Workflow kits anatomy

Every successful tvbo workflow {slurm,snakemake,nextflow} invocation writes a self-contained reproducibility kit. This page documents the layout and the contract each file fulfils.

Layout

<out_dir>/
├── <artefact>             # Snakefile / run.sbatch / main.nf
├── scripts/
│   └── <experiment>.<ext> # Frozen backend code (record + direct-run)
├── spec/
│   └── <experiment>.yaml  # Frozen YAML snapshot; the artefact's run target
├── code/                  # Custom callable/builder modules the recipe references
│   └── <module>.py        #   (only when the recipe uses `module: my_code`)
├── requirements.txt       # pip deps, from environment.requirements
├── environment.yml        # conda env, from environment.requirements
└── README.md              # Provenance + how-to-run

The directory is portable: tar it up, copy it to another machine, install tvbo (or just the runtime needed by the backend), and re-run end to end.

Files

Workflow artefact

Filename depends on the engine:

Engine Filename
slurm run.sbatch
snakemake Snakefile
nextflow main.nf

Each artefact:

  • Sets up engine-specific job parameters (Slurm headers, Snakemake rule all, Nextflow channels) from the merged slurm.* / snakemake.* / nextflow.* workflow blocks.
  • Iterates the workflow-fanned axes: one array task / rule instance / process invocation per cell.
  • Runs the kit’s own frozen spec: tvbo run spec/<exp>.yaml --backend=… --slurm-chunk=$SLURM_ARRAY_TASK_ID/N -o …, so each array task simulates its 1/N share of the sweep and the job is self-contained. Any custom callable/builder modules the recipe references (callable: {module: …}, builder: {module: …}) are bundled into code/ and the artefact puts it on PYTHONPATH, so no external recipe or code is needed at run time; only tvbo plus the backend runtime must be provisioned (from the emitted environment.yml). (If the source recipe is preferred over the frozen spec, the fallback form is tvbo run <recipe> --experiment "<label>" --slurm-chunk=….) A Snakemake study kit can additionally run the pre-rendered scripts/<exp> instead of re-generating from the spec — see Code source: frozen vs spec below.
  • Optionally wraps the inner command in singularity exec <image> when --set container=… was passed.

scripts/<experiment>.<ext>

Frozen backend code, produced by experiment.render(format=BACKEND). The extension is taken from the backend’s registered ExportFormat.extension (.py for jax/tvb/tvboptim/pde, .jl for networkdynamics/modelingtoolkit, etc.).

All Python backends generate scripts with an if __name__ == "__main__": block, so they are directly runnable without tvbo:

python scripts/JR_MEG_FrequencyGradient_Optimization.py --help

The block typically:

  1. Parses --spec PATH (defaults to the sibling spec/*.yaml)
  2. Loads the experiment via SimulationExperiment.from_yaml(spec)
  3. Calls experiment.collect_state() and runs the generated kernel
  4. Saves the result to --output DIR

Julia backends emit top-level solve(...) calls and are runnable as:

julia --project=. scripts/JR_MEG_FrequencyGradient_Optimization.jl

spec/<experiment>.yaml

The frozen YAML snapshot of the experiment, produced by experiment.render(format='yaml'), and the run target the artefact points tvbo run at. It round-trips: private/provenance keys (e.g. _source_file) are stripped on export so the spec is portable and reloads cleanly (tvbo run spec/<exp>.yaml). Do not edit by hand unless you intend to change the rerun behaviour.

requirements.txt / environment.yml

Emitted from the experiment’s schema-native environment.requirements (skipped if none are declared). requirements.txt is one pip spec per line (pkg>=x, or a source_url verbatim); environment.yml is a conda env with those under pip:. Provision the run env with conda env create -f environment.yml or pip install -r requirements.txt. Both are rendered from Mako templates (templates/workflow/{requirements.txt,environment.yml}.mako).

README.md

Auto-generated provenance + how-to-run. Sections:

  1. Layout: file inventory.
  2. Run: copy-pasteable command for the engine (sbatch …, snakemake --cores all, nextflow run main.nf).
  3. Reproducibility:
    • backend (key + label)
    • container image (or (none))
    • cell counts: workflow cells, chunk size, array tasks
    • vectorized vs. workflow-fanned axes (with names + ranges)
  4. CLI overrides: every --set KEY=VALUE recorded with its source.

Code source: frozen vs spec

A Snakemake study kit emits both the frozen scripts/<exp> and the spec/<exp>/ for every experiment, so the same kit runs two ways — chosen at emit or submit time:

Mode The rule runs What the node needs
frozen tvbo run spec/<exp>.yaml --rendered scripts/<exp>.<ext> … — executes the pre-rendered code as-is, no codegen the tvbo runtime + backend; the reducer/streaming logic is already baked into the script, so the node’s tvbo needs no matching codegen
spec (default) tvbo run spec/<exp>.yaml … — re-renders the backend code from the spec at run time a tvbo whose codegen matches the emit-time behaviour

Pick the emit-time default baked into each rule (both artefacts are always written, so one kit stays submittable either way):

tvbo workflow snakemake <recipe> --code-source frozen   # default: spec

Override for a single submission:

tvbo workflow submit <kit> --code-source frozen         # sets $TVBO_CODE_SOURCE for this run
# or drive snakemake yourself:
TVBO_CODE_SOURCE=frozen snakemake --profile profile

Under the hood frozen uses tvbo run --rendered <script>: tvbo run keeps all orchestration (subject / shard / output / dataset resolution) but executes the given pre-rendered script instead of re-generating code. frozen and spec are byte-identical for a deterministic experiment — use frozen to pin the exact emitted code (immune to any later codegen drift) or when the node’s tvbo predates a codegen feature the recipe relies on; use spec to re-generate against the node’s tvbo.

Automatic fallback to spec. A rule reverts to spec when no scripts/<exp> exists — e.g. a cross-experiment analysis (an observation whose source: reads another experiment’s result) has no standalone simulation to render, so only its spec is emitted. frozen also cannot honour run-time flags that change codegen (--set integration.*, or --pin on a fan-out axis the backend does not vectorize); use spec for those.

Why a kit (not a single file)?

Earlier prototypes wrote only the workflow artefact (run.sbatch, Snakefile, …). That works when tvbo is reliably installed everywhere the kit runs, but fails for two important reproducibility scenarios:

  1. Long-term archival. Years later, the tvbo version that produced the artefact may not be installable. The frozen scripts/*.py and spec/*.yaml are the durable contract.
  2. Cross-machine / container migration. The kit can be moved to a node that only has the backend’s runtime (e.g. a JAX wheel and Python). The artefact dispatches to the local script, no tvbo needed.

The kit pattern also lets the artefact remain trivially small: the workflow engine only has to call python scripts/<exp>.py rather than know about tvbo’s CLI surface.

Inspecting a kit

# Show the artefact
cat ./kit/Snakefile

# Show the README's reproducibility section
grep -A 99 '## Reproducibility' ./kit/README.md

# Re-run exactly one cell locally
python ./kit/scripts/JR_MEG_FrequencyGradient_Optimization.py \
    --spec ./kit/spec/JR_MEG_FrequencyGradient_Optimization.yaml \
    --output ./local-cell-0

See also