tvbo workflow

Sub-tree that turns a Study or Experiment into a self-contained, portable reproducibility kit for an HPC scheduler or a workflow engine.

Sub-commands

Command Purpose
tvbo workflow plan Show the resolved plan (no files written)
tvbo workflow slurm Emit an sbatch kit
tvbo workflow snakemake Emit a Snakemake kit
tvbo workflow nextflow Emit a Nextflow kit
tvbo workflow run Emit a kit and execute/submit it with the engine
tvbo workflow submit Submit an already-emitted kit — no recipe, no re-emit
tvbo workflow finalize Reassemble an array run’s shard outputs into one result
tvbo workflow backends List backends and their ontology-derived capabilities

The three emitters (slurm/snakemake/nextflow) share the same flag surface and just write a kit; run and submit also launch it. The kit layout is documented on its own page: see Workflow kits anatomy.

Common flags

Flag Default Purpose
SPEC (required) Path, CURIE, or DB name (Study or Experiment).
--backend, -b tvboptim Backend whose code is rendered into the kit’s scripts/ and whose vectorize_axes drive the planner.
--experiment (first in study) If SPEC is a Study, pick the experiment(s) by key/name/label/numeric id. A comma-list (2,3,20,30) emits one kit per experiment (and, with workflow run, submits each).
-o, --output out/<study>/<exp>/<engine>/ Output directory for the kit.
--set KEY=VALUE (none, repeatable) Override a workflow spec key. Dotted keys nest, e.g. --set slurm.account=foo. Booleans/numbers are coerced.
--stdout (off) Print the artefact text only and skip kit creation.
--pack (off) Emit only <kit>.tar.gz (the kit dir at top level) and remove the loose dir, ready to scp + tvbo workflow submit. Omit --pack for the loose dir.
--bundle-dataset (off) Copy the fan-out’s per-subject dataset files (sidecar + payload) into spec/dataset/ and rewrite dataset.bids_root to a relative path, so the kit is self-contained — no separate FC upload, no $TVBO_BIDS_ROOT. Selection is driven by the observation’s declared query; scope subjects via dataset.subjects. See Per-subject dataset fan-out.
--bundle-select KEY=VALUE (none, repeatable) Override to add BIDS entities only when a subject directory holds several files matching the query (the bundler otherwise errors with the ambiguous candidates). Not needed when the query already names one file. Implies --bundle-dataset.

tvbo workflow plan

Compute the plan and print it. --json emits machine-readable output:

tvbo workflow plan experiment:JR_MEG --backend jax

Sample output:

study      : default
experiment : JR_MEG_FrequencyGradient_Optimization
backend    : jax (JAX)
engine     : local
container  : (none)
out_dir    : out/{wildcards}

vectorized inside backend (1 job covers all):
  - G            ReducedWongWang.G   kind=parameters         n=11
  - seed         integrator.noise_seed kind=noise_seed       n=8

workflow-fanned axes (engine spawns 1 task per cell):
  (none)

total workflow cells : 1
chunk                : 1  →  1 array task(s)

If the same experiment is planned against --backend tvb (which has empty vectorize_axes), every axis becomes a workflow-fanned axis and produces 88 cells.

tvbo workflow slurm / snakemake / nextflow

Each command writes a complete kit:

tvbo workflow snakemake experiment:JR_MEG \
    --backend jax \
    --set slurm.account=tvbo \
    --set slurm.time=02:00:00 \
    -o ./kits/jr_meg_jax

Resulting layout (see Workflow kits anatomy for details):

kits/jr_meg_jax/
├── Snakefile
├── scripts/
│   └── JR_MEG_FrequencyGradient_Optimization.py
├── spec/
│   └── JR_MEG_FrequencyGradient_Optimization.yaml
├── requirements.txt        # pip deps, from environment.requirements
├── environment.yml         # conda env, from environment.requirements
└── README.md

--stdout prints the artefact only, suitable for pipelines:

tvbo workflow slurm experiment:JR_MEG --backend jax --stdout > run.sbatch

Run and submit

The emitters only write a kit. Two commands also launch it.

tvbo workflow run <engine> SPEC — emit and execute in one step. For Slurm it submits run.sbatch (the array job) and chains finalize.sbatch with an afterok dependency, so the run converges to one reassembled result. Run it from a login node (it calls sbatch).

tvbo workflow run slurm mystudy.yaml --experiment 30 --backend tvboptim
tvbo workflow run slurm mystudy.yaml --experiment 2,3,20,30 --backend tvboptim  # one parallel job each

Note the engine asymmetry: run slurm submits to the scheduler, but run snakemake executes the DAG locally (snakemake --cores all). To run a Snakefile on the cluster, emit it and launch it with snakemake’s own slurm executor/profile.

tvbo workflow submit KIT — submit a kit you already emitted (and, say, copied to the cluster), with no recipe and no re-emit. It infers the engine from the kit’s artefact (run.sbatch → slurm) and runs the same array + finalize chain as run, so you never type sbatch by hand:

tvbo workflow slurm mystudy.yaml --experiment 30 -o kit --pack   # laptop: emit + pack → kit.tar.gz
scp kit.tar.gz hpc:~/runs/                                        # ship one file
ssh hpc 'cd ~/runs && tvbo workflow submit kit.tar.gz'           # extracts, then launches

submit takes a kit directory or a .tar.gz/.tgz/.tar/.zip of one — an archive is extracted next to itself first (and an already-extracted copy is reused, so a re-submit never clobbers an in-progress results/). --array 0 submits a single smoke task; --engine overrides the auto-detection.

tvbo workflow finalize reassembles an array run’s per-shard outputs into one keyed <stem>.h5 (+ .yaml sidecar). run/submit chain it automatically; call it by hand only to re-gather.

Choosing an engine

The three emitters are not interchangeable; they match two different shapes of parallelism. The planner already picks the right shape from the backend, so this is mostly about understanding what you get.

Slurm array fits a sweep the backend vectorizes. Every axis lives inside one JAX batch, so the tasks are identical and differ only by $SLURM_ARRAY_TASK_ID; task i runs Space[i::N]. There is no dependency graph to resolve and no per-cell file to track, so a bare #SBATCH --array is the whole mechanism. This is the tvboptim path for the Koller grid.

Snakemake / Nextflow fit a sweep the backend fans out: one process per cell, each with its own inputs and outputs. A DAG engine earns its overhead here because it tracks per-cell files, re-runs only what is missing, and can chain a per-cell preprocess → simulate → analyse. This is the path when the backend vectorizes nothing (e.g. tvb), or when you want file-level resume.

Use a Slurm array when Use Snakemake / Nextflow when
Backend vectorizes every swept axis (tvboptim, jax) Backend fans axes out (tvb)
Tasks are identical (Space[i::N] slices) Each cell is a distinct process
You want minimal scheduling overhead You want file-level resume or a per-cell DAG

A vectorized sweep can still run under Snakemake if you prefer its executor, but it adds a driver process and per-rule bookkeeping for no gain over the array. The reverse does not hold: --slurm-chunk refuses a fanned sweep, because there is no in-process batch to slice.

Environment on the node

The kit runs tvbo run on the compute node, so that node needs an environment where tvbo and the backend runtime import. Provision it from the emitted environment.yml / requirements.txt (see Environment files below), then have the job activate it. The artefact runs set -euo pipefail and then activates in this order:

workflow.<engine> field Emits For
modules: [python/3.11] module load python/3.11 environment modules
venv: /path/to/venv source /path/to/venv/bin/activate a virtualenv
setup: ["…"] the lines verbatim, in order anything else — conda, sourcing a profile, computed paths

setup is the general hook. A conda env is not a venv, so activate it with setup. Hardcode the conda base — run conda info --base once to get it. A batch job runs a non-login shell with no conda command (it’s an interactive-shell function, and --export=ALL carries env vars, not shell functions), so $(conda info --base) fails on the node. Point source at the real path, in the recipe:

workflow:
  slurm:
    setup:
      - "source /opt/conda/etc/profile.d/conda.sh"   # ← your `conda info --base`
      - "conda activate mystudy_env"

or at submit time, keeping the recipe portable (the value is JSON, so a list works):

tvbo workflow run slurm mystudy.yaml --experiment 30 \
  --set 'slurm.setup=["source /opt/conda/etc/profile.d/conda.sh","conda activate mystudy_env"]'

For a single command, skip the JSON — a bare string is one line:

tvbo workflow run slurm mystudy.yaml --experiment 30 --set slurm.setup="conda activate mystudy_env"

If a job dies at once with conda: command not found or ModuleNotFoundError: tvbo in the logs, the node never got the env — this is where you fix it (and make sure the base path is hardcoded, not $(conda info --base)).

Recipe code needs no PYTHONPATH. Callables/builders referenced by bare module name (callable: {module: my_analysis}) are bundled into the kit’s code/ and put on PYTHONPATH by the artefact; locally, tvbo adds the recipe’s code/ to sys.path on load. No PYTHONPATH=code prefix, on the node or off. See tvbo run, Custom code in a recipe.

Chunking, dependencies, and notifications

Chunk a sweep into array tasks. --set chunk=N shards the sweep across N Slurm array tasks. This applies even when every axis is backend-vectorized (the common tvboptim case): each task runs --slurm-chunk=$SLURM_ARRAY_TASK_ID/N over its share. See HPC patterns, Chunking a sweep into array tasks.

Environment files. Declare deps on the experiment’s schema-native environment.requirements; the kit emits requirements.txt (pip) and environment.yml (conda) from them:

environment:
  name: mystudy_env
  requirements:
    - {name: libigl, package: libigl, version_spec: ">=2.5"}
    - {name: tvboptim, package: tvboptim}

Slurm resources / email. Every slurm.* key becomes an #SBATCH directive, including notifications:

tvbo workflow slurm … \
    --set chunk=40 \
    --set slurm.partition=medium --set slurm.mem=16G \
    --set slurm.mail_type=END,FAIL --set slurm.mail_user=you@example.org

tvbo workflow backends

Prints the ontology-derived capability table (see Backends for the full discussion):

tvbo workflow backends
tvbo workflow backends --json

Sample row:

jax  (JAX)
  tasks         : GradientBasedOptimization, ODEIntegration, ParameterExploration, SDEIntegration
  capabilities  : Autodiff, CodeGeneration, GPUSupport, JITCompilation, VectorizedRNG
  vectorize_axes: initial_conditions, noise_seed, parameters

How the plan is built

  1. Resolve SPEC → Study (+ Experiment) or Experiment.
  2. Merge study.workflow: block with all --set KEY=VALUE overrides (dotted keys nest).
  3. Extract every ExplorationAxis from experiment.explorations. Each axis is classified by its dotted parameter path:
    • noise_seed if the path contains noise_seed or ends with .seed
    • initial_conditions if the path mentions initial_conditions
    • subjects if it mentions subject or sample
    • otherwise parameters
  4. For each axis, place it as vectorized (handled inside the backend) or workflow-fanned (a wildcard / array index) according to the backend’s vectorize_axes set.
  5. Derive the array-task count: for workflow-fanned axes, ceil(workflow_cells / chunk); for a fully backend-vectorized sweep, chunk itself is the number of array shards (each running --slurm-chunk=$i/N). Read environment.requirements into the kit’s requirements.txt / environment.yml.

See also