Implementationstooling reviewed 2026-08
15 Workflows and Reproducibility: Implementations
Everything on this page is generated from tools/workflows.json. If something here is wrong or out of date, the fix belongs in that file, not in this prose.
For what a pipeline is and what reproducibility actually requires, see Section 14.1.
15.1 Start here
Two engines dominate bioinformatics, and the choice between them is a choice between the two models in Section 14.3 rather than a ranking. One is dataflow: work moves through channels and runs when its inputs arrive. The other works backwards from the output you asked for. Both give you the dependency graph, the caching and the resumption; neither is the wrong answer.
Dataflow rather than file targets: processes are connected by channels, and the engine schedules whatever is ready. That model fits the fan-out of per-sample work better than a rule-based one, and it is the reason Nextflow scales onto clusters and cloud batch services with a change of configuration rather than a change of pipeline. (Di Tommaso et al. 2017)
The pipeline language is a Groovy DSL, so debugging occasionally drops you into JVM stack traces for a language you did not think you were writing.
Make for science, and the closer of the two to how most people already think: you declare the output you want and a rule that produces it, and the engine works backwards. Rules are Python, so the escape hatch from the workflow language into a real one is a line away rather than a rewrite. (Köster and Rahmann 2012; Mölder et al. 2025)
Working backwards from filenames means the dependency graph lives in your naming scheme. That is elegant while the naming is disciplined and painful once it is not.
15.2 The other engines
These are chosen for compatibility rather than on their merits as engines — because a consortium specified the language, because the pipelines you need are written in it, or because the workflow has to outlive the tool that runs it. That is a legitimate reason and it is usually the only reason.
| Tool | Upstream | Languages | Why / why not |
|---|---|---|---|
| Cromwell | active | scala |
The reference engine for WDL, and the one the Broad’s own pipelines are written against. Choose it when you are running those pipelines or publishing into that ecosystem; the reason is compatibility, not the engine. |
| cwltool | active | py |
The reference implementation of the Common Workflow Language, whose purpose is portability between engines rather than being the fastest engine itself. Worth knowing about when a workflow has to outlive the tool that runs it, which is a real requirement in consortium work. (Crusoe et al. 2022) |
| miniwdl | active | py |
Runs the same WDL without the server: a local runner plus a static checker that catches errors before anything is scheduled. The usual pairing is to develop here and deploy on a heavier engine. |
15.3 What this chapter does not survey
Environments and containers are the two rungs above a workflow engine in Section 14.5, and this chapter deliberately does not rank the tools that provide them.
They are general developer infrastructure, not computational biology. Package managers, virtual environments and container runtimes are shared with every other field that writes software, they are argued about vigorously and they turn over fast — and none of that argument is one this book has any standing to settle. A survey here would be a survey of somebody else’s subject, carrying our review date and our implied authority.
It would also not survive contact with the review cycle. Every entry in this registry promises to be current, and that promise is only affordable where the book’s authors are close enough to the field to notice when it stops being true. For bioinformatics tooling we are. For the packaging ecosystem we are not, and an out-of-date recommendation there is worse than no recommendation, because it looks equally confident.
What is worth taking from Section 14.5 is the shape of the decision rather than a product name: an unpinned list of versions records intent, a resolved environment records a solution, and an image records a filesystem. Each pins strictly more and costs strictly more. Choose the rung, then use whatever your institution and your language community currently use to stand on it.
The one detail specific to this field is that a great deal of bioinformatics software is distributed as compiled binaries rather than as language packages, which is why the tooling used here is often not the tooling used for pure Python or pure R work.
15.4 Language coverage: the chips mean something different here
On the tool pages of this book the language chips report which languages can drive a tool. On this page that question barely applies, and the reason is worth stating rather than leaving the reader to infer it from a lone implementation-language chip.
A workflow engine is not called from your analysis language — it is the thing that calls your analysis, in whatever language that happens to be. Its implementation language tells you what you will be reading in a stack trace and what the pipeline DSL feels like, not what you can drive it from. The engines here are written in Groovy, Python and Scala, and the ones written in Python are noticeably easier to escape from when the workflow language runs out, because the escape lands you somewhere you already know.
NoteWhat this means in practice
Pick the engine whose failure modes you can read. Over a long project you will spend more time debugging the pipeline than writing it, and that debugging happens in the engine’s language, not yours.