Episode 57: Pipeline chaining, meta pipelines and automation (Part 2 of 2)

In episode 57 (March 2026), Phil is joined by Edmund Miller and Ben Sherman to discuss how recent and upcoming Nextflow language changes can make pipeline chaining and meta pipelines easier.

The episode was split into two parts: this first part surveyed current solutions, listen to episode 56 to check that out.

In this episode we focus on defining workflow “contracts” through clearer inputs (typed params in Nextflow 25.10 and proposed record types, including streamlined sample sheet/record ingestion from CSV/JSON/YAML or queries) and clearer outputs (workflow outputs leaving preview in 25.10, adding an entry-workflow publish section and an output block to publish channels, route files, and serialize channel contents into JSON/CSV/YAML index files).

We discuss reducing direct params access inside modules, using records to pass validated param bundles, generating/leveraging nextflow_schema.json for external tooling like Seqera Platform to validate chains, exploring glue pipelines, command-line piping of Nextflow runs, and how lineage IDs could support more data-centric, automated workflows.

00:00 Welcome

01:18 Nextflow language

02:09 Inputs and outputs

04:14 Streamlining sample sheet parsing

07:47 Placement of params

12:03 Sharing workflows

14:09 Outputs

20:04 Workflow ouputs to chain pipelines

25:56 Publishing channels

32:03 Schema

34:37 Schema + pipeline chaining

37:13 Piping pipelines

41:20 Nextflow Lineage

45:56 Conclusion and wrap-up