DRY Principle in Nextflow: Reusing Output Path Definitions in `output:` and `script:` sections

[ppreshant (Prashant Kalvapalle)]

Reusing Output Paths in Nextflow

I’m new to Nextflow and doing the Hello-Nextflow training just now. I had this question nagging me for the last couple days and thought I’ll use this to say hi to this helpful community

I’m coming from Snakemake, where we can define output paths as variables and reuse them as $output.name. I’m wondering if there’s a similar pattern in Nextflow to avoid repeating the output path definition.

For example, in this process:

process toUpperCase {
    publishDir 'results', mode: 'copy'

    input:
        path in_file
    output:
        path "${in_file}-upper.txt"
    script:
    """
    echo ${in_file} | tr '[:lower:]' '[:upper:]' > '${in_file}-upper.txt'
    """
}

I notice that I’m repeating the pattern ${in_file}-upper.txt in both the output block and the script block. In Snakemake, I could define this as a variable and reuse it. Is there a similar pattern in Nextflow to:

1. Define the output path pattern once
2. Reuse it in both the output and script blocks
3. Potentially make it easier to maintain if the pattern needs to change

I’m looking for best practices to handle this pattern in Nextflow. Any suggestions or examples would be greatly appreciated!

[robsyme]

Hi Prashat

You can absolutely define new variables, for example:

process toUpperCase {
    publishDir 'results', mode: 'copy'
    input: path in_file
    output: path outputWithSuffix
    script:
    outputWithSuffix = "${in_file}-upper.txt"
    """
    echo ${in_file} | tr '[:lower:]' '[:upper:]' > '${outputWithSuffix}'
    """
}

Given the modest size of most processes, it’s debatable whether this change makes an impact on maintainability. If you prefer this approach, it’s certainly available for you.

You can also use glob patterns in the output to give some flexibility without defining new variables:

process toUpperCase {
    publishDir 'results', mode: 'copy'
    input: path in_file
    output: path "*.txt"
    script:
    """
    echo ${in_file} | tr '[:lower:]' '[:upper:]' > '${in_file}-upper.txt'
    """
}

[ppreshant (Prashant Kalvapalle)]

Wonderful. That makes a lot of sense now. Thank you for these two approaches!