How to deal with optional inputs in a workflow

Ask for help
1

Greetings all - I’m a bit perplexed as to have a bit of functionality working. I will provide a simplified example of my dilemma. Say I have a workflow that may have optional output:

process PRODUCER {
    input:
        val(meta)

    output:
        path('file1.txt'), emit: out1
        path('file2.txt'), optional: true, emit: out2



    script:
    """
    echo "foo" > file1.txt
    # Optionally create file2.txt
    # echo "foo" > file2.txt
    """
}

Now I want to have behavior in the next process tied to the outputs in the PRODUCER process:

process CONSUMER {
    input:
        path f1
        path f2

    debug true

    script:
    """
    cat ${f1}
    if [[ -f "${f2}" ]]; then
        cat ${f2}
    fi
    """
}

This is harder than it seems on the surface and I haven’t been able to find a solution, because the optional output of producer process can’t be optional in the input. I saw a bit of work revolving around having ‘default values’ but I’m unsure of where this ended up based on the git issues. Another approach I can think of is based on manipulating the outputs in a channel? Can someone help or point me in the right direction here? Thank you in advance and apologies if this is a question already posed - was unable to find duplicates.

2

This is not so much a problem of optional inputs, but how to keep two separate input channels synchronized with each other. You might be tempted to call CONSUMER like this:

CONSUMER(PRODUCER.out.out1, PRODUCER.out.out2)

But this is generally very dangerous, as you shouldn’t assume that the elements in channels are in a particular order. What you need to do is join the two output channels based on an unique key and using reminder:true to convert optional output into []. Now you can multiMap the joined channel into two synchronized channels for each input of CONSUMER.

3

I think the big picture idea is following

Leverage nextflow workflow design to conditional handle the 1 or 2 input for CONSUMER, rather than use script inside CONSUMER to conditional handle 1 or 2 inputs.

process CONSUMER {
    input:
        path f1
        path f2, optional: true

    debug true

    script:
    """
    script:
    if (f2) {
        """
        echo "Using f2: $genome_file"
        """
    } else {
        """
        echo "No f2 file provided, skipping"
        """
    }
    """
}

in your producer you can add val in the output tuple, this way your out2 will have a index than if needed can be merged with out1 using join by: [0]

output:
    val(meta), path('file1.txt'), emit: out1
    val(meta), path('file2.txt'), optional: true, emit: out2



script:
"""
echo "foo" > file1.txt
# Optionally create file2.txt
# echo "foo" > file2.txt
"""

}

4

When I use the syntax

path f2, optional: true

I keep getting the following error:

ERROR ~ No such variable: optional

I cannot figure out what I am doing wrong.