Variables declared in function seem to get overridden when called multiple times in parallel

Ask for help
1

I have a function to extract the filename prefix from an absolute path.

def string_prefix(String file_path, String where_from, String sep = '.') {
    
    f = new File(file_path)
    full_filename = f.getName()
    print file_path + " " + where_from + " " + full_filename
    last_dot = full_filename.lastIndexOf(sep)
    if (last_dot > 0) {
        prefix = full_filename.substring(0, last_dot)
    }
    else {
        prefix = full_filename
    }
    
    return prefix
}

This is called several times in my workflow to extract a sample ID prefix from a non-fastq path. Here’s an example:

        channel
            .fromPath(params.sam)
            .map{tuple(string_prefix(it.toString(), "convert_sam_to_bam"), it)}
            .combine(genrf_ch)
            .set{sam_ch}

I’m finding that although file_path and where_from are consistently correct, the full_filename variable does not consistently get named after the prefix in file_path. Instead, it seems to randomly be set to the prefix for any one of the file_path values submitted from the various calls to string_prefix that occur during the workflow.

Here’s an example. In each line, the final path should have the same prefix as the absolute path that precedes it. Instead, the first one is set to the same value as the third one. The mix-up is different with each run.

/home/benjamin/Documents/hich/sambam/KO_br1_tr2.sam convert_sam_to_bam KO_br1_tr1.bam
/home/benjamin/Documents/hich/pairs/test.pairs.lz4 parsing_wf test.pairs.lz4
/home/benjamin/Documents/hich/sambam/KO_br1_tr1.bam start_as_bam KO_br1_tr1.bam

It seems like somehow, the values of variables declared in the string_prefix function are getting overwritten by the values of the same variables in other calls to string_prefix. I assume this is somehow due to a behavior of Nextflow, but I’m not experienced enough with either Groovy or Nextflow to figure it out. Any help is appreciated!

2

Try declaring these variables as local variables by setting them with def. Read more about it here.

3

That solve the problem, thank you so much! And thanks for the link as well.

1 Like