Changing process.executer for specific applications (from SLURM to local)

Ask for help
1

I’m trying to run a tool TOGA, this tool uses SLURM. I’m working with a VM so the cluster management system is not necessary (I believe). With this in mind, I was trying to change the set up for the extract_chain_features_config.nf file to not work with SLUM, for example,

extract_chain_features_config.nf
// SLURM config for chain features extraction jobs
// relatively lightweighted jobs
#process.executor = 'slurm'
process.executor = 'local'
...

However, the tool still uses SLURM, here my question. How can I edit this behavior in the nextflow.config file? Is that possible? If I change, how can I make sure that I do not use the configuration of the extract_chain_features_config.nf ? I know that this is a question related to one specific tool, however, I’m looking for some guidance to change the default behavior with the use of SLURM in Nexftlow configuration files
Any comment is really appreciated!

2

Yes, you should be able to. You can either do it directly in a process, or use the withName: selector in a config to specifically select the process.

See here for an example:

3

Great, thank you for providing the explanation!
Quick question: I assume that each time I use a new tool that relies on Nextflow, a new process is created. So I can find the process related to that tool and change the process.executor for that process only and not the rest. Am I correct?
Or is this is general for all the tools related to Nextflow in my environment, for example?

4

I’m not entirely sure I understand your question.
Nextflow executes scripts in a process that call particular tools. Where those scripts are executed is governed by the process.executor directive. This can be set for each process individually, or all processes as a whole. Let’s do this in a config for ease.

nextflow.config:

process {
    // directives at this level are applied to all process
    executor = 'slurm' // This is for all processes.

    withName: 'MY_PROCESS' {
        // directives at this level override and apply only to this process.
        executor = 'local' // Overrides the more general setting specifically for MY_PROCESS only.
    }
}

This means that you can also set this as default behaviour, but also change it by supplying another config file.

Does that answer the question?

1 Like
5

Yes, it does! Thank you for your comments!