Is there any comprehensive tutorial on config files/options and how to use them?

Ask for help
,
1

I’m trying to understand config files and profiles and how to set them up and use the different options but its a little confusing because theres just different bits of information and files scattered around. Is there any semicomprehensive tutorial around?

For example I want to set the resource usage of a run and google gives me a snippet of a config file that says

   withLabel:process_single {
        cpus   = { check_max( 1                  , 'cpus'    ) }
        memory = { check_max( 6.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 4.h  * task.attempt, 'time'    ) }
    }

And its not entirely clear how I use this. Do I just type process_single on the command line?

Another thing I wish to do is I want to do some nfcore RNAseq runs locally and some on a remote hpc slurm cluster through a job submission. When I google I come up with another bit of config files that looks like this.

nextflow run nextflow-io/elixir-workshop-21 -r master -profile hpc_slurm -with-docker

/*
* This configuration file is the one used when indicating the Nextflow parameter -profile hpc_slurm
*/

process { 
    // definition of the slurm executor. Run the pipeline in a node able to submit jobs to a HPC via sbatch
    executor="slurm"       

    // resources for default process execution
    cpus='1'
    time='6h'

       // resources for execution of processes / modules with the label "two cpus". This override the default ones.
        withLabel: 'twocpus' {
           queue = "bigcpus"
           cpus='2'
        }   

}

And its not clear to me what is being done here. Is hpc_slurm an inbuilt profile being called in the command line and the text above just modifies it. Is the text above itself ‘hpc_slurm.config’ and its saved as a file in the directory? How is the withLabel term used?

2

Hey @A_AD :smile:

For such questions, the first place to check should be the official documentation. You can define a set of configurations for processes with a specific label through the withLabel process selector (more info in the official docs here). If it was a profile (you can read about it in the official docs here), then you’d need to choose which one(s) you want through the -profile command line argument, but for process selectors this is not needed. You just add the label name (with the label process directive) to processes that you want that configuration (the one you set with withLabel) to apply.

nf-core pipelines follow the best guidelines and make the best out of Nextflow. They can be complex to analyze if you haven’t yet grasped basic concepts of Nextflow. I recommend first reading the official docs and the fundamentals Nextflow training here.