Local executor processor utilisation

[icgk523]

Hi everyone

I am trying to run a pipeline that utilises nextflow to parallelise whole insect genome alignments, and my knowledge of how nextflow works is extremely limited.

I have access to a 64 core/128 thread cluster that isn’t setup using a job scheduler, so I would let nextflow default to the local executor in this case.

When starting to run the pipeline on two genomes, nextflow splits this into ~850 jobs, so my question is: as this is running on a local executor, will this process be parallelised across the available 128 threads (sidenote: does nextflow use cores or threads?), or will it wait for each job to be completed before initiating the next? The number of cpus (I’m assuming this is cores/threads?) in the config file is set to 1, but I’m not sure if this is per job, or per nextflow run.

Thanks in advance!

[ewels (Phil Ewels)]

Hi @icgk523,

The local executor does indeed run tasks in parallel. See the docs: Executors — Nextflow 23.10.0 documentation

The processes are parallelised by spawning multiple threads, taking advantage of the multi-core architecture of the CPU.

@icgk523

in the config file is set to 1, but I’m not sure if this is per job, or per nextflow run.

This is per-task (job). See docs: Processes — Nextflow 23.10.0 documentation

I hope this helps!

Phil