Cleanup and publishing performance

Ask for help
,
1

I’m interested in assessing the performance of the publishing and cleanup tasks.

Is there a way to know when the Nextflow process have finished the publishing of all the required outputs? I assume that it starts as soon as the outputs are available.
Is there a way to know when Nextflow starts and ends the cleanup of the work folder once all tasks are finished?

I want to see how the number of threads in the head process affects performance in our typical workloads. According to Optimizing Nextflow for HPC and cloud a scale the head node uses as many threads as

the number of available CPUs

How Nextflow get the number of available CPUs? My particular running environment is a SLURM job. Does it honours the number of CPUs requested to SLURM? Or does it get as many CPUs as the machine have? Is there somewhere in the logs where I can see how many CPUs/threads the head node is using? In case it is taking more CPUs than allocated by SLURM, is there some flag/command line parameter/environment variable I can use to limit this number?

Too many questions in a single post :innocent:

2

It looks like you’re going into plugin territory: Plugins — Nextflow documentation

I would start there and see if any existing plugin’s have features you’re interested in: Nexflow Plugin Stats | Nextflow Plugins

e.g.

nf-co2footprint
nf-cachebrowser
nf-datatrail

are might be where I would start looking for answers to your questions.

I think for publishing take a look at the trace observers. There’s one there for when a file is published: Plugins — Nextflow documentation

3

Ouch! I didn’t wanted to enter the realm of plugins, I’m not looking for new functionality, just information about current one. I was expecting to have all that information in the log files, but I was unable to find it, hence I was asking for it.

I’ll take a look at the plugins list, thanks.

4

I took a look at the plugins in the registry and none seems to offer any of the information I’m looking for :frowning:

5

The latest version of Nextflow lets you make plugin’s a bit more easily. See https://youtu.be/KNPwMMX82EM?si=mFdQI0ABYjVXsFMv&t=457 for a demo. Maybe that helps a bit