Are container pulls and conda environment creations counted in "realtime" in a nextflow trace?

[JustinBellavance]

Hello,

I’m benchmarking time taken for certain processes in my pipeline, and I’m wondering if time taken for a container to get pulled from an registry counted in the ‘realtime’ column in the trace, or if it’s only counted within the 'duration’ column? (Or neither?) What about if it’s cached?

I wasn’t able to find my answer in the documentation.

Same question for the creation of conda environments, though I assume they’re in similar situations.

Thanks!

Justin

[Alexander_Nater]

As far as I know, container pulling is done by the runner task before a process task starts, so it shouldn’t count at all towards process task runtime.

[bentsherman (Ben Sherman)]

realtime is the runtime of just your process script, and nothing else around it (container pulling, conda, file staging/unstaging, etc).

duration is the time from submission to completion, so it also includes the time spent in the scheduling queue. As a result it usually isn’t interesting.

complete - start is the entire execution time of the job, including conda creation and file staging/unstaging. In “container native” executors like AWS Batch, where the job is launched inside the container, the container pulling is not included in this measurement. In executors like SLURM where the container is launched inside the job, the container pulling is included.