I have a pipeline that signals its start and end in a database. In particular, the last job of the pipeline, dependent on all the previous jobs, updates a status in a database. This update is used by downstream processes to know that the calculus have been finished and they can start their own calculus.
The issue is: when the pipeline signals that it is finished, Nextflow can still take a few hours to finish, probably publishing the result files to wherever is required. That means that the downstream tools will fail due to missing or incomplete files.
Is there some way to make some particular job dependent on the effective publication of a specific output(s)?
Have you tried using the workflow.onComplete to update the database instead? Notifications — Nextflow documentation
No, but that would defeat the purpose of doing it this way.
With the current implementation, as soon as a sample is finished it can continue downstream. Waiting for the workflow to complete will cause all the samples to wait for the longest one (despite the fact that the update won’t be run in SLURM inside a process).
My working sets usually contain many small samples (that can be finished in 1h) and a few big ones (that need a couple days). That would cause big delays in the whole process.
publishDir is asynchronous so there’s no way to determine when it’s done for a given process.
It might be better to include a downstream process that checks data into and out of storage e.g. using dvc or rsync or something. Then you know for sure the “publishing” is done.
Doing the result publishing manually is kind of an antipattern in Nextflow. I was expecting some way to wait for an asynchronous result, as most (all?) languages that work with async commands have 