Caching doesn't work always || already processed data fails

[complexgenome (S)]

Hi There,

I am using nextflow for a massive pipeline.
There are more than 25 steps in a workflow.
I add and test one new process as needed and as the development happens.

I use following command:

/mnt/data1/software/nextflow run main.nf -params-file input_params.json -resume

At times it picks up from the new process, sometimes it decides to randomly pick from some step with random sample and analyses everything again from the step/process selected.

Secondly, issue is, the pipeline fails with already processed data that was analysed until the newly process/step is added.

Third, How do I know which samples are failing when such thing happens?

This is confusing, weird and exasperating.
How can I have the pipeline resume without any issues?

[mribeirodantas (Marcel Ribeiro-Dantas)]

It’s difficult to debug such situations without a minimal reproducible example.
Fortunately, we have nice resources to better understand and debug resuming pipelines. Check them below:

• Analyzing caching behavior of pipelines

Some old ones:

• Easy provenance reporting
• Troubleshooting Nextflow resume
• Demystifying Nextflow resume