-resume not effective with S3 input files?

Ask for help
1

My workflow reads input files either from S3 or local disk. When the input files are local, -resume works as expected, and successful steps are not re-run. When the input files are in S3, repeating the workflow submission with -resume downloads the files from S3 again, and re-runs all steps including those that ran successfully on the prior run.

I’m thinking that the repeated S3 download results in local files that differ in timestamp from those in the previous run, so they are assumed to be different input files. Is there anything I can do here? Thanks!

2

Welcome to the community forum, @Andrew_Crabb

For inconsistent file timestamps, which can invalidate the cache, you can avoid it by using the 'lenient' caching mode, which ignores the last modified timestamp and uses only the file path and size.

process MY_PROCESS_NAME {
  cache 'lenient'
  ...
}