My pipeline leverages a ton of custom-built python scripts that I stored in the bin directory. Most python scripts are only used by one process. Ideally, one would expect that if I made a small change to python script A that is called by the script block in nextflow process a, and then resumed a pipeline, that only process a would rerun, as well as processes downstream of process a. But instead, the entire pipeline reruns because a change to a file in bin apparently changes the entire run’s cache fingerprint and invalidates all cached processes. This is a bit silly. Are there any workarounds or plans to account for python usage in cache building in future versions?
in my experience, a small change of the python script indeed only rerun its process and related process during resume.
could you share a toy example to reproduce your observed behavior?
In addition, you can also separate binaries using module binaries
Maybe the problem is that you’re not passing the script paths as path in the input block? Instead, you’re simply using the “python ${params.script1}” substitution?