Retry Input File Staging on Network Failure

Ask for help
1

Using the nf-test framework, I am accessing remote files for inputs to the tests. I have recently begun to encounter errors related to file access / download (e.g., ERROR ~ No such file or directory: https://raw.githubusercontent.com/...). I have tested extensively with these before, and have had no issues. I believe this is related to the degree of activity on my HPC, and it would be nice to simply retry these type of failures with back-off.

Is there a way to have Nextflow retry input file staging?

I know that you can retry processes, but this is not staging in a process, it is at the workflow level:

Here is the error specifics:

Check script '<redacted>/workflows/wgs.nf' at line: 80 or see '.nf-test/tests/85f69340fae4c75830dd11be32039399/meta/nextflow.log' file for more details
2

Nextflow currently does not support automatic retry for file staging. One thing you can do though is to handle that in a process. Something like:

process DOWNLOAD_FILES {
    errorStrategy 'retry'
    maxRetries 3
    stageInMode 'copy'
    
    input:
    val url
    
    output:
    path "downloaded_file"
    
    script:
    """
    wget --retry-connrefused --waitretry=5 --read-timeout=20 --timeout=15 -t 3 "${url}" -O downloaded_file
    """
}