Nxf_unstage_output(): ls fails when dealing with very large number of files

Ask for help
1

Hello,

A bug I encountered when running my pipeline on a large dataset.

The failing process is dockerized (hence the staging in/out) and exports a bunch of files through globing:

    output:

    path("${params.repref_dir}/*/REF-*.gbk", emit: repref_gbk)

nxf_stage_output() fails at the eval "ls ..." step with error

.command.run: line 24521: /usr/bin/ls: Argument list too long
  • I am a bit surprised as I expect about ~66k files from this command, which do not seem that much.
  • I could stage about ~24k files in without any issue

I could circumvent the error by parallelising this process, so that a smaller number of files is created by each process.

From my understanding there isn’t much that could be done to prevent this issue as it is caused by the underlying OS (maybe add some warning in the documentation about globing when expecting a large number of outputs?), but I though it might still be worth bringing it to your attention.

Cheers,

Theo

2

Please open this as a GitHub issue on the Nextflow repo instead :folded_hands:

3

As far as I remember, this is well known issue. You could just put your output files into a tar or zip archive and emit that.