Turn on printing of all executed bash commands and disable deleting temp folder in work folder

Sometimes when a Nextflow task fails in mysterious ways, it can be useful to debug the execution of the bash scripts (.command.run and .command.sh) in the task’s work folder extra carefully.

I have found two things that help a lot:

  1. Commenting out the deletion of the temporary run folder, so that its contents can be explored
  2. Turning on printing of all executed bash commands, by adding set -x in the top of both .command.sh and .command.run

This can be done with the following bash function, which is supposed to be executed inside a task’s work folder:

function debugnf() {
    # Inserts `set -x` on the second line of .command.run and .command.sh
    # so that all executed commands are printed during execution.
    sed -i '2s/^/set -x\n/' .command.{run,sh};
    # Comments out all rm commands in .command.run so temp folder is kept
    sed -i 's/rm /#rm /g' .command.run;

To use it:

  1. Check for a failing task’s work folder (similar to /tmp/nf/work/a3/0e2ea68c421ced4797e00de9e73155), and change directory into it, e.g. with:
    cd /tmp/nf/work/a3/0e2ea68c421ced4797e00de9e73155
  2. Run debugnf
  3. Now execute either .command.run or .command.sh using bash, e.g:
    bash .command.run

Some more context is provided in this blog post.

1 Like

You can also add

trap read DEBUG

to the script to step through the script as it’s executed.

Helpful tip from: bash debugging

You can also set -x on the script directly by doing:

bash -x .command.run

However, putting set -x in the script is useful for example if you need to run it in the workflow execution environment, for example if you submit the task to slurm.

sbatch .command.run

Ah, thank you @mahesh.binzerpanchal ! That’s extremely helpful tips, and will immediately go into my toolbox :blush: