Turn on printing of all executed bash commands and disable deleting temp folder in work folder

Tips & Tricks
1

Sometimes when a Nextflow task fails in mysterious ways, it can be useful to debug the execution of the bash scripts (.command.run and .command.sh) in the task’s work folder extra carefully.

I have found two things that help a lot:

  1. Commenting out the deletion of the temporary run folder, so that its contents can be explored
  2. Turning on printing of all executed bash commands, by adding set -x in the top of both .command.sh and .command.run

This can be done with the following bash function, which is supposed to be executed inside a task’s work folder:

function debugnf() {
    # Inserts `set -x` on the second line of .command.run and .command.sh
    # so that all executed commands are printed during execution.
    sed -i '2s/^/set -x\n/' .command.{run,sh};
    # Comments out all rm commands in .command.run so temp folder is kept
    sed -i 's/rm /#rm /g' .command.run;
}

To use it:

  1. Check for a failing task’s work folder (similar to /tmp/nf/work/a3/0e2ea68c421ced4797e00de9e73155), and change directory into it, e.g. with:
    cd /tmp/nf/work/a3/0e2ea68c421ced4797e00de9e73155
  2. Run debugnf
  3. Now execute either .command.run or .command.sh using bash, e.g:
    bash .command.run

Some more context is provided in this blog post.

1 Like
2

You can also add

trap read DEBUG

to the script to step through the script as it’s executed.

Helpful tip from: bash debugging

You can also set -x on the script directly by doing:

bash -x .command.run

However, putting set -x in the script is useful for example if you need to run it in the workflow execution environment, for example if you submit the task to slurm.

sbatch .command.run
2 Likes
3

Ah, thank you @mahesh.binzerpanchal ! That’s extremely helpful tips, and will immediately go into my toolbox :blush: