Problem with disk space when running on nf tower

Ask for help
, , ,
1

This problem relates to an nf-core pipeline called scrnaseq that processes single cell RNA seq data. I am hitting a wall with running the scrnaseq pipeline where the locally created /tmp folder is running out of disk space, causing the > pipeline on nf-tower to crash:

2024-08-05 10:56:24 [runtime] (join_complete) ID.SRR7276474.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_REPORTER.CLOUPE_PREPROCESS
2024-08-05 10:56:24 [runtime] (ready) ID.SRR7276474.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_REPORTER.CHOOSE_CLOUPE
2024-08-05 10:56:24 [runtime] (run:local) ID.SRR7276474.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_REPORTER.CHOOSE_CLOUPE.fork0.chnk0.main
2024-08-05 10:56:24 [runtime] (chunks_complete) ID.SRR7276474.SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_REPORTER.CHOOSE_CLOUPE
2024-08-05 10:59:40 [runtime] Pipestance directory out of disk space.
/tmp/nxf.oNCR6caVZb/SRR7276474/SC_RNA_COUNTER_CS has only 2128kB remaining space available.
To ignore this error, set MRO_DISK_SPACE_CHECK=disable in your environment.
2024-08-05 10:59:43 [runtime] Pipestance directory out of disk space.
/tmp/nxf.oNCR6caVZb/SRR7276474/SC_RNA_COUNTER_CS has only 6260kB remaining space available.
To ignore this error, set MRO_DISK_SPACE_CHECK=disable in your environment.
2024-08-05 10:59:46 [runtime] Pipestance directory out of disk space.
/tmp/nxf.oNCR6caVZb/SRR7276474/SC_RNA_COUNTER_CS has only 8280kB remaining space available.
To ignore this error, set MRO_DISK_SPACE_CHECK=disable in your environment.
2024-08-05 10:59:49 [runtime] Pipestance directory out of disk space.

The process that ran was: NFCORE_SCRNASEQ:SCRNASEQ:CELLRANGER_ALIGN:CELLRANGER_COUNT.
When launching the nfcore pipeline from tower, the working directory I gave to the pipeline is an s3 path that has more than enough disk space to process the data. I have a few questions here:

  1. Is the /tmp folder created within the workign directory above?
  2. Why is there a necessity to create and use this /tmp folder?
  3. How do I change the source code to avoid this issue, and should I do so?
    Where in the source code is this /tmp folder even created? I see the following the .command.run:

nxf_mktemp() {
local base=${1:-/tmp}
mkdir -p “base" if [[ (uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX
else TMPDIR=”$base" mktemp -d -t nxf.XXXXXXXXXX
fi
}

nxf_main() {
trap on_exit EXIT
trap on_term TERM INT USR2
trap ‘’ USR1
[[ “${NXF_CHDIR:-}” ]] && cd “NXF_CHDIR" NXF_SCRATCH="(set +u; nxf_mktemp $TMPDIR)”
[[ $NXF_DEBUG > 0 ]] && nxf_env
…}
But where in the code is the pipeline prompted to do this!

2

First things first, what compute platform are you using and what are your settings?

It’s likely Cellranger is using the /tmp drive as a cache. As you can see from the code all the pipeline is doing is calling cellranger count, so the problem is likely to be on the Cellranger side: scrnaseq/modules/nf-core/cellranger/count/templates/cellranger_count.py at c8f02f515d014ee2c98b401950e2cf7141a82b42 · nf-core/scrnaseq · GitHub

cellranger count \
    --id ${prefix} \
     --fastqs $fastq_all \
     --transcriptome ${reference.name} \
     --localcores ${task.cpus} \
     --localmem ${task.memory.toGiga()}

Unfortunately this means we don’t have much control of the code or how it’s behaving.

However, most software will respect the environment variable TMP or TMPDIR and use that instead of /tmp. We can set this to ., which means the current working directory which will prevent it using /tmp. This might remove some problems. Try adding this to your Nextflow config and seeing if it helps:

env {
    TMP = "."
    TMPDIR = "."
}

Of course, you might just be running out of space on your machine, which means you will just need a bigger machine!

1 Like
3

I haven’t tried this particular solution but I have tried changing the cellranger_count.py script that runs in the process causing an error:

#!/usr/bin/env python3
"""
Automatically rename staged files for input into cellranger count.

Copyright (c) Gregor Sturm 2023 - MIT License
"""
import os
tmpdir = os.path.join(os.getcwd(), "tmp")
os.environ["TMPDIR"] = tmpdir
os.makedirs(tmpdir, exist_ok=True)

This simply didnt work and changed nothing in where the tmp directory was assigned. I have also tried using the scratch directive in the process and to change the the TMPDIR within the process script block prior to the cellranger_count.py running. Nothing worked.

I will try this solution as well but is there a reason to think this would work over the others?

4

It will also apply to the cellranger compiled software within the process. When you set the environment variable in cellranger_count.py it’s not clear if this is passed to the compiled cellranger software which is executed by subprocess, not in the Python.

As an added benefit, the env variable will apply to all processes.