Jobs marked as failed by nextflow but slurm exit code 0

Ask for help
,
1

Hi – we’ve been encountering an error with some of our nextflow pipelines (both nf-core and ones we are developing) in which we are getting an error that seems to suggest that one of the jobs failed and was terminated by our hpc job scheduler (we’re on a slurm-based HPC), yet when we go into the work directory it appears to have run successfully (outputs are in the work directory and exit code is 0). I’ve included a snippet of the error message at the end. We see some discussion of this here:

I’d also like to note that we take a look at the execution trace according to slurm the job completed successfully and there is a .exitcode file yet it seems the exit code status is marked null in the execution trace (this is attached as well).
If anyone has suggestions it would be much appreciated.

Thanks,
Asher

======
ERROR ~ Error executing process > ‘SHAHCOMPBIO_BAMBU_NF:BAMBU_NF:BAMBU_MERGE_QUANT (SHAH_H003842_T01_01_TR01_NDR_DEFAULT)’

Caused by:
Process SHAHCOMPBIO_BAMBU_NF:BAMBU_NF:BAMBU_MERGE_QUANT (SHAH_H003842_T01_01_TR01_NDR_DEFAULT) terminated for an unknown reason – Likely it has been terminated by the external system

Command executed:

mkdir -p transcriptome_NDR_DEFAULT
transcript_assembly.R
–rds=SHAH_H003842_T01_01_TR01_R1.sorted.bam
–yieldsize=100000
–ref_genome=GRCh38.primary_assembly.genome.fa
–ref_gtf=extended_annotations.gtf
–out_dir=transcriptome_NDR_DEFAULT
–ncore=1

–discovery=FALSE
cat <<-END_VERSIONS > versions.yml
“SHAHCOMPBIO_BAMBU_NF:BAMBU_NF:BAMBU_MERGE_QUANT”:
r-base: (echo (R --version 2>&1) | sed 's/^.R version //; s/ .//') bambu: (Rscript -e “library(bambu); cat(as.character(packageVersion(‘bambu’)))”)
END_VERSIONS

Command exit status:

Command output:
[1] “processing 1 read classes”

execution_trace_2025-06-03_13-44-19.txt (1.6 KB)

2

Hi Asher, did you guys ever figure this one out? I am working on a SLURM system as well and am having the same issues. My jobs are finishing with all the correct file outputs and all .exitcode files contain 0, yet nextflow is marking them as failed and the pipeline eventually stalls out. Interestingly when I resume, it treats these jobs as correctly finished and cached, and proceeds as normal until it begins to build up “failures” again.