Java heap space error

Ask for help
1

While running a pipeline, Nextflow told me that

ERROR ~ Error executing process > 'SAMTOOLS_FIXMATE (14870)'

Caused by:
  Java heap space



 -- Check '.nextflow.log' file for details

The log file contains a nice stack trace:

May-19 17:59:32.998 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=SAMTOOLS_FIXMATE (14870); work-dir=/scratch/nf/work/bc/6725b85c39ac01b40916619b1f70f0
  error [org.codehaus.groovy.runtime.InvokerInvocationException]: java.lang.OutOfMemoryError: Java heap space
May-19 18:01:16.154 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'SAMTOOLS_FIXMATE (14870)'

Caused by:
  Java heap space


org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.OutOfMemoryError: Java heap space
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:348)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
        at nextflow.executor.AbstractGridExecutor$_getQueueStatus_closure2.doCall(AbstractGridExecutor.groovy:330)
        at nextflow.executor.AbstractGridExecutor$_getQueueStatus_closure2.call(AbstractGridExecutor.groovy)
        at nextflow.util.Throttle.cache(Throttle.groovy:224)
        at nextflow.util.Throttle.cache(Throttle.groovy:208)
        at nextflow.executor.AbstractGridExecutor.getQueueStatus(AbstractGridExecutor.groovy:329)
        at nextflow.executor.AbstractGridExecutor.checkStartedStatus(AbstractGridExecutor.groovy:375)
        at nextflow.executor.GridTaskHandler.isStarted(GridTaskHandler.groovy:448)
        at nextflow.executor.GridTaskHandler.checkIfRunning(GridTaskHandler.groovy:429)
        at nextflow.processor.TaskPollingMonitor.checkTaskStatus(TaskPollingMonitor.groovy:644)
        at nextflow.processor.TaskPollingMonitor.checkAllTasks(TaskPollingMonitor.groovy:573)
        at nextflow.processor.TaskPollingMonitor.pollLoop(TaskPollingMonitor.groovy:443)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
        at nextflow.processor.TaskPollingMonitor$_start_closure2.doCall(TaskPollingMonitor.groovy:318)
        at nextflow.processor.TaskPollingMonitor$_start_closure2.call(TaskPollingMonitor.groovy)
        at groovy.lang.Closure.run(Closure.java:505)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
May-19 18:01:56.761 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.OutOfMemoryError: Java heap space

Any idea of which could be the cause? This pipeline have been running for some months now without any issue like this :frowning:

2

Have a look here: Optimizing Nextflow for HPC and cloud at scale | Seqera
In particular, the variable NXF_JVM_ARGS.

3

This is an error in your process, Nextflow submits the process which fails because it runs out of memory.

Something has changed to cause you to run out of memory. Perhaps you are using a bigger sample than normal, or the data within the sample is arranged in such a way to cause a memory spike. Perhaps your Java version or software changed.

Whatever the cause, the solution is to increase the memory to the process. You can set the memory directive to give memory to each process individually. If you want, you can add a retry strategy to retry the process with more memory on subsequent attempts.

Given this is a Java error, you may need to set the heap size. You can do this by setting a value based on the task directive.

process SAMTOOLS_FIXMATE {
    memory { 4.GB * task.attempt }
    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 3

    script:
    def avail_mem = (task.memory.mega*0.8).intValue()
    """
    java -Xmx${avail_mem}M your_cli_commands --here
    """
}
4

On second thoughts, I see it might be occurring when Nextflow is polling the grid tasks, not in the process itself. Still, I’ll leave the above answer in case it’s useful for someone.

As Alexander says, you can increase the quantity of memory allocated to Nextflow by setting the environment variable:

export NXF_JVM_ARGS ="-Xms500m -Xmx4g"  # Adjust these values as needed

Note you should be able to resume the pipeline with this setting and it will pick up where it left off.

5

Thanks for your answers. I’m pretty sure it was not a process error. These are usually properly caught and the error strategy implemented restart them with more memory and a higher time limit. I’m almost sure that it is the Nextflow process itself the one that fails. I will try with the memory adjustment via the NXF_JVM_ARGS environment variable.

6

Since Samtools is not a Java program, I’m pretty sure that the issue is the Nextflow runner rather than the process.

1 Like