Task submitter is stalling (does not submit new jobs) with many tasks in the submission queue

Ask for help
1

Hello!

I am developing a Nextflow pipeline with a lot of tasks that need to be submitted. When I run the pipeline with a -queue-size=500, it runs perfectly well for a while, but it eventually stalls and does not submit any new tasks. There is no error, and all the tasks that were submitted will complete successfully, but all new tasks remain hung, and I have to resume the pipeline for it to start again.

When I look at the nextflow.log it shows many tasks without job IDs stuck in the task submission queue:

Apr-29 08:14:06.707 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor sge > tasks in the submission queue: 47832 -- tasks to be submitted are shown below
~> TaskHandler[jobId: null; id: 20616; name: render_qc_bundle_images (3470); status: NEW; exit: -; error: -; workDir: ...work/a3/db5ecbf42a895b2a879fad49df0394 started: -; exited: -; ]
...

Here is my config information for the executor:

executor {
    queueGlobalStatus = true
    submitRateLimit = '20 / 1sec' 
    pollInterval = '1 min'
}

I have tried using maxForks to limit the submission queue, but the submission queue does not respond to maxForks or -queue-size.

Has anyone seen this behavior before, or have any suggestions?

Thanks!