Correct use of containerOptions with multiple options

Ask for help
,
1

Hello!
Just wondering, what is the correct way to describe containerOptions in a process block. When I try the following (as recommended by the nvidia docker image):

containerOptions "--gpus all --ipc=host --shm-size=1g --ulimit 'memlock=-1' --ulimit 'stack=67108864'"

with or without the single quotes, it fails with a cryptic error message :

DEBUG nextflow.Session - Session aborted -- Cause: For input string: "true"

The only way I could get my process to work was by removing most of the recommended flags:

containerOptions "--gpus all"

Using Nextflow 23.10.1 on AWS Batch

Thank you!

2

Hi @sunitj.

I believe this issue lies with the --shm-size=1g option. If I’m not mistaken, AWS Batch handles the shm size differently than docker. You must provide Batch with the value in megabytes. In your case, that would mean setting --shm-size=1024. Could you please try that?

3

Thank you @mribeirodantas! Unfortunately, it still failed with the error:

May-28 19:03:37.009 [AWSBatch-executor-36] DEBUG nextflow.Session - Session aborted -- Cause: For input string: "true"

Attaching my diff:

image
image927×136 21.2 KB

Could it be that = are handled differently as well?

4

Now that you mention it, I think it’s indeed without the = sign: --shm-size 1024

Could you please try it this way?

5

Is there actual documentation for this? I can find a number of discussions on github, see #2471,#2282, and was marked as solved as of 22.04.0 release.
Based on #2471, it seems like it should be of the form:

containerOptions '--env MYVAR2=foo2 --env MYVAR3=foo3 --memory-swap 240000 --memory-swappiness 20 --shm-size 16000000 --ulimit nofile=1280:2560 --ulimit nproc=16:32 --privileged'

but there is lots of discussions, updates and changes, so I’m still unsure I have it right.

For my next test i’ll try:

containerOptions '--gpus all --ipc host --shm-size 1024 --ulimit memlock=-1 --ulimit stack=67108864'

and pray to the AWS + nextflow gods. :pray:

6

Instances failed to start.