Setting ephemeral storage in AWS fargate

Ask for help
,
1

Hello!

I’ve been trying to get Nextflow set up with AWS fargate. I am using the nf-core/rnaseq pipeline, and for most samples it is working successfully, but STAR mapping is resulting in errors for maybe 20% of samples.

After much pain, it looks like the ephemeral storage of tasks is set to 50 Gb by default. I had naively assumed that adjusting the ephemeral storage of the “head node” would alter the storage for each process…

Does anyone know of a way in nextflow to alter the default ephemeral storage? Or even better, a way to have this controlled at the process-level? Ideally this would be set to 100+Gb…

Thank you!

Joe

2

I have since tried passing an ephemeralStorage parameter to containerOptions in a nextflow config file, but it looks like this parameter is currently unsupported.

As far as I can see the 50Gb ephemeral storage looks to be hard-coded in nextflow, as I see a line in AwsBatchTaskHandler.groovy that looks to be setting this. (Not that I understand groovy all too well…).

For now I’m assuming setting fargate ephemeral storage is impossible to set, and will have to look to other solutions.