Hello all,
We’ve been advised to try mounting an S3 bucket to our EC2 compute in aws batch.
We run into issues when using nextflow to coordinate all this.
It appears that (and this is from some google-fu so bare with) as all our nextflow processes must be containerized, the mount-s3 package cannot be used. Our docker container has a bare-bones kernel, and “fuse” is not available to let the mount-s3 work as fuse is a dependency.
Workarounds are apparently setting specific docker permissions yet we find that when parsing container.options in our config these don’t get picked up on batch.
Essentially wondering if anyone has seen similar or if anyone is mounting s3 volumes in a different way. We would like to mount s3 as it would speed up our compute significantly in this instance + it seems EFS mounting in AWS batch is removed in our community version of nextflow so this is the only workaround we can think of.
many thanks for any help or discussion