Using fusion with AWS batch and s3 buckets across multiple accounts

Ask for help
,
1

Hi there,

I’ve been trying to configure an AWS batch compute env in seqera cloud that allows me to use fusion v2 and access s3 buckets in multiple aws accounts. This works fine in a compute environment without fusion enabled (where I just add a policy for s3 bucket access to my instance and seqera roles), but gives an almost immediate failed error due to s3 permissions in my fusion-enabled account:

N E X T F L O W  ~  version 23.10.1
Pulling xxx/xxx ...
 downloaded from https://github.com/xxx/xxx.git
Launching `https://github.com/xxx/xxx` [trusting_feynman] DSL2 - revision: xxx [develop]
Downloading plugin nf-validation@1.1.3
ERROR ~ Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; ...

This compute env works totally fine when it’s only accessing files within s3 buckets associated with the same account.

Reading the fusion docs, it’s not at all clear to me what permissions I may be missing.

Any help would be appreciated!

2

It seems a bit odd that this is a Fusion-related issue because, at this point of Nextflow execution, no special Fusion logic requires extra permissions.

Can you share more details on the exception you get in the .nextflow.log file?

3

Totally agree -

A new piece of information, doesn’t solve my problem, but suggests this isn’t a fusion issue:

In all of my testing, I’ve been specifying a params.outdir in the other account’s (not the compute’s) s3 - when I change this to a bucket path of the same account as compute env, the workflow launches and it looks like a process will run until it needs to write nextflow out, log and tower reports, at which point it hits another permission denied s3 error.

Happy to re-tag this issue - but, any ideas?