Reading an input file from a public-only S3 bucket

Ask for help
,
1

I have a Nextflow process that reads in a file from a public S3 bucket using Fusion. As of recently, this returns the following warning upon workflow startup and then fails:

WARN: Unable to get file attributes file: s3://genome-idx/kraken/k2_standard_20240605.tar.gz -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WJEAS9T5FC7E812D; S3 Extended Request ID: PPHK5YeVhoOsksK8VBOIY2sooCQWrlo+reCf/YEqzJ87NIHgKG8FxFVTR/ysVZEb5ehs3dj1bw0=; Proxy: null), S3 Extended Request ID: PPHK5YeVhoOsksK8VBOIY2sooCQWrlo+reCf/YEqzJ87NIHgKG8FxFVTR/ysVZEb5ehs3dj1bw0=

I think this is because the bucket now requires the --no-sign-request option to access using the AWS CLI (i.e. won’t accept signed requests). I could fix it by specifying aws.client.anonymous in my Nextflow config file, but that would break the rest of my workflow which relies on access to Batch and private S3 buckets.

I can’t figure out how to make Nextflow make an unsigned request just for this file; is there a way?

2

Do you get this message from .nextflow.log logs? If the message is there, then it’s not a Fusion issue, it’s a more general problem.

Try to create a mini-pipeline to reproduce it. Then, we can check if there is a solution or open a bug issue at the Nextflow repository.