I have a Nextflow process that reads in a file from a public S3 bucket using Fusion. As of recently, this returns the following warning upon workflow startup and then fails:
WARN: Unable to get file attributes file: s3://genome-idx/kraken/k2_standard_20240605.tar.gz -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WJEAS9T5FC7E812D; S3 Extended Request ID: PPHK5YeVhoOsksK8VBOIY2sooCQWrlo+reCf/YEqzJ87NIHgKG8FxFVTR/ysVZEb5ehs3dj1bw0=; Proxy: null), S3 Extended Request ID: PPHK5YeVhoOsksK8VBOIY2sooCQWrlo+reCf/YEqzJ87NIHgKG8FxFVTR/ysVZEb5ehs3dj1bw0=
I think this is because the bucket now requires the --no-sign-request
option to access using the AWS CLI (i.e. won’t accept signed requests). I could fix it by specifying aws.client.anonymous
in my Nextflow config file, but that would break the rest of my workflow which relies on access to Batch and private S3 buckets.
I can’t figure out how to make Nextflow make an unsigned request just for this file; is there a way?