Dear Support team,
I built a Nextflow pipeline and added publishDir to copy the final results to a local AWS EC2 instance path, and it is working without issues. Now, I would like to publish raw data and results to s3://my-bucket/fastq_standard. However, it is not working and is giving the following error.
ERROR ~ Unexpected error while finalizing task ‘copyFilesToS3 (1)’ - cause: Failed to create publish directory: s3://my-bucket/fastq_standard/test
AWS CLI is installed on the instance, and I have added the AWS access key, secret access key, and region details to the nextflow.config file.
aws {
region = ‘XXXX’ // Change to your AWS region
accessKey = ‘XXXX’
secretKey = ‘XXXX’
}
I have permissions (s3:PutObject, s3:GetObject & s3:ListBucket) to AWS S3, and the Nextflow process copies files when I add it in the script block with aws s3 cp command.
Here is an example workflow that I used to test first before including the publishDir with s3 path in my pipeline.
#!/usr/bin/env nextflow
// Define the process to copy files to S3
process copyFilesToS3 {
// Input: local file path(s) to be copied to S3
input:
path inputFile // Adjust the file pattern as needed
// Output: No output because we're just copying to S3
output:
path "test3/$inputFile"
publishDir 's3://my-bucket/fastq_standard/test', mode: 'copy'
script:
"""
mkdir test3
# Use AWS CLI to copy the file to S3
#aws s3 cp $inputFile s3://my-bucket/fastq_standard/test/
cp $inputFile test3/
"""
}
// Define the workflow execution
workflow {
input = Channel.fromPath(params.inputfile)
// Call the process to copy files from the 'data' directory to S3
copyFilesToS3(input)
}
Please help me to resolve this issue.
Thanks In Advance
Fazulur Rehaman