ERROR ~ Unexpected error while finalizing task 'copyFilesToS3 (1)' - cause: Failed to create publish directory: s3://mybucket/fastq_standard/test

Dear Support team,

I built a Nextflow pipeline and added publishDir to copy the final results to a local AWS EC2 instance path, and it is working without issues. Now, I would like to publish raw data and results to s3://my-bucket/fastq_standard. However, it is not working and is giving the following error.

ERROR ~ Unexpected error while finalizing task 'copyFilesToS3 (1)' - cause: Failed to create publish directory: s3://my-bucket/fastq_standard/test

AWS CLI is installed on the instance, and I have added the AWS access key, secret access key, and region details to the nextflow.config file.

aws {
   region = 'XXXX'  // Change to your AWS region
   accessKey = 'XXXX'
   secretKey = 'XXXX'
}

I have permissions (s3:PutObject, s3:GetObject & s3:ListBucket) to AWS S3, and the Nextflow process copies files when I add it in the script block with aws s3 cp command.

Here is an example workflow that I used to test first before including the publishDir with s3 path in my pipeline.

#!/usr/bin/env nextflow

// Define the process to copy files to S3
process copyFilesToS3 {

    // Input: local file path(s) to be copied to S3
    input:
    path inputFile // Adjust the file pattern as needed

    // Output: No output because we're just copying to S3
    output:
    path "test3/$inputFile"
    publishDir 's3://my-bucket/fastq_standard/test', mode: 'copy'

    script:
    """
    mkdir test3
    # Use AWS CLI to copy the file to S3
    #aws s3 cp $inputFile s3://my-bucket/fastq_standard/test/
    cp $inputFile test3/
    """
}

// Define the workflow execution
workflow {

    input = Channel.fromPath(params.inputfile)
    // Call the process to copy files from the 'data' directory to S3
    copyFilesToS3(input)
}

Please help me to resolve this issue.

Thanks In Advance
Fazulur Rehaman