# Failed to Publish File WARNs for Pipeline That Produces Many Files

**URL:** <https://community.seqera.io/t/failed-to-publish-file-warns-for-pipeline-that-produces-many-files/2181>\
**Category:** Ask for help\
**Tags:** nextflow\
**Created:** [June 10, 2025, 10:06pm UTC](https://community.seqera.io/t/failed-to-publish-file-warns-for-pipeline-that-produces-many-files/2181 "2025-06-10T22:06:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Revant\_Reddy](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/revant_reddy/32/1116_2.png) [@Revant\_Reddy](https://community.seqera.io/u/Revant_Reddy)\
**Post date:** [June 10, 2025, 10:06pm UTC](https://community.seqera.io/t/failed-to-publish-file-warns-for-pipeline-that-produces-many-files/2181/1 "2025-06-10T22:06:54Z")

</div>

Hello,

We get many `"WARN nextflow.processor.PublishDir - Failed to publish file:"` messages in our `.nextflow.log` files. I understand that in the new version of Nextflow we can add the `failOnError` option for the `publishDir` directive [Process reference — Nextflow documentation](https://www.nextflow.io/docs/latest/reference/process.html#publishdir) which will automatically fail the pipeline on these publish failures and you will have to resume the pipeline to get the rest of the files. There are also some options for exponential backoff with jitter with the `workflow.output.retryPolicy.*` [Configuration options — Nextflow documentation](https://nextflow.io/docs/latest/reference/config.html#workflow) set of options but this appears to be an approach to completely replace the `publishDir` directive. **Please keep me honest if you disagree?**

Since we have production pipelines at our company and don’t want to upgrade the nextflow version and replace the `publishDir` directive just yet, **Do you have any thoughts on how best we should tweak our `aws.client` configuration option values in our `nextflow.config` specifically `uploadChunkSize, maxConnections, and uploadMaxThreads` which appear to be used by the nextflow control process where the pipeline is launched from to publish files from the working directory to the output directory?**

Our values today are

uploadChunkSize = ‘100MB’ // The size of a single part in a multipart upload; default  
maxConnections = 5 // The maximum number of allowed open HTTP connections; default:10  
uploadMaxThreads = 5 // The maximum number of threads used for multipart upload; default:10

The Seqera documentation [Nextflow | Seqera Docs](https://docs.seqera.io/platform-cloud/troubleshooting_and_faqs/nextflow) points at below but not sure if this is valid for us (we already use an instance with those specs):

 ![image](https://europe1.discourse-cdn.com/flex013/uploads/seqera/original/2X/6/647a66710121d59c0de538d84a2534d688a62070.jpeg)