# Wait and retry if not ready

**URL:** <https://community.seqera.io/t/wait-and-retry-if-not-ready/1349>\
**Category:** Ask for help\
**Tags:** nextflow\
**Created:** [October 30, 2024, 4:49pm UTC](https://community.seqera.io/t/wait-and-retry-if-not-ready/1349 "2024-10-30T16:49:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![arnaudceol](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/arnaudceol/32/913_2.png) [@arnaudceol](https://community.seqera.io/u/arnaudceol)\
**Post date:** [October 30, 2024, 4:49pm UTC](https://community.seqera.io/t/wait-and-retry-if-not-ready/1349/1 "2024-10-30T16:49:16Z")

</div>

Hi all, I’ve a process, which runs on many files, and which needs to be restarted after some time in case of failure. The use case is the following: my data is archived in the cloud: once I start the rehydratation, I have to wait for the data to be ready; if it is not ready I send a failed message and restart after one hour. I’m using `sleep(60*60 *1000 as long) ; return 'retry'` as an errorStrategy. Unfortunately, it looks like it is applied globally to all tasks of the same process, intead of each single task.This can be tested with this simple pipeline:

```auto
#!/usr/bin/env nextflow

def wait_and_retry = {sleep(30 *1000 as long) ; return 'retry'}

process FAIL_AND_WAIT {

    errorStrategy {task.exitStatus <=2 ? wait_and_retry() : 'terminate'}`
    maxRetries 5
   
     input:
     val(hello)

    script:
    """
    echo "${hello},Attempt ${task.attempt}, \$(date)"
    exit ${task.attempt}
    """
}

workflow {
    ch = channel.from( "Task 1", "Task 3", "Task 2" )
    FAIL_AND_WAIT(ch)`
}

```

> The result is the following:`cat work/*/*/.command.out | sort -t ',' -k3`  
> `Task 1,Attempt 1, Wed Oct 30 13:51:51 CET 2024`  
> `Task 2,Attempt 1, Wed Oct 30 13:51:51 CET 2024`  
> `Task 3,Attempt 1, Wed Oct 30 13:51:51 CET 2024`  
> `Task 3,Attempt 2, Wed Oct 30 13:52:21 CET 2024`  
> `Task 2,Attempt 2, Wed Oct 30 13:52:51 CET 2024`  
> `Task 1,Attempt 2, Wed Oct 30 13:53:21 CET 2024`  
> `Task 2,Attempt 3, Wed Oct 30 13:53:51 CET 2024`  
> `Task 3,Attempt 3, Wed Oct 30 13:54:21 CET 2024`

I would expect the second attempt of all task to start at `13:52:21` (after 30 seconds), instead there is a waiting time of 30 seconds between every single task.Anyone can suggest how to solve it? (edited)

---

<div class="post-metadata">

**Author:** ![arnaudceol](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/arnaudceol/32/913_2.png) [@arnaudceol](https://community.seqera.io/u/arnaudceol)\
**Post date:** [October 30, 2024, 7:47pm UTC](https://community.seqera.io/t/wait-and-retry-if-not-ready/1349/2 "2024-10-30T19:47:14Z")

</div>

As suggested by Thomas Danhorn on Slack ([Slack](https://nextflow.slack.com/archives/C02T98A23U7/p1730311918414069?thread_ts=1730293235.434439&cid=C02T98A23U7)):

> the reason is that the `errorStrategy` needs to be resolved by the main Nextflow process _before_ it can decide what to do with a given worker process (restart or not), so the `sleep()` does not happen in the worker process, but the main Nextflow process as it is resolving the `errorStrategy`

I found the following workaround: I define a global variable that I increment before each “wait” to be sure that the waiting time is triggered only once per “retry cycle”:

```auto
def wait_and_retry = { retryIndex=retryIndex+1; println "WAIT " + new Date(); sleep(20 *1000 as long) ; return 'retry'}

retryIndex=0

process FAIL_AND_WAIT {
    errorStrategy { task.attempt > retryIndex ? wait_and_retry() : 'retry'}

```

---

<div class="post-metadata">

**Author:** ![kenibrewer](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/kenibrewer/32/963_2.png) [@kenibrewer](https://community.seqera.io/u/kenibrewer)\
**Post date:** [November 2, 2024, 4:58am UTC](https://community.seqera.io/t/wait-and-retry-if-not-ready/1349/3 "2024-11-02T04:58:48Z")

</div>

Glad you found a solution. Thanks for posting it here in the forum too!

---

<div class="post-metadata">

**Author:** ![system](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/system/32/2402_2.png) [@system](https://community.seqera.io/u/system)\
**Post date:** [November 9, 2024, 4:59am UTC](https://community.seqera.io/t/wait-and-retry-if-not-ready/1349/4 "2024-11-09T04:59:30Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.