# AWS Batch: Spot Instance - How does Nextflow log an interruption?

**URL:** https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676
**Category:** Ask for help
**Tags:** nextflow, aws
**Created:** [January 29, 2025, 7:44pm UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676 "2025-01-29T19:44:43Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Revant\_Reddy](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/revant_reddy/32/1116_2.png) [@Revant\_Reddy](https://community.seqera.io/u/Revant_Reddy)
#### Post date: [January 29, 2025, 7:44pm UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676/1 "2025-01-29T19:44:43Z")

</div>

General question. I am trying to make the decision about which processes in my nextflow pipeline should utilize spot instances. As a part of this I would like to better understand how Nextflow logs spot instance interruptions? Is there specific information I can look for in the .command.log or trace file that would clue be into these interruptions? Any Nextflow documentation link to back up your thoughts would be appreciated. Sincerely appreciate your time.

---

<div class="post-metadata">

### Author: ![mribeirodantas](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/mribeirodantas/32/235_2.png) [@mribeirodantas](https://community.seqera.io/u/mribeirodantas)
#### Post date: [January 29, 2025, 8:53pm UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676/2 "2025-01-29T20:53:03Z")

</div>

> Iam trying to make the decision about which processes in my nextflow pipeline should utilize spot instances.

In theory, one should pick **fault-tolerant workloads** that can handle interruptions gracefully, or that are not expensive to fail. If you have an extremely long task that will cost a lot of money for each sample, using spot machines may not be the best approach. However, with [Fusion Snapshots](https://seqera.io/blog/fusion-snapshotting/), one shouldn’t worry about this, as when the task is retried, [Fusion](https://seqera.io/fusion/) will make it start where it stopped before the interruption.

> how Nextflow logs spot instance interruptions? Is there specific information I can look for in the .command.log or trace file that would clue be into these interruptions?

The cloud provider is the one killing the task, so we rely on them to give us enough information to identify the task was finished due to the spot machine being reclaimed. For AWS Batch, there is a Nextflow configuration that sets the number of times Nextflow can retry a task run in a spot machine that was reclaimed, which is `aws.batch.maxSpotAttempts`. You can read more about it [here](https://www.nextflow.io/docs/latest/reference/config.html).

Be aware this is different from other cloud providers. For Google Cloud Batch, for example, you have to turn on spot machines in your Nextflow configuration and watch [for a specifi exit status to retry](https://www.nextflow.io/docs/latest/google.html#spot-instances).

---

<div class="post-metadata">

### Author: ![deepank\_korandla](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/deepank_korandla/32/1418_2.png) [@deepank\_korandla](https://community.seqera.io/u/deepank_korandla)
#### Post date: [February 3, 2025, 4:58pm UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676/3 "2025-02-03T16:58:59Z")

</div>

Following up on Revant’s questions about spot instances & Nextflow, I have a few questions about how `aws.batch.maxSpotAttempts` works.

If I

1. set `aws.batch.maxSpotAttempts` to 0,
2. launch a Nextflow pipeline run with AWS Batch, and then
3. manually initiate a Spot Instance interruption ([AWS documentation link](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/initiate-a-spot-instance-interruption.html)) while a Nextflow task is running on a Spot Instance worker,

what should I expect to see happen? Should the Nextflow task fail and require resubmission? Do those sorts of manual spot instance interruptions fall under “Spot Attempts” for Nextflow and therefore trigger use of `aws.batch.maxSpotAttempts`? What information would be captured in the Nextflow log files?

Thanks in advance! - Deepank

---

<div class="post-metadata">

### Author: ![mribeirodantas](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/mribeirodantas/32/235_2.png) [@mribeirodantas](https://community.seqera.io/u/mribeirodantas)
#### Post date: [February 4, 2025, 2:40am UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676/4 "2025-02-04T02:40:38Z")

</div>

Please do not ask new questions in other questions. I posted your reply as a new question authored by you [here](https://community.seqera.io/t/how-does-aws-batch-maxspotattempts-work/1697). I will close this topic now.

---

<div class="post-metadata">

### Author: ![mribeirodantas](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/mribeirodantas/32/235_2.png) [@mribeirodantas](https://community.seqera.io/u/mribeirodantas)
#### Post date: [February 4, 2025, 2:40am UTC](https://community.seqera.io/t/aws-batch-spot-instance-how-does-nextflow-log-an-interruption/1676/5 "2025-02-04T02:40:41Z")

</div>


