Seqera Tower:
If Azure Batch has an issue, a Run can be stuck in the Running state indefinably.
There is no way to terminate it:
Cancel is not available as an option.
tried solution described in Zombie runs, jobs has been killed in local but state is still running on Seqera - #3 by alanwilter
import requests
headers = {
"Authorization": f"Bearer {vars.get_var('TOWER_ACCESS_TOKEN')}",
"Accept": "application/json",
"Content-Type": "application/json"
}
workflow_id = '4XYno27K7Waw3X'
payload = {
"workflowIds": [
workflow_id
]
}
url ="'https://api.cloud.seqera.io/workflow/delete?workspaceId=77936274224094&force=true'"
response = requests.post(url,json=payload, headers=headers)
Respose:
b’{“message”:“Unable to cancel workflow with id: 4XYno27K7Waw3X - reason: Status code 409, {\r\n \“odata.metadata\”:\“https://nextflow35a68628.westus.batch.azure.com/$metadata#Microsoft.Azure.Batch.Protocol.Entities.Container.errors/@Element\\",\\“code\\”:\\“JobCompleted\\”,\\"message\\”:{\r\n \“lang\”:\“en-US\”,\“value\”:\“The specified job is already in a completed state.\\nRequestId:2e0880a3-d3fd-4e61-b07e-69a8582fd446\\nTime:2025-05-17T02:35:02.7352551Z\”\r\n }\r\n}”}’
To reproduce, start an azure batch run.
While it is running terminate the batch node.
Workflow’s will be left in orphan state and not cancel-able or delete-able.
This is legit issue, because if the node runs out of space it gets stuck in a hung state and must be manually terminated. If you forget to cancel Seqera Jobs first, you are stuck. In my case I have code that limits the concurrent Seqera runs, but jobs are stuck running.
The only way I could fix this was to delete the workspace - luckily, I don’t care about old runs. Others will. This also forced me to rebuild my environment - timeconsuming.