# Slurm killing platform jobs

**URL:** https://community.seqera.io/t/slurm-killing-platform-jobs/607
**Category:** Ask for help
**Tags:** slurm, platform, hpc
**Created:** [March 27, 2024, 8:27pm UTC](https://community.seqera.io/t/slurm-killing-platform-jobs/607 "2024-03-27T20:27:29Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![johnvusich](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/johnvusich/32/529_2.png) [@johnvusich](https://community.seqera.io/u/johnvusich)
#### Post date: [March 27, 2024, 8:27pm UTC](https://community.seqera.io/t/slurm-killing-platform-jobs/607/1 "2024-03-27T20:27:29Z")

</div>

I set up a compute environment to run jobs using our HPC. Our cluster uses Slurm. When I submit the head job, I can see that it is running (using `squeue -u`) and that eventually the cluster jobs start to run. However, after a few minutes, it looks like Slurm kills all of the jobs. The error messages are usually `SIGTERM` or

```txt
terminated for unknown reason -- Likely it has been terminated by the external system.

```

The pipelines run when I submit the head job from the command line, but not when I submit from Seqera Platform. This shouldn’t be a permissions issue, right? Because it is submitting the head job from my user account (via ssh key) and all of the input data is in my home directory. Maybe it is a Slurm configuration problem.

When I set up the compute environment, I set the resources using advanced options → head job options. For example, for `MSU_HPCC_short` compute environment, I have the following:

```auto
--time=4:00 --mem=24GB --cpus-per-task=8

```

Is there a better way to set resources that allows me to use the same compute environment every time? And any idea why my jobs are getting killed by Slurm?

Thanks  
John

---

<div class="post-metadata">

### Author: ![ewels](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/ewels/32/1_2.png) [@ewels](https://community.seqera.io/u/ewels)
#### Post date: [March 28, 2024, 8:43am UTC](https://community.seqera.io/t/slurm-killing-platform-jobs/607/2 "2024-03-28T08:43:49Z")

</div>

> [@johnvusich](#):
>
> I set the resources using advanced options → head job options

These options are only for the slurm submission running the main Nextflow process itself, not the jobs that are then submitted by Nextflow. If Slurm kills any of the compute jobs because they went over then the pipeline run will crash and all jobs will exit, including the head job.

If Slurm is killing the compute jobs because they are requesting too many resources, you need to configure Nextflow itself, not Platform. You can do this by passing a Nextflow config in a few places, including [at launch](https://docs.seqera.io/platform/23.4.0/launch/advanced#nextflow-config-file).

The Nextflow config should then use [`process.cpus`](https://nextflow.io/docs/latest/process.html#cpus), [`memory`](https://nextflow.io/docs/latest/process.html#memory) and [`time`](https://nextflow.io/docs/latest/process.html#time), the same as if you were running Nextflow manually on the command line. If you’re using nf-core pipelines (or pipelines made with the nf-core template), you can use the [max-resources parameters](https://nf-co.re/docs/usage/configuration#max-resources).

You can check that your configuration is definitely getting through by looking in the task work directories that are created. The `.command.run` files should have `#SBATCH` header lines with the Slurm arguments.

Phil

---

<div class="post-metadata">

### Author: ![johnvusich](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/johnvusich/32/529_2.png) [@johnvusich](https://community.seqera.io/u/johnvusich)
#### Post date: [April 8, 2024, 6:37pm UTC](https://community.seqera.io/t/slurm-killing-platform-jobs/607/3 "2024-04-08T18:37:47Z")

</div>

Hi Phil,

We figured out the problem is that Seqera is launching Slurm jobs from our gateway node. Is there a way to add a pre-run script to SSH into a development node to launch jobs from there?

Thanks,  
John

---

<div class="post-metadata">

### Author: ![ewels](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/ewels/32/1_2.png) [@ewels](https://community.seqera.io/u/ewels)
#### Post date: [April 8, 2024, 9:31pm UTC](https://community.seqera.io/t/slurm-killing-platform-jobs/607/4 "2024-04-08T21:31:55Z")

</div>

Ah nice, good find!

Yes, you can run pre- and post-run scripts: [Advanced options | Seqera Docs](https://docs.seqera.io/platform/23.4.0/launch/advanced#pre-and-post-run-scripts)

You may also find it easier to run using the [Tower Agent](https://docs.seqera.io/platform/23.4.0/supported_software/agent) - you can then run this process wherever you like and it’ll reach out to Platform and submit jobs wherever it is running.

Phil
