# Nextflow config to switching to GPU in nf-core/sarek pipeline

**URL:** <https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320>\
**Category:** Ask for help\
**Tags:** nextflow, nf-core, platform\
**Created:** [August 7, 2025, 3:03pm UTC](https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320 "2025-08-07T15:03:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![William\_Brandler](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/william_brandler/32/1937_2.png) [@William\_Brandler](https://community.seqera.io/u/William_Brandler)\
**Post date:** [August 7, 2025, 3:03pm UTC](https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320/1 "2025-08-07T15:03:20Z")

</div>

I’m testing `nf-core/sarek` deepvariant pipeline on AWS batch and want it to choose GPUs on the “run deepvariant” step, however, the pipeline keeps choosing CPU instances (e.g. r6). I’ve tried a variety of different things, but it always chooses CPU

Here is my nextflow config file, please could you take a look and suggest a solution?

```auto
aws {
  batch {
    maxSpotAttempts = 2
  }
}

docker {
  enabled = true
  // Remove global runOptions, use process-specific instead
}

params {
  deepvariant_container = 'docker.io/google/deepvariant:1.6.1-gpu'
  deepvariant_options = "--use_gpu --num_shards 8"
}

process {
  executor = 'awsbatch'
  maxRetries = 2

  errorStrategy = {
    (!task.exitStatus || task.exitStatus == 143) ? 'retry' : 'terminate'
  }

  // Default CPU/general-purpose queue routing
  queue = {
    task.attempt <= 2
      ? 'TowerForge-xxx-work' // CPU spot queue
      : 'TowerForge-xxx' // CPU on-demand queue
  }

  // GPU-specific queue logic
  withLabel: 'gpu' {
    queue = {
      task.attempt <= 2
        ? 'TowerForge-xxx-work' // GPU spot queue
        : 'TowerForge-xxx' // GPU on-demand queue
    }
  }

  // Configuration for the DeepVariant GPU process
  withName: 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_DEEPVARIANT:DEEPVARIANT_RUNDEEPVARIANT' {
    label = 'gpu'
    container = params.deepvariant_container
    time = '6h'

    beforeScript = '''
      echo "Checking GPU availability..."
      nvidia-smi || { echo "No GPU found!" >&2; exit 1; }
    '''

    containerOptions = '--gpus all -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_VISIBLE_DEVICES=all'
  }
}

```

---

<div class="post-metadata">

**Author:** ![robsyme](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/robsyme/32/342_2.png) [@robsyme](https://community.seqera.io/u/robsyme)\
**Post date:** [August 7, 2025, 4:12pm UTC](https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320/2 "2025-08-07T16:12:54Z")

</div>

Hi William,

Great question! The key issue here is that you need to use Nextflow’s `accelerator` directive to request GPU instances from AWS Batch.

When you add the `accelerator` directive, Nextflow will augment the AWS Batch SubmitJob API call with a GPU resource requirement, which tells AWS Batch to specifically select GPU-enabled instances. Without this directive, AWS Batch doesn’t know you need a GPU and defaults to CPU instances.

Here’s what you need to add to your config:

```groovy
process {  
  withLabel: 'gpu' {
    accelerator = 1
  }
}

```

A few additional tips:

- **Queue sharing** : You can actually use the same AWS Batch queues for both GPU and CPU jobs - just make sure your Batch Compute Environments have both GPU and CPU instance types available. AWS Batch will automatically select the appropriate instance type based on the job’s resource requirements.

- **Container setup** : If you’re using the GPU-optimized AMI, it includes the NVIDIA Container Toolkit which automatically handles GPU driver mounting and sets the `NVIDIA_DRIVER_CAPABILITIES`. This means you can likely remove the `containerOptions` configuration.

- **Debugging** : Your `beforeScript` with `nvidia-smi` is a clever debugging approach - definitely keep that while testing.

So your simplified config for the DeepVariant process would look like:

```groovy
withName: 'DEEPVARIANT_RUNDEEPVARIANT' {
  label = 'gpu'
  accelerator = 1
  container = params.deepvariant_container
  time = '6h'
  
  beforeScript = '''
    echo "Checking GPU availability..."
    nvidia-smi || { echo "No GPU found!" >&2; exit 1; }
  '''
}

```

Let me know if this resolves the issue!

---

<div class="post-metadata">

**Author:** ![emiller](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/emiller/32/79_2.png) [@emiller](https://community.seqera.io/u/emiller)\
**Post date:** [August 8, 2025, 7:47pm UTC](https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320/3 "2025-08-08T19:47:25Z")

</div>

I just wanted to add that it looks like the [accelerator directive isn’t in the module](https://github.com/nf-core/sarek/blob/2d43ad7973393b83f452d45d230407c7c49cf36c/modules/nf-core/parabricks/fq2bam/main.nf) itself or in any of the config, so we’ll need to add that.

I went ahead and opened a [bug report on the Sarek repo](https://github.com/nf-core/sarek/issues/1951)!

---

<div class="post-metadata">

**Author:** ![William\_Brandler](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/william_brandler/32/1937_2.png) [@William\_Brandler](https://community.seqera.io/u/William_Brandler)\
**Post date:** [August 8, 2025, 11:44pm UTC](https://community.seqera.io/t/nextflow-config-to-switching-to-gpu-in-nf-core-sarek-pipeline/2320/4 "2025-08-08T23:44:27Z")

</div>

thanks @robsyme , with your configuration changes

+ an increase in our AWS quota for g5 series GPUs

+ a configuration to the compute environment (CE) to have 200B on the boot up disk for the GPU CE

we were able to get it running