# Problems with GPU-enabled wave images

**URL:** <https://community.seqera.io/t/problems-with-gpu-enabled-wave-images/1596>\
**Category:** Ask for help\
**Tags:** wave\
**Created:** [January 9, 2025, 5:08pm UTC](https://community.seqera.io/t/problems-with-gpu-enabled-wave-images/1596 "2025-01-09T17:08:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nico\_Trummer](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/nico_trummer/32/978_2.png) [@Nico\_Trummer](https://community.seqera.io/u/Nico_Trummer)\
**Post date:** [January 9, 2025, 5:08pm UTC](https://community.seqera.io/t/problems-with-gpu-enabled-wave-images/1596/1 "2025-01-09T17:08:42Z")

</div>

Hello,

I am trying to build a GPU-enabled image using wave. I encountered several problems:

# Seqera containers (web UI) does not allow selecting a specific build

jaxlib has for each version multiple builds: Some with CUDA, some without. In order to make sure that a GPU (CUDA) version is enabled, I need to specify a specific build. This is not possible with the web tool. Thus I sticked with the wave CLI, which is not a big problem but I just wanted to mention. The yml file used with the CLI is shown [here](https://github.com/nf-core/scdownstream/blob/e8b350099fa282004c65bb496fbde7eb4fb86ff9/modules/local/scvitools/gpu_container/conda.yml).

# Installing cuda requires the virtual \_\_cuda package

Conda virtual packages are explained [here](https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-virtual.html). As the wave servers do not have CUDA installed, it is necessary to set the `CONDA_OVERRIDE_CUDA` environment variable. This is neither possible with Seqera Containers, not in combination with `--conda-file` in the wave CLI (except if one has a `--conda-base-image` that already sets the environment variable.  
Thus, I created a [dockerfile](https://github.com/nf-core/scdownstream/blob/e8b350099fa282004c65bb496fbde7eb4fb86ff9/modules/local/scvitools/gpu_container/Dockerfile) that does this. Not straightforward to figure out, but not a huge problem.

# Error 137 (out of memory)

You can see my final setup [here](https://github.com/nf-core/scdownstream/tree/tool-versions/modules/local/scvitools/gpu_container). The wave build log can be accessed [here](https://wave.seqera.io/view/builds/bd-8410848e83d3b19e_1). Apparently it fails due to an 137 (OOM) error. It is not uncommon for CUDA dependencies to be rather large (several gigabytes).

# Wrapping up

Would love to see them fixed, but I understand if they are not top priority. But GPU computing is on a rise, so maybe these problems will become more prominent with time.

---

<div class="post-metadata">

**Author:** ![paolo](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/paolo/32/72_2.png) [@paolo](https://community.seqera.io/u/paolo)\
**Post date:** [January 10, 2025, 1:39pm UTC](https://community.seqera.io/t/problems-with-gpu-enabled-wave-images/1596/2 "2025-01-10T13:39:44Z")

</div>

> [@Nico\_Trummer](#):
>
> Apparently it fails due to an 137 (OOM) error

I think it’s not an Out out memory issue. More likely it was killed because it reached the 15 minutes build timeout.

If you authenticate your request adding your tower (aka Platform) token, you will gain 25 build minutes timeout

---

<div class="post-metadata">

**Author:** ![Nico\_Trummer](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/nico_trummer/32/978_2.png) [@Nico\_Trummer](https://community.seqera.io/u/Nico_Trummer)\
**Post date:** [January 11, 2025, 2:41pm UTC](https://community.seqera.io/t/problems-with-gpu-enabled-wave-images/1596/3 "2025-01-11T14:41:27Z")

</div>

Hey,  
this is now the build log with a platform token provided: [log](https://wave.seqera.io/view/builds/bd-594b9b9db2065574_1)

It appears to fail in the same way, again after around 15:30, so apparently the additional time limit did not help 🤔