Hi,
I’m wondering about the platform specificity of the Wave containers. I am developing on a Mac, so I need an arm64 image, but I want to test my pipeline on an HPC cluster, which would need amd64. I am just using PyPI packages that are platform-independent. However, with the build function for the containers, I can just build either arm64 or amd64. How can I make sure that my pipeline pulls the correct container, depending on the platform? Do I have to build both Wave containers and make a flag? What is the best way?
Best
Judith
They will be two different images. You can go now to Seqera Containers, pick a tool, pick amd64, get a container image, pick arm64, get the container image again and you will see the URI is different, e.g. community.wave.seqera.io/library/pip_numpy:114dadf77c696fab and community.wave.seqera.io/library/pip_numpy:626440684dec17be.
Docker supports architecture emulation, but I believe the right way to go is indeed to make a flag so that the pipeline will pick the appropriate container image based on the compute environment platform.
1 Like
So how would I find out the platform with Nextflow? Like what would I have to add here:
container "${ workflow.containerEngine == 'docker' && **???** ? 'community.wave.seqera.io/library/pip_numpy:114dadf77c696fab' : 'community.wave.seqera.io/library/pip_numpy:626440684dec17be' }"
I had thought of literally a flag, a pipeline parameter that you provide your pipeline run with, e.g.
nextflow run nf-core/rnaseq -profile docker, test_full --arm
Then check for params.arm being set inside your Nextflow code. Or --amd.
This is all new and we haven’t thought of best practices yet. nf-core is in the process of adopting Wave/Seqera Containers and we will sure have nice tips in the near future.
It’s worth mentioning that multi-architecture docker container images do exist, which means you’d have a single image and Docker would pick the right architecture automatically, depending on where you’re running your container. I will get more info about this and Wave/Seqera Containers and get back to you here.
1 Like
Multi-architecture docker container images have some drawbacks. Multi-arch increases the container image size, which leads to increased time to pull/push, increased time to init once pulled, and limited runtime version compatibility to OCI v2 compliant runtimes. Based on that, I think it’s unlikely we will support this in Seqera Containers for now.
For now, the best approaches I can think of is using a profile or params to let Nextflow know what container image it should pull.
2 Likes