Pushing my first 'mulled' container with wave: quay.io and AWS ECR ✅

Ask for help
1

This is a quick description of me understanding how to use the wave-cli, having seen Paolo’s great demo at the Nextflow 2023 Summit.

wave allows building and deployment of containers with any set of dependencies quickly.

  • By default it builds the containers on seqera lab’s infrastructure
  • These cotainers can then be ‘frozen’ and pushed to another OCI compliant container repository such as docker hub, quay.io or a cloud provider-specific registry such as AWS’s ECR

I will build a container that has both bwa and samtools. This process used to be possible via a ‘mulled’ container in the biocontainer project, but this was not an intuitive process.

First build the container on wave.seqera.io

wave --conda-package samtools=1.18 --conda-package bwa=0.7.17
wave.seqera.io/wt/e4e545776943/wave/build:samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3

Test that this has both tools installed

docker run wave.seqera.io/wt/e4e545776943/wave/build:samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3  sh -c "bwa mem & samtools"
docker run wave.seqera.io/wt/e4e545776943/wave/build:samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3  sh -c "bwa mem & samtools"
Unable to find image 'wave.seqera.io/wt/e4e545776943/wave/build:samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3' locally
samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3: Pulling from wt/e4e545776943/wave/build
Digest: sha256:62ac4c7fbde7b7df962b1fd8518d39fca6c3476bff548d5112fddeb3a511c6a0
Status: Downloaded newer image for wave.seqera.io/wt/e4e545776943/wave/build:samtools-1.18_bwa-0.7.17--2647e1e99ae6c2b3

Usage: bwa mem [options] <idxbase> <in1.fq> [in2.fq]

------ ✂️-------


Program: samtools (Tools for alignments in the SAM format)
Version: 1.18 (using htslib 1.18)

Usage:   samtools <command> [options]
------ ✂️-------

Now to push these to the quay.io and private ECR registries. Credentials need to be added to your Seqera platform account

image
image1034×532 29.6 KB

Select container registry and add your username/password details e.g for quay.io

image
image953×787 25.5 KB

e.g for AWS ECR. In this case username and password or the ACCESS_KEY and SECRET_ACCESS_KEY

image
image946×782 27.6 KB

After this the credentials will show up in the account

image
image960×382 17 KB

Now ensure that you have a repo on the account to receive the push

image
image1041×256 38.8 KB

Make sure that a Tower access token that you created previously

image
image958×374 12.8 KB

is set in your environmental variables

export TOWER_ACCESS_TOKEN=eyJ0aWQiOiA..........

Push the container using the --freeze argument

wave --conda-package samtools=1.18 --conda-package bwa=0.7.17 --freeze --build-repo quay.io/antunderwood/bwa_samtools --await
quay.io/antunderwood/bwa_samtools:samtools-1.18_bwa-0.7.17--3ade26a7c4d975c0

An email confirming the push will be sent to you from wave

image
image871×659 25 KB

This URL can now be used to pull via docker or podman

docker pull quay.io/antunderwood/bwa_samtools:samtools-1.18_bwa-0.7.17--3ade26a7c4d975c0

The same should be possible for AWS ECR. Having created a repo on ECR and seen the push instructions on the AWS console

image

the command fails on the CLI

 wave --conda-package samtools=1.18 --conda-package bwa=0.7.17 --freeze --build-repo 23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com/bwa_samtools --await
java.lang.IllegalStateException: Unexpected response for 'https://wave.seqera.io/v2/wt/f9a00c9bac8f/bwa_samtools/manifests/samtools-1.18_bwa-0.7.17--07b34b9cd5a41cbc': [400] {"errors":[{"message":"error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for \"23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com/bwa_samtools:samtools-1.18_bwa-0.7.17--07b34b9cd5a41cbc\": POST https://23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com/v2/bwa_samtools/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized\n\n"}]}
        at io.seqera.wave.cli.Client.awaitImage(Client.java:174)
        at io.seqera.wave.cli.App.run(App.java:408)
        at io.seqera.wave.cli.App.main(App.java:217)

This is unexpected since the same credentials can push via a manual process

docker login -u AWS -p $(aws ecr get-login-password) 23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com
docker tag 16b9da6fc708 23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com/bwa_samtools:samtools-1.18_bwa-0.7.17--3ade26a7c4d975c0
docker push 23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com/bwa_samtools:samtools-1.18_bwa-0.7.17--3ade26a7c4d975c0
The push refers to repository [23362NNNNNNNdkr.ecr.eu-west-1.amazonaws.com/bwa_samtools]
1f7a95fb4812: Pushed
945a8c321344: Pushed
61b7068cf278: Pushed
503e642d5ebf: Pushed
c3957bc87975: Pushed
5f70bf18a086: Pushed
0eb428f50e73: Pushed
65734c4ecb14: Pushed
5ed208d0a888: Pushed
9f170dfecb6d: Pushed
16dea3230ded: Pushed
cb4596cc1454: Pushed
samtools-1.18_bwa-0.7.17--3ade26a7c4d975c0: digest: sha256:3425b1d0f2c34e45d793f6fba540ae08b1fbd4d4918ca808a712e65bb0e0e1e6 size: 3024

UPDATE: AWS ECR push error solved
I had a TOWER_WORKSPACE_ID environmental variable set and there were no docker login credentials in that workspace. Unsetting this ( unset TOWER_WORKSPACE_ID) enabled a push and the happy DONE email from wave

image
image806×461 18 KB

1 Like
2

@paolo @jordeu Just flagging this as a potential bug when pushing to AWS ECR. I have confirmed that the credentials in my personal account can push the ECR URL I used

3

Hi Anthony, logs report a problem with the creds.

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Make to sure to provide the AWS creds to access that registry, not the ones obtained via aws ecr get-login-password command

2 Likes
4

Hi Paolo

The creds I provided in tower were a ACCESS_KEY and SECRET_ACCESS_KEY not one obtained by aws ecr get-login-password

5

I can confirm that when I remove my .aws/credentials file and set the ACCESS_KEY and SECRET_ACCESS_KEY to a new user with just ECR read/write scope that docker login works and therefore a push succeeds

export AWS_ACCESS_KEY_ID=AKIATMZK.........
export AWS_SECRET_ACCESS_KEY=oFJC5nK1Dn..........
export AWS_DEFAULT_REGION=eu-west-1
docker login -u AWS -p $(aws ecr get-login-password)  23362NNNNNNN.dkr.ecr.eu-west-1.amazonaws.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

These same credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) are those provided in tower

6

Logs report you are trying to access (build) the container using the workspace Id 179114439389631. is this the same workspace where the ECR creds were added?

7

Thanks that was the trick. What makes me puzzled is why the quay.io push worked since I don’t have a docker login for that in that workspace

image
image1161×510 21.8 KB

8

I’ve updated the post in case future users run into the same problem

1 Like