Containers, tls certificates and Seqera plateform

Ask for help
,
1

Hi there,

I am testing a container created through Containers | Seqera with the ncbi-datasets-cli

First container

Simple container, only ncbi-datasets-cli=18.36.0 installed (conda-forge channel), image build details here Container build report

I am encountering a tls certificate issue:

Error: Post "https://api.ncbi.nlm.nih.gov/datasets/v2/taxonomy/taxon_suggest": POST https://api.ncbi.nlm.nih.gov/datasets/v2/taxonomy/taxon_suggest giving up after 11 attempt(s): Post "https://api.ncbi.nlm.nih.gov/datasets/v2/taxonomy/taxon_suggest": tls: failed to verify certificate: x509: certificate signed by unknown authority

Second container

I added the ca-certificates=2026.7.22 package, image build details here Container build report

Same results, tls certificate error.

Third-party container

Using the image available from DockerHub and maintained by the (US) State Public Health Bioinformatics Group staphb/ncbi-datasets - Docker Image

Works fine, no tls error.

Other tests

  • Varying ncbi-datasets-cli version in the environment.yml did not change the result
  • Local conda installation works fine (but I assume this is highly depend on the underlying OS/configuration)

Question

Is there a way to fix this using Seqera’s container service?

Thanks a lot,

Best,

Theo

2

Problem solved (using some AI…)

Root cause

ncbi-datasets are Go static binaries, and look for certificates from a hard-coded path list. conda install ca-certificates writes $CONDA_PREFIX/ssl/cacert.pem, which is on none of Go’s search paths. However, Go uses env overrides SSL_CERT_FILE and SSL_CERT_DIR.

Solution

After exploring Seqera’s container build details and images, ssl certificates installed through ca-certificates are available within the image at /opt/conda/ssl/

Solution used:

  1. create a container with the additional ca-certificates package

  2. tell Nextflow to pass environment variables to the Docker containers, in nextflow.config:

docker.envWhitelist = "SSL_CERT_FILE=/opt/conda/ssl/cacert.pem,SSL_CERT_DIR=/opt/conda/ssl/certs"

3

I am reopening the subject, since the problem reappeared when trying to run my workflow through Seqera.

Not sure why, as Seqera’s plateform resolved configuration does include the envWhitelist docker parameter (of course, I checked that docker is used)

image
image1442×484 22.3 KB

Any help appreciated!

4

So, after using the docker.envWhitelist trick explained above, i added the following to the beginning of my faulty process:

    echo "Printing ENV variables..."
    printenv SSL_CERT_FILE
    printenv SSL_CERT_DIR
    echo "Listing /opt/conda"
    ls -alh /opt/conda/ssl/
    ls -alh /opt/conda/ssl/*
    echo "Done listing"

Locally

The process `.command.out` looks exactly as expected:

Printing ENV variables...
/opt/conda/ssl/cacert.pem
/opt/conda/ssl/certs
Listing /opt/conda
total 292K
drwxr-xr-x  4 root root 4.0K Sep  8 12:02 .
drwxr-xr-x 15 root root 4.0K Sep  8 12:02 ..
-rw-rw-r--  2 root root 235K Jul 22 21:14 cacert.pem
lrwxrwxrwx  1 root root   10 Sep  8 12:02 cert.pem -> cacert.pem
drwxr-xr-x  2 root root 4.0K Sep  8 12:02 certs
-rw-rw-r--  2 root root  412 Aug 25 22:59 ct_log_list.cnf
-rw-rw-r--  2 root root  412 Aug 25 22:59 ct_log_list.cnf.dist
drwxr-xr-x  2 root root 4.0K Sep  8 12:02 misc
-rw-rw-r--  2 root root  13K Aug 25 22:59 openssl.cnf
-rw-rw-r--  2 root root  13K Aug 25 22:59 openssl.cnf.dist
-rw-rw-r-- 2 root root 235K Jul 22 21:14 /opt/conda/ssl/cacert.pem
lrwxrwxrwx 1 root root   10 Sep  8 12:02 /opt/conda/ssl/cert.pem -> cacert.pem
-rw-rw-r-- 2 root root  412 Aug 25 22:59 /opt/conda/ssl/ct_log_list.cnf
-rw-rw-r-- 2 root root  412 Aug 25 22:59 /opt/conda/ssl/ct_log_list.cnf.dist
-rw-rw-r-- 2 root root  13K Aug 25 22:59 /opt/conda/ssl/openssl.cnf
-rw-rw-r-- 2 root root  13K Aug 25 22:59 /opt/conda/ssl/openssl.cnf.dist

/opt/conda/ssl/certs:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep  8 12:02 .
drwxr-xr-x 4 root root 4.0K Sep  8 12:02 ..
-rw-rw-r-- 2 root root    0 Aug 25 22:59 .keep

/opt/conda/ssl/misc:
total 32K
drwxr-xr-x 2 root root 4.0K Sep  8 12:02 .
drwxr-xr-x 4 root root 4.0K Sep  8 12:02 ..
-rwxrwxr-x 2 root root  14K Aug 25 22:59 CA.pl
lrwxrwxrwx 1 root root    8 Sep  8 12:02 tsget -> tsget.pl
-rwxrwxr-x 2 root root 6.6K Aug 25 22:59 tsget.pl
Done listing

On Seqera plateform

The process fails, and the `.command.out` only contains the echo statement output:

Printing ENV variables...

Does this suggests that `docker.envWhitelist` is not actually used on Seqera plateform? Am I missing something?

5

It turns out I misunderstood what env.whiteList actually does.

From the documentation: docker | Seqera Docs

Comma separated list of environment variable names to be included in the container environment.

This GitHub issue helped me understand what this parameter does.

Solution to my problem

Add the environment variables through the env scope

env {
    SSL_CERT_FILE = "/opt/conda/ssl/cacert.pem"
    SSL_CERT_DIR = "/opt/conda/ssl/certs"
    //see https://community.seqera.io/t/containers-and-tls-certificates/2757/2
}

In my case, no need for the env.whiteList parameter which can be removed.

Suggestions

  1. Change the env.whiteList documentation to

Comma separated list of host environment variable names to be exported to the container environment. Host environment variables are not automatically accessible in a containerised task.

  1. (also, while talking about documentation, I could not find any detail of what exactly the docker.fixOwnership parameter does)
  2. Maybe allow process-specific env variables, as requested in this open issue Setting environment variables for labelled process fails · Issue #1187 · nextflow-io/nextflow · GitHub