Nextflow SSL error

A few weeks ago my work laptop started having all sorts of crazy problems related to SSL. I started having issues with conda not being able to install or search for packages due to the errors described here;

Eventually I was able to fix conda by adding this line to my ~/.condarc file;

ssl_verify: truststore

So the issue appears to be that our Zscaler VPN is screwing up the SSL that various cli tools are using when they try to query their own package managers with https.

Well I have not had as much luck getting Nextflow to work. I have Nextflow installed via conda as well in a dedicated conda env and when I try to use it, Nextflow is now completely inoperable;

$ nextflow run main.nf
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

 N E X T F L O W   ~  version 25.04.7

Launching `main.nf` [backstabbing_heyrovsky] DSL2 - revision: 5c721dd9f8

Downloading plugin nf-tower@1.11.4-patch1
ERROR ~ Plugin with id nf-tower not found in any repository

 -- Check '.nextflow.log' file for details

Its now impossible to run nextflow at all because every time you start Nextflow it immediately attempts to query for packages and then fails. I cannot roll back to some other older Nextflow that did not have this behavior (not sure there is one) because I require the latest version of Nextflow for feature testing.

I was digging around online and found this blurb in the repo for tw;

Custom SSL certificate authority store

If you are using a Private CA SSL certificate not recognized by the default Java certificate authorities, use a custom cacerts store:

tw -Djavax.net.ssl.trustStore=/absolute/path/to/cacerts info

So since Nextflow is also running on the JVM this sounds like the same fix I would need to apply to Nextflow, except, how??? In the conda example, I just instructed it to use “truststore" and it figured itself out. In this example here, I need to point it to a location on disk; I have no clue where on disk my “trust store” files are located.

Any ideas how to fix this?

System information;

$ nextflow -version

      N E X T F L O W
      version 25.04.7 build 5955
      created 08-09-2025 13:29 UTC (09:29 EDT)
      cite doi:10.1038/nbt.3820
      http://nextflow.io

$ java -version
openjdk version "17.0.10" 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

$ uname -a
Darwin  24.6.0 Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:55 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6031 arm64 arm Darwin

Heya Steven

It looks like your VPN is performing man-in-the-middle interception on all your traffic.

The problem is that Zscaler’s replacement certificates are signed by Zscaler’s CA, which exists in the system trust store but not in application-specific certificate bundles (like curl).

My guess is that this command also fails, right?

curl -fsSL https://www.nextflow.io/releases/latest/version

The ssl_verify: truststore setting tells conda to use the system’s certificate trust store instead of conda’s built-in certificate bundle for SSL verification. I’m assuming that your VPN’s certificates have been installed already (and the working conda likely confirms that this is the case).

What you need to do is also tell curl to use the system certs. Likely something like:

export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt  # Linux
# or
export CURL_CA_BUNDLE=/etc/ssl/cert.pem                   # macOS

Alternatively you could do it via ~/.curlrc, with something like (Linux example):

cacert = /etc/ssl/certs/ca-certificates.crt
1 Like

Thanks Rob. I dug into this more based on your suggestions and discovered that the system curl actually works fine, turns out my conda is installing its own curl and that one is breaking;

$ curl -fsSL https://www.nextflow.io/releases/latest/version

25.04.7

$ which curl
/usr/bin/curl

$ curl --version
curl 8.7.1 (x86_64-apple-darwin24.0) libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.64.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe UnixSockets


$ conda activate nf-core

$ curl -fsSL https://www.nextflow.io/releases/latest/version

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.


$ which curl
/Users/username/miniforge3/envs/nf-core/bin/curl

$ curl --version
curl 8.14.1 (aarch64-apple-darwin20.0.0) libcurl/8.14.1 OpenSSL/3.5.3 (SecureTransport) zlib/1.3.1 zstd/1.5.7 libssh2/1.11.1 nghttp2/1.67.0
Release-Date: 2025-06-04
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

using the ~/.curlrc file does not seem to work with this conda installed curl, neither does CURL_CA_BUNDLE. It appears that I do have /etc/ssl/cert.pem on this system (with a modification timestamp of last month, about the time the issues started), gonna have to figure out why this conda installed curl is not using it

Maybe something like this?:

  conda activate nf-core
  conda env config vars set CURL_CA_BUNDLE=/etc/ssl/cert.pem
  conda env config vars set SSL_CERT_FILE=/etc/ssl/cert.pem

unfortunately that did not work either…

$ conda env config vars set CURL_CA_BUNDLE=/etc/ssl/cert.pem
To make your changes take effect please reactivate your environment

$ conda env config vars set SSL_CERT_FILE=/etc/ssl/cert.pem
To make your changes take effect please reactivate your environment


$ conda deactivate

$ conda activate nf-core
WARNING: overwriting environment variables set in the machine
overwriting variable ['CURL_CA_BUNDLE', 'SSL_CERT_FILE']

$ curl -fsSL https://www.nextflow.io/releases/latest/version

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

$ which curl
/Users/username/miniforge3/envs/nf-core/bin/curl

ok, made some partial progress

as described, the issue seems to stem from the fact that macOS system curl is using the OS Keychain for TLS verification (libcurl/8.7.1 (SecureTransport)), while conda’s curl is not doing this by default (libcurl/8.14.1 OpenSSL/3.5.3 (SecureTransport)) ; this actually appears to be completely independent of the particular .pem file in use for CURL_CA_BUNDLE and SSL_CERT_FILE and curl --cacert /etc/ssl/cert.pem did not work and neither did adding cacert = /etc/ssl/cert.pem to the file ~/.curlrc. In fact, all of these actually broke things even more.

To fix the conda curl issue, this appears to work;

$ conda activate nf-core

$ export CURL_SSL_BACKEND=secure-transport

$ curl -v -fsSL https://www.nextflow.io/releases/latest/version

Note that this worked ONLY after I went back and un-set all references to CURL_CA_BUNDLE and SSL_CERT_FILE and cleared out the ~/.curlrcfile.

Now, I can get Nextflow to start to work, almost…

$ nextflow run main.nf

 N E X T F L O W   ~  version 25.04.7

Launching `main.nf` [golden_wright] DSL2 - revision: 5c721dd9f8

Downloading plugin nf-tower@1.11.4-patch1
ERROR ~ Plugin with id nf-tower not found in any repository

 -- Check '.nextflow.log' file for details

will have to debug this part next… and will have to figure out the best way to permanently set CURL_SSL_BACKEND=secure-transport, I am guessing just add it to my ~/.zshrc

Ok I think I finally got it to work, for reference the failing log messages from .nextflow.log looked like this;

$ cat .nextflow.log
Sep-19 12:54:03.676 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Sep-19 12:54:03.745 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 25.04.7
Sep-19 12:54:03.756 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/username/.nextflow/plugins; core-plugins: nf-amazon@2.15.0-patch1,nf-azure@1.16.0-patch1,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.1,nf-k8s@1.0.0,nf-tower@1.11.4-patch1,nf-wave@1.12.1
Sep-19 12:54:03.768 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sep-19 12:54:03.769 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sep-19 12:54:03.770 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Sep-19 12:54:03.775 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Sep-19 12:54:03.807 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Sep-19 12:54:03.816 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [admiring_brown] DSL2 - revision: 5c721dd9f8
Sep-19 12:54:03.817 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Sep-19 12:54:03.817 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-tower@1.11.4-patch1]
Sep-19 12:54:03.818 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.11.4-patch1
Sep-19 12:54:03.823 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-tower@1.11.4-patch1
Sep-19 12:54:03.956 [main] ERROR o.p.update.DefaultUpdateRepository - PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:589)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1705)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1629)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1161)
	at org.pf4j.update.DefaultUpdateRepository.initPlugins(DefaultUpdateRepository.java:96)
	at org.pf4j.update.DefaultUpdateRepository.getPlugins(DefaultUpdateRepository.java:80)
	at org.pf4j.update.UpdateManager.getPluginsMap(UpdateManager.java:149)
	at org.pf4j.update.UpdateManager.findReleaseForPlugin(UpdateManager.java:319)
	at org.pf4j.update.UpdateManager.downloadPlugin(UpdateManager.java:268)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
	at nextflow.plugin.PluginUpdater$_safeDownloadPlugin_lambda2.doCall(PluginUpdater.groovy:268)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
	at nextflow.plugin.PluginUpdater.safeDownloadPlugin(PluginUpdater.groovy:270)
	at nextflow.plugin.PluginUpdater.download0(PluginUpdater.groovy:240)
	at nextflow.plugin.PluginUpdater.access$0(PluginUpdater.groovy)
	at nextflow.plugin.PluginUpdater$_safeDownload_closure3.doCall(PluginUpdater.groovy:331)
	at nextflow.plugin.PluginUpdater$_safeDownload_closure3.call(PluginUpdater.groovy)
	at nextflow.file.FileMutex.lock(FileMutex.groovy:106)
	at nextflow.plugin.PluginUpdater.safeDownload(PluginUpdater.groovy:331)
	at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:380)
	at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:224)
	at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:171)
	at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:399)
	at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:289)
	at nextflow.plugin.Plugins.load(Plugins.groovy:55)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:340)
	at nextflow.cli.Launcher.run(Launcher.groovy:513)
	at nextflow.cli.Launcher.main(Launcher.groovy:673)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
	... 57 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 62 common frames omitted
Sep-19 12:54:03.957 [main] INFO  org.pf4j.update.UpdateManager - Plugin with id nf-tower does not exist in any repository
Sep-19 12:54:03.957 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Plugin with id nf-tower not found in any repository
	at org.pf4j.update.UpdateManager.findReleaseForPlugin(UpdateManager.java:322)
	at org.pf4j.update.UpdateManager.downloadPlugin(UpdateManager.java:268)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
	at nextflow.plugin.PluginUpdater$_safeDownloadPlugin_lambda2.doCall(PluginUpdater.groovy:268)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
	at nextflow.plugin.PluginUpdater.safeDownloadPlugin(PluginUpdater.groovy:270)
	at nextflow.plugin.PluginUpdater.download0(PluginUpdater.groovy:240)
	at nextflow.plugin.PluginUpdater.access$0(PluginUpdater.groovy)
	at nextflow.plugin.PluginUpdater$_safeDownload_closure3.doCall(PluginUpdater.groovy:331)
	at nextflow.plugin.PluginUpdater$_safeDownload_closure3.call(PluginUpdater.groovy)
	at nextflow.file.FileMutex.lock(FileMutex.groovy:106)
	at nextflow.plugin.PluginUpdater.safeDownload(PluginUpdater.groovy:331)
	at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:380)
	at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:224)
	at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:171)
	at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:399)
	at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:289)
	at nextflow.plugin.Plugins.load(Plugins.groovy:55)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:340)
	at nextflow.cli.Launcher.run(Launcher.groovy:513)
	at nextflow.cli.Launcher.main(Launcher.groovy:673)

as a sanity check, I was able to curl and wget the Nextflow package list like this

wget https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json

With my conda env activated, I tried doing this;

$ export NXF_OPTS="-Djavax.net.ssl.trustStoreType=KeychainStore \
                 -Djavax.net.ssl.keyStoreType=KeychainStore"

$ nextflow run main.nf

 N E X T F L O W   ~  version 25.04.7

Launching `main.nf` [loving_bernard] DSL2 - revision: 5c721dd9f8

Downloading plugin nf-tower@1.11.4-patch1
executor >  local (13)
....

And it works. So what happens is that when you run nextflow run main.nf with these NXF_OPTS it causes the JVM to attempt to access the macOS Keychain and you get a prompt window like this

After entering my local username/password (thankfully our managed company laptops allow us to have sufficient user permissions for this…) Nextflow was able to run successfully. But, importantly, after exiting the terminal and starting a new fresh session, I did NOT have to add those NXF_OPTS again;

$ conda activate nf-core

$ nextflow run main.nf

 N E X T F L O W   ~  version 25.04.7

Launching `main.nf` [lethal_mccarthy] DSL2 - revision: 5c721dd9f8

executor >  local (13)
[16/ef9314] DO_THING (3)  [100%] 4 of 4, ignored: 1 âś”
...

So it seems like after that initial loading of the credentials from the macOS Keychain, whatever details JVM needed to function must be cached somewhere, or at least I guess macOS is allowing the JVM to access the Keychain on subsequent runs. Regardless, its working now. And for reference, I still have this in my ~/.zshrc;

# Need this for Zscaler to work with conda installed curl that gets pulled in with some conda envs
export CURL_SSL_BACKEND=secure-transport

thanks to @robsyme for the help since that put me on the right track to figure this out

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.