Ok I think I finally got it to work, for reference the failing log messages from .nextflow.log looked like this;
$ cat .nextflow.log
Sep-19 12:54:03.676 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Sep-19 12:54:03.745 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.04.7
Sep-19 12:54:03.756 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/username/.nextflow/plugins; core-plugins: nf-amazon@2.15.0-patch1,nf-azure@1.16.0-patch1,nf-cloudcache@0.4.3,nf-codecommit@0.2.3,nf-console@1.2.1,nf-google@1.21.1,nf-k8s@1.0.0,nf-tower@1.11.4-patch1,nf-wave@1.12.1
Sep-19 12:54:03.768 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Sep-19 12:54:03.769 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Sep-19 12:54:03.770 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Sep-19 12:54:03.775 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Sep-19 12:54:03.807 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Sep-19 12:54:03.816 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [admiring_brown] DSL2 - revision: 5c721dd9f8
Sep-19 12:54:03.817 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Sep-19 12:54:03.817 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-tower@1.11.4-patch1]
Sep-19 12:54:03.818 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-tower version: 1.11.4-patch1
Sep-19 12:54:03.823 [main] INFO nextflow.plugin.PluginUpdater - Downloading plugin nf-tower@1.11.4-patch1
Sep-19 12:54:03.956 [main] ERROR o.p.update.DefaultUpdateRepository - PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:206)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:589)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1705)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1629)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
at java.base/java.net.URL.openStream(URL.java:1161)
at org.pf4j.update.DefaultUpdateRepository.initPlugins(DefaultUpdateRepository.java:96)
at org.pf4j.update.DefaultUpdateRepository.getPlugins(DefaultUpdateRepository.java:80)
at org.pf4j.update.UpdateManager.getPluginsMap(UpdateManager.java:149)
at org.pf4j.update.UpdateManager.findReleaseForPlugin(UpdateManager.java:319)
at org.pf4j.update.UpdateManager.downloadPlugin(UpdateManager.java:268)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
at nextflow.plugin.PluginUpdater$_safeDownloadPlugin_lambda2.doCall(PluginUpdater.groovy:268)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
at nextflow.plugin.PluginUpdater.safeDownloadPlugin(PluginUpdater.groovy:270)
at nextflow.plugin.PluginUpdater.download0(PluginUpdater.groovy:240)
at nextflow.plugin.PluginUpdater.access$0(PluginUpdater.groovy)
at nextflow.plugin.PluginUpdater$_safeDownload_closure3.doCall(PluginUpdater.groovy:331)
at nextflow.plugin.PluginUpdater$_safeDownload_closure3.call(PluginUpdater.groovy)
at nextflow.file.FileMutex.lock(FileMutex.groovy:106)
at nextflow.plugin.PluginUpdater.safeDownload(PluginUpdater.groovy:331)
at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:380)
at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:224)
at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:171)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:399)
at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:289)
at nextflow.plugin.Plugins.load(Plugins.groovy:55)
at nextflow.cli.CmdRun.run(CmdRun.groovy:340)
at nextflow.cli.Launcher.run(Launcher.groovy:513)
at nextflow.cli.Launcher.main(Launcher.groovy:673)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 57 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 62 common frames omitted
Sep-19 12:54:03.957 [main] INFO org.pf4j.update.UpdateManager - Plugin with id nf-tower does not exist in any repository
Sep-19 12:54:03.957 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Plugin with id nf-tower not found in any repository
at org.pf4j.update.UpdateManager.findReleaseForPlugin(UpdateManager.java:322)
at org.pf4j.update.UpdateManager.downloadPlugin(UpdateManager.java:268)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:82)
at nextflow.plugin.PluginUpdater$_safeDownloadPlugin_lambda2.doCall(PluginUpdater.groovy:268)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:236)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:75)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:176)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:437)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:115)
at nextflow.plugin.PluginUpdater.safeDownloadPlugin(PluginUpdater.groovy:270)
at nextflow.plugin.PluginUpdater.download0(PluginUpdater.groovy:240)
at nextflow.plugin.PluginUpdater.access$0(PluginUpdater.groovy)
at nextflow.plugin.PluginUpdater$_safeDownload_closure3.doCall(PluginUpdater.groovy:331)
at nextflow.plugin.PluginUpdater$_safeDownload_closure3.call(PluginUpdater.groovy)
at nextflow.file.FileMutex.lock(FileMutex.groovy:106)
at nextflow.plugin.PluginUpdater.safeDownload(PluginUpdater.groovy:331)
at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:380)
at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:224)
at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:171)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:399)
at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:289)
at nextflow.plugin.Plugins.load(Plugins.groovy:55)
at nextflow.cli.CmdRun.run(CmdRun.groovy:340)
at nextflow.cli.Launcher.run(Launcher.groovy:513)
at nextflow.cli.Launcher.main(Launcher.groovy:673)
as a sanity check, I was able to curl and wget the Nextflow package list like this
wget https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json
With my conda env activated, I tried doing this;
$ export NXF_OPTS="-Djavax.net.ssl.trustStoreType=KeychainStore \
-Djavax.net.ssl.keyStoreType=KeychainStore"
$ nextflow run main.nf
N E X T F L O W ~ version 25.04.7
Launching `main.nf` [loving_bernard] DSL2 - revision: 5c721dd9f8
Downloading plugin nf-tower@1.11.4-patch1
executor > local (13)
....
And it works. So what happens is that when you run nextflow run main.nf with these NXF_OPTS it causes the JVM to attempt to access the macOS Keychain and you get a prompt window like this
After entering my local username/password (thankfully our managed company laptops allow us to have sufficient user permissions for this…) Nextflow was able to run successfully. But, importantly, after exiting the terminal and starting a new fresh session, I did NOT have to add those NXF_OPTS again;
$ conda activate nf-core
$ nextflow run main.nf
N E X T F L O W ~ version 25.04.7
Launching `main.nf` [lethal_mccarthy] DSL2 - revision: 5c721dd9f8
executor > local (13)
[16/ef9314] DO_THING (3) [100%] 4 of 4, ignored: 1 âś”
...
So it seems like after that initial loading of the credentials from the macOS Keychain, whatever details JVM needed to function must be cached somewhere, or at least I guess macOS is allowing the JVM to access the Keychain on subsequent runs. Regardless, its working now. And for reference, I still have this in my ~/.zshrc;
# Need this for Zscaler to work with conda installed curl that gets pulled in with some conda envs
export CURL_SSL_BACKEND=secure-transport
thanks to @robsyme for the help since that put me on the right track to figure this out