Hello!
It seems more and more nf-core pipelines are requiring the nf-validation plugin. Would it be possible to include this one as well in the “-all” binary of nextflow to simplify the process of getting a working nextflow environment up and running on an offline cluster.
Thank you!
ewels
October 26, 2023, 1:10pm
2
Yeah we’ve been discussing this a fair bit recently, we need a better solution.
Inclusion in the -all binary is not likely to happen any time soon, but there are a bunch of open issues and PRs looking into improving the current installation process:
opened 01:37PM - 10 Apr 21 UTC
pinned
env/java
## Summary
Provide a new Nextflow package distribution that runs natively in … the target execution environment ie. deb and rpm on Linux and dmg on MacOs.
## Goals
* Provide a self-contained application package that runs natively in target OS without the need to install the Java virtual machine.
* Streamline the installation of the Nexflow package.
## Description
The core Nextflow application is distributed as a collection of JAR files that are downloaded by the launcher `nextflow` script. Managing the download of the required dependencies can create unexpected problems (see #1806), moreover, the need of having a preinstalled Java VM can be overkill for some users.
Recent versions of the Java (16 and later) includes the `jpackage` tool that allows the creation of a native platform executable including all the application dependencies and the Java runtime. This means that it can be created a self-contained executable that can run natively in the target environment.
See https://openjdk.java.net/jeps/392
opened 10:23AM - 10 Jun 22 UTC
env/java
## Summary
Nextflow application is a packaged as a set of JAR files (Java arch… ives) that are uploaded to [Maven central repository](https://search.maven.org/) with the exception of the launcher app that's stored in http://nextflow.io/releases/
The `nextflow` "binary" is a mere Bash script that checks if the launcher app is stored locally. If it cannot be found it download it from the http://nextflow.io/releases URL and run it.
Then the launcher app, uses the [Capsule package manager](https://github.com/nextflow-io/capsule) to download the main application JARs from Maven along with all required dependencies files.
This system worked quite nicely for many years, however there are some serious open issues:
* The Capsule system has been is not maintained anymore by the original author. Considering the new fast release cadence of the JVM this can because soon a problem (see below)
* Recent Java release has introduced a slightly different version scheme that the Capsule system fail to recognise. This creates problems with some users that need to change Java distribution to use Nextflow. See [here](https://github.com/nextflow-io/nextflow/issues/2856) and [here](https://github.com/nextflow-io/nextflow/issues/2394) and [here](https://github.com/nextflow-io/nextflow/issues/2280)
* The Capsule system breaks when using _JAVA_OPTIONS variable. See [here](https://github.com/nextflow-io/nextflow/issues/1716)
* The Capsule system fails to download and unpack the deps in some systems because it requires to create file system lock, which not all file system supports. See [here](https://github.com/nextflow-io/nextflow/issues/995)
* The new nextflow plugins system makes the capsule system not needed anymore.
## Goals
* Remove the dependency on Capsule package system
* Create instead a standalone nextflow distribution for the core runtime. The Nextflow plugin system will continue to download the required plugins separately
* Keep the compatibility with the existing `nextflow` launcher wrapper
## Non-goals
* the support for [JReleaser](https://jreleaser.org/) or other Linux package managers such as RPM, Debian, etc. is out of the scope of this issue, however, it's desirable that the overall design to be compatible with the packaging those system, that could be adopted in the future
opened 03:24PM - 21 Sep 23 UTC
Talking to @Alexander-Stuckey about running offline with plugins, I pointed to [… these docs](https://github.com/nextflow-io/nextflow/blob/master/docs/plugins.md#offline-usage) for fetching plugins for offline use, but it seems that it doesn't work if you use the `-all` distribution of Nextflow:
```
N E X T F L O W ~ version 23.09.1-edge
Launching `/pgen_int_work/BRS/rif/helix_replacement_project/sarek/resources/nf-core-sarek_3.3.1/3_3_1/main.nf` [distracted_meninsky] DSL2 - revision: 19395081df
WARN: Nextflow self-contained distribution allows only core plugins -- User config plugins will be ignored: nf-validation@0.3.2
```
This is currently the [recommended method](https://nf-co.re/docs/usage/offline#nextflow) from nf-core, so it's causing some confusion.
We're starting to see these kinds of problems a lot now, since @nf-core pipelines are adopting the [nf-validation](https://github.com/nextflow-io/nf-validation) plugin. I think that this is probably some of the first very widespread adoption of a non-core Nextflow plugin.
nextflow-io:master ← nextflow-io:4329-offline-disable-plugin-update
opened 08:34PM - 25 Sep 23 UTC
Close #4329
This PR changes the plugin system to not download plugins in off… line mode (`NXF_OFFLINE`). Instead, Nextflow will search the plugins directory for the latest plugin version and use that one.
This change makes it possible to run a pipeline in offline mode without pinning the version of each plugin, which allows the pipeline to automatically receive plugin updates. The assumption is that users will download all the plugins they need beforehand and transfer them to the offline system, using these plugins as a local cache.
I tested this PR by installing nf-validation 0.3.0 (`nextflow plugin install nf-validation@0.3.0`) and running a script with the following config:
```groovy
plugins {
id 'nf-validation'
}
```
Normally, Nextflow would automatically download the newest plugin version (currently 0.3.2), but now if you run Nextflow with `NXF_OFFLINE=true`, it will use the already downloaded version 0.3.0 and not download anything new. I even disabled my internet connection just to be sure 😉
One limitation however is that the offline version checker doesn't verify that a plugin version is compatible with the current Nextflow version. We can get the version range from the plugin manifest file, but maybe there is some code somewhere that already does this?
opened 09:30PM - 05 Oct 23 UTC
Five years ago there was a closed bug that was asking for Nextflow to be incorpo… rated into Homebrew. After watching my team of mixed skillsets, scientists and engineers, struggle though a series of installation issues, I think it might be worthwhile to revisit this.
Nextflow has dependencies, and some of them are unusual like specific implementations of OpenJDK. A package manager will manage these dependencies, giving them executable access, and linking them.
The current installation strategy is to use wget (not installed by default on MacOS) to download a bash script with suppressed output, which runs a Nextflow installer script with debug messaging turned off. When this runs, the console hangs, and because of underlying bugs with other dependencies (Capsule), it can never return — or print anything. I think this is not the right installation strategy for Nextflow anymore. Folks that tried using SDKMAN! (also installed with arbitrary downloaded code execution) ran into linking issues where their PATH didn't have Java linked up or JAVA_HOME set.
I think `brew install nextflow` `yum install nextflow` and `apt-get install nextflow` would all be very useful depending on the platforms folks are working on. If that's too much, the folks on non-mac unix variants probably have the required skillset to work around the issues they are likely to run into, but scientists working on MacBooks are less likely to be able to navigate this. So if just Homebrew was supported, I think that would make a huge difference.
Thank you,
Ben
What would be super nice is if we can create a custom download bundle creator, where you can specify Nextflow plugins at the point of download. Then we could maintain an nf-core specific package, for example. I think @bentsherman has this use case in mind already.
Phil
Thank you Phil! Looking forward to see these improvements!
It might be mentioned in some of the issues already but let me just point out that the documentation for offline plugin (stable link for future ) usage explicitly mentions using the -all binary and then download extra plugins for that one. But if I understand it correctly, it’s not possible to use non-core plugins for the -all binary at all.
I take it that trying out the regular binary and then attempting to install plugins on that one and then explicitly pinning these in the config is the current strategy?
ewels
October 27, 2023, 8:46am
4
Ah good spot, that needs fixing.
Correct
Hopefully won’t have to explicitly pin in the config for much longer
nextflow-io:master ← nextflow-io:4329-offline-disable-plugin-update
opened 08:34PM - 25 Sep 23 UTC
Close #4329
This PR changes the plugin system to not download plugins in off… line mode (`NXF_OFFLINE`). Instead, Nextflow will search the plugins directory for the latest plugin version and use that one.
This change makes it possible to run a pipeline in offline mode without pinning the version of each plugin, which allows the pipeline to automatically receive plugin updates. The assumption is that users will download all the plugins they need beforehand and transfer them to the offline system, using these plugins as a local cache.
I tested this PR by installing nf-validation 0.3.0 (`nextflow plugin install nf-validation@0.3.0`) and running a script with the following config:
```groovy
plugins {
id 'nf-validation'
}
```
Normally, Nextflow would automatically download the newest plugin version (currently 0.3.2), but now if you run Nextflow with `NXF_OFFLINE=true`, it will use the already downloaded version 0.3.0 and not download anything new. I even disabled my internet connection just to be sure 😉
One limitation however is that the offline version checker doesn't verify that a plugin version is compatible with the current Nextflow version. We can get the version range from the plugin manifest file, but maybe there is some code somewhere that already does this?
ewels
October 27, 2023, 8:51am
5
PR to fix the docs now open here . Thanks for spotting that!
