Is it possible to use Nextflow with the `libmamba` conda solver?

Ask for help
1

Hi all,

I have set up my conda installation to use the libmamba solver (rather than the standalone mamba binaries etc).

However when I try to use conda (v23.3.1) with a Nextflow (v23.04.1) pipeline this configured, get the following error:

    CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic

To reproduce:

  1. Set libmamba solver
conda install -n base conda-libmamba-solver
conda config --set solver libmamba
  1. Then run a nextflow pipeline with the conda profile specified
nextflow run nf-core/taxprofiler -profile conda,test -r 1.1.1 --outdir ./results

Does anyone have the same problem, and if so if they have a solution?

Cheers,
James

3 Likes
2

Yo @jfy133 !

Can you dig up the conda command used by Nextflow in the .command.run file? Let’s see if we get the same error when running it outside of Nextflow.

Cheers,
Robert

2 Likes
3

So the full NXF error is:

ERROR ~ Error executing process > 'NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /home/james/cache/conda/env-f398718d50f0fbf68c336c2cb6f533f8 bioconda::minimap2=2.24
  status : 1
  message:
    CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic

Running the same command results in successfully creation

$ conda create --mkdir --yes --quiet --prefix /home/james/cache/conda/env-f398718d50f0fbf68c336c2cb6f533f8 bioconda::minimap2=2.24
Channels:
 - conda-forge
 - bioconda
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/james/cache/conda/env-f398718d50f0fbf68c336c2cb6f533f8

  added / updated specs:
    - bioconda::minimap2=2.24


The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
  k8                 bioconda/linux-64::k8-0.2.5-hdcf5f25_4 
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_2 
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_2 
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_2 
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5 
  minimap2           bioconda/linux-64::minimap2-2.24-h7132678_1 
  zlib               conda-forge/linux-64::zlib-1.2.13-hd590300_5 


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
4

Hey, James!

Yes, you can. I can’t reproduce your error.

I did the following:

conda update -n base conda
conda install -n base conda-libmamba-solver
conda config --set solver libmamba

I confirmed with conda config --get solver, which returned --set solver libmamba. conda config --show also confirmed it, but much more verbose :laughing:

Then, I created a new folder and added the following Nextflow script:

process FOO {
  conda 'fastqc'
  debug true

  output:
  stdout

  script:
  """
  fastqc --version
  """
}

workflow {
  FOO()
}

Ran with NXF_VER=23.04.1 nextflow run main.nf -with-conda and got as output the following:

CAPSULE: Downloading dependency io.nextflow:nextflow:jar:23.04.1
CAPSULE: Downloading dependency io.nextflow:nf-httpfs:jar:23.04.1
CAPSULE: Downloading dependency io.nextflow:nf-commons:jar:23.04.1
Nextflow 23.10.0 is available - Please consider updating your version to it
dN E X T F L O W  ~  version 23.04.1
Launching `main.nf` [loving_cray] DSL2 - revision: e432c2850d
executor >  local (1)
[d2/28d2f3] process > FOO [100%] 1 of 1 âś”
Creating env using conda: fastqc [cache /Users/mribeirodantas/conda-test/work/conda/env-23037893fdd4cef8d4acc5fe82ba1899]
FastQC v0.12.1

PS: I tried running nextflow run nf-core/taxprofiler -profile conda,test -r 1.1.1 --outdir ./results as you did, and it works past the NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX' step so for some reason I can’t reproduce this either.

[bf/e48529] process > NFCORE_TAXPROFILER:TAXPROFILER:LONGREAD_HOSTREMOVAL:MINIMAP2_INDEX                             [100%] 1 of 1 âś”

I didn’t wait for it to finish, though.

I would guess this issue is related to your conda version. I’m using v23.9.

1 Like
5

Interesting! :thinking: Would you mind sharing the whole conda config? To see what I may have different…

And also try maybe with an nf-core pipeline? The only thing I can now think is there some conda conflict when it tries to make multiple envs at once…

6

Would you mind sharing the whole conda config?

I provided no extra configuration. Empty nextflow.config. Added -with-conda to the command line, and that’s it.

And also try maybe with an nf-core pipeline?

I did. It’s at the bottom of the previous reply.

7

Sorry, I meant the config of conda itself (basically what is stored in ~/.condarc, not the nextflow conda config!

I had the same error on both old and latest version :confused:

Oops sorry about the last bit - the discourse website wouldn’t load on Firefox for mobile for me so I read your response/replied via email but I guess that bit wasn’t displayed… (Now on Chrome for mobile :grimacing:)

8

Oh, sorry. You’ll find my $HOME/.condarc below:

auto_activate_base: false
channels:
  - conda-forge
  - bioconda
  - defaults
channel_priority: strict
changeps1: true
subdir: osx-64
report_errors: true
solver: libmamba
1 Like
9

Nope still not working, even when I copy your config (removing the subdir section). I’m also using 23.09 conda… :frowning:

I’m suspecting it’s a linux thing now… @rpetit3 what OS do you typically use?

10

I tried it now on the training Gitpod instance, which runs Linux, following my steps above, and it also worked correctly :disappointed_relieved:

nf-training -> conda --version
conda 23.9.0
nf-training -> conda config --get solver
--set solver libmamba

nf-training -> rm -rf work; rm -rf nextflow.config; nextflow run main.nf -with-conda
N E X T F L O W  ~  version 23.10.0
Launching `main.nf` [agitated_volta] DSL2 - revision: e432c2850d
executor >  local (1)
[d7/d48364] process > FOO [100%] 1 of 1 âś”
Creating env using conda: fastqc [cache /workspace/gitpod/nf-training/work/conda/env-23037893fdd4cef8d4acc5fe82ba1899]
FastQC v0.12.1
11

Bizarre… must be something to do with my specific set up then… no clue what. Oh well! I’ll just have to live with it.

Thank you for the help investigating @mribeirodantas! :pray:

1 Like
12

You’re welcome!

Let us know in the future if you manage to identify what the issue was :sweat_smile:

1 Like
13

i know this is a bit stale now, but i encountered this recently and found this thread… in my case at least, it seems to have been that the version of libstdc++ available to my os was older than libmamba wanted.

2 Likes