Different containers for different processes (running nextflow in offline mode)

Ask for help
1

I am trying to run a Nextflow workflow offline using two different Apptainer containers that are already downloaded and stored in my home directory.

The workflow is intended to use:

  • One container exclusively for the Manta module/process.
  • A second container for all other tools in the workflow.

I am encountering two issues:

  1. Container path not recognized globally

    When I specify the container directly in the process configuration, for example:

    container = "${System.getenv('HOME')}/sv_caller_new.sif"

    Nextflow does not appear to recognize the container path. However, if I place the same container specification inside a withName: block, Nextflow correctly finds and uses the container.

    Is there a reason why the container path is only being recognized when defined under withName: and not when specified more generally?

  2. Process-specific container assignment not working

    I would like the Manta process/module to use its dedicated container, while all other processes use a different container. I attempted to implement this using withName: blocks, but the workflow always uses the first container and never switches to the Manta-specific container.

    What is the recommended way to configure multiple local containers so that a specific process (Manta) uses one container and all remaining processes use another, particularly in an offline environment?

Below is my configuration file:

apptainer {
enabled = true
autoMounts = true
}

process {
beforeScript = '''
mkdir -p "$PWD/nxf_tmp"
export TMPDIR="$PWD/nxf_tmp"
'''

```
//withName: '.*'{
//    container = "${System.getenv('HOME')}/sv_caller_new.sif"
//}

 withName: '!.*POPGEN48_SV_CALLER:SV_CALLER:MANTA_GERMLINE' {
    container = "${System.getenv('HOME')}/sv_caller_new.sif"
}

withName: '.*POPGEN48_SV_CALLER:SV_CALLER:MANTA_GERMLINE' {
    container = "${System.getenv('HOME')}/manta_python.sif"
}


env{
        TMPDIR = '$PWD/nxf_tmp'
        TMP    = '$PWD/nxf_tmp'
    }

resourceLimits = [
    time: 24.h
]
```

}

timeline {
enabled = true
file = "timeline.html"
overwrite = true
}

trace {
enabled = true
file = "trace.txt"
overwrite = true
}

I am using nextflow version: 25.04.6.5954

2

It’s a bit difficult to help without seeing more of your pipeline. Can you elaborate on what is not working as expected? Is it crashing, or using the wrong container (if so, what processes are using which container?).

Note that it’s better practice to set the NXF_APPTAINER_CACHEDIR environment variable or the apptainer.cacheDir config option to specify where container images are stored, instead of using ${System.getenv('HOME')} directly in your config like that. Also process declarations overwrite based on specificity, so the cleaner is to set a default for all processes and then just overwrite the one you need:

apptainer.cacheDir = ${System.getenv('HOME')}
process {
  container = "sv_caller_new"

  withName: '.*POPGEN48_SV_CALLER:SV_CALLER:MANTA_GERMLINE' {
    container = "manta_python"
  }
}
3

Please accept my apologies for not posting the questions in the required detaills, I have edited my question.

4

Unfortunately, this solution does not work for this specific case.