Python module in process problem

[Iman_Nazari]

Dear Community,
I am using containers in nextflow, without having any problems, the only prblem is, when i want to run python scripts (from the python installed inside the container), it can not imports all installed python libraries.

Command error:
  /opt/conda/envs/drop/bin/python
  Python 3.10.14
  Traceback (most recent call last):
    File "/mainfs/genepy.py", line 1, in <module>
      from numba import njit, cuda, prange
  ModuleNotFoundError: No module named 'numba'

but i checked it, seems python version and path is exactly that i need .
/opt/conda/envs/drop/bin/python
Python 3.10.14

running apptainer outside the Next flow, i can run python inside the container and load all libraries, but in next flow it doesnt work.

any suggestions to overcome it?

thanks in advance
Iman

[mribeirodantas (Marcel Ribeiro-Dantas)]

Hey @Iman_Nazari. Can you please share a minimal reproducible example?

I wonder if the conda environment within your container is not being activated, or something like that.

[Iman_Nazari]

Hi @mribeirodantas , yes sure,
this is my python program,

#!/opt/conda/envs/drop/bin python
from numba import njit, cuda, prange
import numpy as np
import pandas as pd

import math
import pyarrow.csv as pa_csv
from typing import Tuple
import argparse
import time
import os
import sys

and this is my process:

process genepy_g1 {
  
  publishDir "${params.output}/Genepy_score", mode: "copy", overwrite: true

  input:
  file("chunk.lst")
  
  output:
  file("*_gscore_gp1.txt")
  
  script:
    """
    source activate drop
    
    cat ${params.output}/meta_CADD${params.cadd_filter}.txt | head -n 1 > header.meta
    tot_c=\$(awk '{print NF - 26; exit}' header.meta)
    gs=\$((tot_c / 3))
    cut -d\$'\t' -f1-26,27-\$((26+gs)) header.meta > "header.meta_gp1"
    cat "chunk.lst" | while read i; do
      cp "header.meta_gp1" "\${i}_gp1.meta"
      cat ${params.output}/meta_CADD${params.cadd_filter}.txt |grep "\$i" | cut -d\$'\t' -f1-26,27-\$((26+gs)) | awk -F"\t" '{OFS=FS}{for (s=7;s<=16;s++) if(length(\$s)<1 || \$s==0) \$s="3.98e-6"}1'>> "\${i}_gp1.meta"
      which python
      python --version
      python ${params.genepy_py} \${i}_gp1.meta \${i}_gscore_gp1.txt
    done
    """
}

and this is the output:

ERROR ~ Error executing process > 'genepy_g1 (2)'

Caused by:
  Process `genepy_g1 (2)` terminated with an error exit status (1)

Command executed:

  source activate drop

  cat meta_CADDALL.txt | head -n 1 > header.meta
  tot_c=$(awk '{print NF - 26; exit}' header.meta)
  gs=$((tot_c / 3))
  cut -d$' /main' -f1-26,27-$((26+gs)) header.meta > "header.meta_gp1"
  cat "chunk.lst" | while read i; do
    cp "header.meta_gp1" "${i}_gp1.meta"
    cat meta_CADDALL.txt |grep "$i" | cut -d$'  ' -f1-26,27-$((26+gs)) | awk -F"        " '{OFS=FS}{for (s=7;s<=16;s++) if(length($s)<1 || $s==0) $ s="3.98e-6"}1' >> "${i}_gp1.meta"
    which python
    python --version
    python genepy.py
  done

Command exit status:
  1

Command output:
  /opt/conda/envs/drop/bin/python
  Python 3.10.14

Command error:
  /opt/conda/envs/drop/bin/python
  Python 3.10.14
  Traceback (most recent call last):
    File "genepy.py", line 2, in <module>
      from numba import njit, cuda, prange
  ModuleNotFoundError: No module named 'numba'

if run it outside the nextflow , everything works well

also different shebang used and yet it doesnt work
shebangs like :

#!/usr/bin/python 

#!/usr/bin/env python

[Iman_Nazari]

ok , i found the error,
the problem is, when i was making a docker image , numba library installed in the /opt/.local/sitepackage/python path instead of /opt/conda/env.

adding this line to the python code , solved the problem
import sys
sys.path.append(“/drop/.local/lib/python3.10/site-packages”

but my question is , how come running it outside the nextflow everything is fine , but when it runs inside the nextflow, it doesnt work? it should not behave the same?

[ewels (Phil Ewels)]

If you’re using the nf-core pipeline template, you might find that nextflow.config is setting an environment variable PYTHONNOUSERSITE = 1 which we added for exactly this reason - to prevent the user’s Python libraries getting into the container run time.

Otherwise it could be down to what directories are mounted (including the home directory).

[Iman_Nazari]

Thanks ewels, solved!