Failed to import pandas in python process

Ask for help
,
1

Hello,

I have this nf script below that has a simple python process that starts by importing pandas then reading a csv file. The conda environment that I utilize is created is in the config file. However, everytime I run the nf script it produces an error no module names pandas.

#!/usr/bin/env nextflow
process PYMANIFESTCREATOR {
	debug true
	label 'pandasenv'
	input:
	path manifesto_file
	output:
	stdout
	script:
	"""
	#!/usr/bin/env python3
	try:
    	import pandas as pd
    	from pathlib import Path
    	import shutil
    	import glob
    	import os
    	print("All necessary modules imported successfully.")
	except ImportError as e:
    	print(f"An error occurred while importing modules: {e}")
	manifest_csv = "$manifesto_file"
	print(manifest_csv)
	"""
}
workflow {
	input_ch = channel.fromPath("$projectDir/*_Manifest.csv")
	PYMANIFESTCREATOR(input_ch)
}

nextflow.config

profiles{
	singularity{
    	singularity.enabled = true
    	process{
    	process.executor = 'slurm'
    	withLabel: qiime {
        	cpus = 10
        	memory = 100.GB
        	container = '/uufs/chpc.utah.edu/common/home/kapheim-group2/Sief/nextflow_script/singularity/amplicon_latest.sif'
    	}
    	}
	}
	conda{
    	conda.enabled = true
    	process{
        	process.executor = 'slurm'
        	withLabel: pandasenv{
            	cpus = 10
            	memory = 50.GB
            	conda = '/uufs/chpc.utah.edu/common/home/u6060645/software/pkg/miniconda3/envs/pandaenv'
        	}
    	}
	}
}
2

If I activate the environment first manually using conda activate envname and then run nextflow script it works … I thought nextflow config file activates the environment automatically?!

3

Hello @saif_s,

What command line you are typing to run this pipeline? Don’t forget to pick your profile with -profile conda. Would you, by any chance, be picking both the singularity and the conda profiles? Because there’s a conflict here. Either you use containers or conda for a process. You can’t tell Nextflow to use both strategies at the same time.

Could you please share your .nextflow.log file? Running with NXF_TRACE=nextflow before your command line (NXF_TRACE=nextflow nextflow...) enables trace-level logging, which is useful for debugging too. Consider doing that too.

4

Hello,

Thank you so much, I did figure it out and you are totally correct. I didn’t add a profile in my nextflow run command, that solved the main issue.

1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.