Append to params from a .csv

Ask for help
1

My ask is for a way to read values from a .csv and have them appended to params with the code all in nextflow, not some wrapper script. As a convenience for my users, I’d like to let them set some params in a .csv instead of a nextflow.config or command line options. As a stretch, read the .csv directly from S3.

.csv could be row or column oriented, but I’ll go with row oriented for this example. The names of the columns become the names of the elements of params. The values of the one row become the values of the params elements.

s3://bucket/input.csv is:

n_one,s_two,n_three
1,two,3

resulting in params.n_one, params.s_two, params.n_three being visible to the workflow:

// use these, somehow, to read from s3??
// import nextflow.cloud.aws.utils.*
// import nextflow.cloud.aws.nio.*

workflow {
 // some way to read .csv and append values to params

log.info("${params.n_one}") // value set in .csv

}
2

Have you considered the -params-file option (I notice you only mention command-line or config)?
Rather than a csv though, you provide a yml or json, e.g.,

params.yml:

n_one: 1
s_two: 'two'
n_three: 3

and then doing

nextflow run <script> -params-file params.yml [other options]

The parameters are then available as you would expect. They take lower precedence than command-line options, but higher precedence than a config file. See Configuration — Nextflow v24.08.0-edge documentation.

3

Thank you for bringing attention to -params-file. At the moment, the users have .csv config files for their code that we’re wrapping in nextflow and we’d like to leverage those.

What would be really interesting would be some idea of how to extend the logic that handles -params-file to ingest .csv. Subclass? Plugin?

4

You can do it like this, but I think it would be better to use the standard yaml or json format if you can wrangle it. You could also write a plugin too.

params.params_csv = null
params.input = 'input.file'

workflow {
    if ( params.params_csv ) {
        params = params + file( params.params_csv, checkIfExists: true)
            .splitCsv(header: true)
            .head()
        println params
    }
    if ( params.s_two == 'two' ) {
        println "S TWO"
    }
}

output:

$ nextflow run main.nf --params_csv params.csv 

 N E X T F L O W   ~  version 24.04.4

Launching `main.nf` [fabulous_miescher] DSL2 - revision: 2b28f701b9

[params_csv:params.csv, input:input.file, n_one:1, s_two:two, n_three:3]
S TWO
5

Thank you.

6

I think nf-core has this built into the template. I ended up using a simple csv file with one column, so I did not use the nf-core feature.