Adding sample id to resourceLabels in config file

[johandahlberg]

Hi!

I have a question about resourceLabels. I want to add the sample names as tags to allow us to do some cost tracking in AWS, I have something like this right now:

process {
    executor = "awsbatch"
    queue = "nf-batch-spot"
    resourceLabels = {[ 'pixelator-project': params.project_name, 'pixelator-sample-id': { "${task.id.toString()}" }]}
...

In my custom config file, but I can’t seem to figure out how to access the id of the sample in this context. Is it even possible?

[mribeirodantas (Marcel Ribeiro-Dantas)]

If you inspect the task object you will see there is no id property, but index. And maybe that’s not what you’re looking for, as it’s not a workflow-wide unique id.

You could use something like: "$task.process:$ifile" where ifile is the sample name, and you still have the process name, in case this input file is provided to other processes too.

There was a typo in the first version of this message. I meant $ifile not $task.ifile.

[johandahlberg]

Thanks Marcel! In this case I actually don’t want a unique id, but rather one that is the same for all processes for the same sample. Basically the question I’m trying to answer is, “how much have we spent on analyzing sample x in AWS?”

resourceLabels = {[ 'pixelator-project': params.project_name, 'pixelator-sample': task.ifile]}

Based on your suggestion I tried the above. However then the pixelator-sample tag does not show up in AWS. Any idea why that is?

[ewels (Phil Ewels)]

@johandahlberg - @mribeirodantas made a typo, no corrected above. Could you please try again?

resourceLabels = {[ 'pixelator-project': params.project_name, 'pixelator-sample': task.process:$ifile]}

(I think that’s right - @mribeirodantas?)

[robsyme]

No $ needed there, @ewels - that is only needed inside string interpolation.

Given a process

process LabelExample {
    input:
    tuple val(sampleId), path(fastq)

    output:
    tuple val(sampleId), path('out.*.dat')

    script:
    "touch out.${sampleId}.dat" 
}

You can use configuration

process {
    withName: LabelExample {
        resourceLabels = {[
            'pixelator-project': params.project_name,
            'pixelator-sample': sampleId
        ]}
    }
}

Note @johandahlberg, that the sampleId is not a property of task. If the variable is defined as input in the process, it will be available in scope when the closure is evaluated.

[johandahlberg]

Thank you!

With the help of your suggestion I was able to figure it out. This is what I arrived at in the end. We want to attach this to all processes where meta.id is available, so hence the null checking below.

process {
    executor = "awsbatch"
    queue = "nf-batch-spot"
    resourceLabels = {[
            'pixelator-project': params.project_name,
            'pixelator-sample': (meta?.id ?: 'no-sample-info')
        ]}
...