Nextflow Secrets produces empty string due to wrong path

Ask for help
,
1

Hi guys, I have a problem with Nextflow secrets.
My process looks like this:

process FOO {

    time '24h'
    secret 'USERNAME'
    secret 'PASSWORD'

    input:
    [...]


    output:
    [...]

    script:
    """
    #!/usr/bin/env bash
    ## dummy echo for testing
    echo \$USERNAME
    echo \$PASSWORD
       [...]
    """
}

USERNAME and PASSWORD exist in nextflow secrets. When I run the process, both echo commands print empty strings. If I change the name of the secrets variable, it fails, so Nextflow recognizes that USERNAME and PASSWORD exists. But of course this doesn’t help me. When I look into the .command.log, I see the following telling text:

grep: ./secrets/.nf-ec9a753b-9de3-466f-b0ec-c5e6b1a407d6.secrets: No such file or directory

So, Nextflow wants to look into the secrets directory in the current work dir, but the secrets directory is in my project root!
Looking into the .command.run, my suspicion is confirmed, Nextflow changes into the workdir before trying to read from the secrets file in root dir:

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    NXF_SCRATCH=''
    [[ $NXF_DEBUG > 0 ]] && nxf_env
    touch [workdir.... ]/.command.begin
    set +u
    # conda environment
    source $(conda info --json | awk '/conda_prefix/ { gsub(/"|,/, "", $2); print $2 }')/bin/activate foo
    set -u
    export PATH="$PATH:/foo/bar"
    source /dev/stdin <<<"$(cat <(grep -w -e 'USERNAME=.*' -e 'PASSWORD=.*' ./secrets/.nf-ec9a753b-9de3-466f-b0ec-c5e6b1a407d6.secrets))"

Am I using secrets wrong? Do I have to give the secrets dir as input? Because linking it into the workdir would work, but would contradict the use of secrets imo. Or is it a bug?

2

Hi @pwolk ! Welcome to Seqera Community Forum :slight_smile:

How did you create the secrets? I tried to reproduce your situation the following way and it worked:

secrets.nf

process FOO {                                                                      
    debug true                                                                     
                                                                                   
    time '24h'                                                                     
    secret 'USERNAME'                                                              
    secret 'PASSWORD'                                                              
                                                                                   
    script:                                                                        
    """                                                                            
    #!/usr/bin/env bash                                                            
    ## dummy echo for testing                                                      
    echo \$USERNAME                                                                
    echo \$PASSWORD                                                                
    """                                                                                                                                                      
}                                                                                  
                                                                                   
workflow {                                                                         
  FOO()                                                                            
}

# Creating secrets
nextflow secrets set USERNAME Marcel
nextflow secrets set PASSWORD Marcelpass
nextflow run secrets.nf

CleanShot 2025-01-29 at 11.11.22@2x
CleanShot 2025-01-29 at 11.11.22@2x1460Ă—456 82.8 KB

I don’t see in my .command.log or .command.run the same thing you’re seeing.

...
source /dev/stdin <<<"$(cat <(grep -w -e 'USERNAME=.*' -e 'PASSWORD=.*' /Users/mribeirodantas/.nextflow/secrets/.nf-ef20b430-8476-43f7-a171-3b82e5e61    ae1.secrets))"
...
3

Thank you for trying it out!

I just did it like you!
nextflow secrets set USERNAME paul

4

Okay, sorry for wasting your time. :smiley: I just found out the reason:
The bash script that starts the pipeline runs a command that sets the NXF_HOME variable, but the command to determine what to use as home produces just “.”, so this is used also to determine the secrets directory and results in this ./secrets… . ahhh :frowning: using realpath in front of that command solves it.

1 Like
5

Thanks for coming back to share the explanation of what led to the reported scenario, @pwolk ! Great to hear you’ve solved it :partying_face:

1 Like
6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.