How to capture logs from htCondor with nextflow

Hi everyone! First post here, recently started to learn nextflow, so bare with me:)

So we have the htcondor manager on our cluster.

I want to capture the .out, .err and .log that condor is creating.

process {
    executor = 'condor'

    // Creates condor job files in the local work directory
    clusterOptions = '''
        log    = Logs/${task.process}/${task.tag}.log
        output = Logs/${task.process}/${task.tag}.out
        error  = Logs/${task.process}/${task.tag}.err
    '''
}

I got from seqera copilot that I can use {task.process} and {task.tag} to capture information from each process.

The problem here is that {task.process} and {task.tag} is not yet defined in this step of the process so condor puts the work on “hold” saying that the path doesnt exist.

Does anyone have a better advice how to capture these logs?

best /
Jonas

Are the logs not automatically captured?

I’m not familiar with htcondor, but usually for other’s like slurm, the logs are captured automatically.

When you run a Nextflow process, it’ll will create a work directory. In that work directory are several hidden files.

.command.run is the one that contains instructions to submit the job to htcondor. If it’s anything like slurm, then there are headers at the top of file with the options like what to name the log, resource allocation, etc

.command.sh is the script to run in the .command.run environment. When troubleshooting in a work directory you can modify the .command.sh of a failed execution, and then test it’s working with sbatch .command.run or locally with bash .command.run. I hope it’s something similar for htcondor.

.command.out is the captured output stream.
.command.err is the captured error stream.
.command.log is the captured output and error streams together.

There are a few extra hidden files too.
Would that be sufficient or do you have another purpose for capturing the logs?

Ah that’s perfect, solved the problem:) thanks a lot for the help!

Welcome to the community forum, @biomedswe :slight_smile:

You’re correct in that when the configuration is resolved, the pipeline hasn’t run yet and therefore it can’t resolve variables that are defined during a task run (process is merely a definition, a task is the process instance, what is actually run). As @mahesh.binzerpanchal mentioned, Nextflow .command.* files are supposed to capture what you’re after.

One idea that came to me now is that you could use afterScript to copy/move your logs from the task directory to your logs folder (check docs for afterScript here).

I prepared a PoC below that should work for you.

My folder structure contains a main.nf Nextflow script and a move_logs.sh bash script inside a bin folder.

$ tree -a
.
├── bin
│   └── move_logs.sh
└── main.nf

You’ll find below the content of my Nextflow script:

process FOO {
  afterScript 'source move_logs.sh'

  input:
  val string

  output:
  tuple path('out.txt'), stdout

  script:
  """
  echo ${string}
  echo "some ${string}" > out.txt
  """
}

workflow {
  Channel.of('some', 'strings', 'to', 'test').set { my_ch }
  FOO(my_ch)
}

It simply receives a group of strings and prints them to the output and stores to a file named out.txt. There are four strings so we’re talking about four tasks being run and with that 4 task folders. The logs will be there, but I understand you may want to move them somewhere else as the work directory is usually treated as a temporary folder for intermediate files. The afterScript process directive will run the move_logs.sh bash script after the task is finished.

You’ll find the move_logs.sh file content below:

#!/usr/bin/bash

task_path=$(echo "$PWD" | awk -F/ '{print $(NF-1)"_"$NF}')
run_folder=$(echo "$PWD" | rev | cut -d/ -f4- | rev
)

mkdir -p $run_folder/Logs/$task_path
cp .command.err $run_folder/Logs/$task_path
cp .command.out $run_folder/Logs/$task_path
cp .command.log $run_folder/Logs/$task_path

This script basically finds out where you ran your project from and what’s the folder for each task. If you’re running your pipeline from one place but setting a custom work directory to be somewhere else, you must adjust this in the code above. With these two information resolved, we can create the folders (in case they’re not already there) and copy the three log files you’re interested (.err, .out, .log) to this Logs folder, within their respective task directories.

That’s what I get by running the pipeline above:


thx @mribeirodantas and thx for your excellent suggestion! sadly, it seem like afterScript only runs of the script runs to completion and not if it fails:( At least that seems to be the case to me but I might be wrong?

best
Jonas

Not really. I made one of the four tasks fail and I stil see logs in the Logs folder. See the new main.nf code below. Focus on the script block, where I make it fail if the string is to.

process FOO {
  afterScript 'source move_logs.sh'

  input:
  val string

  output:
  tuple path('out.txt'), stdout

  script:
  """
  if [[ "${string}" == to ]]; then
    echo "Error: Variable contains the string 'example'" >&2
    exit 1
  fi
  echo ${string}
  echo "some ${string}" > out.txt
  """
}

workflow {
  Channel.of('some', 'strings', 'to', 'test').set { my_ch }
  FOO(my_ch)
}

Outputs below:

All .command.err are empty (success), apart from the failed task. And the .command.out of the failed task is empty, but the other ones are not.

$ cat Logs/45_dc138901c617d7b41f7f1fc1b601ea/.command.err 
Error: Variable contains the string 'example'
$ cat Logs/45_dc138901c617d7b41f7f1fc1b601ea/.command.out 
$ cat Logs/9b_1316f01c30ec6dbd336fb52ad31d51/.command.out 
test
$ cat Logs/b9_baeae5ed373c56adf12ae3eee798bc/.command.out 
some
$ cat Logs/28_1ffc521881670ca5638988ecb0ee75/.command.out 
strings

Aha I see! However, I have very hard to understand how afterScript works. The documentation is for some reason almost not existent for this section:

with this i get no syntax error but the afterScript causes: Process MergeBamAlignment (HTL214) terminated with an error exit status (127).

process MergeBamAlignment {

    afterScript './Scripts/copy_condor_logfile.sh'

    tag { "${sample_id}" }

    publishDir params.merged_output_dir, mode: params.publish_mode

    input:
    tuple val(sample_id), path(unmapped_bam), path(aligned_bam)

    output:
    path "${sample_id}_merged_unmapped_mapped.bam"

    script:
    """
    # Source shared functions
    source ${file('./Scripts/nextflow_functions.sh')}

    # export for use in copy_condor_logfile.sh
    export LOG_DIR="${params.log_dir}/${task.process}"
    export TAG="${task.tag}"
    mkdir -p "\$LOG_DIR"

    echo "[DEBUG] Merging BAMs for sample: \$TAG" >> "\$PWD/.command.out"

    if ! picard MergeBamAlignment \\
        UNMAPPED_BAM=${unmapped_bam} \\
        ALIGNED_BAM=${aligned_bam} \\
        REFERENCE_SEQUENCE="${params.reference_genome}" \\
        OUTPUT="${sample_id}_merged_unmapped_mapped.bam" \\
        TMP_DIR="${params.tmp_dir}" \\
        SORT_ORDER=coordinate \\
        INCLUDE_SECONDARY_ALIGNMENTS=false \\
        VALIDATION_STRINGENCY=SILENT; then
        echo "❌ MergeBamAlignment failed for \$TAG" >> "\$PWD/.command.err"
    else
        echo "✅ MergeBamAlignment done for \$TAG" >> "\$PWD/.command.out"
    fi

    move_named_log ".command.condor" ".command.err" ".command.out" ".command.run" ".command.sh"
    """
}

I suspect this might be due to variables not found in the copy_condor_logfile.sh:

#!/bin/bash

# Detect condor log file
log_file=$(ls "$PWD"/[0-9][0-9][0-9][0-9][0-9][0-9]*.log 2>/dev/null)

# Only proceed if the log file was found
if [[ -n "$log_file" ]]; then
  cluster_id=$(basename "$log_file")
  cluster_id="${cluster_id%%.log}"

  # LOG_DIR and TAG must be exported by the process script beforehand!
  cp "$log_file" "${LOG_DIR}/${cluster_id}_${TAG}_condor.log"
else
  echo "[WARN] No Condor log file found" >> "$PWD/.command.err"
fi

So both chatGPT and Seqera copilot suggests that I change the afterScript line to:

afterScript:
  """
  export LOG_DIR="${params.log_dir}/${task.process}"
  export TAG="${task.tag}"
  bash ./Scripts/copy_condor_logfile.sh
  """

But then I get the syntax error in VSC:

Invalid process definition -- check for missing or out-of-order section labels nextflow

When I paste that back to chatgpt and copilot it just starts halucinating that the snippet must be after the script block or inside the workflow etc…sigh.

Thank god it’s still good programmers out there on forums to get decent help from haha:D

Thx in advance!

/Jonas

The 127 exit status means command not found. Nextflow can’t find the script you want to run with afterScript simply because the path you’re providing doesn’t exist (even though it may be in your current folder, but that’s not how Nextflow looks for files).

Try using the default folder (scripts inside bin, as I did) and it should work. Nextflow adds all scripts inside this folder to $PATH so that your Nextflow script can find it (command found :wink: )

aha great, thank you so much for the help!