I started using tag to identify the samples each process are processing and simultaneously I started losing visibility of the process names itself.
It turns out that the output width of a nextflow run is affected by the width of the terminal. That makes sense when the terminal is an interactive one, but when the output is redirected to a file, it makes no sense that it changes following the changes of the enclosing terminal.
Also, my production executions are triggered by a Jenkins server that connects to a slave in the HPC infrastructure. There is no terminal I can actuate on. But the output is reduced to some small number of columns, which causes me not to be able to clearly see what is happening (although the tag is now clearly visible).
Which is the way to tell Nextflow which is the desired width of the output?
I tried with export COLUMNS=200, but it didn’t worked
Are you using the -ansig-log false argument? This should not only solve the column width issue, but will also ensure that each task is printed to a separate line in the output.
For example:
nextflow run hello -ansi-log false
You may also find the .nextflow.log file useful as it also contains the full list of tasks the tags and the work directory paths (in addition to many other useful pieces of information).
No, I’m not using the -ansi-log argument, as this will change the functionality.
I’m interested in the standard tty output, where the processes are grouped by name and tell me how many are finished and how many are in total (with the completion percentage next to it). This gives me an overview of the whole execution.
Having just each task printed in the output is useless to me, as gives me no information on how the overall execution is going. If I’m interested in that kind of information, I can check the .nextflow.log file or the trace file.
My interest is in an overview of the whole situation, and for that I need the summary (not the individual processes information) and enough width to be able to know which process are we talking about. Right now, the tag is most of the view, so I don’t know if it is being preprocessed, aligned, indexed, or any other process.
The code that calculates the terminal width is here:
With the default value here:
There is no way to fix the width I think. I agree that honouring the $COLUMNS environment variable would be good, if you fancy putting in a GitHub issue / PR for that, that’d be great
As @robsyme says, the ANSI output is really not designed for redirection to a file. It repaints the terminal window, the control sequences don’t really make sense in a file and it will be massively bloated - the linear format without ANSI is much better suited to non-interactive file based reading.
Thanks for the pointers. I already looked into the code, but at the end, I didn’t found where jline is getting the width (so I could try to fake that source).
I will open an issue, as I’m not proficient enough in Groovy to write a PR. In any case, a quick fix like honoring an environment variable should be something easy to do.
In my use case, I have a handful of nextflow pipelines running simultaneously in SLURM jobs, all of them writing their output to the files defined in the SLURM allocation. And I want to monitor the running processes interactively. What I do is just tail those files and I get a quick overview of how things are going. But as I said: I need the summary lines, including completion percentage. Just the information about processes starting and finishing (like the linear format without ANSI) is of no use at all to me.
There could be other solutions, like keeping a file with a single copy of that output, just the last one: open the file, truncate, write the full summary; an update would be a file truncation followed by a new summary. That would avoid huge files while still giving summary information to the user.
