Hi all — I’ve just released the first version of nf-prometheus, an open-source plugin that exports workflow and task metrics to Prometheus, built specifically for on-premise HPC clusters.
The itch it scratches: on a shared Slurm cluster I couldn’t answer basic questions while a run was going — how long are my tasks queued, per process? which process over-requests memory? is the run stuck or slow? — without tailing logs and running squeue in a loop. And the head job usually runs on a compute node your Prometheus server can’t reach, so a classic scrape endpoint doesn’t help.
So the plugin is textfile-first: it writes an atomically-updated .prom file for the node_exporter textfile collector on a shared filesystem — zero network requirements from the head job, works on air-gapped clusters. A Pushgateway mode gives you live updates during the run, and a plain HTTP /metrics endpoint exists for when the head node is reachable. A ready-to-import Grafana dashboard ships with it (screenshot below): per-process scheduler queue wait, requested vs used CPUs and peak RSS, task states, run history.
- Registry:
plugins { id 'nf-prometheus' }— Nextflow Registry | nf-prometheus@0.1.0 - Source (Apache 2.0): GitHub - mcallisto/nf-prometheus: Prometheus metrics for Nextflow on HPC clusters: queue wait, resource usage, run status. Grafana dashboard included. · GitHub
- 5-minute setup: https://github.com/mcallisto/nf-prometheus/docs/5-minute-setup.md
- Background & war stories from the test cluster: Where did my task go? Monitoring Nextflow on Slurm without Tower · Callisto Labs
Zero dependencies beyond the JDK, TraceObserverV2 API, min Nextflow 25.10. Metric export can never fail your pipeline.
If you run Nextflow on Slurm/PBS/LSF on-premise I’d genuinely like to hear whether this fits your setup — issues and feature requests very welcome, especially about metrics you’re missing.