Hello, there. I am new to Nextflow and would like some help here. I was wondering the best approach to develop a dashboard application for monitoring Nexfolow jobs running on local HPC environment. Is there any plug-in that provides endpoints to Nextflow? Can Nextflow can be figured to use an external database to store its internal status? Thanks in advance!
Hi @hubin-keio,
A warm welcome to the nextflow community 
!
There is actually already a great solution available to your question. Seqera labs (the makers of nextflow) have developed Seqera Platform (formerly called Nextflow tower): https://tower.nf.
You can monitor your jobs runtime, performance and resource usage using this dashboard live in the browser. This works on cloud and local HPCs. It might be a bit outdated for some of the details, but a general introduction talk was given by Evan Floden at nf-core bytesize a while ago (https://nf-co.re/events/2021/bytesize-20-nextflow-tower).
To get started, simply create a free account with tower (or use one of the sign-in options), set up your access token on your HPC (Seqera Cloud) and add -with-tower to your nextflow command. This will then send the runtime information live to your Seqera Platform account and you should see your current workflow run under Runs in your Seqera platform dashboard.
If you are looking for more of a custom way to show runtime information, Seqera platform also has an API: Tower API - Seqera Platform documentation
I hope this answers your question and hope you will enjoy using Seqera Platform 
!
Thanks, @FloWuenne. We actually need to develop a solution for some special use cases that essentially forbids using external platforms.
Ahh ok I see @hubin-keio !
I believe that with the Enterprise version Seqera Platform can be hosted locally so that no data would be send outside (Seqera Platform Enterprise installation - Seqera Platform documentation). Maybe someone from Seqera can comment on this?
1 Like
Thanks @FloWuenne! You’re right, and I believe that Graham had a pretty length discussion about options with @hubin-keio in person at the hackathon on this topic yesterday. We also discussed some other approaches, such as the -with-weblog flag and more.
2 Likes
Yes, I did chat with Graham and several other folks here. Thank you!
And, yes, “-with-weblog” seems most appropriate at this time. I hope the -with-weblog option will be supported long term. We will take a closer look after the summit. We are developing tools for government agencies so we represent an edge case. An example of what we do: https://edgebioinformatics.org/
1 Like