Hi all,
I’m looking for any advice/tips on optimizing Nextflow performance on an HPC cluster. We’re currently running a benchmarking study and using HyperQueue to interface with the Slurm scheduler. Our plan is to scale out across many nodes and run workflows with anywhere from a few hundred up to several thousand tasks.
Everything is working as expected with our current configuration, but we’d love to hear if there are known best practices or tuning options to improve execution efficiency and overall performance at this scale.
If you’ve done similar large-scale Nextflow runs (especially with Slurm + HyperQueue), any pointers or “gotchas” would be greatly appreciated.
Thanks!