We are developing our own nextflow pipeline and have one question regarding peak_rss in the report.html report.
Specifically, we have a process that was requested to run with 20G memory and 16 threads. The process was run successfully, and the report.html shows peak_rss as 152G. We are not sure how the 152G was calculated. We believe the peak memory use for all the applications in this process was less than 20G. Some application in this process use multi-threading. According to nextflow document:
peak_rss
Peak of real memory. This data is read from field VmHWM in /proc/$pid/status file.
Thank you so much for looking at this, any response are appreciated.
Hi @Li_Teng ,
Great to hear you are developing your own Nextflow pipeline and thanks for posting this question!
Performance metrics such as cpu and memory usage in Nextflow are directly calculated using /proc and passed along by Nextflow. You can find the actual code for how Nextflow does this in this template.
So to find out why peak_rss can be higher than expected, we have to go back to the documentation for proc and :
Peak resident set size (“high water mark”). This
value is inaccurate; see /proc/pid/statm above.
So the answer here is that peak_rss will often be very similar to rss, especially for processes with stable memory consumption. For processes that have big peaks in memory usage (due to whatever reason), peak_rss can be quite high. If your processes are not failing due to insufficient memory due to this, I wouldn’t worry too much about this for now. There are obviously a lot of different tools out there and they can all have different memory usage behaviours, so it’s hard to troubleshoot your particular observation.
If we look again at the code and situations where you have multiple child processes within a Nextflow process, Nextflow will actually sum up the peakRSS of these child processes and thus if they peak at different times, these peaks will be added up, contributing to an overall peakRSS that can be much higher than the actual peak memory of each individual child process.
You can also find an discussion from 2023 about this on the Nextflow Github:
Unfortunately I don’t have more insight into proc than this information but I hope this helps in understanding why you are seeing these reported values.
Thanks a lot for the prompt response and useful insights!