Hello,
I am unsure whether this comes from Nextflow, Seqera platform, AWS, interactions between this components, or me setting bad configurations.
I have the feeling that my workflows do not have the most efficient resource allocations:
- sometimes processes are run in instances that are way oversized. This has two major drawbacks:
- unnecessary costs (e.g. a process which requests 2 CPUs and 8 Gb of RAM is run in an instance with 8 CPUs and 32 Gb of RAM, with no other process run in this instance)
- quota consumption: if my quota is 64 CPUs, I can only run 8 instances with 8 CPUs, but could run 32 with 2 CPUs. This makes a huge difference in runtime.
- I am under the impression that sometimes several processes are run in the same instance, but with inconsistent distribution (sometimes it is the case, sometimes new instances are started even if there would be enough resources left in one instance).
Any pointer to resources for how to control that would be appreciated.
Best,
Theo