We recently upgraded to SLURM 23.11.6, and enabled tmp container creation for jobs. Post upgrade, our Nextflow pipelines have stopped working with this error:
container creation failed: mount tmpfs->/opt/singularity/4.1.2/var/singularity/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /opt/singularity/4.1.2/var/singularity/mnt/session: write unix: write: broken pipe
Has anyone seen this before? Any ideas on how to fix? It appears to be trying to mount the /opt directory?
Thanks in advance