Hi Bogdan. Jumping in quickly to add some colour to Ben’s answer, because even though Fusion is amazing and a great solution (check it out), the race should be fixable in the launch template.
We ran into the same problem at Seqera and we handled it by ordering operations in the launch template rather than by waiting. At the top of the launch template we stop the Docker daemon before installing anything, and only restart Docker and enables the ecs service at the very end, after the AWS CLI and any storage setup (EBS autoscale, FSx, EFS) are done. Roughly:
systemctl stop docker
## install the AWS CLI, mount FSx/EFS, etc.
systemctl start docker
systemctl enable --now --no-block ecs
This works because the ECS agent runs as a container and ecs.service requires docker.service. Stopping Docker takes the agent down with it, and until ecs is started the instance never registers with the cluster, so Batch cannot place a job on it. There’s no window in which a task container can start against a half-installed CLI. If your user data runs from cloud-config runcmd, the agent may already have come up by the time you get there, which is what the systemctl stop docker is for.
How cliPath actually gets into the container
Nextflow doesn’t mount the aws binary. It takes the grandparent directory of the path you give it and bind-mounts that whole directory, read-only, at the same path inside the task container. That’s why the config insists the value end in /bin/aws: the layout is assumed to be <install root>/bin/aws, and <install root> is what gets mounted.
Two things follow from that.
First, the install has to be self-contained under that root. Nothing else from the host filesystem is visible inside the container, so every shared library the binary loads at runtime has to live inside the mounted tree. That’s almost certainly your libz.so.1 error: the v2 binary is finding something it needs on the host but not inside the mount.
Second, pick the root carefully, because it’s mounted over the top of whatever the container has at that path. /opt/aws-cli/bin/aws is fine, it only shadows /opt/aws-cli. But if anyone points cliPath at the v2 installer’s symlink, /usr/local/bin/aws, Nextflow mounts the host’s /usr/local over the container’s /usr/local, which is where a lot of bioinformatics images keep their tools and libraries.
This is why we doesn’t use AWS CLI v2 for cliPath compute environments at all. We unpack a Conda-packaged AWS CLI v1 under /home/ec2-user/miniconda and points cliPath at /home/ec2-user/miniconda/bin/aws. The Conda env carries its own Python and shared libraries (libz included) under miniconda/lib, so it works wherever it’s mounted, and no container image is going to have anything it cares about at /home/ec2-user/miniconda.
So if you want to stay on the launch template route: install a self-contained CLI (a Conda env is the easy way) under a path nothing else will ever use, point cliPath at its bin/aws, and gate the ECS agent behind the install. Otherwise Wave/Fusion or a custom AMI avoid the whole thing.