I have a question about setting up compute environments with Seqera Forge. I noticed that in my own setup, the access policies created for instance roles are programmatically created by Seqera Forge. I think this means that the policy here may not be required (at least when Seqera Forge provisions compute environments and AWS resources). I created a compute environment on Seqera platform using Forge and was able to launch jobs without the s3-bucket-write.json policy.
In my AWS console, I see that the IAM role given to Seqera Forge when creating the environment is used to only build the compute environment and is not the role assumed to run the batch jobs.
Could you confirm that this is how things work? If so, would it be worth it to update the docs (and / or the README) to note this detail? I was confused and spent some time (in vain) changing the policy above, trying to give a jobs launched in a compute environment access to new buckets.
You are correct that those linked IAM roles are attached to the user whose credentials you provide to Seqera Platform. Those credentials are used to either:
submit jobs to existing AWS Batch queues, in which case you only need the “launch” credentials, or
create AWS Batch queues, AWS Batch CEs, instance roles, autoscaling groups, etc and then submit jobs to those queues. In this case, you will need the more elevated “forge” permission set.
The repository you linked contains two groups of policies, matching these two use-cases.
Does this make sense? If so, I’ll make a recommendation to our docs team to make this position more explicit.
For documentation on the Nextflow docs, you can take advantage of the fact that everything is open source and make an issue or even a pull request on the Nextflow repository. My first contribution to Nextflow was via this mechanism . We’d certainly welcome your input.
For Seqera Platform documentation, the best thing to do would be to make a post here on the Community Forum or via the Nextflow Slack workspace.
I wanted to follow up here about this issue. I have been setting up / tearing down on my own setups and have been thinking about this more. I have two questions / concerns.
1) About the docs
In my own AWS environment, I am able to create compute environments and launch jobs only with a user given only the forge-policy.json.
I understand that this is intended behavior, per @robsyme, but I don’t think this is accurately described by the docs here (AWS Batch | Seqera Docs) that asks a user to use both IAM policies and create a custom S3 policy.
2) About “forge” vs “launch” policies.
As far as I can tell, the permissions for a launched job is inherited from the compute environment; and when using Forge, the compute environment uses the same credentials to create resources (IAM roles, Batch queues etc.,) and to launch jobs.
What is seqera’s platform’s recommendation w.r.t separating IAM permissions for Seqera Forge and Seqera Launch? Can one do so?
Our docs team are reviewing the docs now and we’ll work to clear up any confusion there.
We provide two example permission sets because not all users will want Seqera Platform to create the infrastructure for a CE.
In cases where a user already has their AWS Batch infrastructure stood up using other tools, they do not need to provide our platform with an IAM role that has permissions to create AWS Batch queues, for example. If they already have cloud infrastructure, the user might use the launch policy as a template for the policy they provide to Seqera Platform.
If the user would prefer to make use of Seqera Platform’s “Forge” feature to stand up the infrastructure in addition to submitting jobs on that infra, Serera Platform will require the expanded set of permissions outlined in the forge policy.
If you choose to use Forge to create a Compute Environment, Forge will, as part of that process create IAM roles specifically for that CE. For example, I created a Platform Compute Environment with ID 2CeQImw0JjWrMM0tRDGRZ7 and so forge created three IAM roles
TowerForge-2CeQImw0JjWrMM0tRDGRZ7-ExecutionRole
TowerForge-2CeQImw0JjWrMM0tRDGRZ7-InstanceRole
TowerForge-2CeQImw0JjWrMM0tRDGRZ7-ServiceRole
When using Forge, the permissions allowed by the EC2 instances launched in Compute Environment with ID “XX” are set by the role TowerForge-XX-InstanceRole, and not by the policy you give to Seqera Platform.
Thank you very much for the detailed response! I actually think that your response here would sufficiently clarify the details I found confusing if it was shared either in the Docs or in the root and forge related READMEs in this the nf-tower-aws repo.
. We’d certainly welcome your input.