Issue with nf-core/taxprofiler Pipeline Stuck in Runnable Status on AWS Batch

Nextflow Help
,
1

Introduction
I’m experiencing an issue while trying to run the nf-core/taxprofiler pipeline on AWS Batch. Despite following the documentation to the best of my knowledge, my job gets stuck in the runnable status and does not execute. I’ve been tinkering around with this for couple of days now but haven’t had any success. Any guidance is greatly appreciated!

Problem Description
I’m running the pipeline from an AWS EC2 instance with the following command:
nextflow run nf-core/taxprofiler --input samples.csv --databases database.csv --outdir s3://mybucket/mybioproject/nf-core-taxprofiler/paired-end/ -params-file params.json -c nextflow.config

The job process > NFCORE_TAXPROFILER:TAXPROFILER:INPUT_CHECK:SAMPLESHEET_CHECK gets submitted to AWS Batch but doesn’t progress beyond the runnable status. I have consulted the following resources for guidance but haven’t found a solution:

Here’s my nextflow.config file:

process {
    executor = 'awsbatch'
    queue = 'lenni-aws-batch-nextflow-queue'
    container = 'nextflow/nextflow:latest'
}

aws {
    region= 'us-east-1'
    batch {
	cliPath = '/home/ec2-user/miniconda3/bin/aws'
    }
}

workDir = 's3://nextflow-workdir/nextflow_env/'

AWS Batch Setup Details:

  • Compute environment uses spot fleet (details attached below).
  • Custom AMI built as per the STAPHB tutorial (Section 2). Briefly, the AMI is build from Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI with miniconda and aws cli installed.
  • Relevant IAM user permissions are in place as per Nextflow AWS Documentation.
  • The service role permission for the AWS Batch compute environment are as follows:
  1. AmazonEC2SpotFleetRole-lenni-nextflow
    Policies: AmazonEC2SpotFleetTaggingRole
  2. AWSServiceRoleForBatch
    Policies: BatchServiceRolePolicy
  3. ecsInstanceRole
    Policies: AmazonEC2ContainerServiceforEC2Role, AmazonS3FullAccess

Compute environment JSON

{
  "computeEnvironmentName": "lenni-batch3",
  "computeEnvironmentArn": "arn:aws:batch:us-east-1:069562266249:compute-environment/lenni-batch3",
  "ecsClusterArn": "arn:aws:ecs:us-east-1:069562266249:cluster/AWSBatch-lenni-batch3-96bd2a4b-bf8e-3b3f-86c1-74df39f3fb74",
  "tags": {},
  "type": "MANAGED",
  "state": "ENABLED",
  "status": "VALID",
  "statusReason": "ComputeEnvironment Healthy",
  "computeResources": {
    "type": "SPOT",
    "allocationStrategy": "SPOT_PRICE_CAPACITY_OPTIMIZED",
    "minvCpus": 0,
    "maxvCpus": 256,
    "desiredvCpus": 2,
    "instanceTypes": [
      "optimal"
    ],
    "subnets": [
      "subnet-8509028b",
      "subnet-cb2e7fad",
      "subnet-1edb873f",
      "subnet-cda9ccfc",
      "subnet-d9d13095",
      "subnet-d52e7c8a"
    ],
    "securityGroupIds": [
      "sg-d4fbddce"
    ],
    "ec2KeyPair": "lenni-nextflow",
    "instanceRole": "arn:aws:iam::069562266249:instance-profile/ecsInstanceRole",
    "tags": {},
    "bidPercentage": 80,
    "spotIamFleetRole": "arn:aws:iam::069562266249:role/AmazonEC2SpotFleetRole-lenni-nextflow",
    "ec2Configuration": [
      {
        "imageType": "ECS_AL2",
        "imageIdOverride": "ami-0b74aeb97fba885ea"
      }
    ]
  },
  "serviceRole": "arn:aws:iam::069562266249:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch",
  "containerOrchestrationType": "ECS",
  "uuid": "f7147aad-d45a-3708-aaae-4569c5add212"
}
2

Sitting in RUNNABLE implies it’s been correctly submitted but nothing is picking it up. This probably means your issue is on the AWS side of things.

In the AWS console, what are the details of the Compute Environment? Is it currently accepting jobs? You can create a quick Job Definition + Job in the AWS Console to see if it works.