# Nextflow pipeline fails on AWS because it can't find executable in Docker container

**URL:** <https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149>\
**Category:** Ask for help\
**Tags:** nextflow, aws\
**Created:** [August 28, 2024, 10:14pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149 "2024-08-28T22:14:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 28, 2024, 10:14pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/1 "2024-08-28T22:14:34Z")

</div>

Hi I am new to Nextflow and AWS batch. I was able to run the example nf-core pipeline on AWS batch no problem. But I am trying to run my pipeline which requires the UCSC util bigWigToBedGraph.

I made a docker container to that installs it in /usr/local/bin:

```auto
# Install UCSC utilities including bigWigToBedGraph
RUN wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64.v385/bigWigToBedGraph -O /usr/local/bin/bigWigToBedGraph \
    && chmod +x /usr/local/bin/bigWigToBedGraph

```

Then I specify the docker container to use in my nextflow config:

```auto
process.container = '<account>.dkr.ecr.us-east-1.amazonaws.com/aindap-process-horvath:latest'

```

I even specify the full path to the executable, which I double checked by runnign the container interactively on EC2:

```auto
// Define the process to convert BigWig to BedGraph
process bigWigToBedGraph {
    tag "${bw.baseName}"
    publishDir "${params.output_dir}", mode: 'copy'
    
    
    input:
    path bw

    output:
    path "${bw.baseName}.bedGraph"

    script:
    """
    echo "running bigWigToBedGraph for file: ${bw.baseName}.bedGraph"
    /usr/local/bin/bigWigToBedGraph $bw ${bw.baseName}.bedGraph
    """
}

```

But I still get an error:

```auto
running bigWigToBedGraph for file: HG002_rep1.hg38.pbmm2.combined.bedGraph
Error: bigWigToBedGraph not found. Please make sure it's installed and in your PATH.

```

I am not really sure what is going on and would appreciate any pointers. It seems specifying the full path in the nextflow file should work.

---

<div class="post-metadata">

**Author:** ![mribeirodantas](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/mribeirodantas/32/235_2.png) [@mribeirodantas](https://community.seqera.io/u/mribeirodantas)\
**Post date:** [August 29, 2024, 2:17am UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/2 "2024-08-29T02:17:48Z")

</div>

Have you tried running your container (with the same URI you’re passing to Nextflow) in interactive mode and checking if the binary is there and with correct permissions?

---

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 29, 2024, 3:51am UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/3 "2024-08-29T03:51:14Z")

</div>

That’s the strange thing , if I run the container interactively, I’m able to confirm the program is in the PATH and it actually works

But when I run via batch the same doesn’t hold in terms of bigWigToBedgraph

---

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 29, 2024, 4:01am UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/4 "2024-08-29T04:01:17Z")

</div>

![Screen Shot 2024-08-28 at 8.59.59 PM](https://europe1.discourse-cdn.com/flex013/uploads/seqera/original/2X/f/f0397e282bcfc45847e1af27866d5ad2f4f840af.png)  
 ![Screen Shot 2024-08-28 at 9.01.03 PM](https://europe1.discourse-cdn.com/flex013/uploads/seqera/original/2X/f/f63b74969c4c5095295d48e98a59a4e98bac6159.png)

---

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 29, 2024, 4:04am UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/5 "2024-08-29T04:04:02Z")

</div>

When I print out the PATH variable when the nextflow pipeline runs:

```auto
running bigWigToBedGraph for file: HG002_rep1.hg38.pbmm2.combined.bedGraph
//nextflow-bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

```

It shows /usr/local/bin as part of the path

---

<div class="post-metadata">

**Author:** ![Adam\_Talbot](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/adam_talbot/32/62_2.png) [@Adam\_Talbot](https://community.seqera.io/u/Adam_Talbot)\
**Post date:** [August 29, 2024, 10:04am UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/6 "2024-08-29T10:04:29Z")

</div>

That’s unusual. Could you try running `which bigWigToBedGraph` in the Nextflow process? Or perhaps `ls -lha /usr/local/bin` which might indicate something wrong with the binary. I’m assuming this is all on x86 and not ARM (Apple silicon).

My only other thought is that it could be user permissions, with some difference between the user on AWS Batch and your local Docker environment.

Also, you could try the image [community.wave.seqera.io/library/ucsc-bigwigtobedgraph:469--c71de23bcede0988](http://community.wave.seqera.io/library/ucsc-bigwigtobedgraph:469--c71de23bcede0988) built from [Seqera containers](https://seqera.io/containers).

---

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 29, 2024, 3:46pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/7 "2024-08-29T15:46:54Z")

</div>

> [@Adam\_Talbot](#):
>
> ls -lha /usr/local/bin

Thanks for the response, Adam. I did check and make sure I pulled the x86 binary when I made my docker container:

```auto
# Install UCSC utilities including bigWigToBedGraph
RUN wget http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64.v385/bigWigToBedGraph -O /usr/local/bin/bigWigToBedGraph \
    && chmod +x /usr/local/bin/bigWigToBedGraph

```

I tried running bigWigToBedGraph from another image specified in my nextflow file:

```auto

// Define the process to convert BigWig to BedGraph
process bigWigToBedGraph {
    tag "${bw.baseName}"
    publishDir "${params.output_dir}", mode: 'copy'
    container "community.wave.seqera.io/library/ucsc-bigwigtobedgraph:469--c71de23bcede0988"
    
    input:
    path bw

    output:
    path "${bw.baseName}.bedGraph"

    script:
    """
        echo "running bigWigToBedGraph for file: ${bw.baseName}.bedGraph"
        bigWigToBedGraph $bw ${bw.baseName}.bedGraph
        """
}

```

But then I get an error

```auto

ERROR ~ Error executing process > 'bigWigToBedGraph (HG002_rep1.hg38.pbmm2.combined)'

Caused by:
  Task failed to start - CannotStartContainerError: Error response from daemon: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/bin/_entrypoint.sh": stat /usr/local/bin/_entrypoint.sh: no such file or directory: unknown

```

Another idea I was going to try was to pull the executable from my S3 bucket and use it as a param in my config file

```auto
params.ucsc_util = "s3://aindap-demo/bigWigToBedGraph"

```

---

<div class="post-metadata">

**Author:** ![Adam\_Talbot](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/adam_talbot/32/62_2.png) [@Adam\_Talbot](https://community.seqera.io/u/Adam_Talbot)\
**Post date:** [August 29, 2024, 4:04pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/8 "2024-08-29T16:04:10Z")

</div>

> [@Amit\_Indap](#):
>
> `unable to start container process`

This is something more fundamentally wrong with your set up. What is your AWS Batch configuration? How did you set the compute environment and queue up? What AWS machine type is the process running on?

---

<div class="post-metadata">

**Author:** ![Amit\_Indap](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/amit_indap/32/1041_2.png) [@Amit\_Indap](https://community.seqera.io/u/Amit_Indap)\
**Post date:** [August 30, 2024, 4:52pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/9 "2024-08-30T16:52:38Z")

</div>

I was following along the steps described [here](https://staphb.org/resources/2020-04-29-nextflow_batch.html) to set up my AWS batch. The machine type is AWS Linux with compute environment having  
AWSServiceRoleForBatch and ecsInstanceRole attached.

Anyways, I solved my original problem with help from @robsyme on a call. The path to my aws-cli was in /usr/local/bin in my nextflow config. This was mounting /usr/local/bin from the AMI and blocking whatever in my docker container /usr/local/bin (like the bigWigToBedGraph in my docker image)

I updated where my aws-cli was and pointed to a more inncoucrous path in my config and the pipeline ran just fine.

---

<div class="post-metadata">

**Author:** ![system](https://dub1.discourse-cdn.com/flex013/user_avatar/community.seqera.io/system/32/2402_2.png) [@system](https://community.seqera.io/u/system)\
**Post date:** [September 6, 2024, 4:53pm UTC](https://community.seqera.io/t/nextflow-pipeline-fails-on-aws-because-it-cant-find-executable-in-docker-container/1149/10 "2024-09-06T16:53:15Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.