S3 work folder

Ask for help
1

Hi, when trying to use a s3 folder as a work dir when launching a pipeline from seqera, I get this error:

Pipeline work directory must start with slash character - offending value: s3://scratch/work

I tried defining workDir from the nextflow.config but it is still using the one defined during the lunch form (where I had to put a non-s3 one)

2

Hi Miguel

I’m not sure I totally understand your situation. Some quick questions:

  1. These are runs launched from Seqera Platform, correct?
  2. What is the work directory defined in the Compute Environment (CE) in which you are executing the run?
  3. What was the value you trying to set the work directory to?
  4. It sounds like you’d like to override that work directory for a particular pipeline or a particular run. Is that correct?

In general, if you want to override the CE for a particular run, it would be sensible to use the “Work directory” text field as shown below. Is that an option for your situation?

CleanShot 2025-02-06 at 14.52.45@2x
CleanShot 2025-02-06 at 14.52.45@2x2844×1374 301 KB

3

Hi robsyme,

  1. Yes.
  2. It is a general work folder in the netapp
  3. The value is something similar to your screenshot: s3://scratch/work/
  4. Yes
    That’s exactly my situation. When I try overriding using a s3 folder, I get the error: Pipeline work directory must start with slash character - offending value: s3://scratch/work and I can’t submit it. I also tried not overriding the work folder in the “Run parameters” tab but in the Advanced settings, in the nextflow.config, but it doesn’t work either.
4

Thanks for the info, Miguel.

Are you launching the run using a compute environment configured with an HPC executor (SLURM/PBS/SGE, etc)?

Using an S3 path as the work directory only makes sense when using the AWS Batch executor. It would be more efficient to use a networked filesystems on your HPC rather than reading from and writing to a (potentially) remote object store like S3.

5

Hi robsyme,

  • Yes, we are using SLURM.
  • We are using minio configured in a local S3. Since it is local, the performance could be enough, right?
6

Just to clarify, I’m using Slurm with the Tower Agent