More information: Here’s essentially my nextflow run
command, wrapped into a Batch job submission
aws batch submit-job \
--job-name scallops-nf \
--job-queue <queue-name>\
--job-definition <jd>:97 \
--tags "Name=scallops-nf-headnode" \
--container-overrides '{
"vcpus": 1,
"memory": 2048,
"command": [
"nextflow",
"-log", "s3://sandbox-lindsay/nextflow/pilot/scallops/logs/nextflow.log",
"run", "main.nf",
"-bucket-dir", "s3://sandbox-lindsay/nextflow/scallops/cache/",
"--input", "s3://sandbox-lindsay/nextflow/pilot/scallops/scallops_test.csv",
"--barcodes", "s3://bigdipir-ctg-s3/CRC/CRC057/library_with_UBE2Z_guides_c8.csv",
"--publish_uri", "s3://sandbox-lindsay/nextflow/pilot/publish/",
"-with-report", "s3://sandbox-lindsay/nextflow/pilot/scallops/logs/report.html",
"-with-trace", "s3://sandbox-lindsay/nextflow/pilot/scallops/logs/trace.txt",
"-with-dag", "s3://sandbox-lindsay/nextflow/pilot/scallops/logs/flowcharg.png",
"-c", "/app/nextflow.config",
"-resume", "59537859-0d69-42de-a52f-8242c6a18249",
"-stub" ]
}'
The session id following -resume
was taken from the report.html
from a previous run.
Additionally the contents of -bucket-dir
look like
(base) liangl29@liangl29-R61DJX scallops-nf-workflow % aws s3 ls s3://sandbox-lindsay/nextflow/scallops/cache/ --recursive | head -n 5
2024-09-18 11:01:17 0 nextflow/scallops/cache/10/082321589291368b7201481d7d2905/
2024-09-18 11:01:27 6 nextflow/scallops/cache/10/082321589291368b7201481d7d2905/.command.begin
2024-09-18 11:01:30 0 nextflow/scallops/cache/10/082321589291368b7201481d7d2905/.command.err
2024-09-18 11:01:32 328 nextflow/scallops/cache/10/082321589291368b7201481d7d2905/.command.log
2024-09-18 11:01:29 328 nextflow/scallops/cache/10/082321589291368b7201481d7d2905/.command.out
And here is my nextflow.config
nextflow.enable.dsl=2
process.executor = 'awsbatch'
process.queue = '<queue-name>'
process.cache = 'lenient'
aws.region = 'us-west-2'
process {
withLabel "scallops" {
container = '<ecr-arn>'
cpus = 2
memory = "4 GB"
}
}
report.overwrite = true
timeline.overwrite = true
dag.overwrite = true
trace.overwrite = true