Hello!
This question is about how to structure a workflow which consists of multiple different studies, which I want to run separately. For example, suppose that I have the following workflows
Workflow 1
This workflow produces 3 variants of an object A, resulting in A1, A2, A3. I produce different kinds of plots and conclude that A2 is the best variant.
Workflow 2
This workflow takes a param like best_A_variant = 2 and performs another modification, resulting in A21, A22, A23. When I run this workflow, I do not want to reproduce A2.
I’m sorry if this is a bit abstract. My specific context is training reinforcement learning agents where each workflow performs an ablation study of one specific component. I then have a “main” workflow where I set all the parameters that were found to be optimal in the other workflows. Currently, my folder structure is
nextflow.config
workflows/
workflow1.nf
workflow2.nf
...
processes/
process1/
main.nf
process2/
main.nf
...
results/
workflow1/
workflow2/
params/
workflow1/
workflow2/
work/
I then use a separate git worktree for each pipeline so that I can reuse code but still have independent work and results folders.
I know that I would get reusability if I just merged everything into one workflow, but I don’t understand how I would do that while still keeping the ability to run and configure the separate workflows independently.
Curious to hear how this can be improved. Let’s discuss!