Platform behaving differently for participants in same workspace

Ask for help
1

Hi Team,
I am experiencing a strange behaviour with the platform and need some help on this

  1. Being a part of same workspace, and having only one compute environment configured, I am able to launch workflows but my professor is unable to do the same. He gets the error “No tower agent online”
  2. I am able to add/remove/modify participants in a workspace which I have left, but I was previously a part of. I can see the workspace and I can add myself without being a part of it!

Details:
My professor and I are a part of a workspace.
Earlier we both were able to launch workflows without any issues. Recently, my professor is unable to launch the workflow. He gets the error message "No agent online".

We only have one compute environment configured and I can launch the workflow on that same environment without any issues.

Now here is when things start getting weird,
We tried leaving the workspace and re-joining it to see if the issue persists. Even when I had left the workspace, I was able to see the workspace, its workflows, and I was able to launch them without any errors.

But my professor is still not able to launch them.

2

Hi Jash,

Thanks for posting.

I’ll start with the easier question 2. It sounds like you may be an Owner of your Organization which gives you the ability to see all workspaces and manage participants.

For question 2, I would:

  • Under your Organization, do you have any Managed Identites or is the Compute Environment using Managed identity cluster as the credential type?
  • Examine any differences in the launch of the pipelines. Are there any user-specific parameters or configuration such as the workdir that only you can access in the HPC

Hope this helps!

Evan

3

Hi Evan,
Thanks for replying.
You’re right about the owner privilege. I was the owner of the workspace. But is this expected behavior that even if I leave the workspace, I can access and modify it?

Initially when setting up the compute environment, I was unable to connect to the compute environment using SSH (probably because the servers are present in my university’s data center and they require an additional authentication before SSH), so I am using tower agent to connect to the platform.

So we don’t have any Managed identities at the organization level. We have a workspace created under the organization and have a single compute environment (SLURM based) and credentials configured in that workspace.

Like I mentioned in the original post, everything was working perfectly earlier. But suddenly, we started facing these issues.

Regarding the parameters or configuration passed to the workflow, nothing user-specific is passed. We also tried launching a simple hello world workflow but faced the same issue.

4

Hi Jash,

Evan is talking about the Organization, not the Workspace (which belongs to the Organization). If you are the Owner of an Organization then by default you can manage all it’s Workspaces and their users, even yourself.

Regarding the problem with your professor launching pipelines - can you please confirm the Role he has in the Workspace? At a minimum, he will need at least the Launch role to execute any pipeline.

Hope this aids you in resolving,
Rob N

5

Hi Rob,
Apologies for the misunderstanding. My and professor’s role in the organization and the workspace is Owner.

Let me know if more details are required.

Thanks!

6

Hi Jash,

Thanks for clarifying. If this is using our multi-tenant Cloud deployment (https://cloud.seqera.io) could you please provide the Organization and Workspace names so we can take a closer look under the hood?

Warm regards,
Rob N

7

Sure,
Organization: North_Carolina_State_University
Workspace: CIFR

8

Rob - Any idea what could be the problem with our setup? Thanks -ic

9

I understand you are using the Tower Agent to launch the jobs.

You should confirm with your professor they are setup using their own token and has the agent running on your cluster.

export TOWER_ACCESS_TOKEN=<YOUR TOKEN>
curl -fSL https://github.com/seqeralabs/tower-agent/releases/latest/download/tw-agent-linux-x86_64 > tw-agent
chmod +x tw-agent
./tw-agent XXXXXXX-XXXX-XXXX-XXXXXXXXXX  --work-dir=./work

where the XXXXX is the specific tower agent id.

1 Like
10

Hi Evan,
Just to confirm, even for the same compute environment, the tower agent should be running per user with the user’s unique tw-agent token?

11

Hi Evan, Rob,
Creating a new token solved the problem.
I created a new tower access token and tower agent credential and changed the compute environment configuration to use that tower agent credential.
Now my professor is also able to launch workflows and using the same compute environment config, I am also able to launch the workflows.

So it seems like the error was related to some user-level permissions corresponding to my user on the server.

Thanks again to both of you for your help!

1 Like
12

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.