Question: Scaling listener pod replicas to avoid single point of failure

[sundeepgarg]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

10.0.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:
- Can increasing the number of listener pod replicas to two lead to a race condition?
- Will both listeners listen to the same job, potentially causing the same job to run twice?
- Is there any listener pod election, or will both listener pods run at the same time?
- Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

Describe the bug

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:

• Can increasing the number of listener pod replicas to two lead to a race condition?
• Will both listeners listen to the same job, potentially causing the same job to run twice?
• Is there any listener pod election, or will both listener pods run at the same time?
• Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

Describe the expected behavior

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:

• Can increasing the number of listener pod replicas to two lead to a race condition?
• Will both listeners listen to the same job, potentially causing the same job to run twice?
• Is there any listener pod election, or will both listener pods run at the same time?
• Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

Additional Context

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:
- Can increasing the number of listener pod replicas to two lead to a race condition?
- Will both listeners listen to the same job, potentially causing the same job to run twice?
- Is there any listener pod election, or will both listener pods run at the same time?
- Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

Controller Logs

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:
- Can increasing the number of listener pod replicas to two lead to a race condition?
- Will both listeners listen to the same job, potentially causing the same job to run twice?
- Is there any listener pod election, or will both listener pods run at the same time?
- Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

Runner Pod Logs

I want to know if replica of listener pod can be increased to two instead of default one so that we can avoid single point of failure.

Specifically, I am concerned about the following:
- Can increasing the number of listener pod replicas to two lead to a race condition?
- Will both listeners listen to the same job, potentially causing the same job to run twice?
- Is there any listener pod election, or will both listener pods run at the same time?
- Is increasing the listener pod to two a good idea, or could it introduce unwanted side effects?

Please provide guidance on best practices for scaling the listener pod and any potential risks or mitigations involved.

[nikola-jokic]

Since this is a question and not an issue, I will convert it to a discussion.

To answer it, you cannot have multiple instances of the listener, and there is no need for it. Just like you cannot have a duplicate runner session, you cannot have a duplicate listener session.
You don't need to worry about availability, since the autoscaling listener controller will spin up the listener pod as soon as it exits. In order to have a multiple pods ready, we would have to use some sort of leader election, keeping one listener as a leader handling the communication with the server.

However, if you want to have multiple active scale sets where each one can pick up the job, you can use active-active deployment: https://docs.github.com/en/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets#high-availability-and-automatic-failover

[jjacobelli]

Hey @nikola-jokic ,

You don't need to worry about availability, since the autoscaling listener controller will spin up the listener pod as soon as it exits.

I would say this is true only if the listener is doing a graceful exit. In the case a node goes NotReady without any graceful shutdown (crash, power loss, ...), the listener pods may be blocked in Terminating state for a long time. In this situation, the controller wont recreate a new pod as the current pod still exists. This is preventing new jobs to be scheduled on the cluster until the Terminting listener is deleted. Having two listener pods, with leader election, running on two different nodes/zones, would prevent this issue.

Can you revisit this request?

Thank you

[Nhqml]

Yes, I'm experiencing the same issue, with the listener pod being stuck in "Terminating" and the controller not creating a new one.

What are my options to prevent such SPOF?