All nodes of label ‘macos15-arm64’ are offline

[richardlau]

https://ci.nodejs.org/job/node-test-commit-osx/nodes=macos15-arm64/ is stuck because

All nodes of label ‘macos15-arm64’ are offline

cc @ryanaslett

[richardlau]

[sxa]

Assumption: The Orka resources are shared between the main ci and the release CI

The logs of ci.nodejs.org are showing:

Started provisioning cc00012e-eab7-457c-8f77-0e66344f3bf6 from Orka Arm Cloud with 1 executors. Remaining excess workload: 26.771
Aug 21, 2026 6:01:27 AM INFO io.jenkins.plugins.orka.OrkaCloud provision
[provisionId=6c1e0d05-2b8e-4f13-91fc-cb829a6fd348] Provisioning for label osx13-arm64. Workload: 2
Aug 21, 2026 6:01:27 AM WARNING io.jenkins.plugins.orka.AgentTemplate provision
Deploying VM failed with: HTTP Code: 500, Error: Internal error occurred: Cannot deploy more than 2 VMs on an ARM host.

Which suggests that Orka believes all of the available arm64 resources are in use and therefore no more can be deployed.

Analysis of the release CI showed several jobs which were running jobs on macos/arm64 but had seemingly hung, including one that had probably been stuck for over two days.

I have aborted those jobs on the release CI as they were not in use for any release work, only testing.

The orka plugin has a 15 minute timeout after which those slots can be reused by the test CI.

At the time of writing this, three macos15-arm64 nodes have been provisioned and are in use by the node-test-commit-osx job

There is currently a backlog of about 60 jobs targetting that label. They typically take about 20 minutes each so I would expect that it will take about 6-7 hours to fully clear the backlog (20*20 minutes). I think leaving them to execute naturally is preferable to clearing the backlog and require collaborators to reschedule things so I am not taking any further action on those jobs.

[sxa]

One thing I will note is that due to this all of the executors on the jenkins-workspace machines (3 machines each with 25 executors) are in use, preventing more things from being scheduled. Each currently running instance of node-test-pull-request and the subjob node-test-commit use an executor on a workspace node.

Since the post-build-status-update job also requires an executor on a jenkins-workspace node those are also all "stuck". I will leave it for a while to see if any executors get freed up naturally once the osx jobs start going through but I suspect that is unlikely since those are subjobs of node-test-pull-request and node-test-commit jobs which are already holding onto executors themselves so we are probable already in a deadlock situation. The currently in-flight fanned jobs (e.g. node-test-commit-arm-fanned are also currently deadlocked as a result of trying to run git-rebase which requires an executor.

As a result of the above it is therefore likely that I will have to temporarily increase the number of allowable executors to hopefully permit some of the post-build-status-update jobs to run though to completion, freeing up the two parent jobs. I hope that the post-build-status-update jobs will be able to grab the executors before other node-test-pull-request jump in.

Just for reference at the time of writing there are 436 entries in the build queue on ci.nodejs.org

[sxa]

The last post-build-status-update job to have ran through was https://ci.nodejs.org/job/post-build-status-update/3197629/ over 13 hours ago.

I have added another five executors to test-mnx-ubuntu2204-x64-1 and that appears to be allowing some of the status-updates jobs to run (I note that 3197630 seemed to run and then disappear quickly as per the screenshot below where I was watching one - likely due to job retention policies only keeping 100 and we probably had more than that number in the queue:

It looks like some other jobs had been able to acquire some executors so I've increased the number that can be used on the mnx-1 machine to 50 and I have confirmed that the post-build-status-update jobs appear to be succeeding (For example this message) in one of the PRs.

Status: The number of in-use executors on the mnx machine is at 50/50 - I will now double the number on the second mnx jenkins-workspace machine to 50 as well. At the time of writing post-build-status-update has been able to run up to job 3198005 but up to 3198090 are still waiting. The overall number of jobs in the jenkins queue is now 199 so has halved as a result of freeing the executors

[sxa]

50/50 executors on mnx-2 are now in use including several running post-build-status-update tasks (which are short so should complete quickly)

screenshot of post-build-status-update executions

EDIT: post-build-status-update backlog has almost cleared (<10 left). All jenkins-workspace executors are still in use. Jenkins queue now at 131 after dropping to around 100 (due to node-test-commit jobs spawning off the platform specific ones) I have temporarily throttled node-test-commit so it cannot spawn more than 10 in parallel (other than the ones already in flight) to stop it chewing up executors - with hindsight I should have done that before the initial increase from 25 to 50. The last few post-build-status-update jobs are now managing to progress.

[richardlau]

50/50 executors on mnx-2 are now in use including several running post-build-status-update tasks (which are short so should complete quickly)

@sxa Please keep an eye on nodejs/jenkins-alerts#6555 -- disk space on the machine is now 90% full.

[sxa]

@sxa Please keep an eye on nodejs/jenkins-alerts#6555 -- disk space on the machine is now 90% full.

Yep that's also part of why I've tried to throttle the parallelism on node-test-commit as it will potentially be generating additional @ workspaces on the workspace servers

[sxa]

Executor count on both machines has been dropped back to 25.

I'm tempted to leave the parallelism restrictions on node-test-commit but I'll raise a separate issue to discuss at the next build WG meeting...