test_0053_hang_watchdog crashes its xdist worker intermittently (1 in ~15 CI runs)

[lmoresi]

tests/test_0053_hang_watchdog.py::test_watching_restores_the_previous_watchdog failed in CI with a crashed worker:

__________________________ test_0053_hang_watchdog.py __________________________
worker 'gw0' crashed while running 'test_0053_hang_watchdog.py::test_watching_restores_the_previous_watchdog'
FAILED tests/test_0053_hang_watchdog.py::test_watching_restores_the_previous_watchdog

Run 33142970081, on feature/pin-y20-relaxation-rate at 6c46c783.

It is flaky, not deterministic

Re-running the same commit with no changes passed. The commit that failed touched only tests/test_1072_free_surface_spherical.py, which scripts/test.sh does not run at all (test_106*py/test_107*py are excluded pending the #504 triage), so the change could not have influenced it. Every other test_uw3 run in the preceding ~15, across seven branches, was green.

Why it is worth a ticket rather than a re-run

The failure mode is a crashed worker, not an assertion — the test process dies rather than reporting. That is the same class as #638's root cause (faulthandler repeat-dump against a hot interpreter loops or SEGVs) and #639's orphan leak, both in this watchdog machinery. An intermittent worker crash under xdist is the symptom you would expect if a dump can still land on an interpreter that is not in a safe state.

Two consequences for anyone reading CI:

• A red run here says nothing about the PR under test, so it invites re-run-until-green, which is how a real regression gets waved through.
• Because the crash takes the worker rather than the test, the tests xdist had scheduled behind it on gw0 do not report either.

Suggested next step

Not a re-run policy — a native stack. Per #638, theorising about this class without one has burned several wrong hypotheses. The reproduction there needed a cold .pyc cache and piped children; worth checking whether the CI runner meets those conditions more often than a warm local run does.

Underworld development team with AI support from Claude Code

[lmoresi]

Second occurrence, next run, different test in the same file:

replacing crashed worker gw0
worker 'gw0' crashed while running 'test_0053_hang_watchdog.py::test_report_carries_the_main_thread_stack'
FAILED tests/test_0053_hang_watchdog.py::test_report_carries_the_main_thread_stack

Run 33145737901, feature/pin-y20-relaxation-rate at 713fc40f.

So it is not one bad test. Across two runs it has taken test_watching_restores_the_previous_watchdog and then test_report_carries_the_main_thread_stack; the constant is the file and the crashed gw0 worker. That points at the watchdog machinery the file exercises rather than at either test's assertions, and it fits the #638 diagnosis — a dump landing on an interpreter that is not in a safe state — better than anything test-specific.

Note the second run also logs replacing crashed worker gw0, so xdist restarted the worker and carried on. Whatever it had scheduled behind it still does not report.

Two runs in a row makes this frequent enough to mislead: the branch under test changes only test_1072, which now runs in its own batch and passed (3 passed, 116.7s), yet the job is red both times for a reason unrelated to it.

Underworld development team with AI support from Claude Code