fix: guard protocol job thread against DLL exceptions - #80
Conversation
An unguarded exception from j1939_dll.async_job_thread() would fall out of the while loop and silently kill _protocol_job_thread for the ECU's lifetime, permanently stopping TP/BAM timeout handling. Catch and log the exception the same way the dispatch and timer threads already do, and retry after a short backoff instead of terminating. Fixes #76 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The ecu.stop() call for this test's ECU was dropped when a prior commit inserted a new test right after it, causing the ECU's three background threads to leak past the end of the test and intermittently fail the autouse thread-leak fixture in CI. Also explicitly wait for all three threads to exit rather than relying on stop()'s join-with-timeout, whose warn-only behavior on timeout can otherwise race the (shorter) leak-detection window in conftest.py. Fixes the CI failure reported on this PR; see #81 for the general flakiness analysis this pattern guards against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Re #81: the CI failure on this PR turned out to have two separate causes, worth untangling.
This does not fully close #81. The broader issue — |
khauersp
left a comment
There was a problem hiding this comment.
Maybe just a style preference here so I don't mind approving for now
| except Exception: | ||
| logger.exception("Exception in protocol job thread") |
There was a problem hiding this comment.
Small nitpick but it may be good if we can get a more descriptive exception here to help users know more of what was going on
There was a problem hiding this comment.
Fixed in 4236872 — the log message now names the DLL class and states the retry behavior: "%s.async_job_thread() raised; TP/BAM timeout handling will retry in 1s instead of stopping". Thanks for the catch.
Address review feedback on #80: name the DLL class and call that raised, and state the retry behavior, so the log message alone tells an operator what happened without needing to read the source.
Summary
_protocol_job_threadinelectronic_control_unit.pycalledself.j1939_dll.async_job_thread(now)with no exception handling. An uncaught exception there would fall out of thewhileloop and silently terminate the thread for the lifetime of the ECU, permanently stopping all TP/BAM timeout handling — with no log output indicating anything had gone wrong.notify(...)(logger.exception(...)+ continue), consistent with the existing pattern in_timer_job_threadfor callback exceptions.next_wakeupfalls back tonow + 1.0so the loop retries shortly instead of spinning or waiting indefinitely.Fixes #76
Test plan
test_protocol_job_thread_survives_dll_exceptionintest/test_threading.py: monkeypatchesasync_job_threadto raise once, asserts the thread is still alive and gets called again afterward.pytest test/test_threading.py -q— passes (one pre-existing, unrelated flaky thread-leak failure intest_dispatch_queue_drop_on_fullreproduces identically onmaster, not introduced by this change).pytest . --pyargs -q— 528 passed, same pre-existing failure.ruff checkon changed files — clean.