Skip to content

test(voice): repair and re-enable turn-detection tests in CI - #6715

Open
brandonbodily wants to merge 1 commit into
livekit:mainfrom
brandonbodily:fix/reenable-turn-detection-tests
Open

test(voice): repair and re-enable turn-detection tests in CI#6715
brandonbodily wants to merge 1 commit into
livekit:mainfrom
brandonbodily:fix/reenable-turn-detection-tests

Conversation

@brandonbodily

Copy link
Copy Markdown

Summary

Four test modules are marked pytest.mark.audio_eot, covering 83 tests of turn-detection behaviour:

Module Tests
tests/test_audio_recognition_turn_detection.py 29
tests/test_audio_turn_detector_fallback.py 40
tests/test_turn_detection_fsm.py 11
tests/test_turn_detection_cloud_stream.py 3

CI runs make unit-testspytest --unit, which deselects all of them, and no workflow passes --audio_eot:

$ pytest tests/test_audio_recognition_turn_detection.py --unit
29 deselected in 0.06s

$ grep -rn "audio_eot" .github/
(no matches)

None of these 83 tests has been running in CI.

Consequence

test_audio_recognition_turn_detection.py drifted out of sync with the source. On a clean checkout of main, 8 of its 29 fail:

# Test expects Source has
1 AudioRecognition.update_turn_detector _update_turn_detector (renamed private)
2 AudioRecognition.clear_user_turn _clear_user_turn (renamed private)
3 _make_full_recognition_for_eou stub missing _turn_backchannel_over_agent, _stt_consumer_atask
4 log assertion on "already flushed" warning now reads "transcript arrives after turn has been committed"

Each is a one-line repair. The other three modules were already green — they simply never ran.

Changes

  • Repair the four drift points.
  • Add --audio_eot to the unit-tests target so these run alongside --unit.

No library code is touched — only the test module and the make target.

Verification

$ pytest --audio_eot          # before
8 failed, 75 passed

$ pytest --audio_eot          # after
83 passed

pytest --unit and pytest --unit --audio_eot produce identical results across every other test in the suite.

Why this matters beyond the repair

This is the coverage for turn-detection commit policy, which is where several open correctness issues live (#6504, #3515). With the suite green, adding a single assertion to the existing test_late_stt_final_after_flush_short_circuits reproduces #6504 deterministically:

ar._hooks.on_end_of_turn.assert_not_called()
# AssertionError: Expected 'on_end_of_turn' to not have been called. Called 2 times.

That test already drives the exact #6504 scenario — a late STT final after _turn_detector_flushed is set — and asserts on predict, on on_eot_prediction, and on the warning, but never on whether a turn is committed. The sibling test two functions below (test_predict_timeout_signals_fallback_and_drops_future) does assert on_end_of_turn.assert_called_once(), so the pattern is established in this file.

I've deliberately left that assertion out of this PR — it would fail until #6504 is fixed, and PR #6516 is already open against it. This PR is only the CI repair, so it lands independently and green.

Two other markers look dark, but I didn't touch them

While tracing this I noticed plugin (26 modules) and docs (1) are also never selected by any workflow — they're only reachable through tests/Makefile:unit-tests (--unit --plugin --evals --docs), and CI's make unit-tests step runs from the repo root, so it hits the root makefile target instead.

To be clear about what still does work: the plugin matrix isn't idle — tests/Makefile:test runs pytest --tts, and test_tts.py reads the PLUGIN env var (lines 286, 551) to filter its parametrized provider list, so each matrix job genuinely exercises that provider's TTS. What it doesn't run are the 26 dedicated test_plugin_* modules, including the *_stt.py ones.

I've left those out of this PR on purpose — re-enabling 26 provider-integration modules is a credentials-and-CI-cost decision for maintainers, not something to slip into a test repair. Happy to open a separate issue if that's useful.

@brandonbodily
brandonbodily requested a review from a team as a code owner August 5, 2026 18:35
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Four test modules are marked `pytest.mark.audio_eot`, covering 83 tests of
turn-detection behaviour:

  tests/test_audio_recognition_turn_detection.py   29
  tests/test_audio_turn_detector_fallback.py       40
  tests/test_turn_detection_fsm.py                 11
  tests/test_turn_detection_cloud_stream.py         3

CI runs `make unit-tests` -> `pytest --unit`, which deselects all of them, and
no workflow passes `--audio_eot`. None of these 83 tests has been running.

test_audio_recognition_turn_detection.py drifted out of sync with the source as
a result; 8 of its 29 fail on a clean checkout:

- `AudioRecognition.update_turn_detector` -> `_update_turn_detector`
- `AudioRecognition.clear_user_turn` -> `_clear_user_turn`
- the `_make_full_recognition_for_eou` stub was missing
  `_turn_backchannel_over_agent` and `_stt_consumer_atask`
- the "already flushed" log assertion no longer matched the warning text,
  which now reads "transcript arrives after turn has been committed"

This repairs all four drift points and adds `--audio_eot` to the `unit-tests`
target. The other three modules were already green -- they simply never ran.

Verified: `pytest --audio_eot` goes from 8 failed/75 passed to 83 passed.
`pytest --unit` and `pytest --unit --audio_eot` are identical on every other test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@brandonbodily
brandonbodily force-pushed the fix/reenable-turn-detection-tests branch from d0ef2b2 to 3a6664f Compare August 5, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants