Skip to content

TEST: fix flaky CreateTargetDialog accessibility test - #2335

Open
romanlutz wants to merge 1 commit into
mainfrom
copilot/fix-flaky-target-dialog-test
Open

TEST: fix flaky CreateTargetDialog accessibility test#2335
romanlutz wants to merge 1 commit into
mainfrom
copilot/fix-flaky-target-dialog-test

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

CreateTargetDialog > should keep full long registry names accessible after selecting RoundRobin targets fails intermittently in CI with:

Unable to find an accessible element with the role "button" and name "Remove openai-production-eastus2-red-team-evaluation-primary-deployment"

Seen on PR #2325 (a uv.lock-only dependabot bump, so unrelated to any product change); a plain re-run of the same commit passed.

Root cause. Fluent's Dropdown focus modalizer (tabster) sets aria-hidden="true" on the dialog's ancestors while the listbox popover is open. In a real browser it removes that on close; under jsdom it is not reliably restored. Because getByRole searches the accessibility tree and defaults to hidden: false, anything under an aria-hidden ancestor becomes invisible to it.

The test file already had a restoreDialogAccessibility() workaround, but it was a one-shot snapshot called once at the end of selectTargetType(). If tabster re-applied the attribute afterwards (for example on a later focus change from a subsequent click), nothing stripped it again, and every later *ByRole query in that test went blind. Under CI load the timing shifted just enough for that to happen.

The failure signature matches exactly: the getByLabelText("Selected target: ...") assertion immediately above passed (label queries do no accessibility filtering) while the getByRole("button", { name: "Remove ..." }) assertion failed, even though both elements render from the same JSX block in CreateTargetDialog.tsx and always appear together.

Fix. Replace the point-in-time cleanup with a MutationObserver (watchDialogAccessibility()) that re-runs the cleanup whenever aria-hidden reappears above the dialog, installed in beforeEach and disconnected in afterEach. This is centralized, so future *ByRole assertions in this file do not need to remember a workaround.

The helper is now a no-op while a listbox is actually open, so tabster keeps its real behavior and only the restore that jsdom skips is compensated for.

Notably, the assertions are unchanged. No swap to getByLabelText and no { hidden: true } escape hatch, since the point of the assertion is that the Remove button is genuinely reachable in the accessibility tree.

Scope check: this is the only test file that opens a Fluent listbox popover inside a modal Dialog, so no other file has the same latent race. ChatWindow's converter Combobox is not inside a dialog and therefore has no modalizer.

Tests and Documentation

Test-only change; no product code and no documentation affected, so JupyText was not applicable.

Verification:

  • Deterministic repro: re-adding aria-hidden="true" to a dialog ancestor from a setTimeout right before the failing assertion reproduces the exact CI error with the observer disabled, and passes with it enabled. The temporary sabotage was reverted before committing.
  • Ran the suite 15 consecutive times: 51/51 tests passing on every run, 0 failures.
  • npm run lint and npm run type-check both clean.

Fluent's Dropdown focus modalizer (tabster) puts aria-hidden on the
dialog's ancestors while the listbox popover is open, and under jsdom it
is not reliably restored when the listbox closes. The existing
restoreDialogAccessibility() helper stripped it as a one-shot snapshot at
the end of selectTargetType(), so any aria-hidden re-applied afterwards
(e.g. on a later focus change) left every subsequent *ByRole query blind
- *ByRole defaults to hidden: false. That is what made "should keep full
long registry names accessible after selecting RoundRobin targets" fail
intermittently in CI while the *ByLabelText assertion just above it
passed.

Replace the point-in-time cleanup with a MutationObserver installed per
test that re-runs the cleanup whenever aria-hidden reappears above the
dialog. The helper is now a no-op while a listbox is open, so the
modalizer keeps its real behavior and only the missing restore is
compensated for. Assertions are unchanged - the Remove button is still
required to be reachable by role.

Verified by re-adding aria-hidden to a dialog ancestor from a timer
before the failing assertion: reproduces the exact CI error without the
observer, passes with it. Suite run 15x with no failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 48ec2fc7-3fd7-4188-a3ff-df3cc2a64790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants