Skip to content

Fail loudly when browser commands never reach the page - #154

Merged
DavertMik merged 5 commits into
mainfrom
fix/recorder-silent-skip
Aug 28, 2026
Merged

Fail loudly when browser commands never reach the page#154
DavertMik merged 5 commits into
mainfrom
fix/recorder-silent-skip

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

The session that prompted this

Trace d4605420f17fe6240a7cf05e33f6eb91, session RainyGlobalScarlet355 — 3.5 minutes trying to fill one text input, then giving up. Six fillField variants, six clicks, two Escapes and a hover all reported success and left every pageDiff empty.

Two observations from that trace show the commands never reached the browser:

  • I.click({role:'link',text:'RainyGlobalScarlet355 All Tests Run'}) returned success for an element xpathCheck proved absent moments later.
  • verify passed I.seeElement(…) for text that was not on the page. An assertion can only pass by not running.

Cause

recorder.add() returns a resolved promise without invoking the task when the recorder is not running (codeceptjs/lib/recorder.js:198). Every I.* reaches the page only through that call (lib/step/record.js:37), but event.step.before / after fire outside it (:31, :52).

So a stopped recorder still logs and reports every step while executing none. action.lastError stays null, attempt() returns true, and every tool reports success against a page that never changed. Nothing throws, so the restart in executeOnce's catch (src/action.ts:378) is never reached, and the session stays phantom to the end.

The change

One line at the single CodeceptJS dispatch site:

this.lastValue = await returned;
if (!recorder.isRunning()) throw new Error('CodeceptJS recorder is stopped, commands were skipped and never reached the browser');

After dispatch, not before: one check covers a recorder that was already dead and one that died mid-command. The throw lands in the existing catch, which restarts the recorder — so lastError is set, the tool reports an honest failure, and the retry actually executes.

The isPlaywright branch drives the page directly and needs no guard. recorder.stop() has no legitimate mid-session caller: the only two in src/ are both in Explorer.stop().

tests/unit/explorer.test.ts mocks codeceptjs with a partial recorder; isRunning was added to the double.

What this does not do

It does not identify what stopped the recorder in that run. Navigation at 11:01:25 worked and the first fill at 11:01:32 was already a phantom; isolated repros of a failing step recover correctly through Action, and store.dryRun is not involved. The command that kills the channel can still report one false success — only the ones after it turn loud. This makes the next occurrence report a failure with a stack trace instead of a silent 3.5-minute run, which is what will pin the trigger.

Verification

  • tests/unit/action-recorder-recovery.test.ts — new. Fails on main, passes here.
  • bun test tests/unit/ — 1113 pass, 0 fail
  • bun test tests/integration/ — 80 pass, 1 skip, 0 fail
  • bun run lint:fix, bun run format — clean

Separate findings, not fixed here

  • cap() (src/ai/tools.ts:1132) keeps the first 4000/6000 chars of ARIA/HTML. In this session the runs list ate the whole budget and the overlay under test sat in the 91,806 truncated characters — context() returned a snapshot with zero occurrences of run-title, detail-view or New Manual Run. Late-DOM overlays are structurally invisible to the model.
  • store.dryRun is set by dryRunTestFile (src/utils/test-files.ts:85) and never reset — a second silent-no-op path into the same helper layer.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts

DavertMik and others added 5 commits August 28, 2026 03:27
CodeceptJS returns a resolved promise from recorder.add() without
invoking the task when the recorder is not running, while the
step.before/after events still fire. Every I.* command reaches the page
only through that call, so a stopped recorder turns clicks, form fills
and assertions into silent successes: the step is logged, no error is
raised, lastError stays null, and the page never changes.

Session RainyGlobalScarlet355 ran 3.5 minutes that way. Six fillField
variants, six clicks, two Escapes and a hover all reported success and
left every pageDiff empty. A click on a link that did not exist
succeeded, and verify() passed I.seeElement for text that was not on the
page — an assertion can only pass by not running.

Check the recorder after dispatching, so both a channel that was already
dead and one that died mid-command are caught. The throw lands in the
existing catch, which restarts the recorder, so lastError is set and the
retry actually executes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The guard throws into executeOnce's catch, which restarts the recorder,
so the next command reaches the browser. Nothing asserted that, leaving
the recovery an accident of the catch block rather than guaranteed
behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The codeceptjs double in explorer.test.ts answered isRunning() with a
hardcoded true. mock.module is global to the bun test process, so that
answer reached every file loaded after it, and the new guard could never
fire there. CI orders test files differently from a local run, which is
why this passed locally and failed there.

Track start/stop on the double instead, and drop the recovery test that
needed the real recorder's catch semantics: it passed with and without
the guard, so it protected nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The double already defined reset and stop further down the object; the
previous commit added duplicates above them, so the later definitions
won and stop() never cleared the running flag. Biome caught it as
duplicate object keys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
Two process-global leaks the new file tripped, both of which only show
up in the file order CI happens to pick:

- recorder.add('label', fn) is the real recorder's signature; the double
  in explorer.test.ts takes the function first, so the probe task never
  ran. Use the single-argument form both accept.
- Executing an action sets a global activity that clearActivity() clears
  on a timer up to a second later, and Activity.addListener replays
  whatever is current to every new listener. remote.test.ts attaches
  inside that window and saw a stale "Browsing..." frame ahead of its
  own. Force-clear after each test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
@DavertMik
DavertMik merged commit 97a8bbc into main Aug 28, 2026
2 checks passed
@DavertMik
DavertMik deleted the fix/recorder-silent-skip branch August 28, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant