Fail loudly when browser commands never reach the page - #154
Merged
Conversation
CodeceptJS returns a resolved promise from recorder.add() without invoking the task when the recorder is not running, while the step.before/after events still fire. Every I.* command reaches the page only through that call, so a stopped recorder turns clicks, form fills and assertions into silent successes: the step is logged, no error is raised, lastError stays null, and the page never changes. Session RainyGlobalScarlet355 ran 3.5 minutes that way. Six fillField variants, six clicks, two Escapes and a hover all reported success and left every pageDiff empty. A click on a link that did not exist succeeded, and verify() passed I.seeElement for text that was not on the page — an assertion can only pass by not running. Check the recorder after dispatching, so both a channel that was already dead and one that died mid-command are caught. The throw lands in the existing catch, which restarts the recorder, so lastError is set and the retry actually executes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The guard throws into executeOnce's catch, which restarts the recorder, so the next command reaches the browser. Nothing asserted that, leaving the recovery an accident of the catch block rather than guaranteed behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The codeceptjs double in explorer.test.ts answered isRunning() with a hardcoded true. mock.module is global to the bun test process, so that answer reached every file loaded after it, and the new guard could never fire there. CI orders test files differently from a local run, which is why this passed locally and failed there. Track start/stop on the double instead, and drop the recovery test that needed the real recorder's catch semantics: it passed with and without the guard, so it protected nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
The double already defined reset and stop further down the object; the previous commit added duplicates above them, so the later definitions won and stop() never cleared the running flag. Biome caught it as duplicate object keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
Two process-global leaks the new file tripped, both of which only show
up in the file order CI happens to pick:
- recorder.add('label', fn) is the real recorder's signature; the double
in explorer.test.ts takes the function first, so the probe task never
ran. Use the single-argument form both accept.
- Executing an action sets a global activity that clearActivity() clears
on a timer up to a second later, and Activity.addListener replays
whatever is current to every new listener. remote.test.ts attaches
inside that window and saw a stale "Browsing..." frame ahead of its
own. Force-clear after each test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The session that prompted this
Trace
d4605420f17fe6240a7cf05e33f6eb91, sessionRainyGlobalScarlet355— 3.5 minutes trying to fill one text input, then giving up. SixfillFieldvariants, six clicks, two Escapes and a hover all reported success and left everypageDiffempty.Two observations from that trace show the commands never reached the browser:
I.click({role:'link',text:'RainyGlobalScarlet355 All Tests Run'})returned success for an elementxpathCheckproved absent moments later.verifypassedI.seeElement(…)for text that was not on the page. An assertion can only pass by not running.Cause
recorder.add()returns a resolved promise without invoking the task when the recorder is not running (codeceptjs/lib/recorder.js:198). EveryI.*reaches the page only through that call (lib/step/record.js:37), butevent.step.before/afterfire outside it (:31,:52).So a stopped recorder still logs and reports every step while executing none.
action.lastErrorstays null,attempt()returns true, and every tool reports success against a page that never changed. Nothing throws, so the restart inexecuteOnce's catch (src/action.ts:378) is never reached, and the session stays phantom to the end.The change
One line at the single CodeceptJS dispatch site:
After dispatch, not before: one check covers a recorder that was already dead and one that died mid-command. The throw lands in the existing catch, which restarts the recorder — so
lastErroris set, the tool reports an honest failure, and the retry actually executes.The
isPlaywrightbranch drives the page directly and needs no guard.recorder.stop()has no legitimate mid-session caller: the only two insrc/are both inExplorer.stop().tests/unit/explorer.test.tsmockscodeceptjswith a partial recorder;isRunningwas added to the double.What this does not do
It does not identify what stopped the recorder in that run. Navigation at 11:01:25 worked and the first fill at 11:01:32 was already a phantom; isolated repros of a failing step recover correctly through
Action, andstore.dryRunis not involved. The command that kills the channel can still report one false success — only the ones after it turn loud. This makes the next occurrence report a failure with a stack trace instead of a silent 3.5-minute run, which is what will pin the trigger.Verification
tests/unit/action-recorder-recovery.test.ts— new. Fails onmain, passes here.bun test tests/unit/— 1113 pass, 0 failbun test tests/integration/— 80 pass, 1 skip, 0 failbun run lint:fix,bun run format— cleanSeparate findings, not fixed here
cap()(src/ai/tools.ts:1132) keeps the first 4000/6000 chars of ARIA/HTML. In this session the runs list ate the whole budget and the overlay under test sat in the 91,806 truncated characters —context()returned a snapshot with zero occurrences ofrun-title,detail-vieworNew Manual Run. Late-DOM overlays are structurally invisible to the model.store.dryRunis set bydryRunTestFile(src/utils/test-files.ts:85) and never reset — a second silent-no-op path into the same helper layer.🤖 Generated with Claude Code
https://claude.ai/code/session_01CW3rvmre6ZEbCh3LX5jhts