Skip to content

Report each command a batch ran, with its status - #150

Merged
DavertMik merged 4 commits into
mainfrom
pilot-tool-failure-evidence
Aug 28, 2026
Merged

Report each command a batch ran, with its status#150
DavertMik merged 4 commits into
mainfrom
pilot-tool-failure-evidence

Conversation

@DavertMik

@DavertMik DavertMik commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

The bug

form() sends many commands and gets back one pass/fail. A command that errors erases the evidence that everything before it worked.

Session OtherYappiestIndigo973 (trace 33fe2b45ba23ed568a2dcc0769deacb3):

I.amOnPage('/projects/.../suite/28989496')                  // ran, navigated
I.waitForElement('Description Persistence Suite Other…')    // a title, not a locator — 30s timeout

The tool reported only Form execution FAILED!. Pilot concluded the reload never happened, rejected finish() three times, and failed a scenario whose own verifications had passed.

The fix

attachStepLogger already listened to both step.passed and step.failed, so it always saw which commands ran and which one broke — the list was just discarded on the error path.

The executors own the information; the tools only read it:

  • Action.executedSteps{command, success, error?}, the shape click() already returns.
  • Navigator.executedSteps — accumulated across the attempts it drives, alongside the existing lastFailureReason.
  • form() / interact() — read those and report, in the message (the channel that reaches the tester conversation and Pilot's session log) and as structured attempts, which pilot's formatActions already reads.
Form execution FAILED! element (…) still not present on page after 30 sec
  OK I.amOnPage('/projects/.../suite/28989496')
  FAILED I.waitForElement('Description Persistence Suite Other…')

No prompt changes: with the commands enumerated, neither the Tester nor Pilot has to infer what ran.

ActionResult was considered as the carrier and rejected — it is a page-state snapshot, persisted and rebuilt via fromState(WebPageState), and on the failure path executeOnce throws before captureOnce() runs, so the only ActionResult available is the previous state object StateManager owns and reuses.

Verification

  • tests/unit/executed-steps.test.ts — the failing command is recorded with success:false, nothing after it appears, and the report lists OK/FAILED plus how many never ran.
  • bun test tests/unit/ tests/integration/ — 1193 pass, 1 skip. The one failure (remote > announces the run…) reproduces on clean main under the full suite and passes standalone: pre-existing ordering issue, unrelated.

🤖 Generated with Claude Code

@DavertMik
DavertMik force-pushed the pilot-tool-failure-evidence branch from 30acadf to f61e677 Compare August 28, 2026 00:32
@DavertMik DavertMik changed the title Stop Pilot from failing a test because one batched command errored Report each command a batch ran, with its status Aug 28, 2026
form() sent many commands and got back one pass/fail, so a command that
errored erased the evidence that everything before it had worked. In
session OtherYappiestIndigo973 a reload executed and navigated, but a
bad locator in the wait bundled after it timed out; the tool reported
only "FAILED", and Pilot concluded the reload never happened, rejected
finish() three times, and failed a scenario whose verifications passed.

attachStepLogger already saw every step and whether it failed — the list
was discarded on the error path. The executors now own it: Action keeps
it as executedSteps, Navigator accumulates it across the attempts it
drives. form() and interact() only read it, reporting each command as OK
or FAILED plus how many never ran — in the message (which reaches
Pilot's session log) and as structured attempts, the shape click()
already returns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DavertMik
DavertMik force-pushed the pilot-tool-failure-evidence branch from f61e677 to 2a63acb Compare August 28, 2026 00:43
DavertMik and others added 3 commits August 28, 2026 03:44
tests/unit/explorer.test.ts replaces the codeceptjs module process-wide
via mock.module, and its fake dispatcher only forwarded `emit`'s first
argument. `step.failed` carries the error as the second one, so once that
mock was loaded, attachStepLogger saw a failed step with no error and
recorded it as a success. Locally bun ran executed-steps.test.ts before
explorer.test.ts and the test passed; in CI the order is reversed and it
failed. The double now forwards all arguments, like the EventEmitter it
stands in for.

attachStepLogger returns its own detach closure instead of a listener a
separate detachStepLogger unregisters, matching captureResponses three
lines above it in executeOnce. Neither helper was exported before this
branch; only one needs to be now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtBpWHRKXtB7ixUqquurto
@DavertMik
DavertMik merged commit fa07319 into main Aug 28, 2026
2 checks passed
@DavertMik
DavertMik deleted the pilot-tool-failure-evidence branch August 28, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant