Report each command a batch ran, with its status - #150
Merged
Conversation
DavertMik
force-pushed
the
pilot-tool-failure-evidence
branch
from
August 28, 2026 00:32
30acadf to
f61e677
Compare
form() sent many commands and got back one pass/fail, so a command that errored erased the evidence that everything before it had worked. In session OtherYappiestIndigo973 a reload executed and navigated, but a bad locator in the wait bundled after it timed out; the tool reported only "FAILED", and Pilot concluded the reload never happened, rejected finish() three times, and failed a scenario whose verifications passed. attachStepLogger already saw every step and whether it failed — the list was discarded on the error path. The executors now own it: Action keeps it as executedSteps, Navigator accumulates it across the attempts it drives. form() and interact() only read it, reporting each command as OK or FAILED plus how many never ran — in the message (which reaches Pilot's session log) and as structured attempts, the shape click() already returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DavertMik
force-pushed
the
pilot-tool-failure-evidence
branch
from
August 28, 2026 00:43
f61e677 to
2a63acb
Compare
tests/unit/explorer.test.ts replaces the codeceptjs module process-wide via mock.module, and its fake dispatcher only forwarded `emit`'s first argument. `step.failed` carries the error as the second one, so once that mock was loaded, attachStepLogger saw a failed step with no error and recorded it as a success. Locally bun ran executed-steps.test.ts before explorer.test.ts and the test passed; in CI the order is reversed and it failed. The double now forwards all arguments, like the EventEmitter it stands in for. attachStepLogger returns its own detach closure instead of a listener a separate detachStepLogger unregisters, matching captureResponses three lines above it in executeOnce. Neither helper was exported before this branch; only one needs to be now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtBpWHRKXtB7ixUqquurto
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
form()sends many commands and gets back one pass/fail. A command that errors erases the evidence that everything before it worked.Session
OtherYappiestIndigo973(trace33fe2b45ba23ed568a2dcc0769deacb3):The tool reported only
Form execution FAILED!. Pilot concluded the reload never happened, rejectedfinish()three times, and failed a scenario whose own verifications had passed.The fix
attachStepLoggeralready listened to bothstep.passedandstep.failed, so it always saw which commands ran and which one broke — the list was just discarded on the error path.The executors own the information; the tools only read it:
Action.executedSteps—{command, success, error?}, the shapeclick()already returns.Navigator.executedSteps— accumulated across the attempts it drives, alongside the existinglastFailureReason.form()/interact()— read those and report, in themessage(the channel that reaches the tester conversation and Pilot's session log) and as structuredattempts, which pilot'sformatActionsalready reads.No prompt changes: with the commands enumerated, neither the Tester nor Pilot has to infer what ran.
ActionResultwas considered as the carrier and rejected — it is a page-state snapshot, persisted and rebuilt viafromState(WebPageState), and on the failure pathexecuteOncethrows beforecaptureOnce()runs, so the only ActionResult available is the previous state object StateManager owns and reuses.Verification
tests/unit/executed-steps.test.ts— the failing command is recorded withsuccess:false, nothing after it appears, and the report lists OK/FAILED plus how many never ran.bun test tests/unit/ tests/integration/— 1193 pass, 1 skip. The one failure (remote > announces the run…) reproduces on cleanmainunder the full suite and passes standalone: pre-existing ordering issue, unrelated.🤖 Generated with Claude Code