Let Pilot, not Planner, name the item a test acts on

[DavertMik]

Problem

Diagnosed from Langfuse trace aa327eeaef94d32e8582a7802368ce92 — a 3m34s run that ended in stop → fail without ever attempting the action it was testing.

The Planner had written a concrete record name into the scenario title. By the time the test ran, that name was not on the page — the closest item differed. The tester noticed the mismatch in its first turn and went looking for the literal anyway, because Pilot's opening plan told it to search for that name:

• 17 click tool calls (20 I.click steps), 4 see, 3 xpathCheck
• three search/clear cycles, each returning "0 tests found"
• one see call answered "yes, it's there" for a name that was not on the page, its own reasoning admitting it had inferred a character it could not read
• the tester did open the right detail overlay — one step from the goal — and Pilot ordered it closed as "the wrong entity"
• two xpathCheck calls then proved the name absent, and the guidance still did not change

The scenario title is written once and read much later. Anything named in it is stale by the time it is used.

Change

Two lines.

src/ai/planner.ts — the scenario field description now asks for the behavior, with the item described by kind and role rather than by title, id, or value.

src/ai/pilot.ts — one line in Pilot's system prompt: scenarios name behavior, not data; Pilot picks the concrete item (from the page, or via precondition()) and gives Tester its exact name.

This moves item selection to the agent that is looking at the page when the test runs. It closes a gap rather than adding machinery: the reset prompt already prefers precondition() when the page lacks target data (pilot.ts:392), the precondition-unavailable fallback already says "continue with what the page already shows" (pilot.ts:737), and the skip-on-missing-data path already exists (pilot.ts:807).

Not addressed here

• src/ai/planner.ts:96 (focusExistingDataDirective) still tells the planner to use item names visible in page research. It fires only under --focus and did not cause this trace.
• The see tool over-committing on a name it could not read is downstream of the hunt.
• I.pressKey(['Meta','a']) on a search input deleted one character per press instead of selecting all — the query shortened by one character on each of three clear attempts.

Testing

• bun test tests/integration/ — 82 pass, 1 skip, 0 fail
• bun run format clean

Prompt-only change; behavior is best confirmed by replaying the scenario. Worth the regression label if you want that coverage.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XsPDpDQrWdgDLGXkzkgGQr