Skip to content

Close the latent LLM-booking loophole in research crews - #130

Open
atc964 wants to merge 2 commits into
mainfrom
fix/crew-order-tools
Open

atc964 wants to merge 2 commits into
mainfrom
fix/crew-order-tools

Conversation

@atc964

@atc964 atc964 commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Every channel crew (branding, mobile, CTV, performance) and the social crew placed an execution agent in the crew's agents=[] list, wired with live OpenDirect order-writing tools (CreateOrderTool, CreateLineTool, ReserveLineTool, BookLineTool), even though no task was ever assigned to that agent. All real booking already goes through the deterministic DealBookingFlow / MultiSellerOrchestrator path.

In crewai's hierarchical process, the manager LLM may delegate a task to any agent present in agents=[], whether or not it was ever assigned one. So an idle, live-tooled execution agent sitting in a research crew was one delegation away from writing a real order outside the deterministic booking path. The existing LLM-free CI guard (tests/unit/test_booking_path_llm_free.py) only proved the booking path can't reach an LLM; it said nothing about a crew's own manager LLM reaching booking, since it never inspected crew tool inventories.

Changes

  1. src/ad_buyer/crews/channel_crews.py: removed execution_agent (and its _create_execution_tools wiring) from _build_channel_crew and from create_social_crew. Research/recommendation crews now contain only agents that actually have a task assigned. create_execution_agent itself is untouched — it's still exercised directly by its own unit tests (test_agents.py, test_agent_hierarchy.py, test_llm_pricing_guardrails.py), so it wasn't left dangling and didn't need deletion.

  2. tests/unit/test_booking_path_llm_free.py: added TestNoCrewAgentCarriesAnOrderWritingTool, which builds every production crew (branding, mobile, CTV, performance, social, portfolio) the way the app does and asserts no agent — manager included — carries CreateOrderTool/CreateLineTool/ReserveLineTool/BookLineTool. A second control test cross-checks the tool classes against the crew module's own _create_execution_tools bundle so the guard can't silently go stale if an execution tool is renamed or added. I verified locally that reverting the crew fix makes this test fail with a clear per-agent message.

  3. tests/unit/test_crew_factories.py: updated the branding/mobile/CTV/performance agent-count assertions from 2 to 1, reflecting the removed execution agent.

Test plan

  • uv run ruff check clean on all changed files (3 pre-existing, unrelated ruff findings in demo/ and patches/ confirmed present on origin/main before this change)
  • Full tests/unit suite: 3502 passed, 1 skipped
  • Manually reverted the crew fix and confirmed the new guard test fails with a clear message identifying the offending crew, agent role, and tool class; restored the fix and reran green

…ree booking guard

The four channel crews and the social crew each carried an execution
agent in their agents=[] list, wired with live OpenDirect order-writing
tools (CreateOrderTool, CreateLineTool, ReserveLineTool, BookLineTool)
even though no task was ever assigned to it. In crewai's hierarchical
process the manager LLM may delegate to any agent present in that list,
so this was a live-tooled agent one delegation away from writing a real
order outside the deterministic DealBookingFlow booking path.

Remove the execution agent (and its tool wiring) from every channel
crew and the social crew. All real booking continues to go through
DealBookingFlow / MultiSellerOrchestrator. The create_execution_agent
factory itself is left in place since it is still exercised directly
by its own unit tests.

Extend tests/unit/test_booking_path_llm_free.py with a guard that
builds every production crew and asserts no agent, manager included,
carries an order-writing tool class. A control test cross-checks the
tool classes against the crew module's own execution-tool bundle so
the guard cannot silently go stale.
@atc964
atc964 requested a review from Sirajmx September 14, 2026 19:40

@Sirajmx Sirajmx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: 3612 passed, 83 skipped, 0 failed (+2 from the new guard tests). Confirmed the real
risk this closes: every research crew carried a live, order-writing execution agent
(CreateOrderTool/CreateLineTool/ReserveLineTool/BookLineTool) with no task ever assigned
to it — one manager-LLM delegation away from a real booking outside DealBookingFlow. The new
guard test (TestNoCrewAgentCarriesAnOrderWritingTool) is a real regression check, not just a
snapshot — it inspects actual crew tool inventories rather than asserting agent counts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants