Submit the agent's execution results to TestRail - #6659
Open
jpangas wants to merge 1 commit into
Open
Conversation
This was referenced Aug 19, 2026
There was a problem hiding this comment.
Pull request overview
Extends the TestRail submission flow so the test-plan-generator’s execution outcomes are persisted in TestRail by creating a run for the newly created cases and posting one result per case, and updates the agent schema + UI to consume the new “nested per-case result” structure.
Changes:
- Add TestRail client + handler support for fetching statuses, creating runs, and submitting per-case results (including status mapping and run metadata).
- Update the recordable
testrail.submit_test_planaction schema to require per-case execution results (and prevent recording multiple plans in one run). - Update the UI to prefer rendering the test plan from the recorded TestRail action params and to read results from each case (with legacy fallback).
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/hackbot-ui/components/TestPlanView.tsx | Prefer parsing from recorded action params; render nested per-case results with legacy fallback. |
| services/hackbot-ui/components/RunDetail.tsx | Render FindingsView even when findings are empty but a TestRail test-plan action exists. |
| services/hackbot-ui/components/FindingsView.tsx | Plumb recorded actions into parseTestPlan for action-based rendering. |
| libs/testrail-client/testrail_client/client.py | Add get_statuses, add_run, and add_results_for_cases client methods. |
| libs/testrail-client/tests/test_client.py | Add endpoint wrapper tests for new TestRail client methods. |
| libs/hackbot-runtime/hackbot_runtime/actions/testrail.py | Require per-case execution results in the recordable action; add summary and prevent duplicate recordings. |
| libs/hackbot-runtime/hackbot_runtime/actions/handlers/testrail_handler.py | Resolve status IDs up-front; create a run and submit results for created case IDs. |
| libs/hackbot-runtime/tests/test_testrail_handler.py | Update handler tests to cover statuses, run creation, per-case results, and run-failure behavior. |
| libs/hackbot-runtime/tests/test_testrail_action.py | Update action tests for nested case results, bounds, sequential IDs, and single-submission enforcement. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/result.py | Remove legacy MCP “submit_result” result schema/tool. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/prompts/system.md | Update agent workflow instructions to record the TestRail action with nested results. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/config.py | Restrict enabled actions to testrail.submit_test_plan. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/agent.py | Swap legacy result server for the actions MCP server; enforce TestRail action recording occurred. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/main.py | Pass runtime actions recorder into the agent and remove post-hoc record_test_plan call. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+76
to
+83
| response = await client.get_statuses() | ||
| statuses = response.get("statuses", []) if isinstance(response, dict) else response | ||
| ids = {status["name"]: int(status["id"]) for status in statuses} | ||
| return { | ||
| "passed": ids["passed"], | ||
| "failed": ids["failed"], | ||
| "unsuitable": ids["blocked"], | ||
| } |
Comment on lines
+29
to
+43
| class TestRailCaseResultInput(BaseModel): | ||
| status: Literal["passed", "failed", "unsuitable"] | ||
| summary: str | ||
| failure_reason: str | None = Field( | ||
| default=None, | ||
| description="Required when status is failed or unsuitable.", | ||
| ) | ||
|
|
||
| @model_validator(mode="after") | ||
| def failure_reason_required_for_non_passing_cases( | ||
| self, | ||
| ) -> "TestRailCaseResultInput": | ||
| if self.status in {"failed", "unsuitable"} and not self.failure_reason: | ||
| raise ValueError("failed or unsuitable cases must include failure_reason") | ||
| return self |
Submitting a test plan created a suite of cases and stopped there, so TestRail recorded what the agent generated but never what happened when it ran them, leaving "Test Runs and Results" empty. After the cases are created, create a run over exactly those case ids and post one result per case. Case statuses map onto TestRail's own: passed and failed match by name, and unsuitable posts as Blocked, the closest status a result can carry. The status ids are resolved before anything is created, so an unmapped status fails the submission rather than leaving a half-populated suite behind. The action's summary becomes the run description, and the case summary and failure reason become the result comment. The run id joins the other created ids on the action result. Fixes mozilla#6436
jpangas
force-pushed
the
testrail-run-results
branch
from
August 22, 2026 00:42
b1b11f0 to
0c3e65e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6436
Depends on #6657 and #6658 — please merge those first.
What changed
After the cases are created, the handler creates a run over exactly those case ids and posts one result per case.
passedandfailedmatch TestRail's own status names;unsuitableposts as Blocked, the closest status a result can carry. Statuses are matched on the lowercasename, not the displaylabel.summarybecomes the run description, and each case's summary and failure reason become the result comment.NOte
If run creation fails after the cases are created, the whole submission is marked failed, so a retry duplicates the suite. There is a test pinning that behavior (
test_submit_test_plan_fails_when_the_run_cannot_be_created). Worth a follow-up issue on de-duplicating retries rather than blocking this.Stack
test-plan-generatoragent's execution results in TestRail #6436