Report each TestRail case result with the case itself - #6658
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the TestRail “submit test plan” data model so each generated test case carries its own execution result (status/summary/failure_reason), eliminating the separate parallel results list and reducing the chance of mismatched case/result sets. It also updates the agent to record the TestRail action directly and updates the UI to render results from the recorded action (with a legacy fallback).
Changes:
- Nest execution results inside each generated test case and validate them via the
testrail.submit_test_planaction schema. - Update the test-plan-generator agent + prompt to record the TestRail action (instead of submitting a separate structured result), and block multiple submissions.
- Update hackbot-ui to parse/render nested per-case results from the recorded action, while still supporting legacy runs.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/hackbot-ui/components/TestPlanView.tsx | Parses nested per-case result, prefers recorded TestRail action params, falls back to legacy results. |
| services/hackbot-ui/components/RunDetail.tsx | Shows Findings panel when a TestRail submit action exists even if findings is empty; passes actions through. |
| services/hackbot-ui/components/FindingsView.tsx | Accepts actions and uses them when parsing the test plan. |
| libs/hackbot-runtime/tests/test_testrail_handler.py | Updates test fixtures to include nested per-case result and removes legacy fields. |
| libs/hackbot-runtime/tests/test_testrail_action.py | Expands tests to cover nested results, sequential IDs, max cases, and single-submission enforcement. |
| libs/hackbot-runtime/hackbot_runtime/actions/testrail.py | Adds per-case result model, sequential-id + max-case validation, optional run summary, and rejects multiple submissions. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/result.py | Removes the old structured result MCP server/model. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/prompts/system.md | Updates agent workflow to record the TestRail action and include nested results. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/config.py | Enables the TestRail submit action type for the agent. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/agent.py | Uses the actions MCP server/tools and asserts the TestRail action was recorded. |
| agents/test-plan-generator/hackbot_agents/test_plan_generator/main.py | Passes the run’s ActionsRecorder into the agent and removes post-hoc recording. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Execution results travelled as a parallel `results` list keyed by case id, so reading a case meant looking up its outcome somewhere else, and the two lists could disagree about which cases exist. Nest the result inside each generated test case instead. Every case now carries a required `result` with its status, summary and failure reason, which makes an unexecuted case unrepresentable rather than merely absent from a second list. The run-wide write-up moves to a `summary` on the action, where it describes the plan as a whole instead of competing with the per-case summaries. The UI reads the nested result and still falls back to the old `results` list so previously recorded runs render unchanged. Fixes mozilla#6511
The opening paragraph described the agent as recording a test plan, dropping the reporting half of the job that the old "report only pass/fail/unsuitable results" line carried. The status vocabulary is still enforced by the tool schema, which exposes status as an enum and rejects anything else, so this is not a correctness fix. It just means the first thing the agent reads describes what it actually does.
jpangas
force-pushed
the
testrail-nested-results
branch
from
August 21, 2026 20:40
c48b7b3 to
7588cbc
Compare
suhaibmujahid
enabled auto-merge (squash)
August 21, 2026 23:26
The step-level `StepResult` model went away with result.py in mozilla#6657, so `TestRailStepInput` now holds only an action and an expectation. The execution rules still told the agent to mark a failing step, which it has no field to do. Point that failure at the case result instead, and have the agent name the offending step in the summary so the attribution the step status used to carry is not lost.
auto-merge was automatically disabled
August 21, 2026 23:59
Head branch was pushed to by a user without write access
suhaibmujahid
enabled auto-merge (squash)
August 22, 2026 00:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6511
Depends on #6657, please merge that first.
What changed
The result is nested inside each generated test case. Every case carries a required
resultwith its status, summary and failure reason, which makes an unexecuted case unrepresentable rather than merely absent from a second list.The run wide write up moves to a
summaryon the action, where it describes the plan as a whole instead of competing with the per-case summaries. It becomes the TestRail run description in #6659.The UI reads the nested result and still falls back to the old
resultslist, so previously recorded runs render unchanged.Stack
test-plan-generatoragent's execution results in TestRail #6436, submit execution results to TestRailI couldn't use the GitHub stack since it requires push access to bases branches on
mozilla/bugbug). So These targetmasterrather than each other (no push access for base branches onmozilla/bugbug), so Files-changed is cumulative until #6657 merges.