Improve Azure DevOps test summaries

[Amaury Levé (Evangelink)]

Summary

• enrich the Azure DevOps job summary with pass rate, current flaky tests, historical pass/fail rates, p95/p99 duration comparisons, and expandable failure diagnostics
• surface MSTest dependency edges by carrying bounded dependency metadata through VSTestBridge
• preserve additive mixed-version summary fragments while bounding dependency data and report-wide expanded diagnostics

Validation

• built Microsoft.Testing.Extensions.UnitTests across its configured target frameworks
• built Microsoft.Testing.Extensions.VSTestBridge.UnitTests across its configured target frameworks
• passed 14 Azure DevOps summary tests, 25 VSTest converter tests, and 21 CI summary aggregation tests

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings remain in summary rendering, lifecycle handling, and aggregation validation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite (auto)
Findings: 1 · 3

Note

Copilot is running an experiment and ran this review at Lite.

Open (4)

• Guard history percentiles before converting to TimeSpan · New
• Use the canonical test name for native MSTest nodes · New
• Backslashes do not escape backticks in inline code · New
• Validate null dependency entries as malformed fragments · New

What changed in this PR

This PR enriches Azure DevOps test summaries with historical metrics, dependency graphs, and expandable failure diagnostics.

Changes:

• Adds pass-rate, flaky-test, duration-history, and failure-detail reporting.
• Propagates bounded MSTest dependencies through VSTestBridge.
• Adds aggregation limits, validation, API tracking, and tests.

File	Reviewed changes and final findings
test/​UnitTests/​Microsoft.Testing.Extensions.VSTestBridge.UnitTests/​ObjectModel/​ObjectModelConvertersTests.cs	Tests dependency transport limits.
test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​AzureDevOpsSummaryReporterTests.cs	Tests enriched summaries, history, retries, and diagnostics.
src/​Platform/​SharedExtensionHelpers/​CiRunSummaryModels.cs	Defines history and dependency metadata.
src/​Platform/​SharedExtensionHelpers/​CiRunSummaryAggregateReader.cs	Validates and bounds fragments. Findings: one nit (1 vote) and two moderate issues (1 and 2 votes).
src/​Platform/​Microsoft.Testing.Extensions.VSTestBridge/​ObjectModel/​ObjectModelConverters.cs	Carries bounded MSTest dependency metadata.
src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​InternalAPI/​InternalAPI.Unshipped.txt	Tracks the new API surface.
src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs	Renders enhanced summaries. Findings: one critical issue (1 vote) and three moderate issues (2, 1, and 1 votes).
src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Lifecycle.cs	Captures history, dependencies, and failure details. Findings: four moderate issues (2, 1, 1, and 1 votes).
src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.cs	Adds bounded reporter state and history integration.
src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsExtensions.cs	Wires shared history services.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved lifecycle, validation, bounding, and rendering issues remain.

Review effort: Lite (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Lite.

Resolved since last review (4)

• Guard history percentiles before converting to TimeSpan
• Validate null dependency entries as malformed fragments
• Backslashes do not escape backticks in inline code
• Use the canonical test name for native MSTest nodes

Previously missed (5)

In code that hasn't changed since last review

Do not mark folded mixed outcomes as recovered flaky tests

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Lifecycle.cs:128

Because records are keyed by uid, folded data-driven rows can share this key. If one shared-UID row is a final failure or skip and a later row passes, the non-passing path removes the only failure marker and this expression can mark the pass as flaky, even though the UID had a mixed final outcome rather than a recovered retry. The GitHub Actions reporter keeps a sticky _notRecoveredTests state and clears flaky records for this case; mirror that behavior here or distinguish final rows.

Fall back to exception messages for blank explanations

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Lifecycle.cs:338

Explanation can be an empty or whitespace-only string. In that case ?? selects the blank value and discards exception.Message, so the expandable summary may show only an exception type and no diagnostic message. The corresponding GitHub capture treats whitespace-only explanations as absent; use the same fallback here.

Use a wider type for historical result totals

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs:287

Each historical count is validated independently as a non-negative int, so a valid fragment can contain int.MaxValue for both counts. Their sum overflows here and can render a negative total or fail while producing the report. Accumulate the two counts as long before formatting.

Disambiguate history and dependency rows across modules

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs:389

When more than one module is aggregated, this helper qualifies rows only with the assembly name. Two target frameworks or architectures of the same test assembly therefore produce indistinguishable history or dependency rows even though their statistics and edges can differ. Include the target framework/architecture (or the full module discriminator used by the section headings) in this label.

Reject null history entries as malformed fragments

src/​Platform/​SharedExtensionHelpers/​CiRunSummaryAggregateReader.cs:149

Unlike the dependency validation below, this new HistoryTests predicate dereferences every element without checking for null. A fragment containing historyTests: [null] therefore throws NullReferenceException instead of being rejected as a FormatException, preventing malformed-fragment handling from working as intended.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical row aggregation and null-validation issues remain, along with nondeterministic dependency truncation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite (auto)
Findings: 2

Note

Copilot is running an experiment and ran this review at Lite.

Open (2)

• Preserve data-driven rows that share a UID · New
• Reject null dependency arrays before bounding · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Six moderate review issues remain unresolved.

Review effort: Lite (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Lite.

Resolved since last review (2)

• Reject null dependency arrays before bounding
• Preserve data-driven rows that share a UID

[github-actions]

Final test-triage resolution — Azure Pipelines build 1600003

Superseding my preliminary comment. Build leg microsoft.testfx (Build Windows Debug) — job "Windows Debug" / task "Test (unit tests only)" failed on both the net462 and .NET Framework 4.8 matrix legs (net8.0/net9.0 legs passed).

Failing test: MSTestAdapter.PlatformServices.UnitTests.Services.AssertionFailureDiagnosticsTests.CaptureShouldAttachBoundedContextForFailedTest

Both legs failed with the same assertion signature: the captured JSON diagnostics artifact was missing the "cpuPercentDuringTest" field.

Not caused by this PR — PR #11337 does not touch TestContextImplementation.AssertionFailureDiagnostics.* or this test file. History shows the same test previously failed on main build 1599509 (net462 leg only), so this is a pre-existing, recurring defect rather than a one-off caused by this change.

Root cause (high confidence): in src/Adapter/MSTestAdapter.PlatformServices/Services/TestContextImplementation.AssertionFailureDiagnostics.Writer.cs, CreateProcessArtifact computes:

double? normalizedCpuPercent = elapsed > TimeSpan.Zero && current.TotalProcessorTime >= start.TotalProcessorTime
    ? Math.Min(100, ...)
    : null;

When the captured test runs fast enough that elapsed rounds to TimeSpan.Zero (common on .NET Framework, which has coarser timer resolution), CpuPercentDuringTest is set to null. Since the [DataMember(..., EmitDefaultValue = false)] attribute on that property omits null values from serialization, the "cpuPercentDuringTest" key is dropped from the JSON entirely, which trips the test's json.Should().Contain("\"cpuPercentDuringTest\"") assertion. This is a deterministic race between measured elapsed time and serialization behavior, not environmental flakiness.

Next step: either always emit cpuPercentDuringTest (drop EmitDefaultValue = false for this property, or emit 0 instead of null when elapsed is effectively zero), or relax the test assertion to tolerate a missing field when elapsed time rounds to zero. Creating a tracking issue since this is a durable, previously recurring defect in the shared framework code.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 127.1 AIC · ⌖ 6.46 AIC · ⊞ 9.4K