Improve Azure DevOps test summary presentation

[Amaury Levé (Evangelink)]

Azure DevOps renders uploaded test summaries as Markdown cards in the Extensions tab, but the current output is difficult to scan: cards use generated filenames, metrics are vertically expanded, and multi-module runs repeat large sections.

This change optimizes the supported Markdown experience:

• uploads summaries with friendly card names using Distributedtask.Core.Summary
• replaces vertical metric tables with compact status strips and smaller headings
• adds clear pass/fail icons and reports nonzero exit codes without routine noise
• renders a concise multi-module table with failures first and deterministic ordering
• focuses detailed sections on failing classes, failed tests, coverage, and the globally slowest tests
• avoids HTML-only collapsible elements that Azure DevOps does not reliably render
• updates CLI help, package documentation, tests, and generated localization files

The Markdown and self-contained HTML report options remain independent and can be enabled together. HTML is still emitted as a report file; this change does not misuse or replace the native Code Coverage tab.

Before and after

Before

After

Validation included the focused Azure DevOps summary unit tests, the all-extensions help acceptance tests, and a repository build/pack.

[Copilot]

Copilot review overview

🟡 Changes recommended

Markdown encoding and ambiguous module labels can produce misleading summaries.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced (auto)
Findings: 2

Note

Copilot is running an experiment and ran this review at Balanced.

Open (2)

• Identify architecture in global slowest entries · New
• Do not HTML-encode text inside code spans · New

What changed in this PR

Improves Azure DevOps test summaries with compact Markdown dashboards and friendly attachment names.

Changes:

• Adds status strips, failure-first module tables, and global slowest tests.
• Uses named Distributedtask.Core.Summary attachments.
• Updates documentation, localization, CLI expectations, and tests.

Review coverage

Reviewed rendering, attachment commands, localization, compatibility, tests, and documentation.

File	Description
azure-pipelines.yml	Updates summary location documentation.
AzureDevOpsSummaryReporter.Markdown.cs	Redesigns Markdown rendering.
AzureDevOpsSummaryReporter.Lifecycle.cs	Adds exit codes and friendly attachment names.
AzureDevOpsSummaryArtifactPostProcessor.cs	Names aggregate summary attachments.
PACKAGE.md	Documents summary and HTML behavior.
AzureDevOpsResources.resx	Updates CLI description.
AzureDevOpsResources.cs.xlf	Regenerates Czech localization.
AzureDevOpsResources.de.xlf	Regenerates German localization.
AzureDevOpsResources.es.xlf	Regenerates Spanish localization.
AzureDevOpsResources.fr.xlf	Regenerates French localization.
AzureDevOpsResources.it.xlf	Regenerates Italian localization.
AzureDevOpsResources.ja.xlf	Regenerates Japanese localization.
AzureDevOpsResources.ko.xlf	Regenerates Korean localization.
AzureDevOpsResources.pl.xlf	Regenerates Polish localization.
AzureDevOpsResources.pt-BR.xlf	Regenerates Portuguese localization.
AzureDevOpsResources.ru.xlf	Regenerates Russian localization.
AzureDevOpsResources.tr.xlf	Regenerates Turkish localization.
AzureDevOpsResources.zh-Hans.xlf	Regenerates Simplified Chinese localization.
AzureDevOpsResources.zh-Hant.xlf	Regenerates Traditional Chinese localization.
HelpInfoAllExtensionsTests.cs	Updates CLI output expectations.
AzureDevOpsSummaryReporterTests.cs	Updates Markdown and attachment tests.
CiRunSummaryAggregationTests.cs	Updates aggregate summary assertions.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Failed or truncated aggregate runs can incorrectly display a successful heading, and one ordering test can pass when a module row is absent.

Review effort: Balanced (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Balanced.

Resolved since last review (2)

• Do not HTML-encode text inside code spans
• Identify architecture in global slowest entries

Previously missed (2)

In code that hasn't changed since last review

Do not show a successful icon for failed or truncated runs

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs:126

When an authoritative exit code is present, this branch ignores both aggregate.FailedTests and aggregate.IsPartial. A valid summary with failed tests and an ignored exit code of 0 therefore gets a green heading while its status strip and module rows report failures; a truncated run can likewise appear successful. Include those run-level failure signals before consulting the exit verdict.

Require both module rows before comparing their positions

test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​AzureDevOpsSummaryReporterTests.cs:331

IndexOf returns -1 when a row is absent, so this ordering assertion still passes if the failed-module row disappears (-1 is before the passing row). Assert that both names were found before comparing their positions so the test detects either omission.

[github-actions]

Final test-triage resolution — Azure Pipelines build 1599763

This is the final resolution for the completed build (analysis mode: full). Two Windows failures were found among 83 candidate results across all legs; everything else passed or was within normal duration variance.

1. CaptureShouldAttachBoundedContextForFailedTest (Windows Debug, .NET Framework 4.8 leg) — actionable, tracked as a new issue.
Root cause is deterministic: TestContextImplementation.AssertionFailureDiagnostics.Writer.CreateProcessArtifact only computes CpuPercentDuringTest when elapsed > TimeSpan.Zero. When the captured test finishes fast enough that the Stopwatch-measured elapsed time rounds to exactly zero, the field stays null and is dropped from the JSON payload (EmitDefaultValue = false), so the test's json.Should().Contain("\"cpuPercentDuringTest\"") assertion fails. The same signature (same message, "elapsedMilliseconds":0) also occurred on main build 20260916.21 on the net462 leg, confirming recurrence across builds/TFMs rather than a one-off. Filed as a new issue with the fix guidance (default CpuPercentDuringTest to 0 instead of null for zero-duration windows, or adjust the test contract).

2. Dispose_FromANotificationHandler_DoesNotSelfWaitOnTheReadLoop (Windows Release, net8.0 leg) — environmental one-off, no issue created.
Actual elapsed time was 6896 ms against a 4000 ms bound. Historical samples (4 builds, net8.0 Release leg) show durations of 14–402 ms with no prior failures, and this exact disposal-timing scenario was already hardened by #11074 (closed 2026-09-07). A single outlier against a machine/agent-load-sensitive bound on a loaded Windows Release job does not meet the recurrence or regression threshold for a new issue; no code change is indicated at this time. If this recurs on a future build with the same signature, it should be reopened/escalated.

No crash dumps or hangs were present (CrashDumps artifacts were empty on both Windows jobs); no slowness regressions met the historical threshold (0 of 81 slow candidates flagged).

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 197 AIC · ⌖ 7.3 AIC · ⊞ 9.4K

[Copilot]

Copilot review overview

🟡 Changes recommended

Attachment-name collisions can discard summaries, and non-authoritative runs can appear successful.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced (auto)
Findings: 3 · 1

Note

Copilot is running an experiment and ran this review at Balanced.

Open (4)

• Make aggregate attachment names unique · New
• Disambiguate fallback summary attachment names · New
• Use an unknown status for non-authoritative summaries · New
• Avoid reporting the per-module exit code twice · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Aggregate summaries duplicate per-module slow tests and flatten the failure heading hierarchy.

Review effort: Balanced (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Balanced.

Resolved since last review (4)

• Use an unknown status for non-authoritative summaries
• Disambiguate fallback summary attachment names
• Make aggregate attachment names unique
• Avoid reporting the per-module exit code twice

Previously missed (2)

In code that hasn't changed since last review

Nest failure diagnostics under their module heading

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs:145

The module heading above this call is level 4, but AppendModuleDiagnostics now emits headings at exactly the supplied level, so “Top failing classes”, “Failure details”, and the next module all become sibling level-4 headings. With multiple failed modules, the document outline no longer associates diagnostics with their module, which makes screen-reader navigation ambiguous. Pass heading level 5 here and update the corresponding expectation.

Avoid repeating per-module slowest tests in the aggregate

src/​Platform/​Microsoft.Testing.Extensions.AzureDevOpsReport/​AzureDevOpsSummaryReporter.Markdown.cs:148

This aggregate call still lets AppendModuleDiagnostics emit each failed module's SlowestTests, after which lines 152–175 render the global top 10 again. A run with many failing modules therefore repeats tests and can add up to ten local slow-test rows per module, contrary to the advertised globally-slowest-only summary. Add a control that disables the helper's slowest-test section for aggregate failure diagnostics while retaining it for direct per-module summaries.