fix(cli): say which runs a rate was counted over when filtered runs are set aside (#204)

[xping-admin]

Closes #204.

The defect

#202 taught Vanished to set a dotnet test --filter run aside, and stopped there on purpose. What it left behind is a report that publishes population: allExecutions — rendered all runs, "nothing was set aside" — on the one kind that sets something aside.

PopulationRules.For was not touched by that change (git show --stat c6d073e), so the marker whose whole job is to say which runs a rate was counted over said the opposite of what the provider did. Two shipped documents asserted it in words:

• docs/internals/finding-populations.md — Vanished … "none — nothing is set aside"
• docs/cli/command-reference.md — "an all runs finding carries no discount counts at all — there is nothing for it to report having set aside"

This is the failure PopulationRule's own remark exists to prevent, arriving through the one route it does not guard: an existing kind changing what it counts, rather than a new kind arriving without a decision.

The fix

Vanished maps to a fourth rule, ExcludesPartialRuns, rendered -partial. Eight columns — the same width as all runs — so the trailer budget and the source path are unmoved.

A fourth enum member rather than a second, orthogonal marker on the finding: the dimensions are mutually exclusive in practice. A kind either reads execution outcomes, and must then count every run the test appeared in because a filtered run's outcomes are as true as any other run's, or it reads appearances, where discounting executions means nothing to it. There is no ExcludesEnvironmentalAndClusteredAndPartial waiting to be written. A second marker would also double the comparison a marker exists to make cheap.

LOW   stopped running  Removed
      ran in 14 of 14 earlier full runs, absent from the last 3 full runs
      evidence moderate | -partial | f_cf818034 | SampleTests.cs:10

Measured before changed, as the issue asked

Seventeen full runs of a seventeen-test suite, then three --filter runs of one test, against the same store with the filtered runs removed:

	17 full	+3 filtered
tests	17	17
excludedLowEvidence	0	0
DurationRegression on the planted slow test	present	gone
notMeasured.DurationRegression.awaitingRuns	0	0

The issue's third bullet is false as stated. Filtered runs do not push tests under the per-test evidence floor, and cannot: MeetsReportingFloor reads the subject's own runs, and the window arm beside it is logically implied by the per-test arm while both constants are 5 — so the count a filtered run inflates is not the count that decides. Recorded in finding-populations.md and pinned by a test that fails if either constant moves, because it is true while they are equal and not in general.

What the measurement did find

DurationProvider skips any test with no execution in the current slice, charging it to nothing, on the stated grounds that Vanished reports the disappearance instead. After #202 that is false exactly when it matters: Vanished sets filtered runs aside so it makes no claim about them, so with three filtered runs at the head of the store both duration kinds go silent across the whole suite and the summary goes on reporting that they read every test they were offered. The identical false inference #204 is about, in the one other place it lives.

Where every run in the current slice covered part of the suite, both kinds now count the test as awaitingRuns — a measurement waiting on a run of the whole suite, which is what it is. The predicate is the slice and not the window: one full run in the "now" did ask, so a test missing from it has genuinely stopped and belongs to Vanished again. No arm arithmetic moves; a tally that already existed starts counting.

Deliberately not changed

• healthy, neither recounted nor renamed. Counting it over covering runs breaks tests − healthy = flagged, because a test seen only in filtered runs stays in tests and could not be healthy. The number is unmoved by partiality; what moved it in the measurement was the lost duration finding, now fixed.
• The header, which still says 20 runs. Twenty runs were analysed. The caveat line gains the denominator instead — 3 of 20 runs covered part of the suite — which is the only thing a header change would have added.
• DurationProvider.ReferenceMedian, which handles filtered runs per test and needs no threshold. Unifying it onto IsPartial would discard a valid speed reading for the test that was selected in the filtered run. Recorded as a remark rather than changed.

Schema moves to 1.18: population is a string consumers switch on and there is now a fourth value.

Verification

• dotnet build Xping.Sdk.sln — 0 warnings, 0 errors.
• 986 CLI + 928 Core + 295 adapter tests pass.
• 6 new tests: the marker and that Vanished is the only kind carrying it, the token table, the caveat denominator, the floor invariant, both duration slice shapes, and the issue's reproduction end to end in both directions.
• End to end on a real SampleApp.MSTest store, seventeen full runs then three --filter runs:

Xping | SampleApp.MSTest | 20 runs | 2026-09-10 16:54 -> 16:55
6 findings (5 high, 1 medium) | 21 tests | 14 healthy
* nothing to measure: concurrency 21 | time sensitive 21
! 3 of 20 runs covered part of the suite

notMeasured.DurationRegression.awaitingRuns and .DurationUnstable.awaitingRuns go 0 → 20; partialSessions goes 0 → 3.

One thing worth a reviewer's eye

In that real-store diff, every outcome-reading kind's sessionsConsidered goes 17 → 20 and every headline reads "in N of 20 runs" rather than "in N of 17". That is not a regression — it is the decision #202 recorded, that a filtered run's outcomes are as true as any other run's, and it was already the behaviour before this change. What is new is that each finding now carries a marker saying so, instead of leaving a reader to infer which of two denominators they are looking at.

docs/internals/implementation-specs/ is untracked on main; I amended cli-report-format-spec.md there to record the fourth token, but left it out of the commit since tracking those specs is a separate call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TMQcZLBGFTd8wz3mcfn9vc

[codecov]

Codecov Report

❌ Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/Xping.Cli/Report/Providers/DurationProvider.cs	90.00%	0 Missing and 1 partial ⚠️

Files with missing lines	Coverage Δ
src/Xping.Cli/Report/AnalysisContext.cs	90.00% <ø> (ø)
src/Xping.Cli/Report/Contract/ReportEnvelope.cs	100.00% <ø> (ø)
src/Xping.Cli/Report/Model/PopulationRule.cs	88.88% <100.00%> (ø)
src/Xping.Cli/Report/Rendering/ReportVocabulary.cs	93.65% <100.00%> (+0.66%)	⬆️
...c/Xping.Cli/Report/Rendering/TextReportRenderer.cs	88.23% <100.00%> (+2.67%)	⬆️
src/Xping.Cli/Report/Providers/DurationProvider.cs	97.12% <90.00%> (-0.21%)	⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Requested documentation and API contract wording updates remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates filtered-run reporting with a new -partial population marker, improved duration handling, schema 1.18, and related tests and documentation.

Changes:

• Adds ExcludesPartialRuns / -partial.
• Tracks duration findings awaiting full-suite runs.
• Updates caveats, schema, tests, and documentation.

File summaries

File	Summary
tests/Xping.Cli.Tests/Report/ShareableOutputTests.cs	Tests markers and caveat denominators.
tests/Xping.Cli.Tests/Report/ReportVocabularyTests.cs	Tests population token coverage.
tests/Xping.Cli.Tests/Report/ReportEnvelopeTests.cs	Tests schema and filtered-run behavior.
tests/Xping.Cli.Tests/Report/EvidenceLevelResolverTests.cs	Pins evidence-floor invariants.
tests/Xping.Cli.Tests/Report/DurationProviderTests.cs	Tests partial-slice duration behavior.
tests/Xping.Cli.Tests/Commands/CliSurfaceTests.cs	Updates CLI schema assertions.
src/Xping.Cli/Report/Rendering/TextReportRenderer.cs	Renders partial-session caveats.
src/Xping.Cli/Report/Rendering/ReportVocabulary.cs	Maps the new population token.
src/Xping.Cli/Report/Providers/DurationProvider.cs	Tracks awaiting runs for partial slices.
src/Xping.Cli/Report/Model/PopulationRule.cs	Adds the partial-run rule.
src/Xping.Cli/Report/Contract/ReportEnvelope.cs	Bumps the report schema.
src/Xping.Cli/Report/AnalysisContext.cs	Documents partial-session semantics.
docs/known-limitations.md	Documents duration limitations.
docs/internals/finding-populations.md	Updates population rules and rationale.
docs/cli/command-reference.md	Documents the new marker and schema value.

Review details

Suppressed comments (5)

docs/cli/command-reference.md:105

• This sample now labels the finding -partial, but its headline still says earlier runs/last 3 runs. When Vanished has a partial run to set aside, the renderer emits full runs in both clauses; otherwise the example contradicts the output it is documenting. Please update the sample wording to match the new marker.

      ran in 12 of 17 earlier runs, absent from the last 3
      evidence moderate | -partial | f_1d77e3f5 | .../ImportTests.cs:41

docs/cli/command-reference.md:365

• The new excludesPartialRuns description says the population covers all counts inside evidence, but VanishedEvidence.ExecutionsInWindow intentionally remains a whole-window count, including partial sessions; only the appearance/rate denominator is restricted to covering runs. Narrow this to the rate/denominator (or explicitly call out the exception) so consumers do not misinterpret that field.

Every finding carries a `headline` — the same sentence the rendered report prints — plus `metrics`, the labelled pairs behind it, and the raw `evidence` the two were resolved from. It also carries `population`, which is one of `allExecutions`, `excludesEnvironmental`, `excludesEnvironmentalAndClustered` or `excludesPartialRuns` and says which executions — or, for `excludesPartialRuns`, which runs — the counts inside `evidence` were taken over, and `evidenceSessions`, the number `evidenceLevel` was banded from.

docs/known-limitations.md:363

• The new heading is missing the relationship between the test and the selected run; please add “When No Recent Run Has Selected It” so the limitation is grammatically clear.

### The Duration Kinds Cannot Measure A Test No Recent Run Selected, And Say So Rather Than Reporting It

docs/known-limitations.md:369

• The final sentence overstates the new behavior: one full-suite run only removes the currentSliceIsAllPartial special case. DurationRegression still requires three current comparable sessions (MinimumCurrentSessions), so after one full run it can remain in awaitingRuns; DurationUnstable can also still be awaiting or unreadable. Please say that a full-suite run makes the test eligible for the normal duration gates, after which the usual evidence requirements apply.

tests are counted in `summary.notMeasured.DurationRegression.awaitingRuns` and
`.DurationUnstable.awaitingRuns`, and one run of the whole suite restores them.

src/Xping.Cli/Report/AnalysisContext.cs:102

• This new remark says the window's session count is “the one every rate below is read against,” but the -partial marker introduced here means Vanished rates use only full runs, while the other markers use execution populations. Please describe this as the denominator of the caveat/header and leave each finding's population marker as the rate denominator.

    /// Reported in the summary as an observation rather than a discount, against the window's own
    /// run count so the reader has the fraction and not just the fact. Only the kinds that read
    /// absence set these sessions aside — every other kind reads outcomes of executions that
    /// happened, and a filtered run's outcomes are as true as any other's — so the line says what
    /// the window contains and does not claim every number was adjusted.

• Files reviewed: 15/15 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[xping-admin]

Valid, and verified — thank you. select(.population != "allExecutions") did start matching Vanished the moment it left allExecutions, and the projection asked it for executionsConsidered and the two execution discounts, none of which it publishes. The output was a row of nulls.

Fixed in 8ec1028 by naming the execution populations positively rather than excluding one, and adding a second example for the run-based kind over the fields it actually carries. Both run against a real envelope:

$ jq ... select(.population | startswith("excludesEnvironmental")) ...
{"kind":"Flaky","population":"excludesEnvironmentalAndClustered","evidence":{"executionsConsidered":17,"discountedEnvironmental":0,"discountedClustered":0}}

$ jq ... select(.population == "excludesPartialRuns") ...
{"kind":"Vanished","population":"excludesPartialRuns","evidence":{"baselineSessions":14,"baselineSessionCount":14,"currentSessionCount":3,"partialSessionsSetAside":3}}

The line 365 half of this is addressed too, in the same commit — see the reply on that thread.

[xping-admin]

Valid — the matrix gained a run-based rule and the prose around it did not follow. Fixed in 8ec1028.

The title is now "Which executions, or runs, each finding kind counts", and the opening paragraph says which kinds do which and why: most count executions, Vanished counts runs because it reads appearances rather than outcomes. The introduction to the not-measured section at the old line 158 had the same problem and is fixed with it.

[xping-admin]

Valid — PopulationRule's own summary was widened in the first commit and FindingDto.Population was missed. Fixed in 8ec1028.

Taken slightly further than asked, because reviewing it surfaced the sharper version of the same problem (also raised on command-reference.md:365): the field qualifies the denominator the rate was counted out of, not every figure in evidence beside it. VanishedEvidence.ExecutionsInWindow is deliberately a whole-window count with nothing set aside — "how often did this test run" and "over how many runs was its absence measured" are different questions, and folding both under one marker would make it mean less rather than more. The contract now says so explicitly instead of leaving the exception to be found.

[xping-admin]

Went through all eight review comments. Six were valid and are fixed in 8ec1028; one was already covered; one I am declining, with reasoning below. No behaviour changed — 986 CLI + 928 Core tests still pass.

The three inline comments are answered on their own threads. The five suppressed ones have no thread to reply to, so:

docs/cli/command-reference.md:365 — population over-claims its scope. Valid, and the sharpest comment of the set. VanishedEvidence.ExecutionsInWindow is deliberately a whole-window count including partial sessions; only the appearance denominator is restricted to covering runs. Saying "the counts inside evidence" was therefore wrong for exactly the kind this PR adds. Narrowed to the published rate, with the exception called out by name, in both the command reference and the FindingDto.Population contract.

docs/known-limitations.md:369 — "one run of the whole suite restores them" overstates it. Valid. A full run lifts the currentSliceIsAllPartial case and nothing else; DurationRegression still wants 7 and 3 comparable runs across its arms and DurationUnstable more than one normalised reading, so a test can clear this and stay in awaitingRuns for the ordinary reason. Reworded to say the run makes the test eligible again, after which the usual gates apply.

docs/known-limitations.md:363 — heading grammar. Taken, as "…Cannot Measure A Test When No Recent Run Has Selected It, And Say So".

src/Xping.Cli/Report/AnalysisContext.cs:102 — the "denominator every rate is read against" claim. Valid, and the phrase is one line off from where it was quoted — it lives in TextReportRenderer.WriteCaveats, not in the AnalysisContext remark. Fixed at its actual site, and the correction is the one suggested: the caveat line scopes the window, and each finding's population marker scopes that finding's rate. The AnalysisContext remark already said the numbers were not all adjusted and already pointed at the marker, so it is left as it is.

docs/cli/command-reference.md:105 — the sample shows -partial beside a headline reading "earlier runs" rather than "full runs". Declining this one. The pairing is correct, and it is correct for the reason the same page gives two sections earlier:

The marker states the rule, not whether it fired. A finding marked -env-cluster in a window with no outages and no clusters has had nothing removed from it, and its denominator is simply how many times the test ran.

-partial is on every Vanished finding, because a marker emitted only where something was set aside would leave a reader unable to tell counted everything from this build did not say — which is the argument the marker mechanism rests on. The full runs wording, by contrast, is conditional on PartialSessionsSetAside > 0, deliberately: in a store with no filtered runs there is nothing for the word to distinguish the runs from, and the shorter sentence is the true one. So -partial next to "earlier runs" is precisely a window that holds no filtered run, which is the common case and the right thing for a sample to show. Changing the sample to full runs would document a store the marker does not imply, and would quietly contradict the rule above it.

Happy to revisit that last one if you read the sample differently.

[xping-admin]

Code review found one real defect in the duration change. Fixed in 5b088da.

The arm decided from a property of the window, not of the test. currentSliceIsAllPartial asked "is every run in the current slice partial?" and then charged awaitingRuns for every test missing from that slice. But such a slice holds two kinds of absent test at once, and only one of them is waiting on anything:

• a test a --filter passed over — still runs whenever the suite is run in full;
• a test that had already stopped — which Vanished re-slices over the covering runs precisely so it can still report.

Charging both put every vanished test in the tally and in a Vanished finding. That is the same disappearance under two names, which is exactly what the skip exists to avoid, and what both its own comment and finding-populations.md claimed it did avoid.

The PR's own test had baked it in, which is the part worth flagging: FilteredRunsMoveWhatCouldBeMeasuredAndNotTheEvidenceFloor asserted awaitingRuns == 16 over a 17-test suite where the 16 were the fifteen Stable tests plus Removed — the vanished test the very same test asserts is reported as a Vanished finding. Confirmed against a stored envelope before changing anything:

{"vanished":["Removed"],"durationAwaiting":16,"tests":17}

The fix asks per test, against the runs that did cover the suite: a test the most recent of those ran is waiting on a full run; a test they asked about and did not find has genuinely gone and stays with Vanished alone. The set comes from AnalysisWindowSlices.From — the same re-split Vanished reads, so the two cannot drift on which runs were in a position to ask — and it is derived only when the slice is wholly partial, which is the only time the distinction arises.

The envelope test drops to 15 and now also pins that the one missing is Removed. A new provider test stops a test three covering runs before the filtered tail and asserts it is absent from the tally; it fails at 12 against the old per-slice predicate and passes at 11 against this one, so it pins the distinction rather than the arithmetic.

987 CLI + 928 Core tests pass. End to end on the real SampleApp.MSTest store the figure is unchanged at 20, because nothing vanished there — which is the point: this only ever removes tests another kind is already reporting.

Everything else in the review came back clean: the marker plumbing end to end, the MeetsReportingFloor parameter order, the -partial trailer width, the schema bump, and that the caveat's "N of M" cannot invert since both numbers derive from window.Sessions.