Add capability-scoped revocation conformance vectors

[astrogilda]

What changed

This is the separate pull request I promised in #30. It adds revocation conformance vectors for category 3, with a reference adapter that answers them. It also adds a runner that scopes vectors to a declared capability, and negative controls that show the scoring can fail.

Everything lands under conformance/revocation/. It shares no file with #30. The two are reviewable independently and can land in either order.

#30 declared the revocation half of category 3 an open gap. It asked for contributions from implementations that exercise revocation. This is that contribution. I have deliberately not touched anything #30 already claims.

Type of change

Tooling and CI. Nothing under specification/ or docs/spec/ is touched, and no schema, hook, event or AgBOM definition changes.

The two failure codes below live in the new directory's README as a proposed enumeration, the way #30 handles its own. Moving either into normative text is a maintainers' call, and this pull request does not ask for it.

What is here

Nine vectors: six refusals and three positive controls. They cover withdrawal inside the freshness window, an unreachable registry, and a registry stale past its own declared publication interval. They also cover a back-dated revocation window, a re-issued mandate naming no predecessor, and a revoked mandate with no authority-signed instant to measure against.

Two failure codes. MANDATE_REVOKED is a withdrawal that was read and found. REVOCATION_UNCHECKABLE is a withdrawal state that could not be read. Neither collapses into a staleness code. A mandate revoked inside its freshness bound is not old, and calling it stale sends an operator to look at clocks. That is the measured-and-wrong versus could-not-measure rule, applied to a second axis.

A capability mechanism, which answers the integration question I raised on #30. Each vector declares what it needs with requires. Each implementation declares what it has, either as CAPABILITIES in the adapter or as --profile on the command line. The runner runs the intersection and reports the rest as out of profile. An implementation with no revocation mechanism gets an unfilled gap rather than a red result.

The flag is not a way to opt out of being measured. A capability inside the profile that no positive control exercises fails the suite outright. Declaring revocation therefore obliges you to pass the must-pass vectors, not only to refuse the must-reject ones. There is no third state either. An adapter that declares nothing, given no profile, is a fatal error rather than a silent run of everything.

Seven vectors became nine, and the reason is the interesting part

I described seven vectors in my review. There are nine here.

selftest.py mutates the reference adapter, one deleted check per mutant, and requires the suite to notice each one. Two mutants survived against the original seven. One trusted a timestamp the presenting party controls. The other rejected on the presence of an identifier in the revoked list, without ever measuring the withdrawal window.

That second one is a real bug in the fail-closed direction. It refuses every act a later-revoked mandate ever authorized, which breaks recovery.

Both survived because no vector exercised the branch. A surviving mutant is a vector nobody wrote, so I wrote the two. pos-cat3-004 is the control that separates an implementation measuring the window from one rejecting on list membership. I would rather report that the count moved than quietly ship the number I announced.

What the entry-point requirement does, and what it does not

Your entry-point fix is the right mechanism and I kept it. Both constraints from my last comment are in. The runner compares the reported value for equality and never parses it. A blank or absent value is a refusal rather than a pass. Empty equals empty, so the check would otherwise hold vacuously in the case it exists to fire on.

It does not close the answer-key hole. I said I would settle that with a run rather than by reading the diff, so here it is, against your branch at 459ec820:

$ python conformance/negative/runner.py --adapter conformance/negative/reference_adapter.py
18/18 vectors conform (6 positive controls across 6 categories)     exit 0

$ python conformance/negative/runner.py --adapter answer_key.py
18/18 vectors conform (6 positive controls across 6 categories)     exit 0

Here is the whole of the second adapter, which enforces nothing:

def evaluate(vector):
    e = vector["expected"]
    return {"verdict": e["verdict"], "code": e.get("code"),
            "reason": " ".join(e.get("reason_must_mention", [])),
            "entry_point": "acme.enforcement.decide"}

The output is byte for byte the reference adapter's. That is the same result as before the fix, because a self-reported name compared only against itself costs one line to satisfy.

The cheap structural answer is to stop handing the adapter the answer. My runner passes the vector identifier, its category and its input, and keeps the expected verdict and the control flag to itself:

question = {"id": v["id"], "category": v["category"], "input": v["input"]}
out = adapter.evaluate(question) or {}

An adapter that would derive its verdict from the expected one then has nothing to derive it from. The whole family stops being something to detect and becomes something that cannot be expressed. That is why my runner is a separate file rather than a patch to yours. Folding the same change into conformance/negative/runner.py is yours to make whenever #30 is otherwise settled.

One smaller thing, and it is a review note rather than a request. b3ef6a13 fixed the positive-control predicate where the structural gate counts. The entry-point bucketing a few lines below still reads the declared flag directly. A must-reject vector carrying that flag is therefore refused a place in the count, and still lands in the positive bucket for the entry-point comparison. Deciding it once, in one function both sites call, is what I did here.

Verification

$ python conformance/revocation/runner.py --adapter conformance/revocation/revocation_adapter.py
9/9 vectors conform (3 positive controls across 1 categories, profile ['revocation'])

$ python conformance/revocation/selftest.py
15/15 negative controls held

The fifteen are one implementation that must pass and five fake adapters that must be refused. Then seven single-line mutants of the reference adapter that must be killed. Then a capability declared with no positive control, and a vector file edited away from its pinned digest.

A gate nobody has watched fail is indistinguishable from one that is not running. So the suite ships with evidence that it discriminates, rather than the assertion.

MANIFEST.json carries a SHA-256 for every file in the directory. The runner refuses a vector file whose digest does not match. A suite editable between publication and use proves nothing about the implementation that ran it.

CI runs all of it on any change under the new directory. It also validates the vectors against their schema and checks that every file still matches its pinned digest. The workflow is path-scoped, so it stays green independently of anything else in the repository.

Checklist

• Commits are signed off, as the DCO requires.
• Prose follows STYLE.md.
• No secrets, tokens, or internal URLs in the diff.
• The strict mkdocs build passes.

On the last one: nothing here is under docs/ or in the mkdocs config, so the site build takes no input from this change. Say the word if you want the directory in the docs tree, and I will wire it up and run the build.

Security

This change has no security impact. It adds test vectors, a reference adapter, and a runner. Nothing here executes against a live system, and nothing changes what an implementation does in production.

Sankalp.

[avp9-nexus]

Ran it before writing, same as you did for #30. Fresh clone, your branch at c957fdfe, offline sandbox. Six claims from your description, six results.

The suite runs as announced. 9/9 vectors conform (3 positive controls across 1 categories, profile ['revocation']), exit 0 - byte for byte your line. selftest.py: 15/15 negative controls held.

The answer-key family cannot be expressed against your runner, and I proved it with the exact adapter from your description. Your six-line answer-key adapter, run against conformance/revocation/runner.py: 0/9 vectors conform, every vector failing with adapter raised KeyError: 'expected'. The code says why (runner.py:110-112): the adapter receives {"id", "category", "input"} and nothing else. Read and executed, both.

The digest gate bites, and it names both sides. One character edited in revocation_vectors.json: SUITE INVALID: ... digest sha256:505e6af9... does not match pinned sha256:14b9cd83.... Restored the file, 9/9 again. A gate watched failing and watched recovering - which is the evidence your description said it ships with.

Both no-silent-third-state properties hold, with the exit codes to match. An adapter declaring no CAPABILITIES, given no --profile: FATAL: ... An undeclared profile is not an empty one, exit 1 - it refuses to guess rather than silently running everything. A capability inside the profile that no positive control exercises (--profile revocation,attestation): SUITE INVALID: capabilities declared with no positive control: ['attestation'], exit 2. The flag is measurably not a way to opt out of being measured.

And "shares no file with #30" is exact: the intersection of git diff --name-only against main for the two branches is empty, so they do land in either order.

One observation rather than a request: your runner.py:81-87 already decides is_positive_control once, in one function both sites call - the same shape you recommended for #30's second site. The two runners will read as one design when both land, which is worth something to a reviewer arriving cold.

On your #30 verdict: accepted as measured, and acted on. The three changes - the question-without-answer handoff, the single predicate, the dead conditional - are built and proven in an isolated clone of the branch: reference adapter still 18/18 exit 0, your answer-key adapter drops from 18/18 exit 0 to 0/18 exit 1, and positive_control is now read in exactly one place. The README's adapter contract and its answer-key note move in the same change, since both describe the old contract and the branch should not contradict itself. Not pushed yet: your own line was "take it whenever this PR is otherwise settled", and I am keeping to that order.

Avp9

[astrogilda]

This pull request's own reference adapter fails the check I am about to propose, and so does the one in #30. That is the argument for the check, so I am leading with it.

Both adapters declare an entry point ending in decide, and neither tree defines a function by that name. Nothing noticed, because entry_point is free-form and the runner only ever compares it against itself. Under an observed check both score zero, mine at 0 of 9 and #30's at 0 of 18. So 27 of 27 vectors across the two pull requests are currently certified by implementations whose one non-stipulated claim corresponds to nothing that ran.

Nobody acted in bad faith. The field cannot be wrong. That is a different problem and a worse one, because a field that cannot be wrong reads exactly like a field that happens to be right.

The fix is a vantage, not a stricter rule. Arm sys.monitoring on PY_START before the adapter is imported, record the qualified name of every function that executes during each step, and require the declared entry point to appear in the observed set. The adapter runs inside the vantage, so nothing it does in-process suppresses what the monitor records. An out-of-process forgery is a different matter and I come back to it below.

The demonstration is two adapters with identical decision logic, differing only in whether the entry point each declares is one it actually calls, so the check is exercised in both directions on inputs that agree about everything else. The dishonest adapter decides every vector correctly and names a function it never calls. Under the current runner it scores 9 of 9 and exits zero. Under the vantage it scores 0 of 9 and names the step, the declared entry point and how many functions actually ran. An honest one, same logic and a truthful name, scores 9 of 9 either way, so the gate passes as readily as it fails.

In the vocabulary of #36, what changes is that entry_point moves from a claim the observed party makes about itself to one the runner observed from outside it. That is the whole distinction, and it turns out the conformance harness becomes a working instance of the primitive it exists to test for.

Two boundaries.

Of six evasions I tried, four die against the check as described. Two win. Define a real function called decide and call it as a no-op, and out-of-process forgery of the record. The second dies if the runner observes from outside the interpreter. The first does not die at any vantage, because "a function of this name ran" is not "this function decided", and no amount of observation closes the gap between those two sentences. That is a real limit and it is worth writing into the text now; discovering it later costs more.

The second boundary is narrower: the check compares a name against execution and nothing else. It says nothing about whether the function that ran is the one that made the decision, which is the same gap one level up.

I can push the runner, the two demonstration adapters and the six evasions to this branch. Say the word if you would rather see them separately.

[avp9-nexus]

Ran the vantage before writing, on CPython 3.13.15. The demonstration pair is the right shape and the boundary you already name is real. One result goes against the sentence the design rests on, and one constraint lands on your CI.

In-process suppression works. I armed sys.monitoring on PY_START as you describe, then had the observed function call set_events(tool_id, 0) and register_callback(tool_id, PY_START, None). Either one alone blinds the monitor: a witness function invoked immediately afterwards is not recorded. The positive control armed the same way sees three functions when nothing interferes.

It does not depend on the adapter knowing your tool id. use_tool_id accepts 0 through 5 and raises ValueError at 6, so the whole space is six slots. An adapter that sweeps all six and clears each one neutralises the vantage without knowing anything about it: same witness, not recorded.

Checking the state afterwards does not catch it. My first thought was that the runner could read get_events(tool_id) after each step and refuse a step where it no longer matched what was armed. That fails. An adapter that clears the six slots, does its work unobserved, then re-registers the callback and restores the events mask before returning leaves get_events reporting exactly the armed value, while the work it did in between was never recorded. Post-call self-inspection is blind to a restore.

So the gap is not that a function named decide might be a no-op. It is that an in-process vantage cannot attest to its own continuity, which is the same reason a span cannot attest to the runtime that emitted it.

Which is why the answer is already in your own #36 comment: interception records from the substrate rather than the runtime, something the observed party executes inside and cannot suppress, plus a seal at run end proving the vantage stayed armed for the whole run. That seal is precisely the arm-observe-disarm attack you named there, and the sys.monitoring vantage has it. A witness outside the interpreter the adapter runs in keeps the property; the in-process version does not.

The CI constraint, separately. sys.monitoring and PY_START arrived in Python 3.12, and .github/workflows/revocation-conformance.yml on this branch pins python-version: "3.11" at line 24, so the vantage would not arm on the job that runs the suite. Pinning the conformance job to 3.12 and stating the floor in the directory README confines the requirement to the suite. The fallback I would avoid is sys.settrace on 3.11: measured the same way, a traced function that calls sys.settrace(None) silences the tracer from inside, four events on an honest target against two on one that hides, so the 3.11 path would be weaker than the thing it stands in for.

Happy to hand over the probes, they are short and each carries its own positive control.

The answer-key change is on #30 now, at e70f677. I had been holding it for the maintainers to settle the pull request first, per your line about taking it whenever #30 is otherwise settled, and changed my mind for one reason: none of it is new scope. It repairs things the pull request already claims, and one of them is the hole you demonstrated. Leaving a runner in an open pull request that is known to certify an adapter enforcing nothing seemed worse than repairing it while nobody is reviewing.

Four holes, all measured before the change and re-measured against the branch afterwards. Your answer-key adapter now drops from 18/18 exit 0 to 0/18 exit 1 on KeyError: 'expected', and the reference adapter still passes 18/18. An empty vector file used to print 0/0 vectors conform and exit 0, which is the same shape as the field that cannot be wrong; it is now refused. An action naming a required_signer with no evidence at all about the connected device used to reach PASS, because the guard read not in (None, required) and None was in the tuple; absence of a measurement is now UNMEASURABLE rather than a match. And a category-level failure was being subtracted from the vector total, so 17/18 vectors conform could print on a suite whose eighteen vectors all conformed; the two counts are now separate.

One I could not close, and the README now says so instead of promising it. REFERENCE_MISSING was documented as covering a reference that is "absent or zeroed". Nothing in a vector declares that a reference was due, and commit_funds appears in the suite both with one and without on vectors that must pass, so the absent arm is not expressible against the current vector shape. Requiring it by action type breaks two positive controls, which I found by trying. Closing it properly needs a schema field, which is outside what this pull request touches.

Avp9.