The formal gates printed FAILED and exited 0 - #787
Merged
Conversation
CI reads the process status, not the report text. All three
dsm_vertical_validation steps summarized their own failures and
returned Ok, so a violated invariant, a falsification config that
stopped falsifying, a failed bridge property and a failed
implementation trace were every one of them GREEN.
Verified before changing anything:
collect_tla_results computes all_passed, eprintln!s
"FAILED specs: {}", and returns Ok(results) unconditionally.
run_tla_check returns Ok(results). main does
`run_tla_check(...).await?` and falls through to Ok(()).
main.rs contains no process::exit, no ExitCode and no bail! on a
verdict. The CI step is a bare `cargo run ... -- tla-check` with no
grep on the log.
run_property_tests and run_implementation_traces return () --
they collect, render and return, with no failure check at all.
So the only things that could redden those jobs were a missing java,
a missing .tla/.cfg, or a build failure.
WHAT WAS ALREADY CORRECT, and is worth stating precisely: the
expected-violation SEMANTICS were fully implemented. run_all already
normalizes every verdict and inverts the falsification configs --
expected invariant violated is a pass, no violation is "the invariant
is decoration", a different violation is "a different failure". None
of that changes here. The defect was never the classification; it was
that nothing turned the aggregate into an exit status.
This adds:
enforce(gate, total, failing) -- bails with the failing labels.
Called AFTER the report prints, so the failing spec and the
invariant it violated stay visible in the log.
tla-check, property-tests and implementation-traces all route
through it. property-tests and implementation-traces now return
Result and their call sites propagate.
EXPECTED_STANDARD_SPECS = 13, asserted in run_all. The TLA registry
is Rust, not a glob, so a spec deleted from standard_specs leaves
nothing for a file count to notice -- the suite just gets smaller
and stays green. This is the analogue of the Lean gate's
expected=12, which had no TLA counterpart.
Two unit tests: the registry size, and that every falsification
config names an invariant its own spec declares (a typo there
degrades silently into "a different failure", which reads as a
defect in the model rather than in the registry).
MUTATION CONTROLS, all executed, all restored:
A ordinary spec pointed at a falsifying config
-> exit 1, "FAILED specs: EconRegisterObservation"
B falsification config neutered so nothing violates
-> exit 1, "must violate EmptinessIsGrounded, but saw no
violation at all - the invariant is decoration"
C falsification config names the WRONG invariant
-> exit 1, "saw a different failure: ConflictUnreachable"
D unmutated 13-spec suite
-> exit 0, "All 13 specs PASSED"
E one spec dropped from the registry
-> exit 1, "registers 12 specs, expected 13"
F one bridge property forced to fail
-> exit 1, "1 of 7 FAILED - hash_chain_continuity"
G one implementation trace forced to fail
-> exit 1, "1 of 16 FAILED - state_machine_transfer_chain"
The baseline suite is genuinely green, so this does not paper over a
red spec -- it makes 13 specs, 7 properties and 16 traces load-bearing
for the first time.
Amendment 2c-C2's record is corrected in the same change. It said the
five falsification configs "are machine-gated on the invariant each
must violate". That was true of the runner's bookkeeping and false of
CI. The sentence now says "classified", with a recorded correction
naming the exit-status defect and this fix.
Board: make lint exit 0 on the pinned 1.98.0 toolchain; cargo fmt
--check clean; clippy --all-targets clean; the 2 new unit tests pass;
tla-check exits 0 on the clean suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI reads the process status, not the report text. All three
dsm_vertical_validationsteps summarized their own failures and returnedOk, so a violated invariant, a falsification config that stopped falsifying, a failed bridge property and a failed implementation trace were every one of them green.Found while surveying the Rust surface for 2c-C3 Phase D. Split out as infrastructure debt rather than folded into C3, because it predates C3 and C3 adds no TLA+ specs.
What was verified before anything changed
collect_tla_resultscomputesall_passed, printsFAILED specs: …, and returnsOk(results)unconditionally.run_tla_checkreturnsOk(results).maindoesrun_tla_check(...).await?and falls through toOk(()).main.rshas noprocess::exit, noExitCodeand nobail!on a verdict anywhere. The CI step is a barecargo run … -- tla-checkwith no grep on the log.run_property_testsandrun_implementation_tracesreturn()— collect, render, return, no failure check at all.So the only things that could redden those three jobs were a missing
java, a missing.tla/.cfg, or a build failure.What was already correct
The expected-violation semantics were fully implemented.
run_allalready normalizes every verdict and inverts the falsification configs: expected invariant violated is a pass, no violation is "the invariant is decoration", a different violation is "a different failure". None of that changes here.The defect was never the classification. It was that nothing turned the aggregate into an exit status.
The change
enforce(gate, total, failing)bails with the failing labels, called after the report prints so the failing spec and its invariant stay visible.Result.EXPECTED_STANDARD_SPECS = 13, asserted inrun_all. The TLA registry is Rust, not a glob, so a spec deleted fromstandard_specsleaves nothing for a file count to notice — the suite just gets smaller and stays green. The Lean gate hasexpected=12in CI; TLA+ had no counterpart.Mutation controls — all executed, all restored
FAILED specs: EconRegisterObservationAll 13 specs PASSEDregisters 12 specs, expected 131 of 7 FAILED — hash_chain_continuity1 of 16 FAILED — state_machine_transfer_chainThe baseline suite is genuinely green, so this does not paper over a red spec — it makes 13 specs, 7 properties and 16 traces load-bearing for the first time.
Record correction
Amendment 2c-C2 (merged as #785) said the five falsification configs "are machine-gated on the invariant each must violate." That was true of the runner's bookkeeping and false of CI. The sentence now reads "classified", with a recorded correction naming the exit-status defect and this fix. The classification it describes was real; the enforcement was not.
Verification
make lintexit 0 on the pinned 1.98.0 toolchain;cargo fmt --checkclean;clippy --all-targetsclean; the 2 new unit tests pass;tla-checkexits 0 on the clean 13-spec suite.