Skip to content

Read the Catch2 dialect, so the guard covers a DuckDB consumer - #126

Merged
estebanzimanyi merged 1 commit into
MobilityDB:masterfrom
estebanzimanyi:tooling/catch2-test-dialect
Sep 3, 2026
Merged

Read the Catch2 dialect, so the guard covers a DuckDB consumer#126
estebanzimanyi merged 1 commit into
MobilityDB:masterfrom
estebanzimanyi:tooling/catch2-test-dialect

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

WITNESS: a MobilityDuck CI run writes its whole test summary as one line of
the Catch2 console reporter DuckDB's test runner builds, reading

... (2695 assertions in 102 test cases)

The rules read surefire, pytest, Go and VSTest, so that log names no dialect
and reaches the no-summary failure: a sqllogictest consumer cannot use the action at all.

MEASURED over that job log: the rules read catch2, total 102, 0 skipped, and
a floor of 102 is accepted while 103 is refused. The passing form's test-case
count EXCLUDES the skipped cases -- the reporter writes
testCases.passed - skippedTests -- so 3 skipped tests, ... in 99 test cases
is a suite of 102 with 3 skipped, and the two are added rather than read off
the printed count. The failure table and the wholly-skipped form are read too.
tests/test_test_outcome.py goes 9 cases to 16, and the suite floor 293 to 300.

The order is load-bearing and one of those cases holds it: the failure table's
101 passed satisfies the pytest pattern, so Catch2 is read BEFORE pytest --
pytest-first reports 101 as the total and misses the failure and every skip
beside it.

WHY, and what this dialect does NOT do: under this runner a sqllogictest FILE
is one test case, and a mode skip inside a file silences the statements after
it while the file still reports as a passing test case. The skip count
therefore cannot see a mode skip, and only the floor holds -- it moves when a
whole file leaves the suite. The module says so where a reader meets the
dialect, because a guard whose green answer is silence about the very construct
a consumer uses is worse than none. A sqllogictest consumer needs a census of
mode skip in the test sources beside this.

WITNESS: a MobilityDuck CI run writes its whole test summary as one line of
the Catch2 console reporter DuckDB's test runner builds, reading

    ... (2695 assertions in 102 test cases)

The rules read surefire, pytest, Go and VSTest, so that log names no dialect
and reaches the no-summary failure: a sqllogictest consumer cannot use the action at all.

MEASURED over that job log: the rules read `catch2`, total 102, 0 skipped, and
a floor of 102 is accepted while 103 is refused. The passing form's test-case
count EXCLUDES the skipped cases -- the reporter writes
`testCases.passed - skippedTests` -- so `3 skipped tests, ... in 99 test cases`
is a suite of 102 with 3 skipped, and the two are added rather than read off
the printed count. The failure table and the wholly-skipped form are read too.
tests/test_test_outcome.py goes 9 cases to 16, and the suite floor 293 to 300.

The order is load-bearing and one of those cases holds it: the failure table's
`101 passed` satisfies the pytest pattern, so Catch2 is read BEFORE pytest --
pytest-first reports 101 as the total and misses the failure and every skip
beside it.

WHY, and what this dialect does NOT do: under this runner a sqllogictest FILE
is one test case, and a `mode skip` inside a file silences the statements after
it while the file still reports as a passing test case. The skip count
therefore cannot see a `mode skip`, and only the floor holds -- it moves when a
whole file leaves the suite. The module says so where a reader meets the
dialect, because a guard whose green answer is silence about the very construct
a consumer uses is worse than none. A sqllogictest consumer needs a census of
`mode skip` in the test sources beside this.
@estebanzimanyi
estebanzimanyi merged commit cdbf8f4 into MobilityDB:master Sep 3, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the tooling/catch2-test-dialect branch September 3, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant