test: pin the threshold evaluator's behaviour before restructuring it - #799
test: pin the threshold evaluator's behaviour before restructuring it#799somethingwithproof wants to merge 4 commits into
Conversation
|
The four Integration Test failures are the upstream break, not this PR — which touches no production file at all. The PHPUnit job passes. #776 pins the Cacti checkout to |
There was a problem hiding this comment.
Pull request overview
Adds a PHPUnit-based characterization test harness for thold_check_threshold() (hi/low and time-based arms) so the evaluator can be safely restructured later while preserving current side-effect behavior, and wires it into CI via a Dockerized test runner and patch-coverage gate.
Changes:
- Introduces a Cacti-function stub/recorder (
CactiStub) plusThresholdScenario/ThresholdOutcomehelpers to exercisethold_check_threshold()by observing side effects (mail/log/traps/DB writes). - Adds characterization tests covering key hi/low and time-based behaviors, including documented “recorded, not endorsed” quirks.
- Adds PHPUnit + Docker + GitHub Actions workflow, plus a patch-coverage script to enforce coverage of changed PHP lines.
Reviewed changes
Copilot reviewed 14 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/Unit/ThresholdTimeBasedCharacterizationTest.php |
Adds time-based arm characterization tests for one-poll behavior and drift vs hi/low. |
tests/Unit/ThresholdHiLowCharacterizationTest.php |
Adds hi/low arm characterization tests including notification/restoral/re-alert behaviors. |
tests/TestCase.php |
Base test case ensuring stub reset + plugin source/constants loading helpers. |
tests/Support/ThresholdScenario.php |
Scenario builder that stubs required DB/core calls and runs one poll. |
tests/Support/ThresholdOutcome.php |
Reads recorded stub calls to expose emitted outcomes in assertion-friendly form. |
tests/Support/CactiStub.php |
Recording + programmable shim backing the global Cacti function stubs. |
tests/fixtures/optional-core-functions.php |
Optional core functions for exercising function_exists() branches in isolated runs. |
tests/fixtures/cacti-lib/variables.php |
Placeholder for Cacti lib/variables.php so include_once paths resolve in tests. |
tests/docker/Dockerfile |
Docker image for running the unit test suite with coverage tooling. |
tests/docker/docker-compose.yml |
Local developer wrapper to run the same Dockerized test job as CI. |
tests/bootstrap.php |
Global-function shims and minimal Cacti environment bootstrap for unit tests. |
tests/bin/patch-coverage.php |
Computes coverage specifically for changed PHP lines by diffing vs base ref. |
phpunit.xml |
PHPUnit config + focused coverage scope for thold_functions.php. |
composer.json |
Adds PHPUnit dependency, dev autoloading for test support classes, and test scripts. |
.gitignore |
Ignores vendor/cache/coverage artifacts introduced by the test toolchain. |
.github/workflows/php-unit-tests.yml |
GitHub Actions workflow running Dockerized PHPUnit + patch-coverage enforcement. |
Suppressed comments (1)
tests/Support/CactiStub.php:112
- CactiStub::record() phpdoc for $params should allow associative keys (it is used for both bound parameter lists and associative row data like sql_save).
* @param string $fn Cacti function name.
* @param string $sql SQL text, or '' for non-query calls.
* @param array<int, mixed> $params Bound parameters, if any.
*
* @return void
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /** | ||
| * Every Cacti function call the plugin made, in order. | ||
| * | ||
| * @var array<int, array{fn: string, sql: string, params: array<int, mixed>}> | ||
| */ | ||
| public static $calls = []; |
| if (!is_dir($maint)) { | ||
| mkdir($maint, 0777, true); | ||
| } |
e421002 to
5f1ead7
Compare
|
Phase 2 is up in #800, stacked on this one. It passes these 35 tests unchanged, which is the check that the move was behaviour-preserving. |
thold_check_threshold() has no return value: everything it decides is a side effect through seven global Cacti functions. ThresholdScenario builds the fixture those need and runs one poll; ThresholdOutcome reads back what was emitted, so a test asserts on behaviour rather than on the SQL text. No production file is touched. Several assertions record behaviour that is wrong rather than intended, each with a comment saying so, so that fixing it later is a deliberate edit here. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Records where this arm has drifted from the hi/low one, notably that a restoral writes the log row and clears the state but sends no mail, so an operator sees the alert and never the all-clear. Cacti's cell fetchers return false rather than '' when a query matches no row. The stub now does the same: on PHP 8 the difference is a TypeError in the re-alert arithmetic, so the old default invented a failure production does not have. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Adopts the conventions from Cacti core: tests/bootstrap-unit.php, tests/Helpers for the stubs, a phpunit.xml carrying error_reporting -1 and CACTI_TEST_BOOTSTRAP, and composer lint / test / test:coverage scripts so CI runs the same commands a developer does. The dev toolchain is Cacti's, pinned to the same platform php 8.1.0. Cacti core runs Pest and this suite does not, because pest ^2 does not resolve on PHP 8.1 -- the platform Cacti's own composer.json pins. Releases up to v2.36.0 conflict with phpunit 10.5.62 and later, every earlier 10.x release is blocked by advisory PKSA-z3gr-8qht-p93v, and v2.36.1, which does resolve, requires PHP 8.2. The stack installs on 8.2 and above; 8.1 is the floor this plugin's CI matrix targets. The tests are written in the plain PHPUnit class style that Cacti's tests/Pest.php explicitly supports, so they run unchanged under Pest wherever it is installable.
This arm compares the reading against statistics rrdtool reports for a reference window, so the scenario supplies those rather than static bounds. Reaching them means answering the three rrdtool calls thold makes -- file existence, an info block describing the data sources and consolidation functions, and a graph command whose printed values are decoded by position -- which the helper now does. Completes the three arms, so Phase 2 can start moving code. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
5f1ead7 to
8138470
Compare
Phase 1 of restructuring
thold_check_threshold(). No production file is touched — this only adds tests.Why this comes first
thold_check_threshold()is 1,431 lines, and after a 157-line preamble it is aswitchwith three arms that are near-copies of each other:thold_mailthold_snmptrapdb_execute_preparedThey have drifted, and the drift is what #742 through #752 report. Collapsing them onto one implementation is the fix, but the function has six side effects per transition and no tests at all — so a structural change now would be a rewrite by hope.
These tests pin present behaviour, bugs included, so the later phases can move code and know nothing shifted.
How it works
The function returns nothing; everything it decides is a side effect through global Cacti functions.
ThresholdScenariobuilds the fixture those need and runs one poll.ThresholdOutcomereads back what was emitted — mail, log rows, traps, and the columns written tothold_data— so a test asserts on behaviour rather than on SQL text and does not break when a query is reworded.persistedColumns()parses the SET clause and resolves each?against the bound parameters, because these statements mix placeholders and literals in one clause.What the tests found
Writing them turned up behaviour worth recording. Each is pinned with a comment saying it is recorded rather than endorsed:
ST_NOTIFYRSlog row and clears the state, but never mails, so an operator watching a time-based threshold sees the alert and never the all-clear. The hi/low arm mails.ST_TRIGGERAexists for this case but only the time-based arm writes it.ops@example.org,,— because the legacy contact list is joined with the global and device addresses whether or not those are set.One harness correction worth calling out
Cacti's
db_fetch_cell_prepared()returnsfalse, not'', when a query matches no row. The stub originally returned'', and on PHP 8 that difference is aTypeErrorin the time-based re-alert arithmetic ($realerttime + $lastemailtime). I nearly filed that as a fatal before checkinglib/database.php; it is not one, and the stub now matches core. Recording it here because it is the kind of thing a stub can invent.Scope
25 tests over the hi/low and time-based arms. The baseline arm needs reference-value fixtures and is not covered yet; it will follow before Phase 2 begins.
The harness commit matches #773, #788, #790 and #791, extended with SQL-matched stub responses and
gmp. Whichever lands first, the others merge cleanly.Full plan for the remaining phases is written up separately; Phase 2 lifts the preamble, Phase 3 extracts the emitter, Phase 4 fixes the filed bugs one commit each, and Phase 5 — collapsing the arms — is optional and gated on maintainer appetite.