Skip to content

fix(roles/monitoring_plugins)!: align the source install with the packages - #315

Merged
NavidSassan merged 2 commits into
mainfrom
fix/monitoring-plugins-source-install-parity
Aug 19, 2026
Merged

fix(roles/monitoring_plugins)!: align the source install with the packages#315
NavidSassan merged 2 commits into
mainfrom
fix/monitoring-plugins-source-install-parity

Conversation

@markuslf

Copy link
Copy Markdown
Member

Pulls the source install of monitoring_plugins back in line with what the packages produce. Reference for the layout is build/install-plugins.sh in the monitoring-plugins repo (what rpm and deb run), and tools/install-monitoring-plugins for the permission and venv handling.

The package path and Windows are untouched.

What was broken

Defect Since Impact
venv missing httpx, pymysql, pypsrp, PySmbClient, pywinrm 2026-07-01 (venv rewrite) every check that speaks HTTP, MySQL, SMB or WinRM returns UNKNOWN
notification plugins in notifications/, event plugins nowhere 2025-02-14 the shipped Director basket points at the flat path, so notifications never ran
/etc/bash_completion.d not created 2026-08-13 run aborts on a host without the bash-completion package
python3-packaging missing always run aborts on a minimal installation (RHEL 8 is fine, it has pkg_resources)
modes taken from the controller checkout always a controller umask of 027 or 077 leaves the plugins unusable for the monitoring user
four tasks changed on every run 2026-07-01 not idempotent

The dependency defect is a regression: before the venv rewrite the role installed lockfiles/pyXX/requirements.txt, which carries the resolved closure of linuxfabrik-lib including httpx. The rewrite replaced it with requirements.in minus linuxfabrik-lib and dropped the transitive dependencies with it. It became visible when lib.url moved to httpx on 2026-05-10.

Why the test did not catch it

monitoring_plugins_source/verify.yml runs cpu-usage, which only needs psutil, and asserts on Traceback / ModuleNotFoundError. Neither ever appears: lib/url.py imports httpx lazily inside fetch() and answers a missing module with a plain sentence. This PR adds nginx-status, which reaches lib.url.fetch() with no arguments, without nginx and without the host needing internet access (its default URL is on localhost, so a refused connection is the expected result). The assertion goes by the message, since both outcomes are UNKNOWN and the exit code cannot tell them apart.

Verified

install_method: 'source', version: 'dev' against systemd containers of Rocky 8, Rocky 9, Rocky 10, Debian 13 and Ubuntu 24.04:

  • 292 plugins deployed, changed=0 on the second run,
  • uptime and githubstatus run as the monitoring user (icinga / nagios),
  • the new assertion fails on a venv with httpx removed and passes after a re-run of the role.

Molecule itself was not run (needs the libvirt VMs).

Two decisions worth a second opinion

  1. Breaking Changes or Fixed for the notification plugin path. I put it under Breaking Changes because the deployed path changes and the old directory is removed, which is migration wording. It never matched our own Director basket, so nothing built on our config can have depended on it.
  2. requirements.in or the lockfiles. I kept the "newest unpinned" policy the role README promises and install the library's requirements.in on top. The one-liner installer instead uses the hash-pinned lockfiles of both repos and then removes linuxfabrik-lib from the venv. Switching to the lockfiles would close this class of drift for good, since nothing would have to be filtered or reassembled, but it changes the documented policy.

Not in this PR

  • install-monitoring-plugins.ps1 verifies the Authenticode signature of the MSI before installing, windows-package.yml does not. Worth pulling through, but I have no Windows host to verify it on.
  • tools/install-monitoring-plugins itself does not deploy the event plugins, while the packages do. Currently one plugin (cloudflare-security-level). Upstream issue, not this repo.

…kages

The source install diverged from what build/install-plugins.sh and the
one-liner installer produce, in five ways that each break a monitored host:

- the venv was missing httpx, pymysql, pypsrp, PySmbClient and pywinrm.
  They enter through linuxfabrik-lib, which the role filters out of
  requirements.in because it deploys the library from GitHub. Install the
  library's own requirements.in alongside; the two lists overlap, so they
  need separate pip calls. A regression since the venv rewrite, which
  replaced the lockfile (carrying the resolved closure) with requirements.in
- notification plugins went to notifications/ and event plugins nowhere.
  The packages and the shipped Director basket expect all of them flat in
  the plugin directory. Mirror build/install-plugins.sh, example included
- modes came from the controller checkout, so a hardened umask left the
  plugins unreadable for the monitoring user. Force them via rsync --chmod
  and chmod the venv, without touching the shared plugin directory
  recursively (setuid check_icmp)
- ansible.builtin.pip needs packaging or setuptools on the managed node,
  and the bash completion needs /etc/bash_completion.d to exist. Neither
  is present on a minimal installation
- rsync pushed the controller's numeric owner and a freshly built staging
  tree's mtimes, so four tasks reported a change on every run

Also drop a venv built with another Python, and abort when the checkout
yields no plugins.

The molecule scenario missed the missing dependencies because it runs
cpu-usage, which only needs psutil, and asserts on tracebacks: lib.url
imports httpx lazily and answers a missing module with a plain sentence.
Add nginx-status, which reaches lib.url.fetch() with no arguments, no
nginx and no internet access, and assert on the message instead of the
exit code, since both outcomes are UNKNOWN.

Verified with install_method 'source' against systemd containers of Rocky
8, 9 and 10, Debian 13 and Ubuntu 24.04: 292 plugins, changed=0 on the
second run, and uptime plus githubstatus run as the monitoring user. The
new assertion fails on a venv with httpx removed and passes after a
re-run of the role.
@markuslf
markuslf requested a review from NavidSassan August 17, 2026 19:01
@NavidSassan
NavidSassan merged commit 004d997 into main Aug 19, 2026
9 checks passed
@NavidSassan
NavidSassan deleted the fix/monitoring-plugins-source-install-parity branch August 19, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants