Pgbench kernel ab benchmark - #31
Open
nmanthey wants to merge 10 commits into
Open
Conversation
On a system with multiple python environments, we might want to run with a different version. Therefore, allow the script to select a python version. This change also helps when adding support to new python versions. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
summarize_unixbench_log built the metric name from fields 1..NF-4, which pulled the numeric value, unit and part of the timing info into the metric name — producing malformed rows like 'Arithmetic_Test_(double)_385400605.9_lps' with a single sample each, instead of one 'Arithmetic_Test_(double)' metric aggregated across VMs. Use NF-6 for the metric name (matching unixbench-kernel-regression) so value=$(NF-5) and unit=$(NF-4) line up, and drop the index-section parsing that emitted duplicate/derived score metrics. Verified against sample UnixBench output: clean metric names, correct value/unit. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
UnixBench (and other CPU-bound benchmarks) run for many minutes with no new console output during the benchmark phase. The 600s hang-detection default tripped mid-run and killed all VMs as a false-positive stall. Raise the PULLAB_TASK_HANG_THRESHOLD_SEC default from 600 to 1200s so these benchmarks complete, while still catching genuine hangs within a reasonable window. The value remains env-overridable for lighter workloads that want faster detection. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
The kernel A/B tests each carried their own copy of the kernel install/reboot helpers, so any change had to be made in every test. Introduce vm-tests/lib/kernel_helpers.sh as the single home for that logic (environment validation, kernel RPM download/selection, install_kernel_rpm with grubby boot-entry management, and the get_running_kernel / assert_kernel_changed helpers). Each kernel test includes it via a kernel_helpers.sh symlink and sources it, keeping only its test-specific functions: - example-kernel-reboot-test: none (pure kernel install/reboot). - simple-source-reboot: source-RPM build helpers. - unixbench-kernel-regression: UnixBench prepare/run/summarize. The symlink is stored by the payload zip as real content, so the VM sees a normal file; no pipeline change is needed. Subsequent fixes to the kernel logic now land once in the shared lib. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
A kernel built with make binrpm-pkg and LOCALVERSION=-nogup has an RPM VERSION of 6.18.41_nogup (underscore) but installs vmlinuz-6.18.41-nogup (dash). The vmlinuz lookup missed the file because it used the RPM VERSION verbatim. Compute an alternate version string with underscores replaced by dashes, try both in the grubby --info and vmlinuz globs, and derive the kernel version for dracut/initramfs from the actual vmlinuz filename. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
AL2023 enables FIPS by default. A custom kernel built with make binrpm-pkg carries unsigned modules (e.g. ghash_clmulni_intel) that fail FIPS signature verification, causing a kernel panic reboot loop. Add fips=0 to the grubby boot-entry args and run fips-mode-setup --disable after installing the kernel, before the reboot, so unsigned modules load without panic. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
On an AL2023 AMI whose default kernel is a different series than the RPM under test (e.g. a 6.18 AMI installing a 6.1 kernel), the distro kernel<N>-tools package declares 'conflicts with kernel-uname-r < <N>', so a plain dnf/yum install is refused with 'conflicting requests'. Verified on a live 6.18 AMI: plain install fails, but 'dnf install --allowerasing' removes the conflicting kernel<N>-tools package and installs the requested kernel; both vmlinuz files remain in /boot so the target kernel boots normally. Add --allowerasing as the final fallback in install_kernel_rpm, making the kernel A/B tests robust to base-AMI kernel-series drift. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
UnixBench's 'System Call Overhead' result is reported in lps — the count
of syscall iterations completed in a fixed time window — like every other
first-section UnixBench metric. Despite the name 'Overhead', a higher
value means more syscalls/sec, i.e. faster.
Confirmed in the bundled UnixBench 6.0.0 source:
* src/syscall.c: the test loops calling syscalls and increments a
counter until a timer fires, then reports the count:
iter = 0; wake_me(duration, report);
while (1) { close(dup(fd)); syscall(SYS_getpid);
getuid(); umask(022); iter++; }
void report() { fprintf(stderr,"COUNT|%ld|1|lps\n", iter); }
So 'iter' is iterations-per-run — higher is faster.
* UnixBench/Run: the syscall test is a plain count metric
("repeat" => 'long', "options" => "10"), scored via the
count-based branch 'product += log(count)' (Run:1194), not the
time-inverted branch — bigger count yields a bigger index.
The tests marked it more_is_better=false, which inverted its meaning: a
genuine syscall-throughput improvement was flagged as a regression (and a
real slowdown would have been mislabeled an improvement).
Set more_is_better=true for all first-section UnixBench metrics in both
unixbench-kernel-regression and simple-unixbench.
Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
Same-NVR kernel RPMs (e.g. two builds sharing 6.18.41-94.142.amzn2023.x86_64 but differing in compiler/toolchain) were skipped by dnf as "already installed", so the second kernel was never actually written and a kernel-regression test compared a kernel against itself. Add get_running_kernel_id() to the shared kernel_helpers.sh library: a composite build identity (uname -r | uname -v | vmlinuz sha256) that detects a real kernel switch even when two builds share an NVR. Make install_kernel_rpm force 'dnf reinstall' when the NVR is already present, and have assert_kernel_changed compare identities with a clearer message. Wire the unixbench-kernel-regression run scripts to use the new helper, and name their benchmark CSVs by $(uname -r) (clean/human-readable) rather than the composite identity, which would embed '|' and spaces. The logic lives once in vm-tests/lib/kernel_helpers.sh; tests only call the shared helper. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
Add a pgbench-kernel-regression VM test that installs a base kernel, runs a PostgreSQL 16 pgbench read-only and read-write suite, reboots into a second (tip) kernel, re-runs the suite, and emits benchmark-base-*.csv and benchmark-tip-*.csv for the regression analyzer to compare (read-only/read-write TPS and average latency). Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
nmanthey
force-pushed
the
pgbench-kernel-AB-benchmark
branch
from
August 20, 2026 08:57
ab17330 to
b4be7da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding the pgbench postgresql benchmark as a new highlevel regression benchmark. Coefficient of variance is low enough to consider this benchmark as a stable benchmark to detect regressions (CV 0.7–1.7%). Since the same postgresql version used. Running multiple test entries in one pipeline works.
Testing Done
Ran the pgbench-kernel-regression test end-to-end on AWS (local-orchestrator path), base kernel 6.1.141-165.249.amzn2023 vs tip 6.1.150-174.273.amzn2023, on c8i.4xlarge in us-west-2 with the al2023-ami-kernel-6.1-x86_64 AMI.
Single-VM run
postgresql.readwrite.latency_avg.
Multi-VM statistical comparison (5 VMs)
Combined config (all touched tests, 2 VMs each)
One config running pgbench-kernel-regression, unixbench-kernel-regression, and simple-unixbench at min_count: 2. 6/6 VMs succeeded. pgbench produced 8 base + 8 tip rows and compared correctly.