Add Valgrind memcheck (ruby_memcheck) and run it in CI - #152
Open
Watson1978 wants to merge 2 commits into
Open
Conversation
Running the suite under Valgrind on a Ruby C extension is normally impractical:
the interpreter itself produces a large volume of reports that have nothing to
do with the extension. ruby_memcheck wraps Valgrind and only surfaces errors
whose stack trace passes through the extension's .so, which makes the output
usable. It is the same tool Shopify runs on nokogiri and liquid-c.
rake spec:valgrind
This reworks SpringMT#147, which was reverted in SpringMT#150 because it broke CI: the Gemfile
guarded the dependency on Linux but not on the Ruby version, and ruby_memcheck
3.x requires Ruby >= 3.0, so `bundle install` failed on the 2.7 entry of the
matrix. The guard now covers both. Verified on Ruby 2.7.8 locally: bundle
install, rake compile and rspec all pass, and the Rakefile simply does not
define the task there.
The memcheck run gets its own workflow rather than a job inside Ruby.yml, so a
report from it cannot turn the main test matrix red. It pins one Ruby, installs
Valgrind, and carries a 30 minute job timeout; a cold run -- extension build
plus the suite under memcheck -- takes about 1m45s locally.
The suite reports nothing on the current tree, so no suppression file is
needed. The task does have teeth: run against the use-after-free of a stream's
borrowed CDict/DDict it reports 113 Invalid read records, and against the
ZSTD_DCtx leak it reports the context as definitely lost, both attributed
inside zstdruby.so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Watson1978
force-pushed
the
feature/valgrind-memcheck-retry
branch
from
August 9, 2026 19:18
08311ec to
5225cd7
Compare
Random.bytes(1<<17 + 15) parses as Random.bytes(1 << 32), because + binds tighter than <<, so the spec allocates 4 GiB instead of the 131,087 bytes it means -- just past the 128 KiB block boundary the neighbouring specs exercise. Adding the parentheses cuts the whole suite from about 8 seconds to 0.3, since that one example was nearly all of it. Under Valgrind it matters more: the suite goes from 1m18s to 1.4s, and on a CI runner the 4 GiB version cannot finish at all -- it stalls for minutes and the job is killed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds Shopify's
ruby_memcheckso the spec suite can be run under Valgrind's memcheck, and runs it in CI:Running Valgrind against a Ruby C extension is normally impractical, because the interpreter itself produces a large volume of reports that have nothing to do with the extension.
ruby_memcheckwraps Valgrind and applies a heuristic that only surfaces errors whose stack trace passes through the extension's.so, which makes the output usable. It is the same tool Shopify runs on nokogiri and liquid-c.What went wrong last time, and what changed
This reworks #147, which was merged and then reverted in #150 because it broke CI. The cause was the
Gemfile: it guarded the dependency on the platform but not on the Ruby version, andruby_memcheck3.x requires Ruby >= 3.0, sobundle installfailed on the2.7entry of the matrix:The guard now covers both:
Verified on Ruby 2.7.8 locally rather than by inspection:
bundle installresolves,rake compileandrspecboth pass (66 examples), andrake -Tshows nospec:valgrindtask — theRakefileskips defining it when the gem is absent, instead of failing to load.What is in this PR
ruby_memcheckas a development dependency in theGemfile, guarded on Linux and Ruby >= 3.0. It goes in theGemfilerather than the gemspec so the packaged gem stays platform-agnostic.rake spec:valgrindtask in theRakefile. It disables YJIT under Valgrind (they interfere, adding noise and slowdown) and fails early with a clear message when Valgrind is not installed.Valgrindworkflow (.github/workflows/valgrind.yml) that installs Valgrind, pins one Ruby, and runs the task.No suppression file is included: on the current tree the suite reports nothing that needs silencing.
Why a separate workflow
The memcheck job is deliberately not a job inside
Ruby.yml. A report from it should not turn the main test matrix red — that is exactly what led to the revert last time. As its own workflow it is an independent status check, so it can be read, required, or disabled on its own.It carries
timeout-minutes: 30so a job that stops making progress is cut off rather than holding a runner for the GitHub default of 6 hours. A cold run locally — extension build plus the suite under memcheck — takes about 1m45s, so the limit is generous.A spec fix that came out of this
The first CI run of the memcheck job was killed after stalling for three minutes, and it turned up a pre-existing bug in
spec/zstd-ruby_spec.rb:+binds tighter than<<, so this parses asRandom.bytes(1 << 32)and allocates 4 GiB rather than the 131,087 bytes it means — just past the 128 KiB block boundary the neighbouring specs exercise. It is in its own commit here.Adding the parentheses cuts the whole suite from about 8 seconds to 0.3, since that one example was nearly all of it, so the existing test matrix gets faster too. Under Valgrind it is the difference between working and not: the suite goes from 1m18s to 1.4s, and the 4 GiB version cannot finish on a CI runner at all.
It stands on its own and is happy to be taken separately; it is in this PR because the memcheck job cannot pass without it.
Current result
Clean.
rake spec:valgrindexits 0 with zero reports attributed to the extension — no invalid accesses, no leaks — for the 69 examples.Limitations
ruby_memcheckis a regression net, not an audit. Its heuristic only surfaces errors whose stack passes through the extension's.so, so a report count of zero does not mean the extension is free of memory bugs — anything that manifests purely inside Ruby frames, or that never trips Valgrind's checks on the inputs the specs happen to exercise, will not show up. It is also only as good as the suite's coverage. This complements rather than replaces ASan/UBSan builds, fuzzing, and manual review; its value is catching regressions in paths the specs already cover.🤖 Generated with Claude Code