Skip to content

Add Valgrind memcheck (ruby_memcheck) and run it in CI - #152

Open
Watson1978 wants to merge 2 commits into
SpringMT:mainfrom
Watson1978:feature/valgrind-memcheck-retry
Open

Add Valgrind memcheck (ruby_memcheck) and run it in CI#152
Watson1978 wants to merge 2 commits into
SpringMT:mainfrom
Watson1978:feature/valgrind-memcheck-retry

Conversation

@Watson1978

@Watson1978 Watson1978 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

This adds Shopify's ruby_memcheck so the spec suite can be run under Valgrind's memcheck, and runs it in CI:

rake spec:valgrind

Running Valgrind against a Ruby C extension is normally impractical, because the interpreter itself produces a large volume of reports that have nothing to do with the extension. ruby_memcheck wraps Valgrind and applies a heuristic that only surfaces errors whose stack trace passes through the extension's .so, which makes the output usable. It is the same tool Shopify runs on nokogiri and liquid-c.

What went wrong last time, and what changed

This reworks #147, which was merged and then reverted in #150 because it broke CI. The cause was the Gemfile: it guarded the dependency on the platform but not on the Ruby version, and ruby_memcheck 3.x requires Ruby >= 3.0, so bundle install failed on the 2.7 entry of the matrix:

Because ruby_memcheck >= 3.0.0 depends on Ruby >= 3.0.0
  and Gemfile depends on ruby_memcheck ~> 3.0,

The guard now covers both:

if RUBY_PLATFORM.include?('linux') && Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('3.0.0')
  gem 'ruby_memcheck', '~> 3.0'
end

Verified on Ruby 2.7.8 locally rather than by inspection: bundle install resolves, rake compile and rspec both pass (66 examples), and rake -T shows no spec:valgrind task — the Rakefile skips defining it when the gem is absent, instead of failing to load.

What is in this PR

  • ruby_memcheck as a development dependency in the Gemfile, guarded on Linux and Ruby >= 3.0. It goes in the Gemfile rather than the gemspec so the packaged gem stays platform-agnostic.
  • A rake spec:valgrind task in the Rakefile. It disables YJIT under Valgrind (they interfere, adding noise and slowdown) and fails early with a clear message when Valgrind is not installed.
  • A Valgrind workflow (.github/workflows/valgrind.yml) that installs Valgrind, pins one Ruby, and runs the task.

No suppression file is included: on the current tree the suite reports nothing that needs silencing.

Why a separate workflow

The memcheck job is deliberately not a job inside Ruby.yml. A report from it should not turn the main test matrix red — that is exactly what led to the revert last time. As its own workflow it is an independent status check, so it can be read, required, or disabled on its own.

It carries timeout-minutes: 30 so a job that stops making progress is cut off rather than holding a runner for the GitHub default of 6 hours. A cold run locally — extension build plus the suite under memcheck — takes about 1m45s, so the limit is generous.

A spec fix that came out of this

The first CI run of the memcheck job was killed after stalling for three minutes, and it turned up a pre-existing bug in spec/zstd-ruby_spec.rb:

large_string = Random.bytes(1<<17 + 15)

+ binds tighter than <<, so this parses as Random.bytes(1 << 32) and allocates 4 GiB rather than the 131,087 bytes it means — just past the 128 KiB block boundary the neighbouring specs exercise. It is in its own commit here.

Adding the parentheses cuts the whole suite from about 8 seconds to 0.3, since that one example was nearly all of it, so the existing test matrix gets faster too. Under Valgrind it is the difference between working and not: the suite goes from 1m18s to 1.4s, and the 4 GiB version cannot finish on a CI runner at all.

It stands on its own and is happy to be taken separately; it is in this PR because the memcheck job cannot pass without it.

Current result

Clean. rake spec:valgrind exits 0 with zero reports attributed to the extension — no invalid accesses, no leaks — for the 69 examples.

Limitations

ruby_memcheck is a regression net, not an audit. Its heuristic only surfaces errors whose stack passes through the extension's .so, so a report count of zero does not mean the extension is free of memory bugs — anything that manifests purely inside Ruby frames, or that never trips Valgrind's checks on the inputs the specs happen to exercise, will not show up. It is also only as good as the suite's coverage. This complements rather than replaces ASan/UBSan builds, fuzzing, and manual review; its value is catching regressions in paths the specs already cover.

🤖 Generated with Claude Code

Running the suite under Valgrind on a Ruby C extension is normally impractical:
the interpreter itself produces a large volume of reports that have nothing to
do with the extension. ruby_memcheck wraps Valgrind and only surfaces errors
whose stack trace passes through the extension's .so, which makes the output
usable. It is the same tool Shopify runs on nokogiri and liquid-c.

    rake spec:valgrind

This reworks SpringMT#147, which was reverted in SpringMT#150 because it broke CI: the Gemfile
guarded the dependency on Linux but not on the Ruby version, and ruby_memcheck
3.x requires Ruby >= 3.0, so `bundle install` failed on the 2.7 entry of the
matrix. The guard now covers both. Verified on Ruby 2.7.8 locally: bundle
install, rake compile and rspec all pass, and the Rakefile simply does not
define the task there.

The memcheck run gets its own workflow rather than a job inside Ruby.yml, so a
report from it cannot turn the main test matrix red. It pins one Ruby, installs
Valgrind, and carries a 30 minute job timeout; a cold run -- extension build
plus the suite under memcheck -- takes about 1m45s locally.

The suite reports nothing on the current tree, so no suppression file is
needed. The task does have teeth: run against the use-after-free of a stream's
borrowed CDict/DDict it reports 113 Invalid read records, and against the
ZSTD_DCtx leak it reports the context as definitely lost, both attributed
inside zstdruby.so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Watson1978
Watson1978 force-pushed the feature/valgrind-memcheck-retry branch from 08311ec to 5225cd7 Compare August 9, 2026 19:18
Random.bytes(1<<17 + 15) parses as Random.bytes(1 << 32), because + binds
tighter than <<, so the spec allocates 4 GiB instead of the 131,087 bytes it
means -- just past the 128 KiB block boundary the neighbouring specs exercise.

Adding the parentheses cuts the whole suite from about 8 seconds to 0.3, since
that one example was nearly all of it. Under Valgrind it matters more: the
suite goes from 1m18s to 1.4s, and on a CI runner the 4 GiB version cannot
finish at all -- it stalls for minutes and the job is killed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant