Skip to content

[SDK] fix: fixed size exemplar reservoir works correctly. - #4429

Open
proost wants to merge 4 commits into
open-telemetry:mainfrom
proost:fix-fixed-size-exemplar-reservoir
Open

[SDK] fix: fixed size exemplar reservoir works correctly.#4429
proost wants to merge 4 commits into
open-telemetry:mainfrom
proost:fix-fixed-size-exemplar-reservoir

Conversation

@proost

@proost proost commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Changes

Looks like current fixed size exemplar reservoir works wrong.

What i fixed:

  1. reset the stored reservoir cells by reference not value copied.
  2. reset selector state without destroying the selector.
  3. reset the simple reservoir’s measurement counter after collection.
  4. exclude empty cells from collection results.
  5. "ReservoirCellSelector" isn't thread-safe. But OfferMeasurement can be called concurrently so, we should guard it.
  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@proost
proost requested a review from a team as a code owner August 14, 2026 14:37
@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.68%. Comparing base (60c3d11) to head (a978029).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4429      +/-   ##
==========================================
+ Coverage   82.61%   82.68%   +0.07%     
==========================================
  Files         511      511              
  Lines       20132    20137       +5     
==========================================
+ Hits        16631    16648      +17     
+ Misses       3501     3489      -12     
Files with missing lines Coverage Δ
...k/metrics/exemplar/fixed_size_exemplar_reservoir.h 91.18% <100.00%> (+1.53%) ⬆️
...cs/exemplar/simple_fixed_size_exemplar_reservoir.h 83.34% <100.00%> (+50.01%) ⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

return;
}

std::lock_guard<std::mutex> lock{mutex_};

@lalitb lalitb Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This serializes offers and collection for each reservoir. That is the right correctness fix, and the feature is still ENABLE_METRICS_EXEMPLAR_PREVIEW-gated. It does add contention to the exemplar-enabled record path, so please mention that tradeoff in the PR description and open a follow-up to benchmark or explore a less contended design before exemplars become stable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because resetting "reservoir_cell_selector_" is called in "CollectAndReset". "CollectAndReset" is never called in the this repo, But i can't find what is contract about thread-safe.

It does add contention to the exemplar-enabled record path

Absolutely right. I bit more digging out atomic way.

}

void reset() override {}
void reset() override { measurements_seen_ = 0; }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reset is correct, but the sampling calculation just above is still off by one. measurement_num is zero-based, so the random choice must cover [0, measurement_num]. With a size-one reservoir, % measurement_num makes the second measurement replace the first every time instead of with 50% probability; size 0 also reaches modulo zero. Could we fix that here and add a deterministic test for the selection bounds?

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes correctness and thread-safety issues in the preview metrics exemplar fixed-size reservoirs (including the simple fixed-size and aligned histogram bucket variants), ensuring reservoirs can be reused across collection intervals and behave correctly under concurrent OfferMeasurement/CollectAndReset usage.

Changes:

  • Serialize OfferMeasurement and CollectAndReset in FixedSizeExemplarReservoir with a mutex, and reset selector state (without destroying it) after collection.
  • Fix collection/reset semantics: reset stored cells by reference (not by value-copy iteration) and omit empty (null) exemplar results from CollectAndReset.
  • Add unit tests (CMake + Bazel) covering multi-interval correctness, simple reservoir sampling restart, and concurrency serialization; update CHANGELOG.md.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
sdk/test/metrics/exemplar/fixed_size_exemplar_reservoir_test.cc Adds tests validating interval reuse, selector reset behavior, and offer/collect serialization.
sdk/test/metrics/exemplar/CMakeLists.txt Registers the new exemplar reservoir test in the CMake test target list.
sdk/test/metrics/exemplar/BUILD Adds a Bazel cc_test target for the new fixed-size exemplar reservoir tests.
sdk/include/opentelemetry/sdk/metrics/exemplar/simple_fixed_size_exemplar_reservoir.h Implements selector reset() to restart sampling each collection interval.
sdk/include/opentelemetry/sdk/metrics/exemplar/fixed_size_exemplar_reservoir.h Fixes reset semantics, filters empty cells, and adds locking to make offer/collect thread-safe.
CHANGELOG.md Documents the exemplar reservoir fixes as a metrics SDK change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants