Skip to content

fix(core): Snapshot feature flags before merging - #5994

Open
adinauer wants to merge 5 commits into
mainfrom
fix/feature-flag-buffer-snapshots
Open

fix(core): Snapshot feature flags before merging#5994
adinauer wants to merge 5 commits into
mainfrom
fix/feature-flag-buffer-snapshots

Conversation

@adinauer

@adinauer adinauer commented Aug 25, 2026

Copy link
Copy Markdown
Member

📜 Description

Snapshot each scope's FeatureFlagBuffer before merging feature flags into an event. This keeps the existing mutable CopyOnWriteArrayList write path while ensuring that the merge traverses structurally stable lists.

💡 Motivation and Context

Fixes #5988

FeatureFlagBuffer keeps a mutable CopyOnWriteArrayList wrapper. Writers update that wrapper through separate remove, add, and eviction operations, while merged() previously captured only the wrapper reference and then used separate size() and indexed get() calls.

A CopyOnWriteArrayList makes each individual operation thread-safe, but retaining its wrapper does not retain one backing-array snapshot. A concurrent writer can therefore produce this interleaving:

  1. A writer adds a flag and the list reaches 101 entries.
  2. The merging thread reads a size of 101.
  3. The writer evicts the oldest flag and the list returns to 100 entries.
  4. The merging thread calls get(100) and throws ArrayIndexOutOfBoundsException during event capture.

The same structural race is possible when another thread clears a buffer. Constructing an independent COW wrapper for each input buffer gives the merge a stable size and stable indexed reads without acquiring the global, isolation, and current buffer locks together. This avoids lock-ordering risk and preserves the benchmark-favored write path.

This intentionally preserves the existing best-effort semantics: a snapshot may observe an intermediate step of one compound feature-flag update, but it is structurally safe and the merged output remains bounded.

Refs JAVA-704

💚 How did you test it?

  • Added a concurrent regression test that pre-fills the buffer, verifies writer progress during add/evict churn, and performs 1,000 merges.
  • Confirmed the regression test fails on main with ArrayIndexOutOfBoundsException and passes with this change.
  • Ran ./gradlew ':sentry:test' --tests='*FeatureFlagBufferTest*' --info (26 tests, 0 failures).
  • Ran ./gradlew spotlessApply apiDump successfully.

📝 Checklist

  • I added GH Issue ID & Linear ID
  • I added tests to verify the changes.
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • Review from the native team if needed.
  • No breaking change or entry added to the changelog.
  • No breaking change for hybrid SDKs or communicated to hybrid SDKs.
  • Public API changes reviewed by another Mobile SDK team member or implemented according to the develop docs spec.

🔮 Next steps

None.

Concurrent writes can change a CopyOnWriteArrayList backing array between
merged() reading its size and indexed entries, crashing event capture.

Snapshot each scope buffer before merging so indexed reads use a stable
view. Add a regression test that verifies writer progress during merging.

Refs JAVA-704
Co-Authored-By: Claude <noreply@anthropic.com>
@linear-code

linear-code Bot commented Aug 25, 2026

Copy link
Copy Markdown

JAVA-704

@adinauer

Copy link
Copy Markdown
Member Author

Feature flag buffer strategy benchmarks

I benchmarked the three implementations discussed here:

  • A — Current main (buggy): mutable CopyOnWriteArrayList, with merged() traversing the live wrapper.
  • B — This PR: same mutable write path, but merged() creates stable COW wrapper snapshots for the three scope buffers.
  • C — PR fix(core): Make feature flag buffer merging thread-safe #5989: immutable lists published through a volatile field; writers copy and publish a completed list.

The benchmark reproduces the historical add() workloads from #4812 and also measures merging three populated scopes and cloning.

Add workloads

Each operation creates a buffer and performs the complete workload.

Workload A: Current B: This PR C: PR #5989
Unique 10.09 µs / 30,848 B 10.35 µs / 30,848 B 19.14 µs / 41,137 B
Duplicates 2.17 µs / 7,312 B 2.16 µs / 7,312 B 2.43 µs / 16,016 B
Full + duplicates 25.72 µs / 117,249 B 30.00 µs / 117,249 B 51.26 µs / 97,137 B
Realistic 6.90 µs / 27,168 B 6.95 µs / 27,168 B 13.97 µs / 32,256 B

A and B have identical add() implementations, so their timing differences are benchmark variance. This PR preserves the existing write-path performance.

Compared with current main, PR #5989 was approximately:

  • 2× slower for unique, full-duplicate, and realistic workloads;
  • 12% slower for duplicate-only updates;
  • 19% more allocation for the realistic workload.

Three-scope merge

Each scope contains the same flag names, modeling substantial deduplication between global, isolation, and current scopes.

Entries per scope A: Current B: This PR C: PR #5989
0 2.42 ns / ~0 B 2.90 ns / 48 B 1.34 ns / ~0 B
10 262.96 ns / 1,304 B 273.63 ns / 1,424 B 248.96 ns / 1,232 B
100 2,143.05 ns / 7,024 B 2,258.90 ns / 7,144 B 1,945.93 ns / 6,592 B

At 100 entries per scope:

  • This PR costs 5.4% more merge time and 120 additional bytes versus current main.
  • On the benchmark's OpenJDK 17 runtime, the fixed 120-byte cost is three COW wrapper allocations; their backing arrays are shared rather than copied.
  • PR fix(core): Make feature flag buffer merging thread-safe #5989 is 9.2% faster than current main for merging and allocates 432 fewer bytes.

Clone at 100 entries

Strategy Time Allocation
A: Current 6.56 ns 112 B
B: This PR 6.82 ns 112 B
C: PR #5989 4.16 ns 72 B

Trade-off

This PR keeps the hot feature-flag evaluation/write path unchanged and pays a small cost when capturing an event. It fixes the structural size()/indexed-get() crash, but a lock-free snapshot can still observe an intermediate step of the existing compound add() operation.

PR #5989 provides stronger completed-write snapshot semantics and cheaper merge/clone operations, but moves substantially more allocation and CPU cost onto feature-flag additions. Based on these measurements, this PR favors preserving write performance while applying the smallest fix for the demonstrated crash.

Methodology and caveats

  • JMH on OpenJDK 17, with 3 warmup iterations, 5 measurement iterations, and 2 forks.
  • GC profiler used for normalized allocation measurements.
  • Single-threaded microbenchmark; lock contention is not measured.
  • The harness mirrors the implementations rather than invoking SDK classes directly.
  • Android's CopyOnWriteArrayList implementation may have different constructor costs; sharing the backing array is an implementation optimization, not a public complexity guarantee.

@sentry

sentry Bot commented Aug 25, 2026

Copy link
Copy Markdown

📲 Install Builds

Android

🔗 App Name App ID Version Configuration
SDK Size io.sentry.tests.size 8.53.0 (1) release

⚙️ sentry-android Build Distribution Settings

@runningcode runningcode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we still remove the volatile on the flags ?

// Capture snapshots to avoid inconsistencies from concurrent modifications
final @Nullable CopyOnWriteArrayList<FeatureFlagEntry> globalFlags =
globalBuffer == null ? null : globalBuffer.flags;
globalBuffer == null ? null : new CopyOnWriteArrayList<>(globalBuffer.flags);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this need to be a CopyOnWriteArrayList ? could it be Collections.unmodifiableList or just an ArrayList ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We make use of the optimization in modern JDK / Android:

    public CopyOnWriteArrayList(Collection<? extends E> c) {
        Object[] es;
        if (c.getClass() == CopyOnWriteArrayList.class)
            es = ((CopyOnWriteArrayList<?>)c).getArray();
        else {
            es = c.toArray();
            if (c.getClass() != java.util.ArrayList.class)
                es = Arrays.copyOf(es, es.length, Object[].class);
        }
        setArray(es);
    }

This avoids copying the underlying array and instead reuses it until the next modification.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah gotcha. I think we should add a comment about that for future readers of the codebase since this is somewhat unexpected here since we never modify the list.

adinauer and others added 3 commits August 25, 2026 14:36
The buffer mutates one CopyOnWriteArrayList wrapper and never replaces the
field after construction. Mark the reference final while relying on the
list's own memory visibility guarantees for its contents.

Refs JAVA-704
Co-Authored-By: Claude <noreply@anthropic.com>
Use 1,000 merge iterations to reduce CI time while retaining regression
coverage. The old implementation reproduced the merge crash in 20 out of
20 clean local runs at this iteration count.

Refs JAVA-704
Co-Authored-By: Claude <noreply@anthropic.com>
Document why merge snapshots pass CopyOnWriteArrayList instances directly
to the collection constructor and how runtimes can avoid copying elements.

Co-Authored-By: Claude <noreply@anthropic.com>
@adinauer
adinauer marked this pull request as ready for review August 25, 2026 13:56
@adinauer
adinauer enabled auto-merge (squash) August 25, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FeatureFlagBuffer.merged() can throw and drop events under concurrency

2 participants