Skip to content

fix(stream): keep pending counters consistent - #3596

Draft
wengsht wants to merge 1 commit into
apache:unstablefrom
wengsht:fix-stream-pending-accounting
Draft

fix(stream): keep pending counters consistent#3596
wengsht wants to merge 1 commit into
apache:unstablefrom
wengsht:fix-stream-pending-accounting

Conversation

@wengsht

@wengsht wengsht commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Problem

Stream pending counters can diverge from the real PEL and wrap past INT64_MAX, making XINFO GROUPS and XINFO CONSUMERS undecodable.

Three related command behaviors cause or amplify the divergence:

  1. XACK key group id id reads the same PEL record twice from one snapshot, reports two acknowledgements, deletes one record, and decrements the cached group/consumer counters twice.
  2. XAUTOCLAIM returns an expired entry already owned by the target consumer without refreshing its delivery time. Repeating the command returns the same ID again.
  3. XCLAIM to the current consumer decrements and increments the same consumer through separate metadata writes; the final write increments its cached pending count.

Together, a reclaim loop can produce multiple deliveries of one ID, a client can coalesce those duplicate IDs into one XACK, and the unsigned counters underflow.

Minimal reproduction before this patch:

XREADGROUP ... >              # one real PEL entry
XAUTOCLAIM ... consumer ...   # returns 1-0
XAUTOCLAIM ... consumer ...   # returns 1-0 again
XACK key group 1-0 1-0       # returns 2
XPENDING key group            # real PEL = 0
XINFO GROUPS key              # Bad integer value

Change

  • Deduplicate IDs within XACK and XCLAIM.
  • Make group and consumer pending decrements saturating as a final corruption guard.
  • Aggregate XCLAIM ownership decrements per original consumer.
  • Keep same-consumer XCLAIM ownership accounting neutral.
  • Refresh XAUTOCLAIM delivery time for every claimed entry, including entries already owned by the target consumer; preserve JUSTID retry-count semantics.

Tests

Command-level regressions verify:

  • duplicate XACK IDs acknowledge one PEL record and leave all pending views at zero;
  • same-consumer XCLAIM leaves group and consumer counts unchanged;
  • same-consumer XAUTOCLAIM refreshes idle time and cannot immediately return the same ID again.

Validation:

  • ./x.py format
  • ./x.py check format
  • fresh ./x.py build build-make --unittest -j 8
  • targeted Go stream regressions pass against the fresh binary

./x.py check tidy could not run locally because run-clang-tidy is not installed.

AI assistance was used for diagnosis and drafting; I reviewed the code, reproduction, and tests.

@wengsht
wengsht force-pushed the fix-stream-pending-accounting branch 2 times, most recently from 7368c48 to 4a48cae Compare August 18, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant