[Blob] __blobstorage__ grows without bound: GC cannot reclaim partially-referenced extents

[JeelGajera]

Which service(blob, file, queue, table) does this issue concern?

Blob (the queue service shares FSExtentStore and is affected in the same way).

Which version of the Azurite was used?

3.37.0, and reproduced against main.

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub in development; reproduced locally from a build of main (npm).

What's the Node.js version?

v22.22.2 (Linux x64).

What problem was encountered?

On a long-running Azurite container (~month, persistent volume, continuous
overwrite/delete churn), __blobstorage__ grew to ~52 GB while the live blob
data was only ~500 MB. The GC never reclaims it.

This is not GC lag and not a GC failure — the GC is behaving exactly as
designed. The space is structurally unreclaimable:

• FSExtentStore.appendExtent() packs many blobs into shared extent files of
  up to DEFAULT_MAX_EXTENT_SIZE (64 MB, src/common/utils/constants.ts).
• BlobGCManager.markSweep() is whole-file mark-and-sweep. It builds a set of
  all extent IDs, removes every ID that is still referenced, and deletes what
  remains. There is no chunk-level accounting, so an extent is deleted only
  when every reference into it is gone.
• BlobReferredExtentsAsyncIterator yields only persistency.id, discarding
  offset/count, so per-extent live-byte usage is not computed anywhere.

Consequently a single surviving 1 KB blob pins its entire 64 MB extent
forever. Under sustained churn, long-lived blobs get scattered across many
extents that were mostly filled with data that has since been deleted, and disk
usage grows without bound. ~52 GB across ~830 extents at ~1% utilization each
is exactly what this predicts after a months.

The class comment on BlobGCManager already anticipates the missing piece:

In the future, GC manager can also help merging small extent mapped files
into one big file to improve the performance.

Measured on a build of main with the attached script — 10 rounds of "write
one 1 MB blob that is kept, then 56 MB of blobs written and deleted
immediately":

--- immediately after churn
  extent files on disk                        : 9 (570.0 MB)
  live referenced bytes                       : 10.0 MB
  dead bytes pinned in referenced extents     : 512.0 MB  <-- never reclaimable
  bytes in fully-unreferenced extents         : 48.0 MB

--- after 22 minutes of GC running
  extent files on disk                        : 9 (570.0 MB)
  live referenced bytes                       : 10.0 MB
  dead bytes pinned in referenced extents     : 512.0 MB
  bytes in fully-unreferenced extents         : 48.0 MB

  per-extent utilization:
    28ff88b7  size=66.0 MB  live=2.0 MB  util=3.03%
    545789d7  size=65.0 MB  live=1.0 MB  util=1.54%
    247365e3  size=65.0 MB  live=1.0 MB  util=1.54%
    86f669c4  size=65.0 MB  live=1.0 MB  util=1.54%
    57dd5047  size=65.0 MB  live=1.0 MB  util=1.54%
    839e2170  size=65.0 MB  live=1.0 MB  util=1.54%
    39b06b69  size=65.0 MB  live=1.0 MB  util=1.54%
    4c9020d5  size=66.0 MB  live=2.0 MB  util=3.03%
    48932e93  size=48.0 MB  live=0.0 MB  util=0.00%

570 MB of disk holding 10 MB of live data. 22 minutes covers more than two full
mark-sweep cycles past the 10-minute DEFAULT_EXTENT_GC_PROTECT_TIME_IN_MS
window, and the GC deleted nothing — correctly, since the only fully
unreferenced extent (48932e93) is the active write extent that
FSExtentStore.isActiveExtent() deliberately skips.

The only workaround today is deleting __blobstorage__, which destroys the
live data along with the dead bytes.

Steps to reproduce the issue?

Attached: `

azurite-extent-fragmentation-repro.js

`. It drives the churn through
the public blob API and then classifies every byte in the extent store as live,
dead-but-pinned, or reclaimable by reading Azurite's own metadata.

npm install @azure/storage-blob
npx azurite-blob --location ./azurite-data --silent &
node azurite-extent-fragmentation-repro.js ./azurite-data

ROUNDS and WAIT_MINUTES are configurable; the default 22-minute wait is
chosen to exceed the GC interval plus the extent protect time.

Have you found a mitigation/solution?

Yes — extent compaction, built on top of the existing GC rather than replacing
it. I have a working prototype and would be glad to open a PR if the approach
looks acceptable.

The idea: identify extents whose live-byte ratio is below a threshold, copy
their live chunks out through the normal appendExtent() path, and rewrite the
blob metadata references to the new locations.

The key design property is that compaction never deletes anything. Once its
live chunks have been relocated, the emptied extent is simply unreferenced, and
the existing mark-and-sweep deletes it on a later pass — still behind the
protect window and the isActiveExtent() guard. That makes the concurrency
story tractable:

• in-flight reads against the old extent keep working, because the file is
  still there;
• chunks shared between blobs (snapshots and startCopyFromURL both copy
  persistency references) are safe — relocating only some references means
  the extent just isn't reclaimed this round;
• if a blob is deleted or overwritten mid-relocation, the store re-reads the
  document, sees the chunks no longer match, and skips the update; the
  freshly written copy becomes ordinary garbage that the GC reclaims.

Additionally, each relocated chunk would be read back from its new location and
compared against the source before its metadata reference is updated, so a
faulty relocation costs disk space rather than corrupting a blob.

Prototype result on the data directory above — 570 MB to 58 MB, with all blobs
verified byte-identical after a restart (the residual 48 MB is the old active
write extent, which normal GC removes once it is no longer active):

before: 570.0 MB on disk, 9 extents, 10.0 MB live
selected 8 extents below 25% utilization
moved 10.0 MB of live data
after:   58.0 MB on disk, 2 extents
verified 10 blobs, 0 mismatches

Sketch of the change: a new IBlobMetadataStore method to relocate chunk
references (implemented for Loki and SQL — SQL only needs the persistency and
committedBlocksInOrder columns, since it does not persist page ranges), a new
BlobExtentCompactor under src/blob/gc/, a call site in markSweep(), and an
opt-in CLI flag so existing behaviour is unchanged by default. Extents
referenced by uncommitted blocks would be excluded from candidacy rather than
partially relocated.

Questions before I invest in a PR:

1. Is compaction in the GC the direction you would want, or would you prefer an
   offline/maintenance-mode command that operates on a stopped data directory?
2. Should it be opt-in behind a flag, or on by default with a conservative
   utilization threshold and a per-cycle byte budget?
3. Would you want the same treatment for the queue service in the same change,
   or kept separate?

Two unrelated minor things noticed while investigating, happy to fold into a
separate PR:

• LokiExtentMetadataStore.updateExtent() writes lastModifiedInMS on insert
  but LastModifyInMS on update, so a freshly inserted extent does not match
  the protect-time predicate in listExtents() in the intended way.
• BlobGCManager.markSweep() logs allExtents.entries (a function reference)
  where the count was intended.