diff --git a/docs/source/deployment/ssd/ssd-offload.md b/docs/source/deployment/ssd/ssd-offload.md index cd10b3c618..d7426da173 100644 --- a/docs/source/deployment/ssd/ssd-offload.md +++ b/docs/source/deployment/ssd/ssd-offload.md @@ -156,6 +156,9 @@ Applies when `MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=bucket_storage_backend | `MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT` | `500` | Max keys per bucket | | `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` | `0` | Eviction threshold in bytes. When set to `0`, the backend uses **90% of the physical disk capacity** as the quota — it does not mean unlimited. Set an explicit value to control disk usage precisely. | | `MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY` | `fifo` | Eviction policy: `none` / `fifo` / `lru` | +| `MOONCAKE_OFFLOAD_BUCKET_GC_ENABLE` | `true` | Enable the single background worker that reclaims tombstoned records | +| `MOONCAKE_OFFLOAD_BUCKET_GC_INTERVAL_SECONDS` | `10` | Normal GC scan interval | +| `MOONCAKE_OFFLOAD_BUCKET_GC_DELETED_RATIO` | `0.25` | Minimum deleted-byte ratio for normal GC; disk pressure bypasses this threshold | ### File-per-key backend settings @@ -250,7 +253,7 @@ Eviction is two-phase: the bucket is removed from metadata and master is notifie ### Proactive watermark eviction -When `MOONCAKE_OFFLOAD_ENABLE_DISK_WATERMARK_EVICTION=true`, the FileStorage heartbeat asks the backend to check local-disk usage every `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS` seconds. If usage exceeds `MOONCAKE_OFFLOAD_DISK_EVICTION_HIGH_WATERMARK_RATIO`, the backend evicts toward `MOONCAKE_OFFLOAD_DISK_EVICTION_LOW_WATERMARK_RATIO`. +When `MOONCAKE_OFFLOAD_ENABLE_DISK_WATERMARK_EVICTION=true`, the FileStorage heartbeat asks the backend to check local-disk usage every `MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS` seconds. If usage exceeds `MOONCAKE_OFFLOAD_DISK_EVICTION_HIGH_WATERMARK_RATIO`, the bucket backend first asks GC to reclaim tombstoned records toward the low watermark. Live-bucket eviction remains the fallback when GC is disabled or no reclaimable bytes remain. | Backend | Behavior | |---------|----------| @@ -262,6 +265,28 @@ The watermark ratios apply to each backend's quota. For `bucket_storage_backend` For watermark eviction, the real client notifies the master before deleting local files. If the notification fails, the selected files remain tracked locally and are retried by a later heartbeat. +### Object deletion and bucket GC + +For `bucket_storage_backend`, `Remove` and `BatchRemove` also schedule deletion +for completed `LOCAL_DISK` replicas. The master stops exposing the logical +object first. During a subsequent real-client heartbeat, the SSD holder +persists a tombstone for the matching object incarnation and acknowledges the +task. A delayed task for an older incarnation cannot delete a newly created +object with the same key. + +Tombstoning makes the deleted object unavailable but does not immediately +shrink its immutable bucket file. The single background GC worker reclaims that +space by unlinking fully dead buckets or merging live records from up to eight +partially dead buckets. Normal GC uses +`MOONCAKE_OFFLOAD_BUCKET_GC_DELETED_RATIO`; after disk usage reaches the high +watermark, GC bypasses that ratio and works toward the low watermark. + +Deletion tasks survive HA failover through the master OpLog. Interrupted local +bucket replacement is recovered independently from `.bucket_gc_intent`. Keep +that file with the bucket data when inspecting or recovering an SSD directory. +See [Reliable SSD Object Deletion and Bucket Garbage Collection](../../design/ssd-object-deletion-and-gc.md) +for protocol, crash-recovery, and validation details. + --- ## Example diff --git a/docs/source/design/index.md b/docs/source/design/index.md index 6b69421b4e..f6de772fdb 100644 --- a/docs/source/design/index.md +++ b/docs/source/design/index.md @@ -24,6 +24,7 @@ distributed execution components. | [Engram](engram) | Distributed serving and cache architecture. | | [Unified Parallel Tensor I/O](unified-parallel-tensor-io) | Parallel tensor storage and transfer model. | | [SSD Offload](ssd-offload) | SSD-backed cache hierarchy design. | +| [SSD Object Deletion and GC](ssd-object-deletion-and-gc) | Durable per-object deletion and bucket-space reclamation. | | [SSD Free-Ratio-First Allocation](ssd-free-ratio-first-allocation) | Capacity-aware replica placement strategy. | ## Distributed Execution and Routing diff --git a/docs/source/design/ssd-object-deletion-and-gc.md b/docs/source/design/ssd-object-deletion-and-gc.md new file mode 100644 index 0000000000..b31f157451 --- /dev/null +++ b/docs/source/design/ssd-object-deletion-and-gc.md @@ -0,0 +1,272 @@ +# Reliable SSD Object Deletion and Bucket Garbage Collection + +## Overview + +Mooncake Store can keep a `LOCAL_DISK` replica after its logical object has +been removed from the master. For the bucket storage backend, unlinking that +replica immediately is not possible because one immutable bucket file contains +multiple objects. + +This design connects `Remove` and `BatchRemove` to durable per-replica delete +tasks. The local-disk holder first records a tombstone in bucket metadata, then +a single background worker reclaims the dead records. The worker can remove +fully dead buckets or merge live records from several partially dead buckets +into one replacement bucket. + +The design separates two responsibilities: + +- the master records which object incarnation must be deleted; and +- the local-disk holder records how that deletion changes its local files. + +This separation keeps object deletion correct across master failover and holder +restart without putting bucket-copy I/O on the remove RPC path. + +## Scope + +The implementation covers: + +- `Remove` and `BatchRemove`; +- completed `LOCAL_DISK` replicas that advertise tombstone support; +- the bucket storage backend; +- durable tombstones, background reclamation, and interrupted-GC recovery; +- HA replication of delete tasks and acknowledgements. + +It does not change `RemoveAll` or `RemoveByRegex`, and it does not add +per-object reclamation to the file-per-key, offset-allocator, distributed, P2P, +NoF, or DFS backends. It also deliberately uses one GC worker rather than a +general-purpose task scheduler. + +## End-to-end flow + +```mermaid +flowchart LR + A["Remove or BatchRemove"] --> B["Master validates the object and completed LOCAL_DISK replicas"] + B --> C["Reserve bounded delete tasks"] + C --> D{"HA enabled?"} + D -- "Yes" --> E["Persist REMOVE payload in OpLog"] + D -- "No" --> F["Publish pending tasks"] + E --> G["Durable callback publishes tasks"] + E --> H["Standby replays the same tasks"] + F --> I["Holder fetches tasks during heartbeat"] + G --> I + H --> I + I --> J["Persist matching tombstones by bucket"] + J --> K["Acknowledge terminal task IDs"] + K --> L{"HA enabled?"} + L -- "Yes" --> M["Persist LOCAL_DELETE_ACK"] + L -- "No" --> N["Remove acknowledged tasks"] + M --> N + J --> O["Wake the background GC worker"] + O --> P["Unlink fully dead buckets"] + O --> Q["Merge partially dead buckets"] +``` + +Task delivery is at least once. Fetching does not remove a task; only an +acknowledgement removes it. Repeated delivery is safe because both the task and +the bucket entry carry the same object incarnation. + +## Delete identity and fencing + +### Object incarnation + +Every logical object has a 128-bit `ObjectIncarnation`, generated when the +master creates the object. The incarnation is propagated through master +metadata, snapshots, offload tasks, local-disk replica descriptors, bucket +metadata, and delete tasks. + +The holder writes a tombstone only when both the key and incarnation match. If +an old task for incarnation N arrives after the same key has been recreated as +incarnation N+1, the task is terminal but does not modify the new object. This +prevents delayed work from causing an ABA-style deletion. + +### Stable local-disk identity + +The bucket directory stores a stable `local_disk_segment_id` in +`.mooncake_local_disk_segment_id`. A master-issued mount epoch fences old +processes after the directory is remounted, and a local advisory lock prevents +two processes on the same host from using the directory concurrently. + +Fetch and acknowledgement requests include both the stable identity and mount +epoch. A stale holder therefore cannot consume or acknowledge work belonging +to the current mount. + +## Master-side durability + +Before changing logical object visibility, the master reserves enough bounded +registry capacity for all eligible local-disk replicas. If the complete set +cannot be reserved, the remove operation fails without publishing a partial +set of physical-delete tasks. + +In HA mode, the versioned `REMOVE` OpLog payload contains the object +incarnation and delete intents. Tasks become visible after the OpLog entry is +durable, and the standby applies the same metadata removal and tasks. +Acknowledgements use a separate `LOCAL_DELETE_ACK` entry. Pending tasks are +also included in master snapshots, so snapshot bootstrap and OpLog replay +produce the same registry state. + +In non-HA mode, the reservation is published directly after validation. + +## Durable tombstones + +The holder groups a fetched batch by bucket. Each affected bucket requires at +most one metadata rewrite: + +```mermaid +flowchart TD + A["Copy bucket metadata"] --> B["Match key and incarnation"] + B --> C["Set tombstoned = true"] + C --> D["Write a temporary metadata file"] + D --> E["fsync the file"] + E --> F["Rename over the active metadata"] + F --> G["fsync the directory"] + G --> H["Remove the exact incarnation from the live index"] + H --> I["Return a terminal result"] + D -. "Failure" .-> J["Retryable failure; do not ACK"] + E -. "Failure" .-> J + F -. "Failure" .-> J + G -. "Failure" .-> J +``` + +The result determines acknowledgement behavior: + +| Result | Meaning | Acknowledge | +|---|---|---| +| `Removed` | The matching tombstone was persisted by this attempt. | Yes | +| `AlreadyRemoved` | The same incarnation was already tombstoned. | Yes | +| `StaleVersion` | That incarnation is no longer present. | Yes | +| `RetryableFailure` | A file or internal operation failed. | No | + +Once the tombstone is durable, the entry is absent from local lookup, +`BatchLoad`, metadata scans, and restart re-registration. The bucket data file +still occupies its original physical space until GC completes. + +## Bucket garbage collection + +### Scheduling + +Each `BucketStorageBackend` owns one background GC worker. A new tombstone +wakes it, and the worker also scans periodically. Under normal usage, a bucket +becomes eligible when its deleted-byte ratio reaches +`MOONCAKE_OFFLOAD_BUCKET_GC_DELETED_RATIO`. + +Candidates are ordered by: + +1. fully dead buckets; +2. higher deleted-byte ratio; +3. more reclaimable bytes; +4. lower bucket ID. + +The current configuration is documented in the +[SSD Offload deployment guide](../deployment/ssd/ssd-offload.md). + +### Watermark behavior + +GC reuses the existing SSD high and low watermarks. When accounted usage +reaches the high watermark, the deleted-ratio threshold is bypassed and the +worker continues reclaiming eligible dead records until one of these +conditions is met: + +- usage reaches the low watermark; +- no reclaimable candidate remains; +- a reclamation operation fails; or +- the worker is stopping. + +The heartbeat only signals the GC worker; compaction I/O runs in the background. +When GC is disabled or no dead bytes remain, the existing live-bucket eviction +path remains available for disk-pressure protection. + +### Multi-bucket merge + +One GC operation selects at most eight source buckets. The combined live data +must fit the existing per-bucket byte and key limits. Fully dead buckets do not +need a replacement; partially dead buckets use copy-on-write: + +```mermaid +flowchart TD + A["Select and lock up to eight sources"] --> B["Persist PREPARED GC intent"] + B --> C["Stream live records to a replacement"] + C --> D["fsync replacement data and metadata"] + D --> E["Stage replacement index entries"] + E --> F["Persist COMMITTED GC intent"] + F --> G["Switch live mappings"] + G --> H["Remove source metadata"] + H --> I["Wait for existing readers"] + I --> J["Remove source data"] + J --> K["Clear the GC intent"] +``` + +Live records are copied through a fixed 1 MiB buffer, so memory usage does not +grow with bucket size. The replacement is counted in physical usage before the +sources are subtracted; source bytes are released from accounting only after +their files are removed. + +## Failure recovery + +The master OpLog and local GC intent solve different failure domains: + +| Failure | Durable state | Recovery | +|---|---|---| +| Primary fails before publishing a durable remove | No committed remove entry | No partial task set becomes visible. | +| Primary fails after durable `REMOVE` | OpLog or snapshot contains the tasks | The standby restores and redelivers them. | +| ACK response is lost | Tombstone is durable; task may remain pending | Redelivery returns `AlreadyRemoved`, then ACK is retried. | +| Holder fails after tombstone but before ACK | Tombstone is in synchronized bucket metadata | Restart does not re-register the object; the task can be redelivered. | +| Holder fails with a PREPARED GC intent | Sources remain authoritative | Recovery removes an incomplete replacement and keeps the sources. | +| Holder fails with a COMMITTED GC intent | Replacement is authoritative | Recovery validates the replacement and removes the sources. | + +The OpLog cannot describe which local source or replacement file is +authoritative during compaction. That is why `.bucket_gc_intent` is required in +addition to the cluster-level delete log. + +## Performance boundaries + +- `Remove` and `BatchRemove` do not copy bucket data. +- Tombstone persistence rewrites bucket metadata in the holder heartbeat path. +- GC data copying runs on one background worker. +- A merge reads at most eight source buckets and uses a fixed 1 MiB buffer. +- The global bucket lock is not held while copying data or waiting for readers. +- Below the high watermark, watermark checks do not start compaction unless + reclaimable data is present. + +GC necessarily introduces read and write traffic for live records. Production +evaluation should compare read latency, GC throughput, write amplification, +disk bandwidth, and resident memory with representative bucket sizes. + +## Validation + +The focused tests require no CUDA, RDMA, or UB hardware. Enable failpoints to +include the primary-process termination cases: + +```bash +cmake -S . -B build-ssd-delete -G Ninja \ + -DWITH_STORE=ON \ + -DWITH_EP=OFF \ + -DUSE_CUDA=OFF \ + -DBUILD_UNIT_TESTS=ON \ + -DBUILD_EXAMPLES=OFF \ + -DMOONCAKE_ENABLE_TEST_FAILPOINTS=ON + +cmake --build build-ssd-delete --target \ + storage_backend_bucket_delete_test \ + local_delete_test \ + oplog_applier_test \ + local_delete_process_kill_test + +ctest --test-dir build-ssd-delete --output-on-failure -R \ + '^(storage_backend_bucket_delete_test|local_delete_test|oplog_applier_test|local_delete_process_kill_test)$' +``` + +The focused suite covers tombstone durability, stale-incarnation protection, +at-least-once task delivery, OpLog replay, process termination, fully dead +buckets, multi-bucket merge, watermark override, and PREPARED/COMMITTED intent +recovery. Changes to this protocol should also run the existing Store snapshot, +HA, SSD, storage-backend, remove, and client integration regression tests. + +## Code organization + +| Area | Primary files | +|---|---| +| Delete task registry and identifiers | `mooncake-store/include/local_delete.h`, `mooncake-store/src/local_delete.cpp` | +| Remove, fetch/ACK, OpLog, and snapshots | `mooncake-store/src/master_service.cpp` and `mooncake-store/src/ha/` | +| Holder task processing and watermark signaling | `mooncake-store/src/file_storage.cpp` | +| Tombstones, GC, and local intent recovery | `mooncake-store/src/storage_backend.cpp` | +| Focused tests | `mooncake-store/tests/local_delete_test.cpp`, `mooncake-store/tests/storage_backend_bucket_delete_test.cpp`, and `mooncake-store/tests/ha/oplog/local_delete_process_kill_test.cpp` | diff --git a/docs/source/design/ssd-offload.md b/docs/source/design/ssd-offload.md index 1ca2607499..e80c6a0803 100644 --- a/docs/source/design/ssd-offload.md +++ b/docs/source/design/ssd-offload.md @@ -271,3 +271,7 @@ To prevent `io_uring`'s `FOLL_LONGTERM` page pinning from failing on systems wit On startup, `FileStorage::Init` calls `StorageBackend::ScanMeta`, which reads on-disk metadata and invokes a callback for each discovered object. The callback calls `MasterClient::NotifyOffloadSuccess` to re-register the objects with the master. This restores the full disk-replica view without any application-level intervention for the backends that preserve restart metadata, namely `BucketStorageBackend` and the file-per-key backend. `OffsetAllocatorStorageBackend` is the exception. It truncates its pre-allocated data file during initialization and clears its in-memory metadata, so previously offloaded objects are not recoverable after a real client restart. + +For durable `Remove` and `BatchRemove` handling in the bucket backend, including +tombstones, watermark-triggered reclamation, and multi-bucket compaction, see +[Reliable SSD Object Deletion and Bucket Garbage Collection](ssd-object-deletion-and-gc.md). diff --git a/docs/source/index.md b/docs/source/index.md index 4f980bcc30..720a4cbcfe 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -136,6 +136,7 @@ design/tent/overview design/tent/tebench design/conductor/conductor-architecture-design design/ssd-offload +design/ssd-object-deletion-and-gc design/ssd-free-ratio-first-allocation ::: diff --git a/mooncake-store/include/local_delete.h b/mooncake-store/include/local_delete.h new file mode 100644 index 0000000000..2f4fbe5817 --- /dev/null +++ b/mooncake-store/include/local_delete.h @@ -0,0 +1,86 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "types.h" + +namespace mooncake { + +class LocalDeleteRegistry { + public: + class Reservation { + public: + ~Reservation(); + Reservation(const Reservation&) = delete; + Reservation& operator=(const Reservation&) = delete; + + const std::vector& tasks() const { return tasks_; } + void Publish(); + + private: + friend class LocalDeleteRegistry; + Reservation(LocalDeleteRegistry* registry, + std::vector tasks); + + LocalDeleteRegistry* registry_; + std::vector tasks_; + bool released_{false}; + }; + + explicit LocalDeleteRegistry(size_t capacity = 50000) + : capacity_(capacity) {} + + tl::expected, ErrorCode> Reserve( + std::vector tasks); + + LocalDiskMountInfo Mount(const UUID& client_id, + const std::string& local_disk_segment_id, + uint32_t capabilities); + void Unmount(const UUID& client_id); + + tl::expected, ErrorCode> Fetch( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch, uint32_t limit) const; + + tl::expected ValidateMount( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch) const; + + size_t Erase(const std::string& local_disk_segment_id, + const std::vector& task_ids); + + bool ApplyDurableTasks(const std::vector& tasks); + std::vector Snapshot() const; + bool Restore(const std::vector& tasks); + void Reset(); + size_t Size() const; + + private: + struct MountState { + UUID client_id{0, 0}; + uint64_t mount_epoch{0}; + uint32_t capabilities{0}; + }; + + void ReleaseReservation(size_t count); + void PublishReservation(std::vector tasks); + + mutable std::mutex mutex_; + const size_t capacity_; + size_t reserved_{0}; + std::unordered_map mounts_; + std::map storage_by_client_; + std::unordered_map> + pending_; +}; + +ObjectIncarnation GenerateObjectIncarnation(); +LocalDeleteTaskId GenerateLocalDeleteTaskId(); + +} // namespace mooncake diff --git a/mooncake-store/include/master_service.h b/mooncake-store/include/master_service.h index f1f14e1990..b8f02ebe0f 100644 --- a/mooncake-store/include/master_service.h +++ b/mooncake-store/include/master_service.h @@ -39,6 +39,7 @@ #include "ha/snapshot/object/snapshot_object_store.h" #include "task_manager.h" #include "kv_event/kv_event_publisher.h" +#include "local_delete.h" #include "ha/oplog/oplog_types.h" #include "ha/oplog/ordered_oplog_writer.h" #include "allocator.h" @@ -77,6 +78,10 @@ class SnapshotChildProcessTest; // exposing test-only accessors on MasterService itself. class PromotionOnHitTest; class MasterServiceTenantQuotaTest; +// Friended so the BatchEvict correctness tests can invoke the private +// BatchEvict entry point and seed lease timestamps directly, instead of +// relying on segment pressure plus the background eviction thread. +class BatchEvictTest; class MasterServiceHATest; } // namespace test namespace benchmarks { @@ -107,6 +112,7 @@ class MasterService { friend class test::PromotionOnHitTest; friend class benchmarks::BatchEvictBench; friend class test::MasterServiceTenantQuotaTest; + friend class test::BatchEvictTest; friend class MasterSnapshotManager; // Allow access to internal state for // snapshot friend class ha::MasterSnapshotCodec; // Allow codec to access private @@ -673,8 +679,11 @@ class MasterService { * @brief Mounts a file storage segment into the master. * @param enable_offloading If true, enables offloading (write-to-file). */ - auto MountLocalDiskSegment(const UUID& client_id, bool enable_offloading) - -> tl::expected; + auto MountLocalDiskSegment( + const UUID& client_id, bool enable_offloading, + const std::string& local_disk_segment_id = std::string(), + uint32_t capabilities = 0) + -> tl::expected; /** * @brief Heartbeat call to collect object-level statistics and retrieve the @@ -685,6 +694,23 @@ class MasterService { auto OffloadObjectHeartbeat(const UUID& client_id, bool enable_offloading) -> tl::expected, ErrorCode>; + auto FetchLocalDeleteTasks(const UUID& client_id, + const std::string& local_disk_segment_id, + uint64_t mount_epoch, uint32_t limit) + -> tl::expected, ErrorCode>; + + auto AckLocalDeleteTasks(const UUID& client_id, + const std::string& local_disk_segment_id, + uint64_t mount_epoch, + const std::vector& task_ids) + -> tl::expected; + + auto ReconcileLocalDiskObjects(const UUID& client_id, + const std::string& local_disk_segment_id, + uint64_t mount_epoch, + const std::vector& objects) + -> tl::expected, ErrorCode>; + /** * @brief Client polls whether master has requested a full SSD clear * (triggered by RemoveAll). Atomically checks and clears the flag. @@ -710,13 +736,6 @@ class MasterService { const std::vector& metadatas) -> tl::expected; - /** - * @brief Returns unique RPC endpoints that own completed LOCAL_DISK - * replicas. Used by clients to warm up offload RPC pools before traffic. - */ - auto GetOffloadEndpoints() - -> tl::expected, ErrorCode>; - /** * @brief Heartbeat-driven pull of pending promotion work for a client. * Returns tenant-scoped promotion tasks for the holder client and clears @@ -727,13 +746,6 @@ class MasterService { auto PromotionObjectHeartbeat(const UUID& client_id) -> tl::expected, ErrorCode>; - /** Fetch pending remove tasks without removing them from the queue. */ - auto RemoveObjectHeartbeat(const UUID& client_id) - -> tl::expected, ErrorCode>; - auto AckRemoveObjectHeartbeat( - const UUID& client_id, const std::vector& tasks) - -> tl::expected; - /** * @brief Stage a PROCESSING MEMORY replica for an existing key. Allocates * DRAM via the existing AllocationStrategy, optionally biased toward the @@ -832,10 +844,11 @@ class MasterService { * @brief Restore primary state from standby promotion context. * Called once at promotion time before serving requests. */ - void RestoreFromStandbySnapshot( + tl::expected RestoreFromStandbySnapshot( const std::vector& objects, uint64_t initial_oplog_sequence_id, - const std::vector& segments); + const std::vector& segments, + const std::vector& pending_local_deletes = {}); /** * @brief Query the status of a task @@ -953,7 +966,8 @@ class MasterService { bool enable_soft_pin, bool enable_hard_pin = false, ObjectDataType data_type_ = ObjectDataType::UNKNOWN, std::string group_id_ = "", TenantId tenant_id_ = TenantId(), - std::string user_key_ = {}) + std::string user_key_ = {}, + std::optional object_incarnation_ = std::nullopt) : client_id(client_id_), put_start_time(put_start_time_), size(value_length), @@ -961,6 +975,8 @@ class MasterService { group_id(std::move(group_id_)), tenant_id(std::move(tenant_id_)), user_key(std::move(user_key_)), + object_incarnation( + object_incarnation_.value_or(GenerateObjectIncarnation())), lease_timeout(), soft_pin_timeout(std::nullopt), hard_pinned(enable_hard_pin), @@ -988,6 +1004,7 @@ class MasterService { const std::string group_id; const TenantId tenant_id; const std::string user_key; + const ObjectIncarnation object_incarnation; mutable SpinLock lock; // Default constructor, creates a time_point representing @@ -1256,11 +1273,6 @@ class MasterService { struct OffloadingTask { ReplicaID source_id; std::chrono::system_clock::time_point start_time; - // Client whose LOCAL_DISK segment owns the queued offload task. - // A key may have one task per offloading node (one per MEMORY - // replica), so tasks are matched by (key, source_client_id) when - // workers report completion or failure. - UUID source_client_id; }; // Tracks an in-flight LOCAL_DISK -> MEMORY copy. The source @@ -1331,8 +1343,7 @@ class MasterService { std::unordered_set processing_keys; std::unordered_map replication_tasks; - std::unordered_map> - offloading_tasks; + std::unordered_map offloading_tasks; std::unordered_map promotion_tasks; std::unordered_map promotion_candidates; @@ -1539,9 +1550,14 @@ class MasterService { MetadataShardAccessorRW* shard); void FinalizeRemovedReplicasAfterDurable( const OpLogEntry& durable_entry, - const std::vector& replica_ids, QuotaEraseMode quota_mode); - void EnqueueRemoveTasks(const std::vector& holder_ids, - const RemoveTaskItem& task); + const std::vector& replica_ids, QuotaEraseMode quota_mode, + std::shared_ptr delete_reservation = + nullptr); + tl::expected, ErrorCode> + ReserveLocalDeleteTasks(const ObjectIdentity& object_id, + const ObjectMetadata& metadata); + void PublishLocalDeleteReservation( + const std::shared_ptr& reservation); void FinalizeMetadataEraseAfterDurable(const OpLogEntry& durable_entry, QuotaEraseMode quota_mode); void FinalizeExpiredProcessingReplicasAfterDurable( @@ -1608,12 +1624,9 @@ class MasterService { bool ProbeNoFSegment(const std::string& te_endpoint, std::string* error_reason); - // Queues an offload task for `replica` on the LOCAL_DISK segment(s) - // mapped from the replica's MEMORY segment name(s). Returns the - // client_id(s) of the segment(s) the task was actually enqueued on; - // callers use them to record one protected OffloadingTask per queue. - tl::expected, ErrorCode> PushOffloadingQueue( - const ObjectIdentity& object_id, Replica& replica); + tl::expected PushOffloadingQueue( + const ObjectIdentity& object_id, Replica& replica, + ObjectIncarnation object_incarnation); struct GracefulUnmountDeadlineRecord { UUID segment_id; @@ -2058,11 +2071,6 @@ class MasterService { // offload_on_evict_=true) bool offload_force_evict_{false}; - // Strict replica allocation: memory-only multi-replica requests must - // allocate exactly replica_num replicas instead of best-effort - // degradation (config: strict_replica_allocation) - bool strict_replica_allocation_{false}; - // Promotion-on-hit: opt-in flag enabling LOCAL_DISK -> MEMORY promotion // when a Get observes a key with only LOCAL_DISK replicas. bool promotion_on_hit_{false}; @@ -2214,6 +2222,7 @@ class MasterService { std::list discarded_replicas_ GUARDED_BY(discarded_replicas_mutex_); size_t offloading_queue_limit_ = 50000; + LocalDeleteRegistry local_delete_registry_{50000}; double offload_cap_ratio_ = 0.5; // Task manager @@ -2304,10 +2313,8 @@ class MasterService { std::string SerializeMetadataForOpLogWithoutMemReplicas( const ObjectMetadata& metadata) const; std::string SerializeMetadataForOpLogFromReplicaDescriptors( - const UUID& client_id, uint64_t size, - const std::vector& replicas, - const std::string& group_id = "", - ObjectDataType data_type = ObjectDataType::UNKNOWN) const; + const ObjectMetadata& metadata, + const std::vector& replicas) const; ErrorCode InitializeBatchOpLogWriter(std::shared_ptr backend); tl::expected AppendOpLogVisibleBeforeDurable( OpType type, const std::string& tenant_id, const std::string& key, @@ -2335,7 +2342,6 @@ class MasterService { ErrorCode ValidateStandbyRemountSegment(const Segment& segment) const; bool IsReplicaReadable(const Replica& replica) const; - bool IsMemoryReplicaEvictable(const Replica& replica) const; /** * Segment lifecycle persist helper. Tries to durably persist the diff --git a/mooncake-store/include/segment.h b/mooncake-store/include/segment.h index 0fc34a841a..f4641db31c 100644 --- a/mooncake-store/include/segment.h +++ b/mooncake-store/include/segment.h @@ -90,6 +90,9 @@ inline std::ostream& operator<<( struct LocalDiskSegment { mutable Mutex offloading_mutex_; bool enable_offloading; + std::string local_disk_segment_id; + uint64_t mount_epoch{0}; + uint32_t capabilities{0}; int64_t ssd_total_capacity_bytes = 0; // last reported by client heartbeat std::atomic ssd_used_bytes{0}; std::unordered_map GUARDED_BY( @@ -100,19 +103,19 @@ struct LocalDiskSegment { // offloading_objects (offloading_mutex_). std::unordered_map GUARDED_BY( offloading_mutex_) promotion_objects; - // Keys removed via Remove/BatchRemove that had LOCAL_DISK replicas on - // this client. Populated by master's Remove when the key has a - // LOCAL_DISK replica. Drained by RemoveObjectHeartbeat RPC. Same locking as - // offloading_objects (offloading_mutex_). - std::vector GUARDED_BY( - offloading_mutex_) removed_keys; // Set by master's RemoveAll. When the client sees this flag via // PollRemoveAll, it calls FileStorage::RemoveAll() to physically // delete all SSD files. Same locking as offloading_objects // (offloading_mutex_). bool GUARDED_BY(offloading_mutex_) pending_remove_all = false; - explicit LocalDiskSegment(bool enable_offloading) - : enable_offloading(enable_offloading) {} + explicit LocalDiskSegment(bool enable_offloading, + std::string local_disk_segment_id = {}, + uint64_t mount_epoch = 0, + uint32_t capabilities = 0) + : enable_offloading(enable_offloading), + local_disk_segment_id(std::move(local_disk_segment_id)), + mount_epoch(mount_epoch), + capabilities(capabilities) {} LocalDiskSegment(const LocalDiskSegment&) = delete; LocalDiskSegment& operator=(const LocalDiskSegment&) = delete; @@ -142,8 +145,10 @@ class ScopedSegmentAccess { */ ErrorCode MountSegment(const Segment& segment, const UUID& client_id); - ErrorCode MountLocalDiskSegment(const UUID& client_id, - bool enable_offloading); + ErrorCode MountLocalDiskSegment( + const UUID& client_id, bool enable_offloading, + const std::string& local_disk_segment_id = {}, uint64_t mount_epoch = 0, + uint32_t capabilities = 0); /** * @brief Re-mount a segment. To avoid infinite remount trying, only the diff --git a/mooncake-store/include/storage_backend.h b/mooncake-store/include/storage_backend.h index b417c3c950..de08abdfcb 100644 --- a/mooncake-store/include/storage_backend.h +++ b/mooncake-store/include/storage_backend.h @@ -33,36 +33,26 @@ struct BucketObjectMetadata { int64_t offset; int64_t key_size; int64_t data_size; + ObjectIncarnation object_incarnation; + bool tombstoned{false}; }; -YLT_REFL(BucketObjectMetadata, offset, key_size, data_size); +YLT_REFL(BucketObjectMetadata, offset, key_size, data_size, object_incarnation, + tombstoned); struct BucketMetadata { - int64_t meta_size; - int64_t data_size; + int64_t meta_size{0}; + int64_t data_size{0}; std::vector keys; std::vector metadatas; - // Persisted tombstones. Init filters these keys before rebuilding the - // in-memory object index. - std::vector tombstones; - // Runtime-only fields (not serialized) for safe deletion support // Tracks number of in-flight reads to enable safe bucket deletion mutable std::atomic inflight_reads_{0}; // Last access timestamp in nanoseconds; used by LRU eviction policy. // Updated on every read with relaxed ordering (approximate is sufficient). mutable std::atomic last_access_ns_{0}; - - // Runtime-only (not serialized): bytes marked removed via MarkRemoved. - // Drives GC candidate selection (compact when deleted_bytes_ > 0). - mutable std::atomic deleted_bytes_{0}; - // Runtime-only (not serialized): true while a GC compaction is in flight - // for this bucket, preventing re-entrant compaction. - mutable std::atomic compacting_{false}; - // Runtime-only version of the bucket's deletion state. Compaction captures - // this value with its read snapshot and validates it before publishing a - // new bucket, so a concurrent deletion cannot publish stale data. - uint64_t generation_{0}; + mutable std::mutex operation_mutex_; + mutable std::atomic mutation_in_progress_{false}; // Default constructor BucketMetadata() = default; @@ -73,12 +63,9 @@ struct BucketMetadata { data_size(other.data_size), keys(other.keys), metadatas(other.metadatas), - tombstones(other.tombstones), inflight_reads_(0), last_access_ns_(0), - deleted_bytes_(0), - compacting_(false), - generation_(0) {} + mutation_in_progress_(false) {} // Move constructor BucketMetadata(BucketMetadata&& other) noexcept @@ -86,12 +73,9 @@ struct BucketMetadata { data_size(other.data_size), keys(std::move(other.keys)), metadatas(std::move(other.metadatas)), - tombstones(std::move(other.tombstones)), inflight_reads_(0), last_access_ns_(0), - deleted_bytes_(0), - compacting_(false), - generation_(0) {} + mutation_in_progress_(false) {} // Copy assignment BucketMetadata& operator=(const BucketMetadata& other) { @@ -100,7 +84,6 @@ struct BucketMetadata { data_size = other.data_size; keys = other.keys; metadatas = other.metadatas; - tombstones = other.tombstones; // Don't copy runtime state } return *this; @@ -113,13 +96,38 @@ struct BucketMetadata { data_size = other.data_size; keys = std::move(other.keys); metadatas = std::move(other.metadatas); - tombstones = std::move(other.tombstones); // Don't move runtime state } return *this; } }; -YLT_REFL(BucketMetadata, data_size, keys, metadatas, tombstones); +YLT_REFL(BucketMetadata, data_size, keys, metadatas); + +struct BucketGcIntent { + uint32_t version{1}; + bool committed{false}; + int64_t target_bucket_id{-1}; + std::vector source_bucket_ids; +}; +YLT_REFL(BucketGcIntent, version, committed, target_bucket_id, + source_bucket_ids); + +enum class LocalDeleteResult { + kRemoved, + kAlreadyRemoved, + kStaleVersion, + kRetryableFailure, +}; + +struct LocalDeleteTaskResult { + LocalDeleteTaskId task_id; + LocalDeleteResult result{LocalDeleteResult::kRetryableFailure}; + ErrorCode error{ErrorCode::OK}; + + [[nodiscard]] bool IsTerminal() const { + return result != LocalDeleteResult::kRetryableFailure; + } +}; /** * @brief RAII guard for tracking in-flight bucket reads. @@ -226,26 +234,9 @@ struct BucketBackendConfig { int64_t max_total_size = 0; // 0 = unlimited; evict when total_size_ // exceeds this threshold (bytes) - bool disable_ssd_eviction = - false; // Force disable eviction regardless of - // eviction_policy. Set via - // MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION. - - // --- Explicit-delete-only GC config --- - // Enable background tombstone compaction GC. bool gc_enable = true; - // GC scan interval in milliseconds. - int64_t gc_interval_ms = 1000; - // Compact a bucket when deleted bytes / bucket data size >= this ratio. + int64_t gc_interval_seconds = 10; double gc_deleted_ratio = 0.25; - // Trigger GC when total_size / max_total_size >= this ratio. - double gc_high_watermark_ratio = 0.90; - // Max old buckets collected per GC round for cross-bucket merge. - int64_t gc_max_buckets_per_round = 1; - // Enable cross-bucket merge compaction (collect live keys from multiple - // tombstone buckets into one new bucket). When false, each bucket is - // compacted independently (no merge). - bool gc_merge_enable = true; bool Validate() const; @@ -390,29 +381,6 @@ struct FileStorageConfig { static FileStorageConfig FromEnvironment(); }; -struct StorageReadStats { - uint64_t plan_us{0}; - uint64_t file_open_us{0}; - uint64_t disk_read_us{0}; - uint64_t slowest_disk_read_us{0}; - std::string slowest_key{"-"}; - std::string io_mode{"unknown"}; - std::string status{"ok"}; - std::string error_key{"-"}; - ErrorCode error_code{ErrorCode::OK}; -}; - -StorageReadStats* CurrentStorageReadStats(); - -class ScopedStorageReadStats { - public: - explicit ScopedStorageReadStats(StorageReadStats* stats); - ~ScopedStorageReadStats(); - - private: - StorageReadStats* previous_; -}; - class StorageBackendInterface { public: StorageBackendInterface(const FileStorageConfig& file_storage_config); @@ -455,20 +423,6 @@ class StorageBackendInterface { // Default: no-op (no test failures injected) } - // Mark a key as removed (tombstone) for explicit-delete-only GC. - // Default no-op: only BucketStorageBackend implements tombstone + GC. - // File-per-key and other backends inherit the no-op (do not delete files). - // Safe to call for keys not present in local storage (idempotent). - virtual tl::expected MarkRemoved( - const std::string& /* key */) { - return {}; - } - - // Batch variant: mark multiple keys as removed in one lock acquisition. - virtual tl::expected BatchMarkRemoved( - const std::vector& /* keys */) { - return {}; - } // Remove all persisted objects from disk. Called during RemoveAll to // clean up physical SSD files alongside master metadata deletion. virtual void RemoveAll() {} @@ -480,6 +434,23 @@ class StorageBackendInterface { return std::vector{}; } + virtual std::vector BatchMarkDeleted( + const std::vector& tasks) { + std::vector results; + results.reserve(tasks.size()); + for (const auto& task : tasks) { + results.push_back({task.task_id, + LocalDeleteResult::kRetryableFailure, + ErrorCode::UNAVAILABLE_IN_CURRENT_MODE}); + } + return results; + } + virtual int64_t GetReclaimableBytes() const { return 0; } + virtual bool RequestGarbageCollection( + bool /* require_disk_pressure */ = false) { + return false; + } + FileStorageConfig file_storage_config_; }; @@ -1012,7 +983,9 @@ class BucketStorageBackend : public StorageBackendInterface { */ tl::expected AllocateOffloadingBuckets( const std::unordered_map& offloading_objects, - std::vector>& buckets_keys); + std::vector>& buckets_keys, + const std::unordered_map* + object_incarnations = nullptr); void ClearUngroupedOffloadingObjects(); @@ -1062,53 +1035,16 @@ class BucketStorageBackend : public StorageBackendInterface { */ tl::expected DeleteBucket(int64_t bucket_id); - // Explicit-delete-only GC: mark a key as tombstone (no disk IO). - // Removes key from object_bucket_map_ (immediately invisible to - // BatchLoad/IsExist) and bumps bucket deleted_bytes_. - // Idempotent: no-op if key not in local storage. - tl::expected MarkRemoved( - const std::string& key) override; - tl::expected BatchMarkRemoved( - const std::vector& keys) override; - - // Compact a single bucket: copy-on-write live keys to a new bucket, - // atomically swap mappings, delete old bucket file after reads drain. - // Returns true on success (or no-op), false on transient failure - // (will retry next round). Public to allow explicit compaction and - // testing (analogous to DeleteBucket). - bool CompactBucket(int64_t bucket_id); - - // Compact multiple buckets into one new bucket (cross-bucket merge). - // Collects live keys from all given old buckets, groups them by - // bucket_keys_limit/bucket_size_limit, and writes ONE new bucket per - // round (the first group that fills up). If the first group doesn't - // fill a full bucket and there's no space pressure, the merge is - // deferred to the next round. Old buckets whose live keys are all - // migrated are deleted. - // Returns true on success (or deferred), false on transient failure. - bool CompactBuckets(const std::vector& bucket_ids, - bool space_pressure = false); tl::expected, ErrorCode> EvictAboveDiskWatermark( double high_watermark_ratio, double low_watermark_ratio, EvictionHandler eviction_handler = nullptr) override; - private: - // --- Background GC --- - // Background GC thread entry point. - void GCThreadFunc(); - - // Wait for in-flight reads on a bucket to drain (up to 10s). - void WaitForInflightReads(std::shared_ptr bucket); - - // Delete .bucket and .meta files for a bucket_id, ignore missing. - void DeleteBucketFiles(int64_t bucket_id); - - // GC thread lifecycle members - std::atomic gc_running_{false}; - std::thread gc_thread_; - std::mutex gc_mutex_; - std::condition_variable gc_cv_; + std::vector BatchMarkDeleted( + const std::vector& tasks) override; + int64_t GetReclaimableBytes() const override; + bool RequestGarbageCollection(bool require_disk_pressure = false) override; + private: tl::expected, ErrorCode> BuildBucket( int64_t bucket_id, const std::unordered_map>& batch_object, @@ -1122,9 +1058,32 @@ class BucketStorageBackend : public StorageBackendInterface { tl::expected StoreBucketMetadata( int64_t bucket_id, std::shared_ptr bucket_metadata); + tl::expected StoreBucketMetadataAtomically( + int64_t bucket_id, BucketMetadata& bucket_metadata); + tl::expected LoadBucketMetadata( int64_t bucket_id, std::shared_ptr bucket_metadata); + struct GcCandidate { + int64_t bucket_id; + std::shared_ptr bucket; + int64_t live_bytes; + int64_t reclaimable_bytes; + size_t live_keys; + }; + + std::vector SelectGcCandidates(bool under_pressure); + tl::expected RunGarbageCollectionOnce(bool under_pressure); + tl::expected, ErrorCode> WriteGcReplacement( + int64_t target_bucket_id, const std::vector& sources); + tl::expected RecoverGarbageCollection(); + tl::expected StoreGcIntent(const BucketGcIntent& intent); + tl::expected RemoveGcIntent(); + tl::expected SyncStorageDirectory() const; + bool FinalizeGcSource(const GcCandidate& source); + void GarbageCollectionThreadFunc(); + static int64_t BucketReclaimableBytes(const BucketMetadata& bucket); + tl::expected CreateBucketId(); tl::expected GetBucketMetadataPath( @@ -1151,7 +1110,7 @@ class BucketStorageBackend : public StorageBackendInterface { * Used by write rollback and startup recovery of incomplete buckets. * @param bucket_id The bucket ID whose files should be deleted. */ - void CleanupOrphanedBucket(int64_t bucket_id); + bool CleanupOrphanedBucket(int64_t bucket_id); /** * @brief Rollback a committed bucket from the local index when @@ -1267,6 +1226,7 @@ class BucketStorageBackend : public StorageBackendInterface { mutable Mutex iterator_mutex_; std::string storage_path_; int64_t total_size_ GUARDED_BY(mutex_) = 0; + std::atomic reclaimable_bytes_{0}; std::unordered_map GUARDED_BY(mutex_) object_bucket_map_; std::unordered_set GUARDED_BY(mutex_) pending_eviction_keys_; @@ -1282,9 +1242,17 @@ class BucketStorageBackend : public StorageBackendInterface { int64_t GUARDED_BY(mutex_) next_bucket_ = -1; BucketBackendConfig bucket_backend_config_; + std::mutex gc_mutex_; + std::condition_variable gc_cv_; + std::thread gc_thread_; + std::atomic gc_stop_{false}; + bool gc_requested_{false}; + mutable Mutex offloading_mutex_; std::unordered_map GUARDED_BY(offloading_mutex_) ungrouped_offloading_objects_; + std::unordered_map GUARDED_BY( + offloading_mutex_) offloading_object_incarnations_; // File handle cache for UringFile to avoid repeated open/close overhead mutable Mutex file_cache_mutex_; diff --git a/mooncake-store/include/types.h b/mooncake-store/include/types.h index eb53a52cab..5e349c6672 100644 --- a/mooncake-store/include/types.h +++ b/mooncake-store/include/types.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include #include @@ -93,14 +94,9 @@ static constexpr double DEFAULT_EVICTION_RATIO = 0.05; static constexpr double DEFAULT_EVICTION_HIGH_WATERMARK_RATIO = 0.90; static constexpr double DEFAULT_NOF_EVICTION_RATIO = 0.05; static constexpr double DEFAULT_NOF_EVICTION_HIGH_WATERMARK_RATIO = 0.90; -static constexpr int64_t DEFAULT_MASTER_VIEW_LEASE_TTL_SEC = 3; // in seconds, old value is 5 +static constexpr int64_t DEFAULT_MASTER_VIEW_LEASE_TTL_SEC = 5; // in seconds static constexpr int64_t DEFAULT_CLIENT_LIVE_TTL_SEC = 10; // in seconds static constexpr int64_t DEFAULT_NOF_HEARTBEAT_INTERVAL_SEC = 10; - -// Metrics reporter defaults (push master storage metrics to HA backend) -static constexpr bool DEFAULT_ENABLE_METRICS_REPORT_TO_BACKEND = false; -static constexpr int DEFAULT_METRICS_REPORT_INTERVAL_SEC = 5; -static constexpr int DEFAULT_METRICS_REPORT_LEASE_TTL_SEC = 10; static constexpr uint32_t DEFAULT_NOF_HEARTBEAT_PROBE_TIMEOUT_MS = 1000; static constexpr uint32_t DEFAULT_NOF_HEARTBEAT_FAILURES_THRESHOLD = 3; static constexpr uint64_t DEFAULT_SNAPSHOT_INTERVAL_SEC = @@ -234,17 +230,78 @@ constexpr const char* DEFAULT_PROTOCOL = "tcp"; constexpr const char* DEFAULT_MASTER_SERVER_ADDR = "127.0.0.1:50051"; static constexpr int DEFAULT_CLIENT_HTTP_PORT = 9300; +struct ObjectIncarnation { + uint64_t high{0}; + uint64_t low{0}; + + [[nodiscard]] bool IsZero() const { return high == 0 && low == 0; } + auto operator<=>(const ObjectIncarnation&) const = default; +}; +YLT_REFL(ObjectIncarnation, high, low); + +struct LocalDeleteTaskId { + uint64_t high{0}; + uint64_t low{0}; + + auto operator<=>(const LocalDeleteTaskId&) const = default; +}; +YLT_REFL(LocalDeleteTaskId, high, low); + +inline constexpr uint32_t kLocalDiskCapabilityObjectTombstoneV1 = 1U << 0; +inline constexpr uint32_t kMaxLocalDeleteTasksPerBatch = 256; + +struct LocalDiskMountInfo { + uint64_t mount_epoch{0}; + uint32_t capabilities{0}; +}; +YLT_REFL(LocalDiskMountInfo, mount_epoch, capabilities); + +struct LocalDeleteTask { + LocalDeleteTaskId task_id; + std::string local_disk_segment_id; + std::string tenant_id; + std::string key; + ObjectIncarnation object_incarnation; + int64_t expected_bucket_id{-1}; + + bool operator==(const LocalDeleteTask&) const = default; +}; +YLT_REFL(LocalDeleteTask, task_id, local_disk_segment_id, tenant_id, key, + object_incarnation, expected_bucket_id); + +struct LocalDeleteRemovePayloadV1 { + uint32_t schema_version{1}; + ObjectIncarnation object_incarnation; + std::vector delete_intents; +}; +YLT_REFL(LocalDeleteRemovePayloadV1, schema_version, object_incarnation, + delete_intents); + +struct LocalDeleteAckPayloadV1 { + uint32_t schema_version{1}; + std::string local_disk_segment_id; + std::vector task_ids; +}; +YLT_REFL(LocalDeleteAckPayloadV1, schema_version, local_disk_segment_id, + task_ids); + struct OffloadTaskItem { std::string tenant_id; std::string key; int64_t size; + struct_pack::compatible object_incarnation{}; + + [[nodiscard]] ObjectIncarnation GetObjectIncarnation() const { + return object_incarnation.value_or(ObjectIncarnation{}); + } bool operator==(const OffloadTaskItem& other) const { return tenant_id == other.tenant_id && key == other.key && - size == other.size; + size == other.size && + GetObjectIncarnation() == other.GetObjectIncarnation(); } }; -YLT_REFL(OffloadTaskItem, tenant_id, key, size); +YLT_REFL(OffloadTaskItem, tenant_id, key, size, object_incarnation); struct PromotionTaskItem { std::string tenant_id; @@ -258,16 +315,6 @@ struct PromotionTaskItem { }; YLT_REFL(PromotionTaskItem, tenant_id, key, size); -struct RemoveTaskItem { - std::string tenant_id; - std::string key; - - bool operator==(const RemoveTaskItem& other) const { - return tenant_id == other.tenant_id && key == other.key; - } -}; -YLT_REFL(RemoveTaskItem, tenant_id, key); - // Store client configuration validation limits static constexpr size_t MIN_SEGMENT_SIZE = 1024; // 1KB static constexpr size_t MAX_SEGMENT_SIZE = 1024ULL * 1024 * 1024 * 1024; // 1TB @@ -537,8 +584,14 @@ struct StorageObjectMetadata { int64_t key_size; int64_t data_size; std::string transport_endpoint; + struct_pack::compatible object_incarnation{}; + + [[nodiscard]] ObjectIncarnation GetObjectIncarnation() const { + return object_incarnation.value_or(ObjectIncarnation{}); + } + YLT_REFL(StorageObjectMetadata, bucket_id, offset, key_size, data_size, - transport_endpoint); + transport_endpoint, object_incarnation); }; } // namespace mooncake diff --git a/mooncake-store/src/file_storage.cpp b/mooncake-store/src/file_storage.cpp index c637b096c6..66f1dcbd9c 100644 --- a/mooncake-store/src/file_storage.cpp +++ b/mooncake-store/src/file_storage.cpp @@ -1,36 +1,157 @@ #include "file_storage.h" +#include +#include +#include + +#include #include +#include +#include #include #include -#include #include #include #include #include #include "aligned_client_buffer.h" -#include "mooncake_logging.h" -#include "storage_backend.h" +#include "bool_parser.h" #include "client_metric.h" +#include "environ.h" +#include "local_delete.h" +#include "storage_backend.h" #include "utils.h" #include "device/accelerator_registry.h" #ifdef USE_URING #include "file_interface.h" #endif -// spdiag perf points for the offload owner-side SSD path. SPDIAG_PROGRAM_NAME -// must match the other TUs (store_py/real_client/client_service); each TU's -// static initializer writes the shared detail::AutoProgramName(), and leaving -// it nullptr here could clobber the program name depending on init order. -#define SPDIAG_PERF_DEF_FILE "mooncake_perf_points.def" -#define SPDIAG_PROGRAM_NAME "mooncake_store" -#include "spdiag/auto_perf.h" - namespace mooncake { namespace { +constexpr std::string_view kLocalDiskSegmentIdMarker = + ".mooncake_local_disk_segment_id"; + +const char* LocalDeleteResultLabel(LocalDeleteResult result) { + switch (result) { + case LocalDeleteResult::kRemoved: + return "removed"; + case LocalDeleteResult::kAlreadyRemoved: + return "already_removed"; + case LocalDeleteResult::kStaleVersion: + return "stale_version"; + case LocalDeleteResult::kRetryableFailure: + return "retryable_failure"; + } + return "unknown"; +} + +tl::expected ReadLocalDiskSegmentId( + const std::filesystem::path& marker_path) { + std::ifstream input(marker_path); + std::string id; + if (!input || !std::getline(input, id) || id.empty()) { + LOG(ERROR) << "Invalid LOCAL_DISK identity marker: " << marker_path; + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + return id; +} + +tl::expected LoadOrCreateLocalDiskSegmentId( + const std::string& storage_path) { + namespace fs = std::filesystem; + const fs::path directory(storage_path); + const fs::path marker_path = + directory / fs::path(kLocalDiskSegmentIdMarker); + std::error_code ec; + if (fs::exists(marker_path, ec)) { + return ReadLocalDiskSegmentId(marker_path); + } + if (ec) { + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + + bool has_unmarked_data = false; + for (const auto& entry : fs::directory_iterator(directory, ec)) { + if (entry.path() != marker_path) { + has_unmarked_data = true; + break; + } + } + if (ec) { + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + const char* adopt_unmarked = + std::getenv("MC_STORE_ADOPT_UNMARKED_LOCAL_DISK"); + if (has_unmarked_data && (adopt_unmarked == nullptr || + std::string_view(adopt_unmarked) != "1")) { + LOG(ERROR) << "LOCAL_DISK path contains data without identity marker: " + << storage_path + << "; set MC_STORE_ADOPT_UNMARKED_LOCAL_DISK=1 once to " + "adopt this directory"; + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + if (has_unmarked_data) { + LOG(WARNING) << "Creating LOCAL_DISK identity marker for existing path " + << storage_path; + } + + const std::string id = UuidToString(generate_uuid()); + const std::string contents = id + "\n"; + const int fd = + ::open(marker_path.c_str(), O_WRONLY | O_CREAT | O_EXCL, 0644); + if (fd < 0) { + if (errno == EEXIST) { + return ReadLocalDiskSegmentId(marker_path); + } + LOG(ERROR) << "Failed to create LOCAL_DISK identity marker: " + << std::strerror(errno); + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + + size_t written = 0; + while (written < contents.size()) { + const ssize_t n = + ::write(fd, contents.data() + written, contents.size() - written); + if (n <= 0) { + ::close(fd); + fs::remove(marker_path, ec); + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + written += static_cast(n); + } + const bool file_synced = ::fsync(fd) == 0; + const bool file_closed = ::close(fd) == 0; + const int directory_fd = ::open(directory.c_str(), O_RDONLY | O_DIRECTORY); + const bool directory_synced = + directory_fd >= 0 && ::fsync(directory_fd) == 0; + if (directory_fd >= 0) { + ::close(directory_fd); + } + if (!file_synced || !file_closed || !directory_synced) { + fs::remove(marker_path, ec); + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + return id; +} + +tl::expected AcquireLocalDiskLock( + const std::string& storage_path) { + const auto lock_path = + std::filesystem::path(storage_path) / ".mooncake_local_disk.lock"; + const int fd = ::open(lock_path.c_str(), O_RDWR | O_CREAT, 0644); + if (fd < 0 || ::flock(fd, LOCK_EX | LOCK_NB) != 0) { + if (fd >= 0) { + ::close(fd); + } + LOG(ERROR) << "LOCAL_DISK path is already mounted: " << storage_path; + return tl::unexpected(ErrorCode::PERSISTENT_FAIL); + } + return fd; +} + double ParseEnvRatioOr(const std::string& raw_value, double default_value) { if (raw_value.empty()) { return default_value; @@ -51,13 +172,13 @@ double ParseEnvRatioOr(const std::string& raw_value, double default_value) { } double GetEnvRatioOr(const char* name, double default_value) { - const auto raw_value = GetEnvStringOr(name, ""); + const auto raw_value = Environ::GetString(name, ""); return ParseEnvRatioOr(raw_value, default_value); } double GetEnvRatioOr(const char* preferred_name, const char* fallback_name, double default_value) { - const auto preferred_value = GetEnvStringOr(preferred_name, ""); + const auto preferred_value = Environ::GetString(preferred_name, ""); if (!preferred_value.empty()) { return ParseEnvRatioOr(preferred_value, default_value); } @@ -66,16 +187,10 @@ double GetEnvRatioOr(const char* preferred_name, const char* fallback_name, bool GetEnvBoolStringOr(const char* name, bool default_value) { const auto raw_value = - GetEnvStringOr(name, default_value ? "true" : "false"); - if (raw_value == "1" || raw_value == "true" || raw_value == "TRUE" || - raw_value == "True") { - return true; - } - if (raw_value == "0" || raw_value == "false" || raw_value == "FALSE" || - raw_value == "False") { - return false; - } - return default_value; + Environ::GetString(name, default_value ? "true" : "false"); + return TryParseBool(raw_value, {.token_set = BoolTokenSet::kTrueFalse, + .trim_ascii_whitespace = false}) + .value_or(default_value); } std::vector BuildOffloadTasksFromStorageKeys( @@ -87,9 +202,14 @@ std::vector BuildOffloadTasksFromStorageKeys( auto [tenant_id, key] = TenantId::ParseScopedKey(storage_keys[i]); const int64_t size = i < metadatas.size() ? metadatas[i].data_size : int64_t{0}; - tasks.push_back(OffloadTaskItem{.tenant_id = tenant_id.value(), - .key = std::move(key), - .size = size}); + const ObjectIncarnation object_incarnation = + i < metadatas.size() ? metadatas[i].GetObjectIncarnation() + : ObjectIncarnation{}; + tasks.push_back( + OffloadTaskItem{.tenant_id = tenant_id.value(), + .key = std::move(key), + .size = size, + .object_incarnation = object_incarnation}); } return tasks; } @@ -100,8 +220,8 @@ FileStorageConfig FileStorageConfig::FromEnvironment() { FileStorageConfig config; auto storage_backend_descriptor = - GetEnvStringOr("MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR", - "bucket_storage_backend"); + Environ::GetString("MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR", + "bucket_storage_backend"); if (storage_backend_descriptor == "bucket_storage_backend") { config.storage_backend_type = StorageBackendType::kBucket; @@ -116,32 +236,32 @@ FileStorageConfig FileStorageConfig::FromEnvironment() { LOG(ERROR) << "Unknown storage backend."; } - config.storage_filepath = GetEnvStringOr( + config.storage_filepath = Environ::GetString( "MOONCAKE_OFFLOAD_FILE_STORAGE_PATH", config.storage_filepath); - config.local_buffer_size = GetEnvOr( + config.local_buffer_size = Environ::GetInt64( "MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES", config.local_buffer_size); - config.scanmeta_iterator_keys_limit = GetEnvOr( + config.scanmeta_iterator_keys_limit = Environ::GetInt64( "MOONCAKE_OFFLOAD_SCANMETA_ITERATOR_KEYS_LIMIT", - GetEnvOr("MOONCAKE_SCANMETA_ITERATOR_KEYS_LIMIT", + Environ::GetInt64("MOONCAKE_SCANMETA_ITERATOR_KEYS_LIMIT", config.scanmeta_iterator_keys_limit)); - config.total_keys_limit = GetEnvOr( + config.total_keys_limit = Environ::GetInt64( "MOONCAKE_OFFLOAD_TOTAL_KEYS_LIMIT", config.total_keys_limit); - config.total_size_limit = GetEnvOr( + config.total_size_limit = Environ::GetInt64( "MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES", config.total_size_limit); config.heartbeat_interval_seconds = - GetEnvOr("MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS", + Environ::GetUInt32("MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS", config.heartbeat_interval_seconds); config.client_buffer_gc_interval_seconds = - GetEnvOr("MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS", + Environ::GetUInt32("MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS", config.client_buffer_gc_interval_seconds); config.client_buffer_gc_ttl_ms = - GetEnvOr("MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_TTL_MS", + Environ::GetUInt64("MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_TTL_MS", config.client_buffer_gc_ttl_ms); config.enable_disk_watermark_eviction = @@ -156,10 +276,13 @@ FileStorageConfig FileStorageConfig::FromEnvironment() { "MOONCAKE_DISK_EVICTION_LOW_WATERMARK_RATIO", config.disk_eviction_low_watermark_ratio); - auto use_uring_str = - GetEnvStringOr("MOONCAKE_OFFLOAD_USE_URING", - GetEnvStringOr("MOONCAKE_USE_URING", "false")); - config.use_uring = (use_uring_str == "true" || use_uring_str == "1"); + const auto use_uring_str = + Environ::GetString("MOONCAKE_OFFLOAD_USE_URING", + Environ::GetString("MOONCAKE_USE_URING", "false")); + config.use_uring = + TryParseBool(use_uring_str, {.token_set = BoolTokenSet::kTrueFalse, + .trim_ascii_whitespace = false}) + .value_or(false); return config; } @@ -318,6 +441,12 @@ FileStorage::~FileStorage() { if (client_buffer_gc_thread_.joinable()) { client_buffer_gc_thread_.join(); } + storage_backend_.reset(); + if (local_disk_lock_fd_ >= 0) { + ::flock(local_disk_lock_fd_, LOCK_UN); + ::close(local_disk_lock_fd_); + local_disk_lock_fd_ = -1; + } } tl::expected FileStorage::Init() { @@ -327,10 +456,29 @@ tl::expected FileStorage::Init() { << register_memory_result.error(); return register_memory_result; } + if (config_.storage_backend_type == StorageBackendType::kBucket) { + auto identity = + LoadOrCreateLocalDiskSegmentId(config_.storage_filepath); + if (!identity) { + return tl::unexpected(identity.error()); + } + local_disk_segment_id_ = std::move(identity.value()); + auto disk_lock = AcquireLocalDiskLock(config_.storage_filepath); + if (!disk_lock) { + return tl::unexpected(disk_lock.error()); + } + local_disk_lock_fd_ = disk_lock.value(); + local_disk_capabilities_ = kLocalDiskCapabilityObjectTombstoneV1; + } auto init_storage_backend_result = storage_backend_->Init(); if (!init_storage_backend_result) { LOG(ERROR) << "Failed to init storage backend: " << init_storage_backend_result.error(); + if (local_disk_lock_fd_ >= 0) { + ::flock(local_disk_lock_fd_, LOCK_UN); + ::close(local_disk_lock_fd_); + local_disk_lock_fd_ = -1; + } return init_storage_backend_result; } auto enable_offloading_result = IsEnableOffloading(); @@ -349,13 +497,16 @@ tl::expected FileStorage::Init() { { MutexLocker locker(&offloading_mutex_); enable_offloading_ = enable_offloading_result.value(); - auto mount_file_storage_result = - client_->MountLocalDiskSegment(enable_offloading_); + auto mount_file_storage_result = client_->MountLocalDiskSegment( + enable_offloading_, local_disk_segment_id_, + local_disk_capabilities_); if (!mount_file_storage_result) { LOG(ERROR) << "Failed to mount file storage: " << mount_file_storage_result.error(); - return mount_file_storage_result; + return tl::unexpected(mount_file_storage_result.error()); } + local_disk_mount_epoch_ = mount_file_storage_result->mount_epoch; + local_disk_capabilities_ = mount_file_storage_result->capabilities; } // Report configured SSD capacity to Master so it can populate // file_total_capacity_ (the denominator in "SSD Storage: X / Y"). @@ -372,18 +523,7 @@ tl::expected FileStorage::Init() { auto scan_meta_result = storage_backend_->ScanMeta( [this](const std::vector& keys, std::vector& metadatas) { - for (auto& metadata : metadatas) { - metadata.transport_endpoint = local_rpc_addr_; - } - auto tasks = BuildOffloadTasksFromStorageKeys(keys, metadatas); - auto add_object_result = - client_->NotifyOffloadSuccess(tasks, metadatas); - if (!add_object_result) { - LOG(ERROR) << "Failed to add object to master: " - << add_object_result.error(); - return add_object_result.error(); - } - return ErrorCode::OK; + return ReconcileScannedObjects(keys, metadatas); }); if (!scan_meta_result) { @@ -412,72 +552,15 @@ tl::expected FileStorage::Init() { tl::expected FileStorage::BatchGet( const std::vector& keys, const std::vector& sizes) { auto start_time = std::chrono::steady_clock::now(); - const uint64_t trace_id = mooncake::logging::CurrentTraceId(); - const bool breakdown_log = - mooncake::logging::ShouldSampleHiFreqLog(trace_id); - const uint64_t total_bytes = - std::accumulate(sizes.begin(), sizes.end(), uint64_t{0}); - uint64_t alloc_us = 0; - StorageReadStats read_stats; - auto log_breakdown = [&](const char* fallback_status, - ErrorCode fallback_error) { - if (!breakdown_log) return; - const auto total_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - start_time) - .count(); - const char* status = - read_stats.status == "ok" ? fallback_status - : read_stats.status.c_str(); - const ErrorCode error = - read_stats.error_code == ErrorCode::OK ? fallback_error - : read_stats.error_code; - MC_LOG(INFO) << "storage_read_breakdown num_keys=" << keys.size() - << " total_bytes=" << total_bytes - << " alloc_us=" << alloc_us - << " plan_us=" << read_stats.plan_us - << " file_open_us=" << read_stats.file_open_us - << " disk_read_us=" << read_stats.disk_read_us - << " total_us=" << total_us - << " slowest_key=" << read_stats.slowest_key - << " slowest_disk_read_us=" - << read_stats.slowest_disk_read_us - << " io_mode=" << read_stats.io_mode - << " status=" << status - << " error_key=" << read_stats.error_key - << " error_code=" << static_cast(error); - }; - // Owner-side SSD read total (OwnerSsdRead); broken down into buffer - // allocation (OwnerAllocBuffer) and disk load (OwnerDiskLoad). End() is on - // the success path only — the error early-returns let the dtor Abandon the - // unfinished total sample. - SpDiag::PerfPoint pt_read(PerfKey::GET_SSD_OWNER_READ, - SpDiag::PerfLevel::MODULE); - pt_read.Start(); - SpDiag::PerfPoint pt_alloc(PerfKey::GET_SSD_OWNER_ALLOC, - SpDiag::PerfLevel::MODULE); - pt_alloc.Start(); - const auto alloc_start = std::chrono::steady_clock::now(); auto allocate_res = AllocateBatch(keys, sizes); - alloc_us = std::chrono::duration_cast( - std::chrono::steady_clock::now() - alloc_start) - .count(); - pt_alloc.End(allocate_res ? 0 : -1); if (!allocate_res) { LOG(ERROR) << "Failed to allocate batch objects"; - log_breakdown("alloc_fail", allocate_res.error()); return tl::make_unexpected(allocate_res.error()); } auto allocated_batch = allocate_res.value(); - SpDiag::PerfPoint pt_load(PerfKey::GET_SSD_OWNER_LOAD, - SpDiag::PerfLevel::MODULE); - pt_load.Start(); - ScopedStorageReadStats stats_scope(breakdown_log ? &read_stats : nullptr); auto result = BatchLoad(allocated_batch->slices); - pt_load.End(result ? 0 : -1); if (!result) { LOG(ERROR) << "Batch load object failed,err_code = " << result.error(); - log_breakdown("read_fail", result.error()); return tl::make_unexpected(result.error()); } @@ -504,24 +587,30 @@ tl::expected FileStorage::BatchGet( .count(); VLOG(1) << "Time taken for FileStorage::BatchGet: " << elapsed_time << "us, key size: " << keys.size() << ", batch_id: " << batch_id; - log_breakdown("ok", ErrorCode::OK); - pt_read.End(0); return batch_result; } +bool FileStorage::IsPerBucketSoftOffloadError(ErrorCode error) { + return error == ErrorCode::INVALID_READ || + error == ErrorCode::OBJECT_ALREADY_EXISTS; +} + tl::expected FileStorage::OffloadObjects( const std::vector& offloading_objects) { if (offloading_objects.empty()) { return {}; } std::unordered_map storage_object_sizes; + std::unordered_map object_incarnations; std::unordered_map task_by_storage_key; storage_object_sizes.reserve(offloading_objects.size()); + object_incarnations.reserve(offloading_objects.size()); task_by_storage_key.reserve(offloading_objects.size()); for (const auto& task : offloading_objects) { const auto storage_key = TenantId(task.tenant_id).MakeScopedKey(task.key); storage_object_sizes.emplace(storage_key, task.size); + object_incarnations.emplace(storage_key, task.GetObjectIncarnation()); task_by_storage_key.emplace(storage_key, task); } @@ -529,7 +618,7 @@ tl::expected FileStorage::OffloadObjects( if (auto bucket_backend = std::dynamic_pointer_cast(storage_backend_)) { auto allocate_res = bucket_backend->AllocateOffloadingBuckets( - storage_object_sizes, buckets_keys); + storage_object_sizes, buckets_keys, &object_incarnations); if (!allocate_res) { LOG(ERROR) << "AllocateOffloadingBuckets failed with error: " << allocate_res.error(); @@ -543,23 +632,27 @@ tl::expected FileStorage::OffloadObjects( } buckets_keys.emplace_back(std::move(keys)); } + auto complete_handler = [this, &task_by_storage_key]( const std::vector& keys, std::vector& metadatas) -> ErrorCode { VLOG(1) << "Success to store objects, keys count: " << keys.size(); - for (auto& metadata : metadatas) { - metadata.transport_endpoint = local_rpc_addr_; - } std::vector tasks; tasks.reserve(keys.size()); - for (const auto& key : keys) { + for (size_t i = 0; i < keys.size(); ++i) { + const auto& key = keys[i]; auto it = task_by_storage_key.find(key); if (it == task_by_storage_key.end()) { LOG(ERROR) << "Offload task not found for storage key"; return ErrorCode::INVALID_KEY; } tasks.push_back(it->second); + if (i < metadatas.size()) { + metadatas[i].transport_endpoint = local_rpc_addr_; + metadatas[i].object_incarnation = + it->second.GetObjectIncarnation(); + } } auto result = client_->NotifyOffloadSuccess(tasks, metadatas); if (!result) { @@ -575,6 +668,10 @@ tl::expected FileStorage::OffloadObjects( // orphaned offloading_tasks and release source replica refcounts. std::vector failed_tasks; std::unordered_set all_bucket_keys; + // Set when a whole-cycle error aborts the bucket loop early. We still fall + // through to the NACK flush below before returning it, so no drained key is + // left waiting on the TTL reaper. + std::optional abort_error; for (const auto& keys : buckets_keys) { for (const auto& k : keys) all_bucket_keys.insert(k); @@ -652,6 +749,21 @@ tl::expected FileStorage::OffloadObjects( } } + // If every object in this bucket failed D2H staging, host_batch_object + // is empty (those keys are already in failed_tasks). Skip BatchOffload, + // which rejects an empty map as INVALID_KEY and would otherwise trip + // the whole-cycle abort below for a bucket that has nothing left to + // persist. staging_bufs can still be non-empty here (an object whose + // first slices copied fine but a later one failed), so hand those + // buffers back before continuing: the release loop after BatchOffload + // is unreachable on this path. + if (host_batch_object.empty()) { + for (auto& buf : staging_bufs) { + pinned_buffer_pool_->Release(std::move(buf)); + } + continue; + } + auto offload_start = std::chrono::steady_clock::now(); auto bucket_complete_handler = [this, offload_start, complete_handler]( @@ -691,18 +803,29 @@ tl::expected FileStorage::OffloadObjects( if (!offload_res) { LOG(ERROR) << "Failed to store objects with error: " << offload_res.error(); + // This bucket did not persist, so report its keys back to the + // master as failed regardless of whether we continue or abort. + // Doing it here (rather than only on the soft path) keeps their + // offloading tasks and source-replica refcounts from leaking until + // the put_start_release_timeout_sec_ TTL reaper fires. + for (const auto& [key, _] : host_batch_object) { + failed_tasks.push_back(task_by_storage_key.at(key)); + } if (offload_res.error() == ErrorCode::KEYS_ULTRA_LIMIT) { + // Disk is over the key-count limit: stop offloading entirely. MutexLocker locker(&offloading_mutex_); enable_offloading_ = false; - return tl::make_unexpected(offload_res.error()); } - if (offload_res.error() == ErrorCode::INVALID_READ) { - for (const auto& [key, _] : host_batch_object) { - failed_tasks.push_back(task_by_storage_key.at(key)); - } - } else { - return tl::make_unexpected(offload_res.error()); + if (!IsPerBucketSoftOffloadError(offload_res.error())) { + // Whole-cycle error (KEYS_ULTRA_LIMIT or any hard failure): + // stop processing further buckets, but fall through to the NACK + // flush below so every drained key is released. Unvisited + // buckets are not yet in all_bucket_keys, so the sweep NACKs + // them too; this bucket's keys were just pushed above. + abort_error = offload_res.error(); + break; } + // Soft per-bucket error: keep processing the remaining buckets. } } @@ -718,7 +841,8 @@ tl::expected FileStorage::OffloadObjects( std::vector failed_metadatas; failed_metadatas.reserve(failed_tasks.size()); for (size_t i = 0; i < failed_tasks.size(); ++i) { - failed_metadatas.push_back(StorageObjectMetadata{-1, 0, 0, -1, ""}); + failed_metadatas.push_back( + StorageObjectMetadata{-1, 0, 0, -1, "", {}}); } auto result = client_->NotifyOffloadSuccess(failed_tasks, failed_metadatas); @@ -728,7 +852,9 @@ tl::expected FileStorage::OffloadObjects( << result.error() << " count: " << failed_tasks.size(); } } - + if (abort_error) { + return tl::make_unexpected(*abort_error); + } return {}; } @@ -786,6 +912,15 @@ tl::expected FileStorage::RunDiskWatermarkEviction() { return {}; } + // Reclaim dead records before evicting live LOCAL_DISK replicas. The + // backend only signals its GC worker here; no compaction I/O runs on the + // heartbeat thread. + if (storage_backend_->GetReclaimableBytes() > 0 && + storage_backend_->RequestGarbageCollection( + /* require_disk_pressure = */ true)) { + return {}; + } + auto eviction_result = storage_backend_->EvictAboveDiskWatermark( config_.disk_eviction_high_watermark_ratio, config_.disk_eviction_low_watermark_ratio, @@ -815,16 +950,6 @@ tl::expected FileStorage::IsEnableOffloading() { return enable_offloading; } -tl::expected FileStorage::MarkRemoved( - const std::string& key) { - return storage_backend_->MarkRemoved(key); -} - -tl::expected FileStorage::BatchMarkRemoved( - const std::vector& keys) { - return storage_backend_->BatchMarkRemoved(keys); -} - tl::expected FileStorage::Heartbeat() { if (client_ == nullptr) { LOG(ERROR) << "client is nullptr"; @@ -851,41 +976,6 @@ tl::expected FileStorage::Heartbeat() { }); } - // === STEP 0: Drain removed keys from master === - // Master pushes {tenant_id, key} pairs to this client's removed_keys - // queue when a Remove/BatchRemove deletes a key that had a LOCAL_DISK - // replica here. We mark each as a tombstone so GC can reclaim SSD space. - { - auto remove_result = - client_->RemoveObjectHeartbeat(client_->getClientId()); - if (remove_result) { - bool all_marked = true; - for (const auto& item : remove_result.value()) { - auto storage_key = - TenantId(item.tenant_id).MakeScopedKey(item.key); - auto mark_result = storage_backend_->MarkRemoved(storage_key); - if (!mark_result) { - all_marked = false; - LOG(ERROR) << "Failed to persist remove tombstone: " - << mark_result.error(); - break; - } - } - if (all_marked && !remove_result.value().empty()) { - auto ack_result = client_->AckRemoveObjectHeartbeat( - client_->getClientId(), remove_result.value()); - if (!ack_result) { - LOG(ERROR) << "Failed to ACK remove tasks: " - << ack_result.error(); - } - VLOG(1) << "RemoveObjectHeartbeat processed " - << remove_result.value().size() - << " removed key(s) from master"; - } - } - // Errors are non-fatal: removed keys will be retried next heartbeat. - } - std::vector offloading_objects; // Objects selected for offloading @@ -904,9 +994,12 @@ tl::expected FileStorage::Heartbeat() { << "SEGMENT_NOT_FOUND, attempting to " << "re-register local disk segment and " << "re-register object metadata"; - auto remount_result = - client_->MountLocalDiskSegment(enable_offloading_); + auto remount_result = client_->MountLocalDiskSegment( + enable_offloading_, local_disk_segment_id_, + local_disk_capabilities_); if (remount_result) { + local_disk_mount_epoch_ = remount_result->mount_epoch; + local_disk_capabilities_ = remount_result->capabilities; // Report configured SSD capacity so the Master can // restore file_total_capacity_ (the denominator in // "SSD Storage: X / Y"). This was lost on restart; @@ -976,6 +1069,12 @@ tl::expected FileStorage::Heartbeat() { VLOG(1) << "Completed heartbeat with offloaded objects count: " << offloading_objects.size(); + auto delete_result = ProcessLocalDeleteTasks(); + if (!delete_result) { + LOG(WARNING) << "LOCAL_DISK delete processing failed: " + << delete_result.error(); + } + // Drive any pending L2->L1 promotion work for this client. Failures // inside ProcessPromotionTasks are logged per-key and do not propagate; // promotion is best-effort and must never break offload. @@ -992,6 +1091,116 @@ tl::expected FileStorage::Heartbeat() { return {}; } +tl::expected FileStorage::ProcessLocalDeleteTasks() { + if (client_ == nullptr || local_disk_segment_id_.empty() || + (local_disk_capabilities_ & kLocalDiskCapabilityObjectTombstoneV1) == + 0) { + return {}; + } + + constexpr uint32_t kDeleteBatchLimit = 128; + std::vector tasks; + auto fetch = client_->FetchLocalDeleteTasks(local_disk_segment_id_, + local_disk_mount_epoch_, + kDeleteBatchLimit, tasks); + if (!fetch || tasks.empty()) { + return fetch; + } + + for (auto& task : tasks) { + task.key = TenantId(task.tenant_id).MakeScopedKey(task.key); + } + const auto results = storage_backend_->BatchMarkDeleted(tasks); + if (results.size() != tasks.size()) { + return tl::unexpected(ErrorCode::INTERNAL_ERROR); + } + + std::vector terminal_task_ids; + terminal_task_ids.reserve(results.size()); + bool tombstone_created = false; + for (size_t i = 0; i < results.size(); ++i) { + const auto& result = results[i]; + const auto& task = tasks[i]; + VLOG(1) << "LOCAL_DISK delete task=" << task.task_id.high << "-" + << task.task_id.low + << ", storage_id=" << task.local_disk_segment_id + << ", tenant=" << task.tenant_id + << ", key_hash=" << std::hash{}(task.key) + << ", incarnation=" << task.object_incarnation.high << "-" + << task.object_incarnation.low + << ", bucket=" << task.expected_bucket_id + << ", result=" << LocalDeleteResultLabel(result.result); + if (result.IsTerminal()) { + terminal_task_ids.push_back(result.task_id); + } + tombstone_created |= result.result == LocalDeleteResult::kRemoved; + } + if (tombstone_created) { + storage_backend_->RequestGarbageCollection(); + } + if (terminal_task_ids.empty()) { + return {}; + } + return client_->AckLocalDeleteTasks( + local_disk_segment_id_, local_disk_mount_epoch_, terminal_task_ids); +} + +ErrorCode FileStorage::ReconcileScannedObjects( + const std::vector& keys, + std::vector& metadatas) { + for (auto& metadata : metadatas) { + metadata.transport_endpoint = local_rpc_addr_; + } + auto objects = BuildOffloadTasksFromStorageKeys(keys, metadatas); + if (local_disk_segment_id_.empty() || + (local_disk_capabilities_ & kLocalDiskCapabilityObjectTombstoneV1) == + 0) { + auto notify = client_->NotifyOffloadSuccess(objects, metadatas); + return notify ? ErrorCode::OK : notify.error(); + } + auto decisions = client_->ReconcileLocalDiskObjects( + local_disk_segment_id_, local_disk_mount_epoch_, objects); + if (!decisions || decisions->size() != objects.size()) { + return decisions ? ErrorCode::INTERNAL_ERROR : decisions.error(); + } + + std::vector stale; + std::vector retained_objects; + std::vector retained_metadatas; + for (size_t i = 0; i < objects.size(); ++i) { + if ((*decisions)[i] != 0) { + retained_objects.push_back(objects[i]); + retained_metadatas.push_back(metadatas[i]); + continue; + } + stale.push_back(LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = local_disk_segment_id_, + .tenant_id = objects[i].tenant_id, + .key = TenantId(objects[i].tenant_id).MakeScopedKey(objects[i].key), + .object_incarnation = objects[i].GetObjectIncarnation(), + .expected_bucket_id = metadatas[i].bucket_id, + }); + } + + bool tombstone_created = false; + for (const auto& result : storage_backend_->BatchMarkDeleted(stale)) { + if (!result.IsTerminal()) { + return result.error; + } + tombstone_created |= result.result == LocalDeleteResult::kRemoved; + } + if (tombstone_created) { + storage_backend_->RequestGarbageCollection(); + } + if (retained_objects.empty()) { + return ErrorCode::OK; + } + auto notify = + client_->NotifyOffloadSuccess(retained_objects, retained_metadatas); + return notify ? ErrorCode::OK : notify.error(); +} + void FileStorage::RemoveAll() { // TODO(tenant-isolation): This performs a tenant-UNAWARE global wipe of the // storage directory. Storage backends store physical files without a @@ -1214,9 +1423,15 @@ tl::expected FileStorage::BatchQuerySegmentSlices( } tl::expected FileStorage::RegisterLocalMemory() { + // The buffer pool backs SSD-offload read results that are fetched by + // remote peers via RDMA READ. It must therefore be registered with + // remote_accessible=true so its BufferDesc publishes an rkey; otherwise + // every remote read of an offloaded object fails with "No rkey for MR + // access" (the pool is only reachable through the address-range lookup, + // and with remote_accessible=false the rkey array is left empty). auto error_code = client_->RegisterLocalMemory( client_buffer_allocator_->getBase(), config_.local_buffer_size, - kWildcardLocation, false, true); + kWildcardLocation, true, true); if (!error_code) { LOG(ERROR) << "Failed to register local memory: " << error_code.error(); return error_code; @@ -1327,36 +1542,17 @@ void FileStorage::ClientBufferGCThreadFunc() { } bool FileStorage::ReleaseBuffer(uint64_t batch_id) { - // Owner-side ClientBuffer release (OwnerReleaseBuffer); shared by the pull - // path (release_offload_buffer RPC) and the push path (inline after WRITE). - SpDiag::PerfPoint pt_release(PerfKey::GET_SSD_OWNER_RELEASE, - SpDiag::PerfLevel::MODULE); - pt_release.Start(); - const auto start = std::chrono::steady_clock::now(); - const bool breakdown_log = mooncake::logging::ShouldSampleHiFreqLog( - mooncake::logging::CurrentTraceId()); MutexLocker locker(&client_buffer_mutex_); auto it = client_buffer_allocated_batches_.find(batch_id); - const bool found = it != client_buffer_allocated_batches_.end(); - if (found) { + if (it != client_buffer_allocated_batches_.end()) { VLOG(1) << "Releasing buffer for batch_id: " << batch_id << " (transfer completed)"; client_buffer_allocated_batches_.erase(it); - } else { - VLOG(1) << "batch_id " << batch_id - << " not found (may have been GC'd already)"; - } - const auto total_us = std::chrono::duration_cast( - std::chrono::steady_clock::now() - start) - .count(); - if (breakdown_log) { - MC_LOG(INFO) << "storage_release_breakdown batch_id=" << batch_id - << " total_us=" << total_us - << " found=" << (found ? 1 : 0) - << " status=" << (found ? "ok" : "not_found"); - } - pt_release.End(found ? 0 : -1); - return found; + return true; + } + VLOG(1) << "batch_id " << batch_id + << " not found (may have been GC'd already)"; + return false; } tl::expected FileStorage::ReRegisterOffloadedObjects() { @@ -1372,33 +1568,24 @@ tl::expected FileStorage::ReRegisterOffloadedObjects() { storage_backend_->ResetScanIterator(); LOG(INFO) << "ReRegisterOffloadedObjects: about to call " "storage_backend_->ScanMeta()"; - auto scan_meta_result = - storage_backend_->ScanMeta( - [this, &total_keys, &total_batches, &total_failures]( - const std::vector& keys, - std::vector& metadatas) { - total_batches++; - total_keys += keys.size(); - for (auto& metadata : metadatas) { - metadata.transport_endpoint = local_rpc_addr_; - } - auto tasks = BuildOffloadTasksFromStorageKeys(keys, metadatas); - auto add_object_result = - client_->NotifyOffloadSuccess(tasks, metadatas); - if (!add_object_result) { - total_failures++; - LOG(ERROR) - << "ReRegisterOffloadedObjects: NotifyOffloadSuccess " - << "failed for batch " << total_batches << " with " - << keys.size() - << " keys, error: " << add_object_result.error(); - return add_object_result.error(); - } - LOG(INFO) << "ReRegisterOffloadedObjects: NotifyOffloadSuccess " - << "succeeded for batch " << total_batches << " with " - << keys.size() << " keys"; - return ErrorCode::OK; - }); + auto scan_meta_result = storage_backend_->ScanMeta( + [this, &total_keys, &total_batches, &total_failures]( + const std::vector& keys, + std::vector& metadatas) { + total_batches++; + total_keys += keys.size(); + const auto reconcile = ReconcileScannedObjects(keys, metadatas); + if (reconcile != ErrorCode::OK) { + total_failures++; + LOG(ERROR) << "ReRegisterOffloadedObjects: reconciliation " + << "failed for batch " << total_batches << " with " + << keys.size() << " keys, error: " << reconcile; + return reconcile; + } + LOG(INFO) << "ReRegisterOffloadedObjects: reconciled batch " + << total_batches << " with " << keys.size() << " keys"; + return ErrorCode::OK; + }); LOG(INFO) << "ReRegisterOffloadedObjects: ScanMeta returned. success=" << scan_meta_result.has_value(); diff --git a/mooncake-store/src/ha/oplog/oplog_applier.cpp b/mooncake-store/src/ha/oplog/oplog_applier.cpp index b715b30d06..f66e5045e6 100644 --- a/mooncake-store/src/ha/oplog/oplog_applier.cpp +++ b/mooncake-store/src/ha/oplog/oplog_applier.cpp @@ -2,6 +2,8 @@ #include +#include + #include "ha_metric_manager.h" #include "metadata_store.h" #include "ha/oplog/oplog_types.h" @@ -68,7 +70,14 @@ bool OpLogApplier::ApplyOpLogEntry(const OpLogEntry& entry) { ApplyPutRevoke(entry); break; case OpType::REMOVE: - ApplyRemove(entry); + if (!ApplyRemove(entry)) { + return false; + } + break; + case OpType::LOCAL_DELETE_ACK: + if (!ApplyLocalDeleteAck(entry)) { + return false; + } break; case OpType::SEGMENT_MOUNT: ApplySegmentMount(entry); @@ -189,7 +198,40 @@ void OpLogApplier::ApplyPutRevoke(const OpLogEntry& entry) { } } -void OpLogApplier::ApplyRemove(const OpLogEntry& entry) { +bool OpLogApplier::ApplyRemove(const OpLogEntry& entry) { + if (!entry.payload.empty()) { + LocalDeleteRemovePayloadV1 payload; + if (struct_pack::deserialize_to(payload, entry.payload) != + struct_pack::errc::ok) { + LOG(ERROR) << "OpLogApplier: invalid LOCAL_DISK delete intents, " + << "sequence_id=" << entry.sequence_id; + return false; + } + if (payload.schema_version != 1) { + LOG(ERROR) << "OpLogApplier: unsupported LOCAL_DISK delete " + "schema, sequence_id=" + << entry.sequence_id; + return false; + } + if (payload.delete_intents.size() > kMaxLocalDeleteTasksPerBatch || + std::any_of( + payload.delete_intents.begin(), payload.delete_intents.end(), + [&](const LocalDeleteTask& task) { + return task.local_disk_segment_id.empty() || + (task.task_id.high == 0 && task.task_id.low == 0) || + task.tenant_id != entry.tenant_id || + task.key != entry.object_key || + task.object_incarnation != + payload.object_incarnation; + }) || + !metadata_store_->ApplyRemoveWithLocalDeleteTasks( + entry.tenant_id, entry.object_key, payload.delete_intents)) { + LOG(ERROR) << "OpLogApplier: invalid LOCAL_DISK delete intents, " + << "sequence_id=" << entry.sequence_id; + return false; + } + return true; + } if (!metadata_store_->Remove(entry.tenant_id, entry.object_key)) { LOG(WARNING) << "OpLogApplier: failed to Remove key=" << entry.object_key @@ -199,6 +241,35 @@ void OpLogApplier::ApplyRemove(const OpLogEntry& entry) { VLOG(1) << "OpLogApplier: applied REMOVE, key=" << entry.object_key << ", sequence_id=" << entry.sequence_id; } + return true; +} + +bool OpLogApplier::ApplyLocalDeleteAck(const OpLogEntry& entry) { + LocalDeleteAckPayloadV1 payload; + if (struct_pack::deserialize_to(payload, entry.payload) != + struct_pack::errc::ok) { + LOG(ERROR) << "OpLogApplier: invalid LOCAL_DELETE_ACK payload, " + << "sequence_id=" << entry.sequence_id; + return false; + } + if (payload.schema_version != 1) { + LOG(ERROR) << "OpLogApplier: unsupported LOCAL_DELETE_ACK schema, " + << "sequence_id=" << entry.sequence_id; + return false; + } + if (payload.local_disk_segment_id.empty() || + payload.task_ids.size() > kMaxLocalDeleteTasksPerBatch || + std::any_of(payload.task_ids.begin(), payload.task_ids.end(), + [](const LocalDeleteTaskId& task_id) { + return task_id.high == 0 && task_id.low == 0; + })) { + LOG(ERROR) << "OpLogApplier: invalid LOCAL_DELETE_ACK payload, " + << "sequence_id=" << entry.sequence_id; + return false; + } + metadata_store_->AckLocalDeleteTasks(payload.local_disk_segment_id, + payload.task_ids); + return true; } const StandbySegmentRegistry& OpLogApplier::GetSegmentRegistry() const { diff --git a/mooncake-store/src/local_delete.cpp b/mooncake-store/src/local_delete.cpp new file mode 100644 index 0000000000..6589877568 --- /dev/null +++ b/mooncake-store/src/local_delete.cpp @@ -0,0 +1,270 @@ +#include "local_delete.h" + +#include +#include + +namespace mooncake { + +namespace { + +template +T GenerateNamedId() { + UUID id{0, 0}; + do { + id = generate_uuid(); + } while (id == UUID{0, 0}); + return T{.high = id.first, .low = id.second}; +} + +size_t PendingCount( + const std::unordered_map< + std::string, std::map>& pending) { + size_t count = 0; + for (const auto& entry : pending) { + count += entry.second.size(); + } + return count; +} + +bool IsValidTask(const LocalDeleteTask& task) { + return !task.local_disk_segment_id.empty() && + (task.task_id.high != 0 || task.task_id.low != 0); +} + +} // namespace + +ObjectIncarnation GenerateObjectIncarnation() { + return GenerateNamedId(); +} + +LocalDeleteTaskId GenerateLocalDeleteTaskId() { + return GenerateNamedId(); +} + +LocalDeleteRegistry::Reservation::Reservation( + LocalDeleteRegistry* registry, std::vector tasks) + : registry_(registry), tasks_(std::move(tasks)) {} + +LocalDeleteRegistry::Reservation::~Reservation() { + if (!released_) { + registry_->ReleaseReservation(tasks_.size()); + } +} + +void LocalDeleteRegistry::Reservation::Publish() { + if (released_) { + return; + } + released_ = true; + registry_->PublishReservation(std::move(tasks_)); +} + +tl::expected, ErrorCode> +LocalDeleteRegistry::Reserve(std::vector tasks) { + std::lock_guard lock(mutex_); + if (!std::all_of(tasks.begin(), tasks.end(), IsValidTask)) { + return tl::unexpected(ErrorCode::INVALID_PARAMS); + } + const size_t pending_count = PendingCount(pending_); + if (pending_count > capacity_ || reserved_ > capacity_ - pending_count || + tasks.size() > capacity_ - pending_count - reserved_) { + return tl::unexpected(ErrorCode::TASK_PENDING_LIMIT_EXCEEDED); + } + reserved_ += tasks.size(); + return std::shared_ptr( + new Reservation(this, std::move(tasks))); +} + +LocalDiskMountInfo LocalDeleteRegistry::Mount( + const UUID& client_id, const std::string& local_disk_segment_id, + uint32_t capabilities) { + if (local_disk_segment_id.empty()) { + return {}; + } + std::lock_guard lock(mutex_); + auto previous_storage = storage_by_client_.find(client_id); + if (previous_storage != storage_by_client_.end() && + previous_storage->second != local_disk_segment_id) { + auto previous_mount = mounts_.find(previous_storage->second); + if (previous_mount != mounts_.end() && + previous_mount->second.client_id == client_id) { + mounts_.erase(previous_mount); + } + storage_by_client_.erase(previous_storage); + } + + auto& state = mounts_[local_disk_segment_id]; + if (state.mount_epoch == 0) { + do { + const auto epoch_id = generate_uuid(); + state.mount_epoch = epoch_id.first ^ epoch_id.second; + } while (state.mount_epoch == 0); + } else if (state.client_id != client_id) { + storage_by_client_.erase(state.client_id); + ++state.mount_epoch; + if (state.mount_epoch == 0) state.mount_epoch = 1; + } + state.client_id = client_id; + state.capabilities = capabilities; + storage_by_client_[client_id] = local_disk_segment_id; + return {state.mount_epoch, state.capabilities}; +} + +void LocalDeleteRegistry::Unmount(const UUID& client_id) { + std::lock_guard lock(mutex_); + auto storage_it = storage_by_client_.find(client_id); + if (storage_it == storage_by_client_.end()) { + return; + } + auto mount_it = mounts_.find(storage_it->second); + if (mount_it != mounts_.end() && mount_it->second.client_id == client_id) { + mounts_.erase(mount_it); + } + storage_by_client_.erase(storage_it); +} + +tl::expected LocalDeleteRegistry::ValidateMount( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch) const { + std::lock_guard lock(mutex_); + auto it = mounts_.find(local_disk_segment_id); + if (it == mounts_.end() || it->second.client_id != client_id || + it->second.mount_epoch != mount_epoch || + (it->second.capabilities & kLocalDiskCapabilityObjectTombstoneV1) == + 0) { + return tl::unexpected(ErrorCode::ILLEGAL_CLIENT); + } + return {}; +} + +tl::expected, ErrorCode> +LocalDeleteRegistry::Fetch(const UUID& client_id, + const std::string& local_disk_segment_id, + uint64_t mount_epoch, uint32_t limit) const { + if (limit == 0) { + return std::vector{}; + } + std::lock_guard lock(mutex_); + auto mount_it = mounts_.find(local_disk_segment_id); + if (mount_it == mounts_.end() || mount_it->second.client_id != client_id || + mount_it->second.mount_epoch != mount_epoch || + (mount_it->second.capabilities & + kLocalDiskCapabilityObjectTombstoneV1) == 0) { + return tl::unexpected(ErrorCode::ILLEGAL_CLIENT); + } + + std::vector result; + auto pending_it = pending_.find(local_disk_segment_id); + if (pending_it == pending_.end()) { + return result; + } + result.reserve(std::min(limit, pending_it->second.size())); + for (const auto& entry : pending_it->second) { + if (result.size() == limit) { + break; + } + result.push_back(entry.second); + } + return result; +} + +size_t LocalDeleteRegistry::Erase( + const std::string& local_disk_segment_id, + const std::vector& task_ids) { + std::lock_guard lock(mutex_); + auto pending_it = pending_.find(local_disk_segment_id); + if (pending_it == pending_.end()) { + return 0; + } + size_t erased = 0; + for (const auto& task_id : task_ids) { + erased += pending_it->second.erase(task_id); + } + if (pending_it->second.empty()) { + pending_.erase(pending_it); + } + return erased; +} + +bool LocalDeleteRegistry::ApplyDurableTasks( + const std::vector& tasks) { + std::lock_guard lock(mutex_); + if (!std::all_of(tasks.begin(), tasks.end(), IsValidTask)) { + return false; + } + const size_t pending_count = PendingCount(pending_); + std::set> new_tasks; + for (const auto& task : tasks) { + auto storage_it = pending_.find(task.local_disk_segment_id); + if (storage_it == pending_.end() || + !storage_it->second.contains(task.task_id)) { + new_tasks.emplace(task.local_disk_segment_id, task.task_id); + } + } + const size_t new_count = new_tasks.size(); + if (pending_count > capacity_ || new_count > capacity_ - pending_count) { + return false; + } + for (const auto& task : tasks) { + pending_[task.local_disk_segment_id].try_emplace(task.task_id, task); + } + return true; +} + +std::vector LocalDeleteRegistry::Snapshot() const { + std::lock_guard lock(mutex_); + std::vector result; + result.reserve(PendingCount(pending_)); + for (const auto& storage_entry : pending_) { + for (const auto& task_entry : storage_entry.second) { + result.push_back(task_entry.second); + } + } + return result; +} + +bool LocalDeleteRegistry::Restore(const std::vector& tasks) { + std::lock_guard lock(mutex_); + if (tasks.size() > capacity_ || + !std::all_of(tasks.begin(), tasks.end(), IsValidTask)) { + return false; + } + pending_.clear(); + mounts_.clear(); + storage_by_client_.clear(); + reserved_ = 0; + for (const auto& task : tasks) { + pending_[task.local_disk_segment_id].try_emplace(task.task_id, task); + } + return true; +} + +void LocalDeleteRegistry::Reset() { + std::lock_guard lock(mutex_); + pending_.clear(); + mounts_.clear(); + storage_by_client_.clear(); + reserved_ = 0; +} + +size_t LocalDeleteRegistry::Size() const { + std::lock_guard lock(mutex_); + return PendingCount(pending_); +} + +void LocalDeleteRegistry::ReleaseReservation(size_t count) { + std::lock_guard lock(mutex_); + reserved_ -= std::min(reserved_, count); +} + +void LocalDeleteRegistry::PublishReservation( + std::vector tasks) { + std::lock_guard lock(mutex_); + reserved_ -= std::min(reserved_, tasks.size()); + for (auto& task : tasks) { + pending_[task.local_disk_segment_id].try_emplace(task.task_id, + std::move(task)); + } +} + +} // namespace mooncake diff --git a/mooncake-store/src/master_service.cpp b/mooncake-store/src/master_service.cpp index 2aef820aa5..a5e0c73e30 100644 --- a/mooncake-store/src/master_service.cpp +++ b/mooncake-store/src/master_service.cpp @@ -1,5 +1,6 @@ #include "master_service.h" +#include #include #include #include @@ -24,8 +25,6 @@ #include "http_metadata_server.h" #include "master_metric_manager.h" -#include "master_perf.h" -#include "mooncake_logging.h" #include "common.h" #include "segment.h" #ifdef USE_HTTP @@ -105,16 +104,12 @@ uint64_t SaturatingMultiply(uint64_t lhs, uint64_t rhs) { // Decides whether PutStart may proceed with the replicas that were // actually allocated. Three deliberately different policies apply: // -// - Memory-only (nof_replica_num == 0): best-effort by default. Fewer -// than config.replica_num replicas (but at least one) still succeed, -// even though DetermineReplicaWriteMode() classifies such configs as +// - Memory-only (nof_replica_num == 0): best-effort. Fewer than +// config.replica_num replicas (but at least one) still succeed, even +// though DetermineReplicaWriteMode() classifies such configs as // RELIABLE_MULTI_REPLICA. The shortfall is surfaced via a WARNING log // (action=put_start_partial_allocation) and the // master_put_start_partial_allocations_total metric. -// When strict_memory_only is true (master flag -// --strict_replica_allocation), the allocation must match -// config.replica_num exactly, otherwise PutStart fails with -// NO_AVAILABLE_HANDLE instead of degrading. // - FLEXIBLE_DUAL_REPLICA (1 memory + 1 NoF): allocating either side // alone is sufficient. // - Any other config with nof_replica_num > 0: strict. Both replica @@ -123,16 +118,11 @@ uint64_t SaturatingMultiply(uint64_t lhs, uint64_t rhs) { // // The "reliable" guarantee of RELIABLE_MULTI_REPLICA is enforced at the // transfer stage (all allocated replicas must complete or the put is -// revoked), not at the allocation stage for memory-only configs unless -// strict_memory_only is enabled. +// revoked), not at the allocation stage for memory-only configs. bool HasExpectedReplicaAllocation(const ReplicateConfig& config, size_t allocated_memory_replicas, - size_t allocated_nof_replicas, - bool strict_memory_only) { + size_t allocated_nof_replicas) { if (config.nof_replica_num == 0) { - if (strict_memory_only) { - return allocated_memory_replicas == config.replica_num; - } return allocated_memory_replicas > 0; } if (DetermineReplicaWriteMode(config) == @@ -149,9 +139,9 @@ tl::expected GetGroupIdForKey( return ""; } if (config.group_ids->size() != key_count || key_index >= key_count) { - MC_LOG(ERROR) << "group_ids.size()=" << config.group_ids->size() - << ", key_count=" << key_count - << ", error=invalid_group_ids"; + LOG(ERROR) << "group_ids.size()=" << config.group_ids->size() + << ", key_count=" << key_count + << ", error=invalid_group_ids"; return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } return config.group_ids->at(key_index); @@ -351,14 +341,6 @@ MasterService::MasterService(const MasterServiceConfig& config) }; #endif - // Strict replica allocation: memory-only multi-replica requests must - // allocate exactly replica_num replicas instead of degrading. - strict_replica_allocation_ = config.strict_replica_allocation; - if (strict_replica_allocation_) { - MC_LOG(INFO) << "Strict replica allocation enabled: memory-only " - "multi-replica requests must be fully satisfied"; - } - // Offload-on-evict: defer LOCAL_DISK offload to eviction time offload_on_evict_ = enable_offload_ && config.offload_on_evict; if (offload_on_evict_) { @@ -1025,9 +1007,6 @@ auto MasterService::ReMountSegment(const std::vector& segments, bool ambiguous_endpoint = false; bool unsupported_cxl = false; - // Track pre-remount readability so the first successful recovery - // starts a fresh lease without extending already-readable objects. - std::unordered_map restored_objects; for (size_t shard_index = 0; shard_index < kNumShards; ++shard_index) { MetadataShardAccessorRW shard(this, shard_index); @@ -1041,8 +1020,7 @@ auto MasterService::ReMountSegment(const std::vector& segments, replica.status() != ReplicaStatus::REMOVED && replica.status() != - ReplicaStatus::FAILED && - !replica.has_invalid_mem_handle(); + ReplicaStatus::FAILED; }, [&](Replica& replica) { auto descriptor = replica.get_descriptor() @@ -1054,24 +1032,6 @@ auto MasterService::ReMountSegment(const std::vector& segments, restore.segment.te_endpoint || descriptor.transport_endpoint_ == restore.segment.name) { - // When multiple segments share the - // same endpoint (e.g. UB per-NUMA - // segments), disambiguate by - // checking whether the replica's - // buffer address falls within this - // segment's virtual address range - // [base, base+size). Each - // per-NUMA segment occupies a - // contiguous and non-overlapping - // range, so at most one segment - // matches. - if (descriptor.buffer_address_ < - restore.segment.base || - descriptor.buffer_address_ >= - restore.segment.base + - restore.segment.size) { - continue; - } if (match != nullptr) { ambiguous_endpoint = true; return; @@ -1086,18 +1046,6 @@ auto MasterService::ReMountSegment(const std::vector& segments, } descriptor.transport_endpoint_ = match->segment.te_endpoint; - if (!restored_objects.contains(&metadata)) { - restored_objects.emplace( - &metadata, - metadata.HasReplica( - [this]( - const Replica& candidate) { - return candidate - .is_memory_replica() && - IsReplicaReadable( - candidate); - })); - } match->replicas.push_back(&replica); match->descriptors.push_back(descriptor); } @@ -1193,16 +1141,6 @@ auto MasterService::ReMountSegment(const std::vector& segments, standby_allocator_keepalive_.erase(restore.segment.te_endpoint); standby_allocator_keepalive_.erase(restore.segment.name); } - for (const auto& [metadata, was_readable] : restored_objects) { - if (!was_readable && - metadata->HasReplica([this](const Replica& replica) { - return replica.is_memory_replica() && - IsReplicaReadable(replica); - })) { - metadata->GrantLease(default_kv_lease_ttl_, - default_kv_soft_pin_ttl_); - } - } } // Change the client status to OK @@ -1722,16 +1660,22 @@ size_t MasterService::EraseReplicasWithCacheTotalAccounting( void MasterService::FinalizeRemovedReplicasAfterDurable( const OpLogEntry& durable_entry, const std::vector& replica_ids, - QuotaEraseMode quota_mode) { + QuotaEraseMode quota_mode, + std::shared_ptr delete_reservation) { + std::shared_lock shared_lock(snapshot_mutex_); if (replica_ids.empty()) { + if (delete_reservation) { + PublishLocalDeleteReservation(delete_reservation); + } return; } - - std::shared_lock shared_lock(snapshot_mutex_); const TenantId tenant_id(durable_entry.tenant_id); const size_t shard_idx = getMetadataShardIndex(tenant_id, durable_entry.object_key); MetadataShardAccessorRW shard(this, shard_idx); + if (delete_reservation) { + PublishLocalDeleteReservation(delete_reservation); + } auto tenant_it = shard->tenants.find(tenant_id); if (tenant_it == shard->tenants.end()) { return; @@ -1763,14 +1707,6 @@ void MasterService::FinalizeRemovedReplicasAfterDurable( const bool erased_local_disk = std::any_of( erased_replicas.begin(), erased_replicas.end(), [](const Replica& replica) { return replica.is_local_disk_replica(); }); - std::vector local_disk_holders; - for (const auto& replica : erased_replicas) { - if (!replica.is_local_disk_replica()) continue; - auto client_id = replica.get_local_disk_client_id(); - if (client_id.has_value()) { - local_disk_holders.push_back(client_id.value()); - } - } ReleaseLocalDiskUsage(erased_replicas); if (erased_local_disk) { shard.OnDiskReplicaRemoved(erased_local_disk, metadata); @@ -1781,11 +1717,52 @@ void MasterService::FinalizeRemovedReplicasAfterDurable( shard->tenants.erase(tenant_it); } } - if (erased_local_disk) { - EnqueueRemoveTasks( - local_disk_holders, - RemoveTaskItem{tenant_id.value(), durable_entry.object_key}); +} + +tl::expected, ErrorCode> +MasterService::ReserveLocalDeleteTasks(const ObjectIdentity& object_id, + const ObjectMetadata& metadata) { + if (enable_ha_ && !enable_oplog_) { + return local_delete_registry_.Reserve({}); + } + std::vector tasks; + std::unordered_set seen_storage_ids; + metadata.VisitReplicas( + [](const Replica& replica) { + return replica.is_local_disk_replica() && replica.is_completed(); + }, + [&](const Replica& replica) { + const auto descriptor = replica.get_descriptor(); + const auto* local_disk = std::get_if( + &descriptor.descriptor_variant); + if (local_disk == nullptr || + local_disk->GetLocalDiskSegmentId().empty() || + (local_disk->GetCapabilities() & + kLocalDiskCapabilityObjectTombstoneV1) == 0 || + local_disk->GetObjectIncarnation() != + metadata.object_incarnation || + !seen_storage_ids.insert(local_disk->GetLocalDiskSegmentId()) + .second) { + return; + } + tasks.push_back(LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = local_disk->GetLocalDiskSegmentId(), + .tenant_id = object_id.tenant_id.value(), + .key = object_id.user_key, + .object_incarnation = metadata.object_incarnation, + .expected_bucket_id = local_disk->GetBucketId(), + }); + }); + if (tasks.size() > kMaxLocalDeleteTasksPerBatch) { + return tl::unexpected(ErrorCode::TASK_PENDING_LIMIT_EXCEEDED); } + return local_delete_registry_.Reserve(std::move(tasks)); +} + +void MasterService::PublishLocalDeleteReservation( + const std::shared_ptr& reservation) { + reservation->Publish(); } void MasterService::FinalizeMetadataEraseAfterDurable( @@ -1822,7 +1799,6 @@ void MasterService::FinalizeExpiredProcessingReplicasAfterDurable( } auto& metadata = accessor.Get(); - auto replicas = PopReplicasWithCacheTotalAccounting( metadata, &Replica::fn_is_processing); if (!replicas.empty()) { @@ -1912,11 +1888,9 @@ tl::expected MasterService::PersistStaleHandleCleanupForHA( const auto op_type = plan.would_invalidate ? OpType::REMOVE : OpType::PUT_END; const std::string payload = - plan.would_invalidate - ? std::string{} - : SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, plan.remaining, - metadata.group_id, metadata.data_type); + plan.would_invalidate ? std::string{} + : SerializeMetadataForOpLogFromReplicaDescriptors( + metadata, plan.remaining); auto reservation = ReserveBatchOpLogSlot(); if (!reservation) { @@ -1944,54 +1918,6 @@ tl::expected MasterService::PersistStaleHandleCleanupForHA( return {}; } -namespace { - -constexpr int kOplogRetryMaxAttempts = 10; -constexpr int kOplogRetryMaxAttemptsUnavailable = 5; -constexpr int kOplogRetryMaxDelayMs = 16; - -template -auto RetryOplogPersist(F&& persist_fn) -> decltype(std::declval()()) { - for (int attempt = 0; attempt <= kOplogRetryMaxAttempts; ++attempt) { - auto result = persist_fn(); - if (result) { - return result; - } - const ErrorCode err = result.error(); - - if (err == ErrorCode::TASK_PENDING_LIMIT_EXCEEDED) { - // Slots full: writer is sealing the current batch and will - // free capacity imminently. Worth waiting. - if (attempt == kOplogRetryMaxAttempts) { - return result; - } - std::this_thread::sleep_for(std::chrono::milliseconds( - std::min(1 << attempt, kOplogRetryMaxDelayMs))); - continue; - } - - if (err == ErrorCode::UNAVAILABLE_IN_CURRENT_STATUS) { - // Writer not accepting: write_batch is retrying against the - // KV backend. Recovery depends on the backend; bail out - // earlier to avoid spinning on a persistent outage. - if (attempt >= kOplogRetryMaxAttemptsUnavailable) { - LOG(WARNING) << "Oplog writer not accepting after " - << attempt << " retries, falling back to local"; - return result; - } - std::this_thread::sleep_for(std::chrono::milliseconds( - std::min(1 << attempt, kOplogRetryMaxDelayMs))); - continue; - } - - // Non-backpressure errors (INVALID_PARAMS etc.) — no retry. - return result; - } - return tl::unexpected(ErrorCode::INTERNAL_ERROR); -} - -} // namespace - std::unordered_map::iterator MasterService::EraseMetadata( TenantState& tenant_state, @@ -2025,17 +1951,15 @@ MasterService::EraseMetadata( const std::string group_id = it->second.group_id; auto& metadata = it->second; - // Clean up offloading_tasks + dec_refcnt before erasing metadata. + // Clean up offloading_task + dec_refcnt before erasing metadata. // When BatchEvict deletes metadata, Store Worker may still have an // in-flight offload for this key. Without this cleanup the task // becomes an orphan that only expires after 600s. auto offload_it = tenant_state.offloading_tasks.find(key); if (offload_it != tenant_state.offloading_tasks.end()) { - for (const auto& task : offload_it->second) { - auto source = metadata.GetReplicaByID(task.source_id); - if (source != nullptr) { - source->dec_refcnt(); - } + auto source = metadata.GetReplicaByID(offload_it->second.source_id); + if (source != nullptr) { + source->dec_refcnt(); } tenant_state.offloading_tasks.erase(offload_it); @@ -2197,16 +2121,15 @@ void MasterService::ClearInvalidHandles( if (enable_ha_) { if (enable_oplog_) { auto persist_result = - RetryOplogPersist([&]() { - return PersistStaleHandleCleanupForHA( - "ClearInvalidHandles", - tenant_it->first, it->first, - it->second, cleanup_plan); - }); - if (persist_result) { + PersistStaleHandleCleanupForHA( + "ClearInvalidHandles", tenant_it->first, + it->first, it->second, cleanup_plan); + if (!persist_result) { ++it; continue; } + ++it; + continue; } } if (CleanupStaleHandles(it->second, alive_clients, @@ -2220,29 +2143,25 @@ void MasterService::ClearInvalidHandles( if (enable_ha_) { if (enable_oplog_) { auto persist_result = - RetryOplogPersist([&]() { - return AppendOpLogWithDurableFinalize( - OpType::REMOVE, - tenant_it->first.value(), - it->first, {}, - [this]( - const OpLogEntry& durable_entry) { - FinalizeMetadataEraseAfterDurable( - durable_entry, - QuotaEraseMode::kFull); - }); - }); - if (persist_result) { - // OPLog path succeeded – skip local erase. + AppendOpLogWithDurableFinalize( + OpType::REMOVE, tenant_it->first.value(), + it->first, {}, + [this](const OpLogEntry& durable_entry) { + FinalizeMetadataEraseAfterDurable( + durable_entry, + QuotaEraseMode::kFull); + }); + if (!persist_result) { + LOG(WARNING) + << "ClearInvalidHandles(last replica)" + << ": REMOVE persist failed for key=" + << it->first << ", err=" + << static_cast(persist_result.error()); ++it; continue; } - LOG(WARNING) - << "ClearInvalidHandles(last replica)" - << ": REMOVE persist failed for key=" - << it->first << ", err=" - << static_cast(persist_result.error()); - // Fall through to local erase. + ++it; + continue; } } it = EraseMetadata(tenant_state, it, tenant_it->first, @@ -2427,7 +2346,7 @@ auto MasterService::ExistKey(const std::string& key, const TenantId& tenant_id) MetadataAccessorRO accessor(this, MakeObjectIdentityForRequest(key, tenant_id)); if (!accessor.Exists()) { - MC_VLOG(1) << "key=" << key << ", info=object_not_found"; + VLOG(1) << "key=" << key << ", info=object_not_found"; return false; } @@ -2634,13 +2553,18 @@ auto MasterService::QuerySegmentStatusById(const UUID& segment_id) return status; } -void MasterService::RestoreFromStandbySnapshot( +tl::expected MasterService::RestoreFromStandbySnapshot( const std::vector& objects, uint64_t initial_oplog_sequence_id, - const std::vector& segments) { + const std::vector& segments, + const std::vector& pending_local_deletes) { // The ordered writer initializes its sequence from durable_prefix. (void)initial_oplog_sequence_id; - + if (!local_delete_registry_.Restore(pending_local_deletes)) { + LOG(ERROR) << "Standby snapshot contains too many pending LOCAL_DISK " + "delete tasks"; + return tl::unexpected(ErrorCode::INVALID_PARAMS); + } // 2. Build allocator keepalive map for standby segments. for (const auto& [segment, bytes] : standby_accounted_memory_bytes_) { MasterMetricManager::instance().dec_allocated_mem_size( @@ -2701,6 +2625,8 @@ void MasterService::RestoreFromStandbySnapshot( const auto& entry = *entry_ptr; auto [tenant_id, user_key] = resolve_standby_object(entry); const auto& standby_meta = entry.metadata; + ObjectIncarnation restored_incarnation = + standby_meta.GetObjectIncarnation(); std::vector replicas; replicas.reserve(standby_meta.replicas.size()); @@ -2746,19 +2672,29 @@ void MasterService::RestoreFromStandbySnapshot( } else if (desc.is_local_disk_replica()) { const auto& local_disk_desc = desc.get_local_disk_descriptor(); + if (restored_incarnation.IsZero()) { + restored_incarnation = + local_disk_desc.GetObjectIncarnation(); + } replicas.emplace_back( local_disk_desc.client_id, local_disk_desc.object_size, - local_disk_desc.transport_endpoint, desc.status); + local_disk_desc.transport_endpoint, desc.status, + local_disk_desc.GetLocalDiskSegmentId(), + local_disk_desc.GetMountEpoch(), + local_disk_desc.GetCapabilities(), + local_disk_desc.GetBucketId(), + local_disk_desc.GetObjectIncarnation()); } } auto& tenant_state = shard->tenants[tenant_id]; tenant_state.metadata.emplace( std::piecewise_construct, std::forward_as_tuple(user_key), - std::forward_as_tuple( - standby_meta.client_id, now, standby_meta.size, - std::move(replicas), false, false, standby_meta.data_type, - standby_meta.group_id, tenant_id, user_key)); + std::forward_as_tuple(standby_meta.client_id, now, + standby_meta.size, std::move(replicas), + false, false, standby_meta.data_type, + standby_meta.group_id, tenant_id, + user_key, restored_incarnation)); if (!standby_meta.group_id.empty()) { RegisterGroupMember(tenant_state, tenant_id, user_key, standby_meta.group_id); @@ -2772,6 +2708,7 @@ void MasterService::RestoreFromStandbySnapshot( << segments.size() << " segments, initial_seq_id=" << initial_oplog_sequence_id << ", invalid_endpoints=" << invalid_replica_endpoints_.size(); + return {}; } auto MasterService::QueryIp(const UUID& client_id) @@ -3005,8 +2942,7 @@ auto MasterService::BatchReplicaClear( std::move(reservation.value()), OpType::PUT_END, normalized_tenant.value(), key, SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + metadata, remaining), [this, removed_ids = std::move(removed_ids)]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( @@ -3075,11 +3011,6 @@ bool MasterService::IsReplicaReadable(const Replica& replica) const { return !endpoint || !invalid_replica_endpoints_.contains(*endpoint); } -bool MasterService::IsMemoryReplicaEvictable(const Replica& replica) const { - return replica.is_memory_replica() && replica.is_completed() && - replica.get_refcnt() == 0 && IsReplicaReadable(replica); -} - auto MasterService::GetReplicaListByRegex(const std::string& regex_pattern, const TenantId& tenant_id) -> tl::expected< @@ -3097,16 +3028,6 @@ auto MasterService::GetReplicaListByRegex(const std::string& regex_pattern, } std::shared_lock shared_lock(snapshot_mutex_); - - // Build the set of currently alive client UUIDs - std::unordered_set> alive_clients; - { - std::shared_lock client_lock(client_mutex_); - for (const auto& [client_id, host] : client_host_id_) { - alive_clients.insert(client_id); - } - } - const TenantId& normalized_tenant = ResolveRequestTenantId(tenant_id); for (size_t i = 0; i < kNumShards; ++i) { MetadataShardAccessorRO shard(this, i); @@ -3118,10 +3039,8 @@ auto MasterService::GetReplicaListByRegex(const std::string& regex_pattern, if (std::regex_search(key, pattern)) { std::vector replica_list; metadata.VisitReplicas( - [this, &alive_clients](const Replica& replica) { - return IsReplicaReadable(replica) && - !replica.has_stale_local_disk_client( - alive_clients); + [this](const Replica& replica) { + return IsReplicaReadable(replica); }, [&replica_list](const Replica& replica) { replica_list.emplace_back(replica.get_descriptor()); @@ -3143,53 +3062,6 @@ auto MasterService::GetReplicaListByRegex(const std::string& regex_pattern, return results; } -auto MasterService::GetOffloadEndpoints() - -> tl::expected, ErrorCode> { - std::unordered_set unique_endpoints; - - // Build the set of currently alive client UUIDs - std::unordered_set> alive_clients; - { - std::shared_lock client_lock(client_mutex_); - for (const auto& [client_id, host] : client_host_id_) { - alive_clients.insert(client_id); - } - } - - std::shared_lock shared_lock(snapshot_mutex_); - for (size_t i = 0; i < kNumShards; ++i) { - MetadataShardAccessorRO shard(this, i); - for (const auto& tenant_it : shard->tenants) { - for (const auto& metadata_it : tenant_it.second.metadata) { - const auto& metadata = metadata_it.second; - metadata.VisitReplicas( - [&alive_clients](const Replica& replica) { - return replica.is_completed() && - replica.is_local_disk_replica() && - !replica.has_stale_local_disk_client( - alive_clients); - }, - [&unique_endpoints](const Replica& replica) { - const auto desc = replica.get_descriptor(); - const auto& endpoint = - desc.get_local_disk_descriptor() - .transport_endpoint; - if (!endpoint.empty()) { - unique_endpoints.emplace(endpoint); - } - }); - } - } - } - - std::vector endpoints; - endpoints.reserve(unique_endpoints.size()); - for (const auto& endpoint : unique_endpoints) { - endpoints.emplace_back(endpoint); - } - return endpoints; -} - auto MasterService::GetReplicaList(const std::string& key, const TenantId& tenant_id) -> tl::expected { @@ -3209,20 +3081,10 @@ auto MasterService::GetReplicaList(const std::string& key, } const auto& metadata = accessor.Get(); - // Build the set of currently alive client UUIDs - std::unordered_set> alive_clients; - { - std::shared_lock client_lock(client_mutex_); - for (const auto& [client_id, host] : client_host_id_) { - alive_clients.insert(client_id); - } - } - std::vector replica_list; metadata.VisitReplicas( - [this, &alive_clients](const Replica& replica) { - return IsReplicaReadable(replica) && - !replica.has_stale_local_disk_client(alive_clients); + [this](const Replica& replica) { + return IsReplicaReadable(replica); }, [&replica_list](const Replica& replica) { replica_list.emplace_back(replica.get_descriptor()); @@ -3567,7 +3429,6 @@ auto MasterService::AllocateAndInsertMetadata( const auto write_mode = DetermineReplicaWriteMode(config); size_t allocated_memory_replicas = 0; size_t allocated_nof_replicas = 0; - bool memory_eviction_may_help = false; if (config.replica_num > 0) { const bool use_local_first = allocation_strategy_type_ == AllocationStrategyType::LOCAL_FIRST && @@ -3581,19 +3442,6 @@ auto MasterService::AllocateAndInsertMetadata( ScopedAllocatorAccess allocator_access = segment_manager_.getAllocatorAccess(); const auto& allocator_manager = allocator_access.getAllocatorManager(); - if (allocator_manager.getNames().size() >= config.replica_num) { - for (const auto& name : allocator_manager.getNames()) { - const auto* allocators = allocator_manager.getAllocators(name); - if (allocators != nullptr && - std::any_of(allocators->begin(), allocators->end(), - [](const auto& allocator) { - return allocator && allocator->size() > 0; - })) { - memory_eviction_may_help = true; - break; - } - } - } std::vector preferred_segments; auto append_preferred_segment = [&preferred_segments]( @@ -3633,14 +3481,10 @@ auto MasterService::AllocateAndInsertMetadata( ssd_provider = &*ssd_access; } - SpDiag::PerfPoint pt_alloc_mem(PerfKey::MASTER_PUT_ALLOCATE_MEM, - SpDiag::PerfLevel::KEY_MODULE); - pt_alloc_mem.Start(); auto allocation_result = allocation_strategy_->Allocate( allocator_manager, value_length, config.replica_num, preferred_segments, std::set(), ReplicaType::MEMORY, ssd_provider); - pt_alloc_mem.End(allocation_result.has_value() ? 0 : -1); if (!allocation_result.has_value()) { VLOG(1) << "Failed to allocate replicas for key=" << key @@ -3651,9 +3495,7 @@ auto MasterService::AllocateAndInsertMetadata( } if (write_mode != ReplicaWriteMode::FLEXIBLE_DUAL_REPLICA) { MasterMetricManager::instance().inc_put_start_alloc_failures(); - if (memory_eviction_may_help) { - need_mem_eviction_ = true; - } + need_mem_eviction_ = true; abort_reserved_quota(); return tl::make_unexpected(ErrorCode::NO_AVAILABLE_HANDLE); } @@ -3673,13 +3515,9 @@ auto MasterService::AllocateAndInsertMetadata( std::vector preferred_segments = config.preferred_nof_segments; - SpDiag::PerfPoint pt_alloc_nof(PerfKey::MASTER_PUT_ALLOCATE_NOF, - SpDiag::PerfLevel::KEY_MODULE); - pt_alloc_nof.Start(); auto allocation_result = allocation_strategy_->Allocate( allocator_manager, value_length, config.nof_replica_num, preferred_segments, std::set(), ReplicaType::NOF_SSD); - pt_alloc_nof.End(allocation_result.has_value() ? 0 : -1); if (!allocation_result.has_value()) { VLOG(1) << "Failed to allocate nof replicas for key=" << key @@ -3704,16 +3542,14 @@ auto MasterService::AllocateAndInsertMetadata( #endif if (!HasExpectedReplicaAllocation(config, allocated_memory_replicas, - allocated_nof_replicas, - strict_replica_allocation_)) { + allocated_nof_replicas)) { if ((config.replica_num > 0 && allocated_memory_replicas != config.replica_num) || (config.nof_replica_num > 0 && allocated_nof_replicas != config.nof_replica_num)) { MasterMetricManager::instance().inc_put_start_alloc_failures(); if (config.replica_num > 0 && - allocated_memory_replicas != config.replica_num && - memory_eviction_may_help) { + allocated_memory_replicas != config.replica_num) { need_mem_eviction_ = true; } if (config.nof_replica_num > 0 && @@ -3755,24 +3591,24 @@ auto MasterService::AllocateAndInsertMetadata( std::vector replica_list; replica_list.reserve(replicas.size()); int i = 0; - MC_VLOG(1) << "PutStart, create replicas: client_id=" << client_id - << ", key=" << key << ", value_length=" << value_length; + VLOG(1) << "PutStart, create replicas: client_id=" << client_id + << ", key=" << key << ", value_length=" << value_length; for (const auto& replica : replicas) { const auto desc = replica.get_descriptor(); replica_list.emplace_back(desc); if (replica.is_memory_replica()) { const auto& mem_desc = desc.get_memory_descriptor(); - MC_VLOG(1) << "Replica #" << ++i << ": buffer_address=" - << mem_desc.buffer_descriptor.buffer_address_ - << ", transport_endpoint=" - << mem_desc.buffer_descriptor.transport_endpoint_; + VLOG(1) << "Replica #" << ++i << ": buffer_address=" + << mem_desc.buffer_descriptor.buffer_address_ + << ", transport_endpoint=" + << mem_desc.buffer_descriptor.transport_endpoint_; } else if (replica.is_nof_replica()) { const auto& nof_desc = desc.get_nof_descriptor(); - MC_VLOG(1) << "Replica #" << ++i << ": buffer_address=" - << nof_desc.buffer_descriptor.buffer_address_ - << ", transport_endpoint=" - << nof_desc.buffer_descriptor.transport_endpoint_; + VLOG(1) << "Replica #" << ++i << ": buffer_address=" + << nof_desc.buffer_descriptor.buffer_address_ + << ", transport_endpoint=" + << nof_desc.buffer_descriptor.transport_endpoint_; } } @@ -3840,8 +3676,8 @@ auto MasterService::PutStart(const UUID& client_id, const std::string& key, return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } - MC_VLOG(1) << "key=" << key << ", value_length=" << slice_length - << ", config=" << config << ", action=put_start_begin"; + VLOG(1) << "key=" << key << ", value_length=" << slice_length + << ", config=" << config << ", action=put_start_begin"; auto group_id_result = GetGroupIdForKey(config, 1, 0); if (!group_id_result) { @@ -4056,21 +3892,24 @@ auto MasterService::PutEnd(const UUID& client_id, const ObjectMeta& object_meta, if (enable_offload_ && !offload_on_evict_) { auto& tenant_state = accessor.GetTenantState(); + bool task_created = false; metadata.VisitReplicas( [](const Replica& replica) { return replica.is_completed() && replica.is_memory_replica(); }, - [this, &object_id, &tenant_state](Replica& replica) { - auto result = PushOffloadingQueue(object_id, replica); - if (!result) { - return; - } - auto& tasks = tenant_state.offloading_tasks[object_id.user_key]; - const auto now = std::chrono::system_clock::now(); - for (const auto& client_id : result.value()) { - replica.inc_refcnt(); - tasks.push_back( - OffloadingTask{replica.id(), now, client_id}); + [this, &object_id, &tenant_state, &task_created, + &metadata](Replica& replica) { + auto result = PushOffloadingQueue(object_id, replica, + metadata.object_incarnation); + if (result) { + if (!task_created) { + replica.inc_refcnt(); + tenant_state.offloading_tasks.emplace( + object_id.user_key, + OffloadingTask{replica.id(), + std::chrono::system_clock::now()}); + task_created = true; + } } }); } @@ -4120,35 +3959,46 @@ auto MasterService::AddReplica(const UUID& client_id, const std::string& key, std::shared_lock shared_lock(snapshot_mutex_); const ObjectIdentity object_id{std::move(normalized_tenant), key}; MetadataAccessorRW accessor(this, object_id); - if (!accessor.Exists()) { - accessor.Create( - client_id, - replica.get_descriptor().get_local_disk_descriptor().object_size, - std::vector{}, false); - } - auto& metadata = accessor.Get(); if (replica.type() != ReplicaType::LOCAL_DISK) { LOG(ERROR) << "Invalid replica type: " << replica.type() << ". Expected ReplicaType::LOCAL_DISK."; return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } + const auto incoming = replica.get_descriptor().get_local_disk_descriptor(); + if (!accessor.Exists()) { + if ((incoming.GetCapabilities() & + kLocalDiskCapabilityObjectTombstoneV1) != 0) { + return tl::make_unexpected(ErrorCode::OBJECT_NOT_FOUND); + } + accessor.Create(client_id, incoming.object_size, std::vector{}, + false); + } + auto& metadata = accessor.Get(); + if ((incoming.GetCapabilities() & kLocalDiskCapabilityObjectTombstoneV1) != + 0 && + incoming.GetObjectIncarnation() != metadata.object_incarnation) { + return tl::make_unexpected(ErrorCode::INVALID_VERSION); + } - const bool replacing_existing = + const auto matches_incoming = [&](const Replica& existing) { + if (!existing.is_local_disk_replica()) { + return false; + } + const auto descriptor = + existing.get_descriptor().get_local_disk_descriptor(); + return descriptor.client_id == client_id || + (!incoming.GetLocalDiskSegmentId().empty() && + descriptor.GetLocalDiskSegmentId() == + incoming.GetLocalDiskSegmentId()); + }; + const bool has_existing_local_disk = metadata.HasReplica(&Replica::fn_is_local_disk_replica); - // Build OPLog payload BEFORE moving the replica, so that - // get_descriptor() is still valid on `replica`. - std::string oplog_payload; - bool oplog_required = false; if (enable_oplog_ && ordered_oplog_writer_) { std::vector post; for (const auto& existing : metadata.GetAllReplicas()) { if (existing.status() != ReplicaStatus::COMPLETE) continue; - if (replacing_existing && - existing.type() == ReplicaType::LOCAL_DISK && - existing.get_descriptor() - .get_local_disk_descriptor() - .client_id == client_id) { + if (has_existing_local_disk && matches_incoming(existing)) { // Substitute with the updated descriptor. Replica::Descriptor updated = existing.get_descriptor(); updated.get_local_disk_descriptor().transport_endpoint = @@ -4159,84 +4009,47 @@ auto MasterService::AddReplica(const UUID& client_id, const std::string& key, replica.get_descriptor() .get_local_disk_descriptor() .object_size; + updated.get_local_disk_descriptor().SetDeleteMetadata( + incoming.GetLocalDiskSegmentId(), incoming.GetMountEpoch(), + incoming.GetCapabilities(), incoming.GetBucketId(), + incoming.GetObjectIncarnation()); post.push_back(std::move(updated)); } else { post.push_back(existing.get_descriptor()); } } - if (!replacing_existing) { + if (!has_existing_local_disk) { // The new LOCAL_DISK replica is COMPLETE upon AddReplica. post.push_back(replica.get_descriptor()); } - oplog_payload = SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post, metadata.group_id, - metadata.data_type); - oplog_required = true; + + auto persist_result = AppendOpLogVisibleBeforeDurable( + OpType::PUT_END, object_id.tenant_id.value(), key, + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, post)); + if (!persist_result) { + return tl::make_unexpected(persist_result.error()); + } } - // Step 1: Update metadata first for LOCAL_DISK replicas. - // This ensures the replica is registered immediately, even when the - // OPLog writer is backed up (e.g. during standby promotion recovery). - // LOCAL_DISK is a secondary replica type: losing its OPLog entry is - // recoverable because the client will re-register on remount. - if (!replacing_existing) { + if (!has_existing_local_disk) { std::vector replicas; replicas.emplace_back(std::move(replica)); metadata.AddReplicas(std::move(replicas)); auto& shard = accessor.GetShard(); shard.OnDiskReplicaAdded(metadata); SyncCacheTotalAccounting(metadata); - - // Step 2: Best-effort OPLog write. - if (oplog_required) { - auto persist_result = AppendOpLogVisibleBeforeDurable( - OpType::PUT_END, object_id.tenant_id.value(), key, - oplog_payload); - if (!persist_result) { - LOG(WARNING) << "AddReplica: OpLog skipped for local_disk" - << " (metadata already updated), key=" << key - << ", err=" - << static_cast(persist_result.error()); - } - } return true; } - // Replace-existing path: update the existing LOCAL_DISK replica. - // Record every LOCAL_DISK replica per owning client. First try to refresh - // an existing replica owned by THIS client (idempotent re-offload or - // restart re-registration only changes the endpoint/size). If this client - // has no LOCAL_DISK replica yet, append a new one so that a key offloaded - // by multiple nodes keeps one LOCAL_DISK replica per node. - size_t updated = metadata.VisitReplicas( - [client_id](const Replica& rep) { - return rep.type() == ReplicaType::LOCAL_DISK && - rep.get_descriptor().get_local_disk_descriptor().client_id == - client_id; - }, - [&replica](Replica& rep) { - const auto desc = - replica.get_descriptor().get_local_disk_descriptor(); - rep.update_local_disk_location(desc.transport_endpoint, - desc.object_size); - }); - - if (updated == 0) { - std::vector replicas; - replicas.emplace_back(std::move(replica)); - metadata.AddReplicas(std::move(replicas)); - } - - // Best-effort OPLog write for replace-existing path. - if (oplog_required) { - auto persist_result = AppendOpLogVisibleBeforeDurable( - OpType::PUT_END, object_id.tenant_id.value(), key, oplog_payload); - if (!persist_result) { - LOG(WARNING) << "AddReplica: OpLog skipped for local_disk" - << " (metadata already updated), key=" << key - << ", err=" << static_cast(persist_result.error()); - } - } + metadata.VisitReplicas(matches_incoming, [&replica](Replica& rep) { + const auto descriptor = + replica.get_descriptor().get_local_disk_descriptor(); + rep.update_local_disk_location( + descriptor.object_size, descriptor.transport_endpoint, + descriptor.GetLocalDiskSegmentId(), descriptor.GetMountEpoch(), + descriptor.GetCapabilities(), descriptor.GetBucketId(), + descriptor.GetObjectIncarnation()); + }); return false; } @@ -4248,7 +4061,7 @@ auto MasterService::PutRevoke(const UUID& client_id, const std::string& key, const auto object_id = MakeObjectIdentityForRequest(key, tenant_id); MetadataAccessorRW accessor(this, object_id); if (!accessor.Exists()) { - MC_LOG(INFO) << "key=" << key << ", info=object_not_found"; + LOG(INFO) << "key=" << key << ", info=object_not_found"; return tl::make_unexpected(ErrorCode::OBJECT_NOT_FOUND); } @@ -4307,9 +4120,8 @@ auto MasterService::PutRevoke(const UUID& client_id, const std::string& key, persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::PUT_END, tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, + remaining), [this, removed_ids = std::move(removed_ids)]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( @@ -4440,8 +4252,8 @@ auto MasterService::UpsertStart(const UUID& client_id, const std::string& key, return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } - MC_VLOG(1) << "key=" << key << ", value_length=" << slice_length - << ", config=" << config << ", action=upsert_start_begin"; + VLOG(1) << "key=" << key << ", value_length=" << slice_length + << ", config=" << config << ", action=upsert_start_begin"; auto group_id_result = GetGroupIdForKey(config, 1, 0); if (!group_id_result) { @@ -4866,9 +4678,8 @@ auto MasterService::EvictDiskReplica(const UUID& client_id, persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::PUT_END, metadata.tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, + remaining), [this, removed_ids = std::move(removed_ids)]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( @@ -4888,9 +4699,8 @@ auto MasterService::EvictDiskReplica(const UUID& client_id, } else { persist_result = AppendOpLogWithDurableFinalize( OpType::PUT_END, metadata.tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, + remaining), nullptr); } if (!persist_result) { @@ -5163,9 +4973,8 @@ tl::expected MasterService::CopyEnd( [&post](const Replica& replica) { post.push_back(replica.get_descriptor()); }); - auto payload = SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post, metadata.group_id, - metadata.data_type); + auto payload = + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, post); if (batch_reservation) { auto persist_result = AppendReservedOpLogWithDurableFinalize( std::move(*batch_reservation), OpType::PUT_END, @@ -5493,9 +5302,7 @@ tl::expected MasterService::MoveEnd( persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::PUT_END, metadata.tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post, metadata.group_id, - metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, post), [this, removed_ids = std::vector{source_id}]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( @@ -5504,9 +5311,7 @@ tl::expected MasterService::MoveEnd( } else { persist_result = AppendOpLogWithDurableFinalize( OpType::PUT_END, metadata.tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post, metadata.group_id, - metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, post), nullptr); } if (!persist_result) { @@ -5612,17 +5417,6 @@ auto MasterService::Remove(const std::string& key, const TenantId& tenant_id, } auto& metadata = accessor.Get(); - std::vector local_disk_holders; - metadata.VisitReplicas( - [](const Replica& replica) { - return replica.is_local_disk_replica(); - }, - [&local_disk_holders](Replica& replica) { - auto client_id = replica.get_local_disk_client_id(); - if (client_id.has_value()) { - local_disk_holders.push_back(client_id.value()); - } - }); if (!force && !metadata.IsLeaseExpired()) { VLOG(1) << "key=" << key << ", error=object_has_lease"; @@ -5645,6 +5439,19 @@ auto MasterService::Remove(const std::string& key, const TenantId& tenant_id, return tl::make_unexpected(ErrorCode::OBJECT_HAS_REPLICATION_TASK); } + auto delete_reservation = ReserveLocalDeleteTasks(object_id, metadata); + if (!delete_reservation) { + return tl::unexpected(delete_reservation.error()); + } + const auto remove_payload_bytes = + struct_pack::serialize(LocalDeleteRemovePayloadV1{ + .schema_version = 1, + .object_incarnation = metadata.object_incarnation, + .delete_intents = delete_reservation.value()->tasks(), + }); + const std::string remove_payload(remove_payload_bytes.begin(), + remove_payload_bytes.end()); + if (enable_ha_) { if (enable_oplog_) { auto reservation = ReserveBatchOpLogSlot(); @@ -5659,32 +5466,28 @@ auto MasterService::Remove(const std::string& key, const TenantId& tenant_id, }); auto persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::REMOVE, - object_id.tenant_id.value(), key, {}, - [this, removed_ids = std::move(removed_ids), - local_disk_holders, - tenant_id_for_task = object_id.tenant_id.value(), key]( + object_id.tenant_id.value(), key, remove_payload, + [this, removed_ids, + delete_reservation = delete_reservation.value()]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( - durable_entry, removed_ids, QuotaEraseMode::kFull); - EnqueueRemoveTasks( - local_disk_holders, - RemoveTaskItem{tenant_id_for_task, key}); + durable_entry, removed_ids, QuotaEraseMode::kFull, + std::move(delete_reservation)); }); if (!persist_result) { + for (const auto replica_id : removed_ids) { + if (auto* replica = metadata.GetReplicaByID(replica_id)) { + replica->restore_removed(); + } + } return tl::make_unexpected(persist_result.error()); } return {}; } } + PublishLocalDeleteReservation(delete_reservation.value()); PublishKvRemoved(key, metadata, object_id.tenant_id); - - // Before erasing metadata, collect LOCAL_DISK replica holders so we - // can notify them to reclaim SSD space via RemoveObjectHeartbeat. accessor.Erase(); - - // Push removed key to each LOCAL_DISK holder's removed_keys queue. - EnqueueRemoveTasks(local_disk_holders, RemoveTaskItem{tenant_id.value(), key}); - return {}; } @@ -6081,18 +5884,6 @@ auto MasterService::BatchRemove(const std::vector& keys, auto& metadata = it->second; - std::vector batch_local_disk_holders; - metadata.VisitReplicas( - [](const Replica& replica) { - return replica.is_local_disk_replica(); - }, - [&batch_local_disk_holders](Replica& replica) { - auto cid = replica.get_local_disk_client_id(); - if (cid.has_value()) { - batch_local_disk_holders.push_back(cid.value()); - } - }); - if (!force && !metadata.IsLeaseExpired(now)) { VLOG(1) << "key=" << key << ", error=object_has_lease"; results[original_idx] = @@ -6115,6 +5906,23 @@ auto MasterService::BatchRemove(const std::vector& keys, continue; } + const auto object_id = MakeObjectIdentity(key, normalized_tenant); + auto delete_reservation = + ReserveLocalDeleteTasks(object_id, metadata); + if (!delete_reservation) { + results[original_idx] = + tl::unexpected(delete_reservation.error()); + continue; + } + const auto remove_payload_bytes = + struct_pack::serialize(LocalDeleteRemovePayloadV1{ + .schema_version = 1, + .object_incarnation = metadata.object_incarnation, + .delete_intents = delete_reservation.value()->tasks(), + }); + const std::string remove_payload(remove_payload_bytes.begin(), + remove_payload_bytes.end()); + // Remove object metadata if (enable_ha_) { if (enable_oplog_) { @@ -6134,19 +5942,22 @@ auto MasterService::BatchRemove(const std::vector& keys, auto persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::REMOVE, - normalized_tenant.value(), key, {}, - [this, removed_ids = std::move(removed_ids), - batch_local_disk_holders, - tenant_id = normalized_tenant.value(), key]( + normalized_tenant.value(), key, remove_payload, + [this, removed_ids, + delete_reservation = delete_reservation.value()]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( durable_entry, removed_ids, - QuotaEraseMode::kFull); - EnqueueRemoveTasks( - batch_local_disk_holders, - RemoveTaskItem{tenant_id, key}); + QuotaEraseMode::kFull, + std::move(delete_reservation)); }); if (!persist_result) { + for (const auto replica_id : removed_ids) { + if (auto* replica = + metadata.GetReplicaByID(replica_id)) { + replica->restore_removed(); + } + } results[original_idx] = tl::make_unexpected(persist_result.error()); continue; @@ -6155,20 +5966,12 @@ auto MasterService::BatchRemove(const std::vector& keys, continue; } } - - // Collect LOCAL_DISK replica holders before erasing, so we - // can notify them to reclaim SSD space via RemoveObjectHeartbeat. + PublishLocalDeleteReservation(delete_reservation.value()); EraseMetadata(tenant_state, it, normalized_tenant, QuotaEraseMode::kFull, &shard); if (tenant_state.Empty()) { shard->tenants.erase(tenant_it); } - - // Push removed key to each LOCAL_DISK holder's removed_keys queue. - EnqueueRemoveTasks( - batch_local_disk_holders, - RemoveTaskItem{normalized_tenant.value(), key}); - results[original_idx] = {}; // Success } } @@ -6264,21 +6067,33 @@ MasterService::GetStorageConfig() const { return GetStorageConfigResponse(fsdir, enable_disk_eviction_, quota_bytes_); } -auto MasterService::MountLocalDiskSegment(const UUID& client_id, - bool enable_offloading) - -> tl::expected { +auto MasterService::MountLocalDiskSegment( + const UUID& client_id, bool enable_offloading, + const std::string& local_disk_segment_id, uint32_t capabilities) + -> tl::expected { if (!enable_offload_) { LOG(ERROR) << " The offload functionality is not enabled"; return tl::make_unexpected(ErrorCode::UNABLE_OFFLOAD); } + const bool durable_delete_ack_supported = !enable_ha_ || enable_oplog_; + const uint32_t negotiated_capabilities = + local_disk_segment_id.empty() || !durable_delete_ack_supported + ? 0 + : capabilities & kLocalDiskCapabilityObjectTombstoneV1; std::shared_lock shared_lock(snapshot_mutex_); - ScopedSegmentAccess segment_access = segment_manager_.getSegmentAccess(); - - auto err = - segment_access.MountLocalDiskSegment(client_id, enable_offloading); + const auto mount_info = local_delete_registry_.Mount( + client_id, local_disk_segment_id, negotiated_capabilities); + auto err = ErrorCode::OK; + { + ScopedSegmentAccess segment_access = + segment_manager_.getSegmentAccess(); + err = segment_access.MountLocalDiskSegment( + client_id, enable_offloading, local_disk_segment_id, + mount_info.mount_epoch, mount_info.capabilities); + } if (err == ErrorCode::SEGMENT_ALREADY_EXISTS) { // Return OK because this is an idempotent operation - return {}; + return mount_info; } else if (err != ErrorCode::OK) { return tl::make_unexpected(err); } @@ -6296,36 +6111,56 @@ auto MasterService::MountLocalDiskSegment(const UUID& client_id, return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); } - return {}; + return mount_info; } auto MasterService::OffloadObjectHeartbeat(const UUID& client_id, bool enable_offloading) -> tl::expected, ErrorCode> { std::shared_lock shared_lock(snapshot_mutex_); - ScopedLocalDiskSegmentAccess local_disk_segment_access = - segment_manager_.getLocalDiskSegmentAccess(); - auto& client_local_disk_segment = - local_disk_segment_access.getClientLocalDiskSegment(); - auto local_disk_segment_it = client_local_disk_segment.find(client_id); - if (local_disk_segment_it == client_local_disk_segment.end()) { - LOG(ERROR) << "Local disk segment not found with client id = " - << client_id; + std::shared_ptr local_disk_segment; + { + ScopedLocalDiskSegmentAccess local_disk_segment_access = + segment_manager_.getLocalDiskSegmentAccess(); + auto& client_local_disk_segment = + local_disk_segment_access.getClientLocalDiskSegment(); + auto local_disk_segment_it = client_local_disk_segment.find(client_id); + if (local_disk_segment_it == client_local_disk_segment.end()) { + LOG(ERROR) << "Local disk segment not found with client id = " + << client_id; + return tl::make_unexpected(ErrorCode::SEGMENT_NOT_FOUND); + } + local_disk_segment = local_disk_segment_it->second; + } + + std::string local_disk_segment_id; + uint64_t mount_epoch = 0; + uint32_t capabilities = 0; + { + MutexLocker locker(&local_disk_segment->offloading_mutex_); + local_disk_segment_id = local_disk_segment->local_disk_segment_id; + mount_epoch = local_disk_segment->mount_epoch; + capabilities = local_disk_segment->capabilities; + } + if ((capabilities & kLocalDiskCapabilityObjectTombstoneV1) != 0 && + !local_delete_registry_ + .ValidateMount(client_id, local_disk_segment_id, mount_epoch) + .has_value()) { return tl::make_unexpected(ErrorCode::SEGMENT_NOT_FOUND); } + std::vector result; std::unordered_map offloading_objects_copy; { - MutexLocker locker(&local_disk_segment_it->second->offloading_mutex_); - local_disk_segment_it->second->enable_offloading = enable_offloading; + MutexLocker locker(&local_disk_segment->offloading_mutex_); + local_disk_segment->enable_offloading = enable_offloading; if (enable_offloading) { - result.reserve( - local_disk_segment_it->second->offloading_objects.size()); + result.reserve(local_disk_segment->offloading_objects.size()); for (const auto& [_, task] : - local_disk_segment_it->second->offloading_objects) { + local_disk_segment->offloading_objects) { result.push_back(task); } - local_disk_segment_it->second->offloading_objects.clear(); + local_disk_segment->offloading_objects.clear(); return result; } // Offloading is disabled: clear the pending queue to prevent @@ -6338,7 +6173,7 @@ auto MasterService::OffloadObjectHeartbeat(const UUID& client_id, // must release offloading_mutex_ before taking shard locks via // MetadataAccessorRW. offloading_objects_copy = - std::move(local_disk_segment_it->second->offloading_objects); + std::move(local_disk_segment->offloading_objects); } for (auto& [_, task] : offloading_objects_copy) { @@ -6350,29 +6185,95 @@ auto MasterService::OffloadObjectHeartbeat(const UUID& client_id, auto task_it = tenant_state.offloading_tasks.find(object_id.user_key); if (task_it != tenant_state.offloading_tasks.end()) { - auto& tasks = task_it->second; - auto offload_it = - std::find_if(tasks.begin(), tasks.end(), - [&client_id](const OffloadingTask& t) { - return t.source_client_id == client_id; - }); - if (offload_it != tasks.end()) { - auto source = - accessor.Get().GetReplicaByID(offload_it->source_id); - if (source) { - source->dec_refcnt(); - } - tasks.erase(offload_it); - if (tasks.empty()) { - tenant_state.offloading_tasks.erase(task_it); - } + auto source = + accessor.Get().GetReplicaByID(task_it->second.source_id); + if (source) { + source->dec_refcnt(); } + tenant_state.offloading_tasks.erase(task_it); } } } return result; } +auto MasterService::FetchLocalDeleteTasks( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch, uint32_t limit) + -> tl::expected, ErrorCode> { + return local_delete_registry_.Fetch( + client_id, local_disk_segment_id, mount_epoch, + std::min(limit, kMaxLocalDeleteTasksPerBatch)); +} + +auto MasterService::AckLocalDeleteTasks( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch, const std::vector& task_ids) + -> tl::expected { + auto validation = local_delete_registry_.ValidateMount( + client_id, local_disk_segment_id, mount_epoch); + if (!validation) { + return validation; + } + if (task_ids.empty()) { + return {}; + } + if (task_ids.size() > kMaxLocalDeleteTasksPerBatch) { + return tl::unexpected(ErrorCode::INVALID_PARAMS); + } + + if (enable_ha_ && enable_oplog_) { + LocalDeleteAckPayloadV1 payload{ + .schema_version = 1, + .local_disk_segment_id = local_disk_segment_id, + .task_ids = task_ids, + }; + const auto payload_bytes = struct_pack::serialize(payload); + auto append = AppendOpLogWithDurableFinalize( + OpType::LOCAL_DELETE_ACK, TenantId::Default().value(), + local_disk_segment_id, + std::string(payload_bytes.begin(), payload_bytes.end()), + [this, local_disk_segment_id, task_ids](const OpLogEntry&) { + std::shared_lock snapshot_lock( + snapshot_mutex_); + local_delete_registry_.Erase(local_disk_segment_id, task_ids); + }); + if (!append) { + return tl::unexpected(append.error()); + } + return {}; + } + + local_delete_registry_.Erase(local_disk_segment_id, task_ids); + return {}; +} + +auto MasterService::ReconcileLocalDiskObjects( + const UUID& client_id, const std::string& local_disk_segment_id, + uint64_t mount_epoch, const std::vector& objects) + -> tl::expected, ErrorCode> { + auto validation = local_delete_registry_.ValidateMount( + client_id, local_disk_segment_id, mount_epoch); + if (!validation) { + return tl::unexpected(validation.error()); + } + + std::shared_lock snapshot_lock(snapshot_mutex_); + std::vector keep(objects.size(), 0); + for (size_t i = 0; i < objects.size(); ++i) { + const auto& object = objects[i]; + const auto object_id = + MakeObjectIdentity(object.key, TenantId(object.tenant_id)); + MetadataAccessorRO accessor(this, object_id); + if (!accessor.Exists() || accessor.Get().object_incarnation != + object.GetObjectIncarnation()) { + continue; + } + keep[i] = 1; + } + return keep; +} + auto MasterService::PollRemoveAll(const UUID& client_id) -> tl::expected { std::shared_lock shared_lock(snapshot_mutex_); @@ -6393,64 +6294,6 @@ auto MasterService::PollRemoveAll(const UUID& client_id) return result; } -auto MasterService::RemoveObjectHeartbeat(const UUID& client_id) - -> tl::expected, ErrorCode> { - std::shared_lock shared_lock(snapshot_mutex_); - ScopedLocalDiskSegmentAccess local_disk_segment_access = - segment_manager_.getLocalDiskSegmentAccess(); - auto& client_local_disk_segment = - local_disk_segment_access.getClientLocalDiskSegment(); - auto local_disk_segment_it = client_local_disk_segment.find(client_id); - if (local_disk_segment_it == client_local_disk_segment.end()) { - return tl::make_unexpected(ErrorCode::SEGMENT_NOT_FOUND); - } - { - MutexLocker locker(&local_disk_segment_it->second->offloading_mutex_); - return local_disk_segment_it->second->removed_keys; - } -} - -void MasterService::EnqueueRemoveTasks( - const std::vector& holder_ids, const RemoveTaskItem& task) { - if (holder_ids.empty()) return; - ScopedLocalDiskSegmentAccess access = - segment_manager_.getLocalDiskSegmentAccess(); - auto& segments = access.getClientLocalDiskSegment(); - for (const auto& holder_id : holder_ids) { - auto it = segments.find(holder_id); - if (it == segments.end()) continue; - MutexLocker locker(&it->second->offloading_mutex_); - if (std::find(it->second->removed_keys.begin(), - it->second->removed_keys.end(), task) == - it->second->removed_keys.end()) { - it->second->removed_keys.push_back(task); - } - } -} - -auto MasterService::AckRemoveObjectHeartbeat( - const UUID& client_id, const std::vector& tasks) - -> tl::expected { - std::shared_lock shared_lock(snapshot_mutex_); - ScopedLocalDiskSegmentAccess access = - segment_manager_.getLocalDiskSegmentAccess(); - auto& segments = access.getClientLocalDiskSegment(); - auto it = segments.find(client_id); - if (it == segments.end()) { - return tl::make_unexpected(ErrorCode::SEGMENT_NOT_FOUND); - } - MutexLocker locker(&it->second->offloading_mutex_); - auto& pending = it->second->removed_keys; - pending.erase(std::remove_if(pending.begin(), pending.end(), - [&tasks](const RemoveTaskItem& task) { - return std::find(tasks.begin(), - tasks.end(), task) != - tasks.end(); - }), - pending.end()); - return {}; -} - auto MasterService::ReportSsdCapacity(const UUID& client_id, int64_t ssd_total_capacity_bytes) -> tl::expected { @@ -6494,6 +6337,9 @@ auto MasterService::NotifyOffloadSuccess( return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } std::shared_ptr local_disk_segment; + std::string local_disk_segment_id; + uint64_t mount_epoch = 0; + uint32_t capabilities = 0; { ScopedLocalDiskSegmentAccess ssd_access = segment_manager_.getLocalDiskSegmentAccess(); @@ -6501,6 +6347,17 @@ auto MasterService::NotifyOffloadSuccess( auto disk_it = client_segments.find(client_id); if (disk_it != client_segments.end()) { local_disk_segment = disk_it->second; + MutexLocker locker(&local_disk_segment->offloading_mutex_); + local_disk_segment_id = local_disk_segment->local_disk_segment_id; + mount_epoch = local_disk_segment->mount_epoch; + capabilities = local_disk_segment->capabilities; + } + } + if ((capabilities & kLocalDiskCapabilityObjectTombstoneV1) != 0) { + auto mount_validation = local_delete_registry_.ValidateMount( + client_id, local_disk_segment_id, mount_epoch); + if (!mount_validation) { + return tl::unexpected(mount_validation.error()); } } @@ -6523,30 +6380,21 @@ auto MasterService::NotifyOffloadSuccess( auto task_it = tenant_state.offloading_tasks.find( request_object_id.user_key); if (task_it != tenant_state.offloading_tasks.end()) { - auto& tasks = task_it->second; - auto offload_it = - std::find_if(tasks.begin(), tasks.end(), - [&client_id](const OffloadingTask& t) { - return t.source_client_id == client_id; - }); - if (offload_it != tasks.end()) { - auto source = - accessor.Get().GetReplicaByID(offload_it->source_id); - if (source != nullptr) { - source->dec_refcnt(); - } - tasks.erase(offload_it); - if (tasks.empty()) { - tenant_state.offloading_tasks.erase(task_it); - } + auto source = accessor.Get().GetReplicaByID( + task_it->second.source_id); + if (source != nullptr) { + source->dec_refcnt(); } + tenant_state.offloading_tasks.erase(task_it); } } continue; } Replica replica(client_id, metadata.data_size, - metadata.transport_endpoint, ReplicaStatus::COMPLETE); + metadata.transport_endpoint, ReplicaStatus::COMPLETE, + local_disk_segment_id, mount_epoch, capabilities, + metadata.bucket_id, task.GetObjectIncarnation()); bool handled_existing_object = false; bool added_new_local_disk_replica = false; { @@ -6555,6 +6403,12 @@ auto MasterService::NotifyOffloadSuccess( if (accessor.Exists()) { auto& obj_metadata = accessor.Get(); auto& tenant_state = accessor.GetTenantState(); + if ((capabilities & kLocalDiskCapabilityObjectTombstoneV1) != + 0 && + task.GetObjectIncarnation() != + obj_metadata.object_incarnation) { + return tl::make_unexpected(ErrorCode::INVALID_VERSION); + } auto task_it = tenant_state.offloading_tasks.find( request_object_id.user_key); if (task_it != tenant_state.offloading_tasks.end() && @@ -6564,52 +6418,19 @@ auto MasterService::NotifyOffloadSuccess( return tl::make_unexpected(ErrorCode::INVALID_PARAMS); } - // Clean up the offloading task if present. + // Existing orphan objects can only bypass tenant registration + // for a master-admitted offload completion. Without this task + // marker, fall through to the regular registration check. if (task_it != tenant_state.offloading_tasks.end()) { - auto& tasks = task_it->second; - auto offload_it = - std::find_if(tasks.begin(), tasks.end(), - [&client_id](const OffloadingTask& t) { - return t.source_client_id == client_id; - }); - if (offload_it != tasks.end()) { - auto source = - obj_metadata.GetReplicaByID(offload_it->source_id); - if (source != nullptr) { - source->dec_refcnt(); - } - tasks.erase(offload_it); - if (tasks.empty()) { - tenant_state.offloading_tasks.erase(task_it); - } + auto source = + obj_metadata.GetReplicaByID(task_it->second.source_id); + if (source != nullptr) { + source->dec_refcnt(); } - } + tenant_state.offloading_tasks.erase(task_it); - // Register / update the LOCAL_DISK replica for this - // object. Handles both the offload-completion case and - // the remount / re-registration case. - if (!obj_metadata.HasReplica( - &Replica::fn_is_local_disk_replica)) { - std::vector replicas; - replicas.emplace_back(std::move(replica)); - obj_metadata.AddReplicas(std::move(replicas)); - auto& shard = accessor.GetShard(); - shard.OnDiskReplicaAdded(obj_metadata); - SyncCacheTotalAccounting(obj_metadata); - added_new_local_disk_replica = true; - } else { - size_t updated = obj_metadata.VisitReplicas( - [client_id](const Replica& rep) { - return rep.type() == ReplicaType::LOCAL_DISK && - rep.get_local_disk_client_id() == - client_id; - }, - [&metadata](Replica& rep) { - rep.update_local_disk_location( - metadata.transport_endpoint, - metadata.data_size); - }); - if (updated == 0) { + if (!obj_metadata.HasReplica( + &Replica::fn_is_local_disk_replica)) { std::vector replicas; replicas.emplace_back(std::move(replica)); obj_metadata.AddReplicas(std::move(replicas)); @@ -6617,9 +6438,30 @@ auto MasterService::NotifyOffloadSuccess( shard.OnDiskReplicaAdded(obj_metadata); SyncCacheTotalAccounting(obj_metadata); added_new_local_disk_replica = true; + } else { + obj_metadata.VisitReplicas( + [client_id](const Replica& rep) { + return rep.type() == ReplicaType::LOCAL_DISK && + rep.get_descriptor() + .get_local_disk_descriptor() + .client_id == client_id; + }, + [&replica](Replica& rep) { + const auto descriptor = + replica.get_descriptor() + .get_local_disk_descriptor(); + rep.update_local_disk_location( + descriptor.object_size, + descriptor.transport_endpoint, + descriptor.GetLocalDiskSegmentId(), + descriptor.GetMountEpoch(), + descriptor.GetCapabilities(), + descriptor.GetBucketId(), + descriptor.GetObjectIncarnation()); + }); } + handled_existing_object = true; } - handled_existing_object = true; } } @@ -6639,12 +6481,11 @@ auto MasterService::NotifyOffloadSuccess( if (res.error() == ErrorCode::OBJECT_NOT_FOUND) { continue; } - LOG(WARNING) << "Failed to add replica, skipping object: " - << "error=" << res.error() - << ", client_id=" << client_id - << ", tenant_id=" << object_id.tenant_id.value() - << ", key=" << object_id.user_key; - continue; + LOG(ERROR) << "Failed to add replica: error=" << res.error() + << ", client_id=" << client_id + << ", tenant_id=" << object_id.tenant_id.value() + << ", key=" << object_id.user_key; + return tl::make_unexpected(res.error()); } added_new_local_disk_replica = res.value(); } @@ -6658,14 +6499,13 @@ auto MasterService::NotifyOffloadSuccess( return {}; } -tl::expected, ErrorCode> MasterService::PushOffloadingQueue( - const ObjectIdentity& object_id, Replica& replica) { +tl::expected MasterService::PushOffloadingQueue( + const ObjectIdentity& object_id, Replica& replica, + ObjectIncarnation object_incarnation) { const auto& segment_names = replica.get_segment_names(); if (segment_names.empty()) { return {}; } - std::vector queued_clients; - queued_clients.reserve(segment_names.size()); for (const auto& segment_name_it : segment_names) { if (!segment_name_it.has_value()) { continue; @@ -6700,13 +6540,13 @@ tl::expected, ErrorCode> MasterService::PushOffloadingQueue( object_id.tenant_id.MakeScopedKey(object_id.user_key), OffloadTaskItem{.tenant_id = object_id.tenant_id.value(), .key = object_id.user_key, - .size = size}); + .size = size, + .object_incarnation = object_incarnation}); if (!res.second) { return tl::make_unexpected(ErrorCode::OBJECT_ALREADY_EXISTS); } - queued_clients.push_back(client_id_it->second); } - return queued_clients; + return {}; } // Promotion-on-hit @@ -7382,9 +7222,7 @@ auto MasterService::NotifyPromotionSuccess(const UUID& client_id, }); const auto payload = - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post, metadata.group_id, - metadata.data_type); + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, post); if (batch_reservation) { auto persist_result = AppendReservedOpLogWithDurableFinalize( std::move(*batch_reservation), OpType::PUT_END, @@ -7560,9 +7398,6 @@ void MasterService::EvictionThreadFunc() { // Try discarding expired processing keys and ongoing replication // tasks if we have not done this for a long time. { - SpDiag::PerfPoint pt_discard(PerfKey::MASTER_BG_DISCARD_EXPIRED, - SpDiag::PerfLevel::MODULE); - pt_discard.Start(); std::shared_lock shared_lock( snapshot_mutex_); for (size_t i = 0; i < kNumShards; i++) { @@ -7570,7 +7405,6 @@ void MasterService::EvictionThreadFunc() { DiscardExpiredProcessingReplicas(shard, now); } ReleaseExpiredDiscardedReplicas(now); - pt_discard.End(0); } last_discard_time = now; } @@ -7663,9 +7497,7 @@ void MasterService::DiscardExpiredProcessingReplicas( persist_result = AppendOpLogWithDurableFinalize( OpType::PUT_END, tenant_it->first.value(), *key_it, SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, - post_descriptors, metadata.group_id, - metadata.data_type), + metadata, post_descriptors), enable_oplog_ ? [this, ttl](const OpLogEntry& durable_entry) { FinalizeExpiredProcessingReplicasAfterDurable( @@ -7765,8 +7597,7 @@ void MasterService::DiscardExpiredProcessingReplicas( OpType::PUT_END, tenant_it->first.value(), task_it->first, SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, post_descriptors, - metadata.group_id, metadata.data_type), + metadata, post_descriptors), enable_oplog_ ? [this, source_id, target_ids = std::move(target_ids), @@ -7815,31 +7646,23 @@ void MasterService::DiscardExpiredProcessingReplicas( for (auto task_it = tenant_state.offloading_tasks.begin(); task_it != tenant_state.offloading_tasks.end();) { - auto& tasks = task_it->second; + const auto ttl = + task_it->second.start_time + put_start_release_timeout_sec_; + if (ttl > now) { + task_it++; + continue; + } auto metadata_it = tenant_state.metadata.find(task_it->first); - for (auto t = tasks.begin(); t != tasks.end();) { - const auto ttl = - t->start_time + put_start_release_timeout_sec_; - if (ttl > now) { - t++; - continue; - } - if (metadata_it != tenant_state.metadata.end()) { - auto source = metadata_it->second.GetReplicaByID( - t->source_id); - if (source != nullptr) { - source->dec_refcnt(); - } + if (metadata_it != tenant_state.metadata.end()) { + auto source = metadata_it->second.GetReplicaByID( + task_it->second.source_id); + if (source != nullptr) { + source->dec_refcnt(); } - LOG(WARNING) << "Offloading task expired for key: " - << task_it->first << " tenant=" << tenant_it->first; - t = tasks.erase(t); - } - if (tasks.empty()) { - task_it = tenant_state.offloading_tasks.erase(task_it); - } else { - task_it++; } + LOG(WARNING) << "Offloading task expired for key: " + << task_it->first << " tenant=" << tenant_it->first; + task_it = tenant_state.offloading_tasks.erase(task_it); } for (auto task_it = tenant_state.promotion_tasks.begin(); @@ -8163,8 +7986,9 @@ MasterService::EvictTenantMemoryForQuota(const TenantId& tenant_id, auto now = std::chrono::system_clock::now(); std::shared_lock shared_lock(snapshot_mutex_); - auto is_evictable_memory_replica = [this](const Replica& replica) { - return IsMemoryReplicaEvictable(replica); + auto is_evictable_memory_replica = [](const Replica& replica) { + return replica.is_memory_replica() && replica.is_completed() && + replica.get_refcnt() == 0; }; auto can_evict_replicas = [&](const ObjectMetadata& metadata) { return metadata.HasReplica(is_evictable_memory_replica); @@ -8219,20 +8043,18 @@ MasterService::EvictTenantMemoryForQuota(const TenantId& tenant_id, bool queued = false; metadata.VisitReplicas( is_evictable_memory_replica, - [this, &key, &normalized_tenant, &tenant_state, &queued, - &now](Replica& replica) { + [this, &key, &normalized_tenant, &tenant_state, &queued, &now, + &metadata](Replica& replica) { if (queued) { return; } auto result = PushOffloadingQueue( - MakeObjectIdentity(key, normalized_tenant), replica); - if (result && !result.value().empty()) { - auto& tasks = tenant_state.offloading_tasks[key]; - for (const auto& client_id : result.value()) { - replica.inc_refcnt(); - tasks.push_back( - OffloadingTask{replica.id(), now, client_id}); - } + MakeObjectIdentity(key, normalized_tenant), replica, + metadata.object_incarnation); + if (result) { + replica.inc_refcnt(); + tenant_state.offloading_tasks.emplace( + key, OffloadingTask{replica.id(), now}); queued = true; } }); @@ -8387,10 +8209,6 @@ MasterService::EvictTenantMemoryForQuota(const TenantId& tenant_id, void MasterService::BatchEvict(double evict_ratio_target, double evict_ratio_lowerbound) { - SpDiag::PerfPoint pt_evict(PerfKey::MASTER_BG_BATCH_EVICT, - SpDiag::PerfLevel::KEY_MODULE); - pt_evict.Start(); - if (evict_ratio_target < evict_ratio_lowerbound) { LOG(ERROR) << "evict_ratio_target=" << evict_ratio_target << ", evict_ratio_lowerbound=" << evict_ratio_lowerbound @@ -8400,8 +8218,9 @@ void MasterService::BatchEvict(double evict_ratio_target, auto now = std::chrono::system_clock::now(); - auto is_evictable_memory_replica = [this](const Replica& replica) { - return IsMemoryReplicaEvictable(replica); + auto is_evictable_memory_replica = [](const Replica& replica) { + return replica.is_memory_replica() && replica.is_completed() && + replica.get_refcnt() == 0; }; auto can_evict_replicas = [&](const ObjectMetadata& metadata) { @@ -8478,18 +8297,20 @@ void MasterService::BatchEvict(double evict_ratio_target, // Queue one MEMORY replica for offload; others will be evicted below. bool queued = false; metadata.VisitReplicas( - is_evictable_memory_replica, [this, &tenant_id, &key, &tenant_state, - &queued, &now](Replica& replica) { + [](const Replica& r) { + return r.is_memory_replica() && r.is_completed() && + r.get_refcnt() == 0; + }, + [this, &tenant_id, &key, &tenant_state, &queued, &now, + &metadata](Replica& replica) { if (queued) return; // only need to pin one replica for offload - auto result = PushOffloadingQueue( - MakeObjectIdentity(key, tenant_id), replica); - if (result && !result.value().empty()) { - auto& tasks = tenant_state.offloading_tasks[key]; - for (const auto& client_id : result.value()) { - replica.inc_refcnt(); - tasks.push_back( - OffloadingTask{replica.id(), now, client_id}); - } + auto result = + PushOffloadingQueue(MakeObjectIdentity(key, tenant_id), + replica, metadata.object_incarnation); + if (result) { + replica.inc_refcnt(); + tenant_state.offloading_tasks.emplace( + key, OffloadingTask{replica.id(), now}); queued = true; } }); @@ -8514,26 +8335,26 @@ void MasterService::BatchEvict(double evict_ratio_target, return 0; }; - // kSubmitted means accepted by the ordered writer, not yet durable. - enum class EvictOpLogSubmissionResult { - kNotRequired, - kSubmitted, - kReservationFailed, - kSubmissionFailed, - }; - - // HA strong-consistency: submit the post-eviction state before the caller - // proceeds with eviction. - auto submit_evict_oplog_if_needed = + // HA strong-consistency: persist the post-eviction state BEFORE the + // helper mutates `metadata`. Returns true on success (or when HA is + // disabled). On false, the caller must NOT call try_evict_or_offload + // and must NOT erase the metadata entry — local state must stay in + // sync with what was published. + auto persist_evict_oplog_or_skip = [&, this](const TenantId& tenant_id, const std::string& key, - ObjectMetadata& metadata) -> EvictOpLogSubmissionResult { + ObjectMetadata& metadata) -> bool { if (!enable_oplog_ || !ordered_oplog_writer_) { - return EvictOpLogSubmissionResult::kNotRequired; + return true; } - // Predict the descriptor list after evict_replicas() runs. - auto remaining = BuildRemainingReplicaDescriptors( - metadata, is_evictable_memory_replica); + // Predict the descriptor list after evict_replicas() runs: + // drop COMPLETE memory replicas with refcnt==0; keep everything else + // that is COMPLETE. + auto remaining = + BuildRemainingReplicaDescriptors(metadata, [](const Replica& r) { + return r.is_memory_replica() && r.is_completed() && + r.get_refcnt() == 0; + }); if (enable_oplog_) { auto reservation = ReserveBatchOpLogSlot(); @@ -8541,8 +8362,8 @@ void MasterService::BatchEvict(double evict_ratio_target, LOG(WARNING) << "BatchEvict: OpLog reservation failed for key=" << key << ", err=" << static_cast(reservation.error()) - << ", stopping eviction cycle"; - return EvictOpLogSubmissionResult::kReservationFailed; + << ", skipping eviction"; + return false; } std::vector removed_ids; metadata.VisitReplicas(is_evictable_memory_replica, @@ -8550,9 +8371,9 @@ void MasterService::BatchEvict(double evict_ratio_target, removed_ids.push_back(replica.id()); replica.mark_removed(); }); - tl::expected submission_result; + tl::expected persist_result; if (remaining.empty()) { - submission_result = AppendReservedOpLogWithDurableFinalize( + persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::REMOVE, tenant_id.value(), key, {}, [this, removed_ids = std::move(removed_ids)]( @@ -8561,54 +8382,50 @@ void MasterService::BatchEvict(double evict_ratio_target, durable_entry, removed_ids, QuotaEraseMode::kFull); }); } else { - submission_result = AppendReservedOpLogWithDurableFinalize( + persist_result = AppendReservedOpLogWithDurableFinalize( std::move(reservation.value()), OpType::PUT_END, tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, + remaining), [this, removed_ids = std::move(removed_ids)]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( durable_entry, removed_ids, QuotaEraseMode::kFull); }); } - if (!submission_result) { + if (!persist_result) { LOG(WARNING) - << "BatchEvict: OpLog submission failed for key=" << key - << ", err=" << static_cast(submission_result.error()) - << ", skipping object eviction"; - return EvictOpLogSubmissionResult::kSubmissionFailed; + << "BatchEvict: OpLog persist failed for key=" << key + << ", err=" << static_cast(persist_result.error()) + << ", skipping eviction"; + return false; } - return EvictOpLogSubmissionResult::kSubmitted; + return true; } - tl::expected submission_result; + tl::expected persist_result; if (remaining.empty()) { - submission_result = AppendOpLogWithDurableFinalize( + persist_result = AppendOpLogWithDurableFinalize( OpType::REMOVE, tenant_id.value(), key, {}, nullptr); } else { - submission_result = AppendOpLogWithDurableFinalize( + persist_result = AppendOpLogWithDurableFinalize( OpType::PUT_END, tenant_id.value(), key, - SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + SerializeMetadataForOpLogFromReplicaDescriptors(metadata, + remaining), nullptr); } - if (!submission_result) { - LOG(WARNING) << "BatchEvict: OpLog submission failed for key=" - << key << ", err=" - << static_cast(submission_result.error()) - << ", skipping object eviction"; - return EvictOpLogSubmissionResult::kSubmissionFailed; + if (!persist_result) { + LOG(WARNING) << "BatchEvict: OpLog persist failed for key=" << key + << ", err=" << static_cast(persist_result.error()) + << ", skipping eviction"; + return false; } - return EvictOpLogSubmissionResult::kSubmitted; + return true; }; struct EvictionResult { uint64_t freed_bytes{0}; long evicted_objects{0}; - bool stop_cycle{false}; }; auto try_evict_group_or_object = @@ -8618,14 +8435,7 @@ void MasterService::BatchEvict(double evict_ratio_target, std::vector>& deferred_replicas, bool allow_soft_pinned) -> EvictionResult { if (!metadata.IsGrouped()) { - auto submission_result = - submit_evict_oplog_if_needed(tenant_id, key, metadata); - if (submission_result == - EvictOpLogSubmissionResult::kReservationFailed) { - return {.stop_cycle = true}; - } - if (submission_result == - EvictOpLogSubmissionResult::kSubmissionFailed) { + if (!persist_evict_oplog_or_skip(tenant_id, key, metadata)) { return {}; } uint64_t freed = try_evict_or_offload( @@ -8635,14 +8445,7 @@ void MasterService::BatchEvict(double evict_ratio_target, auto group_it = tenant_state.group_members.find(metadata.group_id); if (group_it == tenant_state.group_members.end()) { - auto submission_result = - submit_evict_oplog_if_needed(tenant_id, key, metadata); - if (submission_result == - EvictOpLogSubmissionResult::kReservationFailed) { - return {.stop_cycle = true}; - } - if (submission_result == - EvictOpLogSubmissionResult::kSubmissionFailed) { + if (!persist_evict_oplog_or_skip(tenant_id, key, metadata)) { return {}; } uint64_t freed = try_evict_or_offload( @@ -8674,15 +8477,8 @@ void MasterService::BatchEvict(double evict_ratio_target, continue; } - auto submission_result = submit_evict_oplog_if_needed( - tenant_id, member_key, member_metadata); - if (submission_result == - EvictOpLogSubmissionResult::kReservationFailed) { - result.stop_cycle = true; - break; - } - if (submission_result == - EvictOpLogSubmissionResult::kSubmissionFailed) { + if (!persist_evict_oplog_or_skip(tenant_id, member_key, + member_metadata)) { continue; } uint64_t freed = @@ -8719,13 +8515,31 @@ void MasterService::BatchEvict(double evict_ratio_target, size_t start_idx = randomIndex(kNumShards); std::shared_lock shared_lock(snapshot_mutex_); - // ===== Phase 1: Parallel candidate collection ===== - // N threads each scan a batch of shards, collecting Candidates with - // shard_idx + tenant_id + key for safe re-lookup in Phase 2. + // ===== Phase 1: Parallel candidate census ===== + // N threads each scan a batch of shards. For selective ratios only the + // lease timestamps are collected here; full tenant/key identities are + // materialized afterwards for a bounded frontier around the eviction + // cutoff. High ratios collect full Candidates directly, because a census + // followed by a second scan would cost more than the identities it saves. int num_threads = std::min((int)kNumShards, 16); size_t shards_per_thread = (kNumShards + num_threads - 1) / num_threads; + constexpr size_t kMinReserveSlack = 1024; + constexpr size_t kMinFrontierLimit = 64 * 1024; + constexpr size_t kReserveSlackDivisor = 10; + constexpr size_t kFrontierDivisor = 4; + // Above this target ratio the reserve frontier would already cover a + // large share of the population, so selective materialization stops + // paying for the extra scan it costs. + constexpr double kCompactPrebypassTargetRatio = + static_cast(kReserveSlackDivisor) / + static_cast(kFrontierDivisor * (kReserveSlackDivisor + 1)); + const bool compact_frontier_prebypass = + evict_ratio_target >= kCompactPrebypassTargetRatio; + std::vector> local_candidates(num_threads); + std::vector> + local_no_pin(num_threads); std::vector local_eviction_base(num_threads, 0); std::vector local_object_count(num_threads, 0); std::vector> @@ -8752,9 +8566,14 @@ void MasterService::BatchEvict(double evict_ratio_target, if (!it->second.IsLeaseExpired(now) || !has_evictable) continue; if (!it->second.IsSoftPinned(now)) { - local_candidates[t].push_back( - {s, tenant_id, it->first, - it->second.lease_timeout}); + if (compact_frontier_prebypass) { + local_candidates[t].push_back( + {s, tenant_id, it->first, + it->second.lease_timeout}); + } else { + local_no_pin[t].push_back( + it->second.lease_timeout); + } } else if (allow_evict_soft_pinned_objects_) { local_soft_pin[t].push_back( it->second.lease_timeout); @@ -8776,14 +8595,27 @@ void MasterService::BatchEvict(double evict_ratio_target, for (auto v : local_object_count) object_count += v; std::vector candidates; - { + if (compact_frontier_prebypass) { size_t total = 0; for (auto& v : local_candidates) total += v.size(); candidates.reserve(total); + for (auto& v : local_candidates) { + candidates.insert(candidates.end(), + std::make_move_iterator(v.begin()), + std::make_move_iterator(v.end())); + } } - for (auto& v : local_candidates) { - candidates.insert(candidates.end(), std::make_move_iterator(v.begin()), - std::make_move_iterator(v.end())); + + std::vector no_pin_timeouts; + { + size_t total = 0; + for (auto& v : local_no_pin) total += v.size(); + no_pin_timeouts.reserve(total); + } + for (auto& v : local_no_pin) { + no_pin_timeouts.insert(no_pin_timeouts.end(), + std::make_move_iterator(v.begin()), + std::make_move_iterator(v.end())); } std::vector soft_pin_objects; @@ -8805,17 +8637,116 @@ void MasterService::BatchEvict(double evict_ratio_target, return; } + const long ideal_evict_num = + std::ceil(total_eviction_base * evict_ratio_target); + const size_t no_pin_count = + compact_frontier_prebypass ? candidates.size() : no_pin_timeouts.size(); + const long primary_no_pin_num = + std::min(ideal_evict_num, static_cast(no_pin_count)); + + // Re-scan metadata and copy full identities only for objects inside the + // requested timestamp range. The eligibility conditions are identical to + // the census above, so the selected set matches what the census counted. + auto collect_candidates = [&](bool use_cutoff, + std::chrono::system_clock::time_point cutoff, + bool collect_older_or_equal) { + std::vector> local_frontier(num_threads); + std::vector collectors; + collectors.reserve(num_threads); + + for (int t = 0; t < num_threads; t++) { + collectors.emplace_back([&, t] { + size_t s_start = t * shards_per_thread; + size_t s_end = + std::min(s_start + shards_per_thread, kNumShards); + for (size_t s = s_start; s < s_end; s++) { + MetadataShardAccessorRW shard(this, s); + for (const auto& [tenant_id, tenant_state] : + shard->tenants) { + for (const auto& [key, metadata] : + tenant_state.metadata) { + if (metadata.IsHardPinned() || + !metadata.IsLeaseExpired(now) || + metadata.IsSoftPinned(now) || + !can_evict_replicas(metadata)) { + continue; + } + if (use_cutoff) { + const bool in_range = + collect_older_or_equal + ? metadata.lease_timeout <= cutoff + : metadata.lease_timeout > cutoff; + if (!in_range) continue; + } + local_frontier[t].push_back( + {s, tenant_id, key, metadata.lease_timeout}); + } + } + } + }); + } + for (auto& collector : collectors) collector.join(); + + size_t total = 0; + for (const auto& v : local_frontier) total += v.size(); + std::vector merged; + merged.reserve(total); + for (auto& v : local_frontier) { + merged.insert(merged.end(), std::make_move_iterator(v.begin()), + std::make_move_iterator(v.end())); + } + return merged; + }; + + bool compact_frontier_used = false; + std::chrono::system_clock::time_point reserve_cutoff{}; + + if (primary_no_pin_num > 0 && !compact_frontier_prebypass) { + const size_t primary_count = static_cast(primary_no_pin_num); + // The reserve absorbs objects that stop being evictable between the + // census and the eviction pass, so ordinary churn does not require a + // second materialization pass. + const size_t reserve_slack = std::max( + kMinReserveSlack, + (primary_count + kReserveSlackDivisor - 1) / kReserveSlackDivisor); + const size_t reserve_count = + std::min(no_pin_count, primary_count + reserve_slack); + const size_t frontier_limit = + std::max(kMinFrontierLimit, + (no_pin_count + kFrontierDivisor - 1) / kFrontierDivisor); + + if (reserve_count <= frontier_limit) { + std::nth_element(no_pin_timeouts.begin(), + no_pin_timeouts.begin() + (reserve_count - 1), + no_pin_timeouts.end()); + reserve_cutoff = no_pin_timeouts[reserve_count - 1]; + candidates = collect_candidates(/*use_cutoff=*/true, reserve_cutoff, + /*collect_older_or_equal=*/true); + + // Shortfall guard: if churn left the frontier holding fewer + // objects than the target needs, fall back to the full candidate + // set so evict_num below still derives from the requested target + // rather than from a shrunken frontier. + if (candidates.size() >= primary_count) { + compact_frontier_used = true; + } else { + candidates = + collect_candidates(/*use_cutoff=*/false, {}, + /*collect_older_or_equal=*/true); + } + } else { + candidates = collect_candidates(/*use_cutoff=*/false, {}, + /*collect_older_or_equal=*/true); + } + } // ===== Phase 2: Serial eviction via key lookup ===== long evicted_count = 0; uint64_t total_freed_size = 0; std::vector no_pin_objects; std::vector> deferred_replicas; - bool stop_eviction = false; // First pass: evict candidates with no soft pin if (!candidates.empty()) { - long ideal_evict_num = - std::ceil(total_eviction_base * evict_ratio_target); long evict_num = std::min(ideal_evict_num, (long)candidates.size()); std::nth_element(candidates.begin(), @@ -8829,55 +8760,76 @@ void MasterService::BatchEvict(double evict_ratio_target, // continue trying the next one so actual evicted count reaches // evict_num. This matches the old per-shard over-eviction behavior. long evicted_this_pass = 0; - for (auto& c : candidates) { - if (stop_eviction) break; - if (evicted_this_pass >= evict_num && - c.lease_timeout > target_timeout) { - no_pin_objects.push_back(c.lease_timeout); - continue; - } - { - MetadataShardAccessorRW shard(this, c.shard_idx); - auto tenant_it = shard->tenants.find(c.tenant_id); - if (tenant_it == shard->tenants.end()) continue; - auto& tenant_state = tenant_it->second; - auto it = tenant_state.metadata.find(c.key); - if (it == tenant_state.metadata.end()) continue; - // Re-validate: state may have changed since Phase 1 - if (!it->second.IsLeaseExpired(now) || - it->second.IsSoftPinned(now) || - !can_evict_replicas(it->second)) { + auto evict_candidate_batch = [&](std::vector& batch) { + for (auto& c : batch) { + if (evicted_this_pass >= evict_num && + c.lease_timeout > target_timeout) { no_pin_objects.push_back(c.lease_timeout); continue; } - auto evict_result = try_evict_group_or_object( - c.tenant_id, c.key, it->second, shard, tenant_state, - deferred_replicas, - /*allow_soft_pinned=*/false); - stop_eviction = evict_result.stop_cycle; - total_freed_size += evict_result.freed_bytes; - if (!enable_oplog_ && !it->second.IsGrouped()) { - PublishKvRemovedAfterEvict(c.key, evict_result.freed_bytes, - "cpu", it->second, c.tenant_id); - } - if (!enable_oplog_ && !it->second.IsValid()) { - EraseMetadata(tenant_state, it, c.tenant_id, - QuotaEraseMode::kFull, &shard); - } - if (tenant_state.Empty()) { - shard->tenants.erase(tenant_it); + { + MetadataShardAccessorRW shard(this, c.shard_idx); + auto tenant_it = shard->tenants.find(c.tenant_id); + if (tenant_it == shard->tenants.end()) continue; + auto& tenant_state = tenant_it->second; + auto it = tenant_state.metadata.find(c.key); + if (it == tenant_state.metadata.end()) continue; + + // Re-validate: state may have changed since Phase 1 + if (!it->second.IsLeaseExpired(now) || + it->second.IsSoftPinned(now) || + !can_evict_replicas(it->second)) { + no_pin_objects.push_back(c.lease_timeout); + continue; + } + + auto evict_result = try_evict_group_or_object( + c.tenant_id, c.key, it->second, shard, tenant_state, + deferred_replicas, + /*allow_soft_pinned=*/false); + + total_freed_size += evict_result.freed_bytes; + + if (!enable_oplog_ && !it->second.IsGrouped()) { + PublishKvRemovedAfterEvict( + c.key, evict_result.freed_bytes, "cpu", it->second, + c.tenant_id); + } + + if (!enable_oplog_ && !it->second.IsValid()) { + EraseMetadata(tenant_state, it, c.tenant_id, + QuotaEraseMode::kFull, &shard); + } + + if (tenant_state.Empty()) { + shard->tenants.erase(tenant_it); + } + + evicted_count += evict_result.evicted_objects; + evicted_this_pass += evict_result.evicted_objects; } - evicted_count += evict_result.evicted_objects; - evicted_this_pass += evict_result.evicted_objects; + deferred_replicas.clear(); } - deferred_replicas.clear(); + }; + + evict_candidate_batch(candidates); + + // Metadata may change after the frontier is materialized. If the + // reserve is exhausted before the target is met, refill from the + // remainder of the current no-soft-pin population. This recovery + // scan is paid only on churn and preserves the behavior of + // continuing past the cutoff until evict_num is reached. + if (compact_frontier_used && evicted_this_pass < evict_num) { + auto refill_candidates = collect_candidates( + /*use_cutoff=*/true, reserve_cutoff, + /*collect_older_or_equal=*/false); + evict_candidate_batch(refill_candidates); } } // Try releasing discarded replicas before we decide whether to do the // second pass. - uint64_t released_discarded_cnt = - stop_eviction ? 0 : ReleaseExpiredDiscardedReplicas(now); + uint64_t released_discarded_cnt = ReleaseExpiredDiscardedReplicas(now); // The ideal number of objects to evict in the second pass long target_evict_num = @@ -8891,7 +8843,7 @@ void MasterService::BatchEvict(double evict_ratio_target, // Do second pass eviction only if 1). there are candidates that can be // evicted AND 2). The evicted number in the first pass is less than // evict_ratio_lowerbound. - if (!stop_eviction && target_evict_num > 0) { + if (target_evict_num > 0) { if (target_evict_num <= static_cast(no_pin_objects.size())) { // Second pass A: only evict objects without soft pin. std::nth_element(no_pin_objects.begin(), @@ -8900,19 +8852,17 @@ void MasterService::BatchEvict(double evict_ratio_target, auto target_timeout = no_pin_objects[target_evict_num - 1]; // Evict via key lookup — avoid full metadata traversal - for (size_t i = 0; - i < kNumShards && target_evict_num > 0 && !stop_eviction; - i++) { + for (size_t i = 0; i < kNumShards && target_evict_num > 0; i++) { { MetadataShardAccessorRW shard(this, (start_idx + i) % kNumShards); for (auto tenant_it = shard->tenants.begin(); tenant_it != shard->tenants.end() && - target_evict_num > 0 && !stop_eviction;) { + target_evict_num > 0;) { auto& tenant_state = tenant_it->second; auto it = tenant_state.metadata.begin(); while (it != tenant_state.metadata.end() && - target_evict_num > 0 && !stop_eviction) { + target_evict_num > 0) { if (!it->second.IsHardPinned() && it->second.IsLeaseExpired(now) && it->second.lease_timeout <= target_timeout && @@ -8922,7 +8872,6 @@ void MasterService::BatchEvict(double evict_ratio_target, tenant_it->first, it->first, it->second, shard, tenant_state, deferred_replicas, /*allow_soft_pinned=*/false); - stop_eviction = evict_result.stop_cycle; total_freed_size += evict_result.freed_bytes; if (!enable_oplog_ && !it->second.IsGrouped()) { PublishKvRemovedAfterEvict( @@ -8939,7 +8888,6 @@ void MasterService::BatchEvict(double evict_ratio_target, evicted_count += evict_result.evicted_objects; target_evict_num -= evict_result.evicted_objects; - if (stop_eviction) break; } else { ++it; } @@ -8964,20 +8912,18 @@ void MasterService::BatchEvict(double evict_ratio_target, soft_pin_objects.end()); auto soft_target_timeout = soft_pin_objects[soft_pin_evict_num - 1]; - for (size_t i = 0; - i < kNumShards && target_evict_num > 0 && !stop_eviction; - i++) { + for (size_t i = 0; i < kNumShards && target_evict_num > 0; i++) { { MetadataShardAccessorRW shard(this, (start_idx + i) % kNumShards); for (auto tenant_it = shard->tenants.begin(); tenant_it != shard->tenants.end() && - target_evict_num > 0 && !stop_eviction;) { + target_evict_num > 0;) { auto& tenant_state = tenant_it->second; auto it = tenant_state.metadata.begin(); while (it != tenant_state.metadata.end() && - target_evict_num > 0 && !stop_eviction) { + target_evict_num > 0) { if (it->second.IsHardPinned() || !it->second.IsLeaseExpired(now) || !can_evict_replicas(it->second)) { @@ -8991,7 +8937,6 @@ void MasterService::BatchEvict(double evict_ratio_target, tenant_it->first, it->first, it->second, shard, tenant_state, deferred_replicas, /*allow_soft_pinned=*/true); - stop_eviction = evict_result.stop_cycle; total_freed_size += evict_result.freed_bytes; if (!enable_oplog_ && !it->second.IsGrouped()) { PublishKvRemovedAfterEvict( @@ -9008,7 +8953,6 @@ void MasterService::BatchEvict(double evict_ratio_target, evicted_count += evict_result.evicted_objects; target_evict_num -= evict_result.evicted_objects; - if (stop_eviction) break; } else { ++it; } @@ -9035,19 +8979,7 @@ void MasterService::BatchEvict(double evict_ratio_target, } } - if (stop_eviction) { - // Reservation backpressure is transient; retry remaining work later. - need_mem_eviction_ = true; - if (evicted_count > 0) { - MasterMetricManager::instance().inc_eviction_success( - evicted_count, total_freed_size); - MasterMetricManager::instance().inc_mem_eviction_success( - evicted_count, total_freed_size); - } else { - MasterMetricManager::instance().inc_eviction_fail(); - MasterMetricManager::instance().inc_mem_eviction_fail(); - } - } else if (evicted_count > 0 || released_discarded_cnt > 0) { + if (evicted_count > 0 || released_discarded_cnt > 0) { need_mem_eviction_ = false; MasterMetricManager::instance().inc_eviction_success(evicted_count, total_freed_size); @@ -9111,20 +9043,14 @@ void MasterService::BatchEvict(double evict_ratio_target, << " object(s); force-evicted without disk offload " "(offload_force_evict=true)."; } - - pt_evict.End(0); } void MasterService::NoFBatchEvict(double evict_ratio_target, double evict_ratio_lowerbound) { - SpDiag::PerfPoint pt_nof_evict(PerfKey::MASTER_BG_NOF_BATCH_EVICT, - SpDiag::PerfLevel::KEY_MODULE); - pt_nof_evict.Start(); - if (evict_ratio_target < evict_ratio_lowerbound) { - MC_LOG(ERROR) << "nof_evict_ratio_target=" << evict_ratio_target - << ", nof_evict_ratio_lowerbound=" - << evict_ratio_lowerbound << ", error=invalid_params"; + LOG(ERROR) << "nof_evict_ratio_target=" << evict_ratio_target + << ", nof_evict_ratio_lowerbound=" << evict_ratio_lowerbound + << ", error=invalid_params"; evict_ratio_lowerbound = evict_ratio_target; } @@ -9220,9 +9146,7 @@ void MasterService::NoFBatchEvict(double evict_ratio_target, std::move(reservation.value()), OpType::PUT_END, tenant_it->first.value(), it->first, SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, - remaining, metadata.group_id, - metadata.data_type), + metadata, remaining), [this, removed_ids = std::move(removed_ids)]( const OpLogEntry& durable_entry) { FinalizeRemovedReplicasAfterDurable( @@ -9256,8 +9180,7 @@ void MasterService::NoFBatchEvict(double evict_ratio_target, OpType::PUT_END, tenant_it->first.value(), it->first, SerializeMetadataForOpLogFromReplicaDescriptors( - metadata.client_id, metadata.size, remaining, - metadata.group_id, metadata.data_type), + metadata, remaining), nullptr); } if (!persist_result) { @@ -9315,8 +9238,6 @@ void MasterService::NoFBatchEvict(double evict_ratio_target, VLOG(1) << "action=evict_nof_replicas" << ", evicted_count=" << evicted_count << ", total_freed_size=" << total_freed_size; - - pt_nof_evict.End(0); } void MasterService::ClientMonitorFunc() { @@ -9324,10 +9245,6 @@ void MasterService::ClientMonitorFunc() { boost::hash> client_ttl; while (client_monitor_running_) { - SpDiag::PerfPoint pt_monitor(PerfKey::MASTER_BG_CLIENT_MONITOR, - SpDiag::PerfLevel::MODULE); - pt_monitor.Start(); - auto now = std::chrono::steady_clock::now(); // Update the client ttl @@ -9353,9 +9270,6 @@ void MasterService::ClientMonitorFunc() { // Update the client status to NEED_REMOUNT if (!expired_clients.empty()) { - SpDiag::PerfPoint pt_unmount(PerfKey::MASTER_BG_CLIENT_UNMOUNT, - SpDiag::PerfLevel::MODULE); - pt_unmount.Start(); // Notify graceful unmount scheduler to drop pending records // for expired clients. The actual unmount is handled below. for (auto& cid : expired_clients) { @@ -9435,11 +9349,12 @@ void MasterService::ClientMonitorFunc() { segment_access.UnmountLocalDiskSegment(client_id); } } + for (const auto& client_id : expired_clients) { + local_delete_registry_.Unmount(client_id); + } RecomputeTenantEffectiveQuotas(); - pt_unmount.End(0); } - pt_monitor.End(0); std::this_thread::sleep_for( std::chrono::milliseconds(kClientMonitorSleepMs)); } @@ -9686,9 +9601,9 @@ MasterService::MetadataSerializer::Serialize() { msgpack::sbuffer sbuf; msgpack::packer packer(&sbuf); - // Create top-level map with 3 fields: "shards", "discarded_replicas", - // "replica_next_id" - packer.pack_map(3); + // Local delete intents are snapshot state: metadata removal may already be + // visible while the holder still needs to persist its tombstone. + packer.pack_map(4); // 1. Serialize metadata shards packer.pack("shards"); @@ -9769,6 +9684,13 @@ MasterService::MetadataSerializer::Serialize() { packer.pack("replica_next_id"); packer.pack(static_cast(Replica::next_id_.load())); + packer.pack("pending_local_deletes"); + const auto pending_local_deletes = + struct_pack::serialize(service_->local_delete_registry_.Snapshot()); + packer.pack_bin(pending_local_deletes.size()); + packer.pack_bin_body(pending_local_deletes.data(), + pending_local_deletes.size()); + return std::vector( reinterpret_cast(sbuf.data()), reinterpret_cast(sbuf.data()) + sbuf.size()); @@ -9802,6 +9724,7 @@ MasterService::MetadataSerializer::Deserialize( const msgpack::object* shards_obj = nullptr; const msgpack::object* discarded_replicas_obj = nullptr; const msgpack::object* replica_next_id_obj = nullptr; + const msgpack::object* pending_local_deletes_obj = nullptr; // Extract fields from top-level map for (uint32_t i = 0; i < obj.via.map.size; ++i) { @@ -9814,6 +9737,8 @@ MasterService::MetadataSerializer::Deserialize( discarded_replicas_obj = &obj.via.map.ptr[i].val; } else if (key == "replica_next_id") { replica_next_id_obj = &obj.via.map.ptr[i].val; + } else if (key == "pending_local_deletes") { + pending_local_deletes_obj = &obj.via.map.ptr[i].val; } } } @@ -9895,6 +9820,25 @@ MasterService::MetadataSerializer::Deserialize( auto next_id = replica_next_id_obj->as(); Replica::next_id_.store(next_id); LOG(INFO) << "Restored Replica::next_id_ to " << next_id; + + service_->local_delete_registry_.Reset(); + if (pending_local_deletes_obj != nullptr) { + if (pending_local_deletes_obj->type != msgpack::type::BIN) { + return tl::make_unexpected(SerializationError( + ErrorCode::DESERIALIZE_FAIL, + "Invalid pending_local_deletes snapshot field")); + } + std::vector tasks; + const std::string_view bytes(pending_local_deletes_obj->via.bin.ptr, + pending_local_deletes_obj->via.bin.size); + if (struct_pack::deserialize_to(tasks, bytes) != + struct_pack::errc::ok || + !service_->local_delete_registry_.Restore(tasks)) { + return tl::make_unexpected(SerializationError( + ErrorCode::DESERIALIZE_FAIL, + "Failed to restore pending LOCAL_DISK delete tasks")); + } + } service_->RebuildGroupRoutingIndex(); service_->ClearCandidatesForReload(); return {}; @@ -9914,6 +9858,7 @@ void MasterService::MetadataSerializer::Reset() { std::lock_guard lock(service_->discarded_replicas_mutex_); service_->discarded_replicas_.clear(); } + service_->local_delete_registry_.Reset(); Replica::next_id_.store(1); service_->ClearCandidatesForReload(); } @@ -10046,7 +9991,8 @@ MasterService::MetadataSerializer::DeserializeShard(const msgpack::object& obj, metadata_ptr->size, metadata_ptr->PopReplicas(), metadata_ptr->soft_pin_timeout.has_value(), metadata_ptr->IsHardPinned(), metadata_ptr->data_type, - metadata_ptr->group_id, tenant_id, user_key)); + metadata_ptr->group_id, tenant_id, user_key, + metadata_ptr->object_incarnation)); it->second.lease_timeout = metadata_ptr->lease_timeout; it->second.soft_pin_timeout = metadata_ptr->soft_pin_timeout; @@ -10070,11 +10016,13 @@ MasterService::MetadataSerializer::SerializeMetadata( // Pack ObjectMetadata using array structure for efficiency // Format: [client_id, put_start_time, size, lease_timeout, // has_soft_pin_timeout, soft_pin_timeout, replicas_count, data_type, - // replicas..., hard_pinned, group_id, object_checksum?] + // replicas..., hard_pinned, group_id, object_checksum?, + // incarnation_high, incarnation_low] - size_t array_size = 10; // client_id, put_start_time, size, lease_timeout, + size_t array_size = 12; // client_id, put_start_time, size, lease_timeout, // has_soft_pin_timeout, soft_pin_timeout, - // replicas_count, data_type, hard_pinned, group_id + // replicas_count, data_type, hard_pinned, + // group_id, incarnation high/low array_size += metadata.CountReplicas(); // One element per replica if (metadata.object_checksum.has_value()) { ++array_size; @@ -10134,6 +10082,8 @@ MasterService::MetadataSerializer::SerializeMetadata( if (metadata.object_checksum.has_value()) { packer.pack(*metadata.object_checksum); } + packer.pack(metadata.object_incarnation.high); + packer.pack(metadata.object_incarnation.low); return {}; } @@ -10193,11 +10143,13 @@ MasterService::MetadataSerializer::DeserializeMetadata( // v3: 9 + replicas_count, data_type + hard_pinned or hard_pinned + // group_id v4: 10 + replicas_count, data_type + hard_pinned + group_id // v5: 11 + replicas_count, v4 + object_checksum + // v6: v4 + object incarnation high/low, with an optional + // object_checksum before the incarnation fields // 64-bit arithmetic keeps an attacker-controlled near-UINT32_MAX // replicas_count from wrapping the bounds and slipping an out-of-bounds // index past the size check. constexpr uint64_t kBaseFieldCount = 7; - constexpr uint64_t kMaxOptionalFieldCount = 4; + constexpr uint64_t kMaxOptionalFieldCount = 6; const uint64_t total_elements = obj.via.array.size; const uint64_t min_elements = kBaseFieldCount + replicas_count; if (total_elements < min_elements || @@ -10248,14 +10200,32 @@ MasterService::MetadataSerializer::DeserializeMetadata( } std::optional object_checksum; - if (index < total_elements && - array[index].type == msgpack::type::POSITIVE_INTEGER) { + uint64_t trailing_fields = total_elements - index; + if (trailing_fields == 1 || trailing_fields == 3) { + if (array[index].type != msgpack::type::POSITIVE_INTEGER) { + return tl::unexpected(SerializationError( + ErrorCode::DESERIALIZE_FAIL, + "deserialize ObjectMetadata checksum type mismatch")); + } object_checksum = array[index++].as(); + --trailing_fields; + } + + ObjectIncarnation object_incarnation; + if (trailing_fields == 2) { + if (array[index].type != msgpack::type::POSITIVE_INTEGER || + array[index + 1].type != msgpack::type::POSITIVE_INTEGER) { + return tl::unexpected(SerializationError( + ErrorCode::DESERIALIZE_FAIL, + "deserialize ObjectMetadata incarnation type mismatch")); + } + object_incarnation.high = array[index++].as(); + object_incarnation.low = array[index++].as(); } if (index != total_elements) { return tl::unexpected(SerializationError( ErrorCode::DESERIALIZE_FAIL, - "deserialize ObjectMetadata optional field type mismatch")); + "deserialize ObjectMetadata optional field count mismatch")); } // Create ObjectMetadata instance @@ -10265,7 +10235,7 @@ MasterService::MetadataSerializer::DeserializeMetadata( std::chrono::system_clock::time_point( std::chrono::milliseconds(put_start_time_timestamp)), size, std::move(replicas), enable_soft_pin, is_hard_pinned, data_type, - group_id); + group_id, TenantId(), std::string(), object_incarnation); metadata->object_checksum = object_checksum; metadata->lease_timeout = std::chrono::system_clock::time_point( std::chrono::milliseconds(lease_timestamp)); @@ -11268,6 +11238,7 @@ std::string MasterService::SerializeMetadataForOpLog( payload.size = metadata.size; payload.group_id = metadata.group_id; payload.data_type = metadata.data_type; + payload.object_incarnation = metadata.object_incarnation; // Extract replica descriptors - get them all at once const auto& replicas = metadata.GetAllReplicas(); @@ -11293,6 +11264,7 @@ std::string MasterService::SerializeMetadataForOpLogWithoutMemReplicas( payload.size = metadata.size; payload.group_id = metadata.group_id; payload.data_type = metadata.data_type; + payload.object_incarnation = metadata.object_incarnation; const auto& replicas = metadata.GetAllReplicas(); payload.replicas.reserve(replicas.size()); @@ -11308,15 +11280,15 @@ std::string MasterService::SerializeMetadataForOpLogWithoutMemReplicas( } std::string MasterService::SerializeMetadataForOpLogFromReplicaDescriptors( - const UUID& client_id, uint64_t size, - const std::vector& replicas, - const std::string& group_id, ObjectDataType data_type) const { + const ObjectMetadata& metadata, + const std::vector& replicas) const { MetadataPayload payload; - payload.client_id = client_id; - payload.size = size; + payload.client_id = metadata.client_id; + payload.size = metadata.size; payload.replicas = replicas; - payload.group_id = group_id; - payload.data_type = data_type; + payload.group_id = metadata.group_id; + payload.data_type = metadata.data_type; + payload.object_incarnation = metadata.object_incarnation; auto result = struct_pack::serialize(payload); return std::string(result.begin(), result.end()); } diff --git a/mooncake-store/src/segment.cpp b/mooncake-store/src/segment.cpp index 54389e5778..5a39c2bf57 100644 --- a/mooncake-store/src/segment.cpp +++ b/mooncake-store/src/segment.cpp @@ -219,18 +219,34 @@ ErrorCode ScopedSegmentAccess::MountSegment(const Segment& segment, return ErrorCode::OK; } -ErrorCode ScopedSegmentAccess::MountLocalDiskSegment(const UUID& client_id, - bool enable_offloading) { +ErrorCode ScopedSegmentAccess::MountLocalDiskSegment( + const UUID& client_id, bool enable_offloading, + const std::string& local_disk_segment_id, uint64_t mount_epoch, + uint32_t capabilities) { auto exist_segment_it = segment_manager_->client_local_disk_segment_.find(client_id); if (exist_segment_it != segment_manager_->client_local_disk_segment_.end()) { - LOG(WARNING) << "client_id=" << client_id - << ", warn=local_disk_segment_already_exists"; - return ErrorCode::SEGMENT_ALREADY_EXISTS; + MutexLocker locker(&exist_segment_it->second->offloading_mutex_); + if (exist_segment_it->second->enable_offloading == enable_offloading && + exist_segment_it->second->local_disk_segment_id == + local_disk_segment_id && + exist_segment_it->second->mount_epoch == mount_epoch && + exist_segment_it->second->capabilities == capabilities) { + LOG(WARNING) << "client_id=" << client_id + << ", warn=local_disk_segment_already_exists"; + return ErrorCode::SEGMENT_ALREADY_EXISTS; + } + exist_segment_it->second->enable_offloading = enable_offloading; + exist_segment_it->second->local_disk_segment_id = local_disk_segment_id; + exist_segment_it->second->mount_epoch = mount_epoch; + exist_segment_it->second->capabilities = capabilities; + return ErrorCode::OK; } segment_manager_->client_local_disk_segment_.emplace( - client_id, std::make_shared(enable_offloading)); + client_id, std::make_shared( + enable_offloading, local_disk_segment_id, mount_epoch, + capabilities)); return ErrorCode::OK; } diff --git a/mooncake-store/src/storage_backend.cpp b/mooncake-store/src/storage_backend.cpp index ab2723b897..5db833bcfd 100644 --- a/mooncake-store/src/storage_backend.cpp +++ b/mooncake-store/src/storage_backend.cpp @@ -14,13 +14,12 @@ #include #include #include -#include #include #include +#include #include #include #include -#include #include #include @@ -28,6 +27,9 @@ #include "mutex.h" #include "utils.h" #include "crc32c.h" +#include "ascii_string.h" +#include "bool_parser.h" +#include "environ.h" #include @@ -47,33 +49,27 @@ struct FdGuard { return r; } }; -} // namespace - -#include "storage/distributed/distributed_storage_backend.h" -// spdiag perf points for the offload owner-side disk load breakdown. -#define SPDIAG_PERF_DEF_FILE "mooncake_perf_points.def" -#define SPDIAG_PROGRAM_NAME "mooncake_store" -#include "spdiag/auto_perf.h" +template +class ScopeExit { + public: + explicit ScopeExit(Callback callback) : callback_(std::move(callback)) {} + ScopeExit(const ScopeExit&) = delete; + ScopeExit& operator=(const ScopeExit&) = delete; + ~ScopeExit() { callback_(); } -namespace mooncake { + private: + Callback callback_; +}; -namespace { -thread_local StorageReadStats* current_storage_read_stats = nullptr; -} +constexpr size_t kMaxGcSources = 8; +} // namespace -StorageReadStats* CurrentStorageReadStats() { - return current_storage_read_stats; -} +#include "storage/distributed/distributed_storage_backend.h" -ScopedStorageReadStats::ScopedStorageReadStats(StorageReadStats* stats) - : previous_(current_storage_read_stats) { - current_storage_read_stats = stats; -} +namespace mooncake { -ScopedStorageReadStats::~ScopedStorageReadStats() { - current_storage_read_stats = previous_; -} +static std::optional GetEnvDouble(const char* name); bool FilePerKeyConfig::Validate() const { if (fsdir.empty()) { @@ -92,17 +88,26 @@ bool BucketBackendConfig::Validate() const { LOG(ERROR) << "BucketBackendConfig: bucket_size_limit must > 0"; return false; } + if (gc_enable && gc_interval_seconds <= 0) { + LOG(ERROR) + << "BucketBackendConfig: gc_interval_seconds must be positive"; + return false; + } + if (gc_enable && (gc_deleted_ratio <= 0.0 || gc_deleted_ratio > 1.0)) { + LOG(ERROR) << "BucketBackendConfig: gc_deleted_ratio must be in (0, 1]"; + return false; + } return true; } FilePerKeyConfig FilePerKeyConfig::FromEnvironment() { FilePerKeyConfig config; - config.fsdir = GetEnvStringOr("MOONCAKE_OFFLOAD_FSDIR", config.fsdir); + config.fsdir = Environ::GetString("MOONCAKE_OFFLOAD_FSDIR", config.fsdir); - config.enable_eviction = GetEnvOr( + config.enable_eviction = Environ::GetBool( "MOONCAKE_OFFLOAD_ENABLE_EVICTION", - GetEnvOr("ENABLE_EVICTION", config.enable_eviction)); + Environ::GetBool("ENABLE_EVICTION", config.enable_eviction)); return config; } @@ -110,20 +115,31 @@ FilePerKeyConfig FilePerKeyConfig::FromEnvironment() { BucketBackendConfig BucketBackendConfig::FromEnvironment() { BucketBackendConfig config; - config.bucket_keys_limit = GetEnvOr( + config.bucket_keys_limit = Environ::GetInt64( "MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT", config.bucket_keys_limit); - config.bucket_size_limit = GetEnvOr( + config.bucket_size_limit = Environ::GetInt64( "MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES", config.bucket_size_limit); config.max_total_size = - GetEnvOr("MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE", - GetEnvOr("MOONCAKE_BUCKET_MAX_TOTAL_SIZE", + Environ::GetInt64("MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE", + Environ::GetInt64("MOONCAKE_BUCKET_MAX_TOTAL_SIZE", config.max_total_size)); - const auto policy_str = GetEnvStringOr( + config.gc_enable = + Environ::GetBool("MOONCAKE_OFFLOAD_BUCKET_GC_ENABLE", config.gc_enable); + config.gc_interval_seconds = + Environ::GetInt64("MOONCAKE_OFFLOAD_BUCKET_GC_INTERVAL_SECONDS", + config.gc_interval_seconds); + const auto gc_deleted_ratio = + GetEnvDouble("MOONCAKE_OFFLOAD_BUCKET_GC_DELETED_RATIO"); + if (gc_deleted_ratio.has_value()) { + config.gc_deleted_ratio = *gc_deleted_ratio; + } + + const auto policy_str = Environ::GetString( "MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY", - GetEnvStringOr("MOONCAKE_BUCKET_EVICTION_POLICY", "fifo")); + Environ::GetString("MOONCAKE_BUCKET_EVICTION_POLICY", "fifo")); if (policy_str == "fifo") { config.eviction_policy = BucketEvictionPolicy::FIFO; } else if (policy_str == "lru") { @@ -132,43 +148,6 @@ BucketBackendConfig BucketBackendConfig::FromEnvironment() { config.eviction_policy = BucketEvictionPolicy::NONE; } - config.disable_ssd_eviction = - GetEnvOr("MOONCAKE_OFFLOAD_DISABLE_SSD_EVICTION", false); - - config.gc_enable = GetEnvOr("MOONCAKE_OFFLOAD_BUCKET_GC_ENABLE", - config.gc_enable); - config.gc_interval_ms = - GetEnvOr("MOONCAKE_OFFLOAD_BUCKET_GC_INTERVAL_MS", - config.gc_interval_ms); - // Parse doubles manually: GetEnvOr uses std::stoll which cannot parse - // fractional values like "0.25". - { - const char* ratio_env = - std::getenv("MOONCAKE_OFFLOAD_BUCKET_GC_DELETED_RATIO"); - if (ratio_env && !std::string(ratio_env).empty()) { - try { - config.gc_deleted_ratio = std::stod(std::string(ratio_env)); - } catch (...) { - // keep default - } - } - const char* wm_env = std::getenv( - "MOONCAKE_OFFLOAD_BUCKET_GC_HIGH_WATERMARK_RATIO"); - if (wm_env && !std::string(wm_env).empty()) { - try { - config.gc_high_watermark_ratio = - std::stod(std::string(wm_env)); - } catch (...) { - // keep default - } - } - } - config.gc_max_buckets_per_round = GetEnvOr( - "MOONCAKE_OFFLOAD_BUCKET_GC_MAX_BUCKETS_PER_ROUND", - config.gc_max_buckets_per_round); - config.gc_merge_enable = GetEnvOr( - "MOONCAKE_OFFLOAD_BUCKET_GC_MERGE_ENABLE", config.gc_merge_enable); - return config; } @@ -234,8 +213,7 @@ OffsetAllocatorBackendConfig OffsetAllocatorBackendConfig::FromEnvironment() { const char* pol = std::getenv("MOONCAKE_OFFSET_EVICTION_POLICY"); if (pol) { - std::string s(pol); - if (s == "fifo" || s == "FIFO" || s == "Fifo") { + if (AsciiCaseInsensitiveEquals(pol, "fifo")) { cfg.eviction_policy = OffsetEvictionPolicy::FIFO; } // NONE is default; LRU reserved for phase 2 @@ -248,13 +226,13 @@ OffsetAllocatorBackendConfig OffsetAllocatorBackendConfig::FromEnvironment() { cfg.keys_high_ratio = cfg.high_ratio; cfg.keys_low_ratio = cfg.low_ratio; - cfg.max_capacity_nodes = GetEnvOr( + cfg.max_capacity_nodes = Environ::GetInt64( "MOONCAKE_OFFSET_MAX_CAPACITY_NODES", cfg.max_capacity_nodes); // Read eviction cap as int64_t to guard against negative env values - // which would wrap to SIZE_MAX with GetEnvOr. + // which would wrap to SIZE_MAX with an unsigned parser. auto max_evict_raw = - GetEnvOr("MOONCAKE_OFFSET_MAX_EVICT_PER_OFFLOAD", + Environ::GetInt64("MOONCAKE_OFFSET_MAX_EVICT_PER_OFFLOAD", static_cast(cfg.max_evict_per_offload)); if (max_evict_raw > 0) { cfg.max_evict_per_offload = static_cast(max_evict_raw); @@ -268,11 +246,11 @@ OffsetAllocatorBackendConfig OffsetAllocatorBackendConfig::FromEnvironment() { const char* persist = std::getenv("MOONCAKE_OFFSET_PERSIST_MODE"); if (persist) { std::string s(persist); - if (s == "disabled" || s == "DISABLED") { + if (AsciiCaseInsensitiveEquals(s, "disabled")) { cfg.persist_mode = OffsetPersistMode::kDisabled; - } else if (s == "relaxed" || s == "RELAXED") { + } else if (AsciiCaseInsensitiveEquals(s, "relaxed")) { cfg.persist_mode = OffsetPersistMode::kRelaxed; - } else if (s == "strict" || s == "STRICT") { + } else if (AsciiCaseInsensitiveEquals(s, "strict")) { cfg.persist_mode = OffsetPersistMode::kStrict; } else { LOG(WARNING) << "Unknown MOONCAKE_OFFSET_PERSIST_MODE=" << s @@ -281,15 +259,14 @@ OffsetAllocatorBackendConfig OffsetAllocatorBackendConfig::FromEnvironment() { } cfg.persist_interval_seconds = - GetEnvOr("MOONCAKE_OFFSET_PERSIST_INTERVAL_SECONDS", + Environ::GetInt64("MOONCAKE_OFFSET_PERSIST_INTERVAL_SECONDS", cfg.persist_interval_seconds); // Record CRC-32C: "0"/"false"/"off" disables per-record checksums. const char* crc_env = std::getenv("MOONCAKE_OFFSET_RECORD_CRC"); if (crc_env) { - std::string s(crc_env); - for (auto& c : s) c = static_cast(std::tolower(c)); - if (s == "0" || s == "false" || s == "off") { + const auto parsed = TryParseBool(crc_env); + if (parsed.has_value() && !*parsed) { cfg.enable_record_crc = false; } } @@ -1524,8 +1501,12 @@ tl::expected StorageBackendAdaptor::BatchOffload( } metadatas.emplace_back( - StorageObjectMetadata{-1, 0, static_cast(kv.key.size()), - static_cast(kv.value.size()), ""}); + StorageObjectMetadata{-1, + 0, + static_cast(kv.key.size()), + static_cast(kv.value.size()), + "", + {}}); keys.emplace_back(kv.key); } @@ -1568,10 +1549,7 @@ tl::expected StorageBackendAdaptor::IsExist( tl::expected StorageBackendAdaptor::BatchLoad( std::unordered_map& batched_slices) { - auto* stats = CurrentStorageReadStats(); - if (stats) stats->io_mode = "preadv"; for (const auto& [key, slice] : batched_slices) { - const auto io_start = std::chrono::steady_clock::now(); KVEntry kv; kv.key = key; auto path = @@ -1585,11 +1563,6 @@ tl::expected StorageBackendAdaptor::BatchLoad( auto r = storage_backend_->LoadObject(path, kv_buf, kv_buf.size()); if (!r) { - if (stats) { - stats->status = "read_fail"; - stats->error_key = key; - stats->error_code = r.error(); - } LOG(ERROR) << "Failed to load from file"; return tl::make_unexpected(r.error()); } @@ -1599,17 +1572,6 @@ tl::expected StorageBackendAdaptor::BatchLoad( if (!kv.value.empty()) { std::memcpy(slice.ptr, kv.value.data(), kv.value.size()); } - if (stats) { - const auto io_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - io_start) - .count(); - stats->disk_read_us += io_us; - if (io_us > stats->slowest_disk_read_us) { - stats->slowest_disk_read_us = io_us; - stats->slowest_key = key; - } - } } return {}; } @@ -1700,9 +1662,13 @@ tl::expected StorageBackendAdaptor::ScanMeta( total_size += buf.size(); keys.emplace_back(std::move(kv.key)); - metas.emplace_back(StorageObjectMetadata{ - -1, 0, (int64_t)keys.back().size(), - static_cast(kv.value.size()), ""}); + metas.emplace_back( + StorageObjectMetadata{-1, + 0, + (int64_t)keys.back().size(), + static_cast(kv.value.size()), + "", + {}}); if ((int64_t)keys.size() >= file_storage_config_.scanmeta_iterator_keys_limit) { @@ -1775,11 +1741,13 @@ BucketStorageBackend::BucketStorageBackend( } BucketStorageBackend::~BucketStorageBackend() { - // Stop background GC thread first, before clearing any state it touches. - if (gc_running_.load(std::memory_order_acquire)) { - gc_running_.store(false, std::memory_order_release); - gc_cv_.notify_all(); - if (gc_thread_.joinable()) gc_thread_.join(); + { + std::lock_guard lock(gc_mutex_); + gc_stop_.store(true, std::memory_order_release); + } + gc_cv_.notify_all(); + if (gc_thread_.joinable()) { + gc_thread_.join(); } // Clear file cache to release UringFile instances before destruction // This ensures orderly cleanup of io_uring resources @@ -1945,8 +1913,6 @@ tl::expected BucketStorageBackend::BatchQuery( tl::expected BucketStorageBackend::BatchLoad( std::unordered_map& batch_object) { - auto* stats = CurrentStorageReadStats(); - const auto plan_start = std::chrono::steady_clock::now(); // Step 1: Build read plan by copying metadata under lock // BucketReadGuard increments inflight_reads_ to prevent deletion during IO. // When the guard goes out of scope, it decrements the counter. @@ -1964,27 +1930,12 @@ tl::expected BucketStorageBackend::BatchLoad( std::unordered_map> bucket_read_plans; std::vector bucket_guards; // RAII guards for all buckets - // Phase 1: build read plan under lock (OwnerLoadPlan). Pure in-memory - // metadata lookups; the error early-returns let the dtor Abandon the - // unfinished sample. - SpDiag::PerfPoint pt_plan(PerfKey::GET_SSD_OWNER_LOAD_PLAN, - SpDiag::PerfLevel::MODULE); - pt_plan.Start(); { SharedMutexLocker lock(&mutex_, shared_lock); for (const auto& [key, dest_slice] : batch_object) { // Lookup key -> metadata auto object_it = object_bucket_map_.find(key); if (object_it == object_bucket_map_.end()) { - if (stats) { - stats->plan_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - plan_start) - .count(); - stats->status = "plan_fail"; - stats->error_key = key; - stats->error_code = ErrorCode::INVALID_KEY; - } LOG(ERROR) << "Key not found: " << key; return tl::make_unexpected(ErrorCode::INVALID_KEY); } @@ -1993,15 +1944,6 @@ tl::expected BucketStorageBackend::BatchLoad( // Lookup bucket -> BucketMetadata auto bucket_it = buckets_.find(metadata.bucket_id); if (bucket_it == buckets_.end()) { - if (stats) { - stats->plan_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - plan_start) - .count(); - stats->status = "plan_fail"; - stats->error_key = key; - stats->error_code = ErrorCode::BUCKET_NOT_FOUND; - } LOG(ERROR) << "Bucket not found for key: " << key << ", bucket_id=" << metadata.bucket_id; return tl::make_unexpected(ErrorCode::BUCKET_NOT_FOUND); @@ -2009,15 +1951,6 @@ tl::expected BucketStorageBackend::BatchLoad( // Validate size if (metadata.data_size != static_cast(dest_slice.size)) { - if (stats) { - stats->plan_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - plan_start) - .count(); - stats->status = "plan_fail"; - stats->error_key = key; - stats->error_code = ErrorCode::INVALID_PARAMS; - } LOG(ERROR) << "Size mismatch for key: " << key << ", expected: " << metadata.data_size << ", got: " << dest_slice.size; @@ -2047,13 +1980,6 @@ tl::expected BucketStorageBackend::BatchLoad( metadata.key_size, metadata.data_size, dest_slice}); } } - pt_plan.End(0); - if (stats) { - stats->plan_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - plan_start) - .count(); - } // Lock released here - bucket files protected by BucketReadGuards // which remain alive until this function returns (~line bucket_guards // destructor), keeping inflight_reads_ > 0 throughout the I/O phase. @@ -2061,34 +1987,15 @@ tl::expected BucketStorageBackend::BatchLoad( // Step 2: Perform IO without holding any locks for (auto& [bucket_id, read_plans] : bucket_read_plans) { // Open file for this bucket (cheap syscall, no lock needed) - const auto open_start = std::chrono::steady_clock::now(); auto filepath_res = GetBucketDataPath(bucket_id); if (!filepath_res) { - if (stats) { - stats->file_open_us += - std::chrono::duration_cast( - std::chrono::steady_clock::now() - open_start) - .count(); - stats->status = "open_fail"; - stats->error_code = ErrorCode::INTERNAL_ERROR; - } LOG(ERROR) << "Failed to get bucket data path, bucket_id=" << bucket_id; return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); } auto file_res = OpenFile(filepath_res.value(), FileMode::Read); - if (stats) { - stats->file_open_us += - std::chrono::duration_cast( - std::chrono::steady_clock::now() - open_start) - .count(); - } if (!file_res) { - if (stats) { - stats->status = "open_fail"; - stats->error_code = file_res.error(); - } LOG(ERROR) << "Failed to open bucket file: " << filepath_res.value(); return tl::make_unexpected(file_res.error()); @@ -2099,19 +2006,16 @@ tl::expected BucketStorageBackend::BatchLoad( for (const auto& plan : read_plans) { int64_t actual_offset = plan.offset + plan.key_size; tl::expected read_res; - const auto disk_start = std::chrono::steady_clock::now(); - const char* io_mode = "preadv"; #ifdef USE_URING // Try to use read_aligned for O_DIRECT I/O if file is UringFile UringFile* uring_file = dynamic_cast(file.get()); if (uring_file != nullptr) { - io_mode = "io_uring_direct"; // Calculate aligned read range int64_t aligned_offset = align_down(actual_offset, kDirectIOAlignment); - int64_t data_end = actual_offset + - static_cast(plan.dest_slice.size); + int64_t data_end = + actual_offset + static_cast(plan.dest_slice.size); int64_t aligned_end = static_cast(align_up( static_cast(data_end), kDirectIOAlignment)); size_t aligned_size = @@ -2121,53 +2025,49 @@ tl::expected BucketStorageBackend::BatchLoad( // Zero-copy path: read directly into the slice buffer. // dest_slice.ptr is 4096-aligned and oversized (from // AllocateBatch) to accommodate the full aligned read range. - SpDiag::PerfPoint pt_uring(PerfKey::GET_SSD_OWNER_LOAD_URING, - SpDiag::PerfLevel::MODULE); - pt_uring.Start(); read_res = uring_file->read_aligned( plan.dest_slice.ptr, aligned_size, aligned_offset); - pt_uring.End(read_res ? 0 : -1); if (read_res) { - // Adjust ptr to point to actual data start (no memcpy) + // Verify the aligned read returned enough bytes + // to cover the actual data region. read_aligned + // reads the full aligned range [aligned_offset, + // aligned_end); the caller-visible data starts at + // offset_in_buffer into that buffer and spans + // plan.dest_slice.size bytes. + size_t min_required = + static_cast(offset_in_buffer) + + plan.dest_slice.size; + if (read_res.value() < min_required) { + LOG(ERROR) + << "read_aligned short read for key: " << plan.key + << ", bucket_id=" << plan.bucket_id + << ", expected at least: " << min_required + << " (aligned_size=" << aligned_size + << ", data_size=" << plan.dest_slice.size << ")" + << ", got: " << read_res.value(); + return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); + } + // Adjust ptr to point to actual data start + // (zero-copy: no memcpy, buffer was oversized by + // AllocateBatch to accommodate the aligned read) batch_object.at(plan.key).ptr = static_cast(plan.dest_slice.ptr) + offset_in_buffer; + // Normalize read_res so the common validation + // below passes. The real short-read check was + // already done above (min_required). read_res = plan.dest_slice.size; } } else #endif - { - // Fallback to per-key vector_read for non-UringFile (PosixFile). - iovec iov{plan.dest_slice.ptr, plan.dest_slice.size}; - SpDiag::PerfPoint pt_posix(PerfKey::GET_SSD_OWNER_LOAD_POSIX, - SpDiag::PerfLevel::MODULE); - pt_posix.Start(); - read_res = file->vector_read(&iov, 1, actual_offset); - pt_posix.End(read_res ? 0 : -1); - if (stats) { - const auto read_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - disk_start) - .count(); - stats->disk_read_us += read_us; - if (read_us > stats->slowest_disk_read_us) { - stats->slowest_disk_read_us = read_us; - stats->slowest_key = plan.key; - } - if (stats->io_mode == "unknown") { - stats->io_mode = io_mode; - } else if (stats->io_mode != io_mode) { - stats->io_mode = "mixed"; - } + { + // Fallback to vector_read for non-UringFile + iovec iov{plan.dest_slice.ptr, plan.dest_slice.size}; + read_res = file->vector_read(&iov, 1, actual_offset); } if (!read_res) { - if (stats) { - stats->status = "read_fail"; - stats->error_key = plan.key; - stats->error_code = read_res.error(); - } LOG(ERROR) << "vector_read failed for key: " << plan.key << ", bucket_id=" << plan.bucket_id << ", error: " << read_res.error(); @@ -2175,18 +2075,12 @@ tl::expected BucketStorageBackend::BatchLoad( } if (read_res.value() != plan.dest_slice.size) { - if (stats) { - stats->status = "short_read"; - stats->error_key = plan.key; - stats->error_code = ErrorCode::FILE_READ_FAIL; - } LOG(ERROR) << "Read size mismatch for key: " << plan.key << ", expected: " << plan.dest_slice.size << ", got: " << read_res.value(); return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); } } - } } // bucket_guards go out of scope here, decrementing inflight_reads_ @@ -2213,11 +2107,16 @@ tl::expected BucketStorageBackend::Init() { LOG(ERROR) << "Storage backend already initialized"; return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); } + auto recovery = RecoverGarbageCollection(); + if (!recovery) { + return tl::make_unexpected(recovery.error()); + } SharedMutexLocker lock(&mutex_); object_bucket_map_.clear(); buckets_.clear(); lru_index_.clear(); total_size_ = 0; + reclaimable_bytes_.store(0, std::memory_order_relaxed); int64_t max_bucket_id = BucketIdGenerator::INIT_NEW_START_ID; for (const auto& entry : @@ -2246,25 +2145,13 @@ tl::expected BucketStorageBackend::Init() { auto load_bucket_metadata_result = LoadBucketMetadata(bucket_id, metadata_it->second); if (!load_bucket_metadata_result) { - LOG(ERROR) - << "Failed to load metadata for bucket: " - << bucket_id_str - << ", will delete the bucket's data and metadata"; - - auto bucket_data_path_res = GetBucketDataPath(bucket_id); - if (bucket_data_path_res) { - fs::remove(bucket_data_path_res.value()); - } - - auto bucket_meta_path_res = - GetBucketMetadataPath(bucket_id); - if (bucket_meta_path_res) { - fs::remove(bucket_meta_path_res.value()); - } - + LOG(ERROR) << "Failed to load metadata for bucket: " + << bucket_id_str + << "; preserving files for operator recovery"; lru_index_.erase({0LL, bucket_id}); buckets_.erase(bucket_id); - continue; + return tl::make_unexpected( + load_bucket_metadata_result.error()); } auto bucket_data_path_res = GetBucketDataPath(bucket_id); if (!bucket_data_path_res) { @@ -2285,9 +2172,9 @@ tl::expected BucketStorageBackend::Init() { } if (bucket_data_ec == std::errc::no_such_file_or_directory || !fs::is_regular_file(bucket_data_status)) { - LOG(ERROR) << "Bucket metadata has no valid data file: " - << entry.path().string() - << ", will delete the bucket's remaining files"; + LOG(ERROR) + << "Bucket metadata has no valid data file: " + << entry.path().string() << "; removing stale metadata"; CleanupOrphanedBucket(bucket_id); lru_index_.erase({0LL, bucket_id}); buckets_.erase(bucket_id); @@ -2298,8 +2185,8 @@ tl::expected BucketStorageBackend::Init() { meta.metadatas.empty() || meta.keys.empty()) { LOG(ERROR) << "Metadata validation failed for bucket: " << bucket_id_str - << ", will delete the bucket's data and " - "metadata. Detailed values:"; + << "; preserving files for operator recovery. " + "Detailed values:"; LOG(ERROR) << " data_size: " << meta.data_size << " (should not be 0)"; LOG(ERROR) << " meta_size: " << meta.meta_size @@ -2313,44 +2200,63 @@ tl::expected BucketStorageBackend::Init() { << " keys.size(): " << meta.keys.size() << " (empty: " << (meta.keys.empty() ? "true" : "false") << ")"; - auto bucket_data_path_res = GetBucketDataPath(bucket_id); - if (bucket_data_path_res) { - fs::remove(bucket_data_path_res.value()); - } - - auto bucket_meta_path_res = - GetBucketMetadataPath(bucket_id); - if (bucket_meta_path_res) { - fs::remove(bucket_meta_path_res.value()); - } - lru_index_.erase({0LL, bucket_id}); buckets_.erase(bucket_id); - continue; + return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); + } + if (meta.keys.size() != meta.metadatas.size()) { + LOG(ERROR) << "Bucket metadata key/object count mismatch: " + << bucket_id_str; + lru_index_.erase({0LL, bucket_id}); + buckets_.erase(bucket_id); + return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); + } + const uintmax_t physical_data_size = + fs::file_size(bucket_data_path_res.value(), bucket_data_ec); + if (bucket_data_ec || + physical_data_size < + static_cast(meta.data_size)) { + LOG(ERROR) << "Bucket data file is shorter than metadata: " + << bucket_id_str; + lru_index_.erase({0LL, bucket_id}); + buckets_.erase(bucket_id); + return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); + } + std::unordered_set unique_keys; + for (size_t i = 0; i < meta.keys.size(); ++i) { + const auto& object = meta.metadatas[i]; + if (object.offset < 0 || object.key_size < 0 || + object.data_size < 0 || + object.offset > meta.data_size - object.key_size || + object.offset + object.key_size > + meta.data_size - object.data_size || + !unique_keys.insert(meta.keys[i]).second) { + LOG(ERROR) << "Bucket object metadata is invalid: " + << bucket_id_str << ", object_index=" << i; + lru_index_.erase({0LL, bucket_id}); + buckets_.erase(bucket_id); + return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); + } } if (bucket_id > max_bucket_id) { max_bucket_id = bucket_id; } total_size_ += metadata_it->second->data_size + metadata_it->second->meta_size; + reclaimable_bytes_.fetch_add( + BucketReclaimableBytes(*metadata_it->second), + std::memory_order_relaxed); for (size_t i = 0; i < metadata_it->second->keys.size(); i++) { - const auto& key = metadata_it->second->keys[i]; - if (std::find(metadata_it->second->tombstones.begin(), - metadata_it->second->tombstones.end(), key) != - metadata_it->second->tombstones.end()) { - metadata_it->second->deleted_bytes_.fetch_add( - metadata_it->second->metadatas[i].key_size + - metadata_it->second->metadatas[i].data_size, - std::memory_order_relaxed); + const auto& object_meta = metadata_it->second->metadatas[i]; + if (object_meta.tombstoned) { continue; } object_bucket_map_.emplace( - key, StorageObjectMetadata{ - metadata_it->first, - metadata_it->second->metadatas[i].offset, - metadata_it->second->metadatas[i].key_size, - metadata_it->second->metadatas[i].data_size, - ""}); + metadata_it->second->keys[i], + StorageObjectMetadata{ + metadata_it->first, object_meta.offset, + object_meta.key_size, object_meta.data_size, "", + object_meta.object_incarnation}); } } } @@ -2451,12 +2357,10 @@ tl::expected BucketStorageBackend::Init() { return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); } - // Start background GC thread if enabled. if (bucket_backend_config_.gc_enable) { - gc_running_.store(true, std::memory_order_release); - gc_thread_ = std::thread(&BucketStorageBackend::GCThreadFunc, this); + gc_thread_ = std::thread( + &BucketStorageBackend::GarbageCollectionThreadFunc, this); } - return {}; } @@ -2520,24 +2424,32 @@ tl::expected BucketStorageBackend::BucketScan( SharedMutexLocker lock(&mutex_, shared_lock); auto bucket_it = buckets_.lower_bound(bucket_id); for (; bucket_it != buckets_.end(); ++bucket_it) { - if (static_cast(bucket_it->second->keys.size()) > limit) { + const auto live_count = static_cast( + std::count_if(bucket_it->second->metadatas.begin(), + bucket_it->second->metadatas.end(), + [](const BucketObjectMetadata& metadata) { + return !metadata.tombstoned; + })); + if (live_count > limit) { LOG(ERROR) << "Bucket key count exceeds limit: " << "bucket_id=" << bucket_it->first - << ", current_size=" << bucket_it->second->keys.size() + << ", current_size=" << live_count << ", limit=" << limit; return tl::make_unexpected(ErrorCode::KEYS_EXCEED_BUCKET_LIMIT); } - if (static_cast(bucket_it->second->keys.size() + keys.size()) > - limit) { + if (live_count + static_cast(keys.size()) > limit) { return bucket_it->first; } buckets.emplace_back(bucket_it->first); for (size_t i = 0; i < bucket_it->second->keys.size(); i++) { + const auto& object_meta = bucket_it->second->metadatas[i]; + if (object_meta.tombstoned) { + continue; + } keys.emplace_back(bucket_it->second->keys[i]); metadatas.emplace_back(StorageObjectMetadata{ - bucket_it->first, bucket_it->second->metadatas[i].offset, - bucket_it->second->metadatas[i].key_size, - bucket_it->second->metadatas[i].data_size, ""}); + bucket_it->first, object_meta.offset, object_meta.key_size, + object_meta.data_size, "", object_meta.object_incarnation}); } } return 0; @@ -2552,13 +2464,33 @@ BucketStorageBackend::GetStoreMetadata() { tl::expected BucketStorageBackend::AllocateOffloadingBuckets( const std::unordered_map& offloading_objects, - std::vector>& buckets_keys) { - return GroupOffloadingKeysByBucket(offloading_objects, buckets_keys); + std::vector>& buckets_keys, + const std::unordered_map* + object_incarnations) { + auto grouped = + GroupOffloadingKeysByBucket(offloading_objects, buckets_keys); + if (!grouped || object_incarnations == nullptr) { + return grouped; + } + + std::unordered_set grouped_keys; + for (const auto& bucket_keys : buckets_keys) { + grouped_keys.insert(bucket_keys.begin(), bucket_keys.end()); + } + MutexLocker locker(&offloading_mutex_); + for (const auto& [key, incarnation] : *object_incarnations) { + if (grouped_keys.contains(key) || + ungrouped_offloading_objects_.contains(key)) { + offloading_object_incarnations_[key] = incarnation; + } + } + return {}; } void BucketStorageBackend::ClearUngroupedOffloadingObjects() { MutexLocker locker(&offloading_mutex_); ungrouped_offloading_objects_.clear(); + offloading_object_incarnations_.clear(); } size_t BucketStorageBackend::UngroupedOffloadingObjectsSize() const { @@ -2668,6 +2600,7 @@ BucketStorageBackend::BuildBucket( std::vector& iovs, std::vector& metadatas) { auto bucket = std::make_shared(); int64_t storage_offset = 0; + MutexLocker offloading_locker(&offloading_mutex_); for (const auto& object : batch_object) { if (object.second.empty()) { LOG(ERROR) << "Failed to create bucket, object is empty"; @@ -2681,12 +2614,20 @@ BucketStorageBackend::BuildBucket( iovs.emplace_back(iovec{slice.ptr, slice.size}); } bucket->data_size += object_total_size + object.first.size(); + ObjectIncarnation object_incarnation; + auto incarnation_it = + offloading_object_incarnations_.find(object.first); + if (incarnation_it != offloading_object_incarnations_.end()) { + object_incarnation = incarnation_it->second; + offloading_object_incarnations_.erase(incarnation_it); + } bucket->metadatas.emplace_back(BucketObjectMetadata{ storage_offset, static_cast(object.first.size()), - object_total_size}); - metadatas.emplace_back(StorageObjectMetadata{ - bucket_id, storage_offset, - static_cast(object.first.size()), object_total_size, ""}); + object_total_size, object_incarnation, false}); + metadatas.emplace_back( + StorageObjectMetadata{bucket_id, storage_offset, + static_cast(object.first.size()), + object_total_size, "", object_incarnation}); bucket->keys.push_back(object.first); storage_offset += object_total_size + object.first.size(); } @@ -2834,9 +2775,10 @@ tl::expected BucketStorageBackend::WriteBucket( return {}; } -void BucketStorageBackend::CleanupOrphanedBucket(int64_t bucket_id) { +bool BucketStorageBackend::CleanupOrphanedBucket(int64_t bucket_id) { namespace fs = std::filesystem; std::error_code ec; + bool cleaned = true; auto data_path_res = GetBucketDataPath(bucket_id); if (data_path_res) { @@ -2855,7 +2797,10 @@ void BucketStorageBackend::CleanupOrphanedBucket(int64_t bucket_id) { LOG(WARNING) << "Failed to cleanup bucket data file: " << data_path_res.value() << ", error: " << ec.message(); + cleaned = false; } + } else { + cleaned = false; } auto meta_path_res = GetBucketMetadataPath(bucket_id); @@ -2868,8 +2813,12 @@ void BucketStorageBackend::CleanupOrphanedBucket(int64_t bucket_id) { LOG(WARNING) << "Failed to cleanup bucket metadata file: " << meta_path_res.value() << ", error: " << ec.message(); + cleaned = false; } + } else { + cleaned = false; } + return cleaned; } void BucketStorageBackend::RollbackCommittedBucket( @@ -2898,14 +2847,11 @@ void BucketStorageBackend::RollbackCommittedBucket( auto obj_it = object_bucket_map_.find(key); if (obj_it != object_bucket_map_.end() && obj_it->second.bucket_id == bucket_id) { - total_size_ -= - obj_it->second.data_size + obj_it->second.key_size; object_bucket_map_.erase(obj_it); } } - // Remove bucket metadata - total_size_ -= bucket_meta->meta_size; + total_size_ -= bucket_meta->data_size + bucket_meta->meta_size; lru_index_.erase({0LL, bucket_id}); buckets_.erase(bucket_it); } @@ -2970,40 +2916,43 @@ BucketStorageBackend::SelectEvictionCandidate() { case BucketEvictionPolicy::FIFO: // buckets_ is ordered by bucket_id (monotonically increasing), // so begin() is always the oldest bucket. - return buckets_.begin(); - - case BucketEvictionPolicy::LRU: - // Use lru_index_ (a std::set ordered by {last_access_ns_, - // bucket_id}) for O(log N) candidate selection. - // - // The index may be stale: BatchLoad updates last_access_ns_ - // atomically under a shared lock without touching lru_index_. - // We repair lazily here (called under exclusive lock): - // - If the top entry's timestamp matches the actual - // last_access_ns_, it is the true LRU candidate. - // - If stale, re-insert with the correct timestamp and retry. - // - If the bucket no longer exists, discard the entry. + return std::find_if( + buckets_.begin(), buckets_.end(), [](const auto& entry) { + return !entry.second->mutation_in_progress_.load( + std::memory_order_acquire); + }); + + case BucketEvictionPolicy::LRU: { + std::vector> skipped; while (!lru_index_.empty()) { - auto top_it = lru_index_.begin(); - auto [ts, id] = *top_it; - auto bucket_it = buckets_.find(id); - if (bucket_it == buckets_.end()) { - lru_index_.erase(top_it); + auto top = lru_index_.begin(); + auto [timestamp, bucket_id] = *top; + auto bucket = buckets_.find(bucket_id); + if (bucket == buckets_.end()) { + lru_index_.erase(top); continue; } - int64_t actual_ts = bucket_it->second->last_access_ns_.load( - std::memory_order_relaxed); - if (actual_ts == ts) { - // Correct entry: remove from index (bucket is about to be - // evicted) and return. - lru_index_.erase(top_it); - return bucket_it; + const int64_t actual_timestamp = + bucket->second->last_access_ns_.load( + std::memory_order_relaxed); + if (bucket->second->mutation_in_progress_.load( + std::memory_order_acquire)) { + skipped.emplace_back(actual_timestamp, bucket_id); + lru_index_.erase(top); + continue; + } + if (actual_timestamp != timestamp) { + lru_index_.erase(top); + lru_index_.emplace(actual_timestamp, bucket_id); + continue; } - // Stale: repair and retry to find the true minimum. - lru_index_.erase(top_it); - lru_index_.emplace(actual_ts, id); + lru_index_.erase(top); + lru_index_.insert(skipped.begin(), skipped.end()); + return bucket; } + lru_index_.insert(skipped.begin(), skipped.end()); return buckets_.end(); + } default: return buckets_.end(); @@ -3105,20 +3054,19 @@ BucketStorageBackend::PrepareEviction( std::move(evict_it->second); buckets_.erase(evict_it); - int64_t evicted_size = evict_meta->meta_size; + const int64_t evicted_size = + evict_meta->data_size + evict_meta->meta_size; // Remove all keys belonging to this bucket from the object map. for (const auto& key : evict_meta->keys) { auto obj_it = object_bucket_map_.find(key); if (obj_it != object_bucket_map_.end() && obj_it->second.bucket_id == evict_id) { - const int64_t object_size = - obj_it->second.data_size + obj_it->second.key_size; - total_size_ -= object_size; - evicted_size += object_size; object_bucket_map_.erase(obj_it); } } - total_size_ -= evict_meta->meta_size; + total_size_ -= evicted_size; + reclaimable_bytes_.fetch_sub(BucketReclaimableBytes(*evict_meta), + std::memory_order_relaxed); result.evicted_size += evicted_size; // Collect for notification and file deletion. @@ -3174,12 +3122,20 @@ void BucketStorageBackend::RestorePreparedEvictionLocked( for (size_t i = 0; i < bucket_meta->keys.size(); ++i) { const auto& key = bucket_meta->keys[i]; const auto& object_meta = bucket_meta->metadatas[i]; - object_bucket_map_[key] = StorageObjectMetadata{ - bucket_id, object_meta.offset, object_meta.key_size, - object_meta.data_size, ""}; - total_size_ += object_meta.data_size + object_meta.key_size; - } - total_size_ += bucket_meta->meta_size; + if (object_meta.tombstoned) { + continue; + } + object_bucket_map_[key] = + StorageObjectMetadata{bucket_id, + object_meta.offset, + object_meta.key_size, + object_meta.data_size, + "", + object_meta.object_incarnation}; + } + total_size_ += bucket_meta->data_size + bucket_meta->meta_size; + reclaimable_bytes_.fetch_add(BucketReclaimableBytes(*bucket_meta), + std::memory_order_relaxed); if (bucket_backend_config_.eviction_policy == BucketEvictionPolicy::LRU) { lru_index_.emplace( @@ -3232,6 +3188,8 @@ tl::expected BucketStorageBackend::FinalizeEviction( size_t cleanup_failed_count = 0; for (const auto& [bucket_id, bucket_meta] : pending.buckets) { + std::unique_lock operation_lock( + bucket_meta->operation_mutex_); bool bucket_cleanup_failed = false; // The master has already committed the replica removal. Attempt to @@ -3383,6 +3341,11 @@ tl::expected BucketStorageBackend::DeleteBucket( << bucket_id; return tl::make_unexpected(ErrorCode::BUCKET_NOT_FOUND); } + if (bucket_it->second->mutation_in_progress_.load( + std::memory_order_acquire)) { + return tl::make_unexpected( + ErrorCode::UNAVAILABLE_IN_CURRENT_STATUS); + } // Move the shared_ptr out - we now own it bucket_metadata = std::move(bucket_it->second); @@ -3397,17 +3360,19 @@ tl::expected BucketStorageBackend::DeleteBucket( auto obj_it = object_bucket_map_.find(key); if (obj_it != object_bucket_map_.end() && obj_it->second.bucket_id == bucket_id) { - total_size_ -= - obj_it->second.data_size + obj_it->second.key_size; object_bucket_map_.erase(obj_it); } } - // Subtract metadata size - total_size_ -= bucket_metadata->meta_size; + total_size_ -= bucket_metadata->data_size + bucket_metadata->meta_size; + reclaimable_bytes_.fetch_sub(BucketReclaimableBytes(*bucket_metadata), + std::memory_order_relaxed); } // Lock released - new readers can't find this bucket anymore + std::unique_lock operation_lock( + bucket_metadata->operation_mutex_); + // Step 2: Wait for in-flight reads to complete // Readers that started before we removed from buckets_ still hold guards constexpr int kMaxSpinIterations = 1000; @@ -3490,561 +3455,973 @@ void BucketStorageBackend::RemoveAll() { LOG(INFO) << "RemoveAll: removed " << bucket_ids.size() << " bucket(s)"; } -// --- Explicit-delete-only GC --- -tl::expected BucketStorageBackend::MarkRemoved( - const std::string& key) { - SharedMutexLocker lock(&mutex_); - auto it = object_bucket_map_.find(key); - if (it == object_bucket_map_.end()) { - return {}; +tl::expected BucketStorageBackend::StoreBucketMetadata( + int64_t id, std::shared_ptr metadata) { + auto meta_path_res = GetBucketMetadataPath(id); + if (!meta_path_res) { + LOG(ERROR) << "Failed to get bucket metadata path, bucket_id=" << id; + return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); } - int64_t bucket_id = it->second.bucket_id; - int64_t freed = it->second.data_size + it->second.key_size; - auto bucket_it = buckets_.find(bucket_id); - if (bucket_it == buckets_.end()) { - return tl::make_unexpected(ErrorCode::BUCKET_NOT_FOUND); + auto meta_path = meta_path_res.value(); + auto open_file_result = OpenFile(meta_path, FileMode::Write); + if (!open_file_result) { + LOG(ERROR) << "Failed to open file for bucket writing: " << meta_path; + return tl::make_unexpected(ErrorCode::FILE_OPEN_FAIL); + } + auto file = std::move(open_file_result.value()); + std::string str; + struct_pb::to_pb(*metadata, str); + auto write_result = file->write(str, str.size()); + if (!write_result) { + LOG(ERROR) << "Write failed for: " << meta_path + << ", error: " << write_result.error(); + return tl::make_unexpected(write_result.error()); } - auto bucket = bucket_it->second; - const auto object_metadata = it->second; - object_bucket_map_.erase(it); - bucket->tombstones.push_back(key); - auto persist_result = StoreBucketMetadata(bucket_id, bucket); - if (!persist_result) { - object_bucket_map_.emplace(key, object_metadata); - bucket->tombstones.pop_back(); - return tl::make_unexpected(persist_result.error()); - } - bucket->deleted_bytes_.fetch_add(freed, std::memory_order_relaxed); - ++bucket->generation_; + if (write_result.value() != str.size()) { + LOG(ERROR) << "Write size mismatch for: " << meta_path + << ", expected: " << str.size() + << ", got: " << write_result.value(); + return tl::make_unexpected(ErrorCode::FILE_WRITE_FAIL); + } + metadata->meta_size = str.size(); return {}; } -tl::expected BucketStorageBackend::BatchMarkRemoved( - const std::vector& keys) { - for (const auto& key : keys) { - auto result = MarkRemoved(key); - if (!result) { - return result; +tl::expected +BucketStorageBackend::StoreBucketMetadataAtomically(int64_t id, + BucketMetadata& metadata) { + namespace fs = std::filesystem; + auto meta_path_res = GetBucketMetadataPath(id); + if (!meta_path_res) { + return tl::unexpected(meta_path_res.error()); + } + const fs::path meta_path(meta_path_res.value()); + const fs::path temp_path = + meta_path.string() + ".delete." + UuidToString(generate_uuid()); + + std::string bytes; + struct_pb::to_pb(metadata, bytes); + const int fd = ::open(temp_path.c_str(), O_WRONLY | O_CREAT | O_EXCL, 0644); + if (fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard file(fd); + size_t written = 0; + while (written < bytes.size()) { + const ssize_t n = + ::write(file.get(), bytes.data() + written, bytes.size() - written); + if (n < 0 && errno == EINTR) { + continue; + } + if (n <= 0) { + fs::remove(temp_path); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } + written += static_cast(n); + } + if (::fsync(file.get()) != 0 || ::close(file.release()) != 0) { + fs::remove(temp_path); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + if (::rename(temp_path.c_str(), meta_path.c_str()) != 0) { + fs::remove(temp_path); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } + const int directory_fd = + ::open(meta_path.parent_path().c_str(), O_RDONLY | O_DIRECTORY); + if (directory_fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard directory(directory_fd); + if (::fsync(directory.get()) != 0) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + metadata.meta_size = static_cast(bytes.size()); return {}; } -bool BucketStorageBackend::CompactBucket(int64_t bucket_id) { - return CompactBuckets({bucket_id}, true); -} - -bool BucketStorageBackend::CompactBuckets( - const std::vector& bucket_ids, bool space_pressure) { - if (bucket_ids.empty()) return true; +std::vector BucketStorageBackend::BatchMarkDeleted( + const std::vector& tasks) { + std::vector results; + results.reserve(tasks.size()); + for (const auto& task : tasks) { + results.push_back({task.task_id, LocalDeleteResult::kRetryableFailure, + ErrorCode::INTERNAL_ERROR}); + } - // Step 1: lock-snapshot live keys from ALL old buckets + mark compacting_ - struct LiveKeyInfo { - std::string key; - int64_t old_bucket_id; - BucketObjectMetadata meta; - }; - std::vector live_keys_info; - // old_bucket_id -> {shared_ptr, last_access_ns} - std::unordered_map, int64_t>> - old_buckets; - std::unordered_map old_bucket_generations; + std::unordered_map> by_bucket; { - SharedMutexLocker lock(&mutex_); - for (int64_t bid : bucket_ids) { - auto it = buckets_.find(bid); - if (it == buckets_.end()) continue; - auto& bucket = it->second; - if (bucket->compacting_.load(std::memory_order_relaxed)) continue; - bucket->compacting_.store(true, std::memory_order_relaxed); - int64_t ts = bucket->last_access_ns_.load( - std::memory_order_relaxed); - old_buckets[bid] = {bucket, ts}; - old_bucket_generations[bid] = bucket->generation_; - - for (size_t i = 0; i < bucket->keys.size(); ++i) { - const auto& key = bucket->keys[i]; - auto map_it = object_bucket_map_.find(key); - if (map_it != object_bucket_map_.end() && - map_it->second.bucket_id == bid) { - live_keys_info.push_back( - {key, bid, bucket->metadatas[i]}); + SharedMutexLocker lock(&mutex_, shared_lock); + for (size_t i = 0; i < tasks.size(); ++i) { + const auto& task = tasks[i]; + auto live_it = object_bucket_map_.find(task.key); + if (live_it != object_bucket_map_.end()) { + if (live_it->second.object_incarnation != + task.object_incarnation) { + results[i] = {task.task_id, + LocalDeleteResult::kStaleVersion, + ErrorCode::OK}; + continue; } + by_bucket[live_it->second.bucket_id].push_back(i); + continue; } + if (task.expected_bucket_id < 0) { + results[i] = {task.task_id, LocalDeleteResult::kStaleVersion, + ErrorCode::OK}; + continue; + } + by_bucket[task.expected_bucket_id].push_back(i); } } - if (old_buckets.empty()) return true; - - auto reset_compacting = [&]() { - for (auto& [bid, pr] : old_buckets) { - pr.first->compacting_.store(false, std::memory_order_relaxed); + for (const auto& [bucket_id, indexes] : by_bucket) { + std::shared_ptr bucket; + { + SharedMutexLocker lock(&mutex_, shared_lock); + auto bucket_it = buckets_.find(bucket_id); + if (bucket_it != buckets_.end()) { + bucket = bucket_it->second; + } + } + if (!bucket) { + for (size_t index : indexes) { + results[index] = {tasks[index].task_id, + LocalDeleteResult::kStaleVersion, + ErrorCode::OK}; + } + continue; } - }; - // Identify buckets with zero live keys — delete them immediately - // without waiting for merge. Buckets with live keys participate in - // the merge. - std::set buckets_with_live; - for (const auto& info : live_keys_info) { - buckets_with_live.insert(info.old_bucket_id); - } + std::lock_guard operation_lock(bucket->operation_mutex_); + BucketMetadata updated; + std::vector changed_indexes; + int64_t newly_reclaimable = 0; + { + // Claim the bucket while holding the metadata lock. Eviction and + // GC skip claimed buckets, so the durable rewrite cannot race + // with file retirement. + SharedMutexLocker active_lock(&mutex_, shared_lock); + auto active_it = buckets_.find(bucket_id); + if (active_it == buckets_.end() || active_it->second != bucket) { + for (size_t index : indexes) { + results[index] = {tasks[index].task_id, + LocalDeleteResult::kRetryableFailure, + ErrorCode::INTERNAL_ERROR}; + } + // GC may have relocated the same incarnation into a + // replacement bucket after the initial resolution. Do not + // report StaleVersion here: the caller would ACK the task and + // lose the delete intent. Returning a retryable result makes + // the next fetch resolve object_bucket_map_ again. + continue; + } + bool expected = false; + if (!bucket->mutation_in_progress_.compare_exchange_strong( + expected, true, std::memory_order_acq_rel)) { + continue; + } + + updated = *bucket; + for (size_t index : indexes) { + const auto& task = tasks[index]; + auto key_it = std::find(updated.keys.begin(), + updated.keys.end(), task.key); + if (key_it == updated.keys.end()) { + results[index] = {task.task_id, + LocalDeleteResult::kStaleVersion, + ErrorCode::OK}; + continue; + } + const size_t object_index = static_cast( + std::distance(updated.keys.begin(), key_it)); + auto& object_meta = updated.metadatas[object_index]; + if (object_meta.object_incarnation != task.object_incarnation) { + results[index] = {task.task_id, + LocalDeleteResult::kStaleVersion, + ErrorCode::OK}; + continue; + } + if (object_meta.tombstoned) { + results[index] = {task.task_id, + LocalDeleteResult::kAlreadyRemoved, + ErrorCode::OK}; + continue; + } + object_meta.tombstoned = true; + newly_reclaimable += + object_meta.key_size + object_meta.data_size; + changed_indexes.push_back(index); + results[index] = {task.task_id, LocalDeleteResult::kRemoved, + ErrorCode::OK}; + } + if (changed_indexes.empty()) { + bucket->mutation_in_progress_.store(false, + std::memory_order_release); + continue; + } + } - std::vector empty_buckets; - for (auto& [bid, pr] : old_buckets) { - if (buckets_with_live.find(bid) == buckets_with_live.end()) { - empty_buckets.push_back(bid); + auto persist = StoreBucketMetadataAtomically(bucket_id, updated); + if (!persist) { + bucket->mutation_in_progress_.store(false, + std::memory_order_release); + for (size_t index : changed_indexes) { + results[index] = {tasks[index].task_id, + LocalDeleteResult::kRetryableFailure, + persist.error()}; + } + continue; } - } - if (!empty_buckets.empty()) { - std::vector> to_drain; { SharedMutexLocker lock(&mutex_); - for (int64_t bid : empty_buckets) { - auto it = buckets_.find(bid); - if (it != buckets_.end()) { - total_size_ -= - it->second->data_size + it->second->meta_size; - buckets_.erase(it); - int64_t ts = old_buckets[bid].second; - lru_index_.erase({ts, bid}); - to_drain.push_back(old_buckets[bid].first); + auto active_it = buckets_.find(bucket_id); + if (active_it == buckets_.end() || active_it->second != bucket) { + for (size_t index : changed_indexes) { + results[index] = {tasks[index].task_id, + LocalDeleteResult::kRetryableFailure, + ErrorCode::INTERNAL_ERROR}; } + bucket->mutation_in_progress_.store(false, + std::memory_order_release); + // The bucket was replaced while the metadata rewrite was in + // flight. The durable result cannot be associated with the + // active bucket, so leave all tasks pending for re-resolution + // instead of ACKing kRemoved against a stale bucket. + continue; } + total_size_ += updated.meta_size - bucket->meta_size; + reclaimable_bytes_.fetch_add(newly_reclaimable, + std::memory_order_relaxed); + *bucket = std::move(updated); + for (size_t index : changed_indexes) { + const auto& task = tasks[index]; + auto live_it = object_bucket_map_.find(task.key); + if (live_it != object_bucket_map_.end() && + live_it->second.bucket_id == bucket_id && + live_it->second.object_incarnation == + task.object_incarnation) { + object_bucket_map_.erase(live_it); + } + } + bucket->mutation_in_progress_.store(false, + std::memory_order_release); } - for (auto& bucket : to_drain) { - WaitForInflightReads(bucket); - } - for (int64_t bid : empty_buckets) { - DeleteBucketFiles(bid); - } - // Remove deleted buckets from old_buckets so they don't interfere - // with the merge logic below. - for (int64_t bid : empty_buckets) { - old_buckets.erase(bid); + } + + return results; +} + +int64_t BucketStorageBackend::GetReclaimableBytes() const { + return reclaimable_bytes_.load(std::memory_order_relaxed); +} + +int64_t BucketStorageBackend::BucketReclaimableBytes( + const BucketMetadata& bucket) { + int64_t bytes = 0; + for (const auto& metadata : bucket.metadatas) { + if (metadata.tombstoned) { + bytes += metadata.key_size + metadata.data_size; } } + return bytes; +} - // If no live keys remain (all buckets were empty), we're done. - if (live_keys_info.empty()) { - return true; +bool BucketStorageBackend::RequestGarbageCollection( + bool require_disk_pressure) { + if (!bucket_backend_config_.gc_enable) { + return false; } + if (require_disk_pressure) { + SharedMutexLocker lock(&mutex_, shared_lock); + const int64_t capacity = bucket_backend_config_.max_total_size; + const int64_t high_watermark = static_cast( + capacity * file_storage_config_.disk_eviction_high_watermark_ratio); + if (capacity <= 0 || total_size_ < high_watermark) { + return false; + } + } + { + std::lock_guard lock(gc_mutex_); + gc_requested_ = true; + } + gc_cv_.notify_one(); + return true; +} - // Step 2: group live keys by bucket_keys_limit / bucket_size_limit - // using metadata only (no file IO). Only the FIRST group that fills up - // will be written as a new bucket; remaining keys are deferred to the - // next round. - struct GroupedKey { - std::string key; - int64_t old_bucket_id; - BucketObjectMetadata meta; - int64_t total_size; // key_size + data_size - }; - std::vector first_group_keys; - int64_t group_count = 0; - int64_t group_size = 0; - - for (const auto& info : live_keys_info) { - int64_t key_total = - info.meta.key_size + info.meta.data_size; - if (group_count >= bucket_backend_config_.bucket_keys_limit || - (group_count > 0 && group_size + key_total > - bucket_backend_config_.bucket_size_limit)) { - break; // first group is full - } - first_group_keys.push_back( - {info.key, info.old_bucket_id, info.meta, key_total}); - group_size += key_total; - ++group_count; - } - - // Check if first group is full enough to write. - bool group_full = - (group_count >= bucket_backend_config_.bucket_keys_limit) || - (group_size >= bucket_backend_config_.bucket_size_limit); - if (!group_full && !space_pressure) { - // Not enough live keys to fill a bucket; defer to next round. - LOG(INFO) << "[GC] CompactBuckets deferred: group_count=" - << group_count - << " group_size=" << group_size - << " bucket_keys_limit=" - << bucket_backend_config_.bucket_keys_limit - << " bucket_size_limit=" - << bucket_backend_config_.bucket_size_limit - << " total_live_keys=" << live_keys_info.size(); - reset_compacting(); - return true; +tl::expected BucketStorageBackend::SyncStorageDirectory() + const { + const int fd = ::open(storage_path_.c_str(), O_RDONLY | O_DIRECTORY); + if (fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard directory(fd); + if (::fsync(directory.get()) != 0) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } + return {}; +} - // Step 3: read ONLY the first group's live key data from old buckets - // (lock-free IO). Open each old bucket file once, read only the keys - // that are in the first group. - std::unordered_map live_data_buffers; - std::unordered_map> first_group; +tl::expected BucketStorageBackend::StoreGcIntent( + const BucketGcIntent& intent) { + namespace fs = std::filesystem; + const fs::path path = fs::path(storage_path_) / ".bucket_gc_intent"; + const fs::path temp = + path.string() + ".tmp." + UuidToString(generate_uuid()); + std::string bytes; + struct_pb::to_pb(intent, bytes); - // Group first_group_keys by old_bucket_id to open each file once. - std::unordered_map> keys_by_bucket; - for (const auto& gk : first_group_keys) { - keys_by_bucket[gk.old_bucket_id].push_back(&gk); + const int fd = ::open(temp.c_str(), O_WRONLY | O_CREAT | O_EXCL, 0644); + if (fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard file(fd); + size_t written = 0; + while (written < bytes.size()) { + const ssize_t n = + ::write(file.get(), bytes.data() + written, bytes.size() - written); + if (n < 0 && errno == EINTR) { + continue; + } + if (n <= 0) { + fs::remove(temp); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + written += static_cast(n); + } + if (::fsync(file.get()) != 0 || ::close(file.release()) != 0) { + fs::remove(temp); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + if (::rename(temp.c_str(), path.c_str()) != 0) { + fs::remove(temp); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } + return SyncStorageDirectory(); +} - for (auto& [bid, keys_ptr] : keys_by_bucket) { - auto& old_bucket = old_buckets[bid].first; - bool read_ok = true; - { - BucketReadGuard guard(old_bucket); - auto data_path = GetBucketDataPath(bid); - if (!data_path) { - read_ok = false; - } else { - auto file_result = - OpenFile(data_path.value(), FileMode::Read); - if (!file_result) { - read_ok = false; - } else { - auto& file = file_result.value(); - for (const auto* gk : keys_ptr) { - std::string data; - data.resize(gk->meta.data_size); - int64_t actual_offset = - gk->meta.offset + gk->meta.key_size; - iovec iov{data.data(), - static_cast(gk->meta.data_size)}; - auto read_res = - file->vector_read(&iov, 1, actual_offset); - if (!read_res || - read_res.value() != - static_cast(gk->meta.data_size)) { - LOG(ERROR) - << "CompactBuckets: read failed for key: " - << gk->key << ", bucket_id=" << bid; - read_ok = false; - break; - } - live_data_buffers[gk->key] = std::move(data); - first_group.emplace( - gk->key, - std::vector{Slice{ - live_data_buffers[gk->key].data(), - live_data_buffers[gk->key].size()}}); - } - } - } - } // guard released +tl::expected BucketStorageBackend::RemoveGcIntent() { + namespace fs = std::filesystem; + const fs::path path = fs::path(storage_path_) / ".bucket_gc_intent"; + std::error_code ec; + fs::remove(path, ec); + if (ec && ec != std::errc::no_such_file_or_directory) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + return SyncStorageDirectory(); +} - if (!read_ok) { - reset_compacting(); - return false; +tl::expected BucketStorageBackend::RecoverGarbageCollection() { + namespace fs = std::filesystem; + const fs::path path = fs::path(storage_path_) / ".bucket_gc_intent"; + std::error_code ec; + if (!fs::exists(path, ec)) { + if (ec) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); } + return {}; } - // Step 4: write the first group as a new bucket. - int64_t new_bucket_id = bucket_id_generator_->NextId(); - std::vector iovs; - std::vector new_metas; - auto build_result = - BuildBucket(new_bucket_id, first_group, iovs, new_metas); - if (!build_result) { - LOG(ERROR) << "CompactBuckets: BuildBucket failed"; - reset_compacting(); - return false; + const int fd = ::open(path.c_str(), O_RDONLY); + if (fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); } - auto write_result = - WriteBucket(new_bucket_id, build_result.value(), iovs); - if (!write_result) { - LOG(ERROR) << "CompactBuckets: WriteBucket failed"; - reset_compacting(); - return false; + FdGuard file(fd); + struct stat stat_buf{}; + if (::fstat(file.get(), &stat_buf) != 0 || stat_buf.st_size <= 0) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + std::string bytes(static_cast(stat_buf.st_size), '\0'); + size_t read_bytes = 0; + while (read_bytes < bytes.size()) { + const ssize_t n = ::read(file.get(), bytes.data() + read_bytes, + bytes.size() - read_bytes); + if (n < 0 && errno == EINTR) { + continue; + } + if (n <= 0) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + read_bytes += static_cast(n); } - // Step 5: atomic swap under lock with re-validation. - auto& new_bucket = build_result.value(); - // Determine which old buckets have ALL their live keys migrated. - // A key is "migrated" if it's in the first_group AND still maps to an - // old bucket being compacted. Old buckets with no remaining live keys - // (all migrated or removed) can be deleted. - std::set old_bucket_ids_set(bucket_ids.begin(), - bucket_ids.end()); - // Use the coldest last_access_ns among old buckets for the new bucket. - int64_t new_last_access_ns = std::numeric_limits::max(); - for (auto& [bid, pr] : old_buckets) { - new_last_access_ns = std::min(new_last_access_ns, pr.second); + BucketGcIntent intent; + try { + struct_pb::from_pb(intent, bytes); + } catch (const std::exception& e) { + LOG(ERROR) << "Failed to decode bucket GC intent: " << e.what(); + return tl::unexpected(ErrorCode::FILE_READ_FAIL); } - if (new_last_access_ns == std::numeric_limits::max()) { - new_last_access_ns = 0; + if (intent.version != 1 || intent.target_bucket_id < -1 || + intent.source_bucket_ids.empty() || + intent.source_bucket_ids.size() > kMaxGcSources) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + std::unordered_set source_ids; + for (int64_t source_id : intent.source_bucket_ids) { + if (source_id < 0 || source_id == intent.target_bucket_id || + !source_ids.insert(source_id).second) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } } - std::vector buckets_to_delete; - { - SharedMutexLocker lock(&mutex_); - for (const auto& [bid, pr] : old_buckets) { - auto current_it = buckets_.find(bid); - auto generation_it = old_bucket_generations.find(bid); - if (current_it == buckets_.end() || - generation_it == old_bucket_generations.end() || - current_it->second->generation_ != generation_it->second) { - // A delete changed the source bucket after the snapshot. The - // data file was built from a stale view, so do not publish it - // or replace any object mappings with it. - lock.unlock(); - CleanupOrphanedBucket(new_bucket_id); - reset_compacting(); - LOG(INFO) << "CompactBuckets discarded stale snapshot for " - << "bucket_id=" << bid; - return true; - } - } - // Re-validate and remap each key in the new bucket. - for (size_t i = 0; i < new_bucket->keys.size(); ++i) { - const auto& key = new_bucket->keys[i]; - auto map_it = object_bucket_map_.find(key); - if (map_it != object_bucket_map_.end() && - old_bucket_ids_set.count(map_it->second.bucket_id)) { - // Still live at an old bucket -> remap to new bucket. - map_it->second = new_metas[i]; - } - // else: key was removed or remapped -> skip. - } - - // Insert new bucket. - total_size_ += new_bucket->data_size + new_bucket->meta_size; - new_bucket->last_access_ns_.store( - new_last_access_ns, std::memory_order_relaxed); - buckets_.emplace(new_bucket_id, new_bucket); - lru_index_.emplace(new_last_access_ns, new_bucket_id); - - // Determine which old buckets can be deleted: those whose live keys - // are all now absent from object_bucket_map_ or remapped to the new - // bucket (i.e., no key still points at the old bucket). - for (auto& [bid, pr] : old_buckets) { - auto& old_bucket = pr.first; - bool has_remaining_live = false; - for (const auto& key : old_bucket->keys) { - auto map_it = object_bucket_map_.find(key); - if (map_it != object_bucket_map_.end() && - map_it->second.bucket_id == bid) { - // This key is still live at the old bucket — it was not - // in the first group (deferred to next round). - has_remaining_live = true; - break; - } + if (!intent.committed) { + if (intent.target_bucket_id >= 0) { + if (!CleanupOrphanedBucket(intent.target_bucket_id)) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } - if (!has_remaining_live) { - buckets_to_delete.push_back(bid); - } else { - // Reset compacting_ so this bucket can be compacted later. - old_bucket->compacting_.store(false, - std::memory_order_relaxed); + } + } else { + if (intent.target_bucket_id >= 0) { + auto target_meta = GetBucketMetadataPath(intent.target_bucket_id); + auto target_data = GetBucketDataPath(intent.target_bucket_id); + if (!target_meta || !target_data || + !fs::is_regular_file(target_meta.value(), ec) || ec || + !fs::is_regular_file(target_data.value(), ec) || ec) { + LOG(ERROR) << "Committed GC replacement is incomplete: " + << intent.target_bucket_id; + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + auto replacement = std::make_shared(); + auto loaded = + LoadBucketMetadata(intent.target_bucket_id, replacement); + const auto data_size = fs::file_size(target_data.value(), ec); + if (!loaded || ec || replacement->keys.empty() || + replacement->keys.size() != replacement->metadatas.size() || + data_size < static_cast(replacement->data_size)) { + LOG(ERROR) << "Committed GC replacement is invalid: " + << intent.target_bucket_id; + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + std::unordered_set unique_keys; + for (size_t i = 0; i < replacement->keys.size(); ++i) { + const auto& object = replacement->metadatas[i]; + if (object.tombstoned || object.offset < 0 || + object.key_size < 0 || object.data_size < 0 || + object.offset > replacement->data_size - object.key_size || + object.offset + object.key_size > + replacement->data_size - object.data_size || + !unique_keys.insert(replacement->keys[i]).second) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } } } - - // Remove old buckets that are fully migrated. - for (int64_t bid : buckets_to_delete) { - auto it = buckets_.find(bid); - if (it != buckets_.end()) { - total_size_ -= - it->second->data_size + it->second->meta_size; - buckets_.erase(it); - int64_t ts = old_buckets[bid].second; - lru_index_.erase({ts, bid}); + for (int64_t source_id : intent.source_bucket_ids) { + if (!CleanupOrphanedBucket(source_id)) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); } } } + return RemoveGcIntent(); +} - // Step 6: wait for inflight reads + delete old bucket files. - for (int64_t bid : buckets_to_delete) { - WaitForInflightReads(old_buckets[bid].first); - DeleteBucketFiles(bid); +std::vector +BucketStorageBackend::SelectGcCandidates(bool under_pressure) { + std::vector candidates; + SharedMutexLocker lock(&mutex_, shared_lock); + for (const auto& [bucket_id, bucket] : buckets_) { + if (bucket->mutation_in_progress_.load(std::memory_order_acquire)) { + continue; + } + int64_t live_bytes = 0; + int64_t reclaimable_bytes = 0; + size_t live_keys = 0; + for (const auto& metadata : bucket->metadatas) { + const int64_t bytes = metadata.key_size + metadata.data_size; + if (metadata.tombstoned) { + reclaimable_bytes += bytes; + } else { + live_bytes += bytes; + ++live_keys; + } + } + if (reclaimable_bytes == 0) { + continue; + } + const double deleted_ratio = + static_cast(reclaimable_bytes) / + static_cast(live_bytes + reclaimable_bytes); + if (!under_pressure && + deleted_ratio < bucket_backend_config_.gc_deleted_ratio) { + continue; + } + candidates.push_back( + {bucket_id, bucket, live_bytes, reclaimable_bytes, live_keys}); } - - return true; + std::sort( + candidates.begin(), candidates.end(), + [](const GcCandidate& lhs, const GcCandidate& rhs) { + if ((lhs.live_keys == 0) != (rhs.live_keys == 0)) { + return lhs.live_keys == 0; + } + const int64_t lhs_total = lhs.live_bytes + lhs.reclaimable_bytes; + const int64_t rhs_total = rhs.live_bytes + rhs.reclaimable_bytes; + const double lhs_ratio = + static_cast(lhs.reclaimable_bytes) / lhs_total; + const double rhs_ratio = + static_cast(rhs.reclaimable_bytes) / rhs_total; + if (lhs_ratio != rhs_ratio) { + return lhs_ratio > rhs_ratio; + } + if (lhs.reclaimable_bytes != rhs.reclaimable_bytes) { + return lhs.reclaimable_bytes > rhs.reclaimable_bytes; + } + return lhs.bucket_id < rhs.bucket_id; + }); + return candidates; } -void BucketStorageBackend::WaitForInflightReads( - std::shared_ptr bucket) { - constexpr int kMaxSpinIterations = 1000; - constexpr auto kMaxWaitTime = std::chrono::seconds(10); - int spin_count = 0; - auto wait_start = std::chrono::steady_clock::now(); - while (bucket->inflight_reads_.load(std::memory_order_acquire) > 0) { - if (++spin_count > kMaxSpinIterations) { - std::this_thread::yield(); - spin_count = 0; - if (std::chrono::steady_clock::now() - wait_start > - kMaxWaitTime) { - LOG(ERROR) << "CompactBucket: timed out waiting for " - "in-flight reads, inflight_reads=" - << bucket->inflight_reads_.load( - std::memory_order_relaxed); - break; +tl::expected, ErrorCode> +BucketStorageBackend::WriteGcReplacement( + int64_t target_bucket_id, const std::vector& sources) { + namespace fs = std::filesystem; + auto target_path_result = GetBucketDataPath(target_bucket_id); + if (!target_path_result) { + return tl::unexpected(target_path_result.error()); + } + const std::string& target_path = target_path_result.value(); + const int target_fd = + ::open(target_path.c_str(), O_WRONLY | O_CREAT | O_EXCL, 0644); + if (target_fd < 0) { + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard target_file(target_fd); + auto replacement = std::make_shared(); + std::vector buffer(1024 * 1024); + int64_t output_offset = 0; + + for (const auto& source : sources) { + auto source_path_result = GetBucketDataPath(source.bucket_id); + if (!source_path_result) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(source_path_result.error()); + } + const int source_fd = + ::open(source_path_result.value().c_str(), O_RDONLY); + if (source_fd < 0) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(ErrorCode::FILE_OPEN_FAIL); + } + FdGuard source_file(source_fd); + for (size_t i = 0; i < source.bucket->metadatas.size(); ++i) { + const auto& metadata = source.bucket->metadatas[i]; + if (metadata.tombstoned) { + continue; } - } else { - PAUSE(); + const int64_t record_size = metadata.key_size + metadata.data_size; + int64_t copied = 0; + while (copied < record_size) { + if (gc_stop_.load(std::memory_order_acquire)) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected( + ErrorCode::UNAVAILABLE_IN_CURRENT_STATUS); + } + const size_t chunk = static_cast( + std::min(record_size - copied, buffer.size())); + ssize_t n = 0; + do { + n = ::pread(source_file.get(), buffer.data(), chunk, + metadata.offset + copied); + } while (n < 0 && errno == EINTR); + if (n <= 0) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(ErrorCode::FILE_READ_FAIL); + } + size_t written = 0; + while (written < static_cast(n)) { + ssize_t out = ::pwrite( + target_file.get(), buffer.data() + written, + static_cast(n) - written, + output_offset + copied + static_cast(written)); + if (out < 0 && errno == EINTR) { + continue; + } + if (out <= 0) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + written += static_cast(out); + } + copied += n; + } + replacement->keys.push_back(source.bucket->keys[i]); + replacement->metadatas.push_back( + {output_offset, metadata.key_size, metadata.data_size, + metadata.object_incarnation, false}); + output_offset += record_size; } } + replacement->data_size = output_offset; + if (::fsync(target_file.get()) != 0 || + ::close(target_file.release()) != 0) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + auto stored = StoreBucketMetadataAtomically(target_bucket_id, *replacement); + if (!stored) { + CleanupOrphanedBucket(target_bucket_id); + return tl::unexpected(stored.error()); + } + return replacement; } -void BucketStorageBackend::DeleteBucketFiles(int64_t bucket_id) { +bool BucketStorageBackend::FinalizeGcSource(const GcCandidate& source) { namespace fs = std::filesystem; std::error_code ec; - auto data_path = GetBucketDataPath(bucket_id); - if (data_path) { - { - MutexLocker cache_locker(&file_cache_mutex_); - file_cache_.erase(data_path.value()); + auto meta_path = GetBucketMetadataPath(source.bucket_id); + if (!meta_path) { + return false; + } + fs::remove(meta_path.value(), ec); + if (ec && ec != std::errc::no_such_file_or_directory) { + return false; + } + if (!SyncStorageDirectory()) { + return false; + } + + constexpr auto kReaderTimeout = std::chrono::seconds(10); + const auto deadline = std::chrono::steady_clock::now() + kReaderTimeout; + while (source.bucket->inflight_reads_.load(std::memory_order_acquire) > 0) { + if (gc_stop_.load(std::memory_order_acquire)) { + return false; } - fs::remove(data_path.value(), ec); - if (ec && ec != std::errc::no_such_file_or_directory) { - LOG(ERROR) << "CompactBucket: failed to remove data file: " - << data_path.value() << ", error: " << ec.message(); + if (std::chrono::steady_clock::now() >= deadline) { + return false; } + std::this_thread::sleep_for(std::chrono::microseconds(100)); } - auto meta_path = GetBucketMetadataPath(bucket_id); - if (meta_path) { - ec.clear(); - fs::remove(meta_path.value(), ec); - if (ec && ec != std::errc::no_such_file_or_directory) { - LOG(ERROR) << "CompactBucket: failed to remove meta file: " - << meta_path.value() << ", error: " << ec.message(); - } + + auto data_path = GetBucketDataPath(source.bucket_id); + if (!data_path) { + return false; + } + { + MutexLocker cache_locker(&file_cache_mutex_); + file_cache_.erase(data_path.value()); + } + ec.clear(); + fs::remove(data_path.value(), ec); + if (ec && ec != std::errc::no_such_file_or_directory) { + return false; + } + if (!SyncStorageDirectory()) { + return false; + } + { + SharedMutexLocker lock(&mutex_); + total_size_ = + std::max(0, total_size_ - source.bucket->data_size - + source.bucket->meta_size); } + return true; } -// GCThreadFunc: background tombstone compaction loop. -void BucketStorageBackend::GCThreadFunc() { - LOG(INFO) << "[GC] background compaction thread started"; - while (gc_running_.load(std::memory_order_acquire)) { - // Sleep for gc_interval_ms or until woken for shutdown. - { - std::unique_lock lock(gc_mutex_); - gc_cv_.wait_for( - lock, - std::chrono::milliseconds( - bucket_backend_config_.gc_interval_ms), - [this]() { - return !gc_running_.load(std::memory_order_relaxed); - }); +tl::expected BucketStorageBackend::RunGarbageCollectionOnce( + bool under_pressure) { + namespace fs = std::filesystem; + const fs::path intent_path = fs::path(storage_path_) / ".bucket_gc_intent"; + std::error_code ec; + if (fs::exists(intent_path, ec) || ec) { + if (ec) { + return tl::unexpected(ErrorCode::FILE_READ_FAIL); } - if (!gc_running_.load(std::memory_order_acquire)) break; + return false; + } - if (!bucket_backend_config_.gc_enable) continue; + auto candidates = SelectGcCandidates(under_pressure); + if (candidates.empty()) { + return false; + } - // Check space pressure under shared lock (total_size_ is - // GUARDED_BY(mutex_)). - bool space_pressure = false; - { - SharedMutexLocker lock(&mutex_, shared_lock); - if (bucket_backend_config_.max_total_size > 0) { - double used_ratio = - static_cast(total_size_) / - static_cast( - bucket_backend_config_.max_total_size); - space_pressure = used_ratio >= - bucket_backend_config_ - .gc_high_watermark_ratio; + std::vector sources; + int64_t live_bytes = 0; + size_t live_keys = 0; + for (const auto& candidate : candidates) { + if (candidate.live_keys == 0) { + if (!sources.empty() && sources.front().live_keys != 0) { + break; } + sources.push_back(candidate); + if (sources.size() == kMaxGcSources) { + break; + } + continue; + } + if (!sources.empty() && sources.front().live_keys == 0) { + break; + } + if (sources.size() == kMaxGcSources || + live_bytes + candidate.live_bytes > + bucket_backend_config_.bucket_size_limit || + live_keys + candidate.live_keys > + static_cast(bucket_backend_config_.bucket_keys_limit)) { + continue; } + sources.push_back(candidate); + live_bytes += candidate.live_bytes; + live_keys += candidate.live_keys; + } + if (sources.empty()) { + return false; + } + std::sort(sources.begin(), sources.end(), + [](const GcCandidate& lhs, const GcCandidate& rhs) { + return lhs.bucket_id < rhs.bucket_id; + }); - // Collect GC candidate buckets (up to gc_max_buckets_per_round). - std::vector candidates; + std::vector> operation_locks; + operation_locks.reserve(sources.size()); + for (const auto& source : sources) { + operation_locks.emplace_back(source.bucket->operation_mutex_); + } + + { + SharedMutexLocker lock(&mutex_); + for (const auto& source : sources) { + auto active = buckets_.find(source.bucket_id); + if (active == buckets_.end() || active->second != source.bucket || + source.bucket->mutation_in_progress_.load( + std::memory_order_relaxed)) { + return false; + } + } + for (auto& source : sources) { + source.live_bytes = 0; + source.reclaimable_bytes = 0; + source.live_keys = 0; + for (const auto& metadata : source.bucket->metadatas) { + const int64_t bytes = metadata.key_size + metadata.data_size; + if (metadata.tombstoned) { + source.reclaimable_bytes += bytes; + } else { + source.live_bytes += bytes; + ++source.live_keys; + } + } + source.bucket->mutation_in_progress_.store( + true, std::memory_order_release); + } + } + auto clear_mutation = [&] { + SharedMutexLocker lock(&mutex_); + for (const auto& source : sources) { + auto active = buckets_.find(source.bucket_id); + if (active != buckets_.end() && active->second == source.bucket) { + source.bucket->mutation_in_progress_.store( + false, std::memory_order_release); + } + } + }; + ScopeExit mutation_guard(clear_mutation); + + const bool fully_dead = std::all_of( + sources.begin(), sources.end(), + [](const GcCandidate& source) { return source.live_keys == 0; }); + const int64_t target_bucket_id = + fully_dead ? -1 : bucket_id_generator_->NextId(); + BucketGcIntent intent{ + .version = 1, .committed = false, .target_bucket_id = target_bucket_id}; + for (const auto& source : sources) { + intent.source_bucket_ids.push_back(source.bucket_id); + } + auto prepared = StoreGcIntent(intent); + if (!prepared) { + return tl::unexpected(prepared.error()); + } + + std::shared_ptr replacement; + int64_t replacement_size = 0; + auto rollback_prepared = [&]() -> tl::expected { + if (target_bucket_id >= 0 && !CleanupOrphanedBucket(target_bucket_id)) { + return tl::unexpected(ErrorCode::FILE_WRITE_FAIL); + } + if (replacement_size > 0) { + SharedMutexLocker lock(&mutex_); + total_size_ = std::max(0, total_size_ - replacement_size); + } + return RemoveGcIntent(); + }; + if (!fully_dead) { + auto written = WriteGcReplacement(target_bucket_id, sources); + if (!written) { + rollback_prepared(); + return tl::unexpected(written.error()); + } + replacement = std::move(written.value()); + replacement_size = replacement->data_size + replacement->meta_size; { SharedMutexLocker lock(&mutex_); - int64_t count = 0; - for (auto it = buckets_.begin(); - it != buckets_.end() && - count < bucket_backend_config_.gc_max_buckets_per_round; - ++it) { - int64_t deleted = - it->second->deleted_bytes_.load( - std::memory_order_relaxed); - if (deleted <= 0) continue; - if (it->second->compacting_.load( - std::memory_order_relaxed)) - continue; - int64_t data_size = it->second->data_size; - double ratio = - (data_size > 0) - ? static_cast(deleted) / - static_cast(data_size) - : 0.0; - if (!space_pressure && - ratio < bucket_backend_config_.gc_deleted_ratio) { + total_size_ += replacement_size; + } + } + + std::unordered_map + staged_object_mappings; + std::map> staged_bucket; + std::set> staged_lru; + if (replacement) { + try { + staged_object_mappings.reserve(replacement->keys.size()); + for (size_t i = 0; i < replacement->keys.size(); ++i) { + const auto& metadata = replacement->metadatas[i]; + auto [_, inserted] = staged_object_mappings.emplace( + replacement->keys[i], + StorageObjectMetadata{target_bucket_id, metadata.offset, + metadata.key_size, metadata.data_size, + "", metadata.object_incarnation}); + if (!inserted) { + rollback_prepared(); + return tl::unexpected(ErrorCode::INTERNAL_ERROR); + } + } + staged_bucket.emplace(target_bucket_id, replacement); + staged_lru.emplace(0, target_bucket_id); + } catch (const std::exception&) { + rollback_prepared(); + return tl::unexpected(ErrorCode::INTERNAL_ERROR); + } + } + { + SharedMutexLocker lock(&mutex_, shared_lock); + for (const auto& source : sources) { + for (size_t i = 0; i < source.bucket->keys.size(); ++i) { + const auto& metadata = source.bucket->metadatas[i]; + if (metadata.tombstoned) { continue; } - candidates.push_back(it->first); - ++count; + auto object = object_bucket_map_.find(source.bucket->keys[i]); + if (object == object_bucket_map_.end() || + object->second.bucket_id != source.bucket_id || + object->second.object_incarnation != + metadata.object_incarnation) { + lock.unlock(); + rollback_prepared(); + return tl::unexpected(ErrorCode::INTERNAL_ERROR); + } } } + } - if (!candidates.empty()) { - if (bucket_backend_config_.gc_merge_enable && - candidates.size() > 1) { - // Cross-bucket merge: collect live keys from multiple - // tombstone buckets into one new bucket. - if (CompactBuckets(candidates, space_pressure)) { - LOG(INFO) << "[GC] merged " << candidates.size() - << " bucket(s)"; - } else { - LOG(WARNING) << "[GC] CompactBuckets failed for " - << candidates.size() - << " bucket(s), will retry next round"; - } - } else { - // Single-bucket compaction (one at a time). - int64_t compacted = 0; - for (int64_t bid : candidates) { - if (CompactBuckets({bid}, space_pressure)) { - ++compacted; - } else { - LOG(WARNING) - << "[GC] CompactBuckets failed for bucket " - << bid << ", will retry next round"; - break; - } + intent.committed = true; + auto committed = StoreGcIntent(intent); + if (!committed) { + rollback_prepared(); + return tl::unexpected(committed.error()); + } + + { + SharedMutexLocker lock(&mutex_); + for (const auto& source : sources) { + reclaimable_bytes_.fetch_sub(source.reclaimable_bytes, + std::memory_order_relaxed); + for (size_t i = 0; i < source.bucket->keys.size(); ++i) { + const auto& key = source.bucket->keys[i]; + auto object = object_bucket_map_.find(key); + if (object != object_bucket_map_.end() && + object->second.bucket_id == source.bucket_id) { + object_bucket_map_.erase(object); } - if (compacted > 0) { - LOG(INFO) << "[GC] compacted " << compacted - << " bucket(s)"; + } + for (auto it = lru_index_.begin(); it != lru_index_.end();) { + if (it->second == source.bucket_id) { + it = lru_index_.erase(it); + } else { + ++it; } } + buckets_.erase(source.bucket_id); + } + if (replacement) { + object_bucket_map_.merge(staged_object_mappings); + buckets_.merge(staged_bucket); + lru_index_.merge(staged_lru); } } - LOG(INFO) << "[GC] background compaction thread stopped"; -} -tl::expected BucketStorageBackend::StoreBucketMetadata( - int64_t id, std::shared_ptr metadata) { - auto meta_path_res = GetBucketMetadataPath(id); - if (!meta_path_res) { - LOG(ERROR) << "Failed to get bucket metadata path, bucket_id=" << id; - return tl::make_unexpected(ErrorCode::INTERNAL_ERROR); - } - auto meta_path = meta_path_res.value(); - auto open_file_result = OpenFile(meta_path, FileMode::Write); - if (!open_file_result) { - LOG(ERROR) << "Failed to open file for bucket writing: " << meta_path; - return tl::make_unexpected(ErrorCode::FILE_OPEN_FAIL); + bool all_removed = true; + int64_t reclaimed = 0; + for (const auto& source : sources) { + if (FinalizeGcSource(source)) { + reclaimed += source.bucket->data_size + source.bucket->meta_size; + } else { + all_removed = false; + } } - auto file = std::move(open_file_result.value()); - std::string str; - struct_pb::to_pb(*metadata, str); - auto write_result = file->write(str, str.size()); - if (!write_result) { - LOG(ERROR) << "Write failed for: " << meta_path - << ", error: " << write_result.error(); - return tl::make_unexpected(write_result.error()); + if (all_removed) { + auto removed = RemoveGcIntent(); + if (!removed) { + return tl::unexpected(removed.error()); + } } - if (write_result.value() != str.size()) { - LOG(ERROR) << "Write size mismatch for: " << meta_path - << ", expected: " << str.size() - << ", got: " << write_result.value(); - return tl::make_unexpected(ErrorCode::FILE_WRITE_FAIL); + LOG(INFO) << "Bucket GC compacted " << sources.size() + << " source bucket(s), replacement=" << target_bucket_id + << ", reclaimed_bytes=" << reclaimed; + return true; +} + +void BucketStorageBackend::GarbageCollectionThreadFunc() { + const auto interval = + std::chrono::seconds(bucket_backend_config_.gc_interval_seconds); + while (true) { + { + std::unique_lock lock(gc_mutex_); + gc_cv_.wait_for(lock, interval, [this] { + return gc_stop_.load(std::memory_order_acquire) || + gc_requested_; + }); + if (gc_stop_.load(std::memory_order_acquire)) { + return; + } + gc_requested_ = false; + } + + bool under_pressure = false; + int64_t low_watermark = 0; + { + SharedMutexLocker lock(&mutex_, shared_lock); + const int64_t capacity = bucket_backend_config_.max_total_size; + under_pressure = + capacity > 0 && + total_size_ >= + static_cast( + capacity * file_storage_config_ + .disk_eviction_high_watermark_ratio); + low_watermark = static_cast( + capacity * + file_storage_config_.disk_eviction_low_watermark_ratio); + } + + while (true) { + tl::expected result = + tl::unexpected(ErrorCode::INTERNAL_ERROR); + try { + result = RunGarbageCollectionOnce(under_pressure); + } catch (const std::exception& e) { + LOG(ERROR) << "Bucket GC aborted by exception: " << e.what(); + break; + } catch (...) { + LOG(ERROR) << "Bucket GC aborted by unknown exception"; + break; + } + if (!result) { + LOG(WARNING) + << "Bucket GC failed: " << toString(result.error()); + break; + } + if (gc_stop_.load(std::memory_order_acquire)) { + return; + } + if (!result.value() || !under_pressure) { + break; + } + SharedMutexLocker lock(&mutex_, shared_lock); + if (total_size_ <= low_watermark) { + break; + } + } } - metadata->meta_size = str.size(); - return {}; } tl::expected BucketStorageBackend::LoadBucketMetadata( @@ -5765,9 +6142,12 @@ tl::expected OffsetAllocatorStorageBackend::BatchOffload( keys.push_back(key); metadatas.push_back( - StorageObjectMetadata{0, static_cast(offset), + StorageObjectMetadata{0, + static_cast(offset), static_cast(header.key_len), - static_cast(value_size), ""}); + static_cast(value_size), + "", + {}}); } // ---- Post-loop flush: notify master of any evicted keys that @@ -5887,8 +6267,6 @@ tl::expected OffsetAllocatorStorageBackend::BatchOffload( tl::expected OffsetAllocatorStorageBackend::BatchLoad( std::unordered_map& batched_slices) { - auto* stats = CurrentStorageReadStats(); - const auto plan_start = std::chrono::steady_clock::now(); if (!initialized_.load(std::memory_order_acquire)) { LOG(ERROR) << "Storage backend is not initialized. Call Init() before use."; @@ -5948,15 +6326,7 @@ tl::expected OffsetAllocatorStorageBackend::BatchLoad( // Step 2: Perform disk I/O without holding any locks // Allocations and data file are kept alive by shared_ptr refs in read_plans - if (stats) { - stats->plan_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - plan_start) - .count(); - stats->io_mode = "preadv"; - } for (const auto& plan : read_plans) { - const auto disk_start = std::chrono::steady_clock::now(); // Read header first. The CRC is NOT verified here: records are // CRC-validated once during recovery, and during normal operation // a key only becomes visible after its write completed, so the @@ -6024,17 +6394,6 @@ tl::expected OffsetAllocatorStorageBackend::BatchLoad( << ", got: " << read_value_result.value(); return tl::make_unexpected(ErrorCode::FILE_READ_FAIL); } - if (stats) { - const auto read_us = - std::chrono::duration_cast( - std::chrono::steady_clock::now() - disk_start) - .count(); - stats->disk_read_us += read_us; - if (read_us > stats->slowest_disk_read_us) { - stats->slowest_disk_read_us = read_us; - stats->slowest_key = plan.key; - } - } } // read_plans destructor releases all AllocationPtr references @@ -6147,7 +6506,9 @@ tl::expected OffsetAllocatorStorageBackend::ScanMeta( // contains header and alignment padding, so it cannot // be derived arithmetically. static_cast(key.size()), - static_cast(entry.value_size), ""}); + static_cast(entry.value_size), + "", + {}}); // Call handler when batch limit is reached if (static_cast(keys.size()) >= diff --git a/mooncake-store/tests/ha/oplog/oplog_applier_test.cpp b/mooncake-store/tests/ha/oplog/oplog_applier_test.cpp index 9bbc975230..b83efae865 100644 --- a/mooncake-store/tests/ha/oplog/oplog_applier_test.cpp +++ b/mooncake-store/tests/ha/oplog/oplog_applier_test.cpp @@ -51,6 +51,17 @@ std::string MakeValidPayload(uint64_t client_id_first = 1, return std::string(result.begin(), result.end()); } +LocalDeleteTask MakeLocalDeleteTask(const std::string& key) { + return LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = key, + .object_incarnation = {11, 12}, + .expected_bucket_id = 7, + }; +} + class OpLogApplierTest : public ::testing::Test { protected: void SetUp() override { @@ -110,6 +121,67 @@ TEST_F(OpLogApplierTest, TestApplyRemove) { EXPECT_FALSE(mock_metadata_store_->Exists("key1")); } +TEST_F(OpLogApplierTest, RemoveAndAckReplicateLocalDeleteIntent) { + const auto task = MakeLocalDeleteTask("key1"); + LocalDeleteRemovePayloadV1 remove_payload{ + .schema_version = 1, + .object_incarnation = task.object_incarnation, + .delete_intents = {task}, + }; + const auto remove_bytes = struct_pack::serialize(remove_payload); + const std::string remove_data(remove_bytes.begin(), remove_bytes.end()); + + ASSERT_TRUE(applier_->ApplyOpLogEntry( + MakeEntry(1, OpType::PUT_END, "key1", MakeValidPayload()))); + EXPECT_TRUE(applier_->ApplyOpLogEntry( + MakeEntry(2, OpType::REMOVE, "key1", remove_data))); + EXPECT_FALSE(mock_metadata_store_->Exists("key1")); + ASSERT_EQ(mock_metadata_store_->SnapshotLocalDeleteTasks().size(), 1); + EXPECT_EQ(mock_metadata_store_->SnapshotLocalDeleteTasks().front(), task); + + LocalDeleteAckPayloadV1 ack_payload{ + .schema_version = 1, + .local_disk_segment_id = task.local_disk_segment_id, + .task_ids = {task.task_id}, + }; + const auto ack_bytes = struct_pack::serialize(ack_payload); + const std::string ack_data(ack_bytes.begin(), ack_bytes.end()); + EXPECT_TRUE(applier_->ApplyOpLogEntry( + MakeEntry(3, OpType::LOCAL_DELETE_ACK, "", ack_data))); + EXPECT_TRUE(mock_metadata_store_->SnapshotLocalDeleteTasks().empty()); +} + +TEST_F(OpLogApplierTest, InvalidDeleteIntentDoesNotAdvanceSequence) { + auto task = MakeLocalDeleteTask("another-key"); + LocalDeleteRemovePayloadV1 payload{ + .schema_version = 1, + .object_incarnation = task.object_incarnation, + .delete_intents = {task}, + }; + const auto bytes = struct_pack::serialize(payload); + const std::string data(bytes.begin(), bytes.end()); + + EXPECT_FALSE( + applier_->ApplyOpLogEntry(MakeEntry(1, OpType::REMOVE, "key1", data))); + EXPECT_EQ(applier_->GetExpectedSequenceId(), 1); + EXPECT_TRUE(mock_metadata_store_->SnapshotLocalDeleteTasks().empty()); +} + +TEST_F(OpLogApplierTest, UnknownLocalDeleteAckVersionDoesNotAdvanceSequence) { + const auto task = MakeLocalDeleteTask("key1"); + LocalDeleteAckPayloadV1 payload{ + .schema_version = 2, + .local_disk_segment_id = task.local_disk_segment_id, + .task_ids = {task.task_id}, + }; + const auto bytes = struct_pack::serialize(payload); + const std::string data(bytes.begin(), bytes.end()); + + EXPECT_FALSE(applier_->ApplyOpLogEntry( + MakeEntry(1, OpType::LOCAL_DELETE_ACK, "", data))); + EXPECT_EQ(applier_->GetExpectedSequenceId(), 1); +} + TEST_F(OpLogApplierTest, TestApplyOpLogEntry_InvalidOpType) { OpLogEntry entry = MakeEntry(1, OpType::PUT_END, "key1", MakeValidPayload()); diff --git a/mooncake-store/tests/local_delete_test.cpp b/mooncake-store/tests/local_delete_test.cpp new file mode 100644 index 0000000000..8a01ff5c41 --- /dev/null +++ b/mooncake-store/tests/local_delete_test.cpp @@ -0,0 +1,280 @@ +#include "local_delete.h" + +#include + +#include "replica.h" +#include "segment.h" +#include "serialize/serializer.h" +#include "utils.h" + +namespace mooncake::test { + +namespace { + +struct LegacyOffloadTaskItem { + std::string tenant_id; + std::string key; + int64_t size; +}; +YLT_REFL(LegacyOffloadTaskItem, tenant_id, key, size); + +struct LegacyStorageObjectMetadata { + int64_t bucket_id; + int64_t offset; + int64_t key_size; + int64_t data_size; + std::string transport_endpoint; +}; +YLT_REFL(LegacyStorageObjectMetadata, bucket_id, offset, key_size, data_size, + transport_endpoint); + +struct LegacyLocalDiskDescriptor { + UUID client_id; + uint64_t object_size; + std::string transport_endpoint; +}; +YLT_REFL(LegacyLocalDiskDescriptor, client_id, object_size, transport_endpoint); + +LocalDeleteTask MakeTask(std::string storage_id, std::string key) { + return LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = std::move(storage_id), + .tenant_id = "default", + .key = std::move(key), + .object_incarnation = GenerateObjectIncarnation(), + .expected_bucket_id = 7, + }; +} + +} // namespace + +TEST(LocalDeleteRegistryTest, MountEpochFencesPreviousOwner) { + LocalDeleteRegistry registry; + const UUID first_client{1, 2}; + const UUID second_client{3, 4}; + + const auto first = registry.Mount(first_client, "disk-a", + kLocalDiskCapabilityObjectTombstoneV1); + const auto retry = registry.Mount(first_client, "disk-a", + kLocalDiskCapabilityObjectTombstoneV1); + const auto replacement = registry.Mount( + second_client, "disk-a", kLocalDiskCapabilityObjectTombstoneV1); + + EXPECT_NE(first.mount_epoch, 0); + EXPECT_EQ(retry.mount_epoch, first.mount_epoch); + EXPECT_NE(replacement.mount_epoch, first.mount_epoch); + EXPECT_FALSE(registry.Fetch(first_client, "disk-a", first.mount_epoch, 1)); + EXPECT_TRUE( + registry.Fetch(second_client, "disk-a", replacement.mount_epoch, 1)); +} + +TEST(LocalDeleteRegistryTest, UnmountRejectsFormerOwnerWithoutDroppingTasks) { + LocalDeleteRegistry registry; + const UUID client{1, 2}; + const auto mount = + registry.Mount(client, "disk-a", kLocalDiskCapabilityObjectTombstoneV1); + const auto task = MakeTask("disk-a", "key"); + ASSERT_TRUE(registry.ApplyDurableTasks({task})); + + registry.Unmount(client); + EXPECT_FALSE(registry.Fetch(client, "disk-a", mount.mount_epoch, 1)); + EXPECT_EQ(registry.Size(), 1); + + const auto remount = + registry.Mount(client, "disk-a", kLocalDiskCapabilityObjectTombstoneV1); + auto fetched = registry.Fetch(client, "disk-a", remount.mount_epoch, 1); + ASSERT_TRUE(fetched); + ASSERT_EQ(fetched->size(), 1); + EXPECT_EQ(fetched->front(), task); +} + +TEST(LocalDeleteRegistryTest, ClientRemountingAnotherDiskLosesOldBinding) { + LocalDeleteRegistry registry; + const UUID client{1, 2}; + const auto first = + registry.Mount(client, "disk-a", kLocalDiskCapabilityObjectTombstoneV1); + const auto second = + registry.Mount(client, "disk-b", kLocalDiskCapabilityObjectTombstoneV1); + + EXPECT_FALSE(registry.Fetch(client, "disk-a", first.mount_epoch, 1)); + EXPECT_TRUE(registry.Fetch(client, "disk-b", second.mount_epoch, 1)); +} + +TEST(LocalDeleteRegistryTest, CapabilityNegotiationIsFailClosed) { + LocalDeleteRegistry registry; + const UUID legacy_client{1, 2}; + const auto mount = registry.Mount(legacy_client, "disk-a", 0); + EXPECT_FALSE(registry.Fetch(legacy_client, "disk-a", mount.mount_epoch, 1)); +} + +TEST(LocalDeleteRegistryTest, ReservationPublishesOnlyAfterCommit) { + LocalDeleteRegistry registry(2); + const UUID client{1, 2}; + const auto mount = + registry.Mount(client, "disk-a", kLocalDiskCapabilityObjectTombstoneV1); + auto task = MakeTask("disk-a", "key"); + + auto reservation = registry.Reserve({task}); + ASSERT_TRUE(reservation); + EXPECT_TRUE( + registry.Fetch(client, "disk-a", mount.mount_epoch, 8)->empty()); + + reservation.value()->Publish(); + auto fetched = registry.Fetch(client, "disk-a", mount.mount_epoch, 8); + ASSERT_TRUE(fetched); + ASSERT_EQ(fetched->size(), 1); + EXPECT_EQ(fetched->front(), task); + + registry.Erase("disk-a", {task.task_id}); + EXPECT_EQ(registry.Size(), 0); +} + +TEST(LocalDeleteRegistryTest, SnapshotRestoreKeepsPendingIntent) { + LocalDeleteRegistry source; + auto first = MakeTask("disk-a", "first"); + auto second = MakeTask("disk-b", "second"); + ASSERT_TRUE(source.ApplyDurableTasks({first, second})); + + LocalDeleteRegistry restored; + ASSERT_TRUE(restored.Restore(source.Snapshot())); + EXPECT_EQ(restored.Size(), 2); + EXPECT_TRUE(restored.ApplyDurableTasks({first})); + EXPECT_EQ(restored.Size(), 2); +} + +TEST(LocalDeleteRegistryTest, CapacityIncludesReservations) { + LocalDeleteRegistry registry(1); + { + auto first = registry.Reserve({MakeTask("disk-a", "first")}); + ASSERT_TRUE(first); + auto second = registry.Reserve({MakeTask("disk-a", "second")}); + ASSERT_FALSE(second); + EXPECT_EQ(second.error(), ErrorCode::TASK_PENDING_LIMIT_EXCEEDED); + } + EXPECT_TRUE(registry.Reserve({MakeTask("disk-a", "second")})); +} + +TEST(LocalDeleteRegistryTest, DuplicateDurableTasksDoNotConsumeCapacity) { + LocalDeleteRegistry registry(1); + const auto task = MakeTask("disk-a", "key"); + + EXPECT_TRUE(registry.ApplyDurableTasks({task, task})); + EXPECT_EQ(registry.Size(), 1); + EXPECT_TRUE(registry.ApplyDurableTasks({task})); + EXPECT_EQ(registry.Size(), 1); + EXPECT_FALSE( + registry.ApplyDurableTasks({MakeTask("disk-a", "another-key")})); +} + +TEST(LocalDeleteRegistryTest, InvalidTasksAreRejected) { + LocalDeleteRegistry registry; + auto missing_storage = MakeTask("", "key"); + auto missing_task_id = MakeTask("disk-a", "key"); + missing_task_id.task_id = {}; + + auto reservation = registry.Reserve({missing_storage}); + ASSERT_FALSE(reservation); + EXPECT_EQ(reservation.error(), ErrorCode::INVALID_PARAMS); + EXPECT_FALSE(registry.ApplyDurableTasks({missing_task_id})); + EXPECT_FALSE(registry.Restore({missing_storage})); +} + +TEST(LocalDeleteWireCompatibilityTest, AdditiveFieldsUseCompatibleEncoding) { + const LegacyOffloadTaskItem legacy_task{"default", "key", 1024}; + const auto legacy_task_bytes = struct_pack::serialize(legacy_task); + OffloadTaskItem current_task; + ASSERT_EQ(struct_pack::deserialize_to(current_task, legacy_task_bytes), + struct_pack::errc::ok); + EXPECT_TRUE(current_task.GetObjectIncarnation().IsZero()); + + current_task.object_incarnation = ObjectIncarnation{11, 12}; + const auto current_task_bytes = struct_pack::serialize(current_task); + LegacyOffloadTaskItem decoded_legacy_task; + ASSERT_EQ( + struct_pack::deserialize_to(decoded_legacy_task, current_task_bytes), + struct_pack::errc::ok); + EXPECT_EQ(decoded_legacy_task.key, legacy_task.key); + + const LegacyStorageObjectMetadata legacy_storage{7, 1, 3, 1024, "endpoint"}; + const auto legacy_storage_bytes = struct_pack::serialize(legacy_storage); + StorageObjectMetadata current_storage; + ASSERT_EQ( + struct_pack::deserialize_to(current_storage, legacy_storage_bytes), + struct_pack::errc::ok); + EXPECT_TRUE(current_storage.GetObjectIncarnation().IsZero()); + current_storage.object_incarnation = ObjectIncarnation{11, 12}; + const auto current_storage_bytes = struct_pack::serialize(current_storage); + LegacyStorageObjectMetadata decoded_legacy_storage; + ASSERT_EQ(struct_pack::deserialize_to(decoded_legacy_storage, + current_storage_bytes), + struct_pack::errc::ok); + EXPECT_EQ(decoded_legacy_storage.bucket_id, legacy_storage.bucket_id); + + const LegacyLocalDiskDescriptor legacy_descriptor{{1, 2}, 1024, "endpoint"}; + const auto legacy_descriptor_bytes = + struct_pack::serialize(legacy_descriptor); + LocalDiskDescriptor current_descriptor; + ASSERT_EQ(struct_pack::deserialize_to(current_descriptor, + legacy_descriptor_bytes), + struct_pack::errc::ok); + EXPECT_TRUE(current_descriptor.GetLocalDiskSegmentId().empty()); + + current_descriptor.SetDeleteMetadata( + "disk-a", 3, kLocalDiskCapabilityObjectTombstoneV1, 7, {11, 12}); + const auto current_descriptor_bytes = + struct_pack::serialize(current_descriptor); + LegacyLocalDiskDescriptor decoded_legacy_descriptor; + ASSERT_EQ(struct_pack::deserialize_to(decoded_legacy_descriptor, + current_descriptor_bytes), + struct_pack::errc::ok); + EXPECT_EQ(decoded_legacy_descriptor.client_id, legacy_descriptor.client_id); + EXPECT_EQ(decoded_legacy_descriptor.transport_endpoint, + legacy_descriptor.transport_endpoint); +} + +TEST(LocalDeleteWireCompatibilityTest, + ReplicaDeserializerAcceptsLegacyAndCurrentLocalDiskPayloads) { + const UUID client_id{1, 2}; + SegmentManager segment_manager; + + msgpack::sbuffer legacy_buffer; + MsgpackPacker legacy_packer(&legacy_buffer); + legacy_packer.pack_array(4); + legacy_packer.pack(uint64_t{9}); + legacy_packer.pack(static_cast(ReplicaStatus::COMPLETE)); + legacy_packer.pack(static_cast(ReplicaType::LOCAL_DISK)); + legacy_packer.pack_array(3); + legacy_packer.pack(UuidToString(client_id)); + legacy_packer.pack(uint64_t{1024}); + legacy_packer.pack(std::string("endpoint")); + auto legacy_object = + msgpack::unpack(legacy_buffer.data(), legacy_buffer.size()); + auto legacy = Serializer::deserialize(legacy_object.get(), + segment_manager.getView()); + ASSERT_TRUE(legacy); + auto legacy_descriptor = std::get( + legacy.value()->get_descriptor().descriptor_variant); + EXPECT_TRUE(legacy_descriptor.GetLocalDiskSegmentId().empty()); + EXPECT_TRUE(legacy_descriptor.GetObjectIncarnation().IsZero()); + + Replica current(client_id, 1024, "endpoint", ReplicaStatus::COMPLETE, + "disk-a", 3, kLocalDiskCapabilityObjectTombstoneV1, 7, + {11, 12}); + msgpack::sbuffer current_buffer; + MsgpackPacker current_packer(¤t_buffer); + ASSERT_TRUE(Serializer::serialize( + current, segment_manager.getView(), current_packer)); + auto current_object = + msgpack::unpack(current_buffer.data(), current_buffer.size()); + auto decoded = Serializer::deserialize(current_object.get(), + segment_manager.getView()); + ASSERT_TRUE(decoded); + auto current_descriptor = std::get( + decoded.value()->get_descriptor().descriptor_variant); + EXPECT_EQ(current_descriptor.GetLocalDiskSegmentId(), "disk-a"); + EXPECT_EQ(current_descriptor.GetMountEpoch(), 3); + EXPECT_EQ(current_descriptor.GetObjectIncarnation(), + (ObjectIncarnation{11, 12})); +} + +} // namespace mooncake::test diff --git a/mooncake-store/tests/master_service_tenant_quota_test.cpp b/mooncake-store/tests/master_service_tenant_quota_test.cpp index 5016af2b55..4f521d708f 100644 --- a/mooncake-store/tests/master_service_tenant_quota_test.cpp +++ b/mooncake-store/tests/master_service_tenant_quota_test.cpp @@ -532,6 +532,28 @@ TEST_F(MasterServiceTenantQuotaTest, EXPECT_EQ(LocalDiskUsedBytes(service, client_b), 0); } +TEST_F(MasterServiceTenantQuotaTest, + AddReplicaTreatsStableStorageIdentityAsExisting) { + MasterService service(MakeConfig({{TenantId("tenant-a"), 1000}})); + const UUID client_a = generate_uuid(); + const UUID client_b = generate_uuid(); + + Replica first(client_a, 128, "disk-endpoint-a", ReplicaStatus::COMPLETE, + "disk-a", 1); + auto first_result = + service.AddReplica(client_a, "cold", TenantId("tenant-a"), first); + ASSERT_TRUE(first_result.has_value()) << toString(first_result.error()); + EXPECT_TRUE(first_result.value()); + + Replica replacement(client_b, 128, "disk-endpoint-b", + ReplicaStatus::COMPLETE, "disk-a", 2); + auto replacement_result = + service.AddReplica(client_b, "cold", TenantId("tenant-a"), replacement); + ASSERT_TRUE(replacement_result.has_value()) + << toString(replacement_result.error()); + EXPECT_FALSE(replacement_result.value()); +} + TEST_F(MasterServiceTenantQuotaTest, RegisteredTenantQuotaAdmissionDoesNotCreateImplicitTenants) { MasterService service(MakeConfig({{TenantId("tenant-a"), 100}})); diff --git a/mooncake-store/tests/segment_test.cpp b/mooncake-store/tests/segment_test.cpp index 60b7362061..7449c54ebc 100644 --- a/mooncake-store/tests/segment_test.cpp +++ b/mooncake-store/tests/segment_test.cpp @@ -788,4 +788,24 @@ TEST_F(SegmentTest, MountLocalDiskSegmentDuplicate) { ValidateMountedLocalDiskSegments(segment_manager, segments, client_ids); } +TEST_F(SegmentTest, MountLocalDiskSegmentIdentityChanged) { + SegmentManager segment_manager; + UUID client_id = generate_uuid(); + { + auto segment_access = segment_manager.getSegmentAccess(); + ASSERT_EQ(segment_access.MountLocalDiskSegment(client_id, true, + "disk-a", 1, 1), + ErrorCode::OK); + ASSERT_EQ(segment_access.MountLocalDiskSegment(client_id, true, + "disk-b", 2, 1), + ErrorCode::OK); + } + + auto local_disk_access = segment_manager.getLocalDiskSegmentAccess(); + const auto& mounted = + local_disk_access.getClientLocalDiskSegment().at(client_id); + EXPECT_EQ(mounted->local_disk_segment_id, "disk-b"); + EXPECT_EQ(mounted->mount_epoch, 2); +} + } // namespace mooncake diff --git a/mooncake-store/tests/storage_backend_bucket_delete_test.cpp b/mooncake-store/tests/storage_backend_bucket_delete_test.cpp new file mode 100644 index 0000000000..ff3ec30982 --- /dev/null +++ b/mooncake-store/tests/storage_backend_bucket_delete_test.cpp @@ -0,0 +1,633 @@ +#include "storage_backend.h" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "local_delete.h" +#include "utils.h" + +namespace mooncake::test { + +namespace { + +struct LegacyBucketObjectMetadata { + int64_t offset; + int64_t key_size; + int64_t data_size; +}; +YLT_REFL(LegacyBucketObjectMetadata, offset, key_size, data_size); + +struct LegacyBucketMetadata { + int64_t data_size; + std::vector keys; + std::vector metadatas; +}; +YLT_REFL(LegacyBucketMetadata, data_size, keys, metadatas); + +bool WaitUntil(const std::function& predicate, + std::chrono::seconds timeout = std::chrono::seconds(5)) { + const auto deadline = std::chrono::steady_clock::now() + timeout; + while (std::chrono::steady_clock::now() < deadline) { + if (predicate()) { + return true; + } + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + return predicate(); +} + +size_t CountBucketFiles(const std::filesystem::path& path, + std::string_view extension) { + size_t count = 0; + for (const auto& entry : std::filesystem::directory_iterator(path)) { + std::error_code error; + if (entry.is_regular_file(error) && !error && + entry.path().extension() == extension) { + ++count; + } + } + return count; +} + +void WriteGcIntent(const std::filesystem::path& path, + const BucketGcIntent& intent) { + std::string bytes; + struct_pb::to_pb(intent, bytes); + std::ofstream output(path / ".bucket_gc_intent", + std::ios::binary | std::ios::trunc); + ASSERT_TRUE(output.good()); + output.write(bytes.data(), static_cast(bytes.size())); + ASSERT_TRUE(output.good()); +} + +class StorageBackendBucketDeleteTest : public ::testing::Test { + protected: + void SetUp() override { + data_path_ = + std::filesystem::temp_directory_path() / + ("mooncake_bucket_delete_" + UuidToString(generate_uuid())); + ASSERT_TRUE(std::filesystem::create_directories(data_path_)); + } + + void TearDown() override { + std::error_code error; + std::filesystem::remove_all(data_path_, error); + } + + std::filesystem::path data_path_; +}; + +} // namespace + +TEST_F(StorageBackendBucketDeleteTest, + DeleteIsDurableAndCannotDeleteRecreatedKey) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 1; + + const ObjectIncarnation old_incarnation{11, 12}; + const ObjectIncarnation new_incarnation{21, 22}; + StorageObjectMetadata old_location; + LocalDeleteTask remove_old; + int64_t physical_size_after_delete = 0; + + { + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + std::string value(1024, 'A'); + std::vector> grouped; + const std::unordered_map + old_incarnations{{"same-key", old_incarnation}}; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{"same-key", static_cast(value.size())}}, grouped, + &old_incarnations)); + ASSERT_EQ(grouped.size(), 1); + auto stored = backend.BatchOffload( + {{"same-key", {Slice{value.data(), value.size()}}}}, + [&](const std::vector&, + std::vector& metadatas) { + old_location = metadatas.front(); + return ErrorCode::OK; + }); + ASSERT_TRUE(stored); + const auto physical_size_before_delete = backend.GetStoreMetadata(); + ASSERT_TRUE(physical_size_before_delete); + + remove_old = LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = "same-key", + .object_incarnation = old_incarnation, + .expected_bucket_id = old_location.bucket_id, + }; + const auto removed = backend.BatchMarkDeleted({remove_old}); + ASSERT_EQ(removed.size(), 1); + EXPECT_EQ(removed.front().result, LocalDeleteResult::kRemoved); + EXPECT_FALSE(backend.IsExist("same-key").value_or(true)); + EXPECT_EQ(backend.GetReclaimableBytes(), + old_location.key_size + old_location.data_size); + const auto physical_size = backend.GetStoreMetadata(); + ASSERT_TRUE(physical_size); + EXPECT_GE(physical_size->total_size, + physical_size_before_delete->total_size); + physical_size_after_delete = physical_size->total_size; + + const auto redelivered = backend.BatchMarkDeleted({remove_old}); + ASSERT_EQ(redelivered.size(), 1); + EXPECT_EQ(redelivered.front().result, + LocalDeleteResult::kAlreadyRemoved); + } + + BucketStorageBackend restarted(config, bucket_config); + ASSERT_TRUE(restarted.Init()); + ASSERT_TRUE(restarted.GetStoreMetadata()); + EXPECT_EQ(restarted.GetStoreMetadata()->total_size, + physical_size_after_delete); + EXPECT_EQ(restarted.GetReclaimableBytes(), + old_location.key_size + old_location.data_size); + EXPECT_FALSE(restarted.IsExist("same-key").value_or(true)); + const auto after_restart = restarted.BatchMarkDeleted({remove_old}); + ASSERT_EQ(after_restart.size(), 1); + EXPECT_EQ(after_restart.front().result, LocalDeleteResult::kAlreadyRemoved); + + std::vector scanned_keys; + ASSERT_TRUE(restarted.ScanMeta([&](const std::vector& keys, + std::vector&) { + scanned_keys.insert(scanned_keys.end(), keys.begin(), keys.end()); + return ErrorCode::OK; + })); + EXPECT_TRUE(scanned_keys.empty()); + + std::string new_value(1024, 'B'); + std::vector> grouped; + const std::unordered_map new_incarnations{ + {"same-key", new_incarnation}}; + ASSERT_TRUE(restarted.AllocateOffloadingBuckets( + {{"same-key", static_cast(new_value.size())}}, grouped, + &new_incarnations)); + ASSERT_EQ(grouped.size(), 1); + ASSERT_TRUE(restarted.BatchOffload( + {{"same-key", {Slice{new_value.data(), new_value.size()}}}}, + [](const std::vector&, + std::vector&) { return ErrorCode::OK; })); + + LocalDeleteTask delayed_old_delete{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = "same-key", + .object_incarnation = old_incarnation, + .expected_bucket_id = old_location.bucket_id, + }; + const auto stale = restarted.BatchMarkDeleted({delayed_old_delete}); + ASSERT_EQ(stale.size(), 1); + EXPECT_EQ(stale.front().result, LocalDeleteResult::kStaleVersion); + EXPECT_TRUE(restarted.IsExist("same-key").value_or(false)); +} + +TEST_F(StorageBackendBucketDeleteTest, + OneInvalidTaskDoesNotSuppressAnotherBucket) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 1; + + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + std::string first_value(32, 'A'); + std::string second_value(32, 'B'); + const ObjectIncarnation first_incarnation{11, 12}; + const ObjectIncarnation second_incarnation{21, 22}; + std::vector> grouped; + const std::unordered_map incarnations{ + {"first", first_incarnation}, {"second", second_incarnation}}; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{"first", static_cast(first_value.size())}, + {"second", static_cast(second_value.size())}}, + grouped, &incarnations)); + + std::unordered_map locations; + ASSERT_TRUE(backend.BatchOffload( + {{"first", {Slice{first_value.data(), first_value.size()}}}, + {"second", {Slice{second_value.data(), second_value.size()}}}}, + [&](const std::vector& keys, + std::vector& metadatas) { + for (size_t i = 0; i < keys.size(); ++i) { + locations.emplace(keys[i], metadatas[i]); + } + return ErrorCode::OK; + })); + ASSERT_EQ(locations.size(), 2); + + const LocalDeleteTask invalid{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = "first", + .object_incarnation = {31, 32}, + .expected_bucket_id = locations.at("first").bucket_id, + }; + const LocalDeleteTask valid{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = "second", + .object_incarnation = second_incarnation, + .expected_bucket_id = locations.at("second").bucket_id, + }; + const auto results = backend.BatchMarkDeleted({invalid, valid}); + ASSERT_EQ(results.size(), 2); + EXPECT_EQ(results[0].result, LocalDeleteResult::kStaleVersion); + EXPECT_EQ(results[1].result, LocalDeleteResult::kRemoved); + EXPECT_TRUE(backend.IsExist("first").value_or(false)); + EXPECT_FALSE(backend.IsExist("second").value_or(true)); +} + +TEST_F(StorageBackendBucketDeleteTest, + CorruptMetadataFailsClosedAndPreservesBucketFiles) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 1; + + StorageObjectMetadata location; + { + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + std::string value(32, 'A'); + std::vector> grouped; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{"key", static_cast(value.size())}}, grouped)); + ASSERT_TRUE(backend.BatchOffload( + {{"key", {Slice{value.data(), value.size()}}}}, + [&](const std::vector&, + std::vector& metadatas) { + location = metadatas.front(); + return ErrorCode::OK; + })); + } + + const auto metadata_path = + data_path_ / (std::to_string(location.bucket_id) + ".meta"); + const auto data_file_path = + data_path_ / (std::to_string(location.bucket_id) + ".bucket"); + ASSERT_TRUE(std::filesystem::exists(metadata_path)); + ASSERT_TRUE(std::filesystem::exists(data_file_path)); + { + std::ofstream corrupt(metadata_path, + std::ios::binary | std::ios::trunc); + ASSERT_TRUE(corrupt.good()); + corrupt << "not-a-valid-protobuf"; + } + + BucketStorageBackend restarted(config, bucket_config); + EXPECT_FALSE(restarted.Init()); + EXPECT_TRUE(std::filesystem::exists(metadata_path)); + EXPECT_TRUE(std::filesystem::exists(data_file_path)); +} + +TEST_F(StorageBackendBucketDeleteTest, + GarbageCollectionMergesPartiallyDeadBuckets) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 8; + bucket_config.bucket_size_limit = 8 * 1024; + bucket_config.gc_interval_seconds = 3600; + bucket_config.gc_deleted_ratio = 0.25; + + std::unordered_map live_values; + std::unordered_map locations; + int64_t physical_size_before_gc = 0; + { + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + + for (int bucket_index = 0; bucket_index < 3; ++bucket_index) { + const std::string deleted_key = + "deleted-" + std::to_string(bucket_index); + const std::string live_key = "live-" + std::to_string(bucket_index); + std::string deleted_value(7 * 1024, + static_cast('a' + bucket_index)); + std::string live_value(1024, static_cast('A' + bucket_index)); + live_values.emplace(live_key, live_value); + + const ObjectIncarnation deleted_incarnation{ + 100, static_cast(bucket_index + 1)}; + const ObjectIncarnation live_incarnation{ + 200, static_cast(bucket_index + 1)}; + std::vector> grouped; + const std::unordered_map + incarnations{{deleted_key, deleted_incarnation}, + {live_key, live_incarnation}}; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{deleted_key, static_cast(deleted_value.size())}, + {live_key, static_cast(live_value.size())}}, + grouped, &incarnations)); + ASSERT_EQ(grouped.size(), 1); + + std::unordered_map> batch; + batch.emplace(deleted_key, + std::vector{Slice{deleted_value.data(), + deleted_value.size()}}); + batch.emplace(live_key, std::vector{Slice{ + live_value.data(), live_value.size()}}); + ASSERT_TRUE(backend.BatchOffload( + batch, [&](const std::vector& keys, + std::vector& metadatas) { + for (size_t i = 0; i < keys.size(); ++i) { + locations[keys[i]] = metadatas[i]; + } + return ErrorCode::OK; + })); + + const LocalDeleteTask task{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = deleted_key, + .object_incarnation = deleted_incarnation, + .expected_bucket_id = locations.at(deleted_key).bucket_id, + }; + const auto result = backend.BatchMarkDeleted({task}); + ASSERT_EQ(result.size(), 1); + ASSERT_EQ(result.front().result, LocalDeleteResult::kRemoved); + } + + ASSERT_GT(backend.GetReclaimableBytes(), 0); + const auto before_gc = backend.GetStoreMetadata(); + ASSERT_TRUE(before_gc); + physical_size_before_gc = before_gc->total_size; + + backend.RequestGarbageCollection(); + ASSERT_TRUE(WaitUntil([&] { + return backend.GetReclaimableBytes() == 0 && + CountBucketFiles(data_path_, ".meta") == 1 && + CountBucketFiles(data_path_, ".bucket") == 1; + })); + + const auto after_gc = backend.GetStoreMetadata(); + ASSERT_TRUE(after_gc); + EXPECT_LT(after_gc->total_size, physical_size_before_gc); + EXPECT_EQ(CountBucketFiles(data_path_, ".meta"), 1); + EXPECT_EQ(CountBucketFiles(data_path_, ".bucket"), 1); + + for (const auto& [key, value] : live_values) { + std::vector buffer(value.size()); + std::unordered_map load; + load.emplace(key, Slice{buffer.data(), buffer.size()}); + ASSERT_TRUE(backend.BatchLoad(load)); + EXPECT_EQ(std::string(buffer.begin(), buffer.end()), value); + } + } + + bucket_config.gc_enable = false; + BucketStorageBackend restarted(config, bucket_config); + ASSERT_TRUE(restarted.Init()); + EXPECT_EQ(restarted.GetReclaimableBytes(), 0); + EXPECT_FALSE(restarted.RequestGarbageCollection()); + for (const auto& [key, value] : live_values) { + std::vector buffer(value.size()); + std::unordered_map load; + load.emplace(key, Slice{buffer.data(), buffer.size()}); + ASSERT_TRUE(restarted.BatchLoad(load)); + EXPECT_EQ(std::string(buffer.begin(), buffer.end()), value); + } +} + +TEST_F(StorageBackendBucketDeleteTest, + DiskHighWatermarkOverridesDeletedRatioThreshold) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + config.disk_eviction_high_watermark_ratio = 0.90; + config.disk_eviction_low_watermark_ratio = 0.80; + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 10; + bucket_config.bucket_size_limit = 32 * 1024; + bucket_config.max_total_size = 10 * 1024; + bucket_config.gc_interval_seconds = 3600; + bucket_config.gc_deleted_ratio = 0.90; + + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + std::unordered_map> batch; + std::unordered_map values; + std::unordered_map incarnations; + std::unordered_map sizes; + for (int i = 0; i < 10; ++i) { + const std::string key = "key-" + std::to_string(i); + auto [value_it, _] = values.emplace(key, std::string(940, 'A' + i)); + batch.emplace(key, std::vector{Slice{value_it->second.data(), + value_it->second.size()}}); + incarnations.emplace(key, + ObjectIncarnation{300, static_cast(i)}); + sizes.emplace(key, static_cast(value_it->second.size())); + } + std::vector> grouped; + ASSERT_TRUE( + backend.AllocateOffloadingBuckets(sizes, grouped, &incarnations)); + ASSERT_EQ(grouped.size(), 1); + + std::unordered_map locations; + ASSERT_TRUE(backend.BatchOffload( + batch, [&](const std::vector& keys, + std::vector& metadatas) { + for (size_t i = 0; i < keys.size(); ++i) { + locations[keys[i]] = metadatas[i]; + } + return ErrorCode::OK; + })); + const auto before_delete = backend.GetStoreMetadata(); + ASSERT_TRUE(before_delete); + ASSERT_GT(before_delete->total_size, + static_cast(bucket_config.max_total_size * + config.disk_eviction_high_watermark_ratio)); + + std::vector tasks; + for (int i = 0; i < 3; ++i) { + const std::string key = "key-" + std::to_string(i); + tasks.push_back(LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = key, + .object_incarnation = incarnations.at(key), + .expected_bucket_id = locations.at(key).bucket_id, + }); + } + const auto deleted = backend.BatchMarkDeleted(tasks); + ASSERT_EQ(deleted.size(), tasks.size()); + for (const auto& result : deleted) { + ASSERT_EQ(result.result, LocalDeleteResult::kRemoved); + } + + // 30% is below the configured 90% ratio. Crossing the disk high + // watermark must still admit the bucket and reclaim it toward low water. + ASSERT_TRUE(backend.RequestGarbageCollection( + /* require_disk_pressure = */ true)); + ASSERT_TRUE(WaitUntil([&] { + const auto state = backend.GetStoreMetadata(); + return state && backend.GetReclaimableBytes() == 0 && + state->total_size < before_delete->total_size; + })); + const auto after_gc = backend.GetStoreMetadata(); + ASSERT_TRUE(after_gc); + EXPECT_LT(after_gc->total_size, before_delete->total_size); + EXPECT_LE(after_gc->total_size, + static_cast(bucket_config.max_total_size * + config.disk_eviction_low_watermark_ratio)); +} + +TEST_F(StorageBackendBucketDeleteTest, + GarbageCollectionUnlinksFullyDeadBucketWithoutReplacement) { + FileStorageConfig config; + config.storage_filepath = data_path_.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 1; + bucket_config.gc_interval_seconds = 3600; + + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + const std::string key = "dead"; + std::string value(1024, 'D'); + const ObjectIncarnation incarnation{501, 1}; + std::vector> grouped; + const std::unordered_map incarnations{ + {key, incarnation}}; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{key, static_cast(value.size())}}, grouped, &incarnations)); + StorageObjectMetadata location; + ASSERT_TRUE(backend.BatchOffload( + {{key, {Slice{value.data(), value.size()}}}}, + [&](const std::vector&, + std::vector& metadatas) { + location = metadatas.front(); + return ErrorCode::OK; + })); + const auto deleted = backend.BatchMarkDeleted({LocalDeleteTask{ + .task_id = GenerateLocalDeleteTaskId(), + .local_disk_segment_id = "disk-a", + .tenant_id = "default", + .key = key, + .object_incarnation = incarnation, + .expected_bucket_id = location.bucket_id, + }}); + ASSERT_EQ(deleted.size(), 1); + ASSERT_EQ(deleted.front().result, LocalDeleteResult::kRemoved); + + backend.RequestGarbageCollection(); + ASSERT_TRUE(WaitUntil([&] { + return backend.GetReclaimableBytes() == 0 && + CountBucketFiles(data_path_, ".meta") == 0 && + CountBucketFiles(data_path_, ".bucket") == 0; + })); + const auto state = backend.GetStoreMetadata(); + ASSERT_TRUE(state); + EXPECT_EQ(state->total_keys, 0); + EXPECT_EQ(state->total_size, 0); +} + +TEST_F(StorageBackendBucketDeleteTest, + GarbageCollectionIntentRecoversPreparedAndCommittedStates) { + const auto run_case = [&](const std::string& name, bool committed) { + const auto case_path = data_path_ / name; + ASSERT_TRUE(std::filesystem::create_directories(case_path)); + FileStorageConfig config; + config.storage_filepath = case_path.string(); + BucketBackendConfig bucket_config; + bucket_config.bucket_keys_limit = 1; + bucket_config.gc_enable = false; + + int64_t source_id = -1; + int64_t target_id = -1; + { + BucketStorageBackend backend(config, bucket_config); + ASSERT_TRUE(backend.Init()); + for (const auto& [key, incarnation] : + std::vector>{ + {"source", {401, 1}}, {"target", {402, 1}}}) { + std::string value(128, key.front()); + std::vector> grouped; + const std::unordered_map + incarnation_by_key{{key, incarnation}}; + ASSERT_TRUE(backend.AllocateOffloadingBuckets( + {{key, static_cast(value.size())}}, grouped, + &incarnation_by_key)); + StorageObjectMetadata location; + ASSERT_TRUE(backend.BatchOffload( + {{key, + {Slice{value.data(), + static_cast(value.size())}}}}, + [&](const std::vector&, + std::vector& metadatas) { + location = metadatas.front(); + return ErrorCode::OK; + })); + if (key == "source") { + source_id = location.bucket_id; + } else { + target_id = location.bucket_id; + } + } + } + ASSERT_GE(source_id, 0); + ASSERT_GE(target_id, 0); + WriteGcIntent(case_path, + BucketGcIntent{.version = 1, + .committed = committed, + .target_bucket_id = target_id, + .source_bucket_ids = {source_id}}); + + BucketStorageBackend recovered(config, bucket_config); + ASSERT_TRUE(recovered.Init()); + EXPECT_EQ(recovered.IsExist("source").value_or(false), !committed); + EXPECT_EQ(recovered.IsExist("target").value_or(false), committed); + EXPECT_FALSE(std::filesystem::exists(case_path / ".bucket_gc_intent")); + EXPECT_EQ(CountBucketFiles(case_path, ".meta"), 1); + EXPECT_EQ(CountBucketFiles(case_path, ".bucket"), 1); + }; + + run_case("prepared", false); + run_case("committed", true); +} + +TEST_F(StorageBackendBucketDeleteTest, MetadataIsBackwardCompatible) { + LegacyBucketMetadata legacy{ + .data_size = 10, + .keys = {"legacy"}, + .metadatas = {{.offset = 1, .key_size = 2, .data_size = 7}}, + }; + std::string legacy_bytes; + struct_pb::to_pb(legacy, legacy_bytes); + + BucketMetadata upgraded; + ASSERT_NO_THROW(struct_pb::from_pb(upgraded, legacy_bytes)); + ASSERT_EQ(upgraded.metadatas.size(), 1); + EXPECT_TRUE(upgraded.metadatas.front().object_incarnation.IsZero()); + EXPECT_FALSE(upgraded.metadatas.front().tombstoned); + + upgraded.metadatas.front().object_incarnation = {31, 32}; + upgraded.metadatas.front().tombstoned = true; + std::string current_bytes; + struct_pb::to_pb(upgraded, current_bytes); + BucketMetadata round_tripped; + ASSERT_NO_THROW(struct_pb::from_pb(round_tripped, current_bytes)); + ASSERT_EQ(round_tripped.metadatas.size(), 1); + EXPECT_EQ(round_tripped.metadatas.front().object_incarnation, + (ObjectIncarnation{31, 32})); + EXPECT_TRUE(round_tripped.metadatas.front().tombstoned); +} + +} // namespace mooncake::test