feat(CuckooFilter): add cf.del command

[nagisa-kunhah]

part of: #3552

Summary

Add RedisBloom-compatible CF.DEL key item support for Cuckoo Filter.

CF.DEL deletes one matching fingerprint occurrence and returns 1 when a slot is cleared, or 0 when the key/item
is not found. Duplicate inserts require duplicate deletes, matching RedisBloom behavior.

Design

The command layer adds cf.del as a write command and delegates deletion to CuckooChain::Delete.

Deletion loads the Cuckoo Filter metadata, hashes the item, generates the fingerprint, then searches sub-filters from
newest to oldest. The first matching slot in either candidate bucket is cleared, and only one occurrence is removed.

After a successful delete, metadata size is decremented and num_deleted_items is incremented. When the chain has
more than one sub-filter and accumulated deletes exceed 10% of the remaining item count, an internal compact pass is
triggered. Compact tries to move fingerprints from newer sub-filters into older ones, removes fully compacted latest
sub-filters, and deletes their persisted page keys to avoid stale data if the chain expands again later.

Page key construction is shared through small Cuckoo page helpers so compact cleanup uses the same encoding as normal
page access.

This pr is This PR was written using codex and GPT-5.5

[nagisa-kunhah]

Hi @jihuayu, while implementing CF.DEL, I found that the automatic compaction path may introduce serious worker-blocking and OOM risks. Currently, it scans all logical buckets of a sub‑filter and loads every page into CuckooPageCache, including zero‑filled pages for missing keys. Thus, a large sparse sub‑filter can make a single CF.DEL perform work and allocate memory proportional to its logical capacity.

We seek feedback on whether compaction should be part of this PR. Two options:

1. Land CF.DEL without automatic compaction for now.
   Only remove fingerprint and update metadata. Semantics are preserved, but deleted slots and extra sub‑filters remain, causing space and lookup amplification.

2. Move compaction to bounded, incremental background maintenance.
   Use Kvrocks’ TaskRunner (used by AsyncCompactDB, AsyncScanDBSize, etc.) with cursor‑based scanning. Process a bounded number of actually stored pages per slice, commit a bounded write batch, release cache and locks, and requeue if more work remains. Requires actual‑page iteration, bounded cache/batch, deduplication, synchronization, and crash‑safe progress tracking.

[jihuayu]

@nagisa-kunhah Let's leave this out for now. We can implement background compaction in a follow-up PR. Thanks.

PS: when the key does not exist, RedisBloom returns a Not found error, while this PR currently returns success.

[nagisa-kunhah]

CF.DEL Background Compaction

CF.DEL must not compact a complete logical sub-filter on the request path. A large sparse filter can have very few RocksDB pages but a huge logical page range, so a full logical scan can block a worker and materialize excessive page-cache memory.

Compaction

For the current tail Fn, compact oldest targets first:

Fn -> F0
remaining Fn -> F1
...
remaining Fn -> F(n - 1)
verify Fn

For each non-zero fingerprint in Fn, try direct insertion into the two candidate buckets of the current target. If insertion succeeds, write the target fingerprint and clear the source fingerprint in one WriteBatch. Do not use Cuckoo kick-out relocation during compaction.

Scan Fn through its persisted page-key prefix, so sparse filters do not trigger reads for absent logical pages. Delete a source page key when its page becomes all zero.

One Fn -> Fi pass holds the existing logical-key lock. It may flush bounded page batches while keeping that lock. After a pass completes, release the lock, then start the next target pass. If Fn is empty after all targets were tried, decrement n_filters. If that exposes another tail, continue with it in the same task.

Task execution and deduplication

A successful CF.DEL schedules compact work when the chain has more than one sub-filter and delete debt exceeds the compact threshold. The task carries:

Field	Description
namespace and user_key	Reconstruct the Cuckoo Filter in the background callback.
ns_key	Lock key and task-key input.
metadata_version	Reject a key that was deleted and recreated before the task starts.

TaskRunner should gain a generic keyed API:

StatusOr<bool> TryPublishUnique(std::string task_key, Task task);

The compact task key is:

cuckoo-compact:<ns_key>:<metadata_version>

TaskRunner keeps active task keys in memory. The same key is queued only once while it is queued or running. It removes the key when the callback returns, rolls it back if queueing fails, and clears all keys when Join() clears the queue.

The callback locks ns_key, rereads metadata, and exits if metadata_version no longer matches. A restart drops queued work and active task keys. It does not resume compact work. A later qualifying CF.DEL can schedule a new best-effort task.

Trade-offs

• A full Fn -> Fi pass can delay foreground writes for that key.
• One tail is scanned once per older target, and target bucket reads can be random.
• CF.ADD between target passes can make the current cycle less effective, but cannot make tail removal unsafe because the final empty check is locked.
• TaskRunner is shared and single-threaded by default, so long compact tasks can delay other background work.

[nagisa-kunhah]

Hi @jihuayu , sorry for the delay. This PR is ready for review. I've outlined some draft ideas on the background compaction above — would really appreciate your advice when you have time. Thanks!

[jihuayu]

@nagisa-kunhah Overall, it looks good. Thank you!

[jihuayu]

I think these case Go tests is OK, so there’s no need to add C++ test cases.

[jihuayu]

I feel that commitDelete is a bit odd from an architectural perspective, as it duplicates some of the logic in commitSubFilterAndMetadata. Could we reuse commitSubFilterAndMetadata for the commit-related logic instead?

[jihuayu]

The current tests don't seem to cover cases that span multiple sub-filters. Could you add a test case for that?

[jihuayu]

Regarding the background task, I think your overall design looks reasonable. I have a few thoughts:

1. I don't think the compaction operation is very urgent. We could add a debounce mechanism, for example, delaying it by 5 minutes, to avoid triggering multiple compactions after several modifications within a short period.
2. I think we should investigate how long the lock will be held during compaction. A short lock is acceptable, but if compaction may hold the lock for a long time, we should consider whether to add some additional fields to the data structure to support the GC process.
3. Regarding the issue of the task being lost after a restart, could we provide an explicit command that allows users to manually trigger the compaction?

PS: We can open a new discussion or issue to discuss this further.

[Copilot]

Pull request overview

Adds RedisBloom-compatible Cuckoo Filter deletion support by introducing a new CF.DEL command and the underlying delete path in the Cuckoo filter storage implementation.

Changes:

• Add cf.del command plumbing and command registration.
• Implement deletion of a single matching fingerprint occurrence within CuckooChain/CuckooSubFilter.
• Improve page persistence by deleting/omitting fully-empty cuckoo pages, and add Go/C++ tests around delete behavior and page writeback.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/gocase/unit/type/bloom/cuckoo_filter_test.go	Adds Go unit coverage for cf.del argument handling and basic delete flows.
tests/cppunit/types/cuckoo_filter_test.cc	Adds C++ unit coverage for delete semantics and filter-search ordering.
tests/cppunit/types/cuckoo_filter_page_test.cc	Adds C++ unit coverage ensuring empty pages are not written and are deleted when emptied.
src/types/redis_cuckoo_chain.h	Exposes CuckooChain::Delete and related commit helper.
src/types/redis_cuckoo_chain.cc	Implements CuckooChain::Delete and delete commit path updating metadata/pages.
src/types/cuckoo_filter.h	Introduces kEmptyCuckooFingerprint constant for empty-slot representation.
src/types/cuckoo_filter_sub_filter.h	Adds CuckooSubFilter::Delete API.
src/types/cuckoo_filter_sub_filter.cc	Implements deletion by clearing the first matching fingerprint in candidate buckets.
src/types/cuckoo_filter_page.h	Tracks whether a page key previously existed to support delete-vs-skip semantics.
src/types/cuckoo_filter_page.cc	Deletes page keys when a dirty page becomes all-zero; skips writing new empty pages.
src/commands/cmd_cuckoo_filter.cc	Adds CommandCFDel and registers cf.del as a write command.

Suppressed comments (1)

src/types/redis_cuckoo_chain.cc:216

• The PR description states CF.DEL returns 0 when the key/item is not found and also mentions an internal compact pass and sub-filter/page cleanup. The current implementation returns an error for missing keys (see above) and does not implement any compact/sub-filter removal logic (only clears one fingerprint and updates metadata). Please either implement the described compaction behavior or adjust the PR description to match the shipped behavior.

  for (int filter_idx = static_cast<int>(metadata.n_filters) - 1; filter_idx >= 0; --filter_idx) {
    uint32_t num_buckets = 0;
    s = CuckooFilterHelper::GetFilterNumBuckets(metadata.base_capacity, metadata.expansion, metadata.bucket_size,
                                                static_cast<uint16_t>(filter_idx), &num_buckets);
    if (!s.ok()) return s;

    CuckooSubFilter sub_filter(storage_, ctx, ns_key, storage_->IsSlotIdEncoded(), metadata.version,
                               metadata.bucket_size, metadata.page_size, static_cast<uint16_t>(filter_idx),
                               num_buckets);
    bool found = false;
    s = sub_filter.Delete(hash, fingerprint, &found);
    if (!s.ok()) return s;
    if (!found) continue;

    if (metadata.size == 0) return rocksdb::Status::Corruption("invalid metadata: size is 0");
    metadata.size--;
    metadata.num_deleted_items++;
    *deleted = true;
    return commitDelete(ctx, user_key, ns_key, &metadata, &sub_filter);
  }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.