Skip to content

[MOD-14957] Add SQ8 support for tiered HNSW - #1029

Draft
dor-forer wants to merge 12 commits into
mainfrom
dor-forer-MOD-14957-tiered-hnsw-sq8
Draft

[MOD-14957] Add SQ8 support for tiered HNSW#1029
dor-forer wants to merge 12 commits into
mainfrom
dor-forer-MOD-14957-tiered-hnsw-sq8

Conversation

@dor-forer

Copy link
Copy Markdown
Collaborator

Summary

  • carry forward the merged ARM implementation of the three-phase SQ8 tiered-HNSW lifecycle
  • make backend publication and batch-iterator construction use scoped locks
  • make Phase-0 batch iterators deplete after draining the flat frontend
  • warn when tiered SQ8 is configured with dimensions below 64
  • exercise publication and migration concurrently with queries, batch iteration, and metadata reads

Scope

This PR is limited to MOD-14957. Estimator policy, submit-callback failure handling, RediSearch integration, and SQ8 kernel work remain outside this change.

Verification

Run on dorer-intel from an isolated temporary checkout:

  • test_hnsw_sq8: 92/92 passed
  • test_hnsw: 310/310 passed
  • ConcurrentQueriesDuringNormalizationTransition under ThreadSanitizer: passed with no race reports

No compilation or tests were run on the local development machine.

@dor-forer
dor-forer force-pushed the dor-forer-MOD-14957-tiered-hnsw-sq8 branch from 72c1fc1 to 19cb099 Compare September 2, 2026 10:15
@CLAassistant

CLAassistant commented Sep 2, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

* SQ8 for Tired HNSW Index

* Fix after rebase

Don't recreate tiered hnsw backend index

Make isInAccumulationPhase atomic

Add SQAccumulationState

Tiered hnsw fixes

* Add transition tests

* clean up

* Address review

* defered backend index init

* clean up

* fix lock
@dor-forer
dor-forer force-pushed the dor-forer-MOD-14957-tiered-hnsw-sq8 branch 3 times, most recently from 79d3e41 to 2e98a0e Compare September 2, 2026 11:30
@dor-forer
dor-forer force-pushed the dor-forer-MOD-14957-tiered-hnsw-sq8 branch from 2e98a0e to 173b6a0 Compare September 2, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants