Add Inspeximus document store integration - #554
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Someone is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
Pull request overview
- Adds a new integration documentation page for the
inspeximusPython package, describingInspeximusDocumentStoreas a persistent, file-backed HaystackDocumentStorewith provable deletion semantics.
Changes:
- Adds
integrations/inspeximus.mdwith overview, install instructions, and end-to-end usage snippets. - Documents persistence + receipted erasure behavior and positions it as a drop-in alternative to
InMemoryDocumentStore.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
kacperlukawski
left a comment
There was a problem hiding this comment.
Thank you, @DanceNitra Could you please help me understand if the integration comes from an actual need of any user? I would like to know if that's going to be helpful for broader audience.
|
Honest answer: no, this didn't come from a user request. I built the adapter because inspeximus ships a document store and Haystack was a natural target, and I wanted the "drop-in" claim to be tested rather than asserted — so it runs against your own But that's an answer about correctness, not about demand, and demand is what you asked about. I don't have a user to point to. If your bar for the integrations catalogue is demonstrated need, this doesn't clear it, and I'd rather you close or park it than have it sit there implying an adoption that doesn't exist. For what it's worth on the "broader audience" question: the thing inspeximus does differently from an in-memory store is deletion — Happy either way. Thanks for looking at it. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
deepset-ai/haystack-integrations#554 has been open since 22 July. On 2 September the maintainer pushed a fix to our branch rather than closing it: `delete_documents` renamed to `erase_documents` in the paragraph about provable deletion. He was right and the error was ours. That paragraph describes a signed tombstone and a data-subject request, which is `erase_documents(request_id=...)` returning an erasure record. The protocol's `delete_documents` returns None and takes no request id. A doc page is a promise to a stranger with a fresh install, so this executes the promise rather than proof-reading it. It fetches the page at the PR head sha, runs every Python block in it, and checks the one sentence a reviewer cannot verify by reading: that erasure removes the value from the bytes on disk. Measured at 5ed457a on Haystack 3.0.0: the indexing and retrieval example prints ['the invoice is due in March']; erase_documents returns coverage, forgotten, ids, residue_in_store, scrubbed_links and tombstones; the erased text is gone from documents.json and the other document is still there. The two controls are the point. The probe asserts the victim's text IS in the file before erasing, so its later absence means something, and it requires a surviving document, so "the bytes are gone" cannot be satisfied by a truncated or empty file. Both were mutation-tested, along with a neutered erasure and a doc block forced to run unsubstituted. Pinned to the PR head sha rather than the branch name, because a branch moves and a claim about what reviewers will read must not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERJKW2XAjFoCkp8rnWdWJR
|
@kacperlukawski Thanks for the commit. The error was mine, and the autofix corrects it. Copilot The same mix-up does not appear anywhere else on the page. I could not check one of the page's claims by reading it, so I ran the examples at 5ed457a on
That last one asserts the text is in the file before the erase, and it requires the second document Copilot's other finding is still open. It flags On your July question, the answer has not changed. No user asked for this. Since July the page has Written with AI assistance. I maintain inspeximus, so treat the framing as interested. Everything Rastislav |
Adds a Document Store integration page for Inspeximus (MIT, on PyPI as
inspeximus).InspeximusDocumentStoreimplements Haystack'sDocumentStoreprotocol as a drop-in forInMemoryDocumentStore, with two differences that matter for long-running or regulated pipelines: it persists to a file, and itsdelete_documentsremoves the value from disk (with receipts enabled, leaving a signed, content-free tombstone so a deletion is provable).It is a faithful drop-in — the duplicate policies (
SKIP,OVERWRITE,NONE,FAIL) matchInMemoryDocumentStoreexactly (captured from the reference, not guessed), and filtering reuses Haystack's owndocument_matches_filter, soFilterRetrieverand pipeline serialization work unchanged. Parity is verified in CI againstInMemoryDocumentStore, operation by operation, with a falsification control that must fail.The usage example in the page was run end to end against
haystack-ai 3.0.0andinspeximus 1.29.0, and produces the output shown.No logo included for now; happy to add one if you'd like it before merge.
This page was drafted with AI assistance; a human reviewed it and ran the example.