Skip to content

Add Inspeximus document store integration - #554

Open
DanceNitra wants to merge 2 commits into
deepset-ai:mainfrom
DanceNitra:add-inspeximus
Open

Add Inspeximus document store integration#554
DanceNitra wants to merge 2 commits into
deepset-ai:mainfrom
DanceNitra:add-inspeximus

Conversation

@DanceNitra

Copy link
Copy Markdown

Adds a Document Store integration page for Inspeximus (MIT, on PyPI as inspeximus).

InspeximusDocumentStore implements Haystack's DocumentStore protocol as a drop-in for InMemoryDocumentStore, with two differences that matter for long-running or regulated pipelines: it persists to a file, and its delete_documents removes the value from disk (with receipts enabled, leaving a signed, content-free tombstone so a deletion is provable).

It is a faithful drop-in — the duplicate policies (SKIP, OVERWRITE, NONE, FAIL) match InMemoryDocumentStore exactly (captured from the reference, not guessed), and filtering reuses Haystack's own document_matches_filter, so FilterRetriever and pipeline serialization work unchanged. Parity is verified in CI against InMemoryDocumentStore, operation by operation, with a falsification control that must fail.

The usage example in the page was run end to end against haystack-ai 3.0.0 and inspeximus 1.29.0, and produces the output shown.

No logo included for now; happy to add one if you'd like it before merge.

This page was drafted with AI assistance; a human reviewed it and ran the example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@DanceNitra
DanceNitra requested a review from a team as a code owner July 22, 2026 12:43
@vercel

vercel Bot commented Jul 22, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

  • Adds a new integration documentation page for the inspeximus Python package, describing InspeximusDocumentStore as a persistent, file-backed Haystack DocumentStore with provable deletion semantics.

Changes:

  • Adds integrations/inspeximus.md with overview, install instructions, and end-to-end usage snippets.
  • Documents persistence + receipted erasure behavior and positions it as a drop-in alternative to InMemoryDocumentStore.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread integrations/inspeximus.md Outdated
Comment thread integrations/inspeximus.md

@kacperlukawski kacperlukawski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @DanceNitra Could you please help me understand if the integration comes from an actual need of any user? I would like to know if that's going to be helpful for broader audience.

@DanceNitra

Copy link
Copy Markdown
Author

Honest answer: no, this didn't come from a user request. I built the adapter because inspeximus ships a document store and Haystack was a natural target, and I wanted the "drop-in" claim to be tested rather than asserted — so it runs against your own InMemoryDocumentStore in CI, with a control that deliberately breaks ours and requires the comparison to fail.

But that's an answer about correctness, not about demand, and demand is what you asked about. I don't have a user to point to. If your bar for the integrations catalogue is demonstrated need, this doesn't clear it, and I'd rather you close or park it than have it sit there implying an adoption that doesn't exist.

For what it's worth on the "broader audience" question: the thing inspeximus does differently from an in-memory store is deletion — delete_documents removes the value from disk, not just from the index — plus provenance on writes. That matters to people with a retention or erasure obligation and is irrelevant to everyone else, so it's a narrow audience by design.

Happy either way. Thanks for looking at it.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
DanceNitra added a commit to DanceNitra/inspeximus that referenced this pull request Sep 3, 2026
deepset-ai/haystack-integrations#554 has been open since 22 July. On 2 September the maintainer
pushed a fix to our branch rather than closing it: `delete_documents` renamed to `erase_documents` in
the paragraph about provable deletion. He was right and the error was ours. That paragraph describes
a signed tombstone and a data-subject request, which is `erase_documents(request_id=...)` returning
an erasure record. The protocol's `delete_documents` returns None and takes no request id.

A doc page is a promise to a stranger with a fresh install, so this executes the promise rather than
proof-reading it. It fetches the page at the PR head sha, runs every Python block in it, and checks
the one sentence a reviewer cannot verify by reading: that erasure removes the value from the bytes
on disk.

Measured at 5ed457a on Haystack 3.0.0: the indexing and retrieval example prints
['the invoice is due in March']; erase_documents returns coverage, forgotten, ids, residue_in_store,
scrubbed_links and tombstones; the erased text is gone from documents.json and the other document is
still there.

The two controls are the point. The probe asserts the victim's text IS in the file before erasing,
so its later absence means something, and it requires a surviving document, so "the bytes are gone"
cannot be satisfied by a truncated or empty file. Both were mutation-tested, along with a neutered
erasure and a doc block forced to run unsubstituted.

Pinned to the PR head sha rather than the branch name, because a branch moves and a claim about what
reviewers will read must not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ERJKW2XAjFoCkp8rnWdWJR
@DanceNitra

Copy link
Copy Markdown
Author

@kacperlukawski Thanks for the commit. The error was mine, and the autofix corrects it. Copilot
flagged it on 29 July: the paragraph describes a signed tombstone and a provable data-subject
deletion, which is erase_documents(request_id=...) returning an erasure record. The protocol's
delete_documents returns None and takes no request id, so the sentence named the wrong method for
what it described.

The same mix-up does not appear anywhere else on the page. erase_documents is the only method of
ours it calls, and InspeximusDocumentStore the only class. add_component and run are Pipeline
methods.

I could not check one of the page's claims by reading it, so I ran the examples at 5ed457a on
Haystack 3.0.0. Both Python blocks execute:

  • the indexing and retrieval example prints ['the invoice is due in March']
  • erase_documents returns a dict with coverage, forgotten, ids, residue_in_store,
    scrubbed_links and tombstones
  • the erased document's text is no longer in the bytes of documents.json, and the other document
    still is

That last one asserts the text is in the file before the erase, and it requires the second document
to survive, so an empty or truncated file fails it.

Copilot's other finding is still open. It flags version: Haystack 2.0 as conflicting with a 3.0
example. I left the front matter alone: 179 of the 181 pages in integrations/ on main carry exactly
version: Haystack 2.0, one has 2.0+, one has 3.0, and the adapter requires haystack-ai>=2
with no upper bound. If you read the field differently, change it.

On your July question, the answer has not changed. No user asked for this. Since July the page has
been checked against running code rather than my memory of the API, and that is the only difference.
If demand is the bar, closing it is the right call.

Written with AI assistance. I maintain inspeximus, so treat the framing as interested. Everything
above is reproducible from the page at 5ed457a with Haystack 3.0.0.

Rastislav

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants