Make sure deleted documents are not still searchable

Deleting a file from storage does not delete its vectors. Find chunks whose source file no longer exists, then prove whether a search can still return them.

What goes wrong

A document is withdrawn: an outdated contract, a recalled manual, a file a user asked you to erase. It is gone from storage, but the RAG app keeps quoting it. If the deletion was a legal request, the data is not actually deleted.

Why it happens

  • Deleting the file does not trigger anything in the index.
  • The cleanup job only handles new and changed files.
  • The delete ran against the wrong collection or tenant.

How to check (pgvector)

List documents that still have chunks but no longer exist in storage:

SELECT c.document_id, count(*) AS chunks
FROM chunks c
LEFT JOIN source_files s USING (document_id)
WHERE s.document_id IS NULL
GROUP BY c.document_id
ORDER BY c.document_id;

Then prove the content can really be returned: search with one of the document's own stored vectors and see whether it comes back near the top.

SELECT id, document_id
FROM chunks
ORDER BY embedding <=> (SELECT embedding FROM chunks WHERE document_id = 'deleted-doc.pdf' LIMIT 1)
LIMIT 5;

Replace deleted-doc.pdf with a document ID from the first query. The examples use the tables described in the guides introduction.

In Qdrant: compare the document IDs in your payloads with your bucket listing, then run a query using one of the leftover points as the query (Qdrant's query API accepts a point ID).

How to fix it

Delete the chunks by document ID, then run both queries again: the first should return nothing, and the second should no longer return the deleted document.

How to stop it happening again

  • Treat a deletion in storage as an event your pipeline must handle, like an upload.
  • For erasure requests, keep a record of the check showing the content can no longer be retrieved.

RAGGuard checks deletions on every audit and can prove a removed document no longer comes back in search: rag-integrity-platform.vercel.app.

Check all of this automatically

RAGGuard runs every check in these guides against your S3 bucket and your Qdrant or pgvector index, read-only, and shows the evidence and a safe fix for each problem. It is free during the beta.