Find half-indexed documents

An ingestion worker can die after writing only some of a document's chunks. Store the expected chunk count on every chunk and look for gaps.

What goes wrong

Questions about the first part of a long document work. Questions about the rest get "I don't know" or a vague answer. The document is in the index, so a check that only asks "is this document indexed?" says yes.

Why it happens

  • The worker was killed, timed out or ran out of memory partway through a document.
  • A batch write was partly rejected (a rate limit, a payload too large) and the error was swallowed.
  • A retry wrote the first chunks again but never reached the end.

How to check (pgvector)

When you write chunks, store each one's position (chunk_index) and the number of chunks the document should have (chunk_total). Then list documents with fewer chunks than expected:

SELECT document_id,
       max(chunk_total)            AS expected,
       count(DISTINCT chunk_index) AS present
FROM chunks
GROUP BY document_id
HAVING count(DISTINCT chunk_index) < max(chunk_total)
ORDER BY document_id;

Every row is a document that is only partly searchable. The examples use the tables described in the guides introduction.

Without a stored total you can still spot some gaps: a document whose highest chunk_index is larger than its number of chunks minus one is missing chunks in the middle. A missing tail cannot be seen that way, which is why the total matters.

In Qdrant: store chunk_index and chunk_total in the payload, scroll with payloads only, and count the distinct indexes per document.

How to fix it

Re-index the affected documents. If your chunking is deterministic, you can write only the missing chunks.

How to stop it happening again

  • Mark a document done only after counting its chunks in the index and comparing with the total.
  • Alert when this check returns anything after an ingestion run.

RAGGuard finds partly indexed documents on every audit, from the chunk total or from gaps in the numbering: rag-integrity-platform.vercel.app.

Check all of this automatically

RAGGuard runs every check in these guides against your S3 bucket and your Qdrant or pgvector index, read-only, and shows the evidence and a safe fix for each problem. It is free during the beta.