Hybrid Search with Citations: Combining Vectors and Keywords

An assistant over internal documents is usually described in terms of the model. In practice the answer quality tracks the retrieval step: the model can only cite what it was given. A single retrieval strategy fails in predictable ways, which is why most production systems combine two.

This is a reference design for hybrid retrieval with citations. It is not a report of a BuildVault deployment, and it does not evaluate specific embedding models.

Two retrievers, different failures

A query like "why did the 2024 renewal fail for account A-4417" contains an identifier that embeddings tend to smooth away and prose the keyword index cannot rank. Neither retriever is wrong; they are wrong at different things.

Fuse the rankings, not the scores

Vector distances and text-search ranks are not comparable numbers. Reciprocal rank fusion avoids inventing a conversion:

export function fuse(rankings: string[][], k = 60, limit = 20) {
  const scores = new Map<string, number>();
  for (const ranking of rankings) {
    ranking.forEach((docId, index) => {
      scores.set(docId, (scores.get(docId) ?? 0) + 1 / (k + index + 1));
    });
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .slice(0, limit)
    .map(([docId]) => docId);
}

Retrieve more from each side than you intend to return — a wider candidate pool per retriever is what makes fusion useful — then rerank or truncate. If you need a learned reranker, add it after fusion and compare it against the fusion-only baseline on the same evaluation set.

Chunking decisions you cannot undo later

Chunk size determines what a citation can point at. A 2,000-token chunk that spans three sections cannot produce a precise citation, and asking the model to quote from it invites paraphrase drift. Store structure rather than only text:

• document identifier, title, and canonical source URL
• heading path (for example Billing > Proration > Mid-cycle changes)
• character offsets into the extracted text, so a quote can be verified against the source
• extraction timestamp and document version

Citations are only credible if a reader can open the source and find the sentence. Keep the offsets and render the quote from the stored text, not from the model's reproduction of it.

One database is often enough

PostgreSQL can hold both indexes, which keeps a single backup, permission model, and transaction boundary. With the pgvector extension alongside a stored tsvector column:

SELECT id, chunk_text, source_url, heading_path
FROM document_chunk
WHERE tenant_id = $1
ORDER BY embedding <=> $2
LIMIT 40;

SELECT id, chunk_text, source_url, heading_path
FROM document_chunk
WHERE tenant_id = $1
  AND to_tsvector('english', chunk_text) @@ plainto_tsquery('english', $2)
ORDER BY ts_rank_cd(to_tsvector('english', chunk_text), plainto_tsquery('english', $2)) DESC
LIMIT 40;

The PostgreSQL full-text search documentation covers ranking functions and query construction in more detail. Two separate queries fused in the application are easier to reason about than one clever query; start there and combine them only when the round trip measurably matters.

Filter before you rank

Permission and tenancy filters belong in the retrieval query, not in a post-processing step over results. Retrieving chunks the caller may not read and then discarding them leaks information through timing and through any log that records the candidate set. Filtering early also keeps the candidate pool relevant, which improves fusion.

Evaluate retrieval and citations separately

Two different questions, two different measurements:

• Retrieval: on a fixed set of questions with known answers, how often does the correct document appear in the top k? Track this per retriever and for the fused result.
• Citations: what share of answers cite a source that actually contains the claim? This requires human review of a sample; automated judging tends to reward confident phrasing.

Log which retriever contributed each cited chunk. Without that, a regression in one half of the system looks like "the model got worse".

Failure modes worth designing for

• No supporting document. The correct behavior is to say the source was not found. A system that always produces an answer cannot be trusted on the questions that matter.
• Stale index. Documents change or disappear; citations pointing at a deleted page destroy confidence quickly. Reconcile the index against the source of truth on a schedule.
• Tenant leakage. Covered by filtering before ranking, and by testing with a second tenant's data in the index.
• Duplicate chunks. Overlapping windows produce near-identical citations; deduplicate by document and heading path before presenting evidence.

A validation checklist

1. The evaluation set includes questions with identifiers and exact phrases, not only conceptual ones.
2. Fusion is compared against each retriever alone, on the same set.
3. Every citation resolves to stored text, and the quoted span matches it.
4. A question with no supporting document returns an explicit "not found in sources".
5. Retrieval runs under a tenant filter, tested with two tenants in one index.

For the client-side counterpart of this problem — handling documents without sending them to a server at all — see browser-side PDF tooling.