Production RAG for high-stakes document intelligence
Three retrieval models, one grounded pipeline. Mortgages, claims, policies, filings, and long-form text. Runs on your own GPUs, stays inside your network, masks PII before anything reaches a model, and keeps a human reviewer in the loop.
Live pipeline: a 250-page loan package, end to end
Watch one package flow from ingest to cited answer. Counters below show the cost and time it saves.
PII fields masked
·
3 secrets scrubbed
Context tokens sent
·
vs 48,200 naive (96% smaller)
Input cost / query
·
vs $0.41 (93% lower)
Analyst review time
·
vs 52 min / package
Any modality enters, but raw PHI never leaves the trust boundary.
Chunking, embeddings, retrieval, prompting
Retrieval quality decides whether RAG is trustworthy. These are the techniques and frameworks a production system chooses from, and what this project exercises across its three pipelines.
Chunking strategies
Frameworks: LangChain TextSplitter, LlamaIndex NodeParser, Unstructured, Marker, semantic-chunker.
Prompting techniques
Paired with constrained decoding so the model always returns grounded, schema-valid answers with inline citations.
Embedding families
Dense
BGE-M3 · E5-Mistral · NV-Embed-v2 · GTE · Cohere embed v3
Sparse
BM25 · SPLADE · BGE-M3 sparse
Late-interaction
ColBERT v2 · JaColBERT
Multimodal
CLIP · SigLIP · Jina-CLIP
Cost levers
- Matryoshka representation (truncate dimensions at query time)
- int8 / binary quantization (10 to 30x smaller index)
- Deduplication and content-addressed caching
- Domain-adaptive fine-tuning on your corpus
Serving: sentence-transformers, FlagEmbedding, Text Embeddings Inference (TEI), Infinity, FastEmbed.
Retrieval & reranking
- Weighted-alpha fusion and Reciprocal Rank Fusion (RRF)
- ANN indexes: FAISS, HNSW (hnswlib), LanceDB, Qdrant, Milvus, Weaviate, pgvector, Chroma, Vespa
- Query expansion, HyDE, multi-query generation
- Cross-encoder rerank: BGE-reranker, Cohere Rerank
Vector retrieval, accurately
Loading 3D scene…
Where RAG fails
RAG quality is the product of four stages. Each has characteristic failure modes, and each is exactly what an evaluation should measure and a guardrail should catch.
Chunking
Severed boundaries
Fixed-size splits cut mid-sentence or mid-clause, orphaning a fact from its context.
Size trade-off
Too small loses surrounding meaning; too large dilutes relevance and burns the context window.
Structure loss
Tables, lists and forms get flattened and scattered across chunks; fee tables and EOBs suffer most.
Overlap drift
Too little overlap breaks continuity; too much multiplies cost and duplicate retrievals.
Embedding
Domain mismatch
Generic embeddings underperform on clinical, legal and financial vocabulary; synonyms outrank exact codes.
Rare-term blindness
ICD codes, ISINs, policy numbers and proper nouns map to weak, generic vectors.
Multilingual drift
Cross-lingual spaces are noisier; mixed-language docs retrieve inconsistently.
Quantization error
Compressed vectors + ANN approximation compound, returning near-but-wrong neighbors.
Retrieval
Vocabulary mismatch
Sparse misses paraphrase; dense misses exact keywords. Get the hybrid α wrong and both suffer.
Low recall
ANN ef_search / nprobe too small, or top-k too tight, and the right passage never enters context.
No rerank
First-pass scores aren't precision-ordered; relevant chunks sit below noise without a cross-encoder.
Lost in the middle
Even when retrieved, evidence buried mid-context is attended to poorly by the model.
Generation
Hallucination
The model generates plausible text beyond, or contrary to, the retrieved evidence.
Unfaithful synthesis
Numbers, dates and names get transposed or conflated across sources.
Conflicting context
Two passages disagree and the model silently picks one, or blends them into a wrong answer.
Citation failure
Claims go un-cited, or are tagged with the wrong source, defeating verifiability.
The next two disciplines close the loop: evaluation quantifies these failures continuously, and guardrails prevent them from reaching the user.
A grounded, guardrailed chat
A scripted run showing cited answers, PII refusal, and human-in-the-loop escalation. Data is illustrative.
Live this session
Keep it yours, keep it locked down
Guardrails gate both sides of the model. The whole stack can run offline, on your own GPUs, inside your network, with an on-prem vault holding every identifier.
Input guardrails
PII / PHI redaction
Detect and mask identifiers (names, DOB, MRN, SSN, account numbers) before they reach the model. HIPAA, GDPR, GLBA-aligned.
Secret & key scrubbing
API keys, tokens and credentials pasted into a document are stripped so they can never leak into a prompt or response.
Prompt-injection & scope
Flag embedded 'ignore previous instructions', indirect injection from document text, and refuse off-domain requests.
Output guardrails
Leak re-scan
Re-scan every response for identifiers or secrets; redact in-place before delivery.
Groundedness & citations
Faithfulness check plus citation verification so claims trace to a real source span.
Sensitive-request refusal
Requests for identifiers, credentials, or out-of-scope data are blocked and audit-logged.
Deployment & isolation
WiFi / LAN-only
Bind every service to a private subnet, enforce a zero-egress proxy, and require mTLS between components. Nothing leaves the building.
Offline / air-gapped
No internet required. Embeddings (TEI, Infinity), reranker, and LLM (vLLM, TGI, Ollama) run on local GPUs with an on-prem vault.
Your own server GPUs
Self-host the full stack: no per-token fees, no third-party calls. Example node: 4x A100 serves 340 queries/sec at p50 740ms.
Extra encryption layer
TLS in transit, AES-256 at rest, envelope encryption via KMS, field-level encryption for PHI, and tamper-evident audit logs.
Measure it before you trust it
Retrieval and generation are measured independently, then end to end. A golden set per domain (claims, policies, filings) runs on every change, with LLM-as-judge plus human spot-checks.
Retrieval metrics
Recall@k
relevant retrieved / total relevant
Did the right passage make it into context?
Precision@k
relevant retrieved / k
How much of the context is actually useful?
MRR
mean of 1 / rank of first hit
How high does the first relevant chunk rank?
NDCG
DCG / iDCG, graded relevance
Are the most relevant chunks ranked first?
Generation metrics
Faithfulness
grounded claims / total claims
The core anti-hallucination metric.
Context precision
relevant chunks / retrieved chunks
Catches noise diluting the prompt.
Context recall
answer covered by context
Did retrieval supply what was needed?
Answer relevancy
LLM-judge vs. query intent
Does it actually answer the question?
Cheaper, faster, still accountable
Illustrative numbers for a mid-volume production deployment. The pattern: cut token cost and review time, keep a human reviewer on everything that matters.
$2,390
Token input cost
87% lower on 1.2M queries
1,840
Prompt context
96% smaller
7 min
Review time
per 250-page package
4.8x
Analyst capacity
same headcount
99.2%
Citation accuracy
hallucination 0.4% to 0.03%
340 q/s
Throughput
single node, 4x A100
How token cost comes down
- Smaller chunks and rerank send far fewer tokens to the model.
- A semantic cache answers repeat and near-duplicate questions for free.
- Prompt compression (LLMLingua) trims context without losing meaning.
- Quantized embeddings shrink the index 10 to 30x with negligible recall loss.
Human stays in the loop
Every answer below confidence 0.7, every cross-source conflict, and every sensitive request is routed to a reviewer queue with full source traceability. Reviewers see the exact passages and the model's reasoning, approve or edit, and the decision feeds back into the eval set.
0.7
confidence floor
100%
auditable
4.8x
reviewer capacity
Try it with your own models
Configure up to three OpenAI-compatible models, point at a PDF, image, or pasted text, and ask a question. Identifiers and secrets are masked before the model and the answer is scanned for leakage. No keys or data stored server-side.
1 · Models (bring your own)
0/3 configured · OpenAI-compatibleStored only in this browser's localStorage, never sent to the server except as the Authorization header for your own request, over HTTPS.
2 · Source (multimodal)
0 chars
3 · Ask
Configure at least one model above to begin.