Multimodal · self-hostable · human-in-the-loop

Production RAG for high-stakes document intelligence

Three retrieval models, one grounded pipeline. Mortgages, claims, policies, filings, and long-form text. Runs on your own GPUs, stays inside your network, masks PII before anything reaches a model, and keeps a human reviewer in the loop.

PDF · image · text Hybrid retrieval (BM25 + dense) On-prem GPU PII-masked & guardrailed

Live pipeline: a 250-page loan package, end to end

Watch one package flow from ingest to cited answer. Counters below show the cost and time it saves.

Ingest: multimodal source01 / 09

PII fields masked

·

3 secrets scrubbed

Context tokens sent

·

vs 48,200 naive (96% smaller)

Input cost / query

·

vs $0.41 (93% lower)

Analyst review time

·

vs 52 min / package

your VPC · on-prem
PDF
Image
Text
EOB · Claim #4471

Any modality enters, but raw PHI never leaves the trust boundary.

The retrieval stack

Chunking, embeddings, retrieval, prompting

Retrieval quality decides whether RAG is trustworthy. These are the techniques and frameworks a production system chooses from, and what this project exercises across its three pipelines.

Chunking strategies

Fixed-size token windowSentence-awareRecursive characterSemantic (similarity thresholds)Layout / structure-aware (tables & forms)Parent-child / small-to-bigLate chunkingAgentic (LLM-decided boundaries)

Frameworks: LangChain TextSplitter, LlamaIndex NodeParser, Unstructured, Marker, semantic-chunker.

Prompting techniques

Zero / few-shotChain-of-thoughtSelf-consistencyReActChain-of-VerificationStep-back promptingMap-reduce / Refine / StuffCitation-prompting (forced inline tags)Constrained decoding (outlines, lm-format-enforcer)Self-RAG / CRAG reflection

Paired with constrained decoding so the model always returns grounded, schema-valid answers with inline citations.

Embedding families

Dense

BGE-M3 · E5-Mistral · NV-Embed-v2 · GTE · Cohere embed v3

Sparse

BM25 · SPLADE · BGE-M3 sparse

Late-interaction

ColBERT v2 · JaColBERT

Multimodal

CLIP · SigLIP · Jina-CLIP

Cost levers

  • Matryoshka representation (truncate dimensions at query time)
  • int8 / binary quantization (10 to 30x smaller index)
  • Deduplication and content-addressed caching
  • Domain-adaptive fine-tuning on your corpus

Serving: sentence-transformers, FlagEmbedding, Text Embeddings Inference (TEI), Infinity, FastEmbed.

Retrieval & reranking

  • Weighted-alpha fusion and Reciprocal Rank Fusion (RRF)
  • ANN indexes: FAISS, HNSW (hnswlib), LanceDB, Qdrant, Milvus, Weaviate, pgvector, Chroma, Vespa
  • Query expansion, HyDE, multi-query generation
  • Cross-encoder rerank: BGE-reranker, Cohere Rerank
fused = Σ 1 / (k + rank)  ·  hybrid α blend  ·  cross-encoder rerank

Vector retrieval, accurately

Loading 3D scene…

Where it breaks

Where RAG fails

RAG quality is the product of four stages. Each has characteristic failure modes, and each is exactly what an evaluation should measure and a guardrail should catch.

Chunking

  • Severed boundaries

    Fixed-size splits cut mid-sentence or mid-clause, orphaning a fact from its context.

  • Size trade-off

    Too small loses surrounding meaning; too large dilutes relevance and burns the context window.

  • Structure loss

    Tables, lists and forms get flattened and scattered across chunks; fee tables and EOBs suffer most.

  • Overlap drift

    Too little overlap breaks continuity; too much multiplies cost and duplicate retrievals.

Embedding

  • Domain mismatch

    Generic embeddings underperform on clinical, legal and financial vocabulary; synonyms outrank exact codes.

  • Rare-term blindness

    ICD codes, ISINs, policy numbers and proper nouns map to weak, generic vectors.

  • Multilingual drift

    Cross-lingual spaces are noisier; mixed-language docs retrieve inconsistently.

  • Quantization error

    Compressed vectors + ANN approximation compound, returning near-but-wrong neighbors.

Retrieval

  • Vocabulary mismatch

    Sparse misses paraphrase; dense misses exact keywords. Get the hybrid α wrong and both suffer.

  • Low recall

    ANN ef_search / nprobe too small, or top-k too tight, and the right passage never enters context.

  • No rerank

    First-pass scores aren't precision-ordered; relevant chunks sit below noise without a cross-encoder.

  • Lost in the middle

    Even when retrieved, evidence buried mid-context is attended to poorly by the model.

Generation

  • Hallucination

    The model generates plausible text beyond, or contrary to, the retrieved evidence.

  • Unfaithful synthesis

    Numbers, dates and names get transposed or conflated across sources.

  • Conflicting context

    Two passages disagree and the model silently picks one, or blends them into a wrong answer.

  • Citation failure

    Claims go un-cited, or are tagged with the wrong source, defeating verifiability.

The next two disciplines close the loop: evaluation quantifies these failures continuously, and guardrails prevent them from reaching the user.

The assistant, live

A grounded, guardrailed chat

A scripted run showing cited answers, PII refusal, and human-in-the-loop escalation. Data is illustrative.

assistant · online

Live this session

Tokens used410
Est. input cost$0.011
Citations0
PII masked31
Leaks blocked0
Grounded in sources
Human in the loop
Security, guardrails & PII

Keep it yours, keep it locked down

Guardrails gate both sides of the model. The whole stack can run offline, on your own GPUs, inside your network, with an on-prem vault holding every identifier.

Input guardrails

  • PII / PHI redaction

    Detect and mask identifiers (names, DOB, MRN, SSN, account numbers) before they reach the model. HIPAA, GDPR, GLBA-aligned.

  • Secret & key scrubbing

    API keys, tokens and credentials pasted into a document are stripped so they can never leak into a prompt or response.

  • Prompt-injection & scope

    Flag embedded 'ignore previous instructions', indirect injection from document text, and refuse off-domain requests.

Output guardrails

  • Leak re-scan

    Re-scan every response for identifiers or secrets; redact in-place before delivery.

  • Groundedness & citations

    Faithfulness check plus citation verification so claims trace to a real source span.

  • Sensitive-request refusal

    Requests for identifiers, credentials, or out-of-scope data are blocked and audit-logged.

Deployment & isolation

WiFi / LAN-only

Bind every service to a private subnet, enforce a zero-egress proxy, and require mTLS between components. Nothing leaves the building.

Offline / air-gapped

No internet required. Embeddings (TEI, Infinity), reranker, and LLM (vLLM, TGI, Ollama) run on local GPUs with an on-prem vault.

Your own server GPUs

Self-host the full stack: no per-token fees, no third-party calls. Example node: 4x A100 serves 340 queries/sec at p50 740ms.

Extra encryption layer

TLS in transit, AES-256 at rest, envelope encryption via KMS, field-level encryption for PHI, and tamper-evident audit logs.

Default posture: zero data egress, reversible tokenization vault on-prem, and the model only ever sees de-identified tags.
Evals

Measure it before you trust it

Retrieval and generation are measured independently, then end to end. A golden set per domain (claims, policies, filings) runs on every change, with LLM-as-judge plus human spot-checks.

Retrieval metrics

Recall@k

relevant retrieved / total relevant

Did the right passage make it into context?

Precision@k

relevant retrieved / k

How much of the context is actually useful?

MRR

mean of 1 / rank of first hit

How high does the first relevant chunk rank?

NDCG

DCG / iDCG, graded relevance

Are the most relevant chunks ranked first?

Generation metrics

Faithfulness

grounded claims / total claims

The core anti-hallucination metric.

Context precision

relevant chunks / retrieved chunks

Catches noise diluting the prompt.

Context recall

answer covered by context

Did retrieval supply what was needed?

Answer relevancy

LLM-judge vs. query intent

Does it actually answer the question?

Frameworks:RAGASTruLensDeepEvalArize PhoenixLangSmithpromptfoo
Impact

Cheaper, faster, still accountable

Illustrative numbers for a mid-volume production deployment. The pattern: cut token cost and review time, keep a human reviewer on everything that matters.

from $18,400 / mo

$2,390

Token input cost

87% lower on 1.2M queries

from 48,200 tok / query

1,840

Prompt context

96% smaller

from 52 min / package

7 min

Review time

per 250-page package

throughput multiplier

4.8x

Analyst capacity

same headcount

4,000 Q&A eval pairs

99.2%

Citation accuracy

hallucination 0.4% to 0.03%

p50 740ms, p95 2.1s

340 q/s

Throughput

single node, 4x A100

How token cost comes down

  • Smaller chunks and rerank send far fewer tokens to the model.
  • A semantic cache answers repeat and near-duplicate questions for free.
  • Prompt compression (LLMLingua) trims context without losing meaning.
  • Quantized embeddings shrink the index 10 to 30x with negligible recall loss.

Human stays in the loop

Every answer below confidence 0.7, every cross-source conflict, and every sensitive request is routed to a reviewer queue with full source traceability. Reviewers see the exact passages and the model's reasoning, approve or edit, and the decision feeds back into the eval set.

0.7

confidence floor

100%

auditable

4.8x

reviewer capacity

Run the pipeline

Try it with your own models

Configure up to three OpenAI-compatible models, point at a PDF, image, or pasted text, and ask a question. Identifiers and secrets are masked before the model and the answer is scanned for leakage. No keys or data stored server-side.

1 · Models (bring your own)

0/3 configured · OpenAI-compatible

Stored only in this browser's localStorage, never sent to the server except as the Authorization header for your own request, over HTTPS.

2 · Source (multimodal)

0 chars

3 · Ask

Configure at least one model above to begin.