Back to the blog

Private RAG Server: A Production Reference Architecture

A reference architecture for a private RAG server: parsing, chunking, hybrid retrieval, reranking, per-role access control, citations, evaluation and rollout.

Ingestion parses and chunks documents for indexing. Queries filter by role, run hybrid search, rerank results and return answers with citations.

A private RAG server that works in a demo and one that works in production differ in the parts nobody shows in the demo: parsing, permissions, evaluation and index freshness. The model at the end is the easy part. This article describes the reference architecture I use for RAG on a company’s own documents: two pipelines (ingestion and query), access control enforced inside retrieval, citations validated in code rather than trusted from the model, and a golden question set that exists before the first user does.

Key takeaways

  • Treat RAG as a search system with a language model attached. Most wrong answers trace back to parsing, chunking or retrieval, not generation.
  • Combine BM25 and vector search, then rerank. Each catches failures the other misses.
  • Enforce permissions as a filter in the retrieval query, using groups resolved per request from your identity provider.
  • Have the model cite chunk IDs, and map those to document and page on the server. Reject citations that were not retrieved.
  • Build the evaluation set first. Without it, every change to chunking or prompts is a guess.

The architecture at a glance

 INGESTION PIPELINE (async, incremental)

 File shares ─┐   ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌───────────────┐
 SharePoint  ─┼──▶│ Connectors │──▶│ Parsers    │──▶│ Chunker    │──▶│ Embed + index │
 Confluence  ─┤   │ + ACL sync │   │ PDF, DOCX  │   │ + metadata │   │ BM25 + vector │
 Uploads     ─┘   │ + deletes  │   │ XLSX, HTML │   │ + page no. │   │ (versioned)   │
                  └────────────┘   └────────────┘   └────────────┘   └───────┬───────┘
                                                                             ▼
                  ┌──────────────────────────────────────────────────────────────────┐
                  │ INDEX STORE: chunks, vectors, ACLs, page numbers, versions       │
                  └─────────────────────────────────┬────────────────────────────────┘
 QUERY PIPELINE (sync, per request)                 ▼
 ┌────────┐   ┌───────────────┐   ┌─────────────────────────┐   ┌────────────────────┐
 │ User   │──▶│ API gateway   │──▶│ Hybrid retrieval        │──▶│ Reranker           │
 │ (SSO)  │   │ authn, groups │   │ BM25 + vector           │   │ top 100 → 10-20    │
 └────────┘   │ rate limits   │   │ filter: allowed_groups  │   └─────────┬──────────┘
     ▲        └───────────────┘   └─────────────────────────┘             ▼
     │        ┌───────────────┐                                 ┌────────────────────┐
     └────────│ Citation      │◀────────────────────────────────│ Generation         │
              │ validator     │                                 │ answer + chunk IDs │
              └───────┬───────┘                                 └────────────────────┘
                      ▼
              Traces, eval runs, feedback ──▶ observability store

Ingestion is asynchronous and incremental. Query handling is synchronous and stateless. They share only the index store, which makes each side testable on its own.

Ingestion and parsing

Connectors do three jobs: fetch changed content, fetch the permissions on that content, and report deletions. The third is the one that gets forgotten, and it is how a withdrawn policy keeps appearing in answers.

Parsing depends on the format, and one generic text extractor is not enough:

  • PDF. Check whether a text layer exists. Scanned documents need OCR. Keep page numbers attached to every block, because citations depend on them. Multi-column layouts and tables need a layout-aware parser or the reading order comes out wrong.
  • Word. Use the heading styles. They give you the section hierarchy for free, which is the best chunk boundary you will get.
  • Excel. A sheet is a table, not prose. Serialise rows with their column headers repeated, keep sheet names, and consider whether the right answer is a SQL-style tool rather than embedding rows at all.
  • Confluence. Use the API’s storage format instead of scraping HTML. Keep the page tree, labels and last-modified date, and read both space permissions and page-level restrictions.

I normalise everything into one intermediate format (Markdown plus a metadata record) before chunking. Each record carries the source URL, document version hash, page or section, language, and the groups allowed to read it.

Chunking strategies and their trade-offs

Strategy How it works Good for Weakness
Fixed size with overlap Split every N tokens, overlap a little Uniform text, quick baseline Cuts through tables, lists and clauses
Structure-aware Split on headings, paragraphs, list items Policies, manuals, contracts Section sizes vary widely
Parent-child Retrieve small chunks, send the parent section to the model Precise matching with enough context More tokens per answer
Contextual Prepend a short generated description of where the chunk sits in the document Chunks that are ambiguous alone (“the fee is 2%”) One LLM call per chunk at ingestion
Row or table-aware Keep a table together, or one row with its headers Price lists, spec sheets Large tables still need a different tool

I start with structure-aware chunking and add parent-child retrieval when answers lack context. Contextual chunking is worth testing when documents refer to themselves implicitly. Anthropic’s write-up on contextual retrieval reports that it cut their top-20 retrieval failure rate from 5.7% to 3.7% with embeddings alone and to 2.9% when combined with BM25, with an ingestion cost they put at $1.02 per million document tokens under their stated assumptions. Those are their datasets, not yours, which is why the evaluation set comes first.

Embeddings

Three decisions matter more than leaderboard position.

Language coverage. If your documents are in Romanian and English, or any mix, test a multilingual model on your own questions. BGE-M3 is one open option: its model card lists more than 100 languages, inputs up to 8,192 tokens and an MIT licence, and it can run on your own hardware.

Where it runs. An embedding API is simpler to operate. A local model keeps document text inside your network. This choice is independent of where generation runs.

Versioning. Store the model name and version with every vector. Vectors from different models are not comparable, so a model change means a full re-embed into a new index and a switch-over, never a gradual mix.

Hybrid retrieval and reranking

Vector search handles paraphrase. BM25 handles exact strings: contract numbers, product codes, error codes, names. Company documents are full of exact strings, so I run both and fuse the results with Reciprocal Rank Fusion, which only needs ranks, not comparable scores. The original paper fixed its constant at k = 60 and found the exact value not critical.

from collections import defaultdict


def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Reciprocal Rank Fusion over ranked lists of chunk IDs (best first)."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)


def retrieve(query: str, user_groups: set[str], top_k: int = 20) -> list[str]:
    # Groups come from the identity provider for this request, never from the prompt.
    acl = {"allowed_groups": sorted(user_groups)}
    lexical = bm25_search(query, filters=acl, limit=100)
    dense = vector_search(embed(query), filters=acl, limit=100)
    return [chunk_id for chunk_id, _ in rrf([lexical, dense])[:top_k]]

bm25_search, vector_search and embed are your index and model clients. The point of the snippet is the order of operations: the permission filter goes into both searches, before fusion.

A reranker then scores each candidate against the query with a cross-encoder and keeps the best 10 to 20. In Anthropic’s tests, retrieving 150 candidates and reranking down to 20 took the failure rate from 2.9% to 1.9%. Reranking adds latency and a second model to operate, so measure whether it earns its place on your data.

Access control at retrieval time

The rule: a chunk the user may not read must never enter the prompt. Filtering after generation is too late, because the model has already seen the content and may paraphrase it.

  • Store allowed_groups (or an equivalent ACL reference) on every chunk at ingestion.
  • Resolve the user’s groups from the identity provider on every request. Do not accept them from the client or from the conversation.
  • Apply the filter inside the index query, and deny by default when a chunk has no ACL.
  • Sync permission changes more often than content. A revoked permission that takes a day to reach the index is a day-long leak.
  • Key any answer cache by permission set, not only by question.

Check how your vector store combines filters with approximate indexes. In pgvector, for example, WHERE conditions are applied after an approximate index scan, so a restrictive filter can return fewer rows than requested; iterative index scans, available from version 0.8.0, address this. OWASP lists vector and embedding weaknesses as its own entry (LLM08) in the 2025 Top 10 for LLM applications, and cross-tenant or cross-role leakage is the practical form it takes in RAG.

Answers with citations to document and page

Each chunk goes into the prompt with an ID. The model is instructed to answer only from those chunks, to return the IDs it used, and to say that the documents do not contain the answer when they do not. The server, not the model, turns IDs into “Document, page N” links.

def resolve_citations(cited_ids: list[str], retrieved: dict[str, dict]) -> list[dict]:
    unknown = [cid for cid in cited_ids if cid not in retrieved]
    if unknown:
        raise ValueError(f"Cited chunks were not retrieved: {unknown}")
    return [
        {"document": retrieved[cid]["title"], "page": retrieved[cid]["page"]}
        for cid in cited_ids
    ]

This removes one class of hallucinated citation entirely: the model cannot invent a file name or page number, because it never writes one. It can still cite a real chunk that does not support the claim, which is what groundedness evaluation is for.

Evaluation

Build a golden set before tuning anything: real questions from the people who will use the system, each with the expected source document and page. Include questions the documents cannot answer, and permission tests of the form “user in group A must not receive content from document B”.

Layer Metric What it tells you
Retrieval Recall@k, MRR Whether the right chunk is found, and how high
Generation Groundedness / faithfulness Share of claims in the answer that the retrieved context supports (how Ragas defines it, on a 0 to 1 scale)
Citations Citation correctness Whether the cited page contains the claim
Behaviour Abstention rate on unanswerable questions Whether the system admits it does not know
Security Permission test pass rate Must be 100%; anything else blocks release

Run the set on every change to parsers, chunking, embeddings, prompts or models, the same way you run tests in CI.

Observability and cost drivers

Log one trace per query: user and group hash, the query, chunk IDs and scores after each stage, token counts, latency per stage, model versions, and user feedback. Track index freshness (time from a source change to its appearance in the index) and zero-result queries as first-class metrics.

Costs fall into three groups:

  • One-off and per-change: parsing and OCR, embedding, optional contextualisation. Re-embedding after a model change repeats this.
  • Per query: context tokens (chunks × chunk size), output tokens, reranker calls. Sending 20 large parent sections costs far more than 8 small chunks, and a tight context budget matters for the same reasons I describe in MCP tool calls vs. code execution.
  • Fixed: servers or GPUs, storage, backups, and the engineering time to keep connectors and evaluations running. This last item is usually underestimated.

Hosting options

Option Data location Operational effort Fits when
On-prem, local models Your hardware High: GPUs, model serving, updates Strict data residency, steady volume
Private cloud, models in your tenant Your cloud account Medium You already run workloads there
VPS for index and API, external model API Index with you, prompts sent to provider Low Contract terms allow it, volume is variable

Local models give you control and a fixed cost profile, at the price of operations work and, depending on the model, answer quality. API models reverse that trade. A mixed setup (local embeddings and index, API generation under a data processing agreement) is common. If the whole knowledge base is small, the question changes: Anthropic notes that under about 200,000 tokens you can put everything in the prompt and skip retrieval, although per-role permissions usually rule that out.

Failure modes

Failure Typical cause Detection Mitigation
Stale index Missed updates or deletions Freshness metric, source-vs-index diff Incremental sync with tombstones, scheduled full reconciliation
Permission leak ACL sync lag, post-filtering, shared cache Permission tests in the golden set Filter in the query, deny by default, fast revocation sync
Chunk-boundary error Clause or table split across chunks Low groundedness on specific documents Structure-aware or parent-child chunking
Hallucinated citation Model writes references as free text Citation validator failures IDs only, server-side resolution, regenerate on failure
Wrong table values Parser flattened a table Spot checks on numeric answers Table-aware parsing, or a query tool for tabular data

A realistic rollout

  1. Discovery. Pick one document domain and one user group. Collect real questions and write the golden set. Map sources and permission models.
  2. Baseline. Parsing, structure-aware chunks, hybrid retrieval, citations, the trace log. Measure against the golden set.
  3. Hardening. ACL sync, permission tests, reranking if it measurably helps, freshness monitoring, load testing.
  4. Pilot. A small group of real users with a feedback button. Review failed traces weekly and feed them into the golden set.
  5. Expansion. More sources and roles, one at a time, each with its own evaluation questions. Expose the retriever as a tool for agents once it is stable, following the principles in agent-ready API design.

How I build this with clients

I do not start a RAG project from an empty repository. The connectors, parsing pipeline, hybrid retriever, citation validator and evaluation harness described here exist as pre-configured modules that I adapt to a client’s sources, identity provider and hosting constraints. That is what the private RAG server in my AI solutions offer consists of, and it is why the first measurable baseline arrives early instead of after months of plumbing.

What I cannot bring pre-built is your golden question set and your permission model. If you want to go through both for your documents, get in touch for a technical call.

Sources