On-Premise RAG: Self-Hosted and Air-Gapped

Short answer: if your corpus cannot leave your network, that constraint decides the architecture before quality, cost or convenience enter the conversation. Fully on-premise RAG is achievable now — self-hosted embeddings, a vector store you run, an open-weight model behind your firewall — and the retrieval quality gap against hosted APIs has narrowed enough that it rarely decides the outcome. What you take on instead is GPU capacity, model updates and an inference service to operate.

Most teams asking about on-premise RAG have not yet worked out which of three architectures they actually need. Establish that first.

Three architectures, not two

Fully hosted. Documents, embeddings and queries all reach a third party. Cheapest to run, fastest to ship, and eliminated immediately by most data-residency rules.

Hybrid — self-hosted retrieval, hosted generation. Your documents are embedded and indexed inside your network. Only the retrieved passages relevant to a specific query leave, going to a hosted model for generation. The corpus never transits, and neither does anything the user did not ask about.

This is the option teams most often miss, and it is frequently the right one. It answers “our documents cannot go to a vendor” honestly while keeping the operational burden small. Whether it satisfies your rules is a question for your compliance team, not your engineers — the passages that leave are still your content, just a small and query-scoped amount of it.

Fully air-gapped. Everything inside the boundary: embedding model, vector store, LLM, no egress at all. Required for classified work, some clinical and defence settings, and anywhere the answer to “can any content leave?” is a flat no.

The cost difference between these three is large. Ask your compliance team which one the rules actually require rather than assuming the strictest.

Which of the three your compliance position actually requires is the first question we settle in on-premise RAG development.

What self-hosting each layer costs

Embeddings are the easy part and the honest place to start. Open-weight embedding models now match or exceed closed-source on retrieval across most domains, they are small enough to run on modest hardware, and embedding is a batch workload you can schedule. There is very little downside to self-hosting this layer, and it removes the largest data-egress surface, because embedding means sending your entire corpus to a vendor rather than a query.

Check the licence before you commit, though — “open weights” and “free for commercial use” are different claims, and at least one popular family is non-commercial by default. We covered that in choosing an embedding model for RAG.

The vector store is straightforwardly self-hostable. pgvector on Postgres you already operate is the boring correct answer for most corpora, and it puts retrieval inside your existing backup, access-control and audit story rather than adding a new system.

Generation is where the real cost sits. Serving an open-weight model at production latency means GPUs, a serving stack, capacity planning for concurrency, and a team that owns model updates. This is a genuine operational commitment, not a deployment detail — and it is the layer where the hybrid option earns its place.

Where quality actually differs

Not where people expect.

Retrieval quality is dominated by chunking, hybrid search and reranking, none of which depend on where the model runs. A self-hosted pipeline with structure-aware chunking and BM25 alongside vector search will comfortably beat a hosted pipeline with fixed-size splitting. The architecture decision and the quality decision are much less coupled than vendors imply.

Generation quality is where the gap remains, and it is narrower than it was and workload-dependent. For grounded question answering over retrieved passages — which is what RAG generation is — open-weight models perform well, because the hard reasoning was done by the retriever. For long multi-step reasoning over the retrieved context, frontier models still lead.

Test on your own questions. This is measurable in an afternoon with an evaluation set, and it beats any benchmark argument.

What you are actually signing up for

Be clear-eyed about the operational load, because this is where on-premise projects fail rather than at the model:

GPU capacity and utilisation. Idle GPUs are expensive; saturated ones queue. Concurrency planning is a real workstream.
Model updates. A hosted API improves silently. A self-hosted model is exactly as good as the day you deployed it until someone upgrades it, revalidates it and redeploys.
Serving infrastructure. Batching, quantisation, KV caching, failover.
Someone who owns it. An unowned self-hosted stack degrades quietly and nobody notices until answers get worse.

If you do not have or plan to have that capability, the hybrid architecture is a better answer than a fully self-hosted one you cannot maintain.

Permissions matter more, not less

On-premise does not solve access control, and being inside your own network can make it feel like it has.

If different users may see different documents, retrieval must respect that per user — a shared index that any authenticated user can query has flattened your permission model regardless of where it runs. That constraint reaches back into ingestion, and it is the requirement most often discovered late. The patterns are in enterprise RAG architecture.

The takeaway

Decide which of the three architectures your rules actually require before designing anything — the answer is often the hybrid, and the cost difference is large. Self-host embeddings and the vector store in almost every case; they are cheap and they remove the biggest egress surface. Reserve fully air-gapped generation for when the rules genuinely demand it, and be honest about whether you can operate it.

EpochC provides custom RAG development services, including on-premise, self-hosted and air-gapped builds where data residency requires it. See the graph + vector retrieval case study — +40% query accuracy — or start a project.

On-premise RAG: building retrieval that never leaves your network