RAG as a Service vs Building Your Own

Short answer: use RAG as a service when your corpus is small, non-sensitive and needs no custom retrieval logic. Build when permissions matter, when retrieval needs domain logic, or when query volume makes per-query pricing your dominant line item. Most teams should start hosted and only build when they hit one of those three walls.

We build retrieval systems, and we still tell many teams to use a hosted product. Start with the least complex option that meets your security, quality and operating requirements.

The two shapes

	RAG as a service	Building your own
Time to first answer	Hours	Weeks
Cost model	Per query or per seat, ongoing	One-time build, then hosting
Cost at scale	Grows with usage	Roughly flat
Permissions	Usually all-or-nothing	Per-user, modelled properly
Retrieval logic	Vendor’s	Whatever your domain needs
Where data sits	Vendor’s infrastructure	Yours
Who fixes a bad answer	Support ticket	You, same day

The pattern is the familiar one — rent versus own — but RAG has a specific wrinkle that makes the decision less about scale than people expect.

The three walls

Permissions. This is the one that catches teams out. A hosted index that cannot model who is allowed to see which document is a data leak waiting for its first incident. If your corpus contains anything where one user should not see another’s documents — HR records, client files, patient data, deal rooms — you need permission-aware retrieval, and that constraint reaches all the way back into ingestion. It is not a filter you apply to results afterwards, because the model has already seen the passages by then.

Most hosted products treat the corpus as uniformly readable. If yours is not, the wall is immediate and no amount of scale discussion is relevant.

Retrieval logic. Generic semantic search over chunks is a good default and it has a ceiling. When your domain has structure worth exploiting — relationships between entities, temporal ordering, hierarchical documents, code and its call graph — traversing that structure beats similarity search. On a clinical build, combining graph traversal with vector search took query accuracy up 40% against a pure-vector baseline, and no hosted product would have let us express that. We wrote up the trade-off in knowledge graphs vs vector search.

Per-query economics. Hosted RAG prices per query or per seat. That is fine at low volume and becomes the dominant line item at high volume, because your cost now scales with exactly the usage you are trying to grow. A system you own is roughly flat as usage rises. Where the crossover sits depends on your vendor and volume, but the shape is reliable: rent looks cheap until it is the biggest number on the page.

What custom actually buys you

Beyond clearing those walls, three things you cannot get from a hosted pipeline:

Chunking tuned to your documents. Chunking sets the ceiling on everything downstream. Fixed-size splitting cuts tables in half and separates a clause from the sentence that qualifies it. Structure-aware chunking that respects your document format — sections, tables, headers — recovers more accuracy than most model upgrades. Hosted products chunk generically because they have to.

Hybrid retrieval. Embeddings are excellent at meaning and unreliable at exact tokens. Part numbers, error codes, legal citations, drug names — a user searching for ERR_4021 wants that string, and a semantic model will happily return passages about similar-sounding errors. Running BM25 alongside vector search and fusing the results fixes a whole class of failure that looks mysterious otherwise.

Evaluation you own. You cannot improve what you cannot measure, and “it seems better” is not a measurement. A retrieval evaluation set built from real user questions, run on every change, is the difference between engineering and guessing. This matters more than any single architectural choice — see how to stop your RAG system from hallucinating.

Most RAG failures are retrieval failures

Worth stating plainly, because it changes what you should buy.

When a RAG system gives a bad answer, the instinct is to blame the model and reach for a bigger one. Usually the model was fine and the retrieval handed it the wrong passages. A larger model composing an answer from irrelevant context produces a more fluent wrong answer, not a right one.

Diagnose before you spend: take twenty questions the system answered badly and check whether the correct passage was in the retrieved set at all. If it was not, no model change will help, and no hosted vendor’s upgrade tier will either. This is also the cheapest way to establish whether you have a problem worth building for.

A sensible progression

You rarely have to choose once.

Start hosted. Prove the use case, learn what users actually ask, and build your evaluation set from real questions. This costs days, not weeks.
Measure retrieval quality against that set. If it is good and the economics work, stop — you are done, and you have saved a build.
Build when you hit a wall. Permissions, retrieval logic or per-query cost. You will now build against a real evaluation set and real usage patterns instead of assumptions, which makes the build substantially cheaper and better targeted.

The teams who regret building are almost always the ones who skipped step one and built against imagined requirements. The teams who regret not building usually hit the permissions wall in an audit.

What enterprise RAG changes

At enterprise scale two more constraints arrive. Freshness — an index that is a week stale will confidently cite superseded policy, so ingestion has to keep pace with the corpus. And auditability — you need to reconstruct which passages produced which answer months later, which means logging retrieval results, not just responses.

Both are covered in more depth in enterprise RAG architecture, along with the permission-modelling patterns that make multi-tenant retrieval safe.

Compare total cost, not the vendor’s query price

Per-query pricing is visible, but it is only one part of the operating cost. Include ingestion and re-indexing, storage, model tokens, reranking, observability, support, evaluation, access reviews and the time someone spends investigating bad answers. A hosted product may bundle several of these. A custom system may make each one a separate responsibility.

Estimate low, expected and peak usage from actual request logs. Include retries and agent-initiated searches if an application can make multiple retrieval calls per user request. Then compare the hosted quote with an owned-system estimate that includes engineering and on-call time. The point where costs cross is specific to your workload; no universal query threshold decides it.

Test a hosted product before signing

Use a small evaluation set built from questions users actually ask. For every question, record the answer, the expected source passages and who should be allowed to see them. Include exact identifiers, tables, older document versions, misspellings, ambiguous requests and queries that have no answer in the corpus.

Score retrieval separately from answer writing. Check whether the needed passage appeared in the retrieved context, whether restricted content stayed out, whether citations point to the right source, and whether the system admits when it lacks evidence. Repeat the same set after changing chunking, metadata, embeddings or ranking. A polished demo on a handful of clean questions does not establish production fit.

Before committing, confirm export options, deletion behavior, index freshness, audit logs, service limits, data residency and the path for a failed or delayed ingestion job. These details matter more than a feature checklist because they determine whether the service fits your production obligations.

When a hybrid design makes sense

The decision is not always hosted versus fully owned. Some teams keep managed embeddings or a hosted vector database while owning ingestion, permissions and retrieval orchestration. Others keep a hosted prototype for low-risk content and use a separate permission-aware system for sensitive records.

Hybrid designs can reduce migration risk, but they create boundaries to operate. Define which source is authoritative, how permissions synchronize, how stale indexes are detected, and what happens when one component is unavailable. If no team owns those boundaries, a hybrid architecture can cost more than either clear option.

Treat a move as a controlled migration. Export documents, metadata and access rules, then build the replacement index beside the current one. Run the same evaluation questions through both systems, compare retrieved sources and permissions, and investigate each meaningful regression. Shadow traffic can expose differences before users depend on the new path. Keep a rollback route until freshness, latency, answer quality and audit records meet the agreed production thresholds.

If the evaluation shows a hard limit in permission modelling, retrieval control or auditability, involve an engineer to scope that gap. Custom retrieval implementation is appropriate when a team needs to own those controls; it is unnecessary when the managed product passes the same tests.

The takeaway

Rent first, measure honestly, build when a wall makes you. The three walls are permissions, retrieval logic and per-query economics, and only the first is a hard stop from day one. If none of them apply to you, a hosted product is genuinely the right answer and we will say so.

RAG as a service vs custom build questions

What is RAG as a service?

It is a managed product that indexes content and retrieves passages for an application, usually through a hosted API or platform. The vendor operates much of the retrieval stack.

When should a team build its own RAG system?

Build when measured requirements for permissions, retrieval behavior, data control or auditability exceed the hosted product’s capabilities and justify the ongoing ownership cost.

Is hosted RAG cheaper than a custom system?

It is often cheaper to start. Compare ingestion, usage, support and staff time with engineering, hosting, monitoring and maintenance at expected and peak volume.

How do permissions work in RAG?

The system should apply the user’s access rights before restricted passages enter the model context. Filtering only after generation can expose information.

Does a custom RAG system guarantee better answers?

No. It gives a team more control over retrieval and evaluation. Quality still depends on source data, access rules, ranking, tests and operational upkeep.

What should a RAG evaluation set contain?

Use real questions, expected source passages, access identities, ambiguous requests and unanswerable cases. Test retrieval and answer quality separately, then retain the set as a regression check for future changes.

Can a team move from hosted RAG to custom later?

Yes. Preserve source documents, metadata, permissions and evaluation cases so another index can be built and compared before switching traffic.

How often should a RAG index update?

Set freshness expectations from how quickly source changes affect users. Measure ingestion delay and alert when indexing falls behind that service level.

What is hybrid RAG?

It combines managed components with systems a team controls, such as custom ingestion or permission-aware orchestration. Define who owns each boundary.

How can a team tell if retrieval is the problem?

Inspect failed questions and check whether the correct passage was retrieved. If it was absent, changing the answer model will not fix the retrieval gap.

Tools referenced

pgvector
Neo4j
Ragas evaluation

EpochC builds retrieval systems for production teams, plus custom AI agents and AI chatbots. See the graph + vector retrieval case study — +40% query accuracy, sub-200ms — or start a project.

RAG as a service vs custom builds: which one you actually need