Knowledge graphs vs vector search for RAG
Short answer: Vector search finds passages that mean something similar to a question, and is reliably weak on exact identifiers and on how entities relate to each other. A knowledge graph answers relationship questions that embeddings structurally cannot. The strongest retrieval systems run both, with keyword matching alongside, rather than choosing between them.
When people talk about RAG, they usually mean vector search. But vector search has a blind spot, and for a whole class of questions a knowledge graph is the better tool. The strongest systems use both.
What each one is good at
Vector search embeds text into numbers and finds passages that are semantically similar to a question. It is excellent at “find me content about X,” even when the wording differs. It is the right default for most document Q&A.
A knowledge graph stores entities and the relationships between them: this company owns that subsidiary, this patient had that visit, this claim belongs to that policy. It is excellent at questions where the answer lives in the connections, not in a single passage.
Choosing between them is rarely the real question. Combining them is, and that combination is the core of most hybrid retrieval systems we build.
Where vector search falls short
Ask a vector store “which entities are in-network for this plan,” and it will happily return the most similar-sounding passages, which may be completely wrong. The information is relational, and a pile of similar text chunks cannot express it. This is where teams see confident, wrong answers.
Where the graph wins
On a FinTech information-retrieval platform, we combined Neo4j knowledge graphs for entity relationships with vector embeddings for unstructured content. The dual-module approach lifted query accuracy by 40% over a vector-only baseline, because each half answered the questions the other could not.
How to combine them
- Use vector search for open-ended, content-based questions.
- Use the graph for questions about relationships, hierarchy and structure.
- Add a routing or hybrid layer that sends each query to the right retriever, or fuses both.
You do not have to choose. The practical answer is a hybrid retrieval system where vector search and a knowledge graph cover each other’s weaknesses.
What the combination actually looks like in production
The useful pattern is not choosing one. It is using the graph to decide what the vector index should be asked.
A query arrives. Resolve the entities in it against the graph first, which gives you identifiers rather than strings. Use those identifiers to constrain the vector search to the region of the corpus that relates to them. Then rank what comes back using both the embedding score and the graph distance.
That ordering matters. Vector search over the whole corpus, filtered afterwards by graph membership, spends your top-k on documents that were never relevant. Constraining first means every candidate is already in the right neighbourhood.
On our own build, combining a Neo4j knowledge graph with vector search lifted query accuracy 40% and answer relevance 35% against a vector-only baseline, served end to end in under 200ms. The figures and the architecture are in the graph and vector retrieval case study.
What it costs to maintain
This is the part that decides whether the combination is worth it for you, and it is rarely discussed.
A vector index is cheap to maintain because it needs no schema. Documents change, you re-embed, done. A knowledge graph needs an ontology, and an ontology is a set of decisions about what your entities are and how they relate. Those decisions have to be made by someone who understands the domain, revisited as the business changes, and enforced at ingestion.
Entity resolution is the recurring cost. Deciding that two records describe the same customer, drug, part or company, across spelling variants and identifier systems that disagree, is continuous work rather than a setup task.
So the honest rule: add the graph when relationships carry meaning your users actually query on. Supply chains, corporate ownership, clinical pathways, legal citation networks, component hierarchies. If your users ask “what is similar to this”, a vector index alone is the right answer and a graph is expensive decoration.
How to tell which one your problem needs
Take twenty real questions your users ask and sort them into two piles.
Questions answerable by finding a passage that resembles the query go in the first pile. Questions that require traversing a relationship — who owns this, what depends on that, which of these came before — go in the second.
If the second pile is nearly empty, build vector retrieval and stop. If it holds a third or more, the graph will pay for itself, and the ranking above is where to start. That sort takes an afternoon and it is a better basis for the decision than any benchmark.
Tools referenced
- Neo4j
- pgvector
- MongoDB Atlas Vector Search
EpochC builds custom RAG development services. See the graph + vector retrieval case study — +40% query accuracy, sub-200ms — or the enterprise RAG architecture guide.