Choosing an Embedding Model for RAG

Short answer: pick two or three candidates from the leaderboard, then run them against a hundred real questions from your own corpus and choose on that. Benchmark rank is a useful prior and a poor decision rule — a model that tops MTEB on Wikipedia and legal text can underperform on your internal jargon, and you will not find that out from a scoreboard.

The embedding model is one of the few RAG decisions that is genuinely hard to change later, because switching means re-embedding everything. It deserves an afternoon of measurement.

The current landscape, briefly

As of mid-2026 the leaderboard is crowded at the top and the differences are small. Google’s gemini-embedding-001 has held the top MTEB Multilingual spot at around 68.3 overall and 67.7 on retrieval, with a newer preview adding native multimodal support and 3072-dimensional output. Qwen3-Embedding’s 4B and 8B models sit close behind, and its 0.6B variant is remarkable for the size. OpenAI’s text-embedding-3-large remains a solid, well-supported choice at roughly 64.6% average MTEB, now below the leaders on public benchmarks.

The more interesting shift is that open-source models now match or exceed closed-source on retrieval across most domains. That turns the open-vs-closed question from a quality decision into an operational and compliance one, which is a much easier decision to make well.

Four things that should actually decide it

Your corpus, measured. The only reliable signal. Take a hundred real user questions, embed your corpus with each candidate, and check whether the correct passage appears in the top-k retrieved set. That is retrieval recall, it takes an afternoon, and it routinely reorders the leaderboard ranking on domain-specific content.

Context length. OpenAI’s models cap at 8K tokens; several competitors offer 32K. This matters more than it sounds if your documents resist clean chunking — long contracts, technical manuals, transcripts. A longer window is not automatically better, though: embedding a huge chunk dilutes the vector, and precise retrieval usually wants smaller units.

Dimensions and what they cost you. Higher dimensions cost storage and query time. Matryoshka-trained models let you truncate the vector and trade a little accuracy for a lot of index size, which is now practical rather than theoretical. Measure the trade rather than assuming the largest output is correct.

Where it can run. Self-hosting removes per-token cost and keeps data inside your boundary, which is often the deciding constraint in regulated work. It also means you own GPU capacity, model updates and an inference service. Hosted APIs are simpler and meter forever.

These are the four we work through with clients at the start of a custom RAG build, and they settle the choice faster than any leaderboard.

The licensing trap

Worth checking before you build on something: not all open-weight models are commercially usable on the terms you assume.

BAAI’s BGE-M3 is MIT licensed and usable in commercial production without restriction. Jina’s embedding models are released under CC BY-NC-4.0 — non-commercial — so self-hosting them inside a commercial product requires a paid API plan or a separate agreement with Jina.

“Open weights” and “free to use commercially” are different claims. Read the licence before the benchmark.

Embeddings are not the whole retrieval story

The most common mistake we see is treating model choice as the retrieval decision when three other things move the number more.

Chunking sets the ceiling. Fixed-size splitting cuts tables in half and separates a clause from the sentence that qualifies it. Structure-aware chunking that respects your document format recovers more accuracy than any model upgrade will. If retrieval is disappointing, look here first.

Embeddings miss exact tokens. They are excellent at meaning and unreliable at literals. A user searching ERR_4021 wants that exact string, and a semantic model will happily return passages about similar-sounding errors. Running BM25 alongside vector search and fusing the results fixes a whole class of failure that otherwise looks mysterious. No embedding model solves this, because it is not what embeddings do.

Reranking often beats a better embedder. Retrieve twenty candidates cheaply, then rerank them with a cross-encoder that reads query and passage together. This is frequently a larger accuracy gain than moving up the leaderboard, and it is cheaper than re-embedding a corpus.

We covered the surrounding architecture in enterprise RAG architecture and the failure diagnosis in how to stop RAG hallucinations.

A process that works

Build the evaluation set first. A hundred real questions with the passage that should answer each. This is the single highest-value artefact in any RAG project and everything else is guesswork without it.
Shortlist two or three models — one hosted, one open, spanning your context-length requirement.
Measure retrieval recall at k, not answer quality. Answer quality confounds retrieval and generation, and you cannot fix what you cannot isolate.
Check the cheaper option seriously. If a smaller model is within a couple of points, take it — you will spend the difference on chunking and reranking with better returns.
Re-measure after chunking changes. The best model for one chunking strategy is not automatically best for another.

The takeaway

Treat the leaderboard as a shortlist generator and your own corpus as the judge. Verify the licence before you commit. Then spend your remaining effort on chunking, hybrid retrieval and reranking, because that is where the accuracy actually is — most RAG systems that disappoint are failing at retrieval design, not at the embedding model.

Sources

EpochC builds RAG development and retrieval systems with hybrid semantic and structured retrieval over pgvector, Neo4j and BM25. See the graph + vector retrieval case study — +40% query accuracy — or start a project.

Choosing an embedding model for RAG: what MTEB does not tell you