Choosing and Fine-Tuning the Embedding Model Under Your RAG

An embedding model turns a piece of text into a vector — a list of numbers — so that things with similar meaning end up close together in space. It is the quiet foundation under retrieval, search, clustering, and every RAG app: before any re-ranking or clever prompting, this one model decides which documents are even in the running.

Most teams pick the default embedding model on day one and never revisit it, then spend weeks tuning everything downstream. But if the embedder maps your question and the answer to distant points, nothing else can recover them. Choosing — and sometimes fine-tuning — this model is the highest-leverage decision in the whole pipeline.

What you are choosing between

Embedding models differ on a few axes that actually matter, and the popular benchmark leaderboards only hint at how a model will do on your data. The real choice is a balance of nuance, cost, and fit.

• Dimensions: more numbers per vector (768, 1024, 1536 and up) capture more nuance but cost more to store and search. Bigger is not automatically better.
• Context length: how much text fits in one embedding. Too short and your chunks get truncated; match it to how you actually split documents.
• Domain fit: a model trained on web text may not separate near-synonyms in law, medicine, or your own product jargon — the place generic models quietly fail.
• Open versus API: hosted embedders (OpenAI, Cohere, Voyage) are one call away; strong open ones (BGE, E5, GTE, Nomic) you can run, fine-tune, and keep private.

When the generic model is not enough

Off-the-shelf embedders are trained to capture broad, everyday similarity. That breaks down exactly where your app is valuable: in a specialized domain, two terms that look unrelated to a general model may be near-identical to an expert, and two that look similar may be opposites. The embedder confuses precisely the distinctions your users care about.

Fine-tuning fixes this by showing the model what should be close in your world. You feed it pairs of queries and the passages that truly answer them, and it learns to pull those together while pushing unrelated text apart — a technique called contrastive learning. A few thousand good pairs can dramatically sharpen retrieval on a narrow domain.

Fine-tuning, in code

The sentence-transformers library makes this almost anticlimactic. With a set of query-and-correct-passage pairs, in-batch negatives do the rest — every other passage in the batch acts as a wrong answer:

# Fine-tune an embedding model on your own (query, correct-passage) pairs.
from sentence_transformers import SentenceTransformer, losses, InputExample
from torch.utils.data import DataLoader

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
examples = [InputExample(texts=[query, passage]) for query, passage in your_pairs]
loader = DataLoader(examples, batch_size=32, shuffle=True)

# In-batch negatives: every other passage in the batch counts as a wrong answer.
loss = losses.MultipleNegativesRankingLoss(model)
model.fit(train_objectives=[(loader, loss)], epochs=1, warmup_steps=100)
model.save("bge-base-mydomain")

Gather the pairs from real usage — logged questions and the chunk that actually answered them — and even one epoch often moves retrieval quality more than any prompt change you could make.

Re-ranking decides the order of the candidates; the embedding model decides whether the right candidate is a candidate at all. Fix the foundation before you polish the floors above it.

Rules that save you pain

• Evaluate on your data, not the leaderboard. Build a small set of real queries with known answers and measure recall and ranking — benchmark scores rarely predict your domain.
• Changing models means re-embedding everything. Vectors from different models are not comparable, so a swap is a full re-index — plan for it.
• Match the query and document sides. Use the same model (and any required prompt prefixes) for both, or your distances are meaningless.
• Right-size the dimensions. Higher dimensions cost storage and search latency at scale; some models let you safely truncate vectors to trade a little accuracy for a lot of speed.
• Fine-tune when the domain is narrow and the stakes are high — and keep a held-out set so you can prove the tuned model actually retrieves better.

The bottom line

The embedding model is the part of your RAG stack most worth getting right and most often ignored. Start from a strong open or hosted embedder, evaluate it on real queries from your own corpus, and fine-tune on logged query-passage pairs when a specialized domain trips it up. The payoff compounds through every layer above it.

Prompts and re-rankers get the attention because they are easy to change. But retrieval quality is decided down at the vectors. Pick the model that understands your text, teach it your domain when it does not, and the rest of the pipeline suddenly has something good to work with.