HNSW in Production: The Vector Index Nobody Tells You How to Operate

Most HNSW explainers are content to draw the layered graph, wave at "express lanes," and declare the black box opened. That is the algorithm. It is not the thing that keeps you up at night. The thing that keeps you up at night is that HNSW is not really a data structure you query — it is a live, memory-resident graph you have to feed, tune, and occasionally rebuild, and almost nobody writes down the operating manual.

So this post assumes you already believe the graph is clever. Instead I want the operator's mental model: what the two real knobs do to your latency budget, how recall quietly degrades in ways your dashboards will not show, and a decision framework for when HNSW is the right index and when it is actively the wrong one. I have not run a benchmark for you here — treat every number below as illustrative reasoning, not a measurement — but the failure modes are structural, and you can reason your way to them from how the index is built.

The one-sentence mental model that actually helps

Forget "hierarchical navigable small world" for a second. The model I keep in my head: HNSW is a greedy graph walk that is only as good as the neighbors each node was given at insert time. Every query is a chain of "which of my neighbors is closest to the target? " decisions. If the right answer is not reachable through some chain of those local decisions, the walk never finds it — not because the vector is missing, but because no edge points that way. That single fact explains basically every production surprise you will hit.

That reframes everything. HNSW's accuracy is not a property of your query — it is a property of the graph you built. The layers just make the walk start from a good vantage point. So "raise ef to get better recall" really means "spend more of your query budget exploring the graph so a bad edge earlier does not doom you. " The wiring is fixed at insert; ef is you paying at query time to paper over its imperfections.

HNSW does not store your vectors so much as store a set of bets about which vectors are near which. A query is fast when those bets were good, and slow-or-wrong when they were not. Tuning is just deciding how much you pay to hedge them.

The two knobs, and a worked example of tuning ef

There are really only three parameters worth your attention, and two of them are locked in at build time. M is how many neighbors each node keeps — the graph's degree. ef_construction is how hard the builder searches for good neighbors while inserting. Both cost you memory and index time and cannot be changed later without rebuilding. The third, ef (sometimes called ef_search), is the only one you can move at runtime, and it is your entire recall-versus-latency dial.

Here is the whole interface in hnswlib. Notice how little there is — everything your vector database markets sits on top of these few lines:

import hnswlib,import numpy as np,,dim = 768          # e.g. a sentence-embedding dimension,n = 200_000        # vectors in this shard,data = np.random.rand(n, dim).astype(np.float32),,index = hnswlib.Index(space="cosine", dim=dim),,# BUILD-TIME knobs. You are committing to these.,index.init_index(,    max_elements=n,,    M=16,               # neighbors per node: more = better recall, more RAM,    ef_construction=200 # build effort: higher = better graph, slower build,),index.add_items(data),,# QUERY-TIME knob. Change this all day long.,index.set_ef(64)        # must be >= k; this is your recall/latency dial,,labels, distances = index.knn_query(data[:5], k=10)

Now the part tutorials skip: how do you actually pick ef? Not by guessing, and not by copying someone's blog number. You measure recall against ground truth on your own data. Brute-force the true top-k for a sample of, say, 1,000 real queries once (exact search is slow, but you only do it offline), then sweep ef and watch where recall plateaus.

Picture that sweep as a shape, not a promise. Say k is 10. At ef=10 the search barely explores past the candidates it strictly needs, and recall might sit somewhere unpleasant like 0.85 — one in roughly seven queries missing a true neighbor. Bump ef to 40 and recall climbs steeply, maybe to 0.97, for a modest latency cost. Push to 128 and you claw toward 0.99, but latency has roughly doubled from the ef=40 point. Go to 512 and recall barely moves while latency keeps climbing. The curve is concave: cheap gains early, expensive gains late.

The operating decision falls out of that shape. You do not pick "high ef" or "low ef" — you pick the knee of the curve for the recall your product actually needs. A RAG system that stuffs 20 chunks into a prompt and lets the LLM sort them out can live at 0.95 recall and a low ef, because a single missed neighbor rarely changes the answer. A compliance search that must not miss a matching document needs 0.99 or better, and you pay the latency. Same index, same data, completely different ef — and the only way to know the number is to have run the sweep.

The failure modes nobody puts in the quickstart

These are the ones that turn a working prototype into a 2 a. m. page. None show up when you index 10,000 rows on your laptop; all show up at scale.

• Deletes are a lie (mostly). Classic HNSW cannot cleanly remove a node — its edges are load-bearing for other nodes' walks. Most engines soft-delete by marking the vector as a tombstone and skipping it in results, but the node stays in the graph, still consuming RAM and still routing traffic through itself. Churn a collection hard enough and you are walking a graph that is half ghosts. The fix is a periodic full rebuild, which you must budget for as a scheduled operation, not an afterthought.
• Recall drifts when your embeddings drift. The graph was wired for the distribution of vectors you had at build time. Swap your embedding model, or let a year of new-topic documents pile in, and the neighbor bets made for the old distribution get worse. Nothing errors. Recall just quietly sags, and unless you re-run your ground-truth sweep periodically you never see it — the latency dashboards look perfect.
• RAM is the real capacity limit, not disk. The graph edges live in memory on top of the vectors. Back-of-envelope: each node stores on the order of M links per layer (and the base layer commonly gets 2M), so at M=16 you carry tens of extra bytes per vector just for connectivity, before the vectors themselves. At tens of millions of vectors that graph overhead alone can be gigabytes. Size RAM around vectors plus graph, and know that spilling to SSD can quietly wreck your p99 latency.
• Build time is superlinear in the knobs. Doubling ef_construction or M does not double build time — it can be worse, because every insert runs a mini-search through an increasingly connected graph. A generous M=48, ef_construction=500 build over tens of millions of vectors can run for hours. Discover that before your reindex window, not during it.
• The filtered-search trap. "Find nearest neighbors where tenant_id = 42" sounds trivial and is the single most common way HNSW disappoints in production. The graph does not know about your filter, so the engine either walks the full graph and throws away most hits (slow) or constrains the walk and gets badly degraded recall when the filter is selective. If your queries are heavily filtered, benchmark that exact pattern — never extrapolate from unfiltered numbers.

A decision framework: when HNSW, when not

HNSW became the default for good reasons, but "default" is not "always. " Reach for it when your dataset is read-heavy, fits comfortably in RAM, tolerates approximate results, and does not churn violently. That is most semantic search and most RAG — which is exactly why it is everywhere. In that regime it gives the best recall-per-millisecond, and the tuning story is a single runtime dial. The questions I run through before committing:

1. Do you actually need approximate, or exact? If a wrong "nearest" is a correctness bug — dedup, some fraud and security matching, small catalogs — just brute-force it. Under roughly a few hundred thousand vectors, exact search is often fast enough and infinitely simpler, with zero recall to babysit. Do not build an ANN index to solve a problem you do not have.
2. Does it fit in RAM, and will it grow past it? If the index will outgrow memory, HNSW's graph overhead makes it the worst offender. Look at disk-based ANN (DiskANN-style) or IVF variants friendlier to out-of-core layouts before you brute-force your way into a bigger instance every quarter.
3. How write-heavy and delete-heavy is the workload? Streaming inserts with constant deletes are HNSW's weak spot. If you are running something closer to a live feed than a search corpus, factor in rebuild cadence or consider an index designed for churn.
4. Are your queries heavily filtered by metadata? If most queries carry a selective filter, benchmark that path specifically and look hard at whether the engine's filtered-HNSW implementation holds recall — or whether a partitioned/IVF approach that filters before searching fits better.
5. Do you need the tightest possible memory footprint? If RAM is the binding constraint and some recall loss is acceptable, quantization (PQ) or IVF+PQ trades accuracy for dramatic memory savings in ways HNSW alone does not.

Notice the pattern: every "not HNSW" answer is driven by memory pressure, exactness requirements, churn, or filtering. Those are the four axes. If none bind, HNSW wins and you should stop overthinking it.

The bottom line for people who run this thing

The layered-graph picture is true but incomplete. The operator's version is this: HNSW is a RAM-resident graph of fixed neighbor bets, made at insert time, that you walk greedily at query time. Build-time knobs (M, ef_construction) set the quality and cost of those bets, and you are married to them until the next rebuild. The one runtime knob (ef) is you paying to hedge imperfect bets — tune it to the knee of your own measured recall curve, not to a number from someone else's post.

And respect the four things the quickstart hides: deletes accumulate as ghosts, embedding drift silently erodes recall, RAM is your true ceiling, and filtered queries are their own separate benchmark. Do that and HNSW stops being a black box you hope works and becomes a component you actually operate — which is the only version of "understanding an algorithm" that pays rent in production.