Long Context vs RAG Is the Wrong Fight: What I Learned Building a Compression Layer

Every few months a model ships with a bigger context window, and the same argument restarts: if a million tokens fit in the prompt, why keep a retrieval layer at all? Just paste the whole knowledge base in and let the model sort it out. It is a clean story. It is also the wrong frame, and I know because I spent months building the thing that lives in the gap between the two options.

The project is Headroom, a local-first context-compression layer that sits between an agent and its LLM. It intercepts everything the agent is about to read, tool output, logs, RAG chunks, files, history, and shrinks it before it reaches the model. Building it forced me off the two-camps framing and onto a third axis nobody names well: not how much you retrieve, but how much of what you already have you should keep paying full price for.

The framing everyone uses, and where it breaks

The textbook trade-off is real, so let me state it fairly. Long context buys you simplicity and whole-document reasoning: no chunking artifacts, no vector store, and the model can follow a cross-reference from page 2 to page 90 that a chunked system would have severed. RAG buys you scale, cost control, and freshness: you fetch the handful of passages a question needs, so your bill stays flat whether the corpus is a thousand documents or ten million. That part of the debate is settled and correct.

What the debate misses is that in an agent loop, the expensive context is rarely your curated knowledge base at all. It is the junk the agent generates on its own: a tool that returns 100 search hits, a 200-line build log, a JSON blob where 90 fields are identical across every row. None of that came from a retriever you designed. It arrived mid-conversation, and it re-lands in the prompt at full token price on every subsequent turn. Retrieve-or-stuff has no opinion about it, because it is neither the corpus nor the query.

The most expensive tokens in an agent aren't the ones you retrieve, they're the ones the agent hands itself.

The third option: compress, but keep it reversible

Headroom's answer is a pattern I ended up calling CCR: Compress, Cache, Retrieve. Instead of choosing between stuffing everything or retrieving fragments, it compresses the bulky content up front but keeps the full original in a local store the model can pull back on demand. A JSON compressor called SmartCrusher does statistical analysis on arrays: it factors out constant fields, keeps anomalies and spikes and error entries, and collapses the rest. But before it throws anything away, it stashes the untouched original under a 16-character SHA256 key with a default 5-minute TTL and LRU eviction.

The reason that matters is the failure mode. Any compression is a bet that the discarded bytes did not matter for this question. Sometimes that bet is wrong. With CCR, a wrong guess is not fatal: the model can call a headroom_retrieve tool and get the original back, and there is even a BM25 index over the cached content so it can search inside what was compressed. The design principle I kept coming back to was blunt: worst case, retrieve everything. That single guarantee is what makes aggressive compression safe enough to leave on by default.

One design decision I want to be honest about, because it is a constraint and not a brag: compression here does no LLM calls. It is all statistical analysis, pattern matching, and rule-based transforms. That was deliberate. An LLM summarizer is non-deterministic, slow, and can hallucinate a fact that was never in the data. A rule-based compressor is predictable and adds no API cost, but it is dumber, it cannot understand your JSON, it can only notice that a field never changes. CCR is the safety net that lets a dumb-but-safe compressor be useful: it does not have to be right, it only has to be reversible.

Retrieval didn't disappear, it moved inside

Here is the part that reframed the whole long-context-vs-RAG question for me. I did not delete retrieval to build a compressor. I ended up embedding a small retriever inside the compressor. When the model asks for something back, or when the system scores which cached passages are relevant, it runs a hybrid relevance function: BM25 keyword scoring blended with dense-embedding similarity, combined by an alpha weight.

The interesting bit is that alpha is adaptive. If the query contains a UUID, a 4-plus-digit numeric ID, a hostname, or an email, the code pushes alpha toward BM25 (around 0.8, clamped to a [0.3, 0.9] range) because for those tokens exact match is what you want; an embedding will happily return a semantically similar but wrong ID. For prose queries it leans the other way, toward the embeddings. The idea comes from Dynamic Alpha Tuning (Hsu et al. , 2025), which reports roughly 2 to 7.5% retrieval gains from query-aware weighting. And if sentence-transformers is not installed, the whole thing falls back to pure BM25 with zero extra dependencies; a lot of exact-match agent queries never needed the embeddings anyway.

So the honest version of the architecture is not long context beat RAG or the reverse. It is: retrieval got smaller and moved closer to the data. Instead of one big retrieval step at the front of the pipeline, there is a tiny, cache-scoped retriever that decides what to compress and what to hand back.

The internet is full of compression claims with no test behind them, so let me be careful. The README tagline says 60 to 95% fewer tokens, but that is a range across wildly different content, not a single result. The benchmark I trust most for this discussion is the SmartCrusher JSON eval, because it tests the thing that actually scares me about compression: losing the one line that mattered. The setup is 100 production log entries with a critical error deliberately buried at position 67. Compressed, that went from 10,144 tokens down to 1,260, an 87.6% reduction, and all 4 of the eval's questions were still answered correctly, including the one about the buried error. The compressor is allowed to be aggressive precisely because losing position 67 would be caught, and because CCR could retrieve it back if it weren't. It is one benchmark on one dataset, self-reported by the project, not an independent study, but a real reproducible test rather than a vibe.

Where it gets genuinely counterintuitive is what the compressor does to already-tidy input. Feed it grep results or source code and it reports 0.0% compression. The first time a teammate sees that, they assume a bug. It is the opposite: structured, compact content has nothing to factor out, so SmartCrusher passes it through untouched instead of mangling it to hit a number. In this system, 0% is a feature.

Why this touches your caching bill directly

There is a second reason compression and retrieval collide that the long-context debate almost never mentions: provider prompt caching. The three big providers price a cache read very differently. Anthropic gives a 90% read discount (with a 25% write premium and a 5-minute TTL), OpenAI's automatic prefix caching gives 50% off but needs a byte-identical prefix of at least 1024 tokens, and Google's cached-content API gives 75% off but demands a 32,768-token minimum. Those thresholds change the math of just stuff the context in completely. A reused context that hits the cache is cheap; the same context that misses the cache is full price, every call.

Building Headroom's cache layer taught me the hard version of this the expensive way. My first attempt tried to stabilize the cache by pulling volatile content, dates, UUIDs, session IDs, out of the system prompt and re-appending it at the tail, so the cacheable prefix stayed constant. It worked, and then I deleted it, because it violated the one invariant that matters: never mutate a byte inside the cache hot zone. Changing a single cached byte changes the provider's cache key, drops your hit rate to zero, and silently torches the bill you were trying to lower. The rewrite path is gone. What ships now only detects the problem and warns you:

if all_findings:
    counts_str = ", ".join(f"{k}={v}" for k, v in sorted(counts.items()))
    msg_text = (
        f"CacheAligner: detected volatile content in system prompt "
        f"({counts_str}); cache prefix unstable. "
        "Move dynamic values out of the system prompt to recover cache hits."
    )
    warnings.append(msg_text)
    logger.warning(msg_text)

And the detection itself is fussier than you would guess, for a reason that only shows up in production. It uses real parsers, not regex; it defers to the standard-library UUID parser and datetime parsing rather than pattern strings. The subtle case: a 32-character dashless UUID is byte-indistinguishable from an MD5 hex digest, so the code refuses that form and only accepts the canonical 36-character dashed UUID. Otherwise it would flag every content hash in your prompt as a volatile UUID and warn you about cache-busting content that was actually stable.

def _is_uuid(token: str) -> bool:
    # Accepts only the canonical 36-char form with dashes. The 32-char
    # dashless form is indistinguishable from an MD5 hex digest and would
    # misclassify hashes; we treat that case as a hex hash instead.
    if len(token) != _UUID_CANONICAL_LEN:
        return False
    if token.count("-") != 4:
        return False
    try:
        _uuid.UUID(token)
    except (ValueError, AttributeError):
        return False
    return True

So how should you actually decide?

The clean textbook rules still hold, and I would not throw them out. Use them as the first filter:

• Corpus versus window: if your knowledge base comfortably fits the window and rarely changes, long context alone may genuinely be enough. The simplicity is worth a lot.
• Reuse pattern: a context queried once leans long-context; a context hit thousands of times leans retrieval or prompt caching, so you stop paying to re-read it.
• Reasoning span: questions that connect facts across a whole document favor a long window; isolated fact lookups favor retrieval.
• Budget and volume: long context trades engineering simplicity for per-call cost, which is fine at low volume and painful at scale.

But add the axis the debate leaves out. Once you are in an agent loop, the deciding question is not just how much to retrieve before you read; it is how much of what already landed in your context you should keep paying full price for. That is where compression earns its place next to both approaches:

1. For your curated corpus, decide long-context vs RAG the classic way.
2. For the agent's self-generated bulk (tool output, logs, verbose JSON), compress it before it re-enters the prompt on the next turn.
3. Make that compression reversible (a cache the model can pull from) so aggressive shrinking is never a permanent loss.
4. Keep your cacheable prefix byte-identical, because the cheapest tokens of all are the ones the provider already cached.

A million-token window did not kill RAG, and it did not make compression optional either. It raised the ceiling on all three. The real skill now is knowing which of your tokens are corpus, which are query, and which are just expensive noise your agent generated and will re-read on every turn, and treating each differently. The window earns its keep only when you stop paying full price for tokens that were never worth full price to begin with.