Quantization Isn't Compression — It's a Bet on Which Bits Were Bluffing

If you have read one explainer on quantization, you have read them all: weights are 16-bit numbers, storing them in 4 bits makes the model four times smaller, quality drops 'a little,' the end. It is true and it is useless, because it tells you the mechanism and hides the decision. The decision is not 'how few bits. ' It is which of a model's numbers you are willing to be wrong about, and by how much.

That reframing is the whole post. Once you see quantization as choosing where error is allowed to land rather than as generic shrinking, every practical question — which method, which bit-width, what breaks in production — stops being folklore and starts being a decision you can reason about.

The mental model: precision is a budget, and it's mostly slack

Here is the model I actually keep in my head. A trained network stores far more numeric precision than it uses. Most weights are small, cluster near zero, and would give the same answer if you rounded them hard. A tiny minority are large, load-bearing, and would wreck the output if you rounded them the same way. Full precision spends 16 bits on every weight equally — timid ones and load-bearing ones alike. That is the slack.

Quantization is the act of spending your bit budget unevenly on purpose. Good methods are not the ones that use the fewest bits; they are the ones that spend bits where error is expensive and starve the places where it is cheap. 'Round everything to 4 bits' is the naive version and it is why naive 4-bit used to be bad. The reason modern 4-bit is fine is entirely about protecting the load-bearing minority — the outliers — not about the average bit count.

Quantization is not lossy compression applied uniformly. It is a bet that most of a model's precision was bluffing — and the whole craft is knowing which weights were holding the real hand.

This is why 'INT8 is basically free, INT4 is cheap, below 4-bit gets scary' holds so reliably. It is not a magic property of the numbers 8 and 4. It is that at 8 bits you have enough headroom to represent the outliers passably even if you are careless, at 4 bits you can still do it if you are careful, and below 4 bits you run out of room to keep the load-bearing weights honest no matter how clever you are.

A worked example: fitting a 70B on one 48GB card

Let me walk the decision the way it actually happens, not the way the marketing slide draws it. Say you have a single 48GB GPU and you want to serve a 70B-class model. The reflex is to compute the weights: 70 billion params at 4 bits is about 35GB, that leaves 13GB of headroom, done. This is the trap. The weights number is the easy half, and the half people quote is the half that does not OOM you.

# Back-of-envelope VRAM planner. Everyone quotes the weights number and
# forgets the KV cache — which is what actually OOMs you at long context.

params_billion   = 70
bits_per_weight  = 4
weight_gb = params_billion * 1e9 * (bits_per_weight / 8) / 1e9   # ~35 GB

# KV cache = 2 (K and V) * layers * kv_heads * head_dim * seq * batch * dtype_bytes
# For a 70B-class model with GQA (8 kv heads), fp16 cache, one request:
layers, kv_heads, head_dim, dtype_bytes = 80, 8, 128, 2
seq, batch = 8192, 1
kv_gb = 2 * layers * kv_heads * head_dim * seq * batch * dtype_bytes / 1e9  # ~2.7 GB

# Now scale batch to 16 concurrent 8k-context requests:
kv_gb_16 = kv_gb * 16   # ~43 GB  <-- bigger than the quantized weights!
print(round(weight_gb), round(kv_gb, 1), round(kv_gb_16))

Run that reasoning and the picture flips. The quantized weights are 35GB and rock-steady. The KV cache — the per-token memory every request drags along — is about 2.7GB for a single 8k-context request, which looks harmless. But it scales linearly with concurrency, and at 16 concurrent 8k requests it is roughly 43GB. That is larger than the model itself, and it lives in the exact 13GB of headroom you thought you had. Your card does not fall over on load; it falls over on the fourth simultaneous long prompt in production.

So the real decision on a 48GB card is not 'INT4 or INT8' for the weights. INT8 weights (about 70GB) never fit here, so 4-bit weights are forced. The interesting knob is everything the naive framing ignored: how much context you promise, how many concurrent requests you allow, and whether you also quantize the KV cache (fp8 KV cache roughly halves that 43GB). The weight bit-width was the least of it.

That is the general lesson from the worked case: weight quantization decides whether the model loads; KV-cache size and concurrency decide whether it stays up. Plan for the second number and the first takes care of itself. And in practice you rarely quantize a model yourself — you either load an existing checkpoint in low precision for triage, or point a serving engine at a prebuilt quantized build. The bitsandbytes path below is the triage tool: the fastest way to answer 'does 4-bit ruin my task? ' before you invest in a serving pipeline.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# 4-bit weight-only load. Note what is and isn't quantized:
#  - weights: packed to 4-bit NF4
#  - compute: still bf16 (activations stay high precision)
cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',        # NF4 beats plain int4 for weights
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,   # quantize the scales too; ~0.4 bit/param saved
)

model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Meta-Llama-3-70B-Instruct',
    quantization_config=cfg,
    device_map='auto',
)
tok = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-70B-Instruct')

# This is a convenience load, NOT a serving path. bitsandbytes is great
# for 'does 4-bit break my task?' triage on one box. For throughput you
# want a prebuilt AWQ/GPTQ checkpoint loaded by vLLM or TensorRT-LLM.

Note the comment that most tutorials skip: this quantizes the weights but leaves the compute in bf16, so activations stay high precision. That is weight-only quantization, and it is the safe default. The moment you also quantize activations you are in a different, riskier regime — more on that below. And do not ship this exact load as your server: bitsandbytes is a triage and single-box convenience path, not a throughput path. For serving, grab an AWQ or GPTQ build and load it with vLLM or TensorRT-LLM, or a GGUF k-quant for llama. cpp, so the calibration is already baked and the kernels are tuned.

When to use it, when not to, and what breaks

The trade-offs are not one-size. Here is the decision table I would actually hand a teammate, organized by the choice you are really making rather than by bit-width alone.

• Reach for INT8 weight-only when the model already fits and you just want it faster/cheaper — it is close to a free win and rarely needs task-specific eval. If your model fits in full precision comfortably, quantizing purely 'to be safe' is premature; you are adding a variable for no memory you needed.
• Reach for INT4 weight-only (AWQ/GPTQ/NF4) when memory is the actual wall — it is the sweet spot for putting a big model on one card. Use an outlier-aware method (AWQ, GPTQ), not plain round-to-4-bit, and always eval on your real task.
• Reach for activation quantization (INT8 A8W8, or FP8) only when you are throughput-bound at scale and have the eval budget to catch regressions — it saves more but risks more, because activations carry the nasty outliers that weights mostly don't.
• Avoid going below 4-bit unless you have QAT (quantization-aware training) in the loop and a specific reason — sub-4-bit PTQ degrades fast and unpredictably, and the failure shows up on hard tasks first.
• Avoid quantizing when your bottleneck is latency at batch size 1 on a model that already fits and is compute-bound rather than memory-bound — the win from quantization is largely a memory-bandwidth win, so if you are not bandwidth-limited you may see little speedup.

The gotchas nobody demos

These are the ones that pass a quick check and fail a real deployment. None of them are exotic; they are just invisible until you look at the right workload.

1. Hard tasks degrade first, and your demo is an easy task. Long-chain reasoning, multi-step math, and code feel 4-bit quantization long before casual chat does. If you validate on 'write me a haiku' you will ship a model that quietly gets worse at exactly the work you deployed it for. Eval on your hardest real prompts, not a vibe check.
2. Perplexity is a smell test, not a verdict. A near-identical perplexity number reassures you and means little — perplexity can barely move while task accuracy on structured outputs, JSON validity, or tool-call formatting drops noticeably. Measure the thing you actually depend on.
3. The KV cache is the real memory sink at long context, not the weights. As the worked example showed, concurrency times context can dwarf the quantized weights. Teams size for the weights, deploy, and OOM under load. Quantizing the KV cache (fp8) is often a bigger practical lever than shaving another bit off the weights.
4. Weight-only and activation quantization are different risk classes — don't conflate them. 'I'm running 4-bit' usually means weight-only and is safe. A8W8 or aggressive activation quantization can crater on the outlier-heavy layers unless the method (SmoothQuant, AWQ-style scaling) specifically protects them. Know which one you actually enabled.
5. The quantized build and its kernels are a matched pair. An AWQ checkpoint expects AWQ-aware kernels; a GGUF k-quant expects llama. cpp. Mixing a format with an engine that only nominally supports it can silently fall back to a slow path or a subtly wrong dequant. If your 'faster' model got slower, suspect the kernel path before the math.
6. Determinism and quantization don't always coexist. Some fused quantized kernels are not bit-for-bit reproducible across batch sizes or hardware. If you rely on identical outputs for caching or testing, verify it holds after quantizing rather than assuming.

The bottom line

Quantization is the cheapest way to make a big model usable, and the standard advice — download a 4-bit build, point a serving engine at it — is correct. But treat it as a decision, not a dial. INT8 is a near-free win when you already fit; INT4 weight-only is the sweet spot when memory is the wall; activation and sub-4-bit quantization are power tools that demand real evals; and the KV cache, not the weights, is what usually decides whether you survive production.

The one idea worth keeping: most of a model's stored precision is slack, but not all of it, and the entire skill is telling the two apart. Quantization done well is not shrinking a model uniformly — it is spending your bit budget where the model is actually bluffing and refusing to spend it where the model is holding the real hand. Get that right and a model that needed a cluster runs on one card, often faster than before. Get it wrong and you ship a model that aces your demo and fails your users on exactly the tasks you cared about.