Quantization: How to Run Big LLMs on Small Hardware

The single biggest thing standing between you and running a serious open model is memory. Weights are stored as numbers, and by default each number takes 16 bits. Multiply that by 70 billion parameters and you need 140GB just to load the model — more than any single consumer GPU, and most single data-center ones, can hold.

Quantization is the trick that makes the model smaller by storing those numbers in fewer bits — 8 instead of 16, or even 4 — with surprisingly little loss in quality. It is the reason a model that used to need a cluster now runs on one card, or even a laptop. The cost is real but small, and knowing where it lands is the whole skill.

What the bits actually buy you

A model's memory footprint is roughly its parameter count times the bytes per parameter. Cut the bits per number and the footprint drops in direct proportion — and because LLM inference is mostly limited by how fast you can move weights from memory, a smaller model is usually a faster one too.

• FP16 / BF16 (16-bit): the default. A 70B model is about 140GB — full quality, but needs multiple GPUs.
• INT8 (8-bit): halves it to roughly 70GB. Quality is nearly indistinguishable from full precision for most workloads.
• INT4 (4-bit): about 35GB — small enough for a single high-end card. A modest, manageable quality dip with good methods.
• The rule of thumb: bytes per parameter is bits divided by 8, so 4-bit means half a byte each. Memory, bandwidth, and often latency all fall together.

Post-training versus quantization-aware

There are two ways to get there. Post-training quantization (PTQ) takes a finished model and compresses it directly — no retraining, just a calibration pass over a little sample data. This is what almost everyone uses: GPTQ, AWQ, and the GGUF k-quants in llama. cpp all live here. It is fast, cheap, and good enough down to 4-bit.

Quantization-aware training (QAT) instead simulates low precision during training so the model learns to be robust to it. It squeezes out more quality at very low bit-widths but costs a full training run, so it is reserved for cases where every last point matters or you are going below 4-bit.

Doing it in practice

For most people, quantization is a few lines — you load an existing checkpoint in low precision rather than converting anything yourself. With bitsandbytes and Transformers, a 4-bit load looks like this:

# Load a 70B model in 4-bit: ~140GB -> ~35GB, fits on a single 48GB GPU.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # NormalFloat-4, tuned for weight distributions
    bnb_4bit_compute_dtype=torch.bfloat16, # compute in bf16, store in 4-bit
    bnb_4bit_use_double_quant=True,       # quantize the scales too, saving a bit more
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=cfg,
    device_map="auto",
)

For serving, you usually grab a pre-quantized checkpoint instead — a GGUF file for llama. cpp, or an AWQ or GPTQ build that vLLM and TensorRT-LLM load natively — so the calibration work is already done and you just point your engine at it.

Quantization is lossy compression for neural networks: you are betting that most of a model's precision was never doing any work, and for the weights, that bet almost always pays.

What you give up, and how to tell

• A small accuracy hit that grows as bits shrink. INT8 is usually free; INT4 is cheap; below 4-bit, degradation gets real fast.
• Outliers are the hard part. A few weights and activations have huge values that low precision mangles — modern methods (AWQ, SmoothQuant) protect exactly those.
• Activation quantization is harder than weight quantization. Weight-only 4-bit is common and safe; quantizing activations too saves more but risks more.
• Hardest tasks suffer first. Long-chain reasoning and code feel 4-bit before chitchat does, so evaluate on your real workload, not a generic benchmark.
• Always measure twice: perplexity for a quick smell test, then your actual task evals — quality loss is task-specific and easy to miss in a demo.

The bottom line

Quantization is the cheapest way to make a big model usable: download a 4-bit build, point your serving engine at it, and a model that needed a cluster now runs on one GPU, faster than before. INT8 is essentially a free win; INT4 is the sweet spot for fitting large models on small hardware; go lower only with care and good evals.

The deeper lesson is that most of a model's stored precision is slack. Squeeze it out and you trade a sliver of accuracy for a model that is half or a quarter the size and runs anywhere — which is often the difference between a model you can deploy and one you only read about.