Knowledge Distillation: Training a Small Model to Mimic a Big One

Quantization makes a model smaller by storing it in fewer bits. Distillation makes a different bet: train a brand-new, much smaller model — the student — to reproduce the behavior of a big one — the teacher. You end up with a separate model that is a fraction of the size but has learned to act like its mentor on the work you care about.

This is how a lot of capability quietly moves down-market. The frontier model does the expensive thinking once, at training time, and a small student carries the lessons into production — running cheaper, faster, and often on hardware the teacher could never fit on. The surprising part is how much of the teacher's skill survives the shrink.

Soft labels carry hidden knowledge

The original insight is subtle. If you train a small model only on right answers, it learns the answer but not the reasoning around it. The teacher knows more than the label — it knows that a photo of a husky is mostly dog, a little wolf, and definitely not cat. Those relative probabilities are the real prize.

• Hard labels say only this token is correct. Soft labels — the teacher's full probability distribution — say how likely every option was, and that shape is information.
• That extra signal is often called dark knowledge: the teacher's uncertainty and its near-misses, which teach the student far faster than bare answers.
• A temperature setting softens the teacher's distribution so the small probabilities are visible, not crushed to zero, during training.
• The student learns to match that softened distribution, absorbing not just what the teacher answers but how confident it is and why.

How it is actually done for LLMs

For language models, the most common approach today barely looks like the textbook version. You run the teacher over a large set of prompts, collect its outputs, and fine-tune the student on that synthetic dataset — sequence-level distillation. Much of the recent open-model boom is, in effect, students trained on a stronger model's generations.

The more classical form matches the teacher's token-level distributions directly, minimizing the difference between student and teacher probabilities with a divergence loss. It needs access to the teacher's logits, so it is mostly used when you own both models — but it transfers the most signal per example.

The loss, in code

At its core, logit distillation just blends two losses: copy the teacher's soft distribution, and still get the real answer right.

# Distillation loss: match the teacher's soft distribution, plus the true label.
import torch.nn.functional as F

def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    # Soft targets: the teacher's temperature-softened distribution (dark knowledge).
    soft = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.softmax(teacher_logits / T, dim=-1),
        reduction="batchmean",
    ) * (T * T)                                  # rescale so gradients keep their size
    # Hard targets: the ordinary next-token loss on the real labels.
    hard = F.cross_entropy(student_logits, labels)
    return alpha * soft + (1 - alpha) * hard     # blend the two

Everything else — which prompts to distill on, how big the student is, how much teacher data you need — is tuning around that idea: a student that is rewarded for thinking like the teacher, not just for being right.

Distillation is teaching, not compression. The teacher does not hand the student a smaller copy of itself — it hands over its judgment, and the student rebuilds as much of it as it can hold.

Where it helps and where it bites

• Specialize aggressively. A small student distilled for one domain often matches a giant general model there, at a tiny fraction of the cost.
• It is the on-device path. Distillation plus quantization is how frontier-flavored behavior ends up running locally and offline.
• The student inherits the teacher's flaws. Biases, blind spots, and confident mistakes get copied right along with the skill.
• It rarely beats the teacher. A student is bounded by what its teacher knew; distillation transfers capability, it does not create new ceilings.
• Mind the terms and the licenses. Some providers forbid training competing models on their outputs, so check what you are allowed to distill before you ship it.

The bottom line

Distillation is how you turn a model that is too big to deploy into one you can actually run. Generate good teacher data, train a small student to imitate it — ideally on the soft distributions, not just the answers — and you get most of the capability at a fraction of the size and cost, especially when you narrow the target.

Paired with quantization, it is the one-two punch behind nearly every small model that feels smarter than it should. The teacher thinks hard once; the student does the work forever after.