RLHF, DPO, and Preference Tuning: Teaching Models What “Good” Means

A freshly pretrained model is a master of plausible text and a stranger to your preferences. Ask it a question and it will happily continue with something statistically likely — which might be a real answer, a refusal, a list of more questions, or a ramble. It has no idea which of those you actually wanted. Turning that raw capability into a helpful assistant is a separate step, and it runs on human preference.

Preference tuning is how a model learns what good looks like. You show it pairs of responses and which one people preferred, and it adjusts to produce more of the preferred kind. Two approaches dominate: RLHF, the method that made ChatGPT-style assistants possible, and DPO, a simpler technique that reaches a similar place with far less machinery.

Why pretraining is not enough

Pretraining and instruction tuning get you a model that can follow instructions, but following is not the same as aligning with human taste — being helpful, honest, appropriately cautious, well-formatted. Those qualities are hard to write down as rules and easy to recognize by comparison, which is exactly what preference methods exploit.

• Good is easier to compare than to define. People struggle to write a spec for a great answer but can reliably say which of two answers is better.
• There is no single right output. Tone, length, caution, and helpfulness are preferences, not facts — so you train on what humans prefer, not on a gold label.
• Preference data is comparisons. The raw material is pairs: a chosen response and a rejected one for the same prompt.
• It is the final polish. Pretraining gives knowledge, instruction tuning gives obedience, preference tuning gives judgment about which response wins.

RLHF: a reward model and a policy

RLHF — reinforcement learning from human feedback — does it in two stages. First you train a separate reward model on the human comparisons, so it learns to score any response the way people would. Then you fine-tune the language model (the policy) with reinforcement learning to maximize that reward, while a constraint keeps it from drifting too far from the original model and going off the rails.

It works — it is what produced the first genuinely helpful chat assistants — but it is heavy: a second model to train, an RL loop that is famously finicky to stabilize, and several moving parts that can each go wrong. That complexity is exactly what the next method set out to remove.

DPO: the same goal, no reward model

Direct Preference Optimization gets to nearly the same outcome with a single, stable training step and no separate reward model and no RL loop. The insight is mathematical: you can rewrite the RLHF objective so the model is trained directly on the preference pairs, with one loss that simply pushes up the probability of the chosen response and pushes down the rejected one — anchored to the original model so it does not drift.

# DPO: directly push the chosen response up and the rejected one down. No reward model.
import torch.nn.functional as F

def dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta=0.1):
    # log-prob gaps vs the frozen reference model, for the chosen and rejected responses
    chosen_gap = policy_chosen - ref_chosen
    rejected_gap = policy_rejected - ref_rejected
    margin = beta * (chosen_gap - rejected_gap)   # beta keeps the policy near the reference
    return -F.logsigmoid(margin).mean()           # one stable loss, no RL loop

Because it is just a supervised-style loss over preference pairs, DPO is far easier to run and tune than an RL pipeline, which is why much of the open ecosystem now reaches for it (and its variants) first. RLHF still has an edge on the hardest alignment problems, but DPO covers most of what most teams need.

Pretraining teaches a model the world; preference tuning teaches it the room. It is the difference between a model that can answer and one that answers the way you hoped.

What this means in practice

• Your data is preference pairs. The hard, valuable work is collecting good chosen-versus-rejected comparisons; the algorithm is the easy part.
• Start with DPO. For most fine-tuning, it gets you most of the alignment benefit with a fraction of RLHF's complexity and instability.
• Garbage preferences, garbage model. The tuned model inherits whatever your labelers preferred, including their blind spots and biases.
• Mind the drift. Both methods anchor to the original model on purpose; push too hard and you get reward hacking or a model that forgets how to write.
• It is not just for labs. With open base models and a few thousand preference pairs, preference tuning is a realistic way to align a model to your domain and tone.

The bottom line

Preference tuning is the step that turns a capable base model into one that behaves the way people actually want. RLHF pioneered it with a reward model and reinforcement learning; DPO reaches a similar place with a single stable loss on preference pairs and far less to break. For most teams today, DPO is the pragmatic default and RLHF the heavier tool for the hardest cases.

Either way, the lesson is the same: alignment is taught by comparison, not specification. Collect honest judgments of which response is better, train the model to prefer them, and you are no longer hoping the model shares your taste — you are teaching it.