RLHF vs DPO in Practice: A Field Guide to Preference Tuning (and When It Quietly Goes Wrong)

There is a version of this topic that every blog writes: a base model predicts plausible text, preference tuning shows it pairs of answers labeled 'chosen' and 'rejected', RLHF uses a reward model and reinforcement learning, DPO skips the reward model with a clever loss, the end. All true, and all useless the moment you actually try it. This is the other post — the one about what happens after you understand the diagram. Because preference tuning is the one training step where your model can measurably get worse while every dashboard says it is improving, and you will not know why unless someone tells you where the traps are.

The mental model that actually helps: you are hiring a critic

Forget 'reward model' as a term for a second. In RLHF, what you are really doing is hiring a critic. You show a junior editor thousands of examples of 'this answer beat that answer', they internalize your taste, and then they sit next to your writer grading every draft. The writer (the policy) learns to chase high grades. That is RLHF: a critic you trained, and a writer optimizing to please it.

This framing immediately predicts the two biggest RLHF failures. First: the critic is not you. It is a lossy model of your taste, so the writer will eventually find answers the critic loves but you hate — that is reward hacking, and it is not a bug, it is the writer doing its job too well. Second: a critic you trained is a critic you cannot easily fire. It is a whole second model with its own bugs, its own biases, and its own maintenance cost.

DPO's pitch, in this framing, is: skip the critic. Do not hire a middleman to score drafts — bake the preference comparison straight into how the writer learns. There is no separate model to train, so there is no separate model to hack. That is the real reason the open-source world reached for it, more than any elegance argument.

A reward model is a critic you hired and can't fire. DPO is what you do when you realize you never needed the critic — you needed the comparisons the critic was trained on.

A worked example: fixing one bad support answer

Abstract 'chosen vs rejected' means nothing until you see one. Say you run support for a payments API and a user asks: 'Why did my charge fail with error code 402? ' Your base model, being helpful in the statistical sense, produces two candidate replies. You collect a human preference on this exact prompt.

Rejected: 'A 402 error means Payment Required. This is a standard HTTP status code indicating the request cannot be processed until payment is made. You should check your account and try again. Let me know if you have other questions! ' — polite, longer, and almost entirely useless. It restates the error name and gives no path forward.

Chosen: 'A 402 from our API specifically means the card was declined by the issuing bank, not a problem on your end. Check the decline_code field in the response — the three most common are insufficient_funds, card_velocity_exceeded, and do_not_honor. The first two the customer can fix; do_not_honor needs them to call their bank. Want the full decline_code table? ' — shorter on fluff, specific, actionable, and it knows your API.

That single pair encodes a surprising amount of taste: prefer domain-specific truth over generic HTTP trivia, prefer next-step actions over reassurance, do not blame the user by default. You are not writing those rules down anywhere — and you couldn't, cleanly. You are teaching them by contrast. Multiply this by a few thousand pairs across your real support tickets and you have a preference dataset that no off-the-shelf model has, which is exactly the kind of proprietary signal that makes tuning worth doing.

So what does DPO actually do with that pair? Here is the loss in plain, un-optimized Python so you can see the moving parts. The trick is that it needs two copies of the model: the one you are training (policy) and a frozen snapshot from before tuning (reference). Every update is judged relative to that frozen snapshot.

import torch
import torch.nn.functional as F

def dpo_loss(policy_logps, ref_logps, beta=0.1):
    # Each input is a dict with 'chosen' and 'rejected'
    # log-probabilities of the full response under a model.

    # How much MORE the policy likes chosen vs rejected,
    # compared to how much the frozen reference did.
    chosen_delta   = policy_logps['chosen']   - ref_logps['chosen']
    rejected_delta = policy_logps['rejected'] - ref_logps['rejected']

    # The margin we want to be positive and large.
    logits = beta * (chosen_delta - rejected_delta)

    # -log(sigmoid(margin)): pushes chosen up, rejected down,
    # but the reference terms punish drifting too far from base.
    return -F.logsigmoid(logits).mean()

Two things worth staring at. The reference-model terms are not decoration — subtract them and the loss will happily crank up the chosen response's probability by wrecking the model's general fluency, because nothing anchors it. And 'beta' is the single most consequential knob: it controls how hard you are allowed to pull away from the base model. Small beta lets the model move a lot (and possibly forget how to write); large beta keeps it timid and barely tuned. It is the same anchor-vs-drift tension RLHF handles with a KL penalty, just spelled differently.

The gotcha nobody puts in the tutorial: the verbosity tax

Before the decision framework, one failure that will bite you first, and is not in most write-ups. Human labelers, on average, prefer longer answers — they read 'more thorough' as 'better' even when the extra length is padding. So your preference data has a hidden correlation: chosen responses are, say, 30% longer than rejected ones (an illustrative figure — measure your own). The model does not learn 'be better'; it learns the cheapest pattern that predicts 'chosen', which is: be longer. Now flip back to my support example. Length leaking in as the dominant signal would actively push you toward the rejected answer — the rambling 402 explanation was the longer one. You wanted specificity and trained in verbosity, and every completeness metric looks great while your users drown in preamble. This is why serious DPO pipelines length-normalize or balance token counts between chosen and rejected before training. If you take one operational thing from this post, take that.

Choosing: RLHF, DPO, or neither

The honest default is DPO, but 'default' is not 'always'. Here is how I actually decide, and note that the first item is the one most teams skip past too fast.

• Reach for NEITHER first when you have not exhausted supervised fine-tuning (SFT). If the model has never seen good examples of your task, preference tuning is premature — it sharpens a distinction the model can't yet make. A few hundred high-quality demonstrations often beat a preference run. Preference tuning is polish, not foundation.
• Reach for DPO when you have clean preference pairs, want a stable single-loss run you can debug, and your signal is roughly 'this answer is better than that one' on comparable-length responses. This is 80% of real cases.
• Reach for RLHF (or its online variants) when the best answer is genuinely open-ended and hard to pin to a fixed pair — creative work, long-horizon reasoning, safety edge cases — and you can afford to build, monitor, and periodically retrain a reward model. The online exploration RLHF does is a real advantage on the hardest problems; you pay for it in operational pain.
• Reach for a RULE or verifier, not preferences at all, when correctness is checkable — code that must compile, math with a known answer, JSON that must parse. A unit test is a better teacher than a labeler's opinion, and cheaper.

The trade-off in one line: DPO buys you stability and simplicity by freezing the comparison signal into a static dataset; RLHF spends complexity to keep that signal live and exploratory. Static is easier to trust and easier to break subtly; live is powerful and painful.

Whichever method you pick, the verbosity tax hints at a nastier general property of preference tuning: the training loss can look perfect while the model degrades, because the loss only measures 'did I separate chosen from rejected', not 'is the model actually good'. Watch for these too, none of which show up in the loss curve:

1. Contaminated pairs. If your 'chosen' and 'rejected' were written by different systems (say, chosen by a frontier model, rejected by your base model), the model can learn to imitate a style signature rather than the quality difference. Keep the two sides of a pair as comparable as possible in origin.
2. Reference drift over multiple rounds. If you DPO on top of a model that was already DPO'd and reuse the same reference, the anchor no longer matches the policy's starting point and beta stops meaning what you think. Re-snapshot the reference each round.
3. Silent capability regression. Anchoring prevents catastrophic forgetting but not gradual erosion. Always keep a held-out set of tasks you are NOT tuning for (basic math, formatting, a second language) and check them before and after. A model that got more agreeable and less capable is a very common, very quiet outcome.
4. Preferring the labeler's blind spots. The model inherits exactly what your annotators liked, biases included. If your labelers reward confident tone, you will train in confident wrongness. Garbage preferences, garbage model — and confidence is the most seductive garbage.

The through-line: alignment is taught by comparison, not specification — but comparison is a leaky teacher. It teaches whatever most reliably predicts your labels, and if length or tone or origin predicts them better than quality does, that is what you get. The algorithm is the solved part; curating pairs that isolate the thing you actually care about is the whole job, and it is mostly editorial work, not machine learning.

The bottom line

Preference tuning turns a capable base model into one that behaves the way people want, and DPO is the pragmatic default that gets most teams most of the way there without an RL loop to babysit. But the leverage is not in choosing RLHF versus DPO — it is upstream, in whether your preference pairs isolate quality or accidentally encode length, tone, and origin. Build SFT first, length-balance your pairs, keep a held-out capability set, re-snapshot your reference between rounds, and treat a clean loss curve with suspicion. Do that and the algorithm mostly takes care of itself. Skip it and you will ship a model that is longer, friendlier, and quietly worse.