Synthetic Data for LLMs: Why Your Yield Rate Matters More Than Your Prompt

Every synthetic-data tutorial tells you the same thing: generate examples with a strong model, then filter them. True, and useless. It is like saying the secret to a restaurant is 'cook the food, then serve it. ' The interesting question is never the two steps. It is the ratio between them. How many examples do you throw away for every one you keep? That ratio is the whole game, and almost nobody talks about it. If you take away nothing else from this post, take this: synthetic data is a manufacturing-yield problem, not a prompting problem.

The mental model: you are running a factory line, not a data faucet

Picture a factory that stamps out parts. Raw material goes in one end, a stamping machine (your teacher model) produces units, and an inspection station rejects the defective ones. The metric a factory manager lives and dies by is yield: units shipped divided by units produced. A line at 90% yield is healthy. A line at 15% yield is on fire, even if the 15% that pass are flawless.

Synthetic data works exactly like this. Your teacher model is the stamping machine. Your validator (a judge model, a rule, a unit test, an actual code execution) is the inspection station. Your yield rate is the fraction of generated examples that survive filtering. The part nobody tells you is that yield is diagnostic. A low yield is not just wasteful; it is a signal that something upstream is broken.

Yield is not an efficiency stat. It is a smoke detector. When 80% of your generated examples fail validation, the problem is almost never a filter that is too strict. It is a generator that does not understand the task yet.

This reframe changes what you do when things go wrong. The naive engineer sees a bad fine-tune and rewrites the generation prompt to be more elaborate. The engineer who thinks in yield first asks two questions: what was my acceptance rate, and what did the rejects have in common? The rejects are the most information-dense artifact in the entire pipeline, and most people delete them unexamined.

A worked example: 200 real tickets, one classifier, and where it goes wrong

Let us make this concrete. Say you are building a support-ticket classifier that sorts incoming messages into eight categories: billing, bug, feature-request, account-access, and so on. You have exactly 200 real, hand-labeled tickets: enough to see the shape of the problem, nowhere near enough to fine-tune a small model reliably. A classic synthetic-data situation. The tempting move is to prompt a strong model: 'Generate 5,000 realistic support tickets, each labeled with one of these eight categories. ' You will get 5,000 tickets. They will look great. And the fine-tune will underperform in a way that is genuinely hard to debug, for three reasons you can predict in advance.

First, label priors. Ask a model for support tickets with no constraints and it gravitates toward the archetypal ones: password resets and refund requests write themselves. Your real inbox might be 30% account-access, but your synthetic set drifts to 55% because those are the tickets the model finds easiest to imagine. You have taught the classifier a distribution that does not match reality, and it will over-predict the easy classes on real traffic.

Second, degeneracy inside a class. Your 600 'bug' tickets will not be 600 different bugs. They will be maybe 40 distinct scenarios, each rephrased fifteen ways. To a string-level dedup check they look unique: different words, same skeleton. The model learns the 40 skeletons and faceplants on the 41st real bug.

Third, the boundary cases you actually need are the ones that never get generated. A ticket that is arguably both billing and bug ('I was charged twice because the retry button double-fires') is exactly where a classifier earns its keep, and exactly what unconstrained generation avoids, because ambiguous examples are harder to invent than clean ones.

So the fix is not a cleverer one-shot prompt. It is to generate against a plan. Enumerate the eight categories, set target counts that mirror your real 200-ticket distribution, and for each category seed generation with two or three of your real tickets so the model grounds in your actual product's vocabulary. Then generate in small batches per category and measure yield per category, because the yield will not be uniform, and the variation is the signal. Suppose (illustratively, these are made-up numbers to show the reasoning, not a measured result) your validator accepts 88% of generated 'billing' tickets but only 34% of 'account-access' ones. That gap tells you the account-access category is under-specified: the model does not have a crisp enough definition to hit it reliably, so it produces near-misses your validator correctly rejects. The fix lives in the category definition and the seeds, not in the temperature knob. A pipeline that reports one blended yield hides this; per-slice yield surfaces it.

The filter is code, and most of it is deduplication

Here is the shape of the loop, with the part that actually matters (the filter) written out. Note that deduplication is semantic, not string-based. Rephrasings of the same idea must collide even when they share no words.

from sentence_transformers import SentenceTransformer, util,,embedder = SentenceTransformer('all-MiniLM-L6-v2'),SIM_THRESHOLD = 0.92          # tune per task; higher means stricter dedup,,kept = []                     # accepted (text, label) pairs,kept_embeddings = [],,def is_near_duplicate(text):,    if not kept_embeddings:,        return False,    emb = embedder.encode(text, convert_to_tensor=True),    sims = util.cos_sim(emb, kept_embeddings),    return float(sims.max()) >= SIM_THRESHOLD,,def accept(example, validate):,    # validate() is your inspection station: a judge model,,    # a rule, or actually running the ticket through a label check.,    if not validate(example):,        return 'rejected_invalid',    if is_near_duplicate(example['text']):,        return 'rejected_duplicate',    emb = embedder.encode(example['text'], convert_to_tensor=True),    kept.append(example),    kept_embeddings.append(emb),    return 'kept',,# Track WHY things get rejected. The reject reasons are your,# diagnostics: mostly 'invalid' means the generator misunderstands,# the task; mostly 'duplicate' means diversity has collapsed.,outcomes = {'kept': 0, 'rejected_invalid': 0, 'rejected_duplicate': 0},for ex in generated_batch:,    outcomes[accept(ex, validate)] += 1,print('yield:', outcomes['kept'] / sum(outcomes.values()))

The one idea worth internalizing is the last comment block. Splitting rejections into 'invalid' versus 'duplicate' turns your filter into a dashboard. A pile of invalid rejects points at the generator's understanding. A pile of duplicate rejects points at diversity collapse. Same low yield, opposite fixes, and you can only tell them apart if you count them separately.

When to reach for synthetic data, and when it quietly backfires

Synthetic data is not a universal answer. It is a specific tool with a shape, and the shape is verifiability. Here is the decision table I would use.

Reach for it when

• You can cheaply verify a generated example is correct: running code, checking a SQL query executes, matching a deterministic rule. Verifiability is the single best predictor of synthetic-data success.
• The skill is common in the teacher's training but you need it in a smaller, cheaper, or private model. This is distillation, and it is where synthetic data shines.
• You have a small real seed set (even 50 to 200 examples) to ground generation in your actual domain and to hold out as a real evaluation set.
• The failure mode of a wrong example is bounded. A slightly-off paraphrase in an augmentation set will not poison the model the way a wrong label in a reasoning chain will.

Be very careful, or do not bother, when

• Correctness is expensive or subjective to verify. If your only validator is another model's opinion, you are stacking two models' blind spots (see the correlated-judge trap below).
• The task needs knowledge the teacher does not reliably have. A model cannot manufacture ground truth about your internal systems, last quarter's numbers, or a niche regulation; it will confabulate fluently.
• You are generating the evaluation set from the same model family you will evaluate. That is grading the exam with the answer key the student wrote.
• You need real-world distribution and tail behavior. Synthetic traffic clusters around the plausible middle; production breaks in the tails, which are exactly what generation smooths away.

Three gotchas that do not show up until production

The correlated judge. The obvious pipeline uses one strong model to generate and the same model (or its sibling) to judge. The problem: a generator and a judge from the same family share blind spots. An error the generator is prone to make is often an error the judge is prone to wave through. Your yield looks high, your quality is not. Where you can, make the validator a different kind of thing entirely (an executor, a rule, a smaller model with a narrow checklist), not a mirror of the generator.

Contamination by paraphrase. Everyone knows not to put test examples in the training set. Fewer people catch that synthetic generation launders contamination: if a real example seeded generation, and a paraphrase of it lands in your training set while the original sits in your held-out eval, you have leakage no exact-match dedup will catch. Your eval score inflates and you will not know until real traffic disagrees. De-duplicate across the train/eval boundary semantically, not just within the training set.

Silent diversity drift over rounds. Iterative pipelines (generate, train, use the improved model to generate more) feel like a flywheel. They can also drift slowly toward the model's own comfortable outputs, the effect people call model collapse. The tell is subtle: diversity falls a little each round while quality holds, so it looks fine on the dashboard you happen to be watching. Keep real data mixed in every round, and track a diversity number (embedding spread, distinct n-grams, per-class variance) as a first-class metric, not an afterthought.

The bottom line

Generation is cheap and getting cheaper, which is precisely why it is not where the value is. The value is in the inspection station and in reading what it rejects. Treat synthetic data as a factory line with an honest yield number, generate against a plan that mirrors your real distribution, dedup semantically across the train/eval boundary, and keep a strand of real data running through every round. Do that and synthetic data becomes what it should be: a way to manufacture the examples you could not otherwise afford, with a quality line you can actually trust.