Tokenization at the Edges: What a Telugu Voice Clone Taught Me When the Model Kept Dropping My English

A language model never reads your words. Before it sees a single character of your prompt, the text is chopped into tokens — sub-word chunks, turned into a sequence of integer IDs. That invisible step quietly explains a surprising amount of LLM behavior: why you pay per token, why context windows fill up, why models can't count the letters in "strawberry," and why the same sentence in Hindi or Telugu can cost two or three times more than in English.

Most explainers stop at the concept. I want to show you the other end of it — what happens when the tokenizer's blind spots aren't a curiosity but a wall you have to build around. I hit that wall building IndicF5, a Telugu voice-clone text-to-speech system that reads mixed Telugu-and-English tech articles aloud in my own voice, running locally on my Apple Silicon Mac. The model would happily read Telugu. It would silently drop the English. Fixing that meant hand-building an entire text front-end, and it taught me more about tokenization than any diagram ever did.

Words in, tokens out

The standard picture first, because it's correct and it matters. A tokenizer splits text with an algorithm — usually byte-pair encoding — that starts from characters and repeatedly merges the most frequent pairs into larger units. Common words collapse into a single token; rare words, names, and code fragment into several. The result is a fixed vocabulary of sub-word pieces the model was trained on, frozen at training time. You cannot change how a given model chunks your text.

• A token is roughly four characters of English — about three-quarters of a word on average, but it swings wildly with content.
• Common words are one token; rare ones shatter. A long technical term or an unusual name can become many tokens.
• Whitespace and punctuation count. A leading space is often part of the token, so "the" and " the" can tokenize differently.
• The vocabulary is per-model and frozen. Whatever it under-represents, it under-represents for you, permanently.

That last point is the one that bit me. Most tokenizers were built on English-heavy corpora, so non-Latin scripts were under-represented — which is exactly why multilingual text costs more tokens. But in a speech model, an under-served script doesn't just cost more. It can vanish.

When the tokenizer just drops your text on the floor

IndicF5 is the ai4bharat/IndicF5 model — a flow-matching TTS built on an F5-TTS DiT backbone with a Vocos vocoder. It's genuinely good at Indic scripts. The catch: I don't write pure Telugu. A sentence from one of my own posts mixes Telugu grammar with English nouns, a percentage, and a unit dropped right in — think "this model gave 92.5% accuracy at 3.2 GHz," but with the connective words in Telugu script and the technical terms left in English. Feed that straight into IndicF5 and it skips or mangles the Latin-script words entirely. "model," "GPU," "92.5%," "GHz" — the model's front-end simply doesn't have a good representation for them, so they come out wrong or not at all.

This is tokenization failure in its most literal form. The generic version of this lesson is "non-English text costs more tokens. " The lived version is: if a script or symbol is outside what the model's text front-end understands, the safest thing the model can do is ignore it — and "ignore it" is a terrible answer when your whole job is to read the text out loud. There was no prompt trick that would fix this. The model reasons over its tokens, and my English words weren't cleanly in that space.

The tokenizer is not a suggestion you can argue with in the prompt. If a model's vocabulary doesn't cover your input, your only real lever is the text you hand it — so I had to rewrite the input into a form the model could actually see.

Building a normalizer to feed the tokenizer

So I wrote transliterate. py, a from-scratch Telugu normalizer whose entire purpose is to rewrite mixed text into something IndicF5's front-end handles cleanly — before it ever reaches the model. It's the un-glamorous layer that sits in front of every serious tokenizer, and building one by hand is the fastest way I know to understand what "just feed it text" is quietly hiding. It does four jobs:

1. A term dictionary of roughly 200 English-to-Telugu entries for the ML, CS, and hardware jargon that shows up constantly — "model," "GPU," and friends — mapped to fixed Telugu spellings so they're pronounced consistently.
2. A rule-based Latin-to-Telugu fallback transliterator for everything not in the dictionary: it handles digraphs, soft-c, English "-tion/-sion" endings, and the matra-versus-full-vowel distinction, so a novel English word still gets a sane Telugu rendering.
3. An integer-to-Telugu-words converter that spells numbers out up to about 99 crore using Indian crore/lakh grouping — because "92.5%" has to become spoken words, not a numeral the model will fumble.
4. Unit and symbol handling for percent, currency, and units like GHz, GB, and nm, so "Rs1500" or "3.2 GHz" reads as language rather than punctuation soup.

And it goes one level deeper than tokenization, into pronunciation. Even for valid Telugu, the base model mispronounces some hard conjuncts. The Telugu word for "failure" has a la-ya conjunct the model slurs, so the normalizer hard-codes a respelling that inserts a zero-width non-joiner (U+200C) after the virama to break the conjunct. That is about as close to the metal as text normalization gets: reaching into the Unicode itself to steer how a sequence gets tokenized and voiced. It's the same family of problem as an LLM being unable to spell — the model reasons over chunks, not letters, so if you need control at the character level, you engineer it into the input.

The other walls: loading the thing and running it on a Mac

Tokenization was the conceptual wall, but two engineering walls sat right next to it, and they're worth sharing because they cost me real time. The first was a silent weight-loading bug. IndicF5's published checkpoint was saved while wrapped in torch. compile, so every weight key carries an . _orig_mod. prefix. With Dynamo disabled, the un-wrapped modules don't match those keys, and the auto-loader matches nothing — no error, just a random-initialized transformer producing pure noise. The fix is to load the safetensors myself and strip the prefix so the keys line up:

sd = load_file(hf_hub_download(REPO_ID, "model.safetensors"))
remapped = {k.replace("._orig_mod.", "."): v for k, v in sd.items()}
res = model.load_state_dict(remapped, strict=False)
n_miss, n_unexp = len(res.missing_keys), len(res.unexpected_keys)
if n_miss or n_unexp:
    print(f"WARNING: weight load mismatch — {n_miss} missing, {n_unexp} unexpected")
else:
    print("Weights loaded cleanly (0 missing, 0 unexpected).")

After the strip, the log prints "Weights loaded cleanly (0 missing, 0 unexpected). " That one line is the difference between a working voice and static. The same . _orig_mod. trap shows up again in the fine-tune converter and in the voice-blending script — torch. compile checkpoints leak their wrapper straight into your key names.

The second wall was speed control, and it's a neat echo of the tokenization lesson: don't fight the model on its own turf. F5-TTS computes generation duration as ref_audio_len/ref_text_len * gen_text_len / MODEL_SPEED, so if you try to slow the model down, it pads the duration and starts repeating text. The right move is to generate at native speed and time-stretch the output audio afterward, pitch-preserving, via Rubber Band or librosa:

# F5 gen duration = ref_audio_len/ref_text_len * gen_text_len / MODEL_SPEED.
# Slowing the *model* makes F5-TTS pad the duration and REPEAT text, so we slow
# the audio afterwards instead (pitch-preserving, via Rubber Band else librosa).
if SPEED and abs(SPEED - 1.0) > 1e-3 and audio.size:
    audio = _time_stretch(audio, SPEED)

Running it all locally on Apple MPS came with its own tax. I set PYTORCH_ENABLE_MPS_FALLBACK=1 so unsupported ops fall back to CPU instead of crashing, disabled Dynamo so IndicF5's torch. compile wrapper becomes a harmless no-op, and monkey-patched torchaudio. load off its new TorchCodec backend to read the reference clip via soundfile instead. It's not fast: in one uncached run my log shows "Done in 30.6s" to synthesize 5.8s of audio on MPS — roughly five times slower than real time, and that's a single observed run on my machine, not a controlled benchmark. But it's fully offline, private, and speaks in a voice cloned from a single ~15-second reference clip, which is a trade I'll take.

What to take from this

Tokenization is the hidden layer between your text and the model, and it shapes cost, context, and capability. Models read sub-word tokens, not words or letters — which is why you pay per token, why non-English text costs more, and why LLMs are oddly bad at spelling and counting. But the deeper lesson from building IndicF5 is what happens at the edges of the vocabulary, where "costs more" quietly becomes "doesn't work. "

• Inspect the tokens on your actual inputs — code, JSON, and non-English text especially. The abstract count turns concrete fast, and you'll find the rare strings exploding into many tokens.
• When a model can't represent your input, fix the input, not the prompt. A normalizer in front of the model is often the only lever you have, and it's a real one.
• Character-level control lives in the input. If you need exact spelling, pronunciation, or number handling, engineer it — down to inserting a zero-width joiner if that's what it takes.
• Budget for the multilingual tax before it surprises you, in both your bill and your context window.

You can't change a model's tokenizer. But understanding where its vocabulary runs out — and being willing to build the boring front-end that meets it there — is the difference between a demo that reads Telugu and a system that reads what I actually write. Look at the tokens once, at the messy edges, and you'll never treat "just feed it text" as free again.