TL;DR — Whisper is trained on clean transcripts, so it structurally omits pure hesitations. In my measurements this held identically across base, small, and turbo, even though their character error rates differ by 5×. If you’re building filler removal by matching transcript text against a dictionary, the words you want are not in the text. What worked instead: treat the gaps in Whisper’s word timeline as the signal, and confirm them with voice activity detection.

What I was building

VoiceKit removes filler words from recordings — the ums, the uhs, the Japanese えー and えっと — and exports the cleaned audio with subtitles. All on device, WhisperKit for transcription.

The design was the obvious one:

1. Transcribe with word-level timestamps.
2. Match each word against a filler dictionary.
3. Cut those time ranges out of the audio.

Three steps, all of them things Whisper already gives you. I wrote the dictionary first.

The words weren’t there

I ran a spontaneous-speech sample containing twelve fillers I’d identified by ear. The transcript contained almost none of them.

Not misspelled. Not mistimed. Absent. Whisper had heard the hesitation, decided it wasn’t content, and produced fluent text without it.

What did survive into the transcript was the other kind of filler — the ones made of real words. In Japanese: なんか (like), その (that), なんていうか (how do I put it). These are exactly the words that are sometimes fillers and sometimes meaningful. なんか is a filler when it’s padding and a real word when it means “something.”

So the dictionary approach was biased wrong in both directions at once. It missed everything I wanted to cut, and it caught things I needed to keep. A dictionary is only useful if the thing you’re looking for is in the text, and the cleanest fillers — the pure hesitations, the ones users most want gone — are precisely the ones Whisper refuses to write down.

It isn’t a model size problem

My first instinct was that this was an accuracy issue and a bigger model would fix it. It isn’t, and I have the numbers.

Japanese character error rate on a read-script sample, release build:

Model	Size	Japanese CER	RTF
base	74 MB	35.6%	0.05
small	244 MB	17.7%	0.08
large-v3-turbo	626 MB	6.8%	0.13

That’s a 5× spread in accuracy. base is unusable — better than one character in three is wrong. turbo is the only one producing readable Japanese.

The filler-dropping behaviour was identical across all three. Scaling from 74 MB to 626 MB fixes character accuracy dramatically and does nothing at all to hesitation omission.

That’s the tell that this is not an error. It’s the training objective working correctly. Whisper learned from transcripts that humans wrote, and humans writing transcripts don’t type the ums. A model that reproduced every hesitation would be scoring worse against its own training data. The omission isn’t a limitation of the model — it’s the product.

This is also why I expect it to hold beyond Japanese. English um / uh / er are the same category of token — pure hesitation, routinely omitted by human transcribers — so the same mechanism should drop them. To be precise about what I actually measured: the numbers above are Japanese. The English case is a structural expectation, not a result I have.

Reading the gaps instead

Here’s what turned it around. Whisper produces word-level timestamps. If a filler occupies 600ms of audio and Whisper doesn’t transcribe it, that 600ms doesn’t vanish from the recording — it becomes a hole in the word timeline.

And a hole is detectable without any text at all.

filler candidate = (VAD speech regions) − (Whisper word regions)

Run energy-based voice activity detection over the raw audio. Anywhere VAD says “someone is speaking” and Whisper says “no word here,” someone said something the model chose not to write down. Filter by plausible filler duration (0.15–2.5s) and you have your candidates.

The weakness becomes the signal. I don’t need Whisper to recognize fillers — I need it to keep ignoring them consistently, which it does reliably.

The dictionary didn’t disappear, it just got demoted. It now runs as a secondary pass to catch the word-shaped fillers that do make it into the transcript — the exact set gap detection can’t see, because those have word timestamps and leave no hole.

Three layers, in priority order:

1. VAD + word-gap — the primary detector. Catches pure hesitations.
2. Dictionary — secondary. Catches transcribed word-fillers.
3. Manual editing — the backstop. Nothing is deleted without the user seeing it.

On my spontaneous-speech sample: gap detection found 8 of the 9 fillers in its category (~0.89 recall). The dictionary pass added 3 more. 11 of 12 total, with the last one left for manual review. A deliberately planted trap — an instance of その used with real meaning — was correctly classified as not-a-filler.

Cut from the VAD boundary, not the word timestamp

A detail that cost me time. Once you have a filler’s time range, you’d think you cut at Whisper’s word timestamps. You can’t.

Whisper’s word boundaries on filler-adjacent regions were off by up to ±500ms in my measurements. That’s not a cut, that’s a mangling — you either clip the front of the next real word or leave half the um audible.

VAD region boundaries on the same material landed within ±90ms, which a 50–100ms crossfade absorbs cleanly.

So: detect the gap using Whisper’s timeline, but take the actual cut points from VAD. Whisper tells you where to look; the energy envelope tells you where to cut.

Related: don’t protect word endings with a fixed guard interval. I had a +0.15s pad after each real word to avoid clipping, and it managed to fail in both directions simultaneously — too short to protect long word endings, long enough to swallow fillers that started immediately after. Replacing it with an energy-based extension (extend while energy continues, stop at a dip) fixed both.

The verification trap

This is the one I’d most want to warn people about, because it wastes time in a way that feels like progress.

The natural way to check your filler removal worked is to re-transcribe the output and see if the filler is gone.

Do not do this. You are asking a model that capriciously omits fillers whether a filler is present. It will tell you the filler is gone when it’s still audible, because it wasn’t going to transcribe it either way. And it hallucinates in the other direction too — I’ve seen Whisper insert an えー at the start of a segment that had none.

Whisper is unusable as ground truth here, in both directions, for the same reason it was unusable as a detector.

Verify against the actual energy envelope — the VAD frame energies — or with your own ears. Not the transcript.

What I couldn’t solve

I want to be straight about the limits, because I hit a wall and it isn’t moving.

Quiet fillers are not separable. A softly voiced えー sitting 8–16 dB above the noise floor, immediately adjacent to a real word, is continuous with that word in the energy domain. There is no boundary to find. I tried valley-snapping and asymmetric margins; the same instances survived both.

Word-ending protection and filler exposure are the same dial. Every adjustment that better preserves the tail of a real word also better preserves a filler that begins right after it. They aren’t independent problems, so there’s no setting that wins both.

Controlled ground truth is not obtainable. I tried recording scripted samples with fillers inserted at known positions. Performed fillers are acoustically different from real ones — different energy, different duration, different articulation. A model tuned on acted fillers is tuned on the wrong distribution, and real recordings can’t be labelled precisely enough to serve as truth. I don’t think a rigorous accuracy claim for this feature is constructible.

So the honest conclusion is that fully automatic, perfectly correct filler removal is not achievable. That changed the product, not the feature: automatic detection stays the main flow and the main pitch, but it produces candidates a human confirms, never a silent bulk delete. Manual editing isn’t a fallback for edge cases, it’s a required part of the design.

That also settled the precision/recall question. Since every candidate passes through a review screen where rejecting one costs a single tap, recall is worth far more than precision here. A missed filler is a defect the user can’t fix without hunting for it; a false positive is a tap.

Takeaway

When a model won’t give you what you need, check whether its refusal is itself consistent enough to use. I spent a while trying to make Whisper transcribe fillers — better prompts, bigger models, decoding options. None of it was going to work, because not transcribing them is what the model was built to do.

The version that worked treats the omission as a stable, documented behaviour and reads the shape of the hole instead. That reframe is available more often than it looks: a model that fails predictably is still an information source, just not the one printed on the label.

---

I'm Kou (日々野 考), an indie iOS developer in Japan building on-device ML apps under kouworks. I write up the things that cost me days, so they cost you minutes.

Apps:

• VoiceKit — filler-word removal, subtitles, and summarization, entirely on-device.
• KidsMask — automatic face mosaic for photos and video; no face data ever leaves the device.
• VlogMask — real-time face mosaic camera for vlogging.
• VertiCut — turns horizontal video into vertical 9:16, following a person automatically.