TL;DR — While building VertiCut, I collected nine tests that were green while the thing I cared about was broken. None were flaky, and none had bugs in the usual sense. Each one measured a quantity next to the one I needed, and that neighbour was fine. The habit I rely on now is cheap: after writing a test, break the code on purpose and watch the test go red. If it can’t go red, it isn’t testing what you think.

The pattern

A failing test tells you something is wrong. A passing test tells you less than it feels like. It tells you that the thing it measures is fine. Whether that’s the thing you care about is a separate question, and the test can’t answer it.

None of the tests below failed to catch a bug because of a typo or a flaky timer. Each one measured something that could not have failed, because the bug lived one step away from where the test was looking.

Nine cases

1. Two decode paths, compared by geometry. The app reads video two ways — one for the first frame, one for the rest — and a test checked that both paths produced the same image. It compared sizes and positions. They matched. The colours didn’t: one path went through device RGB and the other didn’t, which shifted every pixel by about 8/255 on average. It wasn’t visible and didn’t affect face detection. It mattered only to the tracker, which matches people by colour histogram — so it quietly made the correct face look slightly less like itself.

2. Watermark, tested by geometry. The watermark tests checked position, size and alpha. All correct. But the composite made the image underneath brighter instead of darker, because a CIColorMatrix scaled only the premultiplied alpha. Every geometric fact about the mark was true. The way it was drawn on top was wrong.

3. Watermark visibility, tested by signed mean. A follow-up test checked that the mark visibly changed the region it sat on, by comparing the region’s average before and after. The mark is a dark plate with light text. On a mid-grey background, the two cancel out: the region’s mean moved 6/255 while individual pixels moved by tens. The test reported a clearly visible watermark as “no change.”

4. Localization, tested against the catalog. LocalizationTests checked that every string had an English translation in the string catalog. They all did. But the error messages were plain Swift literals returned from errorDescription, never passed through String(localized:), so the app never asked the catalog for them. 23 error messages had never been shown in English, while the test checking for English stayed green.

5. The scan that caught #4, scoped by hand. The fix for #4 added a test that reads the source and flags raw Japanese literals in errorDescription. It only looked in Services/. The same defect was in 22 more places under Views/. Choosing one directory by hand put everything else outside the net.

6. A diagnostics test that wrote its own input. A test checked that the “Copy Details” report contained the trail of recent failures. It called the trail-writing function itself, then read the report. Real failure sites didn’t write to the trail at all — most only logged to NSLog. The test had created the condition it was supposed to verify.

7. A structural scan that excluded its own target. After #6, a test scanned every screen that used the shared FailureText component and asserted it also had a copy button. The screen that was actually missing the button didn’t use FailureText — it had its own layout. The rule was keyed on the component, so the screen it needed to protect was outside the rule from the start. Reverted to the broken state, the test passed.

8. Thirty device runs on the wrong path. I ran the import-and-export flow thirty times on a real device. The test read a fixture from a read-only path inside the test bundle. The import service, the copy step and the tmp/imports/ directory that the bug reports named were never exercised once.

9. A test with its own copy of the formula. The timeline’s frame-to-position conversion was consolidated into one type. TimelineZoomTests still carried its own copy of the old formula and tested that. It would have stayed green indefinitely, testing math the app no longer used.

Two more that weren’t tests, but fail the same way

Tags aren’t pixels. A colour-space tag comparison reported MATCH on output whose pixels were off by 8.35/255. Matching metadata says the file claims a colour space, not that its contents are in it.

A check that fails on good input isn’t a check. I tried Xcode’s automated accessibility audit for clipped text. It flagged clipping at the default text size, where nothing was clipped, and returned the same list at every size. If a check can’t tell good from bad, it tells you nothing. I only found that out by running it on a known-good screen.

The rules I follow now

Make it go red. After writing a test, revert the fix, or break the code in the way the test claims to catch, and run it. If it stays green, it’s measuring something else. That’s how #7 was found: reverted to the broken state, the scan passed. It also works the other way: when I added the source scan from #4, I checked it against a hand-written list of files, and the scan found two files the list had missed. The proof that a test is green for the right reason is that you can make it red.

Write down what the test doesn’t say. Next to each non-trivial test, I note what it does not check. The source scan for translations, for example, doesn’t see helpers in other files, doesn’t see a literal assigned to a let first, and doesn’t check whether the catalog actually has a translation. That last one is a different test, and both are needed. A test whose gaps are written down is much harder to over-trust.

Measure the quantity you’re claiming. If the claim is “the watermark darkens the image,” assert on brightness, not geometry. If the claim is “these two images are the same,” compare pixels, not tags or sizes.

Use mean absolute difference, not mean difference. A signed average cancels out exactly the changes you most want to see: dark next to light, before next to after.

Don’t derive test inputs from the threshold under test. One tracker test uses a decoy detection with a low confidence score. It’s a literal 0.22, not “threshold minus something.” If you derive it, you can loosen the threshold as far as you like and the test still passes — which is the regression it exists to catch.

Why this matters more now

Most of VertiCut’s code and tests were written with an AI coding agent. That makes this problem bigger, not smaller. An agent produces tests that are plausible, numerous and green, and “it passes” is the easiest thing for it to report. Every case above was plausible. The defence isn’t more tests. It’s asking of each one what it actually says, and making it go red once.

Takeaway

A green test proves the test’s claim, not yours. The gap between the two is invisible from the test output. So before trusting a new test, make it fail on the bug it’s meant to catch, and write one line about what it doesn’t cover. Both take minutes, and neither depends on the test being well written in the first place.

---

I'm Kou (日々野 考), an indie iOS developer in Japan building on-device ML apps under kouworks. I write up the things that cost me days, so they cost you minutes.

Apps:

• VoiceKit — filler-word removal, subtitles, and summarization, entirely on-device.
• KidsMask — automatic face mosaic for photos and video; no face data ever leaves the device.
• VlogMask — real-time face mosaic camera for vlogging.
• VertiCut — turns horizontal video into vertical 9:16, following a person automatically.