TL;DR — My tracker picks its target among face detections by appearance (a colour histogram) and keeps a slowly updating reference histogram of the target. During an occlusion, the real subject went undetected, and a stranger’s low-confidence detection (score 0.22) passed the appearance check. The tracker jumped to it, then gradually learned the stranger’s colours and couldn’t come back. The fix: appearance decides which detection is the target; only a confident detection is allowed to move the tracker. Lowering the learning rate would only have slowed the drift. What fixed it was restricting the input.

What was running

VertiCut follows one person through a horizontal video and crops a vertical 9:16 frame around them. Per frame:

1. A face detector (YuNet) returns boxes with confidence scores.
2. The tracker picks the box whose colour histogram best matches its reference for the target, provided it passes size and colour thresholds.
3. The tracker’s anchor (where it thinks the target is) moves to that box.
4. The reference histogram updates slowly toward what it just saw, so the tracker can follow lighting changes. Rate 0.02, roughly a 50-frame time constant.

The symptom

On a test clip where the subject is briefly occluded, the crop kept jumping between two people. My early notes described it as “19 back-and-forth swings.”

That undersold it. When I measured it properly, the tracker switched to another person at t = 5.93 s and then stayed mostly on that person for about ten seconds, occasionally swinging back.

	Before	After
Jumps (> 0.5 crop-widths in one frame)	20	0
Detection frames on the wrong person	37.1% (219/591)	0.0% (0/462)
Longest wrong-person stretch	73 frames (2.43 s)	0
Final trajectory on the wrong person	39.0% (231/592)	0.0%

What happened, frame by frame

Frame 178. The real subject wasn’t detected. The only box left was the stranger’s, with a YuNet score of 0.22. Its colour similarity to the reference was 0.503, above the 0.45 threshold. Its size ratio was 0.73, above the 0.45 threshold. So it matched. Position smoothing was set to no smoothing, so the anchor jumped 0.43 (in normalized coordinates) in a single frame.

The frames after. Now the reference histogram was updating from the stranger. Slowly, at rate 0.02, but steadily. By frame 268:

	YuNet score	Colour similarity to reference
Real subject	0.93	0.649
Stranger	0.21	0.872

The real subject was back, detected with high confidence. But the tracker chose by colour similarity, and its reference now looked more like the stranger. The stranger won. The tracker had learned the wrong person and couldn’t get back.

The principle I already had, in the wrong place

The odd part is that the code already contained the right rule. A comment on the seeding threshold said, roughly: low-score boxes are for keeping an established track going; they are not evidence for choosing the target.

Seeding enforced it. Tracking didn’t. Once a track existed, a 0.22 box could still move it.

It wasn’t the first time. Earlier, before seeding enforced the threshold, low-confidence boxes were allowed when choosing the initial target, and the tracker once picked a sofa. Same problem, one level deeper.

The fix

Appearance decides which detection is the target. It doesn’t, on its own, decide where the target is.

Two rules now sit between “this box matches” and “move the anchor”:

1. Only a confident detection can move the anchor. The score must be at least 0.6, the same threshold used for seeding. A low-score match no longer moves anything. That frame is marked as coasting. If coasting lasts past a short grace period, the frame is reported downstream as missing and filled in later.

2. Even a confident detection must be physically reachable. If accepting it would mean the subject moved faster than 5.0 crop-widths per second, it’s rejected. This is a backstop for the case where two people are both detected confidently and look alike.

Neither rule restricts where the tracker looks. Matching still searches the whole frame by appearance. Rule 2 limits how far the target can move per unit of time, not its position. A camera pan moves the subject quickly across the frame, but continuously. A one-frame jump is a different kind of event.

Rule 1 also fixed something I didn’t write down

The reference histogram updates inside the same branch that moves the anchor. So rule 1, “only confident detections move the anchor”, also means only confident detections update the reference. That stopped the learned drift.

But that’s an implicit coupling. There’s one if, and nothing says a second rule depends on it. If someone loosens the score threshold later, the drift comes back without any visible sign. So a test pins both behaviours: given a low-score input, the reference histogram, reference size and anchor must all stay unchanged. Given a confident input, they must update.

The decoy in that test has a score of 0.22, written as a literal, not derived from the threshold. If it were written as “threshold minus something,” you could lower the threshold as far as you liked and the test would still pass.

Where the 5.0 came from

The maximum subject speed across all seven test clips, in crop-widths per second:

Clip	Max speed
Static talk	0.42	normal
Subject near the edge	1.69	normal
60 fps clip	1.98	normal
Walking across	2.39	normal, fastest real motion
Camera pan	2.39	normal, must not break
Multiple faces	8.69	2 wrong matches
Occlusion	39.81	the hijack

Normal motion tops out at 2.39. Wrong matches start at 8.69. 5.0 sits in the gap. Even at 2.5, half the shipped value, the camera-pan clip triggers the limit zero times.

One detail: each axis is normalized by its own output dimension. If you measure vertical motion in crop widths, a 9:16 frame inflates it 3.2×, and ordinary hand shake on the camera-pan clip reads as 7.54 instead of 2.39. That would put the threshold inside normal motion.

The 5.0 limit doesn’t fix the occlusion clip. On that clip, it never fires at any value I tried. Rule 1 is what fixed it. The speed limit is insurance, set in a flat region where its exact value barely matters.

What I didn’t change, and why

The learning rate. The obvious fix for “the reference drifted” is to update it more slowly. But 0.02 is already about a 50-frame time constant. Going lower only delays contamination; it doesn’t stop it. It would also make the tracker slower to follow real lighting changes, which is the reason it updates at all. Restrict the input, not the rate.

Position smoothing. Half-trusting a detection by smoothing toward it sounds safer than jumping. But smoothing inside the tracker is a causal low-pass filter, and it adds lag that the zero-phase smoother downstream can’t remove, the same reason I rejected a dead zone. A bad detection should be rejected, not half-trusted.

Reading “0.0%” honestly

The wrong-person rate went from 37.1% to 0.0%. The denominator also changed: 591 detection frames became 462. Rule 1 stopped accepting 129 frames’ worth of weak detections.

So 0.0% doesn’t mean “129 frames got more accurate.” It means 219 frames that showed the wrong person were replaced by 130 frames that are explicitly marked as estimated. That’s a better outcome, because an estimate the user can see beats a confident wrong answer. But it isn’t the same claim, and the before/after table alone would let you believe the stronger one.

Takeaway

When a tracker can learn, what you let it learn from matters more than how fast it learns. A matching threshold decides which detection looks most like the target. It shouldn’t also decide whether that detection is trustworthy enough to move the tracker. Keep those as separate decisions, and test the coupling you’re relying on, so nobody loosens it later without noticing.

---

I'm Kou (日々野 考), an indie iOS developer in Japan building on-device ML apps under kouworks. I write up the things that cost me days, so they cost you minutes.

Apps:

• VoiceKit — filler-word removal, subtitles, and summarization, entirely on-device.
• KidsMask — automatic face mosaic for photos and video; no face data ever leaves the device.
• VlogMask — real-time face mosaic camera for vlogging.
• VertiCut — turns horizontal video into vertical 9:16, following a person automatically.