TL;DR — For a virtual camera that follows a person, “dead zone + EMA” is the intuitive design: hold still until the subject leaves a central band, then follow smoothly. It failed at every threshold I tried. A dead zone leaves a steady-state lag of D + v·τ while the subject moves, and on a clip of someone walking across the frame that reached 0.73 crop-widths: the face left the frame. Putting a zero-phase (forward-backward) filter after it doesn’t help, because a zero-phase filter only cancels the lag its own forward pass introduced. What worked was a plain EMA run forward and backward, with no dead zone at all.

What I was building

VertiCut crops horizontal video to vertical 9:16 by following a person. Per frame, the tracker gives the subject’s horizontal position. The crop window needs to follow it without jitter, the way a camera operator would. The crop moves only horizontally: a full-height 9:16 window cut from 16:9 has no vertical freedom.

This is offline processing. The whole trajectory is known before smoothing starts, so the filter can look at future frames. That makes zero-phase filtering available, which a live camera couldn’t use.

The design that looked right

Good camera operators don’t chase every small movement. They hold the frame and pan only when the subject is about to leave it. A dead zone captures that directly: ignore movement within a band of half-width D around the current crop center, and follow only when the subject moves past it. Put an EMA after it to smooth the pans.

My planning notes rated this design as the most promising. It was wrong.

How I tested it

Seven real clips: static talking, walking across the frame, occlusion, multiple faces, handheld shake, and so on. Nineteen smoothing methods and parameter sets. Numeric metrics were used only to eliminate candidates that were clearly bad. The final choice came from three rounds of watching the output, two of them blind.

Dead zone + EMA failed at every threshold: D = 0.10, 0.15 and 0.20.

Why the dead zone fails: steady-state lag

All distances here are in output crop widths (out_w), because that’s what a viewer perceives: how far the subject is from the center of the frame they’re watching, not of the source video.

While the subject moves at a steady speed v, the dead zone holds the crop back by D. The EMA behind it adds its own lag of about v·τ. Together:

lag ≈ D + v·τ

On the walking clip, the subject’s median speed was 1.15 out_w per second. With D = 0.15 and the dead-zone variant’s τ = 0.5 s:

0.15 + 1.15 × 0.5 ≈ 0.73 out_w

The crop trails the subject by nearly three-quarters of its own width. The face leaves the frame.

Why zero-phase filtering doesn’t rescue it

This is the part I want to stress, because it’s the reasonable objection: “Then put a zero-phase filter after the dead zone. Forward-backward filtering has no phase lag.”

That’s true of the filter’s own lag. A forward-backward filter works like this:

// α computed per frame from the real time step
// forward pass
for i in 1..<x.count { y[i] = y[i-1] + α[i] * (x[i] - y[i-1]) }
// backward pass: the same filter, run in reverse
for i in stride(from: y.count - 2, through: 0, by: -1) { z[i] = z[i+1] + α[i+1] * (y[i] - z[i+1]) }

(A sketch of the shape, not production code.)

The forward pass delays the signal. The backward pass delays it by the same amount in the opposite direction in time. The two cancel, and the result has no net lag.

But it can only cancel lag it created itself. The dead zone runs before it, and it’s a nonlinear stage. By the time the zero-phase filter sees the signal, the dead zone’s lag is part of its input. The filter has no way to tell “the subject was here” apart from “the dead zone held the crop here.” It faithfully smooths an already-late trajectory.

A zero-phase filter cancels the delay of its own causal pass, and nothing upstream of it.

The intuition was right. The mechanism was wrong.

The dead zone’s goal, a camera that stays still most of the time, didn’t need a dead zone. A forward-backward EMA with τ = 0.3 s already holds still. On the static talking clip, the crop was still in 95.7% of frames. On another clip, 80.7%. The stillness comes from the smoothing itself, without a separate stage that introduces lag the smoothing can’t remove.

The same rule showed up again

Later, in the tracker, there was a setting to smooth the tracked position before handing it to the trajectory smoother. Any value below “no smoothing” would make the tracker a causal low-pass filter, which adds lag the zero-phase smoother downstream can’t remove. Same reason, same answer: I left it at no smoothing.

The general rule: if a later stage is zero-phase, don’t put anything causal and lagging in front of it. And when an input is bad, don’t half-trust it by filtering. Reject it.

Other things that mattered

Define τ in seconds, not frames. Compute α = 1 − exp(−Δt / τ) per frame, with Δt taken from the actual presentation timestamps, not the nominal frame rate. Then 30 fps, 60 fps and variable-frame-rate video all look the same for the same τ.

Detect outliers with a median; don’t replace them with it. I used a 5-frame median only to find outliers: detections more than 10% of the crop width away from it. Those are dropped and treated as missing, and every other frame keeps its raw value. When I replaced outliers with the median value instead, the replacement itself counted as tracking error, and every method failed the cut.

Clamp last, and don’t feed the clamp back into the filter. Keeping the crop inside the source frame is the final step. If the clamped value feeds back into the filter state, the filter gets dragged while the crop is pinned to an edge, and it’s late when it comes off.

Lower error isn’t the same as looking better. Gaussian smoothing had small deviation numbers but visibly busier motion. Its acceleration RMS was about 1.8–1.9× that of the forward-backward EMA (3.63 vs 1.92 on the walking clip). Two Gaussian settings had identical deviation metrics, and only one passed when I watched them. That’s why the numbers only eliminated candidates and my eyes made the final call.

Takeaway

A zero-phase filter cancels its own lag, not anyone else’s. Before adding a stage in front of one (a dead zone, a hysteresis band, a causal low-pass), work out its steady-state lag at the speeds you’ll actually see. Then check whether the behaviour you wanted from that stage is already there in the smoothed output. In my case it was, and the dead zone only added a lag nothing downstream could remove.

---

I'm Kou (日々野 考), an indie iOS developer in Japan building on-device ML apps under kouworks. I write up the things that cost me days, so they cost you minutes.

Apps:

• VoiceKit — filler-word removal, subtitles, and summarization, entirely on-device.
• KidsMask — automatic face mosaic for photos and video; no face data ever leaves the device.
• VlogMask — real-time face mosaic camera for vlogging.
• VertiCut — turns horizontal video into vertical 9:16, following a person automatically.