Vision's object tracker leaks 10 MB per call

TL;DR — VNTrackObjectRequest / VNSequenceRequestHandler leaked roughly 10 MB of CPU heap per perform call in my app. With tracking on, memory climbed past 3 GB unbounded; with it off, flat. VNDetectFaceRectanglesRequest does not leak. I removed tracking entirely and replaced it with per-frame detection plus IoU matching for identity continuity.

Measured on device with Instruments; resolved 2026-05-29.

What was running

VlogMask is a real-time camera that mosaics faces as you record. Detect faces in the frame, track them between frames so identity stays stable, blur them, display and record the result — all live, at 30fps, at 4K.

Tracking between frames is what VNTrackObjectRequest is for. You detect once, hand the observation to a VNSequenceRequestHandler, and it follows that object forward. It’s cheaper than detecting every frame and it keeps identity across frames for free. That’s exactly the shape of my problem, so that’s what I used.

The symptom

The app died at about 80 seconds of recording. Every time, roughly the same point.

Memory climbed steadily from launch and never came down. No spike, no single large allocation — just a straight line up until the system killed the process.

A linear climb against wall-clock time in a loop running at 30fps means something per-frame isn’t being released. That much was obvious. Which per-frame thing was not.

The diagnosis that was wrong

My first conclusion was that AVSampleBufferDisplayLayer was retaining the IOSurface backing each preview frame.

This is a completely reasonable thing to believe. The display layer does hold onto buffers. I was feeding it a CVPixelBuffer every frame. There are plenty of reports of exactly this problem. And when I went looking, I found a real CVPixelBuffer leak in the display path.

So I fixed it. And the app still died at 80 seconds.

That’s the part worth sitting with. The diagnosis wasn’t wrong in the sense of being fictional — there genuinely was a leak where I said there was. It just wasn’t the one killing me. A partially correct diagnosis is more expensive than a completely wrong one, because fixing it produces a real improvement that feels like progress and confirms the theory you should be abandoning.

How I actually found it

I stopped reading allocation stacks and started turning subsystems off.

I already had Constants.Debug.disable* flags in the codebase for various pipeline stages. I added one for face tracking and ran the same recording twice, watching memory:

  • Tracking on: unbounded climb, past 3 GB
  • Tracking off: flat

That’s the whole experiment. Two runs, one variable, an unambiguous answer. It took a few minutes and it was conclusive in a way that hours of reading retain/release chains had not been.

Instruments then confirmed the size. Allocations grouped by category showed the persistent bytes accumulating under Malloc 10.56 MiB — one such block per perform call, never freed.

Roughly 10 MB per tracked frame. At 30fps you don’t need to do the arithmetic to see why 80 seconds was the limit.

Detection was fine. VNDetectFaceRectanglesRequest runs through the same framework and leaks nothing. It’s specifically the sequence-handler tracking path.

The fix

I removed tracking. There’s no flag to set and no release pattern that helps — the allocation isn’t mine to free.

What replaced it:

Detect every 2 frames instead of tracking every frame. Detection costs about 50 ms per detection frame at 4K in my pipeline (full-frame plus tiles). At frameInterval = 2, that’s 50 ms of work per 66 ms of wall clock, which holds 30fps with a little room. Detection being more expensive than tracking is the trade I was avoiding by using the tracker; it turns out it’s affordable.

Recover identity with IoU matching. Tracking gave me identity continuity for free, so removing it meant rebuilding that. Matching each new detection to the previous frame’s boxes by intersection-over-union is enough when the gap is 2 frames — faces don’t move far in 66 ms. This is a dozen lines and it leaks nothing.

In the offline app that shares this code, where there’s no realtime budget to protect, the same approach runs every frame and adds a Feature Print similarity check on top of IoU. Live, IoU alone carries it.

Stop producing pixel buffers for the preview. Since I was in there anyway: the preview now renders directly to a CAMetalLayer drawable rather than creating a CVPixelBuffer per frame for the display layer. That removed the separate leak I’d found earlier — properly this time, by not allocating rather than by trying to release.

One thing that made it worse

While hunting this I added ciContext.clearCaches() to the per-frame path, reasoning that if memory is growing, clearing caches should help.

It doesn’t. It destroys CoreImage’s buffer reuse pool, which forces fresh drawable-sized allocations on the next frame. Calling it every frame is strictly counterproductive — you pay allocation cost to throw away the thing that was preventing allocation.

If you need CoreImage to hold less, cacheIntermediates: false on the context is the actual control. clearCaches() is for occasional use on a memory warning, not a per-frame habit.

Takeaway

When memory grows per-frame, toggle subsystems before you read allocation stacks. A leak in a pipeline with six stages has six suspects, and each one can be switched off in about a minute. Two runs with one flag flipped gives you a categorical answer. Reading retain chains gives you a plausible story, and plausible stories are exactly what got me three days into the wrong one.

The generalization I’d actually stand behind: when a fix produces real improvement but doesn’t solve the problem, treat that as evidence against the theory, not for it. I had a leak, I fixed a leak, the app still died — and I spent longer than I should have assuming I’d fixed it incompletely rather than that I’d fixed a different one.

I'm Kou (日々野 考), an indie iOS developer in Japan building on-device ML apps under kouworks. I write up the things that cost me days, so they cost you minutes.

Apps:

  • VoiceKit — filler-word removal, subtitles, and summarization, entirely on-device.
  • KidsMask — automatic face mosaic for photos and video; no face data ever leaves the device.
  • VlogMask — real-time face mosaic camera for vlogging.
  • VertiCut — turns horizontal video into vertical 9:16, following a person automatically.