Apple provides a sample iOS project called AVCamFilter where they use an MTKView to render the camera preview to the screen. The problem is that the frame duration – the amount of time each individual camera frame spends being displayed on the screen – is not stable. The camera stream is running at 30 FPS, that is one frame delivered every 33.3 milliseconds, so it stands to reason that we should be able to display these frames one after another with a frame duration of 33.3 ms. In reality the frame duration is unstable – most of the time it is about 33.3 ms but sometimes it is roughly 16.7 or 50.1 ms.

According to documentation the default behavior for MTKView is to draw to the screen at 60 FPS, that is one frame every 16.7 ms, so each camera frame would be drawn twice before it is replaced by the next one. The fact that MTKView’s draw loop and the camera’s capture loop are not synchronized explains the problem – the time elapsed between a draw and a capture will gradually drift causing frames to go from being captured just before a draw to just after a draw and so on and so forth resulting in some frames being displayed for three draws or one draw instead of the expected two. The MTKView draw loop can be synchronized with the camera by disabling the default behavior and manually calling draw whenever a new camera frame arrives, but all of this is still not synchronized with the device display itself which is rendering at 60 FPS in its own loop. So, we still have the same fundamental timing problem. I know CADisplayLink is meant for synchronizing things to the display but there is no way for the capture stream to use it.

So how do we render the camera preview with a stable 33.3 ms frame duration? Is the instability an expected behavior with no workaround?