Jittery playback of H.264 Level 4.1 movies in macOS (both x86_64 and arm64)

GStreamer version: 1.20.4 (and earlier)
macOS version: 12.5, 11.7 (and earlier)

Reproduce:

1. (Optional) Encode a video like this: ffmpeg -i FrameNumber.mov -pix_fmt yuv420p -c:v libx264 -profile:v high -level:v 4.1 FrameNumberJitter.mp4.
2. (Or) Download the pre-encoded video: H.264_level-4.1_FrameNumberJitter.mp4.
3. Run gst-play-1.0 /path/to/H.264_level-4.1_FrameNumberJitter.mp4, observe the frame numbers and the stripes' motion.

Expected: the frame numbers increase and the stripes move monotonously.

Actual: the frame numbers increase and decrease in some (regular?) pattern, the stripes move right and left, with only the overall trend being correct.

Note: in practice, our application uses GStreamer APIs to work with videos on a frame-by-frame basis, and the frames order is incorrect for some drone-generated footage.