vtenc: encoding RGB..RGBA64 to H.264, ProRes always fails on Apple M1 Max (and likely Pro)
GStreamer version: 1.20.3.
Operating system: macOS 12.5.
Hardware: MacBookPro18,2 (likely earlier MacBookPro18,3 too), NOT Macmini9,1.
Reproduce: run either of the following commands (in a writable directory):
gst-launch-1.0 videotestsrc num-buffers=100 ! video/x-raw, format=RGB ! videoconvert ! vtenc_h264 ! mp4mux ! filesink location=videotestsrc.mp4
or
gst-launch-1.0 videotestsrc num-buffers=100 ! video/x-raw, format=RGB ! videoconvert ! vtenc_prores ! qtmux ! filesink location=videotestsrc.mov
Expected: the resulting files are valid movies of 3.3s of test video.
Actual: the resulting files are empty (0 bytes), the terminal output is attached: Output_H.264.txt, Output_ProRes.txt.
Note: in practice, our application uses GStreamer APIs to write video files in either format, and it works fine on earlier Apple machines (including M1 Mini of 2020), but not on ones with the newer M1 Pro/Max chips.