Hardware h.264 Decode Fails with Library Error on Android (Quest3) with decodebin3 -> amcvideodec-omxqcomvideodecoderavc
Describe your issue
I've gotten GStreamer 1.22.12 and 1.24.4 both running on Meta Quest 3 through an Android Native plugin for Unity3D (v2022.3). I am able to normally view video with the plugin, like video test source, etc. I am also able to view h.264 encoded video using decodebin, such as with the following pipeline:
udpsrc port=7331 ! application/x-rtp,media=video,payload=96 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! video/x-raw,format=RGB ! appsink name=videoSink sync=0
However, the above scenario does not engage hardware decoding via omx plugins. Apparent effects were 1080p 60fps would decode well enough, but 4k video was out of the question. The Quest 3 hw decoder is supposed to be way better than that, so I did some digging and found this stack overflow post about it.
After upgrading omx elements' rankings in the way described in that post, I did get some output from gstreamer regarding hw decode. However, the pipeline completely dies at the end of it. I've hooked up a pad_added_handler function to the decode3 bin and the gldownload element I'm running it into for clarity into what is going on behind the scenes. Log output is below.
After reading GStreamer v1.24 release notes, I saw the omx plugin had been removed from distribution. So, I downgraded to GStreamer v1.22. Unfortunately, I am getting exactly the same output as I was before, which is making me scratch my head.
Expected Behavior
Hardware (FAST) h.264 video decode of a UDP RTP video stream.
Observed Behavior
Software (SLOW) h.264 video decode of a UDP RTP video stream, or a pipeline failure.
Setup
- Operating System:
- Meta Quest OS vSQ3A220605.009.A1 (Android, SDK 31)
- Device: Computer / Tablet / Mobile / Virtual Machine
- Meta Quest 3
- GStreamer Version:
- GStreamer v1.22.12, 1.24.4
- Command line:
Steps to reproduce the bug
Here is the latest pipeline string I've been attempting:
udpsrc port=7331 ! application/x-rtp,media=video,payload=96 ! rtph264depay ! h264parse ! decodebin3 name=decoder ! video/x-raw(memory:GLMemory) ! gldownload name=downloader ! video/x-raw ! videoconvert name=converter ! video/x-raw,format=RGB ! appsink name=videoSink sync=0
However, any pipeline string with decodebin3 in it will either give software video decode output, or playback will fail if I upgrade OMX elements rankings.
How reproducible is the bug?
Always
Screenshots if relevant
Solutions you have tried
Related non-duplicate issues
Additional Information
Info: Upgrading OMX!
Info: Upgragded feature.amcviddec-omxqcomvideodecoderavc
Info: Upgragded feature.amcviddec-omxqcomvideodecoderhevc
Info: Upgragded feature.amcviddec-omxqcomvideodecodervp9
Info: Upgragded feature.amcvidenc-omxqcomvideoencoderavc
Info: Upgragded feature.amcvidenc-omxqcomvideoencoderhevc
Info: Creating Pipeline: udpsrc port=7331 ! application/x-rtp,media=video,payload=96 ! rtph264depay ! h264parse ! decodebin3 name=decoder ! gldownload name=downloader ! videoconvert name=converter ! video/x-raw,format=RGB ! appsink name=videoSink sync=0
Info: Got decodebin element
Info: Got download element
Info: Got converter element
Info: : GstPipelineHandler::CreatePipeline
Info: : Pipeline State Changes: GST_STATE_NULL--> GST_STATE_READY
Info: : Pipeline State Changes: GST_STATE_READY--> GST_STATE_PAUSED
Warning: Can't seek to invalid time
Info: : Playing
Received new pad video_0 from source decoder.
Pad caps: video/x-raw(memory:GLMemory), format=(string)RGBA, texture-target=(string)external-oes
New pad linked to element: downloader
Error: gst_amc_jni_object_local_unref: assertion 'object != NULL' failed
Error: gst_amc_jni_object_local_unref: assertion 'object != NULL' failed
Warning: : GstPipelineHandler::HandleMessage(): error in module amcvideodec-omxqcomvideodecoderavc1 reported: GStreamer encountered a general supporting library error. - ../sys/androidmedia/gstamcvideodec.c(2012): gst_amc_video_dec_set_format (): /GstPipeline:pipeline0/GstDecodebin3:decoder/GstAmcVideoDec-OmxQcomVideoDecoderAvc:amcvideodec-omxqcomvideodecoderavc1:
Could not retrieve application class loader
Warning: : GstPipelineHandler::HandleMessage(): warning in module converter reported: Internal GStreamer error: code not implemented. Please file a bug at https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/new.