androidmedia: Errors and crashes in amcvideodec when releasing frames while flushing
I am using a pipeline which receives an h264 video stream over TCP, decodes it with decodebin and displays it with glimagesink on Android. I am negotiating zerocopy GLMemory caps with autoplug-query, this issue doesn't happen with software buffers. I ran into this when the video stream's resolution changes dynamically, so the caps change and gst_amc_video_dec_set_format stops the decoder, flushes, and starts the decoder back up.
Specifically, _gl_sync_release_buffer can be called for a buffer after the decoder loop has already stopped, and when gst_amc_codec_release_output_buffer fails it sets sync->sink->gl_error which puts the element into a failed state, breaking the pipeline. This happens about 99% of the time, I only saw it not happen once which is how I tracked down this exact cause.
Additionally, occasionally the _amc_gl_wait(_gl) callback on the GLSyncMeta can be fired during a similar window and attempt to g_mutex_lock an invalid lock, which causes a segfault. This only happens maybe 20% of the time.
Unfortunately it may be device/codec specific, I've only tried it on our Android hardware, so I have no idea, but a comment in _amc_gl_possibly_wait_for_gl_sync does mention bad frame listener behaviour in similar cases.
I've attached my patch which simply adds guards on sync->sink->flushing, which is a bit of a shotgun approach. One remaining issue I've noticed is that sometimes after a rotation it spams "could not wait for frame" and doesn't update on screen for a couple seconds but does recover, not sure if that's related to my patch or not.
Does anyone who's more familiar with the androidmedia plugin have any thoughts?