Crash in ruvd_end_frame when calling vaBeginPicture/vaEndPicture without rendering anything
Submitted by 67b..@..ix.org
Assigned to Default DRI bug account
Link to original bug (#105368)
Description
VAAPI testing has revealed that ruvd_end_frame does not handle a particular edge case (see below), i.e. it crashes.
Source of the crash is here: https://cgit.freedesktop.org/mesa/mesa/tree/src/gallium/drivers/radeon/radeon_uvd.c?id=e96e6f60f705c04a3d437eea9fe308826b494c67#n1246
The memset fails when you call vaBeginPicture/vaEndPicture without any relevant vaRenderPicture calls in-between and have previously decoded some frames using the context. Then ruvd_begin_frame (triggered by data buffers) is not called to set up a new bs_ptr, and the old pointer that was unmapped already is still around, so memset will segfault. Inserting dec->bs_ptr = NULL after the buffer_unmap works for me, but I don't know if this is the solution or just a workaround.
ffmpeg seems to do this under certain circumstances, which is how this bug surfaced. The vaapi documentation does not seem to forbid this, even if it does not make a lot of sense.
Version: git