GitLab

I attempted to play a H.264 video using VLC 3.0.8, with hardware acceleration using AMD's VAAPI driver in Mesa, but it fails to work correctly.

I checked the error output for VLC, and this is what's printed:

[00007f34cc001f50] glconv_vaapi_x11 gl error: vaDeriveImage: operation failed
[00007f34dc003320] main video output error: video output creation failed
[00007f34e8d8ada0] main decoder error: failed to create video output

VLC calls vaCreateSurfaces to create a bunch of surfaces, and then uses vaDeriveImage to check if it can get an image object (and hence a buffer object) that represents the surface, so that it can then be accessed via vaAcquireBufferHandle.

VLC's approach works with Intel's VAAPI driver (https://github.com/intel/intel-vaapi-driver), but doesn't work with AMD's VAAPI implementation in Mesa.

I looked at the AMD VAAPI implementation in Mesa, and it looks like there are bugs in the implementation. vaCreateSurfaces creates surfaces with the NV12 format for H.264 video, and also marks the surfaces as interlaced. However, in vaDeriveImage, it returns VA_STATUS_ERROR_OPERATION_FAILED if the surface is interlaced, and it also doesn't handle NV12 format (which is the only kind of surface that vaCreateSurfaces will create in the first place).

That seems like a bug to me.