Skip to content

D3D12: Implement H264 video encode and decode

Sil Vilerino requested to merge sivileri/mesa:user/sivileri/d3d12_video_merge_into_mesa into main

The following merge request implements the necessary changes to implement a d3d12 video gallium driver and enable the following VAAPI entrypoints for H264/NV12 bitstreams. The pipe_video_codec implementation is designed to be extensible in a way that adding new codec support only needs extending/implementing the codec specific structures keeping the core decoder/encoder generic. Tested decode, encode, transcode with ffmpeg using VAAPI and playback using X11 on both mpvand gstreamer vaah264dec ! vaapisink on a WSLg X11 Window.

Enabled VA entrypoints:

vainfo: VA-API version: 1.14 (libva 2.13.0.1)
vainfo: Driver version: Mesa Gallium driver 22.0.0-devel for D3D12
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice

Winsys changes:

  • Add a vl_winsys_xlib_swrast for displaying decoded buffers in system memory with SHM and a vl_winsys_dri_vgem for off-screen video acceleration

Gallium changes:

  • Add multiple slice support to pipe_h264_enc_picture_desc
  • Add multiple reference lists support to pipe_h264_enc_picture_desc
  • Add MinLumaBiPredSize8x8 to pipe_h264_sps, pic_init_qs_minus26 to pipe_h264_pps
  • Add new pipe_video_cap values for video encoding multi-slice and multi-reference support

VDPAU frontend changes:

  • Add initialization path for vl_winsys_xlib_swrast and vl_winsys_dri_vgem in vdp_imp_device_create_x11
  • Fix st_vdpau_resource_from_description: Add format when opening resource from handle
  • Fill value of level_idc,MinLumaBiPredSize8x8 to pipe_h264_sps, pic_init_qs_minus26 to pipe_h264_pps

VA frontend changes:

  • VaDeriveImage to use resource_get_info to get stride and offset independently for both NV12 planes
  • Add support for VAConfigAttribEncSliceStructure, VAConfigAttribEncMaxSlices and VAConfigAttribEncMaxRefFrames using the new pipe_video_cap values . Please note that if the new VA caps are not supported by the gallium driver, the behavior is backward compatible
  • vlVaHandlePictureParameterBufferH264 fill pipe_h264_sps/pps values for: MinLumaBiPredSize8x8, pic_init_qs_minus26, chroma_format_idc, bit_depth_chroma_minus8, bit_depth_luma_minus8
  • vlVaHandleIQMatrixBufferH264 set seq_scaling_matrix_present_flag in pipe_h264_sps when scaling matrix present
  • vlVaHandleVAEncSliceParameterBufferTypeH264: Add support for multiple references encoding . Please note that if the new VA caps did not expose support for more than 1 reference frame, the behavior is backward compatible
  • vlVaHandleVAEncSliceParameterBufferTypeH264: Add support for multiple slices encoding . Please note that if the new VA caps did not expose support for more than 1 slice, the behavior is backward compatible
  • Add initialization path for vl_winsys_xlib_swrast and vl_winsys_dri_vgem in VA_DRIVER_INIT_FUNC

OMX frontend changes:

  • Fill value of MinLumaBiPredSize8x8 to pipe_h264_sps, pic_init_qs_minus26 to pipe_h264_pps
  • Add value of num_ref_idx_lx_active_minus1 and ref_idx_lx_list for both L0/L1 lists

Gallium radeon/r600 changes:

  • Modify reading logic from pipe_h264_enc_picture_desc, change single L0/L1 reference to first of L0/L1 list . Please note if the new VA caps did not expose support for more than 1 reference frame, the behavior is backward compatible using the first L0/L1 element on the lists, previously a single variable for L0/L1 refences

Gallium d3d12 changes:

  • D3D12 implementation of pipe_video_buffer
  • Add video dpb resource pool managers
  • Add video decode implementation of pipe_video_codec
  • Add video encode implementation of pipe_video_codec
  • Add create_video_codec/create_video_buffer entrypoints to d3d12 driver
  • Add util video functions to d3d12_format
  • Planar resources: Add support for multiple plane simultaneous map, individual plane render target creation, specific plane slice copy, implement pipe_screen::resource_get_info
  • Add support for d3d12 video in d3d12_screen

Meson changes:

  • Support d3d12 as a video-supporting driver

D3D12 Video driver architecture:

The following are for common video support:

  • d3d12_context: Entrypoints for video buffer and decoder/encoder creation
  • d3d12_video_screen: Handles pipe video caps using the underlying D3D12 Video Device
  • d3d12_video_buffer: Implements the pipe_video_buffer interface for d3d12 video using a planar (NV12) d3d12_resource
  • d3d12_video_*_dpb_manager: AoT/TextureArray resource pools for Encode/Decode reference frames

Decode specific code:

  • d3d12_video_dec: Core video decoder (codec agnostic) implementation of pipe_video_codec interface for decode
  • d3d12_video_dec_reference_mgr: Codec agnostic reference picture manager for decode
  • d3d12_video_dec_h264: H264 codec specific functions called by d3d12_video_dec

Encode specific code:

  • d3d12_video_enc: Core video decoder (codec agnostic) implementation of pipe_video_codec interface for encode
  • d3d12_video_enc_h264: H264 codec specific functions called by d3d12_video_enc d3d12_video_encoder_bitstream/ ..bitstream_builder(h264) / d3d12_video_encoder_nalu_writer: Helper functions to build the H264 video stream bit-level headers based on D3D12 Encode API structures
  • d3d12_video_encoder_reference_manager(_h264): Manages the H264 codec specific structures for encoding
Edited by Sil Vilerino

Merge request reports