vah265dec: Segfault when accessing P010_10LE buffers / P010_10LE DMABufs not handled zero-copy by glimagesink
Sample file from https://nascompares.com/guide/4k-1080p-hdr-hevc-and-uhd-files-to-download-and-test-your-plex-media-server-nas/, but this happens on any 10-bit H265 from what I can see. 8-bit H265 works fine.
$ gst-launch-1.0 filesrc location=jellyfish-3-mbps-hd-hevc-10bit.mkv ! matroskademux ! h265parse ! vah265dec ! glimagesink
[...]
#0 0x00007ffff7c4488d in __memmove_avx512_unaligned_erms () at /lib64/libc.so.6
#1 0x00007fffe15adb2d in _gl_mem_create (gl_mem=0x7fff94126b40, error=<optimized out>) at ../subprojects/gst-plugins-base/gst-libs/gst/gl/gstglmemorypbo.c:237
#2 0x00007fffe159297c in _mem_create_gl (context=0x6f0bd0, transfer=0x7fffb17f8700) at ../subprojects/gst-plugins-base/gst-libs/gst/gl/gstglbasememory.c:106
#3 0x00007fffe15c5bd3 in _run_message_sync (message=0x7fffb17f8690) at ../subprojects/gst-plugins-base/gst-libs/gst/gl/gstglwindow.c:635
#4 0x00007fffe15c5c32 in _run_message_async (message=0x7fff94348c40) at ../subprojects/gst-plugins-base/gst-libs/gst/gl/gstglwindow.c:702
#5 0x00007ffff7d6748d in g_idle_dispatch () at /lib64/libglib-2.0.so.0
#6 0x00007ffff7d6b48c in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#7 0x00007ffff7dc9648 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#8 0x00007ffff7d6aa8f in g_main_loop_run () at /lib64/libglib-2.0.so.0
#9 0x00007fffe159fe4d in gst_gl_context_create_thread (context=0x6f0bd0) at ../subprojects/gst-plugins-base/gst-libs/gst/gl/gstglcontext.c:1401
#10 0x00007ffff7d99983 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#11 0x00007ffff7b5d907 in start_thread () at /lib64/libc.so.6
#12 0x00007ffff7be3870 in clone3 () at /lib64/libc.so.6
This also shows that glimagesink is actually uploading (or trying to) the memory again instead of directly passing through the DMABufs, which is an additional problem.
Similarly, gst-play-1.0
shows the same exploding in a different place (different sink selected that can't do P010_10LE):
#0 unpack_P010_10LE (info=<optimized out>, flags=GST_VIDEO_PACK_FLAG_NONE, dest=<optimized out>, data=<optimized out>, stride=<optimized out>, x=<optimized out>, y=170, width=1920)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-format.c:5035
#1 0x00007ffff7f08980 in do_unpack_lines (cache=0x7fffac001e70, idx=<optimized out>, out_line=<optimized out>, in_line=170, user_data=0x7fffac005920)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:2952
#2 0x00007ffff7f08bd6 in gst_line_cache_get_lines (cache=0x7fffac001e70, idx=0, out_line=170, in_line=170, n_lines=n_lines@entry=1)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:602
#3 0x00007ffff7f0abf4 in do_convert_lines (cache=0x7fffac0064f0, idx=0, out_line=-1409252608, in_line=-1211484672, user_data=0x7fffac005920)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:3079
#4 0x00007ffff7f08bd6 in gst_line_cache_get_lines (cache=0x7fffac0064f0, idx=0, out_line=170, in_line=in_line@entry=170, n_lines=<optimized out>)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:602
#5 0x00007ffff7f0b9f3 in convert_generic_task (task=0x7fffac0064b0) at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:3240
#6 0x00007ffff7f0b914 in gst_parallelized_task_runner_run (self=0x7fffac007f00, func=0x7ffff7f0b960 <convert_generic_task>, task_data=0x7fffcc000c70)
at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:282
#7 0x00007ffff7f075ef in video_converter_generic (convert=0x7fffac005920, src=<optimized out>, dest=0x7fffd9378738) at ../subprojects/gst-plugins-base/gst-libs/gst/video/video-converter.c:3339
#8 0x00007fffdac939b4 in gst_video_convert_scale_transform_frame (filter=0x7fffd4329260, in_frame=0x7fffd93784a0, out_frame=0x7fffd9378738)
at ../subprojects/gst-plugins-base/gst/videoconvertscale/gstvideoconvertscale.c:1751
#9 0x00007ffff7ef3dbf in gst_video_filter_transform (trans=0x7fffd4329260, inbuf=<optimized out>, outbuf=<optimized out>) at ../subprojects/gst-plugins-base/gst-libs/gst/video/gstvideofilter.c:276
#10 0x00007ffff77d2231 in default_generate_output (trans=0x7fffd4329260, outbuf=0x7fffd9378a50) at ../subprojects/gstreamer/libs/gst/base/gstbasetransform.c:2202
#11 0x00007ffff77d75b6 in gst_base_transform_chain (pad=<optimized out>, parent=parent@entry=0x7fffd4329260, buffer=buffer@entry=0x7fffe03aa800)
at ../subprojects/gstreamer/libs/gst/base/gstbasetransform.c:2355
#12 0x00007ffff7dde191 in gst_pad_chain_data_unchecked (pad=pad@entry=0x7fffd4329640, type=type@entry=4112, data=data@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4463
#13 0x00007ffff7ddedcb in gst_pad_push_data (pad=pad@entry=0x7fffd4098200, type=type@entry=4112, data=data@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4739
#14 0x00007ffff7ddec2d in gst_pad_push (pad=0x7fffd4098200, buffer=buffer@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4858
#15 0x00007ffff7dc497e in gst_proxy_pad_chain_default (pad=pad@entry=0x7fffd408d470, parent=<optimized out>, buffer=buffer@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstghostpad.c:127
#16 0x00007ffff7dde191 in gst_pad_chain_data_unchecked (pad=pad@entry=0x7fffd408d470, type=type@entry=4112, data=data@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4463
#17 0x00007ffff7ddedcb in gst_pad_push_data (pad=pad@entry=0x7fffd4330650, type=type@entry=4112, data=data@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4739
#18 0x00007ffff7ddec2d in gst_pad_push (pad=0x7fffd4330650, buffer=buffer@entry=0x7fffe03aa800) at ../subprojects/gstreamer/gst/gstpad.c:4858
#19 0x00007fffe9d43b79 in gst_queue_push_one (queue=0x7fffd4100510) at ../subprojects/gstreamer/plugins/elements/gstqueue.c:1388
#20 gst_queue_loop (pad=<optimized out>) at ../subprojects/gstreamer/plugins/elements/gstqueue.c:1541
#21 0x00007ffff7e10f5e in gst_task_func (task=0x7fffd4353500) at ../subprojects/gstreamer/gst/gsttask.c:399
#22 0x00007ffff7bb30a2 in g_thread_pool_thread_proxy.lto_priv () at /lib64/libglib-2.0.so.0
#23 0x00007ffff7bb0983 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#24 0x00007ffff7893907 in start_thread () at /lib64/libc.so.6
#25 0x00007ffff7919870 in clone3 () at /lib64/libc.so.6
Output of vainfo
on the device that is used here:
$ vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
libva info: VA-API version 1.18.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.18 (libva 2.18.2)
vainfo: Driver version: Mesa Gallium driver 23.1.5 for AMD Radeon RX 550 / 550 Series (polaris12, LLVM 16.0.6, DRM 3.52, 6.4.9-200.fc38.x86_64)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc