General problems with dmabufs

The current negotiation of formats is not really suitable for the usual workflow with dmabufs.

Example dmabuf workflow

The usual flow between a producer and consumer of dmabufs is as follows:

1. The consumer queries its backend (gbm, vulkan, vaapi, etc.) for supported formats and modifiers.
2. The consumer sends a list of (format, modifier) pairs to the producer.
3. The producer takes the list (or the intersection of lists from multiple consumers), chooses a format and sends a list of modifiers corresponding to the given format to the buffer allocator (gbm, vulkan, ...) along with the size of the buffer.
4. The allocator will choose the "best" modifier from the list taking the buffer size into consideration and returns a buffer.
5. The producer renders to this buffer and sends it to the consumer along with the selected format and modifier.
6. The consumer imports the buffer to its backend with the given metadata.

Problems with the current implementation

Problems regarding a dmabuf only negotiation (I)

Currently both producer and consumer send their capabilities regarding the videostream (dimensions, formats, modifiers, etc.) to pipewire. Pipewire chooses a combination which should work for both and returns the choosen result. This has the following problems:

1. Formats and modifers are not paired. As such pipewire can return format, modifier combinations, which are invalid, see modifier test.
2. Even if we pair formats and modifiers in SPA_PARAM_EnumFormat (for example replacing the SPA_FORMAT_modifer in SPA_PARAM_EnumFormat with a list of pairs. SPA_FORMAT_modifer in SPA_PARAM_Format should stay the selected modifier as a long), the process of choosing the used Formats will still happen inside pipewire. But to select a good Format requires knowledge only available from the driver/hardware. In that case either pipewire has to add an allocator as a dependency (egl, vulkan, gbm, etc.) and queries it for a combination (and could allocate the buffer at this step right away) or it has to move this decision to an instance, which has this information. On options would be to select a format and send its modifier list to the producer which should do the allocation anyway. The producer can then allocate buffers with this list and has to update pipewire with the used modifier, so pipewire can set the correct SPA_PARAM_Format.

Problems regarding dmabufs and shm buffers (II)

While formats are handled with SPA_PARAM_Format, the buffertype is handled by SPA_PARAM_Buffers. This Parameters are evaluated and merged independent from another. This creates the following problem:

1. Pipewire clients might support some formats both as dmabuf and shm, while other formats are only viable with one buffer type. It is not possible to specify this beforehand. This results in clients negotiating a format only to find no viable shared buffertype.

Proposed requirements for dmabuf usage

If pipewire doesn't want to allocate dmabufs the following things would be needed:

• A way to correlate buffertype and format
• The node, which is responsible for creating the buffers, needs to receive a list of (format, modifier) pairs to choose the used modifier and to emit it back.

Ideas

• Replace SPA_FORMAT_modifer in SPA_PARAM_EnumFormat with a list of (format, modifier) pairs. Formats only supported by shm can be marked as (format, SPA_DATA_Invalid). Clients not specifying SPA_FORMAT_modifer would be treated as shm only.
• Add an additional object to SPA_PARAM_EnumFormat for each buffertype (MemPtr and Memfd use the same underlying buffertype and would be identical) to specify format, modifiers an other differences there.

Appendix:

Discussion about modifiers in libva: https://github.com/intel/libva/pull/505

irc discussion with emersion

columbarius	about pipewire: If i didn't have a misconception on how to use pipewire, this should demonstrate the current shortcommings:
columbarius	https://gitlab.freedesktop.org/columbarius/pipewire-modifier-test/-/tree/master
columbarius	linked it to this issue https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/1055
emersion	oh nice!
emersion	that's a pretty big issue indeed
emersion	oh there's also some shm vs. dmabuf format mixing?
emersion	great :/
columbarius	there is no "real" mixing, because they are handled separately, so first you negotiate the format, then the buffer. If the buffer doesn't match the format .... probably try again and remove some possibilities.
columbarius	So pipewire doesn't keep track of format buffertype combinations, you have to
columbarius	*the pipewire client has to
columbarius	does gbm offer me a list of "usefull" format/modifier pairs? because pipewire probably would need a value for any modifier, if a client just imports buffers with gbm and doesn't care / doesn't know better
emersion	why do you want to use gbm?
emersion	ah, because screencopy wants the client to allocate a dmabuf, and pipewire wants the producer to allocate the dmabuf?
emersion	gbm doesn't give you lists for formats/modifiers
emersion	even if it did, you need to pick one supported by the compositor
emersion	i'd suggest using the linux-dmabuf protocol's "modifier" event for that
columbarius	not only. I was more thinking, if there is a way to use dmabufs without an instance, which gives you format/modifier pairs. You probably don't want to give pipewire a list with every modifier for each format
emersion	wlroots won't let you pick the format
emersion	the compositor dictates the format
columbarius	so you probably just want to announce, you can handle almost all
columbarius	in this case you would just announce the formats/modifiers from the compositor
columbarius	i was thinking about pipewire clients like obs
columbarius	if they use vaapi, they will recieve this list in the future, but if they wan't to do sth. with gbm on their own ...
columbarius	is this a possible usecase, without increasing the format/modifier list to all possible combinations?
emersion	i'm not sure i'm following?
columbarius	if i write a consumer, which (for some reason and with good performance) can handle all possible formats and modifier pairs my hardware supports (vendor specific, etc.) via mapping by gbm or some other mechanism. Then I probably can assume, that any producer will also only send format/modifier pairs, which are supported by the hardware and I have two options: Send all combinations of pairs/modifiers, which
columbarius	the hardware supports (crawling the header file of modifiers), and have a really long list, or use a specific "meta" modifier, which signals for every format I support, every modifier will work just fine.
emersion	you cannot assume gbm supports all modifiers
columbarius	but can you assume, that all software working with modifiers on one hardware has the same restrictions wrt. modifiers?
emersion	no
columbarius	hmmm ....
emersion	for instance, on amdgpu, kms can only do a subset of modifiers, gl/gbm can do a bigger subset, vaapi/vulkan can do a smaller subset
emersion	also not all software is released/upgraded at the same time
columbarius	Then the problem is, how to get a usefull list of pairs for every software, which want's to use dmabufs
emersion	one component could add support for a new modifier, when other components still don't support it
emersion	it would probably be sensible for the pipewire consumer to send (format, modifier) pairs it can deal with
columbarius	yes. I just wanted to make sure, if/that any consumer has access to this information
emersion	some parts are still missing
emersion	gl/vulkan def support it
emersion	vaapi is missing it
emersion	(but that's vaapi's issue)
columbarius	If that's the case, we probably only need a fast merging algorythm for ling lists ^^'
columbarius	*long
emersion	the negociation only happens once, so no need to optimize it
emersion	it's not a hot codepath
emersion	negotiation*
columbarius	But it might be a lot of data? (don't have any clue on how many combinations are possible, but uint64_t has a lot of space))
columbarius	s/possible/realistic/
emersion	if you have 100 formats, each with 50 possible modifiers, it's 40KiB
emersion	(100*4 + 100*50*8)
emersion	plus maybe a few more bytes for lengths and whatnot
emersion	wayland just sends the whole thing over the wire, fwiw
emersion	with today's drivers, expect maybe 20 formats, with 5 modifiers each
emersion	see for yourself with wayland-info i guess
columbarius	I think the vaapi changes are in, before pipewire wants to break its api, but i can't think of a simple solution, which doesn't require breakage.
columbarius	nit: shouldn't it be (100*50*(4+8)) = 60KiB since they are in pairs?
columbarius	ok then it is fine. was just a knowledge gap, that's why i could not evaluate it. thankst
emersion	maybe introduce a DMABUF2 buffer type, and let everyone know that DMABUF is broken
emersion	yeah, it kind of depends how you send it
emersion	fwiw, the kernel has an optimized packed struct to store format/modifier pairs
emersion	see drm_format_modifier_blob
columbarius	thing is FORMAT and BUFFER are two separate Objects
columbarius	we either have to pair those, or create a Format and a FORMAT_DMABUF
columbarius	ahh nice
columbarius	* separate Objects without connection/reference
emersion	mind, the packed struct is annoying to decypher
emersion	it's using a 64-bit bitfield with a sliding window
columbarius	pipewire assumes, that all formats are equal, just with differend kind of "packing"
emersion	there is a libdrm MR to add a helper to decypher it
emersion	sorry, i don't remember anything about pipewire's APIs :P
columbarius	nice, when you create sth. that the next person looking at it writes a helper to not get confused xD
emersion	yea :S
columbarius	I think it would be nice, to try to specify how dmabuf and modifiers work in a usual mediapipeline
columbarius	so should producer and consumer send pipewire all supported formats in a somehow prioritised list and pipewire will guess the best one, or would it better for pipewire to only intersect those lists and return the result to the producer in a way, that the producer can choose the "best" format/modifier combination? Or should the consumer do this.
columbarius	additional: how does this work with multiple planes having different modifiers?
columbarius	So probably a specific question for you: When using multiple planes, do they share a format and only can have different modifiers, or is it possible to have an arbitrary formats with an arbitrary modifier per plane?
emersion	it's not possible to have a different modifier per plane
emersion	there cannot be an ordered list of modifiers, because the "best" modifier depends on many parameters (e.g. buffer size)
emersion	only the driver knows which modifier would be the best one, and can only decide at allocation time (where it knows about the size, format, usage, etc)
emersion	some rules are explained here https://github.com/intel/libva/pull/505#issuecomment-814778614
emersion	i'm supposed to write kernel docs for all of this, but haven't got the chance to do so yet
columbarius	is there another example where two programs share a dmabuf and have to negotiate format and modifier?
emersion	many: wayland compositors and clients, the kernel and the compositor, etc
emersion	the consumer sends a list of acceptable (format, modifiers) pairs, then the producer allocates a buffer by intersecting pairs from multiple consumers, picking a format, and providing the list of modifiers for that format to gbm/vaapi/vulkan/?
columbarius	thanks for the explanation. may i paste the relevant bits to the pipewire issue?
columbarius	so the flow would be: obs want's to capture the screen. Asks egl for (format, modifier) pairs, sends them via pipewire to xdpw. xdpw takes the list, queries wlroots. wlroots will return the format of the buffers it uses internally. xdpw queries gbm with that format to get modifiers, intersect the modifiers with those from pipewire and will create a dmabuf. This dmabuf is handed over to wlroots to copy the
columbarius	screen to it. After it returned xdpw will send it via pipewire to obs and has to declare the used modifier via pipewire params?
columbarius	wouldn't wlr-screencopy-unstable-v1 need an extension to send a modifier list in the linux_dmabuf event?
emersion	right now any (format, modifier) pair supported by the linux-dmabuf protocol is also supported by screencopy
emersion	> xdpw queries gbm with that format to get modifiers
emersion	there's no such query, but also no need for it:
emersion	xdpw queries (format, modifier) pairs from the compositor and from the consumer, then picks the format indicated by screencopy, then passes the modifier list to gbm at alloc time (gbm_bo_create)
emersion	yes, xdpw needs to send the whole dmabuf parameters (FDs, offsets, modifier, format, width, height, etc) to the consumer
columbarius	all modifiers by linux-dmabuf supported, will this still be true for pixman or vulkan renderer? or custom renderers?
emersion	pixman doesn't do dmabufs
emersion	in theory, a driver could support texturing from a (format, modifier), but not support rendering to it
emersion	but i'm not aware of such a case, yet
columbarius	so we have two requirements for pipewire (as long as pipewire doesn't want o allocate dmabufs):
emersion	feel free to copy-paste yeah
columbarius	* allow a way to correlate buffertype with the format
columbarius	* allow to send a list of (format, modifier) pairs from the consumer to the producer
emersion	yeah sounds about right