General problems with dmabufs
The current negotiation of formats is not really suitable for the usual workflow with dmabufs.
Example dmabuf workflow
The usual flow between a producer and consumer of dmabufs is as follows:
- The consumer queries its backend (gbm, vulkan, vaapi, etc.) for supported formats and modifiers.
- The consumer sends a list of (format, modifier) pairs to the producer.
- The producer takes the list (or the intersection of lists from multiple consumers), chooses a format and sends a list of modifiers corresponding to the given format to the buffer allocator (gbm, vulkan, ...) along with the size of the buffer.
- The allocator will choose the "best" modifier from the list taking the buffer size into consideration and returns a buffer.
- The producer renders to this buffer and sends it to the consumer along with the selected format and modifier.
- The consumer imports the buffer to its backend with the given metadata.
Problems with the current implementation
Problems regarding a dmabuf only negotiation (I)
Currently both producer and consumer send their capabilities regarding the videostream (dimensions, formats, modifiers, etc.) to pipewire. Pipewire chooses a combination which should work for both and returns the choosen result. This has the following problems:
- Formats and modifers are not paired. As such pipewire can return format, modifier combinations, which are invalid, see modifier test.
- Even if we pair formats and modifiers in
SPA_PARAM_EnumFormat(for example replacing the
SPA_PARAM_EnumFormatwith a list of pairs.
SPA_PARAM_Formatshould stay the selected modifier as a long), the process of choosing the used Formats will still happen inside pipewire. But to select a good Format requires knowledge only available from the driver/hardware. In that case either pipewire has to add an allocator as a dependency (egl, vulkan, gbm, etc.) and queries it for a combination (and could allocate the buffer at this step right away) or it has to move this decision to an instance, which has this information. On options would be to select a format and send its modifier list to the producer which should do the allocation anyway. The producer can then allocate buffers with this list and has to update pipewire with the used modifier, so pipewire can set the correct
Problems regarding dmabufs and shm buffers (II)
While formats are handled with
SPA_PARAM_Format, the buffertype is handled by
SPA_PARAM_Buffers. This Parameters are evaluated and merged independent from another. This creates the following problem:
- Pipewire clients might support some formats both as dmabuf and shm, while other formats are only viable with one buffer type. It is not possible to specify this beforehand. This results in clients negotiating a format only to find no viable shared buffertype.
Proposed requirements for dmabuf usage
If pipewire doesn't want to allocate dmabufs the following things would be needed:
- A way to correlate buffertype and format
- The node, which is responsible for creating the buffers, needs to receive a list of (format, modifier) pairs to choose the used modifier and to emit it back.
SPA_PARAM_EnumFormatwith a list of (format, modifier) pairs. Formats only supported by shm can be marked as (format, SPA_DATA_Invalid). Clients not specifying
SPA_FORMAT_modiferwould be treated as shm only.
- Add an additional object to
SPA_PARAM_EnumFormatfor each buffertype (MemPtr and Memfd use the same underlying buffertype and would be identical) to specify format, modifiers an other differences there.
Discussion about modifiers in libva: https://github.com/intel/libva/pull/505
irc discussion with emersion
columbarius about pipewire: If i didn't have a misconception on how to use pipewire, this should demonstrate the current shortcommings:
columbarius linked it to this issue https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/1055
emersion oh nice!
emersion that's a pretty big issue indeed
emersion oh there's also some shm vs. dmabuf format mixing?
emersion great :/
columbarius there is no "real" mixing, because they are handled separately, so first you negotiate the format, then the buffer. If the buffer doesn't match the format .... probably try again and remove some possibilities.
columbarius So pipewire doesn't keep track of format buffertype combinations, you have to
columbarius *the pipewire client has to
columbarius does gbm offer me a list of "usefull" format/modifier pairs? because pipewire probably would need a value for any modifier, if a client just imports buffers with gbm and doesn't care / doesn't know better
emersion why do you want to use gbm?
emersion ah, because screencopy wants the client to allocate a dmabuf, and pipewire wants the producer to allocate the dmabuf?
emersion gbm doesn't give you lists for formats/modifiers
emersion even if it did, you need to pick one supported by the compositor
emersion i'd suggest using the linux-dmabuf protocol's "modifier" event for that
columbarius not only. I was more thinking, if there is a way to use dmabufs without an instance, which gives you format/modifier pairs. You probably don't want to give pipewire a list with every modifier for each format
emersion wlroots won't let you pick the format
emersion the compositor dictates the format
columbarius so you probably just want to announce, you can handle almost all
columbarius in this case you would just announce the formats/modifiers from the compositor
columbarius i was thinking about pipewire clients like obs
columbarius if they use vaapi, they will recieve this list in the future, but if they wan't to do sth. with gbm on their own ...
columbarius is this a possible usecase, without increasing the format/modifier list to all possible combinations?
emersion i'm not sure i'm following?
columbarius if i write a consumer, which (for some reason and with good performance) can handle all possible formats and modifier pairs my hardware supports (vendor specific, etc.) via mapping by gbm or some other mechanism. Then I probably can assume, that any producer will also only send format/modifier pairs, which are supported by the hardware and I have two options: Send all combinations of pairs/modifiers, which
columbarius the hardware supports (crawling the header file of modifiers), and have a really long list, or use a specific "meta" modifier, which signals for every format I support, every modifier will work just fine.
emersion you cannot assume gbm supports all modifiers
columbarius but can you assume, that all software working with modifiers on one hardware has the same restrictions wrt. modifiers?
columbarius hmmm ....
emersion for instance, on amdgpu, kms can only do a subset of modifiers, gl/gbm can do a bigger subset, vaapi/vulkan can do a smaller subset
emersion also not all software is released/upgraded at the same time
columbarius Then the problem is, how to get a usefull list of pairs for every software, which want's to use dmabufs
emersion one component could add support for a new modifier, when other components still don't support it
emersion it would probably be sensible for the pipewire consumer to send (format, modifier) pairs it can deal with
columbarius yes. I just wanted to make sure, if/that any consumer has access to this information
emersion some parts are still missing
emersion gl/vulkan def support it
emersion vaapi is missing it
emersion (but that's vaapi's issue)
columbarius If that's the case, we probably only need a fast merging algorythm for ling lists ^^'
emersion the negociation only happens once, so no need to optimize it
emersion it's not a hot codepath
columbarius But it might be a lot of data? (don't have any clue on how many combinations are possible, but uint64_t has a lot of space))
emersion if you have 100 formats, each with 50 possible modifiers, it's 40KiB
emersion (100*4 + 100*50*8)
emersion plus maybe a few more bytes for lengths and whatnot
emersion wayland just sends the whole thing over the wire, fwiw
emersion with today's drivers, expect maybe 20 formats, with 5 modifiers each
emersion see for yourself with wayland-info i guess
columbarius I think the vaapi changes are in, before pipewire wants to break its api, but i can't think of a simple solution, which doesn't require breakage.
columbarius nit: shouldn't it be (100*50*(4+8)) = 60KiB since they are in pairs?
columbarius ok then it is fine. was just a knowledge gap, that's why i could not evaluate it. thankst
emersion maybe introduce a DMABUF2 buffer type, and let everyone know that DMABUF is broken
emersion yeah, it kind of depends how you send it
emersion fwiw, the kernel has an optimized packed struct to store format/modifier pairs
emersion see drm_format_modifier_blob
columbarius thing is FORMAT and BUFFER are two separate Objects
columbarius we either have to pair those, or create a Format and a FORMAT_DMABUF
columbarius ahh nice
columbarius * separate Objects without connection/reference
emersion mind, the packed struct is annoying to decypher
emersion it's using a 64-bit bitfield with a sliding window
columbarius pipewire assumes, that all formats are equal, just with differend kind of "packing"
emersion there is a libdrm MR to add a helper to decypher it
emersion sorry, i don't remember anything about pipewire's APIs :P
columbarius nice, when you create sth. that the next person looking at it writes a helper to not get confused xD
emersion yea :S
columbarius I think it would be nice, to try to specify how dmabuf and modifiers work in a usual mediapipeline
columbarius so should producer and consumer send pipewire all supported formats in a somehow prioritised list and pipewire will guess the best one, or would it better for pipewire to only intersect those lists and return the result to the producer in a way, that the producer can choose the "best" format/modifier combination? Or should the consumer do this.
columbarius additional: how does this work with multiple planes having different modifiers?
columbarius So probably a specific question for you: When using multiple planes, do they share a format and only can have different modifiers, or is it possible to have an arbitrary formats with an arbitrary modifier per plane?
emersion it's not possible to have a different modifier per plane
emersion there cannot be an ordered list of modifiers, because the "best" modifier depends on many parameters (e.g. buffer size)
emersion only the driver knows which modifier would be the best one, and can only decide at allocation time (where it knows about the size, format, usage, etc)
emersion some rules are explained here https://github.com/intel/libva/pull/505#issuecomment-814778614
emersion i'm supposed to write kernel docs for all of this, but haven't got the chance to do so yet
columbarius is there another example where two programs share a dmabuf and have to negotiate format and modifier?
emersion many: wayland compositors and clients, the kernel and the compositor, etc
emersion the consumer sends a list of acceptable (format, modifiers) pairs, then the producer allocates a buffer by intersecting pairs from multiple consumers, picking a format, and providing the list of modifiers for that format to gbm/vaapi/vulkan/?
columbarius thanks for the explanation. may i paste the relevant bits to the pipewire issue?
columbarius so the flow would be: obs want's to capture the screen. Asks egl for (format, modifier) pairs, sends them via pipewire to xdpw. xdpw takes the list, queries wlroots. wlroots will return the format of the buffers it uses internally. xdpw queries gbm with that format to get modifiers, intersect the modifiers with those from pipewire and will create a dmabuf. This dmabuf is handed over to wlroots to copy the
columbarius screen to it. After it returned xdpw will send it via pipewire to obs and has to declare the used modifier via pipewire params?
columbarius wouldn't wlr-screencopy-unstable-v1 need an extension to send a modifier list in the linux_dmabuf event?
emersion right now any (format, modifier) pair supported by the linux-dmabuf protocol is also supported by screencopy
emersion > xdpw queries gbm with that format to get modifiers
emersion there's no such query, but also no need for it:
emersion xdpw queries (format, modifier) pairs from the compositor and from the consumer, then picks the format indicated by screencopy, then passes the modifier list to gbm at alloc time (gbm_bo_create)
emersion yes, xdpw needs to send the whole dmabuf parameters (FDs, offsets, modifier, format, width, height, etc) to the consumer
columbarius all modifiers by linux-dmabuf supported, will this still be true for pixman or vulkan renderer? or custom renderers?
emersion pixman doesn't do dmabufs
emersion in theory, a driver could support texturing from a (format, modifier), but not support rendering to it
emersion but i'm not aware of such a case, yet
columbarius so we have two requirements for pipewire (as long as pipewire doesn't want o allocate dmabufs):
emersion feel free to copy-paste yeah
columbarius * allow a way to correlate buffertype with the format
columbarius * allow to send a list of (format, modifier) pairs from the consumer to the producer
emersion yeah sounds about right