Commit c71db353 authored by jasker5183's avatar jasker5183 Committed by Wim Taymans
Browse files

Documentation Updates

parent 8afe5fe0
Pipeline #580745 passed with stages
in 2 minutes and 49 seconds
/** \page page_dma_buf DMA-BUF sharing
/** \page page_dma_buf DMA-BUF Sharing
PipeWire supports sharing Direct Memory Access buffers (DMA-BUFs) between
clients via the `SPA_DATA_DmaBuf` data type. However properly negotiating
DMA-BUF support on both the producer and the consumer side require following
a specific procedure. This page describes said procedure by using events and
methods from the filter or stream API.
Note: This article focuses mostly on DMA-BUF sharing from arbitrary devices,
like discrete GPUs. For using DMA-BUFs created by v4l2 please refer to the
corresponding paragraph.
# Capability negotiations
# Capability Negotiations
The capability negotiation for DMA-BUFs is complicated by the fact, that a
usable and preferred optimal modifier for a given format can only be
......@@ -40,7 +41,7 @@ property.
To prioritise DMA-BUFs place those `SPA_PARAM_EnumFormat` containing modifiers
first, when emitting them to PipeWire.
## param_changed hook
## param_changed Hook
When the `param_changed` hook is called for a `SPA_PARAM_Format` the client
has to parse the `spa_pod` directly. Use
......@@ -57,15 +58,15 @@ format modifier pair. The producer is also responsible to check if all clients
announce sufficient capabilities or fallback to shared memory buffers when
possible.
### For consumers
### For Consumers
Use `spa_format_video_raw_parse` to get the format and modifier.
### For producers
### For Producers
Producers have to handle two cases when it comes to modifiers wrt. to the
previous announced capabilities: Using only the modifier-less API, only the
modifier aware one, or supporting both.
modifier-aware one, or supporting both.
- modifier-less:
In this case only the modifier `DRM_FORMAT_MOD_INVALID` was announced with
......@@ -77,12 +78,12 @@ modifier aware one, or supporting both.
In this case a list with all supported modifiers will be returned in the format.
(using `DRM_FORMAT_MOD_INVALID` as the token for the modifier-less API).
On the `param_changed` event check if the modifier key is present and has the flag
`SPA_POD_PROP_FLAG_DONT_FIXATE`. attached to it. In this case extract all modifiers
`SPA_POD_PROP_FLAG_DONT_FIXATE` attached to it. In this case extract all modifiers
from the list and do a test allocation with your allocator to choose the preferred
modifier. Fixate on that `EnumFormat` by announcing a `SPA_PARAM_EnumFormat` with
only one modifier in the `SPA_CHOICE_Enum` and without the
`SPA_POD_PROP_FLAG_DONT_FIXATE` flag, followed by the previous announced
`EnumFormat`s. This will retrigger the `param_changed` event with an
`EnumFormat`. This will retrigger the `param_changed` event with an
`SPA_PARAM_Format` as described below.
If the `SPA_PARAM_Format` contains a modifier key, without the flag
`SPA_POD_PROP_FLAG_DONT_FIXATE`, it should only contain one value in the
......@@ -96,32 +97,32 @@ pass them all to the graphics API. If the allocation fails and the list of
possible modifiers contains `DRM_FORMAT_MOD_INVALID`, fall back to allocating
without an explicit modifier if the graphics API allows it.
## add_buffer hook
## add_buffer Hook
This is relevant for producers.
Allocate a DMA-BUF only using the negotiated format and modifier.
## on_event hook
## on_event Hook
This is relevant for consumers.
Check the type of the dequeued buffer. If its `SPA_DATA_MemFd` or
`SPA_DATA_MemPtr` use the fallback SHM import mechanism.
If it's `SPA_DATA_DmaBuf`:
Get the DMA-BUF FDs (the plane count is encoded in the `n_datas` variable of the
If it's `SPA_DATA_DmaBuf`
get the DMA-BUF FDs (the plane count is encoded in the `n_datas` variable of the
`spa_buffer` struct) and import them with the graphics API.
Note: Some graphics APIs have separated functions for the modifier-less case
(`DRM_FORMAT_MOD_INVALID`) or are omitting the modifier, since it might be used
for error handling.
## Example programs
## Example Programs
- \ref video-src-fixate.c "": \snippet{doc} video-src-fixate.c title
- \ref video-play-fixate.c "": \snippet{doc} video-play-fixate.c title
# DMA-BUF mapping warning
# DMA-BUF Mapping Warning
It's important to make sure all consumers of the PipeWire stream are prepared
to deal with DMA-BUFs. Most DMA-BUFs cannot be treated like shared memory in general
......
/** \mainpage PipeWire
PipeWire is low-level multimedia framework that provides:
- Graph based processing
- Support for out-of-process processing graphs with minimal overhead
- Flexible and extensible media format negotiation and buffer allocation
- Hard real-time capable plugins
- Very low-latency for both audio and video processing
- Graph based processing.
- Support for out-of-process processing graphs with minimal overhead.
- Flexible and extensible media format negotiation and buffer allocation.
- Hard real-time capable plugins.
- Very low-latency for both audio and video processing.
See \ref page_overview for an overview of PipeWire and \ref page_design
for the design principles guiding PipeWire.
### Components
# Components
PipeWire ships with the following components:
- a \ref page_daemon that implements the IPC and graph processing
- an example \ref page_session_manager that manages objects in the \ref page_daemon
- a set of \ref page_tools to introspect and use the \ref page_daemon
- a \ref page_library to develop PipeWire applications and plugins (\ref
- A \ref page_daemon that implements the IPC and graph processing.
- An example \ref page_session_manager that manages objects in the \ref page_daemon.
- A set of \ref page_tools to introspect and use the \ref page_daemon.
- A \ref page_library to develop PipeWire applications and plugins (\ref
page_tutorial "tutorial").
- the \ref page_spa used by both the \ref page_daemon and in the \ref
- The \ref page_spa used by both the \ref page_daemon and in the \ref
page_library.
### API Documentation
# API Documentation
See \ref page_api
See \ref page_api.
### More Documentation
# More Documentation
See our [Wiki](https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/home) for
More information on how to configure and use PipeWire.
### Resources
# Resources
- [PipeWire and AGL](https://wiki.automotivelinux.org/_media/pipewire_agl_20181206.pdf)
- [LAC 2020 Paper](https://lac2020.sciencesconf.org/307881/document)
......@@ -39,4 +40,5 @@ More information on how to configure and use PipeWire.
- [PipeWire: The Linux audio/video bus (LWN)](https://lwn.net/Articles/847412)
- [PipeWire Wikipedia](https://en.wikipedia.org/wiki/PipeWire)
- [Bluetooth, PipeWire and Whatsapp calls](https://gjhenrique.com/pipewire.html)
*/
/** \page page_overview Overview
PipeWire is a new low-level multimedia framework designed from scratch that
aims to provide
aims to provide:
- graph based processing
- support for out-of-process processing graphs with minimal overhead
- flexible and extensible media format negotiation and buffer allocation
- Hard real-time capable plugins
- achieve very low-latency for both audio and video processing
- Graph based processing.
- Support for out-of-process processing graphs with minimal overhead.
- Flexible and extensible media format negotiation and buffer allocation.
- Hard real-time capable plugins.
- Achieve very low-latency for both audio and video processing.
The framework is used to build a modular daemon that can be configured to:
- be a low-latency audio server with features like pulseaudio and/or jack
- a video capture server that can manage hardware video capture devices and
provide access to them
- a central hub where video can be made available for other applications
- Be a low-latency audio server with features like PulseAudio and/or JACK.
- A video capture server that can manage hardware video capture devices and
provide access to them.
- A central hub where video can be made available for other applications
such as the gnome-shell screencast API.
## Motivation
# Motivation
Linux has no unified framework for exchanging multimedia content between
applications or even devices. In most cases, developers realized that
......@@ -29,14 +29,14 @@ a user-space daemon is needed to make this possible:
- For video capture, we usually go directly to the hardware devices, with
all security implications and inflexible routing that this brings.
- For consumer audio, we use PulseAudio to manage and mix multiple streams
from clients
from clients.
- For Pro audio, we use JACK to manage the graph of nodes.
None of these solutions (except perhaps to some extent Wayland), however,
None of these solutions (except perhaps to some extent Wayland) however
were designed to support the security features that are required when
dealing with flatpaks or other containerized applications. PipeWire
aims to solve this problem and provides a unified framework to run both
consumer and Pro audio as well as video capture and processing in a
consumer and pro audio as well as video capture and processing in a
secure way.
*/
......@@ -14,46 +14,47 @@ Permissions include `R` (read), `W` (write), `X` (execute) and `M` (metadata).
Some of those methods will only query state, others will modify the object.
As said above, modifying the object through one of these methods requires
the `W` permission.
- `M`: An object with M permission can be used as the subject in metadata.
- `M`: An object with `M` permission can be used as the subject in metadata.
Clients with all permissions set are referred to as "ALL" in the
documentation.
# Use cases
### New clients need their permissions configured
# Use Cases
## New Clients Need Their Permissions Configured
A new client is not allowed to communicate with the PipeWire daemon until
it has been configured with permissions.
### Flatpaks can't modify other stream/device volumes
## Flatpaks Can't Modify Other Stream/Device Volumes
An application running as Flatpak should not be able to modify the state of
certain objects. Permissions of the relevant PipeWire objects should not have
the `W` permission to avoid this.
### Flatpaks can't move other streams to different devices
## Flatpaks Can't Move Other Streams To Different Devices
Streams are moved to another device by setting the "target.node" metadata
on the node id. By not setting the `M` bit on the other objects, this can be
Streams are moved to another device by setting the `target.node` metadata
on the node ID. By not setting the `M` bit on the other objects, this can be
avoided.
### Application should be restricted in what they can see
## Application Should Be Restricted In What They Can See
In general, applications should only be able to see the objects that they are
allowed to see. For example, a web browser that was given access to a camera
should not be able to see (and thus receive input data from) audio devices.
### "Manager" applications require full access
## "Manager" Applications Require Full Access
Some applications require full access to the PipeWire graph, including
moving streams between nodes (by setting metadata) and modifying properties
(e.g. volume). These applications must work even when running as Flatpak.
(eg. volume). These applications must work even when running as Flatpak.
# Design
## The PipeWire daemon
## The PipeWire Daemon
Immediately after a new client connects to the PipeWire daemon and updates
its properties, the client will be registered and made visible to other
......@@ -71,8 +72,7 @@ A suspended client can only resume processing after some other client
sets the core permissions to `R`. This other client is usually a session
manager, see e.g. \ref page_session_manager.
## The PipeWire access module
## The PipeWire Access Module
The \ref page_module_access hooks into the `check_access` event that is
emitted when a new client is registered. The module checks the permissions of
......@@ -80,23 +80,23 @@ the client and stores those in the \ref PW_KEY_ACCESS
property on the client object. If this property is already set, the access
module does nothing.
If the property is not set, it will go through a set of checks to determine
the permissions for a client, see the \ref page_module_access documentation
If the property is not set it will go through a set of checks to determine
the permissions for a client. See the \ref page_module_access documentation
for details, particularly on the values documented below. Depending on the
value of the \ref PW_KEY_ACCESS property one the following happens:
- `"allowed"`, `"unrestricted"`: ALL permissions are set on the core
object and the client will be able to resume.
- `"restricted"`, `"flatpak"`, `"$access.force"`: no permissions are set on
- `"restricted"`, `"flatpak"`, `"$access.force"`: No permissions are set on
the core object and the client will be suspended.
- `"rejected"`: an error is sent to the client and the client is
- `"rejected"`: An error is sent to the client and the client is
suspended.
As detailed above, the client may be suspended. In that case the session
manager or another client is required to configure permissions on the object
for it to resume.
## The session manager
## The Session Manager
The session manager listens for new clients to appear. It will use the
\ref PW_KEY_ACCESS property to determine what to do.
......@@ -117,7 +117,7 @@ configure ALL permissions on the client. Possible checks include
permission store checks or ask the user if the application is allowed
full access.
Manager applications (i.e. applications that need to modify the graph) will
Manager applications (ie. applications that need to modify the graph) will
set the \ref PW_KEY_MEDIA_CATEGORY property in the client object to "Manager".
For details on the pipewire-media-session implementation of access control,
......
......@@ -2,19 +2,19 @@
This document explains how Audio is implemented.
# Use cases
# Use Cases
## Audio devices are made available as processing nodes/ports
## Audio Devices Are Made Available As Processing Nodes/Ports
Applications need to be able to see a port for each stream of an
Audio device.
audio device.
### Audio devices can be plugged and unplugged
## Audio Devices Can Be Plugged and Unplugged
When devices are plugged and unplugged the associated nodes/ports
need to be created and removed.
### Audio port in canonical format
## Audio Port In Canonical Format
It must be possible to make individual audio channels available
as a single mono stream with a fixed format and samplerate.
......@@ -22,7 +22,7 @@ as a single mono stream with a fixed format and samplerate.
This makes it possible to link any of the audio ports together
without doing conversions.
### Applications can connect to audio devices
## Applications Can Connect To Audio Devices
Applications can create ports that can connect to the audio ports
so that data can be provided to or consumed from them.
......@@ -30,19 +30,19 @@ so that data can be provided to or consumed from them.
It should be possible to automatically connect an application to
a sink/source when it requests this.
### Default audio sink and sources
## Default Audio Sink and Sources
It should be possible to mark a source or sink as the default source
and sink so that applications are routed to them by default.
It should be possible to change the default audio sink/source.
### Application should be able to move between sinks/sources
## Application Should Be Able To Move Between Sinks/Sources
It should be possible to move an application from one device to
another dynamically.
### Exclusive access
## Exclusive Access
Application should be able to connect to a device in exclusive mode.
This allows the application to negotiate a specific format with the
......@@ -62,13 +62,13 @@ Audio devices are implemented with an \ref spa_device "SPA Device" object.
This object is then responsible for controlling the \ref spa_node "SPA Nodes" that
provide the audio ports to interface with the device.
The Nodes operate on the native audio formats supported by the device.
The nodes operate on the native audio formats supported by the device.
This includes the sample rate as well as the number of channels and
the audio format.
## Audio Adapter
An SPA Node called the "adapter" is usually used with the SPA device Node as
An SPA Node called the "adapter" is usually used with the SPA device node as
the internal node.
The function of the adapter is to convert the device native format to
......@@ -80,14 +80,14 @@ as separate mono ports. This is called the DSP setup.
The audio adapter can also be configured in passthrough mode when it
will not do any conversions but simply pass through the port information
of the internal Node. This can be used to implement exclusive access.
of the internal node. This can be used to implement exclusive access.
Setup of the different configurations of the adapter can be done with
the PortConfig parameter.
## The session manager
## The Session Manager
The session manager is responsible for creating Nodes and Ports for
The session manager is responsible for creating nodes and ports for
the various audio devices. It will need to wrap them into an audio
adapter so that the specific configuration of the node can be
decided by the policy mode.
......@@ -96,33 +96,32 @@ The session manager should create name and description for the
devices and nodes.
The session manager is responsible for assigning priorities to the
Nodes. At least \ref PW_KEY_PRIORITY_SESSION and \ref PW_KEY_PRIORITY_DRIVER
nodes. At least \ref PW_KEY_PRIORITY_SESSION and \ref PW_KEY_PRIORITY_DRIVER
need to be set.
The session manager might need to work with other services to gain
exclusive access to the device (e.g. DBus).
exclusive access to the device (eg. DBus).
# Implementation
## pipewire-media-session (alsa-monitor)
## PipeWire Media Session (alsa-monitor)
PipeWire media session uses the \ref SPA_NAME_API_ALSA_ENUM_UDEV plugin
for enumerating the ALSA devices. For each device it does:
- Try to acquire the DBus device reservation object to gain exclusive
access to the device.
- Create an SPA Device instance for the device and monitor this device instance.
- For each Node created by the device, create an adapter with
- Create an SPA device instance for the device and monitor this device instance.
- For each node created by the device, create an adapter with
an ALSA PCM node in the context of the PipeWire daemon.
The session manager will also create suitable names and descriptions
for the Devices and Nodes that it creates as well as assign session
for the devices and nodes that it creates as well as assign session
and driver priorities.
The session manager has the option to add extra properties on the
Devices and Node that it creates to control their behavior. This
devices and nodes that it creates to control their behavior. This
is implemented with match rules.
*/
......@@ -7,7 +7,7 @@ Typically general, users run one PipeWire daemon that listens for incoming
connections and manages devices. Clients (including the \ref
page_session_manager) are separate processes that talk to the daemon using the
PipeWire socket (default: `$XDG_RUNTIME_DIR/pipewire-0`). This approach
provides provides address-space separation between the privileged daemon and
provides address-space separation between the privileged daemon and
non-privileged clients.
\dot
......@@ -52,7 +52,7 @@ As shown above, the protocol is handled by the \ref
page_module_protocol_native. From PipeWire's point-of-view this module is just
another module.
\section sec_config Configuration Files
# Configuration Files
On startup, the daemon reads a configuration file to configure itself.
It executes a series of commands listed in the config file. The lookup order
......@@ -75,14 +75,14 @@ They are applied to the global configuration file. Properties are overwritten
and array elements are appended. This makes it possible to make small custom customizations
or additions to the main configuration file.
The environment variables `PIPEWIRE_CONFIG_DIR`, `PIPEWIRE_CONFIG_PREFIX`
and `PIPEWIRE_CONFIG_NAME` can be used to specify an alternative config
directory, subdirectory and filename, respectively.
The environment variables `PIPEWIRE_CONFIG_DIR`, `PIPEWIRE_CONFIG_PREFIX`,
and `PIPEWIRE_CONFIG_NAME`. Can be used to specify an alternative configuration
directory, subdirectory, and filename respectively.
\subsection sec_config_format Configuration File Format
## Configuration File Format
PipeWire's configuration file format is JSON. In addition to true JSON,
PipeWire also understands a more compact JSON representation where
PipeWire's configuration file format is JSON. In addition to true JSON
PipeWire also understands a more compact JSON representation. Where
`"` can be omitted around strings, no trailing commas are required and
`:` or `=` can be used to separate object keys from their values.
Also, `#` can be used to start a comment until the end of the line.
......@@ -117,63 +117,60 @@ Allowed configuration file sections are:
- **context.properties** (dictionary): These properties configure the
pipewire instance.
- **context.spa-libs** (dictionary): Maps plugin features with globs to a
spa library.
- **context.modules** (array): Each entry in the array is a dictionary with
the name of the module to load, including optional args and flags. Most
modules support being loaded multiple times.
- **context.objects** (array): Each entry in the array is a dictionary con‐
taining the factory to create an object from and optional extra argu‐
ments specific to that factory.
- **context.exec** (array): Each entry in the array is dictionary containing
the path of a program to execute on startup and optional args. This ar‐
ray usually contains an entry to start the session manager.
\section sec_logging Logging
# Logging
The `PIPEWIRE_DEBUG` environment variable can be used to enable
more debugging. This variable supports one of two formats:
- `PIPEWIRE_DEBUG=<level>` where `<level>` is either a numerical log level or it's
- `PIPEWIRE_DEBUG=<level>` where `<level>` is either a numerical log level or its
respective key, see below.
- `PIPEWIRE_DEBUG=<glob1>:<level1>,<glob2>:<level2>,...` where the globs are
shell-globs to match on log topics and the levels are the respective
log level to set for that topic. Globs are applied in-order and a matching
shell globs to match on log topics and the levels are the respective
log level to set for that topic. Globs are applied in order and a matching
glob overrides an earlier glob for that category. For example,
`PIPEWIRE_DEBUG=*:E,mod.*:D,mod.foo:X" enables global error messages,
`PIPEWIRE_DEBUG=*:E,mod.*:D,mod.foo:X` enables global error messages,
debugging on all modules but no messages on the foo module.
- `<level>` specifies the log level:
+ `X` or `0`: no logging is enabled
+ `E` or `1`: Error logging is enabled
+ `W` or `2`: Warnings are enabled
+ `I` or `3`: Informational messages are enabled
+ `D` or `4`: Debug messages are enabled
+ `X` or `0`: No logging is enabled.
+ `E` or `1`: Error logging is enabled.
+ `W` or `2`: Warnings are enabled.
+ `I` or `3`: Informational messages are enabled.
+ `D` or `4`: Debug messages are enabled.
+ `T` or `5`: Trace messages are enabled. These messages can be logged
from the realtime threads.
PipeWire uses a "category.topic" naming scheme, with the following categories:
- `pw.*`: pipewire-internal topics
- `mod.*`: module topics, for example `mod.foo` would usually refer to the
"foo" module
- `ms.*`: media session topics
- `ms.mod.*`: media session modules, for example `ms.foo` would usually refer
to the "media-session-foo" module
- `conn.*`: connection-specific topics such as printing raw messages sent over
PipeWire uses a `category.topic` naming scheme, with the following categories:
- `pw.*`: PipeWire internal topics.
- `mod.*`: Module topics, for example `mod.foo` would usually refer to the
`foo` module.
- `ms.*`: Media session topics.
- `ms.mod.*`: Media session modules, for example `ms.foo` would usually refer
to the `media-session-foo` module.
- `conn.*`: Connection specific topics such as printing raw messages sent over
a communication socket. These are in a separate namespace as they are
usually vastly more verbose than the normal debugging topics.
This namespace must be explicitly enabled with a `conn.<glob>` glob.
The behavior of the logging can be further controlled with the following
environment variables:
- `PIPEWIRE_LOG_SYSTEMD=false`: disable logging to the systemd journal
- `PIPEWIRE_LOG=<filename>`: redirect the log to the given filename
- `PIPEWIRE_LOG_LINE=false`: don't log filename, function, and source code line
- `PIPEWIRE_LOG_SYSTEMD=false`: Disable logging to the systemd journal.
- `PIPEWIRE_LOG=<filename>`: Redirect the log to the given filename.
- `PIPEWIRE_LOG_LINE=false`: Don't log filename, function, and source code line.
*/
......@@ -6,13 +6,13 @@ PipeWire is a media server that can run graphs of multimedia nodes.
Nodes can run inside the server process or in separate processes,
communicating with the server.
PipeWire was designed to
PipeWire was designed to:
- be efficient for raw video using fd passing and audio with
shared ringbuffers
- be able to provide/consume/process media from any process
- provide policy to restrict access to devices and streams
- be easily extensible
- Be efficient for raw video using fd passing and audio with
shared ringbuffers.
- Be able to provide/consume/process media from any process.
- Provide policy to restrict access to devices and streams.
- Be easily extensible.
Although an initial goal, the design is not limited to raw video
only and should be able to handle compressed video and other
......@@ -23,36 +23,30 @@ SPA is designed for low-latency and efficient processing of any multimedia
format. SPA also provides a number of helper utilities that are not available
in the standard C library.
Some of the application we intend to build
Some of the application we intend to build:
- v4l2 device provider: Provide controlled access to v4l2 devices
and share one device between multiple processes.
- v4l2 device provider: Provide controlled access to v4l2 devices
and share one device between multiple processes.
- gnome-shell video provider: GNOME Shell provides a node that
gives the contents of the frame buffer for screen sharing or
screen recording.
- Audio server: Mix and playback multiple audio streams. The design
is more like CRAS (Chromium audio server) than PulseAudio and with
the added benefit that processing can be arranged in a graph.
- Professional audio graph processing like JACK.
- Media playback backend.
- gnome-shell video provider: GNOME Shell provides a node that
gives the contents of the frame buffer for screen sharing or
screen recording.
- audio server: Mix and playback multiple audio streams. The design
is more like CRAS (Chromium audio server) than PulseAudio and with
the added benefit that processing can be arranged in a graph.
- Pro audio graph processing like JACK.
- Media playback backend
Protocol
--------
# Protocol
The native protocol and object model is similar to
[Wayland](https://wayland.freedesktop.org) but with custom
serialization/deserialization of messages. This is because the datastructures
serialization/deserialization of messages. This is because the data structures
in the messages are more complicated and not easily expressible in XML.
See \ref page_module_protocol_native for details.
Extensibility
-------------
# Extensibility
The functionality of the server is implemented and extended with modules and
extensions. Modules are server side bits of logic that hook into various
......@@ -64,16 +58,13 @@ Extensions are the client side version of the modules. Most extensions provide
both a client side and server side init function. New interfaces or new object
implementation can easily be added with modules/extensions.
Some of the extensions that can be written