Commit cfdaa607 authored by Thibault Saunier's avatar Thibault Saunier 🌵

Merging gst-build

parents 17701b81 4bf0504e
*build*/
cscope.in.out
cscope.out
cscope.po.out
gst-devtools
gst-editing-services
gst-examples
gst-integration-testsuites
gst-libav
gst-omx
gst-plugins-bad
gst-plugins-base
gst-plugins-good
gst-plugins-ugly
gst-python
gst-rtsp-server
gstreamer
gstreamer-sharp
gstreamer-vaapi
libnice
__pycache__
meson/
subprojects/packagecache/
subprojects/gst-integration-testsuites
subprojects/gst-devtools
subprojects/gst-docs
subprojects/gst-editing-services
subprojects/gst-examples
subprojects/gst-libav
subprojects/gst-omx
subprojects/gst-plugins-bad
subprojects/gst-plugins-base
subprojects/gst-plugins-good
subprojects/gst-plugins-ugly
subprojects/gst-python
subprojects/gst-rtsp-server
subprojects/gstreamer
subprojects/gstreamer-sharp
subprojects/gstreamer-vaapi
subprojects/glib
subprojects/gl-headers
subprojects/bindinator
subprojects/dssim
subprojects/gtk-sharp
subprojects/libffi
subprojects/libnice
subprojects/libunwind
subprojects/proxy-libintl
subprojects/zlib
subprojects/openh264
subprojects/pygobject
subprojects/pycairo
subprojects/json-glib
subprojects/orc
subprojects/libsoup
subprojects/FFmpeg
subprojects/x264
prefix/
pygobject
include: 'https://gitlab.freedesktop.org/gstreamer/gst-ci/raw/master/gitlab/ci_template.yml'
This diff is collapsed.
# gst-build
GStreamer [meson](http://mesonbuild.com/) based repositories aggregrator.
Check out this module and run meson on it, and it will git clone the other
GStreamer modules as [meson subprojects](http://mesonbuild.com/Subprojects.html)
and build everything in one go. Once that is done you can switch into an
development environment which allows you to easily develop and test the latest
version of GStreamer without the need to install anything or touch an existing
GStreamer system installation.
## Getting started
### Install git and python 3.5+
If you're on Linux, you probably already have these. On macOS, you can use the
[official Python installer](https://www.python.org/downloads/mac-osx/).
You can find [instructions for Windows below](#windows-prerequisites-setup).
### Install meson and ninja
Meson 0.48 or newer is required.
On Linux and macOS you can get meson through your package manager or using:
$ pip3 install --user meson
This will install meson into `~/.local/bin` which may or may not be included
automatically in your PATH by default.
You should get `ninja` using your package manager or download the [official
release](https://github.com/ninja-build/ninja/releases) and put the `ninja`
binary in your PATH.
You can find [instructions for Windows below](#windows-prerequisites-setup).
### Build GStreamer and its modules
You can get all GStreamer built running:
```
meson build/
ninja -C build/
```
This will automatically create the `build` directory and build everything
inside it.
NOTE: On Windows, you *must* run this from inside the Visual Studio command
prompt of the appropriate architecture and version.
# Development environment
## Building the Qt5 QML plugin
If `qmake` is not in `PATH` and pkgconfig files are not available, you can
point the `QMAKE` env var to the Qt5 installation of your choosing before
running `meson` as shown above.
The plugin will be automatically enabled if possible, but you can ensure that
it is built by passing `-Dgst-plugins-good:qt5=enabled` to `meson`. This will
cause Meson to error out if the plugin could not be enabled. This also works
for all plugins in all GStreamer repositories.
## Development environment target
gst-build also contains a special `devenv` target that lets you enter an
development environment where you will be able to work on GStreamer
easily. You can get into that environment running:
```
ninja -C build/ devenv
```
If your operating system handles symlinks, built modules source code will be
available at the root of `gst-build/` for example GStreamer core will be in
`gstreamer/`. Otherwise they will be present in `subprojects/`. You can simply
hack in there and to rebuild you just need to rerun `ninja -C build/`.
NOTE: In the development environment, a fully usable prefix is also configured
in `gst-build/prefix` where you can install any extra dependency/project.
An external script can be run in development environment with:
```
./gst-env.py external_script.sh
```
## Update git subprojects
We added a special `update` target to update subprojects (it uses `git pull
--rebase` meaning you should always make sure the branches you work on are
following the right upstream branch, you can set it with `git branch
--set-upstream-to origin/master` if you are working on `gst-build` master
branch).
Update all GStreamer modules and rebuild:
```
ninja -C build/ update
```
Update all GStreamer modules without rebuilding:
```
ninja -C build/ git-update
```
## Custom subprojects
We also added a meson option, `custom_subprojects`, that allows the user
to provide a comma-separated list of subprojects that should be built
alongside the default ones.
To use it:
```
cd subprojects
git clone my_subproject
cd ../build
rm -rf * && meson .. -Dcustom_subprojects=my_subproject
ninja
```
## Run tests
You can easily run the test of all the components:
```
meson test -C build
```
To list all available tests:
```
meson test -C build --list
```
To run all the tests of a specific component:
```
meson test -C build --suite gst-plugins-base
```
Or to run a specific test file:
```
meson test -C build/ --suite gstreamer gst_gstbuffer
```
Run a specific test from a specific test file:
```
GST_CHECKS=test_subbuffer meson test -C build/ --suite gstreamer gst_gstbuffer
```
## Optional Installation
`gst-build` has been created primarily for [development usage](#development-environment-target),
but you can also install everything that is built into a predetermined prefix like so:
```
meson --prefix=/path/to/install/prefix build/
ninja -C build/
meson install -C build/
```
Note that the installed files have `RPATH` stripped, so you will need to set
`LD_LIBRARY_PATH`, `DYLD_LIBRARY_PATH`, or `PATH` as appropriate for your
platform for things to work.
## Checkout another branch using worktrees
If you need to have several versions of GStreamer coexisting (eg. `master` and `1.14`),
you can use the `checkout-branch-worktree` script provided by `gst-build`. It allows you
to create a new `gst-build` environment with new checkout of all the GStreamer modules as
[git worktrees](https://git-scm.com/docs/git-worktree).
For example to get a fresh checkout of `gst-1.14` from a `gst-build` in master **already
built** in a `build` directory you can simply run:
```
./checkout-branch-worktree ../gst-build-1.14 origin/1.14 -C build/
```
This will create a new ``gst-build-1.14`` folder at the same level of ``gst-build`` pointing to the given branch ie *1.14*
for all the subprojects ( gstreamer, gst-plugins-base etc.)
## Add information about GStreamer development environment in your prompt line
### Bash prompt
We automatically handle `bash` and set `$PS1` accordingly.
If the automatic `$PS1` override is not desired (maybe you have a fancy custom prompt), set the `$GST_BUILD_DISABLE_PS1_OVERRIDE` environment variable to `TRUE` and use `$GST_ENV` when setting the custom prompt, for example with a snippet like the following:
```bash
...
if [[ -n "${GST_ENV-}" ]];
then
PS1+="[ ${GST_ENV} ]"
fi
...
```
### Zsh prompt
In your `.zshrc`, you should add something like:
```
export PROMPT="$GST_ENV-$PROMPT"
```
### Fish prompt
In your `~/.config/fish/functions/fish_prompt.fish`, you should add something like this at the end of the fish_prompt function body:
```
if set -q GST_ENV
echo -n -s (set_color -b blue white) "(" (basename "$GST_ENV") ")" (set_color normal) " "
end
```
### Using powerline
In your powerline theme configuration file (by default in
`{POWERLINE INSTALLATION DIR}/config_files/themes/shell/default.json`)
you should add a new environment segment as follow:
```
{
"function": "powerline.segments.common.env.environment",
"args": { "variable": "GST_ENV" },
"priority": 50
},
```
## Windows Prerequisites Setup
On Windows, some of the components may require special care.
### Git for Windows
Use the [Git for Windows](https://gitforwindows.org/) installer. It will
install a `bash` prompt with basic shell utils and up-to-date git binaries.
During installation, when prompted about `PATH`, you should select the
following option:
![Select "Git from the command line and also from 3rd-party software"](/data/images/git-installer-PATH.png)
### Python 3.5+ on Windows
Use the [official Python installer](https://www.python.org/downloads/windows/).
You must ensure that Python is installed into `PATH`:
![Enable Add Python to PATH, then click Customize Installation](/data/images/py-installer-page1.png)
You may also want to customize the installation and install it into
a system-wide location such as `C:\PythonXY`, but this is not required.
### Ninja on Windows
The easiest way to install Ninja on Windows is with `pip3`, which will download
the compiled binary and place it into the `Scripts` directory inside your
Python installation:
```
pip3 install ninja
```
You can also download the [official release](https://github.com/ninja-build/ninja/releases)
and place it into `PATH`.
### Meson on Windows
**IMPORTANT**: Do not use the Meson MSI installer since it is experimental and known to not
work with `gst-build`.
You can use `pip3` to install Meson, same as Ninja above:
```
pip3 install meson
```
Note that Meson is written entirely in Python, so you can also run it as-is
from the [git repository](https://github.com/mesonbuild/meson/) if you want to
use the latest master branch for some reason.
### Setup a mingw/wine based development environment on linux
#### Install wine and mingw
##### On fedora x64
``` sh
sudo dnf install mingw64-gcc mingw64-gcc-c++ mingw64-pkg-config mingw64-winpthreads wine
```
FIXME: Figure out what needs to be installed on other distros
#### Get meson from git
This simplifies the process and allows us to use the cross files
defined in meson itself.
``` sh
git clone https://github.com/mesonbuild/meson.git
```
#### Build and install
```
BUILDDIR=$PWD/winebuild/
export WINEPREFIX=$BUILDDIR/wine-prefix/ && mkdir -p $WINEPREFIX
# Setting the prefix is mandatory as it is used to setup symlinks during uninstalled development
meson/meson.py $BUILDDIR --cross-file meson/cross/linux-mingw-w64-64bit.txt -Dgst-plugins-bad:vulkan=disabled -Dorc:gtk_doc=disabled --prefix=$BUILDDIR/wininstall/ -Djson-glib:gtk_doc=disabled
meson/meson.py install -C $BUILDDIR/
```
> __NOTE__: You should use `meson install -C $BUILDDIR` each time you make a change
> instead of the usual `ninja -C build` as the environment is not uninstalled.
#### The development environment
You can get into the development environment the usual way:
```
ninja -C $BUILDDIR/ devenv
```
After setting up [binfmt] to use wine for windows binaries,
you can run GStreamer tools under wine by running:
```
gst-launch-1.0.exe videotestsrc ! glimagesink
```
[binfmt]: http://man7.org/linux/man-pages/man5/binfmt.d.5.html
#!/bin/sh
cd `dirname $0`
case "$MSYSTEM" in
*MINGW*) PYTHON="py -3";;
*) PYTHON="python3";;
esac
$PYTHON gst-worktree.py add "$@"
$i=1
$ppid=(gwmi win32_process -Filter "processid='$pid'").parentprocessid
$pname=(Get-Process -id $ppid).Name
While($true) {
if($pname -eq "cmd" -Or $pname -eq "powershell") {
Write-Host ("{0}.exe" -f $pname)
Break
}
# 10 times iteration seems to be sufficient
if($i -gt 10) {
Break
}
# not found yet, find grand parant
$ppid=(gwmi win32_process -Filter "processid='$ppid'").parentprocessid
$pname=(Get-Process -id $ppid).Name
$i++
}
# Cross compiling GStreamer with gst-build
GStreamer can be cross compiled for various platforms using gst-build. However,
only dependencies that are ported to the Meson build system will be built. It is
recommended to use Cerbero to cross compile GStreamer when other external
dependencies are required.
Once the toolchain is installed and a Meson cross file is created, to build
GStreamer simply run for example: `meson --cross-file cross-files/mingw_w64_x86-64.txt builddir`.
## Android
Requires Android API level >= 28, previous versions are missing *iconv* dependency.
- Download and extract the [NDK](https://developer.android.com/ndk/)
- Create a standalone toolchain targeting your arch and API level:
`./build/tools/make_standalone_toolchain.py --arch $arch --api $api --install-dir $toolchain_path`
- Create a Meson cross file, you can use `android_arm64_api28.txt` as example
and change CPU architectures and toolchain path.
Notes:
- On fedora the Android NDK requires the `ncurses-compat-libs` package.
## Windows
GStreamer can be cross compiled for Windows using mingw packaged in most
distribution.
The Meson cross file `mingw_w64_x86-64.txt` can be used when targeting amd64
architecture, or adapted for i686 arch.
### Fedora
- Install the toolchain packages: `mingw64-gcc`, `mingw64-gcc-c++`. Fedora
provides many other optional dependencies that could be installed as well.
For example: `mingw64-gettext`, `mingw64-libffi`, `mingw64-zlib`.
### Ubuntu
- Install the toolchain package: `gcc-mingw-w64`.
[host_machine]
system = 'android'
cpu_family = 'aarch64'
cpu = 'arm64'
endian = 'little'
[properties]
c_args = ['--sysroot', '/opt/android-arm64-api28/sysroot']
cpp_args = ['--sysroot', '/opt/android-arm64-api28/sysroot']
[binaries]
c = '/opt/android-arm64-api28/bin/aarch64-linux-android-clang'
cpp = '/opt/android-arm64-api28/bin/aarch64-linux-android-clang++'
ar = '/opt/android-arm64-api28/bin/aarch64-linux-android-ar'
strip = '/opt/android-arm64-api28/bin/aarch64-linux-android-strip'
pkgconfig = 'false'
[host_machine]
system = 'windows'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'
[properties]
c_args = []
c_link_args = []
[binaries]
c = 'x86_64-w64-mingw32-gcc'
cpp = 'x86_64-w64-mingw32-g++'
ar = 'x86_64-w64-mingw32-ar'
strip = 'x86_64-w64-mingw32-strip'
pkgconfig = 'x86_64-w64-mingw32-pkg-config'
windres = 'x86_64-w64-mingw32-windres'
#!/usr/bin/env python3
import argparse
import os
import subprocess
import xml.etree.ElementTree as ET
import sys
from scripts.common import git
from scripts.common import Colors
from scripts.common import accept_command
from scripts.common import get_meson
SCRIPTDIR = os.path.normpath(os.path.dirname(__file__))
def manifest_get_commits(manifest):
res = {}
tree = ET.parse(manifest)
root = tree.getroot()
remotes = {}
for child in root:
if child.tag == 'remote':
remotes[child.attrib['name']] = child.attrib['fetch']
if child.tag == 'project':
name = child.attrib['name']
path = child.attrib.get('path', name)
remote = child.attrib.get('remote')
if remote:
res[path] = [child.attrib["revision"], [os.path.join(remotes[remote], name), child.attrib.get('refname', child.attrib["revision"])]]
else:
res[path] = [child.attrib["revision"], []]
return res
def get_branch_name(repo_dir):
return git('-C', repo_dir, 'rev-parse', '--symbolic-full-name', 'HEAD').strip()
def ensure_revision_if_necessary(repo_dir, revision):
"""
Makes sure that @revision is set if the current repo is detached.
"""
if not revision:
if get_branch_name(repo_dir) == 'HEAD':
revision = git('-C', repo_dir, 'rev-parse', 'HEAD').strip()
return revision
def update_subprojects(repos_commits, no_interaction=False):
subprojects_dir = os.path.join(SCRIPTDIR, "subprojects")
for repo_name in os.listdir(subprojects_dir):
repo_dir = os.path.normpath(os.path.join(SCRIPTDIR, subprojects_dir, repo_name))
if not os.path.exists(os.path.join(repo_dir, '.git')):
continue
revision, args = repos_commits.get(repo_name, [None, []])
if not update_repo(repo_name, repo_dir, revision, no_interaction, args):
return False
return True
def update_repo(repo_name, repo_dir, revision, no_interaction, fetch_args=[], recurse_i=0):
revision = ensure_revision_if_necessary(repo_dir, revision)
git("config", "rebase.autoStash", "true", repository_path=repo_dir)
try:
if revision:
print("Checking out %s in %s" % (revision, repo_name))
git("fetch", *fetch_args, repository_path=repo_dir)
git("checkout", "--detach", revision, repository_path=repo_dir)
else:
print("Updating branch %s in %s" % (get_branch_name(repo_dir), repo_name))
git("pull", "--rebase", repository_path=repo_dir)
git("submodule", "update", repository_path=repo_dir)
except Exception as e:
out = getattr(e, "output", b"").decode()
if not no_interaction:
print("====================================="
"\n%s\nEntering a shell in %s to fix that"
" just `exit 0` once done, or `exit 255`"
" to skip update for that repository"
"\n=====================================" % (
out, repo_dir))
try:
if os.name == 'nt':
shell = os.environ.get("COMSPEC", r"C:\WINDOWS\system32\cmd.exe")
else:
shell = os.environ.get("SHELL", os.path.realpath("/bin/sh"))
subprocess.check_call(shell, cwd=repo_dir)
except subprocess.CalledProcessError as e:
if e.returncode == 255:
print("Skipping '%s' update" % repo_name)
return True
except:
# Result of subshell does not really matter
pass
if recurse_i < 3:
return update_repo(repo_name, repo_dir, revision, no_interaction,
recurse_i + 1)
return False
else:
print("\nCould not rebase %s, please fix and try again."
" Error:\n\n%s %s" % (repo_dir, out, e))
return False
commit_message = git("show", "--shortstat", repository_path=repo_dir).split("\n")
print(u" -> %s%s%s - %s" % (Colors.HEADER, commit_message[0][7:14], Colors.ENDC,
commit_message[4].strip()))
return True
if __name__ == "__main__":
parser = argparse.ArgumentParser(prog="git-update")
parser.add_argument("--no-color",
default=False,
action='store_true',
help="Do not output ansi colors.")
parser.add_argument("--builddir",
default=None,
help="Specifies the build directory where to"
" invoke ninja after updating.")
parser.add_argument("--no-interaction",
default=False,
action='store_true',
help="Do not allow interaction with the user.")
parser.add_argument("--manifest",
default=None,
help="Use a android repo manifest to sync repositories"
" Note that it will let all repositories in detached state")
options = parser.parse_args()
if options.no_color or not Colors.can_enable():
Colors.disable()
if options.no_interaction:
sys.stdin.close()
if options.manifest:
meson = get_meson()
targets_s = subprocess.check_output(meson + ['subprojects', 'download'])
repos_commits = manifest_get_commits(options.manifest)
else:
repos_commits = {}
revision, args = repos_commits.get('gst-build', [None, []])
if not update_repo('gst-build', SCRIPTDIR, revision, options.no_interaction, args):
exit(1)
if not update_subprojects(repos_commits, options.no_interaction):
exit(1)
if options.builddir:
ninja = accept_command(["ninja", "ninja-build"])
if not ninja:
print("Can't find ninja, other backends are not supported for rebuilding")
exit(1)
if not os.path.exists(os.path.join (options.builddir, 'build.ninja')):
print("Can't rebuild in %s as no build.ninja file found." % options.builddir)
print("Rebuilding all GStreamer modules.")
exit(subprocess.call([ninja, '-C', options.builddir]))
This diff is collapsed.