...
 
Commits (23)
[submodule "validate/common"]
path = validate/common
url = https://anongit.freedesktop.org/git/gstreamer/common.git
url = https://gitlab.freedesktop.org/gstreamer/common.git
project('gst-devtools', 'c',
version : '1.14.0',
meson_version : '>= 0.36.0',
version : '1.14.5',
meson_version : '>= 0.40.1',
default_options : [ 'warning_level=1',
'c_std=gnu99',
'buildtype=debugoptimized' ])
......
......@@ -49,12 +49,12 @@ build*/
/m4/*m4
/po
*docs/launcher/html
*tools/gst-validate-launcher
tests/check/validate/monitoring
tests/check/validate/overrides
tests/check/validate/reporting
tests/check/validate/padmonitor
/gst/validate/*-enum-types.[ch]
/tests/check/validate/monitoring
/tests/check/validate/overrides
/tests/check/validate/reporting
/tests/check/validate/padmonitor
/launcher/config.py
=== release 1.14.5 ===
2019-05-29 18:09:11 +0100 Tim-Philipp Müller <tim@centricular.com>
* meson.build:
* validate/ChangeLog:
* validate/NEWS:
* validate/configure.ac:
* validate/gst-validate.doap:
Release 1.14.5
2018-11-05 05:54:43 +0000 Matthew Waters <matthew@centricular.com>
* .gitmodules:
* validate/gst-validate.doap:
Update git locations to gitlab
2019-05-02 11:55:38 -0400 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/main.py:
launcher: Update git repo to gitlab
2018-12-06 15:35:18 -0300 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/reporters.py:
validate:launcher: Make failure as <failure> in our xunit reporter
I think it was a mistake to call them <error> as the two notions are
different (we marked failed test as "failures" in the <testuite> node).
Should make gitlab happy with our file!
2018-12-14 12:00:18 +0100 Alexandru Băluț <alexandru.balut@gmail.com>
* validate/launcher/apps/gstvalidate.py:
validate:launcher: Fix error message
2018-12-14 11:17:41 +0100 Alexandru Băluț <alexandru.balut@gmail.com>
* validate/launcher/apps/gstvalidate.py:
validate:launcher: Fix discovery of commands
=== release 1.14.4 ===
2018-10-02 23:13:25 +0100 Tim-Philipp Müller <tim@centricular.com>
* meson.build:
* validate/ChangeLog:
* validate/NEWS:
* validate/configure.ac:
* validate/gst-validate.doap:
Release 1.14.4
=== release 1.14.3 ===
2018-09-16 16:36:43 +0100 Tim-Philipp Müller <tim@centricular.com>
* meson.build:
* validate/ChangeLog:
* validate/NEWS:
* validate/configure.ac:
* validate/gst-validate.doap:
Release 1.14.3
2018-09-07 15:59:49 -0300 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/vfb_server.py:
validate:launcher: Just wait for a while before considering Xvfb is ready if xset is not present
This is what xvfb-run so let's consider it good enough
2018-08-16 16:28:15 -0400 Nicolas Dufresne <nicolas.dufresne@collabora.com>
* meson.build:
meson: Unify required version to 0.40.1
2018-07-10 13:16:36 +0200 Edward Hervey <edward@centricular.com>
* validate/launcher/reporters.py:
validate: Use 'skipped' keyword in xunit xml
It was always meant to be 'skipped' to be 100% compatible with xunit
xsl.
Makes jenkins happy again
=== release 1.14.2 ===
2018-07-20 01:07:44 +0100 Tim-Philipp Müller <tim@centricular.com>
* meson.build:
* validate/ChangeLog:
* validate/NEWS:
* validate/configure.ac:
* validate/gst-validate.doap:
Release 1.14.2
2018-07-01 11:32:10 -0400 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/baseclasses.py:
* validate/launcher/utils.py:
validate:launcher: Allow retrieving coredumps from within flatpak
2018-06-17 08:34:09 -0400 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/baseclasses.py:
validate:launcher: Fix the --forever switch
It was not stopping on error.
https://bugzilla.gnome.org/show_bug.cgi?id=796608
2018-06-05 16:25:46 +0200 Edward Hervey <edward@centricular.com>
* validate/.gitignore:
* validate/docs/.gitignore:
* validate/tools/.gitignore:
validate: Update all gitignore
=== release 1.14.1 ===
2018-05-17 13:35:29 +0100 Tim-Philipp Müller <tim@centricular.com>
* meson.build:
* validate/ChangeLog:
* validate/NEWS:
* validate/configure.ac:
* validate/gst-validate.doap:
Release 1.14.1
2018-05-15 14:40:45 -0400 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/baseclasses.py:
validate:launcher: Error out loudly if the testlist changes
When --fail-on-testlist-change is set.
2018-05-15 14:35:30 -0400 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/reporters.py:
validate:launcher: Do not print time spent if the testsuite never started
2018-04-28 10:15:17 +0200 Thibault Saunier <tsaunier@igalia.com>
* validate/configure.ac:
validate: Set TESTSUITE_VERSION to 1.14 in configure.ac
2018-04-28 10:16:09 +0200 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/baseclasses.py:
validate:launcher: Pass baseclass.py into autopep8
2018-04-28 10:15:17 +0200 Thibault Saunier <tsaunier@igalia.com>
* validate/launcher/baseclasses.py:
validate: launcher: Make sure testsuites are used/configured once only
2018-05-07 17:30:13 +0200 Edward Hervey <edward@centricular.com>
* validate/launcher/apps/gstvalidate.py:
validate: Remove hls.*seek_with_stop blacklisting
The issue is closed upstream (because of concentrating on decodebin3
instead), and initial forever testing seems to show the issue doesn't
happen anymore
2018-04-25 10:57:14 +0800 Kai Kang <kai.kang@windriver.com>
* validate/gst/validate/Makefile.am:
validate: fix out of source tree build error
It fails to generate gst-validate-enum-types.h and gst-validate-enum-types.c
when build out of source tree. Add the path for template files.
https://bugzilla.gnome.org/show_bug.cgi?id=795531
Signed-off-by: Kai Kang <kai.kang@windriver.com>
=== release 1.14.0 ===
2018-03-19 20:29:07 +0000 Tim-Philipp Müller <tim@centricular.com>
......
This diff is collapsed.
......@@ -2,7 +2,7 @@ AC_PREREQ(2.62)
dnl initialize autoconf
dnl when going to/from release please set the nano (fourth number) right !
dnl releases only do Wall, cvs and prerelease does Werror too
AC_INIT(Gst-Validate, 1.14.0,
AC_INIT(Gst-Validate, 1.14.5,
http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer,
gst-validate)
......@@ -10,7 +10,7 @@ dnl This is the version of the testsuite to use with the current version
dnl of gst-validate. During development it should be "master"
dnl during release cycle it should be the release version (1.6 in the 1.6
dnl branch, 1.5.90 for the 1.5.90 release)
GST_VALIDATE_TESTSUITE_VERSION="master"
GST_VALIDATE_TESTSUITE_VERSION="1.14"
AS_AC_EXPAND(GST_VALIDATE_TESTSUITE_VERSION, $GST_VALIDATE_TESTSUITE_VERSION)
AG_GST_INIT
......@@ -49,11 +49,11 @@ AC_SUBST(GST_API_VERSION)
AC_DEFINE_UNQUOTED(GST_API_VERSION, "$GST_API_VERSION",
[GStreamer API Version])
AS_LIBTOOL(GST, 1400, 0, 1400)
AS_LIBTOOL(GST, 1405, 0, 1405)
dnl *** required versions of GStreamer stuff ***
GST_REQ=1.14.0
GSTPB_REQ=1.14.0
GST_REQ=1.14.5
GSTPB_REQ=1.14.5
dnl *** autotools stuff ****
......
......@@ -7,4 +7,4 @@ Makefile.in
.libs
version.entities
tmpl/
/launcher/html
......@@ -47,11 +47,61 @@
<repository>
<GitRepository>
<location rdf:resource="git://anongit.freedesktop.org/gstreamer/gst-devtools"/>
<browse rdf:resource="http://cgit.freedesktop.org/gstreamer/gst-devtools/"/>
<location rdf:resource="https://gitlab.freedesktop.org/gstreamer/gst-devtools"/>
<browse rdf:resource="https://gitlab.freedesktop.org/gstreamer/gst-devtools/"/>
</GitRepository>
</repository>
<release>
<Version>
<revision>1.14.5</revision>
<branch>1.14</branch>
<name></name>
<created>2019-05-29</created>
<file-release rdf:resource="https://gstreamer.freedesktop.org/src/gst-validate/gst-validate-1.14.5.tar.xz" />
</Version>
</release>
<release>
<Version>
<revision>1.14.4</revision>
<branch>1.14</branch>
<name></name>
<created>2018-10-02</created>
<file-release rdf:resource="https://gstreamer.freedesktop.org/src/gst-validate/gst-validate-1.14.4.tar.xz" />
</Version>
</release>
<release>
<Version>
<revision>1.14.3</revision>
<branch>1.14</branch>
<name></name>
<created>2018-09-16</created>
<file-release rdf:resource="https://gstreamer.freedesktop.org/src/gst-validate/gst-validate-1.14.3.tar.xz" />
</Version>
</release>
<release>
<Version>
<revision>1.14.2</revision>
<branch>1.14</branch>
<name></name>
<created>2018-07-20</created>
<file-release rdf:resource="https://gstreamer.freedesktop.org/src/gst-validate/gst-validate-1.14.2.tar.xz" />
</Version>
</release>
<release>
<Version>
<revision>1.14.1</revision>
<branch>1.14</branch>
<name></name>
<created>2018-05-17</created>
<file-release rdf:resource="https://gstreamer.freedesktop.org/src/gst-validate/gst-validate-1.14.1.tar.xz" />
</Version>
</release>
<release>
<Version>
<revision>1.14.0</revision>
......
......@@ -50,12 +50,12 @@ nodist_libgstvalidate_@GST_API_VERSION@include_HEADERS = $(built_header_make)
gst-validate-enum-types.h: $(source_h)
$(AM_V_GEN)$(GLIB_MKENUMS) \
--template gst-validate-enum-types.h.template \
--template $(top_srcdir)/gst/validate/gst-validate-enum-types.h.template \
$^ > gst-validate-enum-types.h
gst-validate-enum-types.c: $(source_h)
$(AM_V_GEN)$(GLIB_MKENUMS) \
--template gst-validate-enum-types.c.template \
--template $(top_srcdir)/gst/validate/gst-validate-enum-types.c.template \
$^ > gst-validate-enum-types.c
EXTRA_DIST= \
......
......@@ -719,11 +719,11 @@ class GstValidateTestManager(GstValidateBaseTestManager):
def init(self):
for command, name in [
(GST_VALIDATE_TRANSCODING_COMMAND, "gst-validate-1.0"),
(GST_VALIDATE_COMMAND, "gst-validate-transcoding-1.0"),
(GST_VALIDATE_TRANSCODING_COMMAND, "gst-validate-transcoding-1.0"),
(GST_VALIDATE_COMMAND, "gst-validate-1.0"),
(G_V_DISCOVERER_COMMAND, "gst-validate-media-check-1.0")]:
if not command:
self.error("%s not found" % command)
self.error("command not found: %s" % name)
return False
return True
......@@ -996,8 +996,8 @@ not been tested and explicitely activated if you set use --wanted-tests ALL""")
def register_default_blacklist(self):
self.set_default_blacklist([
# hls known issues
("hls.playback.seek_with_stop.*",
"https://bugzilla.gnome.org/show_bug.cgi?id=753689"),
# ("hls.playback.seek_with_stop.*",
# "https://bugzilla.gnome.org/show_bug.cgi?id=753689"),
# dash known issues
("dash.media_check.*",
......
This diff is collapsed.
......@@ -139,7 +139,7 @@ http://wiki.pitivi.org/wiki/Bug_reporting#Debug_logs).
QA_ASSETS = "gst-integration-testsuites"
MEDIAS_FOLDER = "medias"
DEFAULT_GST_QA_ASSETS_REPO = "git://anongit.freedesktop.org/gstreamer/gst-integration-testsuites"
DEFAULT_GST_QA_ASSETS_REPO = "https://gitlab.freedesktop.org/gstreamer/gst-integration-testsuites.git"
OLD_DEFAULT_GST_QA_ASSETS_REPO = "https://gitlab.com/thiblahute/gst-integration-testsuites.git"
DEFAULT_TESTSUITES_DIRS = [os.path.join(DEFAULT_MAIN_DIR, QA_ASSETS, "testsuites")]
......
......@@ -106,10 +106,11 @@ class Reporter(Loggable):
print("\n")
lenstat = (len("Statistics") + 1)
printc("Statistics:\n%s" % (lenstat * "-"), Colors.OKBLUE)
printc("\n%sTotal time spent: %s seconds\n" %
((lenstat * " "), datetime.timedelta(
seconds=(time.time() - self._start_time))),
Colors.OKBLUE)
if self._start_time > 0:
printc("\n%sTotal time spent: %s seconds\n" %
((lenstat * " "), datetime.timedelta(
seconds=(time.time() - self._start_time))),
Colors.OKBLUE)
printc("%sPassed: %d" %
(lenstat * " ", self.stats["passed"]), Colors.OKGREEN)
printc("%sFailed: %d" %
......@@ -185,7 +186,7 @@ class XunitReporter(Reporter):
xml_file.write('<?xml version="1.0" encoding="%(encoding)s"?>'
'<testsuite name="gst-validate-launcher" tests="%(total)d" '
'errors="%(timeout)d" failures="%(failures)d" '
'skip="%(skipped)d">' % self.stats)
'skipped="%(skipped)d">' % self.stats)
tmp_xml_file = codecs.open(self.tmp_xml_file.name, 'r',
self.encoding, 'replace')
......@@ -216,8 +217,8 @@ class XunitReporter(Reporter):
self.encoding, 'replace')
xml_file.write(self._forceUnicode(
'<testcase classname=%(cls)s name=%(name)s time="%(taken).3f">'
'<error type=%(errtype)s message=%(message)s>%(stacktrace)s'
'</error>%(systemout)s</testcase>' %
'<failure type=%(errtype)s message=%(message)s>%(stacktrace)s'
'</failure>%(systemout)s</testcase>' %
{'cls': self._quoteattr(test.get_classname()),
'name': self._quoteattr(test.get_name()),
'taken': test.time_taken,
......
......@@ -27,6 +27,7 @@ import os
import platform
import re
import shutil
import shlex
import signal
import subprocess
import sys
......@@ -293,13 +294,25 @@ def get_scenarios():
class BackTraceGenerator(Loggable):
__instance = None
_executable_regex = re.compile(r'Executable: (.*)\n')
_command_line_regex = re.compile(r'Command Line: (.*)\n')
_timestamp_regex = re.compile(r'Timestamp: .*\((\d*)s ago\)')
_pid_regex = re.compile(r'PID: (\d+) \(.*\)')
def __init__(self):
Loggable.__init__(self)
self.coredumpctl = shutil.which('coredumpctl')
self.in_flatpak = os.path.exists("/usr/manifest.json")
if self.in_flatpak:
coredumpctl = ['flatpak-spawn', '--host', 'coredumpctl']
else:
coredumpctl = ['coredumpctl']
try:
subprocess.check_output(coredumpctl)
self.coredumpctl = coredumpctl
except Exception as e:
self.warning(e)
self.coredumpctl = None
self.gdb = shutil.which('gdb')
@classmethod
......@@ -342,10 +355,15 @@ class BackTraceGenerator(Loggable):
# yet.
time.sleep(1)
if not self.in_flatpak:
coredumpctl = self.coredumpctl + ['info', str(test.process.pid)]
else:
newer_than = time.strftime("%a %Y-%m-%d %H:%M:%S %Z", time.localtime(test._starting_time))
coredumpctl = self.coredumpctl + ['info', os.path.basename(test.command[0]),
'--since', newer_than]
try:
info = subprocess.check_output(['coredumpctl', 'info',
str(test.process.pid)],
stderr=subprocess.STDOUT)
info = subprocess.check_output(coredumpctl, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError:
# The trace might not be ready yet
time.sleep(1)
......@@ -353,16 +371,15 @@ class BackTraceGenerator(Loggable):
info = info.decode()
try:
executable = BackTraceGenerator._executable_regex.findall(info)[
0]
pid = self._pid_regex.findall(info)[0]
except IndexError:
self.debug("Backtrace could not be found yet, trying harder.")
# The trace might not be ready yet
continue
if executable != test.application:
command_line = BackTraceGenerator._command_line_regex.findall(info)[0]
if shlex.split(command_line)[0] != test.application:
self.debug("PID: %s -- executable %s != test application: %s" % (
test.process.pid, executable, test.application))
pid, command_line[0], test.application))
# The trace might not be ready yet
continue
......@@ -375,20 +392,22 @@ class BackTraceGenerator(Loggable):
bt_all = None
if self.gdb:
try:
tf = tempfile.NamedTemporaryFile()
subprocess.check_output(['coredumpctl', 'dump',
str(test.process.pid), '--output=' +
tf.name], stderr=subprocess.STDOUT)
gdb = ['gdb', '-ex', 't a a bt', '-ex', 'quit',
test.application, tf.name]
bt_all = subprocess.check_output(
gdb, stderr=subprocess.STDOUT).decode()
info += "\nThread apply all bt:\n\n%s" % (
bt_all.replace('\n', '\n' + 15 * ' '))
with tempfile.NamedTemporaryFile() as stderr:
coredump = subprocess.check_output(self.coredumpctl + ['dump', pid],
stderr=stderr)
with tempfile.NamedTemporaryFile() as tf:
tf.write(coredump)
tf.flush()
gdb = ['gdb', '-ex', 't a a bt', '-ex', 'quit',
test.application, tf.name]
bt_all = subprocess.check_output(
gdb, stderr=subprocess.STDOUT).decode()
info += "\nThread apply all bt:\n\n%s" % (
bt_all.replace('\n', '\n' + 15 * ' '))
except Exception as e:
self.debug("Could not get backtrace from gdb: %s" % e)
self.error("Could not get backtrace from gdb: %s" % e)
return info
......
......@@ -46,12 +46,11 @@ class Xvfb(VirtualFrameBufferServer):
self._logsfile = None
self._command = "Xvfb %s -screen 0 1920x1080x24" % self.display_id
def _check_is_up(self, timeout=60):
""" Check if the xvfb is up, running a simple test based on wget. """
def _check_is_up(self, timeout=3, assume_true=True):
""" Check if the xvfb is up, running a simple test based on xset. """
start = time.time()
while True:
try:
cdisplay = os.environ.get("DISPLAY", None)
os.environ["DISPLAY"] = self.display_id
subprocess.check_output(["xset", "q"],
stderr=self._logsfile)
......@@ -59,6 +58,13 @@ class Xvfb(VirtualFrameBufferServer):
return True
except subprocess.CalledProcessError:
pass
except FileNotFoundError:
if assume_true:
print('WARNING: xset not preset on the system,'
' just wait for %s seconds and hope for the best.'
' (this is what xvfb-run itself does anyway.)' % timeout)
time.sleep(timeout)
return assume_true
if time.time() - start > timeout:
return False
......@@ -69,7 +75,7 @@ class Xvfb(VirtualFrameBufferServer):
""" Start xvfb in a subprocess """
self._logsfile = open(os.path.join(self.options.logsdir,
"xvfb.logs"), 'w+')
if self._check_is_up(timeout=2):
if self._check_is_up(assume_true=False):
print("xvfb already running")
return (True, None)
......
gst-validate-1.0
gst-validate-launcher
gst-validate-transcoding-1.0
gst-validate-media-check-1.0
gst-validate-images-check-1.0
gst-validate-rtsp-server-1.0
gst-validate-1.0-debug
gst-validate-media-check-1.0-debug
gst-validate-transcoding-1.0-debug
gst-validate-images-check-1.0-debug
gst-validate-rtsp-server-1.0-debug