...
 
Commits (4)
  • Jason Ekstrand's avatar
    markdownify the README · b83f1c26
    Jason Ekstrand authored
    Reviewed-by: Jordan Justen's avatarJordan Justen <jordan.l.justen@intel.com>
    b83f1c26
  • Ian Romanick's avatar
    report.py: Gather and log some statistics about the helped / hurt data · 1cba7ba9
    Ian Romanick authored
    A previous method that I tried was treating the helped and hurt data
    as samples from separate populations, and these were compared using a
    T-test.  Since we're applying a common change to "both" sample sets, I
    don't think this is a valid analysis.
    
    Instead, I think it is more valid to look at the entire change set as a
    sample of a single population and compare the mean of that sample to
    zero.  Only the changed samples are examined because the vast majority
    of the sample in unaffected.  If the mean of the entire sample was used,
    the mean confidence interval would always include zero.  It would be
    more valid, I believe, include shaders that were affected but had no
    change in instruction or cycle count.  I don't know of a way to
    determine this using the existing shader-db infrastructure.
    
    These two different methods communicate two different things.  The first
    tries to determine whether the shaders hurt are affected more or less
    than the shaders helped.  This doesn't capture any information about the
    number of shaders affected.  There might be 1,000 shaders helped and 3
    hurt, and the conclusion could still be negative.  The second methods
    trieds to determine whether the sample set is overall helped or hurt.
    This allows the magnitued of hurt (or help) to be overwhelmed by the
    number of helped (or hurt) shaders.  There could be 1,000 shaders helped
    by 1 instruction and 3 shaders hurt by 50 instructions, and the
    conclusion would be positive.
    
    Comparing the declared result with the mean and median, I feel like the
    second method matches my intuitive interpretation of the data.  Here is
    a result of the T-test:
    
        total cycles in shared programs: 559379982 -> 559342256 (<.01%)
        cycles in affected programs: 10791218 -> 10753492 (-0.35%)
        helped: 1952
        HURT: 908
        helped stats (abs) min: 1 max: 5762 x̄: 37.71 x̃: 16
        helped stats (rel) min: <.01% max: 28.57% x̄: 3.54% x̃: 2.09%
        HURT stats (abs)   min: 1 max: 573 x̄: 39.51 x̃: 10
        HURT stats (rel)   min: <.01% max: 27.78% x̄: 1.93% x̃: 0.66%
        abs t: -0.34, p: 73.70%
        rel t: 9.88, p: <.01%
        Inconclusive result (cannot disprove both null hypothoses).
    
    And here is the result of the mean confidence interval tests on the
    same data:
    
        total cycles in shared programs: 559378112 -> 559340386 (<.01%)
        cycles in affected programs: 10791218 -> 10753492 (-0.35%)
        helped: 1952
        HURT: 908
        helped stats (abs) min: 1 max: 5762 x̄: 37.71 x̃: 16
        helped stats (rel) min: <.01% max: 28.57% x̄: 3.54% x̃: 2.09%
        HURT stats (abs)   min: 1 max: 573 x̄: 39.51 x̃: 10
        HURT stats (rel)   min: <.01% max: 27.78% x̄: 1.93% x̃: 0.66%
        95% mean confidence interval for cycles value: -18.27 -8.11
        95% mean confidence interval for cycles %-change: -1.98% -1.63%
        Cycles are helped.
    
    Since the confidence interval is calculated based on the sample mean and
    the sample standard deviation, it can include values out side the sample
    minimum and maximum.  This can lead to unexpected conclusions.  In this
    case all of the affected shaders were helped, but the result is
    inconclusive.
    
        total instructions in shared programs: 7886959 -> 7886925 (<.01%)
        instructions in affected programs: 1340 -> 1306 (-2.54%)
        helped: 4
        HURT: 0
        helped stats (abs) min: 2 max: 15 x̄: 8.50 x̃: 8
        helped stats (rel) min: 0.63% max: 4.30% x̄: 2.45% x̃: 2.43%
        95% mean confidence interval for instructions value: -20.44 3.44
        95% mean confidence interval for instructions %-change: -5.78% 0.89%
        Inconclusive result (value mean confidence interval includes 0).
    
    v2: Don't log statistics for spill or fills.  Simplify T-test logging.
    
    v3: Use confidence interval instead.
    
    Acked-by: Jason Ekstrand jason@jlekstrand.net
    1cba7ba9
  • Ian Romanick's avatar
    run: Add Ice Lake as a platform · 0220a63b
    Ian Romanick authored
    PCI ID directly from Mesa's include/pci_ids/i965_pci_ids.h.
    
    Acked-by: Jason Ekstrand jason@jlekstrand.net
    0220a63b
  • Ian Romanick's avatar
=== What ===
Shader-db: A collection of shaders for offline analysis
=======================================================
A giant pile of shaders from various apps, for whatever purpose. In
particular, we use it to capture assembly output of the shader
compiler for analysis of regressions in compiler behavior.
Shader-db is a giant pile of shaders from various apps, for whatever purpose.
In particular, we use it to capture assembly output of the shader compiler for
analysis of regressions in compiler behavior.
Currently it supports Mesa's i965 and radeonsi drivers.
=== Capturing shaders ===
# create the directory for shader files
mkdir dirpath
## Capturing shaders
# write all shaders that are loaded by "executable" to the directory
MESA_SHADER_CAPTURE_PATH=dirpath executable
Create the directory for shader files
# "fdupes" can be used to remove duplicates
$ mkdir dirpath
=== Compiling ===
Write all shaders that are loaded by "executable" to the directory
$ MESA_SHADER_CAPTURE_PATH=<dirpath> <executable>
Use `fdupes` can be used to remove duplicates
$ fdupes -d <dirpath>
## Compiling
Some libraries are required when building. See section "Dependencies" below.
To build the binary, do:
make
=== i965 Usage ===
## i965 Usage
### Running shaders
=== Running shaders ===
$ ./run shaders 2> err | tee new-run
./run shaders 2> err | tee new-run
### To run just a subset:
# To run just a subset:
./run shaders/supertuxkart 2> err | tee new-run
$ ./run shaders/supertuxkart 2> err | tee new-run
Make sure to check the contents of 'err' after your run.
......@@ -37,51 +45,59 @@ To compile shaders for an i965 PCI ID different from your system, pass
-p {i965,g4x,ilk,snb,ivb,hsw,byt,bdw}
to run.
to `run`.
=== Analysis ===
### Analysis
./report.py old-run new-run
$ ./report.py old-run new-run
=== radeonsi Usage ===
=== Running shaders ===
## radeonsi Usage
./run shaders > new-run 2> /dev/null
### Running shaders
Note that a debug mesa build required (ie. --enable-debug)
$ ./run shaders > new-run 2> /dev/null
=== Analysis ===
Note that a debug mesa build required (ie. `--enable-debug`)
### Analysis
./si-report.py old-run new-run
=== freedreno Usage ===
=== Running shaders ===
## freedreno Usage
### Running shaders
$ ./run -1 shaders 2> new-run
./run -1 shaders 2> new-run
Note that a debug mesa build required (ie. `--enable-debug`)
Note that a debug mesa build required (ie. --enable-debug)
`-1` option for disabling multi-threading is required to avoid garbled shader dumps.
-1 option for disabling multi-threading is required to avoid garbled shader dumps.
=== Analysis ===
### Analysis
./fd-report.py old-run new-run
=== Dependencies ===
## Dependencies
run requires some GNU C extensions, render nodes (/dev/dri/renderD128),
libepoxy, OpenMP, and Mesa configured with --with-egl-platforms=x11,drm
libepoxy, OpenMP, and Mesa configured with `--with-egl-platforms=x11,drm`
=== jemalloc ===
## jemalloc
Since run compiles shaders in different threads, malloc/free locking overhead
from inside Mesa can be expensive. Preloading jemalloc can cut significant
amounts of time:
LD_PRELOAD=/usr/lib64/libjemalloc.so.1 ./run shaders 2> err | tee new-run
$ LD_PRELOAD=/usr/lib64/libjemalloc.so.1 ./run shaders 2> err | tee new-run
=== Deprecated ===
## Deprecated
run.py is obsolete. Use the 'run' binary instead.
`run.py` is obsolete. Use the `run` binary instead.
......@@ -2,6 +2,9 @@
import re
import argparse
import statistics
from scipy import stats
import numpy as np
def get_results(filename):
......@@ -63,6 +66,35 @@ def get_result_string(p, b, a):
def split_list(string):
return string.split(",")
def gather_statistics(changes, before, after, m):
stats = (0.0, 0, 0.0, 0, 0, 0.0, 0.0, 0.0, 0.0, 0.0)
num = len(changes)
if num > 0:
absolute = [abs(before[p][m] - after[p][m]) for p in changes]
relative = [0 if before[p][m] == 0 else abs(before[p][m] - after[p][m]) / before[p][m] for p in changes]
stats = (statistics.mean(absolute),
statistics.median(absolute),
min(absolute),
max(absolute),
statistics.mean(relative),
statistics.median(relative),
min(relative),
max(relative))
return stats
def mean_confidence_interval(data, confidence=0.95):
a = 1.0 * np.array(data)
n = len(a)
m, se = np.mean(a), stats.sem(a)
h = se * stats.t.ppf((1 + confidence) / 2., n - 1)
return m, m - h, m + h
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--measurements", "-m", type=split_list,
......@@ -82,6 +114,9 @@ def main():
affected_after = {}
num_hurt = {}
num_helped = {}
helped_statistics = {}
hurt_statistics = {}
confidence_interval = {}
for m in args.measurements:
total_before[m] = 0
......@@ -135,9 +170,21 @@ def main():
if hurt:
print("")
helped_statistics[m] = gather_statistics(helped, args.before, args.after, m)
hurt_statistics[m] = gather_statistics(hurt, args.before, args.after, m)
num_helped[m] = len(helped)
num_hurt[m] = len(hurt)
# Statistics for spills and fills is usually meaningless.
if m in ["spills", "fills"]:
continue
if num_hurt[m] + num_helped[m] > 3:
A = [args.after[p][m] - args.before[p][m] for p in helped + hurt]
B = [0 if args.before[p][m] == 0 else (args.after[p][m] - args.before[p][m]) / args.before[p][m] for p in helped + hurt]
confidence_interval[m] = (mean_confidence_interval(A), mean_confidence_interval(B))
lost = []
gained = []
......@@ -172,13 +219,65 @@ def main():
print("total {0} in shared programs: {1}\n"
"{0} in affected programs: {2}\n"
"helped: {3}\n"
"HURT: {4}\n".format(
"HURT: {4}".format(
m,
change(total_before[m], total_after[m]),
change(affected_before[m], affected_after[m]),
num_helped[m],
num_hurt[m]))
# Statistics for spills and fills is usually meaningless.
if m in ["spills", "fills"]:
print()
continue
if num_helped[m] > 2 or (num_helped[m] > 0 and num_hurt[m] > 0):
(avg_abs, med_abs, lo_abs, hi_abs, avg_rel, med_rel, lo_rel, hi_rel) = helped_statistics[m]
print("helped stats (abs) min: {} max: {} x\u0304: {:.2f} x\u0303: {}".format(
lo_abs, hi_abs, avg_abs, int(med_abs)))
print("helped stats (rel) min: {} max: {} x\u0304: {} x\u0303: {}".format(
format_percent(lo_rel),
format_percent(hi_rel),
format_percent(avg_rel),
format_percent(med_rel)))
if num_hurt[m] > 2 or (num_hurt[m] > 0 and num_helped[m] > 0):
(avg_abs, med_abs, lo_abs, hi_abs, avg_rel, med_rel, lo_rel, hi_rel) = hurt_statistics[m]
print("HURT stats (abs) min: {} max: {} x\u0304: {:.2f} x\u0303: {}".format(
lo_abs, hi_abs, avg_abs, int(med_abs)))
print("HURT stats (rel) min: {} max: {} x\u0304: {} x\u0303: {}".format(
format_percent(lo_rel),
format_percent(hi_rel),
format_percent(avg_rel),
format_percent(med_rel)))
if m in confidence_interval:
print("95% mean confidence interval for {} value: {:.2f} {:.2f}".format(m,
confidence_interval[m][0][1],
confidence_interval[m][0][2]))
print("95% mean confidence interval for {} %-change: {} {}".format(m,
format_percent(confidence_interval[m][1][1]),
format_percent(confidence_interval[m][1][2])))
# Be very, very conservative about applying results
# based on the confidence intervals. Neither interval
# can include zero, and both intervals must be on the
# same side of zero.
if confidence_interval[m][0][1] < 0 and confidence_interval[m][0][2] > 0:
print("Inconclusive result (value mean confidence interval includes 0).");
elif confidence_interval[m][1][1] < 0 and confidence_interval[m][1][2] > 0:
print("Inconclusive result (%-change mean confidence interval includes 0).");
elif (confidence_interval[m][0][1] < 0) != (confidence_interval[m][1][1] < 0):
print("Inconclusive result (value mean confidence interval and %-change mean confidence interval disagree).");
elif confidence_interval[m][0][1] < 0:
print("{} are helped.".format(m.capitalize()))
else:
print("{} are HURT.".format(m.capitalize()))
print()
if lost or gained or not args.changes_only:
print("LOST: " + str(len(lost)))
......
......@@ -418,6 +418,7 @@ const struct platform platforms[] = {
"byt", "0x0F33",
"bdw", "0x162E",
"skl", "0x191D",
"icl", "0x8A50",
};
void print_usage(const char *prog_name)
......@@ -574,6 +575,7 @@ main(int argc, char **argv)
}
setenv("allow_glsl_extension_directive_midshader", "true", 1);
setenv("allow_glsl_builtin_variable_redeclaration", "true", 1);
setenv("shader_precompile", "true", 1);
setenv("MESA_GLSL_CACHE_DISABLE", "true", 1);
setenv("GALLIUM_THREAD", "0", 1);
......