Project quality assurance

New development means high risk of regression.
Therefore this project require some formal validation procedure to cover the most important features.

Let's use this issue to accumulate the experience ( scenarios, procedures, scripts... ) I hope someday we can make it a part of the CI.

Test 1. Run UIBench <Semi-automated> (Prefer using test 2 instead)

Preconditions:

Test tool: https://github.com/joshuous/uibencher
Use same equipment (Board and Monitor) and same release (for example AOSP-11r27)

Procedure:

Run the test on drm_hwcomposer/main and store baseline values
Run the test with the new changes
Collect the data. Number of Janks should be equal or less than in the baseline run.

Test 2. Run UIBench from AOSP tree

Preconditions:

Use same equipment (Board and Monitor) and same release (for example AOSP-11r27)
Disable flattening in drm_hwc to get true results
Patch AOSP test manifest file to enable all tests instead of 1:

--- a/platform_testing/tests/microbenchmarks/uibench/AndroidTest.xml
+++ b/platform_testing/tests/microbenchmarks/uibench/AndroidTest.xml
@@ -39,3 +39,2 @@
         <option name="package" value="com.android.uibench.microbenchmark" />
-        <option name="class" value="com.android.uibench.microbenchmark.UiBenchDialogListFlingMicrobenchmark#testDialogListFling" />
         <option name="device-listeners" value="android.device.collectors.PerfettoListener,android.device.collectors.SfStatsListener" />

Procedure:

Connect the board
Execute: atest UiBenchMicrobenchmark | tee results.txt
Filter-out useful data: cat results.txt | grep -i "MISS\|CLIENT\|PASSED"

Test 3. Run Composer HAL VTS tests from AOSP tree

Procedure:

Connect the board
Execute:

atest VtsHalGraphicsComposerV2_1TargetTest VtsHalGraphicsComposerV2_2TargetTest VtsHalGraphicsComposerV2_3TargetTest VtsHalGraphicsComposerV2_4TargetTest

Edited May 19, 2021 by Roman Stratiienko