tests: Set clip rect for drm_screenshot_no_damage test

The reference image is 1024x768. If the test is run on an output larger than that it will fail the ROI check. Set the clip rectangle to the reference image size when comparing the reference with the screenshot.

Signed-off-by: Harry Wentland harry.wentland@amd.com

LGTM! Although the doc comment is maybe outdated?

If clip_rect is NULL, the images must be of the same size.

It looks like there other tests that use the ROI function without clip and might or might not rely on the fact that images are exact when the clip rect isn't set.

A better fix is probably to just grab the clip rect from the reference image and pass it down.

added 8 commits

• f28e6c7a...60ab0364 - 7 commits from branch wayland:main
• 4063bae4 - tests: Set clip rect for drm_screenshot_no_damage test

Compare with previous version

changed title from tests: Pick min width/height for ROI check to tests: Set clip rect for drm_screenshot_no_damage test

changed the description

Thank you, this raises some big questions about what Weston's test suite should aim to be.

You probably assumed that DRM-backend tests should be able to run on actual hardware. That is not the case really. Hardware is both uncontrollable (monitor capabilities, connected devices / hotplug, available planes, formats, modifiers, etc.) and hugely varying (no writeback connectors ⇒ no KMS planes testing). We thought about this while @leandrohrb was working on the writeback connector (still WIP) and other KMS tests, and we came to the conclusion that DRM-backend tests shall require VKMS, and are not expected to run on real hardware.

Hence, we know exactly what resolution we get in the drm_screenshot_no_damage test and it must match. Or that was the idea.

I guess the first problem is that we are not documenting the intentions with DRM-backend tests properly.

If you want to turn Weston into a test suite for hardware and drivers, then that's a whole another matter. Does IGT not fulfill that purpose? Would the effort not be better spent there?

I'm hoping that if both VKMS and hardware + drivers pass IGT, and Weston passes on VKMS, then Weston on hardware should be fine. I suppose that is quite optimistic, but the trade-off is writing and maintaining DRM-backend tests that should be completely adapting to the KMS resources at hand and skipping all tests that need control (e.g. hotplug). Would the effort be worth it?

For example, if a test is supposed to test that Weston programs KMS planes correctly, we cannot have Weston falling back to a renderer-based screenshot. We need both the KMS planes and a writeback connector. Since we are usually comparing to a fixed reference image, also the scenegraph is fixed and cannot adapt to at least smaller video modes than expected. One (good) option would be to generate the reference image during the test with a renderer, and compare that to a KMS planes writeback image.

Yeah, I mean the root question in this @hwentland is: what do you want from it? :) As is, it's just making sure that our DRM backend basically works: when we get more capability through VKMS and llvmpipe then we'll be testing things like overlays and dmabuf as well. We outsource the job of 'does this KMS implementation actually behave like KMS should?' to igt right now.

So is there something specific you're after, or is it just looking to see if amdgpu is behaving 'as expected'? If the latter then just skip it as it doesn't exercise anywhere near enough to be meaningful.

This is mostly just me going through the CM&HDR work and learning about Weston from that end. As a DRM/KMS guy I saw the DRM test and figured "let's try running it and learn what I can along the way."

I agree that IGT should be the reference DRM/KMS tests. Is the VKMS requirement/setup documented for Weston tests? If so, I might have missed it. If not, I can take a stab at adding comments to the tests and/or add a blurb in the "running tests" part of the doc?

I don't think it's commented anywhere outside of CI, no. ;) Usually when running tests outside of CI you'll use the headless backend, which is the default unless you override it? Maybe overriding it should come with 'hope you know what you're doing' warnings. Doc patches definitely welcome!

Yeah, CI is the only one where we have the VMKS setup. CI uses Qemu to be able to run a custom kernel with VKMS, but you don't have to do that if you have VKMS in your kernel already, or that's the idea. You set up some environment variables to point to the VKMS device, make sure you are allowed access to it, and then the DRM-backend tests should run instead of skip.

There is no backend overriding. Each test is specifically written for a particular backend. Most of them use headless-backend, and have no benefit from running on any other backend as they don't test backend-specifics. Tests that do test backend-specifics, well, obviously need to run with that backend.

added DRM/KMS backend Testing labels

closed