For reverse PRIME heads, with 'modesetting' as the source, scaling was broken as the compositing of the shared pixmaps wasn't properly configured to take into account scaling (and other transformations). For example, by scaling a reverse PRIME head by 2x2 the user would only see the top-left quadrant of the 2x2 screenspace instead of the full screenspace scaled down to the size of the shared pixmap.
This change brings the RandR transformation available for tracking and updating of dirty regions in the modesetting driver, and subsequently the CompositeRotate() function which renders the transformed screen to the shared pixmap. For transforms with simple translations only a separate execution path is tracked so that direct copying can be used.
These changes should fix scaling, and to some extent, arbitrary transformations, for all heads that are reverse PRIME sinks driven from 'modesetting' source.
Note that these changes only clean up certain parts of the viewport transformation and composition pipeline. A more thorough overhaul of how all the offsets and transformations are managed would be in order to make all combinations of randr screen configurations work seamlessly. I tried to be careful not to break existing functionality but there are things that were broken in the first place and remain to be so.
As a quick test, basic scaling like this should now work properly with reverse PRIME outputs:
xrandr --output --scale 0.5x0.5 xrandr --output --scale 2x2
Arbitrary transforms still exhibit various quirks and artifacts but something like a basic free rotation should now behave a bit better than before with reverse PRIME outputs:
xrandr --output --transform 0.707,-0.707,0,0.707,0.707,0,0,0,1
With this change, this command yields a screen that is at least now rotated yet still wrongly clipped as it was also before this change.