randr: avoid integer truncation in length check of ProcRRChange*Property
Affected are ProcRRChangeProviderProperty and ProcRRChangeOutputProperty. See also 8f454b79 where this same bug was fixed for the core protocol and XI.
This fixes an OOB read and the resulting information disclosure.
Length calculation for the request was clipped to a 32-bit integer. With the correct stuff->nUnits value the expected request size was truncated, passing the REQUEST_FIXED_SIZE check.
The server then proceeded with reading at least stuff->num_items bytes (depending on stuff->format) from the request and stuffing whatever it finds into the property. In the process it would also allocate at least stuff->nUnits bytes, i.e. 4GB.
CVE-2023-6478, ZDI-CAN-22561
This vulnerability was discovered by: Jan-Niklas Sohn working with Trend Micro Zero Day Initiative