transformation matrices converted losing resolution
Submitted by chrysn
Assigned to Keith Packard @keithp
Description
under particular circumstances, the XDoubleToFixed conversion that takes place when using xrandr --transform
loses visually noticably much precision.
details of the problem
take, for example, xrandr --output LVDS1 --transform 1,0,0,0,1,0,-0.00008,0,1
, which slightly skews the lower border of the monitor. a -0.00008 value, when afterwards displayed in xrandr
, becomes -0.000076, showing the limits for XFixed. even smaller values, which would still be visually distinct from an unskewed display (like 0.00003) wind up as 0 at all.
if this was not a projective tranformation matrix, this would be a protocol limitation, and thus hard to overcome, but as projection matrices can be scaled without changing their effect on projective coordinates, this can be overcome by scaling all values up. the example above works flawlessly when put as 100,0,0,0,100,0,-0.008,0,100
.
suggested algorithm
the algorithm i suggest is the following (working on the (double)transform matrix before it gets XDoubleToFixed converted):
MAXFIXED = MAXINT / 65536 maxfactor = MAXFIXED / maximum of transform scalefactor = 2^floor(log2(maxfactor)) transform *= scalefactor
numerical considerations
scalefactor will be >1 unless someone already did that calculation; it will be <1 if someone feeds us values too big for fixed anyway so we have to accept to lose precision. thus, i'll only discuss scalefactor >1 from now:
by scaling not to MAXFIXED, but by a power of two, we're only doing integer addition to the exponent, so the operations are lossless, and we can't hit the upper exponent boundary, as the maximum value we get is MAXINT / 65535 < MAXDOUBLE. the following conversion may trim on the least significant bits of the mantissa depending on the system's mantissa length and int size, but it uses as many bits as possible due to the scaleup.
further development of the bug
if this bug gets accepted, i offer to write a patch to fix it.
it seems to me that there is ongoing effort to refactor xrandr into a thin wrapper around a library[1], something i do appreciate; in that case, this should go to the library as well.
[1] http://lists.x.org/archives/xorg-devel/2012-January/028469.html
Version: git