wlr-output-management: not enough precision for scale

Created by: AdrianVovk

Context:

I'm building a daemon that implements the org.mutter.DisplayConfig protocol on top of wlr-output-management and wf-config

Issue:

Currently, the scale event sends a wl_fixed with the scale. However, wl_fixed does not have enough precision to represent fractional scales correctly. For instance, on my test MacBook one of the ideal scales is 1.6. This is a value that doubles cannot represent exactly; the best they can do is 1.60000...008.... However, since the number of pixels on any axis of the screen are going to be somewhere in the thousands range, a change in scale less than 0.0001 will have no effect on computations.

Trying to represent 1.6 with a wl_fixed_t yields a different result. Trying to convert 1.6 to a wl_fixed and back returns 1.60156.... There is not enough precision to represent values in the 0.0001 range, which is necessary to ensure proper scale values.

Because wlr-output-management uses a wl_fixed to send scale values, it cannot accurately report the current scale to its clients. Any calculations done with that scale will be off.

Potential solution:

The easiest and dirtiest solution would probably be to just multiply the scale by a value like 1000 in the compositor and then have the client divide it by 1000. This could be detailed in the protocol's docs.

The cleanest solution is probably modifying the Wayland wire protocol to be able to send true doubles and use that instead.

Workaround:

In my use case, I compute a list of "good" scales and those are the only scales the user can select in the GUI. Therefore, the scale reported by wlr-output-management should be one of those good scales. Here's the code