gl-renderer: Fix limited range quantization

The highest possible value is 255, not 256, like in previous part of the
calculation. While on it, use the values directly, making the code more
readable.

Fix suggested by Pekka Paalanen and Loïc Molinari.

Signed-off-by: Robert Mader <robert.mader@collabora.com>