Fix errors caught by -fsanitize=undefined and self-tests

With all of these applied, ninja test completes successfully with -fno-sanitize-recover in cflags.

The first patch in this series is maybe questionable. It might make more sense to change wayland_layer_set_mask to take differently-typed arguments, possibly unsigned int for width/height instead, since otherwise it's not possible to create a mask with width or height greater than 1<<31 - 1. Though by the time you're working on an image that's more than two billion pixels wide probably you've encountered some other problems already...

The last two are straightforward.