Remove uneeded xorg-server dependency

This dependency doesn't seem to be used anymore since the x11glvnd Xorg module has been removed. An Xorg server might still be needed for make tests.