configure: Set libdrm flags correctly if only XORG is enabled

This fixes modesetting driver build failure which can be triggered with
the following configure options:

$ ./configure --disable-dri --disable-dri2 --disable-dri3
--disable-config-udev --enable-xorg

Bugzilla: https://bugs.gentoo.org/689768
Signed-off-by: Alexander Tsoy <alexander@tsoy.me>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
3 jobs for fix-libdrm-flags in 4 minutes and 59 seconds (queued for 12 seconds)
latest