configure: Set libdrm flags correctly if only XORG is enabled

This fixes modesetting driver build failure which can be triggered with
the following configure options:

$ ./configure --disable-dri --disable-dri2 --disable-dri3
--disable-config-udev --enable-xorg

Bugzilla: https://bugs.gentoo.org/689768
Signed-off-by: Alexander Tsoy <alexander@tsoy.me>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
3 jobs for fix-libdrm-flags in 5 minutes and 44 seconds (queued for 1 second)
Status Job ID Name Coverage
  Docker Image
passed #705522
debian-testing

00:00:34

 
  Build And Test
passed #705523
autotools-build-and-test

00:05:10

passed #705524
meson-build-and-test

00:01:46