configure: Set libdrm flags correctly if only XORG is enabled

This fixes modesetting driver build failure which can be triggered with
the following configure options:

$ ./configure --disable-dri --disable-dri2 --disable-dri3
--disable-config-udev --enable-xorg

Bugzilla: https://bugs.gentoo.org/689768
Signed-off-by: Alexander Tsoy <alexander@tsoy.me>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
3 jobs for fix-libdrm-flags in 4 minutes and 59 seconds (queued for 12 seconds)
latest
Status Job ID Name Coverage
  Docker Image
passed debian-testing #838449

00:00:28

 
  Build And Test
passed autotools-build-and-test #838450

00:04:30

passed meson-build-and-test #838451

00:02:17