configure: Set libdrm flags correctly if only XORG is enabled

This fixes modesetting driver build failure which can be triggered with
the following configure options:

$ ./configure --disable-dri --disable-dri2 --disable-dri3
--disable-config-udev --enable-xorg

Bugzilla: https://bugs.gentoo.org/689768
Signed-off-by: Alexander Tsoy <alexander@tsoy.me>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
(cherry picked from commit 334f1107)
3 jobs for for-1.20.6 in 6 minutes and 35 seconds (queued for 8 seconds)
Status Job ID Name Coverage
  Docker Image
passed debian-testing #839033

00:00:33

 
  Build And Test
passed autotools-build-and-test #839034

00:06:02

passed meson-build-and-test #839035

00:02:11