If the file descriptor becomes invalid (e.g. because
wl_display_disconnect has been called) exit the event loop.  There's no
value from continuing to poll.

Signed-off-by: Kevin Locke <kevin@kevinlocke.name>