xwayland: Fix user after free on shutdown

We need to remove our listener link before we free the structure
it's inside, or the signal list walk will try to dereference it.

Remove the improbable NULL check at the same time.

Signed-off-by: Derek Foreman <derek.foreman@collabora.com>