fallback: cancel the debounce timers during device remove, not destroy
destroy isn't called until the last libinput_device_unref(), so we may trigger a debounce timer after the device was removed. The same fix is neded for the touchpad interface.
Fixes #72 (closed)