bluetooth: Only remove cards belonging to the device.
Commit f89d64b9 fixed a crash when disabling adapters. However, now if any device is removed ofono card is removed, even if it belongs to different device.
Add a check for the device being unlinked to our callback to fix.