bluetooth: Only remove cards belonging to the device.

Commit f89d64b9 fixed a crash
when disabling adapters. However, now if any device is removed
ofono card is removed, even if it belongs to different device.

Add a check for the device being unlinked to our callback to fix.

Signed-off-by: Juho Hämäläinen <juho.hamalainen@jolla.com>
Part-of: <pulseaudio/pulseaudio!624>