XRef: Fix wrong memory read on destruction

when there's some objects that have been added and then removed
7 jobs for digital_signatures in 4 minutes and 36 seconds (queued for 2 seconds)
Status Job ID Name Coverage
  Build
passed #5838166
build

00:04:36

passed #5838171
build_android

00:02:07

passed #5838167
build_clang_libcpp

00:04:19

passed #5838170
build_clazy

00:04:07

passed #5838169
build_mingw64_fedora

00:02:55

passed #5838168
build_ubuntu_16_04

00:01:21

passed #5838165
clang_format

00:00:50