Getting backtraces against this assertion isn't useful at all.
Fontconfig isn't capable to report who really have a reference
of caches though, this report may be better than assertion.
So we can try analysis tools then.
And report it only when running with FC_DEBUG=16.