Caches contain patterns and character sets which are reference counted and
visible to applications. Reference count the underlying cache object so that
it stays around until all reference objects are no longer in use.

This is less efficient than just leaving all caches around forever, but does
avoid eternal size increases in case applications ever bother to actually
look for changes in the font configuration.