This is the long / more proper version of !22298. It's a lot of churn, touching all Vk drivers and winsys. Is the end result worth the churn? I'm not sure. I'd like @gfxstrand to weigh in.
One of the comments on !22298 mentioned that Wayland needs caching too. I haven't added that here, but the infrastructure is in place so it'd be easy to do so.