ci: avoid trying to cache cargo registry

There have been some very weird issues, where cargo keeps fetching older commits of glib-rs and breaking the build like so [1]

Might be stale cache, or some http/git mirroring shenanigans, so lets stop trying to use the cache for now and check if the issue persists.