rusticl/memory: Fix regressions from memory mapping rework
Fixes a couple of regressions which the "use_pitches" tests in the CTS are hitting, which I don't regularly test against.
Fixes: 7b22bc61 ("rusticl/memory: complete rework on how mapping is implemented")