modesetting: XRandR rotates leaks front_bo and GDM allocated resources
This issue was originally reported at "Pi4: Display goes blank / something inside the kernel or firmware crashes when running xrandr -o too often" https://github.com/raspberrypi/firmware/issues/1345
After running sequences of xrandr rotations Xserver fails displaying messages at kernel log like:
[ 845.945909] [drm:vc4_bo_create [vc4]] ERROR Failed to allocate from CMA:
[ 845.945923] [drm] dumb: 243016kb BOs (31)
[ 845.947141] [drm:vc4_bo_create [vc4]] ERROR Failed to allocate from CMA:
[ 845.947151] [drm] dumb: 243016kb BOs (31)Depending of the memory of the RPI4 and the screen resolution a sequence of xrandr -o left;xrandr -o normal generates that fails and the server needs to be restarted to continue working.
After an analysis at the Mesa and DRM v3d drivers this seems to be a memory leak in xserver, that doesn't free the front_bo in some cases in the drmmode_xf86crtc_resize function.
I'm sending a MR to fix the issue tested in the xserver.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information