GitLab

When first initializing the DIX layer's RRCrtc structures, xf86RandR12Init12 hard-codes a gamma ramp size of 256 elements and then calls into the DDX layer to copy the gamma ramps from the RRCrtc into the xf86Crtc's gamma ramp arrays. When it does this, it assumes that crtc->gamma_size == crtc->randr_crtc->gammaSize == 256. However, commit 245b9db0 modified the modesetting driver to update crtc->gamma_size with the true size of the hardware's gamma ramp when the DRM GAMMA_LUT property is available. This causes xf86RandR12CrtcSetGamma to read past the end of the randr_crtc->gamma{Red,Green,Blue} arrays:

  PreInit
    drmmode_pre_init
      drmmode_crtc_init
        crtc->gamma_size = 1024

  ScreenInit
    xf86CrtcScreenInit
      xf86RandR12Init
        xf86RandR12Init12
          xf86RandR12CreateObjects12
            RRCrtcCreate
              randr_crtc->gammaSize = 0
          xf86RandR12InitGamma(pScrn, 256)
            RRCrtcGammaSetSize
              randr_crtc->gammaSize = 256
          xf86RandR12InitGamma
            xf86RandR12CrtcInitGamma
              RRCrtcGammaSet
                xf86RandR12CrtcSetGamma
                  // crtc->gamma_size is 1024 here, while randr_crtc->gammaRed
                  // is a 256-element array.
                  memcpy(crtc->gamma_red, randr_crtc->gammaRed, crtc->gamma_size * sizeof(crtc->gamma_red[0]));
    drmmode_setup_colormap
      xf86HandleColormaps
        xf86RandR12InitGamma
          RRCrtcGammaSetSize
            randr_crtc->gammaSize = 1024

Rather than passing gammaSize as an argument to xf86RandR12InitGamma, just use the xf86Crtc's gamma size as the argument to RRCrtcGammaSetSize so that the RRCrtc and xf86Crtc agree about the size of the gamma ramp immediately.

Fixes: 245b9db0 - modesetting: Use GAMMA_LUT when available

Closes: #1126 (closed)

Signed-off-by: Aaron Plattner aplattner@nvidia.com