Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • xserver xserver
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 922
    • Issues 922
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 130
    • Merge requests 130
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • xorgxorg
  • xserverxserver
  • Merge requests
  • !573

xserver: fix RGB mask handling

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Leandro Lupori requested to merge luporl/xserver:fix-rgb-mask-handling into master Dec 10, 2020
  • Overview 2
  • Commits 1
  • Pipelines 2
  • Changes 2

On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was crashing in some cases. For instance, when twm was started and the background was clicked to open its menu, X crashed with a segmentation fault, trying to dereference a null pointer at CreatePicture().

There were 2 issues with xorg-server handling of RGB masks that caused the pointer above to be null and thus the crash:

  • wrong use of ffs() to get the RGB offsets from the masks
  • overflow when shifting a 16-bit integer

This change fixes both issues. They happen when the system is BE but has a video adapter using a little-endian (LE) ARGB32 framebuffer. In order to display the correct colors, this setup requires a BE RGBA32 color format to be used by X, by setting the RGB masks appropriately, that didn't work properly because of the issues above.

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: fix-rgb-mask-handling