Screen-specific privates can not be used in CloseScreen

Submitted by Michal Srb

Assigned to Xorg Project Team

Link to original bug (#108762)

Description

I have a setup with nouveau driver, two monitors - one normal, one rotated. When the X server does the internal restart after last client disconnected, it aborts with assertion failure:

dixGetPrivateAddr: Assertion `key->initialized' failed.

Backtrace: #0 0x00007ffff57fb24b in raise () from /lib64/libc.so.6 #1 (closed) 0x00007ffff57e44f1 in abort () from /lib64/libc.so.6 #2 0x00007ffff57e43c1 in __assert_fail_base.cold.0 () from /lib64/libc.so.6 #3 (closed) 0x00007ffff57f3832 in __assert_fail () from /lib64/libc.so.6 #4 (closed) 0x00007ffff2070c89 in dixGetPrivateAddr (privates=0x5555566665e0, key=0x555556292c10) at ../include/privates.h:121 #5 (closed) 0x00007ffff2071647 in exaDestroyPixmap_mixed (pPixmap=0x5555566665c0) at exa_mixed.c:253 #6 (closed) 0x00005555556f5a6e in damageDestroyPixmap (pPixmap=0x5555566665c0) at damage.c:1504 #7 (closed) 0x000055555568e93b in XvDestroyPixmap (pPix=0x5555566665c0) at xvmain.c:369 #8 (closed) 0x00005555555d6ab7 in FreeScratchPixmapHeader (pPixmap=0x5555566665c0) at pixmap.c:82 #9 0x00007ffff24c6f1f in drmmode_crtc_shadow_destroy (crtc=0x555555ab7e30, rotate_pixmap=0x5555566665c0, data=0x7fffe71ef000) at drmmode_display.c:659 #10 (closed) 0x0000555555662e01 in xf86RotateDestroy (crtc=0x555555ab7e30) at xf86Rotate.c:252 #11 (closed) 0x0000555555662ff5 in xf86RotateCloseScreen (screen=0x555555b09840) at xf86Rotate.c:304 #12 (closed) 0x000055555564eef4 in xf86CrtcCloseScreen (screen=0x555555b09840) at xf86Crtc.c:781 #13 0x0000555555604f0f in DGACloseScreen (pScreen=0x555555b09840) at xf86DGA.c:288 #14 0x0000555555614695 in CMapCloseScreen (pScreen=0x555555b09840) at xf86cmap.c:250 #15 0x000055555567888c in CursorCloseScreen (pScreen=0x555555b09840) at cursor.c:205 #16 (closed) 0x00005555556e5c88 in AnimCurCloseScreen (pScreen=0x555555b09840) at animcur.c:100 #17 (closed) 0x000055555567344a in compCloseScreen (pScreen=0x555555b09840) at compinit.c:86 #18 0x00007ffff33f998f in glxCloseScreen (pScreen=0x555555b09840) at glxscreens.c:171 #19 (closed) 0x00005555555b3764 in dix_main (argc=2, argv=0x7fffffffe058, envp=0x7fffffffe070) at main.c:329 #20 (closed) 0x0000555555593ff2 in main (argc=2, argv=0x7fffffffe058, envp=0x7fffffffe070) at stubmain.c:34

Nouveau is destroying its rotate_pixmap during the CloseScreen call, which leads to EXA trying to retrieve its screen-specific private data from the pixmap, but that triggers the assert because the "initialized" flag on the ExaScreenPrivRec::pixmapPrivateKeyRec key is FALSE.

The key was properly initialized, but the flag was then set back to FALSE by dixFreeScreenSpecificPrivates function just before the CloseScreen call.

This behavior was introduced by commit 82eb490b "privates: Clear screen-specific keys during CloseScreen". The reasons why it should be called before CloseScreen is described in here: https://patchwork.freedesktop.org/patch/59716/ and https://patchwork.freedesktop.org/patch/59807/

In short: The initialized flag must be set to FALSE so that the key is registered again after the restart. It can not be done after CloseScreen, because some of the registered keys may be deallocated in CloseScreen. So it is done before, which however means that no screen-specific privates can be accessed in CloseScreen. The above example with nouveau and its rotate_pixmap shows that sometimes it is needed. I expect that there are more situations when it is needed - there are few older bugs that mention the same assert.

I think proper fix is to track the initialized state bit differently. I'll attach two possible solutions.

Version: git