This reverts commit 45ba3571.
Sadly, this causes regressions on amdgpu 1 and even with these fixed, there are fundamental issues with non-blocking modesets 2.