[PATCH xserver 1/5] modesetting: Don't call xf86HandleColorMaps() at screen depth 30.

[Mario Kleiner]

As it turns out, doing so will make any gamma table updates
silently fail, because xf86HandleColorMaps() hooks the
.LoadPalette function to xf86RandR12LoadPalette() if the
.gamma_set function is supported by the ddx, as is in our
case.

Once xf86RandR12LoadPalette() has been called during server
startup, all palette and/or gamma table updates go through
xf86RandR12CrtcComputeGamma() to combine color palette
updates with gamma table updates into a proper hardware lut
for upload into the hw via the .gamma_set function/ioctl,
passing in a palette with palette_red/green/blue_size == the
size given by the visuals red/green/blueMask, which will
be == 1024 for a depth 30 screen.

That in turn is a problem, because the size of the hw lut
crtc->gamma_size is fixed to 256 slots on all kms-drivers
when using the legacy gamma_set ioctl, but
xf86RandR12CrtcComputeGamma() can only handle palettes of a
size <= the hw lut size. As the palette size of 1024 is greater
than the hw lut size of 256, the code silently fails
(gamma_slots == 0 in xf86RandR12CrtcComputeGamma()).

Skipping xf86HandleColormaps() on a depth > 24 screen disables
color palette handling, but keeps at least gamma table updates
via xf86vidmode extension and RandR working.

Signed-off-by: Mario Kleiner <mario.kleiner.de at gmail.com>
---
 hw/xfree86/drivers/modesetting/drmmode_display.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/hw/xfree86/drivers/modesetting/drmmode_display.c b/hw/xfree86/drivers/modesetting/drmmode_display.c
index 9c38563..9b4efe5 100644
--- a/hw/xfree86/drivers/modesetting/drmmode_display.c
+++ b/hw/xfree86/drivers/modesetting/drmmode_display.c
@@ -2265,10 +2265,15 @@ drmmode_setup_colormap(ScreenPtr pScreen, ScrnInfoPtr pScrn)
     xf86DrvMsgVerb(pScrn->scrnIndex, X_INFO, 0, "Initializing kms color map\n");
     if (!miCreateDefColormap(pScreen))
         return FALSE;
-    /* all radeons support 10 bit CLUTs */
-    if (!xf86HandleColormaps(pScreen, 256, 10, drmmode_load_palette, NULL,
-                CMAP_PALETTED_TRUECOLOR |
-                CMAP_RELOAD_ON_MODE_SWITCH))
+
+    /* colormap handling can not handle depth 30 visuals if the hw lut only
+     * has 256 slots, so disable it for screen depth 30.
+     * For non-depth 30, use sigRgbBits 10, as all radeons support 10 bit CLUTs.
+     */
+    if (pScrn->depth != 30 &&
+        !xf86HandleColormaps(pScreen, 256, 10, drmmode_load_palette, NULL,
+                             CMAP_PALETTED_TRUECOLOR |
+                             CMAP_RELOAD_ON_MODE_SWITCH))
         return FALSE;
     return TRUE;
 }
-- 
2.7.4