Overlay colorkey always painted to root window

[Yuan, Shengquan]

Hi,

In current Xorg graphics drivers, including SIS, intel, ATI, MGA,
overlay color key is always painted to root window. The driver may
call Xserver helper function xf86XVFillKeyHelper, or write one similar
function like Intel i830_fill_colorkey. In these functions, the color
key is drawn into root window. It causes some issues when interact
with some window managers (e.g. Moblin UX):
(1) even the overlay window hasn't been brought to frontend, overlay
will penetrate through the root window and display on screen
(2) if the overlay window in frontend, window manager may periodically
override the color key using the overlay window content, it causes
overlay lost.

Painting the color key into the overlay window instead of root window
should avoid above issues, e.g, we can patch Xserver function
xf86XVFillKeyHelper like this:

diff --git a/hw/xfree86/common/xf86xv.c b/hw/xfree86/common/xf86xv.c
index abbe033..9e3fe0f 100644
--- a/hw/xfree86/common/xf86xv.c
+++ b/hw/xfree86/common/xf86xv.c
@@ -1869,9 +1869,8 @@ xf86XVFillKeyHelperDrawable (DrawablePtr pDraw,
CARD32 key, RegionPtr clipboxes)
 }

 void
-xf86XVFillKeyHelper (ScreenPtr pScreen, CARD32 key, RegionPtr clipboxes)
+xf86XVFillKeyHelper (ScreenPtr pScreen, CARD32 key, RegionPtr
clipboxes, DrawablePtr pDraw)
 {
-   DrawablePtr root = &WindowTable[pScreen->myNum]->drawable;
    XID pval[2];
    BoxPtr pbox = REGION_RECTS(clipboxes);
    int i, nbox = REGION_NUM_RECTS(clipboxes);
@@ -1880,23 +1879,23 @@ xf86XVFillKeyHelper (ScreenPtr pScreen, CARD32
key, RegionPtr clipboxes)

    if(!xf86Screens[pScreen->myNum]->vtSema) return;

-   gc = GetScratchGC(root->depth, pScreen);
+   gc = GetScratchGC(pDraw->depth, pScreen);
    pval[0] = key;
    pval[1] = IncludeInferiors;
    (void) ChangeGC(gc, GCForeground|GCSubwindowMode, pval);
-   ValidateGC(root, gc);
+   ValidateGC(pDraw, gc);

    rects = xalloc (nbox * sizeof(xRectangle));

    for(i = 0; i < nbox; i++, pbox++)
    {
-      rects[i].x = pbox->x1;
-      rects[i].y = pbox->y1;
+      rects[i].x = pbox->x1 - pDraw->x;
+      rects[i].y = pbox->y1 - pDraw->y;
       rects[i].width = pbox->x2 - pbox->x1;
       rects[i].height = pbox->y2 - pbox->y1;
    }

-   (*gc->ops->PolyFillRect)(root, gc, nbox, rects);
+   (*gc->ops->PolyFillRect)(pDraw, gc, nbox, rects);

    xfree (rects);
    FreeScratchGC (gc);

Overlay window is passed in by the additional parameter "pDraw", and
modify graphics driver accordingly.

Thanks
Austin