[PATCH] Fix server crash in pixman (to be discussed)

[Matthias Hopf]

The following patch fixes Novell bug 568811:
  VNC Installation aborts right in the middle due to an assertion in Xvnc/libpixman

The bug seems occur only on *very* special occasions (in this case, only
in SLES, but *not* in SLED, which is based on the same code basis...).

Backtrace looks as follows:

#2  0xb71f7baa in __assert_fail () from /lib/libc.so.6
#3  0xb781feab in pixman_region_append_non_o (y2=<value optimized out>,
    y1=<value optimized out>, r_end=<value optimized out>,
    r=<value optimized out>, region=<value optimized out>) at pixman-region.c:670
#4  pixman_op (y2=<value optimized out>, y1=<value optimized out>,
    r_end=<value optimized out>, r=<value optimized out>,
    region=<value optimized out>) at pixman-region.c:996
#5  0xb7820dbe in pixman_region_union (new_reg=0x82ee57c, reg1=0x82ee57c,
    reg2=0xbfd77c90) at pixman-region.c:1439
#6  0x080f023b in miUnion (newReg=0x82ee57c, reg1=0x82ee57c, reg2=0xbfd77c90)
    at miregion.c:1005
#7  0x0806eebe in rfbComposite (op=12 '\f', pSrc=0x8297c08, pMask=0x0,
    pDst=0x82a7a00, xSrc=0, ySrc=0, xMask=0, yMask=0, xDst=20, yDst=8, width=0,
    height=0) at draw.c:1805
#8  0x0815ddad in CompositePicture (op=12 '\f', pSrc=0x8297c08, pMask=0x0,
    pDst=0x82a7a00, xSrc=0, ySrc=0, xMask=<value optimized out>,
    yMask=<value optimized out>, xDst=<value optimized out>,
    yDst=<value optimized out>, width=0, height=0) at picture.c:1675
#9  0x0815a1bf in miGlyphs (op=3 '\003', pSrc=0x82f0810, pDst=0x82a2808,
    maskFormat=0x822f8b8, xSrc=0, ySrc=0, nlist=1, list=0xbfd782a8,
    glyphs=0xbfd77ebc) at glyph.c:726
#10 0x0815a462 in CompositeGlyphs (op=<value optimized out>, pSrc=0x82f0810,
    pDst=0x82a2808, maskFormat=0x822f8b8, xSrc=0, ySrc=0, nlist=1,
    lists=0xbfd782a8, glyphs=0xbfd77ea8) at glyph.c:632
#11 0x0816741e in ProcRenderCompositeGlyphs (client=0x829a860) at render.c:1462
#12 0x08160895 in ProcRenderDispatch (client=0xfcf) at render.c:2089
#13 0x0809bb47 in Dispatch () at dispatch.c:456
#14 0x080b24ba in main (argc=21, argv=0xbfd78704, envp=Cannot access memory at address 0xfd7

What happens is that an assert in pixman_region_append_non_o() fails,
because the region reg2 is degenerated. PIXREGION_NIL as is doesn't
detect this region as being empty, presumably because this should never
ever happen.

The patch is a workaround by enhancing PIXREGION_NIL to detect
degenerated regions.

However, the real reason is probably shaded by this commit.
Reading the source in render/glyph.c in the Xserver it seems that the
glyph to be rendered is empty (glyph->info.width=glyph->info.height=0).
Question is now whether that is allowed or not.

If it is question remains on which level this should be fixed. If it is
not, question remains why this could happen.

Thanks

Matthias

-- 
Matthias Hopf <mhopf at suse.de>      __        __   __
Maxfeldstr. 5 / 90409 Nuernberg   (_   | |  (_   |__          mat at mshopf.de
Phone +49-911-74053-715           __)  |_|  __)  |__  R & D   www.mshopf.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Improve-PIXREGION_NIL-to-return-true-on-degenerated.patch
Type: text/x-patch
Size: 1017 bytes
Desc: not available
URL: <http://lists.x.org/archives/xorg/attachments/20100324/f22e541f/attachment.bin>