[PATCH] randr: first check pScrPriv before using the pointer at RRFirstOutput

[Ricardo Salveti]

On Thu, Jun 21, 2012 at 2:12 AM, Keith Packard <keithp at keithp.com> wrote:
> Ricardo Salveti de Araujo <ricardo.salveti at linaro.org> writes:
>
>> Fix a seg fault in case pScrPriv is NULL at ProcRRGetScreenInfo,
>> which later calls RRFirstOutput.
>
> Do you have a stack trace for this case? It "shouldn't" ever happen, I
> don't think.

This bug happened once I was using the omap driver for Pandaboard,
which for another bug/reason, was initializing both the omap and fbdev
and creating two displays, but only one was initializing randr
properly (the omap one).

Then at userspace, when I was testing Qt 5, it crashed X once it tried
to open a window on all displays available, giving the following stack
trace:

Program received signal SIGSEGV, Segmentation fault.
RRFirstOutput (pScreen=0x2a1880c0) at randr.c:458
458	 if (pScrPriv->primaryOutput && pScrPriv->primaryOutput->crtc)
(gdb) bt full
#0 RRFirstOutput (pScreen=0x2a1880c0) at randr.c:458
        pScrPriv = 0x0
        output = <optimized out>
        i = <optimized out>
        j = <optimized out>
#1 0x2a0a5834 in ProcRRGetScreenInfo (client=0x2a233f08) at rrscreen.c:615
        stuff = <optimized out>
        rep = {type = 232 '\350', setOfRotations = 66 'B',
sequenceNumber = 10777, length = 705951608, root = 3204445656,
timestamp = 706298576, configTimestamp = 704846133, nSizes = 0, sizeID
= 0,
          rotation = 63348, rate = 10771, nrateEnts = 16136, pad = 0}
        pWin = 0x2a1b0258
        n = <optimized out>
        rc = 0
        pScreen = 0x2a1880c0
        pScrPriv = 0x0
        extra = <optimized out>
        extraLen = <optimized out>
        output = <optimized out>
#2 0x2a09f456 in ProcRRDispatch (client=<optimized out>) at randr.c:493
        stuff = <optimized out>
#3 0x2a02eb76 in Dispatch () at dispatch.c:442
        clientReady = 0x2a2e81f0
        result = 0
        client = 0x2a233f08
        nready = 0
        icheck = 0x2a1430c0
        start_tick = 100
#4 0x2a0242ce in main (argc=3, argv=0xbefff824, envp=<optimized out>)
at main.c:287
        i = <optimized out>
        alwaysCheckForInput = {0, 1}

After digging up on similar bugs, I also found that this issue also
happens when you're trying to use 2 different cards with xorg, trying
to load 2 different displays. The thread
http://lists.freedesktop.org/archives/libdlo/2010-March/000616.html is
one example. In my case that's probably because fbdev, once loaded for
the "second" card, is not creating pScrPriv for randr, making X to
crash later on.

I agree that it would also be ideal to fix the bugs on a case by case,
which is why I'm now looking at why X is loading both omap and fbdev
at the same time (probably because there's no bus to claim to control
if it should be initialized or not)

As protecting the pointer is not a big of a deal, and always useful, I
decided to send this patch first (as it also make Qt 5 to work when
both omap and fbdev are loaded at the same time).

Cheers,
-- 
Ricardo Salveti de Araujo