[PATCH] xserver: Don't crash if Xv is not initialized

[Dave Airlie]

On Tue, Feb 8, 2011 at 8:39 PM, Mauro Carvalho Chehab
<mchehab at infradead.org> wrote:
> The xf86-video-v4l video driver calls xf86XVQueryOffscreenImages()
> function in order to probe for the Xv FOURCC formats supported for
> PutVideo ops. However, as this support is deprecated on most of
> the modern drivers, a call to this method will cause a crash:
>
> X: ../../../include/privates.h:115: dixGetPrivateAddr: Assertion `key->initialized' failed.
>
> The reason is that this function calls dixGetPrivate in order to
> locate the map. At include/privates.h, this is declared as:
>
> static inline void *
> dixGetPrivateAddr(PrivatePtr *privates, const DevPrivateKey key)
> {
>    assert(key->initialized);
>    return (char *) (*privates) + key->offset;
> }
>
> As nobody registered it, key->initialized is FALSE, and assert will
> complain, causing the crash.
>
> More information is available on this bugzilla:
>        https://bugzilla.redhat.com/show_bug.cgi?id=675532
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab at redhat.com>
>

I was going to give out about this being a big ugly looking inside the privates,
but the RegisterOffscreenImages function pretty much has a comment on why.

So

Reviewed-by: Dave Airlie <airlied at redhat.com>


> diff --git a/hw/xfree86/common/xf86xv.c b/hw/xfree86/common/xf86xv.c
> index 016db1f..858d3bc 100644
> --- a/hw/xfree86/common/xf86xv.c
> +++ b/hw/xfree86/common/xf86xv.c
> @@ -218,7 +218,14 @@ xf86XVQueryOffscreenImages(
>    ScreenPtr pScreen,
>    int *num
>  ){
> -    OffscreenImageRec *OffscreenImage = GetOffscreenImage(pScreen);
> +    OffscreenImageRec *OffscreenImage;
> +
> +    if (!OffscreenPrivateKey->initialized) {
> +       *num = 0;
> +       return NULL;
> +    }
> +
> +    OffscreenImage = GetOffscreenImage(pScreen);
>     *num = OffscreenImage->num;
>     return OffscreenImage->images;
>  }
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>