[RFC PATCH xserver 0/5] Check glamor-xv usability

[Olivier Fourdan]

It appears that on some hardware/diver combo such as nv30/nouveau, using
GL_ALPHA as format for 8-bit depth will cause an incomplete attachment 
error (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT) when trying to bind the
texture.

As a result, the FBO is NULL and glamor segfaults when trying to access
the FBO width/height in pixmap_priv_get_scale().

This happens with glamor-xv which uses a 8bit pixmap, meaning that on such
hardware/driver, trying to play a video using Xv will lead to a crash of
the Xserver. This affects Xwayland, Xephyr and modesetting driver with
glamor accel:

  https://bugs.freedesktop.org/show_bug.cgi?id=100710
  https://bugzilla.redhat.com/1412814

So this series of RFC patches consist of 3 things:

 1. Check if using a 8bit pixmap causes a incomplete attachment error as
    soon as glamor_xv_core_init() and return a boolean value indicating
    the caller whether or not glamor-xv is usable.
 2. Modify each callers, namely ephyr, modesetting and xwayland to take
    advantage of this boolean value and do not enable xv if not usable.
    One difficulty here is that to check FBO usability, one has to have
    the screen resources created, which means that the changes also
    include moving glamor_xv_core_init() to CreateScreenResources()
    instead of screen init.
 3. Last patch is problably even more controversial, as I found that
    using GL_RGBA for those 8bit format actually works for this hardware,
    even though this is a clear waste of resources. Thins is, though,
    it makes glamor-xv perfectly usable, not only it doesn't crash but
    is's also reasonably fast even on this old hardware. Other thing to
    consider here is that modern hardware won't use than anyway and prefer
    GL_RED and texture swizzle, so for those, this patch makes no difference.
    Now, I am not sure who that flies with other "older" hardware, but
    given point 1. and 2. even if it doesn't work, it won't crash (at 
    least)...

Cheers,
Olivier


Olivier Fourdan (5):
  glamor-xv: Check that FBO is usable
  modesetting: Check that glamor-xv is usable
  Xephyr: Check that glamor-xv is usable
  Xwayland: Check that glamor-xv is usable
  glamor: Use GL_RGBA format for 8-bit depth

 glamor/glamor.c                         |  2 ++
 glamor/glamor_priv.h                    |  3 ++-
 glamor/glamor_utils.h                   | 11 ++++++-----
 glamor/glamor_xv.c                      | 23 ++++++++++++++++++++++-
 hw/kdrive/ephyr/ephyr.c                 | 16 ++++++++++++++--
 hw/kdrive/ephyr/ephyr.h                 |  5 +++--
 hw/kdrive/ephyr/ephyr_glamor_xv.c       |  7 ++++---
 hw/xfree86/drivers/modesetting/driver.c | 26 +++++++++++++-------------
 hw/xfree86/glamor_egl/glamor_xf86_xv.c  |  3 ++-
 hw/xwayland/xwayland-glamor-xv.c        |  3 ++-
 hw/xwayland/xwayland-glamor.c           | 11 ++++++-----
 11 files changed, 76 insertions(+), 34 deletions(-)

-- 
2.9.3