[PATCH xserver 1/2] glamor: glamor_set_destination_drawable() can fail

[Olivier Fourdan]

The fbo_array of a given glamor pixmap can be NULL in some cases, as
glamor_create_fbo_array() can fail to allocate the FBO array.

If this is the case, glamor_pixmap_fbo_at() will return NULL even though
the box index is valid, and glamor_set_destination_drawable() simply
assumes glamor_pixmap_fbo_at() will return an FBO prior to pass the
value to glamor_set_destination_pixmap_fbo(), which will segfault.

We need a way for glamor_set_destination_drawable() to fail safely and
let the caller know about the failure.

Add a boolean return value to glamor_set_destination_drawable() for that
purpose.

Bugzilla: https://bugzilla.redhat.com/1417575
Signed-off-by: Olivier Fourdan <ofourdan at redhat.com>
---
 glamor/glamor_transform.c | 11 +++++++++--
 glamor/glamor_transform.h |  2 +-
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/glamor/glamor_transform.c b/glamor/glamor_transform.c
index eff500c..2d5a634 100644
--- a/glamor/glamor_transform.c
+++ b/glamor/glamor_transform.c
@@ -33,7 +33,7 @@
  * clipping computations can be adjusted as appropriate
  */
 
-void
+Bool
 glamor_set_destination_drawable(DrawablePtr     drawable,
                                 int             box_index,
                                 Bool            do_drawable_translate,
@@ -53,6 +53,11 @@ glamor_set_destination_drawable(DrawablePtr     drawable,
     float scale_x = 2.0f / (float) w;
     float scale_y = 2.0f / (float) h;
     float center_adjust = 0.0f;
+    glamor_pixmap_fbo *pixmap_fbo;
+
+    pixmap_fbo = glamor_pixmap_fbo_at(pixmap_priv, box_index);
+    if (!pixmap_fbo)
+        return FALSE;
 
     glamor_get_drawable_deltas(drawable, pixmap, &off_x, &off_y);
 
@@ -94,8 +99,10 @@ glamor_set_destination_drawable(DrawablePtr     drawable,
                 scale_x, (off_x + center_adjust) * scale_x - 1.0f,
                 scale_y, (off_y + center_adjust) * scale_y - 1.0f);
 
-    glamor_set_destination_pixmap_fbo(glamor_priv, glamor_pixmap_fbo_at(pixmap_priv, box_index),
+    glamor_set_destination_pixmap_fbo(glamor_priv, pixmap_fbo,
                                       0, 0, w, h);
+
+    return TRUE;
 }
 
 /*
diff --git a/glamor/glamor_transform.h b/glamor/glamor_transform.h
index 70d2c16..28855e3 100644
--- a/glamor/glamor_transform.h
+++ b/glamor/glamor_transform.h
@@ -23,7 +23,7 @@
 #ifndef _GLAMOR_TRANSFORM_H_
 #define _GLAMOR_TRANSFORM_H_
 
-void
+Bool
 glamor_set_destination_drawable(DrawablePtr     drawable,
                                 int             box_index,
                                 Bool            do_drawable_translate,
-- 
2.9.3