[PATCH xserver 1/2] randr: Fix rotation check in ProcRRSetScreenSize()

[Alex Goins]

ProcRRSetScreenSize() does bounds checking to ensure that none of the CRTCs have
a viewport that extends beyond the new screen size. In doing so, it accounts for
if the CRTC is rotated 90 or 270 degrees, swapping width and height.

However, it does so by testing if crtc->rotation is equal to RR_Rotate_90 or
RR_Rotate_270. crtc->rotation is a bit mask, and it includes reflection as well
as rotation. If a CRTC is reflected as well as rotated, it will incorrectly fail
this test, resulting in incorrect dimensions being used to verify the validity
of the new screen size. In some cases, this can cause valid uses of
ProcRRSetScreenSize() to fail with BadMatch.

This patch fixes the issue by testing that the bits RR_Rotate_90 or
RR_Rotate_270 are set, rather than testing for equality.

Signed-off-by: Alex Goins <agoins at nvidia.com>
---
 randr/rrscreen.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/randr/rrscreen.c b/randr/rrscreen.c
index d6c4995..f484383 100644
--- a/randr/rrscreen.c
+++ b/randr/rrscreen.c
@@ -272,7 +272,7 @@ ProcRRSetScreenSize(ClientPtr client)
             int source_height = mode->mode.height;
             Rotation rotation = crtc->rotation;
 
-            if (rotation == RR_Rotate_90 || rotation == RR_Rotate_270) {
+            if (rotation & (RR_Rotate_90 | RR_Rotate_270)) {
                 source_width = mode->mode.height;
                 source_height = mode->mode.width;
             }
-- 
2.7.4