[PATCH xserver 1/2] randr: Fix rotation check in ProcRRSetScreenSize()

[Michel Dänzer]

On 2018-01-09 03:44 AM, Alex Goins wrote:
> ProcRRSetScreenSize() does bounds checking to ensure that none of the CRTCs have
> a viewport that extends beyond the new screen size. In doing so, it accounts for
> if the CRTC is rotated 90 or 270 degrees, swapping width and height.
> 
> However, it does so by testing if crtc->rotation is equal to RR_Rotate_90 or
> RR_Rotate_270. crtc->rotation is a bit mask, and it includes reflection as well
> as rotation. If a CRTC is reflected as well as rotated, it will incorrectly fail
> this test, resulting in incorrect dimensions being used to verify the validity
> of the new screen size. In some cases, this can cause valid uses of
> ProcRRSetScreenSize() to fail with BadMatch.
> 
> This patch fixes the issue by testing that the bits RR_Rotate_90 or
> RR_Rotate_270 are set, rather than testing for equality.
> 
> Signed-off-by: Alex Goins <agoins at nvidia.com>
> ---
>  randr/rrscreen.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/randr/rrscreen.c b/randr/rrscreen.c
> index d6c4995..f484383 100644
> --- a/randr/rrscreen.c
> +++ b/randr/rrscreen.c
> @@ -272,7 +272,7 @@ ProcRRSetScreenSize(ClientPtr client)
>              int source_height = mode->mode.height;
>              Rotation rotation = crtc->rotation;
>  
> -            if (rotation == RR_Rotate_90 || rotation == RR_Rotate_270) {
> +            if (rotation & (RR_Rotate_90 | RR_Rotate_270)) {
>                  source_width = mode->mode.height;
>                  source_height = mode->mode.width;
>              }
> 

Reviewed-by: Michel Dänzer <michel.daenzer at amd.com>


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer