[PATCH xserver 1/2] randr: Fix rotation check in ProcRRSetScreenSize()

[Alex Goins]

It sounds like this patch is the one we want. Can it be merged?

Thanks,
Alex

On Wed, 10 Jan 2018, Michel Dänzer wrote:

> On 2018-01-09 03:44 AM, Alex Goins wrote:
> > ProcRRSetScreenSize() does bounds checking to ensure that none of the CRTCs have
> > a viewport that extends beyond the new screen size. In doing so, it accounts for
> > if the CRTC is rotated 90 or 270 degrees, swapping width and height.
> > 
> > However, it does so by testing if crtc->rotation is equal to RR_Rotate_90 or
> > RR_Rotate_270. crtc->rotation is a bit mask, and it includes reflection as well
> > as rotation. If a CRTC is reflected as well as rotated, it will incorrectly fail
> > this test, resulting in incorrect dimensions being used to verify the validity
> > of the new screen size. In some cases, this can cause valid uses of
> > ProcRRSetScreenSize() to fail with BadMatch.
> > 
> > This patch fixes the issue by testing that the bits RR_Rotate_90 or
> > RR_Rotate_270 are set, rather than testing for equality.
> > 
> > Signed-off-by: Alex Goins <agoins at nvidia.com>
> > ---
> >  randr/rrscreen.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/randr/rrscreen.c b/randr/rrscreen.c
> > index d6c4995..f484383 100644
> > --- a/randr/rrscreen.c
> > +++ b/randr/rrscreen.c
> > @@ -272,7 +272,7 @@ ProcRRSetScreenSize(ClientPtr client)
> >              int source_height = mode->mode.height;
> >              Rotation rotation = crtc->rotation;
> >  
> > -            if (rotation == RR_Rotate_90 || rotation == RR_Rotate_270) {
> > +            if (rotation & (RR_Rotate_90 | RR_Rotate_270)) {
> >                  source_width = mode->mode.height;
> >                  source_height = mode->mode.width;
> >              }
> > 
> 
> Reviewed-by: Michel Dänzer <michel.daenzer at amd.com>
> 
> 
> -- 
> Earthling Michel Dänzer               |               http://www.amd.com
> Libre software enthusiast             |             Mesa and X developer
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: https://lists.x.org/mailman/listinfo/xorg-devel