[PATCH] randr: check rotated virtual size limits correctly

[Jeremy Huddleston]

That looks like it does the right thing... (still... get a Tested-by:)

Reviewed-by: Jeremy Huddleston <jeremyhu at apple.com>

On May 11, 2011, at 7:18 AM, Aaron Plattner wrote:

> Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
> the RandR path that cause RRSetScreenConfig requests to fail if the
> size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
> it compares the rotated screen dimensions to the unrotated limits,
> which causes 90- and 270-degree rotation to fail unless your screen
> happens to be square:
> 
>  X Error of failed request:  BadValue (integer parameter out of range for operation)
>    Major opcode of failed request:  153 (RANDR)
>    Minor opcode of failed request:  2 (RRSetScreenConfig)
>    Value in failed request:  0x780
>    Serial number of failed request:  14
>    Current serial number in output stream:  14
> 
> Fix this by moving the check above the code that swaps the dimensions
> based on the rotation.
> 
> Signed-off-by: Aaron Plattner <aplattner at nvidia.com>
> ---
> This patch is against server-1.10-branch.
> 
> randr/rrscreen.c |   12 ++++++------
> 1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/randr/rrscreen.c b/randr/rrscreen.c
> index 1bc1a9e..da6d48d 100644
> --- a/randr/rrscreen.c
> +++ b/randr/rrscreen.c
> @@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
>      */
>     width = mode->mode.width;
>     height = mode->mode.height;
> -    if (rotation & (RR_Rotate_90|RR_Rotate_270))
> -    {
> -	width = mode->mode.height;
> -	height = mode->mode.width;
> -    }
> -
>     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
> 	client->errorValue = width;
> 	free(pData);
> @@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
> 	return BadValue;
>     }
> 
> +    if (rotation & (RR_Rotate_90|RR_Rotate_270))
> +    {
> +	width = mode->mode.height;
> +	height = mode->mode.width;
> +    }
> +
>     if (width != pScreen->width || height != pScreen->height)
>     {
> 	int	c;
> -- 
> 1.7.4.1
> 
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>