[PATCH] xrandr: move transform limit checking after scaling

[Keith Packard]

<#part sign=pgpmime>
On Tue, 20 Mar 2012 16:54:52 -0700, "Pierre-Loup A. Griffais" <pgriffais at nvidia.com> wrote:
> Wasted a good chunk of time on that one.
> From f1a65814f119cb83d16e15093e81946f3adcde61 Mon Sep 17 00:00:00 2001
> From: Pierre-Loup A. Griffais <pgriffais at nvidia.com>
> Date: Tue, 20 Mar 2012 16:46:22 -0700
> Subject: [PATCH] xrandr: move transform limit checking after scaling
> 
> This would trigger for legit scaled matrices, resulting in the wrong
> extents getting computed.
> 
> Signed-off-by: Pierre-Loup A. Griffais <pgriffais at nvidia.com>
> ---
>  xrandr.c |    7 ++++---
>  1 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/xrandr.c b/xrandr.c
> index 35dff3e..622f6a9 100644
> --- a/xrandr.c
> +++ b/xrandr.c
> @@ -432,14 +432,15 @@ transform_point (XTransform *transform, double *xp, double *yp)
>  	v = 0;
>  	for (i = 0; i < 3; i++)
>  	    v += (XFixedToDouble (transform->matrix[j][i]) * vector[i]);
> -	if (v > 32767 || v < -32767)
> -	    return False;
>  	result[j] = v;
>      }
>      if (!result[2])
>  	return False;
> -    for (j = 0; j < 2; j++)
> +    for (j = 0; j < 2; j++) {
>  	vector[j] = result[j] / result[2];
> +        if (vector[j] > 32767 || vector[j] < -32767)
> +            return False;
> +    }

Yeah, that looks right - transform from homogeneous back to pixel
coordinates before doing the range check.

Reviewed-by: Keith Packard <keithp at keithp.com>

(sorry I didn't see this on Tuesday, thanks for the ping!)

-- 
keith.packard at intel.com