[PATCH] xfree86/modes: Take rotation into account when checking mode size

[Ville Syrjälä]

On Thu, Dec 16, 2010 at 01:50:25PM +0200, ext ville.syrjala at nokia.com wrote:
> From: Ville Syrjälä <ville.syrjala at nokia.com>
> 
> Assume that a mode can be used in either landscape or portrait
> orientation. I suppose the correct thing to do would be to collect all
> the supported rotations from the CRTCs that can be used with a specific
> output, but that information doesn't seem to be readily available when
> these checks are done. So just assume that either orientation is fine.
> 
> Signed-off-by: Ville Syrjälä <ville.syrjala at nokia.com>
> ---
>  hw/xfree86/modes/xf86Modes.c |    9 ++++++---
>  1 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/xfree86/modes/xf86Modes.c b/hw/xfree86/modes/xf86Modes.c
> index 05f4319..26db2c2 100644
> --- a/hw/xfree86/modes/xf86Modes.c
> +++ b/hw/xfree86/modes/xf86Modes.c
> @@ -365,13 +365,16 @@ xf86ValidateModesSize(ScrnInfoPtr pScrn, DisplayModePtr modeList,
>      DisplayModePtr mode;
>  
>      for (mode = modeList; mode != NULL; mode = mode->next) {
> -	if (maxPitch > 0 && mode->HDisplay > maxPitch)
> +	if (maxPitch > 0 && xf86ModeWidth(mode, RR_Rotate_0) > maxPitch &&
> +	    xf86ModeWidth(mode, RR_Rotate_90) > maxPitch)
>  	    mode->status = MODE_BAD_WIDTH;
>  
> -	if (maxX > 0 && mode->HDisplay > maxX)
> +	if (maxX > 0 && xf86ModeWidth(mode, RR_Rotate_0) > maxX &&
> +	    xf86ModeWidth(mode, RR_Rotate_90) > maxX)
>  	    mode->status = MODE_VIRTUAL_X;
>  
> -	if (maxY > 0 && mode->VDisplay > maxY)
> +	if (maxY > 0 && xf86ModeHeight(mode, RR_Rotate_0) > maxY &&
> +	    xf86ModeHeight(mode, RR_Rotate_90) > maxY)
>  	    mode->status = MODE_VIRTUAL_Y;

And of course that logic is totally broken. My brain wasn't in gear
when I wrote this. I'll cook up a patch that actually works.

>  
>  	if (mode->next == modeList)
> -- 
> 1.7.2.2
> 
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel

-- 
Ville Syrjälä