[PATCH xserver 2/2 v4] xwayland: Apply output rotation for screen size

[Olivier Fourdan]

Previously, we would swap the width/height of the Xwayland output based
on the output rotation, so that the overall screen size would match the
actual rotation of each output.

Problem is the RandR's ConstrainCursorHarder() handler will also apply
the output rotation, meaning that when the output is rotated, the
pointer will be constrained within the wrong dimension.

Moreover, XRandR assumes the original output width/height are unchanged
when the output is rotated, so by changing the Xwayland output width and
height based on rotation, Xwayland causes XRandr to report the wrong
output sizes (an output of size 1024x768 rotated left or right should
remain 1024x768, not 768x1024).

So to avoid this issue and keep things consistent between Wayland and
Xwayland outputs, leave the actual width/height unchanged but apply the
rotation when computing the screen size. This fixes both the output size
being wrong in "xrandr -q" and the pointer being constrained in the
wrong dimension with rotated with weston.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=99663
Signed-off-by: Olivier Fourdan <ofourdan at redhat.com>
---
 v3: Split the patch in two as there two issues. The second issue being
     the pointer events not being reported when a rotation is applied
     this is because of the RR ConstrainCursorHarder() handler that we
     should not need in Xwayland.
 v4: Do not disable the ConstrainCursorHarder() handler set by
     RRScreenInit(), simply leave the oupout size unchanged and apply 
     the rotation only when needed to compute the overall screen size...

 hw/xwayland/xwayland-output.c | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/hw/xwayland/xwayland-output.c b/hw/xwayland/xwayland-output.c
index bdf270a..7d6c7b0 100644
--- a/hw/xwayland/xwayland-output.c
+++ b/hw/xwayland/xwayland-output.c
@@ -108,14 +108,8 @@ output_handle_mode(void *data, struct wl_output *wl_output, uint32_t flags,
     if (!(flags & WL_OUTPUT_MODE_CURRENT))
         return;
 
-    if (xwl_output->rotation & (RR_Rotate_0 | RR_Rotate_180)) {
-        xwl_output->width = width;
-        xwl_output->height = height;
-    } else {
-        xwl_output->width = height;
-        xwl_output->height = width;
-    }
-
+    xwl_output->width = width;
+    xwl_output->height = height;
     xwl_output->refresh = refresh;
 }
 
@@ -123,11 +117,21 @@ static inline void
 output_get_new_size(struct xwl_output *xwl_output,
                     int *height, int *width)
 {
-    if (*width < xwl_output->x + xwl_output->width)
-        *width = xwl_output->x + xwl_output->width;
+    int output_width, output_height;
+
+    if (xwl_output->rotation & (RR_Rotate_0 | RR_Rotate_180)) {
+        output_width = xwl_output->width;
+        output_height = xwl_output->height;
+    } else {
+        output_width = xwl_output->height;
+        output_height = xwl_output->width;
+    }
+
+    if (*width < xwl_output->x + output_width)
+        *width = xwl_output->x + output_width;
 
-    if (*height < xwl_output->y + xwl_output->height)
-        *height = xwl_output->y + xwl_output->height;
+    if (*height < xwl_output->y + output_height)
+        *height = xwl_output->y + output_height;
 }
 
 /* Approximate some kind of mmpd (m.m. per dot) of the screen given the outputs
-- 
2.9.3