Xorg's software-cursor support + mutter results in a mess

[Hans de Goede]

Hi All,

Note I've also filed an issue for tracking this:
https://gitlab.freedesktop.org/xorg/xserver/issues/828
but this seems like something to discuss on the list.

When Xorg fallsback to software cursor support, for example because
of using a DisplayLink USB2 device (udl kernel driver) or another
"dumb" kms framebuffer device and then running mutter as compositor on
top of Xorg moving the cursor around leads to 2 type of artifacts:

1) There is a pointer trail of the last couple of cursor positions
2) Sometimes the background for one of the older cursor echo's is a
    rectangle (aprox 32x32 pixels) from a Window which is no longer
    shown.

First of all let me sat that although this can be reproduced with
mutter, I do not believe that this is a mutter bug.

Looking at the visual artifacts, so observing the system as a
blackbox, I believe that the problem is that the software-cursor
code in Xorg and the code responsible for presenting framebuffers
are not aware of each other. This combined with mutter now a days
only re-rendering frames when something has changed leads to the
following:

1) Mutter uses multiple framebuffers, at least 3 of them.
2) When ever the cursor (position) changes the software-cursor code
draws the cursor over the current framebuffer.

If one then slowly moves the cursor over an otherwise unchanging
desktop, then each frame its get drawn in a different location
resulting in the 3 framebuffers having the cursor in 3 different
places. Since nothing is changing mutter keeps recylcing these
framebuffers without re-rendering them. This results in the cursor
jumping from location a, to b, to c and back to a again as
framebuffer a. b and c are presented by mutter.

Another problem happens with the code to restore the image which was
below the cursor under its previous position.

Lets say we start with frame "a" with an ok / cancel dialog box on it
with the mouse cursor over its ok button. Now the user clicks the button
and after release moves the cursor a bit. The release of the button
results in the dialog closing and the rendering of frame "b", without the
dialog, when then the cursor moves, Xorg restores the image which was below
the cursor (from frame "a" with the dialog box) on the place where part
of the ok button used to be, putting a rectangle with part of the ok
button over frame-b (which no longer has the dialog) and then also rendering
the cursor over frame-b in its new posiiton.

This results in a rectangle with the part of the ok button below the cursor
flickering in and out of the display as mutter recycles the last few frames
since nothing is changing on the desktop.

I don't know which API mutter (3.32 or newer) is using to display its
frames. but it seems a reasonable assumption to me that a frame handed over
to Xorg for displaying is treated as read-only by Xorg and can be re-used
later without needing to re-render it.

Note that even if mutter was rerendering every frame it presents, we would
still get artifacts from the software-cursor code assuming that there is
only 1 framebuffer (or so it seems).

One way to fix this would be to tie the code presenting the frames
(maybe at the ddx driver level?) and the software-cursor code together
so that on a frame-flip we can have the cursor-code undo its rendering
of the cursor on the frame before "giving it back" to the client and
likewise calling the software code to render the cursor over the
framebuffer before handing the framebuffer to the kms-driver / hardware
for scanout.

Another option which I'm seriously considering is to fake hw-cursor support
at the kms driver level.

Regards,

Hans