[RFC] Multitouch proposal v2 + implementation

[Peter Hutterer]

On Tue, Sep 21, 2010 at 11:30:07AM +0200, Benjamin Tissoires wrote:
> Le 21/09/2010 06:42, Daniel Stone a écrit :
> >Hi,
> >
> >On Mon, Sep 20, 2010 at 04:54:37PM +0200, Benjamin Tissoires wrote:
> >>Le 20/09/2010 06:45, Daniel Stone a écrit :
> >>>During a touchpoint's short lifetime, all motion events will be sent to
> >>>one client and one client only.  Only TouchBegin rather than TouchMotion
> >>>events can be selected for or grabbed, and akin to ButtonPress, only one
> >>>client per window can select for TouchBegin events.  Once a delivery has
> >>>been made to a client without a synchronous grab, all TouchMotion events
> >>>and the TouchFini event will be sent to the same client.
> >>
> >>It seems to me, that for feedback reasons, Peter suggested to send
> >>the events in case of a passive grab to all the stack of clients (2
> >>seemed to be the common case) and not to only one: the underlying
> >>clients can start gesture recognition, early feedbacks even if they
> >>know that this particular touch may not be their but will be
> >>retained by the Window Manager.
> >>
> >>That's all I wanted to add. :-)
> >
> >Hm, do you mean support for replaying events a la XISyncDevice for
> >normal grabs, or do you mean for sending touchstreams to all clients who
> >have selected on a particular window?
> 
> The point here is that I may not be skilled enough to be clear
> enough. I hope that Peter will be able to answer soon....
> 
> From what I understood, in X, there are two types of grabs: active
> and passive.
> In case of an active grabs (even though I still not managed to
> understand it in case of a multitouch device -> an active grab can
> not be on the device itself, but on only the touches in my opinion),
> the server delivers only one queue of events (normal behaviour) per
> touch.
> In case of a passive grab (from what I understood, same think of a
> button press - move - release), Peter suggested to send the events
> to all the clients in the stack. However, only the top most one will
> have the flag that the event is for it. The others will receive the
> events, but with a bit flag that mainly consists in saying: "hey,
> this event stream is currently occurring, but is still not yours".
> The point here is that we should rely on the client not to handle it
> even if the event is still not for it.

The idea was brought up by Keith at XDS: in a normal sync passive grab event
delivery with ReplayPointer, we may send the same event to multiple clients
- first all those that passively grab on a window or its ancestors and then
finally to the window. This is a serial delivery with the server queuing up
events until the AllowEvents request comes through.
Keith's suggestion was to deliver the event stream to all clients that may
possibly get this stream, but with a "Do not use" flag set on all but the
topmost passively grabbing client. Then, when the first client calls
AllowEvents with ReplayTouch (or whatever), instead of sending the whole
queue, the server just sends a "you are free to use event stream X" event to
the next client down. Likewise, if the grabbing client does not replay the
stream, a "discard" event can be sent.
This allows for parallel processing of e.g. gestures, removes the need to
queue potentially vast numbers of events in the server but relies on clients
behaving correctly - if they provide UI feedback for an event stream that is
not yet theirs, it gets tricky. UI problem though.
I haven't scoped out all the corner cases yet, so while I like the idea for
the above advantages in the server, I can't say if it's viable.

> >The former I can be talked around into (especially if someone says that
> >it's vital for their usecase), but as for the latter, it's intended to
> >follow ButtonPress semantics for the same reasons: you really don't want
> >two people attempting to act on the same touch stream.  I'd much, much
> >rather be conservative and drop an event rather than possibly have the
> >same touch stream do two things.
> 
> I may not understood, but is XISyncDevice the keyword for the replay
> events? Peter was talking about such thing.
> I agree one event stream should not be handled by two clients at the
> same time, but to reduce the time between the event and the visual
> feedbacks, we may need to break this:
> in case of Qt, there is a gesture recognition engine embedded. If we
> have to retain the event stream for the top most client, wait for it
> to say "ok, this is not a gesture I can handle", then replay it to
> the toolkit (Qt), and wait for it to say "this is a pinch in that
> case", this will infers a delay that will be very irritating for the
> user.
> 
> Sending the event stream to both the clients will reduce this time,
> but we have to place some confidence in the clients though.

Yes, though as Stephane brought up, the avoidance of a delay is mostly a UI
design issue, less so a technical one. The delay _will_ be there, either
way, it's really a matter of designing your gestures in a way that it
doesn't matter.

> >Would a motion history-like API do the trick here? That would let any
> >client get the entire history of a touch stream they have the touchid
> >for.  Or is the problem that gesture recognition is computationally
> >expensive, so you just need to get a head start for latency reasons?
> 
> I think that the motion history-like API will not help here (see above).
> The point is not that the gesture recognition is computationally
> expensive, but that it has to wait for user events to be sure that
> it is wrong or right.... so it is really time consuming

It's a gesture design decision. If you're accepting e.g. five-finger
gestures you will ahave to wait to see if more fingers appear after the
first finger event is received (similar to middle mouse button emulation).
In that time, your UI cannot always react to input since it is not clear if
you're using one, two, three, four, or five fingers. This can lead to a
noticable delay and a laggy UI.

Adding exit conditions into the gesture design (e.g. once a finger moves, it
cannot become a 5-finger gesture anymore) helps avoiding perceived delays
but that's again not a technical problem.

Cheers,
  Peter