X Gesture Extension protocol - draft proposal v1

[Peter Hutterer]

On Thu, Aug 19, 2010 at 11:35:07AM -0400, Chase Douglas wrote:
> On Thu, 2010-08-19 at 09:10 +1000, Peter Hutterer wrote:
> > On Wed, Aug 18, 2010 at 05:02:57PM -0400, Chase Douglas wrote:
> > > On Wed, 2010-08-18 at 21:54 +0200, Simon Thum wrote:
> > > This is another reason why I like to decouple the GE from X: it allows
> > > people to play with touch to gesture assignment. 
> > 
> > but aren't you coupling GE and X with this protocol? apps now must go
> > through the GE picked by the server* and take it or leave it.
> > 
> > * yeah, not 100% true but you get the point
> 
> By adding the protocol to the server, we just provide a mechanism for
> gesture recognition if people want it. If you don't have recognition and
> you want it, you can just install a recognizer package and it will run
> when you start X. If you have recognition and you don't want it, you can
> remove the recognizer or stop it from starting up. If people want, we
> could define an option so that you could forbid gesture recognizers from
> registering.

now I'm confused again. can you confirm the following:
- there's only one GE at a time. the GE is per server, not per client.
- if one GE is registered, clients have to take it or leave it, because
  replacing it means replacing it for all other clients.
  (so if e.g. GNOME and KDE decide to implement two different ones, then
  mixing the two will be iffy.)

how do I "stop the GE from starting up" if another client already needs
it? 
I seem to be missing something here.

> Now, once a gesture engine is registered to the server it is an extra
> code path for all inputs to all clients. However, there's no way around
> that if you want to support environment/window manager gestures.
> 
> If this is all terribly wrong-headed and you are right that gestures
> should be done above X, we haven't really caused much issue by
> attempting this. Just disable gesture support through X as I described
> above and use a recognizer in a toolkit or some other client-side
> library.

But if we disable the GE in the server and use a client-side library, what
do we have the server one for (and the protocol support, including all the code
pieces required in the server)?

> > > > > When the engine recognizes a gesture primitive, it sends a gesture event to the
> > > > > server with the set of input event sequence numbers that comprise the gesture
> > > > > primitive. The server then selects and propagates the gesture event to clients.
> > > > > If clients were selected for propagation, the input events comprising the
> > > > > gesture primitive are discarded. Otherwise, the input events are released to
> > > > > propagate through to clients as normal XInput events.
> > > > I understand what you want to achieve, but I'd argue that apps shouldn't
> > > > be listening to xinput events when they register for gestures. Or at
> > > > least, only in properly constrained areas. Think of a mouse/pad gesture
> > > > detection - how to avoid the latency implied by that approach?
> > > 
> > > I'm not sure I understand your last sentence, but I'll address the rest.
> > > 
> > > It may be true that a client should choose whether to receive only
> > > gestures or only XInput events on a given window, but I'm not sure. X
> > > was designed to be as flexible as possible, and leave policy up to the
> > > toolkits and libraries that sit on top of X (or so wikipedia tells
> > > me :). This mechanism is following that spirit.
> > > 
> > > Beyond that, It sounds like we're of the same mindset here. You "argue
> > > that apps shouldn't be listening to xinput events when they register for
> > > gestures." This protocol discards XInput events when a gesture is
> > > recognized and an event is sent to a client. I hope I haven't misread
> > > anything :).
> > 
> > You need to clearly define whether this is a "shouldn't" or a "mustn't",
> > because only the latter is something you can safely work with. In
> > particular, you need to define what happens to XI events creating a gesture
> > if a client has a grab on a particular device.
> 
> I don't think it's a "mustn't", and as you note we shouldn't get into
> "shouldn't"s, so I don't state any particular thoughts about it in the
> protocol. I don't see any reason why an app must listen only to gestures
> or only to multitouch events. I don't want to block off an avenue of
> input events if a client is written well enough to take advantage of
> both types of events in the same window.

ok, then this runs into the problem of the GE hogging the events in some
cases (think of that google maps use-case in the last email). The GE will
always zoom, even if both of my fingers were on manipulatable objects and
all I wanted to do was move those.

I don't see yet how you can register for both touch and gesture events
without running into these issues.

> As for defining what happens to events when a client has a grab on a
> device, I do need to state that in the protocol. As I noted in another
> thread, I think XI active grabs take precedence, so I'll state that
> formally in the next revision of the protocol unless there are
> objections. I think gestures should take precedence over passive grabs
> though.

> > > I don't fully understand your last sentence, but I will try to address
> > > latency concerns. I think our current gesture recognition code is less
> > > than 500 lines of code (maybe more near 300 lines, Henrik wrote it and
> > > has more details if you are interested). Obviously, you can do a lot in
> > > a small amount of code to kill latency, but I think Henrik has crafted a
> > > tight and fast algorithm. I was skeptical about latency at first too,
> > > but human interfaces being what they are, we should have plenty of cpu
> > > cycles to do all the gesture primitive recognition we need (please don't
> > > read this and assume we're pegging the processors either :). So far in
> > > testing, I haven't seen any noticeable delay, but it's still rather
> > > early in development.
> > 
> > I don't think the algorithm is what's holding you back anyway, it's the
> > nature of gestures and human input in general.Even if you GE is
> > instantaneous in the recognition, you may not know for N milliseconds if the
> > given input may even translate into a gesture. Example: middle mouse button
> > emulation code - you can't solve it without a timeout.
> 
> Yes that's true. This is an area where I think we need to do some
> research and user testing once we have the foundation implemented. As
> for specifics, I feel Henrik is more qualified to respond. I'll just
> note that in my own testing I think Henrik's implementation works well
> all around. We'll have more useful experiences once the Unity window
> manager is fully integrated and we can test it out. That should occur
> within the next week or two.

how many real multitouch applications do you have? ones that don't use
gestures but direct touch interaction?

Cheers,
  Peter