Multitouch followup: gesture recognition?

[Simon Thum]

Am 24.03.2010 07:36, schrieb Peter Hutterer:
> Only replying to one email, answers to both of your comments.
> 
> On Wed, Mar 24, 2010 at 12:06:30AM +0100, Simon Thum wrote:
>> [CC'ing Pter, see below]
>>
>> Am 23.03.2010 18:42, schrieb Florian Echtler:
>>> Just for my understanding: when talking about a special client, you think of 
>>> something like a (compositing) window manager?
>> Yes, 'special' since it registers itself for rights (and duties) only
>> one client shall possess.
> 
> why? can't the same library be used by multiple clients (thus unifying the
> gesture types) but the clients divide up the available region. This works
> well for button presses, couldn't it work with gestures as well?
Inside one app, there indeed isn't much to it.

It depends on what you want long-term. A special client based extension
allows for gestures to be first-level stuff like key presses,
semantically. This allows for e.g. a gesture to switch apps (or other DE
stuff), which you typically don't want to worry about at application level.

Also, this way you _know_ gestures are unified, which makes them easier
to learn. Development-wise, I think the approach would allow for best
practices to evolve, which could then be retrofitted into the extension.

Long term, apps could just assume gestures to be around and code against
them.

That's basically all I see as weak spots of the library approach. But as
said, it may be retrofitted even in the presence of a range of gesture
libraries. How nice that may work out is another matter :)

>>> Could a client generate new XI events?
>> I don't have the impression this would suit it, though I could be wrong.
>> At any rate, a client can't create a new event _type_. It can create
>> some events though, e.g. via XTst, of predefined types.
>>
>> My idea was that the special client instructs the server what gesture
>> events to generate and how to dispatch them, whenever it thinks it has
>> spotted a gesture. The server tracks some minimal state to ensure
>> consistency and dispatches on behalf of the special gesture client.
>>
>> Peter, maybe you can comment how suitable current mechanisms for input
>> events from clients might be?
> 
> it's reasonably easy to add new events to XI2 for testing and get them
> delivered to all clients. that'd be good enough for a hack, anyway.
> 
> clients can't really do that though. my idea so far in that regard was that
> the library has two interfaces - a preprocessed one for library-internal
> gesture events and a "raw" interface for those clients that like pain.
> 
> One reason why I'd rather not have the gesture library hooked into the
> server is that these libraries require quite some run-time tweaking - such a
> library would be worth its own extension including all disadvantages that
> come from that.
I'm not sure if that is an argument for the approach outlined or against
it? After all, we'd have it in a different process then.

>> In short, dropping in-sync means making it feasible. IMO.
> 
> the order of events is guaranteed for a device but not across multiple
> devices. if one device is frozen, other devices may still send events.
Is the order enforced explicitly? If yes, that would have to be relaxed.

> also, the server does motion event compression so you may lose events if the
> server is busy.
> other than that, we process events in the order they come in to avoid
> hilarities like button releases before presses ;)
>  
>>>> I never fully worked this out, so I can't offer a fancy paper, but it
>>>> seems sensible to me. And since anyone's free to do a library, when it
>>>> comes down to something X.org-endorsed, a special client infrastructure
>>>> might be a preferable choice.
>>> I'm very interested in putting a quick hack together to try this out.
>>> However, my knowledge about X internals is somewhat limited. Are things
>>> like custom events possible, and how would a (special) client go about
>>> sending them?
>> There's the Generic Event Extension:
>> http://www.x.org/wiki/Development/Documentation/XGE
>>
>> I'd make one 'gesture' event, which multiplexes all sorts of gestures.
>> Or maybe three, one for start|cont|end gesture each. Whatever fits the
>> glove.
>>
>> The special client would need to invoke an appropriate gesture dispatch
>> request on the server, maybe as part of a 'X Gesture Extension' (hey
>> that's XGE too :), which would then assemble and dispatch gesture events
>> (only). I don't really see alternatives to this because only the server
>> can properly dispatch events. But XTst should provide some examples to
>> steal from.
> 
> for such an extension, you'd pretty much only need one request that receives
> and event and passes it on. same thing XTEST does, and as Simon said you
> could copy from there.
> 
> to make it simpler on the prototype I'd just extend XI2 and add the request
> + event opcodes. The code in Xi/ isn't always pretty (hardly ever, to be
> precise) but it beats writing and extension from scratch.
Yeah, that way one could knock up a prototype pretty fast.

The only thing missing is interactions, like the app suggesting which
gestures it knows to deal with. Though that could be done declaratively,
e.g. as _WM_ props are handled. For full interaction more requests would
probably be needed, which is a weak spot of the client approach. But
then again, important interactions (like canceling a gesture) could be
handled with the minimal server state I mentioned.

> 
>> Obviously, the event needs to be designed along with the request, and
>> dispatch needs to be worked out. At that point, you should already have
>> yet another half-arsed X protocol extension spec.
>>
>> The server should do some state tracking so you don't get gestures going
>> on without getting their start etc, but that's for when things need to
>> really work.
>>
>> In reality, of course the client should 'be special', i.e. you need
>> "register/unregister gesture client" requests, but for a quick stunt
>> that's optional as well; no-one else will be sending the gesture
>> dispatch request, so there's no contention to prevent. I think there's
>> even a Xi(2) request for getting all the input events to a client, if
>> it's good enough for that case you don't need to do much special for the
>> special client.
>>
>> But it's definitively more work than a library! Still, it may be more
>> rewarding. And these days, there's XCB which reduces the pain of writing
>> extensions. But maybe Peter's OK with just extending Xinput.
> 
> XCB makes it easy to write new extensions, that's true. Unfortunately some
> bits are still missing from it (XKB, XI, XI2) so for now you're probably
> better off biting the bullet and using libXi instead.
> 
>> It's very rough so far. A real impl would probably need to have some
>> opportunity for client interaction too, e.g. an app canceling or
>> grabbing gestures, which I guess you have worked out in your paper.
>>
>> I hope I could give a better picture of the idea. And of course, I'd be
>> delighted to see it realize.
> 
> I really recommend checking if the concept can be realized in a fully
> client-side library. There may be showstoppers that I haven't noticed yet
> but so far I'm still convinced that this is the better solution.
> Maybe Bradley can comment on this too since he should know from his Qt work.
See above ;)

Cheers,

Simon