Multitouch followup: gesture recognition?

[Simon Thum]

[I reordered for readability]

Am 26.03.2010 01:37, schrieb Peter Hutterer:
> On Thu, Mar 25, 2010 at 11:38:59AM +0100, Simon Thum wrote:
>> Also, this way you _know_ gestures are unified, which makes them easier
>> to learn. Development-wise, I think the approach would allow for best
>> practices to evolve, which could then be retrofitted into the extension.
>  
> I claim that gestures _cannot_ be perfectly unified. 
> Unify them, but unify them in gtk-gesture, qt-gesture or libgesture. But not
> in an X extension. A gesture is lossy abstraction of input and what is a
> zoom gesture in one application is a gesture to move two points
> apart in another application. Reducing the latter to a ZoomBy10% gesture
> means to lose the information. So you need to provide both raw data and
> gesture data anyway.
I didn't mean to unify _all_ gestures. Their semantics need to remain in
the application anyway. When I referred to _WM props I thought about a
prop like _NET_WM_ALLOWED_ACTIONS, which would list what gestures are
sensible in the client. So raciness isn't an issue here.

I admit the use cases I had in mind originally weren't as rich as
today's touch interactions. So the approach may be unsuitable, though
I'm not convinced it is.

> sorry. i think you can some extra events in for hacking up a
prototype. For
> an actual solution, I don't think an extension is the solution - but
sending
> events from one client to another one isn't the solution either IMO.

Then it boils down to whether one wants first class gesture events, and
if yes, where shall they be coming from? If no, how to deal with e.g. a
'switch app' gesture?

> 
>> Long term, apps could just assume gestures to be around and code against
>> them.
> 
> The same is true if the gestures are handled in toolkits like GTK. instead
> of connecting to button-press you can connect to zoom-in. where the gesture
> is handled in the end doesn't matter to the application.
> Qt has QGestureEvents already but still provides QTouchEvents too.
> 
>> That's basically all I see as weak spots of the library approach. But as
>> said, it may be retrofitted even in the presence of a range of gesture
>> libraries. How nice that may work out is another matter :)
> 
>>>>> Could a client generate new XI events?
>>>> I don't have the impression this would suit it, though I could be wrong.
>>>> At any rate, a client can't create a new event _type_. It can create
>>>> some events though, e.g. via XTst, of predefined types.
>>>>
>>>> My idea was that the special client instructs the server what gesture
>>>> events to generate and how to dispatch them, whenever it thinks it has
>>>> spotted a gesture. The server tracks some minimal state to ensure
>>>> consistency and dispatches on behalf of the special gesture client.
>>>>
>>>> Peter, maybe you can comment how suitable current mechanisms for input
>>>> events from clients might be?
>>>
>>> it's reasonably easy to add new events to XI2 for testing and get them
>>> delivered to all clients. that'd be good enough for a hack, anyway.
>>>
>>> clients can't really do that though. my idea so far in that regard was that
>>> the library has two interfaces - a preprocessed one for library-internal
>>> gesture events and a "raw" interface for those clients that like pain.
>>>

> 
>>>> In short, dropping in-sync means making it feasible. IMO.
>>>
>>> the order of events is guaranteed for a device but not across multiple
>>> devices. if one device is frozen, other devices may still send events.
>> Is the order enforced explicitly? If yes, that would have to be relaxed.
> 
> there isn't much to it. events come in, are added to the end of the queue
> and processed from the front of the queue. If the driver sends release
> events before press events the server will enqueue those in that order too -
> though some state checking later would drop the release events for
> not-pressed buttons, etc.
> 
>>> also, the server does motion event compression so you may lose events if the
>>> server is busy.
>>> other than that, we process events in the order they come in to avoid
>>> hilarities like button releases before presses ;)
>>>  
>>>>>> I never fully worked this out, so I can't offer a fancy paper, but it
>>>>>> seems sensible to me. And since anyone's free to do a library, when it
>>>>>> comes down to something X.org-endorsed, a special client infrastructure
>>>>>> might be a preferable choice.
>>>>> I'm very interested in putting a quick hack together to try this out.
>>>>> However, my knowledge about X internals is somewhat limited. Are things
>>>>> like custom events possible, and how would a (special) client go about
>>>>> sending them?
>>>> There's the Generic Event Extension:
>>>> http://www.x.org/wiki/Development/Documentation/XGE
>>>>
>>>> I'd make one 'gesture' event, which multiplexes all sorts of gestures.
>>>> Or maybe three, one for start|cont|end gesture each. Whatever fits the
>>>> glove.
>>>>
>>>> The special client would need to invoke an appropriate gesture dispatch
>>>> request on the server, maybe as part of a 'X Gesture Extension' (hey
>>>> that's XGE too :), which would then assemble and dispatch gesture events
>>>> (only). I don't really see alternatives to this because only the server
>>>> can properly dispatch events. But XTst should provide some examples to
>>>> steal from.
>>>
>>> for such an extension, you'd pretty much only need one request that receives
>>> and event and passes it on. same thing XTEST does, and as Simon said you
>>> could copy from there.
>>>
>>> to make it simpler on the prototype I'd just extend XI2 and add the request
>>> + event opcodes. The code in Xi/ isn't always pretty (hardly ever, to be
>>> precise) but it beats writing and extension from scratch.
>> Yeah, that way one could knock up a prototype pretty fast.
>>
>> The only thing missing is interactions, like the app suggesting which
>> gestures it knows to deal with. Though that could be done declaratively,
>> e.g. as _WM_ props are handled. For full interaction more requests would
>> probably be needed, which is a weak spot of the client approach. But
>> then again, important interactions (like canceling a gesture) could be
>> handled with the minimal server state I mentioned.
> 
> I think there's a few people out there that would disagree that _WM props
> are a good way of handling real-time communication :) it's a fairly racy
> approach that leaves more to optimism than to determinism.
> 
> Cheers,
>   Peter
>