Multitouch followup: gesture recognition?

[Simon Thum]

Am 26.03.2010 10:02, schrieb Florian Echtler:
> Hello everyone,
> 
>>>>>> Just for my understanding: when talking about a special client, you think of 
>>>>>> something like a (compositing) window manager?
>>>>> Yes, 'special' since it registers itself for rights (and duties) only
>>>>> one client shall possess.
>>>>
>>>> why? can't the same library be used by multiple clients (thus unifying the
>>>> gesture types) but the clients divide up the available region. This works
>>>> well for button presses, couldn't it work with gestures as well?
>>> Inside one app, there indeed isn't much to it.
>>>
>>> It depends on what you want long-term. A special client based extension
>>> allows for gestures to be first-level stuff like key presses,
>>> semantically. This allows for e.g. a gesture to switch apps (or other DE
>>> stuff), which you typically don't want to worry about at application level.
>>>
>>> Also, this way you _know_ gestures are unified, which makes them easier
>>> to learn. Development-wise, I think the approach would allow for best
>>> practices to evolve, which could then be retrofitted into the extension.
>>  
>> I claim that gestures _cannot_ be perfectly unified. 
>> Unify them, but unify them in gtk-gesture, qt-gesture or libgesture. But not
>> in an X extension. A gesture is lossy abstraction of input and what is a
>> zoom gesture in one application is a gesture to move two points
>> apart in another application. Reducing the latter to a ZoomBy10% gesture
>> means to lose the information. So you need to provide both raw data and
>> gesture data anyway.
> On one hand, I agree. But I believe that this problem is exactly what my
> formalism solves. By a) allowing applications to customize gestures and
> b) restricting them to certain screen regions (*), this isn't a
> contradiction anymore. E.g. what's a zoom gesture to the first app on
Which of course means the extension needs to transport the necessary
information, or describe other means of transport (e.g. the props I
mentioned, though as Peter pointed out, they're not good for live
communication).

This seems essential to your approach, so the feasibility of a server
extension (oranything else, but a extension incurs overhead) depends a
fair bit on the dynamics of your gesture customization.

> one region will split into two separate move gestures for two regions
> for the other one. Of course, the ability to get raw data should always
> be preserved.
> 
> (*) whether these should be XWindows or something different, I don't
> know yet. Is there an extension that allows arbitrary shapes for XWindow
> objects, particularly with respect to input capture?
I know there is the shape extension, it think it also covers input.

> 
>>> Long term, apps could just assume gestures to be around and code against
>>> them.
>> The same is true if the gestures are handled in toolkits like GTK. instead
>> of connecting to button-press you can connect to zoom-in. where the gesture
>> is handled in the end doesn't matter to the application.
>> Qt has QGestureEvents already but still provides QTouchEvents too.
> Correct, but wouldn't it be an advantage to handle it consistently
> across toolkits?
> 
>>>>>> Could a client generate new XI events?
>>>>> I don't have the impression this would suit it, though I could be wrong.
>>>>> At any rate, a client can't create a new event _type_. It can create
>>>>> some events though, e.g. via XTst, of predefined types.
>>>>>
>>>>> My idea was that the special client instructs the server what gesture
>>>>> events to generate and how to dispatch them, whenever it thinks it has
>>>>> spotted a gesture. The server tracks some minimal state to ensure
>>>>> consistency and dispatches on behalf of the special gesture client.
>>>>>
>>>>> Peter, maybe you can comment how suitable current mechanisms for input
>>>>> events from clients might be?
>>>>
>>>> it's reasonably easy to add new events to XI2 for testing and get them
>>>> delivered to all clients. that'd be good enough for a hack, anyway.
>>>>
>>>> clients can't really do that though. my idea so far in that regard was that
>>>> the library has two interfaces - a preprocessed one for library-internal
>>>> gesture events and a "raw" interface for those clients that like pain.
>>>>
>>>> One reason why I'd rather not have the gesture library hooked into the
>>>> server is that these libraries require quite some run-time tweaking - such a
>>>> library would be worth its own extension including all disadvantages that
>>>> come from that.
>>> I'm not sure if that is an argument for the approach outlined or against
>>> it? After all, we'd have it in a different process then.
>>
>> sorry. i think you can some extra events in for hacking up a prototype. For
>> an actual solution, I don't think an extension is the solution - but sending
>> events from one client to another one isn't the solution either IMO.
> 
>>>> also, the server does motion event compression so you may lose events if the
>>>> server is busy.
>>>> other than that, we process events in the order they come in to avoid
>>>> hilarities like button releases before presses ;)
>>>>  
>>>>>>> I never fully worked this out, so I can't offer a fancy paper, but it
>>>>>>> seems sensible to me. And since anyone's free to do a library, when it
>>>>>>> comes down to something X.org-endorsed, a special client infrastructure
>>>>>>> might be a preferable choice.
>>>>>> I'm very interested in putting a quick hack together to try this out.
>>>>>> However, my knowledge about X internals is somewhat limited. Are things
>>>>>> like custom events possible, and how would a (special) client go about
>>>>>> sending them?
>>>>> There's the Generic Event Extension:
>>>>> http://www.x.org/wiki/Development/Documentation/XGE
>>>>>
>>>>> I'd make one 'gesture' event, which multiplexes all sorts of gestures.
>>>>> Or maybe three, one for start|cont|end gesture each. Whatever fits the
>>>>> glove.
>>>>>
>>>>> The special client would need to invoke an appropriate gesture dispatch
>>>>> request on the server, maybe as part of a 'X Gesture Extension' (hey
>>>>> that's XGE too :), which would then assemble and dispatch gesture events
>>>>> (only). I don't really see alternatives to this because only the server
>>>>> can properly dispatch events. But XTst should provide some examples to
>>>>> steal from.
>>>>
>>>> for such an extension, you'd pretty much only need one request that receives
>>>> and event and passes it on. same thing XTEST does, and as Simon said you
>>>> could copy from there.
>>>>
>>>> to make it simpler on the prototype I'd just extend XI2 and add the request
>>>> + event opcodes. The code in Xi/ isn't always pretty (hardly ever, to be
>>>> precise) but it beats writing and extension from scratch.
>>> Yeah, that way one could knock up a prototype pretty fast.
> One important question here, which I've obviously haven't understood
> fully: what's the difference/dividing line between libXi as a library
> and XInput as an extension? AFAICT, libXi is the implementation for
> XInput - correct?
libXi is the client library for talking to X servers having XInput.
Purely from a client perspective, your statement is correct. But more
realistically, libXi is a specialized Network<->API wrapper.

> 
> Let me try to summarize the possible approaches:
> 
> 1. a pure userspace library which just converts gestures locally for one
> specific client without knowledge of others (this is more or less what
> Qt or libTISCH do right now, though in different ways)
> 
> 2. a special client like a WM which intercepts input events and passes
> additional gesture events to other clients (possible, but with some
> caveats I haven't yet understood fully)
> 
> 3. a separate server extension in its own right (possible, also with
> some potential traps)
> 
> 4. a patch to libXi using the X Generic Event Extension (same as 3, but
> fastest to hack together and doesn't require any changes to the server.)
> 
> Would you agree with that summary?
I don't get what 4 might be. 2 and 3 aren't really alternatives, but
different aspects of a server-side implementation. The special client
can make the server deliver events, it can't do that itself. So it needs
an appropriate own server extension or additional Xinput requests to
facilitate delivery. Which you want libXi wrappers for.

So whether a special client detects gestures or the server itself, the
server needs to deliver events, and the client needs to be able to
receive them. This is where XGE and libXi kick in.

Cheers,

Simon