multitouch

[Bradley T. Hughes]

On 02/10/2010 07:37 AM, ext Peter Hutterer wrote:
>>> - send the second touchpoint to the second client?
>>
>> This is the behavior I would like to see. It's the one that promotes
>> the multiple-user and multiple-client use case the best.
>>
>>> This is the main roadblock at the moment, and anytime I try to come up with
>>> a working solution I hit a wall at even quite basic use-cases.
>>
>> Would you mind elaborating a bit on this?
>
> sure, I'm probably repeating myself with some of the points below, but it's
> easier for me to get a train of though going.
>
> The basic assumption for multitouch screens is that it will give us multiple
> touchpoints simultaneously from a single physical device. At this point, we
> have virtually no user-specific (or body-part specific) information attached
> to these points. Furthermore, in X we do not know any context-specific
> information. Any touchpoint other than the first may
> - belong to a different user
> - belong to the same user but a different bodypart that qualifies as "new
>    input device" (think left hand + right hand working independently)
> - belong to the same user but the same bodypart and be auxiliary (think
>    thumb+index during pinching)

This basically hits the nail right on the head. How do we know the context 
of the touch points in the absence of essential information?

> In addition to that, any point may be part of a gesture but without the
> context (i.e. without being the gesture recognizer) it's hard to tell if a
> point is part of a gesture at all. Worse, depending on the context, the same
> coordinates may be part of a different gestures.
> Given two touchpoints that start close to each other and move in diagonally
> opposite directions, this gesture may be a user trying to zoom, a user
> trying to pick to items apart or two users fighting for one object. without
> knowing what's underneath, it's hard to say.

But this kind of operation is really application dependent, isn't it? I 
mean, the application would have to decide what the user is trying to do 
based on the starting/current/final location of the touch points...

> So we need a way handle touchpoints in a way that we're not restricting any
> of these. All while staying compatible to the core, XI1 and XI2 protocol.
> X's main task here is to be an input multiplexer and send the events to the
> right clients since at any point in time we may have more than one
> multitouch application running. Picking the right client is the important
> thing.
>
> A few already discarded ideas:
[snip]

> The current idea, not yet completely discarded is to send touchpoints to the
> client underneath the pointer, with the first touchpoint doing mouse
> emulation. a touchpoint that started in a client is automatically grabbed
> and sent to the client until the release, even if the touch is released.
> thus a gesture moving out of the client doesn't actually go out of the
> client (behaviour similar to implicit passive grabs).  While such a grab is
> active, any more touchpoints in this client go through the same channel,
> while touchpoints outside that client go to the respective client
> underneath.
>
> problem 1: you can't really do multi-mouse emulation since you need a master
> device for that. so you'd have to create master devices on-the-fly for the
> touchpoints in other clients and destroy them again. possible, but costly.

Why not only do mouse emulation for the first client that got the first 
touch point? It does eliminate implicit support for multiple user 
interaction for applications that don't have explicit support for 
multi-touch though. But as a starting point it may work. And then see if 
it's possible to do multiple mouse emulation?

> problem 2: gestures starting outside the client may go to the wrong one. not
> sure how much that is a problem, I think that's more a sideeffect of a UI
> not designed for touch.

I don't think it'll be much of a problem. I doubt we'll see users trying to 
use 2-finger scroll/pinch to manipulate the volume slider in the system 
tray, for example. We made this assumption in Qt... applications/components 
that want multi-touch will be designed in a way (e.g. large enough) that 
allows the user to hit the touch target.

> problem 3: this requires the same device to be grabbed multiple times by
> different clients, but possible not for mouse emulation. And a client
> doesn't necessary own a window and events may be sent to multiple clients at
> the same time, all of which would then need such a grab. I think this is
> where this approach breaks down, you'd get multiple clients getting the same
> event and I'm not sure how that'd work out.

I don't quite understand what you meant by "but possible not for mouse 
emulation"?

I do understand the problem (I think), but unfortunately I'm not familiar 
with the internals to know what this would mean in practice. Considering 
that each touch-point would be given implicit-grabs, not the device, perhaps 
this is an indication that it makes sense (like Carsten mentioned) to have 
each touch-point a separate device in XI2?

But then I wonder... what happens when one client tries to grab the entire 
multi-touch device? hm...

> Oh, and did I mention that we have to be compatible to the core protocol
> grab semantics for mouse emulation?

You did now :)

-- 
Bradley T. Hughes (Nokia-D-Qt/Oslo), bradley.hughes at nokia.com
Sandakervn. 116, P.O. Box 4332 Nydalen, 0402 Oslo, Norway