multitouch

[Matthew Ayres]

Hello all, this is terribly arrogant, but I'm butting into the
conversation. Some of you may recognise my email address, as I've been
discussing MT-related matters with certain developers recently.

I'm nowhere near the level of Xserver competence of the rest of you,
but I'm hoping new eyes will be able to offer new perspective.
However, if I say something that goes completely against X paradigms,
feel free to disregard.  Note that I remember reading that all
windows, widgets, etc are considered windows by X.  Also, in this
email there will be a proposal that may very well go against accepted
X functioning.

> > >The basic assumption for multitouch screens is that it will give us multiple
> > >touchpoints simultaneously from a single physical device. At this point, we
> > >have virtually no user-specific (or body-part specific) information attached
> > >to these points. Furthermore, in X we do not know any context-specific
> > >information. Any touchpoint other than the first may
> > >- belong to a different user
> > >- belong to the same user but a different bodypart that qualifies as "new
> > >   input device" (think left hand + right hand working independently)
> > >- belong to the same user but the same bodypart and be auxiliary (think
> > >   thumb+index during pinching)
> >
> > This basically hits the nail right on the head. How do we know the
> > context of the touch points in the absence of essential information?
>
> We can't. not within the X server. hence the need to find a solution that is
> generic enough that it can forward data to context-aware clients but
> specific enough that you can have more than one such client running at any
> time.

The impression I get from reading through this thread is that the
simplest (and therefore possibly best) approach to grouping touch
events is to group them according to which X window they intersect.
That is, where a second touch event takes place in the same window as
the first, it is part of the same master device; where it takes place
elsewhere, it is another master device.  I'm not sure why this would
not be a useful assumption.

> > >In addition to that, any point may be part of a gesture but without the
> > >context (i.e. without being the gesture recognizer) it's hard to tell if a
> > >point is part of a gesture at all. Worse, depending on the context, the same
> > >coordinates may be part of a different gestures.
> > >Given two touchpoints that start close to each other and move in diagonally
> > >opposite directions, this gesture may be a user trying to zoom, a user
> > >trying to pick to items apart or two users fighting for one object. without
> > >knowing what's underneath, it's hard to say.
> >
> > But this kind of operation is really application dependent, isn't
> > it? I mean, the application would have to decide what the user is
> > trying to do based on the starting/current/final location of the
>
> > touch points...
>
> correct. and that is one of the reasons why I want any context-specific
> information processing (i.e. gestures) in the client. the server cannot have
> enough information to even get started.

Here comes my arrogant proposal:  Suppose that the client application
determines, from a given gesture, that actually the new slave/whatever
is trying to act as a separate master.  I think it would be useful to
provide a mechanism for the application to tell the server that and to
request that the slave be detached and created a new master.  Some
negotiation would be needed of course, but it would be useful (for
instance) if it turns out to be a second user trying to drag something
from another user's window.  So what I imagine would go something like
this:

Touch1 in WindowA (ApplicationX) = MD1 + SD1.
Touch2 in WindowA (ApplicationX) = MD1 + SD2.
ApplicationX determines that Touch2 wants to do something of its own.
ApplicationX tells Xserver to make Touch2 into MD2 + SD1.
Xserver replies with old and new device identifier, allowing smooth hand-off.
ApplicationX may release MD2 and retain MD1.
ApplicationY may now grab MD2.

> > >The current idea, not yet completely discarded is to send touchpoints to the
> > >client underneath the pointer, with the first touchpoint doing mouse
> > >emulation. a touchpoint that started in a client is automatically grabbed
>
> > >and sent to the client until the release, even if the touch is released.
> > >thus a gesture moving out of the client doesn't actually go out of the
> > >client (behaviour similar to implicit passive grabs).  While such a grab is
>
> > >active, any more touchpoints in this client go through the same channel,
> > >while touchpoints outside that client go to the respective client
> > >underneath.
> > >
>
> > >problem 1: you can't really do multi-mouse emulation since you need a master
> > >device for that. so you'd have to create master devices on-the-fly for the
> > >touchpoints in other clients and destroy them again. possible, but costly.
> >
> > Why not only do mouse emulation for the first client that got the
> > first touch point? It does eliminate implicit support for multiple
> > user interaction for applications that don't have explicit support
> > for multi-touch though. But as a starting point it may work. And
> > then see if it's possible to do multiple mouse emulation?
>
> imo, one of the really tempting features of multitouch support in X is to be
> able to use it (within reason) with the default desktop. Yes, we could
> rewrite the whole desktop to support touch but I'd just like to be able to
> do more than that.

That is a very tempting idea.  I suspect that my proposal would not
play well with it.

>

So my apologies for butting in like that, but I felt I might as well
say something.

-Matthew