multitouch

[Peter Hutterer]

On Mon, Mar 01, 2010 at 02:41:51PM +0200, Daniel Stone wrote:
> On Mon, Mar 01, 2010 at 12:42:40PM +0100, Bradley T. Hughes wrote:
> > On 03/01/2010 12:22 PM, ext Daniel Stone wrote:
> >> (What happens in the hybrid system when I get an event from finger 1,
> >> decide I like it, take out a grab, and then finger 2 presses on another
> >> window.  Do I respect the event and give the app the finger 2 press it
> >> likely doesn't want, or break the grab and deliver it to another client?
> >> Neither answer is pleasant.)
> >
> > But what happens in the sub-device system? Finger 1's event goes via it's 
> > MD to the client under the finger, but what about the second finger if 
> > it's over another client's window? Does it go to the client that got 
> > finger1 since it's on the same MD (which precludes the multi-user use 
> > case)?
> 
> You just extend the MD/SD semantics down, so if you have:
> MD -> Touchscreen -> Fingers [123...], then:
>    -> Mouse
> then:
>   * grab on finger 1 results in only finger 1 being grabbed, fingers
>     2345 and the mouse are delivered as normal to any client
>   * grab on the ts results in all fingers being grabbed, but the mouse
>     being delivered as normal
>   * grab on the MD results in everything being grabbed
>   * grab on finger 1 by client A followed by a grab on the MD by client
>     B results in finger 1 being delivered to client A, and all other
>     fingers and the mouse being delivered to client B
> 
> So, you just take the existing MD/SD semantics and extend them to n
> levels instead of 2.  Just walk up the list of parents looking for
> grabs/interest, stopping at the first one you find.
> 
> Peter, correct me if I'm missing something blindingly obvious, or wholly
> non-obvious. :)

Sure. Photo of whiteboard brainstorm is available on the link below, it's
the closest thing I have to a diagram.
http://people.freedesktop.org/~whot/patches/multitouch-brainstorm-03.03.2010.jpg
My notes to this approach below, sorry about the unedited format, they're
just what I jotted down right now but I figured the full train of thought is
still better than just short answers. direct answers to your approach are at
the end of the email.
Spoiler alert: approach breaks down on keyboards during core grabs.

Note that I'm _not_ trying to solve the synaptics multi-finger swipe case,
this one is easy and could be solved by packing the touchpoints into the
event as additional valuators. I'm trying to solve the case where one or
more users interact with one or more apps on the same screen (core, XI, XI2,
multi-touch, or a combination thereof).

Basic premise:
Data from devices comes in terms of Press/Motion/Release from fingers. These
can be grouped, some way or another. If it can't be ID'd, we've already lost
anyway. We have one physical device (the touchsceen) and multiple input
points on that physical devices (call them FDs for FingerDevices). All FDs
come through the same SD and MD.  Assuming we can make the cost of creating
an FD zero, we can do so on the fly as touchpoints show up. They are not
exposed to core or XI clients.  So in the default event delivery, you'd have
FD - SD - MD, which is great for synaptics gestures since you get all the
data you want. But for multi-touch core, you need multiple MDs.

So what about FDs sending core, XI and XI2 events? This is justified since a
FD is a direct interaction device and the distinction between physical
device and virtual device doesn't exist. (This distinction led to the
creation of the MD/SD hierarchy in the first place) 
Hierarchy would then be:
          core event
        MD -/||
        |    ||
        SD --/|
        |     |
        FD ---/

(this diagram best viewed in monospace)

Got to be careful though that the same core event isn't delivered twice from
different devices - may cause doubleclick problem.  XI is severely limited
if FDs are shortlived, since the device is likely to disappear before the
register event comes in - if IDs are reused, the mask may refer to a
different device that has since disappeared and reappeared → make devices
long-lived and re-usable for fingers.

Only propagate core and XI event if the event was not delivered. XI2 events
still need to propagate (what to do with the source_id??) So test case is:
one touchdevice, press one finger.
Init state: 
MD → SD
touch state
MD → SD → FD
FD sticks around until device is removed (??).

After touch, hierarchy event about FD (how will that look like??). Core
event propagated through FD or MD (SDs don't do core),  XI event are
propagated through FD or SD (MDs don't do XI). XI2 event through all three
(XI2 slave grab - double event??).

Assuming core event delivered, grab comes in from client. What device to grab?
 • MD - looses all multitouch if all fingers are grabbed
 • FD - seems like sanest solution, but might be tricky as fuck with
 thaw&replay (MPX semantics take care of this??)
 • Grab FD if ClientPointer is on MD, let other fingers go through MD but
 only for non-interfering grabs. i.e. a core grab grabs through the MD.
 Could be awkward if the FD disappears and the next FD with the same id
 (still grabbed) then untriggers the grab.  Still the most sensible
 solution.
 
this is not an issue for passive grabs, they switch ownership dynamically

What about the keyboard? If one finger is grabbed as sync - freeze the MD's
keyboard? Where do keyboard events go if two fingers are grabbed?  Having
two pointer grabs is easy, but if both grabs wait for a keyboard event, then
you're screwed. You'd have to flip the keyboard over to the second grab when
the first exists but.  DEADLOCK, a client might wait for the key event to
happen. UI ISSUE - user doesn't know where keyboard is

Also, we don't know if a grab comes in response to a FD device or an
SD-generated core event on the MD. so we don't know whether to grab the FD
or the MD from a core client.

>   * grab on finger 1 results in only finger 1 being grabbed, fingers
>     2345 and the mouse are delivered as normal to any client

"grab on finger" isn't defined in your example. Does this mean a specific
XI2 call? We still want core events (well, we do otherwise we might as well
ignore all issues and create our own custom API and ignore the rest)

With core, you cannot know when "Finger 1" is supposed to be grabbed.
example: touch device and mouse on the same MD, you use your finger, then
the mouse button, grab comes in. Do you grab the finger? or the mouse (and
therefore the MD?)
if you grab the MD for all fingers, then you lose multitouch whenever a grab
is active.
even if you would know when to grab the finger - once you have finger 1,
finger 2 and the MD grabbed by three clients, where do you send key events
to? If all three are a sync grab (i.e. waiting for some key combo) you have
a nice confusing half-deadlock.

>   * grab on the ts results in all fingers being grabbed, but the mouse
>     being delivered as normal
>   * grab on the MD results in everything being grabbed

you've just lost the ability to click on anything because some other client
has a menu open. So you're down to single-client multitouch.

>   * grab on finger 1 by client A followed by a grab on the MD by client
>     B results in finger 1 being delivered to client A, and all other
>     fingers and the mouse being delivered to client B

that's what we already do in MPX with the temporary detachment of slave
devices on a direct grab. We can't do so for XI1 devices though, so that'll
likely get tricky on the details again.

Cheers,
  Peter