New approach to multitouch using DIDs and bitmasked events

[Peter Hutterer]

On Thu, Jul 08, 2010 at 04:05:10PM -0400, Chase Douglas wrote:
> On Wed, 2010-07-07 at 15:21 -0400, Chase Douglas wrote:
> > On Tue, 2010-07-06 at 11:33 +1000, Peter Hutterer wrote:
> > > On Fri, Jul 02, 2010 at 09:09:48PM -0400, Chase Douglas wrote:
> > > > > This is a good start. Eventually, we should have DIDs exposed as a new 
> > > > > device class on the protocol though. Likewise, I don't think ET_DM 
> > > > > should not be converted to normal XIDeviceEvents but rather to their own 
> > > > > type that can be expanded in the future to accommodate for additional 
> > > > > features (e.g. the touch area).
> > > > 
> > > > I hacked some more today, and I'm not sure we need a separate XI2 event
> > > > type. I'm going to send out another RFC "pull request" with the work
> > > > soon, but as an overview I wanted to work on the "magicmouse" scenario.
> > > > The result is the following:
> > > > 
> > > > MT Touchscreen:
> > > >   - Device is marked as a DID
> > > >   - Starts out floating, can't be attached
> > > >   - MT touch coordinates are the first two valuators, XIDeviceEvents are
> > > >     in screen coordinates. However, an app can grab XIRawEvents for
> > > >     device coordinates. *
> > > >   - Since it's a DID, button events are also blocked for core, XI1
> > > > 
> > > > Magic Mouse:
> > > >   - Device is *not* marked as a DID
> > > >   - Starts out attached to virtual core master pointer
> > > >   - First two axes are REL_X, REL_Y
> > > >   - Device is marked as relative (for core and XI1 events)
> > > >   - Abs MT axes are marked as absolute
> > > >     * Can only be sent through XI2 since XI can't handle mixed mode
> > > >   - Since Abs MT touches are not the first two axes, they are not
> > > >     translated to screen coordinates
> > > >   - Abs MT axes sent through XI2 are also sent through attached master
> > > >   - Button events are not blocked for core, XI1
> > > > 
> > > > I am not aware of any use case that is not covered by the above scheme
> > > > and requires a new XI2 event type. If you know of any, please let me
> > > > know.
> > > 
> > > How will a client know if a device is a DID? XIQueryDevice must return this
> > > information.
> > > 
> > > One of the things I do also expect from DIDs is that the _first_ touchpoint
> > > is converted into core events. Thus, even on a multitouch device you can
> > > still interact in a traditional way with apps that won't support touch
> > > events  Hoping for Tk and others to be updated to support a new device type
> > > is a bit too optimistic. Worse, it introduces a hard transition between what
> > > we have now and what we can, i.e. you force everyone to update or otherwise
> > > they lose what they have. That's not good IMO, we need a transitional
> > > solution.
> > > 
> > > Anyway. My idea of DIDs was the following:
> > > the driver creates one master pointer device for a physical touchscreen.
> > > This master pointer device is controlled by the first touchpoint on the
> > > screen. At the same time, the driver creates DIDs on-demand as it is being
> > > used. Each DID stands for one touchpoint and sends DirectEvents or whatever
> > > they're called. Obviously, that gives you a duplication but the device
> > > hierarchy should reference which device is a DID device, thus MT aware apps
> > > can just ignore pointer events from that master (or not, in the case of the
> > > magic mouse, possibly need some flag here).
> > > 
> > > DIDs are short-lived and don't show up in the hierarchy, only the parent
> > > device does, indicating that DIDs may be present. Button or key events are
> > > all routed through the master pointer.
> > > 
> > > This is the approach that I think could work. though we won't know until we
> > > write down the spec. Something I recommend for your approach as well, sit
> > > down with XI2proto.txt and write the additions and how clients can use them.
> > > More often than not, you'll likely notice some issues and have to go back to
> > > the code.
> > 
> > I started to think about this, which led to the following:
> > 
> > 1. Instead of allocating devices one at a time each time a touch begins
> > and then deallocating when the touch ends, lets just have a pool of
> > devices.
> > 2. If we have a pool of devices and we have MTDev, why not index the
> > devices in the pool by ABS_MT_SLOT. If we encounter a higher slot than
> > we have allocated for, just allocate a new one on the spot.
> > 3. If we have a pool of slotted devices, then what's the point of DIDs?
> > I was keying all my work off of the concepts Peter pointed out here:
> > 
> >    http://lists.x.org/archives/xorg-devel/2010-June/009757.html
> > 
> >    The arguments boiled down mostly to: input devices can't be transient
> > due to grab issues. But our touch devices aren't transient anymore.
> 
> I thought some more about this again, and I think Peter may have been
> getting at the physical state of the touches as input devices, not the
> fact that the touch input devices literally go away (i.e.
> deallocated/destroyed).
> 
> This leaves us back at trying to figure out what to do about grabs on MT
> devices. It seems apparent that no matter what we do, nothing will be
> backwards compatible for MT touch event grabs. (We can still route one
> touch at a time to the master core pointer for client pointer backwards
> compatibility, but that's a different issue.)

My current thinking is that we may need to allow for two different grabs on
a device - one for touch events, one for traditional pointer/keyboard
events. There is still a use-case for grabs even with direct-touch input.

> Instead, what if we only allow a touch device to be passively grabbed
> through the first mouse button. All other grabs that are attempted would
> return some error code. This would allow typical pointer grabs where you
> mouse down in a window and then drag out of the window before releasing
> the button.

this is the implicit passive grab that we need anyway. A more typical
passive grab initated by the client is the window manager's grab on mouse
buttons. This grab is GrabModeSync to allow the WM to bring the window to
the foreground before the event is replayed.
Though WM's don't feature much in most multi-touch demos I don't see them go
away at all - we need synchronized grabs for touch events.

> Generally, we would still have the concept of DIDs in the sense that the
> touch input devices themselves would be floating, and thus XI2 only. So,
> that leaves us with just the XI2 protocol to worry about.

True, though you still need to cater for XI 2.0 clients accessing the
devices for pointer/keyboard events. These two need to be synchronised.
Also, fwiw, it may not be necessary to float the devices after all, though
I haven't quite figured that out in detail yet.

> To facilitate clients' need to know if the device is transient or not,
> and thus to grab in the proper way, I suggest adding a new "use" type
> called "TransientSlave" that is returned by XIQueryDevice.
> 
> I haven't put everything together yet, but I foresee using the direct
> devices (leaving aside the direct axes) approach in the commits I wrote,
> but renaming "direct" to "transient". Then, combine this with Carlos'
> evdev branch plus mtdev for proper transient input device pooling.
> 
> BTW, a lot of my misunderstandings come from not knowing exactly what
> device grabs are used for, and how they are used. Please let me know if
> there are other reasons for grabbing pointer devices that would break
> this approach.

grabs mean "don't send events to anyone but me". Events during grabs have a
reference to the grab window, hence you know if the event was within the
window or outside. Passive grabs simply do that in response to a button/key
+ modifier combination instead of directly requested. The most common uses
these days are menus and window management.

Cheers,
  Peter