[PATCH inputproto xi 2.1] Updates for pointer emulation and more touch device modes

[Peter Hutterer]

On Tue, Feb 22, 2011 at 10:06:37AM -0500, Chase Douglas wrote:
> Also includes resolutions for dependent devices and implicit grabs and
> how to handle slave touch device attachment and touch selections.
> 
> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
> ---
> 
> Diff is against latest patch sent by Daniel Stone. See
> http://cgit.freedesktop.org/~cndougla/inputproto for the cumulative XI 2.1
> changes.
> 
> Known issues:
>   * Inert grabs and TouchOwnerRejectContinue are not detailed in the spec.
> 
> As a side note, I have implemented this spec, which will be pushed to Ubuntu as
> pre-release functionality today. I am hoping that the ABI can be considered
> stable at this point unless we find bugs with it. My implementation in the
> server and input modules is complete excepting the following:
> 
>   * Inert grabs and TouchOwnerRejectContinue
>   * Proper handling of IndependentPointer and SemiMultitouch devices
>   * Touch selection canceling when attaching a touch device
>   * Implicit grab touch canceling
> 
> I plan to implement all of the above before the release of Ubuntu 11.04 except
> Inert functionality and IndependentPointer handling in the evdev input module.
> 
> I will send patches for the server once I have them cleaned up more, but they
> can be found at http://cgit.freedesktop.org/~cndougla/xserver in the multitouch
> branch.
> 
>  XI2.h        |    9 ++-
>  XI2proto.txt |  186 +++++++++++++++++++++++++++++++++++++++++----------------
>  2 files changed, 139 insertions(+), 56 deletions(-)
> 
> diff --git a/XI2.h b/XI2.h
> index 40c9ca6..dbf14a8 100644
> --- a/XI2.h
> +++ b/XI2.h
> @@ -32,6 +32,7 @@
>  #define Dont_Check                              0
>  #endif
>  #define XInput_2_0                              7
> +#define XInput_2_1                              8

iirc, this needs to move into libXi. these are libXi-only fields and I'm not
sure why I never pushed the patch
http://www.mail-archive.com/xorg-devel@lists.x.org/msg12278.html

>  #define XI_2_Major                              2
> @@ -132,16 +133,16 @@
>  /* Device event flags (common) */
>  /* Device event flags (key events only) */
>  #define XIKeyRepeat                             (1 << 16)
> -/* Device event flags (pointer events only) */
> +/* Device event flags (pointer and touch events only) */
>  #define XIPointerEmulated                       (1 << 16)
>  /* Device event flags (touch events only) */
> -#define XITouchPendingEnd                       (1 << 16)
> -/* Device event flags (touch end events only) */
> -#define XITouchAccepted                         (1 << 17)
> +#define XITouchPendingEnd                       (1 << 17)
>  
>  /* Touch modes */
>  #define XIDirectTouch                           1
>  #define XIDependentTouch                        2
> +#define XIIndependentPointer                    3
> +#define XISemiMultitouch                        4
>  
>  /* XI2 event mask macros */
>  #define XISetMask(ptr, event)   (((unsigned char*)(ptr))[(event)>>3] |=  (1 << ((event) & 7)))
> diff --git a/XI2proto.txt b/XI2proto.txt
> index 5ebe59d..212425d 100644
> --- a/XI2proto.txt
> +++ b/XI2proto.txt
> @@ -52,7 +52,7 @@ not always necessary.
>  The additions in XI 2.1 aim to:
>  - support a dynamic number of simultaneous touch points,
>  - support devices that are both multi-touch and traditional pointer devices,
> -- while supporting pre-XI2.1 clients through emulation of XI 2.0 and core
> +- while supporting pre-XI2.1 clients through emulation of XInput and core
>    pointer events.
>  
>  XI 2.1 caters for two modes of touch input devices:
> @@ -194,6 +194,9 @@ delivered on W. Once an event has been delivered as either XI or core event,
>  event processing stops.
>  
>  4.4 Touch device support
> +
> +4.4.1 Touch event sequences
> +
>  Touch event processing differs from normal event processing in a few ways,
>  most notably in that touch events are processed partially out-of-band from
>  pointer and keyboard events.
> @@ -205,62 +208,145 @@ to touch the device. The init and destroy stage of this sequence are always
>  present, while the move stage is optional. Within this document, the term
>  "touch sequence" is used to describe the above chain of events. A client
>  wishing to receive touch events must register for at least TouchBegin,
> -TouchOwnership, TouchUpdate, and TouchEnd simultaneously; it may also select
> -for TouchUpdateUnowned events if it wants to receive the full touch stream,
> -rather than just the final state.
> +TouchUpdate, and TouchEnd simultaneously. It may also select for
> +TouchUpdateUnowned and TouchOwnership events if it wants to receive the full
> +touch stream while other clients own or have active grabs involving the touch.
> +An example of this usage is a touch painting program that paints touches while
> +a gesture recognizer has an active grab. Such clients must be able to undo state
> +if the touch ends without the client receiving ownership of the touch.

I'm a big fan of examples, but not in the middle of the spec. Examples are
almost always imprecise simplifications and having one example is usually
not enough and two examples too many. The same can be said:

A client selecting for TouchUpdateUnowned will receive touch events even if it is
not the current owner of the event. A client selecting for TouchOwnership
events will receive said event when it becomes the current owner of the
touch.

Examples belong into the appendix. Having said that - if we have two
different event types for UpdateUnowned and Update, why do we need the
Ownership event again? Daniel, can you remember?
[note to self after 5 min: oh right, because there may not be any Update
events after the ownership change]


>  A touch sequence is only considered to be meaningful in its entirety: clients
>  may not capture part of a touch sequence, interpret only that part, and then
>  pass a partial touch sequence on to the next client.  To this end, all clients
> -with active “grabs” on the window hierarchy for a touch sequence receive events
> -from that touch sequence, as well as the client with the deepest selection
> -(i.e. furthest from the root window).  Clients currently in control of the
> -touch sequence receive TouchUpdate events, whereas clients not in control
> -receive TouchUpdateUnowned events.
> +with “grabs” on the window hierarchy for a touch sequence receive events from
> +that touch sequence, as well as potentially the client with the deepest
> +selection (i.e. furthest from the root window).  Clients currently in control of
> +the touch sequence receive TouchUpdate events, whereas clients not in control
> +receive receive TouchUpdateUnowned events.
>  
>  Touch grabs are similar to standard input event grabs in that they take
>  precedence over selections and are searched from the root window to the child
>  window.  The first grab found for a touch sequence will be the owner of that
>  touch sequence, however events for that touch sequence will continue to be
> -delivered to all clients with grabs in the window tree, as well as the client
> -with the deepest selection.  The first client may either “accept” the touch,
> -which claims the touch sequence and stops delivery to all other clients for
> -the duration of the touch sequence, or “reject” the touch sequence, which
> +delivered to all clients with grabs in the window tree, as well as potentially

why "potentially"? I thought the client with the deepest selection would
always get the event.

> +the client with the deepest selection.  The first client may either “accept” the
> +touch, which claims the touch sequence and stops delivery to all other clients
> +for the duration of the touch sequence, or “reject” the touch sequence, which
>  will remove that client from the delivery set and pass ownership on to the
> -next client.
> +next client. When a client, including the initial owner, becomes the owner of a
> +touch, it will receive a TouchOwnership event. When an owning client accepts a
> +touch, further clients receiving unowned events will receive TouchEnd events.
>  
> -Window sets for direct device touches contain the windows from the root to the
> -child in which the touch originated.
> +Clients selecting for touch events may select for either unowned events or only
> +owned events. 

wait, unowend and owned is mutually exclusive? this is a rather severe
change to Daniel's last spec - why the change of heart?

> The event stream for an unowned selection is identical to a touch
> +grab. When a client does not select for unowned and ownership events, it will
> +receive a TouchBegin event when it becomes the owner of a touch stream.

this means you have to buffer Begin + Update events in the server until the
ownership has the right client. Which, IIRC, is the reason we decided not to
re-implement the pointer grab's serial semantics but the parallel ones
sending to all clients.

> +TouchUpdate and TouchEnd events will be received in the same manner as for touch
> +grabs.
> +
> +A TouchEnd event will always be sent to a client when it will receive no more
> +events from a particular touch, regardless of why (grab or selection removed,
> +owner accepted, the client having rejected that touch, etc).
> +
> +Only one client may select or grab touch events for a device on a window. As an
> +example, selecting for AllDevices will prevent any other client from selecting
> +on the same window. When a slave device is attached to a master device, any
> +selections on any windows for touch events for the slave device ID will be
> +canceled. Clients selecting for individual slave devices are suggested to select
> +for HierarchyChanged events to be notified when this occurs.

thanks, this one was definitely missing and needed to be stated explicitly.

> +
> +4.4.2 Touch device modes
> +
> +Touch devices come in many different forms with varying capabilities. The
> +following device modes are defined for this protocol:
> +
> +DirectTouch:
> +    These devices map their input region to a subset of the screen region. Touch
> +    events are delivered according to where the touch occurs in the mapped
> +    screen region. An example of a DirectTouch device is a touchscreen.
> +
> +DependentTouch:
> +    These devices do not have a direct correlation between a touch location and
> +    a position on the screen. Touch events are delivered according to the
> +    location of the pointer on screen. An Example of a DependentTouch device
> +    is a trackpad.
> +
> +IndependentPointer:
> +    These devices do not have any correlation between touch events and pointer
> +    events. IndependentPointer devices are a subset of DependentTouch devices.
> +    An example of an IndependentPointer device is a mouse with a touch surface.

I don't quite understand what the difference to DependentTouch is here.
To me, a mouse with a touch surface is identical to a trackpad in how
gestures would be interpreted. At least that's how I'd use it, so having
this as a separate device type seems confusing.

> +SemiMultitouch:
> +    These devices may report touch events that correlate to the two opposite
> +    corners of the bounding box of all touches. The number of active touch
> +    sequences represents the number of touches on the device, and the position
> +    of any given touch event will be equal to either of the two corners of the
> +    bounding box. However, the physical location of the touches is unknown.
> +    SemiMultitouch devices are a subset of DependentTouch devices. Although
> +    DirectTouch and IndependentPointer devices may also be SemiMultitouch
> +    devices, such devices are not allowed through this protocol.

No. if the device can't do multitouch, hack around it and make it multitouch
for our purpose. we already require devices to track touchpoints through
mtdev, so I don't really see the need for this. 

> +
> +A device is identified as only one of the device modes above at any time. For
> +the purposes of this protocol, IndependentPointer and SemiMultitouch devices are
> +treated the same as DependentTouch devices unless stated otherwise.
> +
> +4.4.3 Touch event delivery
> +
> +Window sets for event propagation for direct device touches contain the windows
> +from the root to the child in which the touch originated.
>  
>  Dependent device window sets depend on whether other touches are active. For
>  the first dependent touch on a device, the window set contains the windows from
>  the root to the current window underneath the position of the device's pointer.
> -For subsequent touches on this device, the window set is identical to
> -the window set of the first touch. Once all touches have been released, the
> -window set is reset and re-calculated on the first subsequent touch.
> -
> -No touches from a dependent device may begin while the device is floating, as
> -it does not have an associated pointer position to focus events.
> -
> -If there is no touch grab on a window and the server supports pointer
> -emulation, it will look for a pointer grab on that window and include it as
> -part of the delivery set; similarly, if no client has selected to receive
> -touch events on a window, the server may look for pointer event selections
> -on that window to add to the delivery set.
> +For subsequent touches on the device, the window set is identical to the window
> +set of the first touch. Once all touches have been released, the window set is
> +reset and re-calculated on the first subsequent touch.
>  
>  The delivery of touch events is not changed by any modifications to the window
>  hierarchy after the window set has been determined for the touch, nor is it
>  affected by new grabs or selections.
>  
> -A TouchEnd event will always be sent to a client when it will receive no more
> -events from a particular touch, regardless of why (grab or selection removed,
> -owner accepted, the client having rejected that touch, etc).
> +No touches from a dependent device may begin while the device is floating, as
> +it does not have an associated pointer position to focus events.
>  
> -A device that sends touch events may also generate pointer events on demand.
> -The decision of which touch events are converted into pointer events is
> -implementation-specific, however any pointer event generated from a touch
> -event will have the PointerEmulated flag set. Emulated pointer/keyboard events
> -follow the core and XI2 grab semantics.
> +In order to prevent touch events delivered to one window while pointer events
> +are implicitly grabbed by another, all touches from indirect devices will end
> +when an implicit grab is activated on the slave or attached master device. New
> +touches may begin while the device is implicitly grabbed.
> +
> +Many touch devices will emit pointer events as well, usually by mapping one
> +touch sequence to pointer events. In these cases, events for both the pointer
> +and its associated touch sequence will have the XIPointerEmulated flag set.
> +
> +4.4.4 Pointer emulation for direct touch devices
> +
> +In order to facilitate backwards compatibility with legacy clients, direct touch
> +devices will emulate pointer events. Pointer emulation events will only be
> +delivered through the attached master device; 

so a pointer emulation event cannot be used on the SD? is this correct?

having said that, we have not specified what happens to pointer emulation on
dependent devices like the magic trackpad. urgh.


> + no pointer events will be emulated
> +for floating touch devices. Further, only one touch from any attached slave
> +touch device may be emulated per master device at any time.


> +
> +A touch event stream must be delivered to clients in a mutually exclusive
> +fashion. This extends to emulated pointer events. For the purposes of
> +exclusivity, emulated pointer events between an emulated button press and
> +button release are considered. An emulated button press event is considered
> +exclusively delivered once it has been delivered through an event selection, an
> +asynchronous pointer grab, or it and a further event are delivered through a
> +synchronous pointer grab.

the last sentence is the meat of the paragraph here, the first three simply
served to confuse me :)

> +
> +Touch and pointer grabs are also mutually exclusive. For a given window, any
> +touch grab is activated first. If the touch grab is rejected, the pointer grab
> +is activated. If an emulated button press event is exclusively delivered to the
> +grabbing client as outlined above, the touch sequence is ended for all clients
> +still listening for unowned events. 

this means you're losing touch events between the grab activating and 
AllowEvents(ReplayPointer).

also, the wording here is ambiguous since "rejecting" a touch is not the
same as "no touch grab" but has the same outcome in this case.

> Otherwise, when the pointer stream is
> +replayed the next window in the window set is checked for touch grabs.
> +
> +If the touch sequence is not exclusively delivered to any client through a grab,

this may be a language barrier issue, but "not exclusively delivered to any
client" doesn't sound exclusive at all to me.

> +the touch and emulated pointer events may be delivered to clients selecting for
> +the events. Event propagation for the touch sequence ends at the first client
> +selecting for touch and/or pointer events. Note that a client may receive both
> +touch and emulated pointer events for the same touch sequence through event
> +selection.

if a window receives touch + pointer, the pointer event will trigger an
implicit passive grab which cancels out the touch (at least for dependent
devices as you stated above). thus, you can never actually send touch events
to any window with both selected.

This section needs to be rewritten (I tried, but it's getting late and my
brain is mushy). I'll try again tomorrow.

>  4.5. The ClientPointer principle
>  
> @@ -425,7 +511,8 @@ are required to be 0.
>                    max:                  FP3232
>                    resolution:           CARD32 }
>  
> -    TOUCHMODE* { DirectTouch, DependentTouch }
> +    TOUCHMODE* { DirectTouch, DependentTouch, IndependentPointer,
> +                 SemiMultitouch }
>  
>      * since XI 2.1
>  
> @@ -540,11 +627,7 @@ are required to be 0.
>      sourceid
>          The device this class originates from.
>      mode
> -        The device type of the touch device. Touch sequences from a device
> -        of type DirectTouch should be interpreted as directly occuring on
> -        the position they are reported on. Touch sequences from a device of
> -        type DependentTouch should be interpreted as dependent of the
> -        current position of the pointer.
> +        The device type of the touch device.
>      num_touches
>          The maximum number of simultaneous touchpoints the device may send.
>          If num_touches is 0, the number of supported touches is unknown or
> @@ -866,6 +949,9 @@ are required to be 0.
>      master
>          The new master device to attach this slave device to.
>  
> +    If any clients are selecting for touch events from the slave device, their
> +    selection will be canceled.
> +
>      DETACHSLAVE detaches a slave device from its current master device.
>      type
>          Always ChangeAttachment.
> @@ -1538,9 +1624,9 @@ are required to be 0.
>      sequence to direct further delivery.
>  
>      deviceid
> -        The grabbed device ID.
> +        The slave device ID for a grabbed touch sequence.
>      touchid
> -        The ID of the currently-grabbed touch sequence.
> +        The ID of the touch sequence to modify.
>      event_mode
>          Given TouchOwnerAccept, the client is deemed to have taken control
>          of the touch sequence; TouchEnd events will be sent to all other
> @@ -1739,9 +1825,8 @@ EVENTHEADER { type:                       BYTE
>  
>      DEVICEEVENTFLAGS (all events): none
>      DEVICEEVENTFLAGS (key events only): { KeyRepeat }
> -    DEVICEEVENTFLAGS (pointer events only): { PointerEmulated }
> +    DEVICEEVENTFLAGS (pointer and touch events only): { PointerEmulated }
>      DEVICEEVENTFLAGS (touch events only): { TouchPendingEnd }
> -    DEVICEEVENTFLAGS (touch end events only): { TouchAccepted }
>  
>      An XIDeviceEvent is generated whenever the logical state of a device
>      changes in response to a button press, a button release, a motion, a key
> @@ -1840,13 +1925,10 @@ EVENTHEADER { type:                       BYTE
>      Touch tracking IDs are provided in the detail field of touch events. Its
>      value is always provided in every touch event. Tracking IDs are
>      represented as unsigned 32-bit values and increase in value for each new
> -    touch, wrapping back to 0 upon reaching the numerical limit of IDs.
> +    touch, wrapping back to 0 upon reaching the numerical limit of IDs. IDs are
> +    unique per each slave touch device.
>  
> -    Touch events do not generate enter/leave events, and are not affected by
> -    other grabs (including device freezing), although any pointer events
> -    generated by emulation will still be subject to all the same constraints
> -    as normal pointer events, including enter/leave events, and being affected
> -    by frozen devices.
> +    Touch events do not generate enter/leave events.

Any reason why you removed the pointer event bit here?

Cheers,
  Peter

>  
>      ┌───
>          RawEvent
> -- 
> 1.7.4.1
>