[PATCH inputproto xi 2.1] Updates for pointer emulation and more touch device modes
Chase Douglas
chase.douglas at canonical.com
Wed Mar 2 06:25:40 PST 2011
On 03/02/2011 02:29 AM, Peter Hutterer wrote:
> On Tue, Feb 22, 2011 at 10:06:37AM -0500, Chase Douglas wrote:
>> Also includes resolutions for dependent devices and implicit grabs and
>> how to handle slave touch device attachment and touch selections.
>>
>> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
>> ---
>>
>> Diff is against latest patch sent by Daniel Stone. See
>> http://cgit.freedesktop.org/~cndougla/inputproto for the cumulative XI 2.1
>> changes.
>>
>> Known issues:
>> * Inert grabs and TouchOwnerRejectContinue are not detailed in the spec.
>>
>> As a side note, I have implemented this spec, which will be pushed to Ubuntu as
>> pre-release functionality today. I am hoping that the ABI can be considered
>> stable at this point unless we find bugs with it. My implementation in the
>> server and input modules is complete excepting the following:
>>
>> * Inert grabs and TouchOwnerRejectContinue
>> * Proper handling of IndependentPointer and SemiMultitouch devices
>> * Touch selection canceling when attaching a touch device
>> * Implicit grab touch canceling
>>
>> I plan to implement all of the above before the release of Ubuntu 11.04 except
>> Inert functionality and IndependentPointer handling in the evdev input module.
>>
>> I will send patches for the server once I have them cleaned up more, but they
>> can be found at http://cgit.freedesktop.org/~cndougla/xserver in the multitouch
>> branch.
>>
>> XI2.h | 9 ++-
>> XI2proto.txt | 186 +++++++++++++++++++++++++++++++++++++++++----------------
>> 2 files changed, 139 insertions(+), 56 deletions(-)
>>
>> diff --git a/XI2.h b/XI2.h
>> index 40c9ca6..dbf14a8 100644
>> --- a/XI2.h
>> +++ b/XI2.h
>> @@ -32,6 +32,7 @@
>> #define Dont_Check 0
>> #endif
>> #define XInput_2_0 7
>> +#define XInput_2_1 8
>
> iirc, this needs to move into libXi. these are libXi-only fields and I'm not
> sure why I never pushed the patch
> http://www.mail-archive.com/xorg-devel@lists.x.org/msg12278.html
Ahh, ok. I'll remove it from here.
>> #define XI_2_Major 2
>> @@ -132,16 +133,16 @@
>> /* Device event flags (common) */
>> /* Device event flags (key events only) */
>> #define XIKeyRepeat (1 << 16)
>> -/* Device event flags (pointer events only) */
>> +/* Device event flags (pointer and touch events only) */
>> #define XIPointerEmulated (1 << 16)
>> /* Device event flags (touch events only) */
>> -#define XITouchPendingEnd (1 << 16)
>> -/* Device event flags (touch end events only) */
>> -#define XITouchAccepted (1 << 17)
>> +#define XITouchPendingEnd (1 << 17)
>>
>> /* Touch modes */
>> #define XIDirectTouch 1
>> #define XIDependentTouch 2
>> +#define XIIndependentPointer 3
>> +#define XISemiMultitouch 4
>>
>> /* XI2 event mask macros */
>> #define XISetMask(ptr, event) (((unsigned char*)(ptr))[(event)>>3] |= (1 << ((event) & 7)))
>> diff --git a/XI2proto.txt b/XI2proto.txt
>> index 5ebe59d..212425d 100644
>> --- a/XI2proto.txt
>> +++ b/XI2proto.txt
>> @@ -52,7 +52,7 @@ not always necessary.
>> The additions in XI 2.1 aim to:
>> - support a dynamic number of simultaneous touch points,
>> - support devices that are both multi-touch and traditional pointer devices,
>> -- while supporting pre-XI2.1 clients through emulation of XI 2.0 and core
>> +- while supporting pre-XI2.1 clients through emulation of XInput and core
>> pointer events.
>>
>> XI 2.1 caters for two modes of touch input devices:
>> @@ -194,6 +194,9 @@ delivered on W. Once an event has been delivered as either XI or core event,
>> event processing stops.
>>
>> 4.4 Touch device support
>> +
>> +4.4.1 Touch event sequences
>> +
>> Touch event processing differs from normal event processing in a few ways,
>> most notably in that touch events are processed partially out-of-band from
>> pointer and keyboard events.
>> @@ -205,62 +208,145 @@ to touch the device. The init and destroy stage of this sequence are always
>> present, while the move stage is optional. Within this document, the term
>> "touch sequence" is used to describe the above chain of events. A client
>> wishing to receive touch events must register for at least TouchBegin,
>> -TouchOwnership, TouchUpdate, and TouchEnd simultaneously; it may also select
>> -for TouchUpdateUnowned events if it wants to receive the full touch stream,
>> -rather than just the final state.
>> +TouchUpdate, and TouchEnd simultaneously. It may also select for
>> +TouchUpdateUnowned and TouchOwnership events if it wants to receive the full
>> +touch stream while other clients own or have active grabs involving the touch.
>> +An example of this usage is a touch painting program that paints touches while
>> +a gesture recognizer has an active grab. Such clients must be able to undo state
>> +if the touch ends without the client receiving ownership of the touch.
>
> I'm a big fan of examples, but not in the middle of the spec. Examples are
> almost always imprecise simplifications and having one example is usually
> not enough and two examples too many. The same can be said:
>
> A client selecting for TouchUpdateUnowned will receive touch events even if it is
> not the current owner of the event. A client selecting for TouchOwnership
> events will receive said event when it becomes the current owner of the
> touch.
This is fine with me. I'll update the text.
> Examples belong into the appendix. Having said that - if we have two
> different event types for UpdateUnowned and Update, why do we need the
> Ownership event again? Daniel, can you remember?
> [note to self after 5 min: oh right, because there may not be any Update
> events after the ownership change]
Yep :).
>> A touch sequence is only considered to be meaningful in its entirety: clients
>> may not capture part of a touch sequence, interpret only that part, and then
>> pass a partial touch sequence on to the next client. To this end, all clients
>> -with active “grabs” on the window hierarchy for a touch sequence receive events
>> -from that touch sequence, as well as the client with the deepest selection
>> -(i.e. furthest from the root window). Clients currently in control of the
>> -touch sequence receive TouchUpdate events, whereas clients not in control
>> -receive TouchUpdateUnowned events.
>> +with “grabs” on the window hierarchy for a touch sequence receive events from
>> +that touch sequence, as well as potentially the client with the deepest
>> +selection (i.e. furthest from the root window). Clients currently in control of
>> +the touch sequence receive TouchUpdate events, whereas clients not in control
>> +receive receive TouchUpdateUnowned events.
>>
>> Touch grabs are similar to standard input event grabs in that they take
>> precedence over selections and are searched from the root window to the child
>> window. The first grab found for a touch sequence will be the owner of that
>> touch sequence, however events for that touch sequence will continue to be
>> -delivered to all clients with grabs in the window tree, as well as the client
>> -with the deepest selection. The first client may either “accept” the touch,
>> -which claims the touch sequence and stops delivery to all other clients for
>> -the duration of the touch sequence, or “reject” the touch sequence, which
>> +delivered to all clients with grabs in the window tree, as well as potentially
>
> why "potentially"? I thought the client with the deepest selection would
> always get the event.
I wasn't sure how to work this in, and "potentially" seemed the easiest
if not the clearest :). Clients can select for touch events without
selecting for unowned events. Thus, the client with the deepest
selection may or may not receive the event depending on whether unowned
events are selected for. This seemed glossed over without adding
"potentially", but perhaps adding "potentially" isn't a satisfactory
addition either. I'll try to think of a better way of stating this.
>> +the client with the deepest selection. The first client may either “accept” the
>> +touch, which claims the touch sequence and stops delivery to all other clients
>> +for the duration of the touch sequence, or “reject” the touch sequence, which
>> will remove that client from the delivery set and pass ownership on to the
>> -next client.
>> +next client. When a client, including the initial owner, becomes the owner of a
>> +touch, it will receive a TouchOwnership event. When an owning client accepts a
>> +touch, further clients receiving unowned events will receive TouchEnd events.
>>
>> -Window sets for direct device touches contain the windows from the root to the
>> -child in which the touch originated.
>> +Clients selecting for touch events may select for either unowned events or only
>> +owned events.
>
> wait, unowend and owned is mutually exclusive? this is a rather severe
> change to Daniel's last spec - why the change of heart?
I'm not sure I understand what you're implying, but it's probably due to
the wording. Logically, it's: (select for unowned and eventually owned
events || select for owned events only). I'll try to think of a better
phrasing for this.
>> The event stream for an unowned selection is identical to a touch
>> +grab. When a client does not select for unowned and ownership events, it will
>> +receive a TouchBegin event when it becomes the owner of a touch stream.
>
> this means you have to buffer Begin + Update events in the server until the
> ownership has the right client. Which, IIRC, is the reason we decided not to
> re-implement the pointer grab's serial semantics but the parallel ones
> sending to all clients.
My implementation has a bounded ring buffer with N events for each
touch. If you overrun the ring buffer, then you'll get the touch begin
event, you'll miss some events from the beginning of the sequence, and
then you'll get the last N events.
The reason this was added was to reduce the need for clients to listen
for unowned events, who may be woken up on every touch event yet never
become the owner. This can be a power drain on mobile devices.
I've been meaning to add a bit of text saying that clients selecting
only for owned events may miss touch update events from the beginning of
the sequence.
>> +TouchUpdate and TouchEnd events will be received in the same manner as for touch
>> +grabs.
>> +
>> +A TouchEnd event will always be sent to a client when it will receive no more
>> +events from a particular touch, regardless of why (grab or selection removed,
>> +owner accepted, the client having rejected that touch, etc).
>> +
>> +Only one client may select or grab touch events for a device on a window. As an
>> +example, selecting for AllDevices will prevent any other client from selecting
>> +on the same window. When a slave device is attached to a master device, any
>> +selections on any windows for touch events for the slave device ID will be
>> +canceled. Clients selecting for individual slave devices are suggested to select
>> +for HierarchyChanged events to be notified when this occurs.
>
> thanks, this one was definitely missing and needed to be stated explicitly.
>
>> +
>> +4.4.2 Touch device modes
>> +
>> +Touch devices come in many different forms with varying capabilities. The
>> +following device modes are defined for this protocol:
>> +
>> +DirectTouch:
>> + These devices map their input region to a subset of the screen region. Touch
>> + events are delivered according to where the touch occurs in the mapped
>> + screen region. An example of a DirectTouch device is a touchscreen.
>> +
>> +DependentTouch:
>> + These devices do not have a direct correlation between a touch location and
>> + a position on the screen. Touch events are delivered according to the
>> + location of the pointer on screen. An Example of a DependentTouch device
>> + is a trackpad.
>> +
>> +IndependentPointer:
>> + These devices do not have any correlation between touch events and pointer
>> + events. IndependentPointer devices are a subset of DependentTouch devices.
>> + An example of an IndependentPointer device is a mouse with a touch surface.
>
> I don't quite understand what the difference to DependentTouch is here.
> To me, a mouse with a touch surface is identical to a trackpad in how
> gestures would be interpreted. At least that's how I'd use it, so having
> this as a separate device type seems confusing.
I'll take Qt as an example. If you have a touchscreen device, it passes
each touch to the widget they landed on. If you have a touchpad, by
default they don't send any touch events until two or more touches are
active. This is due to the duality of the touchpad as a pointer
controlling device and as a touch surface.
Although they don't have any support for other types of devices yet, I
would assume they would handle IndependentPointer devices differently. I
don't see any reason for withholding the first touch for these devices,
so Qt must have a way of knowing about these devices.
>> +SemiMultitouch:
>> + These devices may report touch events that correlate to the two opposite
>> + corners of the bounding box of all touches. The number of active touch
>> + sequences represents the number of touches on the device, and the position
>> + of any given touch event will be equal to either of the two corners of the
>> + bounding box. However, the physical location of the touches is unknown.
>> + SemiMultitouch devices are a subset of DependentTouch devices. Although
>> + DirectTouch and IndependentPointer devices may also be SemiMultitouch
>> + devices, such devices are not allowed through this protocol.
>
> No. if the device can't do multitouch, hack around it and make it multitouch
> for our purpose. we already require devices to track touchpoints through
> mtdev, so I don't really see the need for this.
This is the most "hacking" one can do with these devices. What is stated
above is all you can get out of the hardware. There's a ton of these
devices in use, so I think it is reasonable to cater for them. You can
still do pan and pinch gestures with two and three fingers on synaptics
semi-mt trackpads, and we will want the ability to recognize these in an
X client.
>> +A device is identified as only one of the device modes above at any time. For
>> +the purposes of this protocol, IndependentPointer and SemiMultitouch devices are
>> +treated the same as DependentTouch devices unless stated otherwise.
>> +
>> +4.4.3 Touch event delivery
>> +
>> +Window sets for event propagation for direct device touches contain the windows
>> +from the root to the child in which the touch originated.
>>
>> Dependent device window sets depend on whether other touches are active. For
>> the first dependent touch on a device, the window set contains the windows from
>> the root to the current window underneath the position of the device's pointer.
>> -For subsequent touches on this device, the window set is identical to
>> -the window set of the first touch. Once all touches have been released, the
>> -window set is reset and re-calculated on the first subsequent touch.
>> -
>> -No touches from a dependent device may begin while the device is floating, as
>> -it does not have an associated pointer position to focus events.
>> -
>> -If there is no touch grab on a window and the server supports pointer
>> -emulation, it will look for a pointer grab on that window and include it as
>> -part of the delivery set; similarly, if no client has selected to receive
>> -touch events on a window, the server may look for pointer event selections
>> -on that window to add to the delivery set.
>> +For subsequent touches on the device, the window set is identical to the window
>> +set of the first touch. Once all touches have been released, the window set is
>> +reset and re-calculated on the first subsequent touch.
>>
>> The delivery of touch events is not changed by any modifications to the window
>> hierarchy after the window set has been determined for the touch, nor is it
>> affected by new grabs or selections.
>>
>> -A TouchEnd event will always be sent to a client when it will receive no more
>> -events from a particular touch, regardless of why (grab or selection removed,
>> -owner accepted, the client having rejected that touch, etc).
>> +No touches from a dependent device may begin while the device is floating, as
>> +it does not have an associated pointer position to focus events.
>>
>> -A device that sends touch events may also generate pointer events on demand.
>> -The decision of which touch events are converted into pointer events is
>> -implementation-specific, however any pointer event generated from a touch
>> -event will have the PointerEmulated flag set. Emulated pointer/keyboard events
>> -follow the core and XI2 grab semantics.
>> +In order to prevent touch events delivered to one window while pointer events
>> +are implicitly grabbed by another, all touches from indirect devices will end
>> +when an implicit grab is activated on the slave or attached master device. New
>> +touches may begin while the device is implicitly grabbed.
>> +
>> +Many touch devices will emit pointer events as well, usually by mapping one
>> +touch sequence to pointer events. In these cases, events for both the pointer
>> +and its associated touch sequence will have the XIPointerEmulated flag set.
>> +
>> +4.4.4 Pointer emulation for direct touch devices
>> +
>> +In order to facilitate backwards compatibility with legacy clients, direct touch
>> +devices will emulate pointer events. Pointer emulation events will only be
>> +delivered through the attached master device;
>
> so a pointer emulation event cannot be used on the SD? is this correct?
I don't see any need for pointer emulation on an SD, especially when the
device isn't really emitting pointer events.
> having said that, we have not specified what happens to pointer emulation on
> dependent devices like the magic trackpad. urgh.
Do we need to say anything? There's no pointer "emulation", per se, for
trackpads. Pointer events are sent independently through X for these
devices. The only distinction is that there is usually, but not required
to be, a logical mapping between one of the touches and the pointer motion.
>> + no pointer events will be emulated
>> +for floating touch devices. Further, only one touch from any attached slave
>> +touch device may be emulated per master device at any time.
>
>
>> +
>> +A touch event stream must be delivered to clients in a mutually exclusive
>> +fashion. This extends to emulated pointer events. For the purposes of
>> +exclusivity, emulated pointer events between an emulated button press and
>> +button release are considered. An emulated button press event is considered
>> +exclusively delivered once it has been delivered through an event selection, an
>> +asynchronous pointer grab, or it and a further event are delivered through a
>> +synchronous pointer grab.
>
> the last sentence is the meat of the paragraph here, the first three simply
> served to confuse me :)
I believe it's all necessary. Hopefully the following example will show why:
phyiscal touch begin
emulated pointer motion
emulated pointer button press
touch begin
physical touch move
emulated pointer motion
touch update
physical touch end
emulated pointer button release
touch end
The very first emulated pointer motion moves the cursor to where the
touch begins on the screen. This event is not considered for
exclusivity. It will always be sent whether or not the touch sequence or
further emulated pointer events are delivered. This is why the second
sentence is needed.
>> +Touch and pointer grabs are also mutually exclusive. For a given window, any
>> +touch grab is activated first. If the touch grab is rejected, the pointer grab
>> +is activated. If an emulated button press event is exclusively delivered to the
>> +grabbing client as outlined above, the touch sequence is ended for all clients
>> +still listening for unowned events.
>
> this means you're losing touch events between the grab activating and
> AllowEvents(ReplayPointer).
How? (I've got this working here just fine, with both active and passive
pointer grabs and touch grabs and selecting clients)
> also, the wording here is ambiguous since "rejecting" a touch is not the
> same as "no touch grab" but has the same outcome in this case.
I'll try to clean this up.
>> Otherwise, when the pointer stream is
>> +replayed the next window in the window set is checked for touch grabs.
>> +
>> +If the touch sequence is not exclusively delivered to any client through a grab,
>
> this may be a language barrier issue, but "not exclusively delivered to any
> client" doesn't sound exclusive at all to me.
Part of a touch sequence may be delivered to a client without actually
being "exclusively" delivered to a client. For example, the first button
press may be delivered to a passively grabbing window manager before
being replayed. This is a delivery of part of the touch sequence, but
it's not an "exclusive" delivery.
>> +the touch and emulated pointer events may be delivered to clients selecting for
>> +the events. Event propagation for the touch sequence ends at the first client
>> +selecting for touch and/or pointer events. Note that a client may receive both
>> +touch and emulated pointer events for the same touch sequence through event
>> +selection.
>
> if a window receives touch + pointer, the pointer event will trigger an
> implicit passive grab which cancels out the touch (at least for dependent
> devices as you stated above). thus, you can never actually send touch events
> to any window with both selected.
I suppose the wording should note that it does not apply to implicit
grabs. During an implicit grab, both touch and pointer events may be
sent. I'll add this in.
> This section needs to be rewritten (I tried, but it's getting late and my
> brain is mushy). I'll try again tomorrow.
You'll face no opposition from me if you can make this clearer :). I
worry that it's just very complex and will be difficult to understand no
matter how you phrase it.
>> 4.5. The ClientPointer principle
>>
>> @@ -425,7 +511,8 @@ are required to be 0.
>> max: FP3232
>> resolution: CARD32 }
>>
>> - TOUCHMODE* { DirectTouch, DependentTouch }
>> + TOUCHMODE* { DirectTouch, DependentTouch, IndependentPointer,
>> + SemiMultitouch }
>>
>> * since XI 2.1
>>
>> @@ -540,11 +627,7 @@ are required to be 0.
>> sourceid
>> The device this class originates from.
>> mode
>> - The device type of the touch device. Touch sequences from a device
>> - of type DirectTouch should be interpreted as directly occuring on
>> - the position they are reported on. Touch sequences from a device of
>> - type DependentTouch should be interpreted as dependent of the
>> - current position of the pointer.
>> + The device type of the touch device.
>> num_touches
>> The maximum number of simultaneous touchpoints the device may send.
>> If num_touches is 0, the number of supported touches is unknown or
>> @@ -866,6 +949,9 @@ are required to be 0.
>> master
>> The new master device to attach this slave device to.
>>
>> + If any clients are selecting for touch events from the slave device, their
>> + selection will be canceled.
>> +
>> DETACHSLAVE detaches a slave device from its current master device.
>> type
>> Always ChangeAttachment.
>> @@ -1538,9 +1624,9 @@ are required to be 0.
>> sequence to direct further delivery.
>>
>> deviceid
>> - The grabbed device ID.
>> + The slave device ID for a grabbed touch sequence.
>> touchid
>> - The ID of the currently-grabbed touch sequence.
>> + The ID of the touch sequence to modify.
>> event_mode
>> Given TouchOwnerAccept, the client is deemed to have taken control
>> of the touch sequence; TouchEnd events will be sent to all other
>> @@ -1739,9 +1825,8 @@ EVENTHEADER { type: BYTE
>>
>> DEVICEEVENTFLAGS (all events): none
>> DEVICEEVENTFLAGS (key events only): { KeyRepeat }
>> - DEVICEEVENTFLAGS (pointer events only): { PointerEmulated }
>> + DEVICEEVENTFLAGS (pointer and touch events only): { PointerEmulated }
>> DEVICEEVENTFLAGS (touch events only): { TouchPendingEnd }
>> - DEVICEEVENTFLAGS (touch end events only): { TouchAccepted }
>>
>> An XIDeviceEvent is generated whenever the logical state of a device
>> changes in response to a button press, a button release, a motion, a key
>> @@ -1840,13 +1925,10 @@ EVENTHEADER { type: BYTE
>> Touch tracking IDs are provided in the detail field of touch events. Its
>> value is always provided in every touch event. Tracking IDs are
>> represented as unsigned 32-bit values and increase in value for each new
>> - touch, wrapping back to 0 upon reaching the numerical limit of IDs.
>> + touch, wrapping back to 0 upon reaching the numerical limit of IDs. IDs are
>> + unique per each slave touch device.
>>
>> - Touch events do not generate enter/leave events, and are not affected by
>> - other grabs (including device freezing), although any pointer events
>> - generated by emulation will still be subject to all the same constraints
>> - as normal pointer events, including enter/leave events, and being affected
>> - by frozen devices.
>> + Touch events do not generate enter/leave events.
>
> Any reason why you removed the pointer event bit here?
Because it's explained in more detail with more nuance elsewhere in the
spec. There's no need to repeat it here. We could add a note referring
the reader to the correct section if you feel that's necessary.
Thanks,
-- Chase
More information about the xorg-devel
mailing list