[PATCH xi2.1 inputproto] Many more updates to the XI 2.1 protocol

[Chase Douglas]

On 03/18/2011 02:23 AM, Peter Hutterer wrote:
> On Thu, Mar 10, 2011 at 03:47:41PM -0500, Chase Douglas wrote:
>> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
>> ---
>> To see the full protocol spec as I make changes, go to:
>>
>> http://cgit.freedesktop.org/~cndougla/inputproto
> 
> [...]
>  
>> Areas known to be lacking consensus agreement:
>>
>> * How to handle touch begin for clients who don't want unowned events. I think
>>   defining touch begin as a representation of the begining of a touch, as
>>   opposed to an event sent immediately after a touch begins, clarifies things.
>> * Dropping of TouchUpdateUnowned hasn't been discussed before.
>> * Should IndependentPointer devices be a separate touch device mode? Peter
>>   suggested using labels somehow, but I'm not sure what that means. I've left
>>   things as they were for now.
>> * Addition of active_touches to DeviceEvent has not been formally reviewed
>> * Should clients be able to make multiple grabs/selections per physical device
>>   per window? This also affects whether slave touch selections are canceled on
>>   device attachment.
>> * Should direct touch emulated pointer events send motion then button press
>>   events, or just button press events? I worry about XI 1.x, which turns a
>>   button press into two or more events (button press + 1 or more valuator
>>   events) and how that could affect things. I think it's easier to leave it as
>>   it is.
> 
> Slugging my way through it. fwiw, I didn't look at the patch but rather at
> the protocol spec as a whole to check if it makes sense when you _dont_ see
> what's been there before. I've read through the whole protocol, but these
> comments here affect mainly the top section before the actual protocol
> descriptions start.
> 
> Comments:
> * the different TouchBegin semantics (owner/non-owner) IMO make sense
>   because they simply reduce the semantics to "Begin" is the first event
>   you'll get. this made less sense in the patch but reading the spec in one
>   go it seems to fit nicely
> * there were several sections where I got badly confused. I've tried to
>   improve them, feel free to point out where things aren't clear.

The I looked at this commit:
http://cgit.freedesktop.org/~whot/inputproto/commit/?h=chase-multitouch&id=7ebdae4527382d36c69d999883882566c90b19b0

The changes are fine by me, but it feels like I know the protocol to
well now and I'm just affirming that everything is right rather than
that everything is complete :).

> * it is not clear whether a passive grab can be established for owner-only.
>   does a grab always require the ownership semantics or can there be the
>   default TouchBegin semantics too? If so, how is this decided?

The implementation so far requires that a grabber use unowned event
semantics. A client cannot grab for owner-only events. This is partly
because it's easier and partly because I think touch grabbing clients
either will need the functionality or will be able to handle it easily.

For example, say you have a touch-aware window manager. Generally the WM
only needs to focus a window that's been interacted with or move the
window if you interact with the title bar. Both scenarios are usually
state driven rather than path driven. The WM can raise the window and
can keep track of just the last touch location to move the window when
it becomes the owner.

Given time constraints and no use cases that I'm aware of for the
functionality, I'd suggest leaving it as is. I don't believe this will
inhibit adding the functionality in XI 2.2 if it is deemed necessary either.

> * can we allow more than one TouchObserve grab on the root window? after
>   all, they won't be doing anything with it anyway

I'd say yes, and it's not limited to only the root window.

> * it seems impossible to be an observer that gets ownership. for a situation
>   where you are below the accepting client, this means you're stuck. you can
>   either observe (and never get ownership) or not observe (and lose those
>   touch sequence the parent accepts)

Can't the client have a passive grab and then reject ownership but
continue receiving events as an observer?

> * the TouchObserve grab concept as-is doesn't quite fit. the idea originally
>   came up to solve the problem of a grabber that wants to keep getting
>   events after rejecting. That is what TouchRejectObserve does. The grab
>   mode TouchObserve doesn't fit. if you can't become owner anyway, this is
>   hardly different to RawEvents so we could use those.

I'm hopeful that my last comment would suffice here too :)

>   which brings me to
> * no RawEvents for touches? would be a good time to fix the broken grab
>   semantics for raw events too (deliver them even if a grab is present)

We could add them now, or we could push them off to XI 2.2. I'm
beginning to get worried about landing XI 2.1 in xserver 1.11. I haven't
really thought about raw touch events either. I'd rather let the idea
percolate and/or wait for a use case for them to be more clear.

> * SemiMultitouch simply needs a better name. Why not use BoundingBox or
>   something more descriptive?

That's fine with me.

> * If SemiMultitouch isn't good enough to give you touch points, how is the
>   pointer emulation decided?

The X input module still generates the pointer events. How it does this
is implementation specific. However, SemiMultitouch is limited to
trackpads. Trackpads are also limited in that they only generate pointer
motion events when only one touch is active.

> * as pointed out in the other email, I'm still confused on whether the
>   master device delivers touch events. this isn't mentioned in the spec but
>   the last patchset I reviewed skipped touch events from the master device

I'd just forget any XI 2.1 implementation patchsets you've seen so far
:). The real stuff will look very different.

The master device still delivers touch events from attached slave
devices. Here's an example:

* Slave device S is attached to master device M
* Client A grabs touch events on the root window R on slave device S
* Client B selects for all touch events (including unowned) on top level
window W on master device M

When the user touches on window W, touch events are sent with device id
of S to client A. Touch events are also sent with device id of M to
client B (though the slave id is set to the id of S).

This allows for one client to select for all master devices and not
receive events from a floating slave device.

> * in the touch event delivery section, the sentence "No touches from an
> * indirect device may begin while the
>   device is floating, as it does not have an associated pointer position to
>   focus events." This is incorrect. Run XIQueryPointer on the floating
>   slave, you'll see it does have a pointer position. In fact, it even has a
>   sprite, it's just not rendered.
>   this is in-line with XI1 where clients are expected to manually
>   render the cursor.

What do you suggest as a resolution? Better wording so it matches what
is described, or that we actually send touch events to the pointer
position of the floating device?

> * pointer event handling for indirect devices: what was the motivation for
>   withholding touch events when the pointer leaves the window again? 
>   It isn't clear what "withheld" means, even if you have the ownership
>   selection, you still don't get it? this whole concept feels a bit tacked
>   on the side and breaks ownership assumptions (that touch events are
>   delivered immediately if you can cope with ownership changes).

It is hacky and tacked on the side. The problem it resolves is when a
client has a touch selection on a window, is receiving touch events from
an indirect device, and the cursor then moves outside the window. The
pointer button may then be pressed, activating an implicit grab over a
different window. Now, you've got pointer events going to the new window
and touch events being sent to the original window.

There's a parallel issue when you have touch events from an indirect
device going to the first window and then a pointer grab is activated on
a different window.

I think the most reasonable solution is to have the client ignore touch
events when the pointer "leaves" the selection/grab window. The two
approaches I've come up with are:

1. Leave it up to the client and tell them to ignore touches when they
receive a leave notification event.
2. Enforce it in the server by withholding events while the pointer is away.

Option 2 is easier to implement on the client side and less likely to
cause buggy behavior. This is what I described in the protocol. I'd be
fine with option 1 though, too.

> * it is unclear if a client can grab for pointer + touch, both actively and
>   passively. this is an issue for dependent device, especially because touch
>   events are withheld during pointer grabs.
>   I think the event mask should be honoured during grabs if valid.

I hope the above explanation provides enough detail for how I think this
should work. Let me know if you still think there's an issue.

> * a device that emulates pointer events must also initialize valuators
>   though they may serve no other purpose. should there be a mapping field in
>   the TouchAxisClass to say which valuator this one maps to in case of
>   pointer emulation?
>   for a client it may be important to know that the axis a device advertises
>   is just there because of pointer emulation.
>   same goes for the button class, you need one to send button events.

Lets say a touch device provides pressure information. I think the
pointer device should have a valuator for pressure information too, and
when a touch is emulated the valuator data should be valid.

This is not handled by the X server in any implementation I'm aware of.
However, I see this as an implementation issue, not a protocol issue. If
the valuator axis is listed by the device, then the client should assume
the data exists and will be set appropriately. To do this will require
the proper input module interface, but shouldn't be dependent on any
protocol changes.

> * it is unclear how pointer emulation works. x/y, fair enough. what about
>   pressure and other information potentially available?

I suppose that's answered above :).

> * pointer emulation describes very well how button events are generated very
>   well (see my comment below though) but I predict that sooner or later
>   people will want the feature to emulate third buttons per timeout, etc.
>   any plans?

That's beyond the scope of XI 2.1, imo, and delves into gesture
handling. It would open a very large can of worms to be adding in
pointer event emulation with timeouts so we can enable something like a
right click.

You asked for plans though, for which I suppose the obvious answer is to
integrate our uTouch gesture stack into the toolkits for handling this.
Canonical will only be targetting the major toolkits though (GTK+ and
Qt). However, anything reasonable in X should be possible with only one
primary button. If that were not the case, anyone with an Apple mouse
other than the Magic Mouse would be SOL.

> * in the pointer emulation section the protocol states that "Both the
>   emulated pointer events and their associated touch events will have the
>   PointerEmulated flag set." That is after a paragraph that states that
>   pointer and touch events are mutually exclusive. so you get either pointer
>   or a touch event with that flag but you cannot get the other.
>   that seems rather pointless. if you select for touch events, you won't get
>   the emulated events anyway and if you select for pointer events you never
>   see the touch events. so I'm wondering what that flag conveys other than
>   the server saying "hey, I know more than you about this event" :)

For indirect devices you may receive both pointer and motion events. It
is only for direct devices where you have emulated pointer events that
you will receive events for one type but not both.

>> * Should IndependentPointer devices be a separate touch device mode? Peter
>>   suggested using labels somehow, but I'm not sure what that means. I've left
>>   things as they were for now.
> 
> e.g. we can provide a property on the touch device to label is as
> independent. this isn't something that changes often and it seems to have zero
> effect on the protocol.

That would be fine with me. So we should just remove the type from the
protocol altogether?

>> * Should direct touch emulated pointer events send motion then button press
>>   events, or just button press events? I worry about XI 1.x, which turns a
>>   button press into two or more events (button press + 1 or more valuator
>>   events) and how that could affect things. I think it's easier to leave it as
>>   it is.
> 
> don't confuse input drivers, the input API and the XI protocol. atm, it's
> the drivers converting and afaict there is no requirement in the protocol
> for this.

No confusion here :). I'm referring specifically to the
eventToKeyButtonPointer function in the server. As an example, it takes
one internal button event that has valuators and creates two or more XI
1.x protocol events. I can't give you a specific scenario that will
break if we don't enforce pointer motion then button press in two
separate XI events, but this is all rather complex. I *know* that it
will work separated as described, however.

> Branch is up in my people repo, not counting typos and the need for section
> renumbering.
> 
>     git://people.freedesktop.org/~whot/inputproto.git multitouch

Thanks! I'll base any further work on this branch since it already has
the asciidoc conversion.

-- Chase