[PATCH xi2.1 inputproto] Many more updates to the XI 2.1 protocol

[Peter Hutterer]

sorry, as usual it takes while to get through all this. turns out I had 90%
of it already in my outbox, just never got to the last few paragraphs.

On Tue, Mar 22, 2011 at 03:07:34PM -0400, Chase Douglas wrote:
> On 03/18/2011 02:23 AM, Peter Hutterer wrote:
> > On Thu, Mar 10, 2011 at 03:47:41PM -0500, Chase Douglas wrote:
> >> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
> >> ---
> >> To see the full protocol spec as I make changes, go to:
> >>
> >> http://cgit.freedesktop.org/~cndougla/inputproto
> > 
> > [...]
> >  
> >> Areas known to be lacking consensus agreement:
> >>
> >> * How to handle touch begin for clients who don't want unowned events. I think
> >>   defining touch begin as a representation of the begining of a touch, as
> >>   opposed to an event sent immediately after a touch begins, clarifies things.
> >> * Dropping of TouchUpdateUnowned hasn't been discussed before.
> >> * Should IndependentPointer devices be a separate touch device mode? Peter
> >>   suggested using labels somehow, but I'm not sure what that means. I've left
> >>   things as they were for now.
> >> * Addition of active_touches to DeviceEvent has not been formally reviewed
> >> * Should clients be able to make multiple grabs/selections per physical device
> >>   per window? This also affects whether slave touch selections are canceled on
> >>   device attachment.
> >> * Should direct touch emulated pointer events send motion then button press
> >>   events, or just button press events? I worry about XI 1.x, which turns a
> >>   button press into two or more events (button press + 1 or more valuator
> >>   events) and how that could affect things. I think it's easier to leave it as
> >>   it is.
> > 
> > Slugging my way through it. fwiw, I didn't look at the patch but rather at
> > the protocol spec as a whole to check if it makes sense when you _dont_ see
> > what's been there before. I've read through the whole protocol, but these
> > comments here affect mainly the top section before the actual protocol
> > descriptions start.
> > 
> > Comments:
> > * the different TouchBegin semantics (owner/non-owner) IMO make sense
> >   because they simply reduce the semantics to "Begin" is the first event
> >   you'll get. this made less sense in the patch but reading the spec in one
> >   go it seems to fit nicely
> > * there were several sections where I got badly confused. I've tried to
> >   improve them, feel free to point out where things aren't clear.
> 
> The I looked at this commit:
> http://cgit.freedesktop.org/~whot/inputproto/commit/?h=chase-multitouch&id=7ebdae4527382d36c69d999883882566c90b19b0
> 
> The changes are fine by me, but it feels like I know the protocol to
> well now and I'm just affirming that everything is right rather than
> that everything is complete :).

note that the latest changes are in the multitouch branch, chase-multitouch
was a branch I used locally for the asciidoc merging

http://cgit.freedesktop.org/~whot/inputproto/commit/?h=multitouch&id=ed840d79d3cac60b2fb17448afcc28828236e91b
not that much different, but features some additional changes

> > * it is not clear whether a passive grab can be established for owner-only.
> >   does a grab always require the ownership semantics or can there be the
> >   default TouchBegin semantics too? If so, how is this decided?
> 
> The implementation so far requires that a grabber use unowned event
> semantics. 

does the _protocol_ require it?
remember what I said about implementations?

> A client cannot grab for owner-only events. This is partly
> because it's easier and partly because I think touch grabbing clients
> either will need the functionality or will be able to handle it easily.
> 
> For example, say you have a touch-aware window manager. Generally the WM
> only needs to focus a window that's been interacted with or move the
> window if you interact with the title bar. Both scenarios are usually
> state driven rather than path driven. The WM can raise the window and
> can keep track of just the last touch location to move the window when
> it becomes the owner.

counter-example: popup window. i don't care what happens before I get an
event that's really mine, so why would I listen to it.
 
> Given time constraints and no use cases that I'm aware of for the
> functionality, I'd suggest leaving it as is. I don't believe this will
> inhibit adding the functionality in XI 2.2 if it is deemed necessary either.

uhm. we don't have time constraints. yes, the merge window will end at some
point but if it's not ready by then, there will be another release. given
protocols are written in stone once they're officially released, we'd better
make sure we're happy with it.
 
> > * can we allow more than one TouchObserve grab on the root window? after
> >   all, they won't be doing anything with it anyway
> 
> I'd say yes, and it's not limited to only the root window.
> 
> > * it seems impossible to be an observer that gets ownership. for a situation
> >   where you are below the accepting client, this means you're stuck. you can
> >   either observe (and never get ownership) or not observe (and lose those
> >   touch sequence the parent accepts)
> 
> Can't the client have a passive grab and then reject ownership but
> continue receiving events as an observer?

for observing, you get all events. if you're not observing, if a client
above you accepts the grab, you get a TouchEnd so you never get to opt for
RejectObserve.

> > * the TouchObserve grab concept as-is doesn't quite fit. the idea originally
> >   came up to solve the problem of a grabber that wants to keep getting
> >   events after rejecting. That is what TouchRejectObserve does. The grab
> >   mode TouchObserve doesn't fit. if you can't become owner anyway, this is
> >   hardly different to RawEvents so we could use those.
> 
> I'm hopeful that my last comment would suffice here too :)
> 
> >   which brings me to
> > * no RawEvents for touches? would be a good time to fix the broken grab
> >   semantics for raw events too (deliver them even if a grab is present)
> 
> We could add them now, or we could push them off to XI 2.2. I'm
> beginning to get worried about landing XI 2.1 in xserver 1.11. I haven't
> really thought about raw touch events either. I'd rather let the idea
> percolate and/or wait for a use case for them to be more clear.

planning for 2.2 is hard. I remember pushing off features for 2.1 and then
got sidetracked with a million other things. so I'm somewhat hesitant to
have features pushed off for 2.2 (as opposed to "planned for 2.2". how much
difference one word can make...)
I'm all too worried that once the first version is out everyone moves on to
whatever the next shiny thing on the horizon is.

> > * SemiMultitouch simply needs a better name. Why not use BoundingBox or
> >   something more descriptive?
> 
> That's fine with me.
> 
> > * If SemiMultitouch isn't good enough to give you touch points, how is the
> >   pointer emulation decided?
> 
> The X input module still generates the pointer events. How it does this
> is implementation specific. However, SemiMultitouch is limited to
> trackpads. Trackpads are also limited in that they only generate pointer
> motion events when only one touch is active.

SemiMultitouch is _currently_ limited to trackpads. I wouldn't put it past
hardware vendors to ignore this fine assumption and come out with some other
device that doesn't do multitouch propertly.
but anyway, something like "for devices of type SemiMultitouch, pointer
emulation is implementation dependent" may be useful to clear up confusion.

> > * as pointed out in the other email, I'm still confused on whether the
> >   master device delivers touch events. this isn't mentioned in the spec but
> >   the last patchset I reviewed skipped touch events from the master device
> 
> I'd just forget any XI 2.1 implementation patchsets you've seen so far
> :). The real stuff will look very different.
> 
> The master device still delivers touch events from attached slave
> devices. Here's an example:
> 
> * Slave device S is attached to master device M
> * Client A grabs touch events on the root window R on slave device S
> * Client B selects for all touch events (including unowned) on top level
> window W on master device M
> 
> When the user touches on window W, touch events are sent with device id
> of S to client A. Touch events are also sent with device id of M to
> client B (though the slave id is set to the id of S).
> 
> This allows for one client to select for all master devices and not
> receive events from a floating slave device.

ok, so we're using normal XI2 semantics. fine with me then, though Daniel
expressed some issues with that.

> > * in the touch event delivery section, the sentence "No touches from an
> > * indirect device may begin while the
> >   device is floating, as it does not have an associated pointer position to
> >   focus events." This is incorrect. Run XIQueryPointer on the floating
> >   slave, you'll see it does have a pointer position. In fact, it even has a
> >   sprite, it's just not rendered.
> >   this is in-line with XI1 where clients are expected to manually
> >   render the cursor.
> 
> What do you suggest as a resolution? Better wording so it matches what
> is described, or that we actually send touch events to the pointer
> position of the floating device?

the latter, unless there's a good reason not to. we already expect clients
to know what floating slaves are, so if they select for events from them we
can assume they know what they're doing.

> > * pointer event handling for indirect devices: what was the motivation for
> >   withholding touch events when the pointer leaves the window again? 
> >   It isn't clear what "withheld" means, even if you have the ownership
> >   selection, you still don't get it? this whole concept feels a bit tacked
> >   on the side and breaks ownership assumptions (that touch events are
> >   delivered immediately if you can cope with ownership changes).
> 
> It is hacky and tacked on the side. The problem it resolves is when a
> client has a touch selection on a window, is receiving touch events from
> an indirect device, and the cursor then moves outside the window. The
> pointer button may then be pressed, activating an implicit grab over a
> different window. Now, you've got pointer events going to the new window
> and touch events being sent to the original window.
> 
> There's a parallel issue when you have touch events from an indirect
> device going to the first window and then a pointer grab is activated on
> a different window.
> 
> I think the most reasonable solution is to have the client ignore touch
> events when the pointer "leaves" the selection/grab window. The two
> approaches I've come up with are:
> 
> 1. Leave it up to the client and tell them to ignore touches when they
> receive a leave notification event.
> 2. Enforce it in the server by withholding events while the pointer is away.
> 
> Option 2 is easier to implement on the client side and less likely to
> cause buggy behavior. This is what I described in the protocol. I'd be
> fine with option 1 though, too.

this leaves me with one problem: I can't perform a gesture to virtually lift
an item, move the cursor elsewhere and then drop it. Because anything I do
once the cursor moves outside the window will be thrown away ("withheld").
I don't quite see where the problem is (from a UI POV) to have touch events
go to two differnet windows provided the touch grab was activated first.
if you really want to avoid this situation, disallow pointer grabs from
other clients while touch grabs are active. that may be a sensible solution.

> > * it is unclear if a client can grab for pointer + touch, both actively and
> >   passively. this is an issue for dependent device, especially because touch
> >   events are withheld during pointer grabs.
> >   I think the event mask should be honoured during grabs if valid.
> 
> I hope the above explanation provides enough detail for how I think this
> should work. Let me know if you still think there's an issue.

not quite, different question. my question was - what happens if I call
XIGrabDevice(XITouchBegin | XIButtonPress), i.e. the grab mask selects for
both pointer and touch events.

> > * a device that emulates pointer events must also initialize valuators
> >   though they may serve no other purpose. should there be a mapping field in
> >   the TouchAxisClass to say which valuator this one maps to in case of
> >   pointer emulation?
> >   for a client it may be important to know that the axis a device advertises
> >   is just there because of pointer emulation.
> >   same goes for the button class, you need one to send button events.
> 
> Lets say a touch device provides pressure information. I think the
> pointer device should have a valuator for pressure information too, and
> when a touch is emulated the valuator data should be valid.
> 
> This is not handled by the X server in any implementation I'm aware of.
> However, I see this as an implementation issue, not a protocol issue. If
> the valuator axis is listed by the device, then the client should assume
> the data exists and will be set appropriately. To do this will require
> the proper input module interface, but shouldn't be dependent on any
> protocol changes.

for a dependent device, if I select for both touch and pointer events, how
do I figure out which data is duplicated then? 
if you want to go into the fine details - if pressure isn't emulated
because I'm a touch client and I thus don't get emulation, the pressure
valuator never sends events. I don't get any indication _why_ this is the
case though since I cannot tell whether this axis actually exists or is just
there for of pointer emulation..
 
> > * it is unclear how pointer emulation works. x/y, fair enough. what about
> >   pressure and other information potentially available?
> 
> I suppose that's answered above :).

yep, see above :)

> > * pointer emulation describes very well how button events are generated very
> >   well (see my comment below though) but I predict that sooner or later
> >   people will want the feature to emulate third buttons per timeout, etc.
> >   any plans?
> 
> That's beyond the scope of XI 2.1, imo, and delves into gesture
> handling. It would open a very large can of worms to be adding in
> pointer event emulation with timeouts so we can enable something like a
> right click.
> 
> You asked for plans though, for which I suppose the obvious answer is to
> integrate our uTouch gesture stack into the toolkits for handling this.
> Canonical will only be targetting the major toolkits though (GTK+ and
> Qt). However, anything reasonable in X should be possible with only one
> primary button. If that were not the case, anyone with an Apple mouse
> other than the Magic Mouse would be SOL.
> 
> > * in the pointer emulation section the protocol states that "Both the
> >   emulated pointer events and their associated touch events will have the
> >   PointerEmulated flag set." That is after a paragraph that states that
> >   pointer and touch events are mutually exclusive. so you get either pointer
> >   or a touch event with that flag but you cannot get the other.
> >   that seems rather pointless. if you select for touch events, you won't get
> >   the emulated events anyway and if you select for pointer events you never
> >   see the touch events. so I'm wondering what that flag conveys other than
> >   the server saying "hey, I know more than you about this event" :)
> 
> For indirect devices you may receive both pointer and motion events. It
> is only for direct devices where you have emulated pointer events that
> you will receive events for one type but not both.

so the gist of this seems to leave us with two outcomes. On direct devices
we cannot get both anyway, so let's ignore them. on dependent devices a
client may either
- register for touch and receive touch events with the flag set. that client
  will never see the pointer events with the flag set but may see pointer
  events without the flag.
- register for pointer events only and see pointer events with _and_ without
  the flag, depending on whether emulation is active. 

is this correct?

> >> * Should IndependentPointer devices be a separate touch device mode? Peter
> >>   suggested using labels somehow, but I'm not sure what that means. I've left
> >>   things as they were for now.
> > 
> > e.g. we can provide a property on the touch device to label is as
> > independent. this isn't something that changes often and it seems to have zero
> > effect on the protocol.
> 
> That would be fine with me. So we should just remove the type from the
> protocol altogether?

yeah, sounds like a plan. 

Cheers,
  Peter

> >> * Should direct touch emulated pointer events send motion then button press
> >>   events, or just button press events? I worry about XI 1.x, which turns a
> >>   button press into two or more events (button press + 1 or more valuator
> >>   events) and how that could affect things. I think it's easier to leave it as
> >>   it is.
> > 
> > don't confuse input drivers, the input API and the XI protocol. atm, it's
> > the drivers converting and afaict there is no requirement in the protocol
> > for this.
> 
> No confusion here :). I'm referring specifically to the
> eventToKeyButtonPointer function in the server. As an example, it takes
> one internal button event that has valuators and creates two or more XI
> 1.x protocol events. I can't give you a specific scenario that will
> break if we don't enforce pointer motion then button press in two
> separate XI events, but this is all rather complex. I *know* that it
> will work separated as described, however.
> 
> > Branch is up in my people repo, not counting typos and the need for section
> > renumbering.
> > 
> >     git://people.freedesktop.org/~whot/inputproto.git multitouch
> 
> Thanks! I'll base any further work on this branch since it already has
> the asciidoc conversion.