[RFC] Multi-Touch (MT) support - arbitration or not

[Peter Hutterer]

On Mon, Nov 08, 2010 at 03:54:51PM -0600, Chris Bagwell wrote:
> On Mon, Nov 8, 2010 at 2:08 AM, Benjamin Tissoires <tissoire at cena.fr> wrote:
> > Le 08/11/2010 04:51, Peter Hutterer a écrit :
> >>
> >> fwiw, I'm not sure "arbitrate" is the right word here, filtering seems
> >> easier to understand in this context. I guess "arbitrate" would apply more
> >> if we emit the events across multiple devices like in the bamboo case.
> >> that's mostly bikeshedding though, my points below apply regardless of
> >> what
> >> word we choose :)
> >>
> >> note that we also have two different approaches - single kernel device or
> >> multiple kernel devices and depending on the approach the device uses the
> >> options below have different advantages and disadvantages.
> >>
> >> the tablets I've dealt with so far exposed a single event device, so
> >> that's
> >> what I'm focusing on in this email.
> >>
> >> On Fri, Nov 05, 2010 at 11:47:28AM -0700, Ping Cheng wrote:
> >>>
> >>> Recent changes and discussion about MT support at LKML, UDS, and
> >>> xorg-devel encouraged me to migrate Wacom MT devices to the slot-based
> >>> MT protocol (introduced in kernel 2.6.36). Since Wacom supports both
> >>> digitizer and touch devices, I need to decide how to report touch data
> >>> when the pen is in proximity.
> >>>
> >>> My goal is to understand how X server would like the MT data to be
> >>> reported from the kernel. I hope to keep kernel and X server driver MT
> >>> support in sync so we can avoid unnecessary confusion or extra work in
> >>> the userland.
> >>>
> >>> The existing solution for single touch events is to arbitrate touch
> >>> when pen is in prox. This is based on the assumption that we do not
> >>> want to have two cursors competing on the screen.
> >>>
> >>> With the introduction of MT, the touch data are most likely translated
> >>> into something other than pointer events. So, reporting both pen and
> >>> touch data makes sense now. However, I want to assure a smooth
> >>> tansition from single touch to MT for end users so they still get the
> >>> single touch behavior as they used to be. I gathered the following
> >>> approaches:
> >>>
> >>> 1.     Arbitrate all touch data in the kernel.
> >>>
> >>> This is the simplest solution for device driver developers. But I do
> >>> not feel it is end user and userland client friendly.
> >>
> >> I'm strongly opposed to this. kernel filtering of these devices is hard to
> >> circumvent and there _will_ be use-cases where we need more than one tool
> >> to
> >> work simultaneously. right now we're worrying about pen + touch, but what
> >> stops tablets from becoming large enough to be used by 2+ users with 2+
> >> pens simultaneously?
> >>
> >> from a purely event-stream focused viewpoint: why do we even care whether
> >> something is a pen or a touch? both are just tools and how these should be
> >> used is mostly up to the clients anyway.  IMO, the whole point of
> >> MT_TOOL_TYPE is that we don't have to assume use-cases for the tools but
> >> just forward the information to someone who knows how to deal with this.
> >>
> >>> 2.     Report first finger touch as ABS_X/Y events when pen is not in
> >>> prox.  Arbitrating single touch data when pen is in prox. Pen data is
> >>> reported as ABS_X/Y events. Both ABS_X/Y for pen or the first finger
> >>> and ABS_MT_* for MT data are reported.
> >>>
> >>> This approach reduces the overhead in dealing with two cursors in
> >>> userland.
> >>>
> >>> 3.    Report first finger touch as ABS_X/Y events when pen is not in
> >>> prox;
> >>>        Report pen data as ABS_X/Y events when there is no finger touch;
> >>>        Report touch data as MT_TOOL_TOUCH and pen data as MT_TOOL_PEN
> >>> events when both pen and touch data are received. No ABS_X/Y are
> >>> reported when pen and tocuh or multi-touch data are received.
> >>>
> >>> I feel this one makes sense to userland since pen can be considered as
> >>> another touch.
> >>>
> >>> 4.    Report first finger touch as ABS_X/Y events when pen is not in
> >>> prox;
> >>>        Report pen data as ABS_X/Y events when there is no finger touch;
> >>>        Report touch data as MT_TOOL_TOUCH and pen data as MT_TOOL_PEN
> >>> events when both pen and touch data are received. ABS_X/Y are also
> >>> reported for pen when both pen and tocuh data are received.
> >>
> >> I'd vote for this one. It provides all the data necessary for MT clients
> >> (and all the data the device can support) but has a reasonable
> >> single-touch
> >> strategy. Given that wacom tablets are still primarily pen-centric
> >> tablets,
> >> the emphasis on pen overriding touch makes sense to me.
> >
> > Hi,
> >
> > I'd also vote for this.
> >
> > I don't think that the kernel should make any assumption on the final
> > application. The data are available, so we have to pass them.
> >
> > 1. I read that people worry about sending "false" events (touch) while using
> > the pen. But in my mind, this is a _design_ problem of the final
> > application. I think the final application will have to filter these events:
> > for instance, what happens if the user is too lazy to remove his pen (or
> > just want to keep the hover on the application) out of the proximity range
> > and want to move its digital sheet of paper in his (her) design application?
> > The final application will have to choose whether using or not the touch
> > features (depending on the pressure for instance...).
> >
> > The solution 4. (*technical solution*) addresses the problem of the "false"
> > events for the applications (*design problem*) that are not designed to used
> > multitouch. They will just ignore the touch data.
> > So I think, it's a good start
> >
> >
> > 2. I would also add that multitouch is not only available for trackpads:
> > there are also direct devices in absolute coordinate mode. With those
> > device, the touch data can be directed to an other Xclient that is used by
> > an other user if the surface is large enough. Currently we only see
> > relatively small surfaces (bamboo, ntrig devices), but in the future, we can
> > easily imagine a whole table with both pen and touch.
> >
> > And this solve Michal's problem as he will be able to use buttons in the
> > application with the finger.
> >
> > Cheers,
> > Benjamin
> >
> >>
> >>> This one makes sense to userland too. It eases the backward
> >>> compatibility support for those clients that don't support MT at all.
> >>>
> >>> Which approach do you like? Or do you have other suggestions share?
> >>
> 
> I think we may be mixing some topics and so I'd like to try to
> re-frame the discussion.
> 
> There are two different cases and they may have different answers
> because of it.
> 
> Case 1) 1 input device can support multiple tools that are in
> proximity at same time.
> 
> I believe this is currently a theoretical example (no driver exists like this).

if you consider touch to be just another tool, we already have devices that
support proximity of multiple tools. This isn't theoretical anymore.

> In RFC example, this input devices has a pen and 2 finger touches.
> They all share ABS_X/Y/PRESSURE values.  The single touch (ST) input
> filtering breaks being able to support this case and what multitouch
> events (MT) were added for.
> 
> To date, when converting drivers over to MT events the guideline is
> *always* send MT events (because what app wants to randomly switch
> between MT event processing and ST event processing for same
> X/Y/PRESSURE?) and send something sane for ST events to be backwards
> compatible with older apps.
> 
> I think everyone is happy in this thread to always send pen+touch MT
> events and let X drivers or similar filter/arbitrate out unwanted
> touch events as needed.
> 
> The ideal "sane" behavior for touch ST events has been leaning towards
> tracking 1st touch and continue sending 1st touch during multi-touch
> but there is some debate because tracking can be expensive in kernel.
> In case of pen+touch, the sane may change to prefer pen over touch and
> prefer first touch when 2 touches exist.
> 
> Or "sane" can mean let the ST values go crazy during multi-touch and
> hope user can use GUI enough after new kernel install to get a
> MT-aware X driver.
> 
> Its easy to implement preferring pen then preferring 1st touch so I
> suggest doing that.  This is for backwards compatibility only
> (un-modified xf86-input-wacom/synaptics/evdev/etc).  The future is MT
> events, in which case the ST events are meaningless and we are hiding
> nothing to applications that look at MT events.
> 
> Case 2) 2 input devices can support multiple tools in proximity at same time.
> 
> I believe it was Rafi that brought up point that dual pen+touch
> interfaces will have different properties.  Touch will be lower
> resolution then Pen and maybe different fuzz factor.  Also, on tablets
> I would think pretty easy to have different dimensions (one tool works
> over larger area of tablet).  This is easy to expose to user when 2
> input devices.

Yes and no. We're talking about kernel level here and I don't think this
should be done at this level. The current behaviour of the X driver is to
split multiple tools up into multiple X devices, so the points above can
easily be achieved in userspace.


> Combining into single input to user would be nice but at least when
> dimensions are different, we probably do not want to remove that
> visibility to user and so must keep 2 input devices.
 
If we run into issues with different axis ranges/resolutions for multiple
MT_SLOT devices, this should be addressed in the kernel as well.
I feel uncomfortable about splitting up a physical device into multiple
devices, it takes information away that cannot easily be re-created in the
userspace. Even with a method of associating multiple event devices to the
same physical device, the parsing of simultaneous events is harder because
you're essentially deserialising event streams. In userspace, you have to
re-serialize based on parallel inputs.

That said, it also goes counter the whole multi-touch approach - allowing
more than one device on a single physical device.

Cheers,
  Peter

> In this case, the RFC example becomes 2 touches on 1 input device and
> 1 pen on another input device.
> 
> So using same MT guidelines, the touch input device would always send
> MT events and always send ST events tracking the first touch.
> 
> For pen input, ST-only events are OK because its not competing with
> anything being in proximity at same time.  But we may wish to also
> send MT events for this 1 tool/slot as a hint to X drivers that
> somehow this input corresponds with another input device and so it
> needs to do filtering/arbitration.  We also need to somehow give info
> to applications so they can bind these 2 inputs.
> 
> Also, non-MT ware drivers are also same apps that will not know how to
> bind 2 input devices and so can't filter/arbitrate the unwanted
> touches.  So problem, we do want to filter ST events on touch input
> when pen is in proximity.
> 
> There are lots of things needing to be addressed for this 2nd case so
> I'll not really give a personal opinion.  My opinion is likely to
> change as we make individual decisions.
>