[RFC] Multitouch support, step one

[Peter Hutterer]

On Mon, Mar 15, 2010 at 12:55:11PM +0100, Benjamin Tissoires wrote:
> 
> >>>
> >>>Disclaimer:
> >>>I believe this is pretty much what Win 7 or OS X do so I won't bother
> >>>claiming this being innovative. This isn't exactly my idea either, I'm just
> >>>writing up what I got from talking to Benjamin, Bradley, Henrik, Stephane,
> >>>and many more.
> >>>
> >>>Details:
> >>>The data we get from the (Linux) kernel includes essentially all the ABS_MT
> >>>events, x, y, w, h, etc. We can pack this data into valuators on the device.
> >>>In the simplest case, a device with two touchpoints would thus send 4
> >>>valuators - the first two being the coordinate pair for the first touch
> >>>point, the latter two the coordinates for the second touch point.
> 
> As Carsten said, we could send all the MT-valuators for each touch.
> Thus, having the description of valuators as:
> Val0_not_mt, Val1_not_mt, ... Val_i_not_mt, TrackingID0, ABS_MT_X0,
> ABS_MT_Y0, other_mt_0, TrackingID1, ABS_MT_X1, .... etc
 
yes, this was the plan. sorry if that wasn't specified as such but the evdev
changes required would essentially tack on the MT axes to the already
existing ones resulting in an axis setup exactly as you described.

> >>>
> >>>XI2 provides us with axis labels, so we can label the axes accordingly.
> >>>Clients that don't read axis labels are left guessing what the fancy values
> >>>mean, which is exactly what they're doing already anyway.
> >>
> >>ok. here is where i ask.. what are these labels to be?pointless knowing there
> >>are labels - unless we know what they should be to indicate what is what. it's
> >>still guessing if we don't know what they should be :)
> >
> >See Benjamin's commit a34812b09000db2ff2a1dc6182602839123edd4e on master.
> >The idea is that your 2-touchpoint device provides the following labels,
> >"Abs X", "Abs Y", "Abs MT Position X", "Abs MT Position Y", "Abs MT Position
> >X", "Abs MT Potision Y".
> 
> agree, but remember to send the other axes.
> 
> >
> >Having said that, I now realise that it's hard to tell them apart this way,
> >you can only look for repetition in axes. This is a fair assumption I guess,
> >I'm not sure if any devices have different capabilities on different
> >touchpoints.
> >
> 
> I never seen such a device, and I don't understand how could it be.
> The most strange device I get is the Apple Magic Mouse, which is
> both relative and absolute, but all the touch points are the same.
> 
> By the way I don't think that the kernel could handle such a device
> as it serializes the valuators, and if a touch does not report an
> axe, it will have its value at 0.
> 
> >>>XI2 DeviceEvents provide a bitmask for the valuators present in a device.
> >>>Hence, a driver can dynamically add and remove valuators from events, thus
> >>>providing information about the presence of these valuators.
> >>>e.g. DeviceEvent with valuators [1-4] means two touchpoints down, if the
> >>>next event only includes valuators [3-4], the first touchpoint has
> >>>disappeared.
> 
> I made a few test yesterday (involuntary though) with the magic
> mouse. As we can send only part of the valuators, why not packing
> the different available touches at the beginning. It will require to
> move some valuators, but if we keep sending the trackingID, it won't
> hurt the client.

Packing the MT valuators towards the end has the advantage of being able to
use the repetition algorithm I mentioned elsewhere. I.e. any axes from now
on is a multitouch axes and repeating identifiers specify new touchpoints.
This is less obvious when the MT axes are towards the front of the set.
 
> An other point for keeping the valuator trackingID. Some device
> (stantum and magicmouse) send a trackingID different than touch
> point. i.e. the trackingID is between 1 and 255 on the stantum, and
> between 1 and 16 on the magic mouse. I don't know if the user app
> should use this, or a higher level id could do the job.

the tracking ID can be abstracted away in the driver. In fact, if we chose
the valuators to be representative the tracking ID can fall away
immediately. i.e. valuator n is _always_ touchpoint m.

Of course, if we instead choose to have touchpoints dynamically assigned,
then we need to send the tracking ID down the wire as well. I'm not a big
fan of this at this point but I can be convinced otherwise.

> >>>XI2 allows devices to change at runtime. Hence a device may add or remove
> >>>valuators on-the-fly as touchpoints appear and disappear. There is a chance
> >>>of a race condition here. If a driver decides to add/remove valuators
> >>>together with the touchpoints, a client that skips events may miss out.
> >>>e.g. if a DeviceChanged event that removes an axis is followed by one that
> >>>adds an axis, a client may only take the second one as current, thus
> >>>thinking the axis was never removed. There is nothing in the XI2 specs that
> >>>prohibits this. Anyways, adding removing axes together with touchpoints
> >>>seems superfluous if we use the presence of an axis as indicator for touch.
> >>>Rather, I think a device should be set up with a fixed number of valuators
> >>>describing the default maximum number of touchpoints. Additional ones can be
> >>>added at runtime if necessary.
> >>
> >>agreed. i really see this having a fixed # of touch points - and not changing -
> >>unless you literally unplug/plug in new hardware that has different features
> >>(has more or less in the way of touch point support).
> 
> We can have a fixed number of touch point but send only the required
> ones. So agreed too. The point is: how many touch point do we have.
> The kernel knows how many touches a device can send as the data are
> not serialized. But after that, we have no idea of how many touches
> the device support.
> 
> With the mask system (or the packing of the touches at the
> beginning), we will send only the right number of touches, but the
> description will be very heavy. If each point has 5 axes
> (trackingID, x, y, width, height for instance) we will have 50
> valuators if we support 10 touches ;-) By the way, it's not the
> point here.

Unless you modify the number of valuators, that description needs to only be
sent once to a client. Only when you exceed the number of currently assigned
touchpoints do you need to add new valuators, resulting in the description
being sent to the clients again.

> >>>Work needed:
> >>>- drivers: updated to parse ABS_MT_FOO and forward it on.
> >>>- X server: the input API still uses the principle of first + num_valuators
> >>>   instead of the bitmask that the XI2 protocol uses. These calls need to be
> >>>   added and then used by the drivers.
> >>>- Protocol: no protocol changes are necessary, though care must be taken in
> >>>   regards to XI1 clients.
> >>>   Although the XI2 protocol does allow device changes, this is not specified
> >>>   in the XI1 protocol, suggesting that once a device changes, potential XI1
> >>>   clients should be either ignored or limited to the set of axes present
> >>>   when they issued the ListInputDevices request. Alternatively, the option
> >>>   is to just encourage XI1 clients to go the way of the dodo.
> >>>
> >>>Corner cases:
> >>>We currently have a MAX_VALUATORS define of 32. This may or may not be
> >>>arbitrary and interesting things may or may not happen if we increase that.
> >>
> >>another problem - no ability to do "pressure" here. ie have each touch point
> >>have a radius for example (x and y radius) etc. etc. ??? what happened to that?
> >
> >The kernel's MT API caters for width/height, orientation and a few other
> >things (see linux/input.h, essentiall, we're just mirroring here anyway).
> >what it doesn't cater for yet is MT pressure though IIRC I've either seen a
> >patch float past or at least the talk about it. Since we only need to add
> >another label, that's easy enough. But good point, we mustn't forget this.
> 
> Currently, we don't aim at modifying the data the device send. If it
> provides a pressure (starting from kernel 2.6.33 I think), we should
> transfer it to the client. But, I don't think we should create an
> arbitrary value depending on sizeX and sizeY.

oh, I'm not suggesting modifying the data from the kernel. I'm suggesting
to add a label for ABS_MT_PRESSURE in case the kernel will get this in the
next revision and the devices will start sending this information. If a
device doesn't have it, then we'll never see it anywhere anyway.

Cheers,
  Peter