[RFC] Multitouch support, step one

[Henrik Rydberg]

Peter Hutterer wrote:
> On Tue, Mar 16, 2010 at 02:42:15PM +0100, Henrik Rydberg wrote:
[...]
>> 1. Consistent behavior for all devices
>>
>> The hardware stack supporting multitouch is diverse, and several different
>> mechanisms and abstraction levels exists. The tracking ID is a good example. It
>> may or may not be present in the driver output, and it may work poorly even if
>> it exists. Thus, in order to support hardware consistently, there must be a
>> middle layer outside of the kernel, parsing the driver data and patching it up
>> to produce the same level of detail for all devices. This task can be quite
>> complicated and uses some cpu, so having it in one place is imperative. Luckily,
>> there exists such a solution in the multitouch X driver (see link above). This
>> code can either be broken out as a standalone module or be placed in the X core.
>> If there is a license issue, it can be resolved for the benefit of the X
>> community. In the text below, this piece of code will be referred to as the
>> contact driver.
> 
> I think this is where we mostly agree, maybe not it wording but in spirit.
> I want the data to come out of the protocol in a generic fashion, or at
> least generalised up to a point that clients can work with it. 
> IMO dealing with touchpoint (or "contact") width and height can be left to
> the clients. what the driver should however do is assign tracking IDs. I
> don't think this approach would be useful without being able to track
> touchpoints.

Yes. All that is needed is to put the contact-driver part of the current
multitouch X driver into evdev, and this issue will be resolved.


>> 2. Bandwidth reduction should be made as early as possible
>>
>> The MT events from the kernel are non-filtered, bypassing the normal input
>> filtering by necessity. Duplicating this behavior further into the food chain
>> would be a mistake. After parsing the MT stream in the contact driver, the event
>> stream can be filtered substantially, thereby restoring bandwidth usage to
>> something more similar to non-mt devices.
> 
> I don't understand what you mean by filtering here. Can you detail this?

The kernel input layer has a filtering mechanism that removes an arbitrary large
portion of the driver event stream. Each ABS event is registered together with
an estimate of the signal-to-noise ratio of that particular event. The driver
can then send data as often as it likes to the input core, events will only be
emitted when the change is "big enough". As an example, sitting at a computer,
resting a finger at the trackpad. You consider your hand to be resting, but in
fact the detected finger position changes slightly all the time, in effect
creating a continous stream of events from the driver.

The ABS_MT events, on the other hand, bypass the filtering mechanism, for two
reasons. Firstly, the input filtering requires absolute identity of the ABS
event, which is not the case in the sequential MT stream. Secondly, the
signal-to-noise ratio of a combined finger movement is different from each
finger separately. It is first when considered as a whole a reasonable filtering
can be performed.

And we *do* want filtering. The difference in number of events could easily be
an order of magnitude.


>> 3. The contact driver produces the more digested contact events
>>
>> The contact driver takes the flora of driver MT events and produces a consistent
>> stream of contact events. The contact event stream is less bandwidth-consuming,
>> and follows the init-move-destroy concept we discussed last summer, if you
>> recall. We are still talking about a low-level stream, there are no gesture or
>> other high-level derivatives. Just a consistent stream of data.
> 
> Same as above. You've worked more with the kernel's multitouch interface
> than I did. can you give us an example of what the data from the kernel
> would look like and how it would be "digested"?

Pick a random kernel driver which supports some finger touches, but not yet MT
events. Now we want to make this driver work with multitouch gestures. We have
available in the driver some finger positions not yet reported as events, but
nothing more. The MT protocol allows us to go ahead and simply add
ABS_MT_POSITION_X and ABS_MT_POSITION_Y for the individual fingers, insert a few
 input_mt_sync()'s, and we are done with the kernel changes.

Enters the Contact Driver. It will now see ABS_MT_POSITION events appearing in
the stream, so it knows the driver is MT-enabled. However, there is no tracking
id, so the driver computes how to distribute the changes onto individual,
identifiable fingers. Perhaps the resulting motion is below some signal-to-noise
threshold, so it holds on to the change a while longer. Some time later an
additional change makes it worthwhile to emit an event, so it pushes the changes
onto the appropriate valuators.


[...]
>> The Contact Driver
>> ------------------
>>
>> The general structure of the MT events is that of contacts appearing, changing
>> and disappearing. Because of the diversity of capabilities of the drivers, this
>> structure is quite relaxed in the kernel stream, to the point that it requires
>> work to fully impose this structure further down the stream. That is the job of
>> the Contact Driver. It translates the relaxed kernel MT events into a steady
>> stream of contact events, containing the same level of information for all
>> drivers. The contact events follow the same logic as the MT events, but because
>> all data is present, the init-move-destroy mechanism can be employed fully. Here
>> is an example of what a two-finger scroll would look like:
>>
>> init id = 588, x = -234, y = 42
>> init id = 123, x = 933,	y = 3
>> sync
>> move id = 588, x = -211, y = 529
>> move id = 123, x = 863,	y = 732
>> sync
>> destroy	id = 588
>> destroy	id = 123
>> sync
>  
> This is exactly what I had in mind to handle MT in the evdev driver. The
> driver handles this, then forwards it on to the client. The only difference
> I see so far is that the init/destroy is implicit by the presence of the
> valuators. Is this different to what you are proposing? If so, can you
> provide more detail?

No difference. I believe we have converged.

Cheers,
Henrik