[RFC] Multitouch support, step one

[Henrik Rydberg]

Benjamin Tissoires wrote:
> I do not want to break this discussion, but I think that the work you
> mentioned is a little bit aside of the original subject.
> 
> I rapidly had a look at the code and it seems to be a driver for
> trackpads with multitouch enabled. It's really a good job, but I thought
> the original proposal of Peter was to treat 'all' multitouch devices,
> including multitouch-screen.

That is just plain wrong and a red herring. The driver handles all MT event
drivers, there is no difference between touch pads and touch screens in that
respect.

> 
> Personally, I would say that the work you mentioned is of great
> interest, and has to co-exist with the work we are making in this
> thread. But I think it is not enough broad as you treat only a sub-case
> of mt.

Another red herring. The discussion here is about the kernel MT events, and how
to make use of them outside the kernel. Since I wrote the kernel MT protocol, we
can safely assume the scope of the MT events is well represented in the proposed
Contact Driver.

> 
> I will now comment some of your speach...
> 
> Le 16/03/2010 14:42, Henrik Rydberg a écrit :
>> Peter Hutterer wrote:
>>> [snip]
>>
>> 1. Consistent behavior for all devices
>>
>> The hardware stack supporting multitouch is diverse, and several
>> different
>> mechanisms and abstraction levels exists. The tracking ID is a good
>> example. It
>> may or may not be present in the driver output, and it may work poorly
>> even if
>> it exists. Thus, in order to support hardware consistently, there must
>> be a
>> middle layer outside of the kernel, parsing the driver data and
>> patching it up
>> to produce the same level of detail for all devices. This task can be
>> quite
>> complicated and uses some cpu, so having it in one place is
>> imperative. Luckily,
>> there exists such a solution in the multitouch X driver (see link
>> above). This
>> code can either be broken out as a standalone module or be placed in
>> the X core.
>> If there is a license issue, it can be resolved for the benefit of the X
>> community. In the text below, this piece of code will be referred to
>> as the
>> contact driver.
> 
> really like the idea of offering the same interface for all devices. But
> it may just be a piece of code that can be reused by all the different
> drivers (I think it's what you already got, even if I did not found the
> tracking for N-Trig-like devices)

There is no plan to incorporate software finger tracking in the kernel. The
simple solution is to collect the data from the kernel drivers, patch them up to
the same level using the mentioned code (ntrig has a tracking id thus no extra
work for that device), then present it via X to userland. In other words, the
Contact Driver.

> 
>>
>> 2. Bandwidth reduction should be made as early as possible
>>
>> The MT events from the kernel are non-filtered, bypassing the normal
>> input
>> filtering by necessity. Duplicating this behavior further into the
>> food chain
>> would be a mistake. After parsing the MT stream in the contact driver,
>> the event
>> stream can be filtered substantially, thereby restoring bandwidth
>> usage to
>> something more similar to non-mt devices.
>>
> 
> It depends on the different uses. In case of a single user trackpad, ok.
> In case of a large multitouch display, not sure.

This is also wrong, the filtering depends on the signal-to-noise ratio of the
device.

> 
>> 3. The contact driver produces the more digested contact events
>>
>> The contact driver takes the flora of driver MT events and produces a
>> consistent
>> stream of contact events. The contact event stream is less
>> bandwidth-consuming,
>> and follows the init-move-destroy concept we discussed last summer, if
>> you
>> recall. We are still talking about a low-level stream, there are no
>> gesture or
>> other high-level derivatives. Just a consistent stream of data.
>>
> 
> You mean that the client can directly talk to the contact driver?

If the contact driver is an integral part of X, that would more or less be the case.

> 
>> 4. ABI, memory and cpu burden for nothing
>>
>> Although the currently hard limit of valuators most likely can be
>> programmed
>> away, it just feels wrong to burden all other applications with the
>> additional
>> memory and cpu usage implied by raising a comfortable limit to
>> something much
>> higher, only to satisfy the request of a completely different
>> interface, which
>> strains the existing concept to the limit of breakage.
>>
> 
> ~256 valuators should be enough as long as we don't have device that
> reports more than 20 touches. (I think that the current limit of the
> devices is around 10)
> The point is that 32 axes is not enough for multitouch. We want it to be
> raised at 256 (the hard limit if I read correctly), but we don't know
> the side-effects.

Famous last words... 256 is a number close enough to the *first* use case that
it will most certainly bite "us" soon enough.

> 
>> 5. Use appropriate data structures to solve the problem
>>
>> By defining a handful of contact api functions, operating on simple
>> structures,
>> the whole problem of forward compatibility with multi-user multitouch
>> can be
>> solved in one go, without changing a single bit in the existing
>> interfaces. Yes,
>> it means a new interface, but the functionality is new, so this is the
>> way it
>> should be.
>>
>> ---
>>
>> X Multitouch Support Proposal
>> -----------------------------
>>
>> Introduction
>> ------------
>>
>> Back in summer 2009, when this was discussed informally between some
>> in the
>> present party, the general structure that emerged was a split into a
>> low level
>> protocol and a gesture library, here citing two of the formulations:
>>
>>> Henrik Rydberg:
>>> X multipointer via init-move-destroy to X gesture driver
>>> X gesture driver via enhanced X events to X application
>>>
>>> Peter Hutterer:
>>> X server ->  protocol ->  application
>>>                            |->   x gesture library
>>
>> The change I am proposing today simply means inserting the contact driver
>> discussed above before or in conjunction with the X server in Peter's
>> chain:
>>
>> kernel ->  [contact driver | X server] ->  protocol ->  application
>>                                                        |->   x gesture
>> library
>>
>> The details of the protocol in this chain depends on the output of the
>> contact
>> driver, which is only slightly higher level than the kernel events.
>>
>> The Contact Driver
>> ------------------
>>
>> The general structure of the MT events is that of contacts appearing,
>> changing
>> and disappearing. Because of the diversity of capabilities of the
>> drivers, this
>> structure is quite relaxed in the kernel stream, to the point that it
>> requires
>> work to fully impose this structure further down the stream. That is
>> the job of
>> the Contact Driver. It translates the relaxed kernel MT events into a
>> steady
>> stream of contact events, containing the same level of information for
>> all
>> drivers. The contact events follow the same logic as the MT events,
>> but because
>> all data is present, the init-move-destroy mechanism can be employed
>> fully. Here
>> is an example of what a two-finger scroll would look like:
>>
>> init id = 588, x = -234, y = 42
>> init id = 123, x = 933,    y = 3
>> sync
>> move id = 588, x = -211, y = 529
>> move id = 123, x = 863,    y = 732
>> sync
>> destroy    id = 588
>> destroy    id = 123
>> sync
>>
> 
> currently the structure is the following:
> 
> - the device send a consistent packet of N touches.

General wrong, but right for a few devices.

> - the kernel serialized it for some reasons (I did not follow the
> discuss on the kernel list)

Then read it.

> - the X driver has to pack them all in order to being able to track the
> touches in the case the device does not send the trackingID

It does not "pack" the touches, it computes the tracking id when it is not
available from the driver.

> 
> You want then to re-serialize the data? To which end?

Moot question, see above.

Henrik