How to help on multitouch?

[Peter Hutterer]

On Sun, Jun 13, 2010 at 10:28:04PM -0400, Chase Douglas wrote:
> Hi all,
> 
> I've recently become interested in multitouch on linux, and I'd like to
> get involved. I've read some of the back threads, and I agree with
> Peter's Direct Input Devices approach. I came to a nearly identical
> concept when I was thinking about how the iPhone handled multitouch
> (I've done some iPhone development in the past).
> 
> So first, where does the DIDs work stand? 

A few scribbles in my notebook and likely others, some mails on this list
and Carlos/Benjamin's evdev patches are pretty much it, AFAIK.

> I would be happy to lend a
> hand with any coding. I've not done any X work before, the kernel has been
> my mainstay, but I can get my elbows dirty :).
> 
> Second, I was thinking about the DIDs approach, but I have some concerns
> about situations where legacy apps don't listen for them. My thought is
> that if a legacy app has not registered to listen for DIDs, the X server
> should do some work on its behalf. The server could create one master
> pointer for each legacy app and warp it to the location of the first
> touch in the app. 

The core protocol (i.e. legacy apps) does not permit devices to disappear.
This isn't much of an issue with the current devices but touchpoints appear
and disappear all the time. So for each touchpoint that you want to expose
as a core device, you'd have to keep the device around in case the client
requests a grab, etc. This is an issue in the MPX model already but much
less of a problem because one doesn't unplug mice and keyboards all the time.

Example case: touch comes in, sends button press), touch disappears, client
requests grab on this device. Now you can grab any other device to be
protocol-correct but that's not what the user will expect.

> Multiple touches beyond the first inside such an app
> do nothing. This would allow multitouch DIDs to control legacy apps
> while avoiding some of the master pointer dynamic allocation issues that
> required multitouchd to have a pool of pre-allocated pointers for.

How do you define "inside" an application? Clients can register for events
on windows that they do not own - does this count as "inside"?
Note that this is already solved with the Grab Ownership approach in MPX,
but again, touch is trickier than mice/keyboards and grab ownership doesn't
quite apply here unless you somehow tag new touchpoints with the touch group
they belong to.

> Lastly, I have concerns about keyboard focus with multitouch. I can't
> think of a way to have the server manage focus with the help of DIDs due
> to the fact that multiple apps may be used at the same time. However,
> even when an app is used it may not need to become focused. For example,
> if you are just making some motion to scroll a window that is partially
> behind the focused window, you might not want to focus it. So, I think
> the best way to handle focus would be to have the application receiving
> DID events ask for focus if it feels it should gain it.

Keyboard is not handled by the X server, it's handled by the client. So
you're fighting against the legacy clients here. If the WM issues a
SetInputFocus request you need to know which keyboard to focus into the
particular window. The ClientPointer principle tells you that with
reasonable predictabilty but again, touch points are shortlived and if you
pool them together with multiple keyboards that becomes a bit trickier.
I don't have a good solution to this yet other than "fix the WM".

You can't easily deny a SetInputFocus request btw. Though the protocol
allows for some errors (BadMatch, BadValue and BadWindow), sending them may
cause the applications to behave weirdly. Because BadMatch is the only one
you can really pick here anyway but if the application also tracks windows
it's likely to become confused.

> I'd love to hear others' thoughts on these ideas. I'm really new to X
> development, so I may have some concepts wrong in my head, so please let
> me know if I've got anything wrong :).

I really think the only way out of this mess is pure DIDs, which allow us to
ignore most or all of the core protocol range. Of course, that gets
interesting again with XTest, RECORD and other extensions but ... well,
urgh.

Cheers,
  Peter