XInput 2: the big picture, relationship to toolkits

[Owen Taylor]

XInput 2 has a lot of big changes in it, but it's not clear to me that
it's really going to fix the problems with XInput that have caused problems
for applications, toolkits, and users over the last decade or more.
In some cases, the MPX changes in XInput 2 makes things worse. This mail is an 
attempt to write up some of those issues, look at how the current XInput 2
affects them, and propose some solutions.

It's basically from the perspective of graphics tablets used for drawing
programs; I don't have much experience with other devices or use cases.

I also haven't looked here at device keys ... the past experience was that
it was almost impossible to meaningfully use the Wacom soft keys in apps,
but I forget the details of why that was case.

Device Configuration
====================

I think the desired user picture of how devices should behave is pretty
obvious:

 - You plug in a device, it works immediately
 - No editing of xorg.conf is necessary
 - There is a system-wide "control panel" that allows you to see all
   attached devices and configure individual devices.
 - Configuration for a device would include how it's mapped to the
   screen (preserve aspect ratio, map to one monitor, etc.) and might
   also have device specific options like pressure curves.
 - Configuration is stored persistently

In general, most of this isn't in scope for the XInput extension to 
solve... configuration dialogs are in the realm of the desktop; some
support for hotplug was already added in 1.4. There probably needs
to be some set of input device properties documented and handled in
the core for common configuration.

Application View
================

Configuration of devices should not be in the scope of an application.
Once a device is configured by the user globally, it should just work
in any application. Though in some cases, an application may want to 
allow it's users to configure how device functionality maps to application
functionality, see below.

Moreover, in the case where a device is controlling the core pointer
the application should not have to choose between "normal events" and 
"input extension events". If the user moved the pointer somewhere, it's
a motion event. Then, if the application cares, it should be able to
look and see "what device does this event come from?" "does this event 
have pressure information? what was the pressure?"

So, the question for the XInput extension is: are the APIs correct so that
a toolkit can seamlessly provide this functionality?

One thing that occurs to me: with the master pointer support in XInput 2, 
you no longer have to select for input on all devices, you can simply 
select for the master device ... a definite improvement. However, if there 
are multiple master pointers, you have to select for events for every 
pointer, and track the set of master devices, and if that changes, redo 
the selection.

I don't have a great suggestion for fixing this problem. What I'd really
want at the toolkit level is to say "for this client extension send me 
all extended input events selected by my core event masks along with
the core events". But that's pretty radically different than how the XInput 
extension normally works.(Note that this doesn't make sense for floating slave
devices ... there what you really want to is to select for events
*without respect to the window hierarchy*). 

Monitor to screen relationship
==============================

One of the big problems with the original XInput was that you had
XInput devices and you had the core pointer, and they were separate.
This worked for dial boxes, not so much for how people want to use
graphics tablets. XInput 2 is largely about allowing flexible 
relationships between input devices and X pointers. But what isn't
currently flexible is the way that device coordinates map to 
screen coordinates.

Currently:

* The entire space of the device is mapped to the entire space
  of the screen.

But:

* You might want to attach a tablet to a particular monitor. One case
  of this is a device like the Wacom Cintiq where the tablet *is*
  the monitor. (This is also the case for many mobile devices, but
  those are seldom used with multiple monitors so the screen/monitor
  distinction doesn't matter.)

  But I could also see it being useful for a graphics
  artist with multiple monitors ... the total monitor space
  is shaped very different from the tablet, and any window you
  might want to draw in would occupy a small fraction of the tablet.

* You might possibly want to attach the tablet to a window. My 
  experience years ago was that this wasn't actually that useful;
  it forced you to switch between mouse and tablet a lot, and 
  tablets are high resolution devices available at decent sizes
  for reasonable prices ... it's OK if you aren't drawing with the
  *entire* tablet.

I think there should be an API in XInput for attaching a device
to a particular screen coordinate range, and maybe an alternate
to allow attaching it to a window.

Another thing to consider is aspect ratio ... I think right now
if you have a 4x3 tablet and a 3x2 screen a circle you draw or
trace on your tablet will come out squished. Probably the
attachment API needs to have a flag as to whether to preserve
aspect ratio or not.

Subpixel positioning
====================

An important attribute of a graphics tablet is subpixel positioning; if
you draw a line at a small angle, you want that line to come out
smoothly antialiased, not with a step in it as the pen crosses pixel
boundaries.

The current code reports:

 rootX,rootY: integer screen pixel coordinates
 eventX, eventY: integer window-relative coordinates
 valuator[0], valuator[1]: device coordinates *quantized
   to screen pixel coordinates*.

While what you actually want to get is window-relative coordinates
with subpixel resolution, something you can't get.

What I would suggest:

 rootX,rootY: integer screen pixel coordinates
 eventX, eventY: integer window-relative coordinates
 valuator[0],valuator[1]: Screen-pixel coordinates in 24:8 
   fixed point. (24:8 vs. 16:16 is pretty arbitrary here since
   we only have 16-bit range for X screen range but since it's
   arbitrary, why not pick the sensible one)

Then toolkits can compute what they really want as:

 x = valuator[0] / 256. + (eventX - rootX);
 x = valuator[1] / 256. + (eventY - rootY);

Identifying Axes
================

XListInputDevices() tells you, for each device axis:
 
 resolution (counts/meter)
 min-value
 max-value

But what you don't find out is what the axis actually is ...
is it X/Y position? Pressure? Tilt? Something else?
This is information that the driver has. Then it's thrown
away before sending it to X.

The master/slave setup makes this, if anything, worse, since
it continually mutates the master device to look like one or
the other slave device, but there is no actual indication of
what slave event one of these master events come from that
I'm aware of.

One possible fix here would be to add another type of 
device information class .. ExtendedValuatorInfo, say that
contains extra information per axis.

Device Identity
===============

While position is position, pressure is pressure and applications  
shouldn't have to worry about device identity when reading axis values, 
applications do need the ability to determine what device is sending 
an event. A drawing program might want to make tool selection per
device, for example.

This comes up with Wacom tablets, because the Wacom drivers create
a device for each "tool" you can use on the tablet (different pens, 
airbrush, mouse, etc.) That is arguably an abuse of the XInput
system, but without extending XInput in this area, it's the only
feasible way to do it. And it conceivably useful in the case where
there are multiple actual devices sending events (mouse vs. tablet,
say.)

So, there needs some way of telling what slave device sent an event
delivered from the master device. I don't have an easy solution for
this. One thing to consider is getting rid of the wholesale mutation
of the master device (a lot of messy code in the server), and just
send a "CurrentSlaveDeviceChanged" type event. Since master devices
are only revealed to XInput 2 clients, there is some degree of 
compatibility for this.