XInput 2: the big picture, relationship to toolkits

[Daniel Stone]

Hi,

On Wed, Aug 13, 2008 at 05:45:40PM -0400, Owen Taylor wrote:
> I also haven't looked here at device keys ... the past experience was that
> it was almost impossible to meaningfully use the Wacom soft keys in apps,
> but I forget the details of why that was case.

If you find out, let me know, but I'm willing to bet the answer involved
differing keymaps, which is now solved.

> Device Configuration
> ====================
> 
> I think the desired user picture of how devices should behave is pretty
> obvious:
> 
> [...]
> 
> In general, most of this isn't in scope for the XInput extension to 
> solve... configuration dialogs are in the realm of the desktop; some
> support for hotplug was already added in 1.4. There probably needs
> to be some set of input device properties documented and handled in
> the core for common configuration.

Fully agreed.  As long as we provide both a full enumeration of all
devices connected to the system (individually, not aggregated into one
device for all pointers/keyboards), and allow dynamic runtime
configuration, then that's our work done, mostly.

> Application View
> ================
> 
> Configuration of devices should not be in the scope of an application.
> Once a device is configured by the user globally, it should just work
> in any application. Though in some cases, an application may want to 
> allow it's users to configure how device functionality maps to application
> functionality, see below.
> 
> Moreover, in the case where a device is controlling the core pointer
> the application should not have to choose between "normal events" and 
> "input extension events". If the user moved the pointer somewhere, it's
> a motion event. Then, if the application cares, it should be able to
> look and see "what device does this event come from?" "does this event 
> have pressure information? what was the pressure?"

Agreed.  The answer here is, of course, 'if you're on an Xi2-capable
server, forget core events ever existed'.  Of course, you can keep using
core events if you want, and they will continue to work to the full
limitations of the core input protocol.  That is, you won't get the full
range of information from tablets, etc.

> So, the question for the XInput extension is: are the APIs correct so that
> a toolkit can seamlessly provide this functionality?

Hopefully.  Let me know if not.

> One thing that occurs to me: with the master pointer support in XInput 2, 
> you no longer have to select for input on all devices, you can simply 
> select for the master device ... a definite improvement. However, if there 
> are multiple master pointers, you have to select for events for every 
> pointer, and track the set of master devices, and if that changes, redo 
> the selection.
> 
> I don't have a great suggestion for fixing this problem. What I'd really
> want at the toolkit level is to say "for this client extension send me 
> all extended input events selected by my core event masks along with
> the core events". But that's pretty radically different than how the XInput 
> extension normally works.(Note that this doesn't make sense for floating slave
> devices ... there what you really want to is to select for events
> *without respect to the window hierarchy*). 

Yeah, Xi2 wasn't intended to fix the event-selection problem, and thus
doesn't.  AFAICT, the core input actually got this right, so what we
want is pretty much an XSelectInput()-alike
(XSelectExtendedInput(DeviceMotionNotifyMask | DeviceButtonPressMask),
etc), whose semantics are 'send me any events I'd get from core input'.
I assume that'd be fine for you?

> Monitor to screen relationship
> ==============================
> 
> One of the big problems with the original XInput was that you had
> XInput devices and you had the core pointer, and they were separate.
> This worked for dial boxes, not so much for how people want to use
> graphics tablets. XInput 2 is largely about allowing flexible 
> relationships between input devices and X pointers. But what isn't
> currently flexible is the way that device coordinates map to 
> screen coordinates.
> 
> Currently:
> 
> * The entire space of the device is mapped to the entire space
>   of the screen.
> 
> But:
> 
> * You might want to attach a tablet to a particular monitor. One case
>   of this is a device like the Wacom Cintiq where the tablet *is*
>   the monitor. (This is also the case for many mobile devices, but
>   those are seldom used with multiple monitors so the screen/monitor
>   distinction doesn't matter.)
> 
>   But I could also see it being useful for a graphics
>   artist with multiple monitors ... the total monitor space
>   is shaped very different from the tablet, and any window you
>   might want to draw in would occupy a small fraction of the tablet.

Yeah, there's definite call for floating associations.

> * You might possibly want to attach the tablet to a window. My 
>   experience years ago was that this wasn't actually that useful;
>   it forced you to switch between mouse and tablet a lot, and 
>   tablets are high resolution devices available at decent sizes
>   for reasonable prices ... it's OK if you aren't drawing with the
>   *entire* tablet.

Hmm, interesting idea, but I can see that.

> I think there should be an API in XInput for attaching a device
> to a particular screen coordinate range, and maybe an alternate
> to allow attaching it to a window.

Yep.

> Another thing to consider is aspect ratio ... I think right now
> if you have a 4x3 tablet and a 3x2 screen a circle you draw or
> trace on your tablet will come out squished. Probably the
> attachment API needs to have a flag as to whether to preserve
> aspect ratio or not.

Good point.

> Subpixel positioning
> ====================
> 
> [...]
> 
> While what you actually want to get is window-relative coordinates
> with subpixel resolution, something you can't get.
> 
> What I would suggest:
> 
>  rootX,rootY: integer screen pixel coordinates
>  eventX, eventY: integer window-relative coordinates
>  valuator[0],valuator[1]: Screen-pixel coordinates in 24:8 
>    fixed point. (24:8 vs. 16:16 is pretty arbitrary here since
>    we only have 16-bit range for X screen range but since it's
>    arbitrary, why not pick the sensible one)
> 
> Then toolkits can compute what they really want as:
> 
>  x = valuator[0] / 256. + (eventX - rootX);
>  x = valuator[1] / 256. + (eventY - rootY);

Yeah, I think that's pretty much the best plan.

> Identifying Axes
> ================
> 
> XListInputDevices() tells you, for each device axis:
>  
>  resolution (counts/meter)
>  min-value
>  max-value
> 
> But what you don't find out is what the axis actually is ...
> is it X/Y position? Pressure? Tilt? Something else?
> This is information that the driver has. Then it's thrown
> away before sending it to X.
> 
> The master/slave setup makes this, if anything, worse, since
> it continually mutates the master device to look like one or
> the other slave device, but there is no actual indication of
> what slave event one of these master events come from that
> I'm aware of.
> 
> One possible fix here would be to add another type of 
> device information class .. ExtendedValuatorInfo, say that
> contains extra information per axis.

Meh.  I think the best thing to do here is what we've done with buttons:
rather than sending ExtendedButtonInfo, everyone knows what button n
means.  No need to XKBify Xi.

> Device Identity
> ===============
> 
> While position is position, pressure is pressure and applications  
> shouldn't have to worry about device identity when reading axis values, 
> applications do need the ability to determine what device is sending 
> an event. A drawing program might want to make tool selection per
> device, for example.
> 
> This comes up with Wacom tablets, because the Wacom drivers create
> a device for each "tool" you can use on the tablet (different pens, 
> airbrush, mouse, etc.) That is arguably an abuse of the XInput
> system, but without extending XInput in this area, it's the only
> feasible way to do it. And it conceivably useful in the case where
> there are multiple actual devices sending events (mouse vs. tablet,
> say.)
> 
> So, there needs some way of telling what slave device sent an event
> delivered from the master device. I don't have an easy solution for
> this. One thing to consider is getting rid of the wholesale mutation
> of the master device (a lot of messy code in the server), and just
> send a "CurrentSlaveDeviceChanged" type event. Since master devices
> are only revealed to XInput 2 clients, there is some degree of 
> compatibility for this.

My thinking was just to have two device IDs: originating device (the
physical device actually responsible for the event), and delivering
device (i.e. if you select on a master and not a slave, the master).
This, AFAICT, gives the clients all the information they need while
avoiding the inherent raciness and complexity that comes with having to
constantly round-trip to work out how their input event should be
interpreted (and the associated spike in input latency).

Thanks a lot for your feedback: it's very much appreciated.

Cheers,
Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://lists.x.org/archives/xorg/attachments/20080814/133b46c9/attachment.pgp>