xf86EnableIO & ExtendedEnabled vs. the input thread

[Alan Coopersmith]

Our friends at VMWare have recently tracked down issues on Solaris using
the xf86-input-vmmouse driver to a subtle difference in how IOPL is set
between OS'es.

They found:

    It's indeed outl() that segfaults when trying to program the hardware cursor.

    Now prepending vmwareWriteReg with xf86EnableIO() doesn't help, so the
    X server thinks IO is enabled.

    But it seems like prepending vmwareWriteReg with
    xf86DisableIO()
    xf86EnableIO()
    makes things work again

    So a pretty qualified guess at what's happening here is that when the
    Xserver became multithreaded not too long ago, with a separate thread
    for mouse movement, setting xf86EnableIO() on Solaris doesn't apply to
    all threads, or if the threads are created *after* setting xf86EnableIO,
    the IOPL status doesn't propagate to the new threads.

    Note that Solaris xf86EnableIO() keeps a static variable to determine
    whether IO support was already enabled, so calling it from multiple
    threads only enables IO for the first thread.

I've confirmed with our kernel folks that the syscall we call from
xf86EnableIO() on Solaris has always only set the IOPL for the calling
thread and not other threads.  They believe the primary difference between
Linux & Solaris is that when a new thread is spawned, it inherits the IOPL
on Linux, but on Solaris we reset the IOPL for the new thread instead of
inheriting it.

While we could update this in the kernel, that won't solve the problem for
people on older kernels, or one of the OpenSolaris off-shoots (and wearing
my security hat, I actually prefer the least-privilege method of not having
all new threads inherit the iopl from the main thread).

The one-line solution in the Xorg sources would be to make the
ExtendedEnabled boolean that tracks if the IOPL is set be a
thread-specific variable.

But I have to ask if anyone remembers why we keep track of this flag in
userspace at all - it's not just Solaris, but it seems to have been
cargo-culted to most platforms:

hw/xfree86/os-support/bsd/arm_video.c
hw/xfree86/os-support/bsd/i386_video.c
hw/xfree86/os-support/linux/lnx_video.c
hw/xfree86/os-support/solaris/sun_vid.c

Is there any reason we just don't drop ExtendedEnabled altogether and
just always pass the calls through to the kernel?   That'd leave us
still needing to call xf86EnableIO() in the input thread on Solaris,
unlike Linux, but that's more sensible than forcing a DisableIO()
first just to reset the ExtendedEnabled state.

-- 
	-Alan Coopersmith-               alan.coopersmith at oracle.com
	 Oracle Solaris Engineering - https://blogs.oracle.com/alanc