[PATCH v3 1/3] Add xorg-gtest integration test framework

[Peter Hutterer]

On Thu, May 24, 2012 at 10:25:00AM -0700, Chase Douglas wrote:
> On 05/23/2012 08:32 PM, Peter Hutterer wrote:
> > pulled the branch you sent me that contained (amongst others) this patchset.
> > 
> > On Fri, May 18, 2012 at 02:02:04PM -0700, Chase Douglas wrote:
> >> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
> >> ---
> >> Changes since v2:
> >> * Refreshed xorg-gtest.m4 and Makefile-xorg-gtest.am for xorg-gtest changes
> >>
> >>  configure.ac                            |   21 +++++-
> >>  m4/xorg-gtest.m4                        |  110 +++++++++++++++++++++++++++++++
> >>  test/integration/Makefile-xorg-gtest.am |   64 ++++++++++++++++++
> >>  test/integration/Makefile.am            |    4 ++
> > 
> > What's missing here is that test/Makefile.am needs integration in the
> > SUBDIRS, otherwise the tests won't be built.
> 
> I could have sworn I did this, but I guess not. I've added it in.
> 
> > and when building them, I get 
> > xi2.cpp: In member function ‘virtual void
> > XInput2Test_DisableDeviceEndTouches_Test::TestBody()’:
> > xi2.cpp:308:177: error: ‘wait_for_device’ was not declared in this scope
> > xi2.cpp:313:162: error: ‘wait_for_event_of_type’ was not declared in this
> > scope
> > xi2.cpp:337:162: error: ‘wait_for_event_of_type’ was not declared in this
> > scope
> > 
> > I suspect they are supposed to be in there somewhere, but neither xorg-gtest
> > nor this tree had it.
> 
> I fixed up the XIQueryPointer test, but then when I rebased the "end
> touches on device disable" branch on top I forgot to fix up its test
> too. I guess I then didn't bother to recompile and test :(. My fault.
> 
> I'll resend all five patches together so you can review the changes. The
> branch has been updated if you want to pull:
> 
> git://people.freedesktop.org/~cndougla/xserver integration-tests

this took me a _long_ time to get running and it still fails. A few of the
issues I encountered:

- For some reason the default :133 display doesn't work here. Could be some
  stale socket interfering, i'm not sure at this point.
  If the server fails to start up this way, there is no log file despite the
  message suggesting to look at it. I spent quite a while recompiling trying
  to figure out why the input ABIs of the drivers weren't working with the
  test but worked in my normal server - turns out the log file was leftover
  and didn't get updated at all here.
  Suggestion: move the log file before trying to start the server.

- We require root for evemu, the current error message is confusing there.
  Suggestion: skip the test with a warning if evemu is required but the test
  is not run as root.

- I got segfaults when running the tests (approx every second run), see the
  patches for xorg-gtest to send out but even after those I still get the
  sporadic:
unknown file: Failure
C++ exception with description "Failed to open connection to display" thrown in SetUp().
[  FAILED  ] XInput2Test.DisableDeviceEndTouches/1, where GetParam() = 1 (51 ms)

Now I'm down to 3 errors that suggest either missing patches in the server
or broken tests:

make[1]: Entering directory `/home/whot/xorg/xserver/test/integration'
[==========] Running 6 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 6 tests from XInput2Test
[ RUN      ] XInput2Test.XIQueryPointerTouchscreen/0
xi2.cpp:264: Failure
Value of: XIMaskIsSet(buttons.mask, 0)
  Actual: false
Expected: true
[  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/0, where GetParam() = 0 (417 ms)
[ RUN      ] XInput2Test.XIQueryPointerTouchscreen/1
xi2.cpp:264: Failure
Value of: XIMaskIsSet(buttons.mask, 0)
  Actual: false
Expected: true
[  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/1, where GetParam() = 1 (366 ms)
[ RUN      ] XInput2Test.XIQueryPointerTouchscreen/2
[       OK ] XInput2Test.XIQueryPointerTouchscreen/2 (385 ms)
[ RUN      ] XInput2Test.DisableDeviceEndTouches/0
[       OK ] XInput2Test.DisableDeviceEndTouches/0 (345 ms)
[ RUN      ] XInput2Test.DisableDeviceEndTouches/1
[       OK ] XInput2Test.DisableDeviceEndTouches/1 (341 ms)
[ RUN      ] XInput2Test.DisableDeviceEndTouches/2
xi2.cpp:338: Failure
Value of: WaitForEventOfType(Display(), GenericEvent, xi2_opcode_, XI_TouchEnd)
  Actual: false
Expected: true
[  FAILED  ] XInput2Test.DisableDeviceEndTouches/2, where GetParam() = 2 (1396 ms)
[----------] 6 tests from XInput2Test (3251 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 1 test case ran. (6255 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/0, where GetParam() = 0
[  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/1, where GetParam() = 1
[  FAILED  ] XInput2Test.DisableDeviceEndTouches/2, where GetParam() = 2

Cheers,
  Peter