[PATCH v3 1/3] Add xorg-gtest integration test framework

[Chase Douglas]

On 05/24/2012 06:57 PM, Peter Hutterer wrote:
> On Thu, May 24, 2012 at 10:25:00AM -0700, Chase Douglas wrote:
>> On 05/23/2012 08:32 PM, Peter Hutterer wrote:
>>> pulled the branch you sent me that contained (amongst others) this patchset.
>>>
>>> On Fri, May 18, 2012 at 02:02:04PM -0700, Chase Douglas wrote:
>>>> Signed-off-by: Chase Douglas <chase.douglas at canonical.com>
>>>> ---
>>>> Changes since v2:
>>>> * Refreshed xorg-gtest.m4 and Makefile-xorg-gtest.am for xorg-gtest changes
>>>>
>>>>  configure.ac                            |   21 +++++-
>>>>  m4/xorg-gtest.m4                        |  110 +++++++++++++++++++++++++++++++
>>>>  test/integration/Makefile-xorg-gtest.am |   64 ++++++++++++++++++
>>>>  test/integration/Makefile.am            |    4 ++
>>>
>>> What's missing here is that test/Makefile.am needs integration in the
>>> SUBDIRS, otherwise the tests won't be built.
>>
>> I could have sworn I did this, but I guess not. I've added it in.
>>
>>> and when building them, I get 
>>> xi2.cpp: In member function ‘virtual void
>>> XInput2Test_DisableDeviceEndTouches_Test::TestBody()’:
>>> xi2.cpp:308:177: error: ‘wait_for_device’ was not declared in this scope
>>> xi2.cpp:313:162: error: ‘wait_for_event_of_type’ was not declared in this
>>> scope
>>> xi2.cpp:337:162: error: ‘wait_for_event_of_type’ was not declared in this
>>> scope
>>>
>>> I suspect they are supposed to be in there somewhere, but neither xorg-gtest
>>> nor this tree had it.
>>
>> I fixed up the XIQueryPointer test, but then when I rebased the "end
>> touches on device disable" branch on top I forgot to fix up its test
>> too. I guess I then didn't bother to recompile and test :(. My fault.
>>
>> I'll resend all five patches together so you can review the changes. The
>> branch has been updated if you want to pull:
>>
>> git://people.freedesktop.org/~cndougla/xserver integration-tests
> 
> this took me a _long_ time to get running and it still fails. A few of the
> issues I encountered:
> 
> - For some reason the default :133 display doesn't work here. Could be some
>   stale socket interfering, i'm not sure at this point.
>   If the server fails to start up this way, there is no log file despite the
>   message suggesting to look at it. I spent quite a while recompiling trying
>   to figure out why the input ABIs of the drivers weren't working with the
>   test but worked in my normal server - turns out the log file was leftover
>   and didn't get updated at all here.
>   Suggestion: move the log file before trying to start the server.

If you run a test as root once, then /tmp/Xorg.GTest.log will be owned
by root:root. Then, if you run a test as a normal user the server will
fail to start because it failed to open the log file. We won't be able
to move it out of the way either. I'll add a check to see if we can
write to the file before starting the server.

> - We require root for evemu, the current error message is confusing there.
>   Suggestion: skip the test with a warning if evemu is required but the test
>   is not run as root.

I'll look into using try blocks around evemu calls in tests.

> - I got segfaults when running the tests (approx every second run), see the
>   patches for xorg-gtest to send out but even after those I still get the
>   sporadic:
> unknown file: Failure
> C++ exception with description "Failed to open connection to display" thrown in SetUp().
> [  FAILED  ] XInput2Test.DisableDeviceEndTouches/1, where GetParam() = 1 (51 ms)

No idea here :(.

> Now I'm down to 3 errors that suggest either missing patches in the server
> or broken tests:
> 
> make[1]: Entering directory `/home/whot/xorg/xserver/test/integration'
> [==========] Running 6 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 6 tests from XInput2Test
> [ RUN      ] XInput2Test.XIQueryPointerTouchscreen/0
> xi2.cpp:264: Failure
> Value of: XIMaskIsSet(buttons.mask, 0)
>   Actual: false
> Expected: true
> [  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/0, where GetParam() = 0 (417 ms)
> [ RUN      ] XInput2Test.XIQueryPointerTouchscreen/1
> xi2.cpp:264: Failure
> Value of: XIMaskIsSet(buttons.mask, 0)
>   Actual: false
> Expected: true
> [  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/1, where GetParam() = 1 (366 ms)
> [ RUN      ] XInput2Test.XIQueryPointerTouchscreen/2
> [       OK ] XInput2Test.XIQueryPointerTouchscreen/2 (385 ms)
> [ RUN      ] XInput2Test.DisableDeviceEndTouches/0
> [       OK ] XInput2Test.DisableDeviceEndTouches/0 (345 ms)
> [ RUN      ] XInput2Test.DisableDeviceEndTouches/1
> [       OK ] XInput2Test.DisableDeviceEndTouches/1 (341 ms)
> [ RUN      ] XInput2Test.DisableDeviceEndTouches/2
> xi2.cpp:338: Failure
> Value of: WaitForEventOfType(Display(), GenericEvent, xi2_opcode_, XI_TouchEnd)
>   Actual: false
> Expected: true
> [  FAILED  ] XInput2Test.DisableDeviceEndTouches/2, where GetParam() = 2 (1396 ms)
> [----------] 6 tests from XInput2Test (3251 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 6 tests from 1 test case ran. (6255 ms total)
> [  PASSED  ] 3 tests.
> [  FAILED  ] 3 tests, listed below:
> [  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/0, where GetParam() = 0
> [  FAILED  ] XInput2Test.XIQueryPointerTouchscreen/1, where GetParam() = 1
> [  FAILED  ] XInput2Test.DisableDeviceEndTouches/2, where GetParam() = 2

My guess is you are running a server that doesn't have the fixes for
these bugs. I get the exact same failure on Ubuntu 12.04 LTS.

-- Chase