X Integration test suite

[Chase Douglas]

On 08/29/2012 09:13 PM, Peter Hutterer wrote:
> On Wed, Aug 29, 2012 at 04:22:33PM -0700, Chase Douglas wrote:
>> On 08/29/2012 03:36 PM, Peter Hutterer wrote:
>>> On Wed, Aug 29, 2012 at 01:14:35PM -0700, Chase Douglas wrote:
>>>> On 08/28/2012 03:57 AM, Peter Hutterer wrote:
>>>>> One of the things I've spent quite a bit of time on over the last weeks
>>>>> is a test suite. Chase tried to get some integration tests into the X
>>>>> server repo a while ago but I think a standalone repo is best (for now
>>>>> anyway).
>>>>>
>>>>> I've pushed the current set of tests to
>>>>> http://cgit.freedesktop.org/~whot/xorg-integration-tests/, with a
>>>>> lengthier explanation here:
>>>>> http://who-t.blogspot.com.au/2012/08/xorg-integration-test-suite.html
>>>>>
>>>>> Long story short, the xorg integration tests (XIT) are built on
>>>>> googletest and xorg-gtest, i.e. written in C++. Most of them write out a
>>>>> config, fire up a server, and then either check the log or query the
>>>>> server for state. A few tests use evemu devices to send events.
>>>>>
>>>>> Input testing is relatively simple with these tests since we can emulate
>>>>> virtually anything, but I'm not sure yet if such test cases can scale to
>>>>> output tests as well. Or even if there are tests that can be fully
>>>>> automatic without anyone staring at the screen. I know mesa already has
>>>>> such tests.
>>>>>
>>>>> Any feedback is appreciated, I intend to talk about this a bit more at
>>>>> XDC, but in the meantime you can see if it is useful. Right now, the
>>>>> bigger issues I'm facing are the build system and scalability if we end
>>>>> up with a ton of tests. And the ifdef hell that already started with
>>>>> tests covering different X server versions, RHEL support, etc. Any
>>>>> epiphanies on how to avoid a train wreck would be appreciated.
>>>>
>>>> Overall, I'm quite pleased with how things have progressed :).
>>>>
>>>> As for the run time issue, I assume you are mostly hitting it during
>>>> device tests. Could you find a way to set up all the input config
>>>> blocks you need into the same context, and then start the tests
>>>> against a single server?
>>>
>>> most of the tests (so far) require a specific InputDevice section that may
>>> or may not be CorePointer, or require some xorg.conf option to toggle the
>>> defaults. there are plenty of normal bug tests that don't need that, though
>>> then we do run the chance of having state-dependent tests. instead of
>>> starting with a clean slate for all of them.
>>
>> For those that require specific and different InputDevice sections,
>> you could use MatchProductName (or whatever it is) on the name of
>> the device, and use different named devices for each test.
>>
>> Normal tests should be run on the same server instance. If there is
>> any state dependency, that's also the sign of a bug. In the end,
>> we'd want to be running all of the test in a sequential order, and
>> then in a random order. The sequential order will be a stable test
>> suite, the random order will give us clues if some server state gets
>> messed up as we go.
>
> the problem with random orders are that they're hard to reproduce. It
> doesn't help me when a random-order test fails if I can't run the tests in
> exactly the same order to reproduce. so we'd have to store or print the seed
> somewhere.

In the test output you'll be able to see the order they were run in. You 
can then tell gtest to use that same exact order.

The test output may need to be parsed in order to generate the right 
option string to give to gtest, but that can wait until the first 
failure due to a random test order :). The point is that gtest already 
provides this functionality, so why not take advantage of it?

-- Chase