X Integration test suite

[Peter Hutterer]

On Wed, Aug 29, 2012 at 01:14:35PM -0700, Chase Douglas wrote:
> On 08/28/2012 03:57 AM, Peter Hutterer wrote:
> >One of the things I've spent quite a bit of time on over the last weeks
> >is a test suite. Chase tried to get some integration tests into the X
> >server repo a while ago but I think a standalone repo is best (for now
> >anyway).
> >
> >I've pushed the current set of tests to
> >http://cgit.freedesktop.org/~whot/xorg-integration-tests/, with a
> >lengthier explanation here:
> >http://who-t.blogspot.com.au/2012/08/xorg-integration-test-suite.html
> >
> >Long story short, the xorg integration tests (XIT) are built on
> >googletest and xorg-gtest, i.e. written in C++. Most of them write out a
> >config, fire up a server, and then either check the log or query the
> >server for state. A few tests use evemu devices to send events.
> >
> >Input testing is relatively simple with these tests since we can emulate
> >virtually anything, but I'm not sure yet if such test cases can scale to
> >output tests as well. Or even if there are tests that can be fully
> >automatic without anyone staring at the screen. I know mesa already has
> >such tests.
> >
> >Any feedback is appreciated, I intend to talk about this a bit more at
> >XDC, but in the meantime you can see if it is useful. Right now, the
> >bigger issues I'm facing are the build system and scalability if we end
> >up with a ton of tests. And the ifdef hell that already started with
> >tests covering different X server versions, RHEL support, etc. Any
> >epiphanies on how to avoid a train wreck would be appreciated.
> 
> Overall, I'm quite pleased with how things have progressed :).
> 
> As for the run time issue, I assume you are mostly hitting it during
> device tests. Could you find a way to set up all the input config
> blocks you need into the same context, and then start the tests
> against a single server?

most of the tests (so far) require a specific InputDevice section that may
or may not be CorePointer, or require some xorg.conf option to toggle the
defaults. there are plenty of normal bug tests that don't need that, though
then we do run the chance of having state-dependent tests. instead of
starting with a clean slate for all of them.

Cheers,
   Peter