X.org testing with Google Test

[Matt Dew]

On 12/23/2011 04:36 PM, Ian Romanick wrote:
> On 12/21/2011 08:51 AM, Thomas Voß wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi list,
>>
>> my name is Thomas and I'm a software engineer in quality working for
>> Canonical. Our quality team is currently extending the test coverage
>> of Canonical's upstream projects. To this end, we are developing a
>> number of components for providing a unified testing landscape to
>> developers.
>>
>> One of our focus areas is the headless automated testing of
>> applications and components that interact with the X display server.
>> For many testing scenarios, only a dummy X server without any display
>> capabilities is required and we abstracted/integrated start up and
>> tear down of such a configuration within a Google Test environment
>> that is readily usable by developers (see [1]).
>>
>> Google Test [2] is a unit testing framework tailored towards C and
>> C++. It is lightweight and provides a decent set of functionality for
>> writing unit tests:
>
> I'm in favor of gtest. I've just started using it with Mesa, and I have
> some (unposted) unit tests for the GLX_ARB_create_context patch series
> that I just posted to the list.
>
>> * Automatic test registration: Tests are automatically discovered and
>> recognized. That is, no manual maintenance for executing tests is
>> required.
>>
>> * An extensive set of assertion macros is provided.
>>
>> * The XML output is compatible with JUnit's XML dialect and can be
>> integrated easily with continuous integration systems (e.g. Jenkins
>> [3]).
>>
>> * Support for custom test fixtures that unify test setup and cleanup
>> procedures, thus avoiding boiler plate code within the actual test.
>>
>> * Support for custom test environments for bundling problem specific
>> setup and cleanup routines.
>
> The big advantage of a test framework like gtest is that the
> infrastructure makes it easy to write a large number of small, easy to
> understand (WTF is this testing?), easy to debug (WTF is this failing?)
> tests. For the work mentioned above, I wrote about 75 tests. It wasn't
> until I was done that I even realized how many there were. This seems
> like a good thing.
>
>> Our extension to Google Test defines a custom test environment and a
>> custom test fixture that take care of starting up and tearing down a
>> dummy xserver environment (environment.h) as well as opening up a
>> display connection (test.h). Moreover, a custom main is included
>> within our extension that extends Google Test's main and makes sure
>> that our dummy X server testing environment is known to the whole test
>> suite. Please see [4] for a usage scenario within utouch-frame and [5]
>> for xorg-gtest's doxygen documentation.
>>
>> As we think that this kind of unified testing architecture might be of
>> interest to a wider developer audience, we would like to bring it to
>> your attention by this mail. The project is currently hosted in
>> cndougla's freedesktop.org place, but we feel a better home would be
>> on freedesktop.org. It looks like a good landing spot would be in
>> xorg/test on the git.freedesktop.org page. Is there a formal process
>> for proposing projects, or should we just create a repository there?
>>
>> Of course, your feedback is highly appreciated.
>>
>> Kind regards,
>>
>> Thomas
>>

Hi guys,
    I'm all for testing frameworks.   Would this tie into tinderbox at 
all?  From what I understand, tinderbox just does periodic compiles to 
make sure builds didn't break but isn't actually a formal test 
environment.  Would this remain completely independent of that?
If I'm wrong on tinderbox, please correct me.

Matt