X.org testing with Google Test

[Matt Dew]

On 01/02/2012 01:18 AM, Thomas Voß wrote:
> On 12/30/2011 10:24 PM, Matt Dew wrote:
>>
>> On 12/23/2011 04:36 PM, Ian Romanick wrote:
>>> On 12/21/2011 08:51 AM, Thomas Voß wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> Hi list,
>>>>
>>>> my name is Thomas and I'm a software engineer in quality working for
>>>> Canonical. Our quality team is currently extending the test coverage
>>>> of Canonical's upstream projects. To this end, we are developing a
>>>> number of components for providing a unified testing landscape to
>>>> developers.
>>>>
>>>> One of our focus areas is the headless automated testing of
>>>> applications and components that interact with the X display server.
>>>> For many testing scenarios, only a dummy X server without any display
>>>> capabilities is required and we abstracted/integrated start up and
>>>> tear down of such a configuration within a Google Test environment
>>>> that is readily usable by developers (see [1]).
>>>>
>>>> Google Test [2] is a unit testing framework tailored towards C and
>>>> C++. It is lightweight and provides a decent set of functionality for
>>>> writing unit tests:
>>>
>>> I'm in favor of gtest. I've just started using it with Mesa, and I have
>>> some (unposted) unit tests for the GLX_ARB_create_context patch series
>>> that I just posted to the list.
>>>
>>>> * Automatic test registration: Tests are automatically discovered and
>>>> recognized. That is, no manual maintenance for executing tests is
>>>> required.
>>>>
>>>> * An extensive set of assertion macros is provided.
>>>>
>>>> * The XML output is compatible with JUnit's XML dialect and can be
>>>> integrated easily with continuous integration systems (e.g. Jenkins
>>>> [3]).
>>>>
>>>> * Support for custom test fixtures that unify test setup and cleanup
>>>> procedures, thus avoiding boiler plate code within the actual test.
>>>>
>>>> * Support for custom test environments for bundling problem specific
>>>> setup and cleanup routines.
>>>
>>> The big advantage of a test framework like gtest is that the
>>> infrastructure makes it easy to write a large number of small, easy to
>>> understand (WTF is this testing?), easy to debug (WTF is this failing?)
>>> tests. For the work mentioned above, I wrote about 75 tests. It wasn't
>>> until I was done that I even realized how many there were. This seems
>>> like a good thing.
>>>
>>>> Our extension to Google Test defines a custom test environment and a
>>>> custom test fixture that take care of starting up and tearing down a
>>>> dummy xserver environment (environment.h) as well as opening up a
>>>> display connection (test.h). Moreover, a custom main is included
>>>> within our extension that extends Google Test's main and makes sure
>>>> that our dummy X server testing environment is known to the whole test
>>>> suite. Please see [4] for a usage scenario within utouch-frame and [5]
>>>> for xorg-gtest's doxygen documentation.
>>>>
>>>> As we think that this kind of unified testing architecture might be of
>>>> interest to a wider developer audience, we would like to bring it to
>>>> your attention by this mail. The project is currently hosted in
>>>> cndougla's freedesktop.org place, but we feel a better home would be
>>>> on freedesktop.org. It looks like a good landing spot would be in
>>>> xorg/test on the git.freedesktop.org page. Is there a formal process
>>>> for proposing projects, or should we just create a repository there?
>>>>
>>>> Of course, your feedback is highly appreciated.
>>>>
>>>> Kind regards,
>>>>
>>>> Thomas
>>>>
>>
>> Hi guys,
>> I'm all for testing frameworks. Would this tie into tinderbox at all?
>> From what I understand, tinderbox just does periodic compiles to make
>> sure builds didn't break but isn't actually a formal test environment.
>> Would this remain completely independent of that?
>> If I'm wrong on tinderbox, please correct me.
>>
>
> If I understand it correctly, tinderbox is a continuous integration
> system (much like Jenkins) that compiles and executes unit tests. That
> is, it should be no problem to execute the unit test executables built
> with our framework/Google test.
>

I don't see anything on the tinderbox regarding unit tests so I cc'd 
Chris Ball who's listed as the contact on tinderbox, to be safe.


> Google test produces XML output that we are using in Jenkins to
> visualize test results and report on a project's build health. I do not
> know whether this kind of integration would be possible with tinderbox
> as well.

Chris, can you comment on this thread?

thanks,
Matt