[PATCH macros] Add XORG_ENABLE_INTEGRATION_TESTS

[Chase Douglas]

On 03/09/2012 05:33 PM, Gaetan Nadon wrote:
> On 12-03-09 05:49 PM, Chase Douglas wrote:
>> Copied from XORG_ENABLE_UNIT_TESTS.
>>
>> Signed-off-by: Chase Douglas<chase.douglas at canonical.com>
>> ---
>> I pasted in a minimum version of 1.17.0, but I don't know if that's how we do
>> things.
> yes, there is a bit of guess work here but it is the right number.
>>
>> This will be needed for xorg-gtest and integration testing in other projects.
>>
>>   xorg-macros.m4.in |   29 +++++++++++++++++++++++++++++
>>   1 files changed, 29 insertions(+), 0 deletions(-)
>>
>> diff --git a/xorg-macros.m4.in b/xorg-macros.m4.in
>> index 4966df3..9cd73c0 100644
>> --- a/xorg-macros.m4.in
>> +++ b/xorg-macros.m4.in
>> @@ -1056,6 +1056,35 @@ AC_MSG_CHECKING([whether to build unit test cases])
>>   AC_MSG_RESULT([$enable_unit_tests])
>>   ]) # XORG_ENABLE_UNIT_TESTS
>>
>> +# XORG_ENABLE_INTEGRATION_TESTS (enable_unit_tests=auto)
>> +# ------------------------------------------------------
>> +# Minimum version: 1.17.0
>> +#
>> +# This macro enables a builder to enable/disable integration testing
>> +# It makes no assumption about the test cases' implementation
>> +# Test cases may or may not use Automake "Support for test suites"
>> +#
>> +# Interface to module:
>> +# ENABLE_INTEGRATION_TESTS:   used in makefiles to conditionally build tests
>> +# enable_integration_tests:   used in configure.ac for additional configuration
>> +# --enable-integration-tests: 'yes' user instructs the module to build tests
>> +#                             'no' user instructs the module not to build tests
>> +# parm1:                      specify the default value, yes or no.
>> +#
>> +AC_DEFUN([XORG_ENABLE_INTEGRATION_TESTS],[
>> +AC_REQUIRE([XORG_MEMORY_CHECK_FLAGS])
>> +m4_define([_defopt], m4_default([$1], [auto]))
>> +AC_ARG_ENABLE(integration-tests, AS_HELP_STRING([--enable-integration-tests],
>> +	[Enable building integration test cases (default: ]_defopt[)]),
>> +	[enable_integration_tests=$enableval],
>> +	[enable_integration_tests=]_defopt)
>> +m4_undefine([_defopt])
>> +AM_CONDITIONAL([ENABLE_INTEGRATION_TESTS],
>> +	[test "x$enable_integration_tests" != xno])
>> +AC_MSG_CHECKING([whether to build unit test cases])
>> +AC_MSG_RESULT([$enable_integration_tests])
>> +]) # XORG_ENABLE_INTEGRATION_TESTS
>> +
>>   # XORG_WITH_GLIB([MIN-VERSION], [DEFAULT])
>>   # ----------------------------------------
>>   # Minimum version: 1.13.0
>
> You must have considered using ENABLE_UNIT_TESTS but came to the
> conclusion that it would not be suitable. On the other hand it seems to
> do exactly the same thing. Can you share your conclusions?
>
> The motivation behind ENABLE_UNIT_TEST is to solve the problem where a
> few modules started adding unit testing, each of them using a different
> configuration option, working a little differently. This makes it hard
> for builders to run unit testing on all modules as they must read and
> understand each module code and write their build script accordingly. No
> ones fault, it's the nature of X being distributed in over 240 separate
> modules.
>
> The only mission of the macro is to provide a common configure option
> (--enable-unit-tests) with a matching ENABLE_UNIT_TESTS variable.
> Nothing else is planned for this macro. It does not require or depend on
> XORG_WITH_GLIB or XORG_LD_WRAP.
>
> I am not familiar with gtest, it's hard for me to say if
> --enable-unit-tests would be appropriate. You can look at xserver and
> libXt as examples of modules using it.

The issue is that integration testing usually has different dependencies 
from unit testing. Unit testing usually has no dependencies, or maybe a 
dependency on a test framework. However, integration testing usually has 
dependencies of a working stack. For example, input integration testing 
for the X server will depend on:

xorg-gtest
libX11
libXi
libutouch-evemu
linux
root privileges, or special udev rules

And development headers for it all. Integration testing for other 
subsystems would have different stack requirements.

One example where builds will want unit tests but not integration tests 
is package builds. Debian package builds build and run the unit tests 
during the build and fails the build if the unit tests fail. However, 
integration tests during debian package builds isn't really feasible 
without potentially opening security holes.

I think unit and integration tests are sufficiently different that a 
separate configuration option is required.

Thanks,

-- Chase