[PATCH macros] Add XORG_ENABLE_INTEGRATION_TESTS

[Gaetan Nadon]

On 12-03-10 09:02 PM, Chase Douglas wrote:
> On 03/09/2012 05:33 PM, Gaetan Nadon wrote:
>> On 12-03-09 05:49 PM, Chase Douglas wrote:
>>> Copied from XORG_ENABLE_UNIT_TESTS.
>>>
>>> Signed-off-by: Chase Douglas<chase.douglas at canonical.com>
>>> ---
>>> I pasted in a minimum version of 1.17.0, but I don't know if that's
>>> how we do
>>> things.
>> yes, there is a bit of guess work here but it is the right number.
>>>
>>> This will be needed for xorg-gtest and integration testing in other
>>> projects.
>>>
>>>   xorg-macros.m4.in |   29 +++++++++++++++++++++++++++++
>>>   1 files changed, 29 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/xorg-macros.m4.in b/xorg-macros.m4.in
>>> index 4966df3..9cd73c0 100644
>>> --- a/xorg-macros.m4.in
>>> +++ b/xorg-macros.m4.in
>>> @@ -1056,6 +1056,35 @@ AC_MSG_CHECKING([whether to build unit test
>>> cases])
>>>   AC_MSG_RESULT([$enable_unit_tests])
>>>   ]) # XORG_ENABLE_UNIT_TESTS
>>>
>>> +# XORG_ENABLE_INTEGRATION_TESTS (enable_unit_tests=auto)
>>> +# ------------------------------------------------------
>>> +# Minimum version: 1.17.0
>>> +#
>>> +# This macro enables a builder to enable/disable integration testing
>>> +# It makes no assumption about the test cases' implementation
>>> +# Test cases may or may not use Automake "Support for test suites"
>>> +#
>>> +# Interface to module:
>>> +# ENABLE_INTEGRATION_TESTS:   used in makefiles to conditionally
>>> build tests
>>> +# enable_integration_tests:   used in configure.ac for additional
>>> configuration
>>> +# --enable-integration-tests: 'yes' user instructs the module to
>>> build tests
>>> +#                             'no' user instructs the module not to
>>> build tests
>>> +# parm1:                      specify the default value, yes or no.
>>> +#
>>> +AC_DEFUN([XORG_ENABLE_INTEGRATION_TESTS],[
>>> +AC_REQUIRE([XORG_MEMORY_CHECK_FLAGS])
>>> +m4_define([_defopt], m4_default([$1], [auto]))
>>> +AC_ARG_ENABLE(integration-tests,
>>> AS_HELP_STRING([--enable-integration-tests],
>>> +    [Enable building integration test cases (default: ]_defopt[)]),
>>> +    [enable_integration_tests=$enableval],
>>> +    [enable_integration_tests=]_defopt)
>>> +m4_undefine([_defopt])
>>> +AM_CONDITIONAL([ENABLE_INTEGRATION_TESTS],
>>> +    [test "x$enable_integration_tests" != xno])
>>> +AC_MSG_CHECKING([whether to build unit test cases])
>>> +AC_MSG_RESULT([$enable_integration_tests])
>>> +]) # XORG_ENABLE_INTEGRATION_TESTS
>>> +
>>>   # XORG_WITH_GLIB([MIN-VERSION], [DEFAULT])
>>>   # ----------------------------------------
>>>   # Minimum version: 1.13.0
>>
>> You must have considered using ENABLE_UNIT_TESTS but came to the
>> conclusion that it would not be suitable. On the other hand it seems to
>> do exactly the same thing. Can you share your conclusions?
>>
>> The motivation behind ENABLE_UNIT_TEST is to solve the problem where a
>> few modules started adding unit testing, each of them using a different
>> configuration option, working a little differently. This makes it hard
>> for builders to run unit testing on all modules as they must read and
>> understand each module code and write their build script accordingly. No
>> ones fault, it's the nature of X being distributed in over 240 separate
>> modules.
>>
>> The only mission of the macro is to provide a common configure option
>> (--enable-unit-tests) with a matching ENABLE_UNIT_TESTS variable.
>> Nothing else is planned for this macro. It does not require or depend on
>> XORG_WITH_GLIB or XORG_LD_WRAP.
>>
>> I am not familiar with gtest, it's hard for me to say if
>> --enable-unit-tests would be appropriate. You can look at xserver and
>> libXt as examples of modules using it.
>
> The issue is that integration testing usually has different
> dependencies from unit testing. Unit testing usually has no
> dependencies, or maybe a dependency on a test framework. However,
> integration testing usually has dependencies of a working stack. For
> example, input integration testing for the X server will depend on:
>
> xorg-gtest
> libX11
> libXi
> libutouch-evemu
> linux
> root privileges, or special udev rules
>
> And development headers for it all. Integration testing for other
> subsystems would have different stack requirements.
>
> One example where builds will want unit tests but not integration
> tests is package builds. Debian package builds build and run the unit
> tests during the build and fails the build if the unit tests fail.
> However, integration tests during debian package builds isn't really
> feasible without potentially opening security holes.
>
> I think unit and integration tests are sufficiently different that a
> separate configuration option is required.
>
This is what I was hoping for. I assume that this new macro can be used
by many other packages, not just xorg-gtest.

Would you mind adding the description in the commit text? I'd like to
avoid others to randomly pick integration test macro over unit test
macro or vice-versa. This is all the documentation we've got, aside from
this list.

Feel free to push. I'll make a release after I see a few runs on tinderbox.

Reviewed-by: Gaetan Nadon<memsize at videotron.ca>



> Thanks,
>
> -- Chase
>