[PATCH macros] Add XORG_ENABLE_INTEGRATION_TESTS

[Chase Douglas]

On 03/11/2012 06:25 AM, Gaetan Nadon wrote:
> On 12-03-10 09:02 PM, Chase Douglas wrote:
>> On 03/09/2012 05:33 PM, Gaetan Nadon wrote:
>>> On 12-03-09 05:49 PM, Chase Douglas wrote:
>>>> Copied from XORG_ENABLE_UNIT_TESTS.
>>>>
>>>> Signed-off-by: Chase Douglas<chase.douglas at canonical.com>
>>>> ---
>>>> I pasted in a minimum version of 1.17.0, but I don't know if that's
>>>> how we do
>>>> things.
>>> yes, there is a bit of guess work here but it is the right number.
>>>>
>>>> This will be needed for xorg-gtest and integration testing in other
>>>> projects.
>>>>
>>>>    xorg-macros.m4.in |   29 +++++++++++++++++++++++++++++
>>>>    1 files changed, 29 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/xorg-macros.m4.in b/xorg-macros.m4.in
>>>> index 4966df3..9cd73c0 100644
>>>> --- a/xorg-macros.m4.in
>>>> +++ b/xorg-macros.m4.in
>>>> @@ -1056,6 +1056,35 @@ AC_MSG_CHECKING([whether to build unit test
>>>> cases])
>>>>    AC_MSG_RESULT([$enable_unit_tests])
>>>>    ]) # XORG_ENABLE_UNIT_TESTS
>>>>
>>>> +# XORG_ENABLE_INTEGRATION_TESTS (enable_unit_tests=auto)
>>>> +# ------------------------------------------------------
>>>> +# Minimum version: 1.17.0
>>>> +#
>>>> +# This macro enables a builder to enable/disable integration testing
>>>> +# It makes no assumption about the test cases' implementation
>>>> +# Test cases may or may not use Automake "Support for test suites"
>>>> +#
>>>> +# Interface to module:
>>>> +# ENABLE_INTEGRATION_TESTS:   used in makefiles to conditionally
>>>> build tests
>>>> +# enable_integration_tests:   used in configure.ac for additional
>>>> configuration
>>>> +# --enable-integration-tests: 'yes' user instructs the module to
>>>> build tests
>>>> +#                             'no' user instructs the module not to
>>>> build tests
>>>> +# parm1:                      specify the default value, yes or no.
>>>> +#
>>>> +AC_DEFUN([XORG_ENABLE_INTEGRATION_TESTS],[
>>>> +AC_REQUIRE([XORG_MEMORY_CHECK_FLAGS])
>>>> +m4_define([_defopt], m4_default([$1], [auto]))
>>>> +AC_ARG_ENABLE(integration-tests,
>>>> AS_HELP_STRING([--enable-integration-tests],
>>>> +    [Enable building integration test cases (default: ]_defopt[)]),
>>>> +    [enable_integration_tests=$enableval],
>>>> +    [enable_integration_tests=]_defopt)
>>>> +m4_undefine([_defopt])
>>>> +AM_CONDITIONAL([ENABLE_INTEGRATION_TESTS],
>>>> +    [test "x$enable_integration_tests" != xno])
>>>> +AC_MSG_CHECKING([whether to build unit test cases])
>>>> +AC_MSG_RESULT([$enable_integration_tests])
>>>> +]) # XORG_ENABLE_INTEGRATION_TESTS
>>>> +
>>>>    # XORG_WITH_GLIB([MIN-VERSION], [DEFAULT])
>>>>    # ----------------------------------------
>>>>    # Minimum version: 1.13.0
>>>
>>> You must have considered using ENABLE_UNIT_TESTS but came to the
>>> conclusion that it would not be suitable. On the other hand it seems to
>>> do exactly the same thing. Can you share your conclusions?
>>>
>>> The motivation behind ENABLE_UNIT_TEST is to solve the problem where a
>>> few modules started adding unit testing, each of them using a different
>>> configuration option, working a little differently. This makes it hard
>>> for builders to run unit testing on all modules as they must read and
>>> understand each module code and write their build script accordingly. No
>>> ones fault, it's the nature of X being distributed in over 240 separate
>>> modules.
>>>
>>> The only mission of the macro is to provide a common configure option
>>> (--enable-unit-tests) with a matching ENABLE_UNIT_TESTS variable.
>>> Nothing else is planned for this macro. It does not require or depend on
>>> XORG_WITH_GLIB or XORG_LD_WRAP.
>>>
>>> I am not familiar with gtest, it's hard for me to say if
>>> --enable-unit-tests would be appropriate. You can look at xserver and
>>> libXt as examples of modules using it.
>>
>> The issue is that integration testing usually has different
>> dependencies from unit testing. Unit testing usually has no
>> dependencies, or maybe a dependency on a test framework. However,
>> integration testing usually has dependencies of a working stack. For
>> example, input integration testing for the X server will depend on:
>>
>> xorg-gtest
>> libX11
>> libXi
>> libutouch-evemu
>> linux
>> root privileges, or special udev rules
>>
>> And development headers for it all. Integration testing for other
>> subsystems would have different stack requirements.
>>
>> One example where builds will want unit tests but not integration
>> tests is package builds. Debian package builds build and run the unit
>> tests during the build and fails the build if the unit tests fail.
>> However, integration tests during debian package builds isn't really
>> feasible without potentially opening security holes.
>>
>> I think unit and integration tests are sufficiently different that a
>> separate configuration option is required.
>>
> This is what I was hoping for. I assume that this new macro can be used
> by many other packages, not just xorg-gtest.
>
> Would you mind adding the description in the commit text? I'd like to
> avoid others to randomly pick integration test macro over unit test
> macro or vice-versa. This is all the documentation we've got, aside from
> this list.
>
> Feel free to push. I'll make a release after I see a few runs on tinderbox.
>
> Reviewed-by: Gaetan Nadon<memsize at videotron.ca>

I've pushed it. Please review my last two patches on the list before 
releasing 1.17.0. They are required for proper xorg-gtest setup in 
non-c++ projects.

Thanks!

-- Chase