Modularization mailing list and initial strawman proposal

[Roland Mainz]

Kevin E Martin wrote:
> A few weeks ago the Modularization Working Group was announced, and
> several people expressed interest in helping out.  Many of them attended
> the recent X Developers Conference meeting in Boston and talked about
> how modularization might be accomplished.  Since there had not been very
> much discussion on the mailing lists after the initial announcement, we
> decided to take advantage of our face to face time to lay the groundwork
> for creating a proposal.  We have now worked through the issues that we
> discovered in an effort to create a strawman proposal that describes the
> rationale for modularization, attempts to address the known concerns
> people have raised, provides background on other modularization efforts,
> explains what the modularized source tree would look like, outlines the
> steps required to take this idea from concept to release, and suggests
> how future modular releases can be handled.
> 
> Please note that this is only an initial strawman proposal.  Nothing is
> set in stone.  We would like to open the discussion up to the larger
> community and the Architecture Working Group to build consensus.  Feel
> free to comment.
> 
> The initial strawman proposal has been put into the X.Org wiki here:
> 
>     http://wiki.x.org/wiki/ModularizationProposal

OK... some issues&nitpicking after having a quick look on that paper (I
didn't checked all details, it's just what I found after reading once
over it):
-- snip --
>   Introduction
[snip]
>     The cons
> 
>      * The autotools are a new build technology, which throws away
>        existing knowledge of the current imake build system. People that
>        are not already familiar with the autotools will need to learn a
>        new build system. That could slow some developers down for a
>        while.

Additionally there is the question whether all platforms supported by
Imake are supported or can be supported by design (OpenVMS and QNX come
in mind where "autoconf" support can only be described as poor ...).
What should we do here ? The worst case would be that X.org quickly
becomes "upstream" for autoconf support on these platforms... =:-)

>      * To address this issue, we need people who understand both imake
>        and autotools well to write a transition guide for the benefit of
>        all.
> 
>    Some platforms may not "come along" if there are no maintainers.
>      * This should be treated as an opportunity for us to contact the old
>        maintainers and see if they are interested joining the community,
>        or for us to encourage to new maintainers to step forward to
>        support platforms if the old maintainer is no longer interested or
>        available.

What happens if the maintainers of these platforms do not have the
resources (e.g. time) to do the transition ? The Imake and Imakefiles
exist since many many years and it is VERY unlikely that we can expect
to pull over all changes to all platforms within just ... say... three
months (assuming the X11R7 release should appear in summer 2005). That's
pretty much impossible.

The alternative reaction by such maintainers may be to ignore the
existance of the modular tree and move over to other development
platforms which are still compatible with their OSes (such as Xfree86)
...

[snip]
>    Autodetection can add dependencies that you didn't intend. For
>    example, a developer may have a library on their build system that
>    isn't available on the target system.
>      * Dependencies will be explicitly listed and documented. Configure
>        options will be used to control which dependencies are enabled or
>        disabled.

How many "configure" options will that be ? 50... 100... 200 ? It may be
better to look into a way of a config file a lá xc/config/cf/host.def to
avoid having giant command lines which call "configure" (mozilla has a
non-standard solution which may be adoptable in this case).

[snip]
>    With the imake build system, you can build and test in place with the
>    link directory inside the tree. Building self-contained user
>    environments are more difficult with autotools.
>      * jhbuild solves this since it runs a self-contained environment.
>        Other solutions using chroot, LD_LIBRARY_PATH, etc. are also in
>        use today with autotools.

"chroot" is no option. It MUST be possible to build and test the tree
without having "root" (or similar) priviledges. Otherwise it will be
impossible to do testing on systems like "compile farms" (such as
sourceforge's compile farm) as they (usually) NEVER grant "root" access.

[snip]
>    Legacy imake-only applications will need to be converted.
>      * Imake is not going away and will still be needed for legacy
>        applications maintained outside of our source tree. The transition
>        guide that we use internally will be made available externally to
>        help others become familiar with how the new autotool build system
>        works so that they can convert their applications as time permits.
[snip]
>    Some of the issues we encountered:
[snip]
>      * No other DDXes besides the xfree86 DDX were attempted. The xnest,
>        xwin, and xgl DDXes have been autotooled in Keith's xserver
>        project. The darwin, dmx, sun, sunlynx, vfb, and xprint (and
>        probably xrx counts here too) DDXes have not (to my knowledge)

Uhm... there is no "xrx" DDX. "xrx" (the Broadway plugin) is 100% client
side, either via the standalone "xrx" tool, the libxrx.so plugin or the
libxrxnest.so plugin (which embeds Xnest (per IRC comments from ajax
this was the part which led him to the conclusion that XRX is a seperate
DDX) for the case that the main Xserver is inaccessible (firewall
blocking, Xserver started with -nolisten tcp or a paranoid user doesn't
even trust clients running as "untrusted" X clients))

[snip] 
>    Note that some platforms only have 1 or 2 component revisions for
>    their libraries. On these systems, either the second component (y)
>    could be bumped, or symbol versioning could be used by vendors who
>    package the libraries.
> 
> 
> 
>    By default, all libraries are built as shared objects; however, with
>    other build options, static, profiled and debug versions can be built.

What about libraries which "unstable" APIs which are currently build as
"static" versions for that reason (XprintUtils comes in mind from my
side) ? And what about helper libraries which are currently used to
build the Xserver ?

>    The protocol module: proto
>      * As noted above, this module contains the protocol headers that are
>        required to build the lib and xserver modules.
>      * In addition, the protocol specifications will be included in this
>        module.
> 
>    The X server module: xserver
>      * The X server is also one of the more difficult parts of the tree
>        to modularize since it also has external dependencies, there is a
>        strong desire among the vendors to separate the drivers into their
>        own modules, and the different DDXs support different build
>        options.
>      * The XFree86 drivers depend on the X server headers, and since we
>        are splitting the drivers into a separate module, these headers
>        will need to be installed before the xfree86-driver module can
>        built. These headers will installed in /usr/include/.../xserver as
>        part of the normal install target. A special DDK install could
>        also be added, if that proved useful.
>      * At some point we may want to investigate splitting each of the
>        DDXs into separate modules, but not at this time. The current
>        design does not preclude doing this research in the future.
>      * The xserver module depends on the proto module, and some of the
>        DDXs depend on the lib module (e.g., Xnest, DMX).

... and "Xprt", too.

>      * As noted above, some DDXs require different built options, which
>        implies that not all DDXs will be able to use the same dix, os,
>        mi, etc. compile.

Why is this a problem for the modular build ? Right now all servers
(Xorg, Xnest, Xdmx, Xprt, Xvfb) can be build in one "make World" step.
The same should be possible in the "modular" world.

>        However, they can and should share the same
>        source code. For example, the kdrive server does not support
>        XINPUT whereas the Xorg server does, so dix (and other subdirs)
>        will need to be built differently for each server. Note that these
>        conflicts can be detected at configure time.

See above. There should be no such seperation - the "monolithic" Xorg
tree didn't need it so why should the modular build system introduce
such a pain (currently stub functions and *.a files are used to avoid
this and the same could be done for the modular build) ?

>      * The basic tree structure in the xserver module will look similar
>        to the one used in the monolithic tree under xc/programs/Xserver.
>        Notable exceptions include the drivers that will be moved to their
>        own module (see below), and support programs that could be moved
>        to the app module.
> 
>    The XFree86 drivers module: xfree86-driver

Why can't it be named "drivers" (yes, I know, "Xfree86" should be
referenced but that can be done in the docs) ? I don't like to have
giant-overlong-pathnames and the drivers/ subdir will likely host other
kinds of drivers like the kdrive, Sun DDX or the Xprint ones...

>      * Vendors have expressed a strong desire to be able to ship updates
>        to individual drivers as bugs are fixed, so they have been split
>        out into their own module.
>      * The drivers consist of both input and output (i.e., video)
>        drivers.

And print drivers as the Xprint DDXs should be turned into loadable
modules (in the long term the Xprt server is going away then[1],
replaced by a Xorg server which can handle both video and print modules
similar to what the Xserver in HP/UX can do today).

[1]=or more correctly: Xprt isn't going away as it's still needed as
all-in-one X print server, however sepcial-purpose DDX like the SVGprint
DDX may only be available as loadable module.

>      * Each driver will be completely self-contained and placed in a
>        separate subdir within the module.
>      * The xfree86-driver module depends on the headers from the xserver
>        module being installed.
>      * Vendors can choose to distribute the entire "driver" module as a
>        whole, or they can subdivide this module into finer components if
>        desired.
> 
>    The font module: font
>      * This module contains the fonts traditionally distributed with the
>        monolithic releases.
>      * Note that some fonts have more restrictive licenses and these
>        should be fully documented in this module.
>      * The font module depends on the font tools that will be included in
>        the app module be installed.
> 
>    The documentation module: doc
>      * This module contains the documentation traditionally distributed
>        with the monolithic releases that has not been moved to another
>        module.
>      * The majority of documentation in xc/doc will be relocated to the
>        module that they document. For example, the protocol
>        specifications will be moved to the proto module and the library
>        docs will be moved to the lib module.

If this is done then we need a common build infrastructure to build the
documentation in all places (this needs to be done in general a lá
xc/config/cf/X11.tmp && xc/config/X11/Imake.* to have centralised place
for all the build rules to avoid that they are replicated all over the
source tree).

>    The build utilities module: util
>      * This module contains the build tools and configuration files for
>        legacy applications and libraries that have not been converted to
>        use the autotools.
>      * Other utilities will also be placed here (e.g., makedepend, lndir
>        and mkshadow).

-- snip --

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)