Modularization mailing list and initial strawman proposal

[Kevin E Martin]

On Fri, Mar 18, 2005 at 09:39:32PM +0100, Roland Mainz wrote:
> >     The cons
> > 
> >      * The autotools are a new build technology, which throws away
> >        existing knowledge of the current imake build system. People that
> >        are not already familiar with the autotools will need to learn a
> >        new build system. That could slow some developers down for a
> >        while.
> 
> Additionally there is the question whether all platforms supported by
> Imake are supported or can be supported by design (OpenVMS and QNX come
> in mind where "autoconf" support can only be described as poor ...).
> What should we do here ? The worst case would be that X.org quickly
> becomes "upstream" for autoconf support on these platforms... =:-)

The maintainers of those platforms should look at this as an opportunity
to work with the autoconf maintainers to get their platforms better
supported.  Then, not only will they have their platforms supported in
official X.Org releases, but also gain access to a great number of apps,
libs, etc. that are already autotooled.

No one expects a platform to be ported over with zero work.  There is
real work to be done both from the platforms that are well supported and
those there are not.  It is up to each platform maintainer to decide how
best to focus their efforts.

> >    Some platforms may not "come along" if there are no maintainers.
> >      * This should be treated as an opportunity for us to contact the old
> >        maintainers and see if they are interested joining the community,
> >        or for us to encourage to new maintainers to step forward to
> >        support platforms if the old maintainer is no longer interested or
> >        available.
> 
> What happens if the maintainers of these platforms do not have the
> resources (e.g. time) to do the transition ? The Imake and Imakefiles
> exist since many many years and it is VERY unlikely that we can expect
> to pull over all changes to all platforms within just ... say... three
> months (assuming the X11R7 release should appear in summer 2005). That's
> pretty much impossible.

As quoted above and as noted elsewhere in the proposal, we encourage
current maintainers to step forward and help out with the work on
platforms that are not well supported or new maintainers to step forward
for platforms with no current maintainer.  This is an opportunity.  No
one expects that it will be possible to port all platforms immediately;
however, that is no reason to hold up progress for those that do have
the resources to make it happen today.  Also, X.Org has been very good
about having relatively quick releases -- we've had 4 releases in the
last 12 months.  There is no reason to expect that this won't continue,
and with modularization, there is the possibility that releases will
become simpler and more frequent.  This gives those maintainers who
don't have the time to finish the support for their platforms in the
first release cycle many future opportunities to get their platforms
supported in follow-on releases.

> [snip]
> >    Autodetection can add dependencies that you didn't intend. For
> >    example, a developer may have a library on their build system that
> >    isn't available on the target system.
> >      * Dependencies will be explicitly listed and documented. Configure
> >        options will be used to control which dependencies are enabled or
> >        disabled.
> 
> How many "configure" options will that be ? 50... 100... 200 ? It may be
> better to look into a way of a config file a lá xc/config/cf/host.def to
> avoid having giant command lines which call "configure" (mozilla has a
> non-standard solution which may be adoptable in this case).

There are currently hundreds of options in the xc/config/cf/* files.
Some options will no longer be needed as they will be automatically
handled by the autotools.  Others can be handled via configure options.
Just like the cf options are given reasonable defaults, the configure
options will also have reasonable defaults.

As you say above, these could also be handled in other ways to automate
the build process.  We should investigate those and other solutions
during the initial development cycle.

> [snip]
> >    With the imake build system, you can build and test in place with the
> >    link directory inside the tree. Building self-contained user
> >    environments are more difficult with autotools.
> >      * jhbuild solves this since it runs a self-contained environment.
> >        Other solutions using chroot, LD_LIBRARY_PATH, etc. are also in
> >        use today with autotools.
> 
> "chroot" is no option. It MUST be possible to build and test the tree
> without having "root" (or similar) priviledges. Otherwise it will be
> impossible to do testing on systems like "compile farms" (such as
> sourceforge's compile farm) as they (usually) NEVER grant "root" access.

Sure, chroot requires superuser privileges on many OSes; however, that
does not invalidate its ability to solve this problem.  Also, jhbuild
solves this problem very nicely without requiring any root privileges.

> >      * No other DDXes besides the xfree86 DDX were attempted. The xnest,
> >        xwin, and xgl DDXes have been autotooled in Keith's xserver
> >        project. The darwin, dmx, sun, sunlynx, vfb, and xprint (and
> >        probably xrx counts here too) DDXes have not (to my knowledge)
> 
> Uhm... there is no "xrx" DDX. "xrx" (the Broadway plugin) is 100% client
> side, either via the standalone "xrx" tool, the libxrx.so plugin or the
> libxrxnest.so plugin (which embeds Xnest (per IRC comments from ajax
> this was the part which led him to the conclusion that XRX is a seperate
> DDX) for the case that the main Xserver is inaccessible (firewall
> blocking, Xserver started with -nolisten tcp or a paranoid user doesn't
> even trust clients running as "untrusted" X clients))

I'm glad to hear this explanation repeated here in e-mail so that others
understand that this is not going to be a problem with modularization.

> [snip] 
> >    Note that some platforms only have 1 or 2 component revisions for
> >    their libraries. On these systems, either the second component (y)
> >    could be bumped, or symbol versioning could be used by vendors who
> >    package the libraries.
> > 
> > 
> > 
> >    By default, all libraries are built as shared objects; however, with
> >    other build options, static, profiled and debug versions can be built.
> 
> What about libraries which "unstable" APIs which are currently build as
> "static" versions for that reason (XprintUtils comes in mind from my
> side) ? And what about helper libraries which are currently used to
> build the Xserver ?

It would be nice if libraries were all built as shared objects; however,
modularization and autotooling do not preclude building static versions
of libraries, if they are needed.

> >      * As noted above, some DDXs require different built options, which
> >        implies that not all DDXs will be able to use the same dix, os,
> >        mi, etc. compile.
> 
> Why is this a problem for the modular build ? Right now all servers
> (Xorg, Xnest, Xdmx, Xprt, Xvfb) can be build in one "make World" step.
> The same should be possible in the "modular" world.
> 
> >        However, they can and should share the same
> >        source code. For example, the kdrive server does not support
> >        XINPUT whereas the Xorg server does, so dix (and other subdirs)
> >        will need to be built differently for each server. Note that these
> >        conflicts can be detected at configure time.
> 
> See above. There should be no such seperation - the "monolithic" Xorg
> tree didn't need it so why should the modular build system introduce
> such a pain (currently stub functions and *.a files are used to avoid
> this and the same could be done for the modular build) ?

We are being forward looking: There will be DDXs that require different
build options in the future.  Currently, the DDXs in the monolithic tree
all use the same build options, so it is reasonable to expect that the
initial modular tree will also be able to build those servers with the
same build options.

However, there are already examples of X servers that don't support all
of the same extensions (e.g., kdrive -- see the quote from the proposal
above).  When these DDXs are added to the tree, they will need different
build options.  By being forward looking and being flexible, we can
accommodate these new DDXs in the future with minimal effort.

> >      * The basic tree structure in the xserver module will look similar
> >        to the one used in the monolithic tree under xc/programs/Xserver.
> >        Notable exceptions include the drivers that will be moved to their
> >        own module (see below), and support programs that could be moved
> >        to the app module.
> > 
> >    The XFree86 drivers module: xfree86-driver
> 
> Why can't it be named "drivers" (yes, I know, "Xfree86" should be
> referenced but that can be done in the docs) ? I don't like to have
> giant-overlong-pathnames and the drivers/ subdir will likely host other
> kinds of drivers like the kdrive, Sun DDX or the Xprint ones...

Currently, the plan is to not have a hierarchy under xfree86-driver so
that each driver can be built independently, and the interfaces for this
module are the XFree86 XAA and input driver interfaces.  However, we
could expand this module to include other interfaces and even have a
driver hierarchy.  What do other people think about this idea?  I don't
have strong feelings here.

> And print drivers as the Xprint DDXs should be turned into loadable
> modules (in the long term the Xprt server is going away then[1],
> replaced by a Xorg server which can handle both video and print modules
> similar to what the Xserver in HP/UX can do today).
> 
> [1]=or more correctly: Xprt isn't going away as it's still needed as
> all-in-one X print server, however sepcial-purpose DDX like the SVGprint
> DDX may only be available as loadable module.

Interesting idea.  Have you written a design document yet?  These ideas
should be discussed in the Architecture Working Group.