Modularization development notes [was Re: RFA sent to the ArchWG]

[Kevin E Martin]

On Mon, Apr 04, 2005 at 05:11:39PM -0400, Kevin E Martin wrote:
> Moving forward, we should start discussing the development plan here

Well, I got very busy last week, but finally found time to work on the
devel plan today.  Here are my current thoughts.  I have marked a few
places with [***] where I specifically would like to have feedback, but
feel free to comment on anything.


Platform support
----------------

We need to find out who will be working on modularization so that we can
get an idea of the platform coverage.  For the 6.8 release, I created a
wiki page for the platforms that we wanted to test, and I would like to
do the same for modularization.  The list of those platforms that people
signed up to test was:

- AIX (no arch listed)
- Cygwin (IA-32)
- FreeBSD (Alpha, AMD64, IA-32, IA-64)
- HP-UX (PA-RISC)
- Linux (Alpha, AMD64, ARM, IA-32, IA-64, M68k, MIPS, PPC, PPC64,
  PA-RISC, S/390, Sparc)
- MacOS (PPC)
- MacOS/Darwin (IA-32, PPC)
- OpenBSD (AMD64, IA-32)
- Solaris (IA-32, Sparc)

Other platforms that were listed, but didn't have anyone signed up to
test were:

- IRIX
- LynxOS
- NetBSD
- OS/2
- SCO

Ideally, we would like to see at least one person and preferably more
signed up to work on each of these platforms; however, it is more likely
that we will get fairly good coverage on a subset of them.  I expect
good coverage on Linux, but we need much more than that.  So, if you are
interested in working on a platform and have the hardware available,
please speak up now so that we can get an idea of where we stand. [***]


Module creation
---------------

Once the Architecture Working Group gives their approval, we can begin
work, but before we jump in, we should make sure that we are all on the
same page.  As I see it, there are two parts to this project.  The first
involves dividing the tree into separate modules, which I describe next.
The second involves autotooling those modules (see the next section).

Here are the steps that I see for creating the modules:

1. Determine where each piece of code from the monolithic tree will go
   in the modular tree.  I've started this process, but it's not quite
   ready yet and I didn't want that list to hold up this message.  I'll
   forward it to the list a little later.
2. Create the top level modules (i.e., app, lib, proto, xserver, driver,
   font, doc and util) as peers to xc.
3. Create the directory hierarchy for the modularization within each
   module based on the breakdown determined in step #1.
4. Write a script that creates a symlink tree on the developer's local
   system from a checked out monolithic CVS tree.

As I mentioned above, I'm working on #1.  Steps #2 and #3 should fall
directly out of #1.  I've heard from another developer that they have
starting working on #4.


Autotooling the modules
-----------------------

We have a few people that have worked on previous autotooling efforts
and many more that have not.  Based on feedback from those previous
efforts, much of the work will become routine once everyone understands
exactly what needs to be done and how to do it.  The question is how to
fill this knowledge gap so that everyone can be productive initially
(i.e., how do we bootstrap ourselves).

My initial thought is that we should learn from the past efforts by
having those involved explain the autotooling process that they used.
We can then discuss it here on the list, make suggestions and come to
consensus on how we want to proceed for our project.  By discussing it
openly, I think that many more developers will be able to become
involved at an earlier stage since they will understand what is involved
and how to make it happen.  Also, this discussion will form the basis
for part of the guide described in the proposal (see documents section
below for more info).

What I have in mind here is a simple brain dump of the process used to
autotool in the xapps, xlibs, xserver and Debrix projects.  I'd also
like to see us discuss any common conventions that everyone should
follow so that we have a consistent build system and config options name
space.

Other comments/suggestions? [***]


Dependencies
------------

Note that while there are some dependencies that we need to follow for
certain modules to work (lib, xserver, driver), others can be developed
independently by using the monolithic tree for bootstrapping (app, font,
doc, util).  Here's how I currently see the dependencies:

- proto: no dependencies
- lib: depends on proto
- xserver: depends on proto and lib, but the lib dependency can be
  bootstrapped from the monolithic tree
- driver: depends on the xserver module headers
- app: depends on lib, but can be bootstrapped
- font: depends on app, which will contain the tools that build fonts,
  but can be bootstrapped
- doc: depends on (external?) tools to build docs
- util: no dependencies?

In order for bootstrapping to work, we need to have the capability to
install and use missing pieces from a monolithic build.  That build may
be installed in a non-standard place.  For example, I commonly set
ProjectRoot to be /tmp/Xorg-test.  During the modular development, we
should make sure that we can set the prefix to install there as well as
resolve dependencies from that location.


Documents
---------

In the Modularization Proposal, we committed to create the following
documents:

- Transition/Modular Tree Developer's Guide (see note below):
  - Building and using the new modular tree
  - Developing within the modular tree
  - Converting code from Imake to Autotools
- Component dependencies (part of guide?)
- Standardized configuration options (part of guide?)
- Font licenses

I think that the "Transition Guide" is misnamed in the proposal and it
should probably be called the "Modular Tree Developer's Guide" instead.
Using "transition" implies that the guide is only useful during the
transition period, which I think does not reflect the life of this
document.  I expect this doc to be the place where new developers go to
find out how to use and develop within the modular tree.

We should write the initial outlines for these docs before development
begins.  Then, during development, we will fill in the majority of the
information contained within them.  If someone would like to volunteer
to get these docs started (e.g., as wiki pages that everyone can update
as new info is added), that would be incredibly helpful [***].  They do
not need to be anything elaborate, just something that is easy to update
when new information is available.

One particular document that would be very good to start writing is the
"how to convert code from Imake to Autotools" section of the developers
guide.  This section will set the standards for how we autotool the tree
and will help developers that are new to autotools understand what needs
to be done and how to do it.  I think some of this could fall out of the
initial development discussions mentioned above.


Success Criteria
----------------

Ideally, we should be able to build the exact same binaries and install
them in the exact same place from either the monolithic and modular
trees on the same system, but in reality the compiler options will
probably differ and produce slightly different code.

The next best solution would be to run through the same set of tests on
both installed trees.  I think that the testing procedure used in for
the 6.8 release would be a good starting point for the set of tests used
for the combined 6.9/7.0 release.  Here's a link describing the
procedure:

    http://wiki.x.org/wiki/XorgTesting

In addition, we should make sure that the installed trees look very
similar and provide reasons for any discrepancies.

Other success criteria would be the coverage we get on the platforms we
support in the initial release.  As noted above, Linux will most likely
be the best supported, but we need others to make this a successful
project.  I agree with ajax that we should support at least Linux,
Solaris, one of the BSDs, Darwin and Cygwin initially since we have
active participation by people working on these platforms.  However, my
hope is that we will have several others supported as well.