[PATCH RFC]: Generic Linux multi-domain PCI handling

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David S. Miller wrote:
> I've been plagued with having a sparc64 machine which is multi-domain
> PCI and my video card is on domain 1.  Instead of going off and
> forking the linuxPci.c code like was done for Alpha I tried to do
> something that would work on PPC and other non-sparc64 platforms.

Ironically, the first part of the year for me is going to be occupied
working on some similar stuff.  There are some pretty significant
problems with 32-bit X-servers on 64-bit pSeries hardware.  In addition
to non-zero PCI domains, devices in domain 0 get mapped above
0x7fffffff.  This causes problems when doing an mmap of /dev/mem in
32-bit mode. :(

> One thing of note is that it might be a good idea to dynamically
> allocate the pciBusInfo[] array after ARCH_PCI_INIT() has run
> and pciMaxBusNum has it's final stable value.  With even moderate
> domain values like 128 or 256, this static table gets a bit
> large.
> 
> Anyways, the general idea is to run over the dirents under
> /proc/bus/pci/ and if we find colon encoded directory names
> of the form "%x:%x" we use that to determine which domains
> exist on the machine.  This is implemented in the function
> linuxPci.c:probe_domains().
> 
> For each domain found, we allocate a pciBusInfo_t and stick that into
> the pciBusInfo[] entry for bus zero on the found domain.
> 
> If we don't find any domains, that's ok, we just revert to existing
> behavior which is to assign &linuxPci0 to pciBusInfo[0] and set
> pciNumBuses to 1.
> 
> I then set pciMaxBusNum to whatever we calculated pciNumBuses to
> be, otherwise Pci.c goes absolutely bananas probing domains that
> don't exist and results in an enormous number of PCI config space
> accesses to every possible bus on every non-existent domain, which
> makes the X server sit for a few seconds chugging and doing
> nothing but stat() and open() calls. :-)
> 
> The final part of this change is to make linuxPciOpenFile() open
> domain'd files correctly.  Instead of "0000" we put the actual
> domain number there in the file path string, and we track the
> domain of the cached file descriptor so that still works correctly
> as well.

This is similar to the approach that I was going to take.  However, I
was going to go a bit further.  Rather than using /proc, I was going to
add a new version of pciFindNext that iterated over the entries in
/sys/bus/pci/devices.  That way only devices that actually exist get
iterated.  The guess-and-check method used in the existing pciFindNext
implementations has always bothered me.  Of course, if /sys doesn't
exist it would fall back to the older method.

The second part would be to wrap the existing PCI structures and enums
(e.g., PCITAG) with a slightly higher-level structure.  Rather than
mapping by offset, drivers really want to say, "Map this resource of
this device."  Since, on 64-bit platforms, /sys has a separate file for
each BAR, we can mmap that file with a zero offset instead of mapping
/dev/mem with a > 32-bit offset.

I haven't worked through all (or even most) of the details, so this is
still primarilly hand waving.  I have some experiments to try and some
things to research first.  In the mean time, I thought I'd throw this
out there.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFDurbJX1gOwKyEAw8RAo+gAJ9feBDCkeIitqdLNL2uVuMWeEAREQCglEis
zge301ngs2BBjguREJe5pCw=
=fdgg
-----END PGP SIGNATURE-----