PCI rework

[Mark Kettenis]

> Date: Mon, 1 May 2006 09:17:22 +1000
> From: "Dave Airlie" <airlied at gmail.com>
> 
> > > On Sunday, April 30, 2006 5:22 am, Mark Kettenis wrote:
> > > > Matthieu Herb, asked me to have a look at libpciaccess some time ago.
> > > > My conclusion then was that it is too tightly bound to Linux.  It
> > > > takes the Linux sysfs pci access interface as a model instead of the
> > > > actual pci hardware.  I especially feel the support to map memory
> > > > associated with the PCI BAR's does not belong in the library.  It is
> > > > not impossible to implement the libpciaccess backend for other
> > > > operating systems, but it would be rather awkward.
> > >
> 
> Well I've been taking the time recently to try and salvage the current
> X PCI interfaces and really its a quite large unholy mess.. and I've
> just added to it by fixing the ROMs for Linux....

Well, I'm mostly talking about keeping the current interfaces, not the
code that implements them (although some of it might be reusable).
This removes the need to convert all the drivers, at least the ones
that don't go behind the back of the public interfaces.

I'm not opposing the idea that the linux-specific code should use
sysfs for PCI access at all.  I'm just opposing adopting a flawed
interface, and questioning the gratuitous interface changes.

> > > X already assumes it can mmap memory BARs, do you think it needs to
> > > change too?  It seems to me that the interface provided by libpciaccess
> > > is fairly flexible, easy to use, and easily ported to new platforms (it
> > > should be, it's based on the Linux internal PCI interfaces which have
> > > prooven highly portable).  Do you have other examples where you think
> > > Linux specific implementation details have crept into the interface?
> >
> > I'm talking specifically about pci_device_map_region() and
> > pci_device_unmap_region().  These interfaces are clearly there because
> > the Linux sysfs provides these "regions" as files which you can open
> > and then mmap.  Other operating systems don't support this view, and
> > there's a good reason not to do it: Many PCI devices put mappable
> > memory addresses in config space outside the standard PCI BARs.  What
> > you really need is an interface to read from PCI config space and an
> > interface to map physicall addresses on the pci bus into memory.  Some
> > standard helper functions to decode the standard BARs is certainly
> > desirable.
> 
> Can you point me out some of those PCI devices, as I can't see how
> they can possibly work with any OS that does PCI, because there would
> be nothing stopping the OS from mapping something else in that area,
> granted I can see times when we might want to map 0xa0000 or 0xc0000
> areas for things, I've been trying to make it so I can mmap the
> resource files from sysfs for Linux recently, to be honest the
> interface should be dealing with BARs, PCI is about BARs, if there is
> another sneaky interface for direct memory mapping so be it, but from
> a device driver POV, it should be able to request a BAR resource
> mapping, if the underlying code has to fallback to direct mapping
> /dev/mem it should be easy, it can store the PCI space info to do the
> conversion.. I started implementing this in os-support, before I found
> out that os-support doesn't have enough info, common/xf86pciBus.c has
> it..

If the system has decent firmware it will assign the addresses, and if
it doesn't, the OS should take care of it (remember pulg & pray?).
We're talking physical memory addresses here; the OS can map them into
virtual memory anywhere it likes.  X really shouldn't be fiddling with
the BAR, and only decode them on a modern OS.  But apparently not all
OSes initialize stuff properly.

> Another reason the current code is horribly broken is it doesn't
> respect the E820 maps, I don't want X looking at E820 maps at all, but
> really it should if it is going to do think it can re-configure PCI
> addresses, on Linux the kernel does this, my goal is to remove all
> remapping code paths from X for Linux using systems, so it does
> nothing but read the PCI spaces. The kernel is adding code to enable
> devices from userspace so that X can do that first, which will sort
> out all the BARs etc....

Not all the world's Linux.

> > > Note that the prototype is different in that it takes a full pci info
> > > structure rather than just a tag.  This gives arch specific
> > > implementations more flexibility and eases porting.
> >
> > But a PCITAG is already opaque; there's no reason why you could extend
> > it to include the additional information you might need.
> 
> It's not, passing domain info around in the TAG isn't correct, I don't
> think there is enough bits to fill everything in a TAG correctly, I've
> got a patch from last year that rips out all the TAGs and just adds
> domain to the PciInfo...

As I said, don't think of PCITAG as a bus/device/function triplet, but
as an opaque data structure for a specific PCI device.  Then if
becomes PciInfo.

> > >   o the distinction between mapping domain and regular PCI memory is
> > >     arbitrary and should be removed
> >
> > I'm not quite sure what is actually meant by a "domain".  I'm
> > presuming it's similar to what the ACPI specification calls a
> > "segment": a completely seperate PCI bus hierarchy.  Yes, the way the
> > current interface handles that is awkward, but it should be easy to
> > fix this if you add the domain to the PCITAG.
> 
> Been done, I personally think we don't have enough bits to fit
> potential new crazy ass machines with lots of buses...

PCITAG doesn't have to be an unsigned long.

Mark