[PATCH xserver 4/4] xfree86: promote one GPU screen if (NumScreens == 0 && NumGPUScreens > 0)

Tue Sep 6 11:47:57 UTC 2016

Adding Matt and Ard

On 09/06/16 00:32, Dave Airlie wrote:
> On 5 September 2016 at 22:39, Laszlo Ersek <lersek at redhat.com> wrote:
>> On 09/04/16 11:01, Hans de Goede wrote:
>>> HI,
>>>
>>> On 04-09-16 03:11, Laszlo Ersek wrote:
>>>> Aarch64/KVM virtual machines cannot use emulated graphics cards with
>>>> linear framebuffers, due to architectural cache coherency issues. (For
>>>> this reason, "qemu-system-aarch64" doesn't even include the "virtio-vga"
>>>> device model.)
>>>>
>>>> Such guests can use the "virtio-gpu-pci" device well, through the
>>>> "modesetting" driver. However, given that "virtio-gpu-pci" is never
>>>> recognized as a primary graphics card (which is otherwise correct,
>>>> generally speaking),
>>>
>>> Why is this otherwise correct, generally speaking ? AFAIK virtio-gpu-pci
>>> is needed for Virgil, so I would expect it to become the default graphical
>>> card in vms in the future, or is that what virtio-vga is for ?
>>
>> Yes. Both virtio-vga and virtio-gpu-pci share the VirtIO GPU Device
>> part, but only the former has the VGA compat stuff (incl. the legacy
>> framebuffer), and only the former qualifies as "primary graphics card"
>> at the moment.
> 
> Primary means the card the firmware/BIOS shows up on, and I understand
> in ARM world that probably doesn't mean much as the firmware/BIOS might
> not show up on any.

Actually, in the UEFI world (regardless of architecture), it gets a bit
more complex. The EFI_SYSTEM_TABLE has the following fields:

  ConsoleOutHandle  The handle for the active console output device.
                    This handle must support the
                    EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL.

  ConOut            A pointer to the EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL
                    interface that is associated with ConsoleOutHandle.

When a UEFI application (like a boot loader) wants to output text, it
uses the above Simple Text Out protocol instance. That protocol might be
implemented by a driver stack that ultimately multiplexes the console
output to (say) a serial console and a number of different graphics
cards (internally rendering the text to graphics). You would see grub2
show up on said devices.

For example, when I test GPU assignment with VFIO, I can simultaneously
see the firmware display on thee outputs: (a) on the serial console of
the virtual machine, (b) on the emulated graphics console of the virtual
machine (via the emulated QXL card), (c) on the physical monitor that I
connect to the NVIDIA GTX750 that I assign to the guest with VFIO.

So, "the card (singular) that the firmware/BIOS shows up on" is
ill-defined; UEFI shows up on as many cards (plural) as it has drivers
for *and* the platform's UEFI BDS (boot device selection) policy enables.

In edk2's ArmVirtQemu (arm/aarch64) and OvmfPkg (i386/x86_64) platforms,
we enable all PCI devices in the BDS policy whose class code (offset
0x0b) is Display Controller (0x03).

> On x86 the primary GPU is the one the BIOS inits first,

As far as UEFI is concerned, I believe you may be referring to kernel
commit 86fd887b7fe3, "vgaarb: Don't default exclusively to first video
device with mem+io". That is, the vga_set_default_device() call in
vga_arb_device_init().

However, that search loop won't apply to "vfio-gpu-pci" specifically,
because it looks for a framebuffer address, and "vfio-gpu-pci" has no
framebuffer. Furthermore, the code depends on CONFIG_X86 || CONFIG_IA64,
so it is not even built for aarch64.

Let's look at another place in the kernel that tries to find a "first"
display device. The source file "drivers/firmware/efi/libstub/gop.c"
looks relevant, see the functions setup_gop32() and setup_gop64(), and
the "first_gop" variable.

These functions seem to traverse the GOPs in the order the
LocateHandle() UEFI boot service returns the GOPs -- see
efi_setup_gop(). That order is unspecified, so whatever is picked as
"first GOP" is arbitrary.

I see there is some extra smarts in there, dating back to kernel commit
38cb5ef4473c6 ("X86: Improve GOP detection in the EFI boot stub")
originally: the loop searches for a handle that carries both a Graphics
Output Protocol instance (defined by the UEFI spec) and a
ConsoleOutDevice "protocol" instance (which is defined internally to edk2).

However, this "intersection" doesn't define the result uniquely. For
example, reaching back to my GPU assignment example above, if I go to
the UEFI shell, I can see two handles that carry *both* of these
interfaces -- handle A7 corresponds to the emulated QXL device, while
handle AD corresponds to the physical NVIDIA GTX750 card:

> Shell> dh -d -p GraphicsOutput

> A7: ConsoleOut SimpleTextOut GraphicsOutput(GraphicsOutput)
>  DevicePath(..0x2,0x0)/AcpiAdr(0x80010100))
>
>    Controller Name    : PciRoot(0x0)/Pci(0x2,0x0)/AcpiAdr(0x80010100)
>    Device Path        : PciRoot(0x0)/Pci(0x2,0x0)/AcpiAdr(0x80010100)
>    Controller Type    : BUS
>    Configuration      : NO
>    Diagnostics        : NO
>    Managed by         :
>      Drv[5D]          : Platform Console Management Driver
>      Drv[61]          : Console Splitter Driver
>      Drv[66]          : Graphics Console Driver
>    Parent Controllers :
>      Parent[96]       : QEMU Video PCI Adapter
>    Child Controllers  :
>      Child[64]        : Primary Console Output Device

> AD: ConsoleOut SimpleTextOut GraphicsOutput(GraphicsOutput)
>  EDIDActive(EDIDActive)
>  EDIDDiscovered(EDIDDiscovered)
>  DevicePath(..trl(0x0)/AcpiAdr(0x80010300))
>
>    Controller Name    : DFP0
>    Device Path        :
>                PciRoot(0x0)/Pci(0xB,0x0)/Ctrl(0x0)/AcpiAdr(0x80010300)
>    Controller Type    : BUS
>    Configuration      : NO
>    Diagnostics        : NO
>    Managed by         :
>      Drv[5D]          : Platform Console Management Driver
>      Drv[61]          : Console Splitter Driver
>      Drv[66]          : Graphics Console Driver
>    Parent Controllers :
>      Parent[AC]       : Display0
>    Child Controllers  :
>      Child[64]        : Primary Console Output Device

> [...]

Notice how both handles have "ConsoleOut" and "GraphicsOutput.

> not the onboard,
> you can usually pick in the BIOS what the GPU boot order is,

I believe you might be able to control, dependent on firmware, whether
any given Display Controller gets bound or not by its respective GOP
driver (in other words, whether BDS connects the device or not). You
might even be able to control what order BDS connects them in.

But, that has no specified effect on the order that the LocateHandle()
boot service returns the GOP-carrying handles in. Consequently, if at
least two handles satisfy the search criteria in Linux, whichever gets
picked first remains unpredictable.

> and Linux uses
> whatever is the VGA card as the primary to respect that.

I think it's not uniquely determined.

> 
> So on aarch64/arm we could have the kernel pick one GPU is there is only
> one and mark it as primary. I'm not sure if this would need device tree support
> or not etc.
> 
> In the absence of policy from the kernel, I'd rather not put the policy in the X
> server code itself.

Even if we put the policy in the kernel (that is, even if the kernel
marks a primary), it'd remain non-deterministic.

> As boot probe ordering could mean we'd pick different
> primaries on different boots if things raced which doesn't seem ideal.

Picking different primaries on different boots sounds perfectly
acceptable to me as long as:
- no explicit DeviceId.BusID is specified by the user,
- no primary emerges otherwise,
- there are several secondaries.

This behavior is clearly not ideal in the long term (like at the third,
fourth, fifth etc boots of the same machine), but the user can trivially
fix it *after* the installation finishes (by setting Device.BusID).
Plus, the suggested documentation (man page) and the server log are
clear about it.

Importantly, the suggested behavior (even if non-deterministic) is
clearly superior to the X Server not starting at all. "Output somewhere"
is better than no output at all. (And, in 99% of the cases, there's only
one secondary, which gives exactly the expected result.)

... If it made the series acceptable, I'd be happy to tweak patch #4 so
that the code get active only for

  (NumScreens == 0 && NumGPUScreens == 1)

> So I'd really prefer we try and solve this kernel side first, but I realise that
> might not be practical.

Yeah, not practical.

I named kernel commit 38cb5ef4473c6 above as one tweak for the "first
GOP" logic. Its predecessor was commit 291f36325f9f2 ("x86, efi: EFI
boot stub support"), which had tried to remedy the non-uniqueness of the
GOP by intersecting GOP presence with PciIo presence on the handle. It
had proved insufficient, hence commit 38cb5ef4473c6 got written, but as
I've showed above, that one is also insufficient for uniqueness (to
present day).

Ultimately my goal is to dump the non-determinism on the user's head,
because at that price they'll have output "somewhere", at first boot,
without configuring anything. I think convincing the kernel to take such
a policy (assuming I knew where to start with the code, which I don't)
would be harder than getting the same policy into xserver.

Hans, do you think we can carry this at least in Fedora? Perhaps
restricted to aarch64?

Thanks
Laszlo