Diagnosing first vs subsequent performance

[Lloyd Brown]

Hi, all. 

I hope this isn't too dumb of a question, but I'm having trouble finding
anything on it so far.  Not sure if my google-fu is just not up to the
task today, or if it's genuinely an obscure problem.

I'm in the middle of setting up an HPC node with 2 NVIDIA Tesla K80s (4
total GPUs), for some remote rendering tasks via VirtualGL.  But I've
got some strange behavior I can't yet account for, and I'm hoping
someone can point me in the right direction for diagnosing it.

In short, for accounting reasons, we'd prefer to have each GPU be
attached to a separate Xorg PID.  So I've built some very simple
xorg.conf files (example attached), and I can launch Xorg instances with
a simple syntax like this:

> Xorg :0 -config /etc/X11/xorg.conf.gpu0

When I run my tests, I'm also watching the output of "nvidia-smi" so I
can see which Xorg and application PIDs, are using which GPUs.

The first time I do something like "DISPLAY=:0.0 glxgears", I do *not*
see that process (eg. glxgears) show up in the output of "nvidia-smi",
and I see performance numbers consistent with CPU-based rendering.  If I
cancel (Ctrl-C), and run the exact same command again, I *do* see the
process in the output of "nvidia-smi", on the correct GPU, and I see
faster performance numbers consistent with GPU rendering.

If I switch to a different display (eg "DISPLAY=:3.0"), I see the same
behavior: slow the first time, fast on 2nd and subsequent instances. 
The same behavior even repeats when I switch back to a previously-used,
but not most-recently-used, DISPLAY.

I see similar behavior with other benchmarks (eg. glxspheres64,
glmark2): slow first time on a display, faster after that.

I have a sneaking suspicion that I'm just doing something really stupid
with my configs, but right now I can't find it.  I don't see anything
relevant in the Xorg.log files, or stdout/stderr from the servers, but I
can post those too, if needed.

Any pointers where to go from here, would be appreciated.

Thanks,
Lloyd


Other (possibly relevant) Info:
OS Release: RHEL 6.6
Kernel: 2.6.32-504.16.2.el6.x86_64
Xorg server 1.10.4 (from RHEL RPM)
NVIDIA Driver 352.55

Note: The attached example is for only one GPU.  The others configs are
exactly the same, with the exception of the PCI BusID, inside the GPU
device section.  I can verify via nvidia-smi, that the separate Xorg
PIDs are attached to the correct GPUs.


-- 
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu

-------------- next part --------------

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
    FontPath        "/usr/share/fonts/default/Type1"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/input/mice"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from data in "/etc/sysconfig/keyboard"
    Identifier     "Keyboard0"
    Driver         "kbd"
    Option         "XkbLayout" "us"
    Option         "XkbModel" "pc105"
EndSection


Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
    HorizSync       28.0 - 33.0
    VertRefresh     43.0 - 72.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "GPU0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:6:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "GPU0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "UseDisplayDevice" "none"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection