ABGR Framebuffer compatibility with GLX and RENDER layers?

[Stirling Westrup]

I'm currently writing a X video driver for a USB-based video display
device, targeted (for now) at Xorg 1.7.6.

Because the display is connected over a USB cable, we set up a
framebuffer in our driver, and register it with the Damage layer.
Whenever the framebuffer is damaged, we compress the damaged region
and pipe it across the usb to the display device. We have all this
working.

The difficulty comes in that we want to do as little manipulation of
the framebuffer as possible, so we would like it to have the same
layout as that used by the display device, which is either b5g6r5 (in
16-bit depth) or a8b8g8r8 (in 24-bit depth). These are the only two
display formats the device supports, and the only two bit depths we
support.

Now, initially I assumed I would only need to define a single visual
for the current depth, as there is only one desired display mode, but
then libx11 refuses to talk to the render layer and makes, for
example, hardware ARGB cursors inoperative. If I define a set of bogus
visuals (that we could never display) for depths 1,4,8,24 and 32, then
the render layer is happy and hardware cursors work. However, if I do
that, then GLX stops working correctly, as it chooses to use a bogus
visual when in 16-bit depth.

So far I have yet to find a set of visuals that I can set up that
results in both the render and the glx layers working correctly. This
is exacerbated by the fact that its FBScreenInit that does the main
visual setup, and there seems very little control over how it does so.

So far I've tried:
1) default visuals - render and glx work fine, but B and R are
reversed on screen.
2) correct visual only for current bit depth - glx works, but render
doesn't. Colors are correct.
3) correct visual for current bit depth, fake visuals for all other
depths. - render works, but glx chooses a fake visual at 16-bit depth
and displays wrong.

I am nearing my wits end with this. Does anyone have some clues to
throw my way as to how to get this all working?


-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com
http://sourceforge.net/users/stirlingwestrup