Looking for low-latency method to read pixels from X framebuffer

[Carsten Haitzler]

On Tue, 23 Nov 2021 17:42:07 +0100 Andreas Schmid <andreas.schmid at ur.de> said:

> Dear Xorg developers,
> 
> I am trying to build an application that can measure the 
> latency/processing time of graphics frameworks on Linux.
> My idea is, for a first simple use case, to implement test programs 
> gradually changing the screen's color while logging timestamps using 
> different graphics and UI frameworks (e.g. SDL, OpenGL, Qt, ...). More 
> complex test programs rendering user interfaces or 3D scenes will then 
> be implemented in the future.
> A separate measuring program should detect a pixel being updated in some 
> kind of framebuffer (as close to the application as possible) and log 
> another timestamp. This way, the actual rendering time of the test 
> application should be measured.
> The measuring program should be independent of vblank events, as I'm 
> interested in the time the rendering is done, not the time users might 
> be able to see something.
> 
> My problem is getting the framebuffer content with the second program.
> As those measurements are part of a research project, I want to make 
> sure to measure as exactly as possible.

This is a problem... some toolkits (EFL) literally sync their rendering to
vblank events... so your measurements will be unrealistic as the toolkit is
explicitly delaying updates to gather multiple updates together in one batch and
deferring it until next vblank. (or variations of this technique like deferring
until half-vblank point assuming a render budget of < 1/2 a frame thus
producing a frame in time for the next vblank). if your goal is to measure a
toolkit and how long it takes to handle updates and then render you will want
to be doing this internally inside a toolkit as they may all do very different
things. other toolkits or apps that render with a locked vsync with gl (eg an
eglswapinterval of 1) will also not realistically tell you the rendering time
and latency of the toolkit itself but really only of the policy of syncing to
vblank. :)

> Things I already tried:
> 
> 1. Getting framebuffer content with fbdev or libdrm.
> The problem here is that both require the programs to be run in a 
> terminal without an active XServer (which I would prefer due to external 
> validity). Additionally, fbdev is deprecated and accessing a libdrm 
> framebuffer with mmap seems to be supported by embedded graphics chips only.

That's not true - drm/kms are supported by virtually all drivers. Desktop GPUs
too. Also reading back from the framebuffer would mean you also capture
compositor latency if you have a compositor in your xserver too.

> 2. Using MIT-XShm (reading a pixel with XShmGetImage)
> XShm looks promising but I have not found any information regarding its 
> performance. It also looks like XShm gets blocked for a while when 
> another application (in my case the test application) is drawing 
> something. I assume this is due to a "LockDisplay()" call in XShmGetImage.
> Additionally, XShm behaves very strangely (reading out old color values) 
> when the test application is rendering as fast as it can. Adding a delay 
> of at least 10 milliseconds between every update of the test application 
> seems to fix this issue. This behavior also occurs, when a compositor 
> (in my case compton) is running.

xshm is how you read back pixels in x to software so you can look at them. it's
the fastest way to doit and what you should possibly use (unless you are a
compositor - enlightenment cheats for screenshots and reads back from its own
canvas before it gets to x at all to do screenshots - either by just directly
exposing memory of a render buffer in the client side or by gl read pixels
from the backbuffer). this avoids rendering then reading it back from the
xserver later on.

> 3. Using the Xdamage extension
> Xdamage also looks very promising as its notifications arrive quite soon 
> (about one millisecond) after the test application finishes rendering. 
> However, this way it is obviously not possible to read a pixel's color, 
> so I can't tell if the rendering is already finished when the 
> XDamageNotify event arrives.

xdamage doesn't give you pixels. it gives you areas that were updated as
rectangles as events. if you are happy just getting a "some rectangle(s)were
updates and here they are" then this is the extension to use. it's going to be
much lower impact.

> For better understanding, a visualization of my current measuring setup 
> can be found here: https://hci.ur.de/framework_latency_problem
> 
> My questions are:
> a) What is, in your opinion, the best time to stop the measurement timer?
> b) Are there other ways besides XShm and Xdamage to detect a pixel 
> changing its color?
> c) Where does the strange delay of XShm come from?
> d) Is the XDamageNotify event triggered before or after pixels are updated?
> 
> Kind regards,
> Andreas
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - raster at rasterman.com