From andreas.schmid at ur.de Tue Nov 23 16:42:07 2021 From: andreas.schmid at ur.de (Andreas Schmid) Date: Tue, 23 Nov 2021 17:42:07 +0100 Subject: Looking for low-latency method to read pixels from X framebuffer Message-ID: Dear Xorg developers, I am trying to build an application that can measure the latency/processing time of graphics frameworks on Linux. My idea is, for a first simple use case, to implement test programs gradually changing the screen's color while logging timestamps using different graphics and UI frameworks (e.g. SDL, OpenGL, Qt, ...). More complex test programs rendering user interfaces or 3D scenes will then be implemented in the future. A separate measuring program should detect a pixel being updated in some kind of framebuffer (as close to the application as possible) and log another timestamp. This way, the actual rendering time of the test application should be measured. The measuring program should be independent of vblank events, as I'm interested in the time the rendering is done, not the time users might be able to see something. My problem is getting the framebuffer content with the second program. As those measurements are part of a research project, I want to make sure to measure as exactly as possible. Things I already tried: 1. Getting framebuffer content with fbdev or libdrm. The problem here is that both require the programs to be run in a terminal without an active XServer (which I would prefer due to external validity). Additionally, fbdev is deprecated and accessing a libdrm framebuffer with mmap seems to be supported by embedded graphics chips only. 2. Using MIT-XShm (reading a pixel with XShmGetImage) XShm looks promising but I have not found any information regarding its performance. It also looks like XShm gets blocked for a while when another application (in my case the test application) is drawing something. I assume this is due to a "LockDisplay()" call in XShmGetImage. Additionally, XShm behaves very strangely (reading out old color values) when the test application is rendering as fast as it can. Adding a delay of at least 10 milliseconds between every update of the test application seems to fix this issue. This behavior also occurs, when a compositor (in my case compton) is running. 3. Using the Xdamage extension Xdamage also looks very promising as its notifications arrive quite soon (about one millisecond) after the test application finishes rendering. However, this way it is obviously not possible to read a pixel's color, so I can't tell if the rendering is already finished when the XDamageNotify event arrives. For better understanding, a visualization of my current measuring setup can be found here: https://hci.ur.de/framework_latency_problem My questions are: a) What is, in your opinion, the best time to stop the measurement timer? b) Are there other ways besides XShm and Xdamage to detect a pixel changing its color? c) Where does the strange delay of XShm come from? d) Is the XDamageNotify event triggered before or after pixels are updated? Kind regards, Andreas From raster at rasterman.com Thu Nov 25 11:21:34 2021 From: raster at rasterman.com (Carsten Haitzler) Date: Thu, 25 Nov 2021 11:21:34 +0000 Subject: Looking for low-latency method to read pixels from X framebuffer In-Reply-To: References: Message-ID: <20211125112134.74f25bed0ff2dd48e9cc5a7d@rasterman.com> On Tue, 23 Nov 2021 17:42:07 +0100 Andreas Schmid said: > Dear Xorg developers, > > I am trying to build an application that can measure the > latency/processing time of graphics frameworks on Linux. > My idea is, for a first simple use case, to implement test programs > gradually changing the screen's color while logging timestamps using > different graphics and UI frameworks (e.g. SDL, OpenGL, Qt, ...). More > complex test programs rendering user interfaces or 3D scenes will then > be implemented in the future. > A separate measuring program should detect a pixel being updated in some > kind of framebuffer (as close to the application as possible) and log > another timestamp. This way, the actual rendering time of the test > application should be measured. > The measuring program should be independent of vblank events, as I'm > interested in the time the rendering is done, not the time users might > be able to see something. > > My problem is getting the framebuffer content with the second program. > As those measurements are part of a research project, I want to make > sure to measure as exactly as possible. This is a problem... some toolkits (EFL) literally sync their rendering to vblank events... so your measurements will be unrealistic as the toolkit is explicitly delaying updates to gather multiple updates together in one batch and deferring it until next vblank. (or variations of this technique like deferring until half-vblank point assuming a render budget of < 1/2 a frame thus producing a frame in time for the next vblank). if your goal is to measure a toolkit and how long it takes to handle updates and then render you will want to be doing this internally inside a toolkit as they may all do very different things. other toolkits or apps that render with a locked vsync with gl (eg an eglswapinterval of 1) will also not realistically tell you the rendering time and latency of the toolkit itself but really only of the policy of syncing to vblank. :) > Things I already tried: > > 1. Getting framebuffer content with fbdev or libdrm. > The problem here is that both require the programs to be run in a > terminal without an active XServer (which I would prefer due to external > validity). Additionally, fbdev is deprecated and accessing a libdrm > framebuffer with mmap seems to be supported by embedded graphics chips only. That's not true - drm/kms are supported by virtually all drivers. Desktop GPUs too. Also reading back from the framebuffer would mean you also capture compositor latency if you have a compositor in your xserver too. > 2. Using MIT-XShm (reading a pixel with XShmGetImage) > XShm looks promising but I have not found any information regarding its > performance. It also looks like XShm gets blocked for a while when > another application (in my case the test application) is drawing > something. I assume this is due to a "LockDisplay()" call in XShmGetImage. > Additionally, XShm behaves very strangely (reading out old color values) > when the test application is rendering as fast as it can. Adding a delay > of at least 10 milliseconds between every update of the test application > seems to fix this issue. This behavior also occurs, when a compositor > (in my case compton) is running. xshm is how you read back pixels in x to software so you can look at them. it's the fastest way to doit and what you should possibly use (unless you are a compositor - enlightenment cheats for screenshots and reads back from its own canvas before it gets to x at all to do screenshots - either by just directly exposing memory of a render buffer in the client side or by gl read pixels from the backbuffer). this avoids rendering then reading it back from the xserver later on. > 3. Using the Xdamage extension > Xdamage also looks very promising as its notifications arrive quite soon > (about one millisecond) after the test application finishes rendering. > However, this way it is obviously not possible to read a pixel's color, > so I can't tell if the rendering is already finished when the > XDamageNotify event arrives. xdamage doesn't give you pixels. it gives you areas that were updated as rectangles as events. if you are happy just getting a "some rectangle(s)were updates and here they are" then this is the extension to use. it's going to be much lower impact. > For better understanding, a visualization of my current measuring setup > can be found here: https://hci.ur.de/framework_latency_problem > > My questions are: > a) What is, in your opinion, the best time to stop the measurement timer? > b) Are there other ways besides XShm and Xdamage to detect a pixel > changing its color? > c) Where does the strange delay of XShm come from? > d) Is the XDamageNotify event triggered before or after pixels are updated? > > Kind regards, > Andreas > -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- Carsten Haitzler - raster at rasterman.com