Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Pekka Paalanen]

On Mon, 4 Mar 2024 13:24:56 +0000
Terry Barnaby <terry1 at beam.ltd.uk> wrote:

> On 04/03/2024 09:41, Pekka Paalanen wrote:
> > On Mon, 4 Mar 2024 08:12:10 +0000
> > Terry Barnaby <terry1 at beam.ltd.uk> wrote:
> >  
> >> While I am trying to investigate my issue in the QtWayland arena via the
> >> Qt Jira Bug system, I thought I would try taking Qt out of the equation
> >> to simplify the application a bit more to try and gain some
> >> understanding of what is going on and how this should all work.
> >>
> >> So I have created a pure GStreamer/Wayland/Weston application to test
> >> out how this should work. This is at:
> >> https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz
> >>
> >> This tries to implement a C++ Widget style application using native
> >> Wayland. It is rough and could easily be doing things wrong wrt Wayland.
> >> However it does work to a reasonable degree.
> >>
> >> However, I appear to see the same sort of issue I see with my Qt based
> >> system in that when a subsurface of a subsurface is used, the Gstreamer
> >> video is not seen.
> >>
> >> This example normally (UseWidgetTop=0) has a top level xdg_toplevel
> >> desktop surface (Gui), a subsurface to that (Video) and then waylandsink
> >> creates a subsurface to that which it sets to de-sync mode.
> >>
> >> When this example is run with UseWidgetTop=0 the video frames from
> >> gstreamer are only shown shown when the top subsurface is manually
> >> committed with gvideo->update() every second, otherwise the video
> >> pipeline is stalled.  
> > This is intentional. From wl_subsurface specification:
> >
> >        Even if a sub-surface is in desynchronized mode, it will behave as
> >        in synchronized mode, if its parent surface behaves as in
> >        synchronized mode. This rule is applied recursively throughout the
> >        tree of surfaces. This means, that one can set a sub-surface into
> >        synchronized mode, and then assume that all its child and grand-child
> >        sub-surfaces are synchronized, too, without explicitly setting them.
> >
> > This is derived from the design decision that a wl_surface and its
> > immediate sub-surfaces form a seamlessly integrated unit that works
> > like a single wl_surface without sub-surfaces would. wl_subsurface
> > state is state in the sub-surface's parent, so that the parent controls
> > everything as if there was just a single wl_surface. If the parent sets
> > its sub-surface as desynchronized, it explicitly gives the sub-surface
> > the permission to update on screen regardless of the parent's updates.
> > When the sub-surface is in synchronized mode, the parent surface wants
> > to be updated in sync with the sub-surface in an atomic fashion.
> >
> > When your surface stack looks like:
> >
> > - main surface A, top-level, root surface (implicitly desynchronized)
> >    - sub-surface B, synchronized
> >      - sub-surface C, desynchronized
> >
> > Updates to surface C are immediately made part of surface B, because
> > surface C is in desynchronized mode. If B was the root surface, all C
> > updates would simply go through.
> >
> > However, surface B is a part of surface A, and surface B is in
> > synchronized mode. This means that the client wants surface A updates to
> > be explicit and atomic. Nothing must change on screen until A is
> > explicitly committed itself. So any update to surface B requires a
> > commit on surface A to become visible. Surface C does not get to
> > override the atomicity requirement of surface A updates.
> >
> > This has been designed so that software component A can control surface
> > A, and delegate a part of surface A to component B which happens to the
> > using a sub-surface: surface B. If surface B parts are further
> > delegated to another component C, then component A can still be sure
> > that nothing updates on surface A until it says so. Component A sets
> > surface B to synchronized to ensure that.
> >
> > That's the rationale behind the Wayland design.
> >
> >
> > Thanks,
> > pq  
> 
> Ah, thanks for the info, that may be why this is not working even in Qt 
> then.
> 
> This seems a dropoff in Wayland to me. If a software module wants to 
> display Video into an area on the screen at its own rate, setting that 
> surface to de-synced mode is no use in the general case with this 
> policy.

It is of use, if you don't have unnecessary sub-surfaces in synchronized
mode in between, or you set all those extra sub-surfaces to
desynchronized as well.

> I would have thought that if a subsurface was explicitly set to 
> de-synced mode then that would be honoured. I can't see a usage case for 
> it to be ignored and its commits synchronised up the tree ?

Resizing the window is the main use case.

In order to resize surface A, you also need to resize and paint surface
B, and for surface B you also need to resize and paint surface C. Then
you need to guarantee that all the updates from surface C, B and A are
applied atomically on screen.

Either you have component APIs good enough to negotiate the
stop-resize-paint-resume on your own, or if the sub-components are
free-running regardless of frame callbacks, component A can just
temporarily set surface B to synchronized, resize and reposition it,
and resume.

> So is there a way to actually display Video on a subsurface many levels 
> deep in a surface hierarchy, would setting all of the surfaces up to the 
> subsurface just below the desktop top level one work (although not ideal 
> as it would mean overriding other software modules surfaces at the 
> Wayland level) ?

Setting to what?

> Or can desynced subsurfaces really only work to one level deep ?

You can set your middle sub-surface to desynchronized, too, at least
from Wayland perspective. I don't know if Qt let's you.

There are implementation issues with nested sub-surfaces in Weston, but
this does not seem to be one of them.

> If it is just one subsurface level deep that video can be displayed, I 
> guess I will have to get GStreamers waylandsink to create its subsurface 
> off the top most surface and add calls to manage its surface from my 
> app.

That should have been the first idea to try.

Is Gst waylandsink API the kind that it internally creates a new
wl_surface for itself and makes it a sub-surface of the given surface,
or is there an option to tell Gst to just push frames into a given
wl_surface?

If the former, then waylandsink is supposed to somehow give you an API
to set the sub-surface position and z-order wrt. its parent and
siblings. If the latter, you would create wl_subsurface yourself and
keep control of it to set the sub-surface position and z-order.

Either way, the optimal result is one top-level wl_surface, with one
sub wl_surface drawn by Gst, and no surfaces in between in the
hierarchy.

> Or maybe get waylandsinks subsurface and manipulate it behind 
> waylandsinks back. Not sure what this will do to the Qt level though. 
> Using the QWidgets subsurface as its base should have allowed isolation 
> to a degree between the Qt and wayland sink sub systems, its a much more 
> modular approach.

You cannot punt between-components integration to Wayland. The
sub-surfaces design tried and failed, and as a result the sub-surface
protocol is both complex and insufficient. You still need explicit
communication between your client-side components when you resize.

> There is no documentation on this Wayland restriction in waylandsink or 
> other places, I can try and feed this back.
> 
> Oh for X11 !

Yeah, the blink of garbage when windows are resized, or the blue
colorkey following far behind when a window is moved.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240304/3e2e1021/attachment.sig>