Multimedia support (Re: X Developers' Summit)

[Helge Bahmann]

Am Dienstag, 17. Juli 2007 00:43 schrieb Carsten Haitzler:
> On Mon, 16 Jul 2007 16:55:46 +0200 Helge Bahmann <hcb at chaoticmind.net> 
babbled:
> > yes, and frankly I fail to see what PulseAudio is doing fundamentally
> > different from mas when it comes to synchronisation. Not that it is
> > impossible, but it will get very very messy, and I can show that
> > integrating it into X to begin with neatly solves this problem.
>
> personally - i think going via x proto has merit. you have a point. the
> only problem now is that it becomes x's job to syncronise audio within
> itself (and you thought you got rid of the problem!). you basically shifted
> the problem from the app to x. that's fine - but it's not a silver bullet.
> you will probably now need to content with mixing audio in client-space

Actually not quite, mixing is performed in the server, but under the direction 
of a "mixer client" (somewhat analogous to the "compositing manager")

Basically the idea is as follows... the extension provides X requests to 
perform operations on samplebuffers within the server, for 
example "MultiplyAccumulate" which you would probably use for mixing. It is 
now the responsibility of the "mixer client" app to issue commands to properly 
mix all active streams into the master stream (and it may do whatever it 
chooses to do to perform the mixdown, including suppressing/dropping voices, 
or funny effects like panning audio as the window is moved across the screen)

Now, in principle the mixer client has to make sure that all mixing commands 
are executed "in time" by the server which -- unsurprisingly -- turns out to 
be quite problematic due to latency requirements. This is where the second 
extension kicks in -- it allows clients to issue X requests (a "sensible"
subset of all available requests) that are not executed immediately 
but "deferred" to a specific synchronisation point in time, e.g. 
synchronized to the audio device (unlike the XSync extension this mechanism 
does not prevent the client from issuing other requests while the deferred 
requests are pending). This allows the mixer to schedule enough mixing 
commands to overcome any client<->server communication latency, does not 
introduce audio "policy" into the server, but the actual mixdown can still be 
performed with low latency (in theory bounded only by the capabilities of the 
hardware and OS scheduling).

> (for multiple audio streams), mixing buffers (or audio stream priorities to
> block off audio from other clients), need to handle audio device buffer
> sizes for sync (and when you add a mixing buffer this really get a bit
> nasty when combined with needing to run basically realtime as skips in
> audio when you run out of enough cpu to do the mixing really sound horrible
> - dropping frames is almost heavenly bliss in comparison).

Yes you are right, audio poses a number of hard problems, and the fact that 
current xorg server is not exactly well-behaved when it comes to 
(dispatching) latency does not help either :(. Personally, I suspect this is 
the main reason everyone else is writing separate audio servers, but 
hopefully I can demonstrate that this is not really necessary.

> anyway - point is - audio is its own can of worms. moving it to x is good
> from one point - if it IS a remote display the DISPLAY itself will play the
> audio- i.e. - the audio is where it should be - the user. as a user i have
> no desire for my audio to play out of the server in the room 5 miles away.
> i want it here at my terminal where i am, and an x-based audio layer would
> make that simple and easy to achieve. for local clients this is just "yet
> another route to the audio device" (nb - you will want to be considering
> shm transports for local clients).

Shm samplebuffers are on my todo-list, but not implemented yet (and most 
certainly won't be before the first release). From my point of view they are 
desirable mainly because it would allow to bypass X dispatching entirely, so 
latency could in fact be very close to what you can achieve with direct 
hardware access.

Thanks for your comments!
Best regards
Helge Bahmann
-- 
Mathematicians stand on each other's shoulders while computer scientists stand 
on each other's toes.
-- Richard Hamming