[Intel-gfx] gitlab.fd.o financial situation and impact on services

[Dave Airlie]

On Fri, 28 Feb 2020 at 07:27, Daniel Vetter <daniel.vetter at ffwll.ch> wrote:
>
> Hi all,
>
> You might have read the short take in the X.org board meeting minutes
> already, here's the long version.
>
> The good news: gitlab.fd.o has become very popular with our
> communities, and is used extensively. This especially includes all the
> CI integration. Modern development process and tooling, yay!
>
> The bad news: The cost in growth has also been tremendous, and it's
> breaking our bank account. With reasonable estimates for continued
> growth we're expecting hosting expenses totalling 75k USD this year,
> and 90k USD next year. With the current sponsors we've set up we can't
> sustain that. We estimate that hosting expenses for gitlab.fd.o
> without any of the CI features enabled would total 30k USD, which is
> within X.org's ability to support through various sponsorships, mostly
> through XDC.
>
> Note that X.org does no longer sponsor any CI runners themselves,
> we've stopped that. The huge additional expenses are all just in
> storing and serving build artifacts and images to outside CI runners
> sponsored by various companies. A related topic is that with the
> growth in fd.o it's becoming infeasible to maintain it all on
> volunteer admin time. X.org is therefore also looking for admin
> sponsorship, at least medium term.
>
> Assuming that we want cash flow reserves for one year of gitlab.fd.o
> (without CI support) and a trimmed XDC and assuming no sponsor payment
> meanwhile, we'd have to cut CI services somewhere between May and June
> this year. The board is of course working on acquiring sponsors, but
> filling a shortfall of this magnitude is neither easy nor quick work,
> and we therefore decided to give an early warning as soon as possible.
> Any help in finding sponsors for fd.o is very much appreciated.

a) Ouch.

b) we probably need to take a large step back here.

Look at this from a sponsor POV, why would I give X.org/fd.o
sponsorship money that they are just giving straight to google to pay
for hosting credits? Google are profiting in some minor way from these
hosting credits being bought by us, and I assume we aren't getting any
sort of discounts here. Having google sponsor the credits costs google
substantially less than having any other company give us money to do
it.

If our current CI architecture is going to burn this amount of money a
year and we hadn't worked this out in advance of deploying it then I
suggest the system should be taken offline until we work out what a
sustainable system would look like within the budget we have, whether
that be never transferring containers and build artifacts from the
google network, just having local runner/build combos etc.

Dave.