consolidating CI builds

[Peter Hutterer]

On Tue, Jun 11, 2024 at 10:49:35AM +0200, Enrico Weigelt, metux IT consult wrote:
> On 11.06.24 09:09, Peter Hutterer wrote:
> 
> hi folks,
> 
> > With my ci-templates maintainer hat on: no to xorg-specific templates in
> > there, it gets too messy.
> 
> Indeed that would be the most wrong place.
> 
> > Note that you can include any file from another repo (I think) so in
> > theory we could ship the templates in e.g.
> > xserver/.gitlab-templates/foo.yml or whatever. Not saying we should do
> > that, just noting it's possible.
> 
> Actually though of that. It has some logic: drivers are depending on
> Xserver anyways, it already provides some "sdk" for them.
> 
> By the way: most of that stuff isn't actually gitlab specific - they're
> also designed to be called directly, eg. from within chroot or VM.
> So they're also helpful for scenarios that we can't easily integrate
> on gitlab (Solaris, NetBSD, ...). They also hold the logic for OS
> specific tweaks (eg. special pkgconf pathes), detecting the actual build
> system (xserver as well as driver), etc, etc.
> 
> The more I'm thinking about it, the stronger my feeling that xserver
> repo smells like the best candidate.
> 
> > ci-fairy (in ci-templates) also has a generate-templates command that
> > e.g. libinput uses to generate templates from a config.yml and a jinja
> > file. This could also be an option - have a shared *source* template
> > somewhere, then generate the .gitlab-ci.yml it with repo-specific config
> > values.
> 
> Maybe I'm not really understanding your idea, but I don't feel that's
> the best way: there's not really anything to generate, and the way it's
> done with current templates (eg. the generator has to put in blob
> hashes, etc) IMHO isn't the fine art.

the blob hashes are specific to ci-templates and ensure that building
from the templates pulls in the right cbuild scripts, they don't (need to)
exist anywhere else. `ci-fairy generate-templates` is really just a
fairly thin wrapper around a jinja2 template, that's it.

> Speaking of which: why doesn't it just clone the git repo for getting
> scripts like cbuild, etc ? IMHO would make things a lot easier than
> having "hardcoded" hashes in the generated templates. By the way, while
> working on on the CI stuff, learned that a lot of sophisticated things
> can be done via gitlab templates (still far away from jenkins
> capabilities) ... maybe we could do it even w/o generating templates.
> But that's another topic of it's own.

It's a result of how the ci-templates work. ci-templates are used via
include:, with a specific git sha. Those templates need to pull in the
matching cbuild script (since there *may* be a strict dependency between
that and the template itself). But the template cannot know the git sha
of itself, hence the workaround with the extra checksums.

> Back to xorg drivers:
> 
> As things are now, moving that all out to separate repo just doesn't
> work: cbuild (when building the container images) makes it's own git
> clone of the project's repo, and we don't have any means for injecting
> extra steps. So, when $FDO_DISTRIBUTION_EXEC is called, the scripts
> are missing :(
> 
> I've played around with job inheritance and before_script, so the aux
> repo is cloned before anything else. But that failed at cbuild for
> the reasons above.
> 
> Just got one spontanous idea I'll yet have to try: also add those
> commands to $FDO_DISTRIBUTION_EXEC. Clumpsy, but maybe it could work.

FDO_DISTRIBUTION_EXEC is a convenience helper, you can call out into a
shell script which can then do whatever it wants (and IIRC it inherits
the gitlab env too so you can check for the various CI_FOO).

plenty of projects use FDO_DISTRIBUTION_EXEC to e.g. git clone
dependencies and install them.
 
Cheers,
  Peter

> 
> --mtx
> 
> --
> ---
> Hinweis: unverschlüsselte E-Mails können leicht abgehört und manipuliert
> werden ! Für eine vertrauliche Kommunikation senden Sie bitte ihren
> GPG/PGP-Schlüssel zu.
> ---
> Enrico Weigelt, metux IT consult
> Free software and Linux embedded engineering
> info at metux.net -- +49-151-27565287