Re: isar-bootstrap

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 26 Oct 2021 20:58:16 +0200 (CEST)
schrieb ydirson@free.fr:

> Hi,
> 
> Anton wrote:
> > > I was trying to get a grasp on DISTRO_APT_SOURCES handling, and
> > > noticed
> > > a spurious match for "distro_apt_sources" in
> > > DISTRO_APT_PREFERENCES handling.
> > > While this possibly was meant to read "distro_apt_preferences"
> > > instead.  
> 
> Cool :)
> 
> Henning wrote:
> > This was already fixed by `[PATCH v2] isar-bootstrap: Fix SRC_URI
> > inclusion` and will be merged into next soon.  
> > > it raises the question of
> > > whether it is useful to add all of this to SRC_URI if not adding
> > > it did not cause any problem (it does not appear to be used by any
> > > conf
> > > in isar itself, but I guess it is/has been used in downstream
> > > layers).
> > > 
> > > This brought me to where this gets used (both _SOURCES and
> > > _PREFERENCES in fact), ie. do_apt_config_prepare: the whole point
> > > of
> > > SRC_URI is to direct the fetcher to download/copy source files
> > > into $WORKDIR, and then later tasks (unpack/patch in OE) take
> > > files from there.  But here, $WORKDIR is not used, and instead
> > > aggregate_files()
> > > uses resolve_file() on them.  At first sight this seems to break
> > > the
> > > bitbake abstractions (and seems unnecessarily complicated at the
> > > same
> > > time).  Am I missing something, or is it worth for me to try
> > > patching
> > > ?  
> > 
> > Not sure i get your point. In general we have to deal with two
> > worlds in Isar. There is the debian-world and the bitbake world and
> > they are not too tightly integrated at some points.
> > Meaning we have debian fetching stuff and bitbake fetching stuff. We
> > have two namespace for packages which leads to weird statements like
> > 
> > # comma-sep
> > DEBIAN_DEPENDS = "debian-binary-package-a, debian-binary-package-b"
> > # space-sep
> > DEPENDS = "recipe-for-binpkg-a recipe-for-binpkg-b"
> >
> > That two world can not be easily resolved and is found in several
> > places, the one i pointed out being maybe the nasties one.  
> 
> At first sight I don't find this last part too problematic, this looks
> like the standard
> double source/binary namespace, and DEBIAN_DEPENDS rather looks like
> a Yocto RDEPENDS (though processed), and DEPENDS a Debian
> Build-Depends. (indeed Yocto packaging does generate Depends from
> RDEPENDS when building a deb or ipk)

Yes that is pretty much it. But Isar currently does not try to help
here. In practice it is less of "big deal" just very confusing to new
Isar users when they have to say things twice in different lingo.

> > No need to
> > comment on a possible idea, it has been tried and is more complex
> > than i made it look.  
> 
> I guess this refers to the double-downloader problem, I had my share
> of such things some time ago [0] (and finally found a rather
> acceptable solution based on (IIRC) the meta-rust approach).  I'd be
> interested to learn about the approaches you tried.

No i was talking about the src vs binary namespace, which in fact you
have in Debian and OE alike. And Isar has 4 such namespaces i guess.
But as pointed out that is less of a practical problem than a
"confusion topic". If you go mostly debian and litte Isar (like 95/5)
it is really just an academic thing and not worth too much effort.

For the download Isar goes the pragmatic way and lets debian fetch what
it wants. With a few exceptions ... i.e. there is only one "global
apt-get update" so you have to hope that you can apt-get install what
that initial run created your external database for. In practice that
does not fail too ofter ... or you have to clean build again.

If you really need to pin debian down to what it fetches, because for
some reason (like repro build) you need your own mirror. In fact Isar
spits out a partial debian mirror after an "online" build (base-apt).
That can be used for consecutive offline builds, or as a base for
consecutive "online" builds with custom DISTRO_APT_SOURCES.

While snapshots.debian.org can be used as DISTRO_APT_SOURCES mirror in
theory ... in practice it has rate-limiting in place. So you might
succeed in a manual build that you retry over and over (or a small
image), but in CI without caching ... you will never get a big image to
build. That rate-limiting issue will need to be discussed with
snapshots, we are not the first ones to have issues with it.
But i personally would tell people to simply not freeze if they can,
and the ones that need to freeze i would in fact tell to get a full
debian mirror of their own, instead of a partial one produced by isars
base-apt.
As an OSS project you might see less of a need of freezing, tracking in
fact is a security feature ... and debian will not do much more than
security on their stable distros.

regards,
Henning

> > That said you can not see everything in our code, the files placed
> > carefully will be used by debootstrap and later by apt.  
> 
> Sure, such things do not lay themselves on (e)paper in one day, and
> that can make them more difficult for newcomers to see all the
> implications of a given construct - and also makes it easy to break
> things.
> 
> 
> > And the two variables are lists of files which are meant to be
> > customizeable. That way you can switch to another mirror, add more
> > repos (ppa in ubunut speach) and configure what to take from where
> > or what to possibly pin (prefs)  
> 
> Yes, I realized this and was not suggesting to drop them and use
> SRC_URI instead (although the idea did cross my mind ;).  This remark
> was purely from the bitbake side, and I realized thinking about it
> later that some assumptions of mine may have been wrong, I'll have to
> check this later.
> 
> 
> > > On a related note, I assume that aggregate_files() was meant to
> > > support distro versions predating preferences.d/ and
> > > sources.list.d/
> > > - would it be correct to use the latter instead to simply things
> > > further ?  Most recent is preferences.d/, released with 0.7.22 in
> > > 2009, and stretch which seems to be the oldest supported distro
> > > has 1.4.  
> > 
> > I do not know, maybe also to be able to delete things again after
> > the build. Because you want other configuration at build-time than
> > you want
> > on product delivery. i.e. build against a "frozen" mirror but
> > deliver with a public mirror as update source  
> 
> OK I see.
> 
> >    
> > > 
> > > The chroot-setup.sh script there also seems to do things that are
> > > already handled by the debootstrap (policy.d and
> > > start-stop-daemon),
> > > except for the upstart support.  Even if someone still uses
> > > upstart (which has not been developped since 2014), probably the
> > > rest could be omitted ?
> > >   
> > 
> > Thanks for the feedback. But i guess it might be a little much mixed
> > up in one email.  
> 
> Yes, I had to refrain from bringing too many subjects already :)
> 
> > So i mainly answered to give you more information, not
> > because i want to go and write patches.
> > But maybe you at some point might want to.  
> 
> Yes, though I will need more time to get a full picture.
> 
> [0] https://patchwork.openembedded.org/patch/178697/
> 
> Best regards,