Re: [RFC PATCH] Remove isar-apt and the corresponding lock

["Moessbauer, Felix" <felix.moessbauer@siemens.com>]

On Mon, 2022-12-12 at 12:36 +0000, Schmidt, Adriaan wrote:
> Baurzhan Ismagulov, Freitag, 21. Oktober 2022 12:15:
> > On Wed, Oct 19, 2022 at 12:47:45PM +0200, Adriaan Schmidt wrote:
> > > In addition to removing the lock, this also gives us better
> > > control
> > > over what goes into a package build, and eliminates effects due
> > > to
> > "accidental"
> > > packages in isar-apt.
> > 
> > We are aware of this issue. isar-apt is the "Debian way"
> > deliverable from an
> > Isar-based project, containing the locally-built packages. To
> > implement
> > proper
> > Debian Build-Depends functionality (without duplicating DEPENDS on
> > bitbake
> > level), we need the respective introspection of which packages come
> > from
> > Debian
> > and which ones are built locally. For this, we need to enhance
> > base-apt and
> > isar-apt (and maybe bitbake or some new class) to enable such
> > introspection.
> > [1] and [2] are the beginning of this work (they don't cover all of
> > it
> > today).
> 
> I don't see a conflict here. I'd say the .deb introspection and more
> advanced
> detection of dependencies is an independent issue.

I also would consider this as two independent aspects.

> 
> > We've looked at the locking in scope of [2]. We know that isar-apt
> > critical
> > section is too coarse and can be greatly narrowed down. If this is
> > the pain
> > point, we could prioritize that specific issue and provide the
> > respective
> > improvement. We'll check your patch in more detail, but I'm against
> > removing
> > isar-apt altogether.

The point is not about removing isar-apt all together, but removing the
global state which is shared across the tasks. Having this global state
is bad regarding the following aspects:

- reproducibility: We inject data that is not covered in any bitbake
hash
- blocking the scheduler: The access lock reduces the parallelism and
also smudge the statistics
- no way to remove packages between two invocations of bitbake:
  This is mainly relevant with local re-builds and pre-built packages
where only some should be deployed to the isar-apt. I recently debugged
such a case for an hour to understand why not the upstream package of
ntpsec is used, but my broken custom-built one (which I removed but
still lingered in the isar-apt)

I highly vote to further develop this RFC.

PS: I cannot access the mentioned links, looks like your patchwork is
down (or something is broken with our network).

Felix

> 
> I see the value of isar-apt as a deliverable of a build. But I don't
> think
> it's the best tool to transfer packages during build. To transfer
> between
> recipes of the same build, this RFC could be an alternative. And to
> transfer
> to other builds (re-use packages to speed things up), we have sstate.
> I had a look at [2], and one problem I see is that packages are only
> identified
> by their filename and version, and this does not lead to a rebuild
> when
> any of the inputs (sources, recipe) change.
> 
> If we remove isar-apt as a tool to transfer packages during build, I
> would
> definitely bring it back, possibly as a feature of the rootfs or
> image class
> (e.g., bitbake my-image -c export-apt-repo).
> 
> Adriaan
> 
> > 1. base-apt enhancement: https://patchwork.isar-
> > build.org/project/isar/patch/20220325103226.27033-1-ubely@ilbers.de
> > /
> > 2. isar-apt enhancement: https://patchwork.isar-
> > build.org/project/isar/patch/20220613070759.16949-1-ubely@ilbers.de
> > /
> > 
> > With kind regards,
> > Baurzhan
> > 
>