RE: [PATCH 0/5] Separate prebuild activities

["Schmidt, Adriaan" <adriaan.schmidt@siemens.com>]

MOESSBAUER, Felix (T CED INW-CN), Monday, October 30, 2023 5:45 AM:
> On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> > This patchset introduce three new pre-build tasks from the code was
> > already
> > done during the build task:
> >  do_dpkg_source
> >  do_deploy_source
> 
> We need to add sstate caching to the dpkg_source task as well.
> Currently we see some significant slowdown on big source packages (like
> the kernel) on otherwise fully cache-able builds.
> 
> Maybe Adriaan could add this, as he added the caching for dpkg_build as
> well.

Well, currently I expect that whenever we have a situation in which dpkg_build
would have to run (most likely a change in the recipe), that would mean
we need to rebuild the source as well. So I'm skeptical caching the source
would help much at this point.

BUT:
If we take the two-stage build one step further and actually have dpkg_build
work exclusively with the generated source package from isar-apt as input,
we could re-use one src package for all targets and variants (-compat/-native).

I'm currently working in this direction (prompted by bugs we see when installing
both "pkg" and "pkg-compat" in the same image), and so far I've found:

- We currently see a side-effect in the "remove source package" function:
  "reprepro removesrc" not only removes the src package, but also all related binary
  packages from the repository. So we need to make sure that only one
  task in a build deploys a src package (and that src is deployed before any binaries).
  Currently the source would be deployed by both "pkg" and "pkg-compat", with a
  resulting race condition. Already preparing a patch for that.
- The -compat/-native suffix sometimes "leaks" into the source package.
  Basically, there are still locations that should use "${BPN}" instead of "${PN}".
  Already preparing a patch for that.
- currently there are still tasks/dependencies of do_dpkg_build that rely on 
  the extracted source being present (e.g. finding out the name of the src
  package)
- we'd need to add a deb-src entry to the apt sources for sbuild
- to actually be able to share src packages across architectures, we'd need
  to make sure that the src packages are always identical, and don't depend
  on any architecture or -compat/-native variant. I guess in the Debian world
  that already is a rule, but in bitbake users can of course add any number
  of overrides (e.g. changing SRC_URI depending on PACKAGE_ARCH) to break
  this rule...

Adriaan

> Best regards,
> Felix
> 
> >  do_local_isarapt
> > This allows to execute tasks more efficient. On simple targets like
> > qemuamd64
> > the speed up of the build is around 10 second, but on complex
> > downstream layers
> > this effect can be more visible.
> > We also introduce source packages storage in isar-apt repository
> > which allows
> > obtaining package Build-Depends and Provides information in Debian
> > way later.
> >
> > Anton Mikanovich (5):
> >   repository: Add source package remove function
> >   dpkg: Limit dsc searching by workdir
> >   meta: Implement two stage build
> >   dpkg-base: Copy isar_apt to workdir in separate task
> >   RECIPE-API-CHANGELOG.md: Note tasks separation
> >
> >  RECIPE-API-CHANGELOG.md                    | 14 +++++++++
> >  meta/classes/dpkg-base.bbclass             | 15 +++++-----
> >  meta/classes/dpkg-gbp.bbclass              |  2 +-
> >  meta/classes/dpkg-source.bbclass           | 33
> > ++++++++++++++++++++++
> >  meta/classes/dpkg.bbclass                  |  6 ++--
> >  meta/classes/repository.bbclass            | 14 +++++++++
> >  meta/classes/rootfs.bbclass                |  2 ++
> >  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
> >  8 files changed, 74 insertions(+), 14 deletions(-)
> >  create mode 100644 meta/classes/dpkg-source.bbclass
> >
> > --
> > 2.34.1
> >
>