Re: [PATCH v3 3/3] dpkg: cache do_dpkg_source results in sstate

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Thu, 2026-08-13 at 13:29 +0000, 'MOESSBAUER, Felix' via isar-users
wrote:
> On Mon, 2026-08-10 at 10:31 +0200, Felix Moessbauer wrote:
> > As the do_dpkg_source task now runs for each corresponding
> > do_deploy_deb task, we also want to cache the generated results (to
> > avoid pulling in a much bigger dependency tree on otherwise cached
> > rebuilds).
> > 
> > For that, we cache the .dsc and the .tar.* artifacts in the deploy dir.
> > This is implemented similar to how do_dpkg_build artifacts (.debs) are
> > cached, which is also compatible with a hash equivalence server.
> > 
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md                 | 29 +++++++++++++++++++++++++
> >  meta/classes-recipe/dpkg-source.bbclass | 19 ++++++++++++++--
> >  meta/classes-recipe/dpkg.bbclass        |  2 +-
> >  3 files changed, 47 insertions(+), 3 deletions(-)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 7e923e55..2db05169 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -1146,6 +1146,35 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
> >  Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
> >  artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.
> >  
> > +### Debian source packages are deployed to `DEPLOY_DIR_SRC`
> > +
> > +The source packages produced by `do_dpkg_source` are now exported into a shared,
> > +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`.
> > +
> > +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now
> > +reference the local copy in `${DEPLOY_DIR_SRC}` instead.
> > +
> > +### Architecture specific source packages
> > +
> > +Setting `DPKG_ARCH = "<arch>"` makes the source package architecture-specific,
> > +unlike `all` or `any`. Because Debian source packages are identified by name and
> > +version, package providers must ensure that a source package with the same name
> > +and version does not contain different content for different architectures. This
> > +was never allowed, but Isar now enforces it.
> > +
> > +In this case, you can use the following pattern to make the package architecture specific,
> > +while still maintaining backward compatibility:
> > +
> > +```
> > +inherit dpkg-raw
> > +PROVIDES := "${BPN}"
> > +DEBIAN_PROVIDES := "${BPN}"
> > +PN .= "-${DISTRO_ARCH}"
> > +DPKG_ARCH ?= "${DISTRO_ARCH}"
> > +```
> > +
> > +Firmware packages are an exception because they are built for only one architecture.
> > +
> >  ### Add Hyper-V machine support
> >  
> >  A new machine `hyper-v` has been introduced for building images
> > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
> > index 92d06e1b..97cf9714 100644
> > --- a/meta/classes-recipe/dpkg-source.bbclass
> > +++ b/meta/classes-recipe/dpkg-source.bbclass
> > @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \
> >  DPKG_SOURCE_EXTRA_ARGS ?= "-I"
> >  
> >  DEBIAN_SOURCE ?= "${BPN}"
> > +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg"
> > +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}"
> 
> Hi,
> 
> this series revealed another bug regarding source packages with same
> name but different content:
> 
> Similar to upstream Debian, our kernel source packages are architecture
> specific. By that, we also have to encode the architecture (or machine)
> in the name of the package. Otherwise existing packages might get
> reused on rebuilds with other targets (or multiarch)
> 
> In isar, this has ever been modeled incorrectly (the kernels .dsc file
> was arch=any despite being arch specific). This is easy to fix, but
> also depending on the kernels was done incorrectly (a KERNEL_NAME =
> mainline is not specific enough, as mainline on arm64 is technically a
> different kernel (source package) than mainline on amd64.
> 
> To fix this, we have the following options (which I tried):
> 
> 1. encode the MACHINE or DISTRO_ARCH in KERNEL_NAME. Usually arch is
> enough, but if a machine specific defconfig is used, the machine needs
> to be encoded in the name => clean, works but might require downstream
> changes

Hi, after some extensive tests, I conclude that this is the only
feasible approach. The API change is also quite minimal, as we
technically don't even change an API. We just tell the users how to
correctly name their packages - which they theoretically should already
have done correctly.

> 
> 2. internally generate kernel source package with -${DISTRO_ARCH}
> suffix, but keep DEPENDS / PROVIDES and binary package names as-is:
> This is different to how Debian names the kernels. For the kernel build
> itself this can be modeled, but the dependencies of custom modules back
> to the "correct" kernel cannot be modeled, simply because the :native
> and alike multiarch logic does not know about the internal changes and
> is not able to dispatch to the correct kernel recipes. I tried various
> workarounds, but it only got more and more complex without any solution
> that is able to pass the CI.

I tried this, but it does it comes with severe limitations. The main
issue is that we have to add additional provides, but just for the base
variant of the recipe. Detecting what the base variant is either
requires heuristics (bad!) or immediate expansion of BPN on first
parsing. This however breaks if kernel recipes are inherited / included
as then the immediate expansion always results in the first name
instead of the final one.

It also is problematic as we now create the conflicts on the binary
packages.

> 
> 3. Make DEPLOY_DIR_SRC architecture specific. This is a easy change,
> but means we deploy all source packages multiple times. Along that, it
> still means our deployed isar-apt is racy as the last built source
> package wins.

Possible, but a lot of disk-space waste and against the single-source
package principles.

Felix

> 
> 
> Given that, I will send a fix based on (1), along with a an API
> changelog. I'm pretty sure, most downstream layers don't have to change
> anything. If they only build a single target in a single run, only a
> single kernel is built and by that the bug has no effect.
> 
> Best regards,
> Felix
> 
> >  
> > +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}"
> > +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}"
> > +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
> >  do_dpkg_source() {
> >      # Create a .dsc file from source directory to use it with sbuild
> >      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> > @@ -22,9 +27,19 @@ do_dpkg_source() {
> >      fi
> >      find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
> >      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
> > +    # move packages to deploy directory
> > +    find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \;
> >  }
> >  addtask dpkg_source after do_prepare_build
> >  
> > +SSTATETASKS += "do_dpkg_source"
> > +
> > +python do_dpkg_source_setscene() {
> > +    sstate_setscene(d)
> > +}
> > +
> > +addtask dpkg_source_setscene
> > +
> >  CLEANFUNCS += "deb_clean_source"
> >  
> >  deb_clean_source() {
> > @@ -34,11 +49,11 @@ deb_clean_source() {
> >  
> >  do_deploy_source[depends] += "isar-apt:do_cache_config"
> >  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> > -do_deploy_source[dirs] = "${S}"
> > +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}"
> >  do_deploy_source() {
> >      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> >          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
> > -    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
> > +    DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
> >      if [ -n "${DSC_FILE}" ]; then
> >          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> >              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
> > diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
> > index d8e129bb..a42703d5 100644
> > --- a/meta/classes-recipe/dpkg.bbclass
> > +++ b/meta/classes-recipe/dpkg.bbclass
> > @@ -116,7 +116,7 @@ dpkg_runbuild() {
> >      echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
> >      echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
> >  
> > -    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
> > +    DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
> >  
> >      # networking is automatically enabled on older versions of sbuild
> >      sbuild_network_option=""
> > -- 
> > 2.55.0
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/b26fcda922ab9ab2341f11e5c951089ce8bc1fcc.camel%40siemens.com.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/c20b3fbedf9a90c8e036ad035b40536f0b33374d.camel%40siemens.com.