Re: [PATCH v3 3/3] dpkg: cache do_dpkg_source results in sstate

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Mon, 2026-08-10 at 10:31 +0200, Felix Moessbauer wrote:
> As the do_dpkg_source task now runs for each corresponding
> do_deploy_deb task, we also want to cache the generated results (to
> avoid pulling in a much bigger dependency tree on otherwise cached
> rebuilds).
> 
> For that, we cache the .dsc and the .tar.* artifacts in the deploy dir.
> This is implemented similar to how do_dpkg_build artifacts (.debs) are
> cached, which is also compatible with a hash equivalence server.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md                 | 29 +++++++++++++++++++++++++
>  meta/classes-recipe/dpkg-source.bbclass | 19 ++++++++++++++--
>  meta/classes-recipe/dpkg.bbclass        |  2 +-
>  3 files changed, 47 insertions(+), 3 deletions(-)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 7e923e55..2db05169 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -1146,6 +1146,35 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
>  Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
>  artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.
>  
> +### Debian source packages are deployed to `DEPLOY_DIR_SRC`
> +
> +The source packages produced by `do_dpkg_source` are now exported into a shared,
> +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`.
> +
> +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now
> +reference the local copy in `${DEPLOY_DIR_SRC}` instead.
> +
> +### Architecture specific source packages
> +
> +Setting `DPKG_ARCH = "<arch>"` makes the source package architecture-specific,
> +unlike `all` or `any`. Because Debian source packages are identified by name and
> +version, package providers must ensure that a source package with the same name
> +and version does not contain different content for different architectures. This
> +was never allowed, but Isar now enforces it.
> +
> +In this case, you can use the following pattern to make the package architecture specific,
> +while still maintaining backward compatibility:
> +
> +```
> +inherit dpkg-raw
> +PROVIDES := "${BPN}"
> +DEBIAN_PROVIDES := "${BPN}"
> +PN .= "-${DISTRO_ARCH}"
> +DPKG_ARCH ?= "${DISTRO_ARCH}"
> +```
> +
> +Firmware packages are an exception because they are built for only one architecture.
> +
>  ### Add Hyper-V machine support
>  
>  A new machine `hyper-v` has been introduced for building images
> diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
> index 92d06e1b..97cf9714 100644
> --- a/meta/classes-recipe/dpkg-source.bbclass
> +++ b/meta/classes-recipe/dpkg-source.bbclass
> @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \
>  DPKG_SOURCE_EXTRA_ARGS ?= "-I"
>  
>  DEBIAN_SOURCE ?= "${BPN}"
> +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg"
> +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}"

Hi,

this series revealed another bug regarding source packages with same
name but different content:

Similar to upstream Debian, our kernel source packages are architecture
specific. By that, we also have to encode the architecture (or machine)
in the name of the package. Otherwise existing packages might get
reused on rebuilds with other targets (or multiarch)

In isar, this has ever been modeled incorrectly (the kernels .dsc file
was arch=any despite being arch specific). This is easy to fix, but
also depending on the kernels was done incorrectly (a KERNEL_NAME =
mainline is not specific enough, as mainline on arm64 is technically a
different kernel (source package) than mainline on amd64.

To fix this, we have the following options (which I tried):

1. encode the MACHINE or DISTRO_ARCH in KERNEL_NAME. Usually arch is
enough, but if a machine specific defconfig is used, the machine needs
to be encoded in the name => clean, works but might require downstream
changes

2. internally generate kernel source package with -${DISTRO_ARCH}
suffix, but keep DEPENDS / PROVIDES and binary package names as-is:
This is different to how Debian names the kernels. For the kernel build
itself this can be modeled, but the dependencies of custom modules back
to the "correct" kernel cannot be modeled, simply because the :native
and alike multiarch logic does not know about the internal changes and
is not able to dispatch to the correct kernel recipes. I tried various
workarounds, but it only got more and more complex without any solution
that is able to pass the CI.

3. Make DEPLOY_DIR_SRC architecture specific. This is a easy change,
but means we deploy all source packages multiple times. Along that, it
still means our deployed isar-apt is racy as the last built source
package wins.


Given that, I will send a fix based on (1), along with a an API
changelog. I'm pretty sure, most downstream layers don't have to change
anything. If they only build a single target in a single run, only a
single kernel is built and by that the bug has no effect.

Best regards,
Felix

>  
> +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}"
> +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}"
> +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
>  do_dpkg_source() {
>      # Create a .dsc file from source directory to use it with sbuild
>      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> @@ -22,9 +27,19 @@ do_dpkg_source() {
>      fi
>      find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
>      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
> +    # move packages to deploy directory
> +    find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \;
>  }
>  addtask dpkg_source after do_prepare_build
>  
> +SSTATETASKS += "do_dpkg_source"
> +
> +python do_dpkg_source_setscene() {
> +    sstate_setscene(d)
> +}
> +
> +addtask dpkg_source_setscene
> +
>  CLEANFUNCS += "deb_clean_source"
>  
>  deb_clean_source() {
> @@ -34,11 +49,11 @@ deb_clean_source() {
>  
>  do_deploy_source[depends] += "isar-apt:do_cache_config"
>  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> -do_deploy_source[dirs] = "${S}"
> +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}"
>  do_deploy_source() {
>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
> -    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
> +    DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
>      if [ -n "${DSC_FILE}" ]; then
>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
> diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
> index d8e129bb..a42703d5 100644
> --- a/meta/classes-recipe/dpkg.bbclass
> +++ b/meta/classes-recipe/dpkg.bbclass
> @@ -116,7 +116,7 @@ dpkg_runbuild() {
>      echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>      echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
>  
> -    DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
> +    DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
>  
>      # networking is automatically enabled on older versions of sbuild
>      sbuild_network_option=""
> -- 
> 2.55.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/b26fcda922ab9ab2341f11e5c951089ce8bc1fcc.camel%40siemens.com.