* [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate @ 2026-08-06 7:24 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 0 siblings, 2 replies; 7+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer Changes since v1: - move changelog entry to correct patch - rework caching logic to not break gbp or apt fetcher interfaces The series has now been tested in fast CI. Best regards, Felix Moessbauer Felix Moessbauer (2): dpkg: let do_deploy_deb runtime depend on do_deploy_source dpkg: cache do_dpkg_source results in sstate RECIPE-API-CHANGELOG.md | 8 ++++++++ meta/classes-recipe/dpkg-source.bbclass | 21 +++++++++++++++++++-- meta/classes-recipe/dpkg.bbclass | 2 +- 3 files changed, 28 insertions(+), 3 deletions(-) -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-1-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 ` 'Felix Moessbauer' via isar-users 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 1 sibling, 1 reply; 7+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer The isar-apt is an artifact that is deployed by isar and can be consumed as-is as an apt repository by downstream users. As the isar-apt also provides the debian source packages, we have to ensure that these are also deployed when the do_dpkg_build task is provided by the SState cache. Otherwise the isar-apt is not reproducible across runs with and without the SState cache. To fix this, we add a runtime dependency between do_deploy_deb and do_deploy_source, to ensure all sources of our debs are deployed as well. Reported-by: Andreas Ziegler <ziegler.andreas@siemens.com> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- meta/classes-recipe/dpkg-source.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass index b2b45ed3..92d06e1b 100644 --- a/meta/classes-recipe/dpkg-source.bbclass +++ b/meta/classes-recipe/dpkg-source.bbclass @@ -49,6 +49,8 @@ do_deploy_source() { addtask deploy_source after do_dpkg_source do_dpkg_build[depends] += "${BPN}:do_deploy_source" +# ensure that the source package is deployed into isar-apt +do_deploy_deb[rdepends] += "${BPN}:do_deploy_source" SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt" -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-2-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users @ 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 0 siblings, 0 replies; 7+ messages in thread From: 'Andreas Ziegler' via isar-users @ 2026-08-06 8:19 UTC (permalink / raw) To: Felix Moessbauer, isar-users Am 06.08.2026 um 09:24 schrieb Felix Moessbauer: > The isar-apt is an artifact that is deployed by isar and can be consumed > as-is as an apt repository by downstream users. As the isar-apt also > provides the debian source packages, we have to ensure that these are > also deployed when the do_dpkg_build task is provided by the SState > cache. Otherwise the isar-apt is not reproducible across runs with and > without the SState cache. > > To fix this, we add a runtime dependency between do_deploy_deb and > do_deploy_source, to ensure all sources of our debs are deployed as > well. > > Reported-by: Andreas Ziegler <ziegler.andreas@siemens.com> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > meta/classes-recipe/dpkg-source.bbclass | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > index b2b45ed3..92d06e1b 100644 > --- a/meta/classes-recipe/dpkg-source.bbclass > +++ b/meta/classes-recipe/dpkg-source.bbclass > @@ -49,6 +49,8 @@ do_deploy_source() { > addtask deploy_source after do_dpkg_source > > do_dpkg_build[depends] += "${BPN}:do_deploy_source" > +# ensure that the source package is deployed into isar-apt > +do_deploy_deb[rdepends] += "${BPN}:do_deploy_source" > > SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt" > Works like a charm in our pipeline, so: Tested-by: Andreas Ziegler <ziegler.andreas@siemens.com> Thanks! Andreas -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/fac5f582-a383-4b85-b149-afbc34f3a450%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 ` 'Felix Moessbauer' via isar-users 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users 2026-08-07 9:24 ` Zhihang Wei 1 sibling, 2 replies; 7+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer As the do_dpkg_source task now runs for each corresponding do_deploy_deb task, we also want to cache the generated results (to avoid pulling in a much bigger dependency tree on otherwise cached rebuilds). For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. This is implemented similar to how do_dpkg_build artifacts (.debs) are cached, which is also compatible with a hash equivalence server. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- RECIPE-API-CHANGELOG.md | 8 ++++++++ meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- meta/classes-recipe/dpkg.bbclass | 2 +- 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 7e923e55..7386f5d1 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. +### Debian source packages are deployed to `DEPLOY_DIR_SRC` + +The source packages produced by `do_dpkg_source` are now exported into a shared, +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. + +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now +reference the local copy in `${DEPLOY_DIR_SRC}` instead. + ### Add Hyper-V machine support A new machine `hyper-v` has been introduced for building images diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass index 92d06e1b..97cf9714 100644 --- a/meta/classes-recipe/dpkg-source.bbclass +++ b/meta/classes-recipe/dpkg-source.bbclass @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ DPKG_SOURCE_EXTRA_ARGS ?= "-I" DEBIAN_SOURCE ?= "${BPN}" +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" do_dpkg_source() { # Create a .dsc file from source directory to use it with sbuild DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) @@ -22,9 +27,19 @@ do_dpkg_source() { fi find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" + # move packages to deploy directory + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; } addtask dpkg_source after do_prepare_build +SSTATETASKS += "do_dpkg_source" + +python do_dpkg_source_setscene() { + sstate_setscene(d) +} + +addtask dpkg_source_setscene + CLEANFUNCS += "deb_clean_source" deb_clean_source() { @@ -34,11 +49,11 @@ deb_clean_source() { do_deploy_source[depends] += "isar-apt:do_cache_config" do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" -do_deploy_source[dirs] = "${S}" +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" do_deploy_source() { repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") if [ -n "${DSC_FILE}" ]; then repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass index d8e129bb..a42703d5 100644 --- a/meta/classes-recipe/dpkg.bbclass +++ b/meta/classes-recipe/dpkg.bbclass @@ -116,7 +116,7 @@ dpkg_runbuild() { echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) # networking is automatically enabled on older versions of sbuild sbuild_network_option="" -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-3-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users @ 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users 2026-08-07 9:24 ` Zhihang Wei 1 sibling, 0 replies; 7+ messages in thread From: 'Andreas Ziegler' via isar-users @ 2026-08-06 8:20 UTC (permalink / raw) To: Felix Moessbauer, isar-users Am 06.08.2026 um 09:24 schrieb Felix Moessbauer: > As the do_dpkg_source task now runs for each corresponding > do_deploy_deb task, we also want to cache the generated results (to > avoid pulling in a much bigger dependency tree on otherwise cached > rebuilds). > > For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. > This is implemented similar to how do_dpkg_build artifacts (.debs) are > cached, which is also compatible with a hash equivalence server. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > RECIPE-API-CHANGELOG.md | 8 ++++++++ > meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- > meta/classes-recipe/dpkg.bbclass | 2 +- > 3 files changed, 26 insertions(+), 3 deletions(-) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 7e923e55..7386f5d1 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. > Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an > artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. > > +### Debian source packages are deployed to `DEPLOY_DIR_SRC` > + > +The source packages produced by `do_dpkg_source` are now exported into a shared, > +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. > + > +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now > +reference the local copy in `${DEPLOY_DIR_SRC}` instead. > + > ### Add Hyper-V machine support > > A new machine `hyper-v` has been introduced for building images > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > index 92d06e1b..97cf9714 100644 > --- a/meta/classes-recipe/dpkg-source.bbclass > +++ b/meta/classes-recipe/dpkg-source.bbclass > @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ > DPKG_SOURCE_EXTRA_ARGS ?= "-I" > > DEBIAN_SOURCE ?= "${BPN}" > +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" > +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" > > +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" > do_dpkg_source() { > # Create a .dsc file from source directory to use it with sbuild > DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) > @@ -22,9 +27,19 @@ do_dpkg_source() { > fi > find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete > sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" > + # move packages to deploy directory > + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; > } > addtask dpkg_source after do_prepare_build > > +SSTATETASKS += "do_dpkg_source" > + > +python do_dpkg_source_setscene() { > + sstate_setscene(d) > +} > + > +addtask dpkg_source_setscene > + > CLEANFUNCS += "deb_clean_source" > > deb_clean_source() { > @@ -34,11 +49,11 @@ deb_clean_source() { > > do_deploy_source[depends] += "isar-apt:do_cache_config" > do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > -do_deploy_source[dirs] = "${S}" > +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" > do_deploy_source() { > repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > if [ -n "${DSC_FILE}" ]; then > repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ > diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass > index d8e129bb..a42703d5 100644 > --- a/meta/classes-recipe/dpkg.bbclass > +++ b/meta/classes-recipe/dpkg.bbclass > @@ -116,7 +116,7 @@ dpkg_runbuild() { > echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} > echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} > > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > > # networking is automatically enabled on older versions of sbuild > sbuild_network_option="" Tested-by: Andreas Ziegler <ziegler.andreas@siemens.com> Thanks! Andreas -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/2f561e79-2776-4c70-b36b-7e4829884fd6%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users @ 2026-08-07 9:24 ` Zhihang Wei 2026-08-10 7:40 ` 'MOESSBAUER, Felix' via isar-users 1 sibling, 1 reply; 7+ messages in thread From: Zhihang Wei @ 2026-08-07 9:24 UTC (permalink / raw) To: Felix Moessbauer, isar-users; +Cc: ziegler.andreas On 8/6/26 09:24, 'Felix Moessbauer' via isar-users wrote: > As the do_dpkg_source task now runs for each corresponding > do_deploy_deb task, we also want to cache the generated results (to > avoid pulling in a much bigger dependency tree on otherwise cached > rebuilds). > > For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. > This is implemented similar to how do_dpkg_build artifacts (.debs) are > cached, which is also compatible with a hash equivalence server. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > RECIPE-API-CHANGELOG.md | 8 ++++++++ > meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- > meta/classes-recipe/dpkg.bbclass | 2 +- > 3 files changed, 26 insertions(+), 3 deletions(-) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 7e923e55..7386f5d1 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. > Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an > artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. > > +### Debian source packages are deployed to `DEPLOY_DIR_SRC` > + > +The source packages produced by `do_dpkg_source` are now exported into a shared, > +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. > + > +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now > +reference the local copy in `${DEPLOY_DIR_SRC}` instead. > + > ### Add Hyper-V machine support > > A new machine `hyper-v` has been introduced for building images > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > index 92d06e1b..97cf9714 100644 > --- a/meta/classes-recipe/dpkg-source.bbclass > +++ b/meta/classes-recipe/dpkg-source.bbclass > @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ > DPKG_SOURCE_EXTRA_ARGS ?= "-I" > > DEBIAN_SOURCE ?= "${BPN}" > +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" > +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" > > +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" > do_dpkg_source() { > # Create a .dsc file from source directory to use it with sbuild > DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) > @@ -22,9 +27,19 @@ do_dpkg_source() { > fi > find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete > sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" > + # move packages to deploy directory > + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; > } > addtask dpkg_source after do_prepare_build > > +SSTATETASKS += "do_dpkg_source" > + > +python do_dpkg_source_setscene() { > + sstate_setscene(d) > +} > + > +addtask dpkg_source_setscene > + > CLEANFUNCS += "deb_clean_source" > > deb_clean_source() { > @@ -34,11 +49,11 @@ deb_clean_source() { > > do_deploy_source[depends] += "isar-apt:do_cache_config" > do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > -do_deploy_source[dirs] = "${S}" > +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" > do_deploy_source() { > repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > if [ -n "${DSC_FILE}" ]; then > repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ > diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass > index d8e129bb..a42703d5 100644 > --- a/meta/classes-recipe/dpkg.bbclass > +++ b/meta/classes-recipe/dpkg.bbclass > @@ -116,7 +116,7 @@ dpkg_runbuild() { > echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} > echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} > > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > > # networking is automatically enabled on older versions of sbuild > sbuild_network_option="" Hi Felix, This test case builds containers for qemuamd64-bookworm and qemuarm64-bookworm together using multiconfig. They can be built separately. But when they are built together in this test case, the following error occurs: [stdlog] 2026-08-07 09:00:54,845 avocado.app cibuilder L0371 ERROR| ERROR: mc:qemuamd64-bookworm:prebuilt-docker-img-0.1-r0 do_dpkg_build: ExecutionError('/build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/temp/run.dpkg_runbuild.399927', 1, None, None) [stdlog] 2026-08-07 09:00:54,845 avocado.app cibuilder L0371 ERROR| ERROR: Logfile of failure stored in: /build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/temp/log.do_dpkg_build.399927 [stdlog] 2026-08-07 09:00:54,845 avocado.test cibuilder L0369 INFO | Log data follows: ...... +------------------------------------------------------------------------------+ | Check architectures | +------------------------------------------------------------------------------+ Initial Foreign Architectures: i386 E: dsc: amd64 not in arch list or does not match any arch wildcards: arm64 -- skipping +------------------------------------------------------------------------------+ | Cleanup | +------------------------------------------------------------------------------+ Purging /<<BUILDDIR>> Not cleaning session: cloned chroot in use E: dsc: amd64 not in arch list or does not match any arch wildcards: arm64 -- skipping +------------------------------------------------------------------------------+ | Summary | +------------------------------------------------------------------------------+ Build Architecture: amd64 Build Type: binary Build-Space: n/a Build-Time: 0 Distribution: isar Fail-Stage: arch-check Foreign Architectures: i386 Host Architecture: amd64 Install-Time: 2 Job: /build/isar_wzh_devel_1z_others/42/build/tmp/deploy/isar-source/debian-bookworm/prebuilt-docker-img/prebuilt-docker-img_0.1.dsc Machine Architecture: amd64 Package: prebuilt-docker-img Package-Time: 0 Source-Version: 0.1 Space: n/a Status: skipped Version: 0.1 -------------------------------------------------------------------------------- Finished at 2026-08-07T07:00:50Z Build needed 00:00:00, no disk space E: dsc: amd64 not in arch list or does not match any arch wildcards: arm64 -- skipping cat: /build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/rootfs/dpkg_partial.logWARNING: exit code 1 from a shell command. : No such file or directory DEBUG: Executing shell function dpkg_chroot_finalize Zhihang -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/19e5d32e-e1b8-4860-884b-c05a87952006%40ilbers.de. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-07 9:24 ` Zhihang Wei @ 2026-08-10 7:40 ` 'MOESSBAUER, Felix' via isar-users 0 siblings, 0 replies; 7+ messages in thread From: 'MOESSBAUER, Felix' via isar-users @ 2026-08-10 7:40 UTC (permalink / raw) To: Zhihang Wei, isar-users; +Cc: Ziegler, Andreas, jan.kiszka On Fri, 2026-08-07 at 11:24 +0200, Zhihang Wei wrote: > On 8/6/26 09:24, 'Felix Moessbauer' via isar-users wrote: > > As the do_dpkg_source task now runs for each corresponding > > do_deploy_deb task, we also want to cache the generated results (to > > avoid pulling in a much bigger dependency tree on otherwise cached > > rebuilds). > > > > For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. > > This is implemented similar to how do_dpkg_build artifacts (.debs) are > > cached, which is also compatible with a hash equivalence server. > > > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > > --- > > RECIPE-API-CHANGELOG.md | 8 ++++++++ > > meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- > > meta/classes-recipe/dpkg.bbclass | 2 +- > > 3 files changed, 26 insertions(+), 3 deletions(-) > > > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > > index 7e923e55..7386f5d1 100644 > > --- a/RECIPE-API-CHANGELOG.md > > +++ b/RECIPE-API-CHANGELOG.md > > @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. > > Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an > > artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. > > > > +### Debian source packages are deployed to `DEPLOY_DIR_SRC` > > + > > +The source packages produced by `do_dpkg_source` are now exported into a shared, > > +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. > > + > > +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now > > +reference the local copy in `${DEPLOY_DIR_SRC}` instead. > > + > > ### Add Hyper-V machine support > > > > A new machine `hyper-v` has been introduced for building images > > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > > index 92d06e1b..97cf9714 100644 > > --- a/meta/classes-recipe/dpkg-source.bbclass > > +++ b/meta/classes-recipe/dpkg-source.bbclass > > @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ > > DPKG_SOURCE_EXTRA_ARGS ?= "-I" > > > > DEBIAN_SOURCE ?= "${BPN}" > > +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" > > +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" > > > > +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" > > +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" > > +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" > > do_dpkg_source() { > > # Create a .dsc file from source directory to use it with sbuild > > DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) > > @@ -22,9 +27,19 @@ do_dpkg_source() { > > fi > > find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete > > sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" > > + # move packages to deploy directory > > + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; > > } > > addtask dpkg_source after do_prepare_build > > > > +SSTATETASKS += "do_dpkg_source" > > + > > +python do_dpkg_source_setscene() { > > + sstate_setscene(d) > > +} > > + > > +addtask dpkg_source_setscene > > + > > CLEANFUNCS += "deb_clean_source" > > > > deb_clean_source() { > > @@ -34,11 +49,11 @@ deb_clean_source() { > > > > do_deploy_source[depends] += "isar-apt:do_cache_config" > > do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > > -do_deploy_source[dirs] = "${S}" > > +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" > > do_deploy_source() { > > repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" > > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > > if [ -n "${DSC_FILE}" ]; then > > repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ > > diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass > > index d8e129bb..a42703d5 100644 > > --- a/meta/classes-recipe/dpkg.bbclass > > +++ b/meta/classes-recipe/dpkg.bbclass > > @@ -116,7 +116,7 @@ dpkg_runbuild() { > > echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} > > echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} > > > > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > > > > # networking is automatically enabled on older versions of sbuild > > sbuild_network_option="" > > Hi Felix, > > This test case builds containers for qemuamd64-bookworm and > qemuarm64-bookworm > together using multiconfig. They can be built separately. But when they are > built together in this test case, the following error occurs: > > [stdlog] 2026-08-07 09:00:54,845 avocado.app cibuilder L0371 ERROR| > ERROR: mc:qemuamd64-bookworm:prebuilt-docker-img-0.1-r0 do_dpkg_build: > ExecutionError('/build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/temp/run.dpkg_runbuild.399927', > 1, None, None) > [stdlog] 2026-08-07 09:00:54,845 avocado.app cibuilder L0371 ERROR| > ERROR: Logfile of failure stored in: > /build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/temp/log.do_dpkg_build.399927 > [stdlog] 2026-08-07 09:00:54,845 avocado.test cibuilder L0369 INFO | Log > data follows: Hi, thanks for the logs. This patch series seems to expose the underlying bug: We build the same source package with different content for multiple architectures. Even before this patch, this silently raced in isar-apt, as only last-built .dsc file was added there. Now it races on the deb-source deploy dir, which exposes the bug reliably. When building a source package that varies in content across architectures, we must add the architecture to its name. I will send a patch for that. Note, that in general it is problematic to add DPKG_ARCH != {all, any}, but for cases like firmware, where anyways just one architecture is built, it does not matter. Felix > ...... > +------------------------------------------------------------------------------+ > | Check architectures | > +------------------------------------------------------------------------------+ > > Initial Foreign Architectures: i386 > E: dsc: amd64 not in arch list or does not match any arch wildcards: > arm64 -- skipping > > +------------------------------------------------------------------------------+ > | Cleanup | > +------------------------------------------------------------------------------+ > > Purging /<<BUILDDIR>> > Not cleaning session: cloned chroot in use > E: dsc: amd64 not in arch list or does not match any arch wildcards: > arm64 -- skipping > > +------------------------------------------------------------------------------+ > | Summary | > +------------------------------------------------------------------------------+ > > Build Architecture: amd64 > Build Type: binary > Build-Space: n/a > Build-Time: 0 > Distribution: isar > Fail-Stage: arch-check > Foreign Architectures: i386 > Host Architecture: amd64 > Install-Time: 2 > Job: > /build/isar_wzh_devel_1z_others/42/build/tmp/deploy/isar-source/debian-bookworm/prebuilt-docker-img/prebuilt-docker-img_0.1.dsc > Machine Architecture: amd64 > Package: prebuilt-docker-img > Package-Time: 0 > Source-Version: 0.1 > Space: n/a > Status: skipped > Version: 0.1 > -------------------------------------------------------------------------------- > Finished at 2026-08-07T07:00:50Z > Build needed 00:00:00, no disk space > E: dsc: amd64 not in arch list or does not match any arch wildcards: > arm64 -- skipping > cat: > /build/isar_wzh_devel_1z_others/42/build/tmp/work/debian-bookworm-amd64/prebuilt-docker-img/0.1-r0/rootfs/dpkg_partial.logWARNING: > exit code 1 from a shell command. > : No such file or directory > DEBUG: Executing shell function dpkg_chroot_finalize > > Zhihang -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/7c2e0d895fad8086b239c709820269c1a4ea265e.camel%40siemens.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-08-10 7:40 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users 2026-08-07 9:24 ` Zhihang Wei 2026-08-10 7:40 ` 'MOESSBAUER, Felix' via isar-users
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox