* [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate @ 2026-08-06 7:24 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 0 siblings, 2 replies; 5+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer Changes since v1: - move changelog entry to correct patch - rework caching logic to not break gbp or apt fetcher interfaces The series has now been tested in fast CI. Best regards, Felix Moessbauer Felix Moessbauer (2): dpkg: let do_deploy_deb runtime depend on do_deploy_source dpkg: cache do_dpkg_source results in sstate RECIPE-API-CHANGELOG.md | 8 ++++++++ meta/classes-recipe/dpkg-source.bbclass | 21 +++++++++++++++++++-- meta/classes-recipe/dpkg.bbclass | 2 +- 3 files changed, 28 insertions(+), 3 deletions(-) -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-1-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 ` 'Felix Moessbauer' via isar-users 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 1 sibling, 1 reply; 5+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer The isar-apt is an artifact that is deployed by isar and can be consumed as-is as an apt repository by downstream users. As the isar-apt also provides the debian source packages, we have to ensure that these are also deployed when the do_dpkg_build task is provided by the SState cache. Otherwise the isar-apt is not reproducible across runs with and without the SState cache. To fix this, we add a runtime dependency between do_deploy_deb and do_deploy_source, to ensure all sources of our debs are deployed as well. Reported-by: Andreas Ziegler <ziegler.andreas@siemens.com> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- meta/classes-recipe/dpkg-source.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass index b2b45ed3..92d06e1b 100644 --- a/meta/classes-recipe/dpkg-source.bbclass +++ b/meta/classes-recipe/dpkg-source.bbclass @@ -49,6 +49,8 @@ do_deploy_source() { addtask deploy_source after do_dpkg_source do_dpkg_build[depends] += "${BPN}:do_deploy_source" +# ensure that the source package is deployed into isar-apt +do_deploy_deb[rdepends] += "${BPN}:do_deploy_source" SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt" -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-2-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users @ 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 0 siblings, 0 replies; 5+ messages in thread From: 'Andreas Ziegler' via isar-users @ 2026-08-06 8:19 UTC (permalink / raw) To: Felix Moessbauer, isar-users Am 06.08.2026 um 09:24 schrieb Felix Moessbauer: > The isar-apt is an artifact that is deployed by isar and can be consumed > as-is as an apt repository by downstream users. As the isar-apt also > provides the debian source packages, we have to ensure that these are > also deployed when the do_dpkg_build task is provided by the SState > cache. Otherwise the isar-apt is not reproducible across runs with and > without the SState cache. > > To fix this, we add a runtime dependency between do_deploy_deb and > do_deploy_source, to ensure all sources of our debs are deployed as > well. > > Reported-by: Andreas Ziegler <ziegler.andreas@siemens.com> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > meta/classes-recipe/dpkg-source.bbclass | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > index b2b45ed3..92d06e1b 100644 > --- a/meta/classes-recipe/dpkg-source.bbclass > +++ b/meta/classes-recipe/dpkg-source.bbclass > @@ -49,6 +49,8 @@ do_deploy_source() { > addtask deploy_source after do_dpkg_source > > do_dpkg_build[depends] += "${BPN}:do_deploy_source" > +# ensure that the source package is deployed into isar-apt > +do_deploy_deb[rdepends] += "${BPN}:do_deploy_source" > > SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt" > Works like a charm in our pipeline, so: Tested-by: Andreas Ziegler <ziegler.andreas@siemens.com> Thanks! Andreas -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/fac5f582-a383-4b85-b149-afbc34f3a450%40siemens.com. ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 ` 'Felix Moessbauer' via isar-users 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users 1 sibling, 1 reply; 5+ messages in thread From: 'Felix Moessbauer' via isar-users @ 2026-08-06 7:24 UTC (permalink / raw) To: isar-users; +Cc: ziegler.andreas, Felix Moessbauer As the do_dpkg_source task now runs for each corresponding do_deploy_deb task, we also want to cache the generated results (to avoid pulling in a much bigger dependency tree on otherwise cached rebuilds). For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. This is implemented similar to how do_dpkg_build artifacts (.debs) are cached, which is also compatible with a hash equivalence server. Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> --- RECIPE-API-CHANGELOG.md | 8 ++++++++ meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- meta/classes-recipe/dpkg.bbclass | 2 +- 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 7e923e55..7386f5d1 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. +### Debian source packages are deployed to `DEPLOY_DIR_SRC` + +The source packages produced by `do_dpkg_source` are now exported into a shared, +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. + +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now +reference the local copy in `${DEPLOY_DIR_SRC}` instead. + ### Add Hyper-V machine support A new machine `hyper-v` has been introduced for building images diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass index 92d06e1b..97cf9714 100644 --- a/meta/classes-recipe/dpkg-source.bbclass +++ b/meta/classes-recipe/dpkg-source.bbclass @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ DPKG_SOURCE_EXTRA_ARGS ?= "-I" DEBIAN_SOURCE ?= "${BPN}" +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" do_dpkg_source() { # Create a .dsc file from source directory to use it with sbuild DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) @@ -22,9 +27,19 @@ do_dpkg_source() { fi find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" + # move packages to deploy directory + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; } addtask dpkg_source after do_prepare_build +SSTATETASKS += "do_dpkg_source" + +python do_dpkg_source_setscene() { + sstate_setscene(d) +} + +addtask dpkg_source_setscene + CLEANFUNCS += "deb_clean_source" deb_clean_source() { @@ -34,11 +49,11 @@ deb_clean_source() { do_deploy_source[depends] += "isar-apt:do_cache_config" do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" -do_deploy_source[dirs] = "${S}" +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" do_deploy_source() { repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") if [ -n "${DSC_FILE}" ]; then repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass index d8e129bb..a42703d5 100644 --- a/meta/classes-recipe/dpkg.bbclass +++ b/meta/classes-recipe/dpkg.bbclass @@ -116,7 +116,7 @@ dpkg_runbuild() { echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) # networking is automatically enabled on older versions of sbuild sbuild_network_option="" -- 2.55.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260806072428.2409275-3-felix.moessbauer%40siemens.com. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users @ 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users 0 siblings, 0 replies; 5+ messages in thread From: 'Andreas Ziegler' via isar-users @ 2026-08-06 8:20 UTC (permalink / raw) To: Felix Moessbauer, isar-users Am 06.08.2026 um 09:24 schrieb Felix Moessbauer: > As the do_dpkg_source task now runs for each corresponding > do_deploy_deb task, we also want to cache the generated results (to > avoid pulling in a much bigger dependency tree on otherwise cached > rebuilds). > > For that, we cache the .dsc and the .tar.* artifacts in the deploy dir. > This is implemented similar to how do_dpkg_build artifacts (.debs) are > cached, which is also compatible with a hash equivalence server. > > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> > --- > RECIPE-API-CHANGELOG.md | 8 ++++++++ > meta/classes-recipe/dpkg-source.bbclass | 19 +++++++++++++++++-- > meta/classes-recipe/dpkg.bbclass | 2 +- > 3 files changed, 26 insertions(+), 3 deletions(-) > > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index 7e923e55..7386f5d1 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`. > Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an > artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead. > > +### Debian source packages are deployed to `DEPLOY_DIR_SRC` > + > +The source packages produced by `do_dpkg_source` are now exported into a shared, > +sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`. > + > +Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now > +reference the local copy in `${DEPLOY_DIR_SRC}` instead. > + > ### Add Hyper-V machine support > > A new machine `hyper-v` has been introduced for building images > diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass > index 92d06e1b..97cf9714 100644 > --- a/meta/classes-recipe/dpkg-source.bbclass > +++ b/meta/classes-recipe/dpkg-source.bbclass > @@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \ > DPKG_SOURCE_EXTRA_ARGS ?= "-I" > > DEBIAN_SOURCE ?= "${BPN}" > +SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg" > +DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}" > > +do_dpkg_source[cleandirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}" > +do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}" > do_dpkg_source() { > # Create a .dsc file from source directory to use it with sbuild > DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) > @@ -22,9 +27,19 @@ do_dpkg_source() { > fi > find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete > sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}" > + # move packages to deploy directory > + find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \; > } > addtask dpkg_source after do_prepare_build > > +SSTATETASKS += "do_dpkg_source" > + > +python do_dpkg_source_setscene() { > + sstate_setscene(d) > +} > + > +addtask dpkg_source_setscene > + > CLEANFUNCS += "deb_clean_source" > > deb_clean_source() { > @@ -34,11 +49,11 @@ deb_clean_source() { > > do_deploy_source[depends] += "isar-apt:do_cache_config" > do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > -do_deploy_source[dirs] = "${S}" > +do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}" > do_deploy_source() { > repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}" > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc") > if [ -n "${DSC_FILE}" ]; then > repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \ > "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \ > diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass > index d8e129bb..a42703d5 100644 > --- a/meta/classes-recipe/dpkg.bbclass > +++ b/meta/classes-recipe/dpkg.bbclass > @@ -116,7 +116,7 @@ dpkg_runbuild() { > echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG} > echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG} > > - DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > + DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print) > > # networking is automatically enabled on older versions of sbuild > sbuild_network_option="" Tested-by: Andreas Ziegler <ziegler.andreas@siemens.com> Thanks! Andreas -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/2f561e79-2776-4c70-b36b-7e4829884fd6%40siemens.com. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-08-06 8:39 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-08-06 7:24 [PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate 'Felix Moessbauer' via isar-users 2026-08-06 7:24 ` [PATCH v2 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source 'Felix Moessbauer' via isar-users 2026-08-06 8:19 ` 'Andreas Ziegler' via isar-users 2026-08-06 7:24 ` [PATCH v2 2/2] dpkg: cache do_dpkg_source results in sstate 'Felix Moessbauer' via isar-users 2026-08-06 8:20 ` 'Andreas Ziegler' via isar-users
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox