* [PATCH v3 0/3] Don't update apt database on every package build
@ 2022-10-31 6:06 Uladzimir Bely
2022-10-31 6:06 ` [PATCH v3 1/3] sbuild: " Uladzimir Bely
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Uladzimir Bely @ 2022-10-31 6:06 UTC (permalink / raw)
To: isar-users
Here we want to avoid the default behaviour of sbuild when it tries
to update all apt sources.
To have package database consistent, we should rely on one we got
during the debootstrap. The only repo that really needs updates
is local `isar-apt` where custom build depends may be placed.
Also, `isar-apt` should have the high priority over other sources
and allow downgrades (like it was done earlier for buildchroot).
Changes since v2:
- Allow downgrades when installing package dependencies
Changes since v1:
- Disable only `apt update`, but leave `apt distupgrade` unchanged
(enabled by default). This allows package upgrades from isar-apt repo.
- Added a patch that sets high priority for isar-apt.
Uladzimir Bely (3):
sbuild: Don't update apt database on every package build
sbuild: Set high prio for the packages from isar-apt repo
sbuild: Allow downgrades when installing dependencies
meta/classes/dpkg.bbclass | 3 +++
1 file changed, 3 insertions(+)
--
2.20.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v3 1/3] sbuild: Don't update apt database on every package build 2022-10-31 6:06 [PATCH v3 0/3] Don't update apt database on every package build Uladzimir Bely @ 2022-10-31 6:06 ` Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo Uladzimir Bely ` (2 subsequent siblings) 3 siblings, 0 replies; 9+ messages in thread From: Uladzimir Bely @ 2022-10-31 6:06 UTC (permalink / raw) To: isar-users By default, sbuild updates apt database before the build. It may cause an inconsistency between apt databases for different packages built at the different time if external (Debian) mirrors has updated recently. With new changes, sbuild will rely on global apt state that was got during debootstrap stage. Local `isar-apt` repo will still be automatically updated while it is passed via `--extra-repository` argument. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg.bbclass | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index c92ea7db..dfa097f8 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -90,6 +90,7 @@ dpkg_runbuild() { sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ + --no-apt-update \ --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo 2022-10-31 6:06 [PATCH v3 0/3] Don't update apt database on every package build Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 1/3] sbuild: " Uladzimir Bely @ 2022-10-31 6:06 ` Uladzimir Bely 2023-01-11 14:36 ` Jan Kiszka 2022-10-31 6:06 ` [PATCH v3 3/3] sbuild: Allow downgrades when installing dependencies Uladzimir Bely 2022-11-15 7:49 ` [PATCH v3 0/3] Don't update apt database on every package build Anton Mikanovich 3 siblings, 1 reply; 9+ messages in thread From: Uladzimir Bely @ 2022-10-31 6:06 UTC (permalink / raw) To: isar-users This sets high priority for `isar-apt` repo during sbuild session. So, even if some dependency in other repos has higher version than one we previously built and placed in isar-apt, the second one wil be selected. This repeats the behaviour that is implemented in rootfs.bbclass via `rootfs_configure_isar_apt` that is disabled for sbuild chroot. We can't simply enable this task for sbuild while we don't want to have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot while it's handled via sbuild by `--extra-repository` option Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg.bbclass | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index dfa097f8..c7e7dd34 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -91,6 +91,7 @@ dpkg_runbuild() { --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ --no-apt-update \ + --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo 2022-10-31 6:06 ` [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo Uladzimir Bely @ 2023-01-11 14:36 ` Jan Kiszka 2023-01-11 14:52 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2023-01-11 14:36 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 31.10.22 07:06, Uladzimir Bely wrote: > This sets high priority for `isar-apt` repo during sbuild session. > So, even if some dependency in other repos has higher version than > one we previously built and placed in isar-apt, the second one wil > be selected. > > This repeats the behaviour that is implemented in rootfs.bbclass > via `rootfs_configure_isar_apt` that is disabled for sbuild chroot. > > We can't simply enable this task for sbuild while we don't want to > have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot > while it's handled via sbuild by `--extra-repository` option > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > --- > meta/classes/dpkg.bbclass | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index dfa097f8..c7e7dd34 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -91,6 +91,7 @@ dpkg_runbuild() { > --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \ > --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ > --no-apt-update \ > + --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ > --chroot-setup-commands="rm -f /var/log/dpkg.log" \ > --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ > --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as far as I understand the logic so far. I didn't understand yet how isar-apt packages are otherwise injected into the build env, but this is not working at all. What definitely works better here is properly mounting isar-apt into the schroot via its config. BTW, SCHROOT_MOUNTS and insert/remove_mounts seems like dead code as well. dpkg-gbp tries to make use of it but must be fine without it. Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo 2023-01-11 14:36 ` Jan Kiszka @ 2023-01-11 14:52 ` Jan Kiszka 2023-01-11 17:34 ` Jan Kiszka 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2023-01-11 14:52 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 11.01.23 15:36, Jan Kiszka wrote: > On 31.10.22 07:06, Uladzimir Bely wrote: >> This sets high priority for `isar-apt` repo during sbuild session. >> So, even if some dependency in other repos has higher version than >> one we previously built and placed in isar-apt, the second one wil >> be selected. >> >> This repeats the behaviour that is implemented in rootfs.bbclass >> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot. >> >> We can't simply enable this task for sbuild while we don't want to >> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot >> while it's handled via sbuild by `--extra-repository` option >> >> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >> --- >> meta/classes/dpkg.bbclass | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >> index dfa097f8..c7e7dd34 100644 >> --- a/meta/classes/dpkg.bbclass >> +++ b/meta/classes/dpkg.bbclass >> @@ -91,6 +91,7 @@ dpkg_runbuild() { >> --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \ >> --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ >> --no-apt-update \ >> + --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ >> --chroot-setup-commands="rm -f /var/log/dpkg.log" \ >> --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ >> --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ > > This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as > far as I understand the logic so far. I didn't understand yet how > isar-apt packages are otherwise injected into the build env, but this is > not working at all. Finally found it: sbuild --extra-repository ... > What definitely works better here is properly > mounting isar-apt into the schroot via its config. That was too fast, still no preferences in effect here. Could it be that we would need some "--extra-preferences" (which do not exist) because sbuild creates a clean /etc/apt for the build, ignoring the preferences in the schroot? Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo 2023-01-11 14:52 ` Jan Kiszka @ 2023-01-11 17:34 ` Jan Kiszka 2023-01-12 7:46 ` Uladzimir Bely 0 siblings, 1 reply; 9+ messages in thread From: Jan Kiszka @ 2023-01-11 17:34 UTC (permalink / raw) To: Uladzimir Bely, isar-users On 11.01.23 15:52, Jan Kiszka wrote: > On 11.01.23 15:36, Jan Kiszka wrote: >> On 31.10.22 07:06, Uladzimir Bely wrote: >>> This sets high priority for `isar-apt` repo during sbuild session. >>> So, even if some dependency in other repos has higher version than >>> one we previously built and placed in isar-apt, the second one wil >>> be selected. >>> >>> This repeats the behaviour that is implemented in rootfs.bbclass >>> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot. >>> >>> We can't simply enable this task for sbuild while we don't want to >>> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot >>> while it's handled via sbuild by `--extra-repository` option >>> >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> >>> --- >>> meta/classes/dpkg.bbclass | 1 + >>> 1 file changed, 1 insertion(+) >>> >>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >>> index dfa097f8..c7e7dd34 100644 >>> --- a/meta/classes/dpkg.bbclass >>> +++ b/meta/classes/dpkg.bbclass >>> @@ -91,6 +91,7 @@ dpkg_runbuild() { >>> --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \ >>> --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ >>> --no-apt-update \ >>> + --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ >>> --chroot-setup-commands="rm -f /var/log/dpkg.log" \ >>> --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ >>> --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ >> >> This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as >> far as I understand the logic so far. I didn't understand yet how >> isar-apt packages are otherwise injected into the build env, but this is >> not working at all. > > Finally found it: sbuild --extra-repository ... > >> What definitely works better here is properly >> mounting isar-apt into the schroot via its config. > > That was too fast, still no preferences in effect here. > > Could it be that we would need some "--extra-preferences" (which do not > exist) because sbuild creates a clean /etc/apt for the build, ignoring > the preferences in the schroot? No, preferences are working. I'm "just" seeing local conflicts of incomplete essential package rebuilds (missing archs) that were surfaced by my latest sbuild-chroot optimizations. So we are just left with the inconsistency in the devshell which sent me down the wrong road. Patch will follow. Jan -- Siemens AG, Technology Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo 2023-01-11 17:34 ` Jan Kiszka @ 2023-01-12 7:46 ` Uladzimir Bely 0 siblings, 0 replies; 9+ messages in thread From: Uladzimir Bely @ 2023-01-12 7:46 UTC (permalink / raw) To: isar-users, Jan Kiszka In the email from Wednesday, 11 January 2023 20:34:27 +03 user Jan Kiszka wrote: > On 11.01.23 15:52, Jan Kiszka wrote: > > On 11.01.23 15:36, Jan Kiszka wrote: > >> On 31.10.22 07:06, Uladzimir Bely wrote: > >>> This sets high priority for `isar-apt` repo during sbuild session. > >>> So, even if some dependency in other repos has higher version than > >>> one we previously built and placed in isar-apt, the second one wil > >>> be selected. > >>> > >>> This repeats the behaviour that is implemented in rootfs.bbclass > >>> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot. > >>> > >>> We can't simply enable this task for sbuild while we don't want to > >>> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot > >>> while it's handled via sbuild by `--extra-repository` option > >>> > >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > >>> --- > >>> > >>> meta/classes/dpkg.bbclass | 1 + > >>> 1 file changed, 1 insertion(+) > >>> > >>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > >>> index dfa097f8..c7e7dd34 100644 > >>> --- a/meta/classes/dpkg.bbclass > >>> +++ b/meta/classes/dpkg.bbclass > >>> @@ -91,6 +91,7 @@ dpkg_runbuild() { > >>> > >>> --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} > >>> \ > >>> --no-run-lintian --no-run-piuparts --no-run-autopkgtest > >>> --resolve-alternatives \ --no-apt-update \ > >>> > >>> + --chroot-setup-commands="echo \"Package: *\nPin: release > >>> n=${DEBDISTRONAME}\nPin-Priority: 1000\" > > >>> /etc/apt/preferences.d/isar-apt" \>>> > >>> --chroot-setup-commands="rm -f /var/log/dpkg.log" \ > >>> --chroot-setup-commands="cp -n --no-preserve=owner > >>> ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ > >>> --finished-build-commands="rm -f > >>> ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \>> > >> This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as > >> far as I understand the logic so far. I didn't understand yet how > >> isar-apt packages are otherwise injected into the build env, but this is > >> not working at all. > > > > Finally found it: sbuild --extra-repository ... > > > >> What definitely works better here is properly > >> mounting isar-apt into the schroot via its config. > > > > That was too fast, still no preferences in effect here. > > > > Could it be that we would need some "--extra-preferences" (which do not > > exist) because sbuild creates a clean /etc/apt for the build, ignoring > > the preferences in the schroot? > Hi. Yes, sbuild uses clean /etc/apt, so this additional command was required when I was fixing the "isar-apt prio" issue. For "pure" schroot there might be another way to inherit apt preferences, but I didn't test how it would works in devshell. > No, preferences are working. I'm "just" seeing local conflicts of > incomplete essential package rebuilds (missing archs) that were surfaced > by my latest sbuild-chroot optimizations. > > So we are just left with the inconsistency in the devshell which sent me > down the wrong road. Patch will follow. > > Jan ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 3/3] sbuild: Allow downgrades when installing dependencies 2022-10-31 6:06 [PATCH v3 0/3] Don't update apt database on every package build Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 1/3] sbuild: " Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo Uladzimir Bely @ 2022-10-31 6:06 ` Uladzimir Bely 2022-11-15 7:49 ` [PATCH v3 0/3] Don't update apt database on every package build Anton Mikanovich 3 siblings, 0 replies; 9+ messages in thread From: Uladzimir Bely @ 2022-10-31 6:06 UTC (permalink / raw) To: isar-users If the package we are building depends on previously built custom package, but sbuild chroot happens to have it already installed, we need to allow downgrades explicitly. Related commits: 5367ba74d9f2 and 77e62c1e9bbd (for buildchroot). Actually, the issue is not so typical for sbuild (in comparisong to previously used buildchroot) while sbuild base chroot keeps the bare minimum of preinstalled packages. Anyway, the fix might be needed in case we want to recompile some essential package and use it as a dependency for another one. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg.bbclass | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index c7e7dd34..1f55e3e7 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -92,6 +92,7 @@ dpkg_runbuild() { --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ --no-apt-update \ --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \ + --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \ --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ -- 2.20.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 0/3] Don't update apt database on every package build 2022-10-31 6:06 [PATCH v3 0/3] Don't update apt database on every package build Uladzimir Bely ` (2 preceding siblings ...) 2022-10-31 6:06 ` [PATCH v3 3/3] sbuild: Allow downgrades when installing dependencies Uladzimir Bely @ 2022-11-15 7:49 ` Anton Mikanovich 3 siblings, 0 replies; 9+ messages in thread From: Anton Mikanovich @ 2022-11-15 7:49 UTC (permalink / raw) To: Uladzimir Bely, isar-users 31.10.2022 09:06, Uladzimir Bely wrote: > Here we want to avoid the default behaviour of sbuild when it tries > to update all apt sources. > > To have package database consistent, we should rely on one we got > during the debootstrap. The only repo that really needs updates > is local `isar-apt` where custom build depends may be placed. > > Also, `isar-apt` should have the high priority over other sources > and allow downgrades (like it was done earlier for buildchroot). > > Changes since v2: > - Allow downgrades when installing package dependencies > > Changes since v1: > - Disable only `apt update`, but leave `apt distupgrade` unchanged > (enabled by default). This allows package upgrades from isar-apt repo. > - Added a patch that sets high priority for isar-apt. > > Uladzimir Bely (3): > sbuild: Don't update apt database on every package build > sbuild: Set high prio for the packages from isar-apt repo > sbuild: Allow downgrades when installing dependencies > > meta/classes/dpkg.bbclass | 3 +++ > 1 file changed, 3 insertions(+) > Applied to next, thanks. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-01-12 7:46 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-10-31 6:06 [PATCH v3 0/3] Don't update apt database on every package build Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 1/3] sbuild: " Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo Uladzimir Bely 2023-01-11 14:36 ` Jan Kiszka 2023-01-11 14:52 ` Jan Kiszka 2023-01-11 17:34 ` Jan Kiszka 2023-01-12 7:46 ` Uladzimir Bely 2022-10-31 6:06 ` [PATCH v3 3/3] sbuild: Allow downgrades when installing dependencies Uladzimir Bely 2022-11-15 7:49 ` [PATCH v3 0/3] Don't update apt database on every package build Anton Mikanovich
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox