[PATCH v3 0/3] Don't update apt database on every package build

[PATCH v3 0/3] Don't update apt database on every package build

[Uladzimir Bely]

Here we want to avoid the default behaviour of sbuild when it tries
to update all apt sources.

To have package database consistent, we should rely on one we got
during the debootstrap. The only repo that really needs updates
is local `isar-apt` where custom build depends may be placed.

Also, `isar-apt` should have the high priority over other sources
and allow downgrades (like it was done earlier for buildchroot).

Changes since v2:
 - Allow downgrades when installing package dependencies

Changes since v1:
 - Disable only `apt update`, but leave `apt distupgrade` unchanged
(enabled by default). This allows package upgrades from isar-apt repo.
 - Added a patch that sets high priority for isar-apt.

Uladzimir Bely (3):
  sbuild: Don't update apt database on every package build
  sbuild: Set high prio for the packages from isar-apt repo
  sbuild: Allow downgrades when installing dependencies

 meta/classes/dpkg.bbclass | 3 +++
 1 file changed, 3 insertions(+)

-- 
2.20.1

[PATCH v3 1/3] sbuild: Don't update apt database on every package build

[Uladzimir Bely]

By default, sbuild updates apt database before the build. It may cause
an inconsistency between apt databases for different packages built at
the different time if external (Debian) mirrors has updated recently.

With new changes, sbuild will rely on global apt state that was got
during debootstrap stage.

Local `isar-apt` repo will still be automatically updated while it is
passed via `--extra-repository` argument.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c92ea7db..dfa097f8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -90,6 +90,7 @@ dpkg_runbuild() {
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
+        --no-apt-update \
         --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
-- 
2.20.1

[PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo

[Uladzimir Bely]

This sets high priority for `isar-apt` repo during sbuild session.
So, even if some dependency in other repos has higher version than
one we previously built and placed in isar-apt, the second one wil
be selected.

This repeats the behaviour that is implemented in rootfs.bbclass
via `rootfs_configure_isar_apt` that is disabled for sbuild chroot.

We can't simply enable this task for sbuild while we don't want to
have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot
while it's handled via sbuild by `--extra-repository` option

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index dfa097f8..c7e7dd34 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -91,6 +91,7 @@ dpkg_runbuild() {
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
         --no-apt-update \
+        --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
         --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
-- 
2.20.1

[PATCH v3 3/3] sbuild: Allow downgrades when installing dependencies

[Uladzimir Bely]

If the package we are building depends on previously built custom
package, but sbuild chroot happens to have it already installed,
we need to allow downgrades explicitly.

Related commits: 5367ba74d9f2 and 77e62c1e9bbd (for buildchroot).

Actually, the issue is not so typical for sbuild (in comparisong to
previously used buildchroot) while sbuild base chroot keeps the bare
minimum of preinstalled packages.

Anyway, the fix might be needed in case we want to recompile some
essential package and use it as a dependency for another one.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c7e7dd34..1f55e3e7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -92,6 +92,7 @@ dpkg_runbuild() {
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
         --no-apt-update \
         --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
+        --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
         --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
-- 
2.20.1

Re: [PATCH v3 0/3] Don't update apt database on every package build

[Anton Mikanovich]

31.10.2022 09:06, Uladzimir Bely wrote:
> Here we want to avoid the default behaviour of sbuild when it tries
> to update all apt sources.
>
> To have package database consistent, we should rely on one we got
> during the debootstrap. The only repo that really needs updates
> is local `isar-apt` where custom build depends may be placed.
>
> Also, `isar-apt` should have the high priority over other sources
> and allow downgrades (like it was done earlier for buildchroot).
>
> Changes since v2:
>   - Allow downgrades when installing package dependencies
>
> Changes since v1:
>   - Disable only `apt update`, but leave `apt distupgrade` unchanged
> (enabled by default). This allows package upgrades from isar-apt repo.
>   - Added a patch that sets high priority for isar-apt.
>
> Uladzimir Bely (3):
>    sbuild: Don't update apt database on every package build
>    sbuild: Set high prio for the packages from isar-apt repo
>    sbuild: Allow downgrades when installing dependencies
>
>   meta/classes/dpkg.bbclass | 3 +++
>   1 file changed, 3 insertions(+)
>
Applied to next, thanks.

Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo

[Jan Kiszka]

On 31.10.22 07:06, Uladzimir Bely wrote:
> This sets high priority for `isar-apt` repo during sbuild session.
> So, even if some dependency in other repos has higher version than
> one we previously built and placed in isar-apt, the second one wil
> be selected.
> 
> This repeats the behaviour that is implemented in rootfs.bbclass
> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot.
> 
> We can't simply enable this task for sbuild while we don't want to
> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot
> while it's handled via sbuild by `--extra-repository` option
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/dpkg.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index dfa097f8..c7e7dd34 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -91,6 +91,7 @@ dpkg_runbuild() {
>          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
>          --no-apt-update \
> +        --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
>          --chroot-setup-commands="rm -f /var/log/dpkg.log" \
>          --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
>          --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \

This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as
far as I understand the logic so far. I didn't understand yet how
isar-apt packages are otherwise injected into the build env, but this is
not working at all. What definitely works better here is properly
mounting isar-apt into the schroot via its config.

BTW, SCHROOT_MOUNTS and insert/remove_mounts seems like dead code as
well. dpkg-gbp tries to make use of it but must be fine without it.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo

[Jan Kiszka]

On 11.01.23 15:36, Jan Kiszka wrote:
> On 31.10.22 07:06, Uladzimir Bely wrote:
>> This sets high priority for `isar-apt` repo during sbuild session.
>> So, even if some dependency in other repos has higher version than
>> one we previously built and placed in isar-apt, the second one wil
>> be selected.
>>
>> This repeats the behaviour that is implemented in rootfs.bbclass
>> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot.
>>
>> We can't simply enable this task for sbuild while we don't want to
>> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot
>> while it's handled via sbuild by `--extra-repository` option
>>
>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>> ---
>>  meta/classes/dpkg.bbclass | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index dfa097f8..c7e7dd34 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -91,6 +91,7 @@ dpkg_runbuild() {
>>          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
>>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
>>          --no-apt-update \
>> +        --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
>>          --chroot-setup-commands="rm -f /var/log/dpkg.log" \
>>          --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
>>          --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
> 
> This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as
> far as I understand the logic so far. I didn't understand yet how
> isar-apt packages are otherwise injected into the build env, but this is
> not working at all.

Finally found it: sbuild --extra-repository ...

> What definitely works better here is properly
> mounting isar-apt into the schroot via its config.

That was too fast, still no preferences in effect here.

Could it be that we would need some "--extra-preferences" (which do not
exist) because sbuild creates a clean /etc/apt for the build, ignoring
the preferences in the schroot?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo

[Jan Kiszka]

On 11.01.23 15:52, Jan Kiszka wrote:
> On 11.01.23 15:36, Jan Kiszka wrote:
>> On 31.10.22 07:06, Uladzimir Bely wrote:
>>> This sets high priority for `isar-apt` repo during sbuild session.
>>> So, even if some dependency in other repos has higher version than
>>> one we previously built and placed in isar-apt, the second one wil
>>> be selected.
>>>
>>> This repeats the behaviour that is implemented in rootfs.bbclass
>>> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot.
>>>
>>> We can't simply enable this task for sbuild while we don't want to
>>> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot
>>> while it's handled via sbuild by `--extra-repository` option
>>>
>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>> ---
>>>  meta/classes/dpkg.bbclass | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>> index dfa097f8..c7e7dd34 100644
>>> --- a/meta/classes/dpkg.bbclass
>>> +++ b/meta/classes/dpkg.bbclass
>>> @@ -91,6 +91,7 @@ dpkg_runbuild() {
>>>          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
>>>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
>>>          --no-apt-update \
>>> +        --chroot-setup-commands="echo \"Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: 1000\" > /etc/apt/preferences.d/isar-apt" \
>>>          --chroot-setup-commands="rm -f /var/log/dpkg.log" \
>>>          --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
>>>          --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
>>
>> This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as
>> far as I understand the logic so far. I didn't understand yet how
>> isar-apt packages are otherwise injected into the build env, but this is
>> not working at all.
> 
> Finally found it: sbuild --extra-repository ...
> 
>> What definitely works better here is properly
>> mounting isar-apt into the schroot via its config.
> 
> That was too fast, still no preferences in effect here.
> 
> Could it be that we would need some "--extra-preferences" (which do not
> exist) because sbuild creates a clean /etc/apt for the build, ignoring
> the preferences in the schroot?

No, preferences are working. I'm "just" seeing local conflicts of
incomplete essential package rebuilds (missing archs) that were surfaced
by my latest sbuild-chroot optimizations.

So we are just left with the inconsistency in the devshell which sent me
down the wrong road. Patch will follow.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH v3 2/3] sbuild: Set high prio for the packages from isar-apt repo

[Uladzimir Bely]

In the email from Wednesday, 11 January 2023 20:34:27 +03 user Jan Kiszka 
wrote:
> On 11.01.23 15:52, Jan Kiszka wrote:
> > On 11.01.23 15:36, Jan Kiszka wrote:
> >> On 31.10.22 07:06, Uladzimir Bely wrote:
> >>> This sets high priority for `isar-apt` repo during sbuild session.
> >>> So, even if some dependency in other repos has higher version than
> >>> one we previously built and placed in isar-apt, the second one wil
> >>> be selected.
> >>> 
> >>> This repeats the behaviour that is implemented in rootfs.bbclass
> >>> via `rootfs_configure_isar_apt` that is disabled for sbuild chroot.
> >>> 
> >>> We can't simply enable this task for sbuild while we don't want to
> >>> have `/etc/apt/sources.list.d/isar-apt.list` file in sbuild chroot
> >>> while it's handled via sbuild by `--extra-repository` option
> >>> 
> >>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>> ---
> >>> 
> >>>  meta/classes/dpkg.bbclass | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>> 
> >>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >>> index dfa097f8..c7e7dd34 100644
> >>> --- a/meta/classes/dpkg.bbclass
> >>> +++ b/meta/classes/dpkg.bbclass
> >>> @@ -91,6 +91,7 @@ dpkg_runbuild() {
> >>> 
> >>>          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles}
> >>>          \
> >>>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest
> >>>          --resolve-alternatives \ --no-apt-update \
> >>> 
> >>> +        --chroot-setup-commands="echo \"Package: *\nPin: release
> >>> n=${DEBDISTRONAME}\nPin-Priority: 1000\" >
> >>> /etc/apt/preferences.d/isar-apt" \>>> 
> >>>          --chroot-setup-commands="rm -f /var/log/dpkg.log" \
> >>>          --chroot-setup-commands="cp -n --no-preserve=owner
> >>>          ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
> >>>          --finished-build-commands="rm -f
> >>>          ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \>> 
> >> This remains a nop, doesn't it? There is no isar-apt in sbuild-chroot as
> >> far as I understand the logic so far. I didn't understand yet how
> >> isar-apt packages are otherwise injected into the build env, but this is
> >> not working at all.
> > 
> > Finally found it: sbuild --extra-repository ...
> > 
> >> What definitely works better here is properly
> >> mounting isar-apt into the schroot via its config.
> > 
> > That was too fast, still no preferences in effect here.
> > 
> > Could it be that we would need some "--extra-preferences" (which do not
> > exist) because sbuild creates a clean /etc/apt for the build, ignoring
> > the preferences in the schroot?
> 

Hi.
Yes, sbuild uses clean /etc/apt, so this additional command was required when 
I was fixing the "isar-apt prio" issue. For "pure" schroot there might be 
another way to inherit apt preferences, but I didn't test how it would works 
in devshell.

> No, preferences are working. I'm "just" seeing local conflicts of
> incomplete essential package rebuilds (missing archs) that were surfaced
> by my latest sbuild-chroot optimizations.
> 
> So we are just left with the inconsistency in the devshell which sent me
> down the wrong road. Patch will follow.
> 
> Jan