Re: [PATCH v6] dpkg: Restore support for replacing pre-installed packages in sbuild-chroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 19.01.24 08:40, Jan Kiszka wrote:
> On 19.01.24 08:37, Uladzimir Bely wrote:
>> On Fri, 2024-01-19 at 10:20 +0300, Uladzimir Bely wrote:
>>> On Thu, 2024-01-18 at 14:50 +0100, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> During the migration to sbuild, support for using self-built
>>>> packages
>>>> in
>>>> the build environment was lost if those were already part of the
>>>> sbuild-chroot. This restores it by adding --apt-distupgrade to the
>>>> sbuild call. But that is not enough because sbuild will only
>>>> upgrade
>>>> packages from already configured sources, not those specified via
>>>> --extra-repository. We therefore have to switch back to configuring
>>>> isar-apt during sbuild-chroot creation.
>>>>
>>>> As rootfs_configure_isar_apt configures the isar repo under
>>>> /isar-apt, we bind-mount the one in /home/builder/${PN} to that
>>>> folder.
>>>> Another difference is that we now need to run apt-get update
>>>> explicitly,
>>>> but only for isar-apt.
>>>>
>>>> For apt fetching, we neither need nor want isar-apt to be
>>>> available.
>>>> Rebuilding self-generated apt packages is generally pointless as
>>>> the
>>>> needs are better addressed in the generating recipe itself.
>>>> Exposing
>>>> isar-apt to the fetch may furthermore lead to fetching a previously
>>>> built source package of the same recipe, rather than pulling the
>>>> external version.
>>>>
>>>> And because the sbuild-chroots are now left behind with isar-apt
>>>> configured, the configuration in imager_run can be removed (credits
>>>> to
>>>> Srinuvasan Arjunan).
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>
>>>> Changes in v6:
>>>>  - basically, going back to v4 but disabling isar-apt in
>>>> do_apt_fetch
>>>>
>>>> I was able to resolve the scenario that Uladzimir was sharing this
>>>> way. 
>>>> And I also realized that apt fetching as well as unpacking should
>>>> have 
>>>> no deal with isar-apt. So, v5 was taking a wrong turn.
>>>>
>>>>  meta/classes/dpkg-base.bbclass                | 29 +++++++++++----
>>>> --
>>>> --
>>>>  meta/classes/dpkg.bbclass                     |  5 ++--
>>>>  meta/classes/image-tools-extension.bbclass    | 13 ---------
>>>>  .../sbuild-chroot/sbuild-chroot.inc           |  2 --
>>>>  4 files changed, 20 insertions(+), 29 deletions(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-
>>>> base.bbclass
>>>> index 7b054d3f..80686677 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -19,8 +19,6 @@ DEPENDS:append:riscv64 = "${@' crossbuild-
>>>> essential-riscv64' if d.getVar('ISAR_C
>>>>  DEB_BUILD_PROFILES ?= ""
>>>>  DEB_BUILD_OPTIONS ?= ""
>>>>  
>>>> -ISAR_APT_REPO ?= "deb [trusted=yes]
>>>> file:///home/builder/${PN}/isar-
>>>> apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
>>>> -
>>>>  python do_adjust_git() {
>>>>      import subprocess
>>>>  
>>>> @@ -115,6 +113,8 @@ do_apt_fetch() {
>>>>      trap 'exit 1' INT HUP QUIT TERM ALRM USR1
>>>>      trap 'schroot_cleanup' EXIT
>>>>  
>>>> +    schroot -d / -u root -c ${SBUILD_CHROOT} -- \
>>>> +        rm /etc/apt/sources.list.d/isar-apt.list
>>>> /etc/apt/preferences.d/isar-apt
>>>
>>> The issue is still reproducible with the steps I posted below, as
>>> well
>>> as CI still fails with v6.
>>> I think, the steps above are not sufficient to completely stop
>>> dealing
>>> with isar-apt in do_apt_fetch() and we need some kind of 'apt update'
>>> here to consider removed list and preferences. The question is how to
>>> say apt to forget about isar-apt and do not run apt update for other
>>> source lists... maybe, simply remove isar-apt files from
>>> /var/lib/apt/lists/.
>>
>> An addition:
>>
>> It seems that files are not really removed from the session for some
>> reason. I added some kind of "ls" near remove code:
>>
>> ```
>> +    schroot -d / -u root -c ${SBUILD_CHROOT} -- \
>> +        sh -c 'ls -la /etc/apt/sources.list.d/'
>>      schroot -d / -u root -c ${SBUILD_CHROOT} -- \
>>          rm /etc/apt/sources.list.d/isar-apt.list
>> /etc/apt/preferences.d/isar-apt
>> +    schroot -d / -u root -c ${SBUILD_CHROOT} -- \
>> +        sh -c 'ls -la /etc/apt/sources.list.d/'
>> ```
>> , but still see the following in the log:
>> ```
>> | DEBUG: Executing shell function do_apt_fetch
>> | total 16
>> | drwxr-xr-x 2 root root 4096 Jan 19 07:09 .
>> | drwxr-xr-x 8 root root 4096 Jan 19 07:09 ..
>> | -rw-r--r-- 1 root root  569 Jan 19 07:09 bootstrap.list
>> | -rw-r--r-- 1 root root   94 Jan 19 07:09 isar-apt.list
>> | total 16
>> | drwxr-xr-x 2 root root 4096 Jan 19 07:09 .
>> | drwxr-xr-x 8 root root 4096 Jan 19 07:09 ..
>> | -rw-r--r-- 1 root root  569 Jan 19 07:09 bootstrap.list
>> | -rw-r--r-- 1 root root   94 Jan 19 07:09 isar-apt.list
>> | Reading package lists...
>> ```
>> Fith the apt_fetch failure followed, because of existing of previously
>> built hello source package in the isar-apt repo.
>>
> 
> OK... last-minute change of mine, splitting up the purging and the
> actual fetching into two schroot runs to avoid having to run the latter
> with "-u root". I didn't find a way to de-configure isar-apt
> unprivileged. Back to the drawing board. :(
> 

Comparing the split run of schroot with image-tool-extension, I think
I'm just missing a proper session ID here.

Jan

-- 
Siemens AG, Technology
Linux Expert Center