From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7319406651323187200 X-Received: by 2002:a7b:c8c9:0:b0:40e:4683:9d69 with SMTP id f9-20020a7bc8c9000000b0040e46839d69mr2049516wml.132.1705910350181; Sun, 21 Jan 2024 23:59:10 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1394:b0:40d:6f01:c37e with SMTP id u20-20020a05600c139400b0040d6f01c37els124191wmf.2.-pod-prod-06-eu; Sun, 21 Jan 2024 23:59:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEqx8otNX/d7z3wPThge16W4UKfTZMadXSKfEWYZrmTgOhqA6z5gOm7Fi708u24ou1UL5y X-Received: by 2002:a05:600c:5118:b0:40d:3a32:7d51 with SMTP id o24-20020a05600c511800b0040d3a327d51mr1913911wms.119.1705910347823; Sun, 21 Jan 2024 23:59:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705910347; cv=none; d=google.com; s=arc-20160816; b=rX7Vl0Bf6lMd6U9thdNIiTtVpDj/EATFX42k/ET+/OQ3hvj+YPdSud8NEvFoc737ce caRcOGAxsBKT+kvObrlNgkCYjjGEcSsnsIwZ3jyUac6XDdHc+PjLDazuf8Y5OI+QpQzu ouuUQO3JUlmOo6RBAstaf2erVuHtH/N13JkMzcmKeiy73aGtmPOC4DbZpeEqjidrnBgE a/bOlmSIuL59srD3wjT5YwRwMqHRTkOIgcMnLTcN4wJLz6WnQCVsRZn7Olw+HsYA2+9N yi0hF5mS93TZwTnSxltdKiNyGKoVD6xupvosqU+uSwXi3ePy0+vMkdf7l9J3MIHtFG+j TrzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id; bh=A4dLjRd+xu/sD7qTfMinOB+h5ScBmG4UEFbyH+cBm00=; fh=IxbeHM4vWxalf1XNf2rnGcQL+r1v0Lr0C6ICsGNNkYw=; b=nqKuG0IBkU/6/LlMbq3T4DxRqCqTG0iVdjPcGTxZWolZWpa3VfB5tD3+ErieFCKl86 z4mdcAUkYEKW3Y30CnNDPIt7gTlPnFIXCOtdcIZq5t4nDNNPpUtqBlRuQze+0DhTJX53 PTOQpmOrudtqIzmYwBYaeFYhmIsfZSukT5HRM1GOk5PSu0a5XsKIFAZ28I4CYzIqZhET 5ei0zgtV8gwG1uxJ3kcvz+J1IIOiZOqK2PjG1tmN+bEpV+S9LESqRhA+G0VgKKERsclC Wr8zb+5pQSE1px09Nx1q6dtkqnSgxi2oLQ3S1RHZpqvUvtwAkG68t9E0vLJDP7tthhLK H1Cw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id p24-20020a05600c1d9800b0040eaf035fadsi33858wms.2.2024.01.21.23.59.07 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 21 Jan 2024 23:59:07 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 40M7x57o027473 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 22 Jan 2024 08:59:06 +0100 Message-ID: Subject: Re: [PATCH v6] dpkg: Restore support for replacing pre-installed packages in sbuild-chroot From: Uladzimir Bely To: Jan Kiszka , isar-users Date: Mon, 22 Jan 2024 10:59:07 +0300 In-Reply-To: References: <024b7d92d8d9a9b1bd50466d5903086cda462c07.camel@ilbers.de> <0eacc107-7e89-4118-a9d8-f2e9713c64bc@siemens.com> <7767aec5-41f5-44ce-a292-6551c59a6ecb@siemens.com> <932bf7a167b931150719479296a509996c8b8536.camel@ilbers.de> <9f23f168-f5ee-44f5-aaa5-a242632dcbc3@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.50.3 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 5M2z3TuQkLUd On Fri, 2024-01-19 at 10:49 +0100, Jan Kiszka wrote: > On 19.01.24 10:42, Jan Kiszka wrote: > > On 19.01.24 10:15, Uladzimir Bely wrote: > > > On Fri, 2024-01-19 at 08:49 +0100, Jan Kiszka wrote: > > > > On 19.01.24 08:44, Jan Kiszka wrote: > > > > > On 19.01.24 08:40, Jan Kiszka wrote: > > > > > > On 19.01.24 08:37, Uladzimir Bely wrote: > > > > > > > On Fri, 2024-01-19 at 10:20 +0300, Uladzimir Bely wrote: > > > > > > > > On Thu, 2024-01-18 at 14:50 +0100, Jan Kiszka wrote: > > > > > > > > > From: Jan Kiszka > > > > > > > > >=20 > > > > > > > > > During the migration to sbuild, support for using > > > > > > > > > self- > > > > > > > > > built > > > > > > > > > packages > > > > > > > > > in > > > > > > > > > the build environment was lost if those were already > > > > > > > > > part > > > > > > > > > of the > > > > > > > > > sbuild-chroot. This restores it by adding --apt- > > > > > > > > > distupgrade > > > > > > > > > to the > > > > > > > > > sbuild call. But that is not enough because sbuild > > > > > > > > > will > > > > > > > > > only > > > > > > > > > upgrade > > > > > > > > > packages from already configured sources, not those > > > > > > > > > specified via > > > > > > > > > --extra-repository. We therefore have to switch back > > > > > > > > > to > > > > > > > > > configuring > > > > > > > > > isar-apt during sbuild-chroot creation. > > > > > > > > >=20 > > > > > > > > > As rootfs_configure_isar_apt configures the isar repo > > > > > > > > > under > > > > > > > > > /isar-apt, we bind-mount the one in > > > > > > > > > /home/builder/${PN} to > > > > > > > > > that > > > > > > > > > folder. > > > > > > > > > Another difference is that we now need to run apt-get > > > > > > > > > update > > > > > > > > > explicitly, > > > > > > > > > but only for isar-apt. > > > > > > > > >=20 > > > > > > > > > For apt fetching, we neither need nor want isar-apt > > > > > > > > > to be > > > > > > > > > available. > > > > > > > > > Rebuilding self-generated apt packages is generally > > > > > > > > > pointless as > > > > > > > > > the > > > > > > > > > needs are better addressed in the generating recipe > > > > > > > > > itself. > > > > > > > > > Exposing > > > > > > > > > isar-apt to the fetch may furthermore lead to > > > > > > > > > fetching a > > > > > > > > > previously > > > > > > > > > built source package of the same recipe, rather than > > > > > > > > > pulling the > > > > > > > > > external version. > > > > > > > > >=20 > > > > > > > > > And because the sbuild-chroots are now left behind > > > > > > > > > with > > > > > > > > > isar-apt > > > > > > > > > configured, the configuration in imager_run can be > > > > > > > > > removed > > > > > > > > > (credits > > > > > > > > > to > > > > > > > > > Srinuvasan Arjunan). > > > > > > > > >=20 > > > > > > > > > Signed-off-by: Jan Kiszka > > > > > > > > > --- > > > > > > > > >=20 > > > > > > > > > Changes in v6: > > > > > > > > > =C2=A0- basically, going back to v4 but disabling isar-ap= t > > > > > > > > > in > > > > > > > > > do_apt_fetch > > > > > > > > >=20 > > > > > > > > > I was able to resolve the scenario that Uladzimir was > > > > > > > > > sharing this > > > > > > > > > way.=20 > > > > > > > > > And I also realized that apt fetching as well as > > > > > > > > > unpacking > > > > > > > > > should > > > > > > > > > have=20 > > > > > > > > > no deal with isar-apt. So, v5 was taking a wrong > > > > > > > > > turn. > > > > > > > > >=20 > > > > > > > > > =C2=A0meta/classes/dpkg-base.bbclass=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 29 > > > > > > > > > +++++++++++---- > > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > =C2=A0meta/classes/dpkg.bbclass=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 |=C2=A0 5 > > > > > > > > > ++-- > > > > > > > > > =C2=A0meta/classes/image-tools-extension.bbclass=C2=A0=C2= =A0=C2=A0 | 13 - > > > > > > > > > ------ > > > > > > > > > -- > > > > > > > > > =C2=A0.../sbuild-chroot/sbuild-chroot.inc=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 2 - > > > > > > > > > - > > > > > > > > > =C2=A04 files changed, 20 insertions(+), 29 deletions(-) > > > > > > > > >=20 > > > > > > > > > diff --git a/meta/classes/dpkg-base.bbclass > > > > > > > > > b/meta/classes/dpkg- > > > > > > > > > base.bbclass > > > > > > > > > index 7b054d3f..80686677 100644 > > > > > > > > > --- a/meta/classes/dpkg-base.bbclass > > > > > > > > > +++ b/meta/classes/dpkg-base.bbclass > > > > > > > > > @@ -19,8 +19,6 @@ DEPENDS:append:riscv64 =3D "${@' > > > > > > > > > crossbuild- > > > > > > > > > essential-riscv64' if d.getVar('ISAR_C > > > > > > > > > =C2=A0DEB_BUILD_PROFILES ?=3D "" > > > > > > > > > =C2=A0DEB_BUILD_OPTIONS ?=3D "" > > > > > > > > > =C2=A0 > > > > > > > > > -ISAR_APT_REPO ?=3D "deb [trusted=3Dyes] > > > > > > > > > file:///home/builder/${PN}/isar- > > > > > > > > > apt/${DISTRO}- > > > > > > > > > ${DISTRO_ARCH}/apt/${DISTRO}=C2=A0${DEBDISTRONAME} > > > > > > > > > main" > > > > > > > > > - > > > > > > > > > =C2=A0python do_adjust_git() { > > > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 import subprocess > > > > > > > > > =C2=A0 > > > > > > > > > @@ -115,6 +113,8 @@ do_apt_fetch() { > > > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM = ALRM USR1 > > > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > > > > > > > > > =C2=A0 > > > > > > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHRO= OT} -- \ > > > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/s= ources.list.d/isar-apt.list > > > > > > > > > /etc/apt/preferences.d/isar-apt > > > > > > > >=20 > > > > > > > > The issue is still reproducible with the steps I posted > > > > > > > > below, as > > > > > > > > well > > > > > > > > as CI still fails with v6. > > > > > > > > I think, the steps above are not sufficient to > > > > > > > > completely > > > > > > > > stop > > > > > > > > dealing > > > > > > > > with isar-apt in do_apt_fetch() and we need some kind > > > > > > > > of 'apt > > > > > > > > update' > > > > > > > > here to consider removed list and preferences. The > > > > > > > > question > > > > > > > > is how to > > > > > > > > say apt to forget about isar-apt and do not run apt > > > > > > > > update > > > > > > > > for other > > > > > > > > source lists... maybe, simply remove isar-apt files > > > > > > > > from > > > > > > > > /var/lib/apt/lists/. > > > > > > >=20 > > > > > > > An addition: > > > > > > >=20 > > > > > > > It seems that files are not really removed from the > > > > > > > session for > > > > > > > some > > > > > > > reason. I added some kind of "ls" near remove code: > > > > > > >=20 > > > > > > > ``` > > > > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} = -- \ > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c 'ls -la /et= c/apt/sources.list.d/' > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHR= OOT} -- \ > > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/= sources.list.d/isar-apt.list > > > > > > > /etc/apt/preferences.d/isar-apt > > > > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} = -- \ > > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c 'ls -la /et= c/apt/sources.list.d/' > > > > > > > ``` > > > > > > > , but still see the following in the log: > > > > > > > ``` > > > > > > > > DEBUG: Executing shell function do_apt_fetch > > > > > > > > total 16 > > > > > > > > drwxr-xr-x 2 root root 4096 Jan 19 07:09 . > > > > > > > > drwxr-xr-x 8 root root 4096 Jan 19 07:09 .. > > > > > > > > -rw-r--r-- 1 root root=C2=A0 569 Jan 19 07:09 bootstrap.lis= t > > > > > > > > -rw-r--r-- 1 root root=C2=A0=C2=A0 94 Jan 19 07:09 isar-apt= .list > > > > > > > > total 16 > > > > > > > > drwxr-xr-x 2 root root 4096 Jan 19 07:09 . > > > > > > > > drwxr-xr-x 8 root root 4096 Jan 19 07:09 .. > > > > > > > > -rw-r--r-- 1 root root=C2=A0 569 Jan 19 07:09 bootstrap.lis= t > > > > > > > > -rw-r--r-- 1 root root=C2=A0=C2=A0 94 Jan 19 07:09 isar-apt= .list > > > > > > > > Reading package lists... > > > > > > > ``` > > > > > > > Fith the apt_fetch failure followed, because of existing > > > > > > > of > > > > > > > previously > > > > > > > built hello source package in the isar-apt repo. > > > > > > >=20 > > > > > >=20 > > > > > > OK... last-minute change of mine, splitting up the purging > > > > > > and > > > > > > the > > > > > > actual fetching into two schroot runs to avoid having to > > > > > > run the > > > > > > latter > > > > > > with "-u root". I didn't find a way to de-configure isar- > > > > > > apt > > > > > > unprivileged. Back to the drawing board. :( > > > > > >=20 > > > > >=20 > > > > > Comparing the split run of schroot with image-tool-extension, > > > > > I > > > > > think > > > > > I'm just missing a proper session ID here. > > > > >=20 > > > >=20 > > > > Before sending v7: can you try this on top? > > > >=20 > > > > diff --git a/meta/classes/dpkg-base.bbclass > > > > b/meta/classes/dpkg- > > > > base.bbclass > > > > index 80686677..1a3333ac 100644 > > > > --- a/meta/classes/dpkg-base.bbclass > > > > +++ b/meta/classes/dpkg-base.bbclass > > > > @@ -113,10 +113,13 @@ do_apt_fetch() { > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > > > > =C2=A0 > > > > -=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -- \ > > > > +=C2=A0=C2=A0=C2=A0 session_id=3D$(schroot -q -b -c ${SBUILD_CHROOT= }) > > > > +=C2=A0=C2=A0=C2=A0 echo "Started session: ${session_id}" > > > > + > > > > +=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -u root -- \ > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/source= s.list.d/isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > =C2=A0=C2=A0=C2=A0=C2=A0 for uri in "${SRC_APT}"; do > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -d / -c ${SBUIL= D_CHROOT} -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c ${session= _id} -d / -- \ > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd > > > > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only -- > > > > only- > > > > source source "$2"' my_script "${BASE_DISTRO}- > > > > ${BASE_DISTRO_CODENAME}" "${uri}" > > > > =C2=A0=C2=A0=C2=A0=C2=A0 done > > > > =C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs > > > >=20 > > > > Jan > > > >=20 > > >=20 > > > Yes, with persistent session it works, at least with reproduce > > > steps I > > > used. I just added session cleanup (e.g, "schroot -e -c > > > ${session_id}") > > > at the end. > > >=20 > > > Before sending v7 I could also check if it is not reproduced > > > anymore in > > > CI. > > >=20 > > > Also, due to using schroot session we probably also need to care > > > about > > > proper session removing in case commands under schroot fail, > > > similar to > > > how it's done in image-tools-extention by imager_cleanup(). > >=20 > > Yeah, still need to check that - schroot is also still new to me, > > learned a lot about it these days. > >=20 > > Jan > >=20 >=20 > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg- > base.bbclass > index 80686677..85d0a495 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -107,18 +107,24 @@ do_apt_fetch() { > =C2=A0=C2=A0=C2=A0=C2=A0 E=3D"${@ isar_export_proxies(d)}" > =C2=A0=C2=A0=C2=A0=C2=A0 schroot_create_configs > =C2=A0 > +=C2=A0=C2=A0=C2=A0 session_id=3D$(schroot -q -b -c ${SBUILD_CHROOT}) > +=C2=A0=C2=A0=C2=A0 echo "Started session: ${session_id}" > + > =C2=A0=C2=A0=C2=A0=C2=A0 schroot_cleanup() { > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -q -f -e -c ${session= _id} > /dev/null 2>&1 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs > =C2=A0=C2=A0=C2=A0=C2=A0 } > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > =C2=A0 > -=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -- \ > +=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -u root -- \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.list= .d/isar-apt.list > /etc/apt/preferences.d/isar-apt > =C2=A0=C2=A0=C2=A0=C2=A0 for uri in "${SRC_APT}"; do > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -d / -c ${SBUILD_CHRO= OT} -- \ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -= d / -- \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only- > source source "$2"' my_script "${BASE_DISTRO}- > ${BASE_DISTRO_CODENAME}" "${uri}" > =C2=A0=C2=A0=C2=A0=C2=A0 done > + > +=C2=A0=C2=A0=C2=A0 schroot -e -c ${session_id} > =C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs > =C2=A0} > =C2=A0 >=20 > Jan >=20 Hello. Checked in CI during the weekend - the issue seems not to be reproduced anymore. So, v7 should be OK.