From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7319406651323187200 X-Received: by 2002:a19:e04d:0:b0:50e:6b57:4628 with SMTP id g13-20020a19e04d000000b0050e6b574628mr443663lfj.13.1705655760412; Fri, 19 Jan 2024 01:16:00 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4e82:0:b0:50e:82d5:4f2e with SMTP id o2-20020ac24e82000000b0050e82d54f2els407871lfr.1.-pod-prod-03-eu; Fri, 19 Jan 2024 01:15:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTi05jtZm0KJbidxSc5HVx1y/J08JJUPScOCceHscqGQv7d6LOX4sJsfdQe2mrVgvf7uno X-Received: by 2002:ac2:41c8:0:b0:50e:db6c:f014 with SMTP id d8-20020ac241c8000000b0050edb6cf014mr367730lfi.48.1705655757602; Fri, 19 Jan 2024 01:15:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1705655757; cv=none; d=google.com; s=arc-20160816; b=L9ZJMwVRFOTytf5XWfSXwEJeennV3wOcyRXIJzSEdSZ9QWApJNa5Z/oxpoG/uwZdmI KnSFFBYPrNYtiF720BC0xtK3bNiVxOwEo7EWdLBvtNAF90ITrelyugxCiR2cz5EEl9qj sGgaVyZx7lL0/Pl8x22JciEJbiZW7win0GCiTaHi75woU/0ZYFS9ON323B3EoAf/xasE ucZ3GNl7meBZMgly1loDZhKrXoZoxy/BrfCUkeWDnizQuBGjoF74+cicUtZWBrMfP2vd JTcHHdGnzcRsHGIq+F6rspcA4EZZ7vQAGhgd0BA7XBRHxLwmpi0LaIdWnjvM3GDqDZcC Ghwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:to:from:subject:message-id; bh=Vhzss/9GB/KvCtAL5bLtDuTGnF4AWvO0Xn62gfRNPbI=; fh=IxbeHM4vWxalf1XNf2rnGcQL+r1v0Lr0C6ICsGNNkYw=; b=0tXMJd6TPR3Wce+aWEG8meS2vNEnrJHo3bhGTEboAj20S03H5WBC3UVWJwOngsXsRk o1CkE1l5Lun544dB/aoLiIYn17sCBwIIieN7k655FFv+S+8Fl1M3Lbuo+UVkAwATmVXZ iuu/UTJR8FKEi4Sz3DkqFuB5TdekNU+kRGfD2n2klcwrOIH+EL6LFK0sm/cMGdmAeT8J 1kYVANAnKTYPalx7eZ1cjt5hfJOyMmPdIrbz4s/91NS3KwzQSuDljMgGVhnc64QU97pD RDdugR3rBjYtO5h42qCFx3K69nKRdpMo74yFSYBVo4W55y7bTOMdKpKCpjeaTFNVL1lx ring== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id h9-20020a056512220900b0050e69030a77si200787lfu.6.2024.01.19.01.15.57 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 19 Jan 2024 01:15:57 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [IPv6:::1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 40J9Ftm3013419 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 19 Jan 2024 10:15:56 +0100 Message-ID: <932bf7a167b931150719479296a509996c8b8536.camel@ilbers.de> Subject: Re: [PATCH v6] dpkg: Restore support for replacing pre-installed packages in sbuild-chroot From: Uladzimir Bely To: Jan Kiszka , isar-users Date: Fri, 19 Jan 2024 12:15:53 +0300 In-Reply-To: <7767aec5-41f5-44ce-a292-6551c59a6ecb@siemens.com> References: <024b7d92d8d9a9b1bd50466d5903086cda462c07.camel@ilbers.de> <0eacc107-7e89-4118-a9d8-f2e9713c64bc@siemens.com> <7767aec5-41f5-44ce-a292-6551c59a6ecb@siemens.com> Autocrypt: addr=ubely@ilbers.de; prefer-encrypt=mutual; keydata=mQENBGO2eUkBCACtT+T3OrPVSExBmqfgXT3lp9XcdxRzjYp26wezkgYjjBXaf36bxtaAfS471VoQtpar0RVeFfW7WDDdfX9ZclSj36zBQe+RVSJzoNoNQfjOXWuSHb5Z+cpAFtqBY4muxK4+iaIlLJd6CN3ejOsLHATtCeHHq8wi0z2T+KdLQO+wQRgo2hjj0Lp9pGTrKJry50HP/o7Vbdu14dOx2xqr8+wPc6SQbBIrcqaa4MqCQC00vQG7eXvo+k2MOw59FDdpMH0KR9mHgp3u/s4I+4YRBArukt9G9xz/rsEFmxAIBC6N/a6Hzwg4puc91n7ABDsPg8Vp+X3MDraujN0dvR6OKVNtABEBAAG0IFVsYWR6aW1pciBCZWx5IDx1YmVseUBpbGJlcnMuZGU+iQFOBBMBCAA4FiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAmO2eUkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQAUQYBM/2FkprlAgAmuna8Hm9EyoEtTlDBGDg6Zm4ZLp5ffvZBE946h92jepDrteoxsJ7pSzJVC2HmDLa4iZUao7lLLbDsUj5x45/iLJcqBZKk3YnAxP2r6a+kI+1VVQY1pxdG1nlJAbdNzoojm/qmezNPSrqni61KVMQKsXBCWhIjSXDSM9CsBj21a+9qaVqfxovJGTn9lgrZO+xzKQNMKZeOouJlscVuFj21P0ww3/YENiU/nMeTSuYypO76mDtAd08Jonc3yuHa9MJGei5ixN3wT+IrGR2aL2hdw2M6NgH7sYbL2Zi4ugD6RXHJai1Bh2yvFSVqSQ+M6QOInT4ud7wslm1XRB065dXtLkBDQRjtnlJAQgA0nDigNK6FToZn7ezdts22l//+cyPF9k1OQM+wNuhdEYGZvieFPNnH4iPf2C6OVPxOsHxs66X3mpn728s5Mg+0V2pkDqX4E9IU1sdPAFGC9Zl! 5eDGmE0Px qaTrlLG2ZA2lxouoHUe/xwE3fKiHENxiY4RVC8vya8dnU/SMnTc39uHS1nlPk0oPiZd2odHO+YABGz2pMLMShCstX+8LDlV/XxXacBbXYDeR+Xk78mFsjKzavfXWhZ7UoA6vitR7XG5y1koZ9FbtU6kq7JPRDeeYbVf0mIufdGTUBedza12M1Lm4knvzZRX+get1lLFngPstU48t041YfQKZpNK0cePPwARAQABiQE2BBgBCAAgFiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAmO2eUkCGwwACgkQAUQYBM/2Fkq3aQf6AuKj65nqjLIOLIasYQ6D28Jhip65ZFhW6XISv5rcjHkvYfMp4J+LaNOZk2P+1b/QN/2wDVisNGYq6lhkkfpBN3vTQlvMY0ugKDug2YaRYeuoLSFBMFVONa9N6437KQg+vRNd46d5282q9BTXAlOLWR/ZuXg/IR5ZIV3JTwLtqbPCUGof85hFxSPWEHMYeeE7180r66wUjA/Zf46Mar9t+olviBW1ss5XOx1JWt7FpQ5peETwmkoYi4Kq16iyuRB94mQqyy1C4fiqTq8dKr4uUYg759NWwU6j21G0kf31bDd4QgTIHvhY/8AgldyzoeJX5Bz67KzX3+/fAWvPEXAvvg== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.50.3 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: yCCM0Jxg/KiA On Fri, 2024-01-19 at 08:49 +0100, Jan Kiszka wrote: > On 19.01.24 08:44, Jan Kiszka wrote: > > On 19.01.24 08:40, Jan Kiszka wrote: > > > On 19.01.24 08:37, Uladzimir Bely wrote: > > > > On Fri, 2024-01-19 at 10:20 +0300, Uladzimir Bely wrote: > > > > > On Thu, 2024-01-18 at 14:50 +0100, Jan Kiszka wrote: > > > > > > From: Jan Kiszka > > > > > >=20 > > > > > > During the migration to sbuild, support for using self- > > > > > > built > > > > > > packages > > > > > > in > > > > > > the build environment was lost if those were already part > > > > > > of the > > > > > > sbuild-chroot. This restores it by adding --apt-distupgrade > > > > > > to the > > > > > > sbuild call. But that is not enough because sbuild will > > > > > > only > > > > > > upgrade > > > > > > packages from already configured sources, not those > > > > > > specified via > > > > > > --extra-repository. We therefore have to switch back to > > > > > > configuring > > > > > > isar-apt during sbuild-chroot creation. > > > > > >=20 > > > > > > As rootfs_configure_isar_apt configures the isar repo under > > > > > > /isar-apt, we bind-mount the one in /home/builder/${PN} to > > > > > > that > > > > > > folder. > > > > > > Another difference is that we now need to run apt-get > > > > > > update > > > > > > explicitly, > > > > > > but only for isar-apt. > > > > > >=20 > > > > > > For apt fetching, we neither need nor want isar-apt to be > > > > > > available. > > > > > > Rebuilding self-generated apt packages is generally > > > > > > pointless as > > > > > > the > > > > > > needs are better addressed in the generating recipe itself. > > > > > > Exposing > > > > > > isar-apt to the fetch may furthermore lead to fetching a > > > > > > previously > > > > > > built source package of the same recipe, rather than > > > > > > pulling the > > > > > > external version. > > > > > >=20 > > > > > > And because the sbuild-chroots are now left behind with > > > > > > isar-apt > > > > > > configured, the configuration in imager_run can be removed > > > > > > (credits > > > > > > to > > > > > > Srinuvasan Arjunan). > > > > > >=20 > > > > > > Signed-off-by: Jan Kiszka > > > > > > --- > > > > > >=20 > > > > > > Changes in v6: > > > > > > =C2=A0- basically, going back to v4 but disabling isar-apt in > > > > > > do_apt_fetch > > > > > >=20 > > > > > > I was able to resolve the scenario that Uladzimir was > > > > > > sharing this > > > > > > way.=20 > > > > > > And I also realized that apt fetching as well as unpacking > > > > > > should > > > > > > have=20 > > > > > > no deal with isar-apt. So, v5 was taking a wrong turn. > > > > > >=20 > > > > > > =C2=A0meta/classes/dpkg-base.bbclass=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 29 > > > > > > +++++++++++---- > > > > > > -- > > > > > > -- > > > > > > =C2=A0meta/classes/dpkg.bbclass=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 |=C2=A0 5 ++-- > > > > > > =C2=A0meta/classes/image-tools-extension.bbclass=C2=A0=C2=A0=C2= =A0 | 13 ------- > > > > > > -- > > > > > > =C2=A0.../sbuild-chroot/sbuild-chroot.inc=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 2 -- > > > > > > =C2=A04 files changed, 20 insertions(+), 29 deletions(-) > > > > > >=20 > > > > > > diff --git a/meta/classes/dpkg-base.bbclass > > > > > > b/meta/classes/dpkg- > > > > > > base.bbclass > > > > > > index 7b054d3f..80686677 100644 > > > > > > --- a/meta/classes/dpkg-base.bbclass > > > > > > +++ b/meta/classes/dpkg-base.bbclass > > > > > > @@ -19,8 +19,6 @@ DEPENDS:append:riscv64 =3D "${@' > > > > > > crossbuild- > > > > > > essential-riscv64' if d.getVar('ISAR_C > > > > > > =C2=A0DEB_BUILD_PROFILES ?=3D "" > > > > > > =C2=A0DEB_BUILD_OPTIONS ?=3D "" > > > > > > =C2=A0 > > > > > > -ISAR_APT_REPO ?=3D "deb [trusted=3Dyes] > > > > > > file:///home/builder/${PN}/isar- > > > > > > apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}=C2=A0${DEBDISTRONAME= } > > > > > > main" > > > > > > - > > > > > > =C2=A0python do_adjust_git() { > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 import subprocess > > > > > > =C2=A0 > > > > > > @@ -115,6 +113,8 @@ do_apt_fetch() { > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM U= SR1 > > > > > > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > > > > > > =C2=A0 > > > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} --= \ > > > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources= .list.d/isar-apt.list > > > > > > /etc/apt/preferences.d/isar-apt > > > > >=20 > > > > > The issue is still reproducible with the steps I posted > > > > > below, as > > > > > well > > > > > as CI still fails with v6. > > > > > I think, the steps above are not sufficient to completely > > > > > stop > > > > > dealing > > > > > with isar-apt in do_apt_fetch() and we need some kind of 'apt > > > > > update' > > > > > here to consider removed list and preferences. The question > > > > > is how to > > > > > say apt to forget about isar-apt and do not run apt update > > > > > for other > > > > > source lists... maybe, simply remove isar-apt files from > > > > > /var/lib/apt/lists/. > > > >=20 > > > > An addition: > > > >=20 > > > > It seems that files are not really removed from the session for > > > > some > > > > reason. I added some kind of "ls" near remove code: > > > >=20 > > > > ``` > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c 'ls -la /etc/apt/= sources.list.d/' > > > > =C2=A0=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -= - \ > > > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/source= s.list.d/isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > +=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c 'ls -la /etc/apt/= sources.list.d/' > > > > ``` > > > > , but still see the following in the log: > > > > ``` > > > > > DEBUG: Executing shell function do_apt_fetch > > > > > total 16 > > > > > drwxr-xr-x 2 root root 4096 Jan 19 07:09 . > > > > > drwxr-xr-x 8 root root 4096 Jan 19 07:09 .. > > > > > -rw-r--r-- 1 root root=C2=A0 569 Jan 19 07:09 bootstrap.list > > > > > -rw-r--r-- 1 root root=C2=A0=C2=A0 94 Jan 19 07:09 isar-apt.list > > > > > total 16 > > > > > drwxr-xr-x 2 root root 4096 Jan 19 07:09 . > > > > > drwxr-xr-x 8 root root 4096 Jan 19 07:09 .. > > > > > -rw-r--r-- 1 root root=C2=A0 569 Jan 19 07:09 bootstrap.list > > > > > -rw-r--r-- 1 root root=C2=A0=C2=A0 94 Jan 19 07:09 isar-apt.list > > > > > Reading package lists... > > > > ``` > > > > Fith the apt_fetch failure followed, because of existing of > > > > previously > > > > built hello source package in the isar-apt repo. > > > >=20 > > >=20 > > > OK... last-minute change of mine, splitting up the purging and > > > the > > > actual fetching into two schroot runs to avoid having to run the > > > latter > > > with "-u root". I didn't find a way to de-configure isar-apt > > > unprivileged. Back to the drawing board. :( > > >=20 > >=20 > > Comparing the split run of schroot with image-tool-extension, I > > think > > I'm just missing a proper session ID here. > >=20 >=20 > Before sending v7: can you try this on top? >=20 > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg- > base.bbclass > index 80686677..1a3333ac 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -113,10 +113,13 @@ do_apt_fetch() { > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > =C2=A0=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > =C2=A0 > -=C2=A0=C2=A0=C2=A0 schroot -d / -u root -c ${SBUILD_CHROOT} -- \ > +=C2=A0=C2=A0=C2=A0 session_id=3D$(schroot -q -b -c ${SBUILD_CHROOT}) > +=C2=A0=C2=A0=C2=A0 echo "Started session: ${session_id}" > + > +=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -u root -- \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.list= .d/isar-apt.list > /etc/apt/preferences.d/isar-apt > =C2=A0=C2=A0=C2=A0=C2=A0 for uri in "${SRC_APT}"; do > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -d / -c ${SBUILD_CHRO= OT} -- \ > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -= d / -- \ > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only- > source source "$2"' my_script "${BASE_DISTRO}- > ${BASE_DISTRO_CODENAME}" "${uri}" > =C2=A0=C2=A0=C2=A0=C2=A0 done > =C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs >=20 > Jan >=20 Yes, with persistent session it works, at least with reproduce steps I used. I just added session cleanup (e.g, "schroot -e -c ${session_id}") at the end. Before sending v7 I could also check if it is not reproduced anymore in CI. Also, due to using schroot session we probably also need to care about proper session removing in case commands under schroot fail, similar to how it's done in image-tools-extention by imager_cleanup().