From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a05:6638:41a4:b0:38a:3770:2fa2 with SMTP id az36-20020a05663841a400b0038a37702fa2mr1036227jab.208.1674139995470; Thu, 19 Jan 2023 06:53:15 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6e02:e47:b0:30b:f63e:3073 with SMTP id l7-20020a056e020e4700b0030bf63e3073ls622576ilk.5.-pod-prod-gmail; Thu, 19 Jan 2023 06:53:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXtNqr4mB5JnuHnEbTV/REYeiJpDl/eK/ZNU8x4ffIE70vh+xqoNCSNL6aunMC3qnWphSMMp X-Received: by 2002:a92:d9cb:0:b0:30b:fd98:1be3 with SMTP id n11-20020a92d9cb000000b0030bfd981be3mr8713814ilq.4.1674139994906; Thu, 19 Jan 2023 06:53:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674139994; cv=none; d=google.com; s=arc-20160816; b=cidhzZfPMMWD1CGWR3n6dP6sW+7N/ljxOkbuWM0Qse7dO3dZUhOwAvdCZWni+ERwtp DmC3GW2zzCBihL+9zhib61HTybd26VfT0Hj+4noTGYqk692/K1gDAB0e5EkFWygdEa4l P0RvUoSxjEiMg0Th4s5wAueo6n1y7Y8ro0lMinVu51VKpFaCDE/+gV8VGcNPn9kOnEj6 5FG1dUv63dBL6mRZecfxyxRHA3BzQx8QJPRHHEQ+PLqXpA5APHjWlGDQCfQ2oQGLrF82 0WXH4L3g66GwCJNoEib2RMl82l1fP+kZK80a8OEfXyyanxgHFyT3DUmycL0XIxGEIa4I 05Ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=kB4E9STcKLMBn700qGdEBPyYlxG8tjtrDd86mwqvqi8=; b=o6ZSnHlACggo4L5WRqDESNWAgpYPJJzESQGSKr8MAL+szm8tX5D/Roq63/LP75+hTD Jwnz02ZOipdWO+hwIc5j/AHGdWbOosfaEqdkK3QyWUzgb8XN67lNF250Y6Sa8kPIz7Ga Edg3UmECDgpUsHlZBeKd/TNYy7tLykEqMkiYDOG1C7JTV3LBqdQDT9ZBeOaWylUQplti d6j50dpthFISzkmMx/3qEVx9YLBzamzdX6+DsZh4VpxDBKoh98A+l6Qj6XAHVHoYu/c1 XK8+IX9FqIpRjFeIlCMs9XMfnEeH/79+sXVNevEMCPLXQN4cNm8f6W8vAJ38tnIqltqz kKYw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="e/HXHOJP"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72f as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com. [2607:f8b0:4864:20::72f]) by gmr-mx.google.com with ESMTPS id z8-20020a05663803a800b003a0bea971d4si1233398jap.3.2023.01.19.06.53.14 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Jan 2023 06:53:14 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72f as permitted sender) client-ip=2607:f8b0:4864:20::72f; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="e/HXHOJP"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72f as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qk1-x72f.google.com with SMTP id z9so1259059qkl.13 for ; Thu, 19 Jan 2023 06:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kB4E9STcKLMBn700qGdEBPyYlxG8tjtrDd86mwqvqi8=; b=e/HXHOJPndO81QCVuJrwSEAjT8O+FyOtJw9SnmFFI8mGAxTzBtjmmM0W7g1B7EcIhN 1lJfKGJ2KzwezubK+0rd9de50xRIk934UWWzAQR2Ex976aPf3dEmm0QjsVeju1GdCUV7 7TE4/0Zk9h7qcGEbhbuINkjS8nS5RxM5OBzLymoiHEys3ww6Cm7JxCYfqVEIX/EEQQqN goGswJ5Sh8g4dxpyqaZ3ZC1zLQGLBn3hVsIFemTg40Jl/93KsUmQI574GBS7bgCD4R+C 2tPWC0+TyzDs6NStaVUGIDi/jTr7s6gKrx8wwcG8yRAEvUE0W03mnD/3WkSfbo4gEYkg Sdyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kB4E9STcKLMBn700qGdEBPyYlxG8tjtrDd86mwqvqi8=; b=hwccPHm7+DubXwSWDUN8BF4+oZSobZMzxS2v3KVzM4k9wp9HVipwyLRJP1bGVbs9LR lVDXJ8ZJMQxadIdaMgISCwkpA6Yrta4BWDkgsgSqO4baatyEUY6iLgO0oNW0zRq09/bq QrNBV3DpQtZKP/7g1Gp9Qv5JposN4Y9Xi1HatFAbXn1cFsU6gsjYDxMrF1cX+xUN2GIe /W9DGIKtsWJR2VYFBN3fCBfJ5Bef/TlWVwwEzLY89ryqf//PxZL8tywYMIV3HLGYIhUt h50I7HmF3+Zd2JWmRXdCGKugQfvh7l0Kvfp+LK8q5lunsUgvD+sq8ggG1jcxA+Vs3oB3 ZVYg== X-Gm-Message-State: AFqh2kqBfQxh4MbKvs1sFcnQHee0yLwDQ4wf9eM65lMDZ4QgS8YOHySG PoinZkjCQ4GVPZfxoEbFUbb8FGhebrCQIuWzKg== X-Received: by 2002:a05:620a:219b:b0:705:e0a8:1b03 with SMTP id g27-20020a05620a219b00b00705e0a81b03mr417619qka.119.1674139994232; Thu, 19 Jan 2023 06:53:14 -0800 (PST) MIME-Version: 1.0 References: <20230106064809.10412-1-ubely@ilbers.de> <00ca12addc6e3b93aeca6127124cfc2f0a1aabf2.camel@siemens.com> <6842478.jJDZkT8p0M@home> <6100573.eO5KgaWL5Y@home> In-Reply-To: <6100573.eO5KgaWL5Y@home> From: "Roberto A. Foglietta" Date: Thu, 19 Jan 2023 15:52:37 +0100 Message-ID: Subject: Re: [PATCH v2 0/3] Improving apt cache To: Uladzimir Bely Cc: "isar-users@googlegroups.com" , "Moessbauer, Felix" Content-Type: multipart/alternative; boundary="000000000000a9226c05f29f17aa" X-TUID: kdNeytZWh2ZW --000000000000a9226c05f29f17aa Content-Type: text/plain; charset="UTF-8" On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely wrote: > > I got time to get back to this patches and checked this moment. And it > really > does not work as I expected. > > Original patch includes the following: > > - --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/ > *.deb -t ${deb_dir}/ || :" \ > + --chroot-setup-commands="ln -Pf ${ext_deb_dir}/*.deb -t > ${deb_dir}/ > 2>/dev/null || :" \ > > This results to to failing "ReproTest" in CI with the following error (for > libhello, for example): > > sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but > it > is not installable > E: Unable to correct problems, you have held broken packages. > > To debug it, I removed `2>/dev/null` and found, that hardlinks simply > don't > work and the following errors are now seen earlier: > > ln: failed to create hard link > '/var/cache/apt/archives/adduser_3.118_all.deb' > => '/home/builder/libhello/rootfs/var/cache/apt/archives/ > adduser_3.118_all.deb': Invalid cross-device link > ... #tons of similar errors... > ln: failed to create hard link '/var/cache/apt/archives/ > zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' => '/home/builder/libhello/ > rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb': > > Invalid cross-device link > > I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/ > archives/*.deb -t /var/cache/apt/archives/ || :'. > > So, it works for network builds (when missing packages always can be > downloaded by apt), but it fails for local builds from apt-cache (when, at > first network build, sbuild dependencies are simply not exported to > download > directory due to non-working hardlinks, plus with hidden stderr) > Hi, first of all I do not fully understand why we use ln -P instead of ln -Pf in the exporting debs. --finished-build-commands="ln -P ${deb_dir}/*.deb -t ${ext_deb_dir}/ 2>/dev/null || :" \ This code's goal is to build a deb that should be exported otherwise we are continuing using the old package. It seems that breaks things but - again - I did fully not understand what and why but simply accepted the suggestion. Now, I have reverted back to ln -Pf. Second, those lines are supposed to fail - and obviously fail in both directions: import and export - but the following (a) if export fail we will not have the custom packages but we have them (b) removing that lines and the code will always fail These mean that ln -Pf complains to fail but make a difference and make the difference that we want. However, I might not have understood the case in which it fails completely so: mv build/downloads . rm -rf build ./build.sh basic-os (DONE) ./clean.sh isar ./build.sh (DONE) ./clean.sh all ./build.sh (DONE) remove that two lines about ln -Pf that are supposed to do nothing than fail ./clean.sh all ./build.sh (DONE with the same building time) rm -rf build/downloads mv downloads build ./clean.sh isar ./build.sh complete (FAIL) put back those two command lines with ln -Pf and ./clean.sh all ./build.sh (DONE) I do not say you are wrong and I see ln -Pf complains on stderr but nothing that tin my private fork cannot be solved using a stderr redirection to /dev/null However there are some other ways to do this thing: 1. sbuild uses schroot by default, using chroot instead 2. keep the default in schroot, elsewhere it uses upper folder that can be populated in advance I did not make any changes because I did not identified the issue. Best regards, R- --000000000000a9226c05f29f17aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, 19 Jan 2023 at 08:36, Uladzimir B= ely <ubely@ilbers.de> wrote:

I got time to get back to this patches and checked this moment. And it real= ly
does not work as I expected.

Original patch includes the following:

-=C2=A0 =C2=A0 =C2=A0 =C2=A0 --chroot-setup-commands=3D"cp -n --no-pre= serve=3Downer ${ext_deb_dir}/
*.deb -t ${deb_dir}/ || :" \
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 --chroot-setup-commands=3D"ln -Pf ${ext_d= eb_dir}/*.deb -t ${deb_dir}/
2>/dev/null || :" \

This results to to failing "ReproTest" in CI with the following e= rror (for
libhello, for example):

sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 but = it
is not installable
E: Unable to correct problems, you have held broken packages.

To debug it, I removed `2>/dev/null` and found, that hardlinks simply do= n't
work and the following errors are now seen earlier:

ln: failed to create hard link '/var/cache/apt/archives/adduser_3.118_a= ll.deb'
=3D> '/home/builder/libhello/rootfs/var/cache/apt/archives/
adduser_3.118_all.deb': Invalid cross-device link
... #tons of similar errors...
ln: failed to create hard link '/var/cache/apt/archives/
zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' =3D> '/home/builder= /libhello/
rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb&#= 39;:
Invalid cross-device link

I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt= /
archives/*.deb -t /var/cache/apt/archives/ || :'.

So, it works for network builds (when missing packages always can be
downloaded by apt), but it fails for local builds from apt-cache (when, at =
first network build, sbuild dependencies are simply not exported to downloa= d
directory due to non-working hardlinks, plus with hidden stderr)

Hi, first of all I do not fully understand why we= use ln -P instead of ln -Pf in the exporting debs.
=C2=A0
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 --finished-build-commands=3D"ln -P ${d= eb_dir}/*.deb -t ${ext_deb_dir}/ 2>/dev/null || :" \
=
This code's goal is to build a deb that should be export= ed otherwise we are continuing using the old package. It seems that breaks = things but - again - I did fully not understand what and why but simply acc= epted the suggestion. Now, I have reverted back to ln -Pf.
Second, those lines are supposed to fail - and obviously fail = in both directions: import and export - but the following

(a) if export fail we will not have the custom packages but we have= them
(b) removing that lines and the code will always fail
=

These mean that ln -Pf complains to fail but make a dif= ference and make the difference that we want. However, I might not have und= erstood the case in which it fails completely so:

= mv build/downloads .
rm -rf build
./build.sh basic-os (= DONE)
./clean.sh isar
./build.sh (DONE)
.= /clean.sh all
./build.sh (DONE)

remo= ve that two lines about ln -Pf that are supposed to do nothing than fail

./clean.sh all
./build.sh (DONE with the s= ame=C2=A0building time)

rm -rf build/downloads
mv downloads build

./clean.sh isar
./build.sh complete (FAIL)

put back those two co= mmand lines with ln -Pf and

./clean.sh all
./build.sh (DONE)

I do not say you are wrong an= d I see ln -Pf complains on stderr but nothing that tin my private fork can= not be solved using a stderr redirection to /dev/null

<= div>However there are some other ways to do this thing:

1. sbuild uses schroot by default, using chroot instead
2. = keep the default in schroot, elsewhere it uses upper folder that can be pop= ulated in advance

I did not make any changes becau= se I did not identified the issue.

Best regards, R= -
--000000000000a9226c05f29f17aa--