From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a17:90b:3903:b0:225:de08:b714 with SMTP id ob3-20020a17090b390300b00225de08b714mr1308040pjb.193.1674189869671; Thu, 19 Jan 2023 20:44:29 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6a00:1bc9:b0:58d:abd4:7bd2 with SMTP id o9-20020a056a001bc900b0058dabd47bd2ls1070238pfw.2.-pod-prod-gmail; Thu, 19 Jan 2023 20:44:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXuQf5/oll1jd10t6e0s5wSvf7Xcgdzs18qUQQh0LrYUGsYJlr6+/rmeqgy4mpw4MwMX/BUT X-Received: by 2002:a05:6a00:4c0b:b0:58d:acc6:fd2a with SMTP id ea11-20020a056a004c0b00b0058dacc6fd2amr15066922pfb.25.1674189868601; Thu, 19 Jan 2023 20:44:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674189868; cv=none; d=google.com; s=arc-20160816; b=j5tERR/O8VL1V1koc0HSVJv7/gJBvv2W10dAISBL4ozdbrNqNS4lpQZjELzQ9Dt02h 1YHwSlbagg0Wy4rjdMXd1a+R6pDm7bKrF44uonsbTQyosI6LtGsODrNGT8p6DldvlWzS QPenSTmUDb7y+S5pTOc/obxFBFQUEhSU3l15Pw61cGFZ07tJYrK/ZTav+rlBQsxx295z K1qv9OCq79ASiUTHfdMUYzbjOJwMdvbyGB9c6YVrjNJuHNdBsnswQWtre+UNe2fjp1sF vI9TpKabGa+H5TVuszQ2ZLqG9beCOvxP8npIlwD4X964lSVnqhCmIIMIWlmCbRHTXwtL 1DIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=DzelYVKnF95XTZwEVbEyvKbWV521NjN2OzSKEcoTUYo=; b=R5ZBOUjj2cH6CqGOFl0PK/7iErKSbOE1SvDvS3kzBfuWFkP0FRipNZhKpHaV7shFHP YjvZ4uNzy/aEIwE8nAVclWDS9GZ/HglRn9RZvKuQ+GVujns46pWcr7IbK5Ub8XNtlP6P 7cAvOU9wbohEFJ4h1j2vAG9Mw84efUpMJMZsIXAtOMvyRLQIBUiU58BOZ9ocubveynhA RZocTZrG8FUrUiV5XkNuojS5OkB5f+vYzM8DsKlOSS7dSL0a/lt9b28hxCDuut0t6v9n 7UtZC7lN7xhEDAmrwxiH9QOFip9eUQcblb+Z5K1UJtN6OjbuUeyVe2FIFs1manZu+Dqp lSUw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q10-20020a056a0002aa00b0058e08791ba4si302012pfs.4.2023.01.19.20.44.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 19 Jan 2023 20:44:28 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from hp.localnet (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 30K4iMjY000697 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 20 Jan 2023 05:44:23 +0100 From: Uladzimir Bely To: "Roberto A. Foglietta" Cc: "isar-users@googlegroups.com" Subject: Re: [PATCH v2 0/3] Improving apt cache Date: Fri, 20 Jan 2023 07:44:59 +0300 Message-ID: <9466878.eNJFYEL58v@hp> In-Reply-To: References: <20230106064809.10412-1-ubely@ilbers.de> <6100573.eO5KgaWL5Y@home> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 3exxPR1+JvlR In mail from =D1=87=D0=B5=D1=82=D0=B2=D0=B5=D1=80=D0=B3, 19 =D1=8F=D0=BD=D0= =B2=D0=B0=D1=80=D1=8F 2023 =D0=B3. 17:52:37 +03 user Roberto A. Foglietta=20 wrote: > On Thu, 19 Jan 2023 at 08:36, Uladzimir Bely wrote: > > I got time to get back to this patches and checked this moment. And it > > really > > does not work as I expected. > >=20 > > Original patch includes the following: > >=20 > > - --chroot-setup-commands=3D"cp -n --no-preserve=3Downer > > ${ext_deb_dir}/ > > *.deb -t ${deb_dir}/ || :" \ > > + --chroot-setup-commands=3D"ln -Pf ${ext_deb_dir}/*.deb -t > > ${deb_dir}/ > > 2>/dev/null || :" \ > >=20 > > This results to to failing "ReproTest" in CI with the following error (= for > > libhello, for example): > >=20 > > sbuild-build-depends-dose3-dummy:armhf : Depends: dose-distcheck:amd64 = but > > it > > is not installable > > E: Unable to correct problems, you have held broken packages. > >=20 > > To debug it, I removed `2>/dev/null` and found, that hardlinks simply > > don't > > work and the following errors are now seen earlier: > >=20 > > ln: failed to create hard link > > '/var/cache/apt/archives/adduser_3.118_all.deb' > > =3D> '/home/builder/libhello/rootfs/var/cache/apt/archives/ > > adduser_3.118_all.deb': Invalid cross-device link > > ... #tons of similar errors... > > ln: failed to create hard link '/var/cache/apt/archives/ > > zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.deb' =3D> '/home/builder/libhel= lo/ > > rootfs/var/cache/apt/archives/zlib1g_1%3a1.2.11.dfsg-2+deb11u2_mipsel.d= eb' > > : > >=20 > > Invalid cross-device link > >=20 > > I: Finished running 'ln -Pf /home/builder/libhello/rootfs/var/cache/apt/ > > archives/*.deb -t /var/cache/apt/archives/ || :'. > >=20 > > So, it works for network builds (when missing packages always can be > > downloaded by apt), but it fails for local builds from apt-cache (when,= at > > first network build, sbuild dependencies are simply not exported to > > download > > directory due to non-working hardlinks, plus with hidden stderr) >=20 > Hi, first of all I do not fully understand why we use ln -P instead of ln > -Pf in the exporting debs. >=20 > --finished-build-commands=3D"ln -P ${deb_dir}/*.deb -t > ${ext_deb_dir}/ 2>/dev/null || :" \ >=20 > This code's goal is to build a deb that should be exported otherwise we a= re > continuing using the old package. It seems that breaks things but - again= - > I did fully not understand what and why but simply accepted the suggestio= n. > Now, I have reverted back to ln -Pf. >=20 > Second, those lines are supposed to fail - and obviously fail in both > directions: import and export - but the following >=20 > (a) if export fail we will not have the custom packages but we have them > (b) removing that lines and the code will always fail >=20 > These mean that ln -Pf complains to fail but make a difference and make t= he > difference that we want. However, I might not have understood the case in > which it fails completely so: >=20 > mv build/downloads . > rm -rf build > ./build.sh basic-os (DONE) > ./clean.sh isar > ./build.sh (DONE) > ./clean.sh all > ./build.sh (DONE) >=20 > remove that two lines about ln -Pf that are supposed to do nothing than f= ail >=20 > ./clean.sh all > ./build.sh (DONE with the same building time) >=20 > rm -rf build/downloads > mv downloads build >=20 > ./clean.sh isar > ./build.sh complete (FAIL) >=20 > put back those two command lines with ln -Pf and >=20 > ./clean.sh all > ./build.sh (DONE) >=20 > I do not say you are wrong and I see ln -Pf complains on stderr but nothi= ng > that tin my private fork cannot be solved using a stderr redirection to > /dev/null >=20 > However there are some other ways to do this thing: >=20 > 1. sbuild uses schroot by default, using chroot instead > 2. keep the default in schroot, elsewhere it uses upper folder that can be > populated in advance >=20 > I did not make any changes because I did not identified the issue. >=20 > Best regards, R- I'll try to make things clear. Isar supports local builds from 'base-apt' r= epo=20 with. =46irst build user does as usual. Second build is done with the following changes (at least): ISAR_USE_CACHED_BASE_REPO =3D "1" BB_NO_NETWORK =3D "1" In this case every .deb we downloaded from Debian mirrors at first build go= es=20 to local 'base-apt' and second build use it, but not remote mirrors. That's the reason of using this "cp/ln magic" inside of sbuild. While build= ing=20 some package, multipe dependencies might be downloaded inside of sbuild. An= d=20 we need a way to extract them from "internal" apt-cache in sbuild schroot a= nd=20 put to the package workdir, where deb-dl-dir export will pass them to DL_DI= R=20 later. When we replace copying to hardlinks, that actually don't work due to=20 different filesystems, we come to the situation, when multiple debian packa= ges=20 (dependencies of what we are building or some packages that sbuild requires= =20 itself) are not present in DL_DIR. So, second "cached" build from local rep= o=20 simply fails. I see the following possible solutions that allows to save space. 1. Since hardlinks don't work, try to use symlinks. 2. Don't use import with sbuild at all, but only export. But this has a=20 drawback, when all packages we build with sbuild might download the similar= =20 dependencies in parallel.