From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a05:6402:1804:b0:46f:7144:aef3 with SMTP id g4-20020a056402180400b0046f7144aef3mr6894979edy.198.1673026685389; Fri, 06 Jan 2023 09:38:05 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:4413:b0:43d:b3c4:cd21 with SMTP id y19-20020a056402441300b0043db3c4cd21ls4983324eda.2.-pod-prod-gmail; Fri, 06 Jan 2023 09:38:03 -0800 (PST) X-Google-Smtp-Source: AMrXdXvTptVn+ecjgyOnvf7ED4kpU9AwmakT5QbTP+xQuRLTYYReZ46IEJ7wyymuAS23Dk7WIQ2U X-Received: by 2002:a05:6402:914:b0:479:dc9c:1144 with SMTP id g20-20020a056402091400b00479dc9c1144mr10618231edz.24.1673026683567; Fri, 06 Jan 2023 09:38:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673026683; cv=none; d=google.com; s=arc-20160816; b=UZpIVpVhJcCTA6e2dg997gD9KPDVgl/BgLipstaTCEQFYUU8GxL0l6mk4naLnnkSX4 Db6+BvEDu7cF4v3+yJHmZx8zLDLhZaNG8GsWfrYsOaRxr3jM6Cf65ptd8MfL7NywQ3FT eSfUBbRcgTep8siewyk3lFR1YpBtf1673nHeSTyME+nXMzRy8wgHbz84jzlp9tTTlHCf HU7KSnBydL2lPtwk2w0h3VE2iOMWv0M/e3W3bVT+TboXNlsOUznrcTY2OEcMTeCoXRYe 5qHV/6nq6w4TVMCExUy2tJjBkmTggUdLWPvqm6Y0OFQnIRXQKgVnkgllg4pQsRqc0jyL bBcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=w04hDLLrNIvwiDMSGm8QVxycZzy2Emv/yEq9i9LZrmM=; b=kTpBKd7nJqd+QSUlzrFTrXqmMjfQeUOqxOv+MV4ezkjK2VexIrCZOIgNHSV8qmkJ0K qu7TGwsu33+4RFCnGdgDlrwtC60YujRGoF/+kAcrI8ShO/fNNqeyMVYa3WKLxq7XZPSG ybY1xMLmw/39MMHictHPdW3NVQhbBK85AeDfipYOao1yN0cgB1vNp3KNHVQhynmxg3RG zL8mPOy/Ls+3RMguqwugY7X/yaJzaffOf5YijxStYD3vVARIQQCMLFUW4hNA0tjFuff0 PHU6ZVSjuFthEV+1cOCj49H3Qvc8ovfxbxfN2bpDALrZnGQuHDTB+I/6cn+zft54JGzZ ucEA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of postmaster@promwad.com designates 209.85.208.227 as permitted sender) smtp.mailfrom=postmaster@promwad.com Return-Path: Received: from mail-lj1-f227.google.com (mail-lj1-f227.google.com. [209.85.208.227]) by gmr-mx.google.com with ESMTPS id u9-20020aa7d889000000b0048ebe118a43si89342edq.1.2023.01.06.09.38.03 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 Jan 2023 09:38:03 -0800 (PST) Received-SPF: pass (google.com: domain of postmaster@promwad.com designates 209.85.208.227 as permitted sender) client-ip=209.85.208.227; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of postmaster@promwad.com designates 209.85.208.227 as permitted sender) smtp.mailfrom=postmaster@promwad.com Received: by mail-lj1-f227.google.com with SMTP id q2so2159407ljp.6 for ; Fri, 06 Jan 2023 09:38:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=w04hDLLrNIvwiDMSGm8QVxycZzy2Emv/yEq9i9LZrmM=; b=Aevl3/RgLDJP85+6QVQhXPaa/G4joJCPiUcvT1bSdYv7v0p3c6wiiU/fCldxTCsMyR 5Q7Uq06bGlQRdFOkrOqz1plA6CaLxzrNjhXuKJhxaUPv+U3U6oYW9K8SADQRxSzV946V n+/Tx0+OrSryKAScnSVQrnrz1GoX1fuRu3xJNgmM0RoqmS65yMBjhGfJ16aCgYTzIgV8 u0aibK5AVHEvpty5MPxcccBQQGE/dV/r/V8zkn3dbuUt/APeLD2ApxmIScy5gvrayCdh Ekz+RprLJ/g02sgILJCRqog1m0bIRS6F1MhC6OVEGGlQufq8LEO/2ppCd9vBkzDItKVR vaHA== X-Gm-Message-State: AFqh2kqGkTpvrZw21Ht/4qbXkn2VAudKIqLjVdbAiVQabXCnmJpOiLyh UNBIn3AnkL/4gkk1+vZIZO0K97LbK5Oap9MXL74/YxomHIqGuw== X-Received: by 2002:a2e:a265:0:b0:281:1028:6065 with SMTP id k5-20020a2ea265000000b0028110286065mr1840644ljm.15.1673026682706; Fri, 06 Jan 2023 09:38:02 -0800 (PST) Return-Path: Received: from promwad.com ([178.172.133.66]) by smtp-relay.gmail.com with ESMTPS id p10-20020a2eb7ca000000b0027ffe1e38cfsm22181ljo.56.2023.01.06.09.38.02 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Fri, 06 Jan 2023 09:38:02 -0800 (PST) X-Relaying-Domain: promwad.com Received: from home.localnet (10.54.231.20) by webmail.promwad.com (192.168.10.44) with Microsoft SMTP Server (TLS) id 14.1.438.0; Fri, 6 Jan 2023 20:38:01 +0300 From: Uladzimir Bely To: Henning Schild CC: Subject: Re: [PATCH v2 3/3] Changes for a faster build using less disk space Date: Fri, 6 Jan 2023 20:38:05 +0300 Message-ID: <5434767.rdbgypaU67@home> In-Reply-To: <20230106165826.1346da94@md1za8fc.ad001.siemens.net> References: <20230106064809.10412-1-ubely@ilbers.de> <20230106064809.10412-4-ubely@ilbers.de> <20230106165826.1346da94@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3375900.LZWGnKmheA"; micalg=pgp-sha256; protocol="application/pgp-signature" Return-Path: ubely@ilbers.de X-TUID: 1IAgJg1EP6PQ --nextPart3375900.LZWGnKmheA Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii"; protected-headers="v1" From: Uladzimir Bely To: Henning Schild Cc: isar-users@googlegroups.com Date: Fri, 06 Jan 2023 20:38:05 +0300 Message-ID: <5434767.rdbgypaU67@home> In-Reply-To: <20230106165826.1346da94@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 In the email from Friday, 6 January 2023 18:58:26 +03 user Henning Schild wrote: > Am Fri, 6 Jan 2023 07:48:09 +0100 > > schrieb Uladzimir Bely : > > From: "Roberto A. Foglietta" > > > > Faster build for larger projects with a lot of debian packages > > > > This patch leverages few ways to accomplish the task to optimise > > the large buildings. > > > > Put a CACHEDIR.TAG file in each cache folder in such a way every > > tar that uses the option --exclude-caches could be avoided > > to include debian packages saving a lot of time and disk space. > > > > Signed-off-by: Roberto A. Foglietta > > Signed-off-by: Uladzimir Bely > > --- > > > > meta/classes/deb-dl-dir.bbclass | 2 +- > > meta/classes/rootfs.bbclass | 3 ++- > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++- > > 3 files changed, 9 insertions(+), 3 deletions(-) > > > > diff --git a/meta/classes/deb-dl-dir.bbclass > > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644 > > --- a/meta/classes/deb-dl-dir.bbclass > > +++ b/meta/classes/deb-dl-dir.bbclass > > @@ -102,7 +102,7 @@ deb_dl_dir_export() { > > > > -maxdepth 1 -type f -iname '*\.deb' |\ > > > > while read p; do > > > > # skip files from a previous export > > > > - [ -f "${pc}/${p##*/}" ] && continue > > + [ -e "${pc}/${p##*/}" ] && continue > > > > # can not reuse bitbake function here, this is basically > > # "repo_contains_package" > > package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name > > > > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass > > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644 > > --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() { > > > > # so we use some mount magic to prevent that > > mkdir -p ${WORKDIR}/mnt/rootfs > > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > > > > - sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system > > rootfs > > + lopts="--one-file-system --exclude-caches" > > + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs > > > > sudo umount ${WORKDIR}/mnt/rootfs > > sudo chown $(id -u):$(id -g) rootfs.tar > > > > } > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > cb0079ec..44019f95 100644 --- > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6 > > +395,10 @@ EOSUDO > > > > # Cleanup apt cache > > sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean > > > > + > > + # Don't include apt cache into sstate cache by marking the > > directory > > + # with a CACHEDIR.TAG file with a special signature > > + sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55 > > What is that "special signature" all about? Does anyone later care to > remove the file again, or do we risk it ending up in the final rootfs? > > I find this a little weird and would prefer if the calls to tar would > simply exclude this by path and not by magic tool specific file. > Original patch from Roberto included both `--exclude-caches` and `-- exclude=var/cache/apt/archives"` options for `tar`. First one was expected to automatically exclude all directories that contain special CACHEDIR.TAG file [1]. But it appeared not to work. Investigating the problem shown that it should not be empty file: GNU tar expects file starts with "Signature: 8a477f597d28d172789f06886806bc55" line. It means, all the stuff with multiple "touch CACHEDIR.TAG" was useless and the actually only option "--exclude=var/cache/apt/archives" did the trick. I fixed first and removed second option. But you are right, we can do vice versa and it should look even simpler. Final rootfs doesn't have this file since we do "apt-get clean" for it. [1] https://bford.info/cachedir/ > It could be made a central function so we have all those tar calls in > one place. > > In this patch i see three places ... hard to maintain. > > Henning > > > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" } > > > > addtask bootstrap before do_build after do_generate_keyrings > > > > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" > > > > bootstrap_sstate_prepare() { > > > > # this runs in SSTATE_BUILDDIR, which will be deleted > > > > automatically > > - sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar > > --one-file-system $(basename "${ROOTFSDIR}") > > + lopts="--one-file-system --exclude-caches" > > + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts > > $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar > > > > } --nextPart3375900.LZWGnKmheA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAmO4XH0ACgkQAUQYBM/2 FkpDaAf/dEPSYGUytpwI73zbCQWAShH3gaYAYXQpFBWh19n/xl04begk54sH18dp 9WtFkbgUjLyULv9fehzcVHUT394ycyOBHXdIvbKwvZtKK5iZY88nvc9b6rhrx0Dd ucSiuruzscPnfYi0cj1oKx7L/giKlmuMfVreiKSvWI5e4Xw7Im2iCmZtJ6+Qhjqs KWnN0hE8ULfJjWHbl4qfIP91zWDC/A7B4kHm2XbzYzrewaHggq2kFehfRVURr1vt S1OdBXbxuhbkKosBGkZoGI/JQq3Ik7c1IVFt8xPdE0MqVJdGe8aWZVm3HIk72eGJ LrNdT4UVHsvEOd3Xm4CP8Hx1rz48oQ== =Mwei -----END PGP SIGNATURE----- --nextPart3375900.LZWGnKmheA--