From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182122219497062400 X-Received: by 2002:a05:6870:108f:b0:16b:ee3a:129f with SMTP id 15-20020a056870108f00b0016bee3a129fmr438910oaq.176.1676045176887; Fri, 10 Feb 2023 08:06:16 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:241d:b0:14d:3678:867f with SMTP id n29-20020a056870241d00b0014d3678867fls2830750oap.10.-pod-prod-gmail; Fri, 10 Feb 2023 08:06:16 -0800 (PST) X-Google-Smtp-Source: AK7set9y+iz7mpvBrYWWClzh4QzqeRM4jU4FyrkwTfs4NdQ/aVe0NAT3Fw2p9kus1ty6+hm2P1hX X-Received: by 2002:a05:6870:ea89:b0:14c:7959:8c2e with SMTP id s9-20020a056870ea8900b0014c79598c2emr2415416oap.2.1676045176228; Fri, 10 Feb 2023 08:06:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676045176; cv=none; d=google.com; s=arc-20160816; b=M+7zzrg77XI01ECQE6z5G6tbqDUFNe06ZGaizwaMw7YAfPdMpKTs1L4ejPGHdTEW8s b2yCAbl6zQpGUdHr6308tlUIlhLiqc403tHqqoB7D6z0kDyDwlJxIy264y+HUboE9q15 5wsCUMh1rZpl9KtTleenA8phpUoGfDiVPW+xmYbZGWN9txvpebW6wB1n2aObnnkEwzDF QrBD4UpV7dg+f7Tyr3Ed1YZz3QMK3ksXI82rhG4I9szRc9xJscW0N6HSfskP396+4di9 pm9+EzdFywk52769NdATbgrST3RqnJ+OTl4QpVqMKDKSqS1wwIRPBIRaC3kKA6ZyHak6 1zrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=PRbmDnOb3skiwUwMI8chDjcrUx5hxo9QCCVRts/yk68=; b=G7aod1Ou++zP8EgC6URcMxVj2FNERReldfWMWR6ehQbBYvEVfN1c4LXPFxF8t+/Tis uxdPCOL7RhjQR3+mKvRcb/NzDCC3twKLspK/02rErrpT5irONr5ldcCu5FtdJ8C8zHdy 8eNJ3uCS+Qk2msZWM1QrMZbq1/6ePmHzVtE8zFFwtyp5KluiSHA/xy9nKEKMxAeYWy05 ZseWOpPc6KvrABLb2DrauwZU9Bs3qL6dry4v3eTJ7sstIFysdRJmLkmEpldHKYaQfqx+ ASxx0zThSAZF2Sg4oXWzsHAhTCopcNNYu231dCe+Um7Hwrj80GYJ4HLUrj0noNYrVkPw ug7A== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="F0AlgH/F"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72d as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com. [2607:f8b0:4864:20::72d]) by gmr-mx.google.com with ESMTPS id pi7-20020a0568704c8700b0016a33950b80si435922oab.4.2023.02.10.08.06.16 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 10 Feb 2023 08:06:16 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72d as permitted sender) client-ip=2607:f8b0:4864:20::72d; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="F0AlgH/F"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::72d as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qk1-x72d.google.com with SMTP id ay41so2379805qkb.5 for ; Fri, 10 Feb 2023 08:06:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PRbmDnOb3skiwUwMI8chDjcrUx5hxo9QCCVRts/yk68=; b=F0AlgH/FwRHgOUkkkLb9yTiSCemkGaNajdYSbBKVQnf3qUfawpS3PDQxuC2QtsVeqw 0CIlfSkyRKy6UrMmVBMsIXoohVOm4urSxez+/YevPcX4JoJG/tMdfCWTDDOjYagX0vgd H4w9ln99cNznZBcbMMMP19PzfvwR9bjPmAj8qL/yxeZSAoG0Pa4Ytc3VWCnCD5/gxUbd ujpJJEI3Wq7NRBVTF8fviqfnXggilG4auzIelbPj5GcmBej5E6BaYDSs0qAZb1aa6TNG DtLKmHDiTi5ROU5ti8Bc+wRw2Ht/G1hmI0FAMYoZkxiFwSw16brhIDBk4QUGpunJtE8G gHYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PRbmDnOb3skiwUwMI8chDjcrUx5hxo9QCCVRts/yk68=; b=AB/eh269fbVMujZc92ZRhlHi8+qb5o670cFG1VtMQntljLEFQ5XlFgqpDkDYZFg+aM +yQjfwd1rKNvQ81rRMlTqxZWBoMRDdmlJNSFjl2P7o0V9XE6kOk3NVDGr0AqG4yV2Jzs KJ8EnLaCeKmP1tcSDAGpZqfpvBjj2sLrfAhvGbRurCLxe7wCSqTQNCIGucom5qjdW+F3 Rc4ctzlQ+mYPB/4ZFptjU1qMXyy9K/SjtKTmTSTs81mAT2BWcHC3ykAtU5SnBwcQOLZy yCEd358eyxW5YY5JnIIKfdz/8MWmEEuMeOqecI+KOzOe72zoQEPpr5/j3BNYHWTz9vq3 8BFQ== X-Gm-Message-State: AO0yUKXpcudlhSWxQ5mCdk2H5DRygfvh9L8bZ06+HUxeL2iA3U6YAJA/ a4wxOP2dyIciLfhiAL0bddlMKZUrjBPkoLSs1k5cReNDYA== X-Received: by 2002:a37:b644:0:b0:706:6b11:8224 with SMTP id g65-20020a37b644000000b007066b118224mr649698qkf.197.1676045175800; Fri, 10 Feb 2023 08:06:15 -0800 (PST) MIME-Version: 1.0 References: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> <38d18c245baa4f685642eafa9a52ab9b9ae9001c.camel@siemens.com> <4769513.OV4Wx5bFTl@hp> <95542c23ea3bd00e31898fc1080854c58e41a8b2.camel@siemens.com> <13d9675831e4d315390357c3bfd925e66f84d136.camel@siemens.com> In-Reply-To: From: "Roberto A. Foglietta" Date: Fri, 10 Feb 2023 17:05:38 +0100 Message-ID: Subject: Re: Better way to handle apt cache needed To: "Moessbauer, Felix" Cc: "ubely@ilbers.de" , "isar-users@googlegroups.com" , "Bezdeka, Florian" , "ibr@radix50.net" , "Schild, Henning" Content-Type: multipart/mixed; boundary="0000000000005512a705f45aad89" X-TUID: 8arwuWdb9P3i --0000000000005512a705f45aad89 Content-Type: multipart/alternative; boundary="0000000000005512a305f45aad87" --0000000000005512a305f45aad87 Content-Type: text/plain; charset="UTF-8" On Wed, 25 Jan 2023 at 05:48, Roberto A. Foglietta < roberto.foglietta@gmail.com> wrote: > On Thu, 19 Jan 2023 at 19:08, Roberto A. Foglietta SSTATE CACHE REWORK + APT CACHE MOUNT BIND ========================================== These are the results in the first development version that works in every condition. It carries on a lot of debugging code and it is NOT fully optimised both under the code-review and the performances. Nevertheless, it still improved: devel3 (public) + npriv11 (private) ------------------------------------ fresh cache ------------ basic-os -------------- 3949 Mb (max) | 3176 Mb (max) 2968 Mb (rest) | 2277 Mb (rest) 4281 Mb (deb) | 4281 Mb (deb) 803 Mb (wic) | 802 Mb (wic) <-- cache! 520 Mb (cache) | 520 Mb (cache) time: 3m28s | time: 20s 10.40x ------------ original --------------------- time: 7m58s | time: 2m17s 2.30x 6.85x devel3 (public) + npriv11 (private) ------------------------------------ fresh cache ------------ complete -------------- 32466 Mb (max) | 27741 Mb (max) 23298 Mb (rest) | 18657 Mb (rest) 4281 Mb (deb) | 4281 Mb (deb) 8989 Mb (wic) | 8987 Mb (wic) <-- cache! 4398 Mb (cache) | 4398 Mb (cache) time: 12m19s | time: 3m24s 3.62x ------------ original --------------------- time: 21m17s | time: 8m28s 1.73x 2.49x The mount bind approach is completely general but not yet used for apt lists but just for the .deb packages. Moreover, the rootfs tar options (--exclude/s) are too extreme and the cached build is smaller than the fresh build. This obviously destroys the reproducibility but it is just a single point change. --0000000000005512a305f45aad87 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Wed, 25 Jan 2023 at 05:48, Roberto A. = Foglietta <roberto.foglietta@gmail.com> wrote:
On Thu, 19 Jan 20= 23 at 19:08, Roberto A. Foglietta


SSTATE CACHE= REWORK + APT CACHE MOUNT BIND
=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

These are the results = in the first development version that works in every
condition. It carri= es on a lot of debugging code and it is NOT fully optimised
both under t= he code-review and the performances.
Nevertheless, it still improved:

=C2=A0 = =C2=A0 =C2=A0 =C2=A0 devel3 (public) + npriv11 (private)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0------------------------------------
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 fresh =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= cache
=C2=A0 =C2=A0 =C2=A0 =C2=A0------------ basic-os --------------=C2=A0 =C2=A0 =C2=A0 =C2=A0 3949 Mb (max) =C2=A0 | =C2=A03176 Mb (max)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 2968 Mb (rest) =C2=A0| =C2=A02277 Mb (rest)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 4281 Mb (deb) =C2=A0 | =C2=A04281 Mb (deb)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0803 Mb (wic) =C2=A0 | =C2=A0 802 Mb (wic)= <-- cache!
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0520 Mb (cache) | =C2=A0= 520 Mb (cache)
=C2=A0 =C2=A0 =C2=A0 =C2=A0time: =C2=A03m28s =C2=A0 =C2= =A0 | time: =C2=A0 =C2=A020s =C2=A0 =C2=A0 =C2=A010.40x
=C2=A0 =C2=A0 = =C2=A0 =C2=A0------------ original ---------------------
=C2=A0 =C2=A0 = =C2=A0 =C2=A0time: =C2=A07m58s =C2=A0 =C2=A0 | time: =C2=A02m17s
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2.30x =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A06.85x

=C2=A0 =C2=A0 =C2=A0 =C2=A0 devel3 (publi= c) + npriv11 (private)
=C2=A0 =C2=A0 =C2=A0 =C2=A0----------------------= --------------
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fresh =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cache
=C2=A0 =C2=A0 =C2=A0 =C2= =A0------------ complete --------------
=C2=A0 =C2=A0 =C2=A0 =C2=A032466= Mb (max) =C2=A0 | 27741 Mb (max)
=C2=A0 =C2=A0 =C2=A0 =C2=A023298 Mb (r= est) =C2=A0| 18657 Mb (rest)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 4281 Mb (deb) = =C2=A0 | =C2=A04281 Mb (deb)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 8989 Mb (wic) = =C2=A0 | =C2=A08987 Mb (wic) <-- cache!
=C2=A0 =C2=A0 =C2=A0 =C2=A0 4= 398 Mb (cache) | =C2=A04398 Mb (cache)
=C2=A0 =C2=A0 =C2=A0 =C2=A0time: = 12m19s =C2=A0 =C2=A0 | time: =C2=A03m24s =C2=A0 =C2=A0 =C2=A0 3.62x
=C2= =A0 =C2=A0 =C2=A0 =C2=A0------------ original ---------------------
=C2= =A0 =C2=A0 =C2=A0 =C2=A0time: 21m17s =C2=A0 =C2=A0 | time: =C2=A08m28s =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 1.73x =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02.4= 9x

The mount bind approach is completely general but not yet used fo= r apt lists but
just for the .deb packages. Moreover, the rootfs tar opt= ions (--exclude/s) are
too extreme and the cached build is smaller than = the fresh build. This obviously
destroys the reproducibility but it is j= ust a single point change.
--0000000000005512a305f45aad87-- --0000000000005512a705f45aad89 Content-Type: text/plain; charset="US-ASCII"; name="performances-report-v4.txt" Content-Disposition: attachment; filename="performances-report-v4.txt" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_ldypz6aj0 CgpTU1RBVEUgQ0FDSEUgUkVXT1JLICsgQVBUIENBQ0hFIE1PVU5UIEJJTkQgCj09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQoKVGhlc2UgYXJlIHRoZSByZXN1bHRzIGlu IHRoZSBmaXJzdCBkZXZlbG9wbWVudCB2ZXJzaW9uIHRoYXQgd29ya3MgaW4gZXZlcnkKY29uZGl0 aW9uLiBJdCBjYXJyaWVzIG9uIGEgbG90IG9mIGRlYnVnZ2luZyBjb2RlIGFuZCBpdCBpcyBOT1Qg ZnVsbHkgb3B0aW1pc2VkCmJvdGggdW5kZXIgdGhlIGNvZGUtcmV2aWV3IGFuZCB0aGUgcGVyZm9y bWFuY2VzLgpOZXZlcnRoZWxlc3MsIGl0IHN0aWxsIGltcHJvdmVkOgoKICAgICAgICBkZXZlbDMg KHB1YmxpYykgKyBucHJpdjExIChwcml2YXRlKQogICAgICAgLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tCiAgICAgICAgICAgIGZyZXNoICAgICAgICAgICAgICBjYWNoZQogICAg ICAgLS0tLS0tLS0tLS0tIGJhc2ljLW9zIC0tLS0tLS0tLS0tLS0tCiAgICAgICAgMzk0OSBNYiAo bWF4KSAgIHwgIDMxNzYgTWIgKG1heCkKICAgICAgICAyOTY4IE1iIChyZXN0KSAgfCAgMjI3NyBN YiAocmVzdCkKICAgICAgICA0MjgxIE1iIChkZWIpICAgfCAgNDI4MSBNYiAoZGViKQogICAgICAg ICA4MDMgTWIgKHdpYykgICB8ICAgODAyIE1iICh3aWMpIDwtLSBjYWNoZSEKICAgICAgICAgNTIw IE1iIChjYWNoZSkgfCAgIDUyMCBNYiAoY2FjaGUpCiAgICAgICB0aW1lOiAgM20yOHMgICAgIHwg dGltZTogICAgMjBzICAgICAgMTAuNDB4CiAgICAgICAtLS0tLS0tLS0tLS0gb3JpZ2luYWwgLS0t LS0tLS0tLS0tLS0tLS0tLS0tCiAgICAgICB0aW1lOiAgN201OHMgICAgIHwgdGltZTogIDJtMTdz CiAgICAgICAgICAgICAgMi4zMHggICAgICAgICAgICAgIDYuODV4CgogICAgICAgIGRldmVsMyAo cHVibGljKSArIG5wcml2MTEgKHByaXZhdGUpCiAgICAgICAtLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0KICAgICAgICAgICAgZnJlc2ggICAgICAgICAgICAgIGNhY2hlCiAgICAg ICAtLS0tLS0tLS0tLS0gY29tcGxldGUgLS0tLS0tLS0tLS0tLS0KICAgICAgIDMyNDY2IE1iICht YXgpICAgfCAyNzc0MSBNYiAobWF4KQogICAgICAgMjMyOTggTWIgKHJlc3QpICB8IDE4NjU3IE1i IChyZXN0KQogICAgICAgIDQyODEgTWIgKGRlYikgICB8ICA0MjgxIE1iIChkZWIpCiAgICAgICAg ODk4OSBNYiAod2ljKSAgIHwgIDg5ODcgTWIgKHdpYykgPC0tIGNhY2hlIQogICAgICAgIDQzOTgg TWIgKGNhY2hlKSB8ICA0Mzk4IE1iIChjYWNoZSkKICAgICAgIHRpbWU6IDEybTE5cyAgICAgfCB0 aW1lOiAgM20yNHMgICAgICAgMy42MngKICAgICAgIC0tLS0tLS0tLS0tLSBvcmlnaW5hbCAtLS0t LS0tLS0tLS0tLS0tLS0tLS0KICAgICAgIHRpbWU6IDIxbTE3cyAgICAgfCB0aW1lOiAgOG0yOHMg ICAgICAgICAgICAKICAgICAgICAgICAgICAxLjczeCAgICAgICAgICAgICAgMi40OXgKClRoZSBt b3VudCBiaW5kIGFwcHJvYWNoIGlzIGNvbXBsZXRlbHkgZ2VuZXJhbCBidXQgbm90IHlldCB1c2Vk IGZvciBhcHQgbGlzdHMgYnV0Cmp1c3QgZm9yIHRoZSAuZGViIHBhY2thZ2VzLiBNb3Jlb3Zlciwg dGhlIHJvb3RmcyB0YXIgb3B0aW9ucyAoLS1leGNsdWRlL3MpIGFyZQp0b28gZXh0cmVtZSBhbmQg dGhlIGNhY2hlZCBidWlsZCBpcyBzbWFsbGVyIHRoYW4gdGhlIGZyZXNoIGJ1aWxkLiBUaGlzIG9i dmlvdXNseQpkZXN0cm95cyB0aGUgcmVwcm9kdWNpYmlsaXR5IGJ1dCBpdCBpcyBqdXN0IGEgc2lu Z2xlIHBvaW50IGNoYW5nZS4K --0000000000005512a705f45aad89--