From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182122219497062400 X-Received: by 2002:a05:6870:ac1f:b0:148:43a4:7296 with SMTP id kw31-20020a056870ac1f00b0014843a47296mr1841714oab.66.1672375128046; Thu, 29 Dec 2022 20:38:48 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a54:4f0b:0:b0:363:19d3:70f7 with SMTP id e11-20020a544f0b000000b0036319d370f7ls5391635oiy.9.-pod-prod-gmail; Thu, 29 Dec 2022 20:38:47 -0800 (PST) X-Google-Smtp-Source: AMrXdXsE++/MuxYm7TbDmvkFxZdzP22Ne2YnGu9z9DvHBX3ffDRVxpCeyO1WHqdm+YX2iCl/NA3b X-Received: by 2002:aca:2216:0:b0:35e:885e:1c53 with SMTP id b22-20020aca2216000000b0035e885e1c53mr12908316oic.21.1672375127529; Thu, 29 Dec 2022 20:38:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672375127; cv=none; d=google.com; s=arc-20160816; b=I9Gh4PsThUy23/LtnhAoURQRb0xpjNjAjDGSQ3+pdrK+nzZDHGgGR0LSTS33gUo6tj QQ42Vi3ixLfCa0Izrjq97ZC9lr574eBtCd4Cc3b0wKhmHj38kl1C9x8MvqoYLrbKB57L QnywYMYelE2nK6pU2vxxgOt67uoblVptEADC7qmRu4wUeifFDCNEkIIDd55zcy8U5JK1 7v87Qt85NOnJREcKRmkjDW2hYMdPGM8zd9XFoesu/b0+iOwTLQr8bYkBikzfsDu5IVbD Uu0rUDENZ5retMyJW3sIM+2G7pCeCKl0zGXS4QOM1yDufOeXscW+5yKm+zZ4Wv+epCLE esLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=cvtLkGfb/gwSzk2wu6IkOm+0S8oVSfOAAOaLPi0Qg6k=; b=u6tjRjbFtI7PXaepQ5egtI9S4MnBiGBsISXJ6SifgNjyUXudT2S0j1Kp7lLs1uFEQa +SUlG++0UkjI29rBo/wIFHnsmm5ellROxUf5Z1UdTU5nwbvhCLHcrBqOIadjZEC/VgGI hvDRaUpBEpTzNv6dSiPCiZstrBP648G/w2oga2N/k5iLKapb2y7O+HZIkK/zaTPQOm6U ZSNvUGD2wafEtpE4ShM83ueaIsL08iVP+yNKcAglWuF14e3YGVT2eLlOu5U4Rf3NInZW MLMEqbz4RDPncIobxlyZxZ/GE1MvzAegtZN46Mke9yaNRxfZuJ+Wd7KteLakkAjsR8Kd SdlA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id n62-20020acaef41000000b0035446541a0fsi3238702oih.5.2022.12.29.20.38.47 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 29 Dec 2022 20:38:47 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from hp.localnet (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 2BU4cfM0011144 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 30 Dec 2022 05:38:42 +0100 From: Uladzimir Bely To: "Roberto A. Foglietta" Cc: "isar-users@googlegroups.com" Subject: Re: Better way to handle apt cache needed Date: Fri, 30 Dec 2022 07:38:57 +0300 Message-ID: <3494812.dWV9SEqChM@hp> In-Reply-To: References: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: K3xW131uNmoH In mail from =D0=BF=D1=8F=D1=82=D0=BD=D0=B8=D1=86=D0=B0, 30 =D0=B4=D0=B5=D0= =BA=D0=B0=D0=B1=D1=80=D1=8F 2022 =D0=B3. 02:15:33 +03 user Roberto A.=20 =46oglietta wrote: > On Wed, 28 Dec 2022 at 12:04, Moessbauer, Felix >=20 > wrote: > > On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote: > > > In mail from =D1=81=D1=80=D0=B5=D0=B4=D0=B0, 28 =D0=B4=D0=B5=D0=BA=D0= =B0=D0=B1=D1=80=D1=8F 2022 =D0=B3. 12:45:07 +03 user Moessbauer, > > > Felix > > >=20 > > > wrote: > > > > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote: > > > > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix > > > > >=20 > > > > > wrote: > > > > > > The root cause for that behavior is the apt cache > > > > > > (deb_dl_dir_(import|export)), that copies all previously > > > > > > downloaded > > > > > > apt > > > > > > packages into the WORKDIR of each (bitbake) package. > > > > > > Given, that a common apt-cache is around 2GB and 8 tasks are > > > > > > run in > > > > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB > > > > > > for > > > > > > the > > > > > > buildchroots (host and target), in total ~30GB. > > > > >=20 > > > > > Thanks Felix for the report. IIRC, it was previously mounted and > > > > > was > > > > > supposed > > > > > to be converted to per-package hardlinks to parallelize sbuild > > > > > instances and > > > > > ease debugging (by knowing later which exact snapshot was fed to > > > > > a > > > > > specific > > > > > build). We use small (1- / 2-TB) SSDs as job storage and a huge > > > > > increase would > > > > > have been noticeable... We'll check. > > > >=20 > > > > Thanks! > > > >=20 > > > > I just noticed, that we even make an additional copy to have the > > > > packages inside the sbuild chroot. > > > >=20 > > > > This behavior is hard to notice on small and medium sized projects > > > > (and > > > > given that IOPS are not an issue). But any quadratic behavior will > > > > eventually make the build impossible. And as Florian said, many of > > > > our > > > > ISAR users build in VMs on shared filesystems, where IO is > > > > extremely > > > > expensive / slow. If we could optimize that, it would be a huge > > > > benefit > > > > for a lot of users. > > >=20 > > > I've just did some measurements: > > >=20 > > > - run Isar build for qemuarm64 (8 cores =3D max 8 build tasks in > > > parallel) > > > - measured system disk consumption every 5 seconds > > >=20 > > > Results: > > > - 'downloads/deb' directory finally takse 480MiB > > > - After the build finished, disk space decreased by ~9GiB > > > - During the build maximum disk space decrease was ~16GiB) > > >=20 > > > It means, that we really had about 16 - 9 =3D 7GiB of space temporarly > > > used by > > > parallel builds and it really corresponds to 8 * 2 * 480MiB. > > >=20 > > > So, the main goal now is to minimize this value. > >=20 > > Even the 7GiB should be further reduced, as some major contributor to > > that is the apt-cache for the buildchroots and the images itself, which > > is never cleaned up. While this is not so much an issue for CI systems, > > it is definitely annoying on local builds. Especially, when working on > > multiple ISAR based projects. > >=20 > > I would recommend to have a look at the build/tmp folder using ncdu -x > > (-x to not cross filesystems / temporary mounts). >=20 > Hi all, >=20 > I did some changes to reduce disk usage and to speed up the building. > The results are quite impressing so, before everything else I am going > to show you the numbers: >=20 > results before and after the changes >=20 > 43954 Mb (max) | 8657 Mb (max) > 26548 Mb (rest) | 4118 Mb (rest) > 3741 Mb (deb) | 3741 Mb (deb) > 820 Mb (wic) | 820 Mb (wic) > 11789 Mb (cache) | 579 Mb (cache) > time: 8m33s | time: 4m34s >=20 > The changes has been committed on the default branch (evo) on my ISAR fork >=20 > https://github.com/robang74/isar >=20 > * 9fd5282 - changes for a faster build using less disk space, p.1 > * 1b6ac1e - bugfix: no sstate archive obtainable, will run full task inst= ead > * 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e >=20 > The two bug fixes are necessary to test for regression because the top > commit changes the sstate cache creation. The top commit uses hard > physical links to obtain these results, so it may not work in some > cases like distributed or networked filesystems which do not support > hard links (however, they might be smart enough to do a bare copy but > I cannot grant that). >=20 > However, this is the reason because I titled part one the patch. I am > thinking of addressing the issue in a more radical way but for the > moment this seems quite interesting. > I wish to have some feedback about the limitation about using uncommon > filesystems and which one have a nice fallback to address unsupported > features. >=20 > Looking forward your opinions, R- Hello Roberto. I also did some similar improvements. They are not so radical as yours and = the=20 benefit is not so big. But anyway, I'm sending 1st version of patchset. For= =20 the 2nd version I plan to play with your idea with CACHEDIR.TAG and `tar - exclude-caches` option and to improve work with additional internal sbuild= =20 copying that you've already implemented.