From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182122219497062400 X-Received: by 2002:a05:6870:9a95:b0:14c:8d88:de37 with SMTP id hp21-20020a0568709a9500b0014c8d88de37mr1520335oab.19.1672355771848; Thu, 29 Dec 2022 15:16:11 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aca:6c1:0:b0:359:ca69:f473 with SMTP id 184-20020aca06c1000000b00359ca69f473ls5580844oig.10.-pod-prod-gmail; Thu, 29 Dec 2022 15:16:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXtjWt5xE6hMl+RpVb/qcBKLWMGB31K3PcjMAFnRiNheapZvFhsd6RvWomI5/ZYVNVm3Tw7f X-Received: by 2002:a05:6808:20a:b0:360:dae5:7fce with SMTP id l10-20020a056808020a00b00360dae57fcemr19999469oie.33.1672355771310; Thu, 29 Dec 2022 15:16:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672355771; cv=none; d=google.com; s=arc-20160816; b=TXY/8e3NoYkELydWoU+R0S4h0Yf4zUBnAsZdO7cHDvLi0vMEVFMspXb4pGPp1gbVy7 U0+2iANxAUIni7m88CcTNmTbZuYIY1HEi35CWnAZq5rM0/0xd1NscvN8gyR0fcxEddTj r2IlR18ik8ywuFLz4cY+mcnURb7hFa/iHCmFjLq8D1mInLbPppsMjoRhnDBOVAf/qUcz IToFuTAcjWesfLCF0rbvNg2YmVSK8MrGk80sCiycZ0X/MiBPjNBo5WPF4Y1DsQaSHQSF T/bbrsA0nmtx/L5BD36G9AUUrEPrFe0qErfnX1z9XOynslmgjzgKMnPAU1pjxQeehz+z OGeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=GC9GdyPxhi6DrL4///7fvU2QjdaYPzYR4ErtHnvRjtw=; b=B6s+9Je9GsnEdwG2FbS3eTdtyeHIpR4ELWTavwYMWDz9OYVZFH2JE6rEYgZzly7NwQ nLYrtGzzzOKdB6MQwjpM0iAzjZ424AELmY+O3iykjhQctadbeZ2RFE7r92AvmTg9sHis bZD1E0aaj5R6fQ52eviqIfa41Bk+SjQ1K4khNwthC/spQbZhJ+UUyFQ8Pk+ZeMlAhvJN +9pUtDvHSiR/yUVrfXFp6IYDXjFuxyC+Z+5iHwJjLV9amyTMFXZ8Hxzrbqps++22btKS zduG3g0RBsbzomXmKpzNujJMoVZabNz8w9pPnYdkxteneeCra+qJpq2sfSyx4cPMtTk6 NZdg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="e7iCqIR/"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::732 as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com. [2607:f8b0:4864:20::732]) by gmr-mx.google.com with ESMTPS id n62-20020acaef41000000b0035446541a0fsi3169345oih.5.2022.12.29.15.16.11 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 29 Dec 2022 15:16:11 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::732 as permitted sender) client-ip=2607:f8b0:4864:20::732; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="e7iCqIR/"; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::732 as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qk1-x732.google.com with SMTP id a25so9606885qkl.12 for ; Thu, 29 Dec 2022 15:16:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=GC9GdyPxhi6DrL4///7fvU2QjdaYPzYR4ErtHnvRjtw=; b=e7iCqIR/k06ijbwaY3MdP6oiOof29XCjZH7EvVkF5BKwJHuFncgb0bxEV5UGSZk+xQ 4+bN22VvUSxJuTuK1Og7ST4NCJMmUTsyEHASv9gO5OnMOzXDdFLXDd0j/9f0sgISjBPP ZAdgmFcElVlqH/13PElbHKgWDKOSHySlGdaBMa0ULYC1IEVtdijRvScR8V4J4y5CGZFc zA5sQm7u/Y6M82k+MIHQVo55YNJrZMIhi1ptt4G6ZNsDbf9u7d6IIt/8X5YcvJIDgmnn s8H8Ce6Tx1TNsyCLkhWb/tOnmPh7sJxbcUfVxlHAbVt1Xs6XmnIUMiIXtvdYoxQr9Qd4 abNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GC9GdyPxhi6DrL4///7fvU2QjdaYPzYR4ErtHnvRjtw=; b=C/K+cIYaM3blRjQ2N8QlKdthHHdDTdf9Ims4hlQgNYxQQDNFLcjb8zEGH7uKD5PiWw GVn77J+1DKW1LN7RyQWVmM0uSUzjYLIOLsrdlisaoEaLZ3KZiJsyNOnthX9+aZwEOE+J impcuCT22e30BgkcSmSut8bH0FYOv8Z3HUD+rVjRPG5RAaZonOoxs1sMgB1RNEAyhN4x zlR9xfclJ8oTAKH8tS28HntYeliw9CT8p9CcVb/LbqmM6Do7Qhc5Jc9iXQyMpKNy27jx tm3AiXRVt5RVE2MLUFjSgX97xOfAYjvG3Gtzs0IbsF0RS4p9dLwjgbc5erpw5YxVs2uS TdfA== X-Gm-Message-State: AFqh2koTMKi9w2MrM4bbnm5C/f/d9BCht5tICk6vIhRu4JiqrfJ5YUCu l1TuYk1nYLoRLUbaKSWQEUnwKoGuREaDtxKrii6DNl7BABK2 X-Received: by 2002:ae9:ed86:0:b0:702:4c8f:8316 with SMTP id c128-20020ae9ed86000000b007024c8f8316mr974250qkg.533.1672355770675; Thu, 29 Dec 2022 15:16:10 -0800 (PST) MIME-Version: 1.0 References: <371e4d826cca6aaba11a4222fef547b134ed6ce7.camel@siemens.com> <38d18c245baa4f685642eafa9a52ab9b9ae9001c.camel@siemens.com> <4769513.OV4Wx5bFTl@hp> In-Reply-To: From: "Roberto A. Foglietta" Date: Fri, 30 Dec 2022 00:15:33 +0100 Message-ID: Subject: Re: Better way to handle apt cache needed To: "Moessbauer, Felix" Cc: "ubely@ilbers.de" , "isar-users@googlegroups.com" , "ibr@radix50.net" , "Schild, Henning" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-TUID: OhgWJEQmTagR On Wed, 28 Dec 2022 at 12:04, Moessbauer, Felix wrote: > > On Wed, 2022-12-28 at 13:23 +0300, Uladzimir Bely wrote: > > In mail from =D1=81=D1=80=D0=B5=D0=B4=D0=B0, 28 =D0=B4=D0=B5=D0=BA=D0= =B0=D0=B1=D1=80=D1=8F 2022 =D0=B3. 12:45:07 +03 user Moessbauer, > > Felix > > wrote: > > > On Wed, 2022-12-28 at 10:21 +0100, Baurzhan Ismagulov wrote: > > > > On Wed, Dec 28, 2022 at 09:02:13AM +0000, Moessbauer, Felix > > > > wrote: > > > > > The root cause for that behavior is the apt cache > > > > > (deb_dl_dir_(import|export)), that copies all previously > > > > > downloaded > > > > > apt > > > > > packages into the WORKDIR of each (bitbake) package. > > > > > Given, that a common apt-cache is around 2GB and 8 tasks are > > > > > run in > > > > > parallel, this gives already 16GB for the tasks, and 7 * 2GB > > > > > for > > > > > the > > > > > buildchroots (host and target), in total ~30GB. > > > > > > > > Thanks Felix for the report. IIRC, it was previously mounted and > > > > was > > > > supposed > > > > to be converted to per-package hardlinks to parallelize sbuild > > > > instances and > > > > ease debugging (by knowing later which exact snapshot was fed to > > > > a > > > > specific > > > > build). We use small (1- / 2-TB) SSDs as job storage and a huge > > > > increase would > > > > have been noticeable... We'll check. > > > > > > Thanks! > > > > > > I just noticed, that we even make an additional copy to have the > > > packages inside the sbuild chroot. > > > > > > This behavior is hard to notice on small and medium sized projects > > > (and > > > given that IOPS are not an issue). But any quadratic behavior will > > > eventually make the build impossible. And as Florian said, many of > > > our > > > ISAR users build in VMs on shared filesystems, where IO is > > > extremely > > > expensive / slow. If we could optimize that, it would be a huge > > > benefit > > > for a lot of users. > > > > I've just did some measurements: > > > > - run Isar build for qemuarm64 (8 cores =3D max 8 build tasks in > > parallel) > > - measured system disk consumption every 5 seconds > > > > Results: > > - 'downloads/deb' directory finally takse 480MiB > > - After the build finished, disk space decreased by ~9GiB > > - During the build maximum disk space decrease was ~16GiB) > > > > It means, that we really had about 16 - 9 =3D 7GiB of space temporarly > > used by > > parallel builds and it really corresponds to 8 * 2 * 480MiB. > > > > So, the main goal now is to minimize this value. > > Even the 7GiB should be further reduced, as some major contributor to > that is the apt-cache for the buildchroots and the images itself, which > is never cleaned up. While this is not so much an issue for CI systems, > it is definitely annoying on local builds. Especially, when working on > multiple ISAR based projects. > > I would recommend to have a look at the build/tmp folder using ncdu -x > (-x to not cross filesystems / temporary mounts). Hi all, I did some changes to reduce disk usage and to speed up the building. The results are quite impressing so, before everything else I am going to show you the numbers: results before and after the changes 43954 Mb (max) | 8657 Mb (max) 26548 Mb (rest) | 4118 Mb (rest) 3741 Mb (deb) | 3741 Mb (deb) 820 Mb (wic) | 820 Mb (wic) 11789 Mb (cache) | 579 Mb (cache) time: 8m33s | time: 4m34s The changes has been committed on the default branch (evo) on my ISAR fork https://github.com/robang74/isar * 9fd5282 - changes for a faster build using less disk space, p.1 * 1b6ac1e - bugfix: no sstate archive obtainable, will run full task instea= d * 2cc1854 - bugfix: do_copy_boot_files was able to fail but then set -e The two bug fixes are necessary to test for regression because the top commit changes the sstate cache creation. The top commit uses hard physical links to obtain these results, so it may not work in some cases like distributed or networked filesystems which do not support hard links (however, they might be smart enough to do a bare copy but I cannot grant that). However, this is the reason because I titled part one the patch. I am thinking of addressing the issue in a more radical way but for the moment this seems quite interesting. I wish to have some feedback about the limitation about using uncommon filesystems and which one have a nice fallback to address unsupported features. Looking forward your opinions, R-