From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a05:6102:1179:b0:3b1:4c76:b21e with SMTP id k25-20020a056102117900b003b14c76b21emr10301424vsg.49.1673249943248; Sun, 08 Jan 2023 23:39:03 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6102:23c9:b0:3ce:885f:cc2f with SMTP id x9-20020a05610223c900b003ce885fcc2fls1080893vsr.8.-pod-prod-gmail; Sun, 08 Jan 2023 23:39:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXvuh5AqarHp7atr+10Hc7E/Xe0/HCZyTPGj/DY9rrDl1+B3v+oFD0xRUBKdvSmeNuk3wR8r X-Received: by 2002:a67:e354:0:b0:3d0:8c42:75e3 with SMTP id s20-20020a67e354000000b003d08c4275e3mr1950173vsm.24.1673249942505; Sun, 08 Jan 2023 23:39:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673249942; cv=none; d=google.com; s=arc-20160816; b=L0iiPy16Ne+CIY7r/zacKu1fg0Rm5IGuK/3Vwnbbgu941NoL5xI8+u72ouBmKJw5t2 Byt/fkLaEQdSso6lO+r9v81Ir/87jR3WF+wvoPa8Nh2xZacz/YJ2hbsaQwi+8Iwk6Cqv HsvRPAxznzTIP4cCOwlAvsE+/7BfxkbVqWmDerhBMBUodxNZeeuZYYnZ5ThTD4tYktFZ vPgt15BGSIZyQiu+/DRBTpmDf7+VA09ISNfFCZvDLq7UMRZpBjUolNaeVe2Hf7Mxp9pB 83C/VTYpthf6kARAYMmdR7EvgiwdE2NPaRlNWckIpNv9Hc5BdSg6J3IjRdrzMVOjFPWR ykJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=qgdLqPEsK4Z/868NXOVmgpVU6ECj2RvOD9fflCtwtS8=; b=HON+b4OnMUL2niW8u3Oucg17Os4SKWOkAtQEuuAwtbCeempPza0Kv7TGSYiLAMPlA9 8nsVTmJqu3saeFUreVbaYsSU1bdiwFVyvRNNtWTTGHwHRq3EI4ci47bFCOuz+Lp7zXr3 +79kqo4Lyw3juZoqbAgyubkj8yoiQ5NvL8NYuVdefnC6W4iGjvqsTEDbCl/+FkLhmIv0 7thkzY4ekz2wOJ8JkCrWDxychbJGnMDDK6SKtguOUoomWI5iKT8h2QRVQxZ6R2iRqa5T fv8WFvC/z5ZfvLSUx+08No3HxpXruY2akDRAwv7ujtPcqwGk7yVdI7cexJ0F5/d9FRAG om1Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id p13-20020a67f30d000000b003d04209e4e2si392620vsf.0.2023.01.08.23.39.01 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 08 Jan 2023 23:39:02 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 3097cxqi002210 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 9 Jan 2023 08:39:00 +0100 From: Uladzimir Bely To: "isar-users@googlegroups.com" , "Moessbauer, Felix" Subject: Re: [PATCH v2 0/3] Improving apt cache Date: Mon, 09 Jan 2023 10:39:00 +0300 Message-ID: <6842478.jJDZkT8p0M@home> In-Reply-To: <00ca12addc6e3b93aeca6127124cfc2f0a1aabf2.camel@siemens.com> References: <20230106064809.10412-1-ubely@ilbers.de> <00ca12addc6e3b93aeca6127124cfc2f0a1aabf2.camel@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: KwG0AGimQ84w In the email from Monday, 9 January 2023 09:32:29 +03 user Moessbauer, Felix wrote: > On Fri, 2023-01-06 at 07:48 +0100, Uladzimir Bely wrote: > > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and > > export > > functions are not optimal. Multiple files are imported from global > > DL_DIR to package WORKDIR, increasing disk IO and size needed. > > > > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include > > their apt caches to sstate cache files. > > > > This patchset switches to hardlinks instead of copies and removes apt > > cache from bootstrapped images ans sstate caches. > > > > Currently (measured on qemuarm64-bullseye cross-compilation with > > maximum 8 parallel tasks (e.g. 8 CPU cores)): > > - build directory size reduced from 8906 to 6675 MiB > > - runtime maximum disk usage reduced from 15965 to 8501 MiB > > > > TODO: > > - cleanup other rootfs's (sbuild-chroot, buildchroot). > > Actually, this won't bring much benefit, but why not cleanup final > > rootfs's apt cache when the build finishes? > > - deal with additional copying in sbuild routines (patch 3). > > We could use hardlinks instead of copying packages between upper > > layer (where sbuild temporarly keeps them) and workdir rootfs, > > but it is not expected to bring much benefit, since upper layer > > temporary nature. Additinal measurements are required. > > > Hi, > > if the cache is 20GB, we still copy 20GB of data into the upper layer > (per sbuild task). > This has a huge impact and I would really appreciate if we could use > hardlinks for that part. > > Felix > Hi. I need to check if it really works. My doubts are related to overlayfs: if we are really able to create hardlinks between DL_DIR (that is common linux filesystem) and upper layer of overlayfs (that is technically the same partition, but it is a "mount option" of the different "overlay" filesystem). > > > > > Changes since v1: > > - Simplified cleanup of apt cache in debootstrap rootfs. > > - Now "ln" instead of "cp -l" used. > > - Removed apt cache contents from sstate cache. The idea is proposed > > in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG > > can't > > be just a file (e.g. created by 'touch'), it should include some > > specific signature [1]. Secondly, it's easier to just create this tag > > in bootstrapped rootfs and it will be automatically used in all > > derivatives (sbuild-chroot/buildchroot/image). So, the original patch > > from Roberto A. Foglietta was simplified. > > > > This patchset includes (or absorbs) the logic from p1..p3 patches of > > the series Roberto prosposed. What concerns additional patches, they > > don't let us benefit much, but require quite significant changes > > in Isar, so we should check twice if they are worth including. > > > > [1] https://bford.info/cachedir/ > > - > > > > Roberto A. Foglietta (1): > > Changes for a faster build using less disk space > > > > Uladzimir Bely (2): > > Clean apt cache from debootstrapped rootfs dirs > > Use hardlinks in deb-dl-dir functions > > > > meta/classes/deb-dl-dir.bbclass | 6 +++--- > > meta/classes/rootfs.bbclass | 3 ++- > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++- > > 3 files changed, 14 insertions(+), 5 deletions(-) > > > > -- > > 2.20.1 > > > >