From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a05:6808:191:b0:35b:4050:5dd with SMTP id w17-20020a056808019100b0035b405005ddmr2156635oic.79.1672987692828; Thu, 05 Jan 2023 22:48:12 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6870:131a:b0:154:df53:5a8 with SMTP id 26-20020a056870131a00b00154df5305a8ls648887oab.8.-pod-prod-gmail; Thu, 05 Jan 2023 22:48:12 -0800 (PST) X-Google-Smtp-Source: AMrXdXvyxHMopu8glI2d600kFTpQAl/0rRuWeJR2wyfxQ1hivNO7GHcWNxM4FTVCRyfDLD2z2YQ4 X-Received: by 2002:a05:6870:8996:b0:14f:c6f4:ca3e with SMTP id f22-20020a056870899600b0014fc6f4ca3emr20601573oaq.0.1672987691986; Thu, 05 Jan 2023 22:48:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672987691; cv=none; d=google.com; s=arc-20160816; b=ujx+2NQj26fgRyHhyI6z8aO8vvy1chEnj02SqL2bv7toEO5sJCzSFicCkpSQlrv5LJ PSsRBiAezoPSoCwm+zX6S/ODrbDzvStUTHy1un09shMrxXPGsgAlH8YJQeVJ5eCjlLis 5NcIZIZt18SCkzAr4/OqF29SO+SEc219YwQxWsmMeOTTYiSA3IQ+3qyjzF/H7BdcmTv4 9zerk7QCULPEhkD5xzr6Gt3cyb7lFhMWzckMspiBp7FCoYxMUQ3lk662JL8+6qBYJWrC GuwWMERvUZVpNIqTR+8HfnErbmNEzPbQnljfeTbj+6UY3A2KiDysn2ypGdBnClbBDh2R Kh5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=+HDWFMrvnzAn9qPcJoVvduXTpr3v4vW1D8Nl713yhRk=; b=LjxGYulcgUN7zxijHXxtlCo3swsr2s6N/jGTz+3nSpxV5ucb8c9om1UCQeR0Ae0Tn0 8w5ffLwZQy5EDXiJCb+oHeRwCyyzdSyxw+5Lgv4rOKovEcYDRLmjb0URPJY8XPOPufM3 rAGiptWw63FWKEDBxNjo0isb7gTf1kz3S5qWdsyxZF7iGKLn18xXpQF6TXbW6rsnrhEa nM7DFtWOxqITSEJ6AXz4ZbeJK4eKkuAqgifemBW+h2pN1UPCCu71zJhuc7wZRmrorirv A+C8k/VPUBxnV9uReIDaxOxhrBtrUel6Ny8wd5fP9UeB5LxbkNvJR0/PO6cKqaBpyfIJ VTEg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id i8-20020a056871028800b001438dde860esi46536oae.1.2023.01.05.22.48.11 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 05 Jan 2023 22:48:11 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 3066m9J3011292 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 6 Jan 2023 07:48:10 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v2 0/3] Improving apt cache Date: Fri, 6 Jan 2023 07:48:06 +0100 Message-Id: <20230106064809.10412-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: cm15WbuyeM/B Currently, apt cache (e.g. `var/cache/apt/archives`) import and export functions are not optimal. Multiple files are imported from global DL_DIR to package WORKDIR, increasing disk IO and size needed. Also, various chroots (bootstrap, buildchroot, sbuild chroot) include their apt caches to sstate cache files. This patchset switches to hardlinks instead of copies and removes apt cache from bootstrapped images ans sstate caches. Currently (measured on qemuarm64-bullseye cross-compilation with maximum 8 parallel tasks (e.g. 8 CPU cores)): - build directory size reduced from 8906 to 6675 MiB - runtime maximum disk usage reduced from 15965 to 8501 MiB TODO: - cleanup other rootfs's (sbuild-chroot, buildchroot). Actually, this won't bring much benefit, but why not cleanup final rootfs's apt cache when the build finishes? - deal with additional copying in sbuild routines (patch 3). We could use hardlinks instead of copying packages between upper layer (where sbuild temporarly keeps them) and workdir rootfs, but it is not expected to bring much benefit, since upper layer temporary nature. Additinal measurements are required. Changes since v1: - Simplified cleanup of apt cache in debootstrap rootfs. - Now "ln" instead of "cp -l" used. - Removed apt cache contents from sstate cache. The idea is proposed in patch 3, but it was reworked and fixed. Firstly, CACHEDIR.TAG can't be just a file (e.g. created by 'touch'), it should include some specific signature [1]. Secondly, it's easier to just create this tag in bootstrapped rootfs and it will be automatically used in all derivatives (sbuild-chroot/buildchroot/image). So, the original patch from Roberto A. Foglietta was simplified. This patchset includes (or absorbs) the logic from p1..p3 patches of the series Roberto prosposed. What concerns additional patches, they don't let us benefit much, but require quite significant changes in Isar, so we should check twice if they are worth including. [1] https://bford.info/cachedir/ - Roberto A. Foglietta (1): Changes for a faster build using less disk space Uladzimir Bely (2): Clean apt cache from debootstrapped rootfs dirs Use hardlinks in deb-dl-dir functions meta/classes/deb-dl-dir.bbclass | 6 +++--- meta/classes/rootfs.bbclass | 3 ++- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 +++++++++- 3 files changed, 14 insertions(+), 5 deletions(-) -- 2.20.1