From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7068176960994672640 X-Received: by 2002:ac2:58f2:0:b0:43c:6f79:cea5 with SMTP id v18-20020ac258f2000000b0043c6f79cea5mr1109020lfo.104.1645690791952; Thu, 24 Feb 2022 00:19:51 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:15a3:b0:443:7b15:f451 with SMTP id bp35-20020a05651215a300b004437b15f451ls480489lfb.0.gmail; Thu, 24 Feb 2022 00:19:50 -0800 (PST) X-Google-Smtp-Source: ABdhPJxqLefDZ+SSVobiOz4CcrJdJ6uXVriChSGgq2QfWKdc0J2xRnVRQECpXKVNKRm3/AgQKxsS X-Received: by 2002:a19:6456:0:b0:437:974a:f5ce with SMTP id b22-20020a196456000000b00437974af5cemr1152944lfj.394.1645690790749; Thu, 24 Feb 2022 00:19:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645690790; cv=none; d=google.com; s=arc-20160816; b=HY8TjEjO4t4z7ky1VgYFqGaWxo+UTeTaMFn3fBRoJPTxythZVSMwPQq8CYJxwwKiK4 BfySs//Liv4104dk20tmC1YJpWIxVvEe9R4++l6xs0rHB/lbVI1LM+6QBrwxnrjYe2a4 RxxXsRplzQY6EXGhDtQ8z0BaYL/21BwRtjkSl31y3vNcCOSuRs7C5C5Y21GHtA92X8uX ZhzCh78teoNyA8H/8RPBHCHOKSXU8s7wSYJSzG6TSizMcqXuUMWpQXstOjLGZihDCaD+ Qfe26mZheWDeWBn9CV4Po/60apSu4tA3huvUGpgvXezaO457Mo7eCwmFejEh+DDLrlGw 3zwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=4oyVYPo8lMztqbKqpaRRXPDc5LJitJTM2bemX3Ych0g=; b=whfybO09RthcFcGAWEYRts77AWLqETrBcTffZeZG+CuXoF3KpjY9l0tX8vrOW7pk/7 vvOR9nDK81SdbfqVcDIXD3D7uFd+mC0i0APL/HYlMDNlLoMry3ALD707/oeD52wltdoq mcrm9etoOkl4TvV86+/SjiEqisqlwwuYIkZgH+TY5SFJxbAbJHf6OWMlieJtdA8jArsS /N9h0Bhd6VerP44wpR3Hzd1IFTR0YyOvSgoVN03emo6nk3p1nWlwRL5n6HBENEJcp79X mOJsSdSXV85DPPYEEXiEyl4BIVMcHjylEQlt6ZS/xQnJAzCTTTR0NM357Ca2PnSPX1cH sFcw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id b40si55105ljf.7.2022.02.24.00.19.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 Feb 2022 00:19:50 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 21O8JmUt029925 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 24 Feb 2022 09:19:49 +0100 From: Uladzimir Bely To: "isar-users@googlegroups.com" Cc: "Schmidt, Adriaan" Subject: Re: [PATCH v2 0/1] sstate: do not keep intermediate sstate files Date: Thu, 24 Feb 2022 11:19:45 +0300 Message-ID: <3410387.dWV9SEqChM@home> In-Reply-To: References: <20220224073629.23717-1-ubely@ilbers.de> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: u0zgR5cLVACv In the email from Thursday, 24 February 2022 10:56:48 +03 user Schmidt, Adriaan wrote: > Uladzimir Bely, 24. Februar 2022 08:36: > > Sstate is quite demanding for disk space. And it often causes failures > > in CI, related to 'out-of-space'. > > If those few GB cause problems, then you really need a larger disk :) > But I get the point, and I agree that we should try to save space. > These few GBs result to ~100GiB per CI job (when doing full builds) :) > > This patchset minimizes space usage by intermediate state files by > > removing them: > > - after sstate-cache populated: with separate task in task queue > > I would like to avoid adding extra tasks. There seems to be a hook > to insert functions (SSTATEPOSTCREATEFUNCS), but as sstate is not > really documented, I have not yet figured out how to make our use case work. Yes, it would be better if there are some post-sstate functions we could use instead of inserting one more task after one we are handling... > > - after sstate-cache unpacked: by direct call of this task > > > > Some results shown below for three different cases: > > > > - sstate disabled; > > - sstate enabled; > > - sstate enabled + cleanup patch. > > > > bullseye-arm64 cross build with full IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 5,7G build-nosstate > > 12G build-sstate > > 7,6G build-sstate-cleanup > > > > bullseye-arm64 cross build with empty IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 2,5G build-nosstate > > 5,3G build-sstate > > 3,4G build-sstate-cleanup > > > > bullseye-amd64 native build with empty IMAGE_INSTALL list: > > > > $ sudo du -sh build-* > > 3,2G build-nosstate > > 6,4G build-sstate > > 4,2G build-sstate-cleanup > > > > > > Changes since v1: > > - patchset renamed from "sstate: compress rootfs sstate files"; > > - sstate intermediate files compression replaced by removing; > > - dpkg intermediate files are also handled. > > It's unnecessary for dpkg, because we use hardlinks to the *.deb file, > so no extra disk space is consumed for the sstate preparation. Yes, I saw it, just implemented in the similar way. It may be useful if files are copied to another (slow) disk after build finishes - I'm afraid hardlinks won't work in this case. > > Adriaan > > > Uladzimir Bely (1): > > sstate: do not keep intermediate sstate files > > > > meta/classes/dpkg-base.bbclass | 7 +++++++ > > meta/classes/rootfs.bbclass | 7 +++++++ > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 +++++++ > > 3 files changed, 21 insertions(+) > > > > -- > > 2.20.1 > > > > -- > > You received this message because you are subscribed to the Google Groups > > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar-users/20220224073629.23717-1-> > ubely%40ilbers.de. -- Uladzimir Bely