From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067831991103651840 X-Received: by 2002:a17:906:bc8d:b0:6cf:6b41:e48e with SMTP id lv13-20020a170906bc8d00b006cf6b41e48emr664810ejb.372.1645674946966; Wed, 23 Feb 2022 19:55:46 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:907:77da:b0:6cf:8e4e:499d with SMTP id kz26-20020a17090777da00b006cf8e4e499dls124550ejc.5.gmail; Wed, 23 Feb 2022 19:55:46 -0800 (PST) X-Google-Smtp-Source: ABdhPJxQ/mQpcwrBdnjn4USLITOy8vZlt1YNVPqGYJP/DGtd+bWGKo2q57tb5ocL6XkE+65cPGSg X-Received: by 2002:a17:906:22cf:b0:6cd:2c02:45fc with SMTP id q15-20020a17090622cf00b006cd2c0245fcmr669871eja.295.1645674945978; Wed, 23 Feb 2022 19:55:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645674945; cv=none; d=google.com; s=arc-20160816; b=0KLMWuMQ97Y2dA752N4/V/c1NmEkHo5yj5ifawY91iWxNJF1i0E/HvAAZo9GcodJ8i EkM+ld2/8Y2922hB+4u9t1u7l3d6QLRMjo6e8NP+8wSjKAQivlz4RRZ3up9a4C6WbJGI B50Le9ZkVJOwAHLrQULC87pvURhrrppZTTP/rPcXgX+51PlzrVh4XHvAqYxfuCK/6ZM4 NkPv/VDFeMqynX8YKD5QawUsybm/lTwPC4AaJ47Kcuxf1VjVUgfFagUC3uorCXr97XyP C9ZFxbHRPJ6aNuFd7rxH5xjQrCCdd/Orc1e6W/LzatSuF/4mVXIIIgG+G//d/4kDofBJ fhOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=9+Qi0+dVsFe4hlWQNIcTIu/YlwToocKKnvIg4YN6cPo=; b=CGPTRAKhM/+RvKt7a+oUL3WEKz68TyJagKNAKzb4LjGkprzNVqZsJ3AysJ3JxFfZ00 3Pq323o6p7gW77QIT4BCmEfaKPBVH1FgwsaXOfvkjMoF8fWrU7wlT5D1NzreNvUjlXQs vUltJYkb0SLnryc7ew3lwlkBifVwGJvNUwFyxnUvkZ0OtVPsHcdc2uOovF3OGKBbPTHL MNVjkXu5Mug4hQHt6TwBxUSXvyDx6Ed0DPpx0vafHUmhDOAkzQPCf2KRvyB9SdBWnNDl yXM7kapy76uLMP84lMTRkL+7HOFVbllKLuOO50aW+o/6Hbpv2NxxbkBayWqjnuzoeHvh aJtQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id d13si46418ede.0.2022.02.23.19.55.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 23 Feb 2022 19:55:45 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 21O3ti12029044 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 24 Feb 2022 04:55:45 +0100 From: Uladzimir Bely To: Henning Schild Cc: isar-users@googlegroups.com, "Schmidt, Adriaan (T CED SES-DE)" Subject: Re: [PATCH 1/1] sstate: compress rootfs sstate files Date: Thu, 24 Feb 2022 06:55:41 +0300 Message-ID: <25321236.1r3eYUQgxm@home> In-Reply-To: <20220223223306.7b74e270@md1za8fc.ad001.siemens.net> References: <20220223091749.17260-1-ubely@ilbers.de> <1793829.LH7GnMWURc@home> <20220223223306.7b74e270@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: n9HdoHc9l88i In the email from Thursday, 24 February 2022 00:33:06 +03 user Henning Schild wrote: > Am Wed, 23 Feb 2022 18:52:54 +0300 > > schrieb Uladzimir Bely : > > In the email from Wednesday, 23 February 2022 18:43:21 +03 user > > > > Henning Schild wrote: > > > This is wrong. Everything you put into sstate is already compressed > > > there. > > > > > > So what you probably want is to "move" stuff into sstate and delete > > > setscene after unpack so temporary tarballs maybe do not hang > > > around. > > > > > > That will save much more space and time, but i am not sure it would > > > be "allowed". That is why i added Adriaan on cc. > > > > Yes, it may be wrong because this tarballs might be needed to compare > > hashes by sstate. But why not use them already compressed and save > > space in workdirs? > > The real way to save the space is to not keep sstate stuff at all (in > WORKDIR). When putting things into sstate we should remove it from > WORKDIR right after it went into sstate. When getting stuff out we > should remove staging stuff at the end of setscene. > > SSTATE_BUILDDIR seems to take care of some of the bits, i did not > manage to write the correct code for all of that. But compressing bits > we do not actually want to keep does not seem right. > I also tried to remove these files, but didn't succeed. It's easy to do for getting stuff from sstate cache - we can just remove .tar file at the end of *_setscene(). But it's a bit more tricky, when we put stuff to sstate cache - it's the last stage and we can't execute any custom code. Even [postfuncs] are executed before sstate cache is created. Probably, additional task for cleanup sstate dir could be used. For example, do_bootstrap => (do_bootstrap_sstate_cleanup) => do_build. > So a clear NACK from me. > > Henning > > > > Henning > > > > > > Am Wed, 23 Feb 2022 10:17:49 +0100 > > > > > > schrieb Uladzimir Bely : > > > > This saves about 1-2 GiB per multiconfig, at the expense of > > > > approximately 20-40 seconds for packing the files. > > > > > > > > Signed-off-by: Uladzimir Bely > > > > --- > > > > > > > > meta/classes/rootfs.bbclass | 4 ++-- > > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++-- > > > > 2 files changed, 4 insertions(+), 4 deletions(-) > > > > > > > > diff --git a/meta/classes/rootfs.bbclass > > > > b/meta/classes/rootfs.bbclass index 2bdb3b6d..15e87bec 100644 > > > > --- a/meta/classes/rootfs.bbclass > > > > +++ b/meta/classes/rootfs.bbclass > > > > @@ -298,13 +298,13 @@ do_rootfs_install[sstate-interceptfuncs] = > > > > "rootfs_install_sstate_prepare" # the buildchroot is owned by > > > > root, so we need some sudoing to pack and unpack > > > > rootfs_install_sstate_prepare() { sudo mount --bind > > > > ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > > > > - sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar > > > > --one-file-system rootfs > > > > + sudo tar -C ${WORKDIR}/mnt -cpzf > > > > ${ROOTFS_SSTATE}/rootfs.tar.gz --one-file-system rootfs sudo > > > > umount ${WORKDIR}/mnt/rootfs > > > > > > > > } > > > > do_rootfs_install_sstate_prepare[lockfiles] = > > > > > > > > "${REPO_ISAR_DIR}/isar.lock" > > > > > > > > rootfs_install_sstate_finalize() { > > > > > > > > - sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar > > > > + sudo tar -C ${WORKDIR} -xpzf ${ROOTFS_SSTATE}/rootfs.tar.gz > > > > > > > > } > > > > > > > > python do_rootfs_install_setscene() { > > > > > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > > > 2f483f5a..99d1d945 100644 --- > > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -402,11 > > > > +402,11 @@ do_bootstrap[sstate-plaindirs] = "${BOOTSTRAP_SSTATE}" > > > > do_bootstrap[sstate-interceptfuncs] = "bootstrap_sstate_prepare" > > > > > > > > bootstrap_sstate_prepare() { > > > > > > > > - sudo tar -C $(dirname "${ROOTFSDIR}") -cpf > > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar --one-file-system $(basename > > > > "${ROOTFSDIR}") > > > > + sudo tar -C $(dirname "${ROOTFSDIR}") -cpzf > > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar.gz --one-file-system $(basename > > > > "${ROOTFSDIR}") } > > > > > > > > bootstrap_sstate_finalize() { > > > > > > > > - sudo tar -C $(dirname "${ROOTFSDIR}") -xpf > > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar > > > > + sudo tar -C $(dirname "${ROOTFSDIR}") -xpzf > > > > ${BOOTSTRAP_SSTATE}/bootstrap.tar.gz sudo ln -Tfsr "${ROOTFSDIR}" > > > > "${DEPLOY_ISAR_BOOTSTRAP}" } -- Uladzimir Bely