From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7068247660288802816 X-Received: by 2002:aa7:df1a:0:b0:409:5174:68a9 with SMTP id c26-20020aa7df1a000000b00409517468a9mr6469537edy.145.1645782697603; Fri, 25 Feb 2022 01:51:37 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:c45a:0:b0:410:98bf:fa0c with SMTP id n26-20020aa7c45a000000b0041098bffa0cls1727960edr.2.gmail; Fri, 25 Feb 2022 01:51:36 -0800 (PST) X-Google-Smtp-Source: ABdhPJwAsYHouvZ9ivLPO9SsJq7CfTlQcNzUdoPxVxi+n7mbVPalB+kjwQiT8Ut59Aq/GtHy5BcG X-Received: by 2002:a05:6402:51d2:b0:412:e48e:51 with SMTP id r18-20020a05640251d200b00412e48e0051mr6323317edd.220.1645782696633; Fri, 25 Feb 2022 01:51:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645782696; cv=none; d=google.com; s=arc-20160816; b=l1gHrPQm5IgXYDAyht+vxVhc5GCX2AQTg//QZ71ztm3QfrbPlBrSUBqOXPhlTn0mQ+ wCkSn3tUn0MHlIo91lJonIeFmIq43l6I+o+5JzQsZVsjcN6elN5r2HZOJY1LCx2aZq6q qs/hyUs9Gb9h5abVJ0AhPqb636szHrWeBrfvAJK9L94kJ7QOlvi8ftplwyWBDzksPSuR QfdouvaakfDmR8j1tPDnrqUpuw6PbPGXug5s80Re3IYhRAwQJQPc+RcYBcAOkTeqGu6h SQbRb1aY4r3gL6ob9mYN046+zNq8prLMVyxqH4eLAHdDX9uVT6cuOCIA74DbdDhnt/58 HGDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=6Y2JHZW+asfA75qff38M2XDeqwnUIyEmIxAUjnNmRhU=; b=EUB1BUs2JkkfraL3r6lvEDcvLfl9x4tfJhuWQZlii4Uf0oGjKnp7Jsx/kzy22shpSL PM86CLcMvkScHdj81l2gObcBsWLOSi54Sr6O7qqgJdd5VdQc9AITAq7DsFMxvqjz7tWD qIX2a45d9o0h5VzRqmTef397Hj8L4gOS9lj2jjQEe4lMpETDozbFtPINU+w8NRD0D4Eq V4qbLKgCDbj+haU0li7h3GdiU0vhsNeIoh/cz0LFutiYIcxtcQRU7e01xWubsCxjo5SO Wy3EEuRV31u5YLoQqmHjwPIEuzQWYvSuOpfEk34NQ74bvYLsxOSm1GALm9RY62OcSgNa 7TZA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id et1-20020a170907294100b006ce69d31a32si133718ejc.2.2022.02.25.01.51.36 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 25 Feb 2022 01:51:36 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from home.localnet (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 21P9pZII002035 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 25 Feb 2022 10:51:35 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: Re: [PATCH] sstate: refactor to avoid intermediate files Date: Fri, 25 Feb 2022 12:51:30 +0300 Message-ID: <4665451.OV4Wx5bFTl@home> In-Reply-To: <20220224121025.1153508-1-adriaan.schmidt@siemens.com> References: <20220224121025.1153508-1-adriaan.schmidt@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 0y4iDxRhmeJI In the email from Thursday, 24 February 2022 15:10:25 +03 user Adriaan Schmidt wrote: > By using the SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS > hooks (instead of sstate-interceptfuncs) we can avoid > the explicit creation of intermediate/temporary files > during sstate caching/restoring, which saves disk space > during build. > > Signed-off-by: Adriaan Schmidt > --- > meta/classes/dpkg-base.bbclass | 16 ++++++------ > meta/classes/rootfs.bbclass | 25 ++++++++++++------- > .../isar-bootstrap/isar-bootstrap.inc | 22 +++++++++------- > 3 files changed, 36 insertions(+), 27 deletions(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index 928856a9..86933c57 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -237,27 +237,25 @@ python do_dpkg_build() { > addtask dpkg_build > > SSTATETASKS += "do_dpkg_build" > -DPKG_SSTATE = "${WORKDIR}/dpkg-sstate" > -do_dpkg_build[dirs] += "${DPKG_SSTATE} ${S}/.." > -do_dpkg_build[cleandirs] += "${DPKG_SSTATE}" > -do_dpkg_build[sstate-plaindirs] = "${DPKG_SSTATE}" > -do_dpkg_build[sstate-interceptfuncs] = "dpkg_build_sstate_prepare" > +SSTATECREATEFUNCS += "dpkg_build_sstate_prepare" > +SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize" > I tested the patch and it looks and works well. I also tried to find a way to reuse `sstate-install-` directories which are deleted automatically and look much more "native" for sstate that our custom `rootfs-state` or `bootstrap-sstate` or `dpkg-sstate`, but didn't succeed, being not so much familiar with sstate. Reusing "standard" directories really reduces amount of code and makes things simple. > dpkg_build_sstate_prepare() { > + # this runs in SSTATE_BUILDDIR, which will be deleted automatically > if [ -n "$(find ${S}/.. -maxdepth 1 -name '*.deb' -print -quit)" ]; > then - ln -f ${S}/../*.deb -t ${DPKG_SSTATE} > + cp -f ${S}/../*.deb -t . > fi > } > > dpkg_build_sstate_finalize() { > - if [ -n "$(find ${DPKG_SSTATE} -maxdepth 1 -name '*.deb' -print -quit)" > ]; then - ln -f ${DPKG_SSTATE}/*.deb -t ${S}/.. > + # this runs in SSTATE_INSTDIR > + if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then > + mv -f ./*.deb -t ${S}/.. > fi > } > > python do_dpkg_build_setscene() { > sstate_setscene(d) > - bb.build.exec_func('dpkg_build_sstate_finalize', d) > } > > addtask dpkg_build_setscene > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index 2bdb3b6d..ba86c60a 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -289,26 +289,33 @@ addtask rootfs before do_build > do_rootfs_postprocess[depends] = "base-apt:do_cache > isar-apt:do_cache_config" > > SSTATETASKS += "do_rootfs_install" > -ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate" > -do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" > -do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}" > -do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}" > -do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare" > +SSTATECREATEFUNCS += "rootfs_install_sstate_prepare" > +SSTATEPOSTINSTFUNCS += "rootfs_install_sstate_finalize" > > -# the buildchroot is owned by root, so we need some sudoing to pack and > unpack +# the rootfs is owned by root, so we need some sudoing to pack and > unpack rootfs_install_sstate_prepare() { > + # this runs in SSTATE_BUILDDIR, which will be deleted automatically > + # tar --one-file-system will cross bind-mounts to the same filesystem, > + # so we use some mount magic to prevent that > + mkdir -p ${WORKDIR}/mnt/rootfs > sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro > - sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar > --one-file-system rootfs + sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar > --one-file-system rootfs sudo umount ${WORKDIR}/mnt/rootfs > + sudo chown $(id -u):$(id -g) rootfs.tar > } > do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" > > rootfs_install_sstate_finalize() { > - sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar > + # this runs in SSTATE_INSTDIR > + # - after building the rootfs, the tar won't be there, but we also > don't need to unpack + # - after restoring from cache, there will be a > tar which we unpack and then delete + if [ -f rootfs.tar ]; then > + sudo tar -C ${WORKDIR} -xpf rootfs.tar > + rm rootfs.tar > + fi > } > > python do_rootfs_install_setscene() { > sstate_setscene(d) > - bb.build.exec_func('rootfs_install_sstate_finalize', d) > } > addtask do_rootfs_install_setscene > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > 2f483f5a..d6f90f63 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -395,24 +395,28 @@ EOSUDO > addtask bootstrap before do_build after do_generate_keyrings > > SSTATETASKS += "do_bootstrap" > -BOOTSTRAP_SSTATE = "${WORKDIR}/bootstrap-sstate" > -do_bootstrap[dirs] += "${BOOTSTRAP_SSTATE}" > -do_bootstrap[cleandirs] += "${BOOTSTRAP_SSTATE}" > -do_bootstrap[sstate-plaindirs] = "${BOOTSTRAP_SSTATE}" > -do_bootstrap[sstate-interceptfuncs] = "bootstrap_sstate_prepare" > +SSTATECREATEFUNCS += "bootstrap_sstate_prepare" > +SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" > > bootstrap_sstate_prepare() { > - sudo tar -C $(dirname "${ROOTFSDIR}") -cpf > ${BOOTSTRAP_SSTATE}/bootstrap.tar --one-file-system $(basename > "${ROOTFSDIR}") + # this runs in SSTATE_BUILDDIR, which will be deleted > automatically + sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar > --one-file-system $(basename "${ROOTFSDIR}") + sudo chown $(id -u):$(id > -g) bootstrap.tar > } > > bootstrap_sstate_finalize() { > - sudo tar -C $(dirname "${ROOTFSDIR}") -xpf > ${BOOTSTRAP_SSTATE}/bootstrap.tar - sudo ln -Tfsr "${ROOTFSDIR}" > "${DEPLOY_ISAR_BOOTSTRAP}" > + # this runs in SSTATE_INSTDIR > + # - after building the bootstrap, the tar won't be there, but we also > don't need to unpack + # - after restoring from cache, there will be a > tar which we unpack and then delete + if [ -f bootstrap.tar ]; then > + sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar > + sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" > + rm bootstrap.tar > + fi > } > > python do_bootstrap_setscene() { > sstate_setscene(d) > - bb.build.exec_func('bootstrap_sstate_finalize', d) > } > > addtask do_bootstrap_setscene -- Uladzimir Bely