From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7068247660288802816 X-Received: by 2002:a17:902:cecf:b0:14c:da4a:6aec with SMTP id d15-20020a170902cecf00b0014cda4a6aecmr2181287plg.140.1645704652183; Thu, 24 Feb 2022 04:10:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a65:56c3:0:b0:372:dc1e:f246 with SMTP id w3-20020a6556c3000000b00372dc1ef246ls1103033pgs.0.gmail; Thu, 24 Feb 2022 04:10:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJzls7fAtep+y5Nm8H+KV3FGm71ibN+IDmeKFz5PUVg6UuOb2R9Nogi34IiZJVv0/1Xr46hn X-Received: by 2002:aa7:85d7:0:b0:4de:91e0:3302 with SMTP id z23-20020aa785d7000000b004de91e03302mr2618069pfn.22.1645704651455; Thu, 24 Feb 2022 04:10:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645704651; cv=pass; d=google.com; s=arc-20160816; b=g3DO1ZWVVMjixHrzdNcbS/41BifKXjszK2AOVkFhQbU+7ChxOBvP9vKV+6BHpMdyaw Ue2Z52e2470hXS6UOJeYO9itP2VsBQWoAUVhgyYu4ZMhotHp5pUitdiZ4qdGHe79b0lr 3uUO8RcAW6Pcy7dBpD92vKh8jTvJx9SBnIosCg1wLJEMCQ8Rt9Huho8Q0CGphzB1U+GN Jd+DeWsNxg9fWL7KfiqvdZomYTb1XZ9hwgA+sJ9E+69YXLaiMpDLl9Jn2Sr3fzFzpuo5 mt8qtNb7K1LVe8OtuCRMfhNb433WP0yvOhqT/mbOovuilTwzE4sXniZphSSxEdNpjHtJ ZgwQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=lMuGQuLexBjOXqnjnoDsr+1VqcbGvOLU8kkF6DeYjFg=; b=pPNNAurQ15vs65jbC5y8RwIj2kxU/z2NHgzEiz29WgfMSrrWewSLmqweLVgCotpsx7 wlcZhui0z75jCDz8Ca02AkRp+kR78gmdZWo86XUdLFqVWNmWDALt/e0gxf2LHGmqg+NO a1OdWY5Z2/B6WEzB/J+luszJDUT2tYrT6C1H374FYWV1waoA709ZOXbXuT2SE74g2sz+ DzmMo+HkG3GRmr/UcyzwbMItkkkUgYiX3LOwFtjFOUMELuj3sqIR1kv/UQi5vF35XCQD i37if3rQdm3EfQyl9vXDpALP7yPsnlRSyq/xVW6L05xQzmn6R3B9VsPQgUWmnO7mjxWw eGzw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BDw3Uc9j; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::619 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-am5eur03on0619.outbound.protection.outlook.com. [2a01:111:f400:fe08::619]) by gmr-mx.google.com with ESMTPS id a10si128353pgw.5.2022.02.24.04.10.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Feb 2022 04:10:51 -0800 (PST) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::619 as permitted sender) client-ip=2a01:111:f400:fe08::619; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=BDw3Uc9j; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 2a01:111:f400:fe08::619 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RcS6Gq/TmIlE9Bvg5WpZr6R+fJjWZyx+smOZgGoqdBhDzPQHKFpgLT1sKQGi3HHw+/XpRSQJ+W/tIjJoYZ0ghFy+xiK6U+YHfHhy85MYbaiou/oEZr1ch5Vd2SjHAzscNPw3+L2SYIejeFoF//+OSLjWliTT3/oFXxhG+OpyrDVmiWA2uPTJCK5Tt5u47X9SiLoN5/z2HNNSABTmlfxB/p/4A0vuEkZ3860hOTsIM54BuMJyyMTJlXxrzsqMAA2gRAPJrhhZKHuCJhK9+aoaAgfGaJzPZnlAxC9h4Z8s2REfp6IBJe4C9kJ/MVAMgd+2UEHm4NWOUUmHmOroJuIcSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lMuGQuLexBjOXqnjnoDsr+1VqcbGvOLU8kkF6DeYjFg=; b=ewujgtLiTEGivlP7Npmxn+/2OjDGLqiL73Uxjtz8RH3FU1SUTQ4ZFcGOEB6h2VqCadij6FIpoXw+fQdQeRFNfSN88pi+oM0npxHCwtwA4b+0B+zg4onyTW1BiSE1hXXSc2BYdPQTx7UXpPdVuLhXdtqtxi3vL2lEDtLcNNF2X/KU2W7ENRnbjWzOfLTkF+T+8ptDi2+bEHugKUOCyXbiQ17cE+picHnN3vi1W1pAqXaDmqBUlh5Li5B/Q0jnvbPmwtQETiIPlqUvBY0qtvGkHalC63k7CcasMtfCn7q2vgvcs+lICNW+9UNtNkLqSWuPFvD3VsoRCADY1s1kTXdlFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.70) smtp.rcpttodomain=googlegroups.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lMuGQuLexBjOXqnjnoDsr+1VqcbGvOLU8kkF6DeYjFg=; b=BDw3Uc9jTKyxUthsdw33Cn5ByrdhFLbK8ycuSTJSD8PiEKIydkX2V8UDl6Ebob7xuDNQseleWbrBGYo+c7O+ldLUmGSNQiAYorHsihbMzquNnnhXpWjDMkTH4ldLUfXYu1D78K6fkqLvxBreJPaQ/iO/lDHu8+LRHvAybPhdg/mpYUnkP/7VvSZHDXZJE1yOpLf/b4URPnOzHroce8MjTqZvotlGyyR/YguAqlje9HCG0oboHXU4AW/9yDb30l+y0c1Q/CF1j2ggPLaotRYkukXfLwQK9s6cFgL7BT5A7gqRT5lz+/qtAgNEEy+CvBil7hxFUJn7LFi4OUpvmD2VlQ== Received: from DU2PR04CA0047.eurprd04.prod.outlook.com (2603:10a6:10:234::22) by VE1PR10MB3007.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:109::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.24; Thu, 24 Feb 2022 12:10:48 +0000 Received: from DB5EUR01FT010.eop-EUR01.prod.protection.outlook.com (2603:10a6:10:234:cafe::d0) by DU2PR04CA0047.outlook.office365.com (2603:10a6:10:234::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.24 via Frontend Transport; Thu, 24 Feb 2022 12:10:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.70) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.70 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.70; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.70) by DB5EUR01FT010.mail.protection.outlook.com (10.152.4.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 12:10:48 +0000 Received: from DEMCHDC8A1A.ad011.siemens.net (139.25.226.107) by DEMCHDC9SJA.ad011.siemens.net (194.138.21.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Thu, 24 Feb 2022 13:10:47 +0100 Received: from random.ppmd.siemens.net (139.25.68.25) by DEMCHDC8A1A.ad011.siemens.net (139.25.226.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Thu, 24 Feb 2022 13:10:47 +0100 From: Adriaan Schmidt To: CC: Adriaan Schmidt Subject: [PATCH] sstate: refactor to avoid intermediate files Date: Thu, 24 Feb 2022 13:10:25 +0100 Message-ID: <20220224121025.1153508-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Return-Path: adriaan.schmidt@siemens.com X-Originating-IP: [139.25.68.25] X-ClientProxiedBy: DEMCHDC89YA.ad011.siemens.net (139.25.226.104) To DEMCHDC8A1A.ad011.siemens.net (139.25.226.107) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 969359de-4ccd-4d62-41b9-08d9f78eb14a X-MS-TrafficTypeDiagnostic: VE1PR10MB3007:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vKIshj57My11iIzJBL/qAVWxt3obMucq5axGeKzbmzQ8Y2sqrSp62naGXsbIcdlYMXQS7wpLdVZCXL5eAxMud/MahHUloFP9ysVmOYsI40SinwodtmSxxnJbaleEZeMXrO2Yp+84XaSAZ/O/ifZVztYnGBwlIwkAX5JsMQKY64wmSUIvfgrOGX1U7fyhO+45O02PnZcauqUK80Jelj2wMX/Fd7KOwi/msjfSCzyvspDZ71qpy4d7MfMhIXyFO99UFt6BZ7EOr/pWiHqPZVT8GtV6jQKyNEJWhN6yr4bJaFFR/SA8z0lZbIP/swqj+F71pE+2ajQH33yQaZK0xbSHL9Jzjje7TxiXnEJa5Ag8iLZhpom+8mI6iNb7LY+OtLfkYQrEo/79mQlPAYWzxgYYobELcu1hSd/ax2TksJSbPWqpRwvDzV3nm9ND4gsxVjYKs2/qeR9GOEJ3FOQhqdSkZUGZ34+kwwygijgNJulzSzMO7gJphO9Ap54WoUgkVW3UP6v1M6o2ugthrOSNXnWlgXVNGxIEh7Lnj3sl7Pui4zUOur8YC0ZhbfxnkRcKeaFabJq3I0RBKmCN9sYpUad77xUZnK+hWExJSvPc51RSnUxNn1PF/xOu1nfseLQKaF4ug43JVhP0RR+IdqCt2/fl1gyF4qws166kRUoUmDopoFj3Shg7ZpuX5Yy5v/fFJf/x+ruKebsBrzY4ubewCxG0yQ== X-Forefront-Antispam-Report: CIP:194.138.21.70;CTRY:DE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:hybrid.siemens.com;PTR:hybrid.siemens.com;CAT:NONE;SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(82310400004)(186003)(508600001)(107886003)(47076005)(16526019)(86362001)(36860700001)(26005)(956004)(2616005)(1076003)(336012)(83380400001)(2906002)(6666004)(5660300002)(7596003)(40460700003)(82960400001)(7636003)(4326008)(8676002)(70206006)(316002)(70586007)(44832011)(8936002)(356005)(36756003)(6916009)(360504008);DIR:OUT;SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 12:10:48.0861 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 969359de-4ccd-4d62-41b9-08d9f78eb14a X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a;Ip=[194.138.21.70];Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR01FT010.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR10MB3007 X-TUID: s8s6QFA/c/ow By using the SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS hooks (instead of sstate-interceptfuncs) we can avoid the explicit creation of intermediate/temporary files during sstate caching/restoring, which saves disk space during build. Signed-off-by: Adriaan Schmidt --- meta/classes/dpkg-base.bbclass | 16 ++++++------ meta/classes/rootfs.bbclass | 25 ++++++++++++------- .../isar-bootstrap/isar-bootstrap.inc | 22 +++++++++------- 3 files changed, 36 insertions(+), 27 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 928856a9..86933c57 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -237,27 +237,25 @@ python do_dpkg_build() { addtask dpkg_build SSTATETASKS += "do_dpkg_build" -DPKG_SSTATE = "${WORKDIR}/dpkg-sstate" -do_dpkg_build[dirs] += "${DPKG_SSTATE} ${S}/.." -do_dpkg_build[cleandirs] += "${DPKG_SSTATE}" -do_dpkg_build[sstate-plaindirs] = "${DPKG_SSTATE}" -do_dpkg_build[sstate-interceptfuncs] = "dpkg_build_sstate_prepare" +SSTATECREATEFUNCS += "dpkg_build_sstate_prepare" +SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize" dpkg_build_sstate_prepare() { + # this runs in SSTATE_BUILDDIR, which will be deleted automatically if [ -n "$(find ${S}/.. -maxdepth 1 -name '*.deb' -print -quit)" ]; then - ln -f ${S}/../*.deb -t ${DPKG_SSTATE} + cp -f ${S}/../*.deb -t . fi } dpkg_build_sstate_finalize() { - if [ -n "$(find ${DPKG_SSTATE} -maxdepth 1 -name '*.deb' -print -quit)" ]; then - ln -f ${DPKG_SSTATE}/*.deb -t ${S}/.. + # this runs in SSTATE_INSTDIR + if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then + mv -f ./*.deb -t ${S}/.. fi } python do_dpkg_build_setscene() { sstate_setscene(d) - bb.build.exec_func('dpkg_build_sstate_finalize', d) } addtask dpkg_build_setscene diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 2bdb3b6d..ba86c60a 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -289,26 +289,33 @@ addtask rootfs before do_build do_rootfs_postprocess[depends] = "base-apt:do_cache isar-apt:do_cache_config" SSTATETASKS += "do_rootfs_install" -ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate" -do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs" -do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}" -do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}" -do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare" +SSTATECREATEFUNCS += "rootfs_install_sstate_prepare" +SSTATEPOSTINSTFUNCS += "rootfs_install_sstate_finalize" -# the buildchroot is owned by root, so we need some sudoing to pack and unpack +# the rootfs is owned by root, so we need some sudoing to pack and unpack rootfs_install_sstate_prepare() { + # this runs in SSTATE_BUILDDIR, which will be deleted automatically + # tar --one-file-system will cross bind-mounts to the same filesystem, + # so we use some mount magic to prevent that + mkdir -p ${WORKDIR}/mnt/rootfs sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro - sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file-system rootfs + sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar --one-file-system rootfs sudo umount ${WORKDIR}/mnt/rootfs + sudo chown $(id -u):$(id -g) rootfs.tar } do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock" rootfs_install_sstate_finalize() { - sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar + # this runs in SSTATE_INSTDIR + # - after building the rootfs, the tar won't be there, but we also don't need to unpack + # - after restoring from cache, there will be a tar which we unpack and then delete + if [ -f rootfs.tar ]; then + sudo tar -C ${WORKDIR} -xpf rootfs.tar + rm rootfs.tar + fi } python do_rootfs_install_setscene() { sstate_setscene(d) - bb.build.exec_func('rootfs_install_sstate_finalize', d) } addtask do_rootfs_install_setscene diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 2f483f5a..d6f90f63 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,24 +395,28 @@ EOSUDO addtask bootstrap before do_build after do_generate_keyrings SSTATETASKS += "do_bootstrap" -BOOTSTRAP_SSTATE = "${WORKDIR}/bootstrap-sstate" -do_bootstrap[dirs] += "${BOOTSTRAP_SSTATE}" -do_bootstrap[cleandirs] += "${BOOTSTRAP_SSTATE}" -do_bootstrap[sstate-plaindirs] = "${BOOTSTRAP_SSTATE}" -do_bootstrap[sstate-interceptfuncs] = "bootstrap_sstate_prepare" +SSTATECREATEFUNCS += "bootstrap_sstate_prepare" +SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" bootstrap_sstate_prepare() { - sudo tar -C $(dirname "${ROOTFSDIR}") -cpf ${BOOTSTRAP_SSTATE}/bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}") + # this runs in SSTATE_BUILDDIR, which will be deleted automatically + sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}") + sudo chown $(id -u):$(id -g) bootstrap.tar } bootstrap_sstate_finalize() { - sudo tar -C $(dirname "${ROOTFSDIR}") -xpf ${BOOTSTRAP_SSTATE}/bootstrap.tar - sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" + # this runs in SSTATE_INSTDIR + # - after building the bootstrap, the tar won't be there, but we also don't need to unpack + # - after restoring from cache, there will be a tar which we unpack and then delete + if [ -f bootstrap.tar ]; then + sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar + sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" + rm bootstrap.tar + fi } python do_bootstrap_setscene() { sstate_setscene(d) - bb.build.exec_func('bootstrap_sstate_finalize', d) } addtask do_bootstrap_setscene -- 2.30.2