From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a2e:a80c:: with SMTP id l12mr7085607ljq.441.1636026687014; Thu, 04 Nov 2021 04:51:27 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:10b2:: with SMTP id k18ls233393ljn.3.gmail; Thu, 04 Nov 2021 04:51:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwgunbcALLXZWa/iWG3Tvwk1ynPrbsrbD4vLS/zCYbpe7drt7ZOqXySNpiRuRtJ15maCf7R X-Received: by 2002:a05:651c:88d:: with SMTP id d13mr52125628ljq.399.1636026685925; Thu, 04 Nov 2021 04:51:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1636026685; cv=none; d=google.com; s=arc-20160816; b=LGC/JFDnyUi+iQ4HGGRUo88lYvmW77dbROAk1P8NmfMEXQaHROjQ/r7Oj1MxneCPnu muPYbDeU5VSgQDDshN4X44sJ1EJgpvQsxNNZjzsc9fHEjuOFIUPIxRe37D9Oxl/llayx OQeC2eLxqOYBqoUGqqv0glDOAF0t/X/c1oy/1XmS0tbC0yTpJEEZd/Ig8VMgZ24LImqO NZvW+o9pB0CVS2WMN+/D7Ee/lDWtyW+rAgIiHDo81z7k8vPH62Of7iBTBx67Y8Y4/qb8 wAZX8u4Dsadf8qtz6TeP49tAPFrPuA/3dgPMiCeBLwUlZkfC42F0OOatvEJuoFZPPt9D dYDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=jh53i2XsQGwzuWkG3Si2JPmR5t3Quz+I74Hn8Dfy5Sk=; b=aI+URB4O5vo7d8SMB0iRYjiMgIpchPo4zezMp0uBKWt7gHLTGmlL2ItjODOYZsJKUu yGXPETUfcsgwUPtCe5rmwy3yyIPr0VSIK0mA6bPJwqPGLKhtJ0m0WBLE+9/rQivmSuXl ovcSC11yDaGiziMqhOy+Mh5IlH/9z6vVibajRT2lPSh43jRDC8REogvRnzR1BM6uxPHC 7ZX3DPqHX2NsBs2xoznPy6MbntzKxJrGfhc5211v7aS5GE/8HqVoy5rLoIpeVBCfUlNp 0t2tAPp81DxS5ZpDo7omodvGKdctWDW1VV6OxEOn1LzzVRE1rmdSl4YZ7C63ADx4cHaI phgA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id t71si134340lff.6.2021.11.04.04.51.25 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 Nov 2021 04:51:25 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 1A4BpP12007199 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 4 Nov 2021 12:51:25 +0100 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 1A4BpPbE026036; Thu, 4 Nov 2021 12:51:25 +0100 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [PATCH v5 00/13] Add sstate-cache Date: Thu, 4 Nov 2021 12:51:10 +0100 Message-Id: <20211104115123.2519356-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: oviPIOexRzzV This series adds the sstate-cache feature from OE to Isar. The cache holds the results of bootstrapping, rootfs generation (buildchroot, image rootfs), and deb package generation. To use the cache, the only configuration neccessary is setting SSTATE_DIR. The contents of that directory need to be preserved across bitbake invocations. One known weakness is that the package lists of cached rootfs's can run out of sync with upstream ("apt-get update" only happens at bootstrap time). But this also happens with an "old" local build dir, and is something that may be addressed elsewhere. For now, the recommendation is to frequently clear the cache (in one of our projects we run a nightly "clear&populate cache" CI job). Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching, patch 6 copies files from OE, and patches 7..10 add caching to Isar. Patches 11..12 make minor adjustments to the tests, and patch 13 adds a test case for the new feature. --- Changes sinve v4: - rebase on next - add test case Changes since v3: - rebase on next - make sstate understand that images and initramfss are MACHINE specific (fixes a false-sharing bug that occured while testing with multiconfig) Changes since v2: - rebase on next - add the explicit isar-apt dependency to all images (not only wic as previously). - don't move existing code in isar-bootstrap.inc - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that packages depend recursively on their dependencies, even when build results are taken from cache. - generally improve commit messages Changes since v1: - fix copy/paste typo in rootfs.bbclass - add mounting trick to tar rootfs (because --one-file-system does not stop at bind mounts) - have install_imager_deps also depend explicitly on isar-apt *** BLURB HERE *** Adriaan Schmidt (13): oe imports in central location images: create deploy dir rootfs: recursively depend on packages base: remove unneeded "before do_build" task dependencies dpkg: add explicit dependency to isar-apt meta: add sstate feature from oe sstate: configure sstate: add caching to isar-bootstrap sstate: add caching to rootfs sstate: add caching to debian packages test: pass absolute path for build_dir test: make bitbake_args a list sstate: add test case meta/classes/base.bbclass | 32 +- meta/classes/container-img.bbclass | 1 + meta/classes/cpiogz-img.bbclass | 1 + meta/classes/dpkg-base.bbclass | 34 +- meta/classes/dpkg.bbclass | 1 + meta/classes/ext4-img.bbclass | 1 + meta/classes/fit-img.bbclass | 1 + meta/classes/image-tools-extension.bbclass | 2 +- meta/classes/image.bbclass | 3 + meta/classes/initramfs.bbclass | 3 + meta/classes/patch.bbclass | 5 - meta/classes/rootfs.bbclass | 29 +- meta/classes/sstate.bbclass | 1311 +++++++++++++++++ meta/classes/targz-img.bbclass | 1 + meta/classes/ubi-img.bbclass | 1 + meta/classes/ubifs-img.bbclass | 1 + meta/classes/wic-img.bbclass | 1 + meta/conf/bitbake.conf | 10 +- meta/lib/oe/gpg_sign.py | 130 ++ meta/lib/oe/sstatesig.py | 603 ++++++++ .../isar-bootstrap/isar-bootstrap.inc | 24 + scripts/ci_build.sh | 2 +- testsuite/build_test/build_test.py | 101 +- testsuite/build_test/cibase.py | 20 +- testsuite/build_test/cibuilder.py | 2 +- 25 files changed, 2284 insertions(+), 36 deletions(-) create mode 100644 meta/classes/sstate.bbclass create mode 100644 meta/lib/oe/gpg_sign.py create mode 100644 meta/lib/oe/sstatesig.py -- 2.30.2