From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a2e:a5c8:: with SMTP id n8mr5085676ljp.367.1636613259288; Wed, 10 Nov 2021 22:47:39 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:1314:: with SMTP id x20ls1468194lfu.1.gmail; Wed, 10 Nov 2021 22:47:38 -0800 (PST) X-Google-Smtp-Source: ABdhPJzVf6ft85hdWNUP1gWz4DKK5fYi9IKZPet4f2D145LfNWbCqTh5+PzxXfLIPCnC+PSIJYmy X-Received: by 2002:a05:6512:1102:: with SMTP id l2mr4725329lfg.469.1636613258130; Wed, 10 Nov 2021 22:47:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1636613258; cv=none; d=google.com; s=arc-20160816; b=jgMMdcDQIeoArtzvc8yxxQMophzJWGraLzyl9iBCWYy0gNvia6xi0hWTN5c2DjBXXT EaWqzqLmOMX/h2lrGNTV9xT1KabWlJtZgTxJSyirbmSo0k3tUNUnU9MrV8mkObq4OXFV 6hk0mrYEmU6JMCq60A3d5P2cGI7E4CNuOJALH75SYjjloC2SUz/oXn/8fK87Bt3ZFndy E20O/92/RqDjvvW6/vU6qo+n8B5uEk7T4MLzXyAWofAQWi5JPgmUkg24TfmvlgwiD2+x cq08xf6IKrMmXOJtKdpbqdydftaUckh51J2RwWNgPZEK+pDNl4iw/KkQYY/yM45iLUnF upmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=zbN/XQYHIZfpYiSPdJP9yPY3Q6KlW8s5IL/VHQkeC7I=; b=NJrkvOFRbwLDe3EjK03VBrO4SmRS/9ihUOh6ohmcGa7prfKtnK2FJdMFRr1agsY6wF paroM5yn94I8L/uPYVa8quPUlFSbtFyKZS4Rgx8c3TjImehFzAMFRavcncGrRrhSzFL8 kVdlwX1P2Is0RKf++bxoG9J0t7Qx+Cy63YMlEh51oSPgAehG448En0f9BW1KCYoZn69y CWJ05GvI2JN8vFJ37D21A/PUg/ZRyidOJBYArVweBxUJJHCiQq5pZZdaxWRAhLJ7XO7w dAaGESf4smp/Zj1LhcucikX8MWw3jKKyOu5cD5gU4sXLlkmlizBeLXxFHS6cnbdc4s5V VJpg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id t12si208691ljh.0.2021.11.10.22.47.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Nov 2021 22:47:37 -0800 (PST) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 1AB6lbBp017641 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 11 Nov 2021 07:47:37 +0100 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 1AB6lbcT012388; Thu, 11 Nov 2021 07:47:37 +0100 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [PATCH v6 00/13] Add sstate-cache Date: Thu, 11 Nov 2021 07:47:15 +0100 Message-Id: <20211111064728.2375760-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: apLQlTcIJksD This series adds the sstate-cache feature from OE to Isar. The cache holds the results of bootstrapping, rootfs generation (buildchroot, image rootfs), and deb package generation. To use the cache, the only configuration neccessary is setting SSTATE_DIR. The contents of that directory need to be preserved across bitbake invocations. One known weakness is that the package lists of cached rootfs's can run out of sync with upstream ("apt-get update" only happens at bootstrap time). But this also happens with an "old" local build dir, and is something that may be addressed elsewhere. For now, the recommendation is to frequently clear the cache (in one of our projects we run a nightly "clear&populate cache" CI job). Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching, patch 6 copies files from OE, and patches 7..10 add caching to Isar. Patches 11..12 make minor adjustments to the tests, and patch 13 adds a test case for the new feature. --- Changes since v5: - added lib/oe/utils.py because sstate.bbclass uses oe.utils.ThreadedPool when accessing a remote sstate cache (SSTATE_MIRRORS) Changes since v4: - rebase on next - add test case Changes since v3: - rebase on next - make sstate understand that images and initramfss are MACHINE specific (fixes a false-sharing bug that occured while testing with multiconfig) Changes since v2: - rebase on next - add the explicit isar-apt dependency to all images (not only wic as previously). - don't move existing code in isar-bootstrap.inc - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that packages depend recursively on their dependencies, even when build results are taken from cache. - generally improve commit messages Changes since v1: - fix copy/paste typo in rootfs.bbclass - add mounting trick to tar rootfs (because --one-file-system does not stop at bind mounts) - have install_imager_deps also depend explicitly on isar-apt Adriaan Schmidt (13): oe imports in central location images: create deploy dir rootfs: recursively depend on packages base: remove unneeded "before do_build" task dependencies dpkg: add explicit dependency to isar-apt meta: add sstate feature from oe sstate: configure sstate: add caching to isar-bootstrap sstate: add caching to rootfs sstate: add caching to debian packages test: pass absolute path for build_dir test: make bitbake_args a list sstate: add test case meta/classes/base.bbclass | 32 +- meta/classes/container-img.bbclass | 1 + meta/classes/cpiogz-img.bbclass | 1 + meta/classes/dpkg-base.bbclass | 34 +- meta/classes/dpkg.bbclass | 1 + meta/classes/ext4-img.bbclass | 1 + meta/classes/fit-img.bbclass | 1 + meta/classes/image-tools-extension.bbclass | 2 +- meta/classes/image.bbclass | 3 + meta/classes/initramfs.bbclass | 3 + meta/classes/patch.bbclass | 5 - meta/classes/rootfs.bbclass | 29 +- meta/classes/sstate.bbclass | 1311 +++++++++++++++++ meta/classes/targz-img.bbclass | 1 + meta/classes/ubi-img.bbclass | 1 + meta/classes/ubifs-img.bbclass | 1 + meta/classes/wic-img.bbclass | 1 + meta/conf/bitbake.conf | 10 +- meta/lib/oe/gpg_sign.py | 130 ++ meta/lib/oe/sstatesig.py | 603 ++++++++ meta/lib/oe/utils.py | 569 +++++++ .../isar-bootstrap/isar-bootstrap.inc | 24 + scripts/ci_build.sh | 2 +- testsuite/build_test/build_test.py | 101 +- testsuite/build_test/cibase.py | 20 +- testsuite/build_test/cibuilder.py | 2 +- 26 files changed, 2853 insertions(+), 36 deletions(-) create mode 100644 meta/classes/sstate.bbclass create mode 100644 meta/lib/oe/gpg_sign.py create mode 100644 meta/lib/oe/sstatesig.py create mode 100644 meta/lib/oe/utils.py -- 2.30.2