From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a05:6512:907:: with SMTP id e7mr4950163lft.671.1634632164357; Tue, 19 Oct 2021 01:29:24 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3f14:: with SMTP id y20ls1773467lfa.1.gmail; Tue, 19 Oct 2021 01:29:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwkxyT+mUzCJtOy1bt9zDVCWtk16kmOAINVq2dJv5ATLPM+8DyJZrBIDPwHRxsOxMEnJWR1 X-Received: by 2002:a05:6512:203c:: with SMTP id s28mr4590068lfs.531.1634632163342; Tue, 19 Oct 2021 01:29:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634632163; cv=none; d=google.com; s=arc-20160816; b=zLlyn4J35ZZlD9qthQIyJpG8t9tnaEsSIAnudAAe8JFYBegQ3msF8ruTA5K919Ybix 8SUWYDSQRPNSw0BjWQQzgmmfuDkd5U1y4v7Axy6uQIHOB8reU8NN4XQI6FmK/57rRaAS yrZUjv1FyRcbFavT5094974G2dK6aSyoZAHGgLxZ3oqb9IsZ3i9fqrKIM9VIjWVkYqor ZcbwExcUoFBrF+QFWL2dkYhKfTuW3W40VcZ7AXdTbCNlzf98j+pTR2fsAnCmXSK+Wmoy nbgUfFZi4zI2xmb2db8xN/i4GMixmiysC3f4BmT8S3GvCo0vm2ViveU6A7UI5mXLpPcI zR/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=LfRZoUDs0b6WVbi4S4cYD57EuTo7Vfur+/mIgucX8JU=; b=HjqvY5Zvd3BsAWbI4f3/jqR0eaHLD3xlVRpPc1R831MJx2TgtOwtmonMzuyRyyO6Pg QWhpacy4sC0DZvbN1EJKrqzCeXZYiv8yTfqdHX40RvfJKsA266GiieSsHR81h+2pf3B3 UYaRLn8mrU+/zD2FL0XFYY3xYT7LzpdxDQwEgCE5uS+mb8ixTWWg7rG09F9NWGFsstbt uze5ZrwJp1bE/Td492vo8hEYIoM+0ZGtNiaYowAJmWASk6JYZiDA69sjlLpdajewFPBy NXEZDLJ0G6Z6v/VngOiFtyDdbt6qFiFydhwospbr69sO+0eguI+JeLVmIo2cl45yU71g azYg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id z1si1043484lfu.5.2021.10.19.01.29.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 01:29:23 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 19J8TMdD003453 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 19 Oct 2021 10:29:22 +0200 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 19J8TMY7002966; Tue, 19 Oct 2021 10:29:22 +0200 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [PATCH v3 00/10] Add sstate-cache Date: Tue, 19 Oct 2021 10:29:03 +0200 Message-Id: <20211019082913.199818-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: DNlV8UZ7hFHy This series adds the sstate-cache feature from OE to Isar. The cache holds the results of bootstrapping, rootfs generation (buildchroot, image rootfs), and deb package generation. To use the cache, the only configuration neccessary is setting SSTATE_DIR. The contents of that directory need to be preserved across bitbake invocations. One known weakness is that the package lists of cached rootfs's can run out of sync with upstream ("apt-get update" only happens at bootstrap time). But this also happens with an "old" local build dir, and is something that may be addressed elsewhere. For now, the recommendation is to frequently clear the cache (in one of our projects we run a nightly "clear&populate cache" CI job). Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching, patch 6 copies files from OE, and patches 7..10 add caching to Isar. --- Changes since v2: - rebase on next - add the explicit isar-apt dependency to all images (not only wic as previously). - don't move existing code in isar-bootstrap.inc - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that packages depend recursively on their dependencies, even when build results are taken from cache. - generally improve commit messages Changes since v1: - fix copy/paste typo in rootfs.bbclass - add mounting trick to tar rootfs (because --one-file-system does not stop at bind mounts) - have install_imager_deps also depend explicitly on isar-apt Adriaan Schmidt (10): oe imports in central location images: create deploy dir rootfs: recursively depend on packages base: remove unneeded "before do_build" task dependencies dpkg: add explicit dependency to isar-apt meta: add sstate feature from oe sstate: configure sstate: add caching to isar-bootstrap sstate: add caching to rootfs sstate: add caching to debian packages meta/classes/base.bbclass | 32 +- meta/classes/container-img.bbclass | 1 + meta/classes/cpiogz-img.bbclass | 1 + meta/classes/dpkg-base.bbclass | 34 +- meta/classes/dpkg.bbclass | 1 + meta/classes/ext4-img.bbclass | 1 + meta/classes/fit-img.bbclass | 1 + meta/classes/image-tools-extension.bbclass | 2 +- meta/classes/patch.bbclass | 5 - meta/classes/rootfs.bbclass | 29 +- meta/classes/sstate.bbclass | 1311 +++++++++++++++++ meta/classes/targz-img.bbclass | 1 + meta/classes/ubi-img.bbclass | 1 + meta/classes/ubifs-img.bbclass | 1 + meta/classes/wic-img.bbclass | 1 + meta/conf/bitbake.conf | 10 +- meta/lib/oe/gpg_sign.py | 130 ++ meta/lib/oe/sstatesig.py | 603 ++++++++ .../isar-bootstrap/isar-bootstrap.inc | 24 + 19 files changed, 2172 insertions(+), 17 deletions(-) create mode 100644 meta/classes/sstate.bbclass create mode 100644 meta/lib/oe/gpg_sign.py create mode 100644 meta/lib/oe/sstatesig.py -- 2.30.2