From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a2e:9c8d:: with SMTP id x13mr26029725lji.29.1635251300626; Tue, 26 Oct 2021 05:28:20 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:211d:: with SMTP id a29ls2037944ljq.10.gmail; Tue, 26 Oct 2021 05:28:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzwCuoBgp0120YRlu+AH2A6rAdEGdUdI8RFYXj2PaefvSgjolsm9MxbFj/ngpFZne9w90xp X-Received: by 2002:a2e:b0d3:: with SMTP id g19mr23939343ljl.529.1635251299579; Tue, 26 Oct 2021 05:28:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635251299; cv=none; d=google.com; s=arc-20160816; b=q81cBKDIv7w1W1DrYGwsKKX2RS+S3lsR60MtX+dHBXeX3VQeSqqzW8749CqICZ83Bv rpPTIOE8YUEEicnhKfpKgKQ5bCJFIYn9JVUZOW72cKdhjLc4Kjy1TsqEUV0kW9Bru/oB E1nmojcui0WqhC42egPwV8dxJMrA10h30i6tPY4/ThHkyeupRPm3bYriGk+kkQbLsYYW 8UH3eMz7OQ/7J6S54t2akMWjYJ9Wz1G0WEvH4fhAnhqETS74hbmY47imsLLlRDpwrQMp +HcABgmvBykHZtezrWe4uSwQ7NMJlBB8tdsh2ZxMalhaiDEK6GDmUYROe1O79bdUC+29 6YnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=l7hNE7hm2hyJTmnae7Gy6zT+7RCk4CIDkW+oiD8lV1Y=; b=CUFhtyMjvllgm8TlJmqKMD/7v8hfGO6MtdTSxHUqj1HHhVlPh7fAhl8zVxd+5TcFz1 N7BnUgo9ldHoyLDF7MiA3v+8ZYEpvSUekHinMkMn1N7MOjLjNCXLlC3JojeM2yHwpaZa b0miXxTDkRmZiImp1sArSgKBz3dzjC9VgwNjpaWtJrMQYD526PAiNM2JJf4GqkjSBH9t LHT9tJSt9QzRH+5zOi34lbdXeoBLcM4ii7RoZJ6eax73Na2i7LfxE+4A22K71NiBufCP lJ1j8ItIbkq/gA07pdTv8B+mxHiwn4lR7DUEjVG9/F2Nm5W7ND6DwHNgKnLkUaYFH6nA chFg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id v203si245940lfa.13.2021.10.26.05.28.19 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Oct 2021 05:28:19 -0700 (PDT) Received-SPF: pass (google.com: domain of adriaan.schmidt@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of adriaan.schmidt@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=adriaan.schmidt@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 19QCSIGM009029 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 26 Oct 2021 14:28:18 +0200 Received: from random.ppmd.siemens.net (random.ppmd.siemens.net [139.25.68.25]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 19QCSIPI013081; Tue, 26 Oct 2021 14:28:18 +0200 From: Adriaan Schmidt To: isar-users@googlegroups.com Cc: Adriaan Schmidt Subject: [PATCH v4 00/10] Add sstate-cache Date: Tue, 26 Oct 2021 14:28:01 +0200 Message-Id: <20211026122811.2654125-1-adriaan.schmidt@siemens.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: YgNdi6ab4P60 This series adds the sstate-cache feature from OE to Isar. The cache holds the results of bootstrapping, rootfs generation (buildchroot, image rootfs), and deb package generation. To use the cache, the only configuration neccessary is setting SSTATE_DIR. The contents of that directory need to be preserved across bitbake invocations. One known weakness is that the package lists of cached rootfs's can run out of sync with upstream ("apt-get update" only happens at bootstrap time). But this also happens with an "old" local build dir, and is something that may be addressed elsewhere. For now, the recommendation is to frequently clear the cache (in one of our projects we run a nightly "clear&populate cache" CI job). Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching, patch 6 copies files from OE, and patches 7..10 add caching to Isar. --- Changes since v3: - rebase on next - make sstate understand that images and initramfss are MACHINE specific (fixes a false-sharing bug that occured while testing with multiconfig) Changes since v2: - rebase on next - add the explicit isar-apt dependency to all images (not only wic as previously). - don't move existing code in isar-bootstrap.inc - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that packages depend recursively on their dependencies, even when build results are taken from cache. - generally improve commit messages Changes since v1: - fix copy/paste typo in rootfs.bbclass - add mounting trick to tar rootfs (because --one-file-system does not stop at bind mounts) - have install_imager_deps also depend explicitly on isar-apt Adriaan Schmidt (10): oe imports in central location images: create deploy dir rootfs: recursively depend on packages base: remove unneeded "before do_build" task dependencies dpkg: add explicit dependency to isar-apt meta: add sstate feature from oe sstate: configure sstate: add caching to isar-bootstrap sstate: add caching to rootfs sstate: add caching to debian packages meta/classes/base.bbclass | 32 +- meta/classes/container-img.bbclass | 1 + meta/classes/cpiogz-img.bbclass | 1 + meta/classes/dpkg-base.bbclass | 34 +- meta/classes/dpkg.bbclass | 1 + meta/classes/ext4-img.bbclass | 1 + meta/classes/fit-img.bbclass | 1 + meta/classes/image-tools-extension.bbclass | 2 +- meta/classes/image.bbclass | 3 + meta/classes/initramfs.bbclass | 3 + meta/classes/patch.bbclass | 5 - meta/classes/rootfs.bbclass | 29 +- meta/classes/sstate.bbclass | 1311 +++++++++++++++++ meta/classes/targz-img.bbclass | 1 + meta/classes/ubi-img.bbclass | 1 + meta/classes/ubifs-img.bbclass | 1 + meta/classes/wic-img.bbclass | 1 + meta/conf/bitbake.conf | 10 +- meta/lib/oe/gpg_sign.py | 130 ++ meta/lib/oe/sstatesig.py | 603 ++++++++ .../isar-bootstrap/isar-bootstrap.inc | 24 + 21 files changed, 2178 insertions(+), 17 deletions(-) create mode 100644 meta/classes/sstate.bbclass create mode 100644 meta/lib/oe/gpg_sign.py create mode 100644 meta/lib/oe/sstatesig.py -- 2.30.2