From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a05:6512:456:: with SMTP id y22mr11831572lfk.554.1635255812583; Tue, 26 Oct 2021 06:43:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:211d:: with SMTP id a29ls2089191ljq.10.gmail; Tue, 26 Oct 2021 06:43:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzgCr93z/d4y4H9a5+1QuGR/6M1P8RqK767nOmxO/wBtZlXa9X7oZXa65Sc1ByipLxVkmhP X-Received: by 2002:a2e:361a:: with SMTP id d26mr26737708lja.104.1635255811701; Tue, 26 Oct 2021 06:43:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635255811; cv=none; d=google.com; s=arc-20160816; b=B0oQo0fc0Z2RzW0VrRXbl9IfHCjEFD+47V1VJWLs7CcFrTE7Bj/jovrnNXgC9Ecaaz XR5VzOeTeMgvzUNpvn+0C2JAAi8x7S776m74TVzYSl3mc9rWbVPgWWz/4Tjhul33xSvc LnLfPRCQpJErE3ue5bLwEkV2X891vHAlKGolACWhGy7RL82W/x6DbCXnTLItlIz5zWL8 /FjezSt37DdZ51igMP56br93DBvfw4tmU9CpXE3OlETexI9hffODhTvQt6eXgxEgMe+2 fd7jLi4DbTFw3aErV25RYv7ZmkYUcJ0kwDIA0t5FvAg9gdz7fhNyDcjPwc0CL2FAyug0 2A8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=uFFSIb1TdYzgXYAtr7yntYXAAHmzM/mpyyCf/weEalI=; b=t94HWOYvHOf93R0ZMMr0v1QjddHYgAoRDr2U92FmBlGmFeCNkQWpZU+x9y+TdNlJVp BiOQsx+GEOUpAMewURkppeD01vB958QwYcZF2u8RlcGa2fjxhOGdYdXgRXcIhSUaGfHv WVgp+r7yazmaB1Y/0HVigVrfItmeeR9HFFOFSp/UU7jn4Kjc1I1OcqjHzAS+s761rOZe /tfn5PiTC3C9RKvytKJnIjRd2r4Sv1fTLWCB23frR7DazW5cNG/To4lwFiPwplLv0IHO q+Nhm3oPfXp5AAQEq5PWc/NXVTCkQDQXGUgbfrsu5CmDcI0YM35sIR8PF2CanqXN6RSF 0+LQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id h4si1011159lft.8.2021.10.26.06.43.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Oct 2021 06:43:31 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 19QDhUvP017761 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 26 Oct 2021 15:43:31 +0200 Received: from md1za8fc.ad001.siemens.net ([139.22.32.154]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 19QDhUA7021262; Tue, 26 Oct 2021 15:43:30 +0200 Date: Tue, 26 Oct 2021 15:43:27 +0200 From: Henning Schild To: Adriaan Schmidt Cc: isar-users@googlegroups.com Subject: Re: [PATCH v4 00/10] Add sstate-cache Message-ID: <20211026154327.5af07a17@md1za8fc.ad001.siemens.net> In-Reply-To: <20211026122811.2654125-1-adriaan.schmidt@siemens.com> References: <20211026122811.2654125-1-adriaan.schmidt@siemens.com> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 6wLVtK5lu6gZ Can not wait to see that merged. Or maybe i can ... having to wait in isar projects that do not yet use this! It is so cool to see it in place and saving tons of time! I am not sure but maybe p4 might need an entry in the API CHANGELOG, maybe others as well. You fixed several subtile dependency bugs, which might in fact be present in layers again (copied over from a bad example). It might be hard to guess which mistakes people made or which of your changes should be considered an API change ... Maybe a generic section in the CHANGELOG could try and say "sstate enabled" things will be fast on partial rebuilds ... deps might have changed in a weird way ;) Same goes for the user manual, maybe a short section on the new feature. How to clean the cache etc. Or a pointer to OE. Maybe also suggests on how to configure the cache on a CI runner and how to evict it. Not with code but maybe conceptually. "evict your sstate cache on CI runners" regularly because of "outdated apt database from apt-get update in bootstraped rootfss" regards, Henning Am Tue, 26 Oct 2021 14:28:01 +0200 schrieb Adriaan Schmidt : > This series adds the sstate-cache feature from OE to Isar. The cache > holds the results of bootstrapping, rootfs generation (buildchroot, > image rootfs), and deb package generation. > > To use the cache, the only configuration neccessary is setting > SSTATE_DIR. The contents of that directory need to be preserved > across bitbake invocations. > > One known weakness is that the package lists of cached rootfs's can > run out of sync with upstream ("apt-get update" only happens at > bootstrap time). But this also happens with an "old" local build dir, > and is something that may be addressed elsewhere. For now, the > recommendation is to frequently clear the cache (in one of our > projects we run a nightly "clear&populate cache" CI job). > > Patches 1..5 fix (unrelated) issues that would otherwise block sstate > caching, patch 6 copies files from OE, and patches 7..10 add caching > to Isar. > > --- > Changes since v3: > - rebase on next > - make sstate understand that images and initramfss are MACHINE > specific (fixes a false-sharing bug that occured while testing with > multiconfig) > > Changes since v2: > - rebase on next > - add the explicit isar-apt dependency to all images (not only wic as > previously). > - don't move existing code in isar-bootstrap.inc > - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This > is so that packages depend recursively on their dependencies, even > when build results are taken from cache. > - generally improve commit messages > > Changes since v1: > - fix copy/paste typo in rootfs.bbclass > - add mounting trick to tar rootfs (because --one-file-system > does not stop at bind mounts) > - have install_imager_deps also depend explicitly on isar-apt > > Adriaan Schmidt (10): > oe imports in central location > images: create deploy dir > rootfs: recursively depend on packages > base: remove unneeded "before do_build" task dependencies > dpkg: add explicit dependency to isar-apt > meta: add sstate feature from oe > sstate: configure > sstate: add caching to isar-bootstrap > sstate: add caching to rootfs > sstate: add caching to debian packages > > meta/classes/base.bbclass | 32 +- > meta/classes/container-img.bbclass | 1 + > meta/classes/cpiogz-img.bbclass | 1 + > meta/classes/dpkg-base.bbclass | 34 +- > meta/classes/dpkg.bbclass | 1 + > meta/classes/ext4-img.bbclass | 1 + > meta/classes/fit-img.bbclass | 1 + > meta/classes/image-tools-extension.bbclass | 2 +- > meta/classes/image.bbclass | 3 + > meta/classes/initramfs.bbclass | 3 + > meta/classes/patch.bbclass | 5 - > meta/classes/rootfs.bbclass | 29 +- > meta/classes/sstate.bbclass | 1311 > +++++++++++++++++ meta/classes/targz-img.bbclass | > 1 + meta/classes/ubi-img.bbclass | 1 + > meta/classes/ubifs-img.bbclass | 1 + > meta/classes/wic-img.bbclass | 1 + > meta/conf/bitbake.conf | 10 +- > meta/lib/oe/gpg_sign.py | 130 ++ > meta/lib/oe/sstatesig.py | 603 ++++++++ > .../isar-bootstrap/isar-bootstrap.inc | 24 + > 21 files changed, 2178 insertions(+), 17 deletions(-) > create mode 100644 meta/classes/sstate.bbclass > create mode 100644 meta/lib/oe/gpg_sign.py > create mode 100644 meta/lib/oe/sstatesig.py >