From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7016230395466219520 X-Received: by 2002:a19:9109:: with SMTP id t9mr10993501lfd.214.1638436803616; Thu, 02 Dec 2021 01:20:03 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a4c2:: with SMTP id p2ls877478ljm.4.gmail; Thu, 02 Dec 2021 01:20:02 -0800 (PST) X-Google-Smtp-Source: ABdhPJyabM9sRvYg0YkOXw3e245bDORo5e/uoFZ62LEiCg/7jOTVws8ShuDPR5mcoyxah5XNL5Jo X-Received: by 2002:a05:651c:54d:: with SMTP id q13mr11011043ljp.498.1638436802509; Thu, 02 Dec 2021 01:20:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638436802; cv=none; d=google.com; s=arc-20160816; b=pD1G8uoBpvuLrYLXoIHwSrG4ZFB6BjTcEPXuOaRTr4QfqVqO9a20/I5V1lIOayI7v2 FDRgJx9ZG4Zrad8lcsmg8TfncfnqJG3e0Db6J89YFJA+DkwIbMevOkaJvswxLumjSWgf G4F3sUhq9Mmakk0LDtkPNGUCwC6q61YPFJxCGUPL8PvYOEVZQyAvZR0OeodyguECoDZo tU5Lhe0srsmw/GHBwmxrnruW5yICY2StbBOkj9zLDUjzK2W2vzfDuZuR67aTdnbXzRGH 7OzXOi1BzzgT4y8PNUcblS9Nq3KIMr32hloYm8aA2xe7rgCTMQR3HEsej5/OWndu6uOU TH6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=NZBDMgZUWi/GgMl1UcaiyYww4gFt2k5nYNmbnoD5qqg=; b=SeqaGQ6VEKXjqB+GIYZWV5KS5PxT0I/Ue4NSw0FpEq4Rx71eGpFVvBlfNBOHy9n8oR AV81zLxmC5coqwZXnRGO7I3H54od6oUX9tm1kUxg/p7iwjUXPQftktN6ZpREg0wtOIVJ jzU+C2Yb82WGwS/OvbQJnDBlOXKo4kp4lEnSFUR9Z8i+9ISu7JutWNMGZA0oaD3+woyU 34zEqohvF9PXk5itBIRJoJngdYqvNIhOyxub5jSzo8YDbnoeNu8q7HrwwMFYOUGh2pCu lGQJxErnRScYgjBw7MDVshWv1eM7HXKUIFgaXXHcgLpkSK14/OjKbG87fkPZ4RvvSzjZ huTA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id v8si153009ljh.8.2021.12.02.01.20.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 02 Dec 2021 01:20:02 -0800 (PST) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from [192.168.67.164] (mm-140-61-214-37.mgts.dynamic.pppoe.byfly.by [37.214.61.140] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 1B29K0cU015187 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 2 Dec 2021 10:20:01 +0100 Subject: Re: [PATCH v6 00/13] Add sstate-cache To: Adriaan Schmidt , isar-users@googlegroups.com References: <20211111064728.2375760-1-adriaan.schmidt@siemens.com> From: Anton Mikanovich Message-ID: <6ba31d56-e7e7-0ebe-35d7-709e405fbd0f@ilbers.de> Date: Thu, 2 Dec 2021 12:19:55 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211111064728.2375760-1-adriaan.schmidt@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 6kqN/YOR5ZxD 11.11.2021 09:47, Adriaan Schmidt wrote: > This series adds the sstate-cache feature from OE to Isar. The cache holds > the results of bootstrapping, rootfs generation (buildchroot, image rootfs), > and deb package generation. > > To use the cache, the only configuration neccessary is setting SSTATE_DIR. > The contents of that directory need to be preserved across bitbake invocations. > > One known weakness is that the package lists of cached rootfs's can run > out of sync with upstream ("apt-get update" only happens at bootstrap time). > But this also happens with an "old" local build dir, and is something that > may be addressed elsewhere. For now, the recommendation is to frequently > clear the cache (in one of our projects we run a nightly "clear&populate > cache" CI job). > > Patches 1..5 fix (unrelated) issues that would otherwise block sstate caching, > patch 6 copies files from OE, and patches 7..10 add caching to Isar. > Patches 11..12 make minor adjustments to the tests, and patch 13 adds > a test case for the new feature. > > --- > Changes since v5: > - added lib/oe/utils.py because sstate.bbclass uses oe.utils.ThreadedPool > when accessing a remote sstate cache (SSTATE_MIRRORS) > > Changes since v4: > - rebase on next > - add test case > > Changes since v3: > - rebase on next > - make sstate understand that images and initramfss are MACHINE specific > (fixes a false-sharing bug that occured while testing with multiconfig) > > Changes since v2: > - rebase on next > - add the explicit isar-apt dependency to all images (not only wic as previously). > - don't move existing code in isar-bootstrap.inc > - add `do_deploy_deb[deptask] = "do_deploy_deb"` to dpkg-base. This is so that > packages depend recursively on their dependencies, even when build results > are taken from cache. > - generally improve commit messages > > Changes since v1: > - fix copy/paste typo in rootfs.bbclass > - add mounting trick to tar rootfs (because --one-file-system > does not stop at bind mounts) > - have install_imager_deps also depend explicitly on isar-apt > > Adriaan Schmidt (13): > oe imports in central location > images: create deploy dir > rootfs: recursively depend on packages > base: remove unneeded "before do_build" task dependencies > dpkg: add explicit dependency to isar-apt > meta: add sstate feature from oe > sstate: configure > sstate: add caching to isar-bootstrap > sstate: add caching to rootfs > sstate: add caching to debian packages > test: pass absolute path for build_dir > test: make bitbake_args a list > sstate: add test case > > meta/classes/base.bbclass | 32 +- > meta/classes/container-img.bbclass | 1 + > meta/classes/cpiogz-img.bbclass | 1 + > meta/classes/dpkg-base.bbclass | 34 +- > meta/classes/dpkg.bbclass | 1 + > meta/classes/ext4-img.bbclass | 1 + > meta/classes/fit-img.bbclass | 1 + > meta/classes/image-tools-extension.bbclass | 2 +- > meta/classes/image.bbclass | 3 + > meta/classes/initramfs.bbclass | 3 + > meta/classes/patch.bbclass | 5 - > meta/classes/rootfs.bbclass | 29 +- > meta/classes/sstate.bbclass | 1311 +++++++++++++++++ > meta/classes/targz-img.bbclass | 1 + > meta/classes/ubi-img.bbclass | 1 + > meta/classes/ubifs-img.bbclass | 1 + > meta/classes/wic-img.bbclass | 1 + > meta/conf/bitbake.conf | 10 +- > meta/lib/oe/gpg_sign.py | 130 ++ > meta/lib/oe/sstatesig.py | 603 ++++++++ > meta/lib/oe/utils.py | 569 +++++++ > .../isar-bootstrap/isar-bootstrap.inc | 24 + > scripts/ci_build.sh | 2 +- > testsuite/build_test/build_test.py | 101 +- > testsuite/build_test/cibase.py | 20 +- > testsuite/build_test/cibuilder.py | 2 +- > 26 files changed, 2853 insertions(+), 36 deletions(-) > create mode 100644 meta/classes/sstate.bbclass > create mode 100644 meta/lib/oe/gpg_sign.py > create mode 100644 meta/lib/oe/sstatesig.py > create mode 100644 meta/lib/oe/utils.py > Patches 01-10 applied to next, thanks. Test API rebuild and sstate test case can be separated into the new patchset. -- Anton Mikanovich Promwad Ltd. External service provider of ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn, Germany +49 (89) 122 67 24-0 Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov