From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7221425893476925440 X-Received: by 2002:a05:6402:3211:b0:504:a254:ecce with SMTP id g17-20020a056402321100b00504a254eccemr3330378eda.1.1681369238962; Thu, 13 Apr 2023 00:00:38 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:907:784:b0:921:8068:e25e with SMTP id xd4-20020a170907078400b009218068e25els24982853ejb.0.-pod-prod-gmail; Thu, 13 Apr 2023 00:00:37 -0700 (PDT) X-Google-Smtp-Source: AKy350YUW2enA7DaHSQZZupoODR9ClbsXxv6emnTwYG5O00mPEFy9UqcNeXoZKWwOC0p7oqcxBKf X-Received: by 2002:a17:906:e107:b0:94a:9f9a:b3c4 with SMTP id gj7-20020a170906e10700b0094a9f9ab3c4mr1662643ejb.49.1681369237227; Thu, 13 Apr 2023 00:00:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681369237; cv=none; d=google.com; s=arc-20160816; b=LLq8SeFSfBbcIqkrP9E1EOWvwc6/GMA4njj5+Zdqc2sn03er1GFBPEREUmf6rZXje7 R+qDR3e9j5M3zYqqkgz8TTkNCHiigzVHaJVY01boCEN0jymy0jo+9civqjVXX5O6lzVP pafj2zcuTRprQ0rtEIMmNrgnigK8IPDl+q3EVHfjOjq3D2By2qZ+V2/lZFExBqp+4kGY iXsLO7Jy3lnhgCu7xQfYMaFUiCxMX6BTBCvfQCl5R2DgKHQRcxEHnEVVIPCjY1ag1Fi+ GFDoL6hS8qAze+kNNi/YOwYs2PaGTRDn2eAF8Nubk/m3gLUF2zaj5tcSa2Ez6RgAAqWh Eaow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=aecWUN/5dHem7Mwxx8PJMezGnvyptQvy3TXcoDxaesw=; b=kPZ/a2uZYJY4njo2L7m+Q+ZWnLoP46eGZ+LJwOdf6j+IwWqVkLw3nizwSu2fazVnvm to1DnPo2ipxoG1zixOlqUaSVZagLKigvkHhQoQEghSqfqEKseFg+Iab/P9pt7ZpB2PCf tCm0slXFIZ7rIkIv8uh2IngxfrZjUjYDB5sZgZS0KkaxSWd6HgCBHjmTL6hXkn3creYC RqM038d8f1rAEIXv+EuVIe67Za54oQ7N0gsv6DOZD1C5t13SOSJseia8pLkPxj77++jZ EfLPXwT5H10eLWTSOEOwFr7CPWJZU3idnUwbttexKPF5tAywltL9nzrbL+ZwY9Ep21k9 TQUA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=Jl8CQm9p; spf=pass (google.com: domain of fm-72506-202304130700360a581e732b2d2af345-adjjmd@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-72506-202304130700360a581e732b2d2af345-AdJjMD@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from mta-64-227.siemens.flowmailer.net (mta-64-227.siemens.flowmailer.net. [185.136.64.227]) by gmr-mx.google.com with ESMTPS id ti11-20020a170907c20b00b0094e847b3512si42789ejc.0.2023.04.13.00.00.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Apr 2023 00:00:37 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-72506-202304130700360a581e732b2d2af345-adjjmd@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) client-ip=185.136.64.227; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=Jl8CQm9p; spf=pass (google.com: domain of fm-72506-202304130700360a581e732b2d2af345-adjjmd@rts-flowmailer.siemens.com designates 185.136.64.227 as permitted sender) smtp.mailfrom=fm-72506-202304130700360a581e732b2d2af345-AdJjMD@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: by mta-64-227.siemens.flowmailer.net with ESMTPSA id 202304130700360a581e732b2d2af345 for ; Thu, 13 Apr 2023 09:00:36 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=aecWUN/5dHem7Mwxx8PJMezGnvyptQvy3TXcoDxaesw=; b=Jl8CQm9pyiIntBM11nj951a3KfD1Cv4jeIL/xxJB6lxoyt1UO710lJJ5TQ17716xV7FOPa s2ioowwowt28U6AVyvCFJ8TucA58VCQ81VFpjfRof8x4TssLOCt4WzMECdj+Z7IZYMLCdBAz IWczP0GaPR4KnlT2LJ1cMvKiGttw4=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, Felix Moessbauer Subject: [PATCH 1/2] add support for derived sbuild chroots Date: Thu, 13 Apr 2023 07:00:25 +0000 Message-Id: <20230413070026.3511123-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-TUID: isjGJuelBbx8 This patch adds support to create derived sbuild chroots to speedup the build process. For packages that share a large set of common build dependencies, a derived sbuild chroot can be created to avoid the overhead of installing all base build-deps on each sbuild invocation. Signed-off-by: Felix Moessbauer --- doc/user_manual.md | 27 +++++++++++++++++++ meta/classes/crossvars.bbclass | 11 +++++--- .../sbuild-chroot/sbuild-chroot.inc | 7 ++++- 3 files changed, 41 insertions(+), 4 deletions(-) diff --git a/doc/user_manual.md b/doc/user_manual.md index 92075a88..e7a16e78 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -1348,3 +1348,30 @@ To enable it, add the below line to your local.conf file. ``` BASE_REPO_FEATURES = "cache-deb-src" ``` + +## Use a custom sbuild chroot to speedup build + +### Motivation + +There are use-cases, where many packages need to be compiled but all of them +need a similar base of build dependencies. In case the baseline is quite big, +this adds a significant overhead as the build dependencies are installed individually +for each and every package. + +### Solution + +By creating a dedicated sbuild chroot for this use-case, the baseline can be installed +first and then all package builds of this type can use it. For that, create a +new recipe with the name `sbuild-chroot--`. In that recipe, +define the following: + +``` +require recipes-devtools/sbuild-chroot/sbuild-chroot-.bb + +SBUILD_FLAVOR = "" +SBUILD_CHROOT_PREINSTALL_COMMON += "" +``` + +Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = ""`. +To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_COMMON`. +An full example is provided by the `samefile` recipe. diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass index 201d460f..8d1da09c 100644 --- a/meta/classes/crossvars.bbclass +++ b/meta/classes/crossvars.bbclass @@ -10,22 +10,27 @@ python __anonymous() { d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name) mode = d.getVar('ISAR_CROSS_COMPILE') + + # support derived schroots + flavor = d.getVar('SBUILD_FLAVOR') + flavor_suffix = ('-' + flavor) if flavor else '' + distro_arch = d.getVar('DISTRO_ARCH') if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or distro_arch == None: d.setVar('BUILD_HOST_ARCH', distro_arch) schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False) - sbuild_dep = "sbuild-chroot-target:do_build" + sbuild_dep = "sbuild-chroot-target" + flavor_suffix + ":do_build" buildchroot_dir = d.getVar('BUILDCHROOT_TARGET_DIR', False) buildchroot_dep = "buildchroot-target:do_build" sdk_toolchain = "build-essential" else: d.setVar('BUILD_HOST_ARCH', d.getVar('HOST_ARCH')) schroot_dir = d.getVar('SCHROOT_HOST_DIR', False) - sbuild_dep = "sbuild-chroot-host:do_build" + sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build" buildchroot_dir = d.getVar('BUILDCHROOT_HOST_DIR', False) buildchroot_dep = "buildchroot-host:do_build" sdk_toolchain = "crossbuild-essential-" + distro_arch - d.setVar('SCHROOT_DIR', schroot_dir) + d.setVar('SCHROOT_DIR', schroot_dir + flavor_suffix) d.setVar('SCHROOT_DEP', sbuild_dep) d.setVar('BUILDCHROOT_DIR', buildchroot_dir) d.setVar('BUILDCHROOT_DEP', buildchroot_dep) diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index fd8bb648..b786eef5 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -11,6 +11,11 @@ PV = "1.0" inherit rootfs inherit compat +# set the flavor to create derived sbuild chroots +# this maps to a schroot created by a recipe named sbuild-chroot-- +SBUILD_FLAVOR ??= "" +SBUILD_SCHROOT_SUFFIX = "${@ ('-' + d.getVar('SBUILD_FLAVOR')) if d.getVar('SBUILD_FLAVOR') else ''}" + python() { distro_gcc = d.getVar('DISTRO_GCC') distro_arch = d.getVar('DISTRO_ARCH') @@ -52,7 +57,7 @@ ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}" # We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild ROOTFS_CONFIGURE_COMMAND:remove = "rootfs_configure_isar_apt" -DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}" +DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}${SBUILD_SCHROOT_SUFFIX}" do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}" do_sbuildchroot_deploy() { -- 2.34.1