From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7114179385042665472 X-Received: by 2002:a17:906:e256:b0:726:abd9:1063 with SMTP id gq22-20020a170906e25600b00726abd91063mr7169787ejb.741.1656398965958; Mon, 27 Jun 2022 23:49:25 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:907:1c8e:b0:726:d066:4632 with SMTP id nb14-20020a1709071c8e00b00726d0664632ls475893ejc.3.gmail; Mon, 27 Jun 2022 23:49:24 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tU04VGMp5suzO8RpJV72k8DIOYTSrbelSUKNkCdy6yyb+5AEFkbrzpd081NAseuyq/0Q4k X-Received: by 2002:a17:907:6ea3:b0:726:ca39:5d98 with SMTP id sh35-20020a1709076ea300b00726ca395d98mr2495291ejc.400.1656398964816; Mon, 27 Jun 2022 23:49:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656398964; cv=none; d=google.com; s=arc-20160816; b=o/LIETeTDizNRbJjF01hoqdCqa4Aj5A0nmdIeFgVnSgqhp47hLBUI1tS+pmkr9ckt0 OgbwLfT4/DmbFdomV1QstvEMAqOBxYelKVh/wUwNVkEW4x11U54UaZur2hPd2WJLa6ot 4D9u4HQp8lpG12SAg2LVxPEX4OAsKrlVLHRPGWFiIkBCprpdkRzt680gG9vTpb79NlpY Jqk8r6Z4alIHnMDeoUKoZkseZy9yKymP0xDdxckTN44cr4m8ZQ7O3iXBGbwGH2njUnU6 Q9v5ClgNBLft6LxlTuy6tfVLAjCKKz3QPm8CVe0YswLGDdffFJdqzCIZqKTC9U1BemK2 zvyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=7nl8qo6ee3reIXODUA9PnrDBgzSx04FhHCFnI48NPAA=; b=OKUpcbonO7G4Y+ArwsLzQzmHXOEXlplnfV8aAIKRAyymOkdEWyPMhkg6HFCqJDZF4Y DHJTVxidiNRYgzn+3TYUJEcKDk5gHb7nYYXBVs3vbe5193ShSmgcT1e1xR0OZDQQJcK3 1dGiRYly+O+sXfKWp/tNynZJ2sEVdDk/xwHU+4z26mgl6m8+SMVbtfPzQRiHhn9nx+cl /CGeD60ogsbv4PGuzERsCRutSbw2jA1IrMGNYoPOMjfc/hOrRTu/p5+WdbZn6O2FgmLM OVhxxxCJqcxPiWJAb8YysgX2Id6aPMkbiqrcK5UIi5iu63KPnsYUDY4+k7KQPEr4PBEH YtVw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id v21-20020a1709063bd500b006fefa121de4si432434ejf.1.2022.06.27.23.49.24 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Jun 2022 23:49:24 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from localhost.localdomain (mm-172-27-214-37.mgts.dynamic.pppoe.byfly.by [37.214.27.172] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 25S6nMS4030758 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 28 Jun 2022 08:49:23 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v12 00/21] Sbuild/Schroot migration Date: Tue, 28 Jun 2022 09:48:52 +0300 Message-Id: <20220628064913.27938-1-amikan@ilbers.de> X-Mailer: git-send-email 2.17.1 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: pCyd9KFYcSO8 This patchset integrates sbuild/schroot tools into Isar build system. Base schroot images are created with sbuild-chroot-target or, in case of cross-build, sbuild-chroot-host recipes. These images are based on isar-bootstrap ones, but include some build-related stuff preinstalled. To use these changes you need to have sbuild and schroot installed and configured. User who runs build should be added to sbuild group. Currently, sbuild-related stuff is supported in latest kas. If Gilab CI is used, the following configuration changes are required: - external dir for schroot overlay should be set in the config /etc/gitlab-runner/config.toml: volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"] The patchset was tested with 'meta-iot2050' and 'xenomai-images' downstreams in combination with kas image mentioned above. Current limitations: - parallel building of several packages requires more free space in comparison with buildchroot-based architecture. This happens due the sbuild architecture which uses some basic layer (common for all packages) and per-package separate layer (where builddeps are downloaded and installed). Changes sinse v11: - Rebase on current next. Changes sinse v10: - Rebase on current next. Changes sinse v9: - Add caching improvement patch; - Remove per-package deb artifacts dir after usage to reduce disk space; - Improve schroot cleanup on failures. Changes since v8: - Added patch with support of package versions with epochs Changes since v7: - fixed work with proxy by adding some environment filters in sbuild; - DEB_BUILD_PROFILES support simplified and improved - Fixed gbp-based recipes rebuilding (get back --git-ignore-new option previously removed) - fixed base-apt caching for packages and source packages temporary installed in sbuild isolated environment - fixed rare build error related to ccache setup (added locking) Changes since v6: - added patch that switches sbuild from using source directory to using .dsc file. The file is generated before sbuild call by `dpkg-source`. Changes since v5: - warning about shell exports in dpkg_runbuild_prepend. This should avoided be reworked to templates. Changes since v4: - consider shell exports done in dpkg_runbuild_prepend and pass them to sbuild environment; - fixed ccache work. Changes since v3: - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams compatibility; - dpkg_build_export is used for adjusting sbuild environment; - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build. Changes since v2: - patches reworked/squashed for easier reading and understanding; - fixed building foreigh architectures with kas-docker; - implemented support of ccache; - fixed devshell and devshell_nodeps. Changes since v1: - parallel builds with different BUILD_DIR are supported; - parallel multiconfig targets in one build are supported; - per-task schroot configuration in /etc/schroot/ is now used; - patchset now passes Jenkins CI (so patches changes RFC => PATCH). Anton Mikanovich (3): dpkg: Build packages with sbuild dpkg-base: Cleanup on schroot fail sbuild: Add changelog entry Felix Moessbauer (4): fix: support build of packages with epoch version always create apt-cache dirs in deb_dl_dir_import avoid absolute SCHROOT_* paths to improve caching locate generated binary package in WORKDIR Uladzimir Bely (14): dpkg-gbp: Use separate command to export tarball dpkg-gbp: Use host tools for dsc preparation sbuild: Add recipes for host and target rootfs to run sbuild sbuild: Introduce a class for another build method sbuild: Support of DEB_BUILD_PROFILES sbuild: Support of shell exports from dpkg_runbuild_prepend dpkg: Remove builddeps install task sbuild: Add ccache support dpkg-base: Switch devshell to use schroot dpkg-base: Switch apt_fetch and apt_unpack to use schroot doc: Add sbuild-related documentation sbuild: Use .dsc file instead of source directory sbuild: Fixed proxy support sbuild: Fix debsrc_download for packages dependencies RECIPE-API-CHANGELOG.md | 13 ++ doc/user_manual.md | 22 ++- meta/classes/deb-dl-dir.bbclass | 6 +- meta/classes/dpkg-base.bbclass | 90 ++++++---- meta/classes/dpkg-gbp.bbclass | 26 +-- meta/classes/dpkg.bbclass | 109 +++++++++--- meta/classes/sbuild.bbclass | 159 ++++++++++++++++++ meta/conf/bitbake.conf | 2 + .../sbuild-chroot/sbuild-chroot-host.bb | 13 ++ .../sbuild-chroot/sbuild-chroot-target.bb | 10 ++ .../sbuild-chroot/sbuild-chroot.inc | 39 +++++ 11 files changed, 408 insertions(+), 81 deletions(-) create mode 100644 meta/classes/sbuild.bbclass create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc -- 2.17.1