From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7032253102499561472 X-Received: by 2002:a17:907:3f83:: with SMTP id hr3mr7362800ejc.555.1638968746737; Wed, 08 Dec 2021 05:05:46 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:c517:: with SMTP id o23ls1854816edq.2.gmail; Wed, 08 Dec 2021 05:05:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJxMyl8xRD6DiScV9wxQ4B5Wx4hV+GtTNZ9QTEwLUBm7grtfsr7XqQ0Lg5T8G3ZTFJZ1D84C X-Received: by 2002:a05:6402:174c:: with SMTP id v12mr19483118edx.168.1638968745571; Wed, 08 Dec 2021 05:05:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1638968745; cv=none; d=google.com; s=arc-20160816; b=puTH7Rr4WbOFrrwV5D0JFB5v9vZlEgUoyHLM9sP0BN26Y8umj44Mti6n+/sDQdL5mh F1E7f7VNCOCi2y78CYPzvtn92XGDZ7WxrAgmlEpmaqywZAII50Mr5+8vECHDiJz8KA7M XFNtheiPSpLmSJNymJDhEcchumTeTLX/vjqR2gpFfXJlAC6oPT3IjyF3qx10Ueut7BNA n3eLk+NK564MiyQL0o6Ai2BCdBPoa8EPKQRf0l/THIIjCNJkCqgCzN2KKlUA5gVsiiJ6 5/lgwYderRl+LIbzPf3o0GwWFKyvrtov3HzpNBaEioObfyqLpg3NMUanbC1DJslT1iws 5Thw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=XyVlLkPMWFDk8w63NNN2gEgRke3z+7qqFu46NBYiu/U=; b=L43wRuYOnviriTbrhppyJhh+ZlUvywx1c72CVQbm6QonXuUH2O01VDpJ1OBX+gd3nB uCcZrCb3X08Xuqb30BBdIQyBkZafpxRko343YpbGf5BF04usl1R84v0dUp9katblJcAY 8v8GM4i0Ee02d/aq11MvQB+FzfINaB4dUiGoAIjwZ69NG/slfqqT2/OnTh89HDLKsAGC UAeD5O3MVPiXXI7gFx3/7LnOghWWbDfQYeBZ/dsOfRJrT1CvQ7vOYy5Om7UVpQrdL02l jMFJ36w4zm1m/+mS2fBsIw2W4hTMKttB/6IBAv+3Vjd5EJ3IkOD2+vE8/5+BDJSKYQpC WaUw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id e10si213547edz.5.2021.12.08.05.05.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Dec 2021 05:05:45 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 1B8D5iL1009193 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 8 Dec 2021 14:05:44 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v3 00/20] Sbuild/Schroot migration Date: Wed, 8 Dec 2021 14:05:24 +0100 Message-Id: <20211208130544.15133-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: HQjPc5YybLSx This is a patchset showing how sbuild/schroot tools can be integrated into Isar build system. Base schroot image is created with sbuild-chroot-target (or sbuild-chroot-host in case of cross-build) recipe. These images are based on isar-bootstrap ones, but include some build-related stuff preinstalled. To use this changes you need to have sbuild and schroot installed and configured. Current user should be added to sbuild group. If 'kas-container' is used, it should be modified to support sbuild: - /var/lib/schroot/union/overlay should be externally mounted (-v option) - export KAS_IMAGE_VERSION=2.6.2-sbuild - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord Here I uploaded a docker image that has preinstalled software (sbuild, schroot, etc) required to run sbuild. Later, the changes might be included in official kas docker images. If 'gitlab' is used, the similar changes are required: - 'image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild' in .gitlab-ci.yml - external dir for schroot overlay should also be specified in /etc/gitlab-runner/config.toml: volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"] The patchset was tested with 'meta-iot2050' and 'xenomai-images' downstreams. Some patches are required for these downstreams, appropriate pull requests will be created soon. Changes since v2: - patches reworked/squashed for easier reading and understanding - fixed building foreigh architectures with kas-docker - implemented support of ccache - fixed devshell and devshell_nodeps Changes since v1: - parallel builds with different BUILD_DIR are supported - parallel multiconfig targets in one build are supported - per-task schroot configuration in /etc/schroot/ is now used - patchset now passes Jenkins CI (so patches changes RFC => PATCH) Current limitations: - parallel building of several packages requires more free space in comparison with buildchroot-based architecture. This happens due the sbuild architecture which uses some basic layer (common for all packages) and per-package separate layer (where builddeps are downloaded and installed). For experiments, the code can be taken from Isar 'ubely/sbuild' branch. This is a dev branch, so sometimes it can be force-pushed when some fixes are applied of rebased to 'next' branch. First 9 patches (1..9) - are just preparation patches. With them, Isar will continue using buildchroot-based architecture. So, potentially, they can be merged as a subseries. Remaining 11 patches (10..20) represent a sbuild-related changes. Anton Mikanovich (10): template: Make templates passthrough linux-module: Do not use shell environment u-boot: Do not use shell environment trusted-firmware: Do not use shell environment optee-os: Do not use shell environment kselftest: Do not use shell environment linux-mainline: Move cfg fragment test to debian/rules dpkg: Build packages with sbuild sbuild: Introduce environment variables export API linux-custom: Prepare kernel config inside sbuild Uladzimir Bely (10): dpkg-gbp: Use separate command to export tarball dpkg-gbp: Use host tools for dsc preparation sbuild: Add recipes for host and target rootfs to run sbuild sbuild: Introduce a class for another build method dpkg: Remove builddeps install task. dpkg-base: Switch devshell to use schroot dpkg-base: Switch apt_fetch and apt_unpack to use schroot dpkg-base: Cleanup from buildchroot parts. doc: Add sbuild-related documentation sbuild: add ccache support doc/user_manual.md | 22 ++- .../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +- .../linux/linux-mainline_5.4.70.bb | 16 +- meta/classes/dpkg-base.bbclass | 98 ++++------ meta/classes/dpkg-gbp.bbclass | 26 +-- meta/classes/dpkg.bbclass | 53 ++--- meta/classes/sbuild.bbclass | 184 ++++++++++++++++++ meta/classes/template.bbclass | 7 +- meta/conf/bitbake.conf | 2 + .../files/debian/{rules => rules.tmpl} | 4 +- meta/recipes-bsp/optee-os/optee-os-custom.inc | 9 +- .../files/debian/{rules => rules.tmpl} | 4 +- .../trusted-firmware-a-custom.inc | 9 +- .../u-boot/files/debian/{rules => rules.tmpl} | 2 +- meta/recipes-bsp/u-boot/u-boot-custom.inc | 9 +- .../sbuild-chroot/sbuild-chroot-host.bb | 13 ++ .../sbuild-chroot/sbuild-chroot-target.bb | 10 + .../sbuild-chroot/sbuild-chroot.inc | 39 ++++ .../kselftest/files/{rules => rules.tmpl} | 0 meta/recipes-kernel/kselftest/kselftest.inc | 21 +- .../linux-module/files/debian/rules | 37 ---- .../linux-module/files/debian/rules.tmpl | 52 +++++ meta/recipes-kernel/linux-module/module.inc | 19 +- meta/recipes-kernel/linux/linux-custom.inc | 7 +- 24 files changed, 437 insertions(+), 208 deletions(-) create mode 100644 meta/classes/sbuild.bbclass rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%) rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%) rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%) create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%) delete mode 100755 meta/recipes-kernel/linux-module/files/debian/rules create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl -- 2.20.1