From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7237382541983875072 X-Received: by 2002:a05:620a:4481:b0:759:2ac4:af2c with SMTP id x1-20020a05620a448100b007592ac4af2cmr441350qkp.7.1685084435262; Fri, 26 May 2023 00:00:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:75d8:0:b0:3f5:3b17:86f3 with SMTP id z24-20020ac875d8000000b003f53b1786f3ls2582182qtq.1.-pod-prod-01-us; Fri, 26 May 2023 00:00:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5EGDfF9vYIPm191oQpBgdhhCXAzUbu4CWk5NhoDgWApQDazVwfGfoT6IQTWZquU09kk7Sv X-Received: by 2002:a05:622a:651:b0:3f5:1ff7:db72 with SMTP id a17-20020a05622a065100b003f51ff7db72mr755069qtb.22.1685084434581; Fri, 26 May 2023 00:00:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685084434; cv=none; d=google.com; s=arc-20160816; b=Xaz+YY5wdhXwAf9c5Ykj+a8TELF4/qWpsE4tClxNJL+nrQ1/6tFMn4l6l+DZeLdhRX 6b/ZfyD7LZPfL5xyCADeQQi6e4qSkzSe0u6n2GSw17HDb++xr9lW6KyvlTGQQoPSuylc EGdhiXlzvNaBd6BFM2yaMP0996cqdfC6XEydHP4vuPVW5ZnbYIBxa08DMJTD5KgDXrn6 E7EbCTKEFvujXDj2zYp2Gh2z7DmJJoNAD7wKoAo+tdi9KolUhp4oObrfUQoTI9F3Uxzl yoRJC1cUt7XEgWkOE/s1Xw5w5ooMKWwY7fwVWssiKI/ZCvTqjnGXdIMzntxnqnQ3rFZe Cu+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=9IlyKdq/6/GyL+WTq1aDXpzThLnVvPO9vx6q/AaXua0=; b=fYJ4bjB7zaKE9Hl2u9xsCXD0Vfh2cpxi8mjyBd/noRNy0CW+BwH4R+ELfAbgvZ/+Os v3WcyxgPku8AnqMvXxpkDbO60NU9Cy693XZWeX4l7Ems0QQCmQN9qwjWA0VK950U7w3h L9f383BfaCHK7oBSpHb4Layi5gOyRcvkTmQmc9CjSpW7YEIrHD41qP0KB5rfG289QQp1 mkax4UAbbicyLRb6DZEsbk2++UadtcOfFy53rVjJKVTfK+KLpnUBA1GYUF3h4Rq7AZwp 4eq3sopMYGQYMMmyTww0yGp1Y1yFXO1TiZDsGV6UTY/xhDMp8PkRQ5HluWk8jZVEMbxF 7BBw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id v8-20020a05622a188800b003ef46e5143dsi306243qtc.3.2023.05.26.00.00.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 26 May 2023 00:00:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 34Q70R3U013306 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 26 May 2023 09:00:28 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v5 00/12] Improving base-apt usage PoC Date: Fri, 26 May 2023 09:00:15 +0200 Message-Id: <20230526070027.16890-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 7e8T5h4P7aOU Currently, base-apt is used in the following way: - After the first build, all .deb files that took part in the process are is cached in the ${DL_DIR}/deb/. All the packages are downloaded from the remote repositories. - At the second build, `base-apt` repo is first created from the previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set). Futher debootstrap and installing packages are done from this local repo The idea of this patchset is using local `base-apt` repo even at the first build. Before any build step that requires additional packages, these packages are predownloaded with `debrepo` script to the `base-apt` repo and the debootstrapping / installation are always done from it. Potentialy, such an approach allows to stop using deb-del-dir import and export functionality in favour of debian-like repositories. Currently, separate `debrepo` script is used for prefetching packages to the local 'base-apt' repo. It requires python3-apt to be installed on the build host. Potentially, the functionality it provides could be directly integrated to the `debrepo.bbclass`. Changes since v4: - rebased on latest next; - rearranged patches since some of them are already in next; - added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes. - code passes both full and fast CI in both "old" and "new" modes. Changes since v3: - rebased on latest next; - cross-building for raspberry supported; - code passes both full and fast CI. Changes since v2: - populate base-apt before using at all steps of native build. Changes since v1: - rebased on latest next; - updated patchset description. Uladzimir Bely (12): meta: move base-apt from deploy directory to the top ci_build.sh: Install python3-apt if not installed scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to debootstrap meta: Consider global debrepo context base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode Set ISAR_PREFETCH_BASE_APT by default meta: Specify grub-efi packages arch Disable deb-dl-dir in base-apt prefetch mode RECIPE-API-CHANGELOG.md | 9 + meta-isar/conf/local.conf.sample | 4 + meta/classes/buildchroot.bbclass | 5 +- meta/classes/deb-dl-dir.bbclass | 25 + meta/classes/debrepo.bbclass | 86 ++++ meta/classes/dpkg-base.bbclass | 47 +- meta/classes/dpkg.bbclass | 8 + meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 9 + meta/classes/rootfs.bbclass | 13 +- meta/conf/bitbake.conf | 9 +- meta/conf/distro/debian-common.conf | 8 +- .../isar-bootstrap/isar-bootstrap.inc | 111 ++++- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../buildchroot/buildchroot.inc | 11 + .../sbuild-chroot/sbuild-chroot.inc | 11 + scripts/ci_build.sh | 8 +- scripts/debrepo | 443 ++++++++++++++++++ testsuite/citest.py | 3 + 19 files changed, 804 insertions(+), 32 deletions(-) create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo -- 2.20.1