From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7202866696458076160 X-Received: by 2002:a62:d148:0:b0:5a8:be15:db17 with SMTP id t8-20020a62d148000000b005a8be15db17mr1230238pfl.52.1677048089820; Tue, 21 Feb 2023 22:41:29 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:90b:224d:b0:233:cf4f:61f4 with SMTP id hk13-20020a17090b224d00b00233cf4f61f4ls1255471pjb.1.-pod-canary-gmail; Tue, 21 Feb 2023 22:41:28 -0800 (PST) X-Google-Smtp-Source: AK7set9pVSS7nLt+kH95mt4kCEbQc28R36HisnRz6DJW03302UaUu/4a1AhVsaNH/HG4YFbhPx4Y X-Received: by 2002:a17:902:e84b:b0:199:1160:956c with SMTP id t11-20020a170902e84b00b001991160956cmr7525748plg.31.1677048088691; Tue, 21 Feb 2023 22:41:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677048088; cv=none; d=google.com; s=arc-20160816; b=rnx12U6pjzEjkIVoUE2qK1xAttov8puz5QvpHaADlDAXhAwLoVqkQYQ2oKl2pCDgl0 aKdejV5xdAkplch4MUORjDGMAPOg69WjKbENhlcfyNeghzYxWljG5+0vYFMqbUgyqOSE A8oErlE8rO5T9lEIbpR2JUZco33CWtOSxY8UDbvNY97bQHES2lkbWQLSkx5DAgYGF/jH 7PtI1jvYVt26cOOBfg36CarVr8XQdR6aeRPiClk+/sRGnlMsBIEjNyZT+HIy+/4enoKq e1tn0QiApM7CloM1xPT+g76EjcDqVvNbyV08WNGzZTK7uhXnNoGCCGqoLs33oUUpfNup s2wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=km5jt3g4CcIu1Rf32c+keuMv0l/INwGR7/IUh/0tVBk=; b=r0GopruFFldP1GAxBptrtDiiUVCQWo5qAStKXddISEW8bD02JhcB0RU3FK3hmyWBlE qXRFnpG3cShiM83jDAL1cEp0iPffdLbnekdX0+5L21/xaWnL0ODTnlG9fRB6l9qyVUvQ yYK6NFYh+5D6C5lwVwK49zz2U9e7SsUapQTu3AiDKu4SZzM3S6UkMP+LsO7iu/Ga33F7 tCwQ8j6MTyWsEi6zXFc08D8MOg9J7+k3JonuZ4zKwi+oKOUfMD8u5L943LBlkhJxlDaf vFqIEDkz2A7maYLa4OndzS7DfAQtCodkVQJZic5VxuSnYsfljjJf4A35/Nx4CDyPkVVH +fpA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q15-20020a170902eb8f00b0019a97faf63bsi293248plg.12.2023.02.21.22.41.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Feb 2023 22:41:28 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 31M6fPc4011872 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 22 Feb 2023 07:41:25 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v4 00/13] Improving base-apt usage PoC Date: Wed, 22 Feb 2023 07:41:12 +0100 Message-Id: <20230222064125.22754-1-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: vWiAnhrbSb5i Currently, base-apt is used in the following way: - After the first build, all .deb files that took part in the process are is cached in the ${DL_DIR}/deb/. All the packages are downloaded from the remote repositories. - At the second build, `base-apt` repo is first created from the previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set). Futher debootstrap and installing packages are done from this local repo The idea of this patchset is using local `base-apt` repo even at the first build. Before any build step that requires additional packages, these packages are predownloaded with `debrepo` script to the `base-apt` repo and the debootstrapping / installation are always done from it. Potentialy, such an approach allows to stop using deb-del-dir import and export functionality in favour of debian-like repositories. Currently, separate `debrepo` script is used for prefetching packages to the local 'base-apt' repo. It requires python3-apt to be installed on the build host. Potentially, the functionality it provides could be directly integrated to the `debrepo.bbclass`. Changes since v3: - rebased on latest next; - cross-building for raspberry supported; - code passes both full and fast CI. Changes since v2: - populate base-apt before using at all steps of native build. Changes since v1: - rebased on latest next; - updated patchset description. Uladzimir Bely (13): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta-isar: Add local ubuntu-focal public key meta: Always use base-apt repo in local mode meta: move base-apt from deploy directory to the top meta: Use cached base-apt repo to debootstrap meta: Setup debrepo context for dpkg-base meta: Setup debrepo context for buildchroot meta: Setup debrepo context for sbuild-chroot base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality to base-apt meta: Specify grub-efi packages arch isar-apt: Fix copying isar-apt to workdir meta-isar/conf/distro/ubuntu-focal.conf | 5 + meta-isar/conf/distro/ubuntu.public.key | 53 +++ meta/classes/buildchroot.bbclass | 10 +- meta/classes/deb-dl-dir.bbclass | 22 + meta/classes/debrepo.bbclass | 74 +++ meta/classes/dpkg-base.bbclass | 51 +- meta/classes/dpkg.bbclass | 4 +- meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 9 + meta/classes/rootfs.bbclass | 18 +- meta/conf/bitbake.conf | 9 +- meta/conf/distro/debian-common.conf | 8 +- .../isar-bootstrap/isar-bootstrap.inc | 124 +++-- meta/recipes-devtools/base-apt/base-apt.bb | 46 -- .../buildchroot/buildchroot.inc | 11 + .../sbuild-chroot/sbuild-chroot.inc | 11 + scripts/debrepo | 440 ++++++++++++++++++ testsuite/citest.py | 3 + 18 files changed, 799 insertions(+), 104 deletions(-) create mode 100644 meta-isar/conf/distro/ubuntu.public.key create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo -- 2.20.1