* [PATCH v3 0/5] Improving base-apt usage PoC
@ 2022-03-25 10:32 Uladzimir Bely
2022-03-25 10:32 ` [PATCH v3 1/5] Add debrepo python script handling base-apt Uladzimir Bely
` (4 more replies)
0 siblings, 5 replies; 7+ messages in thread
From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw)
To: isar-users
Currently, base-apt is used in the following way:
* At first build every deb file that took part in installing is cached
in directory $DL_DIR/deb/. Debootstrap is done from remote source.
* At second build, `base-apt` repo is created from previously downloaded
debs (if ISAR_USE_CACHED_BASE_REPO is set). Debootstrap and installing
packages later is done from this local repo
The idea of this patchset is to precreate `base-apt` at first build and
use it later for all steps that require any package downloads (build
dependencies. rootfs populate, etc). So, no second build is required
to get ready-to-use base-apt.
Currently, only native builds are supported. Architecture related
dependencies predownloading is not yet ready, so cross-builds are not
working at the moment.
In the future, there are plans to switch from debootstrap to mmdebstrap
that is able to work without sudo and uses more clean (apt/python-apt)
approach in comparison with debootstrap.
Changes since v2:
- populate base-apt before using at all steps of native build.
Changes since v1:
- rebased on latest next;
- updated patchset description.
Uladzimir Bely (5):
Add debrepo python script handling base-apt
meta: Use cached base-apt repo to debootstrap
meta: always use base-apt repo in local mode
base-apt: Predownload packages to base-apt before install.
draft: make isar-apt repo visible for base-apt
meta-isar/conf/distro/ubuntu-focal.conf | 4 +
meta-isar/conf/distro/ubuntu.public.key | 53 +++
meta/classes/buildchroot.bbclass | 10 +-
meta/classes/dpkg-base.bbclass | 8 +
meta/classes/dpkg-gbp.bbclass | 7 +
meta/classes/dpkg.bbclass | 7 +
meta/classes/image-locales-extension.bbclass | 7 +
meta/classes/image-tools-extension.bbclass | 7 +
meta/classes/rootfs.bbclass | 30 +-
.../isar-bootstrap/isar-bootstrap.inc | 59 ++-
scripts/debrepo | 363 ++++++++++++++++++
11 files changed, 526 insertions(+), 29 deletions(-)
create mode 100644 meta-isar/conf/distro/ubuntu.public.key
create mode 100755 scripts/debrepo
--
2.20.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v3 1/5] Add debrepo python script handling base-apt 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely @ 2022-03-25 10:32 ` Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 2/5] meta: Use cached base-apt repo to debootstrap Uladzimir Bely ` (3 subsequent siblings) 4 siblings, 0 replies; 7+ messages in thread From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw) To: isar-users Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- scripts/debrepo | 361 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100755 scripts/debrepo diff --git a/scripts/debrepo b/scripts/debrepo new file mode 100755 index 00000000..fba8342e --- /dev/null +++ b/scripts/debrepo @@ -0,0 +1,361 @@ +#!/usr/bin/env python3 + +# This software is a part of ISAR. +# Copyright (C) 2022 ilbers GmbH + +import os +import sys + +import subprocess +import getopt +import pickle + +import apt_pkg +import apt.progress.base + + +class DebRepo(object): + def __init__(self, workdir, cmdline_opts): + self.workdir = workdir + self.optsfile = self.workdir + "/repo.opts" + + # Set default values + self.distro = "debian" + + self.repo = self.workdir + "/repo/apt" + "/" + self.distro + self.repodb = self.workdir + "/repo/db" + "/" + self.distro + self.mirror = "http://deb.debian.org/debian" + self.arch = "amd64" + self.codename = "bullseye" + self.components = "main contrib non-free" + self.keydir = "/etc/apt/trusted.gpg.d" + self.check_gpg = True + + # Load stored opts + opts = self.load_opts() + print("stored opts: " + str(opts)) + + # Overwrite opts by cmdline_opts + for opt, arg in cmdline_opts.items(): + opts[opt] = arg + + print("all opts: " + str(opts)) + + # Replace by values passed in commandline + for opt, arg in opts.items(): + if opt == "repodir": + self.repo = arg + "/" + self.distro + if opt == "repodbdir": + self.repodb = arg + "/" + self.distro + if opt == "mirror": + self.mirror = arg + if opt == "arch": + self.arch = arg + if opt == "distro": + self.distro = arg + if opt == "codename": + self.codename = arg + if opt == "components": + self.components = arg.replace(",", " ") + if opt == "keydir": + self.keydir = arg + if opt == "check_gpg": + self.check_gpg = arg + + self.save_opts(opts) + + print("workdir: " + str(self.workdir)) + print("repo: " + str(self.repo)) + print("repodb: " + str(self.repodb)) + print("mirror: " + str(self.mirror)) + print("arch: " + str(self.arch)) + print("distro: " + str(self.distro)) + print("codename: " + str(self.codename)) + print("components: " + str(self.components)) + print("keydir: " + str(self.keydir)) + print("check_gpg: " + str(self.check_gpg)) + + self.cache = None + self.depcache = None + self.sr = None + + def create_rootfs(self): + if not os.path.exists(self.workdir + "/var/lib/dpkg"): + os.makedirs(self.workdir + "/var/lib/dpkg") + f = open(self.workdir + "/var/lib/dpkg" + "/status", "w") + f.flush + f.close + + if not os.path.exists(self.workdir + "/etc/apt/"): + os.makedirs(self.workdir + "/etc/apt/") + f = open(self.workdir + "/etc/apt/sources.list", "w") + f.write("deb [arch=" + self.arch + "] " + self.mirror + " " + self.codename + " " + self.components + "\n") + f.write("deb-src [arch=" + self.arch + "] " + self.mirror + " " + self.codename + " " + self.components + "\n") + f.flush + f.close + + if not os.path.exists(self.workdir + "/var/cache/apt/archives/partial"): + os.makedirs(self.workdir + "/var/cache/apt/archives/partial") + + def create_repo_dist(self): + if not os.path.exists(self.repo + "/conf"): + os.makedirs(self.repo + "/conf") + f = open(self.repo + "/conf" + "/distributions", "w") + f.write("Codename: " + self.codename + "\n") + f.write("Architectures: i386 armhf arm64 amd64 mipsel riscv64 source" + "\n") + f.write("Components: " + self.components + "\n") + f.flush + f.close + + def apt_config(self): + # Configure apt to work with empty directory + apt_pkg.config.set("APT::Architecture", self.arch) + apt_pkg.config.set("Dir", self.workdir) + apt_pkg.config.set("Dir::Cache", self.workdir + "/var/cache/apt") + apt_pkg.config.set("Dir::State::status", self.workdir + "/var/lib/dpkg/status") + + apt_pkg.config.set("APT::Install-Recommends", "0") + apt_pkg.config.set("APT::Install-Suggests", "0") + + # Use host keys for authentification + # apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg") + # apt_pkg.config.set("Dir::Etc::TrustedParts", "/etc/apt/trusted.gpg.d") + apt_pkg.config.set("Dir::Etc::TrustedParts", self.keydir) + + # Allow using repositories without keys + if not self.check_gpg: + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def mark_essential(self): + for pkg in self.cache.packages: + if pkg.essential: + self.depcache.mark_install(pkg) + + def mark_by_prio(self, priority): + for pkg in self.cache.packages: + ver = self.depcache.get_candidate_ver(pkg) + if ver and ver.priority <= priority: + self.depcache.mark_install(pkg) + + def mark_list(self, pkglist): + if pkglist: + for pkgname in pkglist: + if pkgname in self.cache: + pkg = self.cache[pkgname] + self.depcache.mark_install(pkg) + + def handle_deb(self, item): + subprocess.run([ + "reprepro", + "--dbdir", self.repodb, + "--outdir", self.repo, + "--confdir", self.repo + "/conf", + "-C", "main", + "includedeb", + self.codename, + item.destfile + ]) + + def handle_repo(self, fetcher): + fetcher.run() + for item in fetcher.items: + if item.status == item.STAT_ERROR: + print("Some error ocured: '%s'" % item.error_text) + pass + else: + self.handle_deb(item) + + def get_filename(self, url): + fragment_removed = url.split("#")[0] # keep to left of first # + query_string_removed = fragment_removed.split("?")[0] + scheme_removed = query_string_removed.split("://")[-1].split(":")[-1] + if scheme_removed.find("/") == -1: + return "" + return os.path.basename(scheme_removed) + + def download_file(self, uri): + filename = self.get_filename(uri) + subprocess.run([ + "wget", + uri, + "-O", + filename + ]) + + def handle_dsc(self, uri): + filename = self.get_filename(uri) + subprocess.run([ + "reprepro", + "--dbdir", self.repodb, + "--outdir", self.repo, + "--confdir", self.repo + "/conf", + "-C", "main", + "-S", "-", "-P" "source", + "--delete", + "includedsc", + self.codename, + os.path.realpath(filename) + ]) + + def handle_src_list(self, pkgs): + if pkgs: + for pkgname in pkgs: + dsc = "" + self.sr.restart() + if self.sr.lookup(pkgname): + print(self.sr.files) + for sr_file in self.sr.files: + print(self.sr.index.archive_uri(sr_file[2])) + self.download_file(self.sr.index.archive_uri(sr_file[2])) + + dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2]) + self.handle_dsc(dsc_uri) + + def apt_run(self, init, srcmode, pkgs, controlfile): + apt_pkg.init() + + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + self.depcache = apt_pkg.DepCache(self.cache) + self.sr = apt_pkg.SourceRecords() + + if init: + self.mark_essential() + # 1(required), 2(important), 3(standard), 4(optional), 5(extra) + self.mark_by_prio(1) + + if srcmode: + self.handle_src_list(pkgs) + else: + self.mark_list(pkgs) + + if controlfile: + fobj = open(controlfile, "r") + try: + Parse = apt_pkg.TagFile(fobj) + while Parse.step() == 1: + for item in apt_pkg.parse_depends(Parse.section.get("Build-Depends", "")): + #self.mark_list([item[0][0]]) + if item[0][0] in self.cache: + pkg = self.cache[item[0][0]] + print("pkg: " + str(pkg)) + self.depcache.mark_install(pkg) + + finally: + fobj.close() + + if init or not srcmode: + fetcher = apt_pkg.Acquire(progress) + pm = apt_pkg.PackageManager(self.depcache) + + recs = apt_pkg.PackageRecords(self.cache) + pm.get_archives(fetcher, sources, recs) + + self.handle_repo(fetcher) + + def load_opts(self): + params = {} + if os.path.isfile(self.optsfile): + with open(self.optsfile, 'rb') as file: + data = file.read() + if data: + params = pickle.loads(data) + + return params + + def save_opts(self, opts): + file = open(self.optsfile, 'wb') + pickle.dump(opts, file) + file.close() + + +class DebRepoArgs(object): + def __init__(self): + self.workdir = "" + self.init = False + self.srcmode = False + self.controlfile = "" + + self.opts = {} + self.pkgs = [] + + try: + opts, args = getopt.getopt(sys.argv[1:], "", [ + "init", + "srcmode", + "workdir=", + "repodir=", + "repodbdir=", + "mirror=", + "arch=", + "distro=", + "codename=", + "components=", + "keydir=", + "no-check-gpg", + "controlfile=", + ]) + except getopt.GetoptError as msg: + print("Error: " + str(msg)) + sys.exit(1) + + for opt, arg in opts: + if opt in ("--workdir"): + self.workdir = arg + if opt in ("--init"): + self.init = True + if opt in ("--srcmode"): + self.srcmode = True + if opt in ("--controlfile"): + self.controlfile = arg + + if opt in ("--repodir", + "--repodbdir", + "--mirror", + "--arch", + "--distro", + "--codename", + "--components", + "--keydir", + "--controlfile", + ): + self.opts[opt[2:]] = arg + if opt in ("--no-check-gpg"): + self.opts['check_gpg'] = False + + if not self.workdir: + print("Error: workdir is not specified") + sys.exit(1) + + self.pkgs = args + + +def main(): + args = DebRepoArgs() + + if not (args.init or args.pkgs or args.controlfile): + print("Nothing to do") + sys.exit(0) + + if not os.path.exists(args.workdir): + os.makedirs(args.workdir) + + debrepo = DebRepo(args.workdir, args.opts) + + if (args.init): + debrepo.create_rootfs() + debrepo.create_repo_dist() + + debrepo.apt_config() + debrepo.apt_run(args.init, args.srcmode, args.pkgs, args.controlfile) + + +if __name__ == "__main__": + main() -- 2.20.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v3 2/5] meta: Use cached base-apt repo to debootstrap 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 1/5] Add debrepo python script handling base-apt Uladzimir Bely @ 2022-03-25 10:32 ` Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 3/5] meta: always use base-apt repo in local mode Uladzimir Bely ` (2 subsequent siblings) 4 siblings, 0 replies; 7+ messages in thread From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw) To: isar-users This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta-isar/conf/distro/ubuntu-focal.conf | 4 ++ meta-isar/conf/distro/ubuntu.public.key | 53 +++++++++++++++++++ .../isar-bootstrap/isar-bootstrap.inc | 35 +++++++++++- 3 files changed, 91 insertions(+), 1 deletion(-) create mode 100644 meta-isar/conf/distro/ubuntu.public.key diff --git a/meta-isar/conf/distro/ubuntu-focal.conf b/meta-isar/conf/distro/ubuntu-focal.conf index 4dfa2014..67a59f6c 100644 --- a/meta-isar/conf/distro/ubuntu-focal.conf +++ b/meta-isar/conf/distro/ubuntu-focal.conf @@ -11,6 +11,10 @@ DISTRO_APT_SOURCES ?= "conf/distro/${DISTRO}.list" DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${DISTRO}-ports.list" HOST_DISTRO_APT_SOURCES_append_arm64 = " ${DISTRO_APT_SOURCES}" +BOOTSTRAP_KEY = "file://${LAYERDIR_isar}/conf/distro/ubuntu.public.key;sha256sum=36a38199a4bf4eae1e7f574891f7dfcb79b91b87a33a499383265e1224b5e989" +DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}" +HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}" + BASE_DISTRO_CODENAME = "focal" # that is what debootstrap_1.0.118ubuntu1 does anyways diff --git a/meta-isar/conf/distro/ubuntu.public.key b/meta-isar/conf/distro/ubuntu.public.key new file mode 100644 index 00000000..994f9f19 --- /dev/null +++ b/meta-isar/conf/distro/ubuntu.public.key @@ -0,0 +1,53 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBFufwdoBEADv/Gxytx/LcSXYuM0MwKojbBye81s0G1nEx+lz6VAUpIUZnbkq +dXBHC+dwrGS/CeeLuAjPRLU8AoxE/jjvZVp8xFGEWHYdklqXGZ/gJfP5d3fIUBtZ +HZEJl8B8m9pMHf/AQQdsC+YzizSG5t5Mhnotw044LXtdEEkx2t6Jz0OGrh+5Ioxq +X7pZiq6Cv19BohaUioKMdp7ES6RYfN7ol6HSLFlrMXtVfh/ijpN9j3ZhVGVeRC8k +KHQsJ5PkIbmvxBiUh7SJmfZUx0IQhNMaDHXfdZAGNtnhzzNReb1FqNLSVkrS/Pns +AQzMhG1BDm2VOSF64jebKXffFqM5LXRQTeqTLsjUbbrqR6s/GCO8UF7jfUj6I7ta +LygmsHO/JD4jpKRC0gbpUBfaiJyLvuepx3kWoqL3sN0LhlMI80+fA7GTvoOx4tpq +VlzlE6TajYu+jfW3QpOFS5ewEMdL26hzxsZg/geZvTbArcP+OsJKRmhv4kNo6Ayd +yHQ/3ZV/f3X9mT3/SPLbJaumkgp3Yzd6t5PeBu+ZQk/mN5WNNuaihNEV7llb1Zhv +Y0Fxu9BVd/BNl0rzuxp3rIinB2TX2SCg7wE5xXkwXuQ/2eTDE0v0HlGntkuZjGow +DZkxHZQSxZVOzdZCRVaX/WEFLpKa2AQpw5RJrQ4oZ/OfifXyJzP27o03wQARAQAB +tEJVYnVudHUgQXJjaGl2ZSBBdXRvbWF0aWMgU2lnbmluZyBLZXkgKDIwMTgpIDxm +dHBtYXN0ZXJAdWJ1bnR1LmNvbT6JAjgEEwEKACIFAlufwdoCGwMGCwkIBwMCBhUI +AgkKCwQWAgMBAh4BAheAAAoJEIcZINGZG8k8LHMQAKS2cnxz/5WaoCOWArf5g6UH +beOCgc5DBm0hCuFDZWWv427aGei3CPuLw0DGLCXZdyc5dqE8mvjMlOmmAKKlj1uG +g3TYCbQWjWPeMnBPZbkFgkZoXJ7/6CB7bWRht1sHzpt1LTZ+SYDwOwJ68QRp7DRa +Zl9Y6QiUbeuhq2DUcTofVbBxbhrckN4ZteLvm+/nG9m/ciopc66LwRdkxqfJ32Cy +q+1TS5VaIJDG7DWziG+Kbu6qCDM4QNlg3LH7p14CrRxAbc4lvohRgsV4eQqsIcdF +kuVY5HPPj2K8TqpY6STe8Gh0aprG1RV8ZKay3KSMpnyV1fAKn4fM9byiLzQAovC0 +LZ9MMMsrAS/45AvC3IEKSShjLFn1X1dRCiO6/7jmZEoZtAp53hkf8SMBsi78hVNr +BumZwfIdBA1v22+LY4xQK8q4XCoRcA9G+pvzU9YVW7cRnDZZGl0uwOw7z9PkQBF5 +KFKjWDz4fCk+K6+YtGpovGKekGBb8I7EA6UpvPgqA/QdI0t1IBP0N06RQcs1fUaA +QEtz6DGy5zkRhR4pGSZn+dFET7PdAjEK84y7BdY4t+U1jcSIvBj0F2B7LwRL7xGp +SpIKi/ekAXLs117bvFHaCvmUYN7JVp1GMmVFxhIdx6CFm3fxG8QjNb5tere/YqK+ +uOgcXny1UlwtCUzlrSaPmQINBE+tgXgBEADfiL1KNFHT4H4Dw0OR9LemR8ebsFl+ +b9E44IpGhgWYDufj0gaM/UJ1Ti3bHfRT39VVZ6cv1P4mQy0bnAKFbYz/wo+GhzjB +Wtn6dThYv7n+KL8bptSCXgg1a6en8dCCIA/pwtS2Ut/g4Eu6Z467dvYNlMgCqvg+ +prKIrXf5ibio48j3AFvd1dDJl2cHfyuON35/83vXKXz0FPohQ7N7kPfI+qrlGBYG +WFzC/QEGje360Q2Yo+rfMoyDEXmPsoZVqf7EE8gjfnXiRqmz/Bg5YQb5bgnGbLGi +HWtjS+ACIdLUq/h+jlSp57jw8oQktMh2xVMX4utDM0UENeZnPllVJSlR0b+ZmZz7 +paeSar8Yxn4wsNlL7GZbpW5A/WmcmWfuMYoPhBo5Fq1V2/siKNU3UKuf1KH+X0p1 +oZ4oOcZ2bS0Zh3YEG8IQce9Bferq4QMKsekcG9IKS6WBIU7BwaElI2ILD0gSwu8K +zvNSEeIJhYSsBIEzrWxIBXoN2AC9PCqqXkWlI5Xr/86RWllB3CsoPwEfO8CLJW2L +lXTen/Fkq4wT+apdhHeiWiSsq/J5OEff0rKHBQ3fK7fyVuVNrJFb2CopaBLyCxTu +pvxs162jjUNopt0c7OqNBoPoUoVFAxUSpeEwAw6xrM5vROyLMSeh/YnTuRy8WviR +apZCYo6naTCY5wARAQABtEJVYnVudHUgQXJjaGl2ZSBBdXRvbWF0aWMgU2lnbmlu +ZyBLZXkgKDIwMTIpIDxmdHBtYXN0ZXJAdWJ1bnR1LmNvbT6JAjgEEwECACIFAk+t +gXgCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEDtP5qzAsh8yXX4QAJHU +dK6eYMyJcrFP3yKXtUYQMpaHRM/floqZtOFhlmcLVMgBNOr0eLvBU0JcZyZpHMvZ +ciTDBMWX8ItCYVjRejf0K0lPvHHRGaE7t6JHVUCeznNbDMnOPYVwlVJdZLOa6PmE +5WXVXpk8uTA8vm6RO2rS23vE7U0pQlV+1GVXMWH4ZLjaQs/Tm7wdvRxeqTbtfOEe +HGLjmsoh0erHfzMV4wA/9Zq86WzuJS1HxXR6OYDC3/aQX7CxYT1MQxEw/PObnHtk +l3PRMWdTW7fSQtulEXzpr2/JCev6Mfc8Uy0aD3jng9byVk9GpdNFEjGgaUqjqyZo +svwAZ4/dmRjmMEibXeNUGC8HeWC3WOVV8L/DiA+miJlwPvwPiA1ZuKBI5A8VF0rN +HW7QVsG8kQ+PDHgRdsmhpzSRgykN1PgK6UxScKX8LqNKCtKpuEPApka7FQ1u4BoZ +KjjpBhY1R4TpfFkMIe7qW8XfqoaP99pED3xXch2zFRNHitNJr+yQJH4z/o+2UvnT +A2niUTHlFSCBoU1MvSq1N2J3qU6oR2cOYJ4ZxqWyCoeQR1x8aPnLlcn4le6HU7To +cYbHaImcIt7qnG4Ni0OWP4giEhjOpgxtrWgl36mdufvriwya+EHXzn36EvQ9O+bm +3fyarsnhPe01rlsRxqBiK1JOw/g4GnpX8iLGEX1V +=kRV1 +-----END PGP PUBLIC KEY BLOCK----- diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index d6f90f63..24925dbb 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -269,6 +269,37 @@ do_bootstrap[vardeps] += " \ do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config" +do_localrepo[lockfiles] = "${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}/isar.lock" +do_localrepo[depends] = "base-apt:do_cache isar-apt:do_cache_config" +do_localrepo() { + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + debrepo_args="$debrepo_args --no-check-gpg" + fi + if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + arch_param="--arch=${DISTRO_ARCH}" + else + arch_param="--arch=${HOST_ARCH}" + fi + ${SCRIPTSDIR}/debrepo --init \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + --repodir="${REPO_BASE_DIR}" \ + --repodbdir="${REPO_BASE_DB_DIR}" \ + --mirror="${@get_distro_source(d)}" \ + ${arch_param} \ + --distro="${BASE_DISTRO}" \ + --codename="${BASE_DISTRO_CODENAME}" \ + --components="main,contrib,non-free" \ + ${debrepo_args} \ + gnupg locales + #deb_dl_dir_export "${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${BOOTSTRAP_DISTRO}" +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then @@ -291,6 +322,7 @@ do_bootstrap() { sudo rm -rf --one-file-system "${ROOTFSDIR}" deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_DISTRO}" + debootstrap_args="$debootstrap_args --no-check-gpg" sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then @@ -301,7 +333,7 @@ do_bootstrap() { ${@get_distro_components_argument(d)} \ "${@get_distro_suite(d)}" \ "${ROOTFSDIR}" \ - "${@get_distro_source(d)}" \ + "file://${REPO_BASE_DIR}/${BASE_DISTRO}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} # Install apt config @@ -393,6 +425,7 @@ EOSUDO } addtask bootstrap before do_build after do_generate_keyrings +addtask localrepo before do_bootstrap after do_generate_keyrings SSTATETASKS += "do_bootstrap" SSTATECREATEFUNCS += "bootstrap_sstate_prepare" -- 2.20.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v3 3/5] meta: always use base-apt repo in local mode 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 1/5] Add debrepo python script handling base-apt Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 2/5] meta: Use cached base-apt repo to debootstrap Uladzimir Bely @ 2022-03-25 10:32 ` Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 5/5] draft: make isar-apt repo visible for base-apt Uladzimir Bely 4 siblings, 0 replies; 7+ messages in thread From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw) To: isar-users This means that only local URLs in apt sources.list* is present during the build. Any installation of packages is done from local base-apt. So, base-apt should be always mounted in *_do_mounts since now. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/buildchroot.bbclass | 10 +++----- meta/classes/rootfs.bbclass | 10 +++----- .../isar-bootstrap/isar-bootstrap.inc | 24 ++++++++----------- 3 files changed, 16 insertions(+), 28 deletions(-) diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass index dd8f4206..23e78c24 100644 --- a/meta/classes/buildchroot.bbclass +++ b/meta/classes/buildchroot.bbclass @@ -50,13 +50,9 @@ buildchroot_do_mounts() { mount --rbind /sys '${BUILDCHROOT_DIR}/sys' mount --make-rslave '${BUILDCHROOT_DIR}/sys' - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] - then - mkdir -p '${BUILDCHROOT_DIR}/base-apt' - mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \ - mount --bind '${REPO_BASE_DIR}' '${BUILDCHROOT_DIR}/base-apt' - fi + mkdir -p '${BUILDCHROOT_DIR}/base-apt' + mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \ + mount --bind '${REPO_BASE_DIR}' '${BUILDCHROOT_DIR}/base-apt' # Refresh or remove /etc/resolv.conf at this chance if [ "${@repr(bb.utils.to_boolean(d.getVar('BB_NO_NETWORK')))}" = 'True' ] diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index ba86c60a..54d1f714 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -52,13 +52,9 @@ rootfs_do_mounts() { mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt' fi - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] - then - mkdir -p '${ROOTFSDIR}/base-apt' - mountpoint -q '${ROOTFSDIR}/base-apt' || \ - mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt' - fi + mkdir -p '${ROOTFSDIR}/base-apt' + mountpoint -q '${ROOTFSDIR}/base-apt' || \ + mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt' EOSUDO } diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 24925dbb..545bd474 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -341,20 +341,16 @@ do_bootstrap() { install -v -m644 "${APTPREFS}" \ "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then - line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" - if [ -z "${BASE_REPO_KEY}" ]; then - line="[trusted=yes] ${line}" - fi - echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" - echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" - - mkdir -p ${ROOTFSDIR}/base-apt - mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt - else - install -v -m644 "${APTSRCS}" \ - "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" - fi + + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + [ -z "${BASE_REPO_KEY}" ] && line="[trusted=yes] ${line}" + + echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + + mkdir -p ${ROOTFSDIR}/base-apt + mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt + install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list" rm -f "${ROOTFSDIR}/etc/apt/sources.list" rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* -- 2.20.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install. 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely ` (2 preceding siblings ...) 2022-03-25 10:32 ` [PATCH v3 3/5] meta: always use base-apt repo in local mode Uladzimir Bely @ 2022-03-25 10:32 ` Uladzimir Bely 2022-03-30 13:41 ` Moessbauer, Felix 2022-03-25 10:32 ` [PATCH v3 5/5] draft: make isar-apt repo visible for base-apt Uladzimir Bely 4 siblings, 1 reply; 7+ messages in thread From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw) To: isar-users This patch uses debrepo script to predownload packages to base-apt repository before they are installed in buildchroot/rootfs. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg-base.bbclass | 8 ++++++++ meta/classes/dpkg-gbp.bbclass | 7 +++++++ meta/classes/dpkg.bbclass | 7 +++++++ meta/classes/image-locales-extension.bbclass | 7 +++++++ meta/classes/image-tools-extension.bbclass | 7 +++++++ meta/classes/rootfs.bbclass | 20 ++++++++++++++++++++ 6 files changed, 56 insertions(+) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 86933c57..6106ca01 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -102,8 +102,16 @@ python() { } do_apt_fetch() { + ${SCRIPTSDIR}/debrepo \ + --srcmode \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + ${SRC_APT} dpkg_do_mounts E="${@ isar_export_proxies(d)}" + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ -o Dir::Etc::SourceParts="-" \ diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass index d956e8c3..4d583a79 100644 --- a/meta/classes/dpkg-gbp.bbclass +++ b/meta/classes/dpkg-gbp.bbclass @@ -13,7 +13,14 @@ GBP_DEPENDS ?= "git-buildpackage pristine-tar" GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" do_install_builddeps_append() { + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + ${GBP_DEPENDS} dpkg_do_mounts + sudo -E chroot '${BUILDCHROOT_DIR}' /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" distro="${DISTRO}" if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_DISTRO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 320102ba..d167a01d 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -7,7 +7,14 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}" # Install build dependencies for package do_install_builddeps() { + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + --controlfile="${WORKDIR}/${PPS}/debian/control" dpkg_do_mounts + sudo -E chroot '${BUILDCHROOT_DIR}' /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" E="${@ isar_export_proxies(d)}" export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}" export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}" diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 25af5407..a05f0917 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -28,6 +28,13 @@ def get_nopurge(d): ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download() { + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + localepurge + sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index b9968139..359991e3 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -22,10 +22,17 @@ do_install_imager_deps() { exit fi + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + ${IMAGER_INSTALL} buildchroot_do_mounts E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO} + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ apt-get update \ -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 54d1f714..4019b216 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -94,6 +94,23 @@ EOF EOSUDO } +ROOTFS_INSTALL_COMMAND += "rootfs_base_apt_populate" +rootfs_base_apt_populate[weight] = "600" +rootfs_base_apt_populate() { + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + ${ROOTFS_PACKAGES} +} + +ROOTFS_INSTALL_COMMAND += "rootfs_base_apt_pkgs_update" +rootfs_base_apt_pkgs_update[weight] = "5" +rootfs_base_apt_pkgs_update() { + sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ + -o Dir::Etc::SourceParts="-" \ + -o APT::Get::List-Cleanup="0" +} + ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_apt" rootfs_configure_apt[weight] = "2" rootfs_configure_apt() { @@ -139,6 +156,9 @@ ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" rootfs_install_pkgs_download[weight] = "600" rootfs_install_pkgs_download[isar-apt-lock] = "release-after" rootfs_install_pkgs_download() { + ${SCRIPTSDIR}/debrepo \ + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" \ + ${ROOTFS_PACKAGES} sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES} } -- 2.20.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install. 2022-03-25 10:32 ` [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install Uladzimir Bely @ 2022-03-30 13:41 ` Moessbauer, Felix 0 siblings, 0 replies; 7+ messages in thread From: Moessbauer, Felix @ 2022-03-30 13:41 UTC (permalink / raw) To: Uladzimir Bely; +Cc: isar-users, jan.kiszka, Schmidt, Adriaan > -----Original Message----- > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On > Behalf Of Uladzimir Bely > Sent: Friday, March 25, 2022 11:32 AM > To: isar-users@googlegroups.com > Subject: [PATCH v3 4/5] base-apt: Predownload packages to base-apt before > install. > > This patch uses debrepo script to predownload packages to base-apt repository > before they are installed in buildchroot/rootfs. > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de> > --- > meta/classes/dpkg-base.bbclass | 8 ++++++++ > meta/classes/dpkg-gbp.bbclass | 7 +++++++ > meta/classes/dpkg.bbclass | 7 +++++++ > meta/classes/image-locales-extension.bbclass | 7 +++++++ > meta/classes/image-tools-extension.bbclass | 7 +++++++ > meta/classes/rootfs.bbclass | 20 ++++++++++++++++++++ > 6 files changed, 56 insertions(+) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index 86933c57..6106ca01 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -102,8 +102,16 @@ python() { > } > > do_apt_fetch() { > + ${SCRIPTSDIR}/debrepo \ > + --srcmode \ > + --workdir="${TMPDIR}/debrepo/${BASE_DISTRO}- > ${BASE_DISTRO_CODENAME}" \ > + ${SRC_APT} > dpkg_do_mounts > E="${@ isar_export_proxies(d)}" > + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ > + -o Dir::Etc::SourceList="sources.list.d/base-apt.list" \ > + -o Dir::Etc::SourceParts="-" \ > + -o APT::Get::List-Cleanup="0" Hi, This looks like a lot of code duplication. We already have a bunch of similar calls to apt spread across ISAR. Just to name a few: - buildchroot/files/deps.sh - deb-dl-dir.bbclass - dpkg-base.bbclass - dpkg-gbp.bbclass - image-locales-extension.bbclass - image-tools-extension.bbclass - rootfs.bbclass - isar-bootstrap.inc With sbuilder the situation will get even worse, as the do_install_builddeps task is no longer available. By that, users that have to prepare their sources in advance to building will manually have to fiddle around with apt to install pre-build dependencies. Affected OSS layers I know of are meta-coral (dpkg-bazel.bbclass) and meta-iot2050 (npm.bbclass). Maybe we should re-think our apt handling and add a proper infrastructure. By that, we can handle all options centrally and give the user an easy to use interface to install a package. Also error prone aspects like mounts and locks should be handled by that infrastructure. Finally, this would significantly reduce future maintenance effort in downstream layers. Felix -- Siemens AG, Technology, T CED SES-DE Otto-Hahn-Ring 6, 81739 München, Germany ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v3 5/5] draft: make isar-apt repo visible for base-apt 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely ` (3 preceding siblings ...) 2022-03-25 10:32 ` [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install Uladzimir Bely @ 2022-03-25 10:32 ` Uladzimir Bely 4 siblings, 0 replies; 7+ messages in thread From: Uladzimir Bely @ 2022-03-25 10:32 UTC (permalink / raw) To: isar-users some packages in isar-apt (like sshd-regen-keys) may depend on debian packages (openssh-server). So, base-apt need access to isar-apt repo to be able to download these deps. TODO: make it work in cross-build mode --- scripts/debrepo | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/debrepo b/scripts/debrepo index fba8342e..f6e95cfe 100755 --- a/scripts/debrepo +++ b/scripts/debrepo @@ -91,6 +91,8 @@ class DebRepo(object): f = open(self.workdir + "/etc/apt/sources.list", "w") f.write("deb [arch=" + self.arch + "] " + self.mirror + " " + self.codename + " " + self.components + "\n") f.write("deb-src [arch=" + self.arch + "] " + self.mirror + " " + self.codename + " " + self.components + "\n") + f.write("deb [trusted=yes] file://" + self.workdir + "/../../deploy/isar-apt/" + self.distro + "-" + self.codename + "-" + self.arch + "/apt/" + self.distro + "-" + self.codename + " isar main\n") + f.flush f.close -- 2.20.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-03-30 13:41 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-25 10:32 [PATCH v3 0/5] Improving base-apt usage PoC Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 1/5] Add debrepo python script handling base-apt Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 2/5] meta: Use cached base-apt repo to debootstrap Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 3/5] meta: always use base-apt repo in local mode Uladzimir Bely 2022-03-25 10:32 ` [PATCH v3 4/5] base-apt: Predownload packages to base-apt before install Uladzimir Bely 2022-03-30 13:41 ` Moessbauer, Felix 2022-03-25 10:32 ` [PATCH v3 5/5] draft: make isar-apt repo visible for base-apt Uladzimir Bely
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox