* [PATCH v6 00/11] Improving base-apt usage
@ 2024-03-14 7:27 Uladzimir Bely
2024-03-14 7:27 ` [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt Uladzimir Bely
` (10 more replies)
0 siblings, 11 replies; 12+ messages in thread
From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw)
To: isar-users
Currently, base-apt is used in the following way:
- After the first build, all .deb files that took part in the process
are is cached in the ${DL_DIR}/deb/. All the packages are downloaded
from the remote repositories.
- At the second build, `base-apt` repo is first created from the
previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set).
Futher debootstrap and installing packages are done from this
local repo.
The idea of this patchset is to use local `base-apt` repo even during
the first build. Before running any task that requires additional
packages, these packages are predownloaded using `debrepo` script to
the `base-apt` repo and the debootstrapping / installation steps are
always done from it.
Such approach allows to stop using deb-del-dir import/export
functionality in favour of using local 'base-apt' repository.
New approach requires 'apt-get update' to be run after every call of
`debrepo` script. It differs from the old approach, when it was run
only once after debootstrapping and Debian package database considered
unchanged during the build.
But, potentially, new approach in combination with the patchset
"Introduce Debian dependencies and provides helper", could allow to
prepopulate 'base-apt' repo before running any bootstrap/build tasks,
even at the first "clean" build.
Currently, separate `debrepo` script is used for prefetching packages
to the local 'base-apt' repo. It requires python3-apt to be installed
on the build host. Potentially, the functionality it provides could be
directly integrated to the `debrepo.bbclass`.
Some examples of using `debrepo` script in standalone mode:
```
# Create local `/build/ba` repository sufficient to debootstrap Debian
# base system with `armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg
# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```
This patchset v6 can be considered as the first "non-POC" version since
it was heavily tested in CI and looks stable enough.
Known issues:
- May stuck when 'BASE_REPO_FEATURES = "cache-deb-src"' set.
Changes since v5:
- Rebased on latest next.
- Changed order of the patches.
- Fixes in `debrepo` script that allow to use it outside Isar in
'standalone' mode.
Changes since v4:
- Rebased on latest next.
- Rearranged patches since some of them are already in next.
- Added possibility to select between new "prefetch" base-apt mode and
old behaviour when it's populated on 2nd build with packages downloaded
during 1st build. New behaviour is disabled by default, but enabled in
local.conf.example for testing purposes.
- Code passes both full and fast CI in both "old" and "new" modes.
Changes since v3:
- Rebased on latest next.
- Cross-building for raspberry supported.
- Code passes both full and fast CI.
Changes since v2:
- Populate base-apt before using at all steps of native build.
Changes since v1:
- Rebased on latest next.
- Updated patchset description.
Uladzimir Bely (11):
scripts: Add debrepo python script handling base-apt
meta: Add debrepo bbclass handling base-apt prefetching
meta: Always use base-apt repo in local mode
meta: Use cached base-apt repo to debootstrap
meta: Consider global debrepo context
base-apt: Predownload packages to base-apt before install
meta: Add cache-deb-src functionality in base-apt mode
testsuite: Set ISAR_PREFETCH_BASE_APT by default
Disable deb-dl-dir in base-apt prefetch mode
kas: Add PREFETCH_BASE_APT config entry
ci_build.sh: Install python3-apt if not installed
RECIPE-API-CHANGELOG.md | 9 +
kas/opt/Kconfig | 13 +
kas/opt/prefetch-base-apt.yaml | 9 +
meta-test/conf/local.conf.sample | 3 +
meta/classes/deb-dl-dir.bbclass | 19 +
meta/classes/debrepo.bbclass | 81 +++
meta/classes/dpkg-base.bbclass | 38 +-
meta/classes/dpkg.bbclass | 8 +
meta/classes/image-locales-extension.bbclass | 5 +
meta/classes/image-tools-extension.bbclass | 9 +
meta/classes/rootfs.bbclass | 12 +-
meta/conf/bitbake.conf | 5 +
.../isar-bootstrap/isar-bootstrap.inc | 106 +++-
meta/recipes-devtools/base-apt/base-apt.bb | 21 +-
.../sbuild-chroot/sbuild-chroot.inc | 11 +
scripts/ci_build.sh | 8 +-
scripts/debrepo | 571 ++++++++++++++++++
testsuite/cibase.py | 4 +
testsuite/cibuilder.py | 8 +-
19 files changed, 916 insertions(+), 24 deletions(-)
create mode 100644 kas/opt/prefetch-base-apt.yaml
create mode 100644 meta/classes/debrepo.bbclass
create mode 100755 scripts/debrepo
--
2.43.0
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching Uladzimir Bely ` (9 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This is the main utility responsible for prefetching packages into local `base-apt` repo from external Debian mirrors. It uses python-apt module and requires some kind of minimal `rootfs` to work (let's call it "debrepo context"). Once initialized with `--init --workdir=<path>`, it stores the initial configuration in `repo.opts` file inside the context and uses it at futher calls. In future, the logic `debrepo` script implements could be directly implemented inside bitbake classes. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- scripts/debrepo | 571 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 571 insertions(+) create mode 100755 scripts/debrepo diff --git a/scripts/debrepo b/scripts/debrepo new file mode 100755 index 00000000..e089cdb1 --- /dev/null +++ b/scripts/debrepo @@ -0,0 +1,571 @@ +#!/usr/bin/env python3 + +""" +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +# debrepo: build Debian-like repo using "python3-apt" library. + +When building the image, Isar downloads required Debian packages from external +mirrors. After build completed, it can pick all downloaded packages from DL_DIR +and build local 'base-apt' Debian-like repo from them. + +This tool allows to download packages and create local repo in advance. So, +Isar just uses this local repository and does not interact with external +mirrors. Such approach makes deb-dl import/export functionality redundant. + +Script `debrepo` works in so-called "context" directory. It means some +rootfs-like directory with bare minimum of directories/files required for +python3-apt to work. + +Context directory path is passed with "--workdir <dir>" command-line option. +On context creating, all passed parameters are stored in the context directory +and picked every time the context is used again. + +1. Repo for building Debian system +``` +debrepo --workdir=d12 --init locales gnupg +``` +Initialize the context in "d12" directory and create a repository in +"d12/repo/apt" directory sufficient to debootstrap default system +(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg" +with all their dependencies will be available in this repo. + +``` +debrepo --workdir=d12 docbook-to-man +``` +Adds "docbook-to-man" packages with its dependencies to earlier created repo. + +``` +debrepo --workdir=d12 --srcmode docbook-to-man +``` +Downloads source package for "docbook-to-man" and adds it to the repo + +2. Repo for building Ubuntu system +``` +debrepo --init --workdir=uf \ +--distro=ubuntu --codename=focal --arch=arm64 \ +--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \ +--repodir=repo/apt --repodbdir=repo/db \ +--mirror=http://ports.ubuntu.com/ubuntu-ports \ +locales gnupg +``` +Initialize the context in "uf" directory and create a repository in "repo/apt" +directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and +source list are specified by corresponding arguments. Packages "locales" and +" gnupg" with the dependencies will be also placed to the repo. + +``` +debrepo --workdir=uf gnupg,locales +``` +Add "gnupg" and "locales" packages with their dependencies to earlier created +ubuntu repo. Other parameters (distro, codename, arch) are ommited since they +are picked from the context. + +3. Repo for cross-building Debian system +``` +debrepo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf +``` +Initialize the context in "d11" directory sufficient to deboostrap Debian +Bullseye (amd64) with foreign "armhf" architecture support + +``` +debrepo --workdir=d11 gcc +``` +Add "gcc" package (amd64 version) to earlier created repo. + +``` +debrepo --workdir=d11 --crossbuild gcc +``` +Add "gcc" package (armhf version) to earlier created repo. +""" + +import os +import sys +import fcntl + +import argparse +import shutil +import subprocess +import pickle +import urllib.parse + +import apt_pkg +import apt.progress.base + + +class DebRepo(object): + class DebRepoCtx(object): + def __init__(self, workdir): + self.distro = "debian" + self.codename = "bullseye" + self.arch = "amd64" + self.mirror = "http://deb.debian.org/debian" + + self.repodir = f"{workdir}/repo/apt" + self.repodbdir = f"{workdir}/repo/db" + + self.crossarch = self.arch + self.compatarch = None + self.keydir = "/etc/apt/trusted.gpg.d" + + def __init__(self, args): + self.workdir = os.path.abspath(args.workdir) + os.makedirs(self.workdir, exist_ok=True) + self.ctx = self.DebRepoCtx(self.workdir) + + self.cache = None + self.depcache = None + self.sr = None + self.extrarepo = None + + self.ctx_load() + self.ctx_update(args) + self.ctx_save() + + print( + f"ctx workdir: {self.workdir}\n" + f" distro: {self.ctx.distro}\n" + f" codename: {self.ctx.codename}\n" + f" arch: {self.ctx.arch}\n" + f" mirror: {self.ctx.mirror}\n" + f" repodir: {self.ctx.repodir}\n" + f" repodbdir: {self.ctx.repodbdir}\n" + f" crossarch: {self.ctx.crossarch}\n" + f" compatarch: {self.ctx.compatarch}\n" + f" keydir: {self.ctx.keydir}" + ) + + if args.extrarepo: + self.extrarepo = os.path.abspath(args.extrarepo) + + def ctx_load(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + if os.path.isfile(ctxfile): + with open(ctxfile, 'rb') as f: + self.ctx = pickle.load(f) + + def ctx_save(self): + ctxfile = f"{self.workdir}/debrepo.ctx" + + with open(ctxfile, 'wb') as f: + pickle.dump(self.ctx, f) + + def ctx_update(self, args): + if args.distro: + self.ctx.distro = args.distro + if args.codename: + self.ctx.codename = args.codename + if args.arch: + self.ctx.arch = args.arch + if args.mirror: + self.ctx.mirror = args.mirror + + if args.repodir: + self.ctx.repodir = os.path.abspath(args.repodir) + if args.repodbdir: + self.ctx.repodbdir = os.path.abspath(args.repodbdir) + + if args.crossarch: + self.ctx.crossarch = args.crossarch + if args.compatarch: + self.ctx.compatarch = args.compatarch + if args.keydir: + self.ctx.keydir = args.keydir + + def create_rootfs(self, aptsrcsfile): + os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True) + with open(f"{self.workdir}/var/lib/dpkg/status", "w"): + pass + + os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True) + + srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list" + if aptsrcsfile and os.path.exists(aptsrcsfile): + shutil.copy(aptsrcsfile, srcfile) + else: + with open(srcfile, "w") as f: + repo = f"{self.ctx.mirror} {self.ctx.codename} main" + f.write(f"deb {repo}\n") + f.write(f"deb-src {repo}\n") + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial", + exist_ok=True) + + os.makedirs(f"{self.workdir}/tmp", exist_ok=True) + + def create_repo_dist(self): + conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf" + os.makedirs(conf_dir, exist_ok=True) + if not os.path.exists(f"{conf_dir}/distributions"): + with open(f"{conf_dir}/distributions", "w") as f: + f.write(f"Codename: {self.ctx.codename}\n") + f.write( + "Architectures: " + "i386 armhf arm64 amd64 mipsel riscv64 source\n") + f.write("Components: main\n") + + def apt_config(self, init, crossbuild): + # Configure apt to work with empty directory + if not init and self.ctx.arch != self.ctx.crossarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch + apt_pkg.config["APT::Architectures::"] = self.ctx.arch + + if not init and self.ctx.compatarch: + apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch + + apt_pkg.config.set("APT::Architecture", self.ctx.arch) + + apt_pkg.config.set("Dir", self.workdir) + + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}") + apt_pkg.config.set("Dir::State::status", + f"{self.workdir}/var/lib/dpkg/status") + + apt_pkg.config.set("APT::Install-Recommends", "0") + apt_pkg.config.set("APT::Install-Suggests", "0") + + # Use host keys for authentification + apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir) + + # Allow using repositories without keys + apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1") + + def mark_essential(self): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + if pkg.essential: + self.depcache.mark_install(pkg) + + def mark_by_prio(self, priority): + for pkg in self.cache.packages: + if pkg.architecture == self.ctx.arch: + ver = self.depcache.get_candidate_ver(pkg) + if ver and ver.priority <= priority: + self.depcache.mark_install(pkg) + + def mark_pkg(self, name, crossbuild): + pkgname = name + + if pkgname and (pkgname not in self.cache): + # Try for cross arch + if (pkgname, self.ctx.crossarch) in self.cache: + pkgname += f":{self.ctx.crossarch}" + + if pkgname not in self.cache: + print(f"Error: package '{name}' not found") + return False + + pkg = self.cache[pkgname] + + if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions): + if (pkg.has_provides) and (not pkg.has_versions): + print("pkgname is virtual package, selecting best provide") + # Select first provide + pkg_provide = pkg.provides_list[0][2] + # Find better provide with higher version + for provide in pkg.provides_list: + if apt_pkg.version_compare(provide[2].ver_str, + pkg_provide.ver_str) > 0: + pkg_provide = provide[2] + self.depcache.mark_install(pkg_provide.parent_pkg) + else: + self.depcache.mark_install(pkg) + else: + version = pkg.version_list[0] + if version.arch == "all": + self.depcache.mark_install(pkg) + else: + if version.multi_arch == version.MULTI_ARCH_FOREIGN: + if (pkgname, self.ctx.arch) in self.cache: + nativepkg = self.cache[pkgname, self.ctx.arch] + self.depcache.mark_install(nativepkg) + else: + return False + else: + if (pkgname, self.ctx.crossarch) in self.cache: + crosspkg = self.cache[pkgname, self.ctx.crossarch] + self.depcache.mark_install(crosspkg) + else: + return False + + return True + + def mark_list(self, pkglist, crossbuild): + ret = True + if pkglist: + for pkgname in pkglist: + ret = ret and self.mark_pkg(pkgname, crossbuild) + + return ret + + def handle_deb(self, item): + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "includedeb", + self.ctx.codename, + item.destfile + ], + stdout=subprocess.PIPE) + fd.close() + + def handle_repo(self, fetcher): + dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}" + fd = open(f"{self.workdir}/{dir_cache}.lock", "w") + fcntl.flock(fd, fcntl.LOCK_EX) + fetcher.run() + fd.close() + for item in fetcher.items: + if item.status == item.STAT_ERROR: + print("Some error ocured: '%s'" % item.error_text) + pass + else: + self.handle_deb(item) + + def get_filename(self, uri): + path = urllib.parse.urlparse(uri).path + unquoted_path = urllib.parse.unquote(path) + basename = os.path.basename(unquoted_path) + return basename + + def fetch_file(self, uri): + filename = self.get_filename(uri) + subprocess.run([ + "wget", + "-H", + "--timeout=30", + "--tries=3", + "-q", + uri, + "-O", + f"{self.workdir}/tmp/{filename}" + ], + stdout=subprocess.PIPE) + + def handle_dsc(self, uri): + filename = self.get_filename(uri) + fd = open(f"{self.ctx.repodir}/repo.lock", 'w') + fcntl.flock(fd, fcntl.LOCK_EX) + subprocess.run([ + "reprepro", + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}", + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}", + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf", + "-C", "main", + "-S", "-", "-P" "source", + "--delete", + "includedsc", + self.ctx.codename, + os.path.realpath(f"{self.workdir}/tmp/{filename}") + ], + stdout=subprocess.PIPE) + fd.close() + + def handle_src_list(self, pkgs): + if pkgs: + for pkg in pkgs: + pkgname = pkg + pkgver = "" + if '=' in pkg: + pkgname = pkg.split("=")[0] + pkgver = pkg.split("=")[1] + + self.sr.restart() + while self.sr.lookup(pkgname): + if pkgver and pkgver != self.sr.version: + continue + + for sr_file in self.sr.files: + print(self.sr.index.archive_uri(sr_file[2])) + self.fetch_file(self.sr.index.archive_uri(sr_file[2])) + + dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2]) + self.handle_dsc(dsc_uri) + break + + def apt_run(self, init, srcmode, pkgs, controlfile, crossbuild): + apt_pkg.init() + + if self.extrarepo: + srcfile = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list" + with open(srcfile, "w") as f: + distdir=os.path.join(self.extrarepo, "dists") + if os.path.isdir(distdir): + for dist in os.listdir(distdir): + repodir = os.path.join(distdir,dist) + if os.path.isdir(repodir): + for repo in os.listdir(repodir): + if os.path.isdir(os.path.join(repodir, repo)): + f.write(f"deb file://{self.extrarepo} " + f"{dist} {repo}\n") + + sources = apt_pkg.SourceList() + sources.read_main_list() + + progress = apt.progress.text.AcquireProgress() + + self.cache = apt_pkg.Cache() + self.cache.update(progress, sources) + self.cache = apt_pkg.Cache() + + if self.extrarepo: + os.remove(f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list") + + self.depcache = apt_pkg.DepCache(self.cache) + self.sr = apt_pkg.SourceRecords() + + ret = True + + if init: + self.mark_essential() + # 1(required), 2(important), 3(standard), 4(optional), 5(extra) + self.mark_by_prio(1) + + pkgs = list(filter(None, ','.join(pkgs).split(','))) + if srcmode: + self.handle_src_list(pkgs) + else: + ret = self.mark_list(pkgs, crossbuild) + + if controlfile: + fobj = open(controlfile, "r") + + try: + tagfile = apt_pkg.TagFile(fobj) + while tagfile.step() == 1: + deps = tagfile.section.get("Build-Depends", "") + # Remove extra commas and spaces - apt_pkg.parse_depends + # doesnt like lines like ", device-tree-compiler" + deps = ', '.join( + [s.strip() for s in deps.split(',') if s.strip()] + ) + print(f"parsed deps: {deps}") + for item in apt_pkg.parse_depends(deps, False): + pkgname = item[0][0] + self.mark_pkg(pkgname, crossbuild) + + finally: + fobj.close() + + if not ret: + sys.exit("Some of requested packages not found") + + if init or not srcmode: + fetcher = apt_pkg.Acquire(progress) + pm = apt_pkg.PackageManager(self.depcache) + + recs = apt_pkg.PackageRecords(self.cache) + pm.get_archives(fetcher, sources, recs) + + self.handle_repo(fetcher) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--init", + default=False, action="store_true", + help="initialize context in WORKDIR") + parser.add_argument( + "--workdir", + type=str, required=True, + help="work directory storing debrepo context") + parser.add_argument( + "--aptsrcsfile", + type=str, metavar="PATH", + help="sources.list file to use when init") + parser.add_argument( + "--srcmode", + default=False, action="store_true", + help="add source packages instead of debs") + parser.add_argument( + "--repodir", + type=str, metavar="REPO", + help="repository directory") + parser.add_argument( + "--repodbdir", + type=str, metavar="REPODB", + help="repository database directory") + parser.add_argument( + "--extrarepo", + type=str, metavar="REPO", + help="extra repository to consider") + parser.add_argument( + "--mirror", + type=str, + help="use custom distro mirror") + parser.add_argument( + "--distro", + type=str, + help="select distro to use") + parser.add_argument( + "--codename", + type=str, + help="distro codename") + parser.add_argument( + "--arch", + type=str, + help="distro arch") + parser.add_argument( + "--compatarch", + type=str, metavar="ARCH", + help="compat arch to use") + parser.add_argument( + "--crossarch", + type=str, metavar="ARCH", + help="cross-build arch") + parser.add_argument( + "--keydir", + type=str, + help="directory with distro keys") + parser.add_argument( + "--no-check-gpg", + default=False, action="store_true", + help="allow insecure repositories") + parser.add_argument( + "--controlfile", + type=str, metavar="PATH", + help="debian control file to parse") + parser.add_argument( + "--crossbuild", + default=False, action="store_true", + help="add packages with cross arch") + + parser.add_argument( + "packages", + nargs='*', type=str, + help="space- or comma-separated list of packages to add") + + args = parser.parse_args() + + return args + + +def main(): + args = parse_arguments() + + if not (args.init or args.packages or args.controlfile): + sys.exit("Nothing to do") + + debrepo = DebRepo(args) + + if args.init: + debrepo.create_rootfs(args.aptsrcsfile) + debrepo.create_repo_dist() + + debrepo.apt_config(args.init, args.crossbuild) + debrepo.apt_run(args.init, args.srcmode, args.packages, + args.controlfile, args.crossbuild) + + +if __name__ == "__main__": + main() -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 03/11] meta: Always use base-apt repo in local mode Uladzimir Bely ` (8 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- RECIPE-API-CHANGELOG.md | 9 ++++ meta/classes/debrepo.bbclass | 81 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 +++ 3 files changed, 95 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 2da7968d..cb466eb7 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -572,3 +572,12 @@ the module build recipe. Remove all uses of the function deb_compat. The functionality was replaced with a dependency to the package debhelper-compat. +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..6c135ef2 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,81 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${workdir}\" \ + ${args} + " +} + +debrepo_handle_controlfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + control_file="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${DEBREPO_WORKDIR}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo \ + --workdir=\"${DEBREPO_WORKDIR}\" \ + --controlfile=\"${control_file}\" \ + ${args} + " +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 91c5c815..1bff8c8e 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable" -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 03/11] meta: Always use base-apt repo in local mode 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap Uladzimir Bely ` (7 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This means only local URLs in apt sources.list* are present during the build. Any installation of packages is done from local base-apt. So, base-apt should be always mounted in *_do_mounts since now. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/rootfs.bbclass | 5 +++-- .../isar-bootstrap/isar-bootstrap.inc | 12 +++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++------- 3 files changed, 25 insertions(+), 13 deletions(-) diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 498fbfd6..ddc5d12c 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -54,8 +54,9 @@ rootfs_do_mounts() { mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt' fi - # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set - if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] + # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set + if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \ + [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ] then mkdir -p '${ROOTFSDIR}/base-apt' mountpoint -q '${ROOTFSDIR}/base-apt' || \ diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 17f19fd8..05c80308 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -42,7 +42,8 @@ python () { # installation afterwards. However, debootstrap will include the key into # the rootfs automatically thus the right place is distro_bootstrap_keys. - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : own_pub_key = d.getVar("BASE_REPO_KEY") if own_pub_key: distro_bootstrap_keys += own_pub_key.split() @@ -110,7 +111,8 @@ def parse_aptsources_list_line(source_list_line): def get_apt_source_mirror(d, aptsources_entry_list): import re - if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \ + bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) : premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n" else: premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or "" @@ -283,7 +285,8 @@ do_bootstrap() { if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" fi - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then debootstrap_args="$debootstrap_args --no-check-gpg" fi E="${@ isar_export_proxies(d)}" @@ -314,7 +317,8 @@ do_bootstrap() { install -v -m644 "${APTPREFS}" \ "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" - if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \ + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" if [ -z "${BASE_REPO_KEY}" ]; then line="[trusted=yes] ${line}" diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index ea885fe6..e62958d1 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -57,9 +57,12 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" + fi if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ @@ -67,14 +70,18 @@ repo() { "${BASE_DISTRO_CODENAME}" \ "${WORKDIR}/distributions.in" \ "${KEYFILES}" - populate_base_apt "${HOST_BASE_DISTRO}" - repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \ + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then + populate_base_apt "${HOST_BASE_DISTRO}" + repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}" + fi fi } python do_cache() { - if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \ + not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): return 0 for key in d.getVar('BASE_REPO_KEY').split(): -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (2 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 03/11] meta: Always use base-apt repo in local mode Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 05/11] meta: Consider global debrepo context Uladzimir Bely ` (6 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This patch makes local base-apt repo to be created before debootstrap task. So, debootstrap is then done from it. The required packages are downloaded via python-apt and reprepro creates debian-like repository from .deb files. For debian targets host keyring is used while ubuntu/raspbian targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable. The goal is have workable base-apt repo before first build completed. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- .../isar-bootstrap/isar-bootstrap.inc | 94 +++++++++++++++++-- 1 file changed, 86 insertions(+), 8 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 05c80308..e5d2523d 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -29,6 +29,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}" + +# For newer distros "usr-is-merged" indirectly required by debootstrap +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged" + FILESEXTRAPATHS:append = ":${BBPATH}" inherit deb-dl-dir @@ -275,12 +282,63 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" inherit compat +inherit debrepo + +debrepo_bootstrap_prepare() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + debrepo_args="" + if [ "${BASE_DISTRO}" != "debian" ]; then + if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + else + if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then + debrepo_args="$debrepo_args --keydir=${WORKDIR}" + fi + fi + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}" + fi + + debrepo_workdir=${DEBREPO_TARGET_DIR} + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi + fi + mkdir -p "${debrepo_workdir}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + fi + + flock -x "${debrepo_workdir}/repo.lock" -c " + ${SCRIPTSDIR}/debrepo --init \ + --workdir=\"${debrepo_workdir}\" \ + --aptsrcsfile=\"${APTSRCS_INIT}\" \ + --repodir=\"${REPO_BASE_DIR}\" \ + --repodbdir=\"${REPO_BASE_DB_DIR}\" \ + --mirror=\"${@get_distro_source(d)}\" \ + --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \ + --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \ + --codename=\"${BASE_DISTRO_CODENAME}\" \ + ${debrepo_args} \ + ${DISTRO_BOOTSTRAP_BASE_PACKAGES} + " +} + do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then if [ -z "${COMPAT_DISTRO_ARCH}" ]; then bbfatal "${DISTRO_ARCH} does not have a compat arch" fi fi + + debrepo_bootstrap_prepare + debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" @@ -302,8 +360,19 @@ do_bootstrap() { sudo -E -s <<'EOSUDO' set -e if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then - arch_param="--arch=${DISTRO_ARCH}" + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}" fi + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + ${DEBOOTSTRAP} $debootstrap_args \ + $arch_param \ + ${@get_distro_components_argument(d)} \ + ${@get_distro_suite(d)} \ + ${ROOTFSDIR} \ + file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \ + ${DISTRO_DEBOOTSTRAP_SCRIPT} + " + else ${DEBOOTSTRAP} $debootstrap_args \ $arch_param \ ${@get_distro_components_argument(d)} \ @@ -311,7 +380,7 @@ do_bootstrap() { "${ROOTFSDIR}" \ "${@get_distro_source(d)}" \ ${DISTRO_DEBOOTSTRAP_SCRIPT} - + fi # Install apt config mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" install -v -m644 "${APTPREFS}" \ @@ -383,12 +452,21 @@ do_bootstrap() { if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} fi - - chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ - -o APT::Update::Error-Mode=any - chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f - chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ - -o Debug::pkgProblemResolver=yes + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + " + else + chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f + chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + fi umount -l "${ROOTFSDIR}/dev/shm" umount -l "${ROOTFSDIR}/dev/pts" -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 05/11] meta: Consider global debrepo context 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (3 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install Uladzimir Bely ` (5 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users For host/target and cross/native modes different debrepo context (e.g., working dir) should be used. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg-base.bbclass | 11 +++++++++++ meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 11 +++++++++++ 2 files changed, 22 insertions(+) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 30caedf9..361a7e9b 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -12,6 +12,17 @@ inherit repository inherit deb-dl-dir inherit essential +python __anonymous() { + distro_arch = d.getVar('DISTRO_ARCH') + host_arch = d.getVar('HOST_ARCH') + cross = d.getVar('ISAR_CROSS_COMPILE', True) + + if cross == "0" or host_arch == distro_arch: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True)) + else: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True)) +} + DEPENDS ?= "" RPROVIDES ?= "${PROVIDES}" diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc index d794c6fd..2755eaa8 100644 --- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc +++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc @@ -8,6 +8,17 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260 PV = "1.0" +python __anonymous() { + distro_arch = d.getVar('DISTRO_ARCH') + host_arch = d.getVar('HOST_ARCH') + variant = d.getVar('SBUILD_VARIANT', True) + + if variant == "target" or host_arch == distro_arch: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True)) + else: + d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True)) +} + inherit rootfs inherit compat -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (4 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 05/11] meta: Consider global debrepo context Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 07/11] meta: Add cache-deb-src functionality in base-apt mode Uladzimir Bely ` (4 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This patch uses debrepo script to predownload packages to base-apt repository before they are installed in buildchroot/rootfs. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/dpkg-base.bbclass | 27 +++++++++++++++++++- meta/classes/dpkg.bbclass | 8 ++++++ meta/classes/image-locales-extension.bbclass | 5 ++++ meta/classes/image-tools-extension.bbclass | 9 +++++++ meta/classes/rootfs.bbclass | 4 +++ 5 files changed, 52 insertions(+), 1 deletion(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 361a7e9b..0df47529 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -11,6 +11,7 @@ inherit terminal inherit repository inherit deb-dl-dir inherit essential +inherit debrepo python __anonymous() { distro_arch = d.getVar('DISTRO_ARCH') @@ -125,6 +126,14 @@ do_apt_fetch() { trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + schroot -r -c ${session_id} -d / -u root -- \ rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt schroot -r -c ${session_id} -d / -- \ @@ -147,18 +156,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}" # Add dependency from the correct schroot: host or target do_apt_fetch[depends] += "${SCHROOT_DEP}" +# Debrepo context is created by target bootstrap, need this dependency too +do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap" do_apt_unpack() { rm -rf ${S} schroot_create_configs + session_id=$(schroot -b -c ${SBUILD_CHROOT}) + echo "Started session: ${session_id}" + schroot_cleanup() { + schroot -q -f -e -c ${session_id} > /dev/null 2>&1 schroot_delete_configs } trap 'exit 1' INT HUP QUIT TERM ALRM USR1 trap 'schroot_cleanup' EXIT - schroot -d / -c ${SBUILD_CHROOT} -- \ + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + schroot -r -c ${session_id} -d / -u root -- \ + sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" ' + " + fi + + schroot -r -c ${session_id} -d / -- \ sh -c ' set -e for uri in $2; do @@ -168,6 +190,9 @@ do_apt_unpack() { dpkg-source -x "${dscfile}" "${PPS}" done' \ my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}" + + # End chroot session + schroot -e -c ${session_id} schroot_delete_configs } do_apt_unpack[network] = "${TASK_USE_SUDO}" diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 3fa9f604..9aba205a 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -43,6 +43,7 @@ dpkg_runbuild() { export PARALLEL_MAKE="${PARALLEL_MAKE}" rm -f ${SBUILD_CONFIG} + debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control" env | while read -r line; do # Filter the same lines @@ -111,6 +112,11 @@ dpkg_runbuild() { DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog) DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print) + locked_update_cmd=":" + if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then + locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'" + fi + sbuild -A -n -c ${SBUILD_CHROOT} \ --host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \ --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \ @@ -121,9 +127,11 @@ dpkg_runbuild() { --chroot-setup-commands="rm -f /var/log/dpkg.log" \ --chroot-setup-commands="mkdir -p ${deb_dir}" \ --chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \ + --chroot-setup-commands="${locked_update_cmd}" \ --chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \ --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \ + --finished-build-commands="mkdir -p ${ext_root}" \ --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \ --debbuildopts="--source-option=-I" \ --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE} diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass index 9149d643..4dd93f1f 100644 --- a/meta/classes/image-locales-extension.bbclass +++ b/meta/classes/image-locales-extension.bbclass @@ -6,6 +6,8 @@ # This class extends the image.bbclass for setting locales and purging unneeded # ones. +inherit debrepo + LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\ en_US ISO-8859-1\n" LOCALE_DEFAULT ?= "en_US.UTF-8" @@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download" image_install_localepurge_download[weight] = "40" image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" image_install_localepurge_download() { + debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge } diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 46bdf78b..4e5bd550 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -6,6 +6,7 @@ # This file extends the image.bbclass to supply tools for futher imager functions inherit sbuild +inherit debrepo IMAGER_INSTALL ??= "" IMAGER_BUILD_DEPS ??= "" @@ -40,12 +41,20 @@ imager_run() { echo "Installing imager deps: ${local_install}" distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + debrepo_workdir=${DEBREPO_TARGET_DIR} if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then + debrepo_workdir=${DEBREPO_HOST_DIR} + fi fi E="${@ isar_export_proxies(d)}" deb_dl_dir_import ${schroot_dir} ${distro} + + debrepo_add_packages --isarapt "${debrepo_workdir}" "${IMAGER_INSTALL}" + debrepo_update_apt_source_list "${schroot_dir}" "base-apt" + schroot -r -c ${session_id} -d / -u root -- sh -c " \ apt-get update \ -o Dir::Etc::SourceList='sources.list.d/isar-apt.list' \ diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index ddc5d12c..13e06ae1 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -2,6 +2,7 @@ # Copyright (c) Siemens AG, 2020 inherit deb-dl-dir +inherit debrepo ROOTFS_ARCH ?= "${DISTRO_ARCH}" ROOTFS_DISTRO ?= "${DISTRO}" @@ -153,6 +154,9 @@ rootfs_install_pkgs_download[weight] = "600" rootfs_install_pkgs_download[isar-apt-lock] = "release-after" rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}" rootfs_install_pkgs_download() { + debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}" + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" + sudo -E chroot '${ROOTFSDIR}' \ /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES} } -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 07/11] meta: Add cache-deb-src functionality in base-apt mode 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (5 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 08/11] testsuite: Set ISAR_PREFETCH_BASE_APT by default Uladzimir Bely ` (3 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users Fill base-apt repo with source packages. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/deb-dl-dir.bbclass | 17 +++++++++++++++++ meta/classes/rootfs.bbclass | 3 +++ 2 files changed, 20 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index d36b7190..2bb05b31 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -41,6 +41,23 @@ debsrc_undo_mounts() { EOSUDO } +debsrc_fill_base_apt() { + export rootfs="$1" + + local srcpkgs="" + for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do + is_not_part_of_current_build "${package}" && continue + local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )" + local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )" + local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')" + local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc") + [ -n "$dscfile" ] && continue + + srcpkgs="${srcpkgs} ${src}=${version}" + done + debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}" +} + debsrc_download() { export rootfs="$1" export rootfs_distro="$2" diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 13e06ae1..ad4e728a 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -235,6 +235,9 @@ cache_deb_src() { sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/" deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} + + debsrc_fill_base_apt ${ROOTFSDIR} + debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt" debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME} sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 08/11] testsuite: Set ISAR_PREFETCH_BASE_APT by default 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (6 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 07/11] meta: Add cache-deb-src functionality in base-apt mode Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 09/11] Disable deb-dl-dir in base-apt prefetch mode Uladzimir Bely ` (2 subsequent siblings) 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This makes Isar use `base-apt` repo in different way. Any package installation is done from `base-apt` repo which is prepopulated from external mirrors. This behaviour is disabled by default for downstreams. To enable it, set the variable to "1", like isar does in local.conf.sample. In order to be able to run CI in old mode, allow CI read the option from the environment. Also, adjust some tests (like repro one) to make them work with ISAR_PREFETCH_BASE_APT set. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta-test/conf/local.conf.sample | 3 +++ testsuite/cibase.py | 4 ++++ testsuite/cibuilder.py | 8 +++++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample index f692f533..941d7cce 100644 --- a/meta-test/conf/local.conf.sample +++ b/meta-test/conf/local.conf.sample @@ -34,6 +34,9 @@ IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsc IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}" IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}" +# Use new base-apt behaviour +ISAR_PREFETCH_BASE_APT ?= "1" + # Users and groups USERS += "root" USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" diff --git a/testsuite/cibase.py b/testsuite/cibase.py index 90591f32..74f6c971 100755 --- a/testsuite/cibase.py +++ b/testsuite/cibase.py @@ -43,9 +43,13 @@ class CIBaseTest(CIBuilder): self.fail('GPG import failed') try: + self.move_in_build_dir('tmp', 'tmp_before_repro') self.bitbake(targets, **kwargs) self.move_in_build_dir('tmp', 'tmp_middle_repro_%s' % ('signed' if signed else 'unsigned')) + + os.makedirs(f"{self.build_dir}/tmp/deploy/") + self.move_in_build_dir('tmp_middle_repro_%s/deploy/base-apt' % ('signed' if signed else 'unsigned'), 'tmp/deploy/base-apt') self.configure(gpg_pub_key=gpg_pub_key if signed else None, offline=True, sstate_dir="", **kwargs) self.bitbake(targets, **kwargs) diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py index fa30c2f5..08f82295 100755 --- a/testsuite/cibuilder.py +++ b/testsuite/cibuilder.py @@ -104,6 +104,9 @@ class CIBuilder(Test): # get parameters from environment distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS') + # get prefetch base apt mode from environment + prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT') + self.log.info(f'===================================================\n' f'Configuring build_dir {self.build_dir}\n' f' compat_arch = {compat_arch}\n' @@ -120,6 +123,7 @@ class CIBuilder(Test): f' sstate_dir = {sstate_dir}\n' f' ccache_dir = {ccache_dir}\n' f' image_install = {image_install}\n' + f' prefetch_base_apt = {prefetch_base_apt}\n' f'===================================================') # determine bitbake_args @@ -165,7 +169,9 @@ class CIBuilder(Test): if sstate_dir: f.write('SSTATE_DIR = "%s"\n' % sstate_dir) if image_install is not None: - f.write('IMAGE_INSTALL = "%s"' % image_install) + f.write('IMAGE_INSTALL = "%s"\n' % image_install) + if prefetch_base_apt == "0": + f.write('ISAR_PREFETCH_BASE_APT = "0"\n') # include ci_build.conf in local.conf with open(self.build_dir + '/conf/local.conf', 'r+') as f: -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 09/11] Disable deb-dl-dir in base-apt prefetch mode 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (7 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 08/11] testsuite: Set ISAR_PREFETCH_BASE_APT by default Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 10/11] kas: Add PREFETCH_BASE_APT config entry Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 11/11] ci_build.sh: Install python3-apt if not installed Uladzimir Bely 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users Since all packages and source packages are placed to base-apt repo during build, there is no need to have them in one more place. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- meta/classes/deb-dl-dir.bbclass | 2 ++ 1 file changed, 2 insertions(+) diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass index 2bb05b31..4e4741be 100644 --- a/meta/classes/deb-dl-dir.bbclass +++ b/meta/classes/deb-dl-dir.bbclass @@ -89,6 +89,7 @@ deb_dl_dir_import() { export rootfs="${1}" sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ [ ! -d "${pc}" ] && return 0 + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 flock -s "${pc}".lock sudo -Es << 'EOSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x @@ -106,6 +107,7 @@ deb_dl_dir_export() { export rootfs="${1}" export owner=$(id -u):$(id -g) mkdir -p "${pc}" + [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0 flock "${pc}".lock sudo -Es << 'EOSUDO' set -e printenv | grep -q BB_VERBOSE_LOGS && set -x -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 10/11] kas: Add PREFETCH_BASE_APT config entry 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (8 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 09/11] Disable deb-dl-dir in base-apt prefetch mode Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 11/11] ci_build.sh: Install python3-apt if not installed Uladzimir Bely 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This option allows to set ISAR_PREFETCH_BASE_APT to "0" or "1" and choose between old and new base-apt behaviour. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- kas/opt/Kconfig | 13 +++++++++++++ kas/opt/prefetch-base-apt.yaml | 9 +++++++++ 2 files changed, 22 insertions(+) create mode 100644 kas/opt/prefetch-base-apt.yaml diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig index 1619a4cf..493f3ec6 100644 --- a/kas/opt/Kconfig +++ b/kas/opt/Kconfig @@ -146,3 +146,16 @@ config KAS_INCLUDE_CACHE_DEB_SRC string default "kas/opt/cache-deb-src.yaml" depends on CACHE_DEB_SRC + + +config PREFETCH_BASE_APT + bool "Prefetch base-apt repo" + default y + help + This makse Isar always take packages from base-apt repository where they + are prefetched by debrepo script before requested. + +config KAS_INCLUDE_PREFETCH_BASE_APT + string + default "kas/opt/prefetch-base-apt.yaml" + depends on PREFETCH_BASE_APT diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml new file mode 100644 index 00000000..0cbdb93f --- /dev/null +++ b/kas/opt/prefetch-base-apt.yaml @@ -0,0 +1,9 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH + +header: + version: 14 + +local_conf_header: + prefetch-base-apt: | + ISAR_PREFETCH_BASE_APT = "1" -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v6 11/11] ci_build.sh: Install python3-apt if not installed 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely ` (9 preceding siblings ...) 2024-03-14 7:27 ` [PATCH v6 10/11] kas: Add PREFETCH_BASE_APT config entry Uladzimir Bely @ 2024-03-14 7:27 ` Uladzimir Bely 10 siblings, 0 replies; 12+ messages in thread From: Uladzimir Bely @ 2024-03-14 7:27 UTC (permalink / raw) To: isar-users This is mostly related to gitlab CI that migth use an image without preinstalled python3-apt. Also, make system python packages available in virtualenv. Signed-off-by: Uladzimir Bely <ubely@ilbers.de> --- scripts/ci_build.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 51ed3920..b2fa6b9c 100755 --- a/scripts/ci_build.sh +++ b/scripts/ci_build.sh @@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then sudo apt-get update -qq sudo apt-get install -y virtualenv rm -rf /tmp/avocado_venv - virtualenv --python python3 /tmp/avocado_venv + virtualenv --python python3 /tmp/avocado_venv --system-site-packages # shellcheck source=/dev/null source /tmp/avocado_venv/bin/activate pip install avocado-framework==100.1 @@ -131,6 +131,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then fi fi +# install python3-apt +if [ ! -f /usr/share/doc/python3-apt/copyright ]; then + sudo apt-get update -qq + sudo apt-get install -y python3-apt +fi + # Provide working path mkdir -p .config/avocado cat <<EOF > .config/avocado/avocado.conf -- 2.43.0 ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2024-03-14 7:30 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-03-14 7:27 [PATCH v6 00/11] Improving base-apt usage Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 01/11] scripts: Add debrepo python script handling base-apt Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 02/11] meta: Add debrepo bbclass handling base-apt prefetching Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 03/11] meta: Always use base-apt repo in local mode Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 04/11] meta: Use cached base-apt repo to debootstrap Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 05/11] meta: Consider global debrepo context Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 06/11] base-apt: Predownload packages to base-apt before install Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 07/11] meta: Add cache-deb-src functionality in base-apt mode Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 08/11] testsuite: Set ISAR_PREFETCH_BASE_APT by default Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 09/11] Disable deb-dl-dir in base-apt prefetch mode Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 10/11] kas: Add PREFETCH_BASE_APT config entry Uladzimir Bely 2024-03-14 7:27 ` [PATCH v6 11/11] ci_build.sh: Install python3-apt if not installed Uladzimir Bely
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox