* [v10 00/12] Improving base-apt usage
@ 2026-09-04 7:52 Aliaksei Karpovich
2026-09-04 7:52 ` [v10 01/12] scripts: Add debrepo python script handling base-apt Aliaksei Karpovich
` (11 more replies)
0 siblings, 12 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Aliaksei Karpovich
`base-apt` is a local apt repository containing all upstream (Debian, Raspberry
Pi OS, Ubuntu...) packages needed for a particular build. This series implements
upfront repository downloading. This is the first step towards local partial
mirror management.
The current approach in `next`:
- On the first build, mmdebstrap and sbuild are used for building Isar
artifacts. The packages downloaded from the Internet are cached in local
directories.
- On the next build:
- Analyze the logs from the previous build, save packages downloaded by the
bootstraps, sbuilds and imagers into `base-apt`.
- Use `base-apt` for bootstrapping, building and image creation.
Some issues with the current approach:
1. Different policies must be followed for the first and the subsequent builds.
2. As we have multiple versions of the same package from the main and security
repositories and rely on build logs and `find` for populating `base-apt`, extra
care must be taken to ensure that the right package version lands in `base-apt`.
3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving
and reusing the packages.
4. Source packages are stored in a different flat directory, `apt-get source`
for upstream packages is not possible.
5. At the moment of `base-apt` creation all we have is the package name. The
knowledge about the upstream repositories is lost and no local repository
policy implementation is possible (e.g., for the "multiple products, multiple
distros" use case).
6. For implementing further use cases like "fetch all sources necessary for
bootstrapping the base system itself", additional logic is necessary.
The new approach:
- On the first build:
- All packages necessary for bootstrapping and building are identified and
downloaded upfront.
- `base-apt` is used for bootstrapping, building and image creation.
- On the next build:
- `base-apt` is used for bootstrapping, building and image creation.
This series addresses issues 1-5 and provides the architecture for
implementing further use cases.
The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to
zero falls back to the old approach.
The implementation uses `debrepo` script which can also be called manually for
pre-fetching packages to the local 'base-apt' repository. It requires
`python3-apt` to be installed on the build host. Some examples of its usage in
standalone mode:
```
# Create local `/build/ba` repository sufficient to bootstrap Debian system with
`armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg
# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```
Changes since v9:
- Rebased on fresh next (436bc96f)
- Removed "Fix python3-apt dependency"
Changes since v8:
- Rebased on fresh next (86c761e4)
- Removed fixes for isar-apt download in rootfs (use Felix's fix)
- Renamed debrepo -> deb-repo
- Fixed failed tests on Full and Fast CI testsets
Aliaksei Karpovich (2):
debrepo: Add override for reprepro
base-apt: Move bullseye-mipsel in separate dir
Uladzimir Bely (10):
scripts: Add debrepo python script handling base-apt
meta: Add debrepo bbclass handling base-apt prefetching
meta: Always use base-apt repo in local mode
meta: Use cached base-apt repo to bootstrap
base-apt: Predownload packages to base-apt before install
meta: Add cache-deb-src functionality in base-apt mode
testsuite: Set ISAR_PREFETCH_BASE_APT by default
Disable deb-dl-dir in base-apt prefetch mode
kas: Add PREFETCH_BASE_APT config entry
ci_build.sh: Install python3-apt if not installed
RECIPE-API-CHANGELOG.md | 9 +
doc/user_manual.md | 1 +
kas/opt/Kconfig | 12 +
kas/opt/prefetch-base-apt.yaml | 9 +
meta-test/conf/local.conf.sample | 3 +
meta/classes-recipe/bootstrap.bbclass | 50 +-
meta/classes-recipe/crossvars.bbclass | 1 +
meta/classes-recipe/deb-dl-dir.bbclass | 43 ++
meta/classes-recipe/dpkg-base.bbclass | 1 +
meta/classes-recipe/dpkg.bbclass | 8 +
.../image-locales-extension.bbclass | 5 +
.../image-tools-extension.bbclass | 13 +
meta/classes-recipe/rootfs.bbclass | 15 +-
meta/classes-recipe/sbuild.bbclass | 3 +-
meta/classes/debrepo.bbclass | 89 +++
meta/conf/bitbake.conf | 10 +-
meta/conf/distro/debian-bullseye.conf | 1 +
meta/lib/aptsrc_fetcher.py | 46 +-
.../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 +
.../isar-mmdebstrap/isar-mmdebstrap.inc | 22 +-
meta/recipes-devtools/base-apt/base-apt.bb | 21 +-
.../sbom-chroot/sbom-chroot.bb | 2 +
.../sbuild-chroot/sbuild-chroot-host.bb | 2 +
scripts/ci_build.sh | 8 +-
scripts/deb-repo | 689 ++++++++++++++++++
testsuite/cibase.py | 4 +
testsuite/cibuilder.py | 7 +
27 files changed, 1047 insertions(+), 29 deletions(-)
create mode 100644 kas/opt/prefetch-base-apt.yaml
create mode 100644 meta/classes/debrepo.bbclass
create mode 100755 scripts/deb-repo
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-1-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 01/12] scripts: Add debrepo python script handling base-apt
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 8:14 ` 'Florian Bezdeka' via isar-users
2026-09-04 7:52 ` [v10 02/12] meta: Add debrepo bbclass handling base-apt prefetching Aliaksei Karpovich
` (10 subsequent siblings)
11 siblings, 1 reply; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich, Aliaksei Laurynovich
From: Uladzimir Bely <ubely@ilbers.de>
This is the main utility responsible for prefetching packages
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").
Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.
In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
Signed-off-by: Aliaksei Laurynovich <alaur@ilbers.de>
---
scripts/deb-repo | 610 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 610 insertions(+)
create mode 100755 scripts/deb-repo
diff --git a/scripts/deb-repo b/scripts/deb-repo
new file mode 100755
index 00000000..23de0461
--- /dev/null
+++ b/scripts/deb-repo
@@ -0,0 +1,610 @@
+#!/usr/bin/env python3
+
+"""
+# This software is a part of Isar.
+# Copyright (C) 2026 ilbers GmbH
+
+# deb-repo: build Debian-like repo using "python3-apt" library.
+
+When building the image, Isar downloads required Debian packages from external
+mirrors. After build completed, it can pick all downloaded packages from DL_DIR
+and build local 'base-apt' Debian-like repo from them.
+
+This tool allows to download packages and create local repo in advance. So,
+Isar just uses this local repository and does not interact with external
+mirrors. Such approach makes deb-dl import/export functionality redundant.
+
+Script `deb-repo` works in so-called "context" directory. It means some
+rootfs-like directory with bare minimum of directories/files required for
+python3-apt to work.
+
+Context directory path is passed with "--workdir <dir>" command-line option.
+On context creating, all passed parameters are stored in the context directory
+and picked every time the context is used again.
+
+1. Repo for building Debian system
+```
+deb-repo --workdir=d12 --init locales gnupg
+```
+Initialize the context in "d12" directory and create a repository in
+"d12/repo/apt" directory sufficient to debootstrap default system
+(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg"
+with all their dependencies will be available in this repo.
+
+```
+deb-repo --workdir=d12 docbook-to-man
+```
+Adds "docbook-to-man" packages with its dependencies to earlier created repo.
+
+```
+deb-repo --workdir=d12 --srcmode docbook-to-man
+```
+Downloads source package for "docbook-to-man" and adds it to the repo
+
+2. Repo for building Ubuntu system
+```
+deb-repo --init --workdir=uf \
+--distro=ubuntu --codename=focal --arch=arm64 \
+--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \
+--repodir=repo/apt --repodbdir=repo/db \
+--mirror=http://ports.ubuntu.com/ubuntu-ports \
+locales gnupg
+```
+Initialize the context in "uf" directory and create a repository in "repo/apt"
+directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and
+source list are specified by corresponding arguments. Packages "locales" and
+" gnupg" with the dependencies will be also placed to the repo.
+
+```
+deb-repo --workdir=uf gnupg,locales
+```
+Add "gnupg" and "locales" packages with their dependencies to earlier created
+ubuntu repo. Other parameters (distro, codename, arch) are ommited since they
+are picked from the context.
+
+3. Repo for cross-building Debian system
+```
+deb-repo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf
+```
+Initialize the context in "d11" directory sufficient to deboostrap Debian
+Bullseye (amd64) with foreign "armhf" architecture support
+
+```
+deb-repo --workdir=d11 gcc
+```
+Add "gcc" package (amd64 version) to earlier created repo.
+
+```
+deb-repo --workdir=d11 --crossbuild gcc
+```
+Add "gcc" package (armhf version) to earlier created repo.
+"""
+
+import os
+import sys
+import fcntl
+
+import argparse
+import shutil
+import subprocess
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+REPREPRO_TIMEOUT = 1200
+
+
+class DebRepo(object):
+ class DebRepoCtx(object):
+ def __init__(self, workdir):
+ self.distro = "debian"
+ self.codename = "bullseye"
+ self.arch = "amd64"
+ self.mirror = "http://deb.debian.org/debian"
+
+ self.repodir = f"{workdir}/repo/apt"
+ self.repodbdir = f"{workdir}/repo/db"
+
+ self.crossarch = self.arch
+ self.compatarch = None
+ self.keydir = "/etc/apt/trusted.gpg.d"
+
+ def __init__(self, args):
+ self.workdir = os.path.abspath(args.workdir)
+ self.ctx = self.DebRepoCtx(self.workdir)
+
+ self.cache = None
+ self.depcache = None
+ self.sr = None
+ self.extrarepo = None
+
+ self.ctx_load()
+ self.ctx_update(args)
+ self.ctx_save()
+
+ print(
+ f"ctx workdir: {self.workdir}\n"
+ f" distro: {self.ctx.distro}\n"
+ f" codename: {self.ctx.codename}\n"
+ f" arch: {self.ctx.arch}\n"
+ f" mirror: {self.ctx.mirror}\n"
+ f" repodir: {self.ctx.repodir}\n"
+ f" repodbdir: {self.ctx.repodbdir}\n"
+ f" crossarch: {self.ctx.crossarch}\n"
+ f" compatarch: {self.ctx.compatarch}\n"
+ f" keydir: {self.ctx.keydir}"
+ )
+
+ if args.extrarepo:
+ self.extrarepo = os.path.abspath(args.extrarepo)
+
+ def ctx_load(self):
+ ctxfile = f"{self.workdir}/deb-repo.ctx"
+
+ if os.path.isfile(ctxfile):
+ with open(ctxfile, 'rb') as f:
+ self.ctx = pickle.load(f)
+
+ def ctx_save(self):
+ ctxfile = f"{self.workdir}/deb-repo.ctx"
+
+ with open(ctxfile, 'wb') as f:
+ pickle.dump(self.ctx, f)
+
+ def ctx_update(self, args):
+ if args.distro:
+ self.ctx.distro = args.distro
+ if args.codename:
+ self.ctx.codename = args.codename
+ if args.arch:
+ self.ctx.arch = args.arch
+ if args.mirror:
+ self.ctx.mirror = args.mirror
+
+ if args.repodir:
+ self.ctx.repodir = os.path.abspath(args.repodir)
+ if args.repodbdir:
+ self.ctx.repodbdir = os.path.abspath(args.repodbdir)
+
+ if args.crossarch:
+ self.ctx.crossarch = args.crossarch
+ if args.compatarch:
+ self.ctx.compatarch = args.compatarch
+ if args.keydir:
+ self.ctx.keydir = args.keydir
+
+ def create_rootfs(self, aptsrcsfile):
+ os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True)
+ with open(f"{self.workdir}/var/lib/dpkg/status", "w"):
+ pass
+
+ os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True)
+
+ srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list"
+ if aptsrcsfile and os.path.exists(aptsrcsfile):
+ shutil.copy(aptsrcsfile, srcfile)
+ else:
+ with open(srcfile, "w") as f:
+ repo = f"{self.ctx.mirror} {self.ctx.codename} main"
+ f.write(f"deb {repo}\n")
+ f.write(f"deb-src {repo}\n")
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial",
+ exist_ok=True)
+
+ os.makedirs(f"{self.workdir}/tmp", exist_ok=True)
+
+ def create_repo_dist(self):
+ conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf"
+ os.makedirs(conf_dir, exist_ok=True)
+ if not os.path.exists(f"{conf_dir}/distributions"):
+ with open(f"{conf_dir}/distributions", "w") as f:
+ f.write(f"Codename: {self.ctx.codename}\n")
+ f.write(
+ "Architectures: "
+ "i386 armhf arm64 amd64 mipsel riscv64 source\n")
+ f.write("Components: main\n")
+
+ def apt_config(self, init, crossbuild):
+ if not init and self.ctx.compatarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch
+
+ if not init and self.ctx.arch != self.ctx.crossarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch
+ apt_pkg.config["APT::Architectures::"] = self.ctx.arch
+
+ apt_pkg.config.set("APT::Architecture", self.ctx.arch)
+
+ apt_pkg.config.set("Dir", self.workdir)
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}")
+ apt_pkg.config.set("Dir::State::status",
+ f"{self.workdir}/var/lib/dpkg/status")
+
+ apt_pkg.config.set("APT::Install-Recommends", "0")
+ apt_pkg.config.set("APT::Install-Suggests", "0")
+
+ # Use host keys for authentification
+ apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir)
+
+ # Allow using repositories without keys
+ apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+ def mark_essential(self):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ if pkg.essential:
+ self.depcache.mark_install(pkg)
+
+ def mark_by_prio(self, priority):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ ver = self.depcache.get_candidate_ver(pkg)
+ if ver and ver.priority <= priority:
+ self.depcache.mark_install(pkg)
+
+ def mark_pkg(self, name, crossbuild):
+ pkgname = name
+
+ if pkgname and (pkgname not in self.cache):
+ # Try for cross arch
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ pkgname += f":{self.ctx.crossarch}"
+
+ if pkgname not in self.cache:
+ print(f"Error: package '{name}' not found")
+ return False
+
+ pkg = self.cache[pkgname]
+
+ if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions):
+ if (pkg.has_provides) and (not pkg.has_versions):
+ print("pkgname is virtual package, selecting best provide")
+ # Select first provide
+ pkg_provide = pkg.provides_list[0][2]
+ # Find better provide with higher version
+ for provide in pkg.provides_list:
+ if apt_pkg.version_compare(provide[2].ver_str,
+ pkg_provide.ver_str) > 0:
+ pkg_provide = provide[2]
+ self.depcache.mark_install(pkg_provide.parent_pkg)
+ else:
+ self.depcache.mark_install(pkg)
+ else:
+ version = pkg.version_list[0]
+ if version.arch == "all":
+ self.depcache.mark_install(pkg)
+ else:
+ if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+ if (pkgname, self.ctx.arch) in self.cache:
+ nativepkg = self.cache[pkgname, self.ctx.arch]
+ self.depcache.mark_install(nativepkg)
+ else:
+ return False
+ else:
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ crosspkg = self.cache[pkgname, self.ctx.crossarch]
+ self.depcache.mark_install(crosspkg)
+ else:
+ return False
+
+ return True
+
+ def mark_list(self, pkglist, crossbuild):
+ ret = True
+ if pkglist:
+ for pkgname in pkglist:
+ ret = ret and self.mark_pkg(pkgname, crossbuild)
+
+ return ret
+
+ def handle_deb(self, item):
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "includedeb",
+ self.ctx.codename,
+ item.destfile
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+
+ def handle_repo(self, fetcher):
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ fd = open(f"{self.workdir}/{dir_cache}.lock", "w")
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ fetcher.run()
+ fd.close()
+ for item in fetcher.items:
+ if item.status == item.STAT_ERROR:
+ print("Some error ocured: '%s'" % item.error_text)
+ pass
+ else:
+ self.handle_deb(item)
+
+ def get_filename(self, uri):
+ path = urllib.parse.urlparse(uri).path
+ unquoted_path = urllib.parse.unquote(path)
+ basename = os.path.basename(unquoted_path)
+ return basename
+
+ def fetch_file(self, uri):
+ filename = self.get_filename(uri)
+ return subprocess.run([
+ "wget",
+ "-H",
+ "--timeout=30",
+ "--tries=3",
+ "-nv",
+ uri,
+ "-O",
+ f"{self.workdir}/tmp/{filename}"
+ ],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+
+ def handle_dsc(self, dsc_file):
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ result = subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "-S", "-", "-P" "source",
+ "--delete",
+ "includedsc",
+ self.ctx.codename,
+ os.path.realpath(f"{self.workdir}/tmp/{dsc_file}")
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+ return result.returncode == 0
+
+ def handle_src_list(self, pkgs):
+ ret = True
+ if pkgs:
+ local_repo = f"{self.ctx.repodir}/{self.ctx.distro}"
+ fetched_files = []
+ for pkg in pkgs:
+ pkgname = pkg
+ pkgver = ""
+ if '=' in pkg:
+ pkgname = pkg.split("=")[0]
+ pkgver = pkg.split("=")[1]
+
+ self.sr.restart()
+ while self.sr.lookup(pkgname):
+ if pkgver and pkgver != self.sr.version:
+ continue
+
+ for sr_file in self.sr.files:
+ uri_file = self.sr.index.archive_uri(sr_file.path)
+ filename = os.path.basename(sr_file.path)
+ path = urllib.parse.urlparse(uri_file).path.lstrip('/')
+ # Strip the distro directory prefix from upstream path
+ # e.g. "debian/pool/main/h/hello/file.dsc" -> "pool/main/h/hello/file.dsc"
+ parts = path.split('/', 1)
+ local_path = parts[1] if len(parts) > 1 else path
+ local_file = f"{local_repo}/{local_path}"
+ print(uri_file)
+ if filename not in fetched_files:
+ if os.path.exists(local_file):
+ shutil.copy2(local_file, f"{self.workdir}/tmp/{filename}")
+ else:
+ result = self.fetch_file(uri_file)
+ if result.returncode != 0:
+ print(f"ERROR: cannot download {uri_file}")
+ print(f"ERROR: {result.stderr}")
+ fetched_files.append(filename)
+ if sr_file.type == 'dsc':
+ dsc_file = filename
+
+ ret = self.handle_dsc(dsc_file)
+ break
+ return ret
+
+ def apt_run(self, init, srcmode, pkgs, dscfile, crossbuild):
+ apt_pkg.init()
+
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ if self.extrarepo:
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ with open(extrarepo_list, "w") as f:
+ distdir=os.path.join(self.extrarepo, "dists")
+ if os.path.isdir(distdir):
+ for dist in os.listdir(distdir):
+ repodir = os.path.join(distdir,dist)
+ if os.path.isdir(repodir):
+ for repo in os.listdir(repodir):
+ if os.path.isdir(os.path.join(repodir, repo)):
+ f.write(f"deb file://{self.extrarepo} "
+ f"{dist} {repo}\n")
+
+ sources = apt_pkg.SourceList()
+ sources.read_main_list()
+
+ progress = apt.progress.text.AcquireProgress()
+
+ self.cache = apt_pkg.Cache()
+ if init:
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+
+ self.sr = apt_pkg.SourceRecords()
+
+ if self.extrarepo:
+ apt_pkg.config.set("Dir::Etc::SourceList", extrarepo_list)
+ apt_pkg.config.set("APT::Get::List-Cleanup", "0")
+ extrarepo_sources = apt_pkg.SourceList()
+ extrarepo_sources.read_main_list()
+ self.cache.update(progress, extrarepo_sources)
+ self.cache = apt_pkg.Cache()
+ os.remove(extrarepo_list)
+
+ self.depcache = apt_pkg.DepCache(self.cache)
+
+ ret = True
+
+ if init:
+ self.mark_essential()
+ # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+ self.mark_by_prio(1)
+
+ pkgs = list(filter(None, ','.join(pkgs).split(',')))
+ if srcmode:
+ ret = self.handle_src_list(set(pkgs))
+ else:
+ ret = self.mark_list(pkgs, crossbuild)
+
+ if dscfile:
+ fobj = open(dscfile, "r")
+
+ try:
+ tagfile = apt_pkg.TagFile(fobj)
+ while tagfile.step() == 1:
+ deps = tagfile.section.get("Build-Depends", "")
+ # Remove extra commas and spaces - apt_pkg.parse_src_depends
+ # doesnt like lines like ", device-tree-compiler"
+ deps = ', '.join(
+ [s.strip() for s in deps.split(',') if s.strip()]
+ )
+ print(f"parsed deps: {deps}")
+ for item in apt_pkg.parse_src_depends(deps, False):
+ pkgname = item[0][0]
+ self.mark_pkg(pkgname, crossbuild)
+
+ finally:
+ fobj.close()
+
+ if not ret:
+ sys.exit("Some of requested packages not found")
+
+ if init or not srcmode:
+ fetcher = apt_pkg.Acquire(progress)
+ pm = apt_pkg.PackageManager(self.depcache)
+
+ recs = apt_pkg.PackageRecords(self.cache)
+ pm.get_archives(fetcher, sources, recs)
+
+ self.handle_repo(fetcher)
+
+
+def parse_arguments():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--init",
+ default=False, action="store_true",
+ help="initialize context in WORKDIR")
+ parser.add_argument(
+ "--workdir",
+ type=str, required=True,
+ help="work directory storing deb-repo context")
+ parser.add_argument(
+ "--aptsrcsfile",
+ type=str, metavar="PATH",
+ help="sources.list file to use when init")
+ parser.add_argument(
+ "--srcmode",
+ default=False, action="store_true",
+ help="add source packages instead of debs")
+ parser.add_argument(
+ "--repodir",
+ type=str, metavar="REPO",
+ help="repository directory")
+ parser.add_argument(
+ "--repodbdir",
+ type=str, metavar="REPODB",
+ help="repository database directory")
+ parser.add_argument(
+ "--extrarepo",
+ type=str, metavar="REPO",
+ help="extra repository to consider")
+ parser.add_argument(
+ "--mirror",
+ type=str,
+ help="use custom distro mirror")
+ parser.add_argument(
+ "--distro",
+ type=str,
+ help="select distro to use")
+ parser.add_argument(
+ "--codename",
+ type=str,
+ help="distro codename")
+ parser.add_argument(
+ "--arch",
+ type=str,
+ help="distro arch")
+ parser.add_argument(
+ "--compatarch",
+ type=str, metavar="ARCH",
+ help="compat arch to use")
+ parser.add_argument(
+ "--crossarch",
+ type=str, metavar="ARCH",
+ help="cross-build arch")
+ parser.add_argument(
+ "--keydir",
+ type=str,
+ help="directory with distro keys")
+ parser.add_argument(
+ "--no-check-gpg",
+ default=False, action="store_true",
+ help="allow insecure repositories")
+ parser.add_argument(
+ "--dscfile",
+ type=str, metavar="PATH",
+ help="Debian source file to parse")
+ parser.add_argument(
+ "--crossbuild",
+ default=False, action="store_true",
+ help="add packages with cross arch")
+
+ parser.add_argument(
+ "packages",
+ nargs='*', type=str,
+ help="space- or comma-separated list of packages to add")
+
+ args = parser.parse_args()
+
+ return args
+
+
+def main():
+ args = parse_arguments()
+
+ if not (args.init or args.packages or args.dscfile):
+ sys.exit("Nothing to do")
+
+ workdir = os.path.abspath(args.workdir)
+ os.makedirs(workdir, exist_ok=True)
+
+ with open(f"{workdir}/deb-repo.lock", "a") as file:
+ fcntl.flock(file.fileno(), fcntl.LOCK_EX)
+
+ deb_repo = DebRepo(args)
+
+ if args.init:
+ deb_repo.create_rootfs(args.aptsrcsfile)
+ deb_repo.create_repo_dist()
+
+ deb_repo.apt_config(args.init, args.crossbuild)
+ deb_repo.apt_run(args.init, args.srcmode, args.packages,
+ args.dscfile, args.crossbuild)
+
+ #Unlock deb-repo context
+ fcntl.flock(file.fileno(), fcntl.LOCK_UN)
+
+
+if __name__ == "__main__":
+ main()
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-2-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 02/12] meta: Add debrepo bbclass handling base-apt prefetching
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
2026-09-04 7:52 ` [v10 01/12] scripts: Add debrepo python script handling base-apt Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 03/12] meta: Always use base-apt repo in local mode Aliaksei Karpovich
` (9 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich
From: Uladzimir Bely <ubely@ilbers.de>
This class uses 'scripts/debrepo' python script to prefetch given
packages or sources to local base-apt repository.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 9 ++++
doc/user_manual.md | 1 +
meta/classes/debrepo.bbclass | 89 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 5 ++
4 files changed, 104 insertions(+)
create mode 100644 meta/classes/debrepo.bbclass
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 2db05169..139fcc8b 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1201,3 +1201,12 @@ To opt out, remove the feature explicitly:
```
ROOTFS_FEATURES:remove = "clean-apt-credentials"
```
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before running any task that need them. Separate
+`deb-repo` script is used for populating base-apt repo.
diff --git a/doc/user_manual.md b/doc/user_manual.md
index f3b1acb9..5cb942ad 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -86,6 +86,7 @@ apt install \
gettext-base \
git \
python3 \
+ python3-apt \
quilt \
qemu-user-static \
reprepro \
diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..79a19144
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,89 @@
+# This software is a part of Isar.
+# Copyright (C) 2026 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+# Prefetch to base-apt repo by default
+ISAR_PREFETCH_BASE_APT ??= "1"
+
+DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+
+ chroot_dir=${1}
+ apt_list=${2}
+
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+ -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+ -o Dir::Etc::SourceParts=\"-\" \
+ -o APT::Get::List-Cleanup=\"0\"
+ "
+}
+
+debrepo_add_packages() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ args=""
+ if [ "${1}" = "--srcmode" ]; then
+ args="${args} --srcmode"
+ shift
+ fi
+
+ if [ "${1}" = "--isarapt" ]; then
+ args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}"
+ shift
+ fi
+
+ workdir="${1}"
+ args="${args} ${2}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ ${SCRIPTSDIR}/deb-repo \
+ --workdir="${workdir}" \
+ ${args}
+}
+
+debrepo_parse_dscfile() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ dscfile="${1}"
+ args=""
+
+ if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+ build_arch=${HOST_ARCH}
+ if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+ args="--crossbuild \
+ crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \
+ dose-distcheck:${build_arch} \
+ libc-dev:${PACKAGE_ARCH} \
+ libstdc++-dev:${PACKAGE_ARCH} \
+ "
+ fi
+ fi
+
+ args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ if [ -n "${DEB_BUILD_PROFILES}" ]; then
+ export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
+ fi
+
+ ${SCRIPTSDIR}/deb-repo \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --dscfile="${dscfile}" \
+ ${args}
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e69ec3bb..c8503ec6 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -67,6 +67,11 @@ SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${BPN}-${DISTRO}-${MACHINE}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "${@ 'vmlinux' if d.getVar('DISTRO_ARCH') in ['mipsel', 'riscv64', 'arm64'] else 'vmlinuz'}"
+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
MACHINEOVERRIDES ?= "${MACHINE}"
DISTROOVERRIDES ?= "${DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}::${BASE_DISTRO}:${ISAR_CHROOT_MODE}:forcevariable"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-3-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 03/12] meta: Always use base-apt repo in local mode
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
2026-09-04 7:52 ` [v10 01/12] scripts: Add debrepo python script handling base-apt Aliaksei Karpovich
2026-09-04 7:52 ` [v10 02/12] meta: Add debrepo bbclass handling base-apt prefetching Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 04/12] meta: Use cached base-apt repo to bootstrap Aliaksei Karpovich
` (8 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich, Aliaksei Laurynovich
From: Uladzimir Bely <ubely@ilbers.de>
This means only local URLs in apt sources.list* are present during
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
Signed-off-by: Aliaksei Laurynovich <alaur@ilbers.de>
---
meta/classes-recipe/bootstrap.bbclass | 6 ++++--
meta/classes-recipe/rootfs.bbclass | 3 ++-
meta/classes-recipe/sbuild.bbclass | 3 ++-
.../isar-mmdebstrap/isar-mmdebstrap.inc | 17 ++++++++++-----
meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++-------
5 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/meta/classes-recipe/bootstrap.bbclass b/meta/classes-recipe/bootstrap.bbclass
index c1a59fd2..f853a932 100644
--- a/meta/classes-recipe/bootstrap.bbclass
+++ b/meta/classes-recipe/bootstrap.bbclass
@@ -39,7 +39,8 @@ python () {
# installation afterwards. However, bootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
own_pub_key = d.getVar("BASE_REPO_KEY")
if own_pub_key:
distro_bootstrap_keys += own_pub_key.split()
@@ -121,7 +122,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
# this is executed during parsing. No error checking possible
use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
elif use_snapshot and snapshot_mirror:
premirrors = snapshot_mirror
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 8d394b05..99328b12 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -62,7 +62,8 @@ ROOTFS_MOUNTS ??= "${REPO_ISAR_DIR}/${DISTRO}:/isar-apt ${WORKDIR}:/isar-work"
python () {
mounts = d.getVar('ROOTFS_MOUNTS', False)
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) and not ':/base-apt' in mounts:
+ if (bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) or
+ bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT'))) and not ':/base-apt' in mounts:
base_apt = '{}:/base-apt'.format(d.getVar('REPO_BASE_DIR'))
d.setVar('ROOTFS_MOUNTS', '{} {}'.format(mounts, base_apt))
}
diff --git a/meta/classes-recipe/sbuild.bbclass b/meta/classes-recipe/sbuild.bbclass
index 6db29251..29a8dd71 100644
--- a/meta/classes-recipe/sbuild.bbclass
+++ b/meta/classes-recipe/sbuild.bbclass
@@ -44,7 +44,8 @@ EOF
cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] || \
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind,private 0 0"
grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
fi
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 994da174..4d61e9be 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -153,16 +153,21 @@ do_bootstrap() {
fi
E="${@ isar_export_proxies(d)}"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"
line="copy://$base_apt_tmp/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
- if [ -z "${BASE_REPO_KEY}" ]; then
+ if [ -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+
line="[trusted=yes] ${line}"
fi
echo "deb ${line}" > "${WORKDIR}/sources.list.d/base-apt.list"
line="copy://$base_apt_tmp/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
- if [ -z "${BASE_REPO_KEY}" ]; then
+ if [ -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+
line="[trusted=yes] ${line}"
fi
echo "deb-src ${line}" >> "${WORKDIR}/sources.list.d/base-apt.list"
@@ -183,9 +188,11 @@ do_bootstrap() {
\$1/etc/apt/sources.list.d/*.list && \
mkdir -p \$1/base-apt && \
mount -o bind,private '${REPO_BASE_DIR}' \$1/base-apt && \
- chroot \$1 apt-get update -y \
+ chroot \$1 mv /etc/apt/sources.list.d/0000base-apt.list \
+ /etc/apt/sources.list.d/base-apt.list
+ flock -x ${REPO_BASE_DIR}/repo.lock -c \"chroot \$1 apt-get update -y \
-o APT::Update::Error-Mode=any \
- ${@'-o APT::Sandbox::User=root' if d.getVar('ISAR_CHROOT_MODE') == 'unshare' else ''} && \
+ ${@'-o APT::Sandbox::User=root' if d.getVar('ISAR_CHROOT_MODE') == 'unshare' else ''} \" && \
chroot \$1 apt-get install -y dpkg && \
umount \$1/base-apt && \
umount \$1/$base_apt_tmp && \
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 06b1f6c8..05ef9741 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -59,9 +59,12 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ fi
if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -69,14 +72,18 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${HOST_BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${HOST_BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ fi
fi
}
python do_cache() {
- if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+ not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
return 0
for key in d.getVar('BASE_REPO_KEY').split():
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-4-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 04/12] meta: Use cached base-apt repo to bootstrap
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (2 preceding siblings ...)
2026-09-04 7:52 ` [v10 03/12] meta: Always use base-apt repo in local mode Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 05/12] base-apt: Predownload packages to base-apt before install Aliaksei Karpovich
` (7 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich
From: Uladzimir Bely <ubely@ilbers.de>
This patch makes local base-apt repo to be created before
bootstrap task. So, bootstrap is then done from it.
The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.
For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.
The goal is have workable base-apt repo before first build completed.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
meta/classes-recipe/bootstrap.bbclass | 44 +++++++++++++++++++
.../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 +
.../isar-mmdebstrap/isar-mmdebstrap.inc | 2 +
3 files changed, 48 insertions(+)
diff --git a/meta/classes-recipe/bootstrap.bbclass b/meta/classes-recipe/bootstrap.bbclass
index f853a932..10cb23be 100644
--- a/meta/classes-recipe/bootstrap.bbclass
+++ b/meta/classes-recipe/bootstrap.bbclass
@@ -16,6 +16,7 @@ BOOTSTRAP_FOR_HOST ?= "0"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
+APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DISTRO_BOOTSTRAP_KEYS ?= ""
@@ -223,8 +224,51 @@ python do_apt_config_prepare() {
aggregate_files(d, apt_preferences_list, apt_preferences_out)
apt_sources_out = d.getVar("APTSRCS")
+ apt_sources_init_out = d.getVar("APTSRCS_INIT")
apt_sources_list = get_aptsources_list(d)
+ aggregate_files(d, apt_sources_list, apt_sources_init_out)
aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
}
addtask apt_config_prepare before do_bootstrap after do_unpack
+
+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ debrepo_args=""
+ if [ "${BASE_DISTRO}" != "debian" ]; then
+ if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ else
+ if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+ fi
+
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+ debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+ fi
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ ${SCRIPTSDIR}/deb-repo --init \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --aptsrcsfile="${APTSRCS_INIT}" \
+ --repodir="${REPO_BASE_DIR}" \
+ --repodbdir="${REPO_BASE_DB_DIR}" \
+ --mirror="${@get_distro_source(d)}" \
+ --arch="${BOOTSTRAP_DISTRO_ARCH}" \
+ --distro="${BOOTSTRAP_BASE_DISTRO}" \
+ --codename="${BASE_DISTRO_CODENAME}" \
+ ${debrepo_args} \
+ ${DISTRO_BOOTSTRAP_BASE_PACKAGES}
+}
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb
index fa4b76a6..52563d33 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb
@@ -17,3 +17,5 @@ require isar-mmdebstrap.inc
HOST_DISTRO_BOOTSTRAP_KEYS ?= ""
DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}"
+
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 4d61e9be..c452035f 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -147,6 +147,8 @@ do_bootstrap() {
fi
fi
bootstrap_args="--verbose --variant=minbase --include=${@','.join(d.getVar('DISTRO_BOOTSTRAP_BASE_PACKAGES').split())}"
+ debrepo_bootstrap_prepare
+
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
cp "${DISTRO_BOOTSTRAP_KEYRING}" "${WORKDIR}/trusted.gpg.d/"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-5-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 05/12] base-apt: Predownload packages to base-apt before install
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (3 preceding siblings ...)
2026-09-04 7:52 ` [v10 04/12] meta: Use cached base-apt repo to bootstrap Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 06/12] meta: Add cache-deb-src functionality in base-apt mode Aliaksei Karpovich
` (6 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich
From: Uladzimir Bely <ubely@ilbers.de>
This patch uses debrepo script to predownload packages to base-apt
repository before they are installed in rootfs.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
meta/classes-recipe/crossvars.bbclass | 1 +
meta/classes-recipe/dpkg-base.bbclass | 1 +
meta/classes-recipe/dpkg.bbclass | 8 ++++
.../image-locales-extension.bbclass | 5 ++
.../image-tools-extension.bbclass | 13 ++++++
meta/classes-recipe/rootfs.bbclass | 5 +-
meta/lib/aptsrc_fetcher.py | 46 +++++++++++++++----
.../sbom-chroot/sbom-chroot.bb | 2 +
.../sbuild-chroot/sbuild-chroot-host.bb | 2 +
9 files changed, 74 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/crossvars.bbclass b/meta/classes-recipe/crossvars.bbclass
index 7cf2d660..6f070714 100644
--- a/meta/classes-recipe/crossvars.bbclass
+++ b/meta/classes-recipe/crossvars.bbclass
@@ -30,6 +30,7 @@ python __anonymous() {
schroot_dir = d.getVar('SCHROOT_HOST_DIR', False)
sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build"
sdk_toolchain = "crossbuild-essential-" + distro_arch
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR'))
else:
d.setVar('BUILD_ARCH', distro_arch)
schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False)
diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index 4a81787e..91b17768 100644
--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -11,6 +11,7 @@ inherit terminal
inherit repository
inherit deb-dl-dir
inherit essential
+inherit debrepo
# Local (WORKDIR-internal) dir where do_dpkg_build collects the built debs.
# Its content is exported into the shared, sstate-tracked DEPLOY_DIR_DEB so
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index a42703d5..323202e4 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -117,6 +117,12 @@ dpkg_runbuild() {
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ debrepo_parse_dscfile "${DSC_FILE}"
+
+ locked_update_cmd=":"
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'"
+ fi
# networking is automatically enabled on older versions of sbuild
sbuild_network_option=""
@@ -139,9 +145,11 @@ dpkg_runbuild() {
--chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="mkdir -p ${deb_dir}" \
--chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \
+ --chroot-setup-commands="${locked_update_cmd}" \
--chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-*-dummy_*.deb" \
--finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \
+ --finished-build-commands="mkdir -p ${ext_root}" \
${@ '--finished-build-commands="cp /var/log/dpkg.log $ext_root/dpkg_partial.log"' if d.getVar('ISAR_CHROOT_MODE') == 'schroot' else '' } \
--build-path="" --build-dir=${WORKDIR} --dist="${DEBDISTRONAME}" ${DSC_FILE}
diff --git a/meta/classes-recipe/image-locales-extension.bbclass b/meta/classes-recipe/image-locales-extension.bbclass
index c03f34c0..32b5e19c 100644
--- a/meta/classes-recipe/image-locales-extension.bbclass
+++ b/meta/classes-recipe/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
# This class extends the image.bbclass for setting locales and purging unneeded
# ones.
+inherit debrepo
+
LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
en_US ISO-8859-1\n"
LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
image_install_localepurge_download[weight] = "40"
image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
image_install_localepurge_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
run_privileged_heredoc <<'EOF'
set -e
${@insert_isar_mounts(d, d.getVar('ROOTFSDIR'), d.getVar('ROOTFS_MOUNTS') if d.getVar('ISAR_CHROOT_MODE') == 'unshare' else '')}
diff --git a/meta/classes-recipe/image-tools-extension.bbclass b/meta/classes-recipe/image-tools-extension.bbclass
index 8f666444..229ce318 100644
--- a/meta/classes-recipe/image-tools-extension.bbclass
+++ b/meta/classes-recipe/image-tools-extension.bbclass
@@ -6,6 +6,11 @@
# This file extends the image.bbclass to supply tools for futher imager functions
inherit sbuild
+inherit debrepo
+
+python __anonymous() {
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR'))
+}
IMAGER_INSTALL ??= ""
IMAGER_BUILD_DEPS ??= ""
@@ -60,12 +65,20 @@ imager_run_schroot() {
echo "Installing imager deps: ${local_install}"
distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
fi
E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${schroot_dir} ${distro}
+
+ debrepo_add_packages --isarapt "${debrepo_workdir}" "${local_install}"
+ debrepo_update_apt_source_list "${schroot_dir}" "base-apt"
+
${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOAPT
schroot -r -c ${session_id} -d / -u root -- sh -c " \
apt-get update \
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 99328b12..d1f24d68 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -2,7 +2,7 @@
# Copyright (c) Siemens AG, 2020
inherit deb-dl-dir
-
+inherit debrepo
inherit sbom
ROOTFS_ARCH ?= "${DISTRO_ARCH}"
@@ -344,6 +344,9 @@ rootfs_install_pkgs_download[progress] = "custom:rootfs_progress.PkgsDownloadPro
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK}"
rootfs_install_pkgs_download() {
+ debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
# download packages using apt in a non-privileged namespace
rootfs_cmd --bind "${ROOTFSDIR}/var/cache/apt/archives" /var/cache/apt/archives \
${ROOTFSDIR} \
diff --git a/meta/lib/aptsrc_fetcher.py b/meta/lib/aptsrc_fetcher.py
index fe8f1cd3..c81517c2 100644
--- a/meta/lib/aptsrc_fetcher.py
+++ b/meta/lib/aptsrc_fetcher.py
@@ -43,18 +43,29 @@ class AptSrcSchroot(AptSrc):
repo_isar_dir = d.getVar('REPO_ISAR_DIR')
lockfile = bb.utils.lockfile(f'{repo_isar_dir}/isar.lock')
+ debrepo_target_dir = d.getVar('DEBREPO_TARGET_DIR')
+ isar_prefetch_base_apt = d.getVar('ISAR_PREFETCH_BASE_APT')
+ repo_base_dir = d.getVar('REPO_BASE_DIR')
+ scriptsdir = d.getVar('SCRIPTSDIR')
+
try:
runfetchcmd(f'''
set -e
+
schroot -r -c {session_id} -d / -u root -- \
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
- schroot -r -c {session_id} -d / -- \
- sh -c '
- set -e
- mkdir -p /downloads/{ud.localfile}
- cd /downloads/{ud.localfile}
- apt-get -y -o Debug::NoLocking=1 --download-only --only-source source {ud.src_package}
- '
+
+ if [ "{isar_prefetch_base_apt}" = "1" ]; then
+ {scriptsdir}/deb-repo --workdir={debrepo_target_dir} --srcmode "{ud.src_package}"
+ else
+ schroot -r -c {session_id} -d / -- \
+ sh -c '
+ set -e
+ mkdir -p /downloads/{ud.localfile}
+ cd /downloads/{ud.localfile}
+ apt-get -y -o Debug::NoLocking=1 --download-only --only-source source {ud.src_package}
+ '
+ fi
''', d)
except (OSError, FetchError):
raise
@@ -72,9 +83,19 @@ class AptSrcSchroot(AptSrc):
pp = d.getVar('PP')
pps = d.getVar('PPS')
+
+ isar_prefetch_base_apt = d.getVar('ISAR_PREFETCH_BASE_APT')
+ repo_base_dir = d.getVar('REPO_BASE_DIR')
+
try:
runfetchcmd(f'''
set -e
+ if [ "{isar_prefetch_base_apt}" = "1" ]; then
+ flock -x "{repo_base_dir}/repo.lock" -c "
+ schroot -r -c {session_id} -d / -u root -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
schroot -r -c {session_id} -d / -u root -- \
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
schroot -r -c {session_id} -d / -- \
@@ -82,7 +103,16 @@ class AptSrcSchroot(AptSrc):
set -e
dscfile=$(apt-get -y -qq --print-uris --only-source source {ud.src_package} | \
cut -d " " -f2 | grep -E "\.dsc")
- cp /downloads/{ud.localfile}/* {pp}
+
+ if [ "{isar_prefetch_base_apt}" = "1" ]; then
+ temp_dir=$(mktemp -d)
+ cd "$temp_dir"
+ apt-get -y -o Debug::NoLocking=1 --download-only --only-source source {ud.src_package}
+ # Getting rid of links
+ cp -L "$temp_dir"/* {pp}
+ else
+ cp /downloads/{ud.localfile}/* {pp}
+ fi
cd {pp}
mv -f {pps} {pps}.prev
dpkg-source -x "$dscfile" {pps}
diff --git a/meta/recipes-devtools/sbom-chroot/sbom-chroot.bb b/meta/recipes-devtools/sbom-chroot/sbom-chroot.bb
index f347327b..5fcc15a7 100644
--- a/meta/recipes-devtools/sbom-chroot/sbom-chroot.bb
+++ b/meta/recipes-devtools/sbom-chroot/sbom-chroot.bb
@@ -26,6 +26,8 @@ SBOM_IMAGE_INSTALL = "python3-debsbom python3-spdx-tools python3-cyclonedx-lib"
ROOTFSDIR = "${WORKDIR}/rootfs"
ROOTFS_PACKAGES = "${SBOM_IMAGE_INSTALL}"
+DEBREPO_WORKDIR = "${@ '${DEBREPO_HOST_DIR}' if d.getVar("DISTRO_ARCH") != d.getVar("HOST_ARCH") else '${DEBREPO_TARGET_DIR}'}"
+
do_sbomchroot_deploy[dirs] = "${SBOM_DIR}"
do_sbomchroot_deploy[network] = "${TASK_USE_SUDO}"
do_sbomchroot_deploy() {
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
index 52a461fe..48ce2209 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -18,3 +18,5 @@ SBUILD_CHROOT_PREINSTALL ?= " \
${@ 'libc6-dev:${DISTRO_ARCH} crossbuild-essential-${DISTRO_ARCH}' if bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')) else ''} \
apt-utils \
"
+
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-6-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 06/12] meta: Add cache-deb-src functionality in base-apt mode
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (4 preceding siblings ...)
2026-09-04 7:52 ` [v10 05/12] base-apt: Predownload packages to base-apt before install Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 07/12] testsuite: Set ISAR_PREFETCH_BASE_APT by default Aliaksei Karpovich
` (5 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich
From: Uladzimir Bely <ubely@ilbers.de>
Fill base-apt repo with source packages.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
meta/classes-recipe/deb-dl-dir.bbclass | 41 ++++++++++++++++++++++++++
meta/classes-recipe/rootfs.bbclass | 7 +++++
2 files changed, 48 insertions(+)
diff --git a/meta/classes-recipe/deb-dl-dir.bbclass b/meta/classes-recipe/deb-dl-dir.bbclass
index 60e2fecc..7e9f89b1 100644
--- a/meta/classes-recipe/deb-dl-dir.bbclass
+++ b/meta/classes-recipe/deb-dl-dir.bbclass
@@ -27,6 +27,47 @@ debsrc_source_version_filter() {
| sort -u
}
+debsrc_fill_base_apt() {
+ export rootfs="$1"
+ export rootfs_distro="$2"
+ mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+
+ ( flock 9
+ set -e
+ printenv | grep -q BB_VERBOSE_LOGS && set -x
+
+ # We need temporary files for our lists of source packages
+ # trap exit of this sub-shell to remove them (this script may exit abruptly
+ # since "set -e" is used)
+ avail=$(mktemp)
+ wanted=$(mktemp)
+ trap "rm -f ${avail} ${wanted}" EXIT
+
+ # List all packages known to apt
+ rootfs_cmd \
+ --bind "${DEBSRCDIR}" "/deb-src" \
+ --bind "${rootfs}" "${rootfs}" \
+ -- \
+ apt-cache dumpavail \
+ | debsrc_source_version_filter > ${avail}
+
+ # Use apt-ftparchive to scan all .deb files found in the download directory
+ # and get the <source> <version> pairs that we wish to download
+ apt-ftparchive --md5=no --sha1=no --sha256=no --sha512=no \
+ -a "${DISTRO_ARCH}" packages \
+ "${REPO_BASE_DIR}" \
+ | debsrc_source_version_filter > ${wanted}
+
+ # We now have two sorted lists: source packages we want and those known to
+ # apt. We will only consider source packages that may be found in both.
+ comm -12 ${wanted} ${avail} \
+ | while read src version; do
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${src}=${version}"
+ done
+
+ ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index d1f24d68..bdca9b42 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -489,6 +489,13 @@ do_cache_deb_src() {
run_privileged tar -xf "${BOOTSTRAP_SRC}" ./var/lib/apt/lists --one-top-level="${ROOTFSDIR}"
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ debsrc_fill_base_apt ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ rootfs_do_mounts_priv
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+ rootfs_do_umounts_priv
+
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
run_privileged rm -f "${ROOTFSDIR}"/etc/resolv.conf
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-7-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 07/12] testsuite: Set ISAR_PREFETCH_BASE_APT by default
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (5 preceding siblings ...)
2026-09-04 7:52 ` [v10 06/12] meta: Add cache-deb-src functionality in base-apt mode Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 08/12] Disable deb-dl-dir in base-apt prefetch mode Aliaksei Karpovich
` (4 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This makes Isar use `base-apt` repo in different way. Any package
installation is done from `base-apt` repo which is prepopulated
from external mirrors.
This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.
In order to be able to run CI in old mode, allow CI read the option
from the environment. Also, adjust some tests (like repro one) to
make them work with ISAR_PREFETCH_BASE_APT set.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta-test/conf/local.conf.sample | 3 +++
testsuite/cibase.py | 4 ++++
testsuite/cibuilder.py | 7 +++++++
3 files changed, 14 insertions(+)
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 2dbe28f1..74ee90b1 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,6 +27,9 @@ BB_DISKMON_DIRS = "\
MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"
+# Use new base-apt behaviour
+ISAR_PREFETCH_BASE_APT ?= "1"
+
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 033b3b19..835593df 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -81,10 +81,14 @@ class CIBaseTest(CIBuilder):
self.fail("GPG import failed")
try:
+ self.move_in_build_dir('tmp', 'tmp_before_repro')
self.bitbake(targets, **kwargs)
repro_type = 'signed' if signed else 'unsigned'
self.move_in_build_dir('tmp', f"tmp_middle_repro_{repro_type}")
+
+ os.makedirs(f"{self.build_dir}/tmp/deploy/")
+ self.move_in_build_dir(f"tmp_middle_repro_{repro_type}/deploy/base-apt", 'tmp/deploy/base-apt')
self.configure(
targets=targets,
gpg_pub_key=gpg_pub_key if signed else None,
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 1d415084..ca75f2e1 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -174,6 +174,10 @@ class CIBuilder(Test):
fail_on_cleanup = os.getenv('ISAR_FAIL_ON_CLEANUP')
strlines = None if lines is None else '\\n'.join(lines)
+
+ # get prefetch base apt mode from environment
+ prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT')
+
self.log.info(
f"===================================================\n"
f"Configuring build_dir {self.build_dir}\n"
@@ -197,6 +201,7 @@ class CIBuilder(Test):
f" generate_sbom = {generate_sbom}\n"
f" rootless = {rootless}\n"
f" lines = {strlines}\n"
+ f" prefetch_base_apt = {prefetch_base_apt}\n"
f"==================================================="
)
@@ -305,6 +310,8 @@ class CIBuilder(Test):
f.write('ISAR_ROOTLESS = "1"\n')
if lines is not None:
f.writelines((line + '\n' if not line.endswith('\n') else line) for line in lines)
+ if prefetch_base_apt == "0":
+ f.write('ISAR_PREFETCH_BASE_APT = "0"\n')
# include ci_build.conf in local.conf
with open(self.build_dir + '/conf/local.conf', 'r+') as f:
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-8-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 08/12] Disable deb-dl-dir in base-apt prefetch mode
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (6 preceding siblings ...)
2026-09-04 7:52 ` [v10 07/12] testsuite: Set ISAR_PREFETCH_BASE_APT by default Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 09/12] kas: Add PREFETCH_BASE_APT config entry Aliaksei Karpovich
` (3 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Aliaksei Karpovich
From: Uladzimir Bely <ubely@ilbers.de>
Since all packages and source packages are placed to base-apt repo
during build, there is no need to have them in one more place.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
meta/classes-recipe/deb-dl-dir.bbclass | 2 ++
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 3 ++-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/deb-dl-dir.bbclass b/meta/classes-recipe/deb-dl-dir.bbclass
index 7e9f89b1..2fafd0a7 100644
--- a/meta/classes-recipe/deb-dl-dir.bbclass
+++ b/meta/classes-recipe/deb-dl-dir.bbclass
@@ -174,6 +174,7 @@ deb_dl_dir_import() {
# nothing to copy if download directory does not exist just yet
[ ! -d "${pc}" ] && return 0
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
# attempt to create hard-links for .deb files from downloads/ into
# /var/cache/apt/archives/ so apt will only download packages we
@@ -196,6 +197,7 @@ deb_dl_dir_export() {
export rootfs="${1}"
export owner=$(id -u):$(id -g)
mkdir -p "${pc}"
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
export isar_debs=$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \
"find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index c452035f..94cc5fe3 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -274,7 +274,8 @@ do_bootstrap() {
- \
"$bootstrap_list" > ${DEPLOYDIR}/${DEPLOY_ISAR_BOOTSTRAP}.tar.zst
- if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
run_privileged find ${WORKDIR}/dl_dir -maxdepth 1 -mindepth 1 -exec rm -rf --one-file-system "{}" \;
rmdir ${WORKDIR}/dl_dir
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-9-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 09/12] kas: Add PREFETCH_BASE_APT config entry
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (7 preceding siblings ...)
2026-09-04 7:52 ` [v10 08/12] Disable deb-dl-dir in base-apt prefetch mode Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 10/12] ci_build.sh: Install python3-apt if not installed Aliaksei Karpovich
` (2 subsequent siblings)
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This option allows to set ISAR_PREFETCH_BASE_APT to "0" or "1" and
choose between old and new base-apt behaviour.
Docker image kas uses should have "python3-apt" preinstalled
in order to have new functionality working.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
kas/opt/Kconfig | 12 ++++++++++++
kas/opt/prefetch-base-apt.yaml | 9 +++++++++
2 files changed, 21 insertions(+)
create mode 100644 kas/opt/prefetch-base-apt.yaml
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index bc4ed997..316b755a 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -213,3 +213,15 @@ config KAS_INCLUDE_USE_DRACUT
string
default "kas/opt/dracut.yaml"
depends on USE_DRACUT
+
+config PREFETCH_BASE_APT
+ bool "Prefetch base-apt repo"
+ default y
+ help
+ This makse Isar always take packages from base-apt repository where they
+ are prefetched by deb-repo script before requested.
+
+config KAS_INCLUDE_PREFETCH_BASE_APT
+ string
+ default "kas/opt/prefetch-base-apt.yaml"
+ depends on PREFETCH_BASE_APT
diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml
new file mode 100644
index 00000000..0cbdb93f
--- /dev/null
+++ b/kas/opt/prefetch-base-apt.yaml
@@ -0,0 +1,9 @@
+# This software is a part of Isar.
+# Copyright (C) 2024 ilbers GmbH
+
+header:
+ version: 14
+
+local_conf_header:
+ prefetch-base-apt: |
+ ISAR_PREFETCH_BASE_APT = "1"
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-10-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 10/12] ci_build.sh: Install python3-apt if not installed
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (8 preceding siblings ...)
2026-09-04 7:52 ` [v10 09/12] kas: Add PREFETCH_BASE_APT config entry Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 11/12] debrepo: Add override for reprepro Aliaksei Karpovich
2026-09-04 7:52 ` [v10 12/12] base-apt: Move bullseye-mipsel in separate dir Aliaksei Karpovich
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely
From: Uladzimir Bely <ubely@ilbers.de>
This is mostly related to gitlab CI that migth use an image without
preinstalled python3-apt.
Also, make system python packages available in virtualenv.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
scripts/ci_build.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 241ff8c8..71f62111 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
sudo apt-get update -qq
sudo apt-get install -y virtualenv
rm -rf /tmp/avocado_venv
- virtualenv --python python3 /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv --system-site-packages
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install setuptools==81.0.0
@@ -138,6 +138,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
fi
fi
+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+ sudo apt-get update -qq
+ sudo apt-get install -y python3-apt
+fi
+
# Provide working path
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-11-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 11/12] debrepo: Add override for reprepro
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (9 preceding siblings ...)
2026-09-04 7:52 ` [v10 10/12] ci_build.sh: Install python3-apt if not installed Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
2026-09-04 7:52 ` [v10 12/12] base-apt: Move bullseye-mipsel in separate dir Aliaksei Karpovich
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Aliaksei Karpovich
It fixes an issue when bootstrap image in case of offline build
is different than online. It happens because some packages has
different Priority field because of override.
F.e. libtext-wrapi18n-perl in trixie:
- inside deb package: 'Priority: required'
- inside Packages from deb repo: 'Priority: optional'
Solution: download the override file from repo and pass it to
reprepro.
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
scripts/deb-repo | 97 +++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 88 insertions(+), 9 deletions(-)
diff --git a/scripts/deb-repo b/scripts/deb-repo
index 23de0461..99913fb4 100755
--- a/scripts/deb-repo
+++ b/scripts/deb-repo
@@ -89,6 +89,8 @@ import shutil
import subprocess
import pickle
import urllib.parse
+import urllib.request
+import gzip
import apt_pkg
import apt.progress.base
@@ -96,8 +98,38 @@ import apt.progress.base
REPREPRO_TIMEOUT = 1200
+def parse_aptsources_list_line(source_list_line):
+ import re
+
+ s = source_list_line.strip()
+
+ if not s or s.startswith("#"):
+ return None
+
+ type, s = re.split(r"\s+", s, maxsplit=1)
+ if type not in ["deb", "deb-src"]:
+ return None
+
+ options = ""
+ options_match = re.match(r"\[\s*(\S+=\S+(?=\s))*\s*(\S+=\S+)\s*\]\s+", s)
+ if options_match:
+ options = options_match.group(0).strip()
+ s = s[options_match.end():]
+
+ source, s = re.split(r"\s+", s, maxsplit=1)
+
+ if s.startswith("/"):
+ suite = ""
+ else:
+ suite, s = re.split(r"\s+", s, maxsplit=1)
+
+ components = " ".join(s.split())
+
+ return [type, options, source, suite, components]
class DebRepo(object):
+ BOOTSTRAP_LIST = "/etc/apt/sources.list.d/bootstrap.list"
+
class DebRepoCtx(object):
def __init__(self, workdir):
self.distro = "debian"
@@ -181,9 +213,9 @@ class DebRepo(object):
with open(f"{self.workdir}/var/lib/dpkg/status", "w"):
pass
- os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True)
+ srcfile = f"{self.workdir}{self.BOOTSTRAP_LIST}"
+ os.makedirs(os.path.dirname(srcfile), exist_ok=True)
- srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list"
if aptsrcsfile and os.path.exists(aptsrcsfile):
shutil.copy(aptsrcsfile, srcfile)
else:
@@ -201,13 +233,60 @@ class DebRepo(object):
def create_repo_dist(self):
conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf"
os.makedirs(conf_dir, exist_ok=True)
- if not os.path.exists(f"{conf_dir}/distributions"):
- with open(f"{conf_dir}/distributions", "w") as f:
- f.write(f"Codename: {self.ctx.codename}\n")
- f.write(
- "Architectures: "
- "i386 armhf arm64 amd64 mipsel riscv64 source\n")
- f.write("Components: main\n")
+ override_fname = "override"
+ self.create_override(f"{conf_dir}/{override_fname}")
+ with open(f"{conf_dir}/distributions", "w") as f:
+ f.write(f"Codename: {self.ctx.codename}\n")
+ f.write(
+ "Architectures: "
+ "i386 armhf arm64 amd64 mipsel riscv64 source\n")
+ f.write("Components: main\n")
+ f.write(f"DebOverride: {override_fname}\n")
+
+ def create_override(self, override_file):
+ deb_override_name = f"override.{self.ctx.codename}.main"
+ deb_override_file = f"{self.workdir}/tmp/{deb_override_name}"
+ if (not os.path.exists(deb_override_file) or
+ os.path.getsize(deb_override_file) < 1):
+ # Extract source URL
+ srcfile = f"{self.workdir}{self.BOOTSTRAP_LIST}"
+ with open(srcfile, "r") as f:
+ for line in f:
+ entry = parse_aptsources_list_line(line)
+ if entry:
+ source_url = entry[2]
+ # Download override file
+ self.fetch_file(f"{source_url}/indices/{deb_override_name}")
+ if (os.path.exists(deb_override_file) and
+ os.path.getsize(deb_override_file) > 1):
+ break
+ # Download packet override file
+ self.fetch_file(f"{source_url}/indices/{deb_override_name}.gz")
+ # Unpack override file
+ with gzip.open(f"{deb_override_file}.gz", "r") as f_in, open(deb_override_file, "wb") as f_out:
+ for line in f_in:
+ f_out.write(line)
+ break
+
+ with open(override_file, "w") as f_out:
+ # Unpack deb override file
+ with open(deb_override_file, "r") as f:
+ # Convert deb override to reprepro override
+ for line in f:
+ fields = line.strip().split()
+
+ if len(fields) < 3:
+ continue
+
+ package = fields[0]
+ priority = fields[1]
+ section = fields[2]
+
+ f_out.write(f"{package} Priority {priority}\n")
+ f_out.write(f"{package} Section {section}\n")
+
+ if f_out.tell() < 1:
+ print("WARNING: override file is empty!")
def apt_config(self, init, crossbuild):
if not init and self.ctx.compatarch:
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-12-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [v10 12/12] base-apt: Move bullseye-mipsel in separate dir
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
` (10 preceding siblings ...)
2026-09-04 7:52 ` [v10 11/12] debrepo: Add override for reprepro Aliaksei Karpovich
@ 2026-09-04 7:52 ` Aliaksei Karpovich
11 siblings, 0 replies; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-04 7:52 UTC (permalink / raw)
To: isar-users; +Cc: Aliaksei Karpovich
As the bullseye-mipsel has separate bullseye repo (archive) it leads to
unresolving dependency (f.e. gnupg) in base-apt repo. It happens because
mipsel satied on deb11u2 version (and dont't have secure update) then other
move formward and have deb11u3 version.
When the bullseye-mipsel is built in the same time with other arch targed
(f.e. bullseye arm) the following dependency issue occurs:
| The following packages have unmet dependencies:
| gnupg : Depends: dirmngr (< 2.2.27-2+deb11u3.1~) but it is not going to be installed
| Depends: dirmngr (>= 2.2.27-2+deb11u3) but it is not going to be installed
| Depends: gnupg-utils (>= 2.2.27-2+deb11u3) but 2.2.27-2+deb11u2 is to be installed
| Depends: gpg (>= 2.2.27-2+deb11u3) but 2.2.27-2+deb11u2 is to be installed
| Depends: gpg-agent (>= 2.2.27-2+deb11u3) but 2.2.27-2+deb11u2 is to be installed
It happens becasue (during arm build) gnupg_2.2.27-2+deb11u3_all.deb was
fetched, but other dependencies (dirmngr, gnupg-utils) for mipsel have old
deb11u2 version.
Fix: separate bullseye-mipsel into separate repo in base-apt.
Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
---
meta/conf/bitbake.conf | 5 +++--
meta/conf/distro/debian-bullseye.conf | 1 +
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index c8503ec6..97f200a0 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -99,8 +99,9 @@ REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}/apt"
REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}/db"
# Base apt repository paths
-REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt"
-REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/db"
+REPO_BASE_DIR_NAME ?= "${DISTRO}"
+REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${REPO_BASE_DIR_NAME}/apt"
+REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${REPO_BASE_DIR_NAME}/db"
# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
diff --git a/meta/conf/distro/debian-bullseye.conf b/meta/conf/distro/debian-bullseye.conf
index c368dcaf..00d50df4 100644
--- a/meta/conf/distro/debian-bullseye.conf
+++ b/meta/conf/distro/debian-bullseye.conf
@@ -2,6 +2,7 @@
HOST_DISTRO_APT_SOURCES:mipsel ?= "conf/distro/${HOST_DISTRO}-mipsel.list"
DISTRO_APT_SOURCES:mipsel ?= "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-mipsel.list"
+REPO_BASE_DIR_NAME:mipsel = "${DISTRO}-mipsel"
require debian-common.conf
--
2.43.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-13-akarpovich%40ilbers.de.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [v10 01/12] scripts: Add debrepo python script handling base-apt
2026-09-04 7:52 ` [v10 01/12] scripts: Add debrepo python script handling base-apt Aliaksei Karpovich
@ 2026-09-04 8:14 ` 'Florian Bezdeka' via isar-users
2026-09-09 9:30 ` Aliaksei Karpovich
0 siblings, 1 reply; 16+ messages in thread
From: 'Florian Bezdeka' via isar-users @ 2026-09-04 8:14 UTC (permalink / raw)
To: Aliaksei Karpovich, isar-users; +Cc: Uladzimir Bely, Aliaksei Laurynovich
On Fri, 2026-09-04 at 09:52 +0200, Aliaksei Karpovich wrote:
> From: Uladzimir Bely <ubely@ilbers.de>
>
> This is the main utility responsible for prefetching packages
> into local `base-apt` repo from external Debian mirrors. It uses
> python-apt module and requires some kind of minimal `rootfs` to work
> (let's call it "debrepo context").
>
> Once initialized with `--init --workdir=<path>`, it stores the initial
> configuration in `repo.opts` file inside the context and uses it at
> futher calls.
>
> In future, the logic `debrepo` script implements could be directly
> implemented inside bitbake classes.
>
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
> Signed-off-by: Aliaksei Laurynovich <alaur@ilbers.de>
> ---
>
[snip]
> + def handle_deb(self, item):
> + fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
> + fcntl.flock(fd, fcntl.LOCK_EX)
> + subprocess.run([
> + "reprepro",
> + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
> + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
> + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
> + "-C", "main",
> + "includedeb",
> + self.ctx.codename,
> + item.destfile
> + ], timeout=REPREPRO_TIMEOUT)
> + fd.close()
> +
Searching for subprocess.run() delivers wget and reprepro as external
dependencies. As I can't see any exception handling: What happens if
those dependencies are not installed / available in the calling context?
Florian
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/0bfc4f717c8a33cdd17227e3d2e07484a0e02e08.camel%40siemens.com.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [v10 01/12] scripts: Add debrepo python script handling base-apt
2026-09-04 8:14 ` 'Florian Bezdeka' via isar-users
@ 2026-09-09 9:30 ` Aliaksei Karpovich
2026-09-09 9:37 ` 'Florian Bezdeka' via isar-users
0 siblings, 1 reply; 16+ messages in thread
From: Aliaksei Karpovich @ 2026-09-09 9:30 UTC (permalink / raw)
To: Florian Bezdeka, isar-users; +Cc: Uladzimir Bely, Aliaksei Laurynovich
[-- Attachment #1: Type: text/plain, Size: 2306 bytes --]
On 9/4/26 11:14, Florian Bezdeka wrote:
> On Fri, 2026-09-04 at 09:52 +0200, Aliaksei Karpovich wrote:
>> From: Uladzimir Bely<ubely@ilbers.de>
>>
>> This is the main utility responsible for prefetching packages
>> into local `base-apt` repo from external Debian mirrors. It uses
>> python-apt module and requires some kind of minimal `rootfs` to work
>> (let's call it "debrepo context").
>>
>> Once initialized with `--init --workdir=<path>`, it stores the initial
>> configuration in `repo.opts` file inside the context and uses it at
>> futher calls.
>>
>> In future, the logic `debrepo` script implements could be directly
>> implemented inside bitbake classes.
>>
>> Signed-off-by: Uladzimir Bely<ubely@ilbers.de>
>> Signed-off-by: Aliaksei Karpovich<akarpovich@ilbers.de>
>> Signed-off-by: Aliaksei Laurynovich<alaur@ilbers.de>
>> ---
>>
> [snip]
>
>> + def handle_deb(self, item):
>> + fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
>> + fcntl.flock(fd, fcntl.LOCK_EX)
>> + subprocess.run([
>> + "reprepro",
>> + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
>> + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
>> + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
>> + "-C", "main",
>> + "includedeb",
>> + self.ctx.codename,
>> + item.destfile
>> + ], timeout=REPREPRO_TIMEOUT)
>> + fd.close()
>> +
> Searching for subprocess.run() delivers wget and reprepro as external
> dependencies. As I can't see any exception handling: What happens if
> those dependencies are not installed / available in the calling context?
>
> Florian
In this case the child_exception_type is raised in subprocess.py.
Aliaksei.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6f6c002a-1301-402e-831f-5bbe06c54d52%40ilbers.de.
[-- Attachment #2: Type: text/html, Size: 3829 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [v10 01/12] scripts: Add debrepo python script handling base-apt
2026-09-09 9:30 ` Aliaksei Karpovich
@ 2026-09-09 9:37 ` 'Florian Bezdeka' via isar-users
0 siblings, 0 replies; 16+ messages in thread
From: 'Florian Bezdeka' via isar-users @ 2026-09-09 9:37 UTC (permalink / raw)
To: Aliaksei Karpovich, isar-users; +Cc: Uladzimir Bely, Aliaksei Laurynovich
On Wed, 2026-09-09 at 12:30 +0300, Aliaksei Karpovich wrote:
> On 9/4/26 11:14, Florian Bezdeka wrote:
> >
> > On Fri, 2026-09-04 at 09:52 +0200, Aliaksei Karpovich wrote:
> > > From: Uladzimir Bely <ubely@ilbers.de>
> > >
> > > This is the main utility responsible for prefetching packages
> > > into local `base-apt` repo from external Debian mirrors. It uses
> > > python-apt module and requires some kind of minimal `rootfs` to work
> > > (let's call it "debrepo context").
> > >
> > > Once initialized with `--init --workdir=<path>`, it stores the initial
> > > configuration in `repo.opts` file inside the context and uses it at
> > > futher calls.
> > >
> > > In future, the logic `debrepo` script implements could be directly
> > > implemented inside bitbake classes.
> > >
> > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > Signed-off-by: Aliaksei Karpovich <akarpovich@ilbers.de>
> > > Signed-off-by: Aliaksei Laurynovich <alaur@ilbers.de>
> > > ---
> >
> > [snip]
> > >
> > > + def handle_deb(self, item):
> > > + fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
> > > + fcntl.flock(fd, fcntl.LOCK_EX)
> > > + subprocess.run([
> > > + "reprepro",
> > > + "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
> > > + "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
> > > + "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
> > > + "-C", "main",
> > > + "includedeb",
> > > + self.ctx.codename,
> > > + item.destfile
> > > + ], timeout=REPREPRO_TIMEOUT)
> > > + fd.close()
> > > +
> >
> > Searching for subprocess.run() delivers wget and reprepro as external
> > dependencies. As I can't see any exception handling: What happens if
> > those dependencies are not installed / available in the calling context?
> >
> > Florian
> >
> >
> >
> >
>
> In this case the child_exception_type is raised in subprocess.py.
>
> Aliaksei.
>
Questions coming to mind:
- But what is the outcome then? Any half-done work where the last
step failed? Unclean file system state, ...
- Can a user derive from the exception handling (stack trace?) what
exactly went wrong?
- Shouldn't we fail early and inform the user about missing
dependencies?
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/c4eb451a7e8e705f72f18c8de8d5e274d1de4bbe.camel%40siemens.com.
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-09-09 9:37 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-04 7:52 [v10 00/12] Improving base-apt usage Aliaksei Karpovich
2026-09-04 7:52 ` [v10 01/12] scripts: Add debrepo python script handling base-apt Aliaksei Karpovich
2026-09-04 8:14 ` 'Florian Bezdeka' via isar-users
2026-09-09 9:30 ` Aliaksei Karpovich
2026-09-09 9:37 ` 'Florian Bezdeka' via isar-users
2026-09-04 7:52 ` [v10 02/12] meta: Add debrepo bbclass handling base-apt prefetching Aliaksei Karpovich
2026-09-04 7:52 ` [v10 03/12] meta: Always use base-apt repo in local mode Aliaksei Karpovich
2026-09-04 7:52 ` [v10 04/12] meta: Use cached base-apt repo to bootstrap Aliaksei Karpovich
2026-09-04 7:52 ` [v10 05/12] base-apt: Predownload packages to base-apt before install Aliaksei Karpovich
2026-09-04 7:52 ` [v10 06/12] meta: Add cache-deb-src functionality in base-apt mode Aliaksei Karpovich
2026-09-04 7:52 ` [v10 07/12] testsuite: Set ISAR_PREFETCH_BASE_APT by default Aliaksei Karpovich
2026-09-04 7:52 ` [v10 08/12] Disable deb-dl-dir in base-apt prefetch mode Aliaksei Karpovich
2026-09-04 7:52 ` [v10 09/12] kas: Add PREFETCH_BASE_APT config entry Aliaksei Karpovich
2026-09-04 7:52 ` [v10 10/12] ci_build.sh: Install python3-apt if not installed Aliaksei Karpovich
2026-09-04 7:52 ` [v10 11/12] debrepo: Add override for reprepro Aliaksei Karpovich
2026-09-04 7:52 ` [v10 12/12] base-apt: Move bullseye-mipsel in separate dir Aliaksei Karpovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox