* [PATCH v5 01/12] meta: move base-apt from deploy directory to the top
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 02/12] ci_build.sh: Install python3-apt if not installed Uladzimir Bely
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
While base-apt is supposed to be always used for deboostrapping and
installing build dependencies, move it out of temporary directory.
This also allows not to change some of CI tests that completely
remove TMPDIR and underlaying DEPLOY_DIR.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/conf/bitbake.conf | 4 ++--
testsuite/citest.py | 3 +++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a85fea77..89b3e6a3 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -97,8 +97,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
# Base apt repository paths
-REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt"
-REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/db"
+REPO_BASE_DIR = "${TOPDIR}/base-apt/${DISTRO}/apt"
+REPO_BASE_DB_DIR = "${TOPDIR}/base-apt/${DISTRO}/db"
# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 17a90244..2393e275 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -86,9 +86,11 @@ class ReproTest(CIBaseTest):
self.init()
try:
+ self.delete_from_build_dir('base-apt')
self.perform_repro_test(targets, signed=True)
finally:
self.move_in_build_dir('tmp', 'tmp_repro_signed')
+ self.move_in_build_dir('base-apt', 'base-apt_repro_signed')
def test_repro_unsigned(self):
targets = [
@@ -101,6 +103,7 @@ class ReproTest(CIBaseTest):
self.perform_repro_test(targets, cross=False)
finally:
self.move_in_build_dir('tmp', 'tmp_repro_unsigned')
+ self.move_in_build_dir('base-apt', 'base-apt_repro_unsigned')
class CcacheTest(CIBaseTest):
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 02/12] ci_build.sh: Install python3-apt if not installed
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 01/12] meta: move base-apt from deploy directory to the top Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 03/12] scripts: Add debrepo python script handling base-apt Uladzimir Bely
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This is mostly related to gitlab CI that migth use an image without
preinstalled python3-apt.
Also, make system python packages available in virtualenv.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
scripts/ci_build.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 2fa5cccd..ff7c114a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
sudo apt-get update -qq
sudo apt-get install -y virtualenv
rm -rf /tmp/avocado_venv
- virtualenv --python python3 /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv --system-site-packages
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install avocado-framework==100.1
@@ -127,6 +127,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
fi
fi
+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+ sudo apt-get update -qq
+ sudo apt-get install -y python3-apt
+fi
+
# Provide working path
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 03/12] scripts: Add debrepo python script handling base-apt
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 01/12] meta: move base-apt from deploy directory to the top Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 02/12] ci_build.sh: Install python3-apt if not installed Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 04/12] meta: Add debrepo bbclass handling base-apt prefetching Uladzimir Bely
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This is the main utility responsible for prefetching packages
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").
Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.
In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
scripts/debrepo | 443 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 443 insertions(+)
create mode 100755 scripts/debrepo
diff --git a/scripts/debrepo b/scripts/debrepo
new file mode 100755
index 00000000..2a5e5b51
--- /dev/null
+++ b/scripts/debrepo
@@ -0,0 +1,443 @@
+#!/usr/bin/env python3
+
+# This software is a part of ISAR.
+# Copyright (C) 2022 ilbers GmbH
+
+import os
+import sys
+import fcntl
+
+import shutil
+import subprocess
+import getopt
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+class DebRepo(object):
+ def __init__(self, workdir, cmdline_opts):
+ self.workdir = workdir
+ self.optsfile = self.workdir + "/repo.opts"
+
+ # Set default values
+ self.distro = "debian"
+
+ self.repo = self.workdir + "/repo/apt" + "/" + self.distro
+ self.repodb = self.workdir + "/repo/db" + "/" + self.distro
+ self.mirror = "http://deb.debian.org/debian"
+ self.arch = "amd64"
+ self.codename = "bullseye"
+ self.keydir = "/etc/apt/trusted.gpg.d"
+ self.check_gpg = True
+ self.isaraptdir = ""
+
+ # Load stored opts
+ opts = self.load_opts()
+ print("stored opts: " + str(opts))
+
+ # Overwrite opts by cmdline_opts
+ for opt, arg in cmdline_opts.items():
+ opts[opt] = arg
+
+ print("all opts: " + str(opts))
+
+ # Replace by values passed in commandline
+ for opt, arg in opts.items():
+ if opt == "mirror":
+ self.mirror = arg
+ if opt == "arch":
+ self.arch = arg
+ if opt == "distro":
+ self.distro = arg
+ if opt == "codename":
+ self.codename = arg
+ if opt == "keydir":
+ self.keydir = arg
+ if opt == "isaraptdir":
+ self.isaraptdir = arg
+ if opt == "check_gpg":
+ self.check_gpg = arg
+
+ self.crossarch = self.arch
+ for opt, arg in opts.items():
+ if opt == "crossarch":
+ self.crossarch = arg
+
+ self.compatarch = ""
+ for opt, arg in opts.items():
+ if opt == "compatarch":
+ self.compatarch = arg
+
+ for opt, arg in opts.items():
+ if opt == "repodir":
+ self.repo = arg + "/" + self.distro
+ if opt == "repodbdir":
+ self.repodb = arg + "/" + self.distro
+
+ self.save_opts(opts)
+
+ print("workdir: " + str(self.workdir))
+ print("repo: " + str(self.repo))
+ print("repodb: " + str(self.repodb))
+ print("mirror: " + str(self.mirror))
+ print("arch: " + str(self.arch))
+ print("crossarch: " + str(self.crossarch))
+ if self.compatarch:
+ print("compatarch: " + str(self.compatarch))
+ print("distro: " + str(self.distro))
+ print("codename: " + str(self.codename))
+ print("keydir: " + str(self.keydir))
+ print("isaraptdir: " + str(self.isaraptdir))
+ print("check_gpg: " + str(self.check_gpg))
+
+ self.cache = None
+ self.depcache = None
+ self.sr = None
+
+ def create_rootfs(self, aptsrcsfile):
+ if not os.path.exists(self.workdir + "/var/lib/dpkg"):
+ os.makedirs(self.workdir + "/var/lib/dpkg")
+ with open(self.workdir + "/var/lib/dpkg" + "/status", "w"):
+ pass
+
+ if not os.path.exists(self.workdir + "/etc/apt/sources.list.d"):
+ os.makedirs(self.workdir + "/etc/apt/sources.list.d")
+ if os.path.exists(aptsrcsfile):
+ shutil.copy(aptsrcsfile, self.workdir + "/etc/apt/sources.list.d/bootstrap.list")
+
+ if self.isaraptdir:
+ with open(self.workdir + "/etc/apt/sources.list.d/isar-apt.list", "w") as f:
+ f.write("deb [trusted=yes] file://" + self.isaraptdir + " isar main\n")
+
+ if not os.path.exists(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial"):
+ os.makedirs(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial")
+
+ if not os.path.exists(self.workdir + "/tmp"):
+ os.makedirs(self.workdir + "/tmp")
+
+ def apt_config(self, init, crossbuild):
+ # Configure apt to work with empty directory
+ if not init and self.arch != self.crossarch:
+ apt_pkg.config["APT::Architectures::"] = self.crossarch
+ apt_pkg.config["APT::Architectures::"] = self.arch
+
+ if not init and self.compatarch:
+ apt_pkg.config["APT::Architectures::"] = self.compatarch
+
+ apt_pkg.config.set("APT::Architecture", self.arch)
+
+ apt_pkg.config.set("Dir", self.workdir)
+ apt_pkg.config.set("Dir::Cache", self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename)
+ apt_pkg.config.set("Dir::State::status", self.workdir + "/var/lib/dpkg/status")
+
+ apt_pkg.config.set("APT::Install-Recommends", "0")
+ apt_pkg.config.set("APT::Install-Suggests", "0")
+
+ # Use host keys for authentification
+ # apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg")
+ # apt_pkg.config.set("Dir::Etc::TrustedParts", "/etc/apt/trusted.gpg.d")
+ apt_pkg.config.set("Dir::Etc::TrustedParts", self.keydir)
+
+ # Allow using repositories without keys
+ if not self.check_gpg:
+ apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+ def mark_essential(self):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.arch:
+ if pkg.essential:
+ self.depcache.mark_install(pkg)
+
+ def mark_by_prio(self, priority):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.arch:
+ ver = self.depcache.get_candidate_ver(pkg)
+ if ver and ver.priority <= priority:
+ self.depcache.mark_install(pkg)
+
+ def mark_pkg(self, pkgname, crossbuild):
+ if pkgname in self.cache:
+ pkg = self.cache[pkgname]
+
+ if not crossbuild or ':' in pkgname or len(pkg.version_list) == 0:
+ if pkg.has_provides and not pkg.has_versions:
+ print("pkgname is virtual package, selecting best provide")
+ # Select first provide
+ pkg_provide = pkg.provides_list[0][2]
+ # Find better provide with higher version
+ for provide in pkg.provides_list:
+ if apt_pkg.version_compare(provide[2].ver_str, pkg_provide.ver_str) > 0:
+ pkg_provide = provide[2]
+ self.depcache.mark_install(pkg_provide.parent_pkg)
+ else:
+ self.depcache.mark_install(pkg)
+ else:
+ version = pkg.version_list[0]
+ if version.arch == "all":
+ self.depcache.mark_install(pkg)
+ else:
+ if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+ if (pkgname, self.arch) in self.cache:
+ nativepkg = self.cache[pkgname, self.arch]
+ self.depcache.mark_install(nativepkg)
+ else:
+ if (pkgname, self.crossarch) in self.cache:
+ crosspkg = self.cache[pkgname, self.crossarch]
+ self.depcache.mark_install(crosspkg)
+
+ def mark_list(self, pkglist, crossbuild):
+ if pkglist:
+ for pkgname in pkglist:
+ self.mark_pkg(pkgname, crossbuild)
+
+ def handle_deb(self, item):
+ lockfd = open(self.repo + '/../repo.lock', 'w')
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", self.repodb,
+ "--outdir", self.repo,
+ "--confdir", self.repo + "/conf",
+ "-C", "main",
+ "includedeb",
+ self.codename,
+ item.destfile
+ ])
+ lockfd.close()
+
+ def handle_repo(self, fetcher):
+ lockfd = open(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + ".lock", "w")
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ fetcher.run()
+ lockfd.close()
+ for item in fetcher.items:
+ if item.status == item.STAT_ERROR:
+ print("Some error ocured: '%s'" % item.error_text)
+ pass
+ else:
+ self.handle_deb(item)
+
+ def get_filename(self, uri):
+ path = urllib.parse.urlparse(uri).path
+ unquoted_path = urllib.parse.unquote(path)
+ basename = os.path.basename(unquoted_path)
+ return basename
+
+ def download_file(self, uri):
+ filename = self.get_filename(uri)
+ subprocess.run([
+ "wget",
+ "-H",
+ "--timeout=30",
+ "--tries=3",
+ "-q",
+ uri,
+ "-O",
+ self.workdir + "/tmp/" + filename
+ ])
+
+ def handle_dsc(self, uri):
+ filename = self.get_filename(uri)
+ lockfd = open(self.repo + '/../repo.lock', 'w')
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", self.repodb,
+ "--outdir", self.repo,
+ "--confdir", self.repo + "/conf",
+ "-C", "main",
+ "-S", "-", "-P" "source",
+ "--delete",
+ "includedsc",
+ self.codename,
+ os.path.realpath(self.workdir + "/tmp/" + filename)
+ ])
+ lockfd.close()
+
+ def handle_src_list(self, pkgs):
+ if pkgs:
+ for pkg in pkgs:
+ pkgname=pkg
+ pkgver=""
+ if '=' in pkg:
+ pkgname = pkg.split("=")[0]
+ pkgver = pkg.split("=")[1]
+
+ self.sr.restart()
+ while self.sr.lookup(pkgname):
+ if pkgver and pkgver != self.sr.version:
+ continue
+
+ for sr_file in self.sr.files:
+ print(self.sr.index.archive_uri(sr_file[2]))
+ self.download_file(self.sr.index.archive_uri(sr_file[2]))
+
+ dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2])
+ self.handle_dsc(dsc_uri)
+ break
+
+ def apt_run(self, init, srcmode, pkgs, controlfile, crossbuild):
+ apt_pkg.init()
+
+ sources = apt_pkg.SourceList()
+ sources.read_main_list()
+
+ progress = apt.progress.text.AcquireProgress()
+
+ self.cache = apt_pkg.Cache()
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+
+ self.depcache = apt_pkg.DepCache(self.cache)
+ self.sr = apt_pkg.SourceRecords()
+
+ if init:
+ self.mark_essential()
+ # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+ self.mark_by_prio(1)
+
+ if srcmode:
+ self.handle_src_list(pkgs)
+ else:
+ self.mark_list(pkgs, crossbuild)
+
+ if controlfile:
+ fobj = open(controlfile, "r")
+
+ try:
+ tagfile = apt_pkg.TagFile(fobj)
+ while tagfile.step() == 1:
+ deps = tagfile.section.get("Build-Depends", "")
+ # Remove extra commas and spaces - apt_pkg.parse_depends doesnt like
+ # lines like ", device-tree-compiler"
+ deps = ', '.join([s.strip() for s in deps.split(',') if s.strip()])
+ print("parsed deps: " + str(deps))
+ for item in apt_pkg.parse_depends(deps, False):
+ pkgname = item[0][0]
+ self.mark_pkg(pkgname, crossbuild)
+
+ finally:
+ fobj.close()
+
+ if init or not srcmode:
+ fetcher = apt_pkg.Acquire(progress)
+ pm = apt_pkg.PackageManager(self.depcache)
+
+ recs = apt_pkg.PackageRecords(self.cache)
+ pm.get_archives(fetcher, sources, recs)
+
+ self.handle_repo(fetcher)
+
+ def load_opts(self):
+ params = {}
+ if os.path.isfile(self.optsfile):
+ with open(self.optsfile, 'rb') as file:
+ data = file.read()
+ if data:
+ params = pickle.loads(data)
+
+ return params
+
+ def save_opts(self, opts):
+ file = open(self.optsfile, 'wb')
+ pickle.dump(opts, file)
+ file.close()
+
+
+class DebRepoArgs(object):
+ def __init__(self):
+ self.workdir = ""
+ self.init = False
+ self.srcmode = False
+ self.controlfile = ""
+ self.aptsrcsfile = ""
+ self.crossbuild = False
+
+ self.opts = {}
+ self.pkgs = []
+
+ try:
+ opts, args = getopt.getopt(sys.argv[1:], "", [
+ "init",
+ "srcmode",
+ "workdir=",
+ "repodir=",
+ "repodbdir=",
+ "mirror=",
+ "arch=",
+ "crossarch=",
+ "compatarch=",
+ "distro=",
+ "codename=",
+ "keydir=",
+ "isaraptdir=",
+ "no-check-gpg",
+ "controlfile=",
+ "aptsrcsfile=",
+ "crossbuild"
+ ])
+ except getopt.GetoptError as msg:
+ print("Error: " + str(msg))
+ sys.exit(1)
+
+ for opt, arg in opts:
+ if opt in ("--workdir"):
+ self.workdir = arg
+ if opt in ("--init"):
+ self.init = True
+ if opt in ("--srcmode"):
+ self.srcmode = True
+ if opt in ("--controlfile"):
+ self.controlfile = arg
+ if opt in ("--aptsrcsfile"):
+ self.aptsrcsfile = arg
+ if opt in ("--crossbuild"):
+ self.crossbuild = True
+
+ if opt in ("--repodir",
+ "--repodbdir",
+ "--mirror",
+ "--arch",
+ "--crossarch",
+ "--compatarch",
+ "--distro",
+ "--codename",
+ "--keydir",
+ "--isaraptdir",
+ "--controlfile",
+ ):
+ self.opts[opt[2:]] = arg
+ if opt in ("--no-check-gpg"):
+ self.opts['check_gpg'] = False
+
+ if not self.workdir:
+ print("Error: workdir is not specified")
+ sys.exit(1)
+
+ self.pkgs = args
+
+
+def main():
+ args = DebRepoArgs()
+
+ if not (args.init or args.pkgs or args.controlfile):
+ print("Nothing to do")
+ sys.exit(0)
+
+ if not os.path.exists(args.workdir):
+ os.makedirs(args.workdir)
+
+ debrepo = DebRepo(args.workdir, args.opts)
+
+ if (args.init):
+ debrepo.create_rootfs(args.aptsrcsfile)
+
+ debrepo.apt_config(args.init, args.crossbuild)
+ debrepo.apt_run(args.init, args.srcmode, args.pkgs, args.controlfile, args.crossbuild)
+
+
+if __name__ == "__main__":
+ main()
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 04/12] meta: Add debrepo bbclass handling base-apt prefetching
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (2 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 03/12] scripts: Add debrepo python script handling base-apt Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 05/12] meta: Always use base-apt repo in local mode Uladzimir Bely
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This class uses 'scripts/debrepo' python script to prefetch given
packages or sources to local base-apt repository.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/debrepo.bbclass | 86 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 5 +++
2 files changed, 91 insertions(+)
create mode 100644 meta/classes/debrepo.bbclass
diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..8f2a3215
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,86 @@
+DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ chroot_dir=${1}
+ apt_list=${2}
+
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+ -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+ -o Dir::Etc::SourceParts=\"-\" \
+ -o APT::Get::List-Cleanup=\"0\"
+ "
+}
+
+debrepo_add_packages() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ return
+ fi
+
+ args=""
+ if [ "${1}" = "--srcmode" ]; then
+ args="${args} --srcmode"
+ shift
+ fi
+
+ workdir="${1}"
+ pkgs="${2}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${workdir}/repo.lock" -c "
+ ${SCRIPTSDIR}/debrepo \
+ ${args} \
+ --workdir=\"${workdir}\" \
+ ${pkgs}
+ "
+}
+
+debrepo_handle_controlfile() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ return
+ fi
+
+ control_file="${1}"
+
+ args=""
+ pkgs=""
+
+ build_arch=${DISTRO_ARCH}
+ if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+ build_arch=${HOST_ARCH}
+ fi
+ if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+ args="--crossbuild"
+ pkgs="${pkgs} crossbuild-essential-${PACKAGE_ARCH}:${build_arch}"
+ pkgs="${pkgs} dose-distcheck:${build_arch}"
+ pkgs="${pkgs} libc-dev:${PACKAGE_ARCH}"
+ pkgs="${pkgs} libstdc++-dev:${PACKAGE_ARCH}"
+ fi
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "
+ ${SCRIPTSDIR}/debrepo \
+ --workdir=\"${DEBREPO_WORKDIR}\" \
+ --controlfile=\"${control_file}\" \
+ ${args} \
+ ${pkgs}
+ "
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 89b3e6a3..91f81bd4 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -71,6 +71,11 @@ KERNEL_FILE:mipsel ?= "vmlinux"
KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"
+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
OVERRIDES = "${PACKAGE_ARCH}:${MACHINE}:${DISTRO}:${BASE_DISTRO_CODENAME}:forcevariable"
FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 05/12] meta: Always use base-apt repo in local mode
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (3 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 04/12] meta: Add debrepo bbclass handling base-apt prefetching Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 06/12] meta: Use cached base-apt repo to debootstrap Uladzimir Bely
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This means only local URLs in apt sources.list* are present during
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/buildchroot.bbclass | 5 +++--
meta/classes/rootfs.bbclass | 5 +++--
.../isar-bootstrap/isar-bootstrap.inc | 12 +++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++-------
4 files changed, 28 insertions(+), 15 deletions(-)
diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e4a7a571..947529b0 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -38,8 +38,9 @@ buildchroot_do_mounts() {
mount --rbind /sys '${BUILDCHROOT_DIR}/sys'
mount --make-rslave '${BUILDCHROOT_DIR}/sys'
- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${BUILDCHROOT_DIR}/base-apt'
mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 22449d71..cedde239 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -54,8 +54,9 @@ rootfs_do_mounts() {
mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
fi
- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${ROOTFSDIR}/base-apt'
mountpoint -q '${ROOTFSDIR}/base-apt' || \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b94ae0bd..0cfe82ab 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -42,7 +42,8 @@ python () {
# installation afterwards. However, debootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
own_pub_key = d.getVar("BASE_REPO_KEY")
if own_pub_key:
distro_bootstrap_keys += own_pub_key.split()
@@ -110,7 +111,8 @@ def parse_aptsources_list_line(source_list_line):
def get_apt_source_mirror(d, aptsources_entry_list):
import re
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
else:
premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
@@ -286,7 +288,8 @@ do_bootstrap() {
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
debootstrap_args="$debootstrap_args --no-check-gpg"
fi
E="${@ isar_export_proxies(d)}"
@@ -313,7 +316,8 @@ do_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
if [ -z "${BASE_REPO_KEY}" ]; then
line="[trusted=yes] ${line}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index ea885fe6..e62958d1 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -57,9 +57,12 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ fi
if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -67,14 +70,18 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${HOST_BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${HOST_BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ fi
fi
}
python do_cache() {
- if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+ not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
return 0
for key in d.getVar('BASE_REPO_KEY').split():
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 06/12] meta: Use cached base-apt repo to debootstrap
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (4 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 05/12] meta: Always use base-apt repo in local mode Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 07/12] meta: Consider global debrepo context Uladzimir Bely
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This patch makes local base-apt repo to be created before
debootstrap task. So, debootstrap is then done from it.
The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.
For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.
The goal is have workable base-apt repo before first build completed.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
.../isar-bootstrap/isar-bootstrap.inc | 99 +++++++++++++++++--
1 file changed, 91 insertions(+), 8 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0cfe82ab..e1ddac42 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -29,6 +29,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+
FILESEXTRAPATHS:append = ":${BBPATH}"
inherit deb-dl-dir
@@ -278,12 +280,73 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
inherit compat
+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ return
+ fi
+
+ debrepo_args=""
+ if [ "${BASE_DISTRO}" != "debian" ]; then
+ if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ else
+ if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ fi
+ #if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ debrepo_args="$debrepo_args --no-check-gpg"
+ #fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+ fi
+
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+ debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
+ else
+ debrepo_args="$debrepo_args --isaraptdir=${REPO_ISAR_DIR}/${DISTRO}"
+ fi
+ mkdir -p "${DEBREPO_WORKDIR}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "
+ ${SCRIPTSDIR}/debrepo --init \
+ --workdir=\"${debrepo_workdir}\" \
+ --aptsrcsfile=\"${APTSRCS_INIT}\" \
+ --repodir=\"${REPO_BASE_DIR}\" \
+ --repodbdir=\"${REPO_BASE_DB_DIR}\" \
+ --mirror=\"${@get_distro_source(d)}\" \
+ --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \
+ --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \
+ --codename=\"${BASE_DISTRO_CODENAME}\" \
+ ${debrepo_args} \
+ gnupg locales usr-is-merged
+ "
+}
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
+
+ debrepo_bootstrap_prepare
+
debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
@@ -301,8 +364,19 @@ do_bootstrap() {
sudo -E -s <<'EOSUDO'
set -e
if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}"
fi
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ ${DEBOOTSTRAP} $debootstrap_args \
+ $arch_param \
+ ${@get_distro_components_argument(d)} \
+ ${@get_distro_suite(d)} \
+ ${ROOTFSDIR} \
+ file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ "
+ else
${DEBOOTSTRAP} $debootstrap_args \
$arch_param \
${@get_distro_components_argument(d)} \
@@ -310,7 +384,7 @@ do_bootstrap() {
"${ROOTFSDIR}" \
"${@get_distro_source(d)}" \
${DISTRO_DEBOOTSTRAP_SCRIPT}
-
+ fi
# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
install -v -m644 "${APTPREFS}" \
@@ -382,12 +456,21 @@ do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
fi
-
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
- -o APT::Update::Error-Mode=any
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ "
+ else
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ fi
umount -l "${ROOTFSDIR}/dev/shm"
umount -l "${ROOTFSDIR}/dev/pts"
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 07/12] meta: Consider global debrepo context
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (5 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 06/12] meta: Use cached base-apt repo to debootstrap Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 08/12] base-apt: Predownload packages to base-apt before install Uladzimir Bely
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
For host/target and cross/native modes different debrepo context
(e.g., working dir) should be used.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 11 +++++++++++
meta/recipes-devtools/buildchroot/buildchroot.inc | 11 +++++++++++
meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 11 +++++++++++
3 files changed, 33 insertions(+)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e3412c82..c3e47a69 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -12,6 +12,17 @@ inherit repository
inherit deb-dl-dir
inherit essential
+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ cross = d.getVar('ISAR_CROSS_COMPILE', True)
+
+ if cross == "0" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index f74896fb..320df55a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -13,6 +13,17 @@ SRC_URI = "file://configscript.sh \
file://deps.sh"
PV = "1.0"
+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ variant = d.getVar('BUILDCHROOT_VARIANT', True)
+
+ if variant == "target" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
inherit rootfs
BUILDCHROOT_DIR = "${WORKDIR}/rootfs"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index fd8bb648..03f812c9 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -8,6 +8,17 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260
PV = "1.0"
+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ variant = d.getVar('SBUILD_VARIANT', True)
+
+ if variant == "target" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
inherit rootfs
inherit compat
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 08/12] base-apt: Predownload packages to base-apt before install
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (6 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 07/12] meta: Consider global debrepo context Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 09/12] meta: Add cache-deb-src functionality in base-apt mode Uladzimir Bely
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This patch uses debrepo script to predownload packages to base-apt
repository before they are installed in buildchroot/rootfs.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 36 ++++++++++++++++++--
meta/classes/dpkg.bbclass | 8 +++++
meta/classes/image-locales-extension.bbclass | 5 +++
meta/classes/image-tools-extension.bbclass | 9 +++++
meta/classes/rootfs.bbclass | 4 +++
5 files changed, 60 insertions(+), 2 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c3e47a69..faf2ec77 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -23,6 +23,8 @@ python __anonymous() {
d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
}
+inherit debrepo
+
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
@@ -126,10 +128,25 @@ do_apt_fetch() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
+ # Begin chroot session
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}"
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ schroot -d / -r -c ${session_id} -u root -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
for uri in "${SRC_APT}"; do
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ schroot -d / -r -c ${session_id} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
done
+
+ # End chroot session
+ schroot -e -c ${session_id}
+
schroot_delete_configs
}
@@ -139,6 +156,8 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"
# Add dependency from the correct schroot: host or target
do_apt_fetch[depends] += "${SCHROOT_DEP}"
+# Debrepo context is created by target bootstrap, need this dependency too
+do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap"
do_apt_unpack() {
rm -rf ${S}
@@ -150,8 +169,18 @@ do_apt_unpack() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
+ # Begin chroot session
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ schroot -d / -r -c ${session_id} -u root -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
for uri in "${SRC_APT}"; do
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ schroot -d / -r -c ${session_id} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -160,6 +189,9 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
done
+
+ # End chroot session
+ schroot -e -c ${session_id}
schroot_delete_configs
}
do_apt_unpack[network] = "${TASK_USE_SUDO}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5c8996d6..ea26b530 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,7 @@ dpkg_runbuild() {
export PARALLEL_MAKE="${PARALLEL_MAKE}"
rm -f ${SBUILD_CONFIG}
+ debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control"
env | while read -r line; do
# Filter the same lines
@@ -101,6 +102,11 @@ dpkg_runbuild() {
sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
+ locked_update_cmd=":"
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'"
+ fi
+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
@@ -109,8 +115,10 @@ dpkg_runbuild() {
--chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
--chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --chroot-setup-commands="${locked_update_cmd}" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="cp -Ln --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+ --finished-build-commands="mkdir -p ${ext_root}" \
--finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index b48bcec8..aa4874c1 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
# This class extends the image.bbclass for setting locales and purging unneeded
# ones.
+inherit debrepo
+
LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
en_US ISO-8859-1\n"
LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
image_install_localepurge_download[weight] = "40"
image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
image_install_localepurge_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
}
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 7c84505f..f98cd759 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -6,6 +6,7 @@
# This file extends the image.bbclass to supply tools for futher imager functions
inherit buildchroot
+inherit debrepo
IMAGER_INSTALL ??= ""
IMAGER_BUILD_DEPS ??= ""
@@ -21,14 +22,22 @@ do_install_imager_deps() {
fi
distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
fi
buildchroot_do_mounts
E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${BUILDCHROOT_DIR} ${distro}
+
+ debrepo_add_packages "${debrepo_workdir}" "${IMAGER_INSTALL}"
+ debrepo_update_apt_source_list "${BUILDCHROOT_DIR}" "base-apt"
+
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get update \
-o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cedde239..47f07129 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -2,6 +2,7 @@
# Copyright (c) Siemens AG, 2020
inherit deb-dl-dir
+inherit debrepo
ROOTFS_ARCH ?= "${DISTRO_ARCH}"
ROOTFS_DISTRO ?= "${DISTRO}"
@@ -148,6 +149,9 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
rootfs_install_pkgs_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 09/12] meta: Add cache-deb-src functionality in base-apt mode
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (7 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 08/12] base-apt: Predownload packages to base-apt before install Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 10/12] Set ISAR_PREFETCH_BASE_APT by default Uladzimir Bely
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
Fill base-apt repo with source packages.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 17 +++++++++++++++++
meta/classes/rootfs.bbclass | 4 ++++
2 files changed, 21 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index f0ab9714..a7a8dcf7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -43,6 +43,23 @@ debsrc_undo_mounts() {
EOSUDO
}
+debsrc_fill_base_apt() {
+ export rootfs="$1"
+
+ local srcpkgs=""
+ for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do
+ is_not_part_of_current_build "${package}" && continue
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+ local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
+ [ -n "$dscfile" ] && continue
+
+ srcpkgs="${srcpkgs} ${src}=${version}"
+ done
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}"
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 47f07129..0ada4721 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -230,8 +230,12 @@ cache_deb_src() {
sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ debsrc_fill_base_apt ${ROOTFSDIR}
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ] ||
[ -h "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 10/12] Set ISAR_PREFETCH_BASE_APT by default
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (8 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 09/12] meta: Add cache-deb-src functionality in base-apt mode Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 11/12] meta: Specify grub-efi packages arch Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 12/12] Disable deb-dl-dir in base-apt prefetch mode Uladzimir Bely
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
This makes Isar use `base-apt` repo in different way. Any package
installation is done from `base-apt` repo, which is prepopulated
from the external mirrors.
This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 9 +++++++++
meta-isar/conf/local.conf.sample | 4 ++++
2 files changed, 13 insertions(+)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7a16b5c2..ae845674 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -492,3 +492,12 @@ Recipes inheriting dpkg-base now automatically have a bitbake target
exists for the current `DISTRO_ARCH`.
In that case the compat package can be built by adding `<foo>-compat`
to `DEPENDS` or `IMAGE_INSTALL`.
+
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before any step they are requested.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 7e0184e4..2d3c9c1e 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -215,6 +215,10 @@ ISAR_CROSS_COMPILE ?= "0"
# does not access the network
#BB_NO_NETWORK ?= "1"
+# Comment out for switching to old base-apt behaviour when it's populated
+# at the beginnig of the second build
+ISAR_PREFETCH_BASE_APT ?= "1"
+
# Set root password to 'root'
# Password was encrypted using following command:
# mkpasswd -m sha512crypt -R 10000
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 11/12] meta: Specify grub-efi packages arch
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (9 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 10/12] Set ISAR_PREFETCH_BASE_APT by default Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
2023-05-26 7:00 ` [PATCH v5 12/12] Disable deb-dl-dir in base-apt prefetch mode Uladzimir Bely
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
When using python-apt, it can't mark it for downloading when used
without ":<arch>" field, so we miss the package in base-apt repo.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/conf/distro/debian-common.conf | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 5610dcd4..ee84b7dd 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -23,10 +23,10 @@ WIC_IMAGER_INSTALL = "parted \
tar \
fdisk"
-GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin"
-GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin"
-GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin"
-GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin"
+GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin:amd64"
+GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin:i386"
+GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin:armhf"
+GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin:arm64"
GRUB_DEBIAN_SB_CHAIN:amd64 = "grub-efi-amd64-signed shim-signed"
GRUB_DEBIAN_SB_MOK:amd64 = "shim-helpers-amd64-signed"
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v5 12/12] Disable deb-dl-dir in base-apt prefetch mode
2023-05-26 7:00 [PATCH v5 00/12] Improving base-apt usage PoC Uladzimir Bely
` (10 preceding siblings ...)
2023-05-26 7:00 ` [PATCH v5 11/12] meta: Specify grub-efi packages arch Uladzimir Bely
@ 2023-05-26 7:00 ` Uladzimir Bely
11 siblings, 0 replies; 13+ messages in thread
From: Uladzimir Bely @ 2023-05-26 7:00 UTC (permalink / raw)
To: isar-users
Since all packages and source packages are placed to base-apt repo
during build, there is no need to have them in one more place.
Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index a7a8dcf7..54110cdb 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -87,6 +87,10 @@ debsrc_download() {
}
deb_dl_dir_import() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ return
+ fi
+
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -104,6 +108,10 @@ EOSUDO
}
deb_dl_dir_export() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ return
+ fi
+
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
export owner=$(id -u):$(id -g)
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread