[PATCH v5 00/12] Improving base-apt usage PoC

[PATCH v5 00/12] Improving base-apt usage PoC

[Uladzimir Bely]

Currently, base-apt is used in the following way:

 - After the first build, all .deb files that took part in the process
are is cached in the ${DL_DIR}/deb/. All the packages are downloaded
from the remote repositories.

 - At the second build, `base-apt` repo is first created from the
previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set).
Futher debootstrap and installing packages are done from this local repo

The idea of this patchset is using local `base-apt` repo even at
the first build. Before any build step that requires additional
packages, these packages are predownloaded with `debrepo` script to
the `base-apt` repo and the debootstrapping / installation are always
done from it.

Potentialy, such an approach allows to stop using deb-del-dir import
and export functionality in favour of debian-like repositories.

Currently, separate `debrepo` script is used for prefetching packages
to the local 'base-apt' repo. It requires python3-apt to be installed
on the build host. Potentially, the functionality it provides could be
directly integrated to the `debrepo.bbclass`.

Changes since v4:
 - rebased on latest next;
 - rearranged patches since some of them are already in next;
 - added possibility to select between new "prefetch" base-apt mode and
old behaviour when it's populated on 2nd build with packages downloaded
during 1st build. New behaviour is disabled by default, but enabled in
local.conf.example for testing purposes.
 - code passes both full and fast CI in both "old" and "new" modes.

Changes since v3:
 - rebased on latest next;
 - cross-building for raspberry supported;
 - code passes both full and fast CI.

Changes since v2:
 - populate base-apt before using at all steps of native build.

Changes since v1:
 - rebased on latest next;
 - updated patchset description.

Uladzimir Bely (12):
  meta: move base-apt from deploy directory to the top
  ci_build.sh: Install python3-apt if not installed
  scripts: Add debrepo python script handling base-apt
  meta: Add debrepo bbclass handling base-apt prefetching
  meta: Always use base-apt repo in local mode
  meta: Use cached base-apt repo to debootstrap
  meta: Consider global debrepo context
  base-apt: Predownload packages to base-apt before install
  meta: Add cache-deb-src functionality in base-apt mode
  Set ISAR_PREFETCH_BASE_APT by default
  meta: Specify grub-efi packages arch
  Disable deb-dl-dir in base-apt prefetch mode

 RECIPE-API-CHANGELOG.md                       |   9 +
 meta-isar/conf/local.conf.sample              |   4 +
 meta/classes/buildchroot.bbclass              |   5 +-
 meta/classes/deb-dl-dir.bbclass               |  25 +
 meta/classes/debrepo.bbclass                  |  86 ++++
 meta/classes/dpkg-base.bbclass                |  47 +-
 meta/classes/dpkg.bbclass                     |   8 +
 meta/classes/image-locales-extension.bbclass  |   5 +
 meta/classes/image-tools-extension.bbclass    |   9 +
 meta/classes/rootfs.bbclass                   |  13 +-
 meta/conf/bitbake.conf                        |   9 +-
 meta/conf/distro/debian-common.conf           |   8 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 111 ++++-
 meta/recipes-devtools/base-apt/base-apt.bb    |  21 +-
 .../buildchroot/buildchroot.inc               |  11 +
 .../sbuild-chroot/sbuild-chroot.inc           |  11 +
 scripts/ci_build.sh                           |   8 +-
 scripts/debrepo                               | 443 ++++++++++++++++++
 testsuite/citest.py                           |   3 +
 19 files changed, 804 insertions(+), 32 deletions(-)
 create mode 100644 meta/classes/debrepo.bbclass
 create mode 100755 scripts/debrepo

-- 
2.20.1

[PATCH v5 01/12] meta: move base-apt from deploy directory to the top

[Uladzimir Bely]

While base-apt is supposed to be always used for deboostrapping and
installing build dependencies, move it out of temporary directory.

This also allows not to change some of CI tests that completely
remove TMPDIR and underlaying DEPLOY_DIR.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/conf/bitbake.conf | 4 ++--
 testsuite/citest.py    | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index a85fea77..89b3e6a3 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -97,8 +97,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
 THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
 
 # Base apt repository paths
-REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt"
-REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/db"
+REPO_BASE_DIR = "${TOPDIR}/base-apt/${DISTRO}/apt"
+REPO_BASE_DB_DIR = "${TOPDIR}/base-apt/${DISTRO}/db"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 17a90244..2393e275 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -86,9 +86,11 @@ class ReproTest(CIBaseTest):
 
         self.init()
         try:
+            self.delete_from_build_dir('base-apt')
             self.perform_repro_test(targets, signed=True)
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_signed')
+            self.move_in_build_dir('base-apt', 'base-apt_repro_signed')
 
     def test_repro_unsigned(self):
         targets = [
@@ -101,6 +103,7 @@ class ReproTest(CIBaseTest):
             self.perform_repro_test(targets, cross=False)
         finally:
             self.move_in_build_dir('tmp', 'tmp_repro_unsigned')
+            self.move_in_build_dir('base-apt', 'base-apt_repro_unsigned')
 
 class CcacheTest(CIBaseTest):
 
-- 
2.20.1

[PATCH v5 02/12] ci_build.sh: Install python3-apt if not installed

[Uladzimir Bely]

This is mostly related to gitlab CI that migth use an image without
preinstalled python3-apt.

Also, make system python packages available in virtualenv.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/ci_build.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 2fa5cccd..ff7c114a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
     sudo apt-get update -qq
     sudo apt-get install -y virtualenv
     rm -rf /tmp/avocado_venv
-    virtualenv --python python3 /tmp/avocado_venv
+    virtualenv --python python3 /tmp/avocado_venv --system-site-packages
     # shellcheck source=/dev/null
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==100.1
@@ -127,6 +127,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
     fi
 fi
 
+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+    sudo apt-get update -qq
+    sudo apt-get install -y python3-apt
+fi
+
 # Provide working path
 mkdir -p .config/avocado
 cat <<EOF > .config/avocado/avocado.conf
-- 
2.20.1

[PATCH v5 03/12] scripts: Add debrepo python script handling base-apt

[Uladzimir Bely]

This is the main utility responsible for prefetching packages
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").

Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.

In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 scripts/debrepo | 443 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 443 insertions(+)
 create mode 100755 scripts/debrepo

diff --git a/scripts/debrepo b/scripts/debrepo
new file mode 100755
index 00000000..2a5e5b51
--- /dev/null
+++ b/scripts/debrepo
@@ -0,0 +1,443 @@
+#!/usr/bin/env python3
+
+# This software is a part of ISAR.
+# Copyright (C) 2022 ilbers GmbH
+
+import os
+import sys
+import fcntl
+
+import shutil
+import subprocess
+import getopt
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+class DebRepo(object):
+    def __init__(self, workdir, cmdline_opts):
+        self.workdir = workdir
+        self.optsfile = self.workdir + "/repo.opts"
+
+        # Set default values
+        self.distro = "debian"
+
+        self.repo = self.workdir + "/repo/apt" + "/" + self.distro
+        self.repodb = self.workdir + "/repo/db" + "/" + self.distro
+        self.mirror = "http://deb.debian.org/debian"
+        self.arch = "amd64"
+        self.codename = "bullseye"
+        self.keydir = "/etc/apt/trusted.gpg.d"
+        self.check_gpg = True
+        self.isaraptdir = ""
+
+        # Load stored opts
+        opts = self.load_opts()
+        print("stored opts: " + str(opts))
+
+        # Overwrite opts by cmdline_opts
+        for opt, arg in cmdline_opts.items():
+            opts[opt] = arg
+
+        print("all opts: " + str(opts))
+
+        # Replace by values passed in commandline
+        for opt, arg in opts.items():
+            if opt == "mirror":
+                self.mirror = arg
+            if opt == "arch":
+                self.arch = arg
+            if opt == "distro":
+                self.distro = arg
+            if opt == "codename":
+                self.codename = arg
+            if opt == "keydir":
+                self.keydir = arg
+            if opt == "isaraptdir":
+                self.isaraptdir = arg
+            if opt == "check_gpg":
+                self.check_gpg = arg
+
+        self.crossarch = self.arch
+        for opt, arg in opts.items():
+            if opt == "crossarch":
+                self.crossarch = arg
+
+        self.compatarch = ""
+        for opt, arg in opts.items():
+            if opt == "compatarch":
+                self.compatarch = arg
+
+        for opt, arg in opts.items():
+            if opt == "repodir":
+                self.repo = arg + "/" + self.distro
+            if opt == "repodbdir":
+                self.repodb = arg + "/" + self.distro
+
+        self.save_opts(opts)
+
+        print("workdir:     " + str(self.workdir))
+        print("repo:        " + str(self.repo))
+        print("repodb:      " + str(self.repodb))
+        print("mirror:      " + str(self.mirror))
+        print("arch:        " + str(self.arch))
+        print("crossarch:   " + str(self.crossarch))
+        if self.compatarch:
+            print("compatarch:  " + str(self.compatarch))
+        print("distro:      " + str(self.distro))
+        print("codename:    " + str(self.codename))
+        print("keydir:      " + str(self.keydir))
+        print("isaraptdir:  " + str(self.isaraptdir))
+        print("check_gpg:   " + str(self.check_gpg))
+
+        self.cache = None
+        self.depcache = None
+        self.sr = None
+
+    def create_rootfs(self, aptsrcsfile):
+        if not os.path.exists(self.workdir + "/var/lib/dpkg"):
+            os.makedirs(self.workdir + "/var/lib/dpkg")
+        with open(self.workdir + "/var/lib/dpkg" + "/status", "w"):
+            pass
+
+        if not os.path.exists(self.workdir + "/etc/apt/sources.list.d"):
+            os.makedirs(self.workdir + "/etc/apt/sources.list.d")
+        if os.path.exists(aptsrcsfile):
+            shutil.copy(aptsrcsfile, self.workdir + "/etc/apt/sources.list.d/bootstrap.list")
+
+        if self.isaraptdir:
+            with open(self.workdir + "/etc/apt/sources.list.d/isar-apt.list", "w") as f:
+                f.write("deb [trusted=yes] file://" + self.isaraptdir + " isar main\n")
+
+        if not os.path.exists(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial"):
+            os.makedirs(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial")
+
+        if not os.path.exists(self.workdir + "/tmp"):
+            os.makedirs(self.workdir + "/tmp")
+
+    def apt_config(self, init, crossbuild):
+        # Configure apt to work with empty directory
+        if not init and self.arch != self.crossarch:
+            apt_pkg.config["APT::Architectures::"] = self.crossarch
+            apt_pkg.config["APT::Architectures::"] = self.arch
+
+        if not init and self.compatarch:
+            apt_pkg.config["APT::Architectures::"] = self.compatarch
+
+        apt_pkg.config.set("APT::Architecture", self.arch)
+
+        apt_pkg.config.set("Dir", self.workdir)
+        apt_pkg.config.set("Dir::Cache", self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename)
+        apt_pkg.config.set("Dir::State::status", self.workdir + "/var/lib/dpkg/status")
+
+        apt_pkg.config.set("APT::Install-Recommends", "0")
+        apt_pkg.config.set("APT::Install-Suggests", "0")
+
+        # Use host keys for authentification
+        # apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg")
+        # apt_pkg.config.set("Dir::Etc::TrustedParts", "/etc/apt/trusted.gpg.d")
+        apt_pkg.config.set("Dir::Etc::TrustedParts", self.keydir)
+
+        # Allow using repositories without keys
+        if not self.check_gpg:
+            apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+    def mark_essential(self):
+        for pkg in self.cache.packages:
+            if pkg.architecture == self.arch:
+                if pkg.essential:
+                    self.depcache.mark_install(pkg)
+
+    def mark_by_prio(self, priority):
+        for pkg in self.cache.packages:
+            if pkg.architecture == self.arch:
+                ver = self.depcache.get_candidate_ver(pkg)
+                if ver and ver.priority <= priority:
+                    self.depcache.mark_install(pkg)
+
+    def mark_pkg(self, pkgname, crossbuild):
+        if pkgname in self.cache:
+            pkg = self.cache[pkgname]
+
+            if not crossbuild or ':' in pkgname or len(pkg.version_list) == 0:
+                    if pkg.has_provides and not pkg.has_versions:
+                        print("pkgname is virtual package, selecting best provide")
+                        # Select first provide
+                        pkg_provide = pkg.provides_list[0][2]
+                        # Find better provide with higher version
+                        for provide in pkg.provides_list:
+                            if apt_pkg.version_compare(provide[2].ver_str, pkg_provide.ver_str) > 0:
+                                pkg_provide = provide[2]
+                        self.depcache.mark_install(pkg_provide.parent_pkg)
+                    else:
+                        self.depcache.mark_install(pkg)
+            else:
+                version = pkg.version_list[0]
+                if version.arch == "all":
+                    self.depcache.mark_install(pkg)
+                else:
+                    if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+                        if (pkgname, self.arch) in self.cache:
+                            nativepkg = self.cache[pkgname, self.arch]
+                            self.depcache.mark_install(nativepkg)
+                    else:
+                        if (pkgname, self.crossarch) in self.cache:
+                            crosspkg = self.cache[pkgname, self.crossarch]
+                            self.depcache.mark_install(crosspkg)
+
+    def mark_list(self, pkglist, crossbuild):
+        if pkglist:
+            for pkgname in pkglist:
+                self.mark_pkg(pkgname, crossbuild)
+
+    def handle_deb(self, item):
+        lockfd = open(self.repo + '/../repo.lock', 'w')
+        fcntl.flock(lockfd,fcntl.LOCK_EX)
+        subprocess.run([
+            "reprepro",
+            "--dbdir", self.repodb,
+            "--outdir", self.repo,
+            "--confdir", self.repo + "/conf",
+            "-C", "main",
+            "includedeb",
+            self.codename,
+            item.destfile
+            ])
+        lockfd.close()
+
+    def handle_repo(self, fetcher):
+        lockfd = open(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + ".lock", "w")
+        fcntl.flock(lockfd,fcntl.LOCK_EX)
+        fetcher.run()
+        lockfd.close()
+        for item in fetcher.items:
+            if item.status == item.STAT_ERROR:
+                print("Some error ocured: '%s'" % item.error_text)
+                pass
+            else:
+                self.handle_deb(item)
+
+    def get_filename(self, uri):
+        path = urllib.parse.urlparse(uri).path
+        unquoted_path = urllib.parse.unquote(path)
+        basename = os.path.basename(unquoted_path)
+        return basename
+
+    def download_file(self, uri):
+        filename = self.get_filename(uri)
+        subprocess.run([
+            "wget",
+            "-H",
+            "--timeout=30",
+            "--tries=3",
+            "-q",
+            uri,
+            "-O",
+            self.workdir + "/tmp/" + filename
+        ])
+
+    def handle_dsc(self, uri):
+        filename = self.get_filename(uri)
+        lockfd = open(self.repo + '/../repo.lock', 'w')
+        fcntl.flock(lockfd,fcntl.LOCK_EX)
+        subprocess.run([
+            "reprepro",
+            "--dbdir", self.repodb,
+            "--outdir", self.repo,
+            "--confdir", self.repo + "/conf",
+            "-C", "main",
+            "-S", "-", "-P" "source",
+            "--delete",
+            "includedsc",
+            self.codename,
+            os.path.realpath(self.workdir + "/tmp/" + filename)
+            ])
+        lockfd.close()
+
+    def handle_src_list(self, pkgs):
+        if pkgs:
+            for pkg in pkgs:
+                pkgname=pkg
+                pkgver=""
+                if '=' in pkg:
+                    pkgname = pkg.split("=")[0]
+                    pkgver = pkg.split("=")[1]
+
+                self.sr.restart()
+                while self.sr.lookup(pkgname):
+                    if pkgver and pkgver != self.sr.version:
+                        continue
+
+                    for sr_file in self.sr.files:
+                        print(self.sr.index.archive_uri(sr_file[2]))
+                        self.download_file(self.sr.index.archive_uri(sr_file[2]))
+
+                    dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2])
+                    self.handle_dsc(dsc_uri)
+                    break
+
+    def apt_run(self, init, srcmode, pkgs, controlfile, crossbuild):
+        apt_pkg.init()
+
+        sources = apt_pkg.SourceList()
+        sources.read_main_list()
+
+        progress = apt.progress.text.AcquireProgress()
+
+        self.cache = apt_pkg.Cache()
+        self.cache.update(progress, sources)
+        self.cache = apt_pkg.Cache()
+
+        self.depcache = apt_pkg.DepCache(self.cache)
+        self.sr = apt_pkg.SourceRecords()
+
+        if init:
+            self.mark_essential()
+            # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+            self.mark_by_prio(1)
+
+        if srcmode:
+            self.handle_src_list(pkgs)
+        else:
+            self.mark_list(pkgs, crossbuild)
+
+        if controlfile:
+            fobj = open(controlfile, "r")
+
+            try:
+                tagfile = apt_pkg.TagFile(fobj)
+                while tagfile.step() == 1:
+                    deps = tagfile.section.get("Build-Depends", "")
+                    # Remove extra commas and spaces - apt_pkg.parse_depends doesnt like
+                    # lines like ", device-tree-compiler"
+                    deps = ', '.join([s.strip() for s in deps.split(',') if s.strip()])
+                    print("parsed deps: " + str(deps))
+                    for item in apt_pkg.parse_depends(deps, False):
+                        pkgname = item[0][0]
+                        self.mark_pkg(pkgname, crossbuild)
+
+            finally:
+                fobj.close()
+
+        if init or not srcmode:
+            fetcher = apt_pkg.Acquire(progress)
+            pm = apt_pkg.PackageManager(self.depcache)
+
+            recs = apt_pkg.PackageRecords(self.cache)
+            pm.get_archives(fetcher, sources, recs)
+
+            self.handle_repo(fetcher)
+
+    def load_opts(self):
+        params = {}
+        if os.path.isfile(self.optsfile):
+            with open(self.optsfile, 'rb') as file:
+                data = file.read()
+                if data:
+                    params = pickle.loads(data)
+
+        return params
+
+    def save_opts(self, opts):
+        file = open(self.optsfile, 'wb')
+        pickle.dump(opts, file)
+        file.close()
+
+
+class DebRepoArgs(object):
+    def __init__(self):
+        self.workdir = ""
+        self.init = False
+        self.srcmode = False
+        self.controlfile = ""
+        self.aptsrcsfile = ""
+        self.crossbuild = False
+
+        self.opts = {}
+        self.pkgs = []
+
+        try:
+            opts, args = getopt.getopt(sys.argv[1:], "", [
+                "init",
+                "srcmode",
+                "workdir=",
+                "repodir=",
+                "repodbdir=",
+                "mirror=",
+                "arch=",
+                "crossarch=",
+                "compatarch=",
+                "distro=",
+                "codename=",
+                "keydir=",
+                "isaraptdir=",
+                "no-check-gpg",
+                "controlfile=",
+                "aptsrcsfile=",
+                "crossbuild"
+                ])
+        except getopt.GetoptError as msg:
+            print("Error: " + str(msg))
+            sys.exit(1)
+
+        for opt, arg in opts:
+            if opt in ("--workdir"):
+                self.workdir = arg
+            if opt in ("--init"):
+                self.init = True
+            if opt in ("--srcmode"):
+                self.srcmode = True
+            if opt in ("--controlfile"):
+                self.controlfile = arg
+            if opt in ("--aptsrcsfile"):
+                self.aptsrcsfile = arg
+            if opt in ("--crossbuild"):
+                self.crossbuild = True
+
+            if opt in ("--repodir",
+                       "--repodbdir",
+                       "--mirror",
+                       "--arch",
+                       "--crossarch",
+                       "--compatarch",
+                       "--distro",
+                       "--codename",
+                       "--keydir",
+                       "--isaraptdir",
+                       "--controlfile",
+                       ):
+                self.opts[opt[2:]] = arg
+            if opt in ("--no-check-gpg"):
+                self.opts['check_gpg'] = False
+
+        if not self.workdir:
+            print("Error: workdir is not specified")
+            sys.exit(1)
+
+        self.pkgs = args
+
+
+def main():
+    args = DebRepoArgs()
+
+    if not (args.init or args.pkgs or args.controlfile):
+        print("Nothing to do")
+        sys.exit(0)
+
+    if not os.path.exists(args.workdir):
+        os.makedirs(args.workdir)
+
+    debrepo = DebRepo(args.workdir, args.opts)
+
+    if (args.init):
+        debrepo.create_rootfs(args.aptsrcsfile)
+
+    debrepo.apt_config(args.init, args.crossbuild)
+    debrepo.apt_run(args.init, args.srcmode, args.pkgs, args.controlfile, args.crossbuild)
+
+
+if __name__ == "__main__":
+    main()
-- 
2.20.1

[PATCH v5 04/12] meta: Add debrepo bbclass handling base-apt prefetching

[Uladzimir Bely]

This class uses 'scripts/debrepo' python script to prefetch given
packages or sources to local base-apt repository.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/debrepo.bbclass | 86 ++++++++++++++++++++++++++++++++++++
 meta/conf/bitbake.conf       |  5 +++
 2 files changed, 91 insertions(+)
 create mode 100644 meta/classes/debrepo.bbclass

diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..8f2a3215
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,86 @@
+DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+        return
+    fi
+
+    chroot_dir=${1}
+    apt_list=${2}
+
+    flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+        sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+            -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+            -o Dir::Etc::SourceParts=\"-\" \
+            -o APT::Get::List-Cleanup=\"0\"
+    "
+}
+
+debrepo_add_packages() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+        return
+    fi
+
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        return
+    fi
+
+    args=""
+    if [ "${1}" = "--srcmode" ]; then
+        args="${args} --srcmode"
+        shift
+    fi
+
+    workdir="${1}"
+    pkgs="${2}"
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+
+    flock -x "${workdir}/repo.lock" -c "
+        ${SCRIPTSDIR}/debrepo \
+            ${args} \
+            --workdir=\"${workdir}\" \
+            ${pkgs}
+        "
+}
+
+debrepo_handle_controlfile() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+        return
+    fi
+
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        return
+    fi
+
+    control_file="${1}"
+
+    args=""
+    pkgs=""
+
+    build_arch=${DISTRO_ARCH}
+    if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+        build_arch=${HOST_ARCH}
+    fi
+    if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+        args="--crossbuild"
+        pkgs="${pkgs} crossbuild-essential-${PACKAGE_ARCH}:${build_arch}"
+        pkgs="${pkgs} dose-distcheck:${build_arch}"
+        pkgs="${pkgs} libc-dev:${PACKAGE_ARCH}"
+        pkgs="${pkgs} libstdc++-dev:${PACKAGE_ARCH}"
+    fi
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+
+    flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "
+        ${SCRIPTSDIR}/debrepo \
+            --workdir=\"${DEBREPO_WORKDIR}\" \
+            --controlfile=\"${control_file}\" \
+            ${args} \
+            ${pkgs}
+    "
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 89b3e6a3..91f81bd4 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -71,6 +71,11 @@ KERNEL_FILE:mipsel ?= "vmlinux"
 KERNEL_FILE:riscv64 ?= "vmlinux"
 KERNEL_FILE:arm64 ?= "vmlinux"
 
+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
 OVERRIDES = "${PACKAGE_ARCH}:${MACHINE}:${DISTRO}:${BASE_DISTRO_CODENAME}:forcevariable"
 FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"
 
-- 
2.20.1

[PATCH v5 05/12] meta: Always use base-apt repo in local mode

[Uladzimir Bely]

This means only local URLs in apt sources.list* are present during
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/buildchroot.bbclass              |  5 +++--
 meta/classes/rootfs.bbclass                   |  5 +++--
 .../isar-bootstrap/isar-bootstrap.inc         | 12 +++++++----
 meta/recipes-devtools/base-apt/base-apt.bb    | 21 ++++++++++++-------
 4 files changed, 28 insertions(+), 15 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e4a7a571..947529b0 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -38,8 +38,9 @@ buildchroot_do_mounts() {
             mount --rbind /sys '${BUILDCHROOT_DIR}/sys'
         mount --make-rslave '${BUILDCHROOT_DIR}/sys'
 
-        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
-        if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+        # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+        if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+           [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
         then
             mkdir -p '${BUILDCHROOT_DIR}/base-apt'
             mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 22449d71..cedde239 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -54,8 +54,9 @@ rootfs_do_mounts() {
                 mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
         fi
 
-        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
-        if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+        # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+        if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+           [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
         then
             mkdir -p '${ROOTFSDIR}/base-apt'
             mountpoint -q '${ROOTFSDIR}/base-apt' || \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b94ae0bd..0cfe82ab 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -42,7 +42,8 @@ python () {
     # installation afterwards. However, debootstrap will include the key into
     # the rootfs automatically thus the right place is distro_bootstrap_keys.
 
-    if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+    if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+       bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
         own_pub_key = d.getVar("BASE_REPO_KEY")
         if own_pub_key:
             distro_bootstrap_keys += own_pub_key.split()
@@ -110,7 +111,8 @@ def parse_aptsources_list_line(source_list_line):
 def get_apt_source_mirror(d, aptsources_entry_list):
     import re
 
-    if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+    if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+       bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
         premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
     else:
         premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
@@ -286,7 +288,8 @@ do_bootstrap() {
     if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
         debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
     fi
-    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \
+       [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
         debootstrap_args="$debootstrap_args --no-check-gpg"
     fi
     E="${@ isar_export_proxies(d)}"
@@ -313,7 +316,8 @@ do_bootstrap() {
         install -v -m644 "${APTPREFS}" \
                          "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
         mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
-        if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+           [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
             line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
             if [ -z "${BASE_REPO_KEY}" ]; then
                 line="[trusted=yes] ${line}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index ea885fe6..e62958d1 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -57,9 +57,12 @@ repo() {
         "${BASE_DISTRO_CODENAME}" \
         "${WORKDIR}/distributions.in" \
         "${KEYFILES}"
-    populate_base_apt "${BASE_DISTRO}"
-    repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
-        "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+       [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+        populate_base_apt "${BASE_DISTRO}"
+        repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+    fi
 
     if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
         repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -67,14 +70,18 @@ repo() {
             "${BASE_DISTRO_CODENAME}" \
             "${WORKDIR}/distributions.in" \
             "${KEYFILES}"
-        populate_base_apt "${HOST_BASE_DISTRO}"
-        repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
-            "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+        if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+           [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+            populate_base_apt "${HOST_BASE_DISTRO}"
+            repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+                "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+        fi
     fi
 }
 
 python do_cache() {
-    if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+    if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+       not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         return 0
 
     for key in d.getVar('BASE_REPO_KEY').split():
-- 
2.20.1

[PATCH v5 06/12] meta: Use cached base-apt repo to debootstrap

[Uladzimir Bely]

This patch makes local base-apt repo to be created before
debootstrap task. So, debootstrap is then done from it.

The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.

For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.

The goal is have workable base-apt repo before first build completed.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 .../isar-bootstrap/isar-bootstrap.inc         | 99 +++++++++++++++++--
 1 file changed, 91 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0cfe82ab..e1ddac42 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -29,6 +29,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
 DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
 BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
 BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+
 FILESEXTRAPATHS:append = ":${BBPATH}"
 
 inherit deb-dl-dir
@@ -278,12 +280,73 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 
 inherit compat
 
+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+        return
+    fi
+
+    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        return
+    fi
+
+    debrepo_args=""
+    if [ "${BASE_DISTRO}" != "debian" ]; then
+        if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+            debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+        fi
+    else
+        if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+            debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+        fi
+    fi
+    #if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+        debrepo_args="$debrepo_args --no-check-gpg"
+    #fi
+    if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+        debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+    fi
+
+    debrepo_workdir=${DEBREPO_TARGET_DIR}
+    if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+        debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+        if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+            debrepo_workdir=${DEBREPO_HOST_DIR}
+        fi
+    else
+        debrepo_args="$debrepo_args --isaraptdir=${REPO_ISAR_DIR}/${DISTRO}"
+    fi
+    mkdir -p "${DEBREPO_WORKDIR}"
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+
+    flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "
+        ${SCRIPTSDIR}/debrepo --init \
+            --workdir=\"${debrepo_workdir}\" \
+            --aptsrcsfile=\"${APTSRCS_INIT}\" \
+            --repodir=\"${REPO_BASE_DIR}\" \
+            --repodbdir=\"${REPO_BASE_DB_DIR}\" \
+            --mirror=\"${@get_distro_source(d)}\" \
+            --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \
+            --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \
+            --codename=\"${BASE_DISTRO_CODENAME}\" \
+            ${debrepo_args} \
+            gnupg locales usr-is-merged
+    "
+}
+
 do_bootstrap() {
     if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
         if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
             bbfatal "${DISTRO_ARCH} does not have a compat arch"
         fi
     fi
+
+    debrepo_bootstrap_prepare
+
     debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
     if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
         debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
@@ -301,8 +364,19 @@ do_bootstrap() {
     sudo -E -s <<'EOSUDO'
         set -e
         if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
-            arch_param="--arch=${DISTRO_ARCH}"
+            arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}"
         fi
+        if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+            flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+                ${DEBOOTSTRAP} $debootstrap_args \
+                            $arch_param \
+                            ${@get_distro_components_argument(d)} \
+                            ${@get_distro_suite(d)} \
+                            ${ROOTFSDIR} \
+                            file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \
+                            ${DISTRO_DEBOOTSTRAP_SCRIPT}
+            "
+        else
         ${DEBOOTSTRAP} $debootstrap_args \
                        $arch_param \
                        ${@get_distro_components_argument(d)} \
@@ -310,7 +384,7 @@ do_bootstrap() {
                        "${ROOTFSDIR}" \
                        "${@get_distro_source(d)}" \
                        ${DISTRO_DEBOOTSTRAP_SCRIPT}
-
+        fi
         # Install apt config
         mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
         install -v -m644 "${APTPREFS}" \
@@ -382,12 +456,21 @@ do_bootstrap() {
         if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
             chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
         fi
-
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
-                                -o APT::Update::Error-Mode=any
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
-        chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-                                -o Debug::pkgProblemResolver=yes
+        if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+            flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+                chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+                                        -o APT::Update::Error-Mode=any
+                chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+                chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+                                        -o Debug::pkgProblemResolver=yes
+            "
+        else
+            chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+                                    -o APT::Update::Error-Mode=any
+            chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+            chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+                                    -o Debug::pkgProblemResolver=yes
+        fi
 
         umount -l "${ROOTFSDIR}/dev/shm"
         umount -l "${ROOTFSDIR}/dev/pts"
-- 
2.20.1

[PATCH v5 07/12] meta: Consider global debrepo context

[Uladzimir Bely]

For host/target and cross/native modes different debrepo context
(e.g., working dir) should be used.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                        | 11 +++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.inc     | 11 +++++++++++
 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 11 +++++++++++
 3 files changed, 33 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e3412c82..c3e47a69 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -12,6 +12,17 @@ inherit repository
 inherit deb-dl-dir
 inherit essential
 
+python __anonymous() {
+    distro_arch = d.getVar('DISTRO_ARCH')
+    host_arch = d.getVar('HOST_ARCH')
+    cross = d.getVar('ISAR_CROSS_COMPILE', True)
+
+    if cross == "0" or host_arch == distro_arch:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+    else:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
 DEPENDS ?= ""
 RPROVIDES ?= "${PROVIDES}"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index f74896fb..320df55a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -13,6 +13,17 @@ SRC_URI = "file://configscript.sh \
            file://deps.sh"
 PV = "1.0"
 
+python __anonymous() {
+    distro_arch = d.getVar('DISTRO_ARCH')
+    host_arch = d.getVar('HOST_ARCH')
+    variant = d.getVar('BUILDCHROOT_VARIANT', True)
+
+    if variant == "target" or host_arch == distro_arch:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+    else:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
 inherit rootfs
 
 BUILDCHROOT_DIR = "${WORKDIR}/rootfs"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index fd8bb648..03f812c9 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -8,6 +8,17 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260
 
 PV = "1.0"
 
+python __anonymous() {
+    distro_arch = d.getVar('DISTRO_ARCH')
+    host_arch = d.getVar('HOST_ARCH')
+    variant = d.getVar('SBUILD_VARIANT', True)
+
+    if variant == "target" or host_arch ==  distro_arch:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+    else:
+        d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
 inherit rootfs
 inherit compat
 
-- 
2.20.1

[PATCH v5 08/12] base-apt: Predownload packages to base-apt before install

[Uladzimir Bely]

This patch uses debrepo script to predownload packages to base-apt
repository before they are installed in buildchroot/rootfs.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg-base.bbclass               | 36 ++++++++++++++++++--
 meta/classes/dpkg.bbclass                    |  8 +++++
 meta/classes/image-locales-extension.bbclass |  5 +++
 meta/classes/image-tools-extension.bbclass   |  9 +++++
 meta/classes/rootfs.bbclass                  |  4 +++
 5 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c3e47a69..faf2ec77 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -23,6 +23,8 @@ python __anonymous() {
         d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
 }
 
+inherit debrepo
+
 DEPENDS ?= ""
 RPROVIDES ?= "${PROVIDES}"
 
@@ -126,10 +128,25 @@ do_apt_fetch() {
     trap 'exit 1' INT HUP QUIT TERM ALRM USR1
     trap 'schroot_cleanup' EXIT
 
+    # Begin chroot session
+    session_id=$(schroot -b -c ${SBUILD_CHROOT})
+
+    debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}"
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+        flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+        schroot -d / -r -c ${session_id} -u root -- \
+            sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+        "
+    fi
+
     for uri in "${SRC_APT}"; do
-        schroot -d / -c ${SBUILD_CHROOT} -- \
+        schroot -d / -r -c ${session_id} -- \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
     done
+
+    # End chroot session
+    schroot -e -c ${session_id}
+
     schroot_delete_configs
 }
 
@@ -139,6 +156,8 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 
 # Add dependency from the correct schroot: host or target
 do_apt_fetch[depends] += "${SCHROOT_DEP}"
+# Debrepo context is created by target bootstrap, need this dependency too
+do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap"
 
 do_apt_unpack() {
     rm -rf ${S}
@@ -150,8 +169,18 @@ do_apt_unpack() {
     trap 'exit 1' INT HUP QUIT TERM ALRM USR1
     trap 'schroot_cleanup' EXIT
 
+    # Begin chroot session
+    session_id=$(schroot -b -c ${SBUILD_CHROOT})
+
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+        flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+        schroot -d / -r -c ${session_id} -u root -- \
+            sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+        "
+    fi
+
     for uri in "${SRC_APT}"; do
-        schroot -d / -c ${SBUILD_CHROOT} -- \
+        schroot -d / -r -c ${session_id} -- \
             sh -c ' \
                 set -e
                 dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -160,6 +189,9 @@ do_apt_unpack() {
                 dpkg-source -x "${dscfile}" "${PPS}"' \
                     my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
     done
+
+    # End chroot session
+    schroot -e -c ${session_id}
     schroot_delete_configs
 }
 do_apt_unpack[network] = "${TASK_USE_SUDO}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5c8996d6..ea26b530 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,7 @@ dpkg_runbuild() {
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
 
     rm -f ${SBUILD_CONFIG}
+    debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control"
 
     env | while read -r line; do
         # Filter the same lines
@@ -101,6 +102,11 @@ dpkg_runbuild() {
     sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
     DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
 
+    locked_update_cmd=":"
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+        locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'"
+    fi
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
@@ -109,8 +115,10 @@ dpkg_runbuild() {
         --chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
         --chroot-setup-commands="rm -f /var/log/dpkg.log" \
         --chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+        --chroot-setup-commands="${locked_update_cmd}" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
         --finished-build-commands="cp -Ln --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+        --finished-build-commands="mkdir -p ${ext_root}" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index b48bcec8..aa4874c1 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
 # This class extends the image.bbclass for setting locales and purging unneeded
 # ones.
 
+inherit debrepo
+
 LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
                en_US ISO-8859-1\n"
 LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
 image_install_localepurge_download[weight] = "40"
 image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 image_install_localepurge_download() {
+    debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"
+    debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
     sudo -E chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
 }
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 7c84505f..f98cd759 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -6,6 +6,7 @@
 # This file extends the image.bbclass to supply tools for futher imager functions
 
 inherit buildchroot
+inherit debrepo
 
 IMAGER_INSTALL ??= ""
 IMAGER_BUILD_DEPS ??= ""
@@ -21,14 +22,22 @@ do_install_imager_deps() {
     fi
 
     distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+    debrepo_workdir=${DEBREPO_TARGET_DIR}
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
         distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+        if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+            debrepo_workdir=${DEBREPO_HOST_DIR}
+        fi
     fi
 
     buildchroot_do_mounts
 
     E="${@ isar_export_proxies(d)}"
     deb_dl_dir_import ${BUILDCHROOT_DIR} ${distro}
+
+    debrepo_add_packages "${debrepo_workdir}" "${IMAGER_INSTALL}"
+    debrepo_update_apt_source_list "${BUILDCHROOT_DIR}" "base-apt"
+
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cedde239..47f07129 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -2,6 +2,7 @@
 # Copyright (c) Siemens AG, 2020
 
 inherit deb-dl-dir
+inherit debrepo
 
 ROOTFS_ARCH ?= "${DISTRO_ARCH}"
 ROOTFS_DISTRO ?= "${DISTRO}"
@@ -148,6 +149,9 @@ rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
 rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 rootfs_install_pkgs_download() {
+    debrepo_add_packages "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"
+    debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
     sudo -E chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
 }
-- 
2.20.1

[PATCH v5 09/12] meta: Add cache-deb-src functionality in base-apt mode

[Uladzimir Bely]

Fill base-apt repo with source packages.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 17 +++++++++++++++++
 meta/classes/rootfs.bbclass     |  4 ++++
 2 files changed, 21 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index f0ab9714..a7a8dcf7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -43,6 +43,23 @@ debsrc_undo_mounts() {
 EOSUDO
 }
 
+debsrc_fill_base_apt() {
+    export rootfs="$1"
+
+    local srcpkgs=""
+    for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do
+        is_not_part_of_current_build "${package}" && continue
+        local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+        local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+        local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
+        local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
+        [ -n "$dscfile" ] && continue
+
+        srcpkgs="${srcpkgs} ${src}=${version}"
+    done
+    debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}"
+}
+
 debsrc_download() {
     export rootfs="$1"
     export rootfs_distro="$2"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 47f07129..0ada4721 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -230,8 +230,12 @@ cache_deb_src() {
     sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
 
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+    debsrc_fill_base_apt ${ROOTFSDIR}
+    debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
     debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
 
+
     sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
     if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ] ||
        [ -h "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
-- 
2.20.1

[PATCH v5 10/12] Set ISAR_PREFETCH_BASE_APT by default

[Uladzimir Bely]

This makes Isar use `base-apt` repo in different way. Any package
installation is done from `base-apt` repo, which is prepopulated
from the external mirrors.

This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 RECIPE-API-CHANGELOG.md          | 9 +++++++++
 meta-isar/conf/local.conf.sample | 4 ++++
 2 files changed, 13 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7a16b5c2..ae845674 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -492,3 +492,12 @@ Recipes inheriting dpkg-base now automatically have a bitbake target
 exists for the current `DISTRO_ARCH`.
 In that case the compat package can be built by adding `<foo>-compat`
 to `DEPENDS` or `IMAGE_INSTALL`.
+
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before any step they are requested.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 7e0184e4..2d3c9c1e 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -215,6 +215,10 @@ ISAR_CROSS_COMPILE ?= "0"
 # does not access the network
 #BB_NO_NETWORK ?= "1"
 
+# Comment out for switching to old base-apt behaviour when it's populated
+# at the beginnig of the second build
+ISAR_PREFETCH_BASE_APT ?= "1"
+
 # Set root password to 'root'
 # Password was encrypted using following command:
 #   mkpasswd -m sha512crypt -R 10000
-- 
2.20.1

[PATCH v5 11/12] meta: Specify grub-efi packages arch

[Uladzimir Bely]

When using python-apt, it can't mark it for downloading when used
without ":<arch>" field, so we miss the package in base-apt repo.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/conf/distro/debian-common.conf | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 5610dcd4..ee84b7dd 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -23,10 +23,10 @@ WIC_IMAGER_INSTALL = "parted \
                       tar \
                       fdisk"
 
-GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin"
-GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin"
-GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin"
-GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin"
+GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin:amd64"
+GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin:i386"
+GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin:armhf"
+GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin:arm64"
 
 GRUB_DEBIAN_SB_CHAIN:amd64 = "grub-efi-amd64-signed shim-signed"
 GRUB_DEBIAN_SB_MOK:amd64 = "shim-helpers-amd64-signed"
-- 
2.20.1

[PATCH v5 12/12] Disable deb-dl-dir in base-apt prefetch mode

[Uladzimir Bely]

Since all packages and source packages are placed to base-apt repo
during build, there is no need to have them in one more place.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/deb-dl-dir.bbclass | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index a7a8dcf7..54110cdb 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -87,6 +87,10 @@ debsrc_download() {
 }
 
 deb_dl_dir_import() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+        return
+    fi
+
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -104,6 +108,10 @@ EOSUDO
 }
 
 deb_dl_dir_export() {
+    if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+        return
+    fi
+
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     export owner=$(id -u):$(id -g)
-- 
2.20.1