From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 27 Jul 2026 13:29:12 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f189.google.com (mail-lj1-f189.google.com [209.85.208.189]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 66RBTBce000983 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Jul 2026 13:29:11 +0200 Received: by mail-lj1-f189.google.com with SMTP id 38308e7fff4ca-39f2320dbd5sf7031601fa.2 for ; Mon, 27 Jul 2026 04:29:11 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1785151746; cv=pass; d=google.com; s=arc-20260327; b=W/6wwjGRnUF7EYk3CjFUkqRHBBrEjrwpA37JLR3tono4Tw/CyD382cEqjH3Rj6ibqJ zDcfX46zyB7B8Ht+Y+dZ97hDGumQ5EQ2VQeI6GxKHtYHiCxyVWmg41+Hln8meesyaCUU qIl1npHocN1PMVGvZtxHdwQqxdf4icZkTUsfXgDvTqEmyCdcP/IXlofJtWqfqkKSlvLP zBjagI5xY4UBEqnXsHM0aSGLOp1Ou80yB57ATGdZjlgVpdfkE8qSiT26zVlOqkqNeBlO ZOXLs00Srjx0N5TPlxGNnECYYgbzVTn2JU9TO/XXA9c0cpv60UXNgqqr5v2+BVK8JXLT YJeQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; fh=eJMC6ujbYnOA7qS4CYb6vhI2usq4p2VYtJxjCUAl8ws=; b=m5/mytYR1cklNFZ/ISgZRW4B08VH14YRBpzRl8fCIak+vDxNxGHt04lxfJnksud11d V5G7YO1KO9mJhv+u1VbqQiN1ctSBXFijSTj3YukKEqFN3ouRM1Jzlwt19rqvUeL16HPM HROV7y89HtK9Vt17s0g61hWwVBFRg6W84ChSOmRE1eNWVQE+LP5AbXMrx6ynIlfkv6DI PonsQ7QM9aGH6GH6oKWcoJn1oBl7KE4qWtF/sH/Vvv06+6x0Wl91P7h0KF43/LGdfysG CimoN/etRwhnIW/iywpgacoXGOafwcQO3B35YBbe+zz98otuHI0Xh5Asg35hjcNW1N9N GZuw==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1785151746; x=1785756546; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; b=pJ2L7xUYtG/xLI+R1KPMfra0JBdT1kMnyRyLpmAfGHwtRNMYHuUNlRv9cjIxhR4sM9 o50+fu8MdSpgjFplS1ozR5gvDG24ENLAEthacqXQ3+LT4JJfWiAUN9lb3hbubwtgHy9+ QAGUS0SkExrZyNna0aLC09AnYEnWS1O4jd3y0kHqdeZTYbWOl+TkkuFOUtYtcTU7zYH4 LDrskmhKMbZWWtGSFjJN5FcfVVR3tEo7GQSivmSSdlxD9xGeXQOeMIAJcEuw2SZs1M1j xxyKSJwK3g1m41zkppBPuvrAtDmOubQW+fxwX2CI2Lh8DFQ3BiIGKoW6UABgEHO+1wZM +0gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785151746; x=1785756546; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=wCR6v0QP6sDHRNth0bg3FtZ3bWDQW2D1YAeTLYUD2Xw=; b=LnZyBoHjMtUVig1SpeJmaQZBDSoEiQL9J54WMvYhk5ia2mUwb3HgtMWcNI7hdGirEV JVyYLERMQ/tNvxHnEDp8/RocnoAZm15wSSXhv42swU/c9OOlumbUa9kxADjwFkRwt8/J 3Ik3JOsGsw3i84D4dbtqE3DTtZQWg54FNAJecoZRCVMdBRToFVRkzW7CgpPtTMfWagyN v86cj1dSwF4RRsZE2RP5u2cEjmtQ2oZZR7c6j1UvXeDPVM6J8LXk87uQppmqSfilL/k1 ZCQY/TD3lzKVMxVgpTZbSVe01Lhs7djkO0m9ubHtsVlxRoKSMNkUYYYJdzXVapKkvLfn l3EQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AHgh+RqNu1fChnwb1SuZCgvDj6Tn2LQgQTrYHVMKBnVjhRwkZsCNN3vXRgDm+tCpN2lZHXUuD6A2@ilbers.de X-Gm-Message-State: AOJu0Yz/OCplB5YcYDVI8Bb2wPj0r5ueaYobc/Q3QC+9Gm7jX6F03RD1 9VCKlO5Jc7VEBJW0n1d9lOEFYQK3IeqmK3hDu3ifaxFNGnk/bug+nu87 X-Received: by 2002:a05:651c:210e:b0:38e:8760:2922 with SMTP id 38308e7fff4ca-39f287dfc2bmr12507051fa.18.1785151745574; Mon, 27 Jul 2026 04:29:05 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="Aa7YSPRfA7GUrxRCJB0/CgMgANlEotMHA7F5vrvrtnLtXZvmyw==" Received: by 2002:a05:651c:20cd:20b0:39e:e8d1:b786 with SMTP id 38308e7fff4ca-39f1925cba8ls8519551fa.0.-pod-prod-04-eu; Mon, 27 Jul 2026 04:29:01 -0700 (PDT) X-Received: by 2002:a2e:a592:0:b0:39b:1b6d:5027 with SMTP id 38308e7fff4ca-39f287bae1emr13949811fa.11.1785151741193; Mon, 27 Jul 2026 04:29:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1785151741; cv=none; d=google.com; s=arc-20260327; b=bY+n+BiQ9BulcLO52M1ahCEmCnIs/OoSBB/ncN6GQNheDIuI1hJ7HI9xACwb0jYJTR dIXapRG+KeND3NulBPeAnWgzW+QNI4nR7pBqfpEXCZa/ez7YzJ3T7Q6aBhGMMRJPa/oY a2D2tEd79yBlTkH2vvPOvtz0Gl+Ohb5ADoaiKY1MPogXObJkQjpcOLFDJ2YKwQhwMeXa 5/9iWQcp6x9oYnH4p68v+jTIc+jLDzpYc9lKZCF0KrgGVSd/ggA5KVCY7oWLJ/doWvJA 2y0JoNsqG1dO8Ve8NvRNREuozcBwUiX0vosDBmHqwxj18syRPQmwgqACeRCwWffHgtx6 tG8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=uYHQ8HZQsA1ND+rhZetA0I8qqB8bQo9bpBtAMO/8mAs=; fh=u+3kXEMzY4uVhDBiujW2QUo7NnpMLrGEgMQBYpCTRN4=; b=KlxvJO3l54K2/Bie7Kno8RQ20LNdqMAlFdutGNyvJIDe8Hv1nfIdXL775p4gv1T1f2 N5E6CyVsUKAYf9WcIK54aUP4n4Ixdxn6Z60p9La9UF7RAnn2DolX0guxa3W43Jlfl6BW nia+X9yVQXzaiFKm3F6CPd/Vn8tHv1xJkt25WNCEIdtNb5MNssejhdu47R+mf9fvtZIj e3o7GGxe759Wyyf5qVHnYmSMBFkAQqXYUgcv9spkPlEG2J5j0RHL1F1XWbWQFfVDPjgq A/v1EbRC4IINKwCpdpX5ONPyyXlOIazRgwa2iZJtevpyQW5lMhV2Myy6cWfZWQzb+WJE wOEQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-39f2223d6e9si1290581fa.4.2026.07.27.04.29.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 Jul 2026 04:29:01 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 66RBSjdQ000930 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Jul 2026 13:28:59 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Aliaksei Karpovich Subject: [v8 00/14] Improving base-apt usage Date: Mon, 27 Jul 2026 13:26:18 +0200 Message-ID: <20260727112812.2255297-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: smrV/rofeOwd `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v7: - Rebased on latest next (70661c40) - Fixed url parsing in rootfs_install_pkgs_isar_download() - Added version and arch for downloaded package in rootfs_install_pkgs_isar_download() - Workaround for base-apt to force to install python3-apt amd64 - Added using override for reprepro to fix offline build Known issues: - Didn't pass the testsuite CI - 50% of fast tests are failed Aliaksei Karpovich (4): rootfs: Fix URL parsing rootfs: Add version and arch for downloaded package sbom-chroot: Fix python3-apt dependency debrepo: Add using override for reprepro Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 10 + doc/user_manual.md | 1 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 47 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass | 5 + .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 21 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 90 +++ meta/conf/bitbake.conf | 5 + meta/lib/aptsrc_fetcher.py | 22 + .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 10 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 +- .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/debrepo | 659 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 26 files changed, 998 insertions(+), 18 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260727112812.2255297-1-akarpovich%40ilbers.de.