From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 04 Sep 2026 09:56:04 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f188.google.com (mail-lj1-f188.google.com [209.85.208.188]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 6847tu0g004149 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 4 Sep 2026 09:55:56 +0200 Received: by mail-lj1-f188.google.com with SMTP id 38308e7fff4ca-3a3130365d5sf3717221fa.1 for ; Fri, 04 Sep 2026 00:55:56 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1788508551; cv=pass; d=google.com; s=arc-20260327; b=fCeq1qXLosAnb81vOa1O+8Kh9F/Q/L6N+ue2pnOCrqk0fn2QtwJ4HA4nmjJVqGL7SM Yt84pS8d9co93zoStK98dTA1t0IlbYUAo7hbDhcjJHAPk3DHwKL5YJXs8bE/9en+a9nW fUuZCdRI69CjmQ7S0n03ulHy+QrKBbOx2k+dhvq2Z/97tUdy3uJrZKl6Eq1yT3zk/nRJ IyzOv1ca0VpB+CnkQNZWlkXaNTsBX57AcjrP5UTp3dvHY3nCOSa6RwbCScy2KKvDAJ71 F6FJ3ZR7nEf0JSGgHJUak8juUiPgR8pgCUd2rV7bOEjliXibiM7NkCQooOGZmidH0WlR mzSg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; fh=jy+GLzP49LX9mjFyWvoLR3pQLlm7S6cWz5Y4qW247UQ=; b=Hb241xTTYuiq+uHC5Nj4qBaUvGn+Mrvu28jLFlEugyYtemcvcwsvLho1MyibY8Ulcy id7/Dq+BQuh7IpdV2mFhkh/qcANUbhp6oAwD38+JKLEjNt8gCHXPsMkYMxCUn+O6vZeG W23QAfpU0MwwD9oHITPKqcNSXV6kX48sohRPCiPhjnmuk1ckylO958pcs3KVFl0DWKMw cbN2EGAj0K4YtxXC/8zO+ttXr98sywzh9FBmdxJNSsudFd6BCSO2P8keRP2oyK1l8XJO zV0At4+BjuS4tOhtV66hSybrwztMuOg5G59kbyzzDmzhnJNxIQ1+3VwC+mzh8c26TCMk agIw==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1788508551; x=1789113351; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; b=Kp/fQgm+h2mzCyekM5YRfdBfaDBYg+XrqNYv6ClBoEZbgyTor6dNdpUntuTEVCjnbL ZD+p2BBsy5VWzlvwS3QxWustLZozh7JJQDOx+j5uOOvLBPj6y8KT2KXCm+sZN9QdiiRT YdgcN+zEBcrNXvcQM0isX387iL3ki2qprGokIb43kElerxCq8J2scp+Z6i+YPb5faYlR KasUqykgARmoFcg/PXPz0rFardt0M0GQz722HRvMD/jOYiqZwT5ECYoysQj4kZ9IGct7 1QeCRMVtu0FqXe91m92Ob5CSJDuFkhY290R5HkIHsSMG6NlV5eRf//tzoQe6A+4oSfrt /N+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788508551; x=1789113351; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=ORrOAqSwN4VapxMGHnJnASCKwJWEoRKmUeOGaD0H+V4=; b=Lv2ET8OT5fJmamNUtHwOK8N2gmmfYiaZzEpRyLfJ9s4YneeE61V8fz8dNKE2pb6G3V pTdkTDUdK09O7DC50aBui3yXvFZzePae9cxdyjz+yRXvRj4139XZCD4yY+lFw6EVKZAV MeVOtiEdi028Pi6WOpHuoHPA/aUmsflEJcY6vDFTMscaO273mjAJL32X+7MMmT/aWguw VzSOPbihEkwJDxRFzc/eU0s/2keTuLIdf/tIdfiKTCx+pq79OlTR03CifOSi1Zqv24D7 86Bv4IUly/bpzmspmfmJccCthfyQrqG23e9Z7dcwzOOZcZ/TE+tAmhl6Vnx9FhfV7MpG bdIg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AKwUvBykKwU7Q0kWUSsj0H90o+gE+GoLPYqcgUrnrAB7KyirYm/TfpDgtEE0bi/88M1C4/WR4kRM@ilbers.de X-Gm-Message-State: AFuF++k19i1Oww3HVvJUyJVNQW2t+U4kVyZOLGTZLCx41PGZ18JPGgg+ dinxk3GLx/27GfiPqRnefgFMe8MhTvNR7Nxvtydt0KWlSa1fxo2T3xf/ X-Received: by 2002:a05:6512:ad1:b0:5b6:1a7c:35 with SMTP id 2adb3069b0e04-5b61a7c023cmr271232e87.56.1788508550374; Fri, 04 Sep 2026 00:55:50 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="ATskLdd2BKyXIojWh+qaTXDH5yoVyKdA2c7KWIbE/ygM5cX3BA==" Received: by 2002:a05:6512:638b:10b0:5b6:1a72:3d10 with SMTP id 2adb3069b0e04-5b61a723e32ls78320e87.2.-pod-prod-06-eu; Fri, 04 Sep 2026 00:55:46 -0700 (PDT) X-Received: by 2002:a05:6512:a34f:b0:5b6:1a7c:3c with SMTP id 2adb3069b0e04-5b61afa3cdfmr118285e87.63.1788508545732; Fri, 04 Sep 2026 00:55:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1788508545; cv=none; d=google.com; s=arc-20260327; b=HQbOwfIc+TKJLIG5znPZx7V0P22SMpnmjPD7FA2hQ1rxL+fTo1CEDs63JpveQ5Exgy HrTzCXxlxvuYTiROsA+/F2K6V1WnafRdOUKzxP3RUtC4nAmJjQqtJ1bLKUh1i3+CvJEM +m63rwZbFg/0jOwSo6o3sHctzkVS4eBcO66PgGLRucPFe1U40XMosQzuFxAug5FAMMI4 qCaoZcHKZs3LpRNbexDt3ifkEydb//TPxtaGj5vzt5/cyTv4zaHiEOAPLJ4Hc1NJqm1N mP7as+aEd5YUubnObmoZxcWEjQ9nitkLS1uJobza9NgBoZNuJRFew+IsZAgYvyJHE1Ib wgUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=3HY4iqBmnhtBAAJT0ep+LICqLIdu1ES88CPrbes1MBo=; fh=u+3kXEMzY4uVhDBiujW2QUo7NnpMLrGEgMQBYpCTRN4=; b=Ws3FfC5MKcw4kuzHfV+hwTWgUq1PJh1L3kUBKQhdFtTD234FTZC5eb/BDUSh+ZON9s 6Ovn7pc+gHS9aLLgP4cw2F8fy1+e/Vy/jqN3dUq/dUvWbID9ArfU5Zq37uChRLCRwuva fAVy/cLH5Us9ygezXGTl9RYTS5bLendixhBPpJUdeymLNn4bwbCrfQJN9EwAcE/kLbG6 q6df7bo9mKXREokclu5p6vJECEgQdfl+Mv1F+QpquWhgQw/WiilP0EVu4/oKht/Wq5CS LkCCWPQHGzaOP4ZPWFzzJCQPOa18FuDkS534aV4zJgPBTiuhaUCcmz+iQ2edHxqLFQ0i z48g==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-5b6166046cbsi35500e87.0.2026.09.04.00.55.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 04 Sep 2026 00:55:45 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from akarpovich-Victus-by-HP-Laptop-16-e0xxx.promwad.corp (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 6847tRSA004124 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 4 Sep 2026 09:55:42 +0200 From: Aliaksei Karpovich To: isar-users@googlegroups.com Cc: Aliaksei Karpovich Subject: [v10 00/12] Improving base-apt usage Date: Fri, 4 Sep 2026 09:52:42 +0200 Message-ID: <20260904075515.25962-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: mfvUF1rnxTv1 `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v9: - Rebased on fresh next (436bc96f) - Removed "Fix python3-apt dependency" Changes since v8: - Rebased on fresh next (86c761e4) - Removed fixes for isar-apt download in rootfs (use Felix's fix) - Renamed debrepo -> deb-repo - Fixed failed tests on Full and Fast CI testsets Aliaksei Karpovich (2): debrepo: Add override for reprepro base-apt: Move bullseye-mipsel in separate dir Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 9 + doc/user_manual.md | 1 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 43 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass | 5 + .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 15 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 89 +++ meta/conf/bitbake.conf | 10 +- meta/conf/distro/debian-bullseye.conf | 1 + meta/lib/aptsrc_fetcher.py | 46 +- .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 22 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 + .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/deb-repo | 689 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 27 files changed, 1047 insertions(+), 29 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/deb-repo -- 2.43.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20260904075515.25962-1-akarpovich%40ilbers.de.