From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 02 Oct 2026 09:05:06 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f184.google.com (mail-lj1-f184.google.com [209.85.208.184]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 692755SS003855 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 2 Oct 2026 09:05:06 +0200 Received: by mail-lj1-f184.google.com with SMTP id 38308e7fff4ca-3a96f2563c1sf335691fa.2 for ; Fri, 02 Oct 2026 00:05:05 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1790924699; cv=pass; d=google.com; s=arc-20260327; b=O4FnuSMLUsr/iTvO7ZOGXO3LzyykkD38sFBPqTIaC9Lme2iCvmNppWfN8V/yqePkOK MB0uYUggoRtyictqFbmYpeqYMqtBqTx6ZpvU3/KsT0UK7x7LMFyjR4gesKr5tMe1PEIF zCAGuJJPRshYAFjL9Z4ch7Dzv81nJCvuECOdn2nLQt6C2ZwjlEM91xHBccHqxpXBdfUZ WpPgpslkWiLviC1Kiqau0AD1dHVGjO4/nV+gT9SfgTNF69sQ9cq+DZF25MLtEXDJczqM 2g0np7t6NmMd5GpVp0tpWe68unaBe+Vea9VVWhC4z87oaPCLfgayLRKG9We8tTVBf6YY m/ag== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=WxatVOQB0pqxeCPXWxRg5oCKvb6xJCPpJeNO6v6Hfiw=; fh=K7IbDyHzuENhlHCWEelbrkCIyFHV14kQ2k2oKwgECWA=; b=PvWwuX1xg7h061eBN48eJm/J9eAwHzr2RtNw1KSAj3TWNLD5N/xcUQTTtfbGMgqh6D jvTS4wR1e1Gsz/GTpJWg6b7BhxHbADZy7Pyk4E9odyroJY/gW4H2ny9S5b5mnwRIhXAy LdrcQi8jrOrnVc3us5cEeJc8iw7IzOIOZ+J0IO1pvKZprrDqstyMXIQbwtGwbT7rJoUZ DzAjNQFp1XwNTtPM6i3HoRxL7PbfxnMs/Gj/ecKjgfYsZmnwRZ1fMrhiXnp9q9XQTcV0 AVP7dU7aKx1SFiUzwoZoTBia3mSdKQSRggqjVDxqcCSNV6QiHb9orzXBKPUTy+xSrRnM g4LQ==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20251104; t=1790924699; x=1791529499; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-type :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WxatVOQB0pqxeCPXWxRg5oCKvb6xJCPpJeNO6v6Hfiw=; b=Iip1zLHuKXAON2IIslbvesNWsfda8LyauUx+JFBlWGdScwJ43QhqIoqQBLXhuBrqh+ tUuXmaQKZJhPNTgjDeRb7aVzsASma3BTIv65yVyT5T0yXde1rHjXMbktv7OGfkOFZAGr qbkXDPinoo14f+KGZhkr1nTE/qvCMsqeAeK7+dC9Qi9YOycFcGRgFmJKAnnb1GtjzsQv lIQIcH9b+dvzBqdr/iQd//+VDh3fR1bbX0l/vhM4GBhzuSqpfdQK5eiRN1T16moTgbxp hXMP38NRFXOaxK26SDsrMijRXwJLF1rZ5U5WRwgC+6DHJF+Q6xsgBU8I/GD1kcGPnm3s bq+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790924699; x=1791529499; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :content-type:x-original-authentication-results:x-original-sender :mime-version:message-id:date:subject:cc:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to:content-type; bh=WxatVOQB0pqxeCPXWxRg5oCKvb6xJCPpJeNO6v6Hfiw=; b=uvj0TxIkU5kuaRhXF8PX5Ad3cU/jKCFdpUPMlD87iEdSqHP0If7fZMsrFG+2Mqdbnj 6l0h+dcjI4HfV/EtPr+VKHIIEFoEgyhFTgGHVK6/k1Da2rOEzVETeHkbuHTXEqSmYC5X NC62PEwjeTDRPltvl1Nz0Csc9EmSqwtabsf4yegPIiAtEzV2KSrOyCXE15jX2RwvIJtC /0IxqMqkGV4zgvlgC2DztPwhre/RIg3Mn77LbMyw37tbFnw2EvwS1vf/VUiPXPFdNdHG HLWzLSYE5i/jqcBHvMef9Dq/a9eNly46pK+SOMYuog4q27VnX8dxyQj2MY8Xblrx+j8q Xczw== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AKwUvBw5sdxiq1a51l094M7ewqoKwwFU4K8HCLGPUd0vnKVLF4wjkbYIjZ32lPFdvc03r+omMtNO@ilbers.de X-Gm-Message-State: AFq9FYJbJukKj/CbOdPmG4uZujTjFDYjroKO/TFOBibhr1rBgk17Eazy ZQLm7EZSEppIA3MZvFes0itIEFnD3c7+HkzKxFL+Vmw3ns80XpAgLM/+ X-Received: by 2002:a2e:be04:0:b0:3a7:72ec:81a7 with SMTP id 38308e7fff4ca-3a8877ad540mr5014811fa.9.1790924698670; Fri, 02 Oct 2026 00:04:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com; h="ATskLdc5qLEgljcJGr6DXRn2mPaxVmqTqDdc78gpSdC983rTiw==" Received: by 2002:a2e:b051:0:b0:3a1:e148:a89f with SMTP id 38308e7fff4ca-3a8a865915cls1560971fa.2.-pod-prod-04-eu; Fri, 02 Oct 2026 00:04:54 -0700 (PDT) X-Received: by 2002:a05:6512:2c93:b0:5b8:eb63:e56a with SMTP id 2adb3069b0e04-5bb9bc625a5mr493119e87.62.1790924693944; Fri, 02 Oct 2026 00:04:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1790924693; cv=none; d=google.com; s=arc-20260327; b=QsgtcCH0NkX7wuuIDlHZ0lA/W1sISBC5Xr8yFdeXLmoTZeU6qU94r9zBiZoBxujOvc hhbb+VhJ4idleTlL5fZ7khzSWaBWivjjufj6tzo/lo4qSBONflYYpe5TBqTNuzTtO3Cg Mipvwj1C3CmsT1vfORMk/yvLYCAYcfi/qcdaUTFuyTzVKQMmScNzsm3Y2h7ztQYKhwhT IHOrwN6ERFx4vuEprSCGu40UPXdxrGSBQKmj1AOkPjMWwvzMINVI6lkFQusuRlFtjmWX PwrxXMkwmdgALrdPlCCUzHsOVhYP3pVMl0VdZgONnc5QUVAwocyHY7VazCGHgeb1oc18 Gpaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=TFqQanmeZuXmbOqM4MwyMV2whNSWn4M3YXTRg4zs6sM=; fh=o/LXPVebB2HvUFXjV79h674hwoXoxg+ZlOnsqOKL20I=; b=ErcPXCvCeeub+RVMJuVhbJc7NNdmCsEwSfrZ9TRRQD48I50Uip+9GzjoRdArI8Z6XG 6aSz0U4h4WOZ8/nf4d0qB/rfFQjkrrSf8r1oOBcqXlc/lhGtsPx8OSvsKPBFLrxEfnWm NmYNxkcNcIgHwPmtGPyaTPKAGYorjEJvhopz9l3KZOFHNncaXkLJfKEhJ0LtZdTXqKOC BIcN6TD/1lqk8R+FRLSPU6jshULBe3DXpCnmteeoHpQ751hiYh4xaIrBxw5GIGygHNcM a8SMM04VHSsmF3zXM7rChC0JUeI3hbSRp3GgD8Ffl8YEHG69aI+EteFoc1WaWjJZLpjw kMQQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 2adb3069b0e04-5bb7c75c859si27688e87.4.2026.10.02.00.04.53 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 02 Oct 2026 00:04:53 -0700 (PDT) Received-SPF: pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from laptop-0xxx (62.122.174.6.serverel.net [62.122.174.6] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 69274a1f003799 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 2 Oct 2026 09:04:52 +0200 From: akarpovich To: isar-users@googlegroups.com Cc: akarpovich Subject: [v11 00/12] Improving base-apt usage Date: Fri, 2 Oct 2026 09:01:03 +0200 Message-ID: <20261002070335.26526-1-akarpovich@ilbers.de> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: akarpovich@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of akarpovich@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=akarpovich@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: sCq/WUlYx/7i `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, mmdebstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `mmdebstrap` and `sbuild` for saving and reusing the packages. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to bootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v10: - Rebased on fresh next (1d8791f5) - Added wget in install section of user manual Changes since v9: - Rebased on fresh next (436bc96f) - Removed "Fix python3-apt dependency" Changes since v8: - Rebased on fresh next (86c761e4) - Removed fixes for isar-apt download in rootfs (use Felix's fix) - Renamed debrepo -> deb-repo - Fixed failed tests on Full and Fast CI testsets Aliaksei Karpovich (2): debrepo: Add override for reprepro base-apt: Move bullseye-mipsel in separate dir Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to bootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 10 + doc/user_manual.md | 2 + kas/opt/Kconfig | 12 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes-recipe/bootstrap.bbclass | 50 +- meta/classes-recipe/crossvars.bbclass | 1 + meta/classes-recipe/deb-dl-dir.bbclass | 43 ++ meta/classes-recipe/dpkg-base.bbclass | 1 + meta/classes-recipe/dpkg.bbclass | 8 + .../image-locales-extension.bbclass.orig | 159 ++++ .../image-tools-extension.bbclass | 13 + meta/classes-recipe/rootfs.bbclass | 15 +- meta/classes-recipe/sbuild.bbclass | 3 +- meta/classes/debrepo.bbclass | 89 +++ meta/conf/bitbake.conf | 10 +- meta/conf/distro/debian-bullseye.conf | 1 + meta/lib/aptsrc_fetcher.py | 46 +- .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 2 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 22 +- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbom-chroot/sbom-chroot.bb | 2 + .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/deb-repo | 689 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 7 + 27 files changed, 1203 insertions(+), 29 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes-recipe/image-locales-extension.bbclass.orig create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/deb-repo -- 2.53.0 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261002070335.26526-1-akarpovich%40ilbers.de.