From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:23 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-wm1-f61.google.com (mail-wm1-f61.google.com [209.85.128.61]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAMCB007115 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:22 +0200 Received: by mail-wm1-f61.google.com with SMTP id 5b1f17b1804b1-42808063c0csf4345125e9.3 for ; Thu, 25 Jul 2024 08:10:22 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920216; cv=pass; d=google.com; s=arc-20160816; b=oJl1azPQV5tcN/1Pqj3qk77ChZVKMfpAv9/w1GaV2bFogbRLcu3ebQIplo+furFIY/ yDxY30UW9d3zuiDlGGDhDxM1qzcoAKs0oTrixnNFn2gWHMrOg1BiIFvRUDHXo3PUJjUI vObP0WvUQgBsH1YJzLpSvRsBiviv8Clm2HuPOCou9HE0W85FkS7NffAzso/gw7VjwQ8y d+FzPzcDplhWGtEUiNv5zEM+P3pCNhhFMnE+9ga3DYsq/YQJkz0I4UMmCki453l4uQ5i /9PqCAHw+ldrUUrBncnQGy2msFT5XbZJENFY8YiHSDtkekWLJqai6UaQHhCmJOQ7Bm0Q QfuQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:to:from:sender:dkim-signature; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; fh=nxVf2qVWuIQcnImviaitmsx3tNoNfJLNY6Yf64nz2xw=; b=JcfrHV7JK3lL1xxBZj4SCXMoxHP5nzH/FYVzkOj1Znq7WiljqYGorMiuv70Wb0tq7I 63CGKcYNrOnsc+Bdh7N0BdBblIKykOmQ1zSjwJ4F5XITw7rmlt6+8G4CgtLN/IdfGQOf KUCAtB+Zdr7ZI8hkxFDt7erZLC7RdbHiBCIJJsXO0I1hKVKpm5/9KshbB2MLOkEBGqn9 jCa46XbTXlVZOpVE3zLlHxyTE2CVIiXdMR4jW5oV2TZe4oeitpC7MXCg7WbFR4GuT94y cR76/hA+PCuXmjoTeyP3+rVWuf78/GsqUpLoZjRmyWKmG1h2U2K4NLZUrfVsMF52TCZG s5iw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920216; x=1722525016; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; b=DEuNyD9s6mbQqv54luIkiu+G5fbxM+6KleJuzeHD2EY0Fq8OWBzCyHfdm776LNZJGe D1kx1WM8pw+nIriJbHDoJAjvTOK/MzfL3wSAie/UCNwOMtK6nYeZTBCzW9DLIYK9upYk Rs+0x5qNFGm0s+povVvIcBVqmgOZKupvmVM229EvuaR+VrlaUuXn+aesS04avIUPMP0Z deZ7LZJgNfvf0WicAEpCFzOon8JEu/TAdPZIuKSjkPGH4EAb4A6oMx4cFvbPraCCrv+J k1p2YCbNYsIsF4kwhPLq9v8aO+Q0HfGT1cVK9tJKoW4vuQaRoF4RqsCBMy+Nvn9Q3qei 04YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920216; x=1722525016; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=AXs3NgCqgz2w6/XaW86ksufTXjE80tcT5JEt/SZn0Mk=; b=VOfcDaby6MOtBt3KLWC1/UV4zZ+oXXE4CYhaMOt0N84N6s7zE92KzvmRngRuwtdsvP ZlcCEStJYYeysubYK4qr6i8QQL1g0dsPxaAsUWoXnvH/gyBMYll/6rsKRJBNLKSHVPX+ 023FflwYSW6cuGiuVle5UlBHg/jZItWr4Av9sn4tfN1tro9g0Xv2wCJ/u2AxAo1UBLRS eLvom6c+Ww5n5BCJqjxGc5tEW4L5khueLYuixQAfCrHs7SffnNSlay60laQzPhyOni/L nW+/F5HtsuMoOgxYdcizNpPxLSx7G09ltAfFapoPurpDd+myOv0srZloBOxwfduG7OJy w7gg== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUXRDGWcUEhrD40+p/+JltdGq8Pv5XrIA0i2nSmyL/g0vKXOlHUo7KAxkC15dEfti/GE6OmfwCR4LfSDjeaPpJxdgU= X-Gm-Message-State: AOJu0YyqwQTgzo9LqnozGxkz0sb8lFQG7dGQ1gqBzaSOcjh0irADOSMh FqItXEjqc2H06dSt20vAaMYFu5gH04DQNfVWdG5X2gYN1C+MfoLW X-Google-Smtp-Source: AGHT+IGXiqh38JdIeFyA5bYVoQkWJeWZm6eS1nlI+R7JFYe6qZItGIJBBxp2I2kP4htOJvO8GvRKKQ== X-Received: by 2002:a05:600c:4f16:b0:426:6ed5:fd5 with SMTP id 5b1f17b1804b1-42806b6be12mr16917455e9.6.1721920215823; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3c86:b0:425:68a1:9da9 with SMTP id 5b1f17b1804b1-42803b6a203ls4663875e9.2.-pod-prod-09-eu; Thu, 25 Jul 2024 08:10:14 -0700 (PDT) X-Received: by 2002:a05:600c:4f16:b0:426:6ed5:fd5 with SMTP id 5b1f17b1804b1-42806b6be12mr16916665e9.6.1721920213754; Thu, 25 Jul 2024 08:10:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920213; cv=none; d=google.com; s=arc-20160816; b=Ed1loDP0GlSMpoB7uFHcojND9/n7yqaBJJKBOdl4mkiYNyaMRYvwQPsXkHR+j0bzr7 CNJYrPO60HLwj0MHXVOT9moMIHFWLbyi29llaNnHz4jf9MMJEGACcaKA0nSAEtbuR5RZ jgpkbu/TbqDDMdbu8ec/PYH/AlzpOsORSVbLhQgp7H09jDHY/5lmCNetCVo24w6t2LSn a2YBteYp4RnreaEWcivmfdPbp0FzTqmui8gp0lLyHF7CHkkudB5o1PbzNRP1zkVMvPpC Oonn64RdUpzcZir7ilspJvFjPYdS6Hezs/iOSkwBC9ppOBQeizIAUm5Zsix3cGfCztDv SztQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from; bh=1PlBqO2ii0s9YxUVnymyOl1EN7IWbB14UWV4LBMpLHk=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=FNzhbQnfr1SIA7pDycooywzAcsduafQHZqA9VnV8KC0B5oURD4Z3Yl4XzkCHoExPXS j7CWZzFKMqQ0uMf0VNxhi8CrH1CCifVhfyNMTM1PW2TFG4xjXAU0HUliZR3UHUjaoRLJ gfpZltCrTstidQanh+wFrRXos7Cl9VNE7Z5Z1sbwBaP3LJ8Ivps8+j9PHanFt5QgTO8O MrBZ3vonmW4cX+NJL6H8kzw85Es4uVQu4+py05V3H8x2bImKI334y/uShJSQ/cHR6N7v jvgYNIobEQpkD7sxLa+MtbOrMUDR0Ha7RCmsA/tQVBOFHSZjUOtHZh3I3Fa3cPcyAEFs KYxA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 5b1f17b1804b1-427ff751fc4si1666845e9.1.2024.07.25.08.10.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:13 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKW007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:12 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 00/10] Improving base-apt usage Date: Thu, 25 Jul 2024 18:07:32 +0300 Message-ID: <20240725151006.2129-1-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: gzgoyvJS1aNZ `base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management. The current approach in `next`: - On the first build, debootstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories. - On the next build: - Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`. - Use `base-apt` for bootstrapping, building and image creation. Some issues with the current approach: 1. Different policies must be followed for the first and the subsequent builds. 2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`. 3. We rely on internal implementation of `debootstrap` and `sbuild` for saving and reusing the packages. Changing to e.g. `mmdebstrap` breaks the unrelated `base-apt` functionality. 4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible. 5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case). 6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary. The new approach: - On the first build: - All packages necessary for bootstrapping and building are identified and downloaded upfront. - `base-apt` is used for bootstrapping, building and image creation. - On the next build: - `base-apt` is used for bootstrapping, building and image creation. This series addresses issues 1-5 and provides the architecture for implementing further use cases. The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach. The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode: ``` # Create local `/build/ba` repository sufficient to debootstrap Debian system with `armhf` architecture: debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf # Add some packages to this repo (e.g., build deps for some recipe): debrepo --workdir=/build/dr locales gnupg # Add srcpackages for some package to the repo: debrepo --workdir=/build/dr --srcmode tzdata ``` Changes since v6: - Rebased on latest next. - Fixed possible stuck when `cache-deb-src` feature was enabled. - Fixed support of debian build profiles. - Moved locking of `debrepo` context from the bbclass to the script itself. This makes usage from Isar simpler. Changes since v5: - Rebased on latest next. - Changed order of the patches. - Fixes in `debrepo` script that allow to use it outside Isar in standalone mode. Changes since v4: - Rebased on latest next. - Rearranged patches since some of them are already in next. - Added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes. - Code passes both full and fast CI in both "old" and "new" modes. Changes since v3: - Rebased on latest next. - Cross-building for raspberry supported. - Code passes both full and fast CI. Changes since v2: - Populate base-apt before using at all steps of native build. Changes since v1: - Rebased on latest next. - Updated patchset description. Uladzimir Bely (10): scripts: Add debrepo python script handling base-apt meta: Add debrepo bbclass handling base-apt prefetching meta: Always use base-apt repo in local mode meta: Use cached base-apt repo to debootstrap base-apt: Predownload packages to base-apt before install meta: Add cache-deb-src functionality in base-apt mode testsuite: Set ISAR_PREFETCH_BASE_APT by default Disable deb-dl-dir in base-apt prefetch mode kas: Add PREFETCH_BASE_APT config entry ci_build.sh: Install python3-apt if not installed RECIPE-API-CHANGELOG.md | 10 + kas/opt/Kconfig | 13 + kas/opt/prefetch-base-apt.yaml | 9 + meta-test/conf/local.conf.sample | 3 + meta/classes/crossvars.bbclass | 1 + meta/classes/deb-dl-dir.bbclass | 21 + meta/classes/debrepo.bbclass | 90 +++ meta/classes/dpkg-base.bbclass | 27 +- meta/classes/dpkg.bbclass | 8 + meta/classes/image-locales-extension.bbclass | 5 + meta/classes/image-tools-extension.bbclass | 13 + meta/classes/rootfs.bbclass | 12 +- meta/conf/bitbake.conf | 5 + .../isar-bootstrap/isar-bootstrap-host.bb | 2 + .../isar-bootstrap/isar-bootstrap.inc | 99 ++- meta/recipes-devtools/base-apt/base-apt.bb | 21 +- .../sbuild-chroot/sbuild-chroot-host.bb | 2 + scripts/ci_build.sh | 8 +- scripts/debrepo | 590 ++++++++++++++++++ testsuite/cibase.py | 4 + testsuite/cibuilder.py | 8 +- 21 files changed, 927 insertions(+), 24 deletions(-) create mode 100644 kas/opt/prefetch-base-apt.yaml create mode 100644 meta/classes/debrepo.bbclass create mode 100755 scripts/debrepo -- 2.44.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240725151006.2129-1-ubely%40ilbers.de.