From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Fri, 02 Aug 2024 07:37:21 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f185.google.com (mail-lj1-f185.google.com [209.85.208.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4725bKBD018819 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 2 Aug 2024 07:37:21 +0200 Received: by mail-lj1-f185.google.com with SMTP id 38308e7fff4ca-2ef311ad4bcsf75631501fa.0 for ; Thu, 01 Aug 2024 22:37:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1722577035; cv=pass; d=google.com; s=arc-20160816; b=GZOQWNyj9y/ksCZT9l3rjCO36TcmP3PoaIvd2b255UkdrbMBLIYFQl0gw7Rg5Mayi8 JfgzVnt0ex1FE/7+5jqp8vGCxVr6xs9fcDroN6Cr2KD8BWk8CYBBsauEToyCxU7P/cUJ yrPloMXHLG2/xc9uFp1rSihwSZQX3Me0S6ZA0iwQLiQP0SJrGVgF0eGjayGnS3n0ENPd BJ9QCFZ5dNfk7vNvBUHiTwxt9yD9GCcPQoxsGxZ/Ptt5LBECRGAcudPvNYt6dL9GrDYw Phew1ig8WsI2/Ejz3rvINkm2nZIfr/+CgM1MXdC0SbC/eAah5kMwmFJErCBpe3hKSwYT Igmw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:user-agent :content-transfer-encoding:autocrypt:references:in-reply-to:date:to :from:subject:message-id:sender:dkim-signature; bh=BbWAnwhFcR3aDOsJSHb1foL0/Gbje32WPooyGZbQqrQ=; fh=6uPKwlhtoemDK/TqMYA9lQHrY2hiX0qZkMxC72bcKCg=; b=p8NHIlv5DMthtui00ejGvwlaz0uL4E8hNS45OilH0t3lK0mr+TrXPe9zXPNmIjWfLP xsFYXaum0buf/qm7UAz7UIJzhhtPcv0ZoRaIbLb1dRKgEjwR9LMjK09td7Aw6NP2Q7WC qsXrjmJmS8N4CVCCKJtjHvWYk4cwqvyOSdWXpoquLhFrha7b2O7axL+H0C2rPH9XBRj7 8oRgwVcFfTEgxUBS+jbnk6USkbojjsmbiNNEtjWeAIXmqQESCRnrx7ilqEvyzVMWTk// Pl5nHcc/OounlmB4zgRlYNCavVKOoQCDTbBdhcWK/xfBbwDdJVPdZNmAXlJ2pnzKRN8v 2Zfw==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1722577035; x=1723181835; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:user-agent:content-transfer-encoding :autocrypt:references:in-reply-to:date:to:from:subject:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=BbWAnwhFcR3aDOsJSHb1foL0/Gbje32WPooyGZbQqrQ=; b=LJXpsMZ3D9oj9W6gKqWujeANxLTG9h0NsqCaqk9/6zkUJUOcNkygeEChGODW51G2mS /frDF54BBQ8SBVoDu/HFWt6p4TLuCuKmHM8vTwMiGpROYO6TfmG0tJrZ66ryzDLrAIB5 WGzrkag4N9l/Ii2nEJnlSo5OXTn7fN1FwOA8+or+JvHJiIuHSeBNqiw8VJtQ4ascU73v D16ZxM6WQJEtxmp4moggnDLNmW3rWDqksMWfvJ/nf0P73tCqT3KLytUa8xZpuvIIJDTB FmcIPpmS9D5rtGEl1syDavtmsiFR8Ukd53qAGHKvaNHQBMuvMY9CMbL8SlgYuvdNWzmV 4HTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722577035; x=1723181835; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :user-agent:content-transfer-encoding:autocrypt:references :in-reply-to:date:to:from:subject:message-id:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=BbWAnwhFcR3aDOsJSHb1foL0/Gbje32WPooyGZbQqrQ=; b=dPLgWFiVmfDi6e0+7S3Ln3mbOt5t0hYw+mjjmRntUF0C8zHD2/AJ9VEewFPBn+4oIO CfTW8uZ/m3ZTjqoudkdk7tIJPFxwkaKTlqGu3EEOQssOgThDPKs6VFOD/lwLd4+Z6VkW E6qgWC3o7HMOFBYGZc9cZLQ7UYA4hWy9wXKevi3XsfKCPt+mabeg5pEMT1CHAITRYYgd r1T4/hcWmW0kcfVqLV8UX7gb1/7JA7+Fy+M2tc9BXVkZdUxg3pA5AWaMY/IvUqA2YFA5 40y1EH1fGt5Hg5dBnEur3RtgAibrdKXRmoVOKaLkJqyH8+cEQZZOIsfy7UpHbfwRlMCz xbGA== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVOER5ixR7e4mjFWUeV5Cvjypd/jvOAH1NtEE/ZnbVUtTeNHncUzxbyp799/kjLwJHA+udtbFgm2qOUymRbKKH77X8= X-Gm-Message-State: AOJu0Yxv0PEWgHhzeR0cBKolH1iaxbf631lDGmZDtW7lIF1Zd3yBz1j+ HCtnlwgadRG6ht6P3WCjumUWOfg4V1VK8BtDmCO5H/wou0ggeXu5 X-Google-Smtp-Source: AGHT+IFNLOHCOZMVBiLswT0MiNPKi0PuBmC7FPLUKluXUwITTx5MRuTJKmmgnvuSbk0yDKRgDIUB8Q== X-Received: by 2002:a2e:b8d6:0:b0:2f1:5d72:60a7 with SMTP id 38308e7fff4ca-2f15d72621cmr9358441fa.5.1722577034699; Thu, 01 Aug 2024 22:37:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:3c86:b0:425:68a1:9da9 with SMTP id 5b1f17b1804b1-42803b6a203ls40528135e9.2.-pod-prod-09-eu; Thu, 01 Aug 2024 22:37:12 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVLWKkD2IGtgVdJOQUlcP/ZkTEVF5UJB9CmVIxIVU4D8id2kqTbb5kGT471RNERUsJ4m3Dg57eY6AmVK+myHflMmGVi9NEtZ+NV8qw= X-Received: by 2002:a05:600c:45d5:b0:428:1007:6068 with SMTP id 5b1f17b1804b1-428e6b924d4mr13704165e9.34.1722577032292; Thu, 01 Aug 2024 22:37:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1722577032; cv=none; d=google.com; s=arc-20160816; b=bA5/OSnnVq5iOt/yTVE0GTxF0zxcdU4YO3A3xCtXQq1XnLzihYT8/StP+toWOX0Uhq FDyI+AAt7K7fIBTuHzGrp1XpfSOPU5WnhEN8/VzB6GxWvqilMqlszZCkHxT0eQPmZ+Gi x8BXAmxzKAIE8w0EOKhRCfUebWQ1uLzaTdTAXQ6h2//kEJ8BhY+O10aL9ON0NfI0DS5w kieiPB6FGePNL8zzfc9oHFDSCCxB3vyWfhhzVKEZu+cyKf416KpChJqSQm+hGhJaaCnO DQJiTWR7F3iWdVQ9jGq1rBd6/fHQ0vN2Sn3p45DO0yIVxO74C+fiEZaiwTR5cUYJuMvg dM+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:to:from:subject:message-id; bh=y6bOdU9yolKj4BxPAG1H2sq9qUakyvVVGukp6BkB5N8=; fh=PICfM4Fo6fj0YFd/Sa2/vzFoNnq31QOnYxZYf2j1CDU=; b=oasioYu11RYRAuP3hH+0FF0Spw9rkacPs0BnvUea4cO0QiqIzV7OOme3+rBvWTx4Sj b9iCo6H40g9hTzCI5S0q5dxvwypCh95171WxM/+8EMSBUz2HsKVm0B4HSOQxRpWjywae 3gIRK9TAs0VSYdE7dqNvqDiPrMIgAfQkQY2bHVeV7dRy87l0Yz0vrHrl2Efyhj3azhwx 5kENpCREh6wBdpl7JkSbeiAT1k9HIJ1VOyVwj9Uiy/RvAIXdp++JBQDmTOIOeVDbHoj1 0dOE1yzfR4OEw7NLLb/JujO4R69v5LA5QoD7TR4y7uDudoC1+RJvt/uL38eHq1LbRDAI xxWw==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id ffacd0b85a97d-36bbd06970asi19373f8f.8.2024.08.01.22.37.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 01 Aug 2024 22:37:12 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [IPv6:::1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 4725bALc018814 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 2 Aug 2024 07:37:11 +0200 Message-ID: <8a9f43a239129e12f09ccfeda8b683b9f36a8fc2.camel@ilbers.de> Subject: Re: [PATCH v7 00/10] Improving base-apt usage From: Uladzimir Bely To: "MOESSBAUER, Felix" , "isar-users@googlegroups.com" Date: Fri, 02 Aug 2024 08:37:09 +0300 In-Reply-To: <06ec8803dfb228fadd7bd03398817d190914de3f.camel@siemens.com> References: <20240725151006.2129-1-ubely@ilbers.de> <06ec8803dfb228fadd7bd03398817d190914de3f.camel@siemens.com> Autocrypt: addr=ubely@ilbers.de; prefer-encrypt=mutual; keydata=mQENBGO2eUkBCACtT+T3OrPVSExBmqfgXT3lp9XcdxRzjYp26wezkgYjjBXaf36bxtaAf S471VoQtpar0RVeFfW7WDDdfX9ZclSj36zBQe+RVSJzoNoNQfjOXWuSHb5Z+cpAFtqBY4muxK4+ia IlLJd6CN3ejOsLHATtCeHHq8wi0z2T+KdLQO+wQRgo2hjj0Lp9pGTrKJry50HP/o7Vbdu14dOx2xq r8+wPc6SQbBIrcqaa4MqCQC00vQG7eXvo+k2MOw59FDdpMH0KR9mHgp3u/s4I+4YRBArukt9G9xz/ rsEFmxAIBC6N/a6Hzwg4puc91n7ABDsPg8Vp+X3MDraujN0dvR6OKVNtABEBAAG0IFVsYWR6aW1pc iBCZWx5IDx1YmVseUBpbGJlcnMuZGU+iQFOBBMBCAA4FiEEJqPNVhVGyk12Eh+PAUQYBM/2FkoFAm O2eUkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQAUQYBM/2FkprlAgAmuna8Hm9EyoEtTl DBGDg6Zm4ZLp5ffvZBE946h92jepDrteoxsJ7pSzJVC2HmDLa4iZUao7lLLbDsUj5x45/iLJcqBZK k3YnAxP2r6a+kI+1VVQY1pxdG1nlJAbdNzoojm/qmezNPSrqni61KVMQKsXBCWhIjSXDSM9CsBj21 a+9qaVqfxovJGTn9lgrZO+xzKQNMKZeOouJlscVuFj21P0ww3/YENiU/nMeTSuYypO76mDtAd08Jo nc3yuHa9MJGei5ixN3wT+IrGR2aL2hdw2M6NgH7sYbL2Zi4ugD6RXHJai1Bh2yvFSVqSQ+M6QOInT 4ud7wslm1XRB065dXtA== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.2 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: VNhw7mMBH/a/ On Thu, 2024-08-01 at 08:57 +0000, MOESSBAUER, Felix wrote: > On Thu, 2024-07-25 at 18:07 +0300, Uladzimir Bely wrote: > > `base-apt` is a local apt repository containing all upstream > > (Debian, > > Raspberry Pi OS, Ubuntu...) packages needed for a particular build. > > This series implements upfront repository downloading. This is the > > first step towards local partial mirror management. > >=20 > > The current approach in `next`: > >=20 > > - On the first build, debootstrap and sbuild are used for building > > Isar artifacts. The packages downloaded from the Internet are > > cached > > in local directories. > > - On the next build: > > =C2=A0 - Analyze the logs from the previous build, save packages > > downloaded by the bootstraps, sbuilds and imagers into `base-apt`. > > =C2=A0 - Use `base-apt` for bootstrapping, building and image creation. > >=20 > > Some issues with the current approach: > >=20 > > 1. Different policies must be followed for the first and the > > subsequent builds. > > 2. As we have multiple versions of the same package from the main > > and > > security repositories and rely on build logs and `find` for > > populating `base-apt`, extra care must be taken to ensure that the > > right package version lands in `base-apt`. > > 3. We rely on internal implementation of `debootstrap` and `sbuild` > > for saving and reusing the packages. Changing to e.g. `mmdebstrap` > > breaks the unrelated `base-apt` functionality. >=20 > Hi, what is the overall plan w.r.t. this patch series? My > understanding > was that we want to switch to mmdebstrap rather sooner than later. > But > now you write that this will be incompatible with mmdebstrap. >=20 Hello. "Rely on internal implementation" means that we currently workaround some things related to deb-dl-import / deb-dl-export : - sbuild: 2-stage deb cache import/export: (downloads/deb =3D> import to workdir/rootfs =3D> symlinking to apt cache in sbuild chroot =3D> build =3D= > export to workdir/rootfs =3D> export to downloads/deb): - mmdebstrap: since it is not possible to mmdebstrap into non-empty dir, we can't simply use deb-dl-import before mmdebstrap called and have to use some hooks to workaround it. With new base-apt approach, deb-dl import/export (and all related workarounds) becomes redundant. When new approach becomes stable enough we could completely remove fallback mode with deb-dl import/export functionality. > For me, this series still looks pretty experimental. That's why currently we saved fallback mode when things should work like before (ISAR_PREFETCH_BASE_APT =3D "0") >=20 > Felix >=20 > > 4. Source packages are stored in a different flat directory, `apt- > > get > > source` for upstream packages is not possible. > > 5. At the moment of `base-apt` creation all we have is the package > > name. The knowledge about the upstream repositories is lost and no > > local repository policy implementation is possible (e.g., for the > > "multiple products, multiple distros" use case). > > 6. For implementing further use cases like "fetch all sources > > necessary for bootstrapping the base system itself", additional > > logic > > is necessary. > >=20 >=20 > --=20 > Siemens AG, Technology > Linux Expert Center >=20 >=20 --=20 Best regards, Uladzimir. --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/= isar-users/8a9f43a239129e12f09ccfeda8b683b9f36a8fc2.camel%40ilbers.de.