From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6750653680817537024 X-Received: by 2002:adf:db42:: with SMTP id f2mr3758676wrj.287.1571918371279; Thu, 24 Oct 2019 04:59:31 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:960a:: with SMTP id y10ls788214wmd.0.gmail; Thu, 24 Oct 2019 04:59:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqysOdoGsLgdZujFAIHLfSFYhaggZsf1idjTim5K1KzR5SaOVQrIsfpGYGfv3sAJG2V2VoJu X-Received: by 2002:a1c:7c16:: with SMTP id x22mr4748718wmc.177.1571918370688; Thu, 24 Oct 2019 04:59:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571918370; cv=none; d=google.com; s=arc-20160816; b=d+8y95BXPfLLmmi6/UUA3/+Ls4yG0fTZD6QVY7y3r1mG24xeWKoqYqUsiU4iqEVbSP i1N1UATgqmUR7/rlrXvoYmGUj2oyIUpsxPz1N6YDjQIAHhwluwd5ldHD4ymGcIFdePj2 m69ZQ/jZ7uNzrDRMcFLJtZcJH8BDqeJtiGWGtvqmlmtqnQyaDmeRyOdwmE6vOVYYoL5E ivuiUP4ecizpR0TrTWqXeslRdfFpDYhIk1YrY+X5q9bvXnZaa/svAPmPpo2E7vYtLPTZ wcnu+7q0WKuzcTZy7lGxprJhS2tiyt7F9g90x+Tgvkll90RPZ3mwRVSrrhB7pX/2FHuE j0xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=47YowkekuQmquFQvkrduPQzFCP1btvUDyE2NDUO2fpU=; b=N6P5I5HUCDq6JmlKNYiFXYmFAC1i18W2oStAWZ8jBw7YMhyNEaURkNBXuFAuJxS4Br KGHXncyM+g4GgP/CUK/TYxMFXi8eicoyXQ8MaUhSX9VdAGc/jWsLkuItm2M3V12hk3wz byZJzCZO2szEz6M6eXC48cX3nNDtAxGtftSXdcvMk1wpWrBaWDEXJ92SFZ6HJ1azGqbM owVDZ5GnnlalezSlnTbthZngi0+hwvZyg5z96+kZW+cQHcfKOUg5F9nvDqm9y+6bd/Gy EWgcPDv3n5sc6UA6chJRJTrwCu//aSk1MeUYhPY7YTZ9c9k/dF+B2cquX/W08KiOHJ9r ZVOg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id s15si154220wme.2.2019.10.24.04.59.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Oct 2019 04:59:30 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x9OBxUIZ027251 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Oct 2019 13:59:30 +0200 Received: from md1za8fc.ad001.siemens.net ([139.25.0.8]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x9OBxT9o008104; Thu, 24 Oct 2019 13:59:29 +0200 Date: Thu, 24 Oct 2019 13:59:29 +0200 From: Henning Schild To: Baurzhan Ismagulov Cc: Subject: Re: base-apt work in progress, RFC Message-ID: <20191024135929.7f67c88c@md1za8fc.ad001.siemens.net> In-Reply-To: <20191023202823.gn2vqsaokj2lfzmz@yssyq.m.ilbers.de> References: <20191022174359.40fb4045@md1za8fc.ad001.siemens.net> <20191023202823.gn2vqsaokj2lfzmz@yssyq.m.ilbers.de> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: Z5NkTt3IxAfn Am Wed, 23 Oct 2019 22:28:24 +0200 schrieb Baurzhan Ismagulov : > On Tue, Oct 22, 2019 at 05:43:59PM +0200, Henning Schild wrote: > > i now have a working version that changed a few things around the > > implementation of base-apt, to make sure it will also work when some > > rootfs clear their package cache etc. > > Ok, good to know. > > > > So far base-apt mixes together packages from several rootfss. If > > these rootfss had different versions of a package the latest > > version of that package would win. See > > meta/classes/image-cache-extension.bbclass, we are mixing the image > > with the two buildchroots. The preferences and sources in these > > three rootfs can lead to such mixing. When this base-apt gets used > > for a rebuild such newer packages can leak to rootfs where they > > were not in the first run. > > Do I understand correctly, you'd like to cover e.g. the use case > below? > > * multiconfig card1-arm64-buster uses buster with pkg1 1.0-10. > > * multiconfig card2-armhf-buster uses buster with an older pkg1 1.0-9. Isar does support preferences and different debian mirrors. And it supports to control what is used for which rootfs. So we happily download packages from "random" mirrors, random "components" etc. In the end we put everything into the base-apt under the BASE_DISTRO_CODENAME into component "main". So two images that are based on buster, but with different preferences, components or mirros will happily "share" packages they probably should not. Say you have a buster 10.0 which uses snapshot.debian.org (or a base-apt from a few weeks ago), that is your product version 1.0. For your devel version you use bleeding edge (upstream mirror). And maybe the components also differ ... The bleeding edge will win and your 1.0 will get those packages when rebuilding from base-apt. > This is a relatively complex policy issue. I'd suggest to start with > the simplest case first, then add features on top of that. > > > > I see two general directions to solve that: > > 1. align the sources and preferences across the 4-tuple image, > > buildchroots, sdk ... and have a base-apt for every image-4-tuple > > Could you please give an example of two such tuples? I don't see how > that differs from the image-MACHINE-DISTRO-ARCH case. Not that kind of tuple, sorry. Every image comes with 2 buildchroots and an sdk ... up to 4 rootfs related to one "image". > > 2. have a base-apt for every rootfs-PN-PV > > For the example above, would that mean the following? > > base-apt/debian-buster-pkg1-1.0-10/... > base-apt/debian-buster-pkg1-1.0-9/... > Every image, and its up to three sibbling rootfss. base-apt/image1-rootfs/ base-apt/image1-sdk-rootfs/ base-apt/image1-buildchroot-target-rootfs/ base-apt/image1-buildchroot-host-rootfs/ base-apt/image2-rootfs/ ... > > 3. if reprepro has deduplication accross distros/suites possibly > > have all "base-apts" in one repo, and make every instance a "suite" > > so instead of "buster" think > > "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0" > > Now if they in the end are all buster and use the same file, we only > > need to store it once and can still store differences. Same for > > sources, where this even works cross-arch. > > > > I think 1 is not a good idea, isar currently allows sources and > > preferences to differ and it is probably a wanted feature. > > I'd suggest to start cache_base_repo without preferences and add the > feature later. And when we add it, keep the cache_base_repo tooling > separate from the bootstrap tooling. I think the users would like to > use a certain codename (e.g., buster) with some explicit pinning > (e.g., pkg1 1.0-9). That could be described statically in e.g. a json > file, rather than resolving dynamically via preferences. After you've > cache_base_repo-ed that, apt would get the version in base-apt, > making preferences unnecessary. Isar has prefs and sources knobs. It is a feature currently not properly working with base-apt. Your rebuild might "work" but might not really use the same packages. We are at the state where base-apt does this "unwanted" mixing. Since i am trying to "fix" issues in base-apt, the question is how to deal with this one. Agreed, prefs do not make any sense once there is only one mirror and only one version each of the selection of packages. > > 2 is "clear" but we have a lot of those repos, which might be hard > > to handle, if we think people pull updates from there > > I'm not in favor of that, either. > > > > 3 is currently me preferred version if deduplication is in place, i > > think even without it > > Deduplication is possible with the Debian repository format. reprepro > supports it. I think the first step should be exactly how Debian does > it, i.e., BASE_DISTRO/dists/CODENAME, with different DISTRO_ARCH > packages in BASE_DISTRO/pool/*/*/*.deb, without preferences. I guess the real point with partially mirroring debian+optional other mirrors at a certain day is that you probably should invent your own CODENAME and must not use the debian one any longer. And i guess that would be "arch+machine+distro+imagePN". > > > I am not sure whether the solution to that problem should be part of > > the first patch-series. It is not a new problem ... > > In fact i will probably sent my current state without the "deep tree > > download", which would also be something on top. > > If it is not worse than the current implementation (which I hope is > the case) and passes the CI, I'd like to see it. No the whole "issue" is already existing but probably should be fixed. Or align the rootfs 4-tuple and ditch multiconfig. Henning > > With kind regards, > Baurzhan. >