From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7023151278367703040 X-Received: by 2002:ac2:51a3:: with SMTP id f3mr10996783lfk.137.1635277455136; Tue, 26 Oct 2021 12:44:15 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3499:: with SMTP id v25ls988110lfr.0.gmail; Tue, 26 Oct 2021 12:44:14 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwcpoi0XUecUi/HvfbqdeBrpd1CctFxID41BXwR+kQVSkMi3SMub01p5hA6kaBuGjZh6eqw X-Received: by 2002:ac2:4f02:: with SMTP id k2mr24228904lfr.635.1635277454036; Tue, 26 Oct 2021 12:44:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635277454; cv=none; d=google.com; s=arc-20160816; b=Cu3fdhSKmDpdnC+1XH2DpDLbXaWxx6JPfu1mBt54k13DZqd7UQdskDg7RwbhW9G7yd HIUN6jFyy2cd8ShLwdhZXHXQ9yN337/OCTj5IhBmrhwizeeT2zdbsFQA9SO9b29M0Npe sGXCDBuaNxu+3Fkrgx0KLk/TMcqc9VV4fKZkPyNOAO5nynPmz0YjE8aifJhc4xtu0ATx YiZM5H3MwkPADUeIAe8jZ0Ml7+ELEBaV/iJal1G41pe7BHmqHUstD/axOa5rZibojLz6 VNbBGtOg/h6dmJDLyOKUOOFDA/yI34LDZRl7N2Q96XmM4SzwNUH1ASzbEgNfIE7gAtHv M+uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=yJrRTOS9QinRkNDhtUInM2htrWItODzpKGF32fhzPU0=; b=fRs7Sn0ew4/FxAYNDS3CbWPeGKtIq+ZPCh4xGqmU7GrKpM4CYoFkP1ptiDlQwhVZul QoQRMW/fa3yvujI42NLDWS+TPOfWHG3LEzuwURNE8qUKuCFMERxMXcaxLAi4cqqOizy2 aQRAzq/kmacLAO3va2rnKiOoc9+VHVXv/O2YqZ6ICfb0ec+r79rH8KOJQqQSZpK77YHR rgBU5mgWsPOUT3m1FyvIt0sHM6+Vy/NQjuF1PnxC8nUr102hPMlmeHzj2YaflW3ljcdm 5xcGJ7zO89pS5G42Vpv0M3dLmLWBGMfItPGJzNP+MQcvtsReBHi5HwjhvOIjKgwOh9LA lz3w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id w6si1062182lfa.7.2021.10.26.12.44.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Oct 2021 12:44:13 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 19QJiCD0020089 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 26 Oct 2021 21:44:12 +0200 Received: from md1za8fc.ad001.siemens.net ([139.22.32.154]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 19QJiCOb024195; Tue, 26 Oct 2021 21:44:12 +0200 Date: Tue, 26 Oct 2021 21:44:08 +0200 From: Henning Schild To: ydirson@free.fr Cc: Anton Mikanovich , isar-users@googlegroups.com Subject: Re: isar-bootstrap Message-ID: <20211026214408.22030b2f@md1za8fc.ad001.siemens.net> In-Reply-To: <1737723933.1347608081.1635274696462.JavaMail.root@zimbra39-e7> References: <20211026094452.33fccc15@md1za8fc.ad001.siemens.net> <1737723933.1347608081.1635274696462.JavaMail.root@zimbra39-e7> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 8NMyA6a0oj7j Am Tue, 26 Oct 2021 20:58:16 +0200 (CEST) schrieb ydirson@free.fr: > Hi, > > Anton wrote: > > > I was trying to get a grasp on DISTRO_APT_SOURCES handling, and > > > noticed > > > a spurious match for "distro_apt_sources" in > > > DISTRO_APT_PREFERENCES handling. > > > While this possibly was meant to read "distro_apt_preferences" > > > instead. > > Cool :) > > Henning wrote: > > This was already fixed by `[PATCH v2] isar-bootstrap: Fix SRC_URI > > inclusion` and will be merged into next soon. > > > it raises the question of > > > whether it is useful to add all of this to SRC_URI if not adding > > > it did not cause any problem (it does not appear to be used by any > > > conf > > > in isar itself, but I guess it is/has been used in downstream > > > layers). > > > > > > This brought me to where this gets used (both _SOURCES and > > > _PREFERENCES in fact), ie. do_apt_config_prepare: the whole point > > > of > > > SRC_URI is to direct the fetcher to download/copy source files > > > into $WORKDIR, and then later tasks (unpack/patch in OE) take > > > files from there. But here, $WORKDIR is not used, and instead > > > aggregate_files() > > > uses resolve_file() on them. At first sight this seems to break > > > the > > > bitbake abstractions (and seems unnecessarily complicated at the > > > same > > > time). Am I missing something, or is it worth for me to try > > > patching > > > ? > > > > Not sure i get your point. In general we have to deal with two > > worlds in Isar. There is the debian-world and the bitbake world and > > they are not too tightly integrated at some points. > > Meaning we have debian fetching stuff and bitbake fetching stuff. We > > have two namespace for packages which leads to weird statements like > > > > # comma-sep > > DEBIAN_DEPENDS = "debian-binary-package-a, debian-binary-package-b" > > # space-sep > > DEPENDS = "recipe-for-binpkg-a recipe-for-binpkg-b" > > > > That two world can not be easily resolved and is found in several > > places, the one i pointed out being maybe the nasties one. > > At first sight I don't find this last part too problematic, this looks > like the standard > double source/binary namespace, and DEBIAN_DEPENDS rather looks like > a Yocto RDEPENDS (though processed), and DEPENDS a Debian > Build-Depends. (indeed Yocto packaging does generate Depends from > RDEPENDS when building a deb or ipk) Yes that is pretty much it. But Isar currently does not try to help here. In practice it is less of "big deal" just very confusing to new Isar users when they have to say things twice in different lingo. > > No need to > > comment on a possible idea, it has been tried and is more complex > > than i made it look. > > I guess this refers to the double-downloader problem, I had my share > of such things some time ago [0] (and finally found a rather > acceptable solution based on (IIRC) the meta-rust approach). I'd be > interested to learn about the approaches you tried. No i was talking about the src vs binary namespace, which in fact you have in Debian and OE alike. And Isar has 4 such namespaces i guess. But as pointed out that is less of a practical problem than a "confusion topic". If you go mostly debian and litte Isar (like 95/5) it is really just an academic thing and not worth too much effort. For the download Isar goes the pragmatic way and lets debian fetch what it wants. With a few exceptions ... i.e. there is only one "global apt-get update" so you have to hope that you can apt-get install what that initial run created your external database for. In practice that does not fail too ofter ... or you have to clean build again. If you really need to pin debian down to what it fetches, because for some reason (like repro build) you need your own mirror. In fact Isar spits out a partial debian mirror after an "online" build (base-apt). That can be used for consecutive offline builds, or as a base for consecutive "online" builds with custom DISTRO_APT_SOURCES. While snapshots.debian.org can be used as DISTRO_APT_SOURCES mirror in theory ... in practice it has rate-limiting in place. So you might succeed in a manual build that you retry over and over (or a small image), but in CI without caching ... you will never get a big image to build. That rate-limiting issue will need to be discussed with snapshots, we are not the first ones to have issues with it. But i personally would tell people to simply not freeze if they can, and the ones that need to freeze i would in fact tell to get a full debian mirror of their own, instead of a partial one produced by isars base-apt. As an OSS project you might see less of a need of freezing, tracking in fact is a security feature ... and debian will not do much more than security on their stable distros. regards, Henning > > That said you can not see everything in our code, the files placed > > carefully will be used by debootstrap and later by apt. > > Sure, such things do not lay themselves on (e)paper in one day, and > that can make them more difficult for newcomers to see all the > implications of a given construct - and also makes it easy to break > things. > > > > And the two variables are lists of files which are meant to be > > customizeable. That way you can switch to another mirror, add more > > repos (ppa in ubunut speach) and configure what to take from where > > or what to possibly pin (prefs) > > Yes, I realized this and was not suggesting to drop them and use > SRC_URI instead (although the idea did cross my mind ;). This remark > was purely from the bitbake side, and I realized thinking about it > later that some assumptions of mine may have been wrong, I'll have to > check this later. > > > > > On a related note, I assume that aggregate_files() was meant to > > > support distro versions predating preferences.d/ and > > > sources.list.d/ > > > - would it be correct to use the latter instead to simply things > > > further ? Most recent is preferences.d/, released with 0.7.22 in > > > 2009, and stretch which seems to be the oldest supported distro > > > has 1.4. > > > > I do not know, maybe also to be able to delete things again after > > the build. Because you want other configuration at build-time than > > you want > > on product delivery. i.e. build against a "frozen" mirror but > > deliver with a public mirror as update source > > OK I see. > > > > > > > > > The chroot-setup.sh script there also seems to do things that are > > > already handled by the debootstrap (policy.d and > > > start-stop-daemon), > > > except for the upstart support. Even if someone still uses > > > upstart (which has not been developped since 2014), probably the > > > rest could be omitted ? > > > > > > > Thanks for the feedback. But i guess it might be a little much mixed > > up in one email. > > Yes, I had to refrain from bringing too many subjects already :) > > > So i mainly answered to give you more information, not > > because i want to go and write patches. > > But maybe you at some point might want to. > > Yes, though I will need more time to get a full picture. > > [0] https://patchwork.openembedded.org/patch/178697/ > > Best regards,