From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6517599416265736192 X-Received: by 10.46.62.15 with SMTP id l15mr2696912lja.0.1517580938568; Fri, 02 Feb 2018 06:15:38 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.21.76 with SMTP id 12ls208215ljv.5.gmail; Fri, 02 Feb 2018 06:15:37 -0800 (PST) X-Google-Smtp-Source: AH8x224g980X1HBz+1mmSVLJ4zND8HhvJuv+QVG19lEhC9vKhcaKXAoQXvDV0DRSy+qS5BTkqtkk X-Received: by 10.46.0.163 with SMTP id e35mr2730287lji.31.1517580937828; Fri, 02 Feb 2018 06:15:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517580937; cv=none; d=google.com; s=arc-20160816; b=G6Lc36G9ue2xloL0Sg3DwsnmoOSYHr74iPuVNfU4MjlRiVcmFe4NFNY6g95CE0w8C3 NDeNicRtkfAuk28qQMHa8maYJO0jmFdjC/oYaVNzW4itPajCSYvw5B6rHA7mDkLGDQHH FrtKndtQsCyVg3ISXj9jHU8Ph/ekzoH/IdVkMZE1y3EOd3kqUq/AaJgDdg8Zk/EHKZYQ 5BgatsLI8lkwJuPmNNv637ebw3YfMoCWQ2Pkw3hKHkV9c2DUJO371crmy1nYngQz8NOH BKfkvGSROAsyx3ubQtOZQAL09L6audLThBbKNGinWFet23Y+ulc4a1CeZCnDTUaNIdFp 75jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:date:cc:to:from:subject :message-id:arc-authentication-results; bh=/UI0g3Aihbs0PNRJ0IIdkgb8ycAsytHukdgyGqnqQyQ=; b=lChKHjKGv1z2fqHNKT20T04h2LXem1WZwGHa6dMcESZNOCgSlD9rK2f3dfb3aIBhK1 l3T8W+5+czY/qoPEytJ5DUeZodawbBbZUsKUAkm1nKUj2DkQ9yl2MJC2Gglz3ahthfo9 42Zmk4nowziF3xFVCesRaoXmlAcTDXz8H1v8r4fduSjro1RQcCXkSMBXfx9v5V8w94MQ pdws6vbVEJM7f1pB9R1hn6Aplm/UMqnW01GPkW9llrrQBBgDewmlmPQQk3ZpJHmJt5DF MBQbnXQBFLOn3ccmbwHMGklQACx/pWOBvVcubiPJzlcT7BH33K9VtNQ5ZeH55pMhiJ8K 1ogA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [2001:a60:0:28:0:1:25:1]) by gmr-mx.google.com with ESMTPS id z11si147683ljc.3.2018.02.02.06.15.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Feb 2018 06:15:37 -0800 (PST) Received-SPF: neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=2001:a60:0:28:0:1:25:1; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 2001:a60:0:28:0:1:25:1 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 3zXzXn0ZR6z1qtNZ; Fri, 2 Feb 2018 15:15:37 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 3zXzXn0FLjz1r2vG; Fri, 2 Feb 2018 15:15:37 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id kVKrYLQ-Hrh4; Fri, 2 Feb 2018 15:15:35 +0100 (CET) X-Auth-Info: 74U2Yu7Q1XEiGl3C5FZQ5E9Hlk6q+WZuw7kIPh04FQE= Received: from Orrorin.lan (ipservice-092-217-126-008.092.217.pools.vodafone-ip.de [92.217.126.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Fri, 2 Feb 2018 15:15:35 +0100 (CET) Message-ID: <1517580931.2646.70.camel@denx.de> Subject: Re: Multi repo support From: Claudius Heine To: Alexander Smirnov , "[ext] Claudius Heine" Cc: isar-users Date: Fri, 02 Feb 2018 15:15:31 +0100 In-Reply-To: <8d4af596-6ad6-a0ac-9acd-bcbe0b435826@ilbers.de> References: <7714f0e1-aaca-add2-eabc-738d4043c21c@siemens.com> <20180201161658.6b0af973@mmd1pvb1c.ad001.siemens.net> <20180201193458.24b6ac3f@mmd1pvb1c.ad001.siemens.net> <1517511108.2646.24.camel@denx.de> <1517519259.2646.39.camel@denx.de> <161535ba350.27ac.034a6b0541ed39b7fb4e17f4ac219eaa@ilbers.de> <1517553656.2646.52.camel@denx.de> <23593adb-8fb1-f777-66bf-ced97dd26650@ilbers.de> <1517577996.2646.65.camel@denx.de> <8d4af596-6ad6-a0ac-9acd-bcbe0b435826@ilbers.de> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-fbfG8jjKwI44MOyGvwcm" X-Mailer: Evolution 3.26.4 Mime-Version: 1.0 X-TUID: P0i071Xm0iFg --=-fbfG8jjKwI44MOyGvwcm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote: > Hi, >=20 > On 02/02/2018 04:26 PM, Claudius Heine wrote: > > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote: > > > Hi again! > > >=20 > > > > Hi, > > > >=20 > > > > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote: > > > > > Hi! > > > > >=20 > > > > > Claudius Heine 2 =D1=84=D0=B5=D0=B2=D1=80=D0=B0=D0= =BB=D1=8F 2018 =D0=B3. 0:07:49 > > > > > =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: > > > > >=20 > > > > > > Hi Alex, > > > > > >=20 > > > > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote: > > > > > > >=20 > > > > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote: > > > > > > > > Hi, > > > > > > > >=20 > > > > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild > > > > > > > > wrote: > > > > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100 > > > > > > > > > schrieb "[ext] Henning Schild" > > > > > > > > s.co > > > > > > > > > m>: > > > > > > > > >=20 > > > > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100 > > > > > > > > > > schrieb "[ext] Claudius Heine" > > > > > > > > > siem > > > > > > > > > > ens. > > > > > > > > > > com> > > > > > > > > > > : > > > > > > > > > >=20 > > > > > > > > > > > Hi, > > > > > > > > > > >=20 > > > > > > > > > > > I would like to start the discussion about how to > > > > > > > > > > > best > > > > > > > > > > > implement > > > > > > > > > > > muti repository support in isar. > > > > > > > > > > >=20 > > > > > > > > > > > Does someone already has some ideas or even > > > > > > > > > > > something > > > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > pipeline for this? > > > > > > > > > > >=20 > > > > > > > > > > > If not then I do have an idea that was outlined > > > > > > > > > > > together > > > > > > > > > > > with > > > > > > > > > > > Jan: > > > > > > > > > > >=20 > > > > > > > > > > > Adding and configuring apt repositories should be > > > > > > > > > > > done > > > > > > > > > > > via > > > > > > > > > > > config > > > > > > > > > > > files. It should be possible to define own > > > > > > > > > > > multiconfigs > > > > > > > > > > > while > > > > > > > > > > > including multiconfigs from other layers. These > > > > > > > > > > > configs > > > > > > > > > > > then > > > > > > > > > > > append > > > > > > > > > > > filepaths to a global variable. > > > > > > > > > > >=20 > > > > > > > > > > > Every file that is added this way contains > > > > > > > > > > > 'sources.list' > > > > > > > > > > > compatible repository definitions. So one repo > > > > > > > > > > > each > > > > > > > > > > > line. > > > > > > > > > > >=20 > > > > > > > > > > > For every line in those files a repository entry > > > > > > > > > > > for > > > > > > > > > > > multistrap.conf > > > > > > > > > > > is created. Here we might need some more complex > > > > > > > > > > > code > > > > > > > > > > > to > > > > > > > > > > > convert > > > > > > > > > > > such a apt repo tripel to the right format > > > > > > > > > > > multistrap > > > > > > > > > > > expects. > > > > > > > > > > > But > > > > > > > > > > > by using the 'sources.lists' format we would be > > > > > > > > > > > independent > > > > > > > > > > > of > > > > > > > > > > > multistrap and become more future proof. > > > > > > > > >=20 > > > > > > > > > We will also need a way to tell apt the priorities of > > > > > > > > > these > > > > > > > > > repos. > > > > > > > > > Multistrap just adds them and apt-get installs > > > > > > > > > packages > > > > > > > > > according > > > > > > > > > to > > > > > > > > > its default behavior. > > > > > > > > > That means that a package with the same name and > > > > > > > > > version > > > > > > > > > will > > > > > > > > > get > > > > > > > > > picked from a "random" repo. Overlays will need to > > > > > > > > > make > > > > > > > > > sure > > > > > > > > > to > > > > > > > > > always > > > > > > > > > have a greater version or we will need apt > > > > > > > > > configuration. > > > > > > > > > https://wiki.debian.org/AptPreferences > > > > > > > > > The big question here is how/whether multistrap will > > > > > > > > > handle > > > > > > > > > apt > > > > > > > > > preference. > > > > > > > > >=20 > > > > > > > > > Alex already ran into that when he wanted to modify > > > > > > > > > "hello". > > > > > > > > > Renaming > > > > > > > > > packages - like Alex suggested - is not the way to > > > > > > > > > go. > > > > > > > > > Because > > > > > > > > > the > > > > > > > > > ones > > > > > > > > > we do overlay could be deps of packages we do not > > > > > > > > > overlay. > > > > > > >=20 > > > > > > > The version collision could be avoided by using epoch, > > > > > > > and if > > > > > > > I > > > > > > > understand it correctly - this is the preferable way to > > > > > > > assign > > > > > > > apt > > > > > > > priority. > > > > > > >=20 > > > > > > > https://www.debian.org/doc/debian-policy/#s-f-version > > > > > >=20 > > > > > > Where did you get that epoch is the preferable way to > > > > > > assign > > > > > > apt > > > > > > priority? > > > > >=20 > > > > > Not the apt, just specific package. Hmm, I'm a bit > > > > > incorrectly > > > > > called > > > > > my > > > > > pov. :-) I see epoch as an easy way to patch existing package > > > > > from > > > > > upstream. You don't care about upstream one's version even if > > > > > it > > > > > goes > > > > > forward, until your epoch is higher - your package will be > > > > > installed. > > > > > Of > > > > > course it doesn't mean that upstream epoch couldn't be > > > > > changed > > > > > too, > > > > > but > > > > > this happens quite rare. > > > > >=20 > > > > > >=20 > > > > > > From you link: > > > > > >=20 > > > > > > epoch > > > > > > This is a single (generally small) unsigned > > > > > > integer. > > > > > > It may > > > > > > be > > > > > > omitted, in which case zero is assumed. If it is omitted > > > > > > then > > > > > > the > > > > > > upstream_version may not contain any colons. > > > > > > It is provided to allow mistakes in the version > > > > > > numbers of > > > > > > older versions of a package, and also a package=E2=80=99s previ= ous > > > > > > version > > > > > > numbering schemes, to be left behind. > > > > > >=20 > > > > > > I see epoch just as a possibility to fix mistakes or if > > > > > > upstream > > > > > > changed their versioning scheme, not use it use it > > > > > > generally > > > > > > for > > > > > > specifying package priorities. Using it that way sound more > > > > > > like a > > > > > > hack > > > > > > to me. Because what happens if debian raises the epoch of a > > > > > > package? > > > > > > Then you would have to raise yours again? Sounds like > > > > > > playing > > > > > > poker. > > > > >=20 > > > > > Could you please describe the case when you have two > > > > > repositories > > > > > which > > > > > contain package with the same name-version but different > > > > > content? > > > >=20 > > > > With apt-preferences we could prefer every package from our own > > > > repo > > > > with the same name AFAIK, but this would make the original > > > > package > > > > no > > > > longer install-able beside that. > > > > And that is a limitation of debian/dpkg and we can do nothing > > > > about > > > > it > > > > in isar. This problem it not ours to solve. > > > >=20 > > > > Ideally debian would introduce package namespaces or similar. > > > > But > > > > that > > > > is completely out of isars scope. On our side we could only try > > > > some > > > > hacks to avoid name collision, but I would rather spend time > > > > solving > > > > other problems in isar than that, since just choosing a > > > > different > > > > name > > > > is pretty easy. Multi-repo or reproducible builds are much more > > > > important issues than trying to play cat and mouse with package > > > > names. > > > >=20 > > >=20 > > > I see this. But what is the practical usecase to have the same > > > packages > > > in several repositories? > >=20 > > Where did this question comes from? It does not really make sense > > to >=20 > From the text above, but I found that it wasn't your message, sorry > :-) >=20 > ... > That means that a package with the same name and version will get > picked from a "random" repo. > ... >=20 > > have exactly the same package in multiple different repositories. > > But > > that is AFAIK not related to any of this. > >=20 >=20 > So if it's not the case, then it's Ok for me. >=20 > > If you have changed some existing debian package put it into your > > own > > repo, maybe change the version of the package to contain something > > like > > "*~isar0" similar to how ubuntu does it. Then set a apt-preference > > like > > this: > >=20 > > Package: * > > Pin: release c=3Disar > > Pin-Priority: 650 > >=20 > > or similar. And apt would prefer to use packages from the isar repo > > instead of upstream if there is a name collision. Even if the > > version > > of the package in isar is lower than the one upstream. >=20 > Do you think it's possible to handle the following scenario: >=20 > 1. You pick current glibc from upstream distro XYZ, let's say v1.0. > 2. Patch it and put glibc-v1.0-isar to local repo. > 3. Wait for some time until glibc is updated in upstream, let's say > to v1.1. > 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be=20 > installed. But glibc is the core library in distro, and lots of > packages=20 > in XYZ now require glibc-v1.1. So you won't be able to generate > image=20 > anymore, it will fail with unmet dependency. Why do we need to handle this? It would be great if image generation fails in this case and not cause the generation of a broken image. Otherwise that is the reason we should have reproducible builds, so that we can still create the old image. Cheers, Claudius >=20 > Alex >=20 > >=20 > > Cheers, > > Claudius > >=20 > >=20 > > >=20 > > > Alex > > >=20 > > > > > > Telling apt to prefer packages with specific versions or > > > > > > from > > > > > > specific > > > > > > repositories via apt-preferences sounds more like the right > > > > > > tool > > > > > > for > > > > > > the job. > > > > > >=20 > > > > > > Cheers, > > > > > > Claudius > > > > > >=20 > > > > > > >=20 > > > > > > > Regarding renaming 'hello' there was a bit different > > > > > > > case, > > > > > > > our > > > > > > > 'hello' > > > > > > > has nothing common with the upstream one, moreover I've > > > > > > > already > > > > > > > modified > > > > > > > it to use 'libhello'. So this is not the case when we > > > > > > > patch > > > > > > > the > > > > > > > upstream > > > > > > > package. So I decided to rename it, to avoid confusion. > > > > > > >=20 > > > > > > > In case of patching the upstream package, we could just > > > > > > > add > > > > > > > epoch > > > > > > > to > > > > > > > version, so our package will be always in first prio. > > > > > > >=20 > > > > > > > Also I'm not sure that when there are 2 packages with the > > > > > > > same > > > > > > > names > > > > > > > and > > > > > > > versions available in source.list - is a good practice. > > > > > > > In > > > > > > > this > > > > > > > case > > > > > > > you > > > > > > > have no way to easily check if your rootfs contains > > > > > > > correct > > > > > > > package, > > > > > > > 'dpkg -l' won't be enough. > > > > > > >=20 > > > > > > > Alex > > > > > > >=20 > > > > > > > > Thanks, yes that is a very good point. > > > > > > > >=20 > > > > > > > > I would prefer the apt preferences settings. Maybe > > > > > > > > handle > > > > > > > > it > > > > > > > > similar to > > > > > > > > how I proposed the multi-repo support. A global > > > > > > > > variable > > > > > > > > where > > > > > > > > apt > > > > > > > > preference file paths are appended. > > > > > > > >=20 > > > > > > > > If multistrap cannot support them with a reasonable > > > > > > > > complex > > > > > > > > conversion > > > > > > > > script then it might be time to say multistrap goodbye. > > > > > > > >=20 > > > > > > > > Claudius > > > > > > > >=20 > > > > > >=20 > > > > > > -- > > > > > > DENX Software Engineering GmbH, Managing Director: > > > > > > Wolfgang > > > > > > Denk > > > > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 > > > > > > Groebenzell, > > > > > > Germany > > > > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email:=20 > > > > > > ch@d > > > > > > enx. > > > > > > de > > > > > >=20 > > > > > > PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 > > > > > > CB19 > > > > > > 9808 > > > > > > B153 > > > > > > Keyserver: hkp://pool.sks- > > > > > > keyservers.net > > > > >=20 > > > > >=20 > > >=20 > > >=20 --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --=-fbfG8jjKwI44MOyGvwcm Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlp0coMACgkQEXPLGZgI sVN5qg//WAuivAQXmkbOmxcTVa6j3of+vUvWAtQPWHu/hwYxBQsij3HtAlAu42qb Fx7xV0laENEkHPTUJZFXAq6BaBpNYvIE0rHJKXIvWYn1gGPKk9lnxwAlYoK2Kxtx RC8JG9yrHtOqfZKOWBN1zrw3P+DbqMuiuUtI4MEKaKIc1pbi9HWpjTOCWR8y0GIZ zIhNo95SqPGkaOKCl+BEa1CcwRy/gv0wU1cKciD4CEkGYsQQHYALSGtdm6nYKrFQ FwA25bsE9379WuFcFG4cYGioADVh4hGBjiHzcDq9vRBK1PPSvnfLTA/KRzIVon8c jsHsX2hL1cIYYiHMsqSYqDrAJU6Kfa7g5RNnm00SV2NS2W9UoCCRgVc0zyC/042x kh81khkrUSEoZ4rSJldOOYNbe1tIXlZnZ7jzUInxbzqNff/XflpNbIKXAaLz7lSx 1wFUsIs6dSpsVGmxkzjED0loa7C4bQcKUziU2pf4GGuMxvXMlP8SUPdO8W+AHaW9 8hXAa5jBmnbpI+zFrqtLCruZ9gaH05tuWB57OHUeETYabVyu6P7sVP7fpOMUA6h8 LRvkOUcWxxGTdY8n0D4kt515ESXLZPo4RKDEAvjq+eQg2ArKKRUNJq/yVTHSb5kh RWO86wPMraZruv0TaoXJgQkf7YEAd6thqTfwd/AsQstDoBCHg+g= =mZlT -----END PGP SIGNATURE----- --=-fbfG8jjKwI44MOyGvwcm--