From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6517599416265736192 X-Received: by 10.223.152.40 with SMTP id v37mr3423839wrb.13.1517582257442; Fri, 02 Feb 2018 06:37:37 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.108.16 with SMTP id h16ls406199wmc.1.gmail; Fri, 02 Feb 2018 06:37:36 -0800 (PST) X-Google-Smtp-Source: AH8x227/JqV2feX+sIXqOOSEF5ad6W1l1PZy7N3Q7eML6SMTkISul1/XvbMkV83Vjv3KLyfTL9oy X-Received: by 10.28.178.207 with SMTP id b198mr4532285wmf.0.1517582256927; Fri, 02 Feb 2018 06:37:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517582256; cv=none; d=google.com; s=arc-20160816; b=QmfvlXPZInnk2f6Px4O6CNIITryV80Z3xQXtFz502rcDEnR/exH4NGvjr1TAL5SABq lEXNMb4AOLAg/Mqp+b4CPlC2A6tOdngOoVbNF/TtKfzSZzg5r/W9XPN2BW9RLrDanb3d qVxAORYvVn0B9EbS9N4HHqViPA1apTYjzlDZYBHatuWTpFXJN2RFKOW8wydF18d4AKvC vCqHlgXWAhB5gHczZRXtckUKGl73iEVVIwzUCuMZY6plhu/fbO+EBU6+eUxwlxHzxIW2 M8aJdVdag0+VYGgD1l7CS94OiydKTGTeUcb/Z4C9RgvM4RltuxsgqkA3L7lC8H8jWrSS YBBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=M1wJEggljyh4471rKZu61ok1JoX1H3eeLolmJe7TNXA=; b=mdx+/vUyLXCpfMVj756mrFnJWwC97M/SWcaeYKOVBFg4oaJvOinob0zyQLfNM+A9pS hrwRKsB5ioTBl+pM5wkKx/aPg4HNT2gwn+w1dCfHtsvrRSuiwXXf1OPBdxHz6HvOzjem t/6J6rcxLLt2mgxRXtJSNZXkAfvtyIXg28ARiXLgsxW2462Ag7YEANQNsJ5vc0cVpuch 4k6ljI7qxEGMFfzW5exopQQoRyRTZdMiKKrskR+nJnWejU1dd1tdtTUB3HOwBorAwxGd GYnino3Nx2+Ze3YN77h7wsmZsTUIA0QLdvRx3+q8mGgl8RV8KwxJ1PQqQDHs9BHiDgU2 5udQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id c11si178180wrb.0.2018.02.02.06.37.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Feb 2018 06:37:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w12EaxIX015255 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 Feb 2018 15:37:34 +0100 Subject: Re: Multi repo support To: Claudius Heine , "[ext] Claudius Heine" Cc: isar-users References: <7714f0e1-aaca-add2-eabc-738d4043c21c@siemens.com> <20180201161658.6b0af973@mmd1pvb1c.ad001.siemens.net> <20180201193458.24b6ac3f@mmd1pvb1c.ad001.siemens.net> <1517511108.2646.24.camel@denx.de> <1517519259.2646.39.camel@denx.de> <161535ba350.27ac.034a6b0541ed39b7fb4e17f4ac219eaa@ilbers.de> <1517553656.2646.52.camel@denx.de> <23593adb-8fb1-f777-66bf-ced97dd26650@ilbers.de> <1517577996.2646.65.camel@denx.de> <8d4af596-6ad6-a0ac-9acd-bcbe0b435826@ilbers.de> <1517580931.2646.70.camel@denx.de> From: Alexander Smirnov Message-ID: <86c3d08e-e93b-0374-92ef-b43f89298815@ilbers.de> Date: Fri, 2 Feb 2018 17:36:54 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1517580931.2646.70.camel@denx.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: Se2TD6eefz+B On 02/02/2018 05:15 PM, Claudius Heine wrote: > On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote: >> Hi, >> >> On 02/02/2018 04:26 PM, Claudius Heine wrote: >>> On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote: >>>> Hi again! >>>> >>>>> Hi, >>>>> >>>>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote: >>>>>> Hi! >>>>>> >>>>>> Claudius Heine 2 февраля 2018 г. 0:07:49 >>>>>> написал: >>>>>> >>>>>>> Hi Alex, >>>>>>> >>>>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote: >>>>>>>> >>>>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild >>>>>>>>> wrote: >>>>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100 >>>>>>>>>> schrieb "[ext] Henning Schild" >>>>>>>>> s.co >>>>>>>>>> m>: >>>>>>>>>> >>>>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100 >>>>>>>>>>> schrieb "[ext] Claudius Heine" >>>>>>>>>> siem >>>>>>>>>>> ens. >>>>>>>>>>> com> >>>>>>>>>>> : >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I would like to start the discussion about how to >>>>>>>>>>>> best >>>>>>>>>>>> implement >>>>>>>>>>>> muti repository support in isar. >>>>>>>>>>>> >>>>>>>>>>>> Does someone already has some ideas or even >>>>>>>>>>>> something >>>>>>>>>>>> in >>>>>>>>>>>> the >>>>>>>>>>>> pipeline for this? >>>>>>>>>>>> >>>>>>>>>>>> If not then I do have an idea that was outlined >>>>>>>>>>>> together >>>>>>>>>>>> with >>>>>>>>>>>> Jan: >>>>>>>>>>>> >>>>>>>>>>>> Adding and configuring apt repositories should be >>>>>>>>>>>> done >>>>>>>>>>>> via >>>>>>>>>>>> config >>>>>>>>>>>> files. It should be possible to define own >>>>>>>>>>>> multiconfigs >>>>>>>>>>>> while >>>>>>>>>>>> including multiconfigs from other layers. These >>>>>>>>>>>> configs >>>>>>>>>>>> then >>>>>>>>>>>> append >>>>>>>>>>>> filepaths to a global variable. >>>>>>>>>>>> >>>>>>>>>>>> Every file that is added this way contains >>>>>>>>>>>> 'sources.list' >>>>>>>>>>>> compatible repository definitions. So one repo >>>>>>>>>>>> each >>>>>>>>>>>> line. >>>>>>>>>>>> >>>>>>>>>>>> For every line in those files a repository entry >>>>>>>>>>>> for >>>>>>>>>>>> multistrap.conf >>>>>>>>>>>> is created. Here we might need some more complex >>>>>>>>>>>> code >>>>>>>>>>>> to >>>>>>>>>>>> convert >>>>>>>>>>>> such a apt repo tripel to the right format >>>>>>>>>>>> multistrap >>>>>>>>>>>> expects. >>>>>>>>>>>> But >>>>>>>>>>>> by using the 'sources.lists' format we would be >>>>>>>>>>>> independent >>>>>>>>>>>> of >>>>>>>>>>>> multistrap and become more future proof. >>>>>>>>>> >>>>>>>>>> We will also need a way to tell apt the priorities of >>>>>>>>>> these >>>>>>>>>> repos. >>>>>>>>>> Multistrap just adds them and apt-get installs >>>>>>>>>> packages >>>>>>>>>> according >>>>>>>>>> to >>>>>>>>>> its default behavior. >>>>>>>>>> That means that a package with the same name and >>>>>>>>>> version >>>>>>>>>> will >>>>>>>>>> get >>>>>>>>>> picked from a "random" repo. Overlays will need to >>>>>>>>>> make >>>>>>>>>> sure >>>>>>>>>> to >>>>>>>>>> always >>>>>>>>>> have a greater version or we will need apt >>>>>>>>>> configuration. >>>>>>>>>> https://wiki.debian.org/AptPreferences >>>>>>>>>> The big question here is how/whether multistrap will >>>>>>>>>> handle >>>>>>>>>> apt >>>>>>>>>> preference. >>>>>>>>>> >>>>>>>>>> Alex already ran into that when he wanted to modify >>>>>>>>>> "hello". >>>>>>>>>> Renaming >>>>>>>>>> packages - like Alex suggested - is not the way to >>>>>>>>>> go. >>>>>>>>>> Because >>>>>>>>>> the >>>>>>>>>> ones >>>>>>>>>> we do overlay could be deps of packages we do not >>>>>>>>>> overlay. >>>>>>>> >>>>>>>> The version collision could be avoided by using epoch, >>>>>>>> and if >>>>>>>> I >>>>>>>> understand it correctly - this is the preferable way to >>>>>>>> assign >>>>>>>> apt >>>>>>>> priority. >>>>>>>> >>>>>>>> https://www.debian.org/doc/debian-policy/#s-f-version >>>>>>> >>>>>>> Where did you get that epoch is the preferable way to >>>>>>> assign >>>>>>> apt >>>>>>> priority? >>>>>> >>>>>> Not the apt, just specific package. Hmm, I'm a bit >>>>>> incorrectly >>>>>> called >>>>>> my >>>>>> pov. :-) I see epoch as an easy way to patch existing package >>>>>> from >>>>>> upstream. You don't care about upstream one's version even if >>>>>> it >>>>>> goes >>>>>> forward, until your epoch is higher - your package will be >>>>>> installed. >>>>>> Of >>>>>> course it doesn't mean that upstream epoch couldn't be >>>>>> changed >>>>>> too, >>>>>> but >>>>>> this happens quite rare. >>>>>> >>>>>>> >>>>>>> From you link: >>>>>>> >>>>>>> epoch >>>>>>> This is a single (generally small) unsigned >>>>>>> integer. >>>>>>> It may >>>>>>> be >>>>>>> omitted, in which case zero is assumed. If it is omitted >>>>>>> then >>>>>>> the >>>>>>> upstream_version may not contain any colons. >>>>>>> It is provided to allow mistakes in the version >>>>>>> numbers of >>>>>>> older versions of a package, and also a package’s previous >>>>>>> version >>>>>>> numbering schemes, to be left behind. >>>>>>> >>>>>>> I see epoch just as a possibility to fix mistakes or if >>>>>>> upstream >>>>>>> changed their versioning scheme, not use it use it >>>>>>> generally >>>>>>> for >>>>>>> specifying package priorities. Using it that way sound more >>>>>>> like a >>>>>>> hack >>>>>>> to me. Because what happens if debian raises the epoch of a >>>>>>> package? >>>>>>> Then you would have to raise yours again? Sounds like >>>>>>> playing >>>>>>> poker. >>>>>> >>>>>> Could you please describe the case when you have two >>>>>> repositories >>>>>> which >>>>>> contain package with the same name-version but different >>>>>> content? >>>>> >>>>> With apt-preferences we could prefer every package from our own >>>>> repo >>>>> with the same name AFAIK, but this would make the original >>>>> package >>>>> no >>>>> longer install-able beside that. >>>>> And that is a limitation of debian/dpkg and we can do nothing >>>>> about >>>>> it >>>>> in isar. This problem it not ours to solve. >>>>> >>>>> Ideally debian would introduce package namespaces or similar. >>>>> But >>>>> that >>>>> is completely out of isars scope. On our side we could only try >>>>> some >>>>> hacks to avoid name collision, but I would rather spend time >>>>> solving >>>>> other problems in isar than that, since just choosing a >>>>> different >>>>> name >>>>> is pretty easy. Multi-repo or reproducible builds are much more >>>>> important issues than trying to play cat and mouse with package >>>>> names. >>>>> >>>> >>>> I see this. But what is the practical usecase to have the same >>>> packages >>>> in several repositories? >>> >>> Where did this question comes from? It does not really make sense >>> to >> >> From the text above, but I found that it wasn't your message, sorry >> :-) >> >> ... >> That means that a package with the same name and version will get >> picked from a "random" repo. >> ... >> >>> have exactly the same package in multiple different repositories. >>> But >>> that is AFAIK not related to any of this. >>> >> >> So if it's not the case, then it's Ok for me. >> >>> If you have changed some existing debian package put it into your >>> own >>> repo, maybe change the version of the package to contain something >>> like >>> "*~isar0" similar to how ubuntu does it. Then set a apt-preference >>> like >>> this: >>> >>> Package: * >>> Pin: release c=isar >>> Pin-Priority: 650 >>> >>> or similar. And apt would prefer to use packages from the isar repo >>> instead of upstream if there is a name collision. Even if the >>> version >>> of the package in isar is lower than the one upstream. >> >> Do you think it's possible to handle the following scenario: >> >> 1. You pick current glibc from upstream distro XYZ, let's say v1.0. >> 2. Patch it and put glibc-v1.0-isar to local repo. >> 3. Wait for some time until glibc is updated in upstream, let's say >> to v1.1. >> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be >> installed. But glibc is the core library in distro, and lots of >> packages >> in XYZ now require glibc-v1.1. So you won't be able to generate >> image >> anymore, it will fail with unmet dependency. > > Why do we need to handle this? If you use upstream sources and prioritize apts, this problem likely will occur. So repo prioritizing is then unpredictable feature. > > It would be great if image generation fails in this case and not cause > the generation of a broken image. > > Otherwise that is the reason we should have reproducible builds, so > that we can still create the old image. If you don't work with upstream and use only local repositories, why can't you just drop duplicated packages from them? Alex >>>>>>> Telling apt to prefer packages with specific versions or >>>>>>> from >>>>>>> specific >>>>>>> repositories via apt-preferences sounds more like the right >>>>>>> tool >>>>>>> for >>>>>>> the job. >>>>>>> >>>>>>> Cheers, >>>>>>> Claudius >>>>>>> >>>>>>>> >>>>>>>> Regarding renaming 'hello' there was a bit different >>>>>>>> case, >>>>>>>> our >>>>>>>> 'hello' >>>>>>>> has nothing common with the upstream one, moreover I've >>>>>>>> already >>>>>>>> modified >>>>>>>> it to use 'libhello'. So this is not the case when we >>>>>>>> patch >>>>>>>> the >>>>>>>> upstream >>>>>>>> package. So I decided to rename it, to avoid confusion. >>>>>>>> >>>>>>>> In case of patching the upstream package, we could just >>>>>>>> add >>>>>>>> epoch >>>>>>>> to >>>>>>>> version, so our package will be always in first prio. >>>>>>>> >>>>>>>> Also I'm not sure that when there are 2 packages with the >>>>>>>> same >>>>>>>> names >>>>>>>> and >>>>>>>> versions available in source.list - is a good practice. >>>>>>>> In >>>>>>>> this >>>>>>>> case >>>>>>>> you >>>>>>>> have no way to easily check if your rootfs contains >>>>>>>> correct >>>>>>>> package, >>>>>>>> 'dpkg -l' won't be enough. >>>>>>>> >>>>>>>> Alex >>>>>>>> >>>>>>>>> Thanks, yes that is a very good point. >>>>>>>>> >>>>>>>>> I would prefer the apt preferences settings. Maybe >>>>>>>>> handle >>>>>>>>> it >>>>>>>>> similar to >>>>>>>>> how I proposed the multi-repo support. A global >>>>>>>>> variable >>>>>>>>> where >>>>>>>>> apt >>>>>>>>> preference file paths are appended. >>>>>>>>> >>>>>>>>> If multistrap cannot support them with a reasonable >>>>>>>>> complex >>>>>>>>> conversion >>>>>>>>> script then it might be time to say multistrap goodbye. >>>>>>>>> >>>>>>>>> Claudius >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> DENX Software Engineering GmbH, Managing Director: >>>>>>> Wolfgang >>>>>>> Denk >>>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 >>>>>>> Groebenzell, >>>>>>> Germany >>>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: >>>>>>> ch@d >>>>>>> enx. >>>>>>> de >>>>>>> >>>>>>> PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 >>>>>>> CB19 >>>>>>> 9808 >>>>>>> B153 >>>>>>> Keyserver: hkp://pool.sks- >>>>>>> keyservers.net