From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6517599416265736192 X-Received: by 10.28.3.11 with SMTP id 11mr4430448wmd.4.1517579312962; Fri, 02 Feb 2018 05:48:32 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.26.132 with SMTP id a126ls369628wma.1.canary-gmail; Fri, 02 Feb 2018 05:48:32 -0800 (PST) X-Google-Smtp-Source: AH8x225dSk5FVTsoGelyZA8W/x/NtHwZsaq5zS0sBY3y1B4+V8/DjNVCR2oKwEUHc7v2wTm+BjBf X-Received: by 10.28.108.12 with SMTP id h12mr4043875wmc.25.1517579312424; Fri, 02 Feb 2018 05:48:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517579312; cv=none; d=google.com; s=arc-20160816; b=cchTwhBkJgowf/RIQxMtOBTcDb4FKUr3gINisNAPohU57nZt7aeiX0OnhK1cweek7V 9FVaihRN8yP9upvcK6rv4ZOBKknVjAzrmEuzgnLlRuTVXBw1QMGr6COglEBo4Fxc2CXt F3ik1LCB3+5viEuhtkIyz1q/nXXwCY6fLvNqUwaV0auDfgUuQtLnqIKuQEsckwQxZgiJ YOEDhc0SILV852cEyXVALWBq++TH/eLGFpxORzRDdLpPiHAOBBZRAts1dOBUhlDtCV2i 0Bybjs+M0pqKNZGC886h16Pro2mkGVgPirZtPIiA/msjFoGKRKA1SK48i5CQHuGh+/w2 M4tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=0nd8RpIz4XWv0WFC6Wf0kw9QcdOQA0riRzBMelZfok4=; b=CdC72h7GSYWI0wRlcpUuW6RMNc71SgsFPRGZMLs0032r8TxIlqNwUaBqvyLGIXe7LE Hzjbb5B3fhxpGtdd+LZCrPYmvBnItWRjKboll/y54lByQLqgUr8eB/f4mEeB8hcIQCk3 qaCZmZ+IglydEbKc6eqNGUOoXbJctG0rav8z4quEXCQYZmVopauoxERD2Qr9689tnsug SVZ6weaZswygi0GMzqnhBooDKb6H3RvnA9y1Akt2xCEs2FmNw9zMkAUd3A0jrraQKEr/ X5YNwaT8l3rICYozvNDPsxy+LgZmwHI4Orqg5jD/3W714jNSWcNkMobchM6YMBhRhiJU ASPA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id k21si131979wrd.1.2018.02.02.05.48.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Feb 2018 05:48:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w12DmSGf015163 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 Feb 2018 14:48:30 +0100 Subject: Re: Multi repo support To: Claudius Heine , "[ext] Claudius Heine" Cc: isar-users References: <7714f0e1-aaca-add2-eabc-738d4043c21c@siemens.com> <20180201161658.6b0af973@mmd1pvb1c.ad001.siemens.net> <20180201193458.24b6ac3f@mmd1pvb1c.ad001.siemens.net> <1517511108.2646.24.camel@denx.de> <1517519259.2646.39.camel@denx.de> <161535ba350.27ac.034a6b0541ed39b7fb4e17f4ac219eaa@ilbers.de> <1517553656.2646.52.camel@denx.de> <23593adb-8fb1-f777-66bf-ced97dd26650@ilbers.de> <1517577996.2646.65.camel@denx.de> From: Alexander Smirnov Message-ID: <8d4af596-6ad6-a0ac-9acd-bcbe0b435826@ilbers.de> Date: Fri, 2 Feb 2018 16:48:23 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1517577996.2646.65.camel@denx.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: JS77Ohwn2WTA Hi, On 02/02/2018 04:26 PM, Claudius Heine wrote: > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote: >> Hi again! >> >>> Hi, >>> >>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote: >>>> Hi! >>>> >>>> Claudius Heine 2 февраля 2018 г. 0:07:49 написал: >>>> >>>>> Hi Alex, >>>>> >>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote: >>>>>> >>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote: >>>>>>> Hi, >>>>>>> >>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote: >>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100 >>>>>>>> schrieb "[ext] Henning Schild" >>>>>>> m>: >>>>>>>> >>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100 >>>>>>>>> schrieb "[ext] Claudius Heine" >>>>>>>> ens. >>>>>>>>> com> >>>>>>>>> : >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I would like to start the discussion about how to >>>>>>>>>> best >>>>>>>>>> implement >>>>>>>>>> muti repository support in isar. >>>>>>>>>> >>>>>>>>>> Does someone already has some ideas or even something >>>>>>>>>> in >>>>>>>>>> the >>>>>>>>>> pipeline for this? >>>>>>>>>> >>>>>>>>>> If not then I do have an idea that was outlined >>>>>>>>>> together >>>>>>>>>> with >>>>>>>>>> Jan: >>>>>>>>>> >>>>>>>>>> Adding and configuring apt repositories should be >>>>>>>>>> done >>>>>>>>>> via >>>>>>>>>> config >>>>>>>>>> files. It should be possible to define own >>>>>>>>>> multiconfigs >>>>>>>>>> while >>>>>>>>>> including multiconfigs from other layers. These >>>>>>>>>> configs >>>>>>>>>> then >>>>>>>>>> append >>>>>>>>>> filepaths to a global variable. >>>>>>>>>> >>>>>>>>>> Every file that is added this way contains >>>>>>>>>> 'sources.list' >>>>>>>>>> compatible repository definitions. So one repo each >>>>>>>>>> line. >>>>>>>>>> >>>>>>>>>> For every line in those files a repository entry for >>>>>>>>>> multistrap.conf >>>>>>>>>> is created. Here we might need some more complex code >>>>>>>>>> to >>>>>>>>>> convert >>>>>>>>>> such a apt repo tripel to the right format multistrap >>>>>>>>>> expects. >>>>>>>>>> But >>>>>>>>>> by using the 'sources.lists' format we would be >>>>>>>>>> independent >>>>>>>>>> of >>>>>>>>>> multistrap and become more future proof. >>>>>>>> >>>>>>>> We will also need a way to tell apt the priorities of >>>>>>>> these >>>>>>>> repos. >>>>>>>> Multistrap just adds them and apt-get installs packages >>>>>>>> according >>>>>>>> to >>>>>>>> its default behavior. >>>>>>>> That means that a package with the same name and version >>>>>>>> will >>>>>>>> get >>>>>>>> picked from a "random" repo. Overlays will need to make >>>>>>>> sure >>>>>>>> to >>>>>>>> always >>>>>>>> have a greater version or we will need apt configuration. >>>>>>>> https://wiki.debian.org/AptPreferences >>>>>>>> The big question here is how/whether multistrap will >>>>>>>> handle >>>>>>>> apt >>>>>>>> preference. >>>>>>>> >>>>>>>> Alex already ran into that when he wanted to modify >>>>>>>> "hello". >>>>>>>> Renaming >>>>>>>> packages - like Alex suggested - is not the way to go. >>>>>>>> Because >>>>>>>> the >>>>>>>> ones >>>>>>>> we do overlay could be deps of packages we do not >>>>>>>> overlay. >>>>>> >>>>>> The version collision could be avoided by using epoch, and if >>>>>> I >>>>>> understand it correctly - this is the preferable way to >>>>>> assign >>>>>> apt >>>>>> priority. >>>>>> >>>>>> https://www.debian.org/doc/debian-policy/#s-f-version >>>>> >>>>> Where did you get that epoch is the preferable way to assign >>>>> apt >>>>> priority? >>>> >>>> Not the apt, just specific package. Hmm, I'm a bit incorrectly >>>> called >>>> my >>>> pov. :-) I see epoch as an easy way to patch existing package >>>> from >>>> upstream. You don't care about upstream one's version even if it >>>> goes >>>> forward, until your epoch is higher - your package will be >>>> installed. >>>> Of >>>> course it doesn't mean that upstream epoch couldn't be changed >>>> too, >>>> but >>>> this happens quite rare. >>>> >>>>> >>>>> From you link: >>>>> >>>>> epoch >>>>> This is a single (generally small) unsigned integer. >>>>> It may >>>>> be >>>>> omitted, in which case zero is assumed. If it is omitted then >>>>> the >>>>> upstream_version may not contain any colons. >>>>> It is provided to allow mistakes in the version >>>>> numbers of >>>>> older versions of a package, and also a package’s previous >>>>> version >>>>> numbering schemes, to be left behind. >>>>> >>>>> I see epoch just as a possibility to fix mistakes or if >>>>> upstream >>>>> changed their versioning scheme, not use it use it generally >>>>> for >>>>> specifying package priorities. Using it that way sound more >>>>> like a >>>>> hack >>>>> to me. Because what happens if debian raises the epoch of a >>>>> package? >>>>> Then you would have to raise yours again? Sounds like playing >>>>> poker. >>>> >>>> Could you please describe the case when you have two repositories >>>> which >>>> contain package with the same name-version but different content? >>> >>> With apt-preferences we could prefer every package from our own >>> repo >>> with the same name AFAIK, but this would make the original package >>> no >>> longer install-able beside that. >>> And that is a limitation of debian/dpkg and we can do nothing about >>> it >>> in isar. This problem it not ours to solve. >>> >>> Ideally debian would introduce package namespaces or similar. But >>> that >>> is completely out of isars scope. On our side we could only try >>> some >>> hacks to avoid name collision, but I would rather spend time >>> solving >>> other problems in isar than that, since just choosing a different >>> name >>> is pretty easy. Multi-repo or reproducible builds are much more >>> important issues than trying to play cat and mouse with package >>> names. >>> >> >> I see this. But what is the practical usecase to have the same >> packages >> in several repositories? > > Where did this question comes from? It does not really make sense to From the text above, but I found that it wasn't your message, sorry :-) ... That means that a package with the same name and version will get picked from a "random" repo. ... > have exactly the same package in multiple different repositories. But > that is AFAIK not related to any of this. > So if it's not the case, then it's Ok for me. > If you have changed some existing debian package put it into your own > repo, maybe change the version of the package to contain something like > "*~isar0" similar to how ubuntu does it. Then set a apt-preference like > this: > > Package: * > Pin: release c=isar > Pin-Priority: 650 > > or similar. And apt would prefer to use packages from the isar repo > instead of upstream if there is a name collision. Even if the version > of the package in isar is lower than the one upstream. Do you think it's possible to handle the following scenario: 1. You pick current glibc from upstream distro XYZ, let's say v1.0. 2. Patch it and put glibc-v1.0-isar to local repo. 3. Wait for some time until glibc is updated in upstream, let's say to v1.1. 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be installed. But glibc is the core library in distro, and lots of packages in XYZ now require glibc-v1.1. So you won't be able to generate image anymore, it will fail with unmet dependency. Alex > > Cheers, > Claudius > > >> >> Alex >> >>>>> Telling apt to prefer packages with specific versions or from >>>>> specific >>>>> repositories via apt-preferences sounds more like the right >>>>> tool >>>>> for >>>>> the job. >>>>> >>>>> Cheers, >>>>> Claudius >>>>> >>>>>> >>>>>> Regarding renaming 'hello' there was a bit different case, >>>>>> our >>>>>> 'hello' >>>>>> has nothing common with the upstream one, moreover I've >>>>>> already >>>>>> modified >>>>>> it to use 'libhello'. So this is not the case when we patch >>>>>> the >>>>>> upstream >>>>>> package. So I decided to rename it, to avoid confusion. >>>>>> >>>>>> In case of patching the upstream package, we could just add >>>>>> epoch >>>>>> to >>>>>> version, so our package will be always in first prio. >>>>>> >>>>>> Also I'm not sure that when there are 2 packages with the >>>>>> same >>>>>> names >>>>>> and >>>>>> versions available in source.list - is a good practice. In >>>>>> this >>>>>> case >>>>>> you >>>>>> have no way to easily check if your rootfs contains correct >>>>>> package, >>>>>> 'dpkg -l' won't be enough. >>>>>> >>>>>> Alex >>>>>> >>>>>>> Thanks, yes that is a very good point. >>>>>>> >>>>>>> I would prefer the apt preferences settings. Maybe handle >>>>>>> it >>>>>>> similar to >>>>>>> how I proposed the multi-repo support. A global variable >>>>>>> where >>>>>>> apt >>>>>>> preference file paths are appended. >>>>>>> >>>>>>> If multistrap cannot support them with a reasonable complex >>>>>>> conversion >>>>>>> script then it might be time to say multistrap goodbye. >>>>>>> >>>>>>> Claudius >>>>>>> >>>>> >>>>> -- >>>>> DENX Software Engineering GmbH, Managing Director: >>>>> Wolfgang >>>>> Denk >>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, >>>>> Germany >>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@d >>>>> enx. >>>>> de >>>>> >>>>> PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 >>>>> 9808 >>>>> B153 >>>>> Keyserver: hkp://pool.sks- >>>>> keyservers.net >>>> >>>> >> >>