Multi repo support

Multi repo support

[Claudius Heine]

Hi,

I would like to start the discussion about how to best implement muti 
repository support in isar.

Does someone already has some ideas or even something in the pipeline 
for this?

If not then I do have an idea that was outlined together with Jan:

Adding and configuring apt repositories should be done via config files. 
It should be possible to define own multiconfigs while including 
multiconfigs from other layers. These configs then append filepaths to a 
global variable.

Every file that is added this way contains 'sources.list' compatible 
repository definitions. So one repo each line.

For every line in those files a repository entry for multistrap.conf is 
created. Here we might need some more complex code to convert such a apt 
repo tripel to the right format multistrap expects. But by using the 
'sources.lists' format we would be independent of multistrap and become 
more future proof.

Comments are welcome.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Multi repo support

[Henning Schild]

Am Thu, 1 Feb 2018 15:54:26 +0100
schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:

> Hi,
> 
> I would like to start the discussion about how to best implement muti 
> repository support in isar.
> 
> Does someone already has some ideas or even something in the pipeline 
> for this?
> 
> If not then I do have an idea that was outlined together with Jan:
> 
> Adding and configuring apt repositories should be done via config
> files. It should be possible to define own multiconfigs while
> including multiconfigs from other layers. These configs then append
> filepaths to a global variable.
> 
> Every file that is added this way contains 'sources.list' compatible 
> repository definitions. So one repo each line.
>
> For every line in those files a repository entry for multistrap.conf
> is created. Here we might need some more complex code to convert such
> a apt repo tripel to the right format multistrap expects. But by
> using the 'sources.lists' format we would be independent of
> multistrap and become more future proof.
> 
> Comments are welcome.

Sounds good to me, cant wait to review the first patches ;).

Henning
 
> Cheers,
> Claudius
> 

Re: Multi repo support

[Henning Schild]

Am Thu, 1 Feb 2018 16:16:58 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Am Thu, 1 Feb 2018 15:54:26 +0100
> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:
> 
> > Hi,
> > 
> > I would like to start the discussion about how to best implement
> > muti repository support in isar.
> > 
> > Does someone already has some ideas or even something in the
> > pipeline for this?
> > 
> > If not then I do have an idea that was outlined together with Jan:
> > 
> > Adding and configuring apt repositories should be done via config
> > files. It should be possible to define own multiconfigs while
> > including multiconfigs from other layers. These configs then append
> > filepaths to a global variable.
> > 
> > Every file that is added this way contains 'sources.list'
> > compatible repository definitions. So one repo each line.
> >
> > For every line in those files a repository entry for multistrap.conf
> > is created. Here we might need some more complex code to convert
> > such a apt repo tripel to the right format multistrap expects. But
> > by using the 'sources.lists' format we would be independent of
> > multistrap and become more future proof.

We will also need a way to tell apt the priorities of these repos.
Multistrap just adds them and apt-get installs packages according to
its default behavior.
That means that a package with the same name and version will get
picked from a "random" repo. Overlays will need to make sure to always
have a greater version or we will need apt configuration.
https://wiki.debian.org/AptPreferences
The big question here is how/whether multistrap will handle apt
preference.

Alex already ran into that when he wanted to modify "hello". Renaming
packages - like Alex suggested - is not the way to go. Because the ones
we do overlay could be deps of packages we do not overlay.

Henning

> > Comments are welcome.  
> 
> Sounds good to me, cant wait to review the first patches ;).
> 
> Henning
>  
> > Cheers,
> > Claudius
> >   
> 

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 2826 bytes --]

Hi,

On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> Am Thu, 1 Feb 2018 16:16:58 +0100
> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> 
> > Am Thu, 1 Feb 2018 15:54:26 +0100
> > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:
> > 
> > > Hi,
> > > 
> > > I would like to start the discussion about how to best implement
> > > muti repository support in isar.
> > > 
> > > Does someone already has some ideas or even something in the
> > > pipeline for this?
> > > 
> > > If not then I do have an idea that was outlined together with
> > > Jan:
> > > 
> > > Adding and configuring apt repositories should be done via config
> > > files. It should be possible to define own multiconfigs while
> > > including multiconfigs from other layers. These configs then
> > > append
> > > filepaths to a global variable.
> > > 
> > > Every file that is added this way contains 'sources.list'
> > > compatible repository definitions. So one repo each line.
> > > 
> > > For every line in those files a repository entry for
> > > multistrap.conf
> > > is created. Here we might need some more complex code to convert
> > > such a apt repo tripel to the right format multistrap expects.
> > > But
> > > by using the 'sources.lists' format we would be independent of
> > > multistrap and become more future proof.
> 
> We will also need a way to tell apt the priorities of these repos.
> Multistrap just adds them and apt-get installs packages according to
> its default behavior.
> That means that a package with the same name and version will get
> picked from a "random" repo. Overlays will need to make sure to
> always
> have a greater version or we will need apt configuration.
> https://wiki.debian.org/AptPreferences
> The big question here is how/whether multistrap will handle apt
> preference.
> 
> Alex already ran into that when he wanted to modify "hello". Renaming
> packages - like Alex suggested - is not the way to go. Because the
> ones
> we do overlay could be deps of packages we do not overlay.

Thanks, yes that is a very good point.

I would prefer the apt preferences settings. Maybe handle it similar to
how I proposed the multi-repo support. A global variable where apt
preference file paths are appended.

If multistrap cannot support them with a reasonable complex conversion
script then it might be time to say multistrap goodbye.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Alexander Smirnov]

On 02/01/2018 09:51 PM, Claudius Heine wrote:
> Hi,
> 
> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>> Am Thu, 1 Feb 2018 16:16:58 +0100
>> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>>
>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:
>>>
>>>> Hi,
>>>>
>>>> I would like to start the discussion about how to best implement
>>>> muti repository support in isar.
>>>>
>>>> Does someone already has some ideas or even something in the
>>>> pipeline for this?
>>>>
>>>> If not then I do have an idea that was outlined together with
>>>> Jan:
>>>>
>>>> Adding and configuring apt repositories should be done via config
>>>> files. It should be possible to define own multiconfigs while
>>>> including multiconfigs from other layers. These configs then
>>>> append
>>>> filepaths to a global variable.
>>>>
>>>> Every file that is added this way contains 'sources.list'
>>>> compatible repository definitions. So one repo each line.
>>>>
>>>> For every line in those files a repository entry for
>>>> multistrap.conf
>>>> is created. Here we might need some more complex code to convert
>>>> such a apt repo tripel to the right format multistrap expects.
>>>> But
>>>> by using the 'sources.lists' format we would be independent of
>>>> multistrap and become more future proof.
>>
>> We will also need a way to tell apt the priorities of these repos.
>> Multistrap just adds them and apt-get installs packages according to
>> its default behavior.
>> That means that a package with the same name and version will get
>> picked from a "random" repo. Overlays will need to make sure to
>> always
>> have a greater version or we will need apt configuration.
>> https://wiki.debian.org/AptPreferences
>> The big question here is how/whether multistrap will handle apt
>> preference.
>>
>> Alex already ran into that when he wanted to modify "hello". Renaming
>> packages - like Alex suggested - is not the way to go. Because the
>> ones
>> we do overlay could be deps of packages we do not overlay.
> 

The version collision could be avoided by using epoch, and if I 
understand it correctly - this is the preferable way to assign apt priority.

https://www.debian.org/doc/debian-policy/#s-f-version

Regarding renaming 'hello' there was a bit different case, our 'hello' 
has nothing common with the upstream one, moreover I've already modified 
it to use 'libhello'. So this is not the case when we patch the upstream 
package. So I decided to rename it, to avoid confusion.

In case of patching the upstream package, we could just add epoch to 
version, so our package will be always in first prio.

Also I'm not sure that when there are 2 packages with the same names and 
versions available in source.list - is a good practice. In this case you 
have no way to easily check if your rootfs contains correct package, 
'dpkg -l' won't be enough.

Alex

> Thanks, yes that is a very good point.
> 
> I would prefer the apt preferences settings. Maybe handle it similar to
> how I proposed the multi-repo support. A global variable where apt
> preference file paths are appended.
> 
> If multistrap cannot support them with a reasonable complex conversion
> script then it might be time to say multistrap goodbye.
> 
> Claudius
>

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 5235 bytes --]

Hi Alex,

On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> 
> On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > Hi,
> > 
> > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> > > 
> > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>
> > > > :
> > > > 
> > > > > Hi,
> > > > > 
> > > > > I would like to start the discussion about how to best
> > > > > implement
> > > > > muti repository support in isar.
> > > > > 
> > > > > Does someone already has some ideas or even something in the
> > > > > pipeline for this?
> > > > > 
> > > > > If not then I do have an idea that was outlined together with
> > > > > Jan:
> > > > > 
> > > > > Adding and configuring apt repositories should be done via
> > > > > config
> > > > > files. It should be possible to define own multiconfigs while
> > > > > including multiconfigs from other layers. These configs then
> > > > > append
> > > > > filepaths to a global variable.
> > > > > 
> > > > > Every file that is added this way contains 'sources.list'
> > > > > compatible repository definitions. So one repo each line.
> > > > > 
> > > > > For every line in those files a repository entry for
> > > > > multistrap.conf
> > > > > is created. Here we might need some more complex code to
> > > > > convert
> > > > > such a apt repo tripel to the right format multistrap
> > > > > expects.
> > > > > But
> > > > > by using the 'sources.lists' format we would be independent
> > > > > of
> > > > > multistrap and become more future proof.
> > > 
> > > We will also need a way to tell apt the priorities of these
> > > repos.
> > > Multistrap just adds them and apt-get installs packages according
> > > to
> > > its default behavior.
> > > That means that a package with the same name and version will get
> > > picked from a "random" repo. Overlays will need to make sure to
> > > always
> > > have a greater version or we will need apt configuration.
> > > https://wiki.debian.org/AptPreferences
> > > The big question here is how/whether multistrap will handle apt
> > > preference.
> > > 
> > > Alex already ran into that when he wanted to modify "hello".
> > > Renaming
> > > packages - like Alex suggested - is not the way to go. Because
> > > the
> > > ones
> > > we do overlay could be deps of packages we do not overlay.
> 
> The version collision could be avoided by using epoch, and if I 
> understand it correctly - this is the preferable way to assign apt
> priority.
> 
> https://www.debian.org/doc/debian-policy/#s-f-version

Where did you get that epoch is the preferable way to assign apt
priority?

From you link:

    epoch
        This is a single (generally small) unsigned integer. It may be
omitted, in which case zero is assumed. If it is omitted then the
upstream_version may not contain any colons.
        It is provided to allow mistakes in the version numbers of
older versions of a package, and also a package’s previous version
numbering schemes, to be left behind.

I see epoch just as a possibility to fix mistakes or if upstream
changed their versioning scheme, not use it use it generally for
specifying package priorities. Using it that way sound more like a hack
to me. Because what happens if debian raises the epoch of a package?
Then you would have to raise yours again? Sounds like playing poker.

Telling apt to prefer packages with specific versions or from specific
repositories via apt-preferences sounds more like the right tool for
the job.

Cheers,
Claudius

> 
> Regarding renaming 'hello' there was a bit different case, our
> 'hello' 
> has nothing common with the upstream one, moreover I've already
> modified 
> it to use 'libhello'. So this is not the case when we patch the
> upstream 
> package. So I decided to rename it, to avoid confusion.
> 
> In case of patching the upstream package, we could just add epoch to 
> version, so our package will be always in first prio.
> 
> Also I'm not sure that when there are 2 packages with the same names
> and 
> versions available in source.list - is a good practice. In this case
> you 
> have no way to easily check if your rootfs contains correct package, 
> 'dpkg -l' won't be enough.
> 
> Alex
> 
> > Thanks, yes that is a very good point.
> > 
> > I would prefer the apt preferences settings. Maybe handle it
> > similar to
> > how I proposed the multi-repo support. A global variable where apt
> > preference file paths are appended.
> > 
> > If multistrap cannot support them with a reasonable complex
> > conversion
> > script then it might be time to say multistrap goodbye.
> > 
> > Claudius
> > 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Alexander Smirnov]

Hi!

Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:

> Hi Alex,
>
> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>
>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>> > Hi,
>> >
>> > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>> > > Am Thu, 1 Feb 2018 16:16:58 +0100
>> > > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>> > >
>> > > > Am Thu, 1 Feb 2018 15:54:26 +0100
>> > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>
>> > > > :
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I would like to start the discussion about how to best
>> > > > > implement
>> > > > > muti repository support in isar.
>> > > > >
>> > > > > Does someone already has some ideas or even something in the
>> > > > > pipeline for this?
>> > > > >
>> > > > > If not then I do have an idea that was outlined together with
>> > > > > Jan:
>> > > > >
>> > > > > Adding and configuring apt repositories should be done via
>> > > > > config
>> > > > > files. It should be possible to define own multiconfigs while
>> > > > > including multiconfigs from other layers. These configs then
>> > > > > append
>> > > > > filepaths to a global variable.
>> > > > >
>> > > > > Every file that is added this way contains 'sources.list'
>> > > > > compatible repository definitions. So one repo each line.
>> > > > >
>> > > > > For every line in those files a repository entry for
>> > > > > multistrap.conf
>> > > > > is created. Here we might need some more complex code to
>> > > > > convert
>> > > > > such a apt repo tripel to the right format multistrap
>> > > > > expects.
>> > > > > But
>> > > > > by using the 'sources.lists' format we would be independent
>> > > > > of
>> > > > > multistrap and become more future proof.
>> > >
>> > > We will also need a way to tell apt the priorities of these
>> > > repos.
>> > > Multistrap just adds them and apt-get installs packages according
>> > > to
>> > > its default behavior.
>> > > That means that a package with the same name and version will get
>> > > picked from a "random" repo. Overlays will need to make sure to
>> > > always
>> > > have a greater version or we will need apt configuration.
>> > > https://wiki.debian.org/AptPreferences
>> > > The big question here is how/whether multistrap will handle apt
>> > > preference.
>> > >
>> > > Alex already ran into that when he wanted to modify "hello".
>> > > Renaming
>> > > packages - like Alex suggested - is not the way to go. Because
>> > > the
>> > > ones
>> > > we do overlay could be deps of packages we do not overlay.
>>
>> The version collision could be avoided by using epoch, and if I
>> understand it correctly - this is the preferable way to assign apt
>> priority.
>>
>> https://www.debian.org/doc/debian-policy/#s-f-version
>
> Where did you get that epoch is the preferable way to assign apt
> priority?

Not the apt, just specific package. Hmm, I'm a bit incorrectly called my 
pov. :-) I see epoch as an easy way to patch existing package from 
upstream. You don't care about upstream one's version even if it goes 
forward, until your epoch is higher - your package will be installed. Of 
course it doesn't mean that upstream epoch couldn't be changed too, but 
this happens quite rare.

>
> From you link:
>
>     epoch
>         This is a single (generally small) unsigned integer. It may be
> omitted, in which case zero is assumed. If it is omitted then the
> upstream_version may not contain any colons.
>         It is provided to allow mistakes in the version numbers of
> older versions of a package, and also a package’s previous version
> numbering schemes, to be left behind.
>
> I see epoch just as a possibility to fix mistakes or if upstream
> changed their versioning scheme, not use it use it generally for
> specifying package priorities. Using it that way sound more like a hack
> to me. Because what happens if debian raises the epoch of a package?
> Then you would have to raise yours again? Sounds like playing poker.

Could you please describe the case when you have two repositories which 
contain package with the same name-version but different content?

Alex

>
> Telling apt to prefer packages with specific versions or from specific
> repositories via apt-preferences sounds more like the right tool for
> the job.
>
> Cheers,
> Claudius
>
>>
>> Regarding renaming 'hello' there was a bit different case, our
>> 'hello'
>> has nothing common with the upstream one, moreover I've already
>> modified
>> it to use 'libhello'. So this is not the case when we patch the
>> upstream
>> package. So I decided to rename it, to avoid confusion.
>>
>> In case of patching the upstream package, we could just add epoch to
>> version, so our package will be always in first prio.
>>
>> Also I'm not sure that when there are 2 packages with the same names
>> and
>> versions available in source.list - is a good practice. In this case
>> you
>> have no way to easily check if your rootfs contains correct package,
>> 'dpkg -l' won't be enough.
>>
>> Alex
>>
>> > Thanks, yes that is a very good point.
>> >
>> > I would prefer the apt preferences settings. Maybe handle it
>> > similar to
>> > how I proposed the multi-repo support. A global variable where apt
>> > preference file paths are appended.
>> >
>> > If multistrap cannot support them with a reasonable complex
>> > conversion
>> > script then it might be time to say multistrap goodbye.
>> >
>> > Claudius
>> >
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>
>             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
>                               Keyserver: hkp://pool.sks-keyservers.net

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 7923 bytes --]

Hi,

On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> Hi!
> 
> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
> 
> > Hi Alex,
> > 
> > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > 
> > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > Hi,
> > > > 
> > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> > > > > 
> > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.
> > > > > > com>
> > > > > > :
> > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I would like to start the discussion about how to best
> > > > > > > implement
> > > > > > > muti repository support in isar.
> > > > > > > 
> > > > > > > Does someone already has some ideas or even something in
> > > > > > > the
> > > > > > > pipeline for this?
> > > > > > > 
> > > > > > > If not then I do have an idea that was outlined together
> > > > > > > with
> > > > > > > Jan:
> > > > > > > 
> > > > > > > Adding and configuring apt repositories should be done
> > > > > > > via
> > > > > > > config
> > > > > > > files. It should be possible to define own multiconfigs
> > > > > > > while
> > > > > > > including multiconfigs from other layers. These configs
> > > > > > > then
> > > > > > > append
> > > > > > > filepaths to a global variable.
> > > > > > > 
> > > > > > > Every file that is added this way contains 'sources.list'
> > > > > > > compatible repository definitions. So one repo each line.
> > > > > > > 
> > > > > > > For every line in those files a repository entry for
> > > > > > > multistrap.conf
> > > > > > > is created. Here we might need some more complex code to
> > > > > > > convert
> > > > > > > such a apt repo tripel to the right format multistrap
> > > > > > > expects.
> > > > > > > But
> > > > > > > by using the 'sources.lists' format we would be
> > > > > > > independent
> > > > > > > of
> > > > > > > multistrap and become more future proof.
> > > > > 
> > > > > We will also need a way to tell apt the priorities of these
> > > > > repos.
> > > > > Multistrap just adds them and apt-get installs packages
> > > > > according
> > > > > to
> > > > > its default behavior.
> > > > > That means that a package with the same name and version will
> > > > > get
> > > > > picked from a "random" repo. Overlays will need to make sure
> > > > > to
> > > > > always
> > > > > have a greater version or we will need apt configuration.
> > > > > https://wiki.debian.org/AptPreferences
> > > > > The big question here is how/whether multistrap will handle
> > > > > apt
> > > > > preference.
> > > > > 
> > > > > Alex already ran into that when he wanted to modify "hello".
> > > > > Renaming
> > > > > packages - like Alex suggested - is not the way to go.
> > > > > Because
> > > > > the
> > > > > ones
> > > > > we do overlay could be deps of packages we do not overlay.
> > > 
> > > The version collision could be avoided by using epoch, and if I
> > > understand it correctly - this is the preferable way to assign
> > > apt
> > > priority.
> > > 
> > > https://www.debian.org/doc/debian-policy/#s-f-version
> > 
> > Where did you get that epoch is the preferable way to assign apt
> > priority?
> 
> Not the apt, just specific package. Hmm, I'm a bit incorrectly called
> my 
> pov. :-) I see epoch as an easy way to patch existing package from 
> upstream. You don't care about upstream one's version even if it
> goes 
> forward, until your epoch is higher - your package will be installed.
> Of 
> course it doesn't mean that upstream epoch couldn't be changed too,
> but 
> this happens quite rare.
> 
> > 
> > From you link:
> > 
> >     epoch
> >         This is a single (generally small) unsigned integer. It may
> > be
> > omitted, in which case zero is assumed. If it is omitted then the
> > upstream_version may not contain any colons.
> >         It is provided to allow mistakes in the version numbers of
> > older versions of a package, and also a package’s previous version
> > numbering schemes, to be left behind.
> > 
> > I see epoch just as a possibility to fix mistakes or if upstream
> > changed their versioning scheme, not use it use it generally for
> > specifying package priorities. Using it that way sound more like a
> > hack
> > to me. Because what happens if debian raises the epoch of a
> > package?
> > Then you would have to raise yours again? Sounds like playing
> > poker.
> 
> Could you please describe the case when you have two repositories
> which 
> contain package with the same name-version but different content?

With apt-preferences we could prefer every package from our own repo
with the same name AFAIK, but this would make the original package no
longer install-able beside that.
And that is a limitation of debian/dpkg and we can do nothing about it
in isar. This problem it not ours to solve.

Ideally debian would introduce package namespaces or similar. But that
is completely out of isars scope. On our side we could only try some
hacks to avoid name collision, but I would rather spend time solving
other problems in isar than that, since just choosing a different name
is pretty easy. Multi-repo or reproducible builds are much more
important issues than trying to play cat and mouse with package names.

Cheers,
Claudius

> 
> Alex
> 
> > 
> > Telling apt to prefer packages with specific versions or from
> > specific
> > repositories via apt-preferences sounds more like the right tool
> > for
> > the job.
> > 
> > Cheers,
> > Claudius
> > 
> > > 
> > > Regarding renaming 'hello' there was a bit different case, our
> > > 'hello'
> > > has nothing common with the upstream one, moreover I've already
> > > modified
> > > it to use 'libhello'. So this is not the case when we patch the
> > > upstream
> > > package. So I decided to rename it, to avoid confusion.
> > > 
> > > In case of patching the upstream package, we could just add epoch
> > > to
> > > version, so our package will be always in first prio.
> > > 
> > > Also I'm not sure that when there are 2 packages with the same
> > > names
> > > and
> > > versions available in source.list - is a good practice. In this
> > > case
> > > you
> > > have no way to easily check if your rootfs contains correct
> > > package,
> > > 'dpkg -l' won't be enough.
> > > 
> > > Alex
> > > 
> > > > Thanks, yes that is a very good point.
> > > > 
> > > > I would prefer the apt preferences settings. Maybe handle it
> > > > similar to
> > > > how I proposed the multi-repo support. A global variable where
> > > > apt
> > > > preference file paths are appended.
> > > > 
> > > > If multistrap cannot support them with a reasonable complex
> > > > conversion
> > > > script then it might be time to say multistrap goodbye.
> > > > 
> > > > Claudius
> > > > 
> > 
> > --
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk
> > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany
> > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.
> > de
> > 
> >             PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
> > B153
> >                               Keyserver: hkp://pool.sks-
> > keyservers.net
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Alexander Smirnov]

Hi again!

> Hi,
> 
> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>> Hi!
>>
>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
>>
>>> Hi Alex,
>>>
>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>
>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>>>>>>
>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.
>>>>>>> com>
>>>>>>> :
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I would like to start the discussion about how to best
>>>>>>>> implement
>>>>>>>> muti repository support in isar.
>>>>>>>>
>>>>>>>> Does someone already has some ideas or even something in
>>>>>>>> the
>>>>>>>> pipeline for this?
>>>>>>>>
>>>>>>>> If not then I do have an idea that was outlined together
>>>>>>>> with
>>>>>>>> Jan:
>>>>>>>>
>>>>>>>> Adding and configuring apt repositories should be done
>>>>>>>> via
>>>>>>>> config
>>>>>>>> files. It should be possible to define own multiconfigs
>>>>>>>> while
>>>>>>>> including multiconfigs from other layers. These configs
>>>>>>>> then
>>>>>>>> append
>>>>>>>> filepaths to a global variable.
>>>>>>>>
>>>>>>>> Every file that is added this way contains 'sources.list'
>>>>>>>> compatible repository definitions. So one repo each line.
>>>>>>>>
>>>>>>>> For every line in those files a repository entry for
>>>>>>>> multistrap.conf
>>>>>>>> is created. Here we might need some more complex code to
>>>>>>>> convert
>>>>>>>> such a apt repo tripel to the right format multistrap
>>>>>>>> expects.
>>>>>>>> But
>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>> independent
>>>>>>>> of
>>>>>>>> multistrap and become more future proof.
>>>>>>
>>>>>> We will also need a way to tell apt the priorities of these
>>>>>> repos.
>>>>>> Multistrap just adds them and apt-get installs packages
>>>>>> according
>>>>>> to
>>>>>> its default behavior.
>>>>>> That means that a package with the same name and version will
>>>>>> get
>>>>>> picked from a "random" repo. Overlays will need to make sure
>>>>>> to
>>>>>> always
>>>>>> have a greater version or we will need apt configuration.
>>>>>> https://wiki.debian.org/AptPreferences
>>>>>> The big question here is how/whether multistrap will handle
>>>>>> apt
>>>>>> preference.
>>>>>>
>>>>>> Alex already ran into that when he wanted to modify "hello".
>>>>>> Renaming
>>>>>> packages - like Alex suggested - is not the way to go.
>>>>>> Because
>>>>>> the
>>>>>> ones
>>>>>> we do overlay could be deps of packages we do not overlay.
>>>>
>>>> The version collision could be avoided by using epoch, and if I
>>>> understand it correctly - this is the preferable way to assign
>>>> apt
>>>> priority.
>>>>
>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>
>>> Where did you get that epoch is the preferable way to assign apt
>>> priority?
>>
>> Not the apt, just specific package. Hmm, I'm a bit incorrectly called
>> my
>> pov. :-) I see epoch as an easy way to patch existing package from
>> upstream. You don't care about upstream one's version even if it
>> goes
>> forward, until your epoch is higher - your package will be installed.
>> Of
>> course it doesn't mean that upstream epoch couldn't be changed too,
>> but
>> this happens quite rare.
>>
>>>
>>>  From you link:
>>>
>>>      epoch
>>>          This is a single (generally small) unsigned integer. It may
>>> be
>>> omitted, in which case zero is assumed. If it is omitted then the
>>> upstream_version may not contain any colons.
>>>          It is provided to allow mistakes in the version numbers of
>>> older versions of a package, and also a package’s previous version
>>> numbering schemes, to be left behind.
>>>
>>> I see epoch just as a possibility to fix mistakes or if upstream
>>> changed their versioning scheme, not use it use it generally for
>>> specifying package priorities. Using it that way sound more like a
>>> hack
>>> to me. Because what happens if debian raises the epoch of a
>>> package?
>>> Then you would have to raise yours again? Sounds like playing
>>> poker.
>>
>> Could you please describe the case when you have two repositories
>> which
>> contain package with the same name-version but different content?
> 
> With apt-preferences we could prefer every package from our own repo
> with the same name AFAIK, but this would make the original package no
> longer install-able beside that.
> And that is a limitation of debian/dpkg and we can do nothing about it
> in isar. This problem it not ours to solve.
> 
> Ideally debian would introduce package namespaces or similar. But that
> is completely out of isars scope. On our side we could only try some
> hacks to avoid name collision, but I would rather spend time solving
> other problems in isar than that, since just choosing a different name
> is pretty easy. Multi-repo or reproducible builds are much more
> important issues than trying to play cat and mouse with package names.
> 

I see this. But what is the practical usecase to have the same packages 
in several repositories?

Alex

>>> Telling apt to prefer packages with specific versions or from
>>> specific
>>> repositories via apt-preferences sounds more like the right tool
>>> for
>>> the job.
>>>
>>> Cheers,
>>> Claudius
>>>
>>>>
>>>> Regarding renaming 'hello' there was a bit different case, our
>>>> 'hello'
>>>> has nothing common with the upstream one, moreover I've already
>>>> modified
>>>> it to use 'libhello'. So this is not the case when we patch the
>>>> upstream
>>>> package. So I decided to rename it, to avoid confusion.
>>>>
>>>> In case of patching the upstream package, we could just add epoch
>>>> to
>>>> version, so our package will be always in first prio.
>>>>
>>>> Also I'm not sure that when there are 2 packages with the same
>>>> names
>>>> and
>>>> versions available in source.list - is a good practice. In this
>>>> case
>>>> you
>>>> have no way to easily check if your rootfs contains correct
>>>> package,
>>>> 'dpkg -l' won't be enough.
>>>>
>>>> Alex
>>>>
>>>>> Thanks, yes that is a very good point.
>>>>>
>>>>> I would prefer the apt preferences settings. Maybe handle it
>>>>> similar to
>>>>> how I proposed the multi-repo support. A global variable where
>>>>> apt
>>>>> preference file paths are appended.
>>>>>
>>>>> If multistrap cannot support them with a reasonable complex
>>>>> conversion
>>>>> script then it might be time to say multistrap goodbye.
>>>>>
>>>>> Claudius
>>>>>
>>>
>>> --
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang
>>> Denk
>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>> Germany
>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.
>>> de
>>>
>>>              PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808
>>> B153
>>>                                Keyserver: hkp://pool.sks-
>>> keyservers.net
>>
>>

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 10216 bytes --]

On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> Hi again!
> 
> > Hi,
> > 
> > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > Hi!
> > > 
> > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
> > > 
> > > > Hi Alex,
> > > > 
> > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > > > 
> > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
> > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > schrieb "[ext] Henning Schild" <henning.schild@siemens.co
> > > > > > > m>:
> > > > > > > 
> > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@siem
> > > > > > > > ens.
> > > > > > > > com>
> > > > > > > > :
> > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > I would like to start the discussion about how to
> > > > > > > > > best
> > > > > > > > > implement
> > > > > > > > > muti repository support in isar.
> > > > > > > > > 
> > > > > > > > > Does someone already has some ideas or even something
> > > > > > > > > in
> > > > > > > > > the
> > > > > > > > > pipeline for this?
> > > > > > > > > 
> > > > > > > > > If not then I do have an idea that was outlined
> > > > > > > > > together
> > > > > > > > > with
> > > > > > > > > Jan:
> > > > > > > > > 
> > > > > > > > > Adding and configuring apt repositories should be
> > > > > > > > > done
> > > > > > > > > via
> > > > > > > > > config
> > > > > > > > > files. It should be possible to define own
> > > > > > > > > multiconfigs
> > > > > > > > > while
> > > > > > > > > including multiconfigs from other layers. These
> > > > > > > > > configs
> > > > > > > > > then
> > > > > > > > > append
> > > > > > > > > filepaths to a global variable.
> > > > > > > > > 
> > > > > > > > > Every file that is added this way contains
> > > > > > > > > 'sources.list'
> > > > > > > > > compatible repository definitions. So one repo each
> > > > > > > > > line.
> > > > > > > > > 
> > > > > > > > > For every line in those files a repository entry for
> > > > > > > > > multistrap.conf
> > > > > > > > > is created. Here we might need some more complex code
> > > > > > > > > to
> > > > > > > > > convert
> > > > > > > > > such a apt repo tripel to the right format multistrap
> > > > > > > > > expects.
> > > > > > > > > But
> > > > > > > > > by using the 'sources.lists' format we would be
> > > > > > > > > independent
> > > > > > > > > of
> > > > > > > > > multistrap and become more future proof.
> > > > > > > 
> > > > > > > We will also need a way to tell apt the priorities of
> > > > > > > these
> > > > > > > repos.
> > > > > > > Multistrap just adds them and apt-get installs packages
> > > > > > > according
> > > > > > > to
> > > > > > > its default behavior.
> > > > > > > That means that a package with the same name and version
> > > > > > > will
> > > > > > > get
> > > > > > > picked from a "random" repo. Overlays will need to make
> > > > > > > sure
> > > > > > > to
> > > > > > > always
> > > > > > > have a greater version or we will need apt configuration.
> > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > The big question here is how/whether multistrap will
> > > > > > > handle
> > > > > > > apt
> > > > > > > preference.
> > > > > > > 
> > > > > > > Alex already ran into that when he wanted to modify
> > > > > > > "hello".
> > > > > > > Renaming
> > > > > > > packages - like Alex suggested - is not the way to go.
> > > > > > > Because
> > > > > > > the
> > > > > > > ones
> > > > > > > we do overlay could be deps of packages we do not
> > > > > > > overlay.
> > > > > 
> > > > > The version collision could be avoided by using epoch, and if
> > > > > I
> > > > > understand it correctly - this is the preferable way to
> > > > > assign
> > > > > apt
> > > > > priority.
> > > > > 
> > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > 
> > > > Where did you get that epoch is the preferable way to assign
> > > > apt
> > > > priority?
> > > 
> > > Not the apt, just specific package. Hmm, I'm a bit incorrectly
> > > called
> > > my
> > > pov. :-) I see epoch as an easy way to patch existing package
> > > from
> > > upstream. You don't care about upstream one's version even if it
> > > goes
> > > forward, until your epoch is higher - your package will be
> > > installed.
> > > Of
> > > course it doesn't mean that upstream epoch couldn't be changed
> > > too,
> > > but
> > > this happens quite rare.
> > > 
> > > > 
> > > >  From you link:
> > > > 
> > > >      epoch
> > > >          This is a single (generally small) unsigned integer.
> > > > It may
> > > > be
> > > > omitted, in which case zero is assumed. If it is omitted then
> > > > the
> > > > upstream_version may not contain any colons.
> > > >          It is provided to allow mistakes in the version
> > > > numbers of
> > > > older versions of a package, and also a package’s previous
> > > > version
> > > > numbering schemes, to be left behind.
> > > > 
> > > > I see epoch just as a possibility to fix mistakes or if
> > > > upstream
> > > > changed their versioning scheme, not use it use it generally
> > > > for
> > > > specifying package priorities. Using it that way sound more
> > > > like a
> > > > hack
> > > > to me. Because what happens if debian raises the epoch of a
> > > > package?
> > > > Then you would have to raise yours again? Sounds like playing
> > > > poker.
> > > 
> > > Could you please describe the case when you have two repositories
> > > which
> > > contain package with the same name-version but different content?
> > 
> > With apt-preferences we could prefer every package from our own
> > repo
> > with the same name AFAIK, but this would make the original package
> > no
> > longer install-able beside that.
> > And that is a limitation of debian/dpkg and we can do nothing about
> > it
> > in isar. This problem it not ours to solve.
> > 
> > Ideally debian would introduce package namespaces or similar. But
> > that
> > is completely out of isars scope. On our side we could only try
> > some
> > hacks to avoid name collision, but I would rather spend time
> > solving
> > other problems in isar than that, since just choosing a different
> > name
> > is pretty easy. Multi-repo or reproducible builds are much more
> > important issues than trying to play cat and mouse with package
> > names.
> > 
> 
> I see this. But what is the practical usecase to have the same
> packages 
> in several repositories?

Where did this question comes from? It does not really make sense to
have exactly the same package in multiple different repositories. But
that is AFAIK not related to any of this.

If you have changed some existing debian package put it into your own
repo, maybe change the version of the package to contain something like
"*~isar0" similar to how ubuntu does it. Then set a apt-preference like
this:

    Package: *
    Pin: release c=isar
    Pin-Priority: 650

or similar. And apt would prefer to use packages from the isar repo
instead of upstream if there is a name collision. Even if the version
of the package in isar is lower than the one upstream.

Cheers,
Claudius


> 
> Alex
> 
> > > > Telling apt to prefer packages with specific versions or from
> > > > specific
> > > > repositories via apt-preferences sounds more like the right
> > > > tool
> > > > for
> > > > the job.
> > > > 
> > > > Cheers,
> > > > Claudius
> > > > 
> > > > > 
> > > > > Regarding renaming 'hello' there was a bit different case,
> > > > > our
> > > > > 'hello'
> > > > > has nothing common with the upstream one, moreover I've
> > > > > already
> > > > > modified
> > > > > it to use 'libhello'. So this is not the case when we patch
> > > > > the
> > > > > upstream
> > > > > package. So I decided to rename it, to avoid confusion.
> > > > > 
> > > > > In case of patching the upstream package, we could just add
> > > > > epoch
> > > > > to
> > > > > version, so our package will be always in first prio.
> > > > > 
> > > > > Also I'm not sure that when there are 2 packages with the
> > > > > same
> > > > > names
> > > > > and
> > > > > versions available in source.list - is a good practice. In
> > > > > this
> > > > > case
> > > > > you
> > > > > have no way to easily check if your rootfs contains correct
> > > > > package,
> > > > > 'dpkg -l' won't be enough.
> > > > > 
> > > > > Alex
> > > > > 
> > > > > > Thanks, yes that is a very good point.
> > > > > > 
> > > > > > I would prefer the apt preferences settings. Maybe handle
> > > > > > it
> > > > > > similar to
> > > > > > how I proposed the multi-repo support. A global variable
> > > > > > where
> > > > > > apt
> > > > > > preference file paths are appended.
> > > > > > 
> > > > > > If multistrap cannot support them with a reasonable complex
> > > > > > conversion
> > > > > > script then it might be time to say multistrap goodbye.
> > > > > > 
> > > > > > Claudius
> > > > > > 
> > > > 
> > > > --
> > > > DENX Software Engineering GmbH,      Managing Director:
> > > > Wolfgang
> > > > Denk
> > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > > > Germany
> > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@d
> > > > enx.
> > > > de
> > > > 
> > > >              PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19
> > > > 9808
> > > > B153
> > > >                                Keyserver: hkp://pool.sks-
> > > > keyservers.net
> > > 
> > > 
> 
> 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Alexander Smirnov]

Hi,

On 02/02/2018 04:26 PM, Claudius Heine wrote:
> On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
>> Hi again!
>>
>>> Hi,
>>>
>>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>>>> Hi!
>>>>
>>>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49 написал:
>>>>
>>>>> Hi Alex,
>>>>>
>>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>>>
>>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild wrote:
>>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemens.co
>>>>>>>> m>:
>>>>>>>>
>>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siem
>>>>>>>>> ens.
>>>>>>>>> com>
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I would like to start the discussion about how to
>>>>>>>>>> best
>>>>>>>>>> implement
>>>>>>>>>> muti repository support in isar.
>>>>>>>>>>
>>>>>>>>>> Does someone already has some ideas or even something
>>>>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>> pipeline for this?
>>>>>>>>>>
>>>>>>>>>> If not then I do have an idea that was outlined
>>>>>>>>>> together
>>>>>>>>>> with
>>>>>>>>>> Jan:
>>>>>>>>>>
>>>>>>>>>> Adding and configuring apt repositories should be
>>>>>>>>>> done
>>>>>>>>>> via
>>>>>>>>>> config
>>>>>>>>>> files. It should be possible to define own
>>>>>>>>>> multiconfigs
>>>>>>>>>> while
>>>>>>>>>> including multiconfigs from other layers. These
>>>>>>>>>> configs
>>>>>>>>>> then
>>>>>>>>>> append
>>>>>>>>>> filepaths to a global variable.
>>>>>>>>>>
>>>>>>>>>> Every file that is added this way contains
>>>>>>>>>> 'sources.list'
>>>>>>>>>> compatible repository definitions. So one repo each
>>>>>>>>>> line.
>>>>>>>>>>
>>>>>>>>>> For every line in those files a repository entry for
>>>>>>>>>> multistrap.conf
>>>>>>>>>> is created. Here we might need some more complex code
>>>>>>>>>> to
>>>>>>>>>> convert
>>>>>>>>>> such a apt repo tripel to the right format multistrap
>>>>>>>>>> expects.
>>>>>>>>>> But
>>>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>>>> independent
>>>>>>>>>> of
>>>>>>>>>> multistrap and become more future proof.
>>>>>>>>
>>>>>>>> We will also need a way to tell apt the priorities of
>>>>>>>> these
>>>>>>>> repos.
>>>>>>>> Multistrap just adds them and apt-get installs packages
>>>>>>>> according
>>>>>>>> to
>>>>>>>> its default behavior.
>>>>>>>> That means that a package with the same name and version
>>>>>>>> will
>>>>>>>> get
>>>>>>>> picked from a "random" repo. Overlays will need to make
>>>>>>>> sure
>>>>>>>> to
>>>>>>>> always
>>>>>>>> have a greater version or we will need apt configuration.
>>>>>>>> https://wiki.debian.org/AptPreferences
>>>>>>>> The big question here is how/whether multistrap will
>>>>>>>> handle
>>>>>>>> apt
>>>>>>>> preference.
>>>>>>>>
>>>>>>>> Alex already ran into that when he wanted to modify
>>>>>>>> "hello".
>>>>>>>> Renaming
>>>>>>>> packages - like Alex suggested - is not the way to go.
>>>>>>>> Because
>>>>>>>> the
>>>>>>>> ones
>>>>>>>> we do overlay could be deps of packages we do not
>>>>>>>> overlay.
>>>>>>
>>>>>> The version collision could be avoided by using epoch, and if
>>>>>> I
>>>>>> understand it correctly - this is the preferable way to
>>>>>> assign
>>>>>> apt
>>>>>> priority.
>>>>>>
>>>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>>>
>>>>> Where did you get that epoch is the preferable way to assign
>>>>> apt
>>>>> priority?
>>>>
>>>> Not the apt, just specific package. Hmm, I'm a bit incorrectly
>>>> called
>>>> my
>>>> pov. :-) I see epoch as an easy way to patch existing package
>>>> from
>>>> upstream. You don't care about upstream one's version even if it
>>>> goes
>>>> forward, until your epoch is higher - your package will be
>>>> installed.
>>>> Of
>>>> course it doesn't mean that upstream epoch couldn't be changed
>>>> too,
>>>> but
>>>> this happens quite rare.
>>>>
>>>>>
>>>>>   From you link:
>>>>>
>>>>>       epoch
>>>>>           This is a single (generally small) unsigned integer.
>>>>> It may
>>>>> be
>>>>> omitted, in which case zero is assumed. If it is omitted then
>>>>> the
>>>>> upstream_version may not contain any colons.
>>>>>           It is provided to allow mistakes in the version
>>>>> numbers of
>>>>> older versions of a package, and also a package’s previous
>>>>> version
>>>>> numbering schemes, to be left behind.
>>>>>
>>>>> I see epoch just as a possibility to fix mistakes or if
>>>>> upstream
>>>>> changed their versioning scheme, not use it use it generally
>>>>> for
>>>>> specifying package priorities. Using it that way sound more
>>>>> like a
>>>>> hack
>>>>> to me. Because what happens if debian raises the epoch of a
>>>>> package?
>>>>> Then you would have to raise yours again? Sounds like playing
>>>>> poker.
>>>>
>>>> Could you please describe the case when you have two repositories
>>>> which
>>>> contain package with the same name-version but different content?
>>>
>>> With apt-preferences we could prefer every package from our own
>>> repo
>>> with the same name AFAIK, but this would make the original package
>>> no
>>> longer install-able beside that.
>>> And that is a limitation of debian/dpkg and we can do nothing about
>>> it
>>> in isar. This problem it not ours to solve.
>>>
>>> Ideally debian would introduce package namespaces or similar. But
>>> that
>>> is completely out of isars scope. On our side we could only try
>>> some
>>> hacks to avoid name collision, but I would rather spend time
>>> solving
>>> other problems in isar than that, since just choosing a different
>>> name
>>> is pretty easy. Multi-repo or reproducible builds are much more
>>> important issues than trying to play cat and mouse with package
>>> names.
>>>
>>
>> I see this. But what is the practical usecase to have the same
>> packages
>> in several repositories?
> 
> Where did this question comes from? It does not really make sense to

 From the text above, but I found that it wasn't your message, sorry :-)

...
That means that a package with the same name and version will get
picked from a "random" repo.
...

> have exactly the same package in multiple different repositories. But
> that is AFAIK not related to any of this.
> 

So if it's not the case, then it's Ok for me.

> If you have changed some existing debian package put it into your own
> repo, maybe change the version of the package to contain something like
> "*~isar0" similar to how ubuntu does it. Then set a apt-preference like
> this:
> 
>      Package: *
>      Pin: release c=isar
>      Pin-Priority: 650
> 
> or similar. And apt would prefer to use packages from the isar repo
> instead of upstream if there is a name collision. Even if the version
> of the package in isar is lower than the one upstream.

Do you think it's possible to handle the following scenario:

1. You pick current glibc from upstream distro XYZ, let's say v1.0.
2. Patch it and put glibc-v1.0-isar to local repo.
3. Wait for some time until glibc is updated in upstream, let's say to v1.1.
4. Due to glibc-v1.0-isar is set to be preferred, it's going to be 
installed. But glibc is the core library in distro, and lots of packages 
in XYZ now require glibc-v1.1. So you won't be able to generate image 
anymore, it will fail with unmet dependency.

Alex

> 
> Cheers,
> Claudius
> 
> 
>>
>> Alex
>>
>>>>> Telling apt to prefer packages with specific versions or from
>>>>> specific
>>>>> repositories via apt-preferences sounds more like the right
>>>>> tool
>>>>> for
>>>>> the job.
>>>>>
>>>>> Cheers,
>>>>> Claudius
>>>>>
>>>>>>
>>>>>> Regarding renaming 'hello' there was a bit different case,
>>>>>> our
>>>>>> 'hello'
>>>>>> has nothing common with the upstream one, moreover I've
>>>>>> already
>>>>>> modified
>>>>>> it to use 'libhello'. So this is not the case when we patch
>>>>>> the
>>>>>> upstream
>>>>>> package. So I decided to rename it, to avoid confusion.
>>>>>>
>>>>>> In case of patching the upstream package, we could just add
>>>>>> epoch
>>>>>> to
>>>>>> version, so our package will be always in first prio.
>>>>>>
>>>>>> Also I'm not sure that when there are 2 packages with the
>>>>>> same
>>>>>> names
>>>>>> and
>>>>>> versions available in source.list - is a good practice. In
>>>>>> this
>>>>>> case
>>>>>> you
>>>>>> have no way to easily check if your rootfs contains correct
>>>>>> package,
>>>>>> 'dpkg -l' won't be enough.
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>> Thanks, yes that is a very good point.
>>>>>>>
>>>>>>> I would prefer the apt preferences settings. Maybe handle
>>>>>>> it
>>>>>>> similar to
>>>>>>> how I proposed the multi-repo support. A global variable
>>>>>>> where
>>>>>>> apt
>>>>>>> preference file paths are appended.
>>>>>>>
>>>>>>> If multistrap cannot support them with a reasonable complex
>>>>>>> conversion
>>>>>>> script then it might be time to say multistrap goodbye.
>>>>>>>
>>>>>>> Claudius
>>>>>>>
>>>>>
>>>>> --
>>>>> DENX Software Engineering GmbH,      Managing Director:
>>>>> Wolfgang
>>>>> Denk
>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
>>>>> Germany
>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@d
>>>>> enx.
>>>>> de
>>>>>
>>>>>               PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19
>>>>> 9808
>>>>> B153
>>>>>                                 Keyserver: hkp://pool.sks-
>>>>> keyservers.net
>>>>
>>>>
>>
>>

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 13294 bytes --]

On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
> Hi,
> 
> On 02/02/2018 04:26 PM, Claudius Heine wrote:
> > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> > > Hi again!
> > > 
> > > > Hi,
> > > > 
> > > > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > > > Hi!
> > > > > 
> > > > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
> > > > > написал:
> > > > > 
> > > > > > Hi Alex,
> > > > > > 
> > > > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
> > > > > > > 
> > > > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
> > > > > > > > wrote:
> > > > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > > > schrieb "[ext] Henning Schild" <henning.schild@siemen
> > > > > > > > > s.co
> > > > > > > > > m>:
> > > > > > > > > 
> > > > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > > > schrieb "[ext] Claudius Heine" <claudius.heine.ext@
> > > > > > > > > > siem
> > > > > > > > > > ens.
> > > > > > > > > > com>
> > > > > > > > > > :
> > > > > > > > > > 
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > I would like to start the discussion about how to
> > > > > > > > > > > best
> > > > > > > > > > > implement
> > > > > > > > > > > muti repository support in isar.
> > > > > > > > > > > 
> > > > > > > > > > > Does someone already has some ideas or even
> > > > > > > > > > > something
> > > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > pipeline for this?
> > > > > > > > > > > 
> > > > > > > > > > > If not then I do have an idea that was outlined
> > > > > > > > > > > together
> > > > > > > > > > > with
> > > > > > > > > > > Jan:
> > > > > > > > > > > 
> > > > > > > > > > > Adding and configuring apt repositories should be
> > > > > > > > > > > done
> > > > > > > > > > > via
> > > > > > > > > > > config
> > > > > > > > > > > files. It should be possible to define own
> > > > > > > > > > > multiconfigs
> > > > > > > > > > > while
> > > > > > > > > > > including multiconfigs from other layers. These
> > > > > > > > > > > configs
> > > > > > > > > > > then
> > > > > > > > > > > append
> > > > > > > > > > > filepaths to a global variable.
> > > > > > > > > > > 
> > > > > > > > > > > Every file that is added this way contains
> > > > > > > > > > > 'sources.list'
> > > > > > > > > > > compatible repository definitions. So one repo
> > > > > > > > > > > each
> > > > > > > > > > > line.
> > > > > > > > > > > 
> > > > > > > > > > > For every line in those files a repository entry
> > > > > > > > > > > for
> > > > > > > > > > > multistrap.conf
> > > > > > > > > > > is created. Here we might need some more complex
> > > > > > > > > > > code
> > > > > > > > > > > to
> > > > > > > > > > > convert
> > > > > > > > > > > such a apt repo tripel to the right format
> > > > > > > > > > > multistrap
> > > > > > > > > > > expects.
> > > > > > > > > > > But
> > > > > > > > > > > by using the 'sources.lists' format we would be
> > > > > > > > > > > independent
> > > > > > > > > > > of
> > > > > > > > > > > multistrap and become more future proof.
> > > > > > > > > 
> > > > > > > > > We will also need a way to tell apt the priorities of
> > > > > > > > > these
> > > > > > > > > repos.
> > > > > > > > > Multistrap just adds them and apt-get installs
> > > > > > > > > packages
> > > > > > > > > according
> > > > > > > > > to
> > > > > > > > > its default behavior.
> > > > > > > > > That means that a package with the same name and
> > > > > > > > > version
> > > > > > > > > will
> > > > > > > > > get
> > > > > > > > > picked from a "random" repo. Overlays will need to
> > > > > > > > > make
> > > > > > > > > sure
> > > > > > > > > to
> > > > > > > > > always
> > > > > > > > > have a greater version or we will need apt
> > > > > > > > > configuration.
> > > > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > > > The big question here is how/whether multistrap will
> > > > > > > > > handle
> > > > > > > > > apt
> > > > > > > > > preference.
> > > > > > > > > 
> > > > > > > > > Alex already ran into that when he wanted to modify
> > > > > > > > > "hello".
> > > > > > > > > Renaming
> > > > > > > > > packages - like Alex suggested - is not the way to
> > > > > > > > > go.
> > > > > > > > > Because
> > > > > > > > > the
> > > > > > > > > ones
> > > > > > > > > we do overlay could be deps of packages we do not
> > > > > > > > > overlay.
> > > > > > > 
> > > > > > > The version collision could be avoided by using epoch,
> > > > > > > and if
> > > > > > > I
> > > > > > > understand it correctly - this is the preferable way to
> > > > > > > assign
> > > > > > > apt
> > > > > > > priority.
> > > > > > > 
> > > > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > > > 
> > > > > > Where did you get that epoch is the preferable way to
> > > > > > assign
> > > > > > apt
> > > > > > priority?
> > > > > 
> > > > > Not the apt, just specific package. Hmm, I'm a bit
> > > > > incorrectly
> > > > > called
> > > > > my
> > > > > pov. :-) I see epoch as an easy way to patch existing package
> > > > > from
> > > > > upstream. You don't care about upstream one's version even if
> > > > > it
> > > > > goes
> > > > > forward, until your epoch is higher - your package will be
> > > > > installed.
> > > > > Of
> > > > > course it doesn't mean that upstream epoch couldn't be
> > > > > changed
> > > > > too,
> > > > > but
> > > > > this happens quite rare.
> > > > > 
> > > > > > 
> > > > > >   From you link:
> > > > > > 
> > > > > >       epoch
> > > > > >           This is a single (generally small) unsigned
> > > > > > integer.
> > > > > > It may
> > > > > > be
> > > > > > omitted, in which case zero is assumed. If it is omitted
> > > > > > then
> > > > > > the
> > > > > > upstream_version may not contain any colons.
> > > > > >           It is provided to allow mistakes in the version
> > > > > > numbers of
> > > > > > older versions of a package, and also a package’s previous
> > > > > > version
> > > > > > numbering schemes, to be left behind.
> > > > > > 
> > > > > > I see epoch just as a possibility to fix mistakes or if
> > > > > > upstream
> > > > > > changed their versioning scheme, not use it use it
> > > > > > generally
> > > > > > for
> > > > > > specifying package priorities. Using it that way sound more
> > > > > > like a
> > > > > > hack
> > > > > > to me. Because what happens if debian raises the epoch of a
> > > > > > package?
> > > > > > Then you would have to raise yours again? Sounds like
> > > > > > playing
> > > > > > poker.
> > > > > 
> > > > > Could you please describe the case when you have two
> > > > > repositories
> > > > > which
> > > > > contain package with the same name-version but different
> > > > > content?
> > > > 
> > > > With apt-preferences we could prefer every package from our own
> > > > repo
> > > > with the same name AFAIK, but this would make the original
> > > > package
> > > > no
> > > > longer install-able beside that.
> > > > And that is a limitation of debian/dpkg and we can do nothing
> > > > about
> > > > it
> > > > in isar. This problem it not ours to solve.
> > > > 
> > > > Ideally debian would introduce package namespaces or similar.
> > > > But
> > > > that
> > > > is completely out of isars scope. On our side we could only try
> > > > some
> > > > hacks to avoid name collision, but I would rather spend time
> > > > solving
> > > > other problems in isar than that, since just choosing a
> > > > different
> > > > name
> > > > is pretty easy. Multi-repo or reproducible builds are much more
> > > > important issues than trying to play cat and mouse with package
> > > > names.
> > > > 
> > > 
> > > I see this. But what is the practical usecase to have the same
> > > packages
> > > in several repositories?
> > 
> > Where did this question comes from? It does not really make sense
> > to
> 
>  From the text above, but I found that it wasn't your message, sorry
> :-)
> 
> ...
> That means that a package with the same name and version will get
> picked from a "random" repo.
> ...
> 
> > have exactly the same package in multiple different repositories.
> > But
> > that is AFAIK not related to any of this.
> > 
> 
> So if it's not the case, then it's Ok for me.
> 
> > If you have changed some existing debian package put it into your
> > own
> > repo, maybe change the version of the package to contain something
> > like
> > "*~isar0" similar to how ubuntu does it. Then set a apt-preference
> > like
> > this:
> > 
> >      Package: *
> >      Pin: release c=isar
> >      Pin-Priority: 650
> > 
> > or similar. And apt would prefer to use packages from the isar repo
> > instead of upstream if there is a name collision. Even if the
> > version
> > of the package in isar is lower than the one upstream.
> 
> Do you think it's possible to handle the following scenario:
> 
> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
> 2. Patch it and put glibc-v1.0-isar to local repo.
> 3. Wait for some time until glibc is updated in upstream, let's say
> to v1.1.
> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be 
> installed. But glibc is the core library in distro, and lots of
> packages 
> in XYZ now require glibc-v1.1. So you won't be able to generate
> image 
> anymore, it will fail with unmet dependency.

Why do we need to handle this?

It would be great if image generation fails in this case and not cause
the generation of a broken image.

Otherwise that is the reason we should have reproducible builds, so
that we can still create the old image.

Cheers,
Claudius

> 
> Alex
> 
> > 
> > Cheers,
> > Claudius
> > 
> > 
> > > 
> > > Alex
> > > 
> > > > > > Telling apt to prefer packages with specific versions or
> > > > > > from
> > > > > > specific
> > > > > > repositories via apt-preferences sounds more like the right
> > > > > > tool
> > > > > > for
> > > > > > the job.
> > > > > > 
> > > > > > Cheers,
> > > > > > Claudius
> > > > > > 
> > > > > > > 
> > > > > > > Regarding renaming 'hello' there was a bit different
> > > > > > > case,
> > > > > > > our
> > > > > > > 'hello'
> > > > > > > has nothing common with the upstream one, moreover I've
> > > > > > > already
> > > > > > > modified
> > > > > > > it to use 'libhello'. So this is not the case when we
> > > > > > > patch
> > > > > > > the
> > > > > > > upstream
> > > > > > > package. So I decided to rename it, to avoid confusion.
> > > > > > > 
> > > > > > > In case of patching the upstream package, we could just
> > > > > > > add
> > > > > > > epoch
> > > > > > > to
> > > > > > > version, so our package will be always in first prio.
> > > > > > > 
> > > > > > > Also I'm not sure that when there are 2 packages with the
> > > > > > > same
> > > > > > > names
> > > > > > > and
> > > > > > > versions available in source.list - is a good practice.
> > > > > > > In
> > > > > > > this
> > > > > > > case
> > > > > > > you
> > > > > > > have no way to easily check if your rootfs contains
> > > > > > > correct
> > > > > > > package,
> > > > > > > 'dpkg -l' won't be enough.
> > > > > > > 
> > > > > > > Alex
> > > > > > > 
> > > > > > > > Thanks, yes that is a very good point.
> > > > > > > > 
> > > > > > > > I would prefer the apt preferences settings. Maybe
> > > > > > > > handle
> > > > > > > > it
> > > > > > > > similar to
> > > > > > > > how I proposed the multi-repo support. A global
> > > > > > > > variable
> > > > > > > > where
> > > > > > > > apt
> > > > > > > > preference file paths are appended.
> > > > > > > > 
> > > > > > > > If multistrap cannot support them with a reasonable
> > > > > > > > complex
> > > > > > > > conversion
> > > > > > > > script then it might be time to say multistrap goodbye.
> > > > > > > > 
> > > > > > > > Claudius
> > > > > > > > 
> > > > > > 
> > > > > > --
> > > > > > DENX Software Engineering GmbH,      Managing Director:
> > > > > > Wolfgang
> > > > > > Denk
> > > > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> > > > > > Groebenzell,
> > > > > > Germany
> > > > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: 
> > > > > > ch@d
> > > > > > enx.
> > > > > > de
> > > > > > 
> > > > > >               PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173
> > > > > > CB19
> > > > > > 9808
> > > > > > B153
> > > > > >                                 Keyserver: hkp://pool.sks-
> > > > > > keyservers.net
> > > > > 
> > > > > 
> > > 
> > > 
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Alexander Smirnov]

On 02/02/2018 05:15 PM, Claudius Heine wrote:
> On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
>> Hi,
>>
>> On 02/02/2018 04:26 PM, Claudius Heine wrote:
>>> On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
>>>> Hi again!
>>>>
>>>>> Hi,
>>>>>
>>>>> On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
>>>>>> Hi!
>>>>>>
>>>>>> Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
>>>>>> написал:
>>>>>>
>>>>>>> Hi Alex,
>>>>>>>
>>>>>>> On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov wrote:
>>>>>>>>
>>>>>>>> On 02/01/2018 09:51 PM, Claudius Heine wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
>>>>>>>>> wrote:
>>>>>>>>>> Am Thu, 1 Feb 2018 16:16:58 +0100
>>>>>>>>>> schrieb "[ext] Henning Schild" <henning.schild@siemen
>>>>>>>>>> s.co
>>>>>>>>>> m>:
>>>>>>>>>>
>>>>>>>>>>> Am Thu, 1 Feb 2018 15:54:26 +0100
>>>>>>>>>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@
>>>>>>>>>>> siem
>>>>>>>>>>> ens.
>>>>>>>>>>> com>
>>>>>>>>>>> :
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to start the discussion about how to
>>>>>>>>>>>> best
>>>>>>>>>>>> implement
>>>>>>>>>>>> muti repository support in isar.
>>>>>>>>>>>>
>>>>>>>>>>>> Does someone already has some ideas or even
>>>>>>>>>>>> something
>>>>>>>>>>>> in
>>>>>>>>>>>> the
>>>>>>>>>>>> pipeline for this?
>>>>>>>>>>>>
>>>>>>>>>>>> If not then I do have an idea that was outlined
>>>>>>>>>>>> together
>>>>>>>>>>>> with
>>>>>>>>>>>> Jan:
>>>>>>>>>>>>
>>>>>>>>>>>> Adding and configuring apt repositories should be
>>>>>>>>>>>> done
>>>>>>>>>>>> via
>>>>>>>>>>>> config
>>>>>>>>>>>> files. It should be possible to define own
>>>>>>>>>>>> multiconfigs
>>>>>>>>>>>> while
>>>>>>>>>>>> including multiconfigs from other layers. These
>>>>>>>>>>>> configs
>>>>>>>>>>>> then
>>>>>>>>>>>> append
>>>>>>>>>>>> filepaths to a global variable.
>>>>>>>>>>>>
>>>>>>>>>>>> Every file that is added this way contains
>>>>>>>>>>>> 'sources.list'
>>>>>>>>>>>> compatible repository definitions. So one repo
>>>>>>>>>>>> each
>>>>>>>>>>>> line.
>>>>>>>>>>>>
>>>>>>>>>>>> For every line in those files a repository entry
>>>>>>>>>>>> for
>>>>>>>>>>>> multistrap.conf
>>>>>>>>>>>> is created. Here we might need some more complex
>>>>>>>>>>>> code
>>>>>>>>>>>> to
>>>>>>>>>>>> convert
>>>>>>>>>>>> such a apt repo tripel to the right format
>>>>>>>>>>>> multistrap
>>>>>>>>>>>> expects.
>>>>>>>>>>>> But
>>>>>>>>>>>> by using the 'sources.lists' format we would be
>>>>>>>>>>>> independent
>>>>>>>>>>>> of
>>>>>>>>>>>> multistrap and become more future proof.
>>>>>>>>>>
>>>>>>>>>> We will also need a way to tell apt the priorities of
>>>>>>>>>> these
>>>>>>>>>> repos.
>>>>>>>>>> Multistrap just adds them and apt-get installs
>>>>>>>>>> packages
>>>>>>>>>> according
>>>>>>>>>> to
>>>>>>>>>> its default behavior.
>>>>>>>>>> That means that a package with the same name and
>>>>>>>>>> version
>>>>>>>>>> will
>>>>>>>>>> get
>>>>>>>>>> picked from a "random" repo. Overlays will need to
>>>>>>>>>> make
>>>>>>>>>> sure
>>>>>>>>>> to
>>>>>>>>>> always
>>>>>>>>>> have a greater version or we will need apt
>>>>>>>>>> configuration.
>>>>>>>>>> https://wiki.debian.org/AptPreferences
>>>>>>>>>> The big question here is how/whether multistrap will
>>>>>>>>>> handle
>>>>>>>>>> apt
>>>>>>>>>> preference.
>>>>>>>>>>
>>>>>>>>>> Alex already ran into that when he wanted to modify
>>>>>>>>>> "hello".
>>>>>>>>>> Renaming
>>>>>>>>>> packages - like Alex suggested - is not the way to
>>>>>>>>>> go.
>>>>>>>>>> Because
>>>>>>>>>> the
>>>>>>>>>> ones
>>>>>>>>>> we do overlay could be deps of packages we do not
>>>>>>>>>> overlay.
>>>>>>>>
>>>>>>>> The version collision could be avoided by using epoch,
>>>>>>>> and if
>>>>>>>> I
>>>>>>>> understand it correctly - this is the preferable way to
>>>>>>>> assign
>>>>>>>> apt
>>>>>>>> priority.
>>>>>>>>
>>>>>>>> https://www.debian.org/doc/debian-policy/#s-f-version
>>>>>>>
>>>>>>> Where did you get that epoch is the preferable way to
>>>>>>> assign
>>>>>>> apt
>>>>>>> priority?
>>>>>>
>>>>>> Not the apt, just specific package. Hmm, I'm a bit
>>>>>> incorrectly
>>>>>> called
>>>>>> my
>>>>>> pov. :-) I see epoch as an easy way to patch existing package
>>>>>> from
>>>>>> upstream. You don't care about upstream one's version even if
>>>>>> it
>>>>>> goes
>>>>>> forward, until your epoch is higher - your package will be
>>>>>> installed.
>>>>>> Of
>>>>>> course it doesn't mean that upstream epoch couldn't be
>>>>>> changed
>>>>>> too,
>>>>>> but
>>>>>> this happens quite rare.
>>>>>>
>>>>>>>
>>>>>>>    From you link:
>>>>>>>
>>>>>>>        epoch
>>>>>>>            This is a single (generally small) unsigned
>>>>>>> integer.
>>>>>>> It may
>>>>>>> be
>>>>>>> omitted, in which case zero is assumed. If it is omitted
>>>>>>> then
>>>>>>> the
>>>>>>> upstream_version may not contain any colons.
>>>>>>>            It is provided to allow mistakes in the version
>>>>>>> numbers of
>>>>>>> older versions of a package, and also a package’s previous
>>>>>>> version
>>>>>>> numbering schemes, to be left behind.
>>>>>>>
>>>>>>> I see epoch just as a possibility to fix mistakes or if
>>>>>>> upstream
>>>>>>> changed their versioning scheme, not use it use it
>>>>>>> generally
>>>>>>> for
>>>>>>> specifying package priorities. Using it that way sound more
>>>>>>> like a
>>>>>>> hack
>>>>>>> to me. Because what happens if debian raises the epoch of a
>>>>>>> package?
>>>>>>> Then you would have to raise yours again? Sounds like
>>>>>>> playing
>>>>>>> poker.
>>>>>>
>>>>>> Could you please describe the case when you have two
>>>>>> repositories
>>>>>> which
>>>>>> contain package with the same name-version but different
>>>>>> content?
>>>>>
>>>>> With apt-preferences we could prefer every package from our own
>>>>> repo
>>>>> with the same name AFAIK, but this would make the original
>>>>> package
>>>>> no
>>>>> longer install-able beside that.
>>>>> And that is a limitation of debian/dpkg and we can do nothing
>>>>> about
>>>>> it
>>>>> in isar. This problem it not ours to solve.
>>>>>
>>>>> Ideally debian would introduce package namespaces or similar.
>>>>> But
>>>>> that
>>>>> is completely out of isars scope. On our side we could only try
>>>>> some
>>>>> hacks to avoid name collision, but I would rather spend time
>>>>> solving
>>>>> other problems in isar than that, since just choosing a
>>>>> different
>>>>> name
>>>>> is pretty easy. Multi-repo or reproducible builds are much more
>>>>> important issues than trying to play cat and mouse with package
>>>>> names.
>>>>>
>>>>
>>>> I see this. But what is the practical usecase to have the same
>>>> packages
>>>> in several repositories?
>>>
>>> Where did this question comes from? It does not really make sense
>>> to
>>
>>   From the text above, but I found that it wasn't your message, sorry
>> :-)
>>
>> ...
>> That means that a package with the same name and version will get
>> picked from a "random" repo.
>> ...
>>
>>> have exactly the same package in multiple different repositories.
>>> But
>>> that is AFAIK not related to any of this.
>>>
>>
>> So if it's not the case, then it's Ok for me.
>>
>>> If you have changed some existing debian package put it into your
>>> own
>>> repo, maybe change the version of the package to contain something
>>> like
>>> "*~isar0" similar to how ubuntu does it. Then set a apt-preference
>>> like
>>> this:
>>>
>>>       Package: *
>>>       Pin: release c=isar
>>>       Pin-Priority: 650
>>>
>>> or similar. And apt would prefer to use packages from the isar repo
>>> instead of upstream if there is a name collision. Even if the
>>> version
>>> of the package in isar is lower than the one upstream.
>>
>> Do you think it's possible to handle the following scenario:
>>
>> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
>> 2. Patch it and put glibc-v1.0-isar to local repo.
>> 3. Wait for some time until glibc is updated in upstream, let's say
>> to v1.1.
>> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be
>> installed. But glibc is the core library in distro, and lots of
>> packages
>> in XYZ now require glibc-v1.1. So you won't be able to generate
>> image
>> anymore, it will fail with unmet dependency.
> 
> Why do we need to handle this?

If you use upstream sources and prioritize apts, this problem likely 
will occur. So repo prioritizing is then unpredictable feature.

> 
> It would be great if image generation fails in this case and not cause
> the generation of a broken image.
> 
> Otherwise that is the reason we should have reproducible builds, so
> that we can still create the old image.

If you don't work with upstream and use only local repositories, why 
can't you just drop duplicated packages from them?

Alex

>>>>>>> Telling apt to prefer packages with specific versions or
>>>>>>> from
>>>>>>> specific
>>>>>>> repositories via apt-preferences sounds more like the right
>>>>>>> tool
>>>>>>> for
>>>>>>> the job.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Claudius
>>>>>>>
>>>>>>>>
>>>>>>>> Regarding renaming 'hello' there was a bit different
>>>>>>>> case,
>>>>>>>> our
>>>>>>>> 'hello'
>>>>>>>> has nothing common with the upstream one, moreover I've
>>>>>>>> already
>>>>>>>> modified
>>>>>>>> it to use 'libhello'. So this is not the case when we
>>>>>>>> patch
>>>>>>>> the
>>>>>>>> upstream
>>>>>>>> package. So I decided to rename it, to avoid confusion.
>>>>>>>>
>>>>>>>> In case of patching the upstream package, we could just
>>>>>>>> add
>>>>>>>> epoch
>>>>>>>> to
>>>>>>>> version, so our package will be always in first prio.
>>>>>>>>
>>>>>>>> Also I'm not sure that when there are 2 packages with the
>>>>>>>> same
>>>>>>>> names
>>>>>>>> and
>>>>>>>> versions available in source.list - is a good practice.
>>>>>>>> In
>>>>>>>> this
>>>>>>>> case
>>>>>>>> you
>>>>>>>> have no way to easily check if your rootfs contains
>>>>>>>> correct
>>>>>>>> package,
>>>>>>>> 'dpkg -l' won't be enough.
>>>>>>>>
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>> Thanks, yes that is a very good point.
>>>>>>>>>
>>>>>>>>> I would prefer the apt preferences settings. Maybe
>>>>>>>>> handle
>>>>>>>>> it
>>>>>>>>> similar to
>>>>>>>>> how I proposed the multi-repo support. A global
>>>>>>>>> variable
>>>>>>>>> where
>>>>>>>>> apt
>>>>>>>>> preference file paths are appended.
>>>>>>>>>
>>>>>>>>> If multistrap cannot support them with a reasonable
>>>>>>>>> complex
>>>>>>>>> conversion
>>>>>>>>> script then it might be time to say multistrap goodbye.
>>>>>>>>>
>>>>>>>>> Claudius
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> DENX Software Engineering GmbH,      Managing Director:
>>>>>>> Wolfgang
>>>>>>> Denk
>>>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194
>>>>>>> Groebenzell,
>>>>>>> Germany
>>>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email:
>>>>>>> ch@d
>>>>>>> enx.
>>>>>>> de
>>>>>>>
>>>>>>>                PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173
>>>>>>> CB19
>>>>>>> 9808
>>>>>>> B153
>>>>>>>                                  Keyserver: hkp://pool.sks-
>>>>>>> keyservers.net

Re: Multi repo support

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 17276 bytes --]

On Fri, 2018-02-02 at 17:36 +0300, Alexander Smirnov wrote:
> On 02/02/2018 05:15 PM, Claudius Heine wrote:
> > On Fri, 2018-02-02 at 16:48 +0300, Alexander Smirnov wrote:
> > > Hi,
> > > 
> > > On 02/02/2018 04:26 PM, Claudius Heine wrote:
> > > > On Fri, 2018-02-02 at 15:28 +0300, Alexander Smirnov wrote:
> > > > > Hi again!
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > On Fri, 2018-02-02 at 00:52 +0300, Alexander Smirnov wrote:
> > > > > > > Hi!
> > > > > > > 
> > > > > > > Claudius Heine <ch@denx.de> 2 февраля 2018 г. 0:07:49
> > > > > > > написал:
> > > > > > > 
> > > > > > > > Hi Alex,
> > > > > > > > 
> > > > > > > > On Thu, 2018-02-01 at 23:47 +0300, Alexander Smirnov
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > On 02/01/2018 09:51 PM, Claudius Heine wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On Thu, 2018-02-01 at 19:34 +0100, Henning Schild
> > > > > > > > > > wrote:
> > > > > > > > > > > Am Thu, 1 Feb 2018 16:16:58 +0100
> > > > > > > > > > > schrieb "[ext] Henning Schild" <henning.schild@si
> > > > > > > > > > > emen
> > > > > > > > > > > s.co
> > > > > > > > > > > m>:
> > > > > > > > > > > 
> > > > > > > > > > > > Am Thu, 1 Feb 2018 15:54:26 +0100
> > > > > > > > > > > > schrieb "[ext] Claudius Heine"
> > > > > > > > > > > > <claudius.heine.ext@
> > > > > > > > > > > > siem
> > > > > > > > > > > > ens.
> > > > > > > > > > > > com>
> > > > > > > > > > > > :
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I would like to start the discussion about
> > > > > > > > > > > > > how to
> > > > > > > > > > > > > best
> > > > > > > > > > > > > implement
> > > > > > > > > > > > > muti repository support in isar.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Does someone already has some ideas or even
> > > > > > > > > > > > > something
> > > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > pipeline for this?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > If not then I do have an idea that was
> > > > > > > > > > > > > outlined
> > > > > > > > > > > > > together
> > > > > > > > > > > > > with
> > > > > > > > > > > > > Jan:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Adding and configuring apt repositories
> > > > > > > > > > > > > should be
> > > > > > > > > > > > > done
> > > > > > > > > > > > > via
> > > > > > > > > > > > > config
> > > > > > > > > > > > > files. It should be possible to define own
> > > > > > > > > > > > > multiconfigs
> > > > > > > > > > > > > while
> > > > > > > > > > > > > including multiconfigs from other layers.
> > > > > > > > > > > > > These
> > > > > > > > > > > > > configs
> > > > > > > > > > > > > then
> > > > > > > > > > > > > append
> > > > > > > > > > > > > filepaths to a global variable.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Every file that is added this way contains
> > > > > > > > > > > > > 'sources.list'
> > > > > > > > > > > > > compatible repository definitions. So one
> > > > > > > > > > > > > repo
> > > > > > > > > > > > > each
> > > > > > > > > > > > > line.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > For every line in those files a repository
> > > > > > > > > > > > > entry
> > > > > > > > > > > > > for
> > > > > > > > > > > > > multistrap.conf
> > > > > > > > > > > > > is created. Here we might need some more
> > > > > > > > > > > > > complex
> > > > > > > > > > > > > code
> > > > > > > > > > > > > to
> > > > > > > > > > > > > convert
> > > > > > > > > > > > > such a apt repo tripel to the right format
> > > > > > > > > > > > > multistrap
> > > > > > > > > > > > > expects.
> > > > > > > > > > > > > But
> > > > > > > > > > > > > by using the 'sources.lists' format we would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > independent
> > > > > > > > > > > > > of
> > > > > > > > > > > > > multistrap and become more future proof.
> > > > > > > > > > > 
> > > > > > > > > > > We will also need a way to tell apt the
> > > > > > > > > > > priorities of
> > > > > > > > > > > these
> > > > > > > > > > > repos.
> > > > > > > > > > > Multistrap just adds them and apt-get installs
> > > > > > > > > > > packages
> > > > > > > > > > > according
> > > > > > > > > > > to
> > > > > > > > > > > its default behavior.
> > > > > > > > > > > That means that a package with the same name and
> > > > > > > > > > > version
> > > > > > > > > > > will
> > > > > > > > > > > get
> > > > > > > > > > > picked from a "random" repo. Overlays will need
> > > > > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > to
> > > > > > > > > > > always
> > > > > > > > > > > have a greater version or we will need apt
> > > > > > > > > > > configuration.
> > > > > > > > > > > https://wiki.debian.org/AptPreferences
> > > > > > > > > > > The big question here is how/whether multistrap
> > > > > > > > > > > will
> > > > > > > > > > > handle
> > > > > > > > > > > apt
> > > > > > > > > > > preference.
> > > > > > > > > > > 
> > > > > > > > > > > Alex already ran into that when he wanted to
> > > > > > > > > > > modify
> > > > > > > > > > > "hello".
> > > > > > > > > > > Renaming
> > > > > > > > > > > packages - like Alex suggested - is not the way
> > > > > > > > > > > to
> > > > > > > > > > > go.
> > > > > > > > > > > Because
> > > > > > > > > > > the
> > > > > > > > > > > ones
> > > > > > > > > > > we do overlay could be deps of packages we do not
> > > > > > > > > > > overlay.
> > > > > > > > > 
> > > > > > > > > The version collision could be avoided by using
> > > > > > > > > epoch,
> > > > > > > > > and if
> > > > > > > > > I
> > > > > > > > > understand it correctly - this is the preferable way
> > > > > > > > > to
> > > > > > > > > assign
> > > > > > > > > apt
> > > > > > > > > priority.
> > > > > > > > > 
> > > > > > > > > https://www.debian.org/doc/debian-policy/#s-f-version
> > > > > > > > 
> > > > > > > > Where did you get that epoch is the preferable way to
> > > > > > > > assign
> > > > > > > > apt
> > > > > > > > priority?
> > > > > > > 
> > > > > > > Not the apt, just specific package. Hmm, I'm a bit
> > > > > > > incorrectly
> > > > > > > called
> > > > > > > my
> > > > > > > pov. :-) I see epoch as an easy way to patch existing
> > > > > > > package
> > > > > > > from
> > > > > > > upstream. You don't care about upstream one's version
> > > > > > > even if
> > > > > > > it
> > > > > > > goes
> > > > > > > forward, until your epoch is higher - your package will
> > > > > > > be
> > > > > > > installed.
> > > > > > > Of
> > > > > > > course it doesn't mean that upstream epoch couldn't be
> > > > > > > changed
> > > > > > > too,
> > > > > > > but
> > > > > > > this happens quite rare.
> > > > > > > 
> > > > > > > > 
> > > > > > > >    From you link:
> > > > > > > > 
> > > > > > > >        epoch
> > > > > > > >            This is a single (generally small) unsigned
> > > > > > > > integer.
> > > > > > > > It may
> > > > > > > > be
> > > > > > > > omitted, in which case zero is assumed. If it is
> > > > > > > > omitted
> > > > > > > > then
> > > > > > > > the
> > > > > > > > upstream_version may not contain any colons.
> > > > > > > >            It is provided to allow mistakes in the
> > > > > > > > version
> > > > > > > > numbers of
> > > > > > > > older versions of a package, and also a package’s
> > > > > > > > previous
> > > > > > > > version
> > > > > > > > numbering schemes, to be left behind.
> > > > > > > > 
> > > > > > > > I see epoch just as a possibility to fix mistakes or if
> > > > > > > > upstream
> > > > > > > > changed their versioning scheme, not use it use it
> > > > > > > > generally
> > > > > > > > for
> > > > > > > > specifying package priorities. Using it that way sound
> > > > > > > > more
> > > > > > > > like a
> > > > > > > > hack
> > > > > > > > to me. Because what happens if debian raises the epoch
> > > > > > > > of a
> > > > > > > > package?
> > > > > > > > Then you would have to raise yours again? Sounds like
> > > > > > > > playing
> > > > > > > > poker.
> > > > > > > 
> > > > > > > Could you please describe the case when you have two
> > > > > > > repositories
> > > > > > > which
> > > > > > > contain package with the same name-version but different
> > > > > > > content?
> > > > > > 
> > > > > > With apt-preferences we could prefer every package from our
> > > > > > own
> > > > > > repo
> > > > > > with the same name AFAIK, but this would make the original
> > > > > > package
> > > > > > no
> > > > > > longer install-able beside that.
> > > > > > And that is a limitation of debian/dpkg and we can do
> > > > > > nothing
> > > > > > about
> > > > > > it
> > > > > > in isar. This problem it not ours to solve.
> > > > > > 
> > > > > > Ideally debian would introduce package namespaces or
> > > > > > similar.
> > > > > > But
> > > > > > that
> > > > > > is completely out of isars scope. On our side we could only
> > > > > > try
> > > > > > some
> > > > > > hacks to avoid name collision, but I would rather spend
> > > > > > time
> > > > > > solving
> > > > > > other problems in isar than that, since just choosing a
> > > > > > different
> > > > > > name
> > > > > > is pretty easy. Multi-repo or reproducible builds are much
> > > > > > more
> > > > > > important issues than trying to play cat and mouse with
> > > > > > package
> > > > > > names.
> > > > > > 
> > > > > 
> > > > > I see this. But what is the practical usecase to have the
> > > > > same
> > > > > packages
> > > > > in several repositories?
> > > > 
> > > > Where did this question comes from? It does not really make
> > > > sense
> > > > to
> > > 
> > >   From the text above, but I found that it wasn't your message,
> > > sorry
> > > :-)
> > > 
> > > ...
> > > That means that a package with the same name and version will get
> > > picked from a "random" repo.
> > > ...
> > > 
> > > > have exactly the same package in multiple different
> > > > repositories.
> > > > But
> > > > that is AFAIK not related to any of this.
> > > > 
> > > 
> > > So if it's not the case, then it's Ok for me.
> > > 
> > > > If you have changed some existing debian package put it into
> > > > your
> > > > own
> > > > repo, maybe change the version of the package to contain
> > > > something
> > > > like
> > > > "*~isar0" similar to how ubuntu does it. Then set a apt-
> > > > preference
> > > > like
> > > > this:
> > > > 
> > > >       Package: *
> > > >       Pin: release c=isar
> > > >       Pin-Priority: 650
> > > > 
> > > > or similar. And apt would prefer to use packages from the isar
> > > > repo
> > > > instead of upstream if there is a name collision. Even if the
> > > > version
> > > > of the package in isar is lower than the one upstream.
> > > 
> > > Do you think it's possible to handle the following scenario:
> > > 
> > > 1. You pick current glibc from upstream distro XYZ, let's say
> > > v1.0.
> > > 2. Patch it and put glibc-v1.0-isar to local repo.
> > > 3. Wait for some time until glibc is updated in upstream, let's
> > > say
> > > to v1.1.
> > > 4. Due to glibc-v1.0-isar is set to be preferred, it's going to
> > > be
> > > installed. But glibc is the core library in distro, and lots of
> > > packages
> > > in XYZ now require glibc-v1.1. So you won't be able to generate
> > > image
> > > anymore, it will fail with unmet dependency.
> > 
> > Why do we need to handle this?
> 
> If you use upstream sources and prioritize apts, this problem likely 
> will occur. So repo prioritizing is then unpredictable feature.

Yes, but that is predictable unpredictable.
If you want predictability don't mix new upstream and old own stuff.

You need to have the reproducibility data from that one point in time
where you own stuff and upstream has worked together and was tested
against each other. Otherwise build fails and you need to update your
code.

To make it absolutely clear, the only method I see to solve this with
isar is implementing reproducibility. And then this is no longer a
problem.

> 
> > 
> > It would be great if image generation fails in this case and not
> > cause
> > the generation of a broken image.
> > 
> > Otherwise that is the reason we should have reproducible builds, so
> > that we can still create the old image.
> 
> If you don't work with upstream and use only local repositories, why 
> can't you just drop duplicated packages from them?

That completely depends on how you are implementing reproducible
builds.

If reproducibilty is implemented the way I suggested, then the local
upstream mirror would have an entry in the package index for the other
package but would not store the package data itself, since it was not
downloaded while building the image. The apt preference entry would
still be needed.

So I don't see prioritizing as an unpredictable feature, since you
either have to make sure that the custom stuff is up to date or that
you have reproducible data from that time when your custom stuff was up
to date.

IMO updating/fetching stuff from upstream should not be the default
only the fallback. This way the dev would always has the reproducible
data that has worked for him once available.

Cheers,
Claudius

> 
> Alex
> 
> > > > > > > > Telling apt to prefer packages with specific versions
> > > > > > > > or
> > > > > > > > from
> > > > > > > > specific
> > > > > > > > repositories via apt-preferences sounds more like the
> > > > > > > > right
> > > > > > > > tool
> > > > > > > > for
> > > > > > > > the job.
> > > > > > > > 
> > > > > > > > Cheers,
> > > > > > > > Claudius
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Regarding renaming 'hello' there was a bit different
> > > > > > > > > case,
> > > > > > > > > our
> > > > > > > > > 'hello'
> > > > > > > > > has nothing common with the upstream one, moreover
> > > > > > > > > I've
> > > > > > > > > already
> > > > > > > > > modified
> > > > > > > > > it to use 'libhello'. So this is not the case when we
> > > > > > > > > patch
> > > > > > > > > the
> > > > > > > > > upstream
> > > > > > > > > package. So I decided to rename it, to avoid
> > > > > > > > > confusion.
> > > > > > > > > 
> > > > > > > > > In case of patching the upstream package, we could
> > > > > > > > > just
> > > > > > > > > add
> > > > > > > > > epoch
> > > > > > > > > to
> > > > > > > > > version, so our package will be always in first prio.
> > > > > > > > > 
> > > > > > > > > Also I'm not sure that when there are 2 packages with
> > > > > > > > > the
> > > > > > > > > same
> > > > > > > > > names
> > > > > > > > > and
> > > > > > > > > versions available in source.list - is a good
> > > > > > > > > practice.
> > > > > > > > > In
> > > > > > > > > this
> > > > > > > > > case
> > > > > > > > > you
> > > > > > > > > have no way to easily check if your rootfs contains
> > > > > > > > > correct
> > > > > > > > > package,
> > > > > > > > > 'dpkg -l' won't be enough.
> > > > > > > > > 
> > > > > > > > > Alex
> > > > > > > > > 
> > > > > > > > > > Thanks, yes that is a very good point.
> > > > > > > > > > 
> > > > > > > > > > I would prefer the apt preferences settings. Maybe
> > > > > > > > > > handle
> > > > > > > > > > it
> > > > > > > > > > similar to
> > > > > > > > > > how I proposed the multi-repo support. A global
> > > > > > > > > > variable
> > > > > > > > > > where
> > > > > > > > > > apt
> > > > > > > > > > preference file paths are appended.
> > > > > > > > > > 
> > > > > > > > > > If multistrap cannot support them with a reasonable
> > > > > > > > > > complex
> > > > > > > > > > conversion
> > > > > > > > > > script then it might be time to say multistrap
> > > > > > > > > > goodbye.
> > > > > > > > > > 
> > > > > > > > > > Claudius
> > > > > > > > > > 
> > > > > > > > 
> > > > > > > > --
> > > > > > > > DENX Software Engineering GmbH,      Managing Director:
> > > > > > > > Wolfgang
> > > > > > > > Denk
> > > > > > > > HRB 165235 Munich, Office: Kirchenstr.5, D-82194
> > > > > > > > Groebenzell,
> > > > > > > > Germany
> > > > > > > > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80
> > > > > > > > Email:
> > > > > > > > ch@d
> > > > > > > > enx.
> > > > > > > > de
> > > > > > > > 
> > > > > > > >                PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1
> > > > > > > > 1173
> > > > > > > > CB19
> > > > > > > > 9808
> > > > > > > > B153
> > > > > > > >                                  Keyserver:
> > > > > > > > hkp://pool.sks-
> > > > > > > > keyservers.net
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Multi repo support

[Claudius Heine]

Hi,

On 02/01/2018 03:54 PM, [ext] Claudius Heine wrote:
> Hi,
> 
> I would like to start the discussion about how to best implement muti 
> repository support in isar.
> 
> Does someone already has some ideas or even something in the pipeline 
> for this?
> 
> If not then I do have an idea that was outlined together with Jan:
> 
> Adding and configuring apt repositories should be done via config files. 
> It should be possible to define own multiconfigs while including 
> multiconfigs from other layers. These configs then append filepaths to a 
> global variable.
> 
> Every file that is added this way contains 'sources.list' compatible 
> repository definitions. So one repo each line.
> 
> For every line in those files a repository entry for multistrap.conf is 
> created. Here we might need some more complex code to convert such a apt 
> repo tripel to the right format multistrap expects. But by using the 
> 'sources.lists' format we would be independent of multistrap and become 
> more future proof.
> 
> Comments are welcome.

So for reference here are the points gathered from the comments in this 
thread:

   - It needs some way to handle package name collision. For instance if
     custom packages have the same name as an upstream package.
     Suggestion: Use apt-preferences to prefer packages from the 'isar'
                 repo. To consider: How to teach multistrap about those
                 preferences?
   - Overwriting source entries with local mirrors.
     Suggestion: Similar to how oe does it with PREMIRRORS.
                 e.g. DEBIAN_APT_MIRROR_append = "regex replace\n"
                 To be implemented later.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Multi repo support

[Jan Kiszka]

On 2018-02-07 10:05, [ext] Claudius Heine wrote:
> Hi,
> 
> On 02/01/2018 03:54 PM, [ext] Claudius Heine wrote:
>> Hi,
>>
>> I would like to start the discussion about how to best implement muti
>> repository support in isar.
>>
>> Does someone already has some ideas or even something in the pipeline
>> for this?
>>
>> If not then I do have an idea that was outlined together with Jan:
>>
>> Adding and configuring apt repositories should be done via config
>> files. It should be possible to define own multiconfigs while
>> including multiconfigs from other layers. These configs then append
>> filepaths to a global variable.
>>
>> Every file that is added this way contains 'sources.list' compatible
>> repository definitions. So one repo each line.
>>
>> For every line in those files a repository entry for multistrap.conf
>> is created. Here we might need some more complex code to convert such
>> a apt repo tripel to the right format multistrap expects. But by using
>> the 'sources.lists' format we would be independent of multistrap and
>> become more future proof.
>>
>> Comments are welcome.
> 
> So for reference here are the points gathered from the comments in this
> thread:
> 
>   - It needs some way to handle package name collision. For instance if
>     custom packages have the same name as an upstream package.
>     Suggestion: Use apt-preferences to prefer packages from the 'isar'
>                 repo. To consider: How to teach multistrap about those
>                 preferences?

https://manpages.debian.org/stretch/multistrap/multistrap.1.en.html#Apt_preferences

>   - Overwriting source entries with local mirrors.
>     Suggestion: Similar to how oe does it with PREMIRRORS.
>                 e.g. DEBIAN_APT_MIRROR_append = "regex replace\n"
>                 To be implemented later.

As discussed offline, this sounds like valuable add-on for later. Until
then, the user should be able to replace repo lists that are pointing to
undesired URLs with own ones with the preferred URLs.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Multi repo support

[Claudius Heine]

Hi Jan,

On 02/07/2018 10:21 AM, Jan Kiszka wrote:
> On 2018-02-07 10:05, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 02/01/2018 03:54 PM, [ext] Claudius Heine wrote:
>>> Hi,
>>>
>>> I would like to start the discussion about how to best implement muti
>>> repository support in isar.
>>>
>>> Does someone already has some ideas or even something in the pipeline
>>> for this?
>>>
>>> If not then I do have an idea that was outlined together with Jan:
>>>
>>> Adding and configuring apt repositories should be done via config
>>> files. It should be possible to define own multiconfigs while
>>> including multiconfigs from other layers. These configs then append
>>> filepaths to a global variable.
>>>
>>> Every file that is added this way contains 'sources.list' compatible
>>> repository definitions. So one repo each line.
>>>
>>> For every line in those files a repository entry for multistrap.conf
>>> is created. Here we might need some more complex code to convert such
>>> a apt repo tripel to the right format multistrap expects. But by using
>>> the 'sources.lists' format we would be independent of multistrap and
>>> become more future proof.
>>>
>>> Comments are welcome.
>>
>> So for reference here are the points gathered from the comments in this
>> thread:
>>
>>    - It needs some way to handle package name collision. For instance if
>>      custom packages have the same name as an upstream package.
>>      Suggestion: Use apt-preferences to prefer packages from the 'isar'
>>                  repo. To consider: How to teach multistrap about those
>>                  preferences?
> 
> https://manpages.debian.org/stretch/multistrap/multistrap.1.en.html#Apt_preferences

Great, thanks!

Claudius

> 
>>    - Overwriting source entries with local mirrors.
>>      Suggestion: Similar to how oe does it with PREMIRRORS.
>>                  e.g. DEBIAN_APT_MIRROR_append = "regex replace\n"
>>                  To be implemented later.
> 
> As discussed offline, this sounds like valuable add-on for later. Until
> then, the user should be able to replace repo lists that are pointing to
> undesired URLs with own ones with the preferred URLs.
> 
> Jan
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de