image-postproc-extension.bbclass modifying /etc/os-release

image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

Hi,

I've noticed that '/etc/os-release' is being changed on the image in meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up changing '/usr/lib/os-release', since it's only a symlink. But both '/etc/os-release' and '/usr/lib/os-release' are owned by 'base-files'...

An upgrade of 'base-files' would be replacing (silently, since is not marked as a configuration file) '/usr/lib/os-release' with the version of the upstream 'base-files' package and possibly breaking any tools in the system relying on certain values in that file.

Is there a reason that I'm missing for doing so instead of the Debian-way (file diversion with dpkg-divert)? Or any hack that I've overseen that avoids the mentioned breakage?

Hasta la vista,

  Silvano

[1] https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta/classes/image-postproc-extension.bbclass#L20

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

I might try to provide a fix, if we agree that the current implementation has an issue.

@Claudius: you wrote the original code [1]. Do you remember why you implemented it this way? Do you remember if you were aware of the issue I mentioned and you provided a mitigation for the issue that I see (assuming my analysis is right)?

[1] https://github.com/ilbers/isar/commit/13ce96e5bc84b60f2fa7ccfe93dde045461884e6

  Silvano

On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:
> Hi,
>
> I've noticed that '/etc/os-release' is being changed on the image in meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up changing '/usr/lib/os-release', since it's only a symlink. But both '/etc/os-release' and '/usr/lib/os-release' are owned by 'base-files'...
>
> An upgrade of 'base-files' would be replacing (silently, since is not marked as a configuration file) '/usr/lib/os-release' with the version of the upstream 'base-files' package and possibly breaking any tools in the system relying on certain values in that file.
>
> Is there a reason that I'm missing for doing so instead of the Debian-way (file diversion with dpkg-divert)? Or any hack that I've overseen that avoids the mentioned breakage?
>
> Hasta la vista,
>
>   Silvano
>
> [1] https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta/classes/image-postproc-extension.bbclass#L20
>

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Claudius Heine]

Hi Silvano,

On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:
> I might try to provide a fix, if we agree that the current implementation has an issue.
> 
> @Claudius: you wrote the original code [1]. Do you remember why you implemented it this way? Do you remember if you were aware of the issue I mentioned and you provided a mitigation for the issue that I see (assuming my analysis is right)?
> 
> [1] https://github.com/ilbers/isar/commit/13ce96e5bc84b60f2fa7ccfe93dde045461884e6
> 
>    Silvano
> 
> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:
>> Hi,
>>
>> I've noticed that '/etc/os-release' is being changed on the image in meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up changing '/usr/lib/os-release', since it's only a symlink. But both '/etc/os-release' and '/usr/lib/os-release' are owned by 'base-files'...
>>
>> An upgrade of 'base-files' would be replacing (silently, since is not marked as a configuration file) '/usr/lib/os-release' with the version of the upstream 'base-files' package and possibly breaking any tools in the system relying on certain values in that file.
>>
>> Is there a reason that I'm missing for doing so instead of the Debian-way (file diversion with dpkg-divert)? Or any hack that I've overseen that avoids the mentioned breakage?

Interesting, I didn't remember that `/etc/os-release` is a symlink, 
could that be something that has changed in more recent debian versions?

If so then, of course that needs to be fixed.

regards,
Claudius

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

I've tested it right now with the Qemu AMD64 image:

1. Modified it so that /etc/os-release has "my information".

2. Confirmed that /etc/os-release in the VM has "my information".

3. Made the current 'base-files' upstream package [1] available to the VM (additional disk, editing the generated image,...).

4. Installed the current 'base-files' upstream package.

5. Confirmed that "my" /etc/os-release has been silently replaced by a symlink to /usr/lib/os-release.

Anybody not doing package updates on 'base-files' won't notice anything, but the bug will still be there. Only covered by a convention, uncover it and it'll bite you...

Please convince me :-) that it's not a bug or that there are no realistic scenarios where it can be triggered... if not I'll contribute a fix.

  Silvano

[1] http://ftp.de.debian.org/debian/pool/main/b/base-files/base-files_10.3+deb10u7_amd64.deb

On 18/01/2021 13:35, Claudius Heine wrote:
> Hi Silvano,
>
> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:
>> I might try to provide a fix, if we agree that the current implementation has an issue.
>>
>> @Claudius: you wrote the original code [1]. Do you remember why you implemented it this way? Do you remember if you were aware of the issue I mentioned and you provided a mitigation for the issue that I see (assuming my analysis is right)?
>>
>> [1] https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C0bf1a83dda684b9e03e408d8bbad9a58%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637465701582684886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rRq%2F%2BKNrHdUn1SV%2FxVTIOSH6BofosIhUrFZWyy5PmSc%3D&reserved=0
>>
>>    Silvano
>>
>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:
>>> Hi,
>>>
>>> I've noticed that '/etc/os-release' is being changed on the image in meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up changing '/usr/lib/os-release', since it's only a symlink. But both '/etc/os-release' and '/usr/lib/os-release' are owned by 'base-files'...
>>>
>>> An upgrade of 'base-files' would be replacing (silently, since is not marked as a configuration file) '/usr/lib/os-release' with the version of the upstream 'base-files' package and possibly breaking any tools in the system relying on certain values in that file.
>>>
>>> Is there a reason that I'm missing for doing so instead of the Debian-way (file diversion with dpkg-divert)? Or any hack that I've overseen that avoids the mentioned breakage?
>
> Interesting, I didn't remember that `/etc/os-release` is a symlink, could that be something that has changed in more recent debian versions?
>
> If so then, of course that needs to be fixed.
>
> regards,
> Claudius

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Mon, 18 Jan 2021 13:35:53 +0100
schrieb Claudius Heine <ch@denx.de>:

> Hi Silvano,
> 
> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:
> > I might try to provide a fix, if we agree that the current
> > implementation has an issue.
> > 
> > @Claudius: you wrote the original code [1]. Do you remember why you
> > implemented it this way? Do you remember if you were aware of the
> > issue I mentioned and you provided a mitigation for the issue that
> > I see (assuming my analysis is right)?
> > 
> > [1]
> > https://github.com/ilbers/isar/commit/13ce96e5bc84b60f2fa7ccfe93dde045461884e6
> > 
> >    Silvano
> > 
> > On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:  
> >> Hi,
> >>
> >> I've noticed that '/etc/os-release' is being changed on the image
> >> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
> >> ends up changing '/usr/lib/os-release', since it's only a symlink.
> >> But both '/etc/os-release' and '/usr/lib/os-release' are owned by
> >> 'base-files'...
> >>
> >> An upgrade of 'base-files' would be replacing (silently, since is
> >> not marked as a configuration file) '/usr/lib/os-release' with the
> >> version of the upstream 'base-files' package and possibly breaking
> >> any tools in the system relying on certain values in that file.
> >>
> >> Is there a reason that I'm missing for doing so instead of the
> >> Debian-way (file diversion with dpkg-divert)? Or any hack that
> >> I've overseen that avoids the mentioned breakage?  
> 
> Interesting, I didn't remember that `/etc/os-release` is a symlink, 
> could that be something that has changed in more recent debian
> versions?
> 
> If so then, of course that needs to be fixed.

the problem seems to be that it is a symlink, otherwise one would
assume that changes in /etc/ are allowed and covered by the config file
exception and will be subject to merging if an updated package comes
around.

My guess would be that we need to
 - make it a copy instead of a symlink
 - modify it

In this case an update of the base-files package should leave it alone
or ask for a merge. And i think that would be OK behaviour.

Henning

> regards,
> Claudius
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Tue, 19 Jan 2021 09:25:31 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Am Mon, 18 Jan 2021 13:35:53 +0100
> schrieb Claudius Heine <ch@denx.de>:
> 
> > Hi Silvano,
> > 
> > On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:  
> > > I might try to provide a fix, if we agree that the current
> > > implementation has an issue.
> > > 
> > > @Claudius: you wrote the original code [1]. Do you remember why
> > > you implemented it this way? Do you remember if you were aware of
> > > the issue I mentioned and you provided a mitigation for the issue
> > > that I see (assuming my analysis is right)?
> > > 
> > > [1]
> > > https://github.com/ilbers/isar/commit/13ce96e5bc84b60f2fa7ccfe93dde045461884e6
> > > 
> > >    Silvano
> > > 
> > > On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:    
> > >> Hi,
> > >>
> > >> I've noticed that '/etc/os-release' is being changed on the image
> > >> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
> > >> ends up changing '/usr/lib/os-release', since it's only a
> > >> symlink. But both '/etc/os-release' and '/usr/lib/os-release'
> > >> are owned by 'base-files'...
> > >>
> > >> An upgrade of 'base-files' would be replacing (silently, since is
> > >> not marked as a configuration file) '/usr/lib/os-release' with
> > >> the version of the upstream 'base-files' package and possibly
> > >> breaking any tools in the system relying on certain values in
> > >> that file.
> > >>
> > >> Is there a reason that I'm missing for doing so instead of the
> > >> Debian-way (file diversion with dpkg-divert)? Or any hack that
> > >> I've overseen that avoids the mentioned breakage?    
> > 
> > Interesting, I didn't remember that `/etc/os-release` is a symlink, 
> > could that be something that has changed in more recent debian
> > versions?
> > 
> > If so then, of course that needs to be fixed.  
> 
> the problem seems to be that it is a symlink, otherwise one would
> assume that changes in /etc/ are allowed and covered by the config
> file exception and will be subject to merging if an updated package
> comes around.
> 
> My guess would be that we need to
>  - make it a copy instead of a symlink
>  - modify it

Alternative would be to create /etc/os-release-isar as that modified
copy, and replace the symlink target /etc/os-release ->
/etc/os-release-isar

We can not have the modified file be part of any package, at least not
with its final content. The version control command has to happen on
every rootfs build and has to come "late", and a package would not have
that property. Plus a package would become subject to potential false
sharing in the multiconfig case ... The fields we change are mostly
image-properties, and not generic

Henning

> 
> In this case an update of the base-files package should leave it alone
> or ask for a merge. And i think that would be OK behaviour.
> 
> Henning
> 
> > regards,
> > Claudius
> >   
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 19/01/2021 09:33, Henning Schild wrote:
> Am Tue, 19 Jan 2021 09:25:31 +0100
> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>
>> Am Mon, 18 Jan 2021 13:35:53 +0100
>> schrieb Claudius Heine <ch@denx.de>:
>>
>>> Hi Silvano,
>>>
>>> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:  
>>>> I might try to provide a fix, if we agree that the current
>>>> implementation has an issue.
>>>>
>>>> @Claudius: you wrote the original code [1]. Do you remember why
>>>> you implemented it this way? Do you remember if you were aware of
>>>> the issue I mentioned and you provided a mitigation for the issue
>>>> that I see (assuming my analysis is right)?
>>>>
>>>> [1]
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc75c1ca076a0460e8a2a08d8bc54e4dd%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466420092677953%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=c3d3xPuWoZC616jMfcDeqo9g1zolQIYgHsN%2FdKc1RC8%3D&amp;reserved=0
>>>>
>>>>    Silvano
>>>>
>>>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:    
>>>>> Hi,
>>>>>
>>>>> I've noticed that '/etc/os-release' is being changed on the image
>>>>> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
>>>>> ends up changing '/usr/lib/os-release', since it's only a
>>>>> symlink. But both '/etc/os-release' and '/usr/lib/os-release'
>>>>> are owned by 'base-files'...
>>>>>
>>>>> An upgrade of 'base-files' would be replacing (silently, since is
>>>>> not marked as a configuration file) '/usr/lib/os-release' with
>>>>> the version of the upstream 'base-files' package and possibly
>>>>> breaking any tools in the system relying on certain values in
>>>>> that file.
>>>>>
>>>>> Is there a reason that I'm missing for doing so instead of the
>>>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
>>>>> I've overseen that avoids the mentioned breakage?    
>>> Interesting, I didn't remember that `/etc/os-release` is a symlink, 
>>> could that be something that has changed in more recent debian
>>> versions?
>>>
>>> If so then, of course that needs to be fixed.  
>> the problem seems to be that it is a symlink, otherwise one would
>> assume that changes in /etc/ are allowed and covered by the config
>> file exception and will be subject to merging if an updated package
>> comes around.
>>
>> My guess would be that we need to
>>  - make it a copy instead of a symlink
>>  - modify it
> Alternative would be to create /etc/os-release-isar as that modified
> copy, and replace the symlink target /etc/os-release ->
> /etc/os-release-isar
No, nothing touching /etc/os-release off-band will block "base-files" from restoring it. Creating another package touching /etc/os-release will create a conflict that will be detected by dpkg. Either upstream "base-files" is not there or we create a file diversion for /etc/os-release. IMO we are creating a Debian Derivative and as therefore I'd consider their re-/de-branding guidelines (see separate thread).
>
> We can not have the modified file be part of any package, at least not
> with its final content. The version control command has to happen on
> every rootfs build and has to come "late", and a package would not have
> that property. Plus a package would become subject to potential false
> sharing in the multiconfig case ... The fields we change are mostly
> image-properties, and not generic

Image post-processing is deemed to fail.

Sorry, I don't understand what you mean with "a package would become subject to potential false sharing in the multiconfig case". A package created on build-time can contain those image properties, right? I might be missing something...

   Silvano

>
> Henning
>
>> In this case an update of the base-files package should leave it alone
>> or ask for a merge. And i think that would be OK behaviour.
>>
>> Henning
>>
>>> regards,
>>> Claudius
>>>   

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Tue, 19 Jan 2021 09:50:27 +0100
schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:

> On 19/01/2021 09:33, Henning Schild wrote:
> > Am Tue, 19 Jan 2021 09:25:31 +0100
> > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> >  
> >> Am Mon, 18 Jan 2021 13:35:53 +0100
> >> schrieb Claudius Heine <ch@denx.de>:
> >>  
> >>> Hi Silvano,
> >>>
> >>> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:    
> >>>> I might try to provide a fix, if we agree that the current
> >>>> implementation has an issue.
> >>>>
> >>>> @Claudius: you wrote the original code [1]. Do you remember why
> >>>> you implemented it this way? Do you remember if you were aware of
> >>>> the issue I mentioned and you provided a mitigation for the issue
> >>>> that I see (assuming my analysis is right)?
> >>>>
> >>>> [1]
> >>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7Cfb6a5b02cdb74a01fe7708d8bc5745c5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466430297681379%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=MLD8tTQoJprCpQsG%2BRL6oSg8jiR%2BUBUHG4J89VAEWDg%3D&amp;reserved=0
> >>>>
> >>>>    Silvano
> >>>>
> >>>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:      
> >>>>> Hi,
> >>>>>
> >>>>> I've noticed that '/etc/os-release' is being changed on the
> >>>>> image in meta/classes/image-postproc-extension.bbclass [1].
> >>>>> What BTW ends up changing '/usr/lib/os-release', since it's
> >>>>> only a symlink. But both '/etc/os-release' and
> >>>>> '/usr/lib/os-release' are owned by 'base-files'...
> >>>>>
> >>>>> An upgrade of 'base-files' would be replacing (silently, since
> >>>>> is not marked as a configuration file) '/usr/lib/os-release'
> >>>>> with the version of the upstream 'base-files' package and
> >>>>> possibly breaking any tools in the system relying on certain
> >>>>> values in that file.
> >>>>>
> >>>>> Is there a reason that I'm missing for doing so instead of the
> >>>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
> >>>>> I've overseen that avoids the mentioned breakage?      
> >>> Interesting, I didn't remember that `/etc/os-release` is a
> >>> symlink, could that be something that has changed in more recent
> >>> debian versions?
> >>>
> >>> If so then, of course that needs to be fixed.    
> >> the problem seems to be that it is a symlink, otherwise one would
> >> assume that changes in /etc/ are allowed and covered by the config
> >> file exception and will be subject to merging if an updated package
> >> comes around.
> >>
> >> My guess would be that we need to
> >>  - make it a copy instead of a symlink
> >>  - modify it  
> > Alternative would be to create /etc/os-release-isar as that modified
> > copy, and replace the symlink target /etc/os-release ->
> > /etc/os-release-isar  
> No, nothing touching /etc/os-release off-band will block "base-files"
> from restoring it. Creating another package touching /etc/os-release
> will create a conflict that will be detected by dpkg. Either upstream
> "base-files" is not there or we create a file diversion for
> /etc/os-release. IMO we are creating a Debian Derivative and as
> therefore I'd consider their re-/de-branding guidelines (see separate
> thread).
> >
> > We can not have the modified file be part of any package, at least
> > not with its final content. The version control command has to
> > happen on every rootfs build and has to come "late", and a package
> > would not have that property. Plus a package would become subject
> > to potential false sharing in the multiconfig case ... The fields
> > we change are mostly image-properties, and not generic  
> 
> Image post-processing is deemed to fail.
> 
> Sorry, I don't understand what you mean with "a package would become
> subject to potential false sharing in the multiconfig case". A
> package created on build-time can contain those image properties,
> right? I might be missing something...

No a package can not do the job, unfortunately. That is why we violate
the rule that every modification should be part of a package. It is not
because that file is potentially already owned by another package, that
can always be solved with overwriting in a package hook.

The package would need to have a unique name and version for every new
output of "git describe" and for every image description. In
multiconfig we build multiple images in one tree, sharing isar-apt.

Henning

>    Silvano
> 
> >
> > Henning
> >  
> >> In this case an update of the base-files package should leave it
> >> alone or ask for a merge. And i think that would be OK behaviour.
> >>
> >> Henning
> >>  
> >>> regards,
> >>> Claudius
> >>>     
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 19/01/2021 10:22, Henning Schild wrote:
> Am Tue, 19 Jan 2021 09:50:27 +0100
> schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:
>
>> On 19/01/2021 09:33, Henning Schild wrote:
>>> Am Tue, 19 Jan 2021 09:25:31 +0100
>>> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>>>  
>>>> Am Mon, 18 Jan 2021 13:35:53 +0100
>>>> schrieb Claudius Heine <ch@denx.de>:
>>>>  
>>>>> Hi Silvano,
>>>>>
>>>>> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:    
>>>>>> I might try to provide a fix, if we agree that the current
>>>>>> implementation has an issue.
>>>>>>
>>>>>> @Claudius: you wrote the original code [1]. Do you remember why
>>>>>> you implemented it this way? Do you remember if you were aware of
>>>>>> the issue I mentioned and you provided a mitigation for the issue
>>>>>> that I see (assuming my analysis is right)?
>>>>>>
>>>>>> [1]
>>>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C7170c79e6ec14d6df1ac08d8bc5bcaf8%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466449720721035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ejDae1qkGm4bNOnsk1oRZeGGkJLs7tXVmhHx%2B4pnv2Q%3D&amp;reserved=0
>>>>>>
>>>>>>    Silvano
>>>>>>
>>>>>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:      
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've noticed that '/etc/os-release' is being changed on the
>>>>>>> image in meta/classes/image-postproc-extension.bbclass [1].
>>>>>>> What BTW ends up changing '/usr/lib/os-release', since it's
>>>>>>> only a symlink. But both '/etc/os-release' and
>>>>>>> '/usr/lib/os-release' are owned by 'base-files'...
>>>>>>>
>>>>>>> An upgrade of 'base-files' would be replacing (silently, since
>>>>>>> is not marked as a configuration file) '/usr/lib/os-release'
>>>>>>> with the version of the upstream 'base-files' package and
>>>>>>> possibly breaking any tools in the system relying on certain
>>>>>>> values in that file.
>>>>>>>
>>>>>>> Is there a reason that I'm missing for doing so instead of the
>>>>>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
>>>>>>> I've overseen that avoids the mentioned breakage?      
>>>>> Interesting, I didn't remember that `/etc/os-release` is a
>>>>> symlink, could that be something that has changed in more recent
>>>>> debian versions?
>>>>>
>>>>> If so then, of course that needs to be fixed.    
>>>> the problem seems to be that it is a symlink, otherwise one would
>>>> assume that changes in /etc/ are allowed and covered by the config
>>>> file exception and will be subject to merging if an updated package
>>>> comes around.
>>>>
>>>> My guess would be that we need to
>>>>  - make it a copy instead of a symlink
>>>>  - modify it  
>>> Alternative would be to create /etc/os-release-isar as that modified
>>> copy, and replace the symlink target /etc/os-release ->
>>> /etc/os-release-isar  
>> No, nothing touching /etc/os-release off-band will block "base-files"
>> from restoring it. Creating another package touching /etc/os-release
>> will create a conflict that will be detected by dpkg. Either upstream
>> "base-files" is not there or we create a file diversion for
>> /etc/os-release. IMO we are creating a Debian Derivative and as
>> therefore I'd consider their re-/de-branding guidelines (see separate
>> thread).
>>> We can not have the modified file be part of any package, at least
>>> not with its final content. The version control command has to
>>> happen on every rootfs build and has to come "late", and a package
>>> would not have that property. Plus a package would become subject
>>> to potential false sharing in the multiconfig case ... The fields
>>> we change are mostly image-properties, and not generic  
>> Image post-processing is deemed to fail.
>>
>> Sorry, I don't understand what you mean with "a package would become
>> subject to potential false sharing in the multiconfig case". A
>> package created on build-time can contain those image properties,
>> right? I might be missing something...
> No a package can not do the job, unfortunately. That is why we violate
> the rule that every modification should be part of a package. It is not
> because that file is potentially already owned by another package, that
> can always be solved with overwriting in a package hook.

What do you mean with "package hook"? A package "postinst" script?

>
> The package would need to have a unique name and version for every new
> output of "git describe" and for every image description. In
> multiconfig we build multiple images in one tree, sharing isar-apt.

We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:

- /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything

- Debian os-release manpage still applies

If going for the other approach (create our own "base-files" package) we can do pretty much the same:

- the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged

- image post-processing to create the file becomes a requirement

  Silvano

>
> Henning
>
>>    Silvano
>>
>>> Henning
>>>  
>>>> In this case an update of the base-files package should leave it
>>>> alone or ask for a merge. And i think that would be OK behaviour.
>>>>
>>>> Henning
>>>>  
>>>>> regards,
>>>>> Claudius
>>>>>     

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Claudius Heine]

Hi Silvano,

On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
> 
> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
> 
> - Debian os-release manpage still applies
> 
> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
> 
> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
> 
> - image post-processing to create the file becomes a requirement
I looked at the manpage of os-release and it does not state that 
`/etc/os-release` can or should not be a file:

```
        The file /etc/os-release takes precedence over 
/usr/lib/os-release. Applications should check for the former, and 
exclusively use its data if it exists, and only fall back to 
/usr/lib/os-release if
        it is missing. Applications should not read data from both files 
at the same time.  /usr/lib/os-release is the recommended place to store 
OS release information as part of vendor trees.
        /etc/os-release should be a relative symlink to 
/usr/lib/os-release, to provide compatibility with applications only 
looking at /etc/. A relative symlink instead of an absolute symlink is 
necessary
        to avoid breaking the link in a chroot or initrd environment 
such as dracut.

        os-release contains data that is defined by the operating system 
vendor and should generally not be changed by the administrator.

        As this file only encodes names and identifiers it should not be 
localized.

        The /etc/os-release and /usr/lib/os-release files might be 
symlinks to other files, but it is important that the file is available 
from earliest boot on, and hence must be located on the root file
        system.
```

So why not make sure `/etc/os-release` is a file and do this:

     dpkg-divert --add --local --no-rename /etc/os-release

in the post processing step? Shouldn't this prevent any future 
base-files update from preventing to overwrite the `/etc/os-release`?

The only issue I would see if someone performs a distribution upgrade 
and now the `/etc/os-release` no longer contains the correct version, 
but other than an apt.conf hook that updates `/etc/os-release` based on 
new information from `/usr/lib/os-release` after an upgrade, I don't see 
a way to do this automatically.

I would rather divert '/etc/os-release' in a postprocessing script than 
`/usr/lib/os-release` because, I see the post-processing more of a local 
configuration.

For multiple reasons we cannot used a package to write the version 
information, so that is out of the picture anyway.

The only other way would be to create a new file just for the image or 
build version, but that would break compatibility now.

regards,
Claudius

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

Hi Claudius,

TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package

Nevertheless, more information inline.

On 22/01/2021 09:52, Claudius Heine wrote:
> Hi Silvano,
>
> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
>>
>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
>>
>> - Debian os-release manpage still applies
>>
>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
>>
>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
>>
>> - image post-processing to create the file becomes a requirement
> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:

That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:

- /etc/os-release symlink to /usr/lib/os-release

- /usr/lib/os-relase only (not /etc/os-release at all)

- /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)

- /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)

...

>
> ```
>        The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
>        it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
>        /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
>        to avoid breaking the link in a chroot or initrd environment such as dracut.
>
>        os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
>
>        As this file only encodes names and identifiers it should not be localized.
>
>        The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
>        system.
> ```
>
> So why not make sure `/etc/os-release` is a file and do this:
>
>     dpkg-divert --add --local --no-rename /etc/os-release
>
> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?

This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.

This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...

>
> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.

Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.

File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).

>
> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.

Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):

1. Keep originating Debian for reference? Then change only /etc/os-release.

2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.

/etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.

[1] http://0pointer.de/blog/projects/os-release

> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.
>
> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.
>
> regards,
> Claudius

  Silvano

-- 
Siemens AG, T RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Claudius Heine]

Hi Silvano

On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
> Hi Claudius,
> 
> TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package
> 
> Nevertheless, more information inline.
> 
> On 22/01/2021 09:52, Claudius Heine wrote:
>> Hi Silvano,
>>
>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
>>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
>>>
>>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
>>>
>>> - Debian os-release manpage still applies
>>>
>>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
>>>
>>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
>>>
>>> - image post-processing to create the file becomes a requirement
>> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:
> 
> That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:
> 
> - /etc/os-release symlink to /usr/lib/os-release
> 
> - /usr/lib/os-relase only (not /etc/os-release at all)
> 
> - /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
> 
> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)

Well I had this in mind:

   - /etc/os-release is a file containing the isar version, 
/usr/lib/os-release is a file containing the original debian version as 
deployed by base-files.

This could also be done similar to your fourth solution:

- /etc/os-release is a symlink to another file like 
/usr/lib/isar-release, original /usr/lib/os-release from base-files

The manual page states that /etc/os-release takes precedence over 
/usr/lib/os-release, so if /etc/os-release exists, it does not matter if 
/usr/lib/os-release does as well or what its content is.

> 
> ...
> 
>>
>> ```
>>         The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
>>         it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
>>         /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
>>         to avoid breaking the link in a chroot or initrd environment such as dracut.
>>
>>         os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
>>
>>         As this file only encodes names and identifiers it should not be localized.
>>
>>         The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
>>         system.
>> ```
>>
>> So why not make sure `/etc/os-release` is a file and do this:
>>
>>      dpkg-divert --add --local --no-rename /etc/os-release
>>
>> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?
> 
> This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.

Why would such a command go into a postinst hook? It contains '--local' 
therefore I would expect the administrator doing it him or herself. Or 
in Isars case a bitbake post installation process.

> 
> This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...

Simple example: Our /etc/os-release contains the build date and git 
commit id. If we put this information into a package, then it we could 
not cache it, because this information will change constantly. That also 
means all dependent tasks and recipes (image creation) cannot be cached, 
breaking the caching of a lot of stuff.

Calling 'bitbake isar-image' two times in short succession without any 
change, would cause 2 full builds of the whole image.

>> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.
> 
> Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.
> 
> File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).

You misunderstood me.

In my example a base-files package would still update 
/usr/lib/os-release to contain a new debian suite name, if you do a 
distribution update. If you want our diverted /etc/os-release to contain 
this new info automatically, you could do this via a apt.conf.d hook.

> 
>>
>> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.
> 
> Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):

I don't want to divert /usr/lib/os-release. Let it contain the upstream 
version for reference. This file does not matter, because 
/etc/os-release takes precedence over it.

> 
> 1. Keep originating Debian for reference? Then change only /etc/os-release.

Yes.

> 
> 2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.

No.

> 
> /etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.

Well its a separate discussion if Isar is in the role of a distribution 
vendor or of an administrator.

I would say its something in between.

> 
> [1] http://0pointer.de/blog/projects/os-release
> 
>> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
> As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.

This was why I proposed a apt.conf.d hook. Maybe we could generally do 
it like this.

Some package changes /usr/lib/os-release and a apt.conf.d hook would 
merge this content with ours into /etc/os-release.

>>
>> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
> Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.

While I agree that Isar in some configurations creates a Debian 
Derivative, currently it puts image version and build date into 
/etc/os-release. Which is strictly speaking not the right place for that.

It would be better to have something like /etc/base-image-version etc. 
for that.

regards,
Claudius

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

Hi Claudius,

On 22/01/2021 11:33, Claudius Heine wrote:
> Hi Silvano
>
> On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
>> Hi Claudius,
>>
>> TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package
>>
>> Nevertheless, more information inline.
>>
>> On 22/01/2021 09:52, Claudius Heine wrote:
>>> Hi Silvano,
>>>
>>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
>>>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
>>>>
>>>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
>>>>
>>>> - Debian os-release manpage still applies
>>>>
>>>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
>>>>
>>>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
>>>>
>>>> - image post-processing to create the file becomes a requirement
>>> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:
>>
>> That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:
>>
>> - /etc/os-release symlink to /usr/lib/os-release
>>
>> - /usr/lib/os-relase only (not /etc/os-release at all)
>>
>> - /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>>
>> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>
> Well I had this in mind:
>
>   - /etc/os-release is a file containing the isar version, /usr/lib/os-release is a file containing the original debian version as deployed by base-files.
>
> This could also be done similar to your fourth solution:
>
> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, original /usr/lib/os-release from base-files
>
> The manual page states that /etc/os-release takes precedence over /usr/lib/os-release, so if /etc/os-release exists, it does not matter if /usr/lib/os-release does as well or what its content is.
I also had foreseen that option as one of my favorites (either that or the first one), therefore that would be fine for me.
>
>>
>> ...
>>
>>>
>>> ```
>>>         The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
>>>         it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
>>>         /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
>>>         to avoid breaking the link in a chroot or initrd environment such as dracut.
>>>
>>>         os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
>>>
>>>         As this file only encodes names and identifiers it should not be localized.
>>>
>>>         The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
>>>         system.
>>> ```
>>>
>>> So why not make sure `/etc/os-release` is a file and do this:
>>>
>>>      dpkg-divert --add --local --no-rename /etc/os-release
>>>
>>> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?
>>
>> This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.
>
> Why would such a command go into a postinst hook? It contains '--local' therefore I would expect the administrator doing it him or herself. Or in Isars case a bitbake post installation process.
Sorry, forgot that we cannot use a package. A bitbake post installation process is what I had in mind. I don't care in this mailing list about admins "doing stuff" that might break their systems ;-)
>
>>
>> This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...
>
> Simple example: Our /etc/os-release contains the build date and git commit id. If we put this information into a package, then it we could not cache it, because this information will change constantly. That also means all dependent tasks and recipes (image creation) cannot be cached, breaking the caching of a lot of stuff.
>
> Calling 'bitbake isar-image' two times in short succession without any change, would cause 2 full builds of the whole image.
The current image post-processing is happening pretty much in the end and not breaking caching, right? Wouldn't it be possible to modify/extend that recipe to create a Debian package and install it chrooting to the root filesystem?
>
>>> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.
>>
>> Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.
>>
>> File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).
>
> You misunderstood me.
>
> In my example a base-files package would still update /usr/lib/os-release to contain a new debian suite name, if you do a distribution update. If you want our diverted /etc/os-release to contain this new info automatically, you could do this via a apt.conf.d hook.
Diverting only /etc/os-release you get /usr/lib/os-release managed (and therefore updated) by base-files and ISAR takes care of /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR creates it and adds it to the root filesystem relying on neither the user nor dpkg modifying it.
>
>>
>>>
>>> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.
>>
>> Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):
>
> I don't want to divert /usr/lib/os-release. Let it contain the upstream version for reference. This file does not matter, because /etc/os-release takes precedence over it.
Hopefully it's clear from my comments above, that I find this approach fine.
>
>>
>> 1. Keep originating Debian for reference? Then change only /etc/os-release.
>
> Yes.
>
>>
>> 2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.
>
> No.
>
>>
>> /etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.
>
> Well its a separate discussion if Isar is in the role of a distribution vendor or of an administrator.

Is this a philosophical discussion? :-) Am I making a fuss?

I might have a different understanding for "admin", but if not distribution builder, I'd say installation builder, but not administrator.

IMO ISAR is a distribution builder and as such a tool for distribution vendors. As a Debian user I find at home with many ISAR images, but not at all with some others.

A distribution according this definition "A Linux distribution may also be described as a particular assortment of application and utility software [...], packaged together with the Linux kernel in such a way that its capabilities meet the needs of many users.", taken from https://en.wikipedia.org/wiki/Linux_distribution

>
> I would say its something in between.
>
>>
>> [1] https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&reserved=0
>>
>>> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
>> As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.
>
> This was why I proposed a apt.conf.d hook. Maybe we could generally do it like this.
No matter how you divert that file (bitbake post processing, apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files packages appear whenever you have a file divert. Unless that vendor-provided base-files package modifies the file diversion being applied by ISAR (I don't know if it's possible) to become the owner of the file.
>
> Some package changes /usr/lib/os-release and a apt.conf.d hook would merge this content with ours into /etc/os-release.
I still don't get the need for such "black magic".
>
>>>
>>> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
>> Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.
>
> While I agree that Isar in some configurations creates a Debian Derivative, currently it puts image version and build date into /etc/os-release. Which is strictly speaking not the right place for that.
If the image version and build date are relevant to have a clear identification of the OS version, then I disagree. If that's informative and not needed to have a unique identification of the OS version, then I agree that it could be somewhere else (although I still think it's not a wrong place for it).
>
> It would be better to have something like /etc/base-image-version etc. for that.
>
> regards,
> Claudius
  Silvano

-- 
Siemens AG, T RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijaikumar....@gmail.com]

[-- Attachment #1.1: Type: text/plain, Size: 14775 bytes --]

Hi All,

On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano Cuesta 
wrote:

> Hi Claudius, 
>
> On 22/01/2021 11:33, Claudius Heine wrote: 
> > Hi Silvano 
> > 
> > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote: 
> >> Hi Claudius, 
> >> 
> >> TL;DR: you're proposing here what appears to be a feasible solution to 
> this issue without using a package 
> >> 
> >> Nevertheless, more information inline. 
> >> 
> >> On 22/01/2021 09:52, Claudius Heine wrote: 
> >>> Hi Silvano, 
> >>> 
> >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote: 
> >>>> We can create a removal file diversion for /usr/lib/os-release 
> (leaving /etc/os-release untouched, that way the Debian manpage for 
> os-release still applies). This approach has following attributes: 
> >>>> 
> >>>> - /usr/lib/os-release is not managed by any package and that way 
> post-processing it is not an issue and dpkg won't break anything 
> >>>> 
> >>>> - Debian os-release manpage still applies 
> >>>> 
> >>>> If going for the other approach (create our own "base-files" package) 
> we can do pretty much the same: 
> >>>> 
> >>>> - the package doesn't provide /usr/lib/os-release, leaving it 
> package-unmanaged 
> >>>> 
> >>>> - image post-processing to create the file becomes a requirement 
> >>> I looked at the manpage of os-release and it does not state that 
> `/etc/os-release` can or should not be a file: 
> >> 
> >> That's probably a misunderstanding. I don't mean that we have to create 
> "a file", but provide the os-release information according the 
> specification. There are multiple different valid combinations: 
> >> 
> >> - /etc/os-release symlink to /usr/lib/os-release 
> >> 
> >> - /usr/lib/os-relase only (not /etc/os-release at all) 
> >> 
> >> - /etc/os-release only, no /usr/lib/os-release at all (it's not the 
> recommended way, but it isn't forbidden either) 
> >> 
> >> - /etc/os-release is a symlink to another file like 
> /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the 
> recommended way, but it isn't forbidden either) 
> > 
> > Well I had this in mind: 
> > 
> >   - /etc/os-release is a file containing the isar version, 
> /usr/lib/os-release is a file containing the original debian version as 
> deployed by base-files. 
> > 
> > This could also be done similar to your fourth solution: 
> > 
> > - /etc/os-release is a symlink to another file like 
> /usr/lib/isar-release, original /usr/lib/os-release from base-files 
> > 
> > The manual page states that /etc/os-release takes precedence over 
> /usr/lib/os-release, so if /etc/os-release exists, it does not matter if 
> /usr/lib/os-release does as well or what its content is. 
> I also had foreseen that option as one of my favorites (either that or the 
> first one), therefore that would be fine for me. 
> > 
> >> 
> >> ... 
> >> 
> >>> 
> >>> ``` 
> >>>         The file /etc/os-release takes precedence over 
> /usr/lib/os-release. Applications should check for the former, and 
> exclusively use its data if it exists, and only fall back to 
> /usr/lib/os-release if 
> >>>         it is missing. Applications should not read data from both 
> files at the same time.  /usr/lib/os-release is the recommended place to 
> store OS release information as part of vendor trees. 
> >>>         /etc/os-release should be a relative symlink to 
> /usr/lib/os-release, to provide compatibility with applications only 
> looking at /etc/. A relative symlink instead of an absolute symlink is 
> necessary 
> >>>         to avoid breaking the link in a chroot or initrd environment 
> such as dracut. 
> >>> 
> >>>         os-release contains data that is defined by the operating 
> system vendor and should generally not be changed by the administrator. 
> >>> 
> >>>         As this file only encodes names and identifiers it should not 
> be localized. 
> >>> 
> >>>         The /etc/os-release and /usr/lib/os-release files might be 
> symlinks to other files, but it is important that the file is available 
> from earliest boot on, and hence must be located on the root file 
> >>>         system. 
> >>> ``` 
> >>> 
> >>> So why not make sure `/etc/os-release` is a file and do this: 
> >>> 
> >>>      dpkg-divert --add --local --no-rename /etc/os-release 
> >>> 
> >>> in the post processing step? Shouldn't this prevent any future 
> base-files update from preventing to overwrite the `/etc/os-release`? 
> >> 
> >> This command would so to say command dpkg not to touch /etc/os-release 
> from the point it's executed on. That way you can have a local version of 
> /etc/os-release and be sure that no package updates is modifying/replacing 
> it. Such a command would go into the postinst hook and another postrm hook 
> would be required to remove the file diversion for /etc/os-release and 
> restoring the original one. 
> > 
> > Why would such a command go into a postinst hook? It contains '--local' 
> therefore I would expect the administrator doing it him or herself. Or in 
> Isars case a bitbake post installation process. 
> Sorry, forgot that we cannot use a package. A bitbake post installation 
> process is what I had in mind. I don't care in this mailing list about 
> admins "doing stuff" that might break their systems ;-) 
> > 
> >> 
> >> This would technically resolve the issue, but looks a bit weird to me. 
> Debian and derivatives get that file managed by a package. But since I 
> haven't understood what the issue mentioned by Henning regarding 
> multiconfig and packages, this might be the only feasible solution... 
> > 
> > Simple example: Our /etc/os-release contains the build date and git 
> commit id. If we put this information into a package, then it we could not 
> cache it, because this information will change constantly. That also means 
> all dependent tasks and recipes (image creation) cannot be cached, breaking 
> the caching of a lot of stuff. 
> > 
> > Calling 'bitbake isar-image' two times in short succession without any 
> change, would cause 2 full builds of the whole image. 
> The current image post-processing is happening pretty much in the end and 
> not breaking caching, right? Wouldn't it be possible to modify/extend that 
> recipe to create a Debian package and install it chrooting to the root 
> filesystem? 
> > 
> >>> The only issue I would see if someone performs a distribution upgrade 
> and now the `/etc/os-release` no longer contains the correct version, but 
> other than an apt.conf hook that updates `/etc/os-release` based on new 
> information from `/usr/lib/os-release` after an upgrade, I don't see a way 
> to do this automatically. 
> >> 
> >> Why should a distribution upgrade change that? I'm not aware that 
> dist-upgrade changes file diversions... To my knowledge, once the file 
> diversion has been created, dpkg should completely ignore that file. 
> >> 
> >> File diversions don't apply to packages, but to files, therefore I'd 
> expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of 
> the packages that are "blocked" from touching a file. It keeps track of the 
> file path and of the package that requested the file diversion (unless 
> "--local" specified, as you propose). 
> > 
> > You misunderstood me. 
> > 
> > In my example a base-files package would still update 
> /usr/lib/os-release to contain a new debian suite name, if you do a 
> distribution update. If you want our diverted /etc/os-release to contain 
> this new info automatically, you could do this via a apt.conf.d hook. 
> Diverting only /etc/os-release you get /usr/lib/os-release managed (and 
> therefore updated) by base-files and ISAR takes care of /etc/os-release. I 
> don't understand the need for an apt.conf.d hook... ISAR creates it and 
> adds it to the root filesystem relying on neither the user nor dpkg 
> modifying it. 
> > 
> >> 
> >>> 
> >>> I would rather divert '/etc/os-release' in a postprocessing script 
> than `/usr/lib/os-release` because, I see the post-processing more of a 
> local configuration. 
> >> 
> >> Diverting /etc/os-release or /usr/lib/os-release depends on what we 
> want to achieve (it would be even better letting the ISAR user decide it): 
> > 
> > I don't want to divert /usr/lib/os-release. Let it contain the upstream 
> version for reference. This file does not matter, because /etc/os-release 
> takes precedence over it. 
> Hopefully it's clear from my comments above, that I find this approach 
> fine. 
> > 
> >> 
> >> 1. Keep originating Debian for reference? Then change only 
> /etc/os-release. 
> > 
> > Yes. 
> > 
> >> 
> >> 2. Wipe away all information about originating Debian? Then change 
> /usr/lib/os-release. 
> > 
> > No. 
> > 
> >> 
> >> /etc/os-release might appear like a local configuration for the ISAR 
> builder, but IMO it's intrinsic information about the distribution being 
> built with ISAR. As the documentation that you are quoting says "defined by 
> the operating system vendor and should generally not be changed by the 
> administrator." Therefore IMO the path /etc/os-release was kept for 
> historical reasons (see footnote 1 of the announcement [1]), but it's an 
> exception to the convention that files under "/etc" are configurations and 
> can be changed. 
> > 
> > Well its a separate discussion if Isar is in the role of a distribution 
> vendor or of an administrator. 
>
> Is this a philosophical discussion? :-) Am I making a fuss? 
>
> I might have a different understanding for "admin", but if not 
> distribution builder, I'd say installation builder, but not administrator. 
>
> IMO ISAR is a distribution builder and as such a tool for distribution 
> vendors. As a Debian user I find at home with many ISAR images, but not at 
> all with some others. 
>
> A distribution according this definition "A Linux distribution may also be 
> described as a particular assortment of application and utility software 
> [...], packaged together with the Linux kernel in such a way that its 
> capabilities meet the needs of many users.", taken from 
> https://en.wikipedia.org/wiki/Linux_distribution 
>
> > 
> > I would say its something in between. 
> > 
> >> 
> >> [1] 
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&reserved=0 
> >> 
> >>> For multiple reasons we cannot used a package to write the version 
> information, so that is out of the picture anyway. 
> >> As said above, if for the reasons you and Henning mention using a 
> package is out of the picture, this is probably the only feasible 
> solution... Except for those people who wrote their own base-files package 
> to resolve this issue, since their package won't work anymore :-) They 
> would have to manage this file using hooks instead of directly declaring 
> the file as managed by their package. 
> > 
> > This was why I proposed a apt.conf.d hook. Maybe we could generally do 
> it like this. 
> No matter how you divert that file (bitbake post processing, apt.conf.d 
> hook,...) the mentioned conflict with vendor-provided base-files packages 
> appear whenever you have a file divert. Unless that vendor-provided 
> base-files package modifies the file diversion being applied by ISAR (I 
> don't know if it's possible) to become the owner of the file. 
> > 
> > Some package changes /usr/lib/os-release and a apt.conf.d hook would 
> merge this content with ours into /etc/os-release. 
> I still don't get the need for such "black magic". 
> > 
> >>> 
> >>> The only other way would be to create a new file just for the image or 
> build version, but that would break compatibility now. 
> >> Do you mean using a completely different file? Something like 
> /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I 
> wouldn't do it. Not only because of the compatibility issue that you 
> mention, but also because ISAR is de-facto creating a Debian Derivative 
> that information belongs logically into /etc/os-release. Someone finding 
> vanilla Debian inputs in /etc/os-release would expect a vanilla Debian 
> installation, not an ISAR-built distro. 
> > 
> > While I agree that Isar in some configurations creates a Debian 
> Derivative, currently it puts image version and build date into 
> /etc/os-release. Which is strictly speaking not the right place for that. 
> If the image version and build date are relevant to have a clear 
> identification of the OS version, then I disagree. If that's informative 
> and not needed to have a unique identification of the OS version, then I 
> agree that it could be somewhere else (although I still think it's not a 
> wrong place for it). 
> > 
> > It would be better to have something like /etc/base-image-version etc. 
> for that. 
> > 
>

After going through this, I feel that we could move the ISAR system details 
to a separate file like /etc/isar_{version/base/info}.
ISAR, as I see could be used to create a custom distribution or in a simple 
case just the functions of what an installer would do in other
distribution(bootstrap and prepare a simple bootable image for(*and) 
install).

If we see the minimal case, it does not make sense to have the ISAR info in 
/etc/os-release. And probably we should leave the /etc/os-release
alone; at least in the main ISAR repo and leave the custom configuration 
part to the users. In this way, we are not enforcing any uncommon
behavior (dpkg-divert+hook) that a new user least expects in ISAR in its 
vanilla form. This leaves more freedom to the downstream users to
choose a method(custom base-files / dpkg-divert / any unofficial way) that 
best fits their need based on what and where they need the information.

If you have built an image with ISAR, you could find some details about the 
ISAR system used from the /etc/isar_{version/base/info}. If you want to
add more information to it, well you are free to do so. If you feel the 
information is crucial, that it needs to be present in /etc/os-release, and
willing to maintain it for your custom distro, well you could do it too. If 
you feel it is unnecessary, well you could remove it too.

And then comes the discussion of backward compatibility. Probably implement 
and maintain dpkg-divert approach and provide a way for existing users to 
switch to them using a flag/feature.

Thanks,
Vijai Kumar K 

> regards, 
> > Claudius 
>   Silvano 
>
> -- 
> Siemens AG, T RDA IOT SES-DE 
> Corporate Competence Center Embedded Linux 
>
>

[-- Attachment #1.2: Type: text/html, Size: 17893 bytes --]

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

Hi Vijai,

I like your proposal.

If we need to provide "something" that keeps modifying the /etc/os-release for backwards compatibility, do we want to keep the current "broken" implementation?

If we want to "fix" that implementation, then we can do it in a way that not only provides that backwards compatibility, but also to provides a useful feature. The default would be not touching /etc/os-release and one recipe could provide (when requested with a flag) a package for "managing" /etc/os-release (assuming it can be somehow accomplished with multiconfigs, or be a mutually exclusive feature).

Ciao,

  Silvano

On 05/02/2021 12:55, vijaikumar....@gmail.com wrote:
> Hi All,
>
> On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano Cuesta wrote:
>
>     Hi Claudius,
>
>     On 22/01/2021 11:33, Claudius Heine wrote:
>     > Hi Silvano
>     >
>     > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
>     >> Hi Claudius,
>     >>
>     >> TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package
>     >>
>     >> Nevertheless, more information inline.
>     >>
>     >> On 22/01/2021 09:52, Claudius Heine wrote:
>     >>> Hi Silvano,
>     >>>
>     >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
>     >>>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
>     >>>>
>     >>>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
>     >>>>
>     >>>> - Debian os-release manpage still applies
>     >>>>
>     >>>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
>     >>>>
>     >>>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
>     >>>>
>     >>>> - image post-processing to create the file becomes a requirement
>     >>> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:
>     >>
>     >> That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:
>     >>
>     >> - /etc/os-release symlink to /usr/lib/os-release
>     >>
>     >> - /usr/lib/os-relase only (not /etc/os-release at all)
>     >>
>     >> - /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>     >>
>     >> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>     >
>     > Well I had this in mind:
>     >
>     >   - /etc/os-release is a file containing the isar version, /usr/lib/os-release is a file containing the original debian version as deployed by base-files.
>     >
>     > This could also be done similar to your fourth solution:
>     >
>     > - /etc/os-release is a symlink to another file like /usr/lib/isar-release, original /usr/lib/os-release from base-files
>     >
>     > The manual page states that /etc/os-release takes precedence over /usr/lib/os-release, so if /etc/os-release exists, it does not matter if /usr/lib/os-release does as well or what its content is.
>     I also had foreseen that option as one of my favorites (either that or the first one), therefore that would be fine for me.
>     >
>     >>
>     >> ...
>     >>
>     >>>
>     >>> ```
>     >>>         The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
>     >>>         it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
>     >>>         /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
>     >>>         to avoid breaking the link in a chroot or initrd environment such as dracut.
>     >>>
>     >>>         os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
>     >>>
>     >>>         As this file only encodes names and identifiers it should not be localized.
>     >>>
>     >>>         The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
>     >>>         system.
>     >>> ```
>     >>>
>     >>> So why not make sure `/etc/os-release` is a file and do this:
>     >>>
>     >>>      dpkg-divert --add --local --no-rename /etc/os-release
>     >>>
>     >>> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?
>     >>
>     >> This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.
>     >
>     > Why would such a command go into a postinst hook? It contains '--local' therefore I would expect the administrator doing it him or herself. Or in Isars case a bitbake post installation process.
>     Sorry, forgot that we cannot use a package. A bitbake post installation process is what I had in mind. I don't care in this mailing list about admins "doing stuff" that might break their systems ;-)
>     >
>     >>
>     >> This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...
>     >
>     > Simple example: Our /etc/os-release contains the build date and git commit id. If we put this information into a package, then it we could not cache it, because this information will change constantly. That also means all dependent tasks and recipes (image creation) cannot be cached, breaking the caching of a lot of stuff.
>     >
>     > Calling 'bitbake isar-image' two times in short succession without any change, would cause 2 full builds of the whole image.
>     The current image post-processing is happening pretty much in the end and not breaking caching, right? Wouldn't it be possible to modify/extend that recipe to create a Debian package and install it chrooting to the root filesystem?
>     >
>     >>> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.
>     >>
>     >> Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.
>     >>
>     >> File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).
>     >
>     > You misunderstood me.
>     >
>     > In my example a base-files package would still update /usr/lib/os-release to contain a new debian suite name, if you do a distribution update. If you want our diverted /etc/os-release to contain this new info automatically, you could do this via a apt.conf.d hook.
>     Diverting only /etc/os-release you get /usr/lib/os-release managed (and therefore updated) by base-files and ISAR takes care of /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR creates it and adds it to the root filesystem relying on neither the user nor dpkg modifying it.
>     >
>     >>
>     >>>
>     >>> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.
>     >>
>     >> Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):
>     >
>     > I don't want to divert /usr/lib/os-release. Let it contain the upstream version for reference. This file does not matter, because /etc/os-release takes precedence over it.
>     Hopefully it's clear from my comments above, that I find this approach fine.
>     >
>     >>
>     >> 1. Keep originating Debian for reference? Then change only /etc/os-release.
>     >
>     > Yes.
>     >
>     >>
>     >> 2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.
>     >
>     > No.
>     >
>     >>
>     >> /etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.
>     >
>     > Well its a separate discussion if Isar is in the role of a distribution vendor or of an administrator.
>
>     Is this a philosophical discussion? :-) Am I making a fuss?
>
>     I might have a different understanding for "admin", but if not distribution builder, I'd say installation builder, but not administrator.
>
>     IMO ISAR is a distribution builder and as such a tool for distribution vendors. As a Debian user I find at home with many ISAR images, but not at all with some others.
>
>     A distribution according this definition "A Linux distribution may also be described as a particular assortment of application and utility software [...], packaged together with the Linux kernel in such a way that its capabilities meet the needs of many users.", taken from https://en.wikipedia.org/wiki/Linux_distribution <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229432997922%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b7fZq0akmNohTagcefD81VInLbBkAv%2FcjQTLZNKqIjc%3D&reserved=0>
>
>     >
>     > I would say its something in between.
>     >
>     >>
>     >> [1] https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&amp;reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hhqb%2FyU4z%2BqBlh4Yz2ocfxsQTaRtQotyAEJ3dG64k24%3D&reserved=0>
>     >>
>     >>> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
>     >> As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.
>     >
>     > This was why I proposed a apt.conf.d hook. Maybe we could generally do it like this.
>     No matter how you divert that file (bitbake post processing, apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files packages appear whenever you have a file divert. Unless that vendor-provided base-files package modifies the file diversion being applied by ISAR (I don't know if it's possible) to become the owner of the file.
>     >
>     > Some package changes /usr/lib/os-release and a apt.conf.d hook would merge this content with ours into /etc/os-release.
>     I still don't get the need for such "black magic".
>     >
>     >>>
>     >>> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
>     >> Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.
>     >
>     > While I agree that Isar in some configurations creates a Debian Derivative, currently it puts image version and build date into /etc/os-release. Which is strictly speaking not the right place for that.
>     If the image version and build date are relevant to have a clear identification of the OS version, then I disagree. If that's informative and not needed to have a unique identification of the OS version, then I agree that it could be somewhere else (although I still think it's not a wrong place for it).
>     >
>     > It would be better to have something like /etc/base-image-version etc. for that.
>     >
>
>
> After going through this, I feel that we could move the ISAR system details to a separate file like /etc/isar_{version/base/info}.
> ISAR, as I see could be used to create a custom distribution or in a simple case just the functions of what an installer would do in other
> distribution(bootstrap and prepare a simple bootable image for(*and) install).
>
> If we see the minimal case, it does not make sense to have the ISAR info in /etc/os-release. And probably we should leave the /etc/os-release
> alone; at least in the main ISAR repo and leave the custom configuration part to the users. In this way, we are not enforcing any uncommon
> behavior (dpkg-divert+hook) that a new user least expects in ISAR in its vanilla form. This leaves more freedom to the downstream users to
> choose a method(custom base-files / dpkg-divert / any unofficial way) that best fits their need based on what and where they need the information.
>
> If you have built an image with ISAR, you could find some details about the ISAR system used from the /etc/isar_{version/base/info}. If you want to
> add more information to it, well you are free to do so. If you feel the information is crucial, that it needs to be present in /etc/os-release, and
> willing to maintain it for your custom distro, well you could do it too. If you feel it is unnecessary, well you could remove it too.
>
> And then comes the discussion of backward compatibility. Probably implement and maintain dpkg-divert approach and provide a way for existing users to switch to them using a flag/feature.
>
> Thanks,
> Vijai Kumar K 
>
>     > regards,
>     > Claudius
>       Silvano
>
>     -- 
>     Siemens AG, T RDA IOT SES-DE
>     Corporate Competence Center Embedded Linux
>
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com <mailto:isar-users+unsubscribe@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%40googlegroups.com <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PKO1KmREqy6GRr20lbuZ%2FsxeINqLIB86%2Fgfd4QeKlQo%3D&reserved=0>.

Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijai kumar]

[-- Attachment #1: Type: text/plain, Size: 19493 bytes --]

On Fri, Feb 5, 2021 at 8:27 PM Silvano Cirujano Cuesta <
silvano.cirujano-cuesta@siemens.com> wrote:

> Hi Vijai,
>
> I like your proposal.
>
> If we need to provide "something" that keeps modifying the /etc/os-release
> for backwards compatibility, do we want to keep the current "broken"
> implementation?
>

I would not suggest we keep the current broken implementation. We should
fix it.


>
> If we want to "fix" that implementation, then we can do it in a way that
> not only provides that backwards compatibility, but also to provides a
> useful feature. The default would be not touching /etc/os-release and one
> recipe could provide (when requested with a flag) a package for "managing"
> /etc/os-release (assuming it can be somehow accomplished with multiconfigs,
> or be a mutually exclusive feature).
>

As Claudius previously mentioned, this might break caching. If the
information is purely build-system related, should we try to put it in a
package? Does this file need updates in the future? Well, from the current
standpoint, I don't see it needing updates. The data is only applicable
when we build using ISAR. If hypothetically, we decided to recreate the
distro with a cached apt-repo generated with ISAR using some other tool,
this info is not relevant.

That said, we need to understand from the community, what are the real-life
scenarios they are using this ISAR information in /etc/os-release for? I am
not personally using that info. If it is being used somewhere, we need to
know why? So that we know how costly it is to break backward compatibility.
Huge or none at all?

Thanks,
VIjai Kumar K


>
> Ciao,
>
>   Silvano
>
> On 05/02/2021 12:55, vijaikumar....@gmail.com wrote:
> > Hi All,
> >
> > On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano
> Cuesta wrote:
> >
> >     Hi Claudius,
> >
> >     On 22/01/2021 11:33, Claudius Heine wrote:
> >     > Hi Silvano
> >     >
> >     > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
> >     >> Hi Claudius,
> >     >>
> >     >> TL;DR: you're proposing here what appears to be a feasible
> solution to this issue without using a package
> >     >>
> >     >> Nevertheless, more information inline.
> >     >>
> >     >> On 22/01/2021 09:52, Claudius Heine wrote:
> >     >>> Hi Silvano,
> >     >>>
> >     >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
> >     >>>> We can create a removal file diversion for /usr/lib/os-release
> (leaving /etc/os-release untouched, that way the Debian manpage for
> os-release still applies). This approach has following attributes:
> >     >>>>
> >     >>>> - /usr/lib/os-release is not managed by any package and that
> way post-processing it is not an issue and dpkg won't break anything
> >     >>>>
> >     >>>> - Debian os-release manpage still applies
> >     >>>>
> >     >>>> If going for the other approach (create our own "base-files"
> package) we can do pretty much the same:
> >     >>>>
> >     >>>> - the package doesn't provide /usr/lib/os-release, leaving it
> package-unmanaged
> >     >>>>
> >     >>>> - image post-processing to create the file becomes a requirement
> >     >>> I looked at the manpage of os-release and it does not state that
> `/etc/os-release` can or should not be a file:
> >     >>
> >     >> That's probably a misunderstanding. I don't mean that we have to
> create "a file", but provide the os-release information according the
> specification. There are multiple different valid combinations:
> >     >>
> >     >> - /etc/os-release symlink to /usr/lib/os-release
> >     >>
> >     >> - /usr/lib/os-relase only (not /etc/os-release at all)
> >     >>
> >     >> - /etc/os-release only, no /usr/lib/os-release at all (it's not
> the recommended way, but it isn't forbidden either)
> >     >>
> >     >> - /etc/os-release is a symlink to another file like
> /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the
> recommended way, but it isn't forbidden either)
> >     >
> >     > Well I had this in mind:
> >     >
> >     >   - /etc/os-release is a file containing the isar version,
> /usr/lib/os-release is a file containing the original debian version as
> deployed by base-files.
> >     >
> >     > This could also be done similar to your fourth solution:
> >     >
> >     > - /etc/os-release is a symlink to another file like
> /usr/lib/isar-release, original /usr/lib/os-release from base-files
> >     >
> >     > The manual page states that /etc/os-release takes precedence over
> /usr/lib/os-release, so if /etc/os-release exists, it does not matter if
> /usr/lib/os-release does as well or what its content is.
> >     I also had foreseen that option as one of my favorites (either that
> or the first one), therefore that would be fine for me.
> >     >
> >     >>
> >     >> ...
> >     >>
> >     >>>
> >     >>> ```
> >     >>>         The file /etc/os-release takes precedence over
> /usr/lib/os-release. Applications should check for the former, and
> exclusively use its data if it exists, and only fall back to
> /usr/lib/os-release if
> >     >>>         it is missing. Applications should not read data from
> both files at the same time.  /usr/lib/os-release is the recommended place
> to store OS release information as part of vendor trees.
> >     >>>         /etc/os-release should be a relative symlink to
> /usr/lib/os-release, to provide compatibility with applications only
> looking at /etc/. A relative symlink instead of an absolute symlink is
> necessary
> >     >>>         to avoid breaking the link in a chroot or initrd
> environment such as dracut.
> >     >>>
> >     >>>         os-release contains data that is defined by the
> operating system vendor and should generally not be changed by the
> administrator.
> >     >>>
> >     >>>         As this file only encodes names and identifiers it
> should not be localized.
> >     >>>
> >     >>>         The /etc/os-release and /usr/lib/os-release files might
> be symlinks to other files, but it is important that the file is available
> from earliest boot on, and hence must be located on the root file
> >     >>>         system.
> >     >>> ```
> >     >>>
> >     >>> So why not make sure `/etc/os-release` is a file and do this:
> >     >>>
> >     >>>      dpkg-divert --add --local --no-rename /etc/os-release
> >     >>>
> >     >>> in the post processing step? Shouldn't this prevent any future
> base-files update from preventing to overwrite the `/etc/os-release`?
> >     >>
> >     >> This command would so to say command dpkg not to touch
> /etc/os-release from the point it's executed on. That way you can have a
> local version of /etc/os-release and be sure that no package updates is
> modifying/replacing it. Such a command would go into the postinst hook and
> another postrm hook would be required to remove the file diversion for
> /etc/os-release and restoring the original one.
> >     >
> >     > Why would such a command go into a postinst hook? It contains
> '--local' therefore I would expect the administrator doing it him or
> herself. Or in Isars case a bitbake post installation process.
> >     Sorry, forgot that we cannot use a package. A bitbake post
> installation process is what I had in mind. I don't care in this mailing
> list about admins "doing stuff" that might break their systems ;-)
> >     >
> >     >>
> >     >> This would technically resolve the issue, but looks a bit weird
> to me. Debian and derivatives get that file managed by a package. But since
> I haven't understood what the issue mentioned by Henning regarding
> multiconfig and packages, this might be the only feasible solution...
> >     >
> >     > Simple example: Our /etc/os-release contains the build date and
> git commit id. If we put this information into a package, then it we could
> not cache it, because this information will change constantly. That also
> means all dependent tasks and recipes (image creation) cannot be cached,
> breaking the caching of a lot of stuff.
> >     >
> >     > Calling 'bitbake isar-image' two times in short succession without
> any change, would cause 2 full builds of the whole image.
> >     The current image post-processing is happening pretty much in the
> end and not breaking caching, right? Wouldn't it be possible to
> modify/extend that recipe to create a Debian package and install it
> chrooting to the root filesystem?
> >     >
> >     >>> The only issue I would see if someone performs a distribution
> upgrade and now the `/etc/os-release` no longer contains the correct
> version, but other than an apt.conf hook that updates `/etc/os-release`
> based on new information from `/usr/lib/os-release` after an upgrade, I
> don't see a way to do this automatically.
> >     >>
> >     >> Why should a distribution upgrade change that? I'm not aware that
> dist-upgrade changes file diversions... To my knowledge, once the file
> diversion has been created, dpkg should completely ignore that file.
> >     >>
> >     >> File diversions don't apply to packages, but to files, therefore
> I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track
> of the packages that are "blocked" from touching a file. It keeps track of
> the file path and of the package that requested the file diversion (unless
> "--local" specified, as you propose).
> >     >
> >     > You misunderstood me.
> >     >
> >     > In my example a base-files package would still update
> /usr/lib/os-release to contain a new debian suite name, if you do a
> distribution update. If you want our diverted /etc/os-release to contain
> this new info automatically, you could do this via a apt.conf.d hook.
> >     Diverting only /etc/os-release you get /usr/lib/os-release managed
> (and therefore updated) by base-files and ISAR takes care of
> /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR
> creates it and adds it to the root filesystem relying on neither the user
> nor dpkg modifying it.
> >     >
> >     >>
> >     >>>
> >     >>> I would rather divert '/etc/os-release' in a postprocessing
> script than `/usr/lib/os-release` because, I see the post-processing more
> of a local configuration.
> >     >>
> >     >> Diverting /etc/os-release or /usr/lib/os-release depends on what
> we want to achieve (it would be even better letting the ISAR user decide
> it):
> >     >
> >     > I don't want to divert /usr/lib/os-release. Let it contain the
> upstream version for reference. This file does not matter, because
> /etc/os-release takes precedence over it.
> >     Hopefully it's clear from my comments above, that I find this
> approach fine.
> >     >
> >     >>
> >     >> 1. Keep originating Debian for reference? Then change only
> /etc/os-release.
> >     >
> >     > Yes.
> >     >
> >     >>
> >     >> 2. Wipe away all information about originating Debian? Then
> change /usr/lib/os-release.
> >     >
> >     > No.
> >     >
> >     >>
> >     >> /etc/os-release might appear like a local configuration for the
> ISAR builder, but IMO it's intrinsic information about the distribution
> being built with ISAR. As the documentation that you are quoting says
> "defined by the operating system vendor and should generally not be changed
> by the administrator." Therefore IMO the path /etc/os-release was kept for
> historical reasons (see footnote 1 of the announcement [1]), but it's an
> exception to the convention that files under "/etc" are configurations and
> can be changed.
> >     >
> >     > Well its a separate discussion if Isar is in the role of a
> distribution vendor or of an administrator.
> >
> >     Is this a philosophical discussion? :-) Am I making a fuss?
> >
> >     I might have a different understanding for "admin", but if not
> distribution builder, I'd say installation builder, but not administrator.
> >
> >     IMO ISAR is a distribution builder and as such a tool for
> distribution vendors. As a Debian user I find at home with many ISAR
> images, but not at all with some others.
> >
> >     A distribution according this definition "A Linux distribution may
> also be described as a particular assortment of application and utility
> software [...], packaged together with the Linux kernel in such a way that
> its capabilities meet the needs of many users.", taken from
> https://en.wikipedia.org/wiki/Linux_distribution <
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229432997922%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b7fZq0akmNohTagcefD81VInLbBkAv%2FcjQTLZNKqIjc%3D&reserved=0
> >
> >
> >     >
> >     > I would say its something in between.
> >     >
> >     >>
> >     >> [1]
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&amp;reserved=0
> <
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hhqb%2FyU4z%2BqBlh4Yz2ocfxsQTaRtQotyAEJ3dG64k24%3D&reserved=0
> >
> >     >>
> >     >>> For multiple reasons we cannot used a package to write the
> version information, so that is out of the picture anyway.
> >     >> As said above, if for the reasons you and Henning mention using a
> package is out of the picture, this is probably the only feasible
> solution... Except for those people who wrote their own base-files package
> to resolve this issue, since their package won't work anymore :-) They
> would have to manage this file using hooks instead of directly declaring
> the file as managed by their package.
> >     >
> >     > This was why I proposed a apt.conf.d hook. Maybe we could
> generally do it like this.
> >     No matter how you divert that file (bitbake post processing,
> apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files
> packages appear whenever you have a file divert. Unless that
> vendor-provided base-files package modifies the file diversion being
> applied by ISAR (I don't know if it's possible) to become the owner of the
> file.
> >     >
> >     > Some package changes /usr/lib/os-release and a apt.conf.d hook
> would merge this content with ours into /etc/os-release.
> >     I still don't get the need for such "black magic".
> >     >
> >     >>>
> >     >>> The only other way would be to create a new file just for the
> image or build version, but that would break compatibility now.
> >     >> Do you mean using a completely different file? Something like
> /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I
> wouldn't do it. Not only because of the compatibility issue that you
> mention, but also because ISAR is de-facto creating a Debian Derivative
> that information belongs logically into /etc/os-release. Someone finding
> vanilla Debian inputs in /etc/os-release would expect a vanilla Debian
> installation, not an ISAR-built distro.
> >     >
> >     > While I agree that Isar in some configurations creates a Debian
> Derivative, currently it puts image version and build date into
> /etc/os-release. Which is strictly speaking not the right place for that.
> >     If the image version and build date are relevant to have a clear
> identification of the OS version, then I disagree. If that's informative
> and not needed to have a unique identification of the OS version, then I
> agree that it could be somewhere else (although I still think it's not a
> wrong place for it).
> >     >
> >     > It would be better to have something like /etc/base-image-version
> etc. for that.
> >     >
> >
> >
> > After going through this, I feel that we could move the ISAR system
> details to a separate file like /etc/isar_{version/base/info}.
> > ISAR, as I see could be used to create a custom distribution or in a
> simple case just the functions of what an installer would do in other
> > distribution(bootstrap and prepare a simple bootable image for(*and)
> install).
> >
> > If we see the minimal case, it does not make sense to have the ISAR info
> in /etc/os-release. And probably we should leave the /etc/os-release
> > alone; at least in the main ISAR repo and leave the custom configuration
> part to the users. In this way, we are not enforcing any uncommon
> > behavior (dpkg-divert+hook) that a new user least expects in ISAR in its
> vanilla form. This leaves more freedom to the downstream users to
> > choose a method(custom base-files / dpkg-divert / any unofficial way)
> that best fits their need based on what and where they need the information.
> >
> > If you have built an image with ISAR, you could find some details about
> the ISAR system used from the /etc/isar_{version/base/info}. If you want to
> > add more information to it, well you are free to do so. If you feel the
> information is crucial, that it needs to be present in /etc/os-release, and
> > willing to maintain it for your custom distro, well you could do it too.
> If you feel it is unnecessary, well you could remove it too.
> >
> > And then comes the discussion of backward compatibility. Probably
> implement and maintain dpkg-divert approach and provide a way for existing
> users to switch to them using a flag/feature.
> >
> > Thanks,
> > Vijai Kumar K
> >
> >     > regards,
> >     > Claudius
> >       Silvano
> >
> >     --
> >     Siemens AG, T RDA IOT SES-DE
> >     Corporate Competence Center Embedded Linux
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+unsubscribe@googlegroups.com <mailto:
> isar-users+unsubscribe@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%40googlegroups.com
> <
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PKO1KmREqy6GRr20lbuZ%2FsxeINqLIB86%2Fgfd4QeKlQo%3D&reserved=0
> >.
>

[-- Attachment #2: Type: text/html, Size: 24944 bytes --]

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 07/02/2021 10:02, vijai kumar wrote:
>
>
> On Fri, Feb 5, 2021 at 8:27 PM Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com <mailto:silvano.cirujano-cuesta@siemens.com>> wrote:
>
>     Hi Vijai,
>
>     I like your proposal.
>
>     If we need to provide "something" that keeps modifying the /etc/os-release for backwards compatibility, do we want to keep the current "broken" implementation?
>
>
> I would not suggest we keep the current broken implementation. We should fix it.
>  
>
>
>     If we want to "fix" that implementation, then we can do it in a way that not only provides that backwards compatibility, but also to provides a useful feature. The default would be not touching /etc/os-release and one recipe could provide (when requested with a flag) a package for "managing" /etc/os-release (assuming it can be somehow accomplished with multiconfigs, or be a mutually exclusive feature).
>
>
> As Claudius previously mentioned, this might break caching. If the information is purely build-system related, should we try to put it in a package? Does this file need updates in the future? Well, fro"m the current standpoint, I don't see it needing updates. The data is only applicable when we build using ISAR. If hypothetically, we decided to recreate the distro with a cached apt-repo generated with ISAR using some other tool, this info is not relevant.

You possibly miss my explanation why simply placing that file there on post-processing is an error: any package-update of base-files would overwrite that file with the Debian vanilla one. In order to avoid that file from being changed on package updates we need to create either a file diversion or package replacement (like Ubuntu does).

BTW, a package doing only "dkpg-divert" for /etc/os-release doesn't need to contain any version information. Then the version information can be added on image postprocessing. That would be a compromise, because it wouldn't break any package caching.

I don't know exactly how caching works, but I wonder which caching would be breaking just because a package changes. Cached packages? Cached rootfs? Cached images? I would expect almost all rebuilds to be modifying images. I would also expect most of the rebuilds to be modifying the rootfs. Please correct me, if I'm wrong.

>
> That said, we need to understand from the community, what are the real-life scenarios they are using this ISAR information in /etc/os-release for? I am not personally using that info. If it is being used somewhere, we need to know why? So that we know how costly it is to break backward compatibility. Huge or none at all?

I'm sort of part of that community :-) I get here from a project where we are using ISAR. A provider of us identified the above mentioned issue and fixed it on one of their layers replacing base-files with their own version, what I consider the right fix, but still a fix of something broken in ISAR. But being myself also a (small) contributor of ISAR I decided to address the issue upstream.

Back to your "real-life scenario" question. We started using it because that what the place to identify which base-system release we hat while integrating other components on the system. Moving that information to a file like "/etc/isar-release" would fix it for us, we would simply have to update our integration to fetch the information from that file.

Here start my personal opinion, nothing to do with my "real-time scenario". ISAR is being used in multiple cases to build Debian-derivates [1] and I would expect a Debian-derivative to provide some hint about it in its /etc/os-release. Therefore I would like to see ISAR providing the means to easily accomplish it. But since I seam to be the only one seeing that need, I might try to convince you with a RFC patch :-)

[1] https://www.debian.org/derivatives/

  Silvano

>
> Thanks,
> VIjai Kumar K
>  
>
>
>     Ciao,
>
>       Silvano
>
>     On 05/02/2021 12:55, vijaikumar....@gmail.com <mailto:vijaikumar....@gmail.com> wrote:
>     > Hi All,
>     >
>     > On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano Cuesta wrote:
>     >
>     >     Hi Claudius,
>     >
>     >     On 22/01/2021 11:33, Claudius Heine wrote:
>     >     > Hi Silvano
>     >     >
>     >     > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
>     >     >> Hi Claudius,
>     >     >>
>     >     >> TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package
>     >     >>
>     >     >> Nevertheless, more information inline.
>     >     >>
>     >     >> On 22/01/2021 09:52, Claudius Heine wrote:
>     >     >>> Hi Silvano,
>     >     >>>
>     >     >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
>     >     >>>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
>     >     >>>>
>     >     >>>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
>     >     >>>>
>     >     >>>> - Debian os-release manpage still applies
>     >     >>>>
>     >     >>>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
>     >     >>>>
>     >     >>>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
>     >     >>>>
>     >     >>>> - image post-processing to create the file becomes a requirement
>     >     >>> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:
>     >     >>
>     >     >> That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:
>     >     >>
>     >     >> - /etc/os-release symlink to /usr/lib/os-release
>     >     >>
>     >     >> - /usr/lib/os-relase only (not /etc/os-release at all)
>     >     >>
>     >     >> - /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>     >     >>
>     >     >> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
>     >     >
>     >     > Well I had this in mind:
>     >     >
>     >     >   - /etc/os-release is a file containing the isar version, /usr/lib/os-release is a file containing the original debian version as deployed by base-files.
>     >     >
>     >     > This could also be done similar to your fourth solution:
>     >     >
>     >     > - /etc/os-release is a symlink to another file like /usr/lib/isar-release, original /usr/lib/os-release from base-files
>     >     >
>     >     > The manual page states that /etc/os-release takes precedence over /usr/lib/os-release, so if /etc/os-release exists, it does not matter if /usr/lib/os-release does as well or what its content is.
>     >     I also had foreseen that option as one of my favorites (either that or the first one), therefore that would be fine for me.
>     >     >
>     >     >>
>     >     >> ...
>     >     >>
>     >     >>>
>     >     >>> ```
>     >     >>>         The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
>     >     >>>         it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
>     >     >>>         /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
>     >     >>>         to avoid breaking the link in a chroot or initrd environment such as dracut.
>     >     >>>
>     >     >>>         os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
>     >     >>>
>     >     >>>         As this file only encodes names and identifiers it should not be localized.
>     >     >>>
>     >     >>>         The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
>     >     >>>         system.
>     >     >>> ```
>     >     >>>
>     >     >>> So why not make sure `/etc/os-release` is a file and do this:
>     >     >>>
>     >     >>>      dpkg-divert --add --local --no-rename /etc/os-release
>     >     >>>
>     >     >>> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?
>     >     >>
>     >     >> This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.
>     >     >
>     >     > Why would such a command go into a postinst hook? It contains '--local' therefore I would expect the administrator doing it him or herself. Or in Isars case a bitbake post installation process.
>     >     Sorry, forgot that we cannot use a package. A bitbake post installation process is what I had in mind. I don't care in this mailing list about admins "doing stuff" that might break their systems ;-)
>     >     >
>     >     >>
>     >     >> This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...
>     >     >
>     >     > Simple example: Our /etc/os-release contains the build date and git commit id. If we put this information into a package, then it we could not cache it, because this information will change constantly. That also means all dependent tasks and recipes (image creation) cannot be cached, breaking the caching of a lot of stuff.
>     >     >
>     >     > Calling 'bitbake isar-image' two times in short succession without any change, would cause 2 full builds of the whole image.
>     >     The current image post-processing is happening pretty much in the end and not breaking caching, right? Wouldn't it be possible to modify/extend that recipe to create a Debian package and install it chrooting to the root filesystem?
>     >     >
>     >     >>> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.
>     >     >>
>     >     >> Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.
>     >     >>
>     >     >> File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).
>     >     >
>     >     > You misunderstood me.
>     >     >
>     >     > In my example a base-files package would still update /usr/lib/os-release to contain a new debian suite name, if you do a distribution update. If you want our diverted /etc/os-release to contain this new info automatically, you could do this via a apt.conf.d hook.
>     >     Diverting only /etc/os-release you get /usr/lib/os-release managed (and therefore updated) by base-files and ISAR takes care of /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR creates it and adds it to the root filesystem relying on neither the user nor dpkg modifying it.
>     >     >
>     >     >>
>     >     >>>
>     >     >>> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.
>     >     >>
>     >     >> Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):
>     >     >
>     >     > I don't want to divert /usr/lib/os-release. Let it contain the upstream version for reference. This file does not matter, because /etc/os-release takes precedence over it.
>     >     Hopefully it's clear from my comments above, that I find this approach fine.
>     >     >
>     >     >>
>     >     >> 1. Keep originating Debian for reference? Then change only /etc/os-release.
>     >     >
>     >     > Yes.
>     >     >
>     >     >>
>     >     >> 2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.
>     >     >
>     >     > No.
>     >     >
>     >     >>
>     >     >> /etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.
>     >     >
>     >     > Well its a separate discussion if Isar is in the role of a distribution vendor or of an administrator.
>     >
>     >     Is this a philosophical discussion? :-) Am I making a fuss?
>     >
>     >     I might have a different understanding for "admin", but if not distribution builder, I'd say installation builder, but not administrator.
>     >
>     >     IMO ISAR is a distribution builder and as such a tool for distribution vendors. As a Debian user I find at home with many ISAR images, but not at all with some others.
>     >
>     >     A distribution according this definition "A Linux distribution may also be described as a particular assortment of application and utility software [...], packaged together with the Linux kernel in such a way that its capabilities meet the needs of many users.", taken from https://en.wikipedia.org/wiki/Linux_distribution <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429358842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=f3qnjYOwI5IVbiDTIp1RVb4yMb46H0cPnC%2BBtkAeSiE%3D&reserved=0>
>     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229432997922%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b7fZq0akmNohTagcefD81VInLbBkAv%2FcjQTLZNKqIjc%3D&reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429368834%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BEXgejPcmGTr32%2BbJ%2BVx7teZ9SPDa5vhC2sprApiCRo%3D&reserved=0>>
>     >
>     >     >
>     >     > I would say its something in between.
>     >     >
>     >     >>
>     >     >> [1] https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&amp;reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429368834%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Ktv25HHI%2Bf7XOQmsb6ViKvFfDQDzhpzR6NUIly1y8Z4%3D&reserved=0>
>     <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hhqb%2FyU4z%2BqBlh4Yz2ocfxsQTaRtQotyAEJ3dG64k24%3D&reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429378828%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=flCLe1V1QbiuuvdS62vC7qYXUVNeJIv0szecFs9C2jg%3D&reserved=0>>
>     >     >>
>     >     >>> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
>     >     >> As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.
>     >     >
>     >     > This was why I proposed a apt.conf.d hook. Maybe we could generally do it like this.
>     >     No matter how you divert that file (bitbake post processing, apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files packages appear whenever you have a file divert. Unless that vendor-provided base-files package modifies the file diversion being applied by ISAR (I don't know if it's possible) to become the owner of the file.
>     >     >
>     >     > Some package changes /usr/lib/os-release and a apt.conf.d hook would merge this content with ours into /etc/os-release.
>     >     I still don't get the need for such "black magic".
>     >     >
>     >     >>>
>     >     >>> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
>     >     >> Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.
>     >     >
>     >     > While I agree that Isar in some configurations creates a Debian Derivative, currently it puts image version and build date into /etc/os-release. Which is strictly speaking not the right place for that.
>     >     If the image version and build date are relevant to have a clear identification of the OS version, then I disagree. If that's informative and not needed to have a unique identification of the OS version, then I agree that it could be somewhere else (although I still think it's not a wrong place for it).
>     >     >
>     >     > It would be better to have something like /etc/base-image-version etc. for that.
>     >     >
>     >
>     >
>     > After going through this, I feel that we could move the ISAR system details to a separate file like /etc/isar_{version/base/info}.
>     > ISAR, as I see could be used to create a custom distribution or in a simple case just the functions of what an installer would do in other
>     > distribution(bootstrap and prepare a simple bootable image for(*and) install).
>     >
>     > If we see the minimal case, it does not make sense to have the ISAR info in /etc/os-release. And probably we should leave the /etc/os-release
>     > alone; at least in the main ISAR repo and leave the custom configuration part to the users. In this way, we are not enforcing any uncommon
>     > behavior (dpkg-divert+hook) that a new user least expects in ISAR in its vanilla form. This leaves more freedom to the downstream users to
>     > choose a method(custom base-files / dpkg-divert / any unofficial way) that best fits their need based on what and where they need the information.
>     >
>     > If you have built an image with ISAR, you could find some details about the ISAR system used from the /etc/isar_{version/base/info}. If you want to
>     > add more information to it, well you are free to do so. If you feel the information is crucial, that it needs to be present in /etc/os-release, and
>     > willing to maintain it for your custom distro, well you could do it too. If you feel it is unnecessary, well you could remove it too.
>     >
>     > And then comes the discussion of backward compatibility. Probably implement and maintain dpkg-divert approach and provide a way for existing users to switch to them using a flag/feature.
>     >
>     > Thanks,
>     > Vijai Kumar K 
>     >
>     >     > regards,
>     >     > Claudius
>     >       Silvano
>     >
>     >     --
>     >     Siemens AG, T RDA IOT SES-DE
>     >     Corporate Competence Center Embedded Linux
>     >
>     > --
>     > You received this message because you are subscribed to the Google Groups "isar-users" group.
>     > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com <mailto:isar-users%2Bunsubscribe@googlegroups.com> <mailto:isar-users+unsubscribe@googlegroups.com <mailto:isar-users%2Bunsubscribe@googlegroups.com>>.
>     > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%40googlegroups.com <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429388823%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6NhouiZuDTLTf%2FYF5%2BdQHEL%2FBkga3PyysBp3xTUV%2FyI%3D&reserved=0>
>     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PKO1KmREqy6GRr20lbuZ%2FsxeINqLIB86%2Fgfd4QeKlQo%3D&reserved=0
>     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429388823%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FVR4TnQcZEodPPHnwzqHpiuBi4tEwUK3hurROqsMlxI%3D&reserved=0>>.
>

Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijai kumar]

On Mon, Feb 8, 2021 at 2:20 PM Silvano Cirujano Cuesta
<silvano.cirujano-cuesta@siemens.com> wrote:
>
>
> On 07/02/2021 10:02, vijai kumar wrote:
> >
> >
> > On Fri, Feb 5, 2021 at 8:27 PM Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com <mailto:silvano.cirujano-cuesta@siemens.com>> wrote:
> >
> >     Hi Vijai,
> >
> >     I like your proposal.
> >
> >     If we need to provide "something" that keeps modifying the /etc/os-release for backwards compatibility, do we want to keep the current "broken" implementation?
> >
> >
> > I would not suggest we keep the current broken implementation. We should fix it.
> >
> >
> >
> >     If we want to "fix" that implementation, then we can do it in a way that not only provides that backwards compatibility, but also to provides a useful feature. The default would be not touching /etc/os-release and one recipe could provide (when requested with a flag) a package for "managing" /etc/os-release (assuming it can be somehow accomplished with multiconfigs, or be a mutually exclusive feature).
> >
> >
> > As Claudius previously mentioned, this might break caching. If the information is purely build-system related, should we try to put it in a package? Does this file need updates in the future? Well, fro"m the current standpoint, I don't see it needing updates. The data is only applicable when we build using ISAR. If hypothetically, we decided to recreate the distro with a cached apt-repo generated with ISAR using some other tool, this info is not relevant.
>
> You possibly miss my explanation why simply placing that file there on post-processing is an error: any package-update of base-files would overwrite that file with the Debian vanilla one. In order to avoid that file from being changed on package updates we need to create either a file diversion or package replacement (like Ubuntu does).

Ah, In the above I was merely talking about the information (i.e the
ISAR version, commit info etc.) regardless of where it is put. I do
agree that the current way of changing the content of /etc/os-release
is an error, since now we know that it belongs to base-files and its
update makes that information disappear.

>
> BTW, a package doing only "dkpg-divert" for /etc/os-release doesn't need to contain any version information. Then the version information can be added on image postprocessing. That would be a compromise, because it wouldn't break any package caching.
>
> I don't know exactly how caching works, but I wonder which caching would be breaking just because a package changes. Cached packages? Cached rootfs? Cached images? I would expect almost all rebuilds to be modifying images. I would also expect most of the rebuilds to be modifying the rootfs. Please correct me, if I'm wrong.
>
> >
> > That said, we need to understand from the community, what are the real-life scenarios they are using this ISAR information in /etc/os-release for? I am not personally using that info. If it is being used somewhere, we need to know why? So that we know how costly it is to break backward compatibility. Huge or none at all?
>
> I'm sort of part of that community :-) I get here from a project where we are using ISAR. A provider of us identified the above mentioned issue and fixed it on one of their layers replacing base-files with their own version, what I consider the right fix, but still a fix of something broken in ISAR. But being myself also a (small) contributor of ISAR I decided to address the issue upstream.
>
> Back to your "real-life scenario" question. We started using it because that what the place to identify which base-system release we hat while integrating other components on the system. Moving that information to a file like "/etc/isar-release" would fix it for us, we would simply have to update our integration to fetch the information from that file.
>
> Here start my personal opinion, nothing to do with my "real-time scenario". ISAR is being used in multiple cases to build Debian-derivates [1] and I would expect a Debian-derivative to provide some hint about it in its /etc/os-release. Therefore I would like to see ISAR providing the means to easily accomplish it. But since I seam to be the only one seeing that need, I might try to convince you with a RFC patch :-)

Agreed. /etc/os-release is supposed to be extendable. And looks like
there is no hard and fast rule which describes what is "OS
information". It could be accomplished by modifying base-files. I
would recommend ISAR support something similar to it instead of
bringing in a new package to handle /etc/os-release, which by the way
is redundant.

Thanks,
Vijai Kumar K






>
> [1] https://www.debian.org/derivatives/
>
>   Silvano
>
> >
> > Thanks,
> > VIjai Kumar K
> >
> >
> >
> >     Ciao,
> >
> >       Silvano
> >
> >     On 05/02/2021 12:55, vijaikumar....@gmail.com <mailto:vijaikumar....@gmail.com> wrote:
> >     > Hi All,
> >     >
> >     > On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano Cuesta wrote:
> >     >
> >     >     Hi Claudius,
> >     >
> >     >     On 22/01/2021 11:33, Claudius Heine wrote:
> >     >     > Hi Silvano
> >     >     >
> >     >     > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
> >     >     >> Hi Claudius,
> >     >     >>
> >     >     >> TL;DR: you're proposing here what appears to be a feasible solution to this issue without using a package
> >     >     >>
> >     >     >> Nevertheless, more information inline.
> >     >     >>
> >     >     >> On 22/01/2021 09:52, Claudius Heine wrote:
> >     >     >>> Hi Silvano,
> >     >     >>>
> >     >     >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
> >     >     >>>> We can create a removal file diversion for /usr/lib/os-release (leaving /etc/os-release untouched, that way the Debian manpage for os-release still applies). This approach has following attributes:
> >     >     >>>>
> >     >     >>>> - /usr/lib/os-release is not managed by any package and that way post-processing it is not an issue and dpkg won't break anything
> >     >     >>>>
> >     >     >>>> - Debian os-release manpage still applies
> >     >     >>>>
> >     >     >>>> If going for the other approach (create our own "base-files" package) we can do pretty much the same:
> >     >     >>>>
> >     >     >>>> - the package doesn't provide /usr/lib/os-release, leaving it package-unmanaged
> >     >     >>>>
> >     >     >>>> - image post-processing to create the file becomes a requirement
> >     >     >>> I looked at the manpage of os-release and it does not state that `/etc/os-release` can or should not be a file:
> >     >     >>
> >     >     >> That's probably a misunderstanding. I don't mean that we have to create "a file", but provide the os-release information according the specification. There are multiple different valid combinations:
> >     >     >>
> >     >     >> - /etc/os-release symlink to /usr/lib/os-release
> >     >     >>
> >     >     >> - /usr/lib/os-relase only (not /etc/os-release at all)
> >     >     >>
> >     >     >> - /etc/os-release only, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
> >     >     >>
> >     >     >> - /etc/os-release is a symlink to another file like /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the recommended way, but it isn't forbidden either)
> >     >     >
> >     >     > Well I had this in mind:
> >     >     >
> >     >     >   - /etc/os-release is a file containing the isar version, /usr/lib/os-release is a file containing the original debian version as deployed by base-files.
> >     >     >
> >     >     > This could also be done similar to your fourth solution:
> >     >     >
> >     >     > - /etc/os-release is a symlink to another file like /usr/lib/isar-release, original /usr/lib/os-release from base-files
> >     >     >
> >     >     > The manual page states that /etc/os-release takes precedence over /usr/lib/os-release, so if /etc/os-release exists, it does not matter if /usr/lib/os-release does as well or what its content is.
> >     >     I also had foreseen that option as one of my favorites (either that or the first one), therefore that would be fine for me.
> >     >     >
> >     >     >>
> >     >     >> ...
> >     >     >>
> >     >     >>>
> >     >     >>> ```
> >     >     >>>         The file /etc/os-release takes precedence over /usr/lib/os-release. Applications should check for the former, and exclusively use its data if it exists, and only fall back to /usr/lib/os-release if
> >     >     >>>         it is missing. Applications should not read data from both files at the same time.  /usr/lib/os-release is the recommended place to store OS release information as part of vendor trees.
> >     >     >>>         /etc/os-release should be a relative symlink to /usr/lib/os-release, to provide compatibility with applications only looking at /etc/. A relative symlink instead of an absolute symlink is necessary
> >     >     >>>         to avoid breaking the link in a chroot or initrd environment such as dracut.
> >     >     >>>
> >     >     >>>         os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
> >     >     >>>
> >     >     >>>         As this file only encodes names and identifiers it should not be localized.
> >     >     >>>
> >     >     >>>         The /etc/os-release and /usr/lib/os-release files might be symlinks to other files, but it is important that the file is available from earliest boot on, and hence must be located on the root file
> >     >     >>>         system.
> >     >     >>> ```
> >     >     >>>
> >     >     >>> So why not make sure `/etc/os-release` is a file and do this:
> >     >     >>>
> >     >     >>>      dpkg-divert --add --local --no-rename /etc/os-release
> >     >     >>>
> >     >     >>> in the post processing step? Shouldn't this prevent any future base-files update from preventing to overwrite the `/etc/os-release`?
> >     >     >>
> >     >     >> This command would so to say command dpkg not to touch /etc/os-release from the point it's executed on. That way you can have a local version of /etc/os-release and be sure that no package updates is modifying/replacing it. Such a command would go into the postinst hook and another postrm hook would be required to remove the file diversion for /etc/os-release and restoring the original one.
> >     >     >
> >     >     > Why would such a command go into a postinst hook? It contains '--local' therefore I would expect the administrator doing it him or herself. Or in Isars case a bitbake post installation process.
> >     >     Sorry, forgot that we cannot use a package. A bitbake post installation process is what I had in mind. I don't care in this mailing list about admins "doing stuff" that might break their systems ;-)
> >     >     >
> >     >     >>
> >     >     >> This would technically resolve the issue, but looks a bit weird to me. Debian and derivatives get that file managed by a package. But since I haven't understood what the issue mentioned by Henning regarding multiconfig and packages, this might be the only feasible solution...
> >     >     >
> >     >     > Simple example: Our /etc/os-release contains the build date and git commit id. If we put this information into a package, then it we could not cache it, because this information will change constantly. That also means all dependent tasks and recipes (image creation) cannot be cached, breaking the caching of a lot of stuff.
> >     >     >
> >     >     > Calling 'bitbake isar-image' two times in short succession without any change, would cause 2 full builds of the whole image.
> >     >     The current image post-processing is happening pretty much in the end and not breaking caching, right? Wouldn't it be possible to modify/extend that recipe to create a Debian package and install it chrooting to the root filesystem?
> >     >     >
> >     >     >>> The only issue I would see if someone performs a distribution upgrade and now the `/etc/os-release` no longer contains the correct version, but other than an apt.conf hook that updates `/etc/os-release` based on new information from `/usr/lib/os-release` after an upgrade, I don't see a way to do this automatically.
> >     >     >>
> >     >     >> Why should a distribution upgrade change that? I'm not aware that dist-upgrade changes file diversions... To my knowledge, once the file diversion has been created, dpkg should completely ignore that file.
> >     >     >>
> >     >     >> File diversions don't apply to packages, but to files, therefore I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track of the packages that are "blocked" from touching a file. It keeps track of the file path and of the package that requested the file diversion (unless "--local" specified, as you propose).
> >     >     >
> >     >     > You misunderstood me.
> >     >     >
> >     >     > In my example a base-files package would still update /usr/lib/os-release to contain a new debian suite name, if you do a distribution update. If you want our diverted /etc/os-release to contain this new info automatically, you could do this via a apt.conf.d hook.
> >     >     Diverting only /etc/os-release you get /usr/lib/os-release managed (and therefore updated) by base-files and ISAR takes care of /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR creates it and adds it to the root filesystem relying on neither the user nor dpkg modifying it.
> >     >     >
> >     >     >>
> >     >     >>>
> >     >     >>> I would rather divert '/etc/os-release' in a postprocessing script than `/usr/lib/os-release` because, I see the post-processing more of a local configuration.
> >     >     >>
> >     >     >> Diverting /etc/os-release or /usr/lib/os-release depends on what we want to achieve (it would be even better letting the ISAR user decide it):
> >     >     >
> >     >     > I don't want to divert /usr/lib/os-release. Let it contain the upstream version for reference. This file does not matter, because /etc/os-release takes precedence over it.
> >     >     Hopefully it's clear from my comments above, that I find this approach fine.
> >     >     >
> >     >     >>
> >     >     >> 1. Keep originating Debian for reference? Then change only /etc/os-release.
> >     >     >
> >     >     > Yes.
> >     >     >
> >     >     >>
> >     >     >> 2. Wipe away all information about originating Debian? Then change /usr/lib/os-release.
> >     >     >
> >     >     > No.
> >     >     >
> >     >     >>
> >     >     >> /etc/os-release might appear like a local configuration for the ISAR builder, but IMO it's intrinsic information about the distribution being built with ISAR. As the documentation that you are quoting says "defined by the operating system vendor and should generally not be changed by the administrator." Therefore IMO the path /etc/os-release was kept for historical reasons (see footnote 1 of the announcement [1]), but it's an exception to the convention that files under "/etc" are configurations and can be changed.
> >     >     >
> >     >     > Well its a separate discussion if Isar is in the role of a distribution vendor or of an administrator.
> >     >
> >     >     Is this a philosophical discussion? :-) Am I making a fuss?
> >     >
> >     >     I might have a different understanding for "admin", but if not distribution builder, I'd say installation builder, but not administrator.
> >     >
> >     >     IMO ISAR is a distribution builder and as such a tool for distribution vendors. As a Debian user I find at home with many ISAR images, but not at all with some others.
> >     >
> >     >     A distribution according this definition "A Linux distribution may also be described as a particular assortment of application and utility software [...], packaged together with the Linux kernel in such a way that its capabilities meet the needs of many users.", taken from https://en.wikipedia.org/wiki/Linux_distribution <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429358842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=f3qnjYOwI5IVbiDTIp1RVb4yMb46H0cPnC%2BBtkAeSiE%3D&reserved=0>
> >     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229432997922%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b7fZq0akmNohTagcefD81VInLbBkAv%2FcjQTLZNKqIjc%3D&reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429368834%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BEXgejPcmGTr32%2BbJ%2BVx7teZ9SPDa5vhC2sprApiCRo%3D&reserved=0>>
> >     >
> >     >     >
> >     >     > I would say its something in between.
> >     >     >
> >     >     >>
> >     >     >> [1] https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&amp;reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429368834%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Ktv25HHI%2Bf7XOQmsb6ViKvFfDQDzhpzR6NUIly1y8Z4%3D&reserved=0>
> >     <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hhqb%2FyU4z%2BqBlh4Yz2ocfxsQTaRtQotyAEJ3dG64k24%3D&reserved=0 <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429378828%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=flCLe1V1QbiuuvdS62vC7qYXUVNeJIv0szecFs9C2jg%3D&reserved=0>>
> >     >     >>
> >     >     >>> For multiple reasons we cannot used a package to write the version information, so that is out of the picture anyway.
> >     >     >> As said above, if for the reasons you and Henning mention using a package is out of the picture, this is probably the only feasible solution... Except for those people who wrote their own base-files package to resolve this issue, since their package won't work anymore :-) They would have to manage this file using hooks instead of directly declaring the file as managed by their package.
> >     >     >
> >     >     > This was why I proposed a apt.conf.d hook. Maybe we could generally do it like this.
> >     >     No matter how you divert that file (bitbake post processing, apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files packages appear whenever you have a file divert. Unless that vendor-provided base-files package modifies the file diversion being applied by ISAR (I don't know if it's possible) to become the owner of the file.
> >     >     >
> >     >     > Some package changes /usr/lib/os-release and a apt.conf.d hook would merge this content with ours into /etc/os-release.
> >     >     I still don't get the need for such "black magic".
> >     >     >
> >     >     >>>
> >     >     >>> The only other way would be to create a new file just for the image or build version, but that would break compatibility now.
> >     >     >> Do you mean using a completely different file? Something like /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I wouldn't do it. Not only because of the compatibility issue that you mention, but also because ISAR is de-facto creating a Debian Derivative that information belongs logically into /etc/os-release. Someone finding vanilla Debian inputs in /etc/os-release would expect a vanilla Debian installation, not an ISAR-built distro.
> >     >     >
> >     >     > While I agree that Isar in some configurations creates a Debian Derivative, currently it puts image version and build date into /etc/os-release. Which is strictly speaking not the right place for that.
> >     >     If the image version and build date are relevant to have a clear identification of the OS version, then I disagree. If that's informative and not needed to have a unique identification of the OS version, then I agree that it could be somewhere else (although I still think it's not a wrong place for it).
> >     >     >
> >     >     > It would be better to have something like /etc/base-image-version etc. for that.
> >     >     >
> >     >
> >     >
> >     > After going through this, I feel that we could move the ISAR system details to a separate file like /etc/isar_{version/base/info}.
> >     > ISAR, as I see could be used to create a custom distribution or in a simple case just the functions of what an installer would do in other
> >     > distribution(bootstrap and prepare a simple bootable image for(*and) install).
> >     >
> >     > If we see the minimal case, it does not make sense to have the ISAR info in /etc/os-release. And probably we should leave the /etc/os-release
> >     > alone; at least in the main ISAR repo and leave the custom configuration part to the users. In this way, we are not enforcing any uncommon
> >     > behavior (dpkg-divert+hook) that a new user least expects in ISAR in its vanilla form. This leaves more freedom to the downstream users to
> >     > choose a method(custom base-files / dpkg-divert / any unofficial way) that best fits their need based on what and where they need the information.
> >     >
> >     > If you have built an image with ISAR, you could find some details about the ISAR system used from the /etc/isar_{version/base/info}. If you want to
> >     > add more information to it, well you are free to do so. If you feel the information is crucial, that it needs to be present in /etc/os-release, and
> >     > willing to maintain it for your custom distro, well you could do it too. If you feel it is unnecessary, well you could remove it too.
> >     >
> >     > And then comes the discussion of backward compatibility. Probably implement and maintain dpkg-divert approach and provide a way for existing users to switch to them using a flag/feature.
> >     >
> >     > Thanks,
> >     > Vijai Kumar K
> >     >
> >     >     > regards,
> >     >     > Claudius
> >     >       Silvano
> >     >
> >     >     --
> >     >     Siemens AG, T RDA IOT SES-DE
> >     >     Corporate Competence Center Embedded Linux
> >     >
> >     > --
> >     > You received this message because you are subscribed to the Google Groups "isar-users" group.
> >     > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com <mailto:isar-users%2Bunsubscribe@googlegroups.com> <mailto:isar-users+unsubscribe@googlegroups.com <mailto:isar-users%2Bunsubscribe@googlegroups.com>>.
> >     > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%40googlegroups.com <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429388823%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6NhouiZuDTLTf%2FYF5%2BdQHEL%2FBkga3PyysBp3xTUV%2FyI%3D&reserved=0>
> >     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PKO1KmREqy6GRr20lbuZ%2FsxeINqLIB86%2Fgfd4QeKlQo%3D&reserved=0
> >     <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Cc4baff9a162d4f6554ba08d8cb471284%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637482853429388823%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FVR4TnQcZEodPPHnwzqHpiuBi4tEwUK3hurROqsMlxI%3D&reserved=0>>.
> >

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Baurzhan Ismagulov]

On Sun, Feb 07, 2021 at 12:31:23AM -0800, vijaikumar....@gmail.com wrote:
> > If we need to provide "something" that keeps modifying the /etc/os-release 
> > for backwards compatibility, do we want to keep the current "broken" 
> > implementation? 
> 
> I would not suggest keeping it as it is. We know it is broken. Better to 
> fix it.

I suggest to patch /usr/lib/os-release as the first step to preserve the
/etc/os-release symlink (we could provide a patch), then see whether we want to
optionally provide base-files in Isar.


> > If we want to "fix" that implementation, then we can do it in a way that 
> > not only provides that backwards compatibility, but also to provides a 
> > useful feature. The default would be not touching /etc/os-release and one 
> > recipe could provide (when requested with a flag) a package for "managing" 
> > /etc/os-release (assuming it can be somehow accomplished with multiconfigs, 
> > or be a mutually exclusive feature). 
> >
> 
> As Claudius previously mentioned, this might break caching. If the 
> information is purely build-system related, should we try to put it in a 
> package? Does this file need updates in the future? Well, from the current 
> standpoint, I don't see it needing updates. The data is only applicable 
> when we build using ISAR. If hypothetically, we decided to recreate the 
> distro with the cached apt-repo generated with ISAR using some other tool, 
> this info is not relevant.

Under "caching" I understand base-apt. Providing base-files in Isar shouldn't
break base-apt. Isar-generated base-files would land in isar-apt. Updating that
file again and again with every new build is a normal project lifecycle.


> That said, we need to understand from the community, what are the real-life 
> scenarios they are using this ISAR information in /etc/os-release for? I am 
> not personally using that info. If it is being used somewhere, we need to 
> know why? So that we know how costly it is to break backward compatibility. 
> Huge or none at all?

I only use it visually when I log into the target.


On Tue, Feb 09, 2021 at 11:32:30AM +0530, vijai kumar wrote:
> I do
> agree that the current way of changing the content of /etc/os-release
> is an error, since now we know that it belongs to base-files and its
> update makes that information disappear.

While I agree that patching os-release is not optimal, overwriting on upgrade
being a bug or not depends IMHO on whether a downstream declares itself a
distro or not. If they don't, let Debian overwrite it -- it isn't an
Isar-generated image anymore. If they do, they should provide their own
base-files.


> Agreed. /etc/os-release is supposed to be extendable. And looks like
> there is no hard and fast rule which describes what is "OS
> information". It could be accomplished by modifying base-files. I
> would recommend ISAR support something similar to it instead of
> bringing in a new package to handle /etc/os-release, which by the way
> is redundant.

If this is about providing base-files in Isar, I'm open to the discussion. A
modified package would be cleaner than patching the rootfs. That said, my itch
isn't currently that strong to provide a patch :) .


With kind regards,
Baurzhan.

Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijaikumar....@gmail.com]

[-- Attachment #1.1: Type: text/plain, Size: 4228 bytes --]

Hi Baurzhan,

On Wednesday, February 10, 2021 at 2:52:52 PM UTC+5:30 i...@radix50.net 
wrote:

> On Sun, Feb 07, 2021 at 12:31:23AM -0800, vijaikumar....@gmail.com wrote: 
> > > If we need to provide "something" that keeps modifying the 
> /etc/os-release 
> > > for backwards compatibility, do we want to keep the current "broken" 
> > > implementation? 
> > 
> > I would not suggest keeping it as it is. We know it is broken. Better to 
> > fix it. 
>
> I suggest to patch /usr/lib/os-release as the first step to preserve the 
> /etc/os-release symlink (we could provide a patch), then see whether we 
> want to 
> optionally provide base-files in Isar. 
>
>
> > > If we want to "fix" that implementation, then we can do it in a way 
> that 
> > > not only provides that backwards compatibility, but also to provides a 
> > > useful feature. The default would be not touching /etc/os-release and 
> one 
> > > recipe could provide (when requested with a flag) a package for 
> "managing" 
> > > /etc/os-release (assuming it can be somehow accomplished with 
> multiconfigs, 
> > > or be a mutually exclusive feature). 
> > > 
> > 
> > As Claudius previously mentioned, this might break caching. If the 
> > information is purely build-system related, should we try to put it in a 
> > package? Does this file need updates in the future? Well, from the 
> current 
> > standpoint, I don't see it needing updates. The data is only applicable 
> > when we build using ISAR. If hypothetically, we decided to recreate the 
> > distro with the cached apt-repo generated with ISAR using some other 
> tool, 
> > this info is not relevant. 
>
> Under "caching" I understand base-apt. Providing base-files in Isar 
> shouldn't 
> break base-apt. Isar-generated base-files would land in isar-apt. Updating 
> that 
> file again and again with every new build is a normal project lifecycle. 
>
>
> > That said, we need to understand from the community, what are the 
> real-life 
> > scenarios they are using this ISAR information in /etc/os-release for? I 
> am 
> > not personally using that info. If it is being used somewhere, we need 
> to 
> > know why? So that we know how costly it is to break backward 
> compatibility. 
> > Huge or none at all? 
>
> I only use it visually when I log into the target. 
>
>
> On Tue, Feb 09, 2021 at 11:32:30AM +0530, vijai kumar wrote: 
> > I do 
> > agree that the current way of changing the content of /etc/os-release 
> > is an error, since now we know that it belongs to base-files and its 
> > update makes that information disappear. 
>
> While I agree that patching os-release is not optimal, overwriting on 
> upgrade 
> being a bug or not depends IMHO on whether a downstream declares itself a 
> distro or not. If they don't, let Debian overwrite it -- it isn't an 
> Isar-generated image anymore. If they do, they should provide their own 
> base-files. 
>

Since ISAR is providing that information by default, and people use it based
on upstream trust, we should probably have that information in a way that it
is usable even after an upgrade. Or not provide it at all. I am more 
inclined to 
base-files approach here since we know that /etc/os-release is maintained 
by it.
Since providing another file is not the recommended way, as per Debian, we
should probably provide the isar information in /etc/os-release through 
custom
base-files in upstream isar for reference.

Do you feel it is a strong enough reason to have custom base-files in ISAR?

If so, I could try and send a patch for RFC sometime later.

Thanks,
Vijai Kumar K
 

>
>
> > Agreed. /etc/os-release is supposed to be extendable. And looks like 
> > there is no hard and fast rule which describes what is "OS 
> > information". It could be accomplished by modifying base-files. I 
> > would recommend ISAR support something similar to it instead of 
> > bringing in a new package to handle /etc/os-release, which by the way 
> > is redundant. 
>
> If this is about providing base-files in Isar, I'm open to the discussion. 
> A 
> modified package would be cleaner than patching the rootfs. That said, my 
> itch 
> isn't currently that strong to provide a patch :) . 
>
>
> With kind regards, 
> Baurzhan. 
>

[-- Attachment #1.2: Type: text/html, Size: 5094 bytes --]

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Baurzhan Ismagulov]

On Wed, Feb 10, 2021 at 09:54:39PM -0800, vijaikumar....@gmail.com wrote:
> Since ISAR is providing that information by default, and people use it based
> on upstream trust, we should probably have that information in a way that it
> is usable even after an upgrade. Or not provide it at all. I am more 
> inclined to 
> base-files approach here since we know that /etc/os-release is maintained 
> by it.
> Since providing another file is not the recommended way, as per Debian, we
> should probably provide the isar information in /etc/os-release through 
> custom
> base-files in upstream isar for reference.
> 
> Do you feel it is a strong enough reason to have custom base-files in ISAR?

Clean or nothing -- fair enough. I don't have strong feelings that we should
provide that information at all :) , but I admit it is handy after we have it.
As long as Isar doesn't become a distro, I'm fine with providing base-files as
it cleans things up, especially long-term (updates).

With kind regards,
Baurzhan.

Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijaikumar....@gmail.com]

[-- Attachment #1.1: Type: text/plain, Size: 1287 bytes --]



On Thursday, February 11, 2021 at 2:19:35 PM UTC+5:30 i...@radix50.net 
wrote:

> On Wed, Feb 10, 2021 at 09:54:39PM -0800, vijaikumar....@gmail.com wrote: 
> > Since ISAR is providing that information by default, and people use it 
> based 
> > on upstream trust, we should probably have that information in a way 
> that it 
> > is usable even after an upgrade. Or not provide it at all. I am more 
> > inclined to 
> > base-files approach here since we know that /etc/os-release is 
> maintained 
> > by it. 
> > Since providing another file is not the recommended way, as per Debian, 
> we 
> > should probably provide the isar information in /etc/os-release through 
> > custom 
> > base-files in upstream isar for reference. 
> > 
> > Do you feel it is a strong enough reason to have custom base-files in 
> ISAR? 
>
> Clean or nothing -- fair enough. I don't have strong feelings that we 
> should 
> provide that information at all :) , but I admit it is handy after we have 
> it. 
> As long as Isar doesn't become a distro, I'm fine with providing 
> base-files as 
> it cleans things up, especially long-term (updates). 


Ok. In that case, I will try preparing a patch and send it for review 
sometime
soon.

Thanks,
Vijai Kumar K
 

>
> With kind regards, 
> Baurzhan. 
>

[-- Attachment #1.2: Type: text/html, Size: 1805 bytes --]

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 19/01/2021 09:25, Henning Schild wrote:
> Am Mon, 18 Jan 2021 13:35:53 +0100
> schrieb Claudius Heine <ch@denx.de>:
>
>> Hi Silvano,
>>
>> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:
>>> I might try to provide a fix, if we agree that the current
>>> implementation has an issue.
>>>
>>> @Claudius: you wrote the original code [1]. Do you remember why you
>>> implemented it this way? Do you remember if you were aware of the
>>> issue I mentioned and you provided a mitigation for the issue that
>>> I see (assuming my analysis is right)?
>>>
>>> [1]
>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7Ce6139ab1b7d14094332e08d8bc53cb38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466415364594960%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=3jkRDpGDsHDLUs315h%2Fyx40841JT4RpHa%2BvqQ7fD%2FXk%3D&amp;reserved=0
>>>
>>>    Silvano
>>>
>>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:  
>>>> Hi,
>>>>
>>>> I've noticed that '/etc/os-release' is being changed on the image
>>>> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
>>>> ends up changing '/usr/lib/os-release', since it's only a symlink.
>>>> But both '/etc/os-release' and '/usr/lib/os-release' are owned by
>>>> 'base-files'...
>>>>
>>>> An upgrade of 'base-files' would be replacing (silently, since is
>>>> not marked as a configuration file) '/usr/lib/os-release' with the
>>>> version of the upstream 'base-files' package and possibly breaking
>>>> any tools in the system relying on certain values in that file.
>>>>
>>>> Is there a reason that I'm missing for doing so instead of the
>>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
>>>> I've overseen that avoids the mentioned breakage?  
>> Interesting, I didn't remember that `/etc/os-release` is a symlink, 
>> could that be something that has changed in more recent debian
>> versions?
>>
>> If so then, of course that needs to be fixed.
> the problem seems to be that it is a symlink, otherwise one would
> assume that changes in /etc/ are allowed and covered by the config file
> exception and will be subject to merging if an updated package comes
> around.

No, that's wrong. The issue is not that it's a symlink, the issue is that it's a package-managed file that hasn't been declared as a configuration file.

I agree with you that files under /etc should be changeable, but... Debian found sort of a compromise between using the standard path /etc/os-release and not expecting it to be changed (symlink to /usr/lib and no configuration). You can like it or not, but that's what we have to live with...

> My guess would be that we need to
>  - make it a copy instead of a symlink
>  - modify it
As long as the file remains managed by "base-files", none of this should be done. Either we replace "base-files" or we create file diversions for those files. I wanted to focus this thread on confirming the issue, I've opened another thread to align on how to fix this issue. My proposal there is to contact the Debian Derivatives mailing list.
>
> In this case an update of the base-files package should leave it alone
> or ask for a merge. And i think that would be OK behaviour.

But that's not the case if updating upstream Debian "base-files", it silently replaces whatever we place there (file or symlink) with a symlink to /usr/lib/os-release.

  Silvano

>
> Henning
>
>> regards,
>> Claudius
>>
-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Tue, 19 Jan 2021 09:43:17 +0100
schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:

> On 19/01/2021 09:25, Henning Schild wrote:
> > Am Mon, 18 Jan 2021 13:35:53 +0100
> > schrieb Claudius Heine <ch@denx.de>:
> >  
> >> Hi Silvano,
> >>
> >> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:  
> >>> I might try to provide a fix, if we agree that the current
> >>> implementation has an issue.
> >>>
> >>> @Claudius: you wrote the original code [1]. Do you remember why
> >>> you implemented it this way? Do you remember if you were aware of
> >>> the issue I mentioned and you provided a mitigation for the issue
> >>> that I see (assuming my analysis is right)?
> >>>
> >>> [1]
> >>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7Cf108d28909264c483e3d08d8bc5646d1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466426022621736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=nXdIXZTWYDOaDvvJkYCD1q1rYY2dsKuKPj0nUexZz5U%3D&amp;reserved=0
> >>>
> >>>    Silvano
> >>>
> >>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:    
> >>>> Hi,
> >>>>
> >>>> I've noticed that '/etc/os-release' is being changed on the image
> >>>> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
> >>>> ends up changing '/usr/lib/os-release', since it's only a
> >>>> symlink. But both '/etc/os-release' and '/usr/lib/os-release'
> >>>> are owned by 'base-files'...
> >>>>
> >>>> An upgrade of 'base-files' would be replacing (silently, since is
> >>>> not marked as a configuration file) '/usr/lib/os-release' with
> >>>> the version of the upstream 'base-files' package and possibly
> >>>> breaking any tools in the system relying on certain values in
> >>>> that file.
> >>>>
> >>>> Is there a reason that I'm missing for doing so instead of the
> >>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
> >>>> I've overseen that avoids the mentioned breakage?    
> >> Interesting, I didn't remember that `/etc/os-release` is a
> >> symlink, could that be something that has changed in more recent
> >> debian versions?
> >>
> >> If so then, of course that needs to be fixed.  
> > the problem seems to be that it is a symlink, otherwise one would
> > assume that changes in /etc/ are allowed and covered by the config
> > file exception and will be subject to merging if an updated package
> > comes around.  
> 
> No, that's wrong. The issue is not that it's a symlink, the issue is
> that it's a package-managed file that hasn't been declared as a
> configuration file.
> 
> I agree with you that files under /etc should be changeable, but...
> Debian found sort of a compromise between using the standard path
> /etc/os-release and not expecting it to be changed (symlink to
> /usr/lib and no configuration). You can like it or not, but that's
> what we have to live with...

We should not follow that symlink, but i would still hope that we are
allowed to recreate that file without the risk of that package
overwriting the file in /etc/ if it became a file and is modified.

Problem with the copy is that a major update would not be reflected.
Say your isar image starts off with debian9, a dist-upgrade might leave
you with 9 in there.

> > My guess would be that we need to
> >  - make it a copy instead of a symlink
> >  - modify it  
> As long as the file remains managed by "base-files", none of this
> should be done. Either we replace "base-files" or we create file
> diversions for those files. I wanted to focus this thread on
> confirming the issue, I've opened another thread to align on how to
> fix this issue. My proposal there is to contact the Debian
> Derivatives mailing list.

Dont know what a file diversion is, but also sounds like a hack that
would not cover the dist-upgrade correctly.

> >
> > In this case an update of the base-files package should leave it
> > alone or ask for a merge. And i think that would be OK behaviour.  
> 
> But that's not the case if updating upstream Debian "base-files", it
> silently replaces whatever we place there (file or symlink) with a
> symlink to /usr/lib/os-release.

Are you sure about that? That would have to be an "ln -sf" in a
postinst or something, maybe you can share a link to the code doing
that.

If there is code that overwrites a file in /etc without config file
protection, that would clearly be a reason to start talking to
upstream. Because it would be a violation of how config files work in
general.

Henning

>   Silvano
> 
> >
> > Henning
> >  
> >> regards,
> >> Claudius
> >>  

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Tue, 19 Jan 2021 10:08:25 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> Am Tue, 19 Jan 2021 09:43:17 +0100
> schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:
> 
> > On 19/01/2021 09:25, Henning Schild wrote:  
> > > Am Mon, 18 Jan 2021 13:35:53 +0100
> > > schrieb Claudius Heine <ch@denx.de>:
> > >    
> > >> Hi Silvano,
> > >>
> > >> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:    
> > >>> I might try to provide a fix, if we agree that the current
> > >>> implementation has an issue.
> > >>>
> > >>> @Claudius: you wrote the original code [1]. Do you remember why
> > >>> you implemented it this way? Do you remember if you were aware
> > >>> of the issue I mentioned and you provided a mitigation for the
> > >>> issue that I see (assuming my analysis is right)?
> > >>>
> > >>> [1]
> > >>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Cde173c00-e982-4fda-8644-47edf4671d63%40ad011.siemens.com%7Cf108d28909264c483e3d08d8bc5646d1%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466426022621736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=nXdIXZTWYDOaDvvJkYCD1q1rYY2dsKuKPj0nUexZz5U%3D&amp;reserved=0
> > >>>
> > >>>    Silvano
> > >>>
> > >>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:      
> > >>>> Hi,
> > >>>>
> > >>>> I've noticed that '/etc/os-release' is being changed on the
> > >>>> image in meta/classes/image-postproc-extension.bbclass [1].
> > >>>> What BTW ends up changing '/usr/lib/os-release', since it's
> > >>>> only a symlink. But both '/etc/os-release' and
> > >>>> '/usr/lib/os-release' are owned by 'base-files'...
> > >>>>
> > >>>> An upgrade of 'base-files' would be replacing (silently, since
> > >>>> is not marked as a configuration file) '/usr/lib/os-release'
> > >>>> with the version of the upstream 'base-files' package and
> > >>>> possibly breaking any tools in the system relying on certain
> > >>>> values in that file.
> > >>>>
> > >>>> Is there a reason that I'm missing for doing so instead of the
> > >>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
> > >>>> I've overseen that avoids the mentioned breakage?      
> > >> Interesting, I didn't remember that `/etc/os-release` is a
> > >> symlink, could that be something that has changed in more recent
> > >> debian versions?
> > >>
> > >> If so then, of course that needs to be fixed.    
> > > the problem seems to be that it is a symlink, otherwise one would
> > > assume that changes in /etc/ are allowed and covered by the config
> > > file exception and will be subject to merging if an updated
> > > package comes around.    
> > 
> > No, that's wrong. The issue is not that it's a symlink, the issue is
> > that it's a package-managed file that hasn't been declared as a
> > configuration file.
> > 
> > I agree with you that files under /etc should be changeable, but...
> > Debian found sort of a compromise between using the standard path
> > /etc/os-release and not expecting it to be changed (symlink to
> > /usr/lib and no configuration). You can like it or not, but that's
> > what we have to live with...  
> 
> We should not follow that symlink, but i would still hope that we are
> allowed to recreate that file without the risk of that package
> overwriting the file in /etc/ if it became a file and is modified.
> 
> Problem with the copy is that a major update would not be reflected.
> Say your isar image starts off with debian9, a dist-upgrade might
> leave you with 9 in there.
> 
> > > My guess would be that we need to
> > >  - make it a copy instead of a symlink
> > >  - modify it    
> > As long as the file remains managed by "base-files", none of this
> > should be done. Either we replace "base-files" or we create file
> > diversions for those files. I wanted to focus this thread on
> > confirming the issue, I've opened another thread to align on how to
> > fix this issue. My proposal there is to contact the Debian
> > Derivatives mailing list.  
> 
> Dont know what a file diversion is, but also sounds like a hack that
> would not cover the dist-upgrade correctly.
> 
> > >
> > > In this case an update of the base-files package should leave it
> > > alone or ask for a merge. And i think that would be OK behaviour.
> > >    
> > 
> > But that's not the case if updating upstream Debian "base-files", it
> > silently replaces whatever we place there (file or symlink) with a
> > symlink to /usr/lib/os-release.  
> 
> Are you sure about that? That would have to be an "ln -sf" in a
> postinst or something, maybe you can share a link to the code doing
> that.
> 
> If there is code that overwrites a file in /etc without config file
> protection, that would clearly be a reason to start talking to
> upstream. Because it would be a violation of how config files work in
> general.

Indeed, it does overwrite a modified copy ignoring config file
protection.

rm /etc/os-release 
cp /usr/lib/os-release /etc/os-release
vim /etc/os-release
apt-get install base-files --reinstall
ls -al /etc/os-release

It is also replaced when it is a symlink to another location.

Henning

> Henning
> 
> >   Silvano
> >   
> > >
> > > Henning
> > >    
> > >> regards,
> > >> Claudius
> > >>    
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 19/01/2021 10:08, Henning Schild wrote:
> Am Tue, 19 Jan 2021 09:43:17 +0100
> schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:
>
>> On 19/01/2021 09:25, Henning Schild wrote:
>>> Am Mon, 18 Jan 2021 13:35:53 +0100
>>> schrieb Claudius Heine <ch@denx.de>:
>>>  
>>>> Hi Silvano,
>>>>
>>>> On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:  
>>>>> I might try to provide a fix, if we agree that the current
>>>>> implementation has an issue.
>>>>>
>>>>> @Claudius: you wrote the original code [1]. Do you remember why
>>>>> you implemented it this way? Do you remember if you were aware of
>>>>> the issue I mentioned and you provided a mitigation for the issue
>>>>> that I see (assuming my analysis is right)?
>>>>>
>>>>> [1]
>>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fcommit%2F13ce96e5bc84b60f2fa7ccfe93dde045461884e6&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2b01ef368ee24a52b9b908d8bc59c9b3%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466441106647009%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=CN8sFDPYqhNRf%2BeDzmc%2Fz%2FOOXunMg6cP4%2FmgEBa%2Bbr4%3D&amp;reserved=0
>>>>>
>>>>>    Silvano
>>>>>
>>>>> On 15/01/2021 15:26, [ext] Silvano Cirujano Cuesta wrote:    
>>>>>> Hi,
>>>>>>
>>>>>> I've noticed that '/etc/os-release' is being changed on the image
>>>>>> in meta/classes/image-postproc-extension.bbclass [1].  What BTW
>>>>>> ends up changing '/usr/lib/os-release', since it's only a
>>>>>> symlink. But both '/etc/os-release' and '/usr/lib/os-release'
>>>>>> are owned by 'base-files'...
>>>>>>
>>>>>> An upgrade of 'base-files' would be replacing (silently, since is
>>>>>> not marked as a configuration file) '/usr/lib/os-release' with
>>>>>> the version of the upstream 'base-files' package and possibly
>>>>>> breaking any tools in the system relying on certain values in
>>>>>> that file.
>>>>>>
>>>>>> Is there a reason that I'm missing for doing so instead of the
>>>>>> Debian-way (file diversion with dpkg-divert)? Or any hack that
>>>>>> I've overseen that avoids the mentioned breakage?    
>>>> Interesting, I didn't remember that `/etc/os-release` is a
>>>> symlink, could that be something that has changed in more recent
>>>> debian versions?
>>>>
>>>> If so then, of course that needs to be fixed.  
>>> the problem seems to be that it is a symlink, otherwise one would
>>> assume that changes in /etc/ are allowed and covered by the config
>>> file exception and will be subject to merging if an updated package
>>> comes around.  
>> No, that's wrong. The issue is not that it's a symlink, the issue is
>> that it's a package-managed file that hasn't been declared as a
>> configuration file.
>>
>> I agree with you that files under /etc should be changeable, but...
>> Debian found sort of a compromise between using the standard path
>> /etc/os-release and not expecting it to be changed (symlink to
>> /usr/lib and no configuration). You can like it or not, but that's
>> what we have to live with...
> We should not follow that symlink, but i would still hope that we are
> allowed to recreate that file without the risk of that package
> overwriting the file in /etc/ if it became a file and is modified.

I was expecting dpkg to report the problem, but I decided to test it before reporting this issue to have as much knowledge about the situation as possible.

What happens is that an update of "base-files" silently restores the symlink to /usr/lib/os-release, removing whatever is there.

>
> Problem with the copy is that a major update would not be reflected.
> Say your isar image starts off with debian9, a dist-upgrade might leave
> you with 9 in there.
Sorry, I don't get what you mean.
>
>>> My guess would be that we need to
>>>  - make it a copy instead of a symlink
>>>  - modify it  
>> As long as the file remains managed by "base-files", none of this
>> should be done. Either we replace "base-files" or we create file
>> diversions for those files. I wanted to focus this thread on
>> confirming the issue, I've opened another thread to align on how to
>> fix this issue. My proposal there is to contact the Debian
>> Derivatives mailing list.
> Dont know what a file diversion is, but also sounds like a hack that
> would not cover the dist-upgrade correctly.

That's exactly the opposite, it's the Debian way to support that a package can manage a file that was being managed previously by another one. And making all these modifications manageable by dpkg, even restoring the original files is supported.

"dpkg-divert --list" gives you all the file diversions you have on a Debian system. Packages doing diversions are "dash", "firefox-esr", "vim-runtime",...

See here: https://wiki.debian.org/DpkgDiversions

>
>>> In this case an update of the base-files package should leave it
>>> alone or ask for a merge. And i think that would be OK behaviour.  
>> But that's not the case if updating upstream Debian "base-files", it
>> silently replaces whatever we place there (file or symlink) with a
>> symlink to /usr/lib/os-release.
> Are you sure about that? That would have to be an "ln -sf" in a
> postinst or something, maybe you can share a link to the code doing
> that.
I've tested it myself. Built an ISAR qemu-amd64 image, uploaded upstream "base-files", installed it and gone it was (silently, not nice). No more ISAR /etc/os-release.
>
> If there is code that overwrites a file in /etc without config file
> protection, that would clearly be a reason to start talking to
> upstream. Because it would be a violation of how config files work in
> general.

You might be right. I agree with Debian on considering that OS information is not configuration. But they probably didn't want to divert from the Linux common path /etc/os-release.

Considering the content of /usr/lib/os-release something that shouldn't be changeable, they had to break one of following too Linux consents:

- files in /etc are configurations and changeable

- OS information is available under /etc/os-release

  Silvano

>
> Henning
>
>>   Silvano
>>
>>> Henning
>>>  
>>>> regards,
>>>> Claudius
>>>>  

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Claudius Heine]

Hi Silvano,

On 2021-01-18 12:35, Silvano Cirujano Cuesta wrote:
> I might try to provide a fix, if we agree that the current implementation has an issue.
> 
> @Claudius: you wrote the original code [1]. Do you remember why you implemented it this way? Do you remember if you were aware of the issue I mentioned and you provided a mitigation for the issue that I see (assuming my analysis is right)?
> 
> [1] https://github.com/ilbers/isar/commit/13ce96e5bc84b60f2fa7ccfe93dde045461884e6

I now looked at the code, and in that commit I just copied the 
`update_etc_os_release` function to fit into the post processing 
framework. The original commit is actually [1], so Henning might have 
more insight into this than me.

regards,
Claudius

[1] 
https://github.com/ilbers/isar/commit/2c99c8d2b821c493dfc4eab65ea0d205c992cd4d

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Hi,

i wonder if there is any command we can run for QA. One that makes sure
that all files in the rootfs are owned by a package and are not
modified. That should take /etc /var /tmp and other such locations into
account, but should clearly detect the problematic change in /usr/lib
that slipped in via that unexpected symlink.

Does anyone know such a command?

Henning

Am Fri, 15 Jan 2021 15:26:14 +0100
schrieb "[ext] Silvano Cirujano Cuesta"
<silvano.cirujano-cuesta@siemens.com>:

> Hi,
> 
> I've noticed that '/etc/os-release' is being changed on the image in
> meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up
> changing '/usr/lib/os-release', since it's only a symlink. But both
> '/etc/os-release' and '/usr/lib/os-release' are owned by
> 'base-files'...
> 
> An upgrade of 'base-files' would be replacing (silently, since is not
> marked as a configuration file) '/usr/lib/os-release' with the
> version of the upstream 'base-files' package and possibly breaking
> any tools in the system relying on certain values in that file.
> 
> Is there a reason that I'm missing for doing so instead of the
> Debian-way (file diversion with dpkg-divert)? Or any hack that I've
> overseen that avoids the mentioned breakage?
> 
> Hasta la vista,
> 
>   Silvano
> 
> [1]
> https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta/classes/image-postproc-extension.bbclass#L20
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Silvano Cirujano Cuesta]

On 19/01/2021 09:43, Henning Schild wrote:
> Hi,
>
> i wonder if there is any command we can run for QA. One that makes sure
> that all files in the rootfs are owned by a package and are not
> modified. That should take /etc /var /tmp and other such locations into
> account, but should clearly detect the problematic change in /usr/lib
> that slipped in via that unexpected symlink.

It's not that easy. Files generated during package post-processing are not owned by any package.

>
> Does anyone know such a command?

Yes, "dpkg -S" is what you're looking for. For example, "dpkg -S /etc/os-release" returns "base-files: /etc/os-release". Or the other way around, "dpkg -L base-files" gives you all the files managed by "base-files". With "dpkg -S" you can use globbing, but it reports also directories. In the end it's easier something like "find <rootfs> -type f -exec dpkg -S {} \;" if you want to scan the whole tree.

 Silvano

>
> Henning
>
> Am Fri, 15 Jan 2021 15:26:14 +0100
> schrieb "[ext] Silvano Cirujano Cuesta"
> <silvano.cirujano-cuesta@siemens.com>:
>
>> Hi,
>>
>> I've noticed that '/etc/os-release' is being changed on the image in
>> meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up
>> changing '/usr/lib/os-release', since it's only a symlink. But both
>> '/etc/os-release' and '/usr/lib/os-release' are owned by
>> 'base-files'...
>>
>> An upgrade of 'base-files' would be replacing (silently, since is not
>> marked as a configuration file) '/usr/lib/os-release' with the
>> version of the upstream 'base-files' package and possibly breaking
>> any tools in the system relying on certain values in that file.
>>
>> Is there a reason that I'm missing for doing so instead of the
>> Debian-way (file diversion with dpkg-divert)? Or any hack that I've
>> overseen that avoids the mentioned breakage?
>>
>> Hasta la vista,
>>
>>   Silvano
>>
>> [1]
>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Filbers%2Fisar%2Fblob%2Fa0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3%2Fmeta%2Fclasses%2Fimage-postproc-extension.bbclass%23L20&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2e50ef427a76472a2c3308d8bc565c87%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637466426394815917%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=DP6mMX8xBVzZ6v0PupvNXCJb8Amrf%2Fwh91dT9XSL6I8%3D&amp;reserved=0
>>
-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Henning Schild]

Am Fri, 15 Jan 2021 15:26:14 +0100
schrieb "[ext] Silvano Cirujano Cuesta"
<silvano.cirujano-cuesta@siemens.com>:

> Hi,
> 
> I've noticed that '/etc/os-release' is being changed on the image in
> meta/classes/image-postproc-extension.bbclass [1].  What BTW ends up
> changing '/usr/lib/os-release', since it's only a symlink. But both
> '/etc/os-release' and '/usr/lib/os-release' are owned by
> 'base-files'...
> 
> An upgrade of 'base-files' would be replacing (silently, since is not
> marked as a configuration file) '/usr/lib/os-release' with the
> version of the upstream 'base-files' package and possibly breaking
> any tools in the system relying on certain values in that file.

We found that we can not even place a file in /etc/ it will always be
replaced with that symlink, no config file protection on this one.
While we could talk with upstream about that violation, we would also
need a way of dealing with it, because it is unlikely to change in
stable distros. 

> Is there a reason that I'm missing for doing so instead of the
> Debian-way (file diversion with dpkg-divert)? Or any hack that I've
> overseen that avoids the mentioned breakage?

So i propose a hack as follows:

- write a script that can create /etc/os-release from a given default
  and a second file /etc/os-release-isar
  - if variable in both, prefer isar
  - if variable only in isar, append
  - if variable only in original, use
- put that script into a package we install in every isar image
- place a dpkg or apt hook into that package
  - test -f /etc/os-release-isar && test -h /etc/os-release
- create/change /etc/os-release-isar in image postproc and run hook 

Henning

> Hasta la vista,
> 
>   Silvano
> 
> [1]
> https://github.com/ilbers/isar/blob/a0d6de4ac7dfb926b38cfcd44f2bc8852f2e43e3/meta/classes/image-postproc-extension.bbclass#L20
> 

Re: image-postproc-extension.bbclass modifying /etc/os-release

[Baurzhan Ismagulov]

Hello Silvano,

On Fri, Jan 15, 2021 at 03:26:14PM +0100, Silvano Cirujano Cuesta wrote:
> I've noticed that '/etc/os-release' is being changed on the image in meta/classes/image-postproc-extension.bbclass [1].� What BTW ends up changing '/usr/lib/os-release', since it's only a symlink. But both '/etc/os-release' and '/usr/lib/os-release' are owned by 'base-files'...
> 
> An upgrade of 'base-files' would be replacing (silently, since is not marked as a configuration file) '/usr/lib/os-release' with the version of the upstream 'base-files' package and possibly breaking any tools in the system relying on certain values in that file.
> 
> Is there a reason that I'm missing for doing so instead of the Debian-way (file diversion with dpkg-divert)? Or any hack that I've overseen that avoids the mentioned breakage?

Thanks for raising this topic. I'd like to address some of the issues.

* I see identification as a downstream issue. However, it would be nice to
  provide something by default. Thus we touch os-release.

* Regarding an Isar-specific identification file: This is advised against in
  http://0pointer.de/blog/projects/os-release.html.

* Breaking /etc/os-release -> ../usr/lib/os-release is a bug. We should modify
  /usr/lib/os-release, in whatever way. We should not break the symlink. I
  agree we should fix this.

* Isar was conceived as a rootfs builder that should be able to work without
  creating a derivative (remember SLIND). If a downstream feels like it, they
  may create a derivative.

* For a given rootfs, the identification information is intrinsically static.
  Thus, the best would be to provide it in a static way, e.g. by patching (the
  current approach) or providing base-files (I see this as a downstream issue).

* os-release contains static information and doesn't have user-modifiable
  contents. Thus, I agree with Debian that this is not a config file.

* Upgrading base-files would overwrite os-release. From Isar point of view, I'm
  fine with that; after all, it isn't the original image, it's image + upgrade
  -- the identification should be changed as well.

* If a downstream wants to preserve the identification through package
  upgrades, they should provide their own base-files. Maybe we could make it
  easier for them in stock Isar.

* Since the information is static, dynamic approaches (config files, postinst,
  alternatives, diversions) are not necessary. They are more complex than they
  seem at the first glance (package upgrade order, package removal order,
  downgrades, failed upgrades, idempotency, etc.).

To summarize, I suggest fixing /etc/os-release -> ../usr/lib/os-release
symlink breakage.

If there is an itch for the upgrade problem, I'd be open for the discussion
on providing base-files with Isar.

With kind regards,
Baurzhan.