Re: image-postproc-extension.bbclass modifying /etc/os-release

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 19493 bytes --]

On Fri, Feb 5, 2021 at 8:27 PM Silvano Cirujano Cuesta <
silvano.cirujano-cuesta@siemens.com> wrote:

> Hi Vijai,
>
> I like your proposal.
>
> If we need to provide "something" that keeps modifying the /etc/os-release
> for backwards compatibility, do we want to keep the current "broken"
> implementation?
>

I would not suggest we keep the current broken implementation. We should
fix it.


>
> If we want to "fix" that implementation, then we can do it in a way that
> not only provides that backwards compatibility, but also to provides a
> useful feature. The default would be not touching /etc/os-release and one
> recipe could provide (when requested with a flag) a package for "managing"
> /etc/os-release (assuming it can be somehow accomplished with multiconfigs,
> or be a mutually exclusive feature).
>

As Claudius previously mentioned, this might break caching. If the
information is purely build-system related, should we try to put it in a
package? Does this file need updates in the future? Well, from the current
standpoint, I don't see it needing updates. The data is only applicable
when we build using ISAR. If hypothetically, we decided to recreate the
distro with a cached apt-repo generated with ISAR using some other tool,
this info is not relevant.

That said, we need to understand from the community, what are the real-life
scenarios they are using this ISAR information in /etc/os-release for? I am
not personally using that info. If it is being used somewhere, we need to
know why? So that we know how costly it is to break backward compatibility.
Huge or none at all?

Thanks,
VIjai Kumar K


>
> Ciao,
>
>   Silvano
>
> On 05/02/2021 12:55, vijaikumar....@gmail.com wrote:
> > Hi All,
> >
> > On Friday, January 22, 2021 at 5:06:34 PM UTC+5:30 Silvano Cirujano
> Cuesta wrote:
> >
> >     Hi Claudius,
> >
> >     On 22/01/2021 11:33, Claudius Heine wrote:
> >     > Hi Silvano
> >     >
> >     > On 2021-01-22 10:47, Silvano Cirujano Cuesta wrote:
> >     >> Hi Claudius,
> >     >>
> >     >> TL;DR: you're proposing here what appears to be a feasible
> solution to this issue without using a package
> >     >>
> >     >> Nevertheless, more information inline.
> >     >>
> >     >> On 22/01/2021 09:52, Claudius Heine wrote:
> >     >>> Hi Silvano,
> >     >>>
> >     >>> On 2021-01-19 11:37, Silvano Cirujano Cuesta wrote:
> >     >>>> We can create a removal file diversion for /usr/lib/os-release
> (leaving /etc/os-release untouched, that way the Debian manpage for
> os-release still applies). This approach has following attributes:
> >     >>>>
> >     >>>> - /usr/lib/os-release is not managed by any package and that
> way post-processing it is not an issue and dpkg won't break anything
> >     >>>>
> >     >>>> - Debian os-release manpage still applies
> >     >>>>
> >     >>>> If going for the other approach (create our own "base-files"
> package) we can do pretty much the same:
> >     >>>>
> >     >>>> - the package doesn't provide /usr/lib/os-release, leaving it
> package-unmanaged
> >     >>>>
> >     >>>> - image post-processing to create the file becomes a requirement
> >     >>> I looked at the manpage of os-release and it does not state that
> `/etc/os-release` can or should not be a file:
> >     >>
> >     >> That's probably a misunderstanding. I don't mean that we have to
> create "a file", but provide the os-release information according the
> specification. There are multiple different valid combinations:
> >     >>
> >     >> - /etc/os-release symlink to /usr/lib/os-release
> >     >>
> >     >> - /usr/lib/os-relase only (not /etc/os-release at all)
> >     >>
> >     >> - /etc/os-release only, no /usr/lib/os-release at all (it's not
> the recommended way, but it isn't forbidden either)
> >     >>
> >     >> - /etc/os-release is a symlink to another file like
> /usr/lib/isar-release, no /usr/lib/os-release at all (it's not the
> recommended way, but it isn't forbidden either)
> >     >
> >     > Well I had this in mind:
> >     >
> >     >   - /etc/os-release is a file containing the isar version,
> /usr/lib/os-release is a file containing the original debian version as
> deployed by base-files.
> >     >
> >     > This could also be done similar to your fourth solution:
> >     >
> >     > - /etc/os-release is a symlink to another file like
> /usr/lib/isar-release, original /usr/lib/os-release from base-files
> >     >
> >     > The manual page states that /etc/os-release takes precedence over
> /usr/lib/os-release, so if /etc/os-release exists, it does not matter if
> /usr/lib/os-release does as well or what its content is.
> >     I also had foreseen that option as one of my favorites (either that
> or the first one), therefore that would be fine for me.
> >     >
> >     >>
> >     >> ...
> >     >>
> >     >>>
> >     >>> ```
> >     >>>         The file /etc/os-release takes precedence over
> /usr/lib/os-release. Applications should check for the former, and
> exclusively use its data if it exists, and only fall back to
> /usr/lib/os-release if
> >     >>>         it is missing. Applications should not read data from
> both files at the same time.  /usr/lib/os-release is the recommended place
> to store OS release information as part of vendor trees.
> >     >>>         /etc/os-release should be a relative symlink to
> /usr/lib/os-release, to provide compatibility with applications only
> looking at /etc/. A relative symlink instead of an absolute symlink is
> necessary
> >     >>>         to avoid breaking the link in a chroot or initrd
> environment such as dracut.
> >     >>>
> >     >>>         os-release contains data that is defined by the
> operating system vendor and should generally not be changed by the
> administrator.
> >     >>>
> >     >>>         As this file only encodes names and identifiers it
> should not be localized.
> >     >>>
> >     >>>         The /etc/os-release and /usr/lib/os-release files might
> be symlinks to other files, but it is important that the file is available
> from earliest boot on, and hence must be located on the root file
> >     >>>         system.
> >     >>> ```
> >     >>>
> >     >>> So why not make sure `/etc/os-release` is a file and do this:
> >     >>>
> >     >>>      dpkg-divert --add --local --no-rename /etc/os-release
> >     >>>
> >     >>> in the post processing step? Shouldn't this prevent any future
> base-files update from preventing to overwrite the `/etc/os-release`?
> >     >>
> >     >> This command would so to say command dpkg not to touch
> /etc/os-release from the point it's executed on. That way you can have a
> local version of /etc/os-release and be sure that no package updates is
> modifying/replacing it. Such a command would go into the postinst hook and
> another postrm hook would be required to remove the file diversion for
> /etc/os-release and restoring the original one.
> >     >
> >     > Why would such a command go into a postinst hook? It contains
> '--local' therefore I would expect the administrator doing it him or
> herself. Or in Isars case a bitbake post installation process.
> >     Sorry, forgot that we cannot use a package. A bitbake post
> installation process is what I had in mind. I don't care in this mailing
> list about admins "doing stuff" that might break their systems ;-)
> >     >
> >     >>
> >     >> This would technically resolve the issue, but looks a bit weird
> to me. Debian and derivatives get that file managed by a package. But since
> I haven't understood what the issue mentioned by Henning regarding
> multiconfig and packages, this might be the only feasible solution...
> >     >
> >     > Simple example: Our /etc/os-release contains the build date and
> git commit id. If we put this information into a package, then it we could
> not cache it, because this information will change constantly. That also
> means all dependent tasks and recipes (image creation) cannot be cached,
> breaking the caching of a lot of stuff.
> >     >
> >     > Calling 'bitbake isar-image' two times in short succession without
> any change, would cause 2 full builds of the whole image.
> >     The current image post-processing is happening pretty much in the
> end and not breaking caching, right? Wouldn't it be possible to
> modify/extend that recipe to create a Debian package and install it
> chrooting to the root filesystem?
> >     >
> >     >>> The only issue I would see if someone performs a distribution
> upgrade and now the `/etc/os-release` no longer contains the correct
> version, but other than an apt.conf hook that updates `/etc/os-release`
> based on new information from `/usr/lib/os-release` after an upgrade, I
> don't see a way to do this automatically.
> >     >>
> >     >> Why should a distribution upgrade change that? I'm not aware that
> dist-upgrade changes file diversions... To my knowledge, once the file
> diversion has been created, dpkg should completely ignore that file.
> >     >>
> >     >> File diversions don't apply to packages, but to files, therefore
> I'd expect it to survive a dist-upgrade. Therefore dpkg doesn't keep track
> of the packages that are "blocked" from touching a file. It keeps track of
> the file path and of the package that requested the file diversion (unless
> "--local" specified, as you propose).
> >     >
> >     > You misunderstood me.
> >     >
> >     > In my example a base-files package would still update
> /usr/lib/os-release to contain a new debian suite name, if you do a
> distribution update. If you want our diverted /etc/os-release to contain
> this new info automatically, you could do this via a apt.conf.d hook.
> >     Diverting only /etc/os-release you get /usr/lib/os-release managed
> (and therefore updated) by base-files and ISAR takes care of
> /etc/os-release. I don't understand the need for an apt.conf.d hook... ISAR
> creates it and adds it to the root filesystem relying on neither the user
> nor dpkg modifying it.
> >     >
> >     >>
> >     >>>
> >     >>> I would rather divert '/etc/os-release' in a postprocessing
> script than `/usr/lib/os-release` because, I see the post-processing more
> of a local configuration.
> >     >>
> >     >> Diverting /etc/os-release or /usr/lib/os-release depends on what
> we want to achieve (it would be even better letting the ISAR user decide
> it):
> >     >
> >     > I don't want to divert /usr/lib/os-release. Let it contain the
> upstream version for reference. This file does not matter, because
> /etc/os-release takes precedence over it.
> >     Hopefully it's clear from my comments above, that I find this
> approach fine.
> >     >
> >     >>
> >     >> 1. Keep originating Debian for reference? Then change only
> /etc/os-release.
> >     >
> >     > Yes.
> >     >
> >     >>
> >     >> 2. Wipe away all information about originating Debian? Then
> change /usr/lib/os-release.
> >     >
> >     > No.
> >     >
> >     >>
> >     >> /etc/os-release might appear like a local configuration for the
> ISAR builder, but IMO it's intrinsic information about the distribution
> being built with ISAR. As the documentation that you are quoting says
> "defined by the operating system vendor and should generally not be changed
> by the administrator." Therefore IMO the path /etc/os-release was kept for
> historical reasons (see footnote 1 of the announcement [1]), but it's an
> exception to the convention that files under "/etc" are configurations and
> can be changed.
> >     >
> >     > Well its a separate discussion if Isar is in the role of a
> distribution vendor or of an administrator.
> >
> >     Is this a philosophical discussion? :-) Am I making a fuss?
> >
> >     I might have a different understanding for "admin", but if not
> distribution builder, I'd say installation builder, but not administrator.
> >
> >     IMO ISAR is a distribution builder and as such a tool for
> distribution vendors. As a Debian user I find at home with many ISAR
> images, but not at all with some others.
> >
> >     A distribution according this definition "A Linux distribution may
> also be described as a particular assortment of application and utility
> software [...], packaged together with the Linux kernel in such a way that
> its capabilities meet the needs of many users.", taken from
> https://en.wikipedia.org/wiki/Linux_distribution <
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FLinux_distribution&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229432997922%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b7fZq0akmNohTagcefD81VInLbBkAv%2FcjQTLZNKqIjc%3D&reserved=0
> >
> >
> >     >
> >     > I would say its something in between.
> >     >
> >     >>
> >     >> [1]
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C4348b0c7cf544189355608d8bec12a38%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637469084129831158%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gobc49NK4Zh02%2BrVOBBU%2BQMg7sAu5XVstVzu39CquOA%3D&amp;reserved=0
> <
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F0pointer.de%2Fblog%2Fprojects%2Fos-release&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Hhqb%2FyU4z%2BqBlh4Yz2ocfxsQTaRtQotyAEJ3dG64k24%3D&reserved=0
> >
> >     >>
> >     >>> For multiple reasons we cannot used a package to write the
> version information, so that is out of the picture anyway.
> >     >> As said above, if for the reasons you and Henning mention using a
> package is out of the picture, this is probably the only feasible
> solution... Except for those people who wrote their own base-files package
> to resolve this issue, since their package won't work anymore :-) They
> would have to manage this file using hooks instead of directly declaring
> the file as managed by their package.
> >     >
> >     > This was why I proposed a apt.conf.d hook. Maybe we could
> generally do it like this.
> >     No matter how you divert that file (bitbake post processing,
> apt.conf.d hook,...) the mentioned conflict with vendor-provided base-files
> packages appear whenever you have a file divert. Unless that
> vendor-provided base-files package modifies the file diversion being
> applied by ISAR (I don't know if it's possible) to become the owner of the
> file.
> >     >
> >     > Some package changes /usr/lib/os-release and a apt.conf.d hook
> would merge this content with ours into /etc/os-release.
> >     I still don't get the need for such "black magic".
> >     >
> >     >>>
> >     >>> The only other way would be to create a new file just for the
> image or build version, but that would break compatibility now.
> >     >> Do you mean using a completely different file? Something like
> /etc/isar-release or /usr/lib/isar-release? If that's what you mean, I
> wouldn't do it. Not only because of the compatibility issue that you
> mention, but also because ISAR is de-facto creating a Debian Derivative
> that information belongs logically into /etc/os-release. Someone finding
> vanilla Debian inputs in /etc/os-release would expect a vanilla Debian
> installation, not an ISAR-built distro.
> >     >
> >     > While I agree that Isar in some configurations creates a Debian
> Derivative, currently it puts image version and build date into
> /etc/os-release. Which is strictly speaking not the right place for that.
> >     If the image version and build date are relevant to have a clear
> identification of the OS version, then I disagree. If that's informative
> and not needed to have a unique identification of the OS version, then I
> agree that it could be somewhere else (although I still think it's not a
> wrong place for it).
> >     >
> >     > It would be better to have something like /etc/base-image-version
> etc. for that.
> >     >
> >
> >
> > After going through this, I feel that we could move the ISAR system
> details to a separate file like /etc/isar_{version/base/info}.
> > ISAR, as I see could be used to create a custom distribution or in a
> simple case just the functions of what an installer would do in other
> > distribution(bootstrap and prepare a simple bootable image for(*and)
> install).
> >
> > If we see the minimal case, it does not make sense to have the ISAR info
> in /etc/os-release. And probably we should leave the /etc/os-release
> > alone; at least in the main ISAR repo and leave the custom configuration
> part to the users. In this way, we are not enforcing any uncommon
> > behavior (dpkg-divert+hook) that a new user least expects in ISAR in its
> vanilla form. This leaves more freedom to the downstream users to
> > choose a method(custom base-files / dpkg-divert / any unofficial way)
> that best fits their need based on what and where they need the information.
> >
> > If you have built an image with ISAR, you could find some details about
> the ISAR system used from the /etc/isar_{version/base/info}. If you want to
> > add more information to it, well you are free to do so. If you feel the
> information is crucial, that it needs to be present in /etc/os-release, and
> > willing to maintain it for your custom distro, well you could do it too.
> If you feel it is unnecessary, well you could remove it too.
> >
> > And then comes the discussion of backward compatibility. Probably
> implement and maintain dpkg-divert approach and provide a way for existing
> users to switch to them using a flag/feature.
> >
> > Thanks,
> > Vijai Kumar K
> >
> >     > regards,
> >     > Claudius
> >       Silvano
> >
> >     --
> >     Siemens AG, T RDA IOT SES-DE
> >     Corporate Competence Center Embedded Linux
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+unsubscribe@googlegroups.com <mailto:
> isar-users+unsubscribe@googlegroups.com>.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%40googlegroups.com
> <
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fisar-users%2F6f4b945f-7763-49ee-a8c5-2e02d450a2e0n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2210c7432f134485959a08d8c9ccf62a%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637481229433007915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PKO1KmREqy6GRr20lbuZ%2FsxeINqLIB86%2Fgfd4QeKlQo%3D&reserved=0
> >.
>

[-- Attachment #2: Type: text/html, Size: 24944 bytes --]