RE: sstate can result in inconsistent kernel and modules

["MOESSBAUER, FELIX JONATHAN" <felix.moessbauer@siemens.com>]

> -----Original Message-----
> From: Uladzimir Bely <ubely@ilbers.de>
> Sent: Wednesday, October 19, 2022 1:37 PM
> To: isar-users@googlegroups.com; Moessbauer, Felix Jonathan (T CED INW-CN)
> <felix.moessbauer@siemens.com>
> Subject: Re: sstate can result in inconsistent kernel and modules
> 
> In mail from понедельник, 17 октября 2022 г. 09:20:25 +03 user
> MOESSBAUER, FELIX JONATHAN wrote:
> > > -----Original Message-----
> > > From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> > > Sent: Monday, October 17, 2022 2:00 PM
> > > To: Moessbauer, Felix Jonathan (T CED INW-CN)
> > > <felix.moessbauer@siemens.com>; isar-users@googlegroups.com;
> > > Schmidt, Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>
> > > Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> > > Subject: Re: sstate can result in inconsistent kernel and modules
> > >
> > > On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
> > > > Dear ISAR users,
> > > >
> > > > I just discovered the following bug on a re-build using the sstate
> > > > cache:
> > > >
> > > > The installed kernel version in the image is 5.19.6-1, while the
> > > > kernel
> > >
> > > modules are built for 5.19.0-2.
> > >
> 
> Hello.
> 
> Could you please clarify where did this '5.19.6.1' version could come from?
> Yesterday I tried to build qemuamd64-bookworm image with example-module in
> order to debug the issue you've described, but what I could see in the resulted
> image is that both module and kernel versions were 5.19.0-2. So, I can't get why
> you could have even newer kernel version few days ago. Some custom recipe
> for the kernel?

The kernel versioning in Debian unfortunately is not very straight forward.
I was referring to the version of the linux-image-amd64 Debian package which is currently "5.19.11-1" (this is the package we directly specify in ISAR by setting the kernel name).
This package currently depends on "linux-image-5.19.0-2-amd64", where "5.19.0-2" is the kernel revision as seen in /usr/lib/modules but "5.19.11-1" is the version of the package. 

Actually I stumbled upon this versioning conflict as well.
While thinking about the situation again, it looks like modules where built for 5.19.0-1 (fetched from sstate) but 5.19.0-2 was installed.

This problem unfortunately is very hard to reproduce as it depends on upstream kernel updates.
But I'll keep an eye on it and report back once it happens again.

Felix

> 
> Also, with sstate-cache enabled, attempt to rebuild the image took everything
> from the sstate cache. So, as expected, id didn't even tried  to update the
> mirrors. And I'm wondering what in your case could trigger the issue (e.g., you
> might was debugging something and editing some recipes at that time).
> 
> > > > The reason is that the modules are extracted from the sstate
> > > > cache, but
> > >
> > > bitbake does not know about the "exact" kernel revision used for building.
> > >
> > > > Actually, I don't know how to easily solve that issue, as bitbake
> > > > does not
> > >
> > > know about the exact Debian package versions.
> > >
> > > > But what could be done is to check for version incompatibilities
> > > > in the
> > > > final>
> > > rootfs.
> > >
> > > > Or we disable the sstate cache for module builds (but I don't know
> > > > how to
> > >
> > > do that).
> > >
> > > How was this inconsistency created? Did you select a different
> > > kernel version from the same isar-bootstrap generated apt database?
> > > Because if the latter changes, all recipes and cached artifacts
> > > downstream should change as well.
> > This happened on a Debian bookworm build where kernel updates happen
> > quite often. The custom modules have been extracted from the cache,
> > but the target rootfs got a more recent kernel. By that, the modules
> > (built for
> > 5.19.0-2) where installed in an image with kernel 5.19.6-1, hence
> > could not be loaded.
> >
> > I don't know if there is an apt upgrade running right after assembling
> > the rootfs, or if simply the target rootfs could not be extracted from
> > the cache.
> >
> > Maybe, the module recipe should add a runtime depends on the exact
> > version of the kernel used for building. This would at least break the
> > build, instead of failing silently.
> >
> > Felix
> >
> > > Jan
> > >
> > > --
> > > Siemens AG, Technology
> > > Competence Center Embedded Linux
> 
> 
>