sstate can result in inconsistent kernel and modules

sstate can result in inconsistent kernel and modules

[MOESSBAUER, FELIX JONATHAN]

Dear ISAR users,

I just discovered the following bug on a re-build using the sstate cache:

The installed kernel version in the image is 5.19.6-1, while the kernel modules are built for 5.19.0-2.
The reason is that the modules are extracted from the sstate cache, but bitbake does not know about the "exact" kernel revision used for building.

Actually, I don't know how to easily solve that issue, as bitbake does not know about the exact Debian package versions.
But what could be done is to check for version incompatibilities in the final rootfs.
Or we disable the sstate cache for module builds (but I don't know how to do that).

Best regards,
Felix

--
Siemens AG, Linux Expert Center
Otto-Hahn-Ring 6, 81739 München, Germany

Re: sstate can result in inconsistent kernel and modules

[Jan Kiszka]

On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
> Dear ISAR users,
> 
> I just discovered the following bug on a re-build using the sstate cache:
> 
> The installed kernel version in the image is 5.19.6-1, while the kernel modules are built for 5.19.0-2.
> The reason is that the modules are extracted from the sstate cache, but bitbake does not know about the "exact" kernel revision used for building.
> 
> Actually, I don't know how to easily solve that issue, as bitbake does not know about the exact Debian package versions.
> But what could be done is to check for version incompatibilities in the final rootfs.
> Or we disable the sstate cache for module builds (but I don't know how to do that).

How was this inconsistency created? Did you select a different kernel
version from the same isar-bootstrap generated apt database? Because if
the latter changes, all recipes and cached artifacts downstream should
change as well.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: sstate can result in inconsistent kernel and modules

[MOESSBAUER, FELIX JONATHAN]

> -----Original Message-----
> From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> Sent: Monday, October 17, 2022 2:00 PM
> To: Moessbauer, Felix Jonathan (T CED INW-CN)
> <felix.moessbauer@siemens.com>; isar-users@googlegroups.com; Schmidt,
> Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>
> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> Subject: Re: sstate can result in inconsistent kernel and modules
> 
> On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
> > Dear ISAR users,
> >
> > I just discovered the following bug on a re-build using the sstate cache:
> >
> > The installed kernel version in the image is 5.19.6-1, while the kernel
> modules are built for 5.19.0-2.
> > The reason is that the modules are extracted from the sstate cache, but
> bitbake does not know about the "exact" kernel revision used for building.
> >
> > Actually, I don't know how to easily solve that issue, as bitbake does not
> know about the exact Debian package versions.
> > But what could be done is to check for version incompatibilities in the final
> rootfs.
> > Or we disable the sstate cache for module builds (but I don't know how to
> do that).
> 
> How was this inconsistency created? Did you select a different kernel version
> from the same isar-bootstrap generated apt database? Because if the latter
> changes, all recipes and cached artifacts downstream should change as well.

This happened on a Debian bookworm build where kernel updates happen quite often.
The custom modules have been extracted from the cache, but the target rootfs got a more recent kernel.
By that, the modules (built for 5.19.0-2) where installed in an image with kernel 5.19.6-1, hence could not be loaded.

I don't know if there is an apt upgrade running right after assembling the rootfs, or if simply the target rootfs could not be extracted from the cache.

Maybe, the module recipe should add a runtime depends on the exact version of the kernel used for building.
This would at least break the build, instead of failing silently.

Felix

> 
> Jan
> 
> --
> Siemens AG, Technology
> Competence Center Embedded Linux

Re: sstate can result in inconsistent kernel and modules

[Jan Kiszka]

On 17.10.22 08:20, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
>> -----Original Message-----
>> From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
>> Sent: Monday, October 17, 2022 2:00 PM
>> To: Moessbauer, Felix Jonathan (T CED INW-CN)
>> <felix.moessbauer@siemens.com>; isar-users@googlegroups.com; Schmidt,
>> Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>
>> Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
>> Subject: Re: sstate can result in inconsistent kernel and modules
>>
>> On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
>>> Dear ISAR users,
>>>
>>> I just discovered the following bug on a re-build using the sstate cache:
>>>
>>> The installed kernel version in the image is 5.19.6-1, while the kernel
>> modules are built for 5.19.0-2.
>>> The reason is that the modules are extracted from the sstate cache, but
>> bitbake does not know about the "exact" kernel revision used for building.
>>>
>>> Actually, I don't know how to easily solve that issue, as bitbake does not
>> know about the exact Debian package versions.
>>> But what could be done is to check for version incompatibilities in the final
>> rootfs.
>>> Or we disable the sstate cache for module builds (but I don't know how to
>> do that).
>>
>> How was this inconsistency created? Did you select a different kernel version
>> from the same isar-bootstrap generated apt database? Because if the latter
>> changes, all recipes and cached artifacts downstream should change as well.
> 
> This happened on a Debian bookworm build where kernel updates happen quite often.
> The custom modules have been extracted from the cache, but the target rootfs got a more recent kernel.
> By that, the modules (built for 5.19.0-2) where installed in an image with kernel 5.19.6-1, hence could not be loaded.
> 
> I don't know if there is an apt upgrade running right after assembling the rootfs, or if simply the target rootfs could not be extracted from the cache.
> 
> Maybe, the module recipe should add a runtime depends on the exact version of the kernel used for building.
> This would at least break the build, instead of failing silently.

If we have a an update of the repo, isar-bootstrap should change, and
all packages built after should get their caches invalidated. Looks like
there is something broken along that line.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: sstate can result in inconsistent kernel and modules

[Uladzimir Bely]

In mail from понедельник, 17 октября 2022 г. 09:20:25 +03 user MOESSBAUER, 
FELIX JONATHAN wrote:
> > -----Original Message-----
> > From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> > Sent: Monday, October 17, 2022 2:00 PM
> > To: Moessbauer, Felix Jonathan (T CED INW-CN)
> > <felix.moessbauer@siemens.com>; isar-users@googlegroups.com; Schmidt,
> > Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>
> > Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> > Subject: Re: sstate can result in inconsistent kernel and modules
> > 
> > On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
> > > Dear ISAR users,
> > > 
> > > I just discovered the following bug on a re-build using the sstate
> > > cache:
> > > 
> > > The installed kernel version in the image is 5.19.6-1, while the kernel
> > 
> > modules are built for 5.19.0-2.
> > 

Hello.

Could you please clarify where did this '5.19.6.1' version could come from? 
Yesterday I tried to build qemuamd64-bookworm image with example-module in 
order to debug the issue you've described, but what I could see in the 
resulted image is that both module and kernel versions were 5.19.0-2. So, I 
can't get why you could have even newer kernel version few days ago. Some 
custom recipe for the kernel?

Also, with sstate-cache enabled, attempt to rebuild the image took everything 
from the sstate cache. So, as expected, id didn't even tried  to update the 
mirrors. And I'm wondering what in your case could trigger the issue (e.g., 
you might was debugging something and editing some recipes at that time).

> > > The reason is that the modules are extracted from the sstate cache, but
> > 
> > bitbake does not know about the "exact" kernel revision used for building.
> > 
> > > Actually, I don't know how to easily solve that issue, as bitbake does
> > > not
> > 
> > know about the exact Debian package versions.
> > 
> > > But what could be done is to check for version incompatibilities in the
> > > final> 
> > rootfs.
> > 
> > > Or we disable the sstate cache for module builds (but I don't know how
> > > to
> > 
> > do that).
> > 
> > How was this inconsistency created? Did you select a different kernel
> > version from the same isar-bootstrap generated apt database? Because if
> > the latter changes, all recipes and cached artifacts downstream should
> > change as well.
> This happened on a Debian bookworm build where kernel updates happen quite
> often. The custom modules have been extracted from the cache, but the
> target rootfs got a more recent kernel. By that, the modules (built for
> 5.19.0-2) where installed in an image with kernel 5.19.6-1, hence could not
> be loaded.
> 
> I don't know if there is an apt upgrade running right after assembling the
> rootfs, or if simply the target rootfs could not be extracted from the
> cache.
> 
> Maybe, the module recipe should add a runtime depends on the exact version
> of the kernel used for building. This would at least break the build,
> instead of failing silently.
> 
> Felix
> 
> > Jan
> > 
> > --
> > Siemens AG, Technology
> > Competence Center Embedded Linux

RE: sstate can result in inconsistent kernel and modules

[MOESSBAUER, FELIX JONATHAN]

> -----Original Message-----
> From: Uladzimir Bely <ubely@ilbers.de>
> Sent: Wednesday, October 19, 2022 1:37 PM
> To: isar-users@googlegroups.com; Moessbauer, Felix Jonathan (T CED INW-CN)
> <felix.moessbauer@siemens.com>
> Subject: Re: sstate can result in inconsistent kernel and modules
> 
> In mail from понедельник, 17 октября 2022 г. 09:20:25 +03 user
> MOESSBAUER, FELIX JONATHAN wrote:
> > > -----Original Message-----
> > > From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> > > Sent: Monday, October 17, 2022 2:00 PM
> > > To: Moessbauer, Felix Jonathan (T CED INW-CN)
> > > <felix.moessbauer@siemens.com>; isar-users@googlegroups.com;
> > > Schmidt, Adriaan (T CED SES-DE) <adriaan.schmidt@siemens.com>
> > > Cc: Schild, Henning (T CED SES-DE) <henning.schild@siemens.com>
> > > Subject: Re: sstate can result in inconsistent kernel and modules
> > >
> > > On 16.10.22 07:48, Moessbauer, Felix Jonathan (T CED INW-CN) wrote:
> > > > Dear ISAR users,
> > > >
> > > > I just discovered the following bug on a re-build using the sstate
> > > > cache:
> > > >
> > > > The installed kernel version in the image is 5.19.6-1, while the
> > > > kernel
> > >
> > > modules are built for 5.19.0-2.
> > >
> 
> Hello.
> 
> Could you please clarify where did this '5.19.6.1' version could come from?
> Yesterday I tried to build qemuamd64-bookworm image with example-module in
> order to debug the issue you've described, but what I could see in the resulted
> image is that both module and kernel versions were 5.19.0-2. So, I can't get why
> you could have even newer kernel version few days ago. Some custom recipe
> for the kernel?

The kernel versioning in Debian unfortunately is not very straight forward.
I was referring to the version of the linux-image-amd64 Debian package which is currently "5.19.11-1" (this is the package we directly specify in ISAR by setting the kernel name).
This package currently depends on "linux-image-5.19.0-2-amd64", where "5.19.0-2" is the kernel revision as seen in /usr/lib/modules but "5.19.11-1" is the version of the package. 

Actually I stumbled upon this versioning conflict as well.
While thinking about the situation again, it looks like modules where built for 5.19.0-1 (fetched from sstate) but 5.19.0-2 was installed.

This problem unfortunately is very hard to reproduce as it depends on upstream kernel updates.
But I'll keep an eye on it and report back once it happens again.

Felix

> 
> Also, with sstate-cache enabled, attempt to rebuild the image took everything
> from the sstate cache. So, as expected, id didn't even tried  to update the
> mirrors. And I'm wondering what in your case could trigger the issue (e.g., you
> might was debugging something and editing some recipes at that time).
> 
> > > > The reason is that the modules are extracted from the sstate
> > > > cache, but
> > >
> > > bitbake does not know about the "exact" kernel revision used for building.
> > >
> > > > Actually, I don't know how to easily solve that issue, as bitbake
> > > > does not
> > >
> > > know about the exact Debian package versions.
> > >
> > > > But what could be done is to check for version incompatibilities
> > > > in the
> > > > final>
> > > rootfs.
> > >
> > > > Or we disable the sstate cache for module builds (but I don't know
> > > > how to
> > >
> > > do that).
> > >
> > > How was this inconsistency created? Did you select a different
> > > kernel version from the same isar-bootstrap generated apt database?
> > > Because if the latter changes, all recipes and cached artifacts
> > > downstream should change as well.
> > This happened on a Debian bookworm build where kernel updates happen
> > quite often. The custom modules have been extracted from the cache,
> > but the target rootfs got a more recent kernel. By that, the modules
> > (built for
> > 5.19.0-2) where installed in an image with kernel 5.19.6-1, hence
> > could not be loaded.
> >
> > I don't know if there is an apt upgrade running right after assembling
> > the rootfs, or if simply the target rootfs could not be extracted from
> > the cache.
> >
> > Maybe, the module recipe should add a runtime depends on the exact
> > version of the kernel used for building. This would at least break the
> > build, instead of failing silently.
> >
> > Felix
> >
> > > Jan
> > >
> > > --
> > > Siemens AG, Technology
> > > Competence Center Embedded Linux
> 
> 
>