SSTATE cache not working for -native recipes due to unexpanded variable

SSTATE cache not working for -native recipes due to unexpanded variable

[Waltl, Jan]

Hello,

I noticed that for -native recipes, SSTATE_PKGARCH variable is not expanded in the .siginfo filename. On rebuild, SSTATE reports mismatches leading to rebuilds even though the recipe and the resulting stamps are exactly the same.

Reproduction steps:
1. Checkout ISAR next 216f0721265fcfe965ac48b51a0128d612d89bf4.
2. Run 'bitbake cowsay-native' from meta-isar layer  (I use KAS for setting up the bitbake directory)
4. Look into sstate directory, I see a files:
sstate-cache/31/1a/sstate:cowsay-native::git:r0::10:311a86032fcbd98d2343b40447e0558b5aae410ff64f727ae56aa2952167f05e_patch.tar.zst.siginfo
sstate-cache/3b/24/sstate:cowsay-native::git:r0::10:3b2452fa0b072ed1488aad9432257dff8bee324659b5f0738ab224d82b1de2fc_unpack.tar.zst.siginfo
sstate-cache/d6/ab/sstate:cowsay-native::git:r0::10:d6abcc3328b93b67b59ef9c0b21449188f1330bc76d6ee6372c76e94bec87756_fetch.tar.zst.siginfo
sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-isar:git:r0:${SSTATE_PKGARCH}:10:64de5a007fac75eebd427a2ea9156cf0142718f68362a06b51c65559fc8045b7_dpkg_build.tar.zst.siginfo'
sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-isar:git:r0::10:64de5a007fac75eebd427a2ea9156cf0142718f68362a06b51c65559fc8045b7_dpkg_build.tar.zst
sstate-cache/isarnative/aa/07/sstate:cowsay-native:amd64-isar:git:r0:${SSTATE_PKGARCH}:10:aa07cf2e6ff17f7004e6dc2b4e7a9a6123b127668c27d4c3cb29d0f7dc65ac5f_deploy_deb.tar.zst.siginfo'
sstate-cache/isarnative/c4/db/sstate:cowsay-native:amd64-isar:git:r0:${SSTATE_PKGARCH}:10:c4dbfbe1c1696f2b690f15cbd179a5795fe48f05a30a76d20783c98786af6d4a_adjust_git.tar.zst.siginfo'
sstate-cache/isarnative/d0/0c/sstate:cowsay-native:amd64-isar:git:r0:${SSTATE_PKGARCH}:10:d00c69fa102b131bc09fb27aa8259fcb447dbe5b51d5f6723d1e12ff43f92fee_transform_template.tar.zst.siginfo'
sstate-cache/isarnative/e7/94/sstate:cowsay-native:amd64-isar:git:r0:${SSTATE_PKGARCH}:10:e794e52fe10ccf60b559089c39eaf86a4990a81991c9b1b0b2672721878a1b8b_prepare_build.tar.zst.siginfo'

5. Remove stamps directory.
6. Build again - SSTATE should be used for the package but it is not and the tasks are run again.

I think it is related to https://github.com/ilbers/isar/blob/61086915e6c83fff22effa85cda64a2ac0c2f100/meta/classes/sstate.bbclass#L136 and BUILD_ARCH being empty according to 'bitbake -e'. 

I changed the line to HOST_ARCH (+expand=True) which seems to work but I do not have enough expertise to guess its correctness. Maybe the line in native.bbclass covers this "if statement" already?

How to best resolve this?


[This is my first message to this mailing list, please correct me if I am doing something wrong]


Kind Regards,
Jan Waltl

Siemens Mobility, s.r.o.
SMO RI LCE CZ SAF 1

Re: SSTATE cache not working for -native recipes due to unexpanded variable

[Uladzimir Bely]

On Tue, 2023-10-03 at 13:30 +0000, 'Waltl, Jan' via isar-users wrote:
> Hello,
> 
> I noticed that for -native recipes, SSTATE_PKGARCH variable is not
> expanded in the .siginfo filename. On rebuild, SSTATE reports
> mismatches leading to rebuilds even though the recipe and the
> resulting stamps are exactly the same.
> 
> Reproduction steps:
> 1. Checkout ISAR next 216f0721265fcfe965ac48b51a0128d612d89bf4.
> 2. Run 'bitbake cowsay-native' from meta-isar layer  (I use KAS for
> setting up the bitbake directory)
> 4. Look into sstate directory, I see a files:
> sstate-cache/31/1a/sstate:cowsay-
> native::git:r0::10:311a86032fcbd98d2343b40447e0558b5aae410ff64f727ae5
> 6aa2952167f05e_patch.tar.zst.siginfo
> sstate-cache/3b/24/sstate:cowsay-
> native::git:r0::10:3b2452fa0b072ed1488aad9432257dff8bee324659b5f0738a
> b224d82b1de2fc_unpack.tar.zst.siginfo
> sstate-cache/d6/ab/sstate:cowsay-
> native::git:r0::10:d6abcc3328b93b67b59ef9c0b21449188f1330bc76d6ee6372
> c76e94bec87756_fetch.tar.zst.siginfo
> sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:64de5a007fac75eebd427a2ea9156cf01427
> 18f68362a06b51c65559fc8045b7_dpkg_build.tar.zst.siginfo'
> sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> isar:git:r0::10:64de5a007fac75eebd427a2ea9156cf0142718f68362a06b51c65
> 559fc8045b7_dpkg_build.tar.zst
> sstate-cache/isarnative/aa/07/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:aa07cf2e6ff17f7004e6dc2b4e7a9a6123b1
> 27668c27d4c3cb29d0f7dc65ac5f_deploy_deb.tar.zst.siginfo'
> sstate-cache/isarnative/c4/db/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:c4dbfbe1c1696f2b690f15cbd179a5795fe4
> 8f05a30a76d20783c98786af6d4a_adjust_git.tar.zst.siginfo'
> sstate-cache/isarnative/d0/0c/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:d00c69fa102b131bc09fb27aa8259fcb447d
> be5b51d5f6723d1e12ff43f92fee_transform_template.tar.zst.siginfo'
> sstate-cache/isarnative/e7/94/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:e794e52fe10ccf60b559089c39eaf86a4990
> a81991c9b1b0b2672721878a1b8b_prepare_build.tar.zst.siginfo'
> 
> 5. Remove stamps directory.
> 6. Build again - SSTATE should be used for the package but it is not
> and the tasks are run again.
> 
> I think it is related to
> https://github.com/ilbers/isar/blob/61086915e6c83fff22effa85cda64a2ac0c2f100/meta/classes/sstate.bbclass#L136
>  and BUILD_ARCH being empty according to 'bitbake -e'. 
> 
> I changed the line to HOST_ARCH (+expand=True) which seems to work
> but I do not have enough expertise to guess its correctness. Maybe
> the line in native.bbclass covers this "if statement" already?
> 
> How to best resolve this?
> 
> 
Hello, Jan.

I've tried to repeat your steps and reproduced the issue in case of
cross-building.

Changing the line to HOST_ARCH is techincally correct. But since we
don't like forking (sstate.bbclass is taken from some OE branch without
any changes), I think it would be better to just define BUILD_ARCH
somewhere else in Isar to the value of HOST_ARCH, for example:

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9eb93e2b..0d737b64 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -90,6 +90,7 @@ NATIVELSBSTRING ?= "isarnative"
 TARGET_VENDOR = ""
 TARGET_OS = "isar"
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
+BUILD_ARCH ?= "${HOST_ARCH}"
 
 # Isar apt repository paths
 REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"

This seems to work and doesn't fork sstate.bbclass, so it can be later
updated to some newer OE revision.

Unfortunately, HOST_ARCH has different meanings in Isar and Yocto...

> [This is my first message to this mailing list, please correct me if
> I am doing something wrong]
> 
> 
> Kind Regards,
> Jan Waltl
> 
> Siemens Mobility, s.r.o.
> SMO RI LCE CZ SAF 1
> 
> 
> 

RE: SSTATE cache not working for -native recipes due to unexpanded variable

[Waltl, Jan]

Hello Uladzimir,

Thank you very, the patch appears to fix the issue for caching of native builds for me - now SSTATE works with them.

Also, I see some related discussions on the mailing list I will try to follow.

Kind Regards
Jan Waltl

Siemens Mobility, s.r.o.
SMO RI LCE CZ SAF 1


-----Original Message-----
From: Uladzimir Bely <ubely@ilbers.de> 
Sent: Tuesday, October 3, 2023 5:41 PM
To: Waltl, Jan (SMO RI LCE CZ SAF 1) <jan.waltl@siemens.com>; isar-users@googlegroups.com
Subject: Re: SSTATE cache not working for -native recipes due to unexpanded variable

On Tue, 2023-10-03 at 13:30 +0000, 'Waltl, Jan' via isar-users wrote:
> Hello,
> 
> I noticed that for -native recipes, SSTATE_PKGARCH variable is not 
> expanded in the .siginfo filename. On rebuild, SSTATE reports 
> mismatches leading to rebuilds even though the recipe and the 
> resulting stamps are exactly the same.
> 
> Reproduction steps:
> 1. Checkout ISAR next 216f0721265fcfe965ac48b51a0128d612d89bf4.
> 2. Run 'bitbake cowsay-native' from meta-isar layer  (I use KAS for 
> setting up the bitbake directory) 4. Look into sstate directory, I see 
> a files:
> sstate-cache/31/1a/sstate:cowsay-
> native::git:r0::10:311a86032fcbd98d2343b40447e0558b5aae410ff64f727ae5
> 6aa2952167f05e_patch.tar.zst.siginfo
> sstate-cache/3b/24/sstate:cowsay-
> native::git:r0::10:3b2452fa0b072ed1488aad9432257dff8bee324659b5f0738a
> b224d82b1de2fc_unpack.tar.zst.siginfo
> sstate-cache/d6/ab/sstate:cowsay-
> native::git:r0::10:d6abcc3328b93b67b59ef9c0b21449188f1330bc76d6ee6372
> c76e94bec87756_fetch.tar.zst.siginfo
> sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:64de5a007fac75eebd427a2ea9156cf01427
> 18f68362a06b51c65559fc8045b7_dpkg_build.tar.zst.siginfo'
> sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> isar:git:r0::10:64de5a007fac75eebd427a2ea9156cf0142718f68362a06b51c65
> 559fc8045b7_dpkg_build.tar.zst
> sstate-cache/isarnative/aa/07/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:aa07cf2e6ff17f7004e6dc2b4e7a9a6123b1
> 27668c27d4c3cb29d0f7dc65ac5f_deploy_deb.tar.zst.siginfo'
> sstate-cache/isarnative/c4/db/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:c4dbfbe1c1696f2b690f15cbd179a5795fe4
> 8f05a30a76d20783c98786af6d4a_adjust_git.tar.zst.siginfo'
> sstate-cache/isarnative/d0/0c/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:d00c69fa102b131bc09fb27aa8259fcb447d
> be5b51d5f6723d1e12ff43f92fee_transform_template.tar.zst.siginfo'
> sstate-cache/isarnative/e7/94/sstate:cowsay-native:amd64-
> isar:git:r0:${SSTATE_PKGARCH}:10:e794e52fe10ccf60b559089c39eaf86a4990
> a81991c9b1b0b2672721878a1b8b_prepare_build.tar.zst.siginfo'
> 
> 5. Remove stamps directory.
> 6. Build again - SSTATE should be used for the package but it is not 
> and the tasks are run again.
> 
> I think it is related to
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Filbers%2Fisar%2Fblob%2F61086915e6c83fff22effa85cda64a2ac0c2f1
> 00%2Fmeta%2Fclasses%2Fsstate.bbclass%23L136&data=05%7C01%7Cjan.waltl%4
> 0siemens.com%7C8d77413c796a4920807b08dbc426e043%7C38ae3bcd95794fd4adda
> b42e1495d55a%7C1%7C0%7C638319443476294267%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%
> 7C%7C%7C&sdata=GrO630mertcLooKbWBUDIS57z7VppVi1ntvVyayCOs4%3D&reserved
> =0
>  and BUILD_ARCH being empty according to 'bitbake -e'. 
> 
> I changed the line to HOST_ARCH (+expand=True) which seems to work but 
> I do not have enough expertise to guess its correctness. Maybe the 
> line in native.bbclass covers this "if statement" already?
> 
> How to best resolve this?
> 
> 
Hello, Jan.

I've tried to repeat your steps and reproduced the issue in case of cross-building.

Changing the line to HOST_ARCH is techincally correct. But since we don't like forking (sstate.bbclass is taken from some OE branch without any changes), I think it would be better to just define BUILD_ARCH somewhere else in Isar to the value of HOST_ARCH, for example:

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 9eb93e2b..0d737b64 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -90,6 +90,7 @@ NATIVELSBSTRING ?= "isarnative"
 TARGET_VENDOR = ""
 TARGET_OS = "isar"
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
+BUILD_ARCH ?= "${HOST_ARCH}"
 
 # Isar apt repository paths
 REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"

This seems to work and doesn't fork sstate.bbclass, so it can be later updated to some newer OE revision.

Unfortunately, HOST_ARCH has different meanings in Isar and Yocto...

> [This is my first message to this mailing list, please correct me if I 
> am doing something wrong]
> 
> 
> Kind Regards,
> Jan Waltl
> 
> Siemens Mobility, s.r.o.
> SMO RI LCE CZ SAF 1
> 
> 
> 

Re: SSTATE cache not working for -native recipes due to unexpanded variable

[MOESSBAUER, Felix]

On Fri, 2023-10-06 at 07:59 +0000, 'Waltl, Jan' via isar-users wrote:
> Hello Uladzimir,
> 
> Thank you very, the patch appears to fix the issue for caching of
> native builds for me - now SSTATE works with them.
> 
> Also, I see some related discussions on the mailing list I will try
> to follow.

I guess this is actually related to the inverse meaning of HOST_ARCH
and DISTRO_ARCH in Yocto and GNU / ISAR. By that, I vote for the patch
proposed by Uladzimir.

@Jan: Could you please send this patch to the ML and put Adriaan in CC,
as he implemented most of the sstate parts for ISAR.

Felix


> 
> Kind Regards
> Jan Waltl
> 
> Siemens Mobility, s.r.o.
> SMO RI LCE CZ SAF 1
> 
> 
> -----Original Message-----
> From: Uladzimir Bely <ubely@ilbers.de> 
> Sent: Tuesday, October 3, 2023 5:41 PM
> To: Waltl, Jan (SMO RI LCE CZ SAF 1) <jan.waltl@siemens.com>;
> isar-users@googlegroups.com
> Subject: Re: SSTATE cache not working for -native recipes due to
> unexpanded variable
> 
> On Tue, 2023-10-03 at 13:30 +0000, 'Waltl, Jan' via isar-users wrote:
> > Hello,
> > 
> > I noticed that for -native recipes, SSTATE_PKGARCH variable is not 
> > expanded in the .siginfo filename. On rebuild, SSTATE reports 
> > mismatches leading to rebuilds even though the recipe and the 
> > resulting stamps are exactly the same.
> > 
> > Reproduction steps:
> > 1. Checkout ISAR next 216f0721265fcfe965ac48b51a0128d612d89bf4.
> > 2. Run 'bitbake cowsay-native' from meta-isar layer  (I use KAS for
> > setting up the bitbake directory) 4. Look into sstate directory, I
> > see 
> > a files:
> > sstate-cache/31/1a/sstate:cowsay-
> > native::git:r0::10:311a86032fcbd98d2343b40447e0558b5aae410ff64f727a
> > e5
> > 6aa2952167f05e_patch.tar.zst.siginfo
> > sstate-cache/3b/24/sstate:cowsay-
> > native::git:r0::10:3b2452fa0b072ed1488aad9432257dff8bee324659b5f073
> > 8a
> > b224d82b1de2fc_unpack.tar.zst.siginfo
> > sstate-cache/d6/ab/sstate:cowsay-
> > native::git:r0::10:d6abcc3328b93b67b59ef9c0b21449188f1330bc76d6ee63
> > 72
> > c76e94bec87756_fetch.tar.zst.siginfo
> > sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> > isar:git:r0:${SSTATE_PKGARCH}:10:64de5a007fac75eebd427a2ea9156cf014
> > 27
> > 18f68362a06b51c65559fc8045b7_dpkg_build.tar.zst.siginfo'
> > sstate-cache/isarnative/64/de/sstate:cowsay-native:amd64-
> > isar:git:r0::10:64de5a007fac75eebd427a2ea9156cf0142718f68362a06b51c
> > 65
> > 559fc8045b7_dpkg_build.tar.zst
> > sstate-cache/isarnative/aa/07/sstate:cowsay-native:amd64-
> > isar:git:r0:${SSTATE_PKGARCH}:10:aa07cf2e6ff17f7004e6dc2b4e7a9a6123
> > b1
> > 27668c27d4c3cb29d0f7dc65ac5f_deploy_deb.tar.zst.siginfo'
> > sstate-cache/isarnative/c4/db/sstate:cowsay-native:amd64-
> > isar:git:r0:${SSTATE_PKGARCH}:10:c4dbfbe1c1696f2b690f15cbd179a5795f
> > e4
> > 8f05a30a76d20783c98786af6d4a_adjust_git.tar.zst.siginfo'
> > sstate-cache/isarnative/d0/0c/sstate:cowsay-native:amd64-
> > isar:git:r0:${SSTATE_PKGARCH}:10:d00c69fa102b131bc09fb27aa8259fcb44
> > 7d
> > be5b51d5f6723d1e12ff43f92fee_transform_template.tar.zst.siginfo'
> > sstate-cache/isarnative/e7/94/sstate:cowsay-native:amd64-
> > isar:git:r0:${SSTATE_PKGARCH}:10:e794e52fe10ccf60b559089c39eaf86a49
> > 90
> > a81991c9b1b0b2672721878a1b8b_prepare_build.tar.zst.siginfo'
> > 
> > 5. Remove stamps directory.
> > 6. Build again - SSTATE should be used for the package but it is
> > not 
> > and the tasks are run again.
> > 
> > I think it is related to
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Filbers%2Fisar%2Fblob%2F61086915e6c83fff22effa85cda64a2ac0c
> > 2f1
> > 00%2Fmeta%2Fclasses%2Fsstate.bbclass%23L136&data=05%7C01%7Cjan.walt
> > l%4
> > 0siemens.com%7C8d77413c796a4920807b08dbc426e043%7C38ae3bcd95794fd4a
> > dda
> > b42e1495d55a%7C1%7C0%7C638319443476294267%7CUnknown%7CTWFpbGZsb3d8e
> > yJW
> > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C30
> > 00%
> > 7C%7C%7C&sdata=GrO630mertcLooKbWBUDIS57z7VppVi1ntvVyayCOs4%3D&reser
> > ved
> > =0
> >  and BUILD_ARCH being empty according to 'bitbake -e'. 
> > 
> > I changed the line to HOST_ARCH (+expand=True) which seems to work
> > but 
> > I do not have enough expertise to guess its correctness. Maybe the 
> > line in native.bbclass covers this "if statement" already?
> > 
> > How to best resolve this?
> > 
> > 
> Hello, Jan.
> 
> I've tried to repeat your steps and reproduced the issue in case of
> cross-building.
> 
> Changing the line to HOST_ARCH is techincally correct. But since we
> don't like forking (sstate.bbclass is taken from some OE branch
> without any changes), I think it would be better to just define
> BUILD_ARCH somewhere else in Isar to the value of HOST_ARCH, for
> example:
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index
> 9eb93e2b..0d737b64 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -90,6 +90,7 @@ NATIVELSBSTRING ?= "isarnative"
>  TARGET_VENDOR = ""
>  TARGET_OS = "isar"
>  PACKAGE_ARCH ?= "${DISTRO_ARCH}"
> +BUILD_ARCH ?= "${HOST_ARCH}"
>  
>  # Isar apt repository paths
>  REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-
> ${DISTRO_ARCH}/apt"
> 
> This seems to work and doesn't fork sstate.bbclass, so it can be
> later updated to some newer OE revision.
> 
> Unfortunately, HOST_ARCH has different meanings in Isar and Yocto...
> 
> > [This is my first message to this mailing list, please correct me
> > if I 
> > am doing something wrong]
> > 
> > 
> > Kind Regards,
> > Jan Waltl
> > 
> > Siemens Mobility, s.r.o.
> > SMO RI LCE CZ SAF 1
> > 
> > 
> > 
> 

Re: SSTATE cache not working for -native recipes due to unexpanded variable

[Jan Kiszka]

On 07.10.23 08:05, 'MOESSBAUER, Felix' via isar-users wrote:
> On Fri, 2023-10-06 at 07:59 +0000, 'Waltl, Jan' via isar-users wrote:
>> Hello Uladzimir,
>>
>> Thank you very, the patch appears to fix the issue for caching of
>> native builds for me - now SSTATE works with them.
>>
>> Also, I see some related discussions on the mailing list I will try
>> to follow.
> 
> I guess this is actually related to the inverse meaning of HOST_ARCH
> and DISTRO_ARCH in Yocto and GNU / ISAR. By that, I vote for the patch
> proposed by Uladzimir.
> 
> @Jan: Could you please send this patch to the ML and put Adriaan in CC,
> as he implemented most of the sstate parts for ISAR.

Are you referring to
https://groups.google.com/d/msgid/isar-users/60dba7199afdc8b71494b55081b1193e091b9bde.1696606502.git.jan.kiszka%40siemens.com
?

Jan

-- 
Siemens AG, Technology
Linux Expert Center