Conditional weak assignments behave unexpectedly

Conditional weak assignments behave unexpectedly

[Moessbauer, Felix]

Dear developers,

I just stumbled upon the following pattern, which needs further
investigation:

- use the qemuamd64 machine on debian-bullseye
- define a custom WKS_FILE in local.conf

While the qemuamd64.conf file contains the following line:
WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"
I expected that I'm able to override that unconditionally in the
local.conf using: WKS_FILE = "foo.wks"

However, the final value of the WKS_FILE variable is "sdimage-efi-
btrfs". In the bitbake documentation, I cannot find any example of
combining weak assignments and overrides. However, the implementation
seems to evaluate the weak assignment on the override first. Then, the
WKS_FILE part from the local conf is evaluated and finally the
WKS_FILE:debian-bullseye overrides what was in WKS_FILE. This suspicion
is also backed by the following test:

local.conf: WKS_FILE:debian-bullseye = "foo.wks"
Now, the final value of WKS_FILE is "foo.wks".

I'm unsure if that has ever been like that and was just unnoticed, or
if the behavior change was introduced in bitbake 2.0.
Anyways, we must not use overrides in combination with weak
assignments. And we also need to fix that on the qemuamd64 machine
config immediately (even if we loose some tests by doing that).
Otherwise it is impossible to override the WKS_FILE of the qemuamd64
machine.

Best regards,
Felix

Re: Conditional weak assignments behave unexpectedly

[Moessbauer, Felix]

On Sun, 2023-04-23 at 07:48 +0000, 'Moessbauer, Felix' via isar-users
wrote:
> Dear developers,
> 
> I just stumbled upon the following pattern, which needs further
> investigation:
> 
> - use the qemuamd64 machine on debian-bullseye
> - define a custom WKS_FILE in local.conf
> 
> While the qemuamd64.conf file contains the following line:
> WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"
> I expected that I'm able to override that unconditionally in the
> local.conf using: WKS_FILE = "foo.wks"
> 
> However, the final value of the WKS_FILE variable is "sdimage-efi-
> btrfs". In the bitbake documentation, I cannot find any example of
> combining weak assignments and overrides. However, the implementation
> seems to evaluate the weak assignment on the override first. Then,
> the
> WKS_FILE part from the local conf is evaluated and finally the
> WKS_FILE:debian-bullseye overrides what was in WKS_FILE. This
> suspicion
> is also backed by the following test:
> 
> local.conf: WKS_FILE:debian-bullseye = "foo.wks"
> Now, the final value of WKS_FILE is "foo.wks".
> 
> I'm unsure if that has ever been like that and was just unnoticed, or
> if the behavior change was introduced in bitbake 2.0.
> Anyways, we must not use overrides in combination with weak
> assignments. And we also need to fix that on the qemuamd64 machine
> config immediately (even if we loose some tests by doing that).
> Otherwise it is impossible to override the WKS_FILE of the qemuamd64
> machine.

I just checked where this pattern is used in meta and meta-isar. There
are quite some occurrences. However, all except for the WKS_FILE seem
to be never customized anyways. With ack and this regex, you can check
by yourself (also in downstream layers).

Felix

ack '[A-Z]+:[a-z0-9-]+\s*\?=' meta meta-isar/
meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
21:SBUILD_CHROOT_PREINSTALL:riscv64 ?= " \

meta/conf/bitbake.conf
66:KERNEL_FILE:mipsel ?= "vmlinux"
67:KERNEL_FILE:riscv64 ?= "vmlinux"
68:KERNEL_FILE:arm64 ?= "vmlinux"

meta-isar/conf/machine/qemuarm64.conf
7:KERNEL_NAME:ubuntu-focal ?= "generic"

meta-isar/conf/machine/qemuamd64.conf
7:KERNEL_NAME:ubuntu-focal ?= "generic"
13:WKS_FILE:ubuntu-focal ?= "sdimage-efi-sd"
16:WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"

meta-isar/conf/distro/ubuntu-focal.conf
13:DISTRO_APT_SOURCES:arm64 ?= "conf/distro/${BASE_DISTRO}-
${BASE_DISTRO_CODENAME}-ports.list"
14:HOST_DISTRO_APT_SOURCES:arm64 ?= "conf/distro/${HOST_DISTRO}.list
conf/distro/${HOST_DISTRO}-ports.list"

meta-isar/conf/distro/raspios-bullseye.conf
16:DISTRO_APT_SOURCES:arm64 ?= "conf/distro/raspios-bullseye-64.list"

> 
> Best regards,
> Felix
> 

Re: Conditional weak assignments behave unexpectedly

[Henning Schild]

Am Sun, 23 Apr 2023 11:11:04 +0200
schrieb "Moessbauer, Felix (T CED INW-CN)"
<felix.moessbauer@siemens.com>:

> On Sun, 2023-04-23 at 07:48 +0000, 'Moessbauer, Felix' via isar-users
> wrote:
> > Dear developers,
> >
> > I just stumbled upon the following pattern, which needs further
> > investigation:
> >
> > - use the qemuamd64 machine on debian-bullseye
> > - define a custom WKS_FILE in local.conf
> >
> > While the qemuamd64.conf file contains the following line:
> > WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"
> > I expected that I'm able to override that unconditionally in the
> > local.conf using: WKS_FILE = "foo.wks"
> >
> > However, the final value of the WKS_FILE variable is "sdimage-efi-
> > btrfs". In the bitbake documentation, I cannot find any example of
> > combining weak assignments and overrides. However, the
> > implementation seems to evaluate the weak assignment on the
> > override first. Then, the
> > WKS_FILE part from the local conf is evaluated and finally the
> > WKS_FILE:debian-bullseye overrides what was in WKS_FILE. This
> > suspicion
> > is also backed by the following test:
> >
> > local.conf: WKS_FILE:debian-bullseye = "foo.wks"
> > Now, the final value of WKS_FILE is "foo.wks".
> >
> > I'm unsure if that has ever been like that and was just unnoticed,
> > or if the behavior change was introduced in bitbake 2.0.
> > Anyways, we must not use overrides in combination with weak
> > assignments. And we also need to fix that on the qemuamd64 machine
> > config immediately (even if we loose some tests by doing that).
> > Otherwise it is impossible to override the WKS_FILE of the qemuamd64
> > machine.  
> 
> I just checked where this pattern is used in meta and meta-isar. There
> are quite some occurrences. However, all except for the WKS_FILE seem
> to be never customized anyways. With ack and this regex, you can check
> by yourself (also in downstream layers).

That distro-suite using another WKS is also just a result of wanting to
test things better in CI. Just to have one target where btrfs is
actually used so we can test the resize in CI.

meta-isar is really not meant to be used and based on as it is, it is a
collection of examples intermixed with CI ... not a good base layer

Henning
 
> Felix
> 
> ack '[A-Z]+:[a-z0-9-]+\s*\?=' meta meta-isar/
> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
> 21:SBUILD_CHROOT_PREINSTALL:riscv64 ?= " \
> 
> meta/conf/bitbake.conf
> 66:KERNEL_FILE:mipsel ?= "vmlinux"
> 67:KERNEL_FILE:riscv64 ?= "vmlinux"
> 68:KERNEL_FILE:arm64 ?= "vmlinux"
> 
> meta-isar/conf/machine/qemuarm64.conf
> 7:KERNEL_NAME:ubuntu-focal ?= "generic"
> 
> meta-isar/conf/machine/qemuamd64.conf
> 7:KERNEL_NAME:ubuntu-focal ?= "generic"
> 13:WKS_FILE:ubuntu-focal ?= "sdimage-efi-sd"
> 16:WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"
> 
> meta-isar/conf/distro/ubuntu-focal.conf
> 13:DISTRO_APT_SOURCES:arm64 ?= "conf/distro/${BASE_DISTRO}-
> ${BASE_DISTRO_CODENAME}-ports.list"
> 14:HOST_DISTRO_APT_SOURCES:arm64 ?= "conf/distro/${HOST_DISTRO}.list
> conf/distro/${HOST_DISTRO}-ports.list"
> 
> meta-isar/conf/distro/raspios-bullseye.conf
> 16:DISTRO_APT_SOURCES:arm64 ?= "conf/distro/raspios-bullseye-64.list"
> 
> >
> > Best regards,
> > Felix
> >  
> 

Re: Conditional weak assignments behave unexpectedly

[Anton Mikanovich]

23/04/2023 10:48, 'Moessbauer, Felix' via isar-users wrote:
> Dear developers,
>
> I just stumbled upon the following pattern, which needs further
> investigation:
>
> - use the qemuamd64 machine on debian-bullseye
> - define a custom WKS_FILE in local.conf
>
> While the qemuamd64.conf file contains the following line:
> WKS_FILE:debian-bullseye ?= "sdimage-efi-btrfs"
> I expected that I'm able to override that unconditionally in the
> local.conf using: WKS_FILE = "foo.wks"
>
> However, the final value of the WKS_FILE variable is "sdimage-efi-
> btrfs". In the bitbake documentation, I cannot find any example of
> combining weak assignments and overrides. However, the implementation
> seems to evaluate the weak assignment on the override first. Then, the
> WKS_FILE part from the local conf is evaluated and finally the
> WKS_FILE:debian-bullseye overrides what was in WKS_FILE. This suspicion
> is also backed by the following test:
>
> local.conf: WKS_FILE:debian-bullseye = "foo.wks"
> Now, the final value of WKS_FILE is "foo.wks".
>
> I'm unsure if that has ever been like that and was just unnoticed, or
> if the behavior change was introduced in bitbake 2.0.
> Anyways, we must not use overrides in combination with weak
> assignments. And we also need to fix that on the qemuamd64 machine
> config immediately (even if we loose some tests by doing that).
> Otherwise it is impossible to override the WKS_FILE of the qemuamd64
> machine.
>
> Best regards,
> Felix
>
Hello Felix,

As Henning already mentioned its all about CI usage.
We are already working on a solution that will move all that corner 
cases from
the layer to CI itself, so we can remove overrides.
This solution requires some testing and then I hope I'll send it to 
maillist.

Re: Conditional weak assignments behave unexpectedly

[Anton Mikanovich]

Hello Felix,

I've just pushed the first version of the fix patchset:

[PATCH v1 00/10] Move testsuite hacks to separate layer