DISTRO is set too late

DISTRO is set too late

[Harald Seiler]

I am currently porting our layer to `next`.  While doing so, I got
a build failure from c8e63614 (base-apt: Introduce base implementation):

bb.data_smart.ExpansionError: Failure expanding variable DISTRO_SUITE,
[...] triggered Exception IndexError: list index out of range

The offending line looks like this (base-apt-helper.bbclass):

DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"

This would work perfectly if $DISTRO was already set to the distro from
multiconf, but because that seems to happen later on, $DISTRO is still
`poky` at this point.

Am I doing something wrong or does this need to be fixed upstream?

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws@denx.de 

Re: DISTRO is set too late

[Maxim Yu. Osipov]

Hi Harald,

On 10/23/18 12:05 PM, Harald Seiler wrote:
> I am currently porting our layer to `next`.  While doing so, I got
> a build failure from c8e63614 (base-apt: Introduce base implementation):
> 
> bb.data_smart.ExpansionError: Failure expanding variable DISTRO_SUITE,
> [...] triggered Exception IndexError: list index out of range
> 
> The offending line looks like this (base-apt-helper.bbclass):
> 
> DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
> 
> This would work perfectly if $DISTRO was already set to the distro from
> multiconf, but because that seems to happen later on, $DISTRO is still
> `poky` at this point.

> Am I doing something wrong or does this need to be fixed upstream?

The list of supported distros is under isar/meta-isar/conf/distro/*.conf
(debian-wheezy is officially not supported but still present for 
convenience).

(see 
https://github.com/ilbers/isar/blob/master/doc/user_manual.md#localconf 
for more detail)

How did you get the value `poky` for DISTRO in your environment?

Kind regards,
Maxim.


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: DISTRO is set too late

[Harald Seiler]

On Tue, 2018-10-23 at 12:48 +0300, Maxim Yu. Osipov wrote:
> Hi Harald,
> 
> The list of supported distros is under isar/meta-isar/conf/distro/*.conf
> (debian-wheezy is officially not supported but still present for 
> convenience).

We are using `debian-stretch`.  If I add

    DISTRO = "debian-stretch"

to `local.conf`, I can build.  But this should not be necessary as
$DISTRO is also set from multiconf.  The issue is that this happens
too late, so when $DISTRO_SUITE is expanded, it hasn't happened yet.

> How did you get the value `poky` for DISTRO in your environment?

Hmm, apparently this is kas adding

    DISTRO ??= "poky"

to `local.conf` if no distro is supplied in its config.  This should,
again, not be an issue, because the multiconf sets

    DISTRO ?= "debian-stretch"

What seems to go wrong is that the multiconf is read too late, so
that base-apt-helper.bb tries to expand $DISTRO before is has gotten
its final value.  I don't think this is the intended behavior ...

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws@denx.de 

Re: DISTRO is set too late

[Maxim Yu. Osipov]

On 10/23/18 1:10 PM, Harald Seiler wrote:
> On Tue, 2018-10-23 at 12:48 +0300, Maxim Yu. Osipov wrote:
>> Hi Harald,
>>
>> The list of supported distros is under isar/meta-isar/conf/distro/*.conf
>> (debian-wheezy is officially not supported but still present for
>> convenience).
> 
> We are using `debian-stretch`.  If I add
> 
>      DISTRO = "debian-stretch"
> 
> to `local.conf`, I can build.  But this should not be necessary as
> $DISTRO is also set from multiconf.  The issue is that this happens
> too late, so when $DISTRO_SUITE is expanded, it hasn't happened yet.
> 
>> How did you get the value `poky` for DISTRO in your environment?
> 
> Hmm, apparently this is kas adding
> 
>      DISTRO ??= "poky"
> 
> to `local.conf` if no distro is supplied in its config.  This should,
> again, not be an issue, because the multiconf sets
> 
>      DISTRO ?= "debian-stretch"
> 
> What seems to go wrong is that the multiconf is read too late, so
> that base-apt-helper.bb tries to expand $DISTRO before is has gotten
> its final value.  I don't think this is the intended behavior ...
> 

I don't use kas.

I've ran `bitbake -e multiconfig:qemuarm-stretch:isar-image-base` and 
DISTRO/DISTRO_SUITE are expanded properly:

# $DISTRO [2 operations]
#   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
#     [_defaultval] "debian-stretch"
#   set? 
/home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
#     "debian-stretch"
# pre-expansion value:
#   "debian-stretch"
DISTRO="debian-stretch"

...

#
# $DISTRO_SUITE
#   set? 
/home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
#     "${@ d.getVar('DISTRO', True).split('-')[1]}"
DISTRO_SUITE="stretch"



If I set in local.conf

DISTRO ??= "debian-jessie"

DISTRO/DISTRO_SUITE are also expanded properly:

# $DISTRO [2 operations]
#   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
#     [_defaultval] "debian-jessie"
#   set? 
/home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
#     "debian-stretch"
# pre-expansion value:
#   "debian-stretch"
DISTRO="debian-stretch"

...

#
# $DISTRO_SUITE
#   set? 
/home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
#     "${@ d.getVar('DISTRO', True).split('-')[1]}"
DISTRO_SUITE="stretch"


Kind regards,
Maxim.


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: DISTRO is set too late

[Harald Seiler]

On Tue, 2018-10-23 at 14:34 +0300, Maxim Yu. Osipov wrote:
> On 10/23/18 1:10 PM, Harald Seiler wrote:
> > On Tue, 2018-10-23 at 12:48 +0300, Maxim Yu. Osipov wrote:
> > > Hi Harald,
> > > 
> > > The list of supported distros is under isar/meta-isar/conf/distro/*.conf
> > > (debian-wheezy is officially not supported but still present for
> > > convenience).
> > 
> > We are using `debian-stretch`.  If I add
> > 
> >      DISTRO = "debian-stretch"
> > 
> > to `local.conf`, I can build.  But this should not be necessary as
> > $DISTRO is also set from multiconf.  The issue is that this happens
> > too late, so when $DISTRO_SUITE is expanded, it hasn't happened yet.
> > 
> > > How did you get the value `poky` for DISTRO in your environment?
> > 
> > Hmm, apparently this is kas adding
> > 
> >      DISTRO ??= "poky"
> > 
> > to `local.conf` if no distro is supplied in its config.  This should,
> > again, not be an issue, because the multiconf sets
> > 
> >      DISTRO ?= "debian-stretch"
> > 
> > What seems to go wrong is that the multiconf is read too late, so
> > that base-apt-helper.bb tries to expand $DISTRO before is has gotten
> > its final value.  I don't think this is the intended behavior ...
> > 
> 
> I don't use kas.
> 
> I've ran `bitbake -e multiconfig:qemuarm-stretch:isar-image-base` and 
> DISTRO/DISTRO_SUITE are expanded properly:
> 
> # $DISTRO [2 operations]
> #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
> #     [_defaultval] "debian-stretch"
> #   set? 
> /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
> #     "debian-stretch"
> # pre-expansion value:
> #   "debian-stretch"
> DISTRO="debian-stretch"
> 
> ...
> 
> #
> # $DISTRO_SUITE
> #   set? 
> /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
> #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
> DISTRO_SUITE="stretch"
> 
> 
> 
> If I set in local.conf
> 
> DISTRO ??= "debian-jessie"
> 
> DISTRO/DISTRO_SUITE are also expanded properly:
> 
> # $DISTRO [2 operations]
> #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
> #     [_defaultval] "debian-jessie"
> #   set? 
> /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
> #     "debian-stretch"
> # pre-expansion value:
> #   "debian-stretch"
> DISTRO="debian-stretch"
> 
> ...
> 
> #
> # $DISTRO_SUITE
> #   set? 
> /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
> #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
> DISTRO_SUITE="stretch"

I did a little more testing.  Apparently, it works fine if the weak default value
contains at least one `-`.  The rest of the value can be anything (though I do get
some warnings if it is not a valid distro).

So this works:

    DISTRO ??= "invalid-distro"

But this doesn't:

    DISTRO ??= "invalid_distro"


-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws@denx.de 

Re: DISTRO is set too late

[Jan Kiszka]

On 23.10.18 13:37, Harald Seiler wrote:
> On Tue, 2018-10-23 at 14:34 +0300, Maxim Yu. Osipov wrote:
>> On 10/23/18 1:10 PM, Harald Seiler wrote:
>>> On Tue, 2018-10-23 at 12:48 +0300, Maxim Yu. Osipov wrote:
>>>> Hi Harald,
>>>>
>>>> The list of supported distros is under isar/meta-isar/conf/distro/*.conf
>>>> (debian-wheezy is officially not supported but still present for
>>>> convenience).
>>>
>>> We are using `debian-stretch`.  If I add
>>>
>>>       DISTRO = "debian-stretch"
>>>
>>> to `local.conf`, I can build.  But this should not be necessary as
>>> $DISTRO is also set from multiconf.  The issue is that this happens
>>> too late, so when $DISTRO_SUITE is expanded, it hasn't happened yet.
>>>
>>>> How did you get the value `poky` for DISTRO in your environment?
>>>
>>> Hmm, apparently this is kas adding
>>>
>>>       DISTRO ??= "poky"
>>>
>>> to `local.conf` if no distro is supplied in its config.  This should,
>>> again, not be an issue, because the multiconf sets
>>>
>>>       DISTRO ?= "debian-stretch"
>>>
>>> What seems to go wrong is that the multiconf is read too late, so
>>> that base-apt-helper.bb tries to expand $DISTRO before is has gotten
>>> its final value.  I don't think this is the intended behavior ...
>>>
>>
>> I don't use kas.
>>
>> I've ran `bitbake -e multiconfig:qemuarm-stretch:isar-image-base` and
>> DISTRO/DISTRO_SUITE are expanded properly:
>>
>> # $DISTRO [2 operations]
>> #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
>> #     [_defaultval] "debian-stretch"
>> #   set?
>> /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
>> #     "debian-stretch"
>> # pre-expansion value:
>> #   "debian-stretch"
>> DISTRO="debian-stretch"
>>
>> ...
>>
>> #
>> # $DISTRO_SUITE
>> #   set?
>> /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
>> #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
>> DISTRO_SUITE="stretch"
>>
>>
>>
>> If I set in local.conf
>>
>> DISTRO ??= "debian-jessie"
>>
>> DISTRO/DISTRO_SUITE are also expanded properly:
>>
>> # $DISTRO [2 operations]
>> #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
>> #     [_defaultval] "debian-jessie"
>> #   set?
>> /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
>> #     "debian-stretch"
>> # pre-expansion value:
>> #   "debian-stretch"
>> DISTRO="debian-stretch"
>>
>> ...
>>
>> #
>> # $DISTRO_SUITE
>> #   set?
>> /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
>> #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
>> DISTRO_SUITE="stretch"
> 
> I did a little more testing.  Apparently, it works fine if the weak default value
> contains at least one `-`.  The rest of the value can be anything (though I do get
> some warnings if it is not a valid distro).
> 
> So this works:
> 
>      DISTRO ??= "invalid-distro"
> 
> But this doesn't:
> 
>      DISTRO ??= "invalid_distro"
> 
> 

I didn't dig into all details, and maybe this is actually resolving the issue 
better, but I had to fix a similar issue with my kas-driven layer lik this: 
https://github.com/siemens/jailhouse-images/commit/e595c6d62ddc6e8a608b199dc767dab9bc12f216

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: DISTRO is set too late

[Harald Seiler]

On Thu, 2018-10-25 at 14:21 +0100, Jan Kiszka wrote:
> On 23.10.18 13:37, Harald Seiler wrote:
> > On Tue, 2018-10-23 at 14:34 +0300, Maxim Yu. Osipov wrote:
> > > On 10/23/18 1:10 PM, Harald Seiler wrote:
> > > > On Tue, 2018-10-23 at 12:48 +0300, Maxim Yu. Osipov wrote:
> > > > > Hi Harald,
> > > > > 
> > > > > The list of supported distros is under isar/meta-isar/conf/distro/*.conf
> > > > > (debian-wheezy is officially not supported but still present for
> > > > > convenience).
> > > > 
> > > > We are using `debian-stretch`.  If I add
> > > > 
> > > >       DISTRO = "debian-stretch"
> > > > 
> > > > to `local.conf`, I can build.  But this should not be necessary as
> > > > $DISTRO is also set from multiconf.  The issue is that this happens
> > > > too late, so when $DISTRO_SUITE is expanded, it hasn't happened yet.
> > > > 
> > > > > How did you get the value `poky` for DISTRO in your environment?
> > > > 
> > > > Hmm, apparently this is kas adding
> > > > 
> > > >       DISTRO ??= "poky"
> > > > 
> > > > to `local.conf` if no distro is supplied in its config.  This should,
> > > > again, not be an issue, because the multiconf sets
> > > > 
> > > >       DISTRO ?= "debian-stretch"
> > > > 
> > > > What seems to go wrong is that the multiconf is read too late, so
> > > > that base-apt-helper.bb tries to expand $DISTRO before is has gotten
> > > > its final value.  I don't think this is the intended behavior ...
> > > > 
> > > 
> > > I don't use kas.
> > > 
> > > I've ran `bitbake -e multiconfig:qemuarm-stretch:isar-image-base` and
> > > DISTRO/DISTRO_SUITE are expanded properly:
> > > 
> > > # $DISTRO [2 operations]
> > > #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
> > > #     [_defaultval] "debian-stretch"
> > > #   set?
> > > /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
> > > #     "debian-stretch"
> > > # pre-expansion value:
> > > #   "debian-stretch"
> > > DISTRO="debian-stretch"
> > > 
> > > ...
> > > 
> > > #
> > > # $DISTRO_SUITE
> > > #   set?
> > > /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
> > > #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
> > > DISTRO_SUITE="stretch"
> > > 
> > > 
> > > 
> > > If I set in local.conf
> > > 
> > > DISTRO ??= "debian-jessie"
> > > 
> > > DISTRO/DISTRO_SUITE are also expanded properly:
> > > 
> > > # $DISTRO [2 operations]
> > > #   set /home/myo/work/isar/src/trunk/isar/build/conf/local.conf:30
> > > #     [_defaultval] "debian-jessie"
> > > #   set?
> > > /home/myo/work/isar/src/trunk/isar/meta-isar/conf/multiconfig/qemuarm-stretch.conf:6
> > > #     "debian-stretch"
> > > # pre-expansion value:
> > > #   "debian-stretch"
> > > DISTRO="debian-stretch"
> > > 
> > > ...
> > > 
> > > #
> > > # $DISTRO_SUITE
> > > #   set?
> > > /home/myo/work/isar/src/trunk/isar/meta/classes/base-apt-helper.bbclass:5
> > > #     "${@ d.getVar('DISTRO', True).split('-')[1]}"
> > > DISTRO_SUITE="stretch"
> > 
> > I did a little more testing.  Apparently, it works fine if the weak default value
> > contains at least one `-`.  The rest of the value can be anything (though I do get
> > some warnings if it is not a valid distro).
> > 
> > So this works:
> > 
> >      DISTRO ??= "invalid-distro"
> > 
> > But this doesn't:
> > 
> >      DISTRO ??= "invalid_distro"
> > 
> > 
> 
> I didn't dig into all details, and maybe this is actually resolving the issue 
> better, but I had to fix a similar issue with my kas-driven layer lik this: 
> https://github.com/siemens/jailhouse-images/commit/e595c6d62ddc6e8a608b199dc767dab9bc12f216

I am also able to reproduce this behavior with poky/sumo, so this is a bitbake issue and
not directly isar related.

I am unsure if it makes sense to follow up on this; this bug will only ever be triggered
if an invalid default value is provided, which is by itself a "bug" because a default
value that is supposed to be overwritten shouldn't be set in the first place ...

What do you think?

-- 
Harald

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62  Fax: +49-8142-66989-80   Email: hws@denx.de