* isar-bootstrap fails to rebuild after config changes
@ 2019-08-12 17:18 Jan Kiszka
2019-08-13 8:01 ` Claudius Heine
0 siblings, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2019-08-12 17:18 UTC (permalink / raw)
To: isar-users, Claudius Heine
Hi,
I noticed today that I'm not getting a new bootstrap rootfs after adding a repo
to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN setting,
but also more:
Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in
do_apt_config_prepare(), Claudius? This obviously exists after a first run, and
that blocks refreshing all the apt settings if they have changed.
There is CLEANFUNCS set with clean_deploy(), but that is only run on true
do_clean(), not on rebuilds.
And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it protects
checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But
do_apt_config_prepare() runs lockless. This cannot work reliably.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes
2019-08-12 17:18 isar-bootstrap fails to rebuild after config changes Jan Kiszka
@ 2019-08-13 8:01 ` Claudius Heine
2019-08-13 8:14 ` Jan Kiszka
0 siblings, 1 reply; 6+ messages in thread
From: Claudius Heine @ 2019-08-13 8:01 UTC (permalink / raw)
To: [ext] Jan Kiszka, isar-users, Claudius Heine
Hi Jan,
On 12/08/2019 19.18, [ext] Jan Kiszka wrote:
> Hi,
>
> I noticed today that I'm not getting a new bootstrap rootfs after adding
> a repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing
> STAMPCLEAN setting, but also more:
>
> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link
> in do_apt_config_prepare(), Claudius? This obviously exists after a
> first run, and that blocks refreshing all the apt settings if they have
> changed.
I do not know. That was added by a patch of Maxim
(ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows.
isar-bootstrap: Eliminate multiple debootstraps for the same
distro/host
After applying this patch only single debootstrap for
particular platform is created and pointed by corresponding
link under DEPLOY_DIR_IMAGE.
For proper locking of parallel builds shell tasks
apt_config_install, set_locale, setup_chroot, apt_update
and deploy were collapsed to common debootstrap helper.
Due to problems in bitbake with shell functions expansion
under quotes these functions were substituted by their
bodies.
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
That is by the way a pretty big patch. Also parts of it could probably
be reverted, since the 'problems in bitbake with shell functions
expansion under quotes' are resolved in the current version by removing
these quotes IIUC.
> There is CLEANFUNCS set with clean_deploy(), but that is only run on
> true do_clean(), not on rebuilds.
Right.
>
> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it
> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But
> do_apt_config_prepare() runs lockless. This cannot work reliably.
When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC
the locks where added when I wasn't active with isar and I haven't yet
looked into why exactly they are needed and what they protect against
etc. So you know probably more about them than me.
regards,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes
2019-08-13 8:01 ` Claudius Heine
@ 2019-08-13 8:14 ` Jan Kiszka
2019-08-20 19:33 ` Jan Kiszka
0 siblings, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2019-08-13 8:14 UTC (permalink / raw)
To: Claudius Heine, isar-users, Claudius Heine, Maksim Osipov
On 13.08.19 10:01, Claudius Heine wrote:
> Hi Jan,
>
> On 12/08/2019 19.18, [ext] Jan Kiszka wrote:
>> Hi,
>>
>> I noticed today that I'm not getting a new bootstrap rootfs after adding a
>> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN
>> setting, but also more:
>>
>> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in
>> do_apt_config_prepare(), Claudius? This obviously exists after a first run,
>> and that blocks refreshing all the apt settings if they have changed.
>
> I do not know. That was added by a patch of Maxim
> (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows.
>
> isar-bootstrap: Eliminate multiple debootstraps for the same distro/host
>
> After applying this patch only single debootstrap for
> particular platform is created and pointed by corresponding
> link under DEPLOY_DIR_IMAGE.
>
> For proper locking of parallel builds shell tasks
> apt_config_install, set_locale, setup_chroot, apt_update
> and deploy were collapsed to common debootstrap helper.
> Due to problems in bitbake with shell functions expansion
> under quotes these functions were substituted by their
> bodies.
>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
>
> That is by the way a pretty big patch. Also parts of it could probably be
> reverted, since the 'problems in bitbake with shell functions expansion under
> quotes' are resolved in the current version by removing these quotes IIUC.
>
>> There is CLEANFUNCS set with clean_deploy(), but that is only run on true
>> do_clean(), not on rebuilds.
>
> Right.
>
>>
>> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it
>> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But
>> do_apt_config_prepare() runs lockless. This cannot work reliably.
>
> When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the
> locks where added when I wasn't active with isar and I haven't yet looked into
> why exactly they are needed and what they protect against etc. So you know
> probably more about them than me.
Ah, sorry, wrongly accounted you for this.
Maxim, can you comment on the design idea and the issues I found?
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes
2019-08-13 8:14 ` Jan Kiszka
@ 2019-08-20 19:33 ` Jan Kiszka
2019-08-21 9:02 ` Baurzhan Ismagulov
0 siblings, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2019-08-20 19:33 UTC (permalink / raw)
To: Claudius Heine, isar-users, Claudius Heine, Maksim Osipov
On 13.08.19 10:14, [ext] Jan Kiszka wrote:
> On 13.08.19 10:01, Claudius Heine wrote:
>> Hi Jan,
>>
>> On 12/08/2019 19.18, [ext] Jan Kiszka wrote:
>>> Hi,
>>>
>>> I noticed today that I'm not getting a new bootstrap rootfs after adding a
>>> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN
>>> setting, but also more:
>>>
>>> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in
>>> do_apt_config_prepare(), Claudius? This obviously exists after a first run,
>>> and that blocks refreshing all the apt settings if they have changed.
>>
>> I do not know. That was added by a patch of Maxim
>> (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows.
>>
>> isar-bootstrap: Eliminate multiple debootstraps for the same distro/host
>>
>> After applying this patch only single debootstrap for
>> particular platform is created and pointed by corresponding
>> link under DEPLOY_DIR_IMAGE.
>>
>> For proper locking of parallel builds shell tasks
>> apt_config_install, set_locale, setup_chroot, apt_update
>> and deploy were collapsed to common debootstrap helper.
>> Due to problems in bitbake with shell functions expansion
>> under quotes these functions were substituted by their
>> bodies.
>>
>> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
>>
>> That is by the way a pretty big patch. Also parts of it could probably be
>> reverted, since the 'problems in bitbake with shell functions expansion under
>> quotes' are resolved in the current version by removing these quotes IIUC.
>>
>>> There is CLEANFUNCS set with clean_deploy(), but that is only run on true
>>> do_clean(), not on rebuilds.
>>
>> Right.
>>
>>>
>>> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it
>>> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But
>>> do_apt_config_prepare() runs lockless. This cannot work reliably.
>>
>> When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the
>> locks where added when I wasn't active with isar and I haven't yet looked into
>> why exactly they are needed and what they protect against etc. So you know
>> probably more about them than me.
>
> Ah, sorry, wrongly accounted you for this.
>
> Maxim, can you comment on the design idea and the issues I found?
>
Actually, I do not even see the point of locking here: If a bootstrap target is
shared, only one multiconfig target will request its build and the others will
wait automatically for the result. That's what bitbake is supposed to do for us.
If it's not shared, there is no conflict to use locks on. We only need locks if
different tasks of different recipes compete for common resources.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes
2019-08-20 19:33 ` Jan Kiszka
@ 2019-08-21 9:02 ` Baurzhan Ismagulov
2019-08-21 9:06 ` Jan Kiszka
0 siblings, 1 reply; 6+ messages in thread
From: Baurzhan Ismagulov @ 2019-08-21 9:02 UTC (permalink / raw)
To: isar-users
On Tue, Aug 20, 2019 at 09:33:29PM +0200, Jan Kiszka wrote:
> Actually, I do not even see the point of locking here: If a bootstrap target
> is shared, only one multiconfig target will request its build and the others
> will wait automatically for the result. That's what bitbake is supposed to
> do for us. If it's not shared, there is no conflict to use locks on. We only
> need locks if different tasks of different recipes compete for common
> resources.
There were build issues. Maybe that is the case when the host and the target
arch are the same. We might be able to reproduce that by trying to build
2c99c8d.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes
2019-08-21 9:02 ` Baurzhan Ismagulov
@ 2019-08-21 9:06 ` Jan Kiszka
0 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2019-08-21 9:06 UTC (permalink / raw)
To: isar-users
On 21.08.19 11:02, Baurzhan Ismagulov wrote:
> On Tue, Aug 20, 2019 at 09:33:29PM +0200, Jan Kiszka wrote:
>> Actually, I do not even see the point of locking here: If a bootstrap target
>> is shared, only one multiconfig target will request its build and the others
>> will wait automatically for the result. That's what bitbake is supposed to
>> do for us. If it's not shared, there is no conflict to use locks on. We only
>> need locks if different tasks of different recipes compete for common
>> resources.
>
> There were build issues. Maybe that is the case when the host and the target
> arch are the same. We might be able to reproduce that by trying to build
> 2c99c8d.
I will rather do this on top of my current queue which cleans up a lot and
possibly also fixes false sharing of build stamps (due to improper STAMP
setting). But I'm still fighting with other issues, will share some soon.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-08-21 9:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-12 17:18 isar-bootstrap fails to rebuild after config changes Jan Kiszka
2019-08-13 8:01 ` Claudius Heine
2019-08-13 8:14 ` Jan Kiszka
2019-08-20 19:33 ` Jan Kiszka
2019-08-21 9:02 ` Baurzhan Ismagulov
2019-08-21 9:06 ` Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox