* isar-bootstrap fails to rebuild after config changes @ 2019-08-12 17:18 Jan Kiszka 2019-08-13 8:01 ` Claudius Heine 0 siblings, 1 reply; 6+ messages in thread From: Jan Kiszka @ 2019-08-12 17:18 UTC (permalink / raw) To: isar-users, Claudius Heine Hi, I noticed today that I'm not getting a new bootstrap rootfs after adding a repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN setting, but also more: Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in do_apt_config_prepare(), Claudius? This obviously exists after a first run, and that blocks refreshing all the apt settings if they have changed. There is CLEANFUNCS set with clean_deploy(), but that is only run on true do_clean(), not on rebuilds. And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But do_apt_config_prepare() runs lockless. This cannot work reliably. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes 2019-08-12 17:18 isar-bootstrap fails to rebuild after config changes Jan Kiszka @ 2019-08-13 8:01 ` Claudius Heine 2019-08-13 8:14 ` Jan Kiszka 0 siblings, 1 reply; 6+ messages in thread From: Claudius Heine @ 2019-08-13 8:01 UTC (permalink / raw) To: [ext] Jan Kiszka, isar-users, Claudius Heine Hi Jan, On 12/08/2019 19.18, [ext] Jan Kiszka wrote: > Hi, > > I noticed today that I'm not getting a new bootstrap rootfs after adding > a repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing > STAMPCLEAN setting, but also more: > > Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link > in do_apt_config_prepare(), Claudius? This obviously exists after a > first run, and that blocks refreshing all the apt settings if they have > changed. I do not know. That was added by a patch of Maxim (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows. isar-bootstrap: Eliminate multiple debootstraps for the same distro/host After applying this patch only single debootstrap for particular platform is created and pointed by corresponding link under DEPLOY_DIR_IMAGE. For proper locking of parallel builds shell tasks apt_config_install, set_locale, setup_chroot, apt_update and deploy were collapsed to common debootstrap helper. Due to problems in bitbake with shell functions expansion under quotes these functions were substituted by their bodies. Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de> That is by the way a pretty big patch. Also parts of it could probably be reverted, since the 'problems in bitbake with shell functions expansion under quotes' are resolved in the current version by removing these quotes IIUC. > There is CLEANFUNCS set with clean_deploy(), but that is only run on > true do_clean(), not on rebuilds. Right. > > And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it > protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But > do_apt_config_prepare() runs lockless. This cannot work reliably. When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the locks where added when I wasn't active with isar and I haven't yet looked into why exactly they are needed and what they protect against etc. So you know probably more about them than me. regards, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes 2019-08-13 8:01 ` Claudius Heine @ 2019-08-13 8:14 ` Jan Kiszka 2019-08-20 19:33 ` Jan Kiszka 0 siblings, 1 reply; 6+ messages in thread From: Jan Kiszka @ 2019-08-13 8:14 UTC (permalink / raw) To: Claudius Heine, isar-users, Claudius Heine, Maksim Osipov On 13.08.19 10:01, Claudius Heine wrote: > Hi Jan, > > On 12/08/2019 19.18, [ext] Jan Kiszka wrote: >> Hi, >> >> I noticed today that I'm not getting a new bootstrap rootfs after adding a >> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN >> setting, but also more: >> >> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in >> do_apt_config_prepare(), Claudius? This obviously exists after a first run, >> and that blocks refreshing all the apt settings if they have changed. > > I do not know. That was added by a patch of Maxim > (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows. > > isar-bootstrap: Eliminate multiple debootstraps for the same distro/host > > After applying this patch only single debootstrap for > particular platform is created and pointed by corresponding > link under DEPLOY_DIR_IMAGE. > > For proper locking of parallel builds shell tasks > apt_config_install, set_locale, setup_chroot, apt_update > and deploy were collapsed to common debootstrap helper. > Due to problems in bitbake with shell functions expansion > under quotes these functions were substituted by their > bodies. > > Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de> > > That is by the way a pretty big patch. Also parts of it could probably be > reverted, since the 'problems in bitbake with shell functions expansion under > quotes' are resolved in the current version by removing these quotes IIUC. > >> There is CLEANFUNCS set with clean_deploy(), but that is only run on true >> do_clean(), not on rebuilds. > > Right. > >> >> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it >> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But >> do_apt_config_prepare() runs lockless. This cannot work reliably. > > When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the > locks where added when I wasn't active with isar and I haven't yet looked into > why exactly they are needed and what they protect against etc. So you know > probably more about them than me. Ah, sorry, wrongly accounted you for this. Maxim, can you comment on the design idea and the issues I found? Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes 2019-08-13 8:14 ` Jan Kiszka @ 2019-08-20 19:33 ` Jan Kiszka 2019-08-21 9:02 ` Baurzhan Ismagulov 0 siblings, 1 reply; 6+ messages in thread From: Jan Kiszka @ 2019-08-20 19:33 UTC (permalink / raw) To: Claudius Heine, isar-users, Claudius Heine, Maksim Osipov On 13.08.19 10:14, [ext] Jan Kiszka wrote: > On 13.08.19 10:01, Claudius Heine wrote: >> Hi Jan, >> >> On 12/08/2019 19.18, [ext] Jan Kiszka wrote: >>> Hi, >>> >>> I noticed today that I'm not getting a new bootstrap rootfs after adding a >>> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN >>> setting, but also more: >>> >>> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in >>> do_apt_config_prepare(), Claudius? This obviously exists after a first run, >>> and that blocks refreshing all the apt settings if they have changed. >> >> I do not know. That was added by a patch of Maxim >> (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows. >> >> isar-bootstrap: Eliminate multiple debootstraps for the same distro/host >> >> After applying this patch only single debootstrap for >> particular platform is created and pointed by corresponding >> link under DEPLOY_DIR_IMAGE. >> >> For proper locking of parallel builds shell tasks >> apt_config_install, set_locale, setup_chroot, apt_update >> and deploy were collapsed to common debootstrap helper. >> Due to problems in bitbake with shell functions expansion >> under quotes these functions were substituted by their >> bodies. >> >> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de> >> >> That is by the way a pretty big patch. Also parts of it could probably be >> reverted, since the 'problems in bitbake with shell functions expansion under >> quotes' are resolved in the current version by removing these quotes IIUC. >> >>> There is CLEANFUNCS set with clean_deploy(), but that is only run on true >>> do_clean(), not on rebuilds. >> >> Right. >> >>> >>> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it >>> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But >>> do_apt_config_prepare() runs lockless. This cannot work reliably. >> >> When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the >> locks where added when I wasn't active with isar and I haven't yet looked into >> why exactly they are needed and what they protect against etc. So you know >> probably more about them than me. > > Ah, sorry, wrongly accounted you for this. > > Maxim, can you comment on the design idea and the issues I found? > Actually, I do not even see the point of locking here: If a bootstrap target is shared, only one multiconfig target will request its build and the others will wait automatically for the result. That's what bitbake is supposed to do for us. If it's not shared, there is no conflict to use locks on. We only need locks if different tasks of different recipes compete for common resources. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes 2019-08-20 19:33 ` Jan Kiszka @ 2019-08-21 9:02 ` Baurzhan Ismagulov 2019-08-21 9:06 ` Jan Kiszka 0 siblings, 1 reply; 6+ messages in thread From: Baurzhan Ismagulov @ 2019-08-21 9:02 UTC (permalink / raw) To: isar-users On Tue, Aug 20, 2019 at 09:33:29PM +0200, Jan Kiszka wrote: > Actually, I do not even see the point of locking here: If a bootstrap target > is shared, only one multiconfig target will request its build and the others > will wait automatically for the result. That's what bitbake is supposed to > do for us. If it's not shared, there is no conflict to use locks on. We only > need locks if different tasks of different recipes compete for common > resources. There were build issues. Maybe that is the case when the host and the target arch are the same. We might be able to reproduce that by trying to build 2c99c8d. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: isar-bootstrap fails to rebuild after config changes 2019-08-21 9:02 ` Baurzhan Ismagulov @ 2019-08-21 9:06 ` Jan Kiszka 0 siblings, 0 replies; 6+ messages in thread From: Jan Kiszka @ 2019-08-21 9:06 UTC (permalink / raw) To: isar-users On 21.08.19 11:02, Baurzhan Ismagulov wrote: > On Tue, Aug 20, 2019 at 09:33:29PM +0200, Jan Kiszka wrote: >> Actually, I do not even see the point of locking here: If a bootstrap target >> is shared, only one multiconfig target will request its build and the others >> will wait automatically for the result. That's what bitbake is supposed to >> do for us. If it's not shared, there is no conflict to use locks on. We only >> need locks if different tasks of different recipes compete for common >> resources. > > There were build issues. Maybe that is the case when the host and the target > arch are the same. We might be able to reproduce that by trying to build > 2c99c8d. I will rather do this on top of my current queue which cleans up a lot and possibly also fixes false sharing of build stamps (due to improper STAMP setting). But I'm still fighting with other issues, will share some soon. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-08-21 9:06 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-12 17:18 isar-bootstrap fails to rebuild after config changes Jan Kiszka 2019-08-13 8:01 ` Claudius Heine 2019-08-13 8:14 ` Jan Kiszka 2019-08-20 19:33 ` Jan Kiszka 2019-08-21 9:02 ` Baurzhan Ismagulov 2019-08-21 9:06 ` Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox