[PATCHv3 0/3] sstate bug fix

[PATCHv3 0/3] sstate bug fix

[henning.schild]

changes since v2:
 - add p3

changes since v1:
 - add p2
 - fix typo in p1
 - move code around in p1, later touched again in p2

This is just a single patch but i think it fixes bugs and should be
considered before a release ... i usually do not do cover letters for
single patches. So here i still do it to point out that it might be
important.

Henning Schild (3):
  sstate: change rootfs task to cache
  sstate: fix task order and deps
  buildchroot: make function buildchroot_install_files idempotent

 meta/classes/rootfs.bbclass                   | 26 +++++++++----------
 .../buildchroot/buildchroot.inc               |  2 +-
 2 files changed, 14 insertions(+), 14 deletions(-)

-- 
2.34.1

[PATCHv3 1/3] sstate: change rootfs task to cache

[henning.schild]

This patch moves sstate caching from the task "do_rootfs" to
"do_rootfs_install". Which is earlier in the task chain.

There are two "bugs" that we fix by doing so, while hopefully not
giving up on too much caching speed.
 - rootfs_postprocess:
   - image_postprocess_mark: potentially outdated BUILD_ID and friends
   - rootfs_generate_manifest: manifest file gone missing in DEPLOY_DIR

Since rootfs_postprocess is kind of an open field it might also fix
issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any function
deriving stuff (like a manifest) or changing stuff (like the mark) could
potentially benefit from "not caching".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 1fc487896f4d..78fb2cc71350 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -270,29 +270,28 @@ python do_rootfs() {
 }
 addtask rootfs before do_build
 
-do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config"
-
-SSTATETASKS += "do_rootfs"
+SSTATETASKS += "do_rootfs_install"
 ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate"
-do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
-do_rootfs[cleandirs] += "${ROOTFS_SSTATE}"
-do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}"
-do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare"
+do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config"
+do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
+do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}"
+do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}"
+do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare"
 
 # the buildchroot is owned by root, so we need some sudoing to pack and unpack
-rootfs_sstate_prepare() {
+rootfs_install_sstate_prepare() {
     sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
     sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file-system rootfs
     sudo umount ${WORKDIR}/mnt/rootfs
 }
-do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 
-rootfs_sstate_finalize() {
+rootfs_install_sstate_finalize() {
     sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar
 }
 
-python do_rootfs_setscene() {
+python do_rootfs_install_setscene() {
     sstate_setscene(d)
-    bb.build.exec_func('rootfs_sstate_finalize', d)
+    bb.build.exec_func('rootfs_install_sstate_finalize', d)
 }
-addtask do_rootfs_setscene
+addtask do_rootfs_install_setscene
-- 
2.34.1

[PATCHv3 2/3] sstate: fix task order and deps

[henning.schild]

With do_rootfs_install being the new sstate task we need to be more
precise on what needs to happen before do_rootfs_postprocess.

Some postprocess functions need files downloaded so we add "after unpack"
 - sdkchroot_install_files
 - buildchroot_install_files
And do_rootfs_postprocess needs the apt caches for rootfs_do_mounts.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/rootfs.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 78fb2cc71350..9f0d345dae03 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -262,7 +262,7 @@ python do_rootfs_postprocess() {
         bb.build.exec_func(cmd, d)
         progress_reporter.update(int(i / len(cmds) * 100))
 }
-addtask rootfs_postprocess before do_rootfs
+addtask rootfs_postprocess before do_rootfs after do_unpack
 
 python do_rootfs() {
     """Virtual task"""
@@ -270,9 +270,10 @@ python do_rootfs() {
 }
 addtask rootfs before do_build
 
+do_rootfs_postprocess[depends] = "base-apt:do_cache isar-apt:do_cache_config"
+
 SSTATETASKS += "do_rootfs_install"
 ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate"
-do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config"
 do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
 do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}"
 do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}"
-- 
2.34.1

[PATCHv3 3/3] buildchroot: make function buildchroot_install_files idempotent

[henning.schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 6d9ced0c3f28..aa190e921ae4 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -61,7 +61,7 @@ buildchroot_install_files() {
     sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/
     sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/
 
-    sudo ln -s /downloads/git "${BUILDCHROOT_DIR}/home/.git-downloads"
+    sudo ln -sf /downloads/git "${BUILDCHROOT_DIR}/home/.git-downloads"
 
     # Configure root filesystem
     sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
-- 
2.34.1

RE: [PATCHv3 0/3] sstate bug fix

[Schmidt, Adriaan]

The root problem here is that the sstate code does not "see" when/which files are put into DEPLOY_DIR_IMAGE as a side-effect of tasks.

Unfortunately, we just found out that this problem is not limited to rootfs recipes. There are examples of dpkg recipes [1] placing artifacts into the deploy dir, which also breaks with sstate caching.

One option I can see is:
  * "forbid" (by convention/documentation) recipes from manually placing files in DEPLOY_DIR_IMAGE
  * instead, recipes need to set a variable (e.g., FILES_TO_DEPLOY)
  * these files will be deployed by isar classes
  * sstate will cache and restore these files

Or we go the OE way:
  * no-one writes directly to DEPLOY_DIR_IMAGE, but instead to IMGDEPLOYDIR, which is below WORKDIR
  * isar classes transfer the deployed artifacts to DEPLOY_DIR_IMAGE (in OE this transfer is actually done implicitly by sstate code)

Q:
  * is DEPLOY_DIR_IMAGE the only location with that problem? Or should we assume that recipes have side-effects writing to arbitrary locations?

Adriaan

[1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-bsp/efibootguard/efibootguard_0.9-git%2Bisar.bb#L42

henning.schild@siemens.com, 4. Februar 2022 17:41:
> 
> changes since v2:
>  - add p3
> 
> changes since v1:
>  - add p2
>  - fix typo in p1
>  - move code around in p1, later touched again in p2
> 
> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.
> 
> Henning Schild (3):
>   sstate: change rootfs task to cache
>   sstate: fix task order and deps
>   buildchroot: make function buildchroot_install_files idempotent
> 
>  meta/classes/rootfs.bbclass                   | 26 +++++++++----------
>  .../buildchroot/buildchroot.inc               |  2 +-
>  2 files changed, 14 insertions(+), 14 deletions(-)
> 
> --
> 2.34.1
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/20220204164124.10396-1-
> henning.schild%40siemens.com.

Re: [PATCHv3 0/3] sstate bug fix

[Anton Mikanovich]

15.02.2022 13:08, Schmidt, Adriaan wrote:
> The root problem here is that the sstate code does not "see" when/which files are put into DEPLOY_DIR_IMAGE as a side-effect of tasks.
>
> Unfortunately, we just found out that this problem is not limited to rootfs recipes. There are examples of dpkg recipes [1] placing artifacts into the deploy dir, which also breaks with sstate caching.
>
> One option I can see is:
>    * "forbid" (by convention/documentation) recipes from manually placing files in DEPLOY_DIR_IMAGE
>    * instead, recipes need to set a variable (e.g., FILES_TO_DEPLOY)
>    * these files will be deployed by isar classes
>    * sstate will cache and restore these files
>
> Or we go the OE way:
>    * no-one writes directly to DEPLOY_DIR_IMAGE, but instead to IMGDEPLOYDIR, which is below WORKDIR
>    * isar classes transfer the deployed artifacts to DEPLOY_DIR_IMAGE (in OE this transfer is actually done implicitly by sstate code)
>
> Q:
>    * is DEPLOY_DIR_IMAGE the only location with that problem? Or should we assume that recipes have side-effects writing to arbitrary locations?
>
> Adriaan
>
> [1] https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/master/recipes-bsp/efibootguard/efibootguard_0.9-git%2Bisar.bb#L42

The way binaries installed to DEPLOY_DIR_IMAGE definitely need to be 
changed, but does anything speak against merging this patchset?

Re: [PATCHv3 0/3] sstate bug fix

[Anton Mikanovich]

4.02.22 19:41, henning.schild@siemens.com wrote:
> changes since v2:
>   - add p3
>
> changes since v1:
>   - add p2
>   - fix typo in p1
>   - move code around in p1, later touched again in p2
>
> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.
>
> Henning Schild (3):
>    sstate: change rootfs task to cache
>    sstate: fix task order and deps
>    buildchroot: make function buildchroot_install_files idempotent
>
>   meta/classes/rootfs.bbclass                   | 26 +++++++++----------
>   .../buildchroot/buildchroot.inc               |  2 +-
>   2 files changed, 14 insertions(+), 14 deletions(-)
>
Applied to next, thanks.