[PATCH 0/1] sstate bug fix

[PATCH 0/1] sstate bug fix

[Henning Schild]

This is just a single patch but i think it fixes bugs and should be
considered before a release ... i usually do not do cover letters for
single patches. So here i still do it to point out that it might be
important.

Henning Schild (1):
  sttate: change rootfs task to cache

 meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

-- 
2.34.1

[PATCH 1/1] sttate: change rootfs task to cache

[Henning Schild]

This patch moves sstate caching from the task "do_rootfs" to
"do_rootfs_install". Which is earlier in the task chain.

There are two "bugs" that we fix by doing so, while hopefully not
giving up on too much caching speed.
 - rootfs_postprocess:
   - image_postprocess_mark: potentially outdated BUILD_ID and friends
   - rootfs_generate_manifest: manifest file gone missing in DEPLOY_DIR

Since rootfs_postprocess is kind of an open field it might also fix
issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any function
deriving stuff (like a manifest) or changing stuff (like the mark) could
potentially benefit from "not caching".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 1fc487896f4d..e368244ce1ca 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -157,6 +157,7 @@ rootfs_install_pkgs_install() {
 do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
 do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}"
 do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
+do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config"
 do_rootfs_install[recrdeptask] = "do_deploy_deb"
 python do_rootfs_install() {
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
@@ -270,29 +271,27 @@ python do_rootfs() {
 }
 addtask rootfs before do_build
 
-do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config"
-
-SSTATETASKS += "do_rootfs"
+SSTATETASKS += "do_rootfs_install"
 ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate"
-do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
-do_rootfs[cleandirs] += "${ROOTFS_SSTATE}"
-do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}"
-do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare"
+do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
+do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}"
+do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}"
+do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare"
 
 # the buildchroot is owned by root, so we need some sudoing to pack and unpack
-rootfs_sstate_prepare() {
+rootfs_install_sstate_prepare() {
     sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
     sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file-system rootfs
     sudo umount ${WORKDIR}/mnt/rootfs
 }
-do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 
-rootfs_sstate_finalize() {
+rootfs_install_sstate_finalize() {
     sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar
 }
 
-python do_rootfs_setscene() {
+python do_rootfs_install_setscene() {
     sstate_setscene(d)
-    bb.build.exec_func('rootfs_sstate_finalize', d)
+    bb.build.exec_func('rootfs_install_sstate_finalize', d)
 }
-addtask do_rootfs_setscene
+addtask do_rootfs_install_setscene
-- 
2.34.1

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Some more testing reveals that buildchroot_install_files does not work
with that. The reason seams to be that files to be installed did not
get fetched when using the cache.

Will have to see how to deal with that. Stay tuned for a v2 and please
not that sstate still has bugs. But those might just be OK to enter a
release, that is up to the maintainer to decide.

regards,
Henning

Am Thu, 27 Jan 2022 21:18:44 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.
> 
> Henning Schild (1):
>   sttate: change rootfs task to cache
> 
>  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
>  1 file changed, 12 insertions(+), 13 deletions(-)
> 

RE: [PATCH 0/1] sstate bug fix

[Schmidt, Adriaan]

Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:
> 
> Some more testing reveals that buildchroot_install_files does not work
> with that. The reason seams to be that files to be installed did not
> get fetched when using the cache.

An easy fix would be to add a dependency in buildchroot:
addtask rootfs_install after do_fetch

Or turn those files into a package so they get installed in rootfs_install
instead of "installing" stuff during "postprocess".

Adriaan

> Will have to see how to deal with that. Stay tuned for a v2 and please
> not that sstate still has bugs. But those might just be OK to enter a
> release, that is up to the maintainer to decide.
> 
> regards,
> Henning
> 
> Am Thu, 27 Jan 2022 21:18:44 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
> > This is just a single patch but i think it fixes bugs and should be
> > considered before a release ... i usually do not do cover letters for
> > single patches. So here i still do it to point out that it might be
> > important.
> >
> > Henning Schild (1):
> >   sttate: change rootfs task to cache
> >
> >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> >  1 file changed, 12 insertions(+), 13 deletions(-)
> >

RE: [PATCH 1/1] sttate: change rootfs task to cache

[Schmidt, Adriaan]

Schild, Henning (T CED SES-DE), 27. Januar 2022 21:19:

> Subject: [PATCH 1/1] sttate: change rootfs task to cache

Typo in commit msg -----^

> This patch moves sstate caching from the task "do_rootfs" to
> "do_rootfs_install". Which is earlier in the task chain.
> 
> There are two "bugs" that we fix by doing so, while hopefully not
> giving up on too much caching speed.
>  - rootfs_postprocess:
>    - image_postprocess_mark: potentially outdated BUILD_ID and friends
>    - rootfs_generate_manifest: manifest file gone missing in DEPLOY_DIR
> 
> Since rootfs_postprocess is kind of an open field it might also fix
> issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any function
> deriving stuff (like a manifest) or changing stuff (like the mark) could
> potentially benefit from "not caching".
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
>  1 file changed, 12 insertions(+), 13 deletions(-)
> 
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 1fc487896f4d..e368244ce1ca 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -157,6 +157,7 @@ rootfs_install_pkgs_install() {
>  do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
>  do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND}
> ${ROOTFS_INSTALL_COMMAND}"
>  do_rootfs_install[depends] = "isar-bootstrap-${@'target' if
> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
> +do_rootfs_install[depends] += " base-apt:do_cache isar-apt:do_cache_config"
>  do_rootfs_install[recrdeptask] = "do_deploy_deb"
>  python do_rootfs_install() {
>      configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or
> "").split()
> @@ -270,29 +271,27 @@ python do_rootfs() {
>  }
>  addtask rootfs before do_build
> 
> -do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config"

You removed that line.
I don't remember all of the details, but there were some cases where
this explicit dependency was needed when restoring from cache to
an empty tmp/ dir. Might need some testing...

Adriaan

> -
> -SSTATETASKS += "do_rootfs"
> +SSTATETASKS += "do_rootfs_install"
>  ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate"
> -do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
> -do_rootfs[cleandirs] += "${ROOTFS_SSTATE}"
> -do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}"
> -do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare"
> +do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
> +do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}"
> +do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}"
> +do_rootfs_install[sstate-interceptfuncs] = "rootfs_install_sstate_prepare"
> 
>  # the buildchroot is owned by root, so we need some sudoing to pack and
> unpack
> -rootfs_sstate_prepare() {
> +rootfs_install_sstate_prepare() {
>      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
>      sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar --one-file-
> system rootfs
>      sudo umount ${WORKDIR}/mnt/rootfs
>  }
> -do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> +do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> 
> -rootfs_sstate_finalize() {
> +rootfs_install_sstate_finalize() {
>      sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar
>  }
> 
> -python do_rootfs_setscene() {
> +python do_rootfs_install_setscene() {
>      sstate_setscene(d)
> -    bb.build.exec_func('rootfs_sstate_finalize', d)
> +    bb.build.exec_func('rootfs_install_sstate_finalize', d)
>  }
> -addtask do_rootfs_setscene
> +addtask do_rootfs_install_setscene
> --
> 2.34.1

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Am Fri, 28 Jan 2022 07:24:50 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:
> > 
> > Some more testing reveals that buildchroot_install_files does not
> > work with that. The reason seams to be that files to be installed
> > did not get fetched when using the cache.  
> 
> An easy fix would be to add a dependency in buildchroot:
> addtask rootfs_install after do_fetch

I will try that. Looking at the code we might well have the same
problem with sdkchroot_install_files, hopefully the fetch dep fixes
both.

> Or turn those files into a package so they get installed in
> rootfs_install instead of "installing" stuff during "postprocess".

Yes both these postprocess functions violate an essential isar rule.
"everything inside rootfs should come from a package"

I think for the stuff from sdkchroot_install_files we can do that
easily and i will add a TODO for that. For buildchroot_install_files it
might be trickier, if we wanted that to be a package we would need to
build it "native" since we do not yet have a buildchroot.

Henning

> Adriaan
> 
> > Will have to see how to deal with that. Stay tuned for a v2 and
> > please not that sstate still has bugs. But those might just be OK
> > to enter a release, that is up to the maintainer to decide.
> > 
> > regards,
> > Henning
> > 
> > Am Thu, 27 Jan 2022 21:18:44 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> > > This is just a single patch but i think it fixes bugs and should
> > > be considered before a release ... i usually do not do cover
> > > letters for single patches. So here i still do it to point out
> > > that it might be important.
> > >
> > > Henning Schild (1):
> > >   sttate: change rootfs task to cache
> > >
> > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > >  
> 

Re: [PATCH 1/1] sttate: change rootfs task to cache

[Henning Schild]

Am Fri, 28 Jan 2022 07:57:31 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 27. Januar 2022 21:19:
> 
> > Subject: [PATCH 1/1] sttate: change rootfs task to cache  
> 
> Typo in commit msg -----^

Thanks!

> > This patch moves sstate caching from the task "do_rootfs" to
> > "do_rootfs_install". Which is earlier in the task chain.
> > 
> > There are two "bugs" that we fix by doing so, while hopefully not
> > giving up on too much caching speed.
> >  - rootfs_postprocess:
> >    - image_postprocess_mark: potentially outdated BUILD_ID and
> > friends
> >    - rootfs_generate_manifest: manifest file gone missing in
> > DEPLOY_DIR
> > 
> > Since rootfs_postprocess is kind of an open field it might also fix
> > issues in ROOTFS_POSTPROCESS_COMMAND functions in layers. Any
> > function deriving stuff (like a manifest) or changing stuff (like
> > the mark) could potentially benefit from "not caching".
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> >  1 file changed, 12 insertions(+), 13 deletions(-)
> > 
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 1fc487896f4d..e368244ce1ca
> > 100644 --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -157,6 +157,7 @@ rootfs_install_pkgs_install() {
> >  do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
> >  do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND}
> > ${ROOTFS_INSTALL_COMMAND}"
> >  do_rootfs_install[depends] = "isar-bootstrap-${@'target' if
> > d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else
> > 'host'}:do_build" +do_rootfs_install[depends] += "
> > base-apt:do_cache isar-apt:do_cache_config"
> > do_rootfs_install[recrdeptask] = "do_deploy_deb" python
> > do_rootfs_install() { configure_cmds =
> > (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
> > @@ -270,29 +271,27 @@ python do_rootfs() {
> >  }
> >  addtask rootfs before do_build
> > 
> > -do_rootfs[depends] = "base-apt:do_cache isar-apt:do_cache_config"  
> 
> You removed that line.
> I don't remember all of the details, but there were some cases where
> this explicit dependency was needed when restoring from cache to
> an empty tmp/ dir. Might need some testing...

It got moved up under a preexisting do_rootfs_install[depends]. In fact
we might want to move all do_rootfs_install[] assignments there for
readability. But hey, i think i will move it back down so we have the
actual function []s and the sstate specific []s.

Henning

> Adriaan
> 
> > -
> > -SSTATETASKS += "do_rootfs"
> > +SSTATETASKS += "do_rootfs_install"
> >  ROOTFS_SSTATE = "${WORKDIR}/rootfs-sstate"
> > -do_rootfs[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
> > -do_rootfs[cleandirs] += "${ROOTFS_SSTATE}"
> > -do_rootfs[sstate-plaindirs] = "${ROOTFS_SSTATE}"
> > -do_rootfs[sstate-interceptfuncs] = "rootfs_sstate_prepare"
> > +do_rootfs_install[dirs] += "${ROOTFS_SSTATE} ${WORKDIR}/mnt/rootfs"
> > +do_rootfs_install[cleandirs] += "${ROOTFS_SSTATE}"
> > +do_rootfs_install[sstate-plaindirs] = "${ROOTFS_SSTATE}"
> > +do_rootfs_install[sstate-interceptfuncs] =
> > "rootfs_install_sstate_prepare"
> > 
> >  # the buildchroot is owned by root, so we need some sudoing to
> > pack and unpack
> > -rootfs_sstate_prepare() {
> > +rootfs_install_sstate_prepare() {
> >      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> >      sudo tar -C ${WORKDIR}/mnt -cpf ${ROOTFS_SSTATE}/rootfs.tar
> > --one-file- system rootfs
> >      sudo umount ${WORKDIR}/mnt/rootfs
> >  }
> > -do_rootfs_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> > +do_rootfs_install_sstate_prepare[lockfiles] =
> > "${REPO_ISAR_DIR}/isar.lock"
> > 
> > -rootfs_sstate_finalize() {
> > +rootfs_install_sstate_finalize() {
> >      sudo tar -C ${WORKDIR} -xpf ${ROOTFS_SSTATE}/rootfs.tar
> >  }
> > 
> > -python do_rootfs_setscene() {
> > +python do_rootfs_install_setscene() {
> >      sstate_setscene(d)
> > -    bb.build.exec_func('rootfs_sstate_finalize', d)
> > +    bb.build.exec_func('rootfs_install_sstate_finalize', d)
> >  }
> > -addtask do_rootfs_setscene
> > +addtask do_rootfs_install_setscene
> > --
> > 2.34.1  
> 

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Am Fri, 28 Jan 2022 07:24:50 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:
> > 
> > Some more testing reveals that buildchroot_install_files does not
> > work with that. The reason seams to be that files to be installed
> > did not get fetched when using the cache.  
> 
> An easy fix would be to add a dependency in buildchroot:
> addtask rootfs_install after do_fetch

In fact we already have it "after do_unpack". I kind of need a way to
say that do_rootfs_postprocess depends on fetch/unpack being actually
done, even if rootfs_install was found in the cache.

Henning

> Or turn those files into a package so they get installed in
> rootfs_install instead of "installing" stuff during "postprocess".
> 
> Adriaan
> 
> > Will have to see how to deal with that. Stay tuned for a v2 and
> > please not that sstate still has bugs. But those might just be OK
> > to enter a release, that is up to the maintainer to decide.
> > 
> > regards,
> > Henning
> > 
> > Am Thu, 27 Jan 2022 21:18:44 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> > > This is just a single patch but i think it fixes bugs and should
> > > be considered before a release ... i usually do not do cover
> > > letters for single patches. So here i still do it to point out
> > > that it might be important.
> > >
> > > Henning Schild (1):
> > >   sttate: change rootfs task to cache
> > >
> > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > >  
> 

RE: [PATCH 0/1] sstate bug fix

[Schmidt, Adriaan]

Schild, Henning (T CED SES-DE), 28. Januar 2022 08:15
> Am Fri, 28 Jan 2022 07:24:50 +0100
> schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:
> 
> > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:
> > >
> > > Some more testing reveals that buildchroot_install_files does not
> > > work with that. The reason seams to be that files to be installed
> > > did not get fetched when using the cache.
> >
> > An easy fix would be to add a dependency in buildchroot:
> > addtask rootfs_install after do_fetch
> 
> I will try that. Looking at the code we might well have the same
> problem with sdkchroot_install_files, hopefully the fetch dep fixes
> both.

Hopefully, running do_fetch will not trigger the complete task chain.
If all goes well, it will produce an unchanged sstate signature and
not see any need to go through do_patch..do_rootfs_install.
But that's something to look out for.

> > Or turn those files into a package so they get installed in
> > rootfs_install instead of "installing" stuff during "postprocess".
> 
> Yes both these postprocess functions violate an essential isar rule.
> "everything inside rootfs should come from a package"
> 
> I think for the stuff from sdkchroot_install_files we can do that
> easily and i will add a TODO for that. For buildchroot_install_files it
> might be trickier, if we wanted that to be a package we would need to
> build it "native" since we do not yet have a buildchroot.

For the SDK case, that's one of the things happening in the proposed
"[PATCH v3 2/3] image: refactor SDK".

Adriaan

> 
> Henning
> 
> > Adriaan
> >
> > > Will have to see how to deal with that. Stay tuned for a v2 and
> > > please not that sstate still has bugs. But those might just be OK
> > > to enter a release, that is up to the maintainer to decide.
> > >
> > > regards,
> > > Henning
> > >
> > > Am Thu, 27 Jan 2022 21:18:44 +0100
> > > schrieb Henning Schild <henning.schild@siemens.com>:
> > >
> > > > This is just a single patch but i think it fixes bugs and should
> > > > be considered before a release ... i usually do not do cover
> > > > letters for single patches. So here i still do it to point out
> > > > that it might be important.
> > > >
> > > > Henning Schild (1):
> > > >   sttate: change rootfs task to cache
> > > >
> > > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > > >
> >

RE: [PATCH 0/1] sstate bug fix

[Schmidt, Adriaan]

Schild, Henning (T CED SES-DE), 28. Januar 2022 08:24:
> Am Fri, 28 Jan 2022 07:24:50 +0100
> schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:
> 
> > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:
> > >
> > > Some more testing reveals that buildchroot_install_files does not
> > > work with that. The reason seams to be that files to be installed
> > > did not get fetched when using the cache.
> >
> > An easy fix would be to add a dependency in buildchroot:
> > addtask rootfs_install after do_fetch
> 
> In fact we already have it "after do_unpack". I kind of need a way to
> say that do_rootfs_postprocess depends on fetch/unpack being actually
> done, even if rootfs_install was found in the cache.

I was wrong in my original comment. We probably need:
addtask rootfs_postprocess after do_unpack

Adriaan

> Henning
> 
> > Or turn those files into a package so they get installed in
> > rootfs_install instead of "installing" stuff during "postprocess".
> >
> > Adriaan
> >
> > > Will have to see how to deal with that. Stay tuned for a v2 and
> > > please not that sstate still has bugs. But those might just be OK
> > > to enter a release, that is up to the maintainer to decide.
> > >
> > > regards,
> > > Henning
> > >
> > > Am Thu, 27 Jan 2022 21:18:44 +0100
> > > schrieb Henning Schild <henning.schild@siemens.com>:
> > >
> > > > This is just a single patch but i think it fixes bugs and should
> > > > be considered before a release ... i usually do not do cover
> > > > letters for single patches. So here i still do it to point out
> > > > that it might be important.
> > > >
> > > > Henning Schild (1):
> > > >   sttate: change rootfs task to cache
> > > >
> > > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > > >
> >

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Am Fri, 28 Jan 2022 08:38:17 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 28. Januar 2022 08:24:
> > Am Fri, 28 Jan 2022 07:24:50 +0100
> > schrieb "Schmidt, Adriaan (T CED SES-DE)"
> > <adriaan.schmidt@siemens.com>: 
> > > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:  
> > > >
> > > > Some more testing reveals that buildchroot_install_files does
> > > > not work with that. The reason seams to be that files to be
> > > > installed did not get fetched when using the cache.  
> > >
> > > An easy fix would be to add a dependency in buildchroot:
> > > addtask rootfs_install after do_fetch  
> > 
> > In fact we already have it "after do_unpack". I kind of need a way
> > to say that do_rootfs_postprocess depends on fetch/unpack being
> > actually done, even if rootfs_install was found in the cache.  
> 
> I was wrong in my original comment. We probably need:
> addtask rootfs_postprocess after do_unpack

Yes, that is what i came up with as well. Seems to work.

Henning

> Adriaan
> 
> > Henning
> >   
> > > Or turn those files into a package so they get installed in
> > > rootfs_install instead of "installing" stuff during "postprocess".
> > >
> > > Adriaan
> > >  
> > > > Will have to see how to deal with that. Stay tuned for a v2 and
> > > > please not that sstate still has bugs. But those might just be
> > > > OK to enter a release, that is up to the maintainer to decide.
> > > >
> > > > regards,
> > > > Henning
> > > >
> > > > Am Thu, 27 Jan 2022 21:18:44 +0100
> > > > schrieb Henning Schild <henning.schild@siemens.com>:
> > > >  
> > > > > This is just a single patch but i think it fixes bugs and
> > > > > should be considered before a release ... i usually do not do
> > > > > cover letters for single patches. So here i still do it to
> > > > > point out that it might be important.
> > > > >
> > > > > Henning Schild (1):
> > > > >   sttate: change rootfs task to cache
> > > > >
> > > > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > > > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > > > >  
> > >  
> 

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Am Fri, 28 Jan 2022 08:33:01 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 28. Januar 2022 08:15
> > Am Fri, 28 Jan 2022 07:24:50 +0100
> > schrieb "Schmidt, Adriaan (T CED SES-DE)"
> > <adriaan.schmidt@siemens.com>: 
> > > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:  
> > > >
> > > > Some more testing reveals that buildchroot_install_files does
> > > > not work with that. The reason seams to be that files to be
> > > > installed did not get fetched when using the cache.  
> > >
> > > An easy fix would be to add a dependency in buildchroot:
> > > addtask rootfs_install after do_fetch  
> > 
> > I will try that. Looking at the code we might well have the same
> > problem with sdkchroot_install_files, hopefully the fetch dep fixes
> > both.  
> 
> Hopefully, running do_fetch will not trigger the complete task chain.
> If all goes well, it will produce an unchanged sstate signature and
> not see any need to go through do_patch..do_rootfs_install.
> But that's something to look out for.
> 
> > > Or turn those files into a package so they get installed in
> > > rootfs_install instead of "installing" stuff during
> > > "postprocess".  
> > 
> > Yes both these postprocess functions violate an essential isar rule.
> > "everything inside rootfs should come from a package"
> > 
> > I think for the stuff from sdkchroot_install_files we can do that
> > easily and i will add a TODO for that. For
> > buildchroot_install_files it might be trickier, if we wanted that
> > to be a package we would need to build it "native" since we do not
> > yet have a buildchroot.  
> 
> For the SDK case, that's one of the things happening in the proposed
> "[PATCH v3 2/3] image: refactor SDK".

Cool! I will not even add the TODO in that case.

Henning

> Adriaan
> 
> > 
> > Henning
> >   
> > > Adriaan
> > >  
> > > > Will have to see how to deal with that. Stay tuned for a v2 and
> > > > please not that sstate still has bugs. But those might just be
> > > > OK to enter a release, that is up to the maintainer to decide.
> > > >
> > > > regards,
> > > > Henning
> > > >
> > > > Am Thu, 27 Jan 2022 21:18:44 +0100
> > > > schrieb Henning Schild <henning.schild@siemens.com>:
> > > >  
> > > > > This is just a single patch but i think it fixes bugs and
> > > > > should be considered before a release ... i usually do not do
> > > > > cover letters for single patches. So here i still do it to
> > > > > point out that it might be important.
> > > > >
> > > > > Henning Schild (1):
> > > > >   sttate: change rootfs task to cache
> > > > >
> > > > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > > > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > > > >  
> > >  
> 

Re: [PATCH 0/1] sstate bug fix

[Henning Schild]

Am Fri, 28 Jan 2022 08:33:01 +0100
schrieb "Schmidt, Adriaan (T CED SES-DE)" <adriaan.schmidt@siemens.com>:

> Schild, Henning (T CED SES-DE), 28. Januar 2022 08:15
> > Am Fri, 28 Jan 2022 07:24:50 +0100
> > schrieb "Schmidt, Adriaan (T CED SES-DE)"
> > <adriaan.schmidt@siemens.com>: 
> > > Schild, Henning (T CED SES-DE), 27. Januar 2022 21:40:  
> > > >
> > > > Some more testing reveals that buildchroot_install_files does
> > > > not work with that. The reason seams to be that files to be
> > > > installed did not get fetched when using the cache.  
> > >
> > > An easy fix would be to add a dependency in buildchroot:
> > > addtask rootfs_install after do_fetch  
> > 
> > I will try that. Looking at the code we might well have the same
> > problem with sdkchroot_install_files, hopefully the fetch dep fixes
> > both.  
> 
> Hopefully, running do_fetch will not trigger the complete task chain.
> If all goes well, it will produce an unchanged sstate signature and
> not see any need to go through do_patch..do_rootfs_install.
> But that's something to look out for.

I checked and it will not trigger rootfs_install again.

Henning

> > > Or turn those files into a package so they get installed in
> > > rootfs_install instead of "installing" stuff during
> > > "postprocess".  
> > 
> > Yes both these postprocess functions violate an essential isar rule.
> > "everything inside rootfs should come from a package"
> > 
> > I think for the stuff from sdkchroot_install_files we can do that
> > easily and i will add a TODO for that. For
> > buildchroot_install_files it might be trickier, if we wanted that
> > to be a package we would need to build it "native" since we do not
> > yet have a buildchroot.  
> 
> For the SDK case, that's one of the things happening in the proposed
> "[PATCH v3 2/3] image: refactor SDK".
> 
> Adriaan
> 
> > 
> > Henning
> >   
> > > Adriaan
> > >  
> > > > Will have to see how to deal with that. Stay tuned for a v2 and
> > > > please not that sstate still has bugs. But those might just be
> > > > OK to enter a release, that is up to the maintainer to decide.
> > > >
> > > > regards,
> > > > Henning
> > > >
> > > > Am Thu, 27 Jan 2022 21:18:44 +0100
> > > > schrieb Henning Schild <henning.schild@siemens.com>:
> > > >  
> > > > > This is just a single patch but i think it fixes bugs and
> > > > > should be considered before a release ... i usually do not do
> > > > > cover letters for single patches. So here i still do it to
> > > > > point out that it might be important.
> > > > >
> > > > > Henning Schild (1):
> > > > >   sttate: change rootfs task to cache
> > > > >
> > > > >  meta/classes/rootfs.bbclass | 25 ++++++++++++-------------
> > > > >  1 file changed, 12 insertions(+), 13 deletions(-)
> > > > >  
> > >  
> 

Re: [PATCH 0/1] sstate bug fix

[Baurzhan Ismagulov]

On Thu, Jan 27, 2022 at 09:18:44PM +0100, Henning Schild wrote:
> This is just a single patch but i think it fixes bugs and should be
> considered before a release ... i usually do not do cover letters for
> single patches. So here i still do it to point out that it might be
> important.

Oops, we've concentrated too much on our original patch list to include in the
release and overlooked this one... We'll review and test the latest version.

We'll consider process changes for a more balanced procedure and migrate our
internal patchwork to a public server.

With kind regards,
Baurzhan.