* Isar can't reuse sstate-cache build from different isar location @ 2023-09-07 11:31 Uladzimir Bely 2023-09-07 13:29 ` Schmidt, Adriaan 0 siblings, 1 reply; 5+ messages in thread From: Uladzimir Bely @ 2023-09-07 11:31 UTC (permalink / raw) To: isar-users I tried to use the same SSTATE_DIR for different isar locations on the same machine under the same user and found that it does not work. Quick steps to reproduce it: ## Prepare build environment with two copies of isar: ``` ~ $ mkdir -m 777 test-sstate ~ $ cd test-sstate/ ~/test-sstate $ docker run --privileged -it -v .:/build ghcr.io/siemens/kas/kas-isar:4.0 builder@ffe5274283d3:~$ cd /build builder@ffe5274283d3:/build$ git clone https://github.com/ilbers/isar.git isar-1 builder@ffe5274283d3:/build$ git clone https://github.com/ilbers/isar.git isar-2 ``` ## Build isar-1: ``` builder@ffe5274283d3:/build$ cd /build/isar-1 builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env builder@ffe5274283d3:/build/isar-1/build$ echo 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf builder@ffe5274283d3:/build/isar-1/build$ echo 'DL_DIR="/build/downloads"' >> conf/local.conf builder@ffe5274283d3:/build/isar-1/build$ time bitbake mc:qemuamd64- bullseye:isar-image-base real 4m38.759s user 0m4.248s sys 0m1.068s builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads 229M /build/downloads builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate-cache 1.1G /build/sstate-cache ``` ## Build isar-2: ``` builder@ffe5274283d3:/build$ cd /build/isar-2 builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env builder@ffe5274283d3:/build/isar-2/build$ echo 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf builder@ffe5274283d3:/build/isar-2/build$ echo 'DL_DIR="/build/downloads"' >> conf/local.conf builder@ffe5274283d3:/build/isar-2/build$ time bitbake mc:qemuamd64- bullseye:isar-image-base real 4m32.314s user 0m3.975s sys 0m0.898s builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads 229M /build/downloads builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate-cache/ 2.2G /build/sstate-cache/ ``` Second build took almost the same time as first one. Sstate cache was not reused, second build just added new files to the cache. I tried similar approach with Yocto, and it works in it. After debugging, I have concluded, that this issue comes from "bootstrap" task that is not sstate-cache compatible. It's caused by our manipulations with SRC_URI we do in it, so that it's different for both builds. E.g.: SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- 1/meta/conf/distro/debian-bullseye.list" SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- 2/meta/conf/distro/debian-bullseye.list" SRC_URI is a variable that is tracked in sstate-cache. Since the last part (that we add "on fly" with get_aptsources_list() function) is different for both builds, sstate-cache can't be reused for "isar- bootstrap-<host,target>" and for all other tasks since they depend on those ones. Any ideas how to properly deal with this? By the way, kas users are not affected, since fixed paths are used (e.g., "/work/isar", "/build") under the container. ^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: Isar can't reuse sstate-cache build from different isar location 2023-09-07 11:31 Isar can't reuse sstate-cache build from different isar location Uladzimir Bely @ 2023-09-07 13:29 ` Schmidt, Adriaan 2023-09-07 15:23 ` Uladzimir Bely 0 siblings, 1 reply; 5+ messages in thread From: Schmidt, Adriaan @ 2023-09-07 13:29 UTC (permalink / raw) To: Uladzimir Bely, isar-users Hi, I believe the problem was introduced in 9e7c63ee6. This causes absolute paths to end up in SRC_URI. Can you try this patch: --- diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 8af73a9b..28264024 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -156,7 +156,7 @@ def get_aptsources_list(d): for p in apt_sources_list: try: f = bb.parse.resolve_file(p, d) - ret.append(f) + ret.append(p) except FileNotFoundError as e: bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') return ret --- I haven't tested it. If this is not it I'll have to dig a little deeper. Generally re-use of sstate artifacts is supported, and we were careful to avoid absolute paths in variables. In fact, the isar-sstate script has a "lint" command to detect them, but that fails here, because the path has a "file://" prefix. Thanks, Adriaan Uladzimir Bely, Thursday, September 7, 2023 1:32 PM: > I tried to use the same SSTATE_DIR for different isar locations on the > same machine under the same user and found that it does not work. > > Quick steps to reproduce it: > > ## Prepare build environment with two copies of isar: > > ``` > ~ $ mkdir -m 777 test-sstate > ~ $ cd test-sstate/ > ~/test-sstate $ docker run --privileged -it -v .:/build > ghcr.io/siemens/kas/kas-isar:4.0 > builder@ffe5274283d3:~$ cd /build > builder@ffe5274283d3:/build$ git clone > https://github.com/ilbers/isar.git isar-1 > builder@ffe5274283d3:/build$ git clone > https://github.com/ilbers/isar.git isar-2 > ``` > > ## Build isar-1: > > ``` > builder@ffe5274283d3:/build$ cd /build/isar-1 > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > builder@ffe5274283d3:/build/isar-1/build$ echo > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > builder@ffe5274283d3:/build/isar-1/build$ echo > 'DL_DIR="/build/downloads"' >> conf/local.conf > builder@ffe5274283d3:/build/isar-1/build$ time bitbake mc:qemuamd64- > bullseye:isar-image-base > real 4m38.759s > user 0m4.248s > sys 0m1.068s > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads > 229M /build/downloads > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate-cache > 1.1G /build/sstate-cache > ``` > > ## Build isar-2: > > ``` > builder@ffe5274283d3:/build$ cd /build/isar-2 > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > builder@ffe5274283d3:/build/isar-2/build$ echo > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > builder@ffe5274283d3:/build/isar-2/build$ echo > 'DL_DIR="/build/downloads"' >> conf/local.conf > builder@ffe5274283d3:/build/isar-2/build$ time bitbake mc:qemuamd64- > bullseye:isar-image-base > real 4m32.314s > user 0m3.975s > sys 0m0.898s > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads > 229M /build/downloads > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate-cache/ > 2.2G /build/sstate-cache/ > ``` > > Second build took almost the same time as first one. Sstate cache was > not reused, second build just added new files to the cache. > > I tried similar approach with Yocto, and it works in it. > > After debugging, I have concluded, that this issue comes from > "bootstrap" task that is not sstate-cache compatible. It's caused by > our manipulations with SRC_URI we do in it, so that it's different for > both builds. E.g.: > > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > 1/meta/conf/distro/debian-bullseye.list" > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > 2/meta/conf/distro/debian-bullseye.list" > > SRC_URI is a variable that is tracked in sstate-cache. Since the last > part (that we add "on fly" with get_aptsources_list() function) is > different for both builds, sstate-cache can't be reused for "isar- > bootstrap-<host,target>" and for all other tasks since they depend on > those ones. > > Any ideas how to properly deal with this? > > By the way, kas users are not affected, since fixed paths are used > (e.g., "/work/isar", "/build") under the container. > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar- > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.de. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Isar can't reuse sstate-cache build from different isar location 2023-09-07 13:29 ` Schmidt, Adriaan @ 2023-09-07 15:23 ` Uladzimir Bely 2023-09-07 15:27 ` Uladzimir Bely 0 siblings, 1 reply; 5+ messages in thread From: Uladzimir Bely @ 2023-09-07 15:23 UTC (permalink / raw) To: Schmidt, Adriaan, isar-users On Thu, 2023-09-07 at 13:29 +0000, Schmidt, Adriaan wrote: > Hi, > > I believe the problem was introduced in 9e7c63ee6. This causes > absolute paths to end up in SRC_URI. > > Can you try this patch: > --- > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 8af73a9b..28264024 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -156,7 +156,7 @@ def get_aptsources_list(d): > for p in apt_sources_list: > try: > f = bb.parse.resolve_file(p, d) > - ret.append(f) > + ret.append(p) > except FileNotFoundError as e: > bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') > return ret > --- > > I haven't tested it. If this is not it I'll have to dig a little > deeper. > While debugging, I was looking at this also. "p" is something like "conf/distro/debian-bullseye.list", "f" just prepends it with an absolute path. I tried this replacement and it resulted to something like "file not found", since LAYERDIR_core and LAYERDIR_isar are not directly avalable in FILEEXTRAPATH and so on. But I'll recheck it. > Generally re-use of sstate artifacts is supported, and we were > careful to avoid absolute paths in variables. > In fact, the isar-sstate script has a "lint" command to detect them, > but that fails here, because the > path has a "file://" prefix. > > Thanks, > Adriaan > > Uladzimir Bely, Thursday, September 7, 2023 1:32 PM: > > I tried to use the same SSTATE_DIR for different isar locations on > > the > > same machine under the same user and found that it does not work. > > > > Quick steps to reproduce it: > > > > ## Prepare build environment with two copies of isar: > > > > ``` > > ~ $ mkdir -m 777 test-sstate > > ~ $ cd test-sstate/ > > ~/test-sstate $ docker run --privileged -it -v .:/build > > ghcr.io/siemens/kas/kas-isar:4.0 > > builder@ffe5274283d3:~$ cd /build > > builder@ffe5274283d3:/build$ git clone > > https://github.com/ilbers/isar.git isar-1 > > builder@ffe5274283d3:/build$ git clone > > https://github.com/ilbers/isar.git isar-2 > > ``` > > > > ## Build isar-1: > > > > ``` > > builder@ffe5274283d3:/build$ cd /build/isar-1 > > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > > builder@ffe5274283d3:/build/isar-1/build$ echo > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-1/build$ echo > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-1/build$ time bitbake > > mc:qemuamd64- > > bullseye:isar-image-base > > real 4m38.759s > > user 0m4.248s > > sys 0m1.068s > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads > > 229M /build/downloads > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate- > > cache > > 1.1G /build/sstate-cache > > ``` > > > > ## Build isar-2: > > > > ``` > > builder@ffe5274283d3:/build$ cd /build/isar-2 > > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > > builder@ffe5274283d3:/build/isar-2/build$ echo > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-2/build$ echo > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-2/build$ time bitbake > > mc:qemuamd64- > > bullseye:isar-image-base > > real 4m32.314s > > user 0m3.975s > > sys 0m0.898s > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads > > 229M /build/downloads > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate- > > cache/ > > 2.2G /build/sstate-cache/ > > ``` > > > > Second build took almost the same time as first one. Sstate cache > > was > > not reused, second build just added new files to the cache. > > > > I tried similar approach with Yocto, and it works in it. > > > > After debugging, I have concluded, that this issue comes from > > "bootstrap" task that is not sstate-cache compatible. It's caused > > by > > our manipulations with SRC_URI we do in it, so that it's different > > for > > both builds. E.g.: > > > > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > > 1/meta/conf/distro/debian-bullseye.list" > > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > > 2/meta/conf/distro/debian-bullseye.list" > > > > SRC_URI is a variable that is tracked in sstate-cache. Since the > > last > > part (that we add "on fly" with get_aptsources_list() function) is > > different for both builds, sstate-cache can't be reused for "isar- > > bootstrap-<host,target>" and for all other tasks since they depend > > on > > those ones. > > > > Any ideas how to properly deal with this? > > > > By the way, kas users are not affected, since fixed paths are used > > (e.g., "/work/isar", "/build") under the container. > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, > > send an > > email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar- > > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.de. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Isar can't reuse sstate-cache build from different isar location 2023-09-07 15:23 ` Uladzimir Bely @ 2023-09-07 15:27 ` Uladzimir Bely 2023-09-07 15:56 ` Uladzimir Bely 0 siblings, 1 reply; 5+ messages in thread From: Uladzimir Bely @ 2023-09-07 15:27 UTC (permalink / raw) To: Schmidt, Adriaan, isar-users On Thu, 2023-09-07 at 18:23 +0300, Uladzimir Bely wrote: > On Thu, 2023-09-07 at 13:29 +0000, Schmidt, Adriaan wrote: > > Hi, > > > > I believe the problem was introduced in 9e7c63ee6. This causes > > absolute paths to end up in SRC_URI. > > > > Can you try this patch: > > --- > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index 8af73a9b..28264024 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -156,7 +156,7 @@ def get_aptsources_list(d): > > for p in apt_sources_list: > > try: > > f = bb.parse.resolve_file(p, d) > > - ret.append(f) > > + ret.append(p) > > except FileNotFoundError as e: > > bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') > > return ret > > --- > > > > I haven't tested it. If this is not it I'll have to dig a little > > deeper. > > > > While debugging, I was looking at this also. "p" is something like > "conf/distro/debian-bullseye.list", "f" just prepends it with an > absolute path. I tried this replacement and it resulted to something > like "file not found", since LAYERDIR_core and LAYERDIR_isar are not > directly avalable in FILEEXTRAPATH and so on. > > But I'll recheck it. > Yes, I checked this again. It just fails during parsing stage with "[Errno 2] No such file or directory: 'conf/distro/debian-buster.list'" > > Generally re-use of sstate artifacts is supported, and we were > > careful to avoid absolute paths in variables. > > In fact, the isar-sstate script has a "lint" command to detect > > them, > > but that fails here, because the > > path has a "file://" prefix. > > > > Thanks, > > Adriaan > > > > Uladzimir Bely, Thursday, September 7, 2023 1:32 PM: > > > I tried to use the same SSTATE_DIR for different isar locations > > > on > > > the > > > same machine under the same user and found that it does not work. > > > > > > Quick steps to reproduce it: > > > > > > ## Prepare build environment with two copies of isar: > > > > > > ``` > > > ~ $ mkdir -m 777 test-sstate > > > ~ $ cd test-sstate/ > > > ~/test-sstate $ docker run --privileged -it -v .:/build > > > ghcr.io/siemens/kas/kas-isar:4.0 > > > builder@ffe5274283d3:~$ cd /build > > > builder@ffe5274283d3:/build$ git clone > > > https://github.com/ilbers/isar.git isar-1 > > > builder@ffe5274283d3:/build$ git clone > > > https://github.com/ilbers/isar.git isar-2 > > > ``` > > > > > > ## Build isar-1: > > > > > > ``` > > > builder@ffe5274283d3:/build$ cd /build/isar-1 > > > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-1/build$ time bitbake > > > mc:qemuamd64- > > > bullseye:isar-image-base > > > real 4m38.759s > > > user 0m4.248s > > > sys 0m1.068s > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads > > > 229M /build/downloads > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate- > > > cache > > > 1.1G /build/sstate-cache > > > ``` > > > > > > ## Build isar-2: > > > > > > ``` > > > builder@ffe5274283d3:/build$ cd /build/isar-2 > > > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-2/build$ time bitbake > > > mc:qemuamd64- > > > bullseye:isar-image-base > > > real 4m32.314s > > > user 0m3.975s > > > sys 0m0.898s > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads > > > 229M /build/downloads > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate- > > > cache/ > > > 2.2G /build/sstate-cache/ > > > ``` > > > > > > Second build took almost the same time as first one. Sstate cache > > > was > > > not reused, second build just added new files to the cache. > > > > > > I tried similar approach with Yocto, and it works in it. > > > > > > After debugging, I have concluded, that this issue comes from > > > "bootstrap" task that is not sstate-cache compatible. It's caused > > > by > > > our manipulations with SRC_URI we do in it, so that it's > > > different > > > for > > > both builds. E.g.: > > > > > > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > > > 1/meta/conf/distro/debian-bullseye.list" > > > SRC_URI="file://locale file://chroot-setup.sh file:///build/isar- > > > 2/meta/conf/distro/debian-bullseye.list" > > > > > > SRC_URI is a variable that is tracked in sstate-cache. Since the > > > last > > > part (that we add "on fly" with get_aptsources_list() function) > > > is > > > different for both builds, sstate-cache can't be reused for > > > "isar- > > > bootstrap-<host,target>" and for all other tasks since they > > > depend > > > on > > > those ones. > > > > > > Any ideas how to properly deal with this? > > > > > > By the way, kas users are not affected, since fixed paths are > > > used > > > (e.g., "/work/isar", "/build") under the container. > > > > > > -- > > > You received this message because you are subscribed to the > > > Google > > > Groups > > > "isar-users" group. > > > To unsubscribe from this group and stop receiving emails from it, > > > send an > > > email to isar-users+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar- > > > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.de. > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Isar can't reuse sstate-cache build from different isar location 2023-09-07 15:27 ` Uladzimir Bely @ 2023-09-07 15:56 ` Uladzimir Bely 0 siblings, 0 replies; 5+ messages in thread From: Uladzimir Bely @ 2023-09-07 15:56 UTC (permalink / raw) To: Schmidt, Adriaan, isar-users On Thu, 2023-09-07 at 18:27 +0300, Uladzimir Bely wrote: > On Thu, 2023-09-07 at 18:23 +0300, Uladzimir Bely wrote: > > On Thu, 2023-09-07 at 13:29 +0000, Schmidt, Adriaan wrote: > > > Hi, > > > > > > I believe the problem was introduced in 9e7c63ee6. This causes > > > absolute paths to end up in SRC_URI. > > > > > > Can you try this patch: > > > --- > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > index 8af73a9b..28264024 100644 > > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > @@ -156,7 +156,7 @@ def get_aptsources_list(d): > > > for p in apt_sources_list: > > > try: > > > f = bb.parse.resolve_file(p, d) > > > - ret.append(f) > > > + ret.append(p) > > > except FileNotFoundError as e: > > > bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') > > > return ret > > > --- > > > > > > I haven't tested it. If this is not it I'll have to dig a little > > > deeper. > > > > > > > While debugging, I was looking at this also. "p" is something like > > "conf/distro/debian-bullseye.list", "f" just prepends it with an > > absolute path. I tried this replacement and it resulted to > > something > > like "file not found", since LAYERDIR_core and LAYERDIR_isar are > > not > > directly avalable in FILEEXTRAPATH and so on. > > > > But I'll recheck it. > > > > Yes, I checked this again. It just fails during parsing stage with > "[Errno 2] No such file or directory: 'conf/distro/debian- > buster.list'" > Anyway, when I first reverted ae3c091c and then 9e7c63ee, sharing sstate-cache between different isar looks working again. > > > Generally re-use of sstate artifacts is supported, and we were > > > careful to avoid absolute paths in variables. > > > In fact, the isar-sstate script has a "lint" command to detect > > > them, > > > but that fails here, because the > > > path has a "file://" prefix. > > > > > > Thanks, > > > Adriaan > > > > > > Uladzimir Bely, Thursday, September 7, 2023 1:32 PM: > > > > I tried to use the same SSTATE_DIR for different isar locations > > > > on > > > > the > > > > same machine under the same user and found that it does not > > > > work. > > > > > > > > Quick steps to reproduce it: > > > > > > > > ## Prepare build environment with two copies of isar: > > > > > > > > ``` > > > > ~ $ mkdir -m 777 test-sstate > > > > ~ $ cd test-sstate/ > > > > ~/test-sstate $ docker run --privileged -it -v .:/build > > > > ghcr.io/siemens/kas/kas-isar:4.0 > > > > builder@ffe5274283d3:~$ cd /build > > > > builder@ffe5274283d3:/build$ git clone > > > > https://github.com/ilbers/isar.git isar-1 > > > > builder@ffe5274283d3:/build$ git clone > > > > https://github.com/ilbers/isar.git isar-2 > > > > ``` > > > > > > > > ## Build isar-1: > > > > > > > > ``` > > > > builder@ffe5274283d3:/build$ cd /build/isar-1 > > > > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > > > builder@ffe5274283d3:/build/isar-1/build$ time bitbake > > > > mc:qemuamd64- > > > > bullseye:isar-image-base > > > > real 4m38.759s > > > > user 0m4.248s > > > > sys 0m1.068s > > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh > > > > /build/downloads > > > > 229M /build/downloads > > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate- > > > > cache > > > > 1.1G /build/sstate-cache > > > > ``` > > > > > > > > ## Build isar-2: > > > > > > > > ``` > > > > builder@ffe5274283d3:/build$ cd /build/isar-2 > > > > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > > 'SSTATE_DIR="/build/sstate-cache"' >> conf/local.conf > > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > > 'DL_DIR="/build/downloads"' >> conf/local.conf > > > > builder@ffe5274283d3:/build/isar-2/build$ time bitbake > > > > mc:qemuamd64- > > > > bullseye:isar-image-base > > > > real 4m32.314s > > > > user 0m3.975s > > > > sys 0m0.898s > > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh > > > > /build/downloads > > > > 229M /build/downloads > > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate- > > > > cache/ > > > > 2.2G /build/sstate-cache/ > > > > ``` > > > > > > > > Second build took almost the same time as first one. Sstate > > > > cache > > > > was > > > > not reused, second build just added new files to the cache. > > > > > > > > I tried similar approach with Yocto, and it works in it. > > > > > > > > After debugging, I have concluded, that this issue comes from > > > > "bootstrap" task that is not sstate-cache compatible. It's > > > > caused > > > > by > > > > our manipulations with SRC_URI we do in it, so that it's > > > > different > > > > for > > > > both builds. E.g.: > > > > > > > > SRC_URI=" > > > > file://locale file://chroot-setup.sh file:///build/isar- > > > > 1/meta/conf/distro/debian-bullseye.list" > > > > SRC_URI=" > > > > file://locale file://chroot-setup.sh file:///build/isar- > > > > 2/meta/conf/distro/debian-bullseye.list" > > > > > > > > SRC_URI is a variable that is tracked in sstate-cache. Since > > > > the > > > > last > > > > part (that we add "on fly" with get_aptsources_list() function) > > > > is > > > > different for both builds, sstate-cache can't be reused for > > > > "isar- > > > > bootstrap-<host,target>" and for all other tasks since they > > > > depend > > > > on > > > > those ones. > > > > > > > > Any ideas how to properly deal with this? > > > > > > > > By the way, kas users are not affected, since fixed paths are > > > > used > > > > (e.g., "/work/isar", "/build") under the container. > > > > > > > > -- > > > > You received this message because you are subscribed to the > > > > Google > > > > Groups > > > > "isar-users" group. > > > > To unsubscribe from this group and stop receiving emails from > > > > it, > > > > send an > > > > email to isar-users+unsubscribe@googlegroups.com. > > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/isar- > > > > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.d > > > > e. > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-09-07 15:56 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-09-07 11:31 Isar can't reuse sstate-cache build from different isar location Uladzimir Bely 2023-09-07 13:29 ` Schmidt, Adriaan 2023-09-07 15:23 ` Uladzimir Bely 2023-09-07 15:27 ` Uladzimir Bely 2023-09-07 15:56 ` Uladzimir Bely
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox