From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7276045224940404736 X-Received: by 2002:a19:5007:0:b0:500:96dd:f95b with SMTP id e7-20020a195007000000b0050096ddf95bmr4919030lfb.59.1694100450809; Thu, 07 Sep 2023 08:27:30 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:511e:b0:3fa:95c3:7e99 with SMTP id o30-20020a05600c511e00b003fa95c37e99ls6041wms.2.-pod-prod-08-eu; Thu, 07 Sep 2023 08:27:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFGtuUBUxCnHWiFT/G2K+GXI9VyxUPsPQuDE9qIe2KVk/Hpr9ed6oOy3t28AGCGG8UMbdMO X-Received: by 2002:a05:600c:214d:b0:401:cbf6:e5cc with SMTP id v13-20020a05600c214d00b00401cbf6e5ccmr33833wml.22.1694100448926; Thu, 07 Sep 2023 08:27:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694100448; cv=none; d=google.com; s=arc-20160816; b=e0S1HqgxuzwSxu27uxdL9kQIVi6YmQBz+MFgh70HfK6/FoBCAxWCZsCz3f076ARCFv uYF0nD1iKzOR/jLEcHzIuPUER4BT1uYhGfq8YN+naUlHNnkikcxy+HlIPFK5glYYPm9E uTcnuo4jUj7ueekQX0fY5pvwsA/dCEfMZlInCEqYC+tw5DDLDzXZSAMWJgfnbbFe59t9 IknZBHBN+UviKyguT63G2gUW0Mthc7fBfNNW/Md5s6YtCN66p2XpZgm9O4ZK/4YZrvxE pxhvuQdk0A627xBDQ+DC3WTVo/hGXfEArYDpPZpcQsrhoKTZjK2obPVL+Myah+L2BkdS Yf2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id; bh=+z/rMFvzWwyd+pprNqXWgsQTCYwPxiseOwkizwM5kls=; fh=+5d5QSUkCcFduM7vOrvo2tuP7AKEM8Gp6bs3W6TI0pA=; b=eAQsg1yi9pceALyLA9QjsBKqGnucoh9D8wHMrbBJrZMx9XpU/Xe14d+jCQjv1nLVZ6 AbLmpa915ro9OpibC5+n8MZAPUrjUCBdxvAxwRMsDJpDzRq9QZnSJ1RjGXOr0GNFstfg SVC8Z6SCzoSvUU9WRWlmeKjMIEb+UGZ+4d1nQkEXiia7fTItpZ3F8KjVDSl2qzjaQ5QT RdCiYL8etyANt7BVgESNgmbsHuJMJ1PlWsDk6D5FTReQMltEbS7Uk2qyK+u4iP6wURUs yBYW8a+vEMi/8J1zgopVJKv1yfalvDzZqvlmSgZT1k5ANj2FB9ryLIlWoWqlX6fndJ5P 4ftg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id fk4-20020a05600c0cc400b003fefb9e1a6asi180098wmb.0.2023.09.07.08.27.28 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Sep 2023 08:27:28 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 387FRRQv019921 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 7 Sep 2023 17:27:27 +0200 Message-ID: <8a9d1f026ddd80dc48207f010e53a4a40c8e5ab0.camel@ilbers.de> Subject: Re: Isar can't reuse sstate-cache build from different isar location From: Uladzimir Bely To: "Schmidt, Adriaan" , "isar-users@googlegroups.com" Date: Thu, 07 Sep 2023 18:27:30 +0300 In-Reply-To: <5cb93d9fd0e8a8109b37d2e4d35481c899c68f2a.camel@ilbers.de> References: <7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel@ilbers.de> <5cb93d9fd0e8a8109b37d2e4d35481c899c68f2a.camel@ilbers.de> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.4 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: f131xeU46ZQd On Thu, 2023-09-07 at 18:23 +0300, Uladzimir Bely wrote: > On Thu, 2023-09-07 at 13:29 +0000, Schmidt, Adriaan wrote: > > Hi, > >=20 > > I believe the problem was introduced in 9e7c63ee6. This causes > > absolute paths to end up in SRC_URI. > >=20 > > Can you try this patch: > > --- > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index 8af73a9b..28264024 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -156,7 +156,7 @@ def get_aptsources_list(d): > > =C2=A0=C2=A0=C2=A0=C2=A0 for p in apt_sources_list: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 f =3D bb.parse.resolve_file(p, d) > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ret= .append(f) > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ret= .append(p) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 except FileNotFoundErr= or as e: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') > > =C2=A0=C2=A0=C2=A0=C2=A0 return ret > > --- > >=20 > > I haven't tested it. If this is not it I'll have to dig a little > > deeper. > >=20 >=20 > While debugging, I was looking at this also. "p" is something like > "conf/distro/debian-bullseye.list",=C2=A0 "f" just prepends it with an > absolute path. I tried this replacement and it resulted to something > like "file not found", since LAYERDIR_core and LAYERDIR_isar are not > directly avalable in FILEEXTRAPATH and so on. >=20 > But I'll recheck it. >=20 Yes, I checked this again. It just fails during parsing stage with "[Errno 2] No such file or directory: 'conf/distro/debian-buster.list'" > > Generally re-use of sstate artifacts is supported, and we were > > careful to avoid absolute paths in variables. > > In fact, the isar-sstate script has a "lint" command to detect > > them, > > but that fails here, because the > > path has a "file://" prefix. > >=20 > > Thanks, > > Adriaan > >=20 > > Uladzimir Bely, Thursday, September 7, 2023 1:32 PM:=20 > > > I tried to use the same SSTATE_DIR for different isar locations > > > on > > > the > > > same machine under the same user and found that it does not work. > > >=20 > > > Quick steps to reproduce it: > > >=20 > > > ## Prepare build environment with two copies of isar: > > >=20 > > > ``` > > > ~ $ mkdir -m 777 test-sstate > > > ~ $ cd test-sstate/ > > > ~/test-sstate $ docker run --privileged -it -v .:/build > > > ghcr.io/siemens/kas/kas-isar:4.0 > > > builder@ffe5274283d3:~$ cd /build > > > builder@ffe5274283d3:/build$ git clone > > > https://github.com/ilbers/isar.git=C2=A0isar-1 > > > builder@ffe5274283d3:/build$ git clone > > > https://github.com/ilbers/isar.git=C2=A0isar-2 > > > ``` > > >=20 > > > ## Build isar-1: > > >=20 > > > ``` > > > builder@ffe5274283d3:/build$ cd /build/isar-1 > > > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > 'SSTATE_DIR=3D"/build/sstate-cache"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-1/build$ echo > > > 'DL_DIR=3D"/build/downloads"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-1/build$ time bitbake > > > mc:qemuamd64- > > > bullseye:isar-image-base > > > real=C2=A0=C2=A0=C2=A0 4m38.759s > > > user=C2=A0=C2=A0=C2=A0 0m4.248s > > > sys=C2=A0=C2=A0=C2=A0=C2=A0 0m1.068s > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads > > > 229M=C2=A0=C2=A0=C2=A0 /build/downloads > > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate- > > > cache > > > 1.1G=C2=A0=C2=A0=C2=A0 /build/sstate-cache > > > ``` > > >=20 > > > ## Build isar-2: > > >=20 > > > ``` > > > builder@ffe5274283d3:/build$ cd /build/isar-2 > > > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > 'SSTATE_DIR=3D"/build/sstate-cache"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-2/build$ echo > > > 'DL_DIR=3D"/build/downloads"' >> conf/local.conf > > > builder@ffe5274283d3:/build/isar-2/build$ time bitbake > > > mc:qemuamd64- > > > bullseye:isar-image-base > > > real=C2=A0=C2=A0=C2=A0 4m32.314s > > > user=C2=A0=C2=A0=C2=A0 0m3.975s > > > sys=C2=A0=C2=A0=C2=A0=C2=A0 0m0.898s > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads > > > 229M=C2=A0=C2=A0=C2=A0 /build/downloads > > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate- > > > cache/ > > > 2.2G=C2=A0=C2=A0=C2=A0 /build/sstate-cache/ > > > ``` > > >=20 > > > Second build took almost the same time as first one. Sstate cache > > > was > > > not reused, second build just added new files to the cache. > > >=20 > > > I tried similar approach with Yocto, and it works in it. > > >=20 > > > After debugging, I have concluded, that this issue comes from > > > "bootstrap" task that is not sstate-cache compatible. It's caused > > > by > > > our manipulations with SRC_URI we do in it, so that it's > > > different > > > for > > > both builds. E.g.: > > >=20 > > > SRC_URI=3D"file://locale=C2=A0file://chroot-setup.sh=C2=A0file:///bui= ld/isar- > > > 1/meta/conf/distro/debian-bullseye.list" > > > SRC_URI=3D"file://locale=C2=A0file://chroot-setup.sh=C2=A0file:///bui= ld/isar- > > > 2/meta/conf/distro/debian-bullseye.list" > > >=20 > > > SRC_URI is a variable that is tracked in sstate-cache. Since the > > > last > > > part (that we add "on fly" with get_aptsources_list() function) > > > is > > > different for both builds, sstate-cache can't be reused for > > > "isar- > > > bootstrap-" and for all other tasks since they > > > depend > > > on > > > those ones. > > >=20 > > > Any ideas how to properly deal with this? > > >=20 > > > By the way, kas users are not affected, since fixed paths are > > > used > > > (e.g., "/work/isar", "/build") under the container. > > >=20 > > > -- > > > You received this message because you are subscribed to the > > > Google > > > Groups > > > "isar-users" group. > > > To unsubscribe from this group and stop receiving emails from it, > > > send an > > > email to isar-users+unsubscribe@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/isar- > > > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.de. >=20