From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7276045224940404736 X-Received: by 2002:a19:3852:0:b0:500:ac0b:8d52 with SMTP id d18-20020a193852000000b00500ac0b8d52mr4751136lfj.7.1694100216807; Thu, 07 Sep 2023 08:23:36 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:1ca:b0:4fe:12c4:b5ac with SMTP id f10-20020a05651201ca00b004fe12c4b5acls778088lfp.0.-pod-prod-05-eu; Thu, 07 Sep 2023 08:23:34 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHcchpaIZomMxqA8Upv/LjxJTQYwbVSmWeOm6kVnvf3C/3r/7jKuI8HomJAxUd7M7neVGG2 X-Received: by 2002:ac2:4a66:0:b0:4fb:89b3:3373 with SMTP id q6-20020ac24a66000000b004fb89b33373mr4676269lfp.43.1694100214403; Thu, 07 Sep 2023 08:23:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694100214; cv=none; d=google.com; s=arc-20160816; b=vQCKbkkXauXEz6gzCCcz+TkAx9J337EtO+IJ52LpjDkHcmKhbQc1dX1eYA0sD3e3Yz dzhZEzqY2tYcwm4CwUjRq7aez6hAOEu1YHsKJiMGcWWIN0xqwWCxgAoQVdb+4dSv84Cd OxL9133fjnDiUGGSroKNbAq0ptuNbkAqhEldBTmkNeKz2ycFH9TWIz5GRjX36pcXs5aL qjwGVinrQHzmMRPD2atSpgUDmonvwB5/M2gklrU4cnlj0L1/OLKOH43+k6cIGst4Tz/W qiNA/UwvqDy1uI55aBN/5rDCsXmJBEHJSUMdf8Cufnwnq7PNbxZJZeCG3FSXlnOkfquV YNXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id; bh=q9i+NN8olrlWFsG3yXxSH2e+bv2uen+6ZiLVeOMLOGg=; fh=+5d5QSUkCcFduM7vOrvo2tuP7AKEM8Gp6bs3W6TI0pA=; b=ZCnTcb9RTBA4hff4gbrc6hKmYaWmOeRId8yCsOw3rkHws8rkC4Qau02fiHHkgA9wKJ dujjB3o46qwuOhGz8XLw3lp3RivGhL5Ll3lkRoZ+KHMbG/Rt12gXu1SdrfCIOQ+amXif gJHLm/T0kNg3DaRAgV7dI3jNeY7bOsiqife/rdefx9Wtehe47eBaM4i5RGYJ6lFgyRcF vap8p8Exnt57dKFJNSdSMo1txx4brWEkBnu5kn94TQueT/6jbTIdQcBXSWflLs+gGyQA 8shRDOiwHbLWEOcR1s//r6nQXBFuzaRx3EPK2ccv4as4Ozhl3UCbDyDkepo7XOiZesme a9LQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id b18-20020a05640202d200b0051fe05f750asi1053376edx.2.2023.09.07.08.23.34 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Sep 2023 08:23:34 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 387FNW0n019848 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 7 Sep 2023 17:23:33 +0200 Message-ID: <5cb93d9fd0e8a8109b37d2e4d35481c899c68f2a.camel@ilbers.de> Subject: Re: Isar can't reuse sstate-cache build from different isar location From: Uladzimir Bely To: "Schmidt, Adriaan" , "isar-users@googlegroups.com" Date: Thu, 07 Sep 2023 18:23:35 +0300 In-Reply-To: References: <7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel@ilbers.de> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.4 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 5XKhcRJ1VkQB On Thu, 2023-09-07 at 13:29 +0000, Schmidt, Adriaan wrote: > Hi, >=20 > I believe the problem was introduced in 9e7c63ee6. This causes > absolute paths to end up in SRC_URI. >=20 > Can you try this patch: > --- > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index 8af73a9b..28264024 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -156,7 +156,7 @@ def get_aptsources_list(d): > =C2=A0=C2=A0=C2=A0=C2=A0 for p in apt_sources_list: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = f =3D bb.parse.resolve_file(p, d) > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ret.a= ppend(f) > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ret.a= ppend(p) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 except FileNotFoundError= as e: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = bb.fatal(os.strerror(errno.ENOENT) + ' "' + p + '"') > =C2=A0=C2=A0=C2=A0=C2=A0 return ret > --- >=20 > I haven't tested it. If this is not it I'll have to dig a little > deeper. >=20 While debugging, I was looking at this also. "p" is something like "conf/distro/debian-bullseye.list", "f" just prepends it with an absolute path. I tried this replacement and it resulted to something like "file not found", since LAYERDIR_core and LAYERDIR_isar are not directly avalable in FILEEXTRAPATH and so on. But I'll recheck it. > Generally re-use of sstate artifacts is supported, and we were > careful to avoid absolute paths in variables. > In fact, the isar-sstate script has a "lint" command to detect them, > but that fails here, because the > path has a "file://" prefix. >=20 > Thanks, > Adriaan >=20 > Uladzimir Bely, Thursday, September 7, 2023 1:32 PM:=20 > > I tried to use the same SSTATE_DIR for different isar locations on > > the > > same machine under the same user and found that it does not work. > >=20 > > Quick steps to reproduce it: > >=20 > > ## Prepare build environment with two copies of isar: > >=20 > > ``` > > ~ $ mkdir -m 777 test-sstate > > ~ $ cd test-sstate/ > > ~/test-sstate $ docker run --privileged -it -v .:/build > > ghcr.io/siemens/kas/kas-isar:4.0 > > builder@ffe5274283d3:~$ cd /build > > builder@ffe5274283d3:/build$ git clone > > https://github.com/ilbers/isar.git=C2=A0isar-1 > > builder@ffe5274283d3:/build$ git clone > > https://github.com/ilbers/isar.git=C2=A0isar-2 > > ``` > >=20 > > ## Build isar-1: > >=20 > > ``` > > builder@ffe5274283d3:/build$ cd /build/isar-1 > > builder@ffe5274283d3:/build/isar-1$ . isar-init-build-env > > builder@ffe5274283d3:/build/isar-1/build$ echo > > 'SSTATE_DIR=3D"/build/sstate-cache"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-1/build$ echo > > 'DL_DIR=3D"/build/downloads"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-1/build$ time bitbake > > mc:qemuamd64- > > bullseye:isar-image-base > > real=C2=A0=C2=A0=C2=A0 4m38.759s > > user=C2=A0=C2=A0=C2=A0 0m4.248s > > sys=C2=A0=C2=A0=C2=A0=C2=A0 0m1.068s > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/downloads > > 229M=C2=A0=C2=A0=C2=A0 /build/downloads > > builder@ffe5274283d3:/build/isar-1/build$ du -sh /build/sstate- > > cache > > 1.1G=C2=A0=C2=A0=C2=A0 /build/sstate-cache > > ``` > >=20 > > ## Build isar-2: > >=20 > > ``` > > builder@ffe5274283d3:/build$ cd /build/isar-2 > > builder@ffe5274283d3:/build/isar-2$ . isar-init-build-env > > builder@ffe5274283d3:/build/isar-2/build$ echo > > 'SSTATE_DIR=3D"/build/sstate-cache"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-2/build$ echo > > 'DL_DIR=3D"/build/downloads"' >> conf/local.conf > > builder@ffe5274283d3:/build/isar-2/build$ time bitbake > > mc:qemuamd64- > > bullseye:isar-image-base > > real=C2=A0=C2=A0=C2=A0 4m32.314s > > user=C2=A0=C2=A0=C2=A0 0m3.975s > > sys=C2=A0=C2=A0=C2=A0=C2=A0 0m0.898s > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/downloads > > 229M=C2=A0=C2=A0=C2=A0 /build/downloads > > builder@ffe5274283d3:/build/isar-2/build$ du -sh /build/sstate- > > cache/ > > 2.2G=C2=A0=C2=A0=C2=A0 /build/sstate-cache/ > > ``` > >=20 > > Second build took almost the same time as first one. Sstate cache > > was > > not reused, second build just added new files to the cache. > >=20 > > I tried similar approach with Yocto, and it works in it. > >=20 > > After debugging, I have concluded, that this issue comes from > > "bootstrap" task that is not sstate-cache compatible. It's caused > > by > > our manipulations with SRC_URI we do in it, so that it's different > > for > > both builds. E.g.: > >=20 > > SRC_URI=3D"file://locale=C2=A0file://chroot-setup.sh=C2=A0file:///build= /isar- > > 1/meta/conf/distro/debian-bullseye.list" > > SRC_URI=3D"file://locale=C2=A0file://chroot-setup.sh=C2=A0file:///build= /isar- > > 2/meta/conf/distro/debian-bullseye.list" > >=20 > > SRC_URI is a variable that is tracked in sstate-cache. Since the > > last > > part (that we add "on fly" with get_aptsources_list() function) is > > different for both builds, sstate-cache can't be reused for "isar- > > bootstrap-" and for all other tasks since they depend > > on > > those ones. > >=20 > > Any ideas how to properly deal with this? > >=20 > > By the way, kas users are not affected, since fixed paths are used > > (e.g., "/work/isar", "/build") under the container. > >=20 > > -- > > You received this message because you are subscribed to the Google > > Groups > > "isar-users" group. > > To unsubscribe from this group and stop receiving emails from it, > > send an > > email to isar-users+unsubscribe@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/isar- > > users/7f715c07ed85772dc5e2eafcb41f3ace734de9d4.camel%40ilbers.de.