From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6788114222392803328 X-Received: by 2002:a17:906:7d6:: with SMTP id m22mr26597968ejc.335.1580833614959; Tue, 04 Feb 2020 08:26:54 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:1b06:: with SMTP id by6ls5711295edb.1.gmail; Tue, 04 Feb 2020 08:26:54 -0800 (PST) X-Google-Smtp-Source: APXvYqyf3FygDtbro17JINWoWCFWNtx/cdFhL4brVX5GVd9+Q0VsK7Dui2My51CcTZUQYVV7lQKs X-Received: by 2002:a05:6402:3132:: with SMTP id dd18mr1025050edb.118.1580833614179; Tue, 04 Feb 2020 08:26:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580833614; cv=none; d=google.com; s=arc-20160816; b=F8XnjBYeuRLOoFzePf9ucsZrGr7IDzKwLzbHbNU/a5DKEE7fPMTxwXAyOw7wbZmcut 86UWSTvWx0VkB1lmK1mMHJNYzc62/E5xLuXliLE06EbXzoOCMtTN39Rs39ez/6O1b8Ve ICsGY7IZirr7cjsNGuFQbOGsU5g3GxymgRzCfvgphCLxtKwgPdksf6YZyQ5w7Xo3Ia/V m+pSELRnU+edP0ORnGiflfD/a20O8B5bS/rjXv9Gsr5n2R0QtPUvGfhbvB1BJ/khU63Z UvQ2JXTio/Sv44oikaCwPkmSb6fkYjkbERRJnOQEJglpjpKrdOszW45ttdcmYFg4JoGE fqEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=nlCDtmKjBnLXk4jOpT+nX1SSv3lFTie9muNqXfZmw50=; b=W21UjRh4Rr/hOfEdq2TL2Ur/oNq9oCzNlQIY8rFIUiBJi/m08cnEEWOxSWT31BFXGy aWjHc2E3VJDkQxKeQxyYKp4Z5pjWsRHKUhHm2i248WhYl4Sln8kkflk92VNAPqRmECdj ey7QxTjBpVk7NuD/KsPQ5Q+tpjYJ4ndIhntuYrugsJRTW3jMj/XBaAd20l8Du8K5Gx/X U/P2qGOv7LNGonOhMx6K3rAE68851IZ0yQRZzHZOeNp/GBtMqmbzratY9z/8jcWuax9Q 1F77yebfTVeSsKqlsvQFuMYdvpM3BS7m3YLQLFx9Ru6EW2VBGwaUuls1qBqwHF2Q5RI/ eQLA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id z20si870762ejx.1.2020.02.04.08.26.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Feb 2020 08:26:54 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 014GQrs4021717 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 4 Feb 2020 17:26:53 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.193]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 014GQrj1005155; Tue, 4 Feb 2020 17:26:53 +0100 Date: Tue, 4 Feb 2020 17:26:52 +0100 From: Henning Schild To: Jan Kiszka Cc: "Su, Bao Cheng (RC-CN DI FA R&D SW)" , "isar-users@googlegroups.com" , Vijai Kumar K Subject: Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from isar-apt Message-ID: <20200204172652.3c3d9039@md1za8fc.ad001.siemens.net> In-Reply-To: <0f7adda2-f4e0-5788-c043-9c182ce9e90e@siemens.com> References: <20200131143000.14873-1-henning.schild@siemens.com> <20200131143000.14873-23-henning.schild@siemens.com> <16977d1f-fb4d-481b-9c55-53ed064df333@siemens.com> <20200203192701.2700782a@md1za8fc.ad001.siemens.net> <20200203195919.02560450@md1za8fc.ad001.siemens.net> <5a62e0f4-f8a9-4e08-8010-5a47f49bd2b6@siemens.com> <2ad5750c-43f6-4b19-aeee-a9978fe39886@siemens.com> <0f7adda2-f4e0-5788-c043-9c182ce9e90e@siemens.com> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: 79l9C0cIfuqc On Tue, 4 Feb 2020 06:14:00 +0100 Jan Kiszka wrote: > On 04.02.20 06:10, Su, Bao Cheng (RC-CN DI FA R&D SW) wrote: > > Still exists. > >=20 > > Under `build/tmp/work/`: > >=20 > > $ grep "command not found" . -r --include=3D"log.do_*" > > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bi= n/bash:=20 > > line 5: repo_contains_package: command not found > > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bi= n/bash:=20 > > line 5: repo_contains_package: command not found > > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bi= n/bash:=20 > > line 5: repo_contains_package: command not found > > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bi= n/bash:=20 > > line 5: repo_contains_package: command not found > >=20 > > $ grep "command not found" . -r --include=3D"log.do_*" | wc -l > > 1596 > >=20 > > This is only a half-way build, for a full build, there are more > > occurrences. =20 >=20 > And if I read the code correctly, the issue only prevents an=20 > optimization, not the correct functioning of the code. Thus, when > fixing this, building should be even faster :). No, it is putting packages we built into the dl-cache where they really do not belong. Having them in there is harmful because it will most likely cause issues when rebuilding, checksum changes on pre-downloaded files ... something that can not happen with real downloads. But something that is probably common in Isar, when people do not bump PV on every change. Henning > Thanks, > Jan >=20 > >=20 > > With best regards > > ------------------------------------------------------------------------ > > *=E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A* Su, Bao Cheng (RC-CN DI FA R&D S= W) > > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A* 2020=E5=B9=B42=E6=9C=88= 4=E6=97=A5=E6=98=9F=E6=9C=9F=E4=BA=8C 10:11 > > *=E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A* Schild, Henning (CT RDA IOT SES-= DE); Kiszka, Jan (CT RDA > > IOT SES-DE) > > *=E6=8A=84=E9=80=81=EF=BC=9A* isar-users@googlegroups.com; Vijai Kumar K > > *=E4=B8=BB=E9=A2=98=EF=BC=9A* Re: [PATCHv4 22/26] meta: deb-dl-dir: do = not cache debs > > from isar-apt > >=20 > > after building, grep "command not found" or "repo_contains_package" > > in the log.do_* file. > >=20 > > I met this problem on an early version of v4 round patch, not sure > > if still exists on the latest version. will run a test build > > against the latest patch to check. > >=20 > > Sorry for mailing via phone, due to coronavirus, not convenient for > > me to use laptop emails. > >=20 > > With best regards > > ------------------------------------------------------------------------ > > *=E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A* Schild, Henning (CT RDA IOT SES-= DE) > > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A* 2020=E5=B9=B42=E6=9C=88= 4=E6=97=A5=E6=98=9F=E6=9C=9F=E4=BA=8C 02:59 > > *=E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A* Kiszka, Jan (CT RDA IOT SES-DE) > > *=E6=8A=84=E9=80=81=EF=BC=9A* isar-users@googlegroups.com; Su, Bao Chen= g (RC-CN DI FA > > R&D SW); Vijai Kumar K > > *=E4=B8=BB=E9=A2=98=EF=BC=9A* Re: [PATCHv4 22/26] meta: deb-dl-dir: do = not cache debs > > from isar-apt > >=20 > > Am Mon, 3 Feb 2020 19:27:01 +0100 > > schrieb "[ext] Henning Schild" : > > =20 > >> Am Mon, 3 Feb 2020 18:20:11 +0100 > >> schrieb Jan Kiszka : > >> =20 > >> > On 31.01.20 15:29, [ext] Henning Schild wrote: =20 > >> > > From: Henning Schild > >> > >=20 > >> > > Packages from isar-apt are not downloaded from the outside and > >> > > should not be cached. > >> > >=20 > >> > > Signed-off-by: Henning Schild > >> > > --- > >> > >=C2=A0=C2=A0 meta/classes/deb-dl-dir.bbclass | 14 ++++++++++---- > >> > >=C2=A0=C2=A0 1 file changed, 10 insertions(+), 4 deletions(-) > >> > >=20 > >> > > diff --git a/meta/classes/deb-dl-dir.bbclass > >> > > b/meta/classes/deb-dl-dir.bbclass index ab4b1f09..f9699603 > >> > > 100644 --- a/meta/classes/deb-dl-dir.bbclass > >> > > +++ b/meta/classes/deb-dl-dir.bbclass > >> > > @@ -3,8 +3,10 @@ > >> > >=C2=A0=C2=A0 # > >> > >=C2=A0=C2=A0 # SPDX-License-Identifier: MIT > >> > > =20 > >> > > +inherit repository > >> > > + > >> > >=C2=A0=C2=A0 deb_dl_dir_import() { > >> > > -=C2=A0=C2=A0=C2=A0 export pc=3D"${DEBDIR}/${DISTRO}" > >> > > +=C2=A0=C2=A0=C2=A0 export pc=3D"${DEBDIR}/${DISTRO}/" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 export rootfs=3D"${1}" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 [ ! -d "${pc}" ] && return 0 > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 flock -s "${pc}".lock -c ' \ > >> > > @@ -14,12 +16,16 @@ deb_dl_dir_import() { > >> > >=C2=A0=C2=A0 } > >> > > =20 > >> > >=C2=A0=C2=A0 deb_dl_dir_export() { > >> > > -=C2=A0=C2=A0=C2=A0 export pc=3D"${DEBDIR}/${DISTRO}" > >> > > +=C2=A0=C2=A0=C2=A0 export pc=3D"${DEBDIR}/${DISTRO}/" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 export rootfs=3D"${1}" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mkdir -p "${pc}" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 flock "${pc}".lock -c ' \ > >> > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo find "${rootfs}"/= var/cache/apt/archives/ -type f > >> > > -iname '*\.deb' \ > >> > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 -exec cp -f '{}' "${pc}" \; > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 find "${rootfs}"/var/c= ache/apt/archives/ -type f > >> > > -iname '*\.deb' |\ > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 while read p; do > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 repo_contains_package > >> > > "${REPO_ISAR_DIR}"/"${DISTRO}" "${p}" && \ =20 > >> >=20 > >> > repo_contains_package may not be found inside the flock shell > >> > context, as Bao Cheng noticed out. He suggests the pattern =20 > >>=20 > >> That is possible indeed. I remember that i struggled re-using the > >> function since i did not want to code it twice. > >>=20 > >> But to be honest i do not understand the problem with the given > >> description. Bao Cheng please go into more detail. =20 > >=20 > > This is a hot code-path and if it is _very_ broken that would be > > very visible. > >=20 > > Just tried a build and am looking at the download cache and isar-apt > >=20 > > find tmp/deploy/isar-apt/ -iname *hello* > > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello > > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-= dbgsym_0.1_amd64.deb > > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-= dev_0.1_amd64.deb > > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello_= 0.1_amd64.deb > > .... > > tmp/deploy/isar-apt/apt/debian-buster/pool/main/h/hello-isar/hello-isar= _0.3_amd64.deb > >=20 > > Ok so we have the hello stuff in isar-apt, the own package as well > > as the rebuild upstream (also own). > >=20 > > And nothing in the download cache on hello -> expected for the image > > find downloads/deb/ -iname *hello* > > > >=20 > > Meaning the filtering worked in my case. > >=20 > > And the caching of all required outside stuff works as well, since > > the offline rebuild works. > >=20 > > Caching and filtering are the two main aspects of that code. > >=20 > > Please make sure to give feedback for sure, i do not want such a > > unclear description slowing down the merge even more. > >=20 > > Henning > > =20 > >> Maybe the filtering indeed does not work, i will try that on a > >> simple test now. And it might be a good idea to make sure that > >> code gets a "set -e". > >>=20 > >> Henning > >> =20 > >> > (flock 8 > >> >=C2=A0=C2=A0 ... > >> > ) 8>${LOCKFILE} > >> > > >> > Bao Check, maybe you can describe how you noticed. > >> >=20 > >> > Jan > >> > =20 > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 continue > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 sudo cp -f "${p}" "${pc}" > >> > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 done > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sudo c= hown -R $(id -u):$(id -g) "${pc}" > >> > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ' > >> > >=C2=A0=C2=A0 } > >> > > =20 > >> > =20 > >> =20 > > =20 >=20