From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6788114222392803328 X-Received: by 2002:a17:906:138f:: with SMTP id f15mr30866272ejc.370.1580904673475; Wed, 05 Feb 2020 04:11:13 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:ce99:: with SMTP id y25ls797399edv.6.gmail; Wed, 05 Feb 2020 04:11:12 -0800 (PST) X-Google-Smtp-Source: APXvYqw31JWYi1BL28I2lpUg9lLwi7ukHkRyhfomuHnanbyUa1kEmWj5Xu3/Oaio67gcmJHs8njJ X-Received: by 2002:a05:6402:144c:: with SMTP id d12mr4608751edx.265.1580904672556; Wed, 05 Feb 2020 04:11:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580904672; cv=none; d=google.com; s=arc-20160816; b=fLMihI8iA0T1UtPYT3yfOw3BYSxxvT7nIMlQgc5WG3frYqowr6fQfKUbQQrgq9IQ45 88AkUqR8JMX62leKswlgj1DbrzFX4UDRkmzbvpqx/0MO1RO5vbA3gPYB8dbHWo99TPAo zEeG1ZdLS5U0xwuZhWbeWXTm4lv5uCNxNRFMZ69JEKNbcMxuWxkbVEQyCeiLy6wUNUnD AWpsb/psOHnlGBwqlJgKft6/J45CstUpYQftA2Wyqi45MoPjU2gzf/ASDHke+Bx8Xxe6 oDBi7Dmbonqxd//fZ60jPSfxZi5J4+2XSw38gOWiEW+FRHZG4fpGWS3eLubnjT4Kw8we MEOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=3Av33SJXUuXmTBUnOeXRio+BuozBk2NoLF6lOAH/axs=; b=tE8TyDDKRTK+eJPoICwmnXmIqAQ9e2lPOAarTxWnFQOidX23OB7kJa4+XfPYJazmBX eKa636D+v4Z8RJEN0tn6iwOM2j5N7cAOgD0E98DbSJf+vKfjc2jdR29/CVOoao95vAFb dhVQr1IcRC+KmRq1nARm65QAE8i74L9cRsK/4k5QF0UEhFaHUMfmNV/8oiGPgK3egBkC 2rKDinWSTd5C8DQ+7FitKFM6hLXDmWVhQ33fmeaVxvIxh5M+1J6ugzDa+9WJi20L8Gdj feCu74x6hlo3QVJ6Fnn21oyFVHm27cPS4zuMVrgpRAO7DaxGCind4upkCMSGC4rkxza5 GhIw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id x18si1032366eds.2.2020.02.05.04.11.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Feb 2020 04:11:12 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 015CBBkS007189 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 5 Feb 2020 13:11:11 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.193]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 015CBBSg018282; Wed, 5 Feb 2020 13:11:11 +0100 Date: Wed, 5 Feb 2020 13:11:11 +0100 From: Henning Schild To: "Su, Bao Cheng (RC-CN DI FA R&D SW)" Cc: "Kiszka, Jan (CT RDA IOT SES-DE)" , "isar-users@googlegroups.com" , Vijai Kumar K Subject: Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from isar-apt Message-ID: <20200205131111.053bdc95@md1za8fc.ad001.siemens.net> In-Reply-To: <2ad5750c-43f6-4b19-aeee-a9978fe39886@siemens.com> References: <20200131143000.14873-1-henning.schild@siemens.com> <20200131143000.14873-23-henning.schild@siemens.com> <16977d1f-fb4d-481b-9c55-53ed064df333@siemens.com> <20200203192701.2700782a@md1za8fc.ad001.siemens.net> <20200203195919.02560450@md1za8fc.ad001.siemens.net> <5a62e0f4-f8a9-4e08-8010-5a47f49bd2b6@siemens.com> <2ad5750c-43f6-4b19-aeee-a9978fe39886@siemens.com> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: cLi9TAwX8uL2 On Tue, 4 Feb 2020 05:10:05 +0000 "Su, Bao Cheng (RC-CN DI FA R&D SW)" wrote: > Still exists. >=20 > Under `build/tmp/work/`: >=20 > $ grep "command not found" . -r --include=3D"log.do_*" > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/= bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/= bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/= bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/= bash: > line 5: repo_contains_package: command not found >=20 > $ grep "command not found" . -r --include=3D"log.do_*" | wc -l > 1596 >=20 > This is only a half-way build, for a full build, there are more > occurrences. Ok i can reproduce the issue and am investigating, will probably take the other flock notation. Funny thing is that despite our efforts to make the build env reproducible we are using "dash" in a plain Isar build and "bash" in a kas based build. That is what dash has to say .. /bin/sh: 9: repo_contains_package: not found So the grep did not find it. What i still do not understand is how the download cache seems to be or is exactly in the state i would expect. You probably did not patch because of the "command not found" but because of a result of some sort. Have packages been missing in the dl cache? What kind of packages? Did this break your offline rebuild, or did it show up in another context (maybe clearing). Thanks, Henning > With best regards > ________________________________ > =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A Su, Bao Cheng (RC-CN DI FA R&D SW) > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A 2020=E5=B9=B42=E6=9C=884=E6= =97=A5=E6=98=9F=E6=9C=9F=E4=BA=8C 10:11 > =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A Schild, Henning (CT RDA IOT SES-DE);= Kiszka, Jan (CT RDA IOT > SES-DE) =E6=8A=84=E9=80=81=EF=BC=9A isar-users@googlegroups.com; Vijai Ku= mar K > =E4=B8=BB=E9=A2=98=EF=BC=9A Re: [PATCHv4 22/26] meta: deb-dl-dir: do not = cache debs from > isar-apt >=20 > after building, grep "command not found" or "repo_contains_package" > in the log.do_* file. >=20 > I met this problem on an early version of v4 round patch, not sure if > still exists on the latest version. will run a test build against the > latest patch to check. >=20 > Sorry for mailing via phone, due to coronavirus, not convenient for > me to use laptop emails. >=20 > With best regards > ________________________________ > =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A Schild, Henning (CT RDA IOT SES-DE) > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A 2020=E5=B9=B42=E6=9C=884=E6= =97=A5=E6=98=9F=E6=9C=9F=E4=BA=8C 02:59 > =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A Kiszka, Jan (CT RDA IOT SES-DE) > =E6=8A=84=E9=80=81=EF=BC=9A isar-users@googlegroups.com; Su, Bao Cheng (R= C-CN DI FA R&D > SW); Vijai Kumar K =E4=B8=BB=E9=A2=98=EF=BC=9A Re: [PATCHv4 22/26] meta: = deb-dl-dir: do > not cache debs from isar-apt >=20 > Am Mon, 3 Feb 2020 19:27:01 +0100 > schrieb "[ext] Henning Schild" : >=20 > > Am Mon, 3 Feb 2020 18:20:11 +0100 > > schrieb Jan Kiszka : > > =20 > > > On 31.01.20 15:29, [ext] Henning Schild wrote: =20 > > > > From: Henning Schild > > > > > > > > Packages from isar-apt are not downloaded from the outside and > > > > should not be cached. > > > > > > > > Signed-off-by: Henning Schild > > > > --- > > > > meta/classes/deb-dl-dir.bbclass | 14 ++++++++++---- > > > > 1 file changed, 10 insertions(+), 4 deletions(-) > > > > > > > > diff --git a/meta/classes/deb-dl-dir.bbclass > > > > b/meta/classes/deb-dl-dir.bbclass index ab4b1f09..f9699603 > > > > 100644 --- a/meta/classes/deb-dl-dir.bbclass > > > > +++ b/meta/classes/deb-dl-dir.bbclass > > > > @@ -3,8 +3,10 @@ > > > > # > > > > # SPDX-License-Identifier: MIT > > > > > > > > +inherit repository > > > > + > > > > deb_dl_dir_import() { > > > > - export pc=3D"${DEBDIR}/${DISTRO}" > > > > + export pc=3D"${DEBDIR}/${DISTRO}/" > > > > export rootfs=3D"${1}" > > > > [ ! -d "${pc}" ] && return 0 > > > > flock -s "${pc}".lock -c ' \ > > > > @@ -14,12 +16,16 @@ deb_dl_dir_import() { > > > > } > > > > > > > > deb_dl_dir_export() { > > > > - export pc=3D"${DEBDIR}/${DISTRO}" > > > > + export pc=3D"${DEBDIR}/${DISTRO}/" > > > > export rootfs=3D"${1}" > > > > mkdir -p "${pc}" > > > > flock "${pc}".lock -c ' \ > > > > - sudo find "${rootfs}"/var/cache/apt/archives/ -type f > > > > -iname '*\.deb' \ > > > > - -exec cp -f '{}' "${pc}" \; > > > > + find "${rootfs}"/var/cache/apt/archives/ -type f -iname > > > > '*\.deb' |\ > > > > + while read p; do > > > > + repo_contains_package > > > > "${REPO_ISAR_DIR}"/"${DISTRO}" "${p}" && \ =20 > > > > > > repo_contains_package may not be found inside the flock shell > > > context, as Bao Cheng noticed out. He suggests the pattern =20 > > > > That is possible indeed. I remember that i struggled re-using the > > function since i did not want to code it twice. > > > > But to be honest i do not understand the problem with the given > > description. Bao Cheng please go into more detail. =20 >=20 > This is a hot code-path and if it is _very_ broken that would be very > visible. >=20 > Just tried a build and am looking at the download cache and isar-apt >=20 > find tmp/deploy/isar-apt/ -iname *hello* > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-db= gsym_0.1_amd64.deb > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-de= v_0.1_amd64.deb > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello_0.= 1_amd64.deb > .... > tmp/deploy/isar-apt/apt/debian-buster/pool/main/h/hello-isar/hello-isar_0= .3_amd64.deb >=20 > Ok so we have the hello stuff in isar-apt, the own package as well as > the rebuild upstream (also own). >=20 > And nothing in the download cache on hello -> expected for the image > find downloads/deb/ -iname *hello* > >=20 > Meaning the filtering worked in my case. >=20 > And the caching of all required outside stuff works as well, since the > offline rebuild works. >=20 > Caching and filtering are the two main aspects of that code. >=20 > Please make sure to give feedback for sure, i do not want such a > unclear description slowing down the merge even more. >=20 > Henning >=20 > > Maybe the filtering indeed does not work, i will try that on a > > simple test now. And it might be a good idea to make sure that code > > gets a "set -e". > > > > Henning > > =20 > > > (flock 8 > > > ... > > > ) 8>${LOCKFILE} > > > > > > Bao Check, maybe you can describe how you noticed. > > > > > > Jan > > > =20 > > > > + continue > > > > + sudo cp -f "${p}" "${pc}" > > > > + done > > > > sudo chown -R $(id -u):$(id -g) "${pc}" > > > > ' > > > > } > > > > =20 > > > =20 > > =20 >=20