From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6788114222392803328 X-Received: by 2002:a05:6402:298:: with SMTP id l24mr15428865edv.70.1580793244609; Mon, 03 Feb 2020 21:14:04 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:584f:: with SMTP id h15ls4847268ejs.9.gmail; Mon, 03 Feb 2020 21:14:03 -0800 (PST) X-Google-Smtp-Source: APXvYqwtJWF2O7y3PAa3I8t78NHDCDyNxaZhvY82DIwxiQlibrMf0w3TzB6HGU2ZeaTE/EiZb87C X-Received: by 2002:a17:906:7d17:: with SMTP id u23mr24529116ejo.318.1580793243863; Mon, 03 Feb 2020 21:14:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580793243; cv=none; d=google.com; s=arc-20160816; b=tx6MWah5dz1JB25B+Lyx4v4ZGb6HbnrQ/FvtXrHL4/9896shuambfLadtgaKCn8Pnm dqWcXowaSsHL5duFibb9OwPUtAfiEN19HSnEOBRgVZlSMn+CPhDmIsQWSuji1siFO0tU MgjpIM71/0t9yzxeGw9ujkiIjcmUTSkfa48wDIggrxFqW8BBlR13FVVf+TW0a55pWNvm 5ICZ216+7OXjrTSLe+HQBS9+c9TLtSC0aAzV9xWEIwZQafS3qD6ZYQxaBOTmWP3JyEt/ wNYlLG4Tdj2iuAWYmJjYFtIYv6qRgr/Db6RQkUNjPZ2i075nDJcB5OsqBO9qJT823mHP jlMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=XRS6kr1E7xvYDLZ2cAtI4aYP0q2nSONDHCfhBCeOSOQ=; b=v/ZosfveGmUagQZxqQaPpFJ6hFpyUwVVIP6XCLuJS67dSHwuK5F3r1nXDLAdEjKk1J Vs3ITBS9EUVUE6KdeHayM2ebSMk8abXfTyZ/TWN5MUJPOEM4yTrJqb6xFdVAtRMFXU8Z 1bsq63N8dD8t53j6bV0VojT70lhhyGmnpHNfxizgfMieQAgWAVonY9Eb4fxiPlCn/acQ MqJxQ9KaMtXVrnn+2DkTRFeEtkTXMbKqko7I0H1I9eCDn6QZGRaNGVT/arP1ctZb5Z1c KsR+dRFLYkJd2tLrPY4Puz+OJRF+oWmv39DyPcD4xmnAjMFdQHOykZdDyylfWatf1SEY nVOw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id ba12si779363edb.3.2020.02.03.21.14.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Feb 2020 21:14:03 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 0145E3tE020938 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 4 Feb 2020 06:14:03 +0100 Received: from [167.87.2.81] ([167.87.2.81]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 0145E1bc023151; Tue, 4 Feb 2020 06:14:02 +0100 Subject: Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from isar-apt To: "Su, Bao Cheng (RC-CN DI FA R&D SW)" , "Schild, Henning (CT RDA IOT SES-DE)" Cc: "isar-users@googlegroups.com" , Vijai Kumar K References: <20200131143000.14873-1-henning.schild@siemens.com> <20200131143000.14873-23-henning.schild@siemens.com> <16977d1f-fb4d-481b-9c55-53ed064df333@siemens.com> <20200203192701.2700782a@md1za8fc.ad001.siemens.net> <20200203195919.02560450@md1za8fc.ad001.siemens.net> <5a62e0f4-f8a9-4e08-8010-5a47f49bd2b6@siemens.com> <2ad5750c-43f6-4b19-aeee-a9978fe39886@siemens.com> From: Jan Kiszka Message-ID: <0f7adda2-f4e0-5788-c043-9c182ce9e90e@siemens.com> Date: Tue, 4 Feb 2020 06:14:00 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <2ad5750c-43f6-4b19-aeee-a9978fe39886@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: lOGgBuPp/iGT On 04.02.20 06:10, Su, Bao Cheng (RC-CN DI FA R&D SW) wrote: > Still exists. > > Under `build/tmp/work/`: > > $ grep "command not found" . -r --include="log.do_*" > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash: > line 5: repo_contains_package: command not found > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash: > line 5: repo_contains_package: command not found > > $ grep "command not found" . -r --include="log.do_*" | wc -l > 1596 > > This is only a half-way build, for a full build, there are more occurrences. And if I read the code correctly, the issue only prevents an optimization, not the correct functioning of the code. Thus, when fixing this, building should be even faster :). Thanks, Jan > > With best regards > ------------------------------------------------------------------------ > *发件人：* Su, Bao Cheng (RC-CN DI FA R&D SW) > *发送时间：* 2020年2月4日星期二 10:11 > *收件人：* Schild, Henning (CT RDA IOT SES-DE); Kiszka, Jan (CT RDA IOT > SES-DE) > *抄送：* isar-users@googlegroups.com; Vijai Kumar K > *主题：* Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from > isar-apt > > after building, grep "command not found" or "repo_contains_package" in > the log.do_* file. > > I met this problem on an early version of v4 round patch, not sure if > still exists on the latest version. will run a test build against the > latest patch to check. > > Sorry for mailing via phone, due to coronavirus, not convenient for me > to use laptop emails. > > With best regards > ------------------------------------------------------------------------ > *发件人：* Schild, Henning (CT RDA IOT SES-DE) > *发送时间：* 2020年2月4日星期二 02:59 > *收件人：* Kiszka, Jan (CT RDA IOT SES-DE) > *抄送：* isar-users@googlegroups.com; Su, Bao Cheng (RC-CN DI FA R&D > SW); Vijai Kumar K > *主题：* Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from > isar-apt > > Am Mon, 3 Feb 2020 19:27:01 +0100 > schrieb "[ext] Henning Schild" : > >> Am Mon, 3 Feb 2020 18:20:11 +0100 >> schrieb Jan Kiszka : >> >> > On 31.01.20 15:29, [ext] Henning Schild wrote: >> > > From: Henning Schild >> > > >> > > Packages from isar-apt are not downloaded from the outside and >> > > should not be cached. >> > > >> > > Signed-off-by: Henning Schild >> > > --- >> > >   meta/classes/deb-dl-dir.bbclass | 14 ++++++++++---- >> > >   1 file changed, 10 insertions(+), 4 deletions(-) >> > > >> > > diff --git a/meta/classes/deb-dl-dir.bbclass >> > > b/meta/classes/deb-dl-dir.bbclass index ab4b1f09..f9699603 100644 >> > > --- a/meta/classes/deb-dl-dir.bbclass >> > > +++ b/meta/classes/deb-dl-dir.bbclass >> > > @@ -3,8 +3,10 @@ >> > >   # >> > >   # SPDX-License-Identifier: MIT >> > > >> > > +inherit repository >> > > + >> > >   deb_dl_dir_import() { >> > > -    export pc="${DEBDIR}/${DISTRO}" >> > > +    export pc="${DEBDIR}/${DISTRO}/" >> > >       export rootfs="${1}" >> > >       [ ! -d "${pc}" ] && return 0 >> > >       flock -s "${pc}".lock -c ' \ >> > > @@ -14,12 +16,16 @@ deb_dl_dir_import() { >> > >   } >> > > >> > >   deb_dl_dir_export() { >> > > -    export pc="${DEBDIR}/${DISTRO}" >> > > +    export pc="${DEBDIR}/${DISTRO}/" >> > >       export rootfs="${1}" >> > >       mkdir -p "${pc}" >> > >       flock "${pc}".lock -c ' \ >> > > -        sudo find "${rootfs}"/var/cache/apt/archives/ -type f >> > > -iname '*\.deb' \ >> > > -            -exec cp -f '{}' "${pc}" \; >> > > +        find "${rootfs}"/var/cache/apt/archives/ -type f -iname >> > > '*\.deb' |\ >> > > +        while read p; do >> > > +             repo_contains_package "${REPO_ISAR_DIR}"/"${DISTRO}" >> > > "${p}" && \ >> > >> > repo_contains_package may not be found inside the flock shell >> > context, as Bao Cheng noticed out. He suggests the pattern >> >> That is possible indeed. I remember that i struggled re-using the >> function since i did not want to code it twice. >> >> But to be honest i do not understand the problem with the given >> description. Bao Cheng please go into more detail. > > This is a hot code-path and if it is _very_ broken that would be very > visible. > > Just tried a build and am looking at the download cache and isar-apt > > find tmp/deploy/isar-apt/ -iname *hello* > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-dbgsym_0.1_amd64.deb > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-dev_0.1_amd64.deb > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello_0.1_amd64.deb > .... > tmp/deploy/isar-apt/apt/debian-buster/pool/main/h/hello-isar/hello-isar_0.3_amd64.deb > > Ok so we have the hello stuff in isar-apt, the own package as well as > the rebuild upstream (also own). > > And nothing in the download cache on hello -> expected for the image > find downloads/deb/ -iname *hello* > > > Meaning the filtering worked in my case. > > And the caching of all required outside stuff works as well, since the > offline rebuild works. > > Caching and filtering are the two main aspects of that code. > > Please make sure to give feedback for sure, i do not want such a > unclear description slowing down the merge even more. > > Henning > >> Maybe the filtering indeed does not work, i will try that on a simple >> test now. And it might be a good idea to make sure that code gets a >> "set -e". >> >> Henning >> >> > (flock 8 >> >   ... >> > ) 8>${LOCKFILE} >> > >> > Bao Check, maybe you can describe how you noticed. >> > >> > Jan >> > >> > > +                 continue >> > > +             sudo cp -f "${p}" "${pc}" >> > > +        done >> > >           sudo chown -R $(id -u):$(id -g) "${pc}" >> > >       ' >> > >   } >> > > >> > >> > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux