RE: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from isar-apt

["Su, Bao Cheng" <baocheng.su@siemens.com>]

Glad of hearing you have spotted the issue.

It seemed to be ok with this "command not found" log and I remember it's ok to rebuild offline even with the command not found, but I am not very sure now because I only have dim memories for it.

I only remember I noticed it while investigating another issue of ours, by accidently looking into the build log.

> -----Original Message-----
> From: Schild, Henning (CT RDA IOT SES-DE) <henning.schild@siemens.com>
> Sent: Wednesday, February 05, 2020 8:11 PM
> To: Su, Bao Cheng (RC-CN DI FA R&D SW) <baocheng.su@siemens.com>
> Cc: Kiszka, Jan (CT RDA IOT SES-DE) <jan.kiszka@siemens.com>; isar-users@googlegroups.com; Vijai Kumar K
> <vijaikumar.kanagarajan@gmail.com>
> Subject: Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from isar-apt
> 
> On Tue, 4 Feb 2020 05:10:05 +0000
> "Su, Bao Cheng (RC-CN DI FA R&D SW)" <baocheng.su@siemens.com> wrote:
> 
> > Still exists.
> >
> > Under `build/tmp/work/`:
> >
> > $ grep "command not found" . -r --include="log.do_*"
> > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash:
> > line 5: repo_contains_package: command not found
> > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash:
> > line 5: repo_contains_package: command not found
> > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash:
> > line 5: repo_contains_package: command not found
> > ./isar-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.329:/bin/bash:
> > line 5: repo_contains_package: command not found
> >
> > $ grep "command not found" . -r --include="log.do_*" | wc -l
> > 1596
> >
> > This is only a half-way build, for a full build, there are more
> > occurrences.
> 
> Ok i can reproduce the issue and am investigating, will probably take
> the other flock notation.
> 
> Funny thing is that despite our efforts to make the build env
> reproducible we are using "dash" in a plain Isar build and "bash" in a
> kas based build.
> 
> That is what dash has to say ..
> /bin/sh: 9: repo_contains_package: not found
> 
> So the grep did not find it.
> 
> What i still do not understand is how the download cache seems to be or
> is exactly in the state i would expect. You probably did not patch
> because of the "command not found" but because of a result of some sort.
> 
> Have packages been missing in the dl cache? What kind of packages? Did
> this break your offline rebuild, or did it show up in another context
> (maybe clearing).
> 
> Thanks,
> Henning
> 
> > With best regards
> > ________________________________
> > 发件人： Su, Bao Cheng (RC-CN DI FA R&D SW)
> > 发送时间： 2020年2月4日星期二 10:11
> > 收件人： Schild, Henning (CT RDA IOT SES-DE); Kiszka, Jan (CT RDA IOT
> > SES-DE) 抄送： isar-users@googlegroups.com; Vijai Kumar K
> > 主题： Re: [PATCHv4 22/26] meta: deb-dl-dir: do not cache debs from
> > isar-apt
> >
> > after building, grep "command not found" or "repo_contains_package"
> > in the log.do_* file.
> >
> > I met this problem on an early version of v4 round patch, not sure if
> > still exists on the latest version. will run a test build against the
> > latest patch to check.
> >
> > Sorry for mailing via phone, due to coronavirus, not convenient for
> > me to use laptop emails.
> >
> > With best regards
> > ________________________________
> > 发件人： Schild, Henning (CT RDA IOT SES-DE)
> > 发送时间： 2020年2月4日星期二 02:59
> > 收件人： Kiszka, Jan (CT RDA IOT SES-DE)
> > 抄送： isar-users@googlegroups.com; Su, Bao Cheng (RC-CN DI FA R&D
> > SW); Vijai Kumar K 主题： Re: [PATCHv4 22/26] meta: deb-dl-dir: do
> > not cache debs from isar-apt
> >
> > Am Mon, 3 Feb 2020 19:27:01 +0100
> > schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> >
> > > Am Mon, 3 Feb 2020 18:20:11 +0100
> > > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> > >
> > > > On 31.01.20 15:29, [ext] Henning Schild wrote:
> > > > > From: Henning Schild <henning.schild@siemens.com>
> > > > >
> > > > > Packages from isar-apt are not downloaded from the outside and
> > > > > should not be cached.
> > > > >
> > > > > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > > > > ---
> > > > >   meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
> > > > >   1 file changed, 10 insertions(+), 4 deletions(-)
> > > > >
> > > > > diff --git a/meta/classes/deb-dl-dir.bbclass
> > > > > b/meta/classes/deb-dl-dir.bbclass index ab4b1f09..f9699603
> > > > > 100644 --- a/meta/classes/deb-dl-dir.bbclass
> > > > > +++ b/meta/classes/deb-dl-dir.bbclass
> > > > > @@ -3,8 +3,10 @@
> > > > >   #
> > > > >   # SPDX-License-Identifier: MIT
> > > > >
> > > > > +inherit repository
> > > > > +
> > > > >   deb_dl_dir_import() {
> > > > > -    export pc="${DEBDIR}/${DISTRO}"
> > > > > +    export pc="${DEBDIR}/${DISTRO}/"
> > > > >       export rootfs="${1}"
> > > > >       [ ! -d "${pc}" ] && return 0
> > > > >       flock -s "${pc}".lock -c ' \
> > > > > @@ -14,12 +16,16 @@ deb_dl_dir_import() {
> > > > >   }
> > > > >
> > > > >   deb_dl_dir_export() {
> > > > > -    export pc="${DEBDIR}/${DISTRO}"
> > > > > +    export pc="${DEBDIR}/${DISTRO}/"
> > > > >       export rootfs="${1}"
> > > > >       mkdir -p "${pc}"
> > > > >       flock "${pc}".lock -c ' \
> > > > > -        sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > > > > -iname '*\.deb' \
> > > > > -            -exec cp -f '{}' "${pc}" \;
> > > > > +        find "${rootfs}"/var/cache/apt/archives/ -type f -iname
> > > > > '*\.deb' |\
> > > > > +        while read p; do
> > > > > +             repo_contains_package
> > > > > "${REPO_ISAR_DIR}"/"${DISTRO}" "${p}" && \
> > > >
> > > > repo_contains_package may not be found inside the flock shell
> > > > context, as Bao Cheng noticed out. He suggests the pattern
> > >
> > > That is possible indeed. I remember that i struggled re-using the
> > > function since i did not want to code it twice.
> > >
> > > But to be honest i do not understand the problem with the given
> > > description. Bao Cheng please go into more detail.
> >
> > This is a hot code-path and if it is _very_ broken that would be very
> > visible.
> >
> > Just tried a build and am looking at the download cache and isar-apt
> >
> > find tmp/deploy/isar-apt/ -iname *hello*
> > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello
> > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-dbgsym_0.1_amd64.deb
> > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello-dev_0.1_amd64.deb
> > tmp/deploy/isar-apt/apt/debian-buster/pool/main/libh/libhello/libhello_0.1_amd64.deb
> > ....
> > tmp/deploy/isar-apt/apt/debian-buster/pool/main/h/hello-isar/hello-isar_0.3_amd64.deb
> >
> > Ok so we have the hello stuff in isar-apt, the own package as well as
> > the rebuild upstream (also own).
> >
> > And nothing in the download cache on hello -> expected for the image
> > find downloads/deb/ -iname *hello*
> > <nothing>
> >
> > Meaning the filtering worked in my case.
> >
> > And the caching of all required outside stuff works as well, since the
> > offline rebuild works.
> >
> > Caching and filtering are the two main aspects of that code.
> >
> > Please make sure to give feedback for sure, i do not want such a
> > unclear description slowing down the merge even more.
> >
> > Henning
> >
> > > Maybe the filtering indeed does not work, i will try that on a
> > > simple test now. And it might be a good idea to make sure that code
> > > gets a "set -e".
> > >
> > > Henning
> > >
> > > > (flock 8
> > > >   ...
> > > > ) 8>${LOCKFILE}
> > > >
> > > > Bao Check, maybe you can describe how you noticed.
> > > >
> > > > Jan
> > > >
> > > > > +                 continue
> > > > > +             sudo cp -f "${p}" "${pc}"
> > > > > +        done
> > > > >           sudo chown -R $(id -u):$(id -g) "${pc}"
> > > > >       '
> > > > >   }
> > > > >
> > > >
> > >
> >