Re: [PATCH v4 2/2] meta: cache deb srcs as part of postprocessing

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

On Tue, Apr 7, 2020 at 2:08 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 07.04.20 09:59, vijai kumar wrote:
> > On Tue, Apr 7, 2020 at 12:34 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >>
> >> On 07.04.20 08:58, vijai kumar wrote:
> >>> On Tue, Apr 7, 2020 at 12:14 PM Jan Kiszka <jan.kiszka@siemens.com> wrote:
> >>>>
> >>>> On 03.04.20 15:05, Vijai Kumar K wrote:
> >>>>> Collect the deb sources of the corresponding deb binaries cached
> >>>>> in DEBDIR as part of postprocess for those to be later included
> >>>>> into the final base-apt by do_cache.
> >>>>>
> >>>>> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> >>>>> ---
> >>>>>     meta/classes/image.bbclass  |  2 +-
> >>>>>     meta/classes/rootfs.bbclass | 46 +++++++++++++++++++++++++++++++++++++
> >>>>>     2 files changed, 47 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> >>>>> index 9fa58f8..1c7a527 100644
> >>>>> --- a/meta/classes/image.bbclass
> >>>>> +++ b/meta/classes/image.bbclass
> >>>>> @@ -60,7 +60,7 @@ image_do_mounts() {
> >>>>>     }
> >>>>>
> >>>>>     ROOTFSDIR = "${IMAGE_ROOTFS}"
> >>>>> -ROOTFS_FEATURES += "copy-package-cache clean-package-cache generate-manifest"
> >>>>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache generate-manifest cache-deb-src"
> >>>>>     ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
> >>>>>     ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
> >>>>>
> >>>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> >>>>> index 8bb003d..7bfdfc9 100644
> >>>>> --- a/meta/classes/rootfs.bbclass
> >>>>> +++ b/meta/classes/rootfs.bbclass
> >>>>> @@ -201,6 +201,52 @@ rootfs_generate_manifest () {
> >>>>>             ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> >>>>>     }
> >>>>>
> >>>>> +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
> >>>>> +cache_deb_src() {
> >>>>> +    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>> +        return 0
> >>>>> +    fi
> >>>>> +
> >>>>> +    mkdir -p "${DEBSRCDIR}"/"${DISTRO}"
> >>>>> +
> >>>>> +    sudo -s <<'EOSUDO'
> >>>>> +    cp -L /etc/resolv.conf '${ROOTFSDIR}/etc'
> >>>>> +    mkdir -p '${ROOTFSDIR}/deb-src'
> >>>>> +    mountpoint -q '${ROOTFSDIR}/deb-src' || \
> >>>>> +    mount --bind '${DEBSRCDIR}' '${ROOTFSDIR}/deb-src'
> >>>>> +EOSUDO
> >>>>> +
> >>>>> +    sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update
> >>>>> +
> >>>>> +    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
> >>>>> +        local src="$( dpkg-deb --show --showformat '${Source}' "${package}" )"
> >>>>> +        # If the binary package version and source package version are different, then the
> >>>>> +        # source package version will be present inside "()" of the Source field.
> >>>>> +        local version="$( echo "$src" | cut -sd "(" -f2 | cut -sd ")" -f1 )"
> >>>>> +        if [ -z ${version} ]; then
> >>>>> +            version="$( dpkg-deb --show --showformat '${Version}' "${package}" )"
> >>>>> +        fi
> >>>>> +        # Now strip any version information that might be available.
> >>>>> +        src="$( echo "$src" | cut -d' ' -f1 )"
> >>>>> +        # If there is no source field, then the source package has the same name as the
> >>>>> +        # binary package.
> >>>>> +        if [ -z "${src}" ];then
> >>>>> +            src="$( dpkg-deb --show --showformat '${Package}' "${package}" )"
> >>>>> +        fi
> >>>>> +
> >>>>> +        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${ROOTFSDIR} \
> >>>>> +            sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && \
> >>>>> +                apt-get -y --download-only --only-source source "$2"="$3"' \
> >>>>> +                download-src "${DISTRO}" "${src}" "${version}"
> >>>>
> >>>> How did you construct this apt-get command? I'm trying to match it
> >>>> against the man page of apt-get but there is no reference to "download-src".
> >>>
> >>> download-src is just the script name.
> >>> The shell script call is in below format to pass the args.
> >>>
> >>> sh -c '<commands>' <script name> <args>
> >>>
> >>
> >> Oh, I see. What's the benefit of this obfuscation? Or is there even any
> >> technical need?
> >
> > Yes. To run multiple commands from a single chroot call. We can either
> > invoke sh -c or have a script that contains the commands that could be
> > called.
> > We are using sh -c and hence the need for arguments to pass in the variables.
> > It could very well be multiple chroot calls or a script but this is
> > more clean I guess.
>
> It's not technically needed, giving the commands a name is optional. You
> could also unfold the arguments (which would make them more readable).

Ah. Just tried that. Looks like unfolding is just possible. The man
page says read from string, so the
variables part of the string are expanded. Its not like a subshell, if
I understand correctly now.
In that case unfolding does make sense.

Thanks,
Vijai Kumar K

>
> > Also, to note, this style was inherited from the base-apt series by Henning.
> >
>
> Well, if we keep that pattern, then please indent in a more
> reader-friendly way:
>
> sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && \
>         apt-get -y --download-only --only-source source "$2"="$3"' \
>     download-src "${DISTRO}" "${src}" "${version}"
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux