Re: [PATCH v4 2/2] meta: cache deb srcs as part of postprocessing

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 3 Apr 2020 18:35:51 +0530
schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:

> Collect the deb sources of the corresponding deb binaries cached
> in DEBDIR as part of postprocess for those to be later included
> into the final base-apt by do_cache.
> 
> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> ---
>  meta/classes/image.bbclass  |  2 +-
>  meta/classes/rootfs.bbclass | 46
> +++++++++++++++++++++++++++++++++++++ 2 files changed, 47
> insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 9fa58f8..1c7a527 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -60,7 +60,7 @@ image_do_mounts() {
>  }
>  
>  ROOTFSDIR = "${IMAGE_ROOTFS}"
> -ROOTFS_FEATURES += "copy-package-cache clean-package-cache
> generate-manifest" +ROOTFS_FEATURES += "copy-package-cache
> clean-package-cache generate-manifest cache-deb-src" ROOTFS_PACKAGES
> += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" ROOTFS_MANIFEST_DEPLOY_DIR
> ?= "${DEPLOY_DIR_IMAGE}" 
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 8bb003d..7bfdfc9 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -201,6 +201,52 @@ rootfs_generate_manifest () {
>          ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
>  }
>  
> +ROOTFS_POSTPROCESS_COMMAND +=
> "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src',
> 'cache_deb_src', '', d)}" +cache_deb_src() {
> +    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> +        return 0
> +    fi

Should the source packages not all end up in the cache, so they can and
probably should be fetched from there.

> +    mkdir -p "${DEBSRCDIR}"/"${DISTRO}"
> +
> +    sudo -s <<'EOSUDO'
> +    cp -L /etc/resolv.conf '${ROOTFSDIR}/etc'
> +    mkdir -p '${ROOTFSDIR}/deb-src'
> +    mountpoint -q '${ROOTFSDIR}/deb-src' || \
> +    mount --bind '${DEBSRCDIR}' '${ROOTFSDIR}/deb-src'
> +EOSUDO
> +
> +    sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update
> +
> +    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read
> package; do
> +        local src="$( dpkg-deb --show --showformat '${Source}'
> "${package}" )"
> +        # If the binary package version and source package version
> are different, then the
> +        # source package version will be present inside "()" of the
> Source field.
> +        local version="$( echo "$src" | cut -sd "(" -f2 | cut -sd
> ")" -f1 )"
> +        if [ -z ${version} ]; then
> +            version="$( dpkg-deb --show --showformat '${Version}'
> "${package}" )"
> +        fi
> +        # Now strip any version information that might be available.
> +        src="$( echo "$src" | cut -d' ' -f1 )"
> +        # If there is no source field, then the source package has
> the same name as the
> +        # binary package.
> +        if [ -z "${src}" ];then
> +            src="$( dpkg-deb --show --showformat '${Package}'
> "${package}" )"
> +        fi
> +
> +        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${ROOTFSDIR}
> \
> +            sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd
> "/deb-src/${1}/${2}" && \
> +                apt-get -y --download-only --only-source source
> "$2"="$3"' \
> +                download-src "${DISTRO}" "${src}" "${version}"
> +    done

Looks like we are going online without proxy configuration here. It
also needs a BB_NO_NETWORK guard.

And i would suggest to generate the list of things you want to fetch,
factor out the fetcher from dpkg-base and reuse is instead of copying
it.

And i would personally like a new series of patches to be sent without
"in-reply-to". Maybe its my client but i find these deeply nested
threads very hard to follow.

Henning

> +    sudo -s <<'EOSUDO'
> +    mountpoint -q '${ROOTFSDIR}/deb-src' && \
> +    umount -l ${ROOTFSDIR}/deb-src
> +    rm -rf '${ROOTFSDIR}/etc/resolv.conf'
> +EOSUDO
> +}
> +
>  do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
>  python do_rootfs_postprocess() {
>      # Take care that its correctly mounted: