Re: [PATCH v4 2/2] meta: cache deb srcs as part of postprocessing

[vijai kumar <vijaikumar.kanagarajan@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 6254 bytes --]



On Tuesday, April 7, 2020 at 2:38:23 PM UTC+5:30, vijai kumar wrote:
>
> On Tue, Apr 7, 2020 at 2:08 PM Jan Kiszka <jan.kiszka@siemens.com> wrote: 
> > 
> > On 07.04.20 09:59, vijai kumar wrote: 
> > > On Tue, Apr 7, 2020 at 12:34 PM Jan Kiszka <jan.kiszka@siemens.com> 
> wrote: 
> > >> 
> > >> On 07.04.20 08:58, vijai kumar wrote: 
> > >>> On Tue, Apr 7, 2020 at 12:14 PM Jan Kiszka <jan.kiszka@siemens.com> 
> wrote: 
> > >>>> 
> > >>>> On 03.04.20 15:05, Vijai Kumar K wrote: 
> > >>>>> Collect the deb sources of the corresponding deb binaries cached 
> > >>>>> in DEBDIR as part of postprocess for those to be later included 
> > >>>>> into the final base-apt by do_cache. 
> > >>>>> 
> > >>>>> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> 
> > >>>>> --- 
> > >>>>>     meta/classes/image.bbclass  |  2 +- 
> > >>>>>     meta/classes/rootfs.bbclass | 46 
> +++++++++++++++++++++++++++++++++++++ 
> > >>>>>     2 files changed, 47 insertions(+), 1 deletion(-) 
> > >>>>> 
> > >>>>> diff --git a/meta/classes/image.bbclass 
> b/meta/classes/image.bbclass 
> > >>>>> index 9fa58f8..1c7a527 100644 
> > >>>>> --- a/meta/classes/image.bbclass 
> > >>>>> +++ b/meta/classes/image.bbclass 
> > >>>>> @@ -60,7 +60,7 @@ image_do_mounts() { 
> > >>>>>     } 
> > >>>>> 
> > >>>>>     ROOTFSDIR = "${IMAGE_ROOTFS}" 
> > >>>>> -ROOTFS_FEATURES += "copy-package-cache clean-package-cache 
> generate-manifest" 
> > >>>>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache 
> generate-manifest cache-deb-src" 
> > >>>>>     ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" 
> > >>>>>     ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" 
> > >>>>> 
> > >>>>> diff --git a/meta/classes/rootfs.bbclass 
> b/meta/classes/rootfs.bbclass 
> > >>>>> index 8bb003d..7bfdfc9 100644 
> > >>>>> --- a/meta/classes/rootfs.bbclass 
> > >>>>> +++ b/meta/classes/rootfs.bbclass 
> > >>>>> @@ -201,6 +201,52 @@ rootfs_generate_manifest () { 
> > >>>>>             ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest 
> > >>>>>     } 
> > >>>>> 
> > >>>>> +ROOTFS_POSTPROCESS_COMMAND += 
> "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', 
> '', d)}" 
> > >>>>> +cache_deb_src() { 
> > >>>>> +    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then 
> > >>>>> +        return 0 
> > >>>>> +    fi 
> > >>>>> + 
> > >>>>> +    mkdir -p "${DEBSRCDIR}"/"${DISTRO}" 
> > >>>>> + 
> > >>>>> +    sudo -s <<'EOSUDO' 
> > >>>>> +    cp -L /etc/resolv.conf '${ROOTFSDIR}/etc' 
> > >>>>> +    mkdir -p '${ROOTFSDIR}/deb-src' 
> > >>>>> +    mountpoint -q '${ROOTFSDIR}/deb-src' || \ 
> > >>>>> +    mount --bind '${DEBSRCDIR}' '${ROOTFSDIR}/deb-src' 
> > >>>>> +EOSUDO 
> > >>>>> + 
> > >>>>> +    sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update 
> > >>>>> + 
> > >>>>> +    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read 
> package; do 
> > >>>>> +        local src="$( dpkg-deb --show --showformat '${Source}' 
> "${package}" )" 
> > >>>>> +        # If the binary package version and source package 
> version are different, then the 
> > >>>>> +        # source package version will be present inside "()" of 
> the Source field. 
> > >>>>> +        local version="$( echo "$src" | cut -sd "(" -f2 | cut -sd 
> ")" -f1 )" 
> > >>>>> +        if [ -z ${version} ]; then 
> > >>>>> +            version="$( dpkg-deb --show --showformat '${Version}' 
> "${package}" )" 
> > >>>>> +        fi 
> > >>>>> +        # Now strip any version information that might be 
> available. 
> > >>>>> +        src="$( echo "$src" | cut -d' ' -f1 )" 
> > >>>>> +        # If there is no source field, then the source package 
> has the same name as the 
> > >>>>> +        # binary package. 
> > >>>>> +        if [ -z "${src}" ];then 
> > >>>>> +            src="$( dpkg-deb --show --showformat '${Package}' 
> "${package}" )" 
> > >>>>> +        fi 
> > >>>>> + 
> > >>>>> +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> ${ROOTFSDIR} \ 
> > >>>>> +            sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd 
> "/deb-src/${1}/${2}" && \ 
> > >>>>> +                apt-get -y --download-only --only-source source 
> "$2"="$3"' \ 
> > >>>>> +                download-src "${DISTRO}" "${src}" "${version}" 
> > >>>> 
> > >>>> How did you construct this apt-get command? I'm trying to match it 
> > >>>> against the man page of apt-get but there is no reference to 
> "download-src". 
> > >>> 
> > >>> download-src is just the script name. 
> > >>> The shell script call is in below format to pass the args. 
> > >>> 
> > >>> sh -c '<commands>' <script name> <args> 
> > >>> 
> > >> 
> > >> Oh, I see. What's the benefit of this obfuscation? Or is there even 
> any 
> > >> technical need? 
> > > 
> > > Yes. To run multiple commands from a single chroot call. We can either 
> > > invoke sh -c or have a script that contains the commands that could be 
> > > called. 
> > > We are using sh -c and hence the need for arguments to pass in the 
> variables. 
> > > It could very well be multiple chroot calls or a script but this is 
> > > more clean I guess. 
> > 
> > It's not technically needed, giving the commands a name is optional. You 
> > could also unfold the arguments (which would make them more readable). 
>
> Ah. Just tried that. Looks like unfolding is just possible. The man 
> page says read from string, so the 
> variables part of the string are expanded. Its not like a subshell, if 
> I understand correctly now. 
> In that case unfolding does make sense. 
>
> Thanks, 
> Vijai Kumar K 
>
> > 
> > > Also, to note, this style was inherited from the base-apt series by 
> Henning. 
> > > 
> > 
> > Well, if we keep that pattern, then please indent in a more 
> > reader-friendly way: 
> > 
> > sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && \ 
> >         apt-get -y --download-only --only-source source "$2"="$3"' \ 
> >     download-src "${DISTRO}" "${src}" "${version}" 
> > 
> > Jan 
>

There is atleast one issue in this series. Need to take care when 
HOST_DISTRO!=DISTRO. Issue first identified in rpi-stretch cross 
compilation.

Thanks,
Vijai Kumar K

> 
> > -- 
> > Siemens AG, Corporate Technology, CT RDA IOT SES-DE 
> > Corporate Competence Center Embedded Linux 
>

[-- Attachment #1.2: Type: text/html, Size: 10079 bytes --]