From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6790334981638979584 Date: Fri, 14 Feb 2020 00:45:20 -0800 (PST) From: vijai kumar To: isar-users Message-Id: <74f8d2e2-3fdd-4399-99b6-34dd8b78af65@googlegroups.com> In-Reply-To: <0898a4b3-c512-48f9-825e-ee112b26f7d7@googlegroups.com> References: <20200213100820.23444-2-Vijaikumar_Kanagarajan@mentor.com> <20200214054801.11870-1-Vijaikumar_Kanagarajan@mentor.com> <20200214054801.11870-2-Vijaikumar_Kanagarajan@mentor.com> <6bb30f45-fc18-f86e-487f-281bb37d589d@siemens.com> <0898a4b3-c512-48f9-825e-ee112b26f7d7@googlegroups.com> Subject: Re: [PATCH v3 2/2] meta: cache deb srcs as part of postprocessing MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_2621_762587159.1581669920361" X-Google-Token: EKC8mfIF33vRh9zygvQ0 X-Google-IP: 139.181.36.34 X-TUID: 08mQtrS7GD4Q ------=_Part_2621_762587159.1581669920361 Content-Type: multipart/alternative; boundary="----=_Part_2622_277230627.1581669920362" ------=_Part_2622_277230627.1581669920362 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On Friday, February 14, 2020 at 2:11:45 PM UTC+5:30, vijai kumar wrote: > > > > On Friday, February 14, 2020 at 1:49:31 PM UTC+5:30, Jan Kiszka wrote: >> >> On 14.02.20 06:48, Vijai Kumar K wrote: >> > Collect the deb sources of the corresponding deb binaries cached >> > in DEBDIR as part of postprocess for those to be later included >> > into the final base-apt by do_cache. >> > >> >> So, inclusion into base-apt will come in a later patch? IOW: It's not >> yet clear to me if this patch alone is already useful. >> > > Hi Jan, > > The new base-apt rework from Henning would download all the deb and > deb-srcs in downloads/deb > download/deb-src directories respectively. The actual repo(using reprepro) > is not created until base-apt:do_cache is called. For which > you have to trigger a offline build and set ISAR_USE_CACHED_BASE_REPO. > > This patch is an extension to the downloads/deb-src downloading, where in > all the source files of all downloaded deb files will cached, > but for it to be converted to repo, base-apt.bb:do_cache needs to be > called. > So, basically this is a middle piece of Henning's workflow. > Thanks, > Vijai Kumar K > > >> Thanks, >> Jan >> >> > Signed-off-by: Vijai Kumar K >> > --- >> > meta/classes/image.bbclass | 2 +- >> > meta/classes/rootfs.bbclass | 29 +++++++++++++++++++++++++++++ >> > 2 files changed, 30 insertions(+), 1 deletion(-) >> > >> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> > index c5fddba..77306ce 100644 >> > --- a/meta/classes/image.bbclass >> > +++ b/meta/classes/image.bbclass >> > @@ -60,7 +60,7 @@ image_do_mounts() { >> > } >> > >> > ROOTFSDIR = "${IMAGE_ROOTFS}" >> > -ROOTFS_FEATURES += "copy-package-cache clean-package-cache >> generate-manifest" >> > +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >> generate-manifest cache-deb-src" >> > ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >> > ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" >> > >> > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >> > index c3af7c1..971a299 100644 >> > --- a/meta/classes/rootfs.bbclass >> > +++ b/meta/classes/rootfs.bbclass >> > @@ -201,6 +201,35 @@ rootfs_generate_manifest () { >> > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest >> > } >> > >> > +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', >> 'cache-deb-src', 'cache_deb_src', '', d)}" >> > +cache_deb_src() { >> > + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >> > + return 0 >> > + fi >> > + sudo -s <<'EOSUDO' >> > + sudo cp -L /etc/resolv.conf '${ROOTFSDIR}/etc' >> > + mkdir -p "${DEBSRCDIR}"/"${DISTRO}" >> > + mkdir -p '${ROOTFSDIR}/deb-src' >> > + mountpoint -q '${ROOTFSDIR}/deb-src' || \ >> > + mount --bind '${DEBSRCDIR}' '${ROOTFSDIR}/deb-src' >> > +EOSUDO >> > + sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update >> > + find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; >> do >> > + local pkg="$( dpkg-deb --show --showformat '${Package}' >> "${package}" )" >> > + local dirname="$( dpkg-deb --show --showformat '${Source}' >> "${package}" )" >> > + if [ -z "${dirname}" ];then >> > + dirname="$pkg" >> > + fi >> > + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${ROOTFSDIR} \ >> > + sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd >> "/deb-src/${1}/${2}" && apt-get -y source --download-only "$3"' >> download-src "${DISTRO}" "${dirname}" "${pkg}" >> > + done >> > + sudo -s <<'EOSUDO' >> > + mountpoint -q '${ROOTFSDIR}/deb-src' && \ >> > + umount -l ${ROOTFSDIR}/deb-src >> > + sudo rm -rf '${ROOTFSDIR}/etc/resolv.conf' >> > +EOSUDO >> > +} >> > + >> > do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >> > python do_rootfs_postprocess() { >> > # Take care that its correctly mounted: >> > >> >> -- >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE >> Corporate Competence Center Embedded Linux >> > ------=_Part_2622_277230627.1581669920362 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable


On Friday, February 14, 2020 at 2:11:45 PM UTC+5:3= 0, vijai kumar wrote:


On Friday, February 14, 2020 at 1:49:31 PM UTC+5:30, Jan K= iszka wrote:
On 14.02.20 06:48, Vija= i Kumar K wrote:
> Collect the deb sources of the corresponding deb binaries cached
> in DEBDIR as part of postprocess for those to be later included
> into the final base-apt by do_cache.
>=20

So, inclusion into base-apt will come in a later patch? IOW: It's n= ot=20
yet clear to me if this patch alone is already useful.
=

Hi Jan,

The new base-apt rewor= k from Henning would download all the deb and deb-srcs in downloads/deb
download/deb-src directories respectively. The actual repo(using rep= repro) is not created until base-apt:do_cache is called. For which
you have to trigger a offline build and set ISAR_USE_CACHED_BASE_REPO.

This patch is an extension to the downloads/deb= -src downloading, where in all the source files of all downloaded deb files= will cached,
but for it to be converted to repo, base-apt.bb:do_= cache needs to be called.

So, b= asically this is a middle piece of Henning's workflow.


=
Thanks,
Vijai Kumar K


Thanks,
Jan

> Signed-off-by: Vijai Kumar K <Vijaikumar_..= .@mentor.com>
> ---
> =C2=A0 meta/classes/image.bbclass =C2=A0| =C2=A02 +-
> =C2=A0 meta/classes/rootfs.bbclass | 29 ++++++++++++++++++++++++++= +++
> =C2=A0 2 files changed, 30 insertions(+), 1 deletion(-)
>=20
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbcla= ss
> index c5fddba..77306ce 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -60,7 +60,7 @@ image_do_mounts() {
> =C2=A0 }
> =C2=A0=20
> =C2=A0 ROOTFSDIR =3D "${IMAGE_ROOTFS}"
> -ROOTFS_FEATURES +=3D "copy-package-cache clean-package-cache= generate-manifest"
> +ROOTFS_FEATURES +=3D "copy-package-cache clean-package-cache= generate-manifest cache-deb-src"
> =C2=A0 ROOTFS_PACKAGES +=3D "${IMAGE_PREINSTALL} ${IMAGE_INST= ALL}"
> =C2=A0 ROOTFS_MANIFEST_DEPLOY_DIR ?=3D "${DEPLOY_DIR_IMAGE}&q= uot;
> =C2=A0=20
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbc= lass
> index c3af7c1..971a299 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -201,6 +201,35 @@ rootfs_generate_manifest () {
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ${ROOTFS_MANIFEST_DEPLOY_DIR}/<= wbr>"${PF}".manifest
> =C2=A0 }
> =C2=A0=20
> +ROOTFS_POSTPROCESS_COMMAND +=3D "${@bb.utils.contains('R= OOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src',= '', d)}"
> +cache_deb_src() {
> + =C2=A0 =C2=A0if [ "${ISAR_USE_CACHED_BASE_REPO}" =3D &= quot;1" ]; then
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0
> + =C2=A0 =C2=A0fi
> + =C2=A0 =C2=A0sudo -s <<'EOSUDO'
> + =C2=A0 =C2=A0sudo cp -L /etc/resolv.conf '${ROOTFSDIR}/etc&#= 39;
> + =C2=A0 =C2=A0mkdir -p "${DEBSRCDIR}"/"${DISTRO}&q= uot;
> + =C2=A0 =C2=A0mkdir -p '${ROOTFSDIR}/deb-src'
> + =C2=A0 =C2=A0mountpoint -q '${ROOTFSDIR}/deb-src' || \
> + =C2=A0 =C2=A0mount --bind '${DEBSRCDIR}' '${ROOTFSDI= R}/deb-src'
> +EOSUDO
> + =C2=A0 =C2=A0sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update
> + =C2=A0 =C2=A0find "${DEBDIR}"/"${DISTRO}" -n= ame '*\.deb' | while read package; do
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0local pkg=3D"$( dpkg-deb --show = --showformat '${Package}' "${package}" )"
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0local dirname=3D"$( dpkg-deb --s= how --showformat '${Source}' "${package}" )"
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0if [ -z "${dirname}" ];then
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dirname=3D"$pkg&qu= ot;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0fi
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0sudo -E chroot --userspec=3D$( id -u = ):$( id -g ) ${ROOTFSDIR} \
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sh -c 'mkdir -p &qu= ot;/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" &a= mp;& apt-get -y source --download-only "$3"' download-src= "${DISTRO}" "${dirname}" "${pkg}"
> + =C2=A0 =C2=A0done
> + =C2=A0 =C2=A0sudo -s <<'EOSUDO'
> + =C2=A0 =C2=A0mountpoint -q '${ROOTFSDIR}/deb-src' &&= amp; \
> + =C2=A0 =C2=A0umount -l ${ROOTFSDIR}/deb-src
> + =C2=A0 =C2=A0sudo rm -rf '${ROOTFSDIR}/etc/resolv.conf'
> +EOSUDO
> +}
> +
> =C2=A0 do_rootfs_postprocess[vardeps] =3D "${ROOTFS_POSTPROCE= SS_COMMAND}"
> =C2=A0 python do_rootfs_postprocess() {
> =C2=A0 =C2=A0 =C2=A0 # Take care that its correctly mounted:
>=20

--=20
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
------=_Part_2622_277230627.1581669920362-- ------=_Part_2621_762587159.1581669920361--