From: Henning Schild <henning.schild@siemens.com>
To: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>
Cc: <isar-users@googlegroups.com>, <ibr@radix50.net>,
<jan.kiszka@siemens.com>,
Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
Subject: Re: [PATCH v5 02/13] deb-dl-dir: Cache host distro debs separately
Date: Wed, 22 Apr 2020 08:52:26 +0200 [thread overview]
Message-ID: <20200422085226.36b3e606@md1za8fc.ad001.siemens.net> (raw)
In-Reply-To: <20200417093040.15130-3-Vijaikumar_Kanagarajan@mentor.com>
Hi Vijai,
you found one of the "known" issues of that implementation
https://groups.google.com/d/msg/isar-users/8djs0Thl0wI/n5Bo2hPeCgAJ
There i talk about "mixing" packages from many sources into one
base-apt under one "suite" like "buster".
In fact such a mix in the download cache is what you found here. But
maybe for the cache we can store everything plain in one folder. Given
that an "apt-get source" without a version would "unpack" only what it
found in its Sources.gz(s) ... which i would assume.
But i think what you suggest here should work as well. And there is
probably not much overlap ... packages with exactly the same versions
in multiple suites.
Henning
On Fri, 17 Apr 2020 15:00:29 +0530
Vijai Kumar K <vijaikumar.kanagarajan@gmail.com> wrote:
> In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
> we were still caching the debs from both the distros into the same
> DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be
> cached in the relevant subdirectory and avoids mixing of debs from
> two separate distros.
>
> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> ---
> meta/classes/deb-dl-dir.bbclass | 4 ++--
> meta/classes/dpkg-gbp.bbclass | 8 ++++++--
> meta/classes/dpkg.bbclass | 8 ++++++--
> meta/classes/image-tools-extension.bbclass | 4 ++--
> meta/classes/rootfs.bbclass | 4 ++--
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++--
> 6 files changed, 28 insertions(+), 12 deletions(-)
>
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index e996703..29a3d67 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -6,7 +6,7 @@
> inherit repository
>
> deb_dl_dir_import() {
> - export pc="${DEBDIR}/${DISTRO}/"
> + export pc="${DEBDIR}/${2}"
> export rootfs="${1}"
> [ ! -d "${pc}" ] && return 0
> sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
> @@ -20,7 +20,7 @@ deb_dl_dir_import() {
> }
>
> deb_dl_dir_export() {
> - export pc="${DEBDIR}/${DISTRO}/"
> + export pc="${DEBDIR}/${2}"
> export rootfs="${1}"
> mkdir -p "${pc}"
> flock "${pc}".lock -c '
> diff --git a/meta/classes/dpkg-gbp.bbclass
> b/meta/classes/dpkg-gbp.bbclass index afa1e19..ba5c3eb 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>
> do_install_builddeps_append() {
> dpkg_do_mounts
> - deb_dl_dir_import "${BUILDCHROOT_DIR}"
> + distro="${DISTRO}"
> + if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> + distro="${HOST_DISTRO}"
> + fi
> + deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends --download-only
> ${GBP_DEPENDS}
> - deb_dl_dir_export "${BUILDCHROOT_DIR}"
> + deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends ${GBP_DEPENDS}
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 585365e..53862b5 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -7,10 +7,14 @@ inherit dpkg-base
> do_install_builddeps() {
> dpkg_do_mounts
> E="${@ isar_export_proxies(d)}"
> - deb_dl_dir_import "${BUILDCHROOT_DIR}"
> + distro="${DISTRO}"
> + if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> + distro="${HOST_DISTRO}"
> + fi
> + deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> - deb_dl_dir_export "${BUILDCHROOT_DIR}"
> + deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH}
> dpkg_undo_mounts
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass index 6590ee7..0b067ff
> 100644 --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -25,7 +25,7 @@ do_install_imager_deps() {
> buildchroot_do_mounts
>
> E="${@ isar_export_proxies(d)}"
> - deb_dl_dir_import ${BUILDCHROOT_DIR}
> + deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
> sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> apt-get update \
> -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
> @@ -35,7 +35,7 @@ do_install_imager_deps() {
> --allow-unauthenticated --allow-downgrades
> --download-only install \ ${IMAGER_INSTALL}'
>
> - deb_dl_dir_export ${BUILDCHROOT_DIR}
> + deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
> sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y \ --allow-unauthenticated
> --allow-downgrades install \ diff --git a/meta/classes/rootfs.bbclass
> b/meta/classes/rootfs.bbclass index 8bb003d..cee358c 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -115,7 +115,7 @@ rootfs_install_resolvconf() {
> ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
> rootfs_import_package_cache[weight] = "5"
> rootfs_import_package_cache() {
> - deb_dl_dir_import ${ROOTFSDIR}
> + deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
> }
>
> ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
> @@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND +=
> "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}" ROOTFS_INSTALL_COMMAND +=
> "rootfs_export_package_cache" rootfs_export_package_cache[weight] =
> "5" rootfs_export_package_cache() {
> - deb_dl_dir_export ${ROOTFSDIR}
> + deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO}
> }
>
> ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if
> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" diff --git
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> da0d436..4061c86 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -247,7
> +247,11 @@ isar_bootstrap() { export IS_HOST debootstrap_args E if [
> ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then sudo rm -rf --one-file-system
> "${ROOTFSDIR}"
> - deb_dl_dir_import "${ROOTFSDIR}"
> + if [ "${IS_HOST}" ];then
> + deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
> + else
> + deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
> + fi
>
> sudo -E -s <<'EOSUDO'
> set -e
> @@ -344,7 +348,11 @@ isar_bootstrap() {
> ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
> EOSUDO
> fi
> - deb_dl_dir_export "${ROOTFSDIR}"
> + if [ "${IS_HOST}" ];then
> + deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
> + else
> + deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
> + fi
> }
>
> CLEANFUNCS = "clean_deploy"
next prev parent reply other threads:[~2020-04-22 6:52 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-17 9:30 [PATCH v5 00/13] Deb-src caching implementation Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 01/13] rootfs: Make rootfs finalize a separate task Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 02/13] deb-dl-dir: Cache host distro debs separately Vijai Kumar K
2020-04-22 6:52 ` Henning Schild [this message]
2020-04-17 9:30 ` [PATCH v5 03/13] meta: cache deb srcs as part of postprocessing Vijai Kumar K
2020-04-22 7:06 ` Henning Schild
2020-04-22 9:57 ` vijai kumar
2020-04-29 6:19 ` Henning Schild
2020-05-04 8:42 ` vijai kumar
2020-05-08 5:44 ` vijai kumar
2020-06-11 11:44 ` vijai kumar
2020-09-03 11:43 ` vijaikumar....@gmail.com
2020-09-03 18:58 ` Henning Schild
2020-09-04 5:12 ` vijaikumar....@gmail.com
2020-04-22 7:28 ` Henning Schild
2020-04-22 10:35 ` vijai kumar
2020-04-17 9:30 ` [PATCH v5 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
2020-04-22 7:15 ` Henning Schild
2020-04-22 10:11 ` vijai kumar
2020-04-29 7:05 ` Henning Schild
2020-04-17 9:30 ` [PATCH v5 05/13] deb-dl-dir: Download files only belonging to the current image Vijai Kumar K
2020-04-22 7:25 ` Henning Schild
2020-04-22 10:20 ` vijai kumar
2020-04-17 9:30 ` [PATCH v5 06/13] deb-dl-dir: Factor out the mounting part Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 07/13] deb-dl-dir: Fix skipping of removed files Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 08/13] repository: Add a sanity test to check missing sources Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 09/13] base-apt: Introduce BASE_REPO_FEATURES Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 10/13] repository: Fix failures due to missing section Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 11/13] scripts/ci_build.sh: Enable deb-src caching Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 12/13] rootfs: Fix possible overwrite of existing resolv.conf Vijai Kumar K
2020-04-17 9:30 ` [PATCH v5 13/13] rootfs: Handle failures when postprocess is rerun Vijai Kumar K
2020-04-17 14:21 ` [PATCH v5 00/13] Deb-src caching implementation vijai kumar
2020-04-22 7:41 ` Henning Schild
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200422085226.36b3e606@md1za8fc.ad001.siemens.net \
--to=henning.schild@siemens.com \
--cc=Vijaikumar_Kanagarajan@mentor.com \
--cc=ibr@radix50.net \
--cc=isar-users@googlegroups.com \
--cc=jan.kiszka@siemens.com \
--cc=vijaikumar.kanagarajan@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox