From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6816610675337461760 X-Received: by 2002:a17:906:7c11:: with SMTP id t17mr25400419ejo.73.1587538354290; Tue, 21 Apr 2020 23:52:34 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:54cd:: with SMTP id c13ls523052ejp.5.gmail; Tue, 21 Apr 2020 23:52:33 -0700 (PDT) X-Google-Smtp-Source: APiQypK9ef8gJTYXzrrpy85hDK/BT6Kywiu4JuGZI9IiXI09pXhHPt1IWeJDEdoFzHWUFKF3ZBC1 X-Received: by 2002:a17:906:310e:: with SMTP id 14mr25688813ejx.177.1587538353756; Tue, 21 Apr 2020 23:52:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587538353; cv=none; d=google.com; s=arc-20160816; b=YZn62XkEMgOJN/XOWwjqq45pOhNBg5LT31KGJksbnI0Le2795sVU7H5nCSYe01jUgX bvBDXyyIgRzKGefw4QVV/MW9/VYbSezBldx4cSReL8ZYwtD4MaL9PokK0H16NH4vxPE6 PrlXsChsr8nnn1HIKdWMSVYQR3hzxTuFVMHThj6ZqN5LYdMghKRwnB/PEOSgY58DlMmt 8CaWxRkRREGUOOOvk8VxxZTFZIF2fjhQbZ2YB8EcXKe8nrSXNRcZ96ZLmuSc94VQ6HfM 0pd+Y0HKztY37ZPhxs6cpdpPtseMxZE8dHFbDlbc8hH6m4trOD+rFyU2jIBa6fuS917z /g8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=6FJ4Chl3KucJzdj5EnjcUkmDiCKbT27lN34PCChXOk8=; b=vdpUpz+jWjn/IAz+SVjXh2Yj2uMH2iVr3C3fLf2BIdsSH33oniS9cD45PqmR212y/H J/BsxJo5wEM4fowg0MTDFii3PC7vhSjpJqvqt1oAa+L/ZUelsEGmk+eVub2jWZ01iwQb XdIyMc2vJemXNRaBZBe9dpJrHXfz95ml3epA9IrX5x+YcsBR0rDavLcIM3q073rvwn6m tWj6je5sljh/x9iOBaGYAMVuPIaGymMTN43UBiG3NjIlGeHu+IhlZakE788pfKePMq92 gAthkImt30jDnzi9DwbkKfAMWhabVgx5JC8izOTj75o9PzGJ1dBEQJva2a95dvlGuk/l BwPg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id r26si258710edb.5.2020.04.21.23.52.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Apr 2020 23:52:33 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 03M6qW3r012560 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 22 Apr 2020 08:52:33 +0200 Received: from md1za8fc.ad001.siemens.net ([167.87.23.203]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 03M6qWXX000353; Wed, 22 Apr 2020 08:52:32 +0200 Date: Wed, 22 Apr 2020 08:52:26 +0200 From: Henning Schild To: Vijai Kumar K Cc: , , , Vijai Kumar K Subject: Re: [PATCH v5 02/13] deb-dl-dir: Cache host distro debs separately Message-ID: <20200422085226.36b3e606@md1za8fc.ad001.siemens.net> In-Reply-To: <20200417093040.15130-3-Vijaikumar_Kanagarajan@mentor.com> References: <20200417093040.15130-1-Vijaikumar_Kanagarajan@mentor.com> <20200417093040.15130-3-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: eFZqIrNsH9kL Hi Vijai, you found one of the "known" issues of that implementation https://groups.google.com/d/msg/isar-users/8djs0Thl0wI/n5Bo2hPeCgAJ There i talk about "mixing" packages from many sources into one base-apt under one "suite" like "buster". In fact such a mix in the download cache is what you found here. But maybe for the cache we can store everything plain in one folder. Given that an "apt-get source" without a version would "unpack" only what it found in its Sources.gz(s) ... which i would assume. But i think what you suggest here should work as well. And there is probably not much overlap ... packages with exactly the same versions in multiple suites. Henning On Fri, 17 Apr 2020 15:00:29 +0530 Vijai Kumar K wrote: > In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch, > we were still caching the debs from both the distros into the same > DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be > cached in the relevant subdirectory and avoids mixing of debs from > two separate distros. > > Signed-off-by: Vijai Kumar K > --- > meta/classes/deb-dl-dir.bbclass | 4 ++-- > meta/classes/dpkg-gbp.bbclass | 8 ++++++-- > meta/classes/dpkg.bbclass | 8 ++++++-- > meta/classes/image-tools-extension.bbclass | 4 ++-- > meta/classes/rootfs.bbclass | 4 ++-- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++-- > 6 files changed, 28 insertions(+), 12 deletions(-) > > diff --git a/meta/classes/deb-dl-dir.bbclass > b/meta/classes/deb-dl-dir.bbclass index e996703..29a3d67 100644 > --- a/meta/classes/deb-dl-dir.bbclass > +++ b/meta/classes/deb-dl-dir.bbclass > @@ -6,7 +6,7 @@ > inherit repository > > deb_dl_dir_import() { > - export pc="${DEBDIR}/${DISTRO}/" > + export pc="${DEBDIR}/${2}" > export rootfs="${1}" > [ ! -d "${pc}" ] && return 0 > sudo mkdir -p "${rootfs}"/var/cache/apt/archives/ > @@ -20,7 +20,7 @@ deb_dl_dir_import() { > } > > deb_dl_dir_export() { > - export pc="${DEBDIR}/${DISTRO}/" > + export pc="${DEBDIR}/${2}" > export rootfs="${1}" > mkdir -p "${pc}" > flock "${pc}".lock -c ' > diff --git a/meta/classes/dpkg-gbp.bbclass > b/meta/classes/dpkg-gbp.bbclass index afa1e19..ba5c3eb 100644 > --- a/meta/classes/dpkg-gbp.bbclass > +++ b/meta/classes/dpkg-gbp.bbclass > @@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" > > do_install_builddeps_append() { > dpkg_do_mounts > - deb_dl_dir_import "${BUILDCHROOT_DIR}" > + distro="${DISTRO}" > + if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > + distro="${HOST_DISTRO}" > + fi > + deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > --no-install-recommends --download-only > ${GBP_DEPENDS} > - deb_dl_dir_export "${BUILDCHROOT_DIR}" > + deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} \ > apt-get install -y -o Debug::pkgProblemResolver=yes \ > --no-install-recommends ${GBP_DEPENDS} > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 585365e..53862b5 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -7,10 +7,14 @@ inherit dpkg-base > do_install_builddeps() { > dpkg_do_mounts > E="${@ isar_export_proxies(d)}" > - deb_dl_dir_import "${BUILDCHROOT_DIR}" > + distro="${DISTRO}" > + if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then > + distro="${HOST_DISTRO}" > + fi > + deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ > ${PP}/${PPS} ${DISTRO_ARCH} --download-only > - deb_dl_dir_export "${BUILDCHROOT_DIR}" > + deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}" > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ > ${PP}/${PPS} ${DISTRO_ARCH} > dpkg_undo_mounts > diff --git a/meta/classes/image-tools-extension.bbclass > b/meta/classes/image-tools-extension.bbclass index 6590ee7..0b067ff > 100644 --- a/meta/classes/image-tools-extension.bbclass > +++ b/meta/classes/image-tools-extension.bbclass > @@ -25,7 +25,7 @@ do_install_imager_deps() { > buildchroot_do_mounts > > E="${@ isar_export_proxies(d)}" > - deb_dl_dir_import ${BUILDCHROOT_DIR} > + deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO} > sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ > apt-get update \ > -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \ > @@ -35,7 +35,7 @@ do_install_imager_deps() { > --allow-unauthenticated --allow-downgrades > --download-only install \ ${IMAGER_INSTALL}' > > - deb_dl_dir_export ${BUILDCHROOT_DIR} > + deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO} > sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ > apt-get -o Debug::pkgProblemResolver=yes > --no-install-recommends -y \ --allow-unauthenticated > --allow-downgrades install \ diff --git a/meta/classes/rootfs.bbclass > b/meta/classes/rootfs.bbclass index 8bb003d..cee358c 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -115,7 +115,7 @@ rootfs_install_resolvconf() { > ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache" > rootfs_import_package_cache[weight] = "5" > rootfs_import_package_cache() { > - deb_dl_dir_import ${ROOTFSDIR} > + deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO} > } > > ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" > @@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND += > "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}" ROOTFS_INSTALL_COMMAND += > "rootfs_export_package_cache" rootfs_export_package_cache[weight] = > "5" rootfs_export_package_cache() { > - deb_dl_dir_export ${ROOTFSDIR} > + deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO} > } > > ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if > (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" diff --git > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > da0d436..4061c86 100644 --- > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -247,7 > +247,11 @@ isar_bootstrap() { export IS_HOST debootstrap_args E if [ > ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then sudo rm -rf --one-file-system > "${ROOTFSDIR}" > - deb_dl_dir_import "${ROOTFSDIR}" > + if [ "${IS_HOST}" ];then > + deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}" > + else > + deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}" > + fi > > sudo -E -s <<'EOSUDO' > set -e > @@ -344,7 +348,11 @@ isar_bootstrap() { > ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" > EOSUDO > fi > - deb_dl_dir_export "${ROOTFSDIR}" > + if [ "${IS_HOST}" ];then > + deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}" > + else > + deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}" > + fi > } > > CLEANFUNCS = "clean_deploy"