From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6752900179667451904 X-Received: by 2002:a50:85cb:: with SMTP id q11mr2517365edh.141.1573046239053; Wed, 06 Nov 2019 05:17:19 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:cd1e:: with SMTP id z30ls569409edi.15.gmail; Wed, 06 Nov 2019 05:17:18 -0800 (PST) X-Google-Smtp-Source: APXvYqxk5CJHgSVqZHEm8gkIRX6YP59lbpvjePscTYzNTxTRHkNDHhRvhZxaA0tXRZOX9dq5Mdar X-Received: by 2002:a50:8f03:: with SMTP id 3mr2658008edy.195.1573046238571; Wed, 06 Nov 2019 05:17:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573046238; cv=none; d=google.com; s=arc-20160816; b=S/DQE3z0c0AcTH0VftmiVCZVkjgOz0tXXIJJcGnxBLRYdzxaHQa3q9qiZS46NPlJOr 9GTTQva+siJKPGhFsv2AmY+b0ffxJWKQugrzM7Tc7No61CEVfKcRJ4rIDe3fR5jkMcb5 mZYclv9BQ0sriSeIaFy5YDdcUk84IWPGEh9Jkt7vPPgcEKukyer8T247Oz7yrKjZVMz8 JnVsYdV5cv39DoEvHyaTD5FY3wm4EeEMyUtwR/tmVh/xdUGg6MiBp3MUjXy1wFuMP+pU EH+iB1oUbD2OG4BXsbvaAcYIq1kU9eZ52CKwQxNBmlrBlMdpNQp2UzIJdaJUaRGOQGVw R0Qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date; bh=kq7BRcVWDw8HDVIm9jN0F73fTKvwLZFajHTO2uYrUsw=; b=aV4ZbBpfFVsunHO2tRCERefUOUIiQ96Ztw6iobKmDmkC8uRGtdYa6Dc//9bmqW+xDB wLMrEGHyAQeSrFAamMfnYTnSubzZMWEQ8M6njJE6eKFfQspdLluAErBpjgcyeObr41PS OQyaBcAVp7v8NgFsDRFIZLftsMQD/oSAhPaSpCCNdRI5FyyPTZAlvsqAu/N5y8KBClOE UIg45uXno14bqAnmjWbYKOQsiJitYwQCH+i707SQUMs4wN+qyPWmHcQgH48krvu1cv8b 9Pv1suPkef9Vf4np9L0mWecMx6FrhHyQjxRDnDZ0Zji/7S61aYNw+LjhDkpaofi2E1jF fs+w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id v57si1728781edc.3.2019.11.06.05.17.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Nov 2019 05:17:18 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id xA6DHIYD030002 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 6 Nov 2019 14:17:18 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.0.32]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id xA6DHHOs027898 for ; Wed, 6 Nov 2019 14:17:17 +0100 Date: Wed, 6 Nov 2019 14:17:09 +0100 From: Henning Schild To: Subject: Re: [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps Message-ID: <20191106141709.74375aad@md1za8fc.ad001.siemens.net> In-Reply-To: <20191030202138.4bb2054d@md1za8fc.ad001.siemens.net> References: <20191028170132.28072-1-henning.schild@siemens.com> <20191028170132.28072-10-henning.schild@siemens.com> <20191030202138.4bb2054d@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 4oNg3QTH70vK Am Wed, 30 Oct 2019 20:21:38 +0100 schrieb "[ext] Henning Schild" : > A future patch will actually populate base-apt out of the files in > DL_DIR, i did not sent that yet. > > Once the creation+population is done purely from the download cache, > we can not exclude packages from isar-apt anymore. > > At the moment i think that would not be a problem because the rebuild > from base-apt will just work. isar-apt will always win and just > rebuild the "same" stuff. In fact the debs coming from isar-apt should never be extracted into the download cache, because from an isar point of view they have never been downloaded. Henning > The only problem i see is that base-apt will potentially contain > product packages, which means you can not share it as widely as you > can a pure "mirror", which base-apt is not anyways. > > Henning > > On Mon, 28 Oct 2019 18:01:24 +0100 > Henning Schild wrote: > > > From: Henning Schild > > > > If we ever downloaded a file there is no need to do that again. We > > can not only use that DL_DIR to later construct a repository but > > also to speed up our rebuilds. > > > > Signed-off-by: Henning Schild > > --- > > meta/classes/deb-dl-dir.bbclass | 10 ++++++++++ > > meta/classes/dpkg-gbp.bbclass | 1 + > > meta/classes/dpkg.bbclass | 1 + > > meta/classes/image-tools-extension.bbclass | 1 + > > meta/classes/rootfs.bbclass | 6 ++++++ > > 5 files changed, 19 insertions(+) > > > > diff --git a/meta/classes/deb-dl-dir.bbclass > > b/meta/classes/deb-dl-dir.bbclass index e58df7a..b137eb9 100644 > > --- a/meta/classes/deb-dl-dir.bbclass > > +++ b/meta/classes/deb-dl-dir.bbclass > > @@ -3,6 +3,16 @@ > > # > > # SPDX-License-Identifier: MIT > > > > +deb_dl_dir_import() { > > + export pc="${DEBDIR}/${DISTRO}" > > + export rootfs="${1}" > > + [ ! -d "${pc}" ] && exit 0 > > + flock -s "${pc}".lock -c ' \ > > + sudo find "${pc}" -type f -iname '*\.deb' -exec \ > > + cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \; > > + ' > > +} > > + > > deb_dl_dir_export() { > > export pc="${DEBDIR}/${DISTRO}" > > export rootfs="${1}" > > diff --git a/meta/classes/dpkg-gbp.bbclass > > b/meta/classes/dpkg-gbp.bbclass index 9e6945b..be0769c 100644 > > --- a/meta/classes/dpkg-gbp.bbclass > > +++ b/meta/classes/dpkg-gbp.bbclass > > @@ -13,6 +13,7 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar" > > do_install_builddeps_append() { > > dpkg_do_mounts > > E="${@ bb.build.exec_func('isar_export_proxies', d)}" > > + deb_dl_dir_import "${BUILDCHROOT_DIR}" > > sudo -E chroot ${BUILDCHROOT_DIR} \ > > apt-get install -y -o Debug::pkgProblemResolver=yes \ > > --no-install-recommends --download-only > > ${GBP_DEPENDS} diff --git a/meta/classes/dpkg.bbclass > > b/meta/classes/dpkg.bbclass index b6844e9..9e10ad8 100644 > > --- a/meta/classes/dpkg.bbclass > > +++ b/meta/classes/dpkg.bbclass > > @@ -7,6 +7,7 @@ inherit dpkg-base > > do_install_builddeps() { > > dpkg_do_mounts > > E="${@ bb.build.exec_func('isar_export_proxies', d)}" > > + deb_dl_dir_import "${BUILDCHROOT_DIR}" > > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ > > ${PP}/${PPS} ${DISTRO_ARCH} --download-only > > deb_dl_dir_export "${BUILDCHROOT_DIR}" > > diff --git a/meta/classes/image-tools-extension.bbclass > > b/meta/classes/image-tools-extension.bbclass index 3906585..08b426d > > 100644 --- a/meta/classes/image-tools-extension.bbclass > > +++ b/meta/classes/image-tools-extension.bbclass > > @@ -25,6 +25,7 @@ do_install_imager_deps() { > > buildchroot_do_mounts > > > > E="${@ bb.build.exec_func('isar_export_proxies', d)}" > > + deb_dl_dir_import ${BUILDCHROOT_DIR} > > sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ > > apt-get update \ > > -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" > > \ diff --git a/meta/classes/rootfs.bbclass > > b/meta/classes/rootfs.bbclass index 841aa89..845304c 100644 > > --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -113,6 +113,12 @@ rootfs_install_resolvconf() { > > fi > > } > > > > +ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache" > > +rootfs_import_package_cache[weight] = "5" > > +rootfs_import_package_cache() { > > + deb_dl_dir_import ${ROOTFSDIR} > > +} > > + > > ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" > > rootfs_install_pkgs_download[weight] = "600" > > rootfs_install_pkgs_download[isar-apt-lock] = "release-after" >