public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
From: Henning Schild <henning.schild@siemens.com>
To: isar-users@googlegroups.com
Cc: Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>,
	Baurzhan Ismagulov <ibr@radix50.net>,
	Jan Kiszka <jan.kiszka@siemens.com>,
	Henning Schild <henning.schild@siemens.com>
Subject: [PATCHv7 10/29] meta: import DL_DIR debs before apt-get download steps
Date: Sat, 21 Mar 2020 09:31:29 +0100	[thread overview]
Message-ID: <20200321083148.26160-11-henning.schild@siemens.com> (raw)
In-Reply-To: <20200321083148.26160-1-henning.schild@siemens.com>

From: Henning Schild <henning.schild@siemens.com>

If we ever downloaded a file there is no need to do that again. We can
not only use that DL_DIR to later construct a repository but also to
speed up our rebuilds.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass            | 10 ++++++++++
 meta/classes/dpkg-gbp.bbclass              |  1 +
 meta/classes/dpkg.bbclass                  |  1 +
 meta/classes/image-tools-extension.bbclass |  1 +
 meta/classes/rootfs.bbclass                |  6 ++++++
 5 files changed, 19 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 066ce1a3..2858fea2 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -3,6 +3,16 @@
 #
 # SPDX-License-Identifier: MIT
 
+deb_dl_dir_import() {
+    export pc="${DEBDIR}/${DISTRO}"
+    export rootfs="${1}"
+    [ ! -d "${pc}" ] && return 0
+    flock -s "${pc}".lock -c ' \
+        sudo find "${pc}" -type f -iname '*\.deb' -exec \
+            cp -f --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ '{}' +
+    '
+}
+
 deb_dl_dir_export() {
     export pc="${DEBDIR}/${DISTRO}"
     export rootfs="${1}"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 7e908eba..afa1e19b 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,6 +12,7 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
 do_install_builddeps_append() {
     dpkg_do_mounts
+    deb_dl_dir_import "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends --download-only ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 7e2c86d6..585365ee 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -7,6 +7,7 @@ inherit dpkg-base
 do_install_builddeps() {
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
+    deb_dl_dir_import "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${DISTRO_ARCH} --download-only
     deb_dl_dir_export "${BUILDCHROOT_DIR}"
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index ec67d940..6590ee7a 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,6 +25,7 @@ do_install_imager_deps() {
     buildchroot_do_mounts
 
     E="${@ isar_export_proxies(d)}"
+    deb_dl_dir_import ${BUILDCHROOT_DIR}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 706b43a7..81759ebc 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -113,6 +113,12 @@ rootfs_install_resolvconf() {
     fi
 }
 
+ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
+rootfs_import_package_cache[weight] = "5"
+rootfs_import_package_cache() {
+    deb_dl_dir_import ${ROOTFSDIR}
+}
+
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
-- 
2.24.1


  parent reply	other threads:[~2020-03-21  8:32 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-21  8:31 [PATCHv7 00/29] base-apt-rework Henning Schild
2020-03-21  8:31 ` [PATCHv7 01/29] repository: new class to deal with repos Henning Schild
2020-03-21  8:31 ` [PATCHv7 02/29] dpkg-base: add download caching of apt:// downloads Henning Schild
2020-03-21  8:31 ` [PATCHv7 03/29] meta: dpkg-base: convert "exit" into "return 0" Henning Schild
2020-03-21  8:31 ` [PATCHv7 04/29] base-apt: change the sources.list to also offer deb-src Henning Schild
2020-03-21  8:31 ` [PATCHv7 05/29] base-apt: add deb-src packages as well Henning Schild
2020-03-21  8:31 ` [PATCHv7 06/29] base-apt: do not skip gpg check when it is signed Henning Schild
2020-03-21  8:31 ` [PATCHv7 07/29] ci: conf: add "hello" to the sample config and every build Henning Schild
2020-03-21  8:31 ` [PATCHv7 08/29] meta: split all apt-get invocations into download and execution Henning Schild
2020-03-21  8:31 ` [PATCHv7 09/29] meta: create DL_DIR support for all apt-get downloaded .debs Henning Schild
2020-03-21  8:31 ` Henning Schild [this message]
2020-03-21  8:31 ` [PATCHv7 11/29] meta: include DL_DIR deb import/export into bootstrap Henning Schild
2020-03-21  8:31 ` [PATCHv7 12/29] base-apt: populate from DEBDIR as well Henning Schild
2020-03-21  8:31 ` [PATCHv7 13/29] base-apt: drop the "apt_cache" feature Henning Schild
2020-03-21  8:31 ` [PATCHv7 14/29] base-apt: do not copy debs directly out of rootfss anymore Henning Schild
2020-03-21  8:31 ` [PATCHv7 15/29] base-apt: rework base-apt population Henning Schild
2020-03-21  8:31 ` [PATCHv7 16/29] base-apt: move class "base-apt-helper" into only user Henning Schild
2020-03-21  8:31 ` [PATCHv7 17/29] CI: include "isar-disable-apt-cache" into all CI images Henning Schild
2020-03-21  8:31 ` [PATCHv7 18/29] CI: include "cowsay" into default build to test dpkg-gbp Henning Schild
2020-03-21  8:31 ` [PATCHv7 19/29] CI: set BB_NO_NETWORK for cached rebuild Henning Schild
2020-03-21  8:31 ` [PATCHv7 20/29] meta: repository: implement repo_contains_package and use it in base_apt Henning Schild
2020-03-21  8:31 ` [PATCHv7 21/29] meta: repository: simplify the matching of packages Henning Schild
2020-03-21  8:31 ` [PATCHv7 22/29] meta: deb-dl-dir: do not cache debs from isar-apt Henning Schild
2020-03-21  8:31 ` [PATCHv7 23/29] base-apt: pull base-apt population to the front of the build chain Henning Schild
2020-03-21  8:31 ` [PATCHv7 24/29] meta: base-apt: remove isar-apt check from population Henning Schild
2020-03-21  8:31 ` [PATCHv7 25/29] meta: deb-dl-dir: only export newly downloaded files Henning Schild
2020-03-21  8:31 ` [PATCHv7 26/29] CI: test a custom kernel build in the base-apt offline rebuild Henning Schild
2020-03-21  8:31 ` [PATCHv7 27/29] meta/deb-dl-dir: make subshell verbose and fail on error Henning Schild
2020-03-21  8:31 ` [PATCHv7 28/29] meta: deb-dl-dir: fix "command not found" error when excluding pkgs Henning Schild
2020-03-21  8:31 ` [PATCHv7 29/29] RECIPE-API-CHANGELOG: add a few bits on the base-apt caching changes Henning Schild
2020-03-21  8:41 ` [PATCHv7 00/29] base-apt-rework Henning Schild
2020-03-23 10:56 ` Baurzhan Ismagulov
2020-03-23 11:01   ` Jan Kiszka
2020-03-23 12:22     ` Baurzhan Ismagulov
2020-03-23 15:25       ` Henning Schild
2020-03-24 10:49         ` Baurzhan Ismagulov
2020-03-25 10:35           ` Henning Schild
2020-03-25 11:13             ` Jan Kiszka
2020-03-25 11:25               ` Henning Schild
2020-03-25 11:14             ` Baurzhan Ismagulov
2020-03-26 12:53               ` Henning Schild

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200321083148.26160-11-henning.schild@siemens.com \
    --to=henning.schild@siemens.com \
    --cc=ibr@radix50.net \
    --cc=isar-users@googlegroups.com \
    --cc=jan.kiszka@siemens.com \
    --cc=vijaikumar.kanagarajan@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox