From: chombourger@gmail.com
To: isar-users <isar-users@googlegroups.com>
Subject: Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs
Date: Tue, 29 Oct 2019 05:48:01 -0700 (PDT) [thread overview]
Message-ID: <5c2908a6-97df-46dd-bb9b-21af77202918@googlegroups.com> (raw)
In-Reply-To: <20191028170132.28072-9-henning.schild@siemens.com>
[-- Attachment #1.1: Type: text/plain, Size: 6449 bytes --]
On Monday, October 28, 2019 at 6:01:36 PM UTC+1, Henning Schild wrote:
>
> From: Henning Schild <hennin...@siemens.com <javascript:>>
>
> Hook in between the "apt-get --download-only"s and the "apt-get"s and
> copy out all the debs for later.
>
> Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>>
> ---
> meta/classes/deb-dl-dir.bbclass | 14 ++++++++++++++
> meta/classes/dpkg-base.bbclass | 1 +
> meta/classes/dpkg-gbp.bbclass | 1 +
> meta/classes/dpkg.bbclass | 1 +
> meta/classes/image-locales-extension.bbclass | 2 +-
> meta/classes/image-tools-extension.bbclass | 1 +
> meta/classes/rootfs.bbclass | 12 ++++++++++--
> meta/conf/bitbake.conf | 1 +
> 8 files changed, 30 insertions(+), 3 deletions(-)
> create mode 100644 meta/classes/deb-dl-dir.bbclass
>
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass
> new file mode 100644
> index 0000000..e58df7a
> --- /dev/null
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -0,0 +1,14 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2019 Siemens AG
> +#
> +# SPDX-License-Identifier: MIT
> +
> +deb_dl_dir_export() {
> + export pc="${DEBDIR}/${DISTRO}"
> + export rootfs="${1}"
> + mkdir -p "${pc}"
> + flock "${pc}".lock -c ' \
> + sudo find "${rootfs}"/var/cache/apt/archives/ -type f -iname
> '*\.deb' \
> + -exec cp -f '{}' "${pc}" \;
>
use -exec command {} + to reduce the number of fork/execvp pairs?
note: you would need to use cp -t "${pc}" since the "+" modifier appends
matches to the end of the command
> + '
> +}
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass
> index b39df5c..8815478 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -8,6 +8,7 @@ inherit buildchroot
> inherit debianize
> inherit terminal
> inherit repository
> +inherit deb-dl-dir
>
> DEPENDS ?= ""
>
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> index d9c3af0..9e6945b 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -16,6 +16,7 @@ do_install_builddeps_append() {
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends --download-only
> ${GBP_DEPENDS}
> + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends ${GBP_DEPENDS}
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index d809cca..b6844e9 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -9,6 +9,7 @@ do_install_builddeps() {
> E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH}
> dpkg_undo_mounts
> diff --git a/meta/classes/image-locales-extension.bbclass
> b/meta/classes/image-locales-extension.bbclass
> index 3c0758f..0f0d0ca 100644
> --- a/meta/classes/image-locales-extension.bbclass
> +++ b/meta/classes/image-locales-extension.bbclass
> @@ -25,7 +25,7 @@ def get_nopurge(d):
> j.split()[0].split(".")[0],
> j.split()[0]))))
>
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN +=
> "image_install_localepurge_download"
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT +=
> "image_install_localepurge_download"
> image_install_localepurge_download[weight] = "40"
> image_install_localepurge_download() {
> sudo -E chroot '${ROOTFSDIR}' \
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass
> index 243fa11..3906585 100644
> --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -34,6 +34,7 @@ do_install_imager_deps() {
> --allow-unauthenticated --allow-downgrades --download-only
> install \
> ${IMAGER_INSTALL}'
>
> + deb_dl_dir_export ${BUILDCHROOT_DIR}
> sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends
> -y \
> --allow-unauthenticated --allow-downgrades install \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 095ebc1..841aa89 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -1,6 +1,8 @@
> # This software is a part of ISAR.
> # Copyright (c) Siemens AG, 2019
>
> +inherit deb-dl-dir
> +
> ROOTFS_ARCH ?= "${DISTRO_ARCH}"
> ROOTFS_DISTRO ?= "${DISTRO}"
> ROOTFS_PACKAGES ?= ""
> @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() {
> /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> ${ROOTFS_PACKAGES}
> }
>
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
> -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> +ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
> +
> +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
> +rootfs_export_package_cache[weight] = "5"
> +rootfs_export_package_cache() {
> + deb_dl_dir_export ${ROOTFSDIR}
> +}
>
> ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if
> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
> rootfs_install_clean_files[weight] = "2"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 4c0a809..56f4fbf 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
> FILESPATH =
> "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
>
> FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> GITDIR = "${DL_DIR}/git"
> +DEBDIR = "${DL_DIR}/deb"
> DEBSRCDIR = "${DL_DIR}/deb-src"
> P = "${PN}-${PV}"
> PF = "${PN}-${PV}-${PR}"
> --
> 2.23.0
>
>
[-- Attachment #1.2: Type: text/html, Size: 8373 bytes --]
next prev parent reply other threads:[~2019-10-29 12:48 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-28 17:01 [PATCH 00/17] base-apt rework Henning Schild
2019-10-28 17:01 ` [PATCH 01/17] repository: new class to deal with repos Henning Schild
2019-11-24 15:29 ` Baurzhan Ismagulov
2019-11-25 12:03 ` Henning Schild
2019-10-28 17:01 ` [PATCH 02/17] dpkg-base: add download caching of apt:// downloads Henning Schild
2019-10-29 12:57 ` chombourger
2019-10-29 16:03 ` Henning Schild
2019-11-08 11:39 ` Henning Schild
2019-11-24 15:30 ` Baurzhan Ismagulov
2019-11-25 10:32 ` Henning Schild
2019-10-28 17:01 ` [PATCH 03/17] base-apt: change the sources.list to also offer deb-src Henning Schild
2019-10-28 17:01 ` [PATCH 04/17] base-apt: add deb-src packages as well Henning Schild
2019-10-30 19:23 ` Henning Schild
2019-10-28 17:01 ` [PATCH 05/17] base-apt: do not skip gpg check when it is signed Henning Schild
2019-11-24 15:35 ` Baurzhan Ismagulov
2019-11-25 10:30 ` Henning Schild
2019-10-28 17:01 ` [PATCH 06/17] ci: conf: add "hello" to the sample config and every build Henning Schild
2019-10-28 17:01 ` [PATCH 07/17] meta: split all apt-get invocations into download and execution Henning Schild
2019-11-24 15:41 ` Baurzhan Ismagulov
2019-11-25 10:20 ` Henning Schild
2019-10-28 17:01 ` [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs Henning Schild
2019-10-29 12:48 ` chombourger [this message]
2019-10-29 16:02 ` Henning Schild
2019-11-06 13:19 ` Henning Schild
2019-11-24 15:57 ` Baurzhan Ismagulov
2019-11-25 10:15 ` Henning Schild
2019-10-28 17:01 ` [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps Henning Schild
2019-10-30 19:21 ` Henning Schild
2019-11-06 13:17 ` Henning Schild
2019-11-24 20:20 ` Baurzhan Ismagulov
2019-11-25 10:13 ` Henning Schild
2019-10-28 17:01 ` [PATCH 10/17] base-apt: populate from DEBDIR as well Henning Schild
2019-10-28 17:01 ` [PATCH 11/17] base-apt: drop the "apt_cache" feature Henning Schild
2019-10-28 17:01 ` [PATCH 12/17] base-apt: do not copy debs directly out of rootfss anymore Henning Schild
2019-10-28 17:01 ` [PATCH 13/17] base-apt: rework base-apt population Henning Schild
2019-10-28 17:01 ` [PATCH 14/17] base-apt: move class "base-apt-helper" into only user Henning Schild
2019-10-28 17:01 ` [PATCH 15/17] CI: include "isar-disable-apt-cache" into all CI images Henning Schild
2019-10-28 17:01 ` [PATCH 16/17] CI: include "cowsay" into default build to test dpkg-gbp Henning Schild
2019-10-28 17:01 ` [PATCH 17/17] CI: set BB_NO_NETWORK for cached rebuild Henning Schild
2019-11-06 16:00 ` [PATCH 00/17] base-apt rework Jan Kiszka
2019-11-24 13:21 ` Baurzhan Ismagulov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5c2908a6-97df-46dd-bb9b-21af77202918@googlegroups.com \
--to=chombourger@gmail.com \
--cc=isar-users@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox