From: vijai kumar <vijaikumar.kanagarajan@gmail.com>
To: isar-users <isar-users@googlegroups.com>
Subject: Re: [PATCHv2 09/25] meta: create DL_DIR support for all apt-get downloaded .debs
Date: Sat, 25 Jan 2020 06:06:59 -0800 (PST) [thread overview]
Message-ID: <8bb53ef9-64ee-449e-a3b2-58eda2dab41e@googlegroups.com> (raw)
In-Reply-To: <20191216115011.17664-10-henning.schild@siemens.com>
[-- Attachment #1.1: Type: text/plain, Size: 6724 bytes --]
On Monday, December 16, 2019 at 5:20:15 PM UTC+5:30, Henning Schild wrote:
>
> From: Henning Schild <hennin...@siemens.com <javascript:>>
>
> Hook in between the "apt-get --download-only"s and the "apt-get"s and
> copy out all the debs for later.
>
> Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>>
> ---
> meta/classes/deb-dl-dir.bbclass | 15 +++++++++++++++
> meta/classes/dpkg-base.bbclass | 1 +
> meta/classes/dpkg-gbp.bbclass | 1 +
> meta/classes/dpkg.bbclass | 1 +
> meta/classes/image-locales-extension.bbclass | 2 +-
> meta/classes/image-tools-extension.bbclass | 1 +
> meta/classes/rootfs.bbclass | 12 ++++++++++--
> meta/conf/bitbake.conf | 1 +
> 8 files changed, 31 insertions(+), 3 deletions(-)
> create mode 100644 meta/classes/deb-dl-dir.bbclass
>
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass
> new file mode 100644
> index 0000000..e41e981
> --- /dev/null
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -0,0 +1,15 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2019 Siemens AG
> +#
> +# SPDX-License-Identifier: MIT
> +
> +deb_dl_dir_export() {
> + export pc="${DEBDIR}/${DISTRO}"
> + export rootfs="${1}"
> + mkdir -p "${pc}"
> + flock "${pc}".lock -c ' \
> + sudo find "${rootfs}"/var/cache/apt/archives/ -type f -iname
> '*\.deb' \
> + -exec cp -f '{}' "${pc}" \;
> + sudo chown -R $(id -u):$(id -g) "${pc}"
> + '
> +}
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass
> index aa11660..fd0b07f 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -8,6 +8,7 @@ inherit buildchroot
> inherit debianize
> inherit terminal
> inherit repository
> +inherit deb-dl-dir
>
> DEPENDS ?= ""
>
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> index 2afe9db..7e908eb 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -15,6 +15,7 @@ do_install_builddeps_append() {
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends --download-only
> ${GBP_DEPENDS}
> + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> sudo -E chroot ${BUILDCHROOT_DIR} \
> apt-get install -y -o Debug::pkgProblemResolver=yes \
> --no-install-recommends ${GBP_DEPENDS}
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index b26c645..7e2c86d 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -9,6 +9,7 @@ do_install_builddeps() {
While testing this series in a downstream project, I do not see a part of
KBUILD_DEPENDS cached.
We don't have the "linux-custom recipe rework" changes by Cedric there. The
old Linux recipe overrides
this task with its own variant.
I have added deb_dl_* there as well to fix the issue.
This might become a problem when more such recipes pop-up. Unlikely, but
possible.
Recipe authors should know they have to take care of this if they decide to
override
do_install_builddeps.
Thanks,
Vijai Kumar K
E="${@ isar_export_proxies(d)}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> ${PP}/${PPS} ${DISTRO_ARCH}
> dpkg_undo_mounts
> diff --git a/meta/classes/image-locales-extension.bbclass
> b/meta/classes/image-locales-extension.bbclass
> index 3c0758f..0f0d0ca 100644
> --- a/meta/classes/image-locales-extension.bbclass
> +++ b/meta/classes/image-locales-extension.bbclass
> @@ -25,7 +25,7 @@ def get_nopurge(d):
> j.split()[0].split(".")[0],
> j.split()[0]))))
>
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN +=
> "image_install_localepurge_download"
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT +=
> "image_install_localepurge_download"
> image_install_localepurge_download[weight] = "40"
> image_install_localepurge_download() {
> sudo -E chroot '${ROOTFSDIR}' \
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass
> index b8531cb..ec67d94 100644
> --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -34,6 +34,7 @@ do_install_imager_deps() {
> --allow-unauthenticated --allow-downgrades --download-only
> install \
> ${IMAGER_INSTALL}'
>
> + deb_dl_dir_export ${BUILDCHROOT_DIR}
> sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends
> -y \
> --allow-unauthenticated --allow-downgrades install \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index b4dbfa3..a6b6110 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -1,6 +1,8 @@
> # This software is a part of ISAR.
> # Copyright (c) Siemens AG, 2019
>
> +inherit deb-dl-dir
> +
> ROOTFS_ARCH ?= "${DISTRO_ARCH}"
> ROOTFS_DISTRO ?= "${DISTRO}"
> ROOTFS_PACKAGES ?= ""
> @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() {
> /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> ${ROOTFS_PACKAGES}
> }
>
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
> -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> +ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
> +
> +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
> +rootfs_export_package_cache[weight] = "5"
> +rootfs_export_package_cache() {
> + deb_dl_dir_export ${ROOTFSDIR}
> +}
>
> ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if
> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
> rootfs_install_clean_files[weight] = "2"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index ab608da..d86c5b9 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
> FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> FILESEXTRAPATHS ?= "__default:"
> GITDIR = "${DL_DIR}/git"
> +DEBDIR = "${DL_DIR}/deb"
> DEBSRCDIR = "${DL_DIR}/deb-src"
> P = "${PN}-${PV}"
> PF = "${PN}-${PV}-${PR}"
> --
> 2.24.1
>
>
[-- Attachment #1.2: Type: text/html, Size: 8738 bytes --]
next prev parent reply other threads:[~2020-01-25 14:06 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-16 11:49 [PATCHv2 00/25] base-apt-rework Henning Schild
2019-12-16 11:49 ` [PATCHv2 01/25] repository: new class to deal with repos Henning Schild
2019-12-16 11:49 ` [PATCHv2 02/25] dpkg-base: add download caching of apt:// downloads Henning Schild
2020-01-09 11:27 ` vijai kumar
2020-01-10 9:51 ` Henning Schild
2020-01-10 9:58 ` vijai kumar
2019-12-16 11:49 ` [PATCHv2 03/25] meta: dpkg-base: convert "exit" into "return 0" Henning Schild
2019-12-16 11:49 ` [PATCHv2 04/25] base-apt: change the sources.list to also offer deb-src Henning Schild
2019-12-16 11:49 ` [PATCHv2 05/25] base-apt: add deb-src packages as well Henning Schild
2019-12-16 11:49 ` [PATCHv2 06/25] base-apt: do not skip gpg check when it is signed Henning Schild
2019-12-16 11:49 ` [PATCHv2 07/25] ci: conf: add "hello" to the sample config and every build Henning Schild
2019-12-16 11:49 ` [PATCHv2 08/25] meta: split all apt-get invocations into download and execution Henning Schild
2019-12-16 11:49 ` [PATCHv2 09/25] meta: create DL_DIR support for all apt-get downloaded .debs Henning Schild
2020-01-25 14:06 ` vijai kumar [this message]
2020-01-28 14:43 ` Henning Schild
2020-01-28 15:41 ` Base-apt rework v3 Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 01/26] repository: new class to deal with repos Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 02/26] dpkg-base: add download caching of apt:// downloads Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 03/26] meta: dpkg-base: convert "exit" into "return 0" Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 04/26] base-apt: change the sources.list to also offer deb-src Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 05/26] base-apt: add deb-src packages as well Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 06/26] base-apt: do not skip gpg check when it is signed Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 07/26] ci: conf: add "hello" to the sample config and every build Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 08/26] meta: split all apt-get invocations into download and execution Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 09/26] meta: create DL_DIR support for all apt-get downloaded .debs Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 10/26] meta: import DL_DIR debs before apt-get download steps Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 11/26] meta: include DL_DIR deb import/export into bootstrap Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 12/26] base-apt: populate from DEBDIR as well Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 13/26] base-apt: drop the "apt_cache" feature Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 14/26] base-apt: do not copy debs directly out of rootfss anymore Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 15/26] base-apt: rework base-apt population Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 16/26] base-apt: move class "base-apt-helper" into only user Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 17/26] CI: include "isar-disable-apt-cache" into all CI images Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 18/26] CI: include "cowsay" into default build to test dpkg-gbp Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 19/26] CI: set BB_NO_NETWORK for cached rebuild Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 20/26] meta: repository: implement repo_contains_package and use it in base_apt Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 21/26] meta: repository: simplify the matching of packages Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 22/26] meta: deb-dl-dir: do not cache debs from isar-apt Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 23/26] base-apt: pull base-apt population to the front of the build chain Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 24/26] meta: base-apt: remove isar-apt check from population Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 25/26] meta: deb-dl-dir: only export newly downloaded files Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 26/26] meta/recipes-kernel: Export KBUILD_DEPENDS to cache Vijai Kumar K
2020-01-28 16:52 ` Henning Schild
2020-01-28 19:45 ` vijai kumar
2020-01-28 21:22 ` Henning Schild
2020-01-29 5:29 ` Vijai Kumar K
2020-01-31 9:27 ` Henning Schild
2019-12-16 11:49 ` [PATCHv2 10/25] meta: import DL_DIR debs before apt-get download steps Henning Schild
2019-12-16 11:49 ` [PATCHv2 11/25] meta: include DL_DIR deb import/export into bootstrap Henning Schild
2019-12-16 11:49 ` [PATCHv2 12/25] base-apt: populate from DEBDIR as well Henning Schild
2019-12-16 11:49 ` [PATCHv2 13/25] base-apt: drop the "apt_cache" feature Henning Schild
2019-12-16 11:50 ` [PATCHv2 14/25] base-apt: do not copy debs directly out of rootfss anymore Henning Schild
2019-12-16 11:50 ` [PATCHv2 15/25] base-apt: rework base-apt population Henning Schild
2019-12-16 11:50 ` [PATCHv2 16/25] base-apt: move class "base-apt-helper" into only user Henning Schild
2019-12-16 11:50 ` [PATCHv2 17/25] CI: include "isar-disable-apt-cache" into all CI images Henning Schild
2019-12-16 11:50 ` [PATCHv2 18/25] CI: include "cowsay" into default build to test dpkg-gbp Henning Schild
2019-12-16 11:50 ` [PATCHv2 19/25] CI: set BB_NO_NETWORK for cached rebuild Henning Schild
2019-12-16 11:50 ` [PATCHv2 20/25] meta: repository: implement repo_contains_package and use it in base_apt Henning Schild
2019-12-16 11:50 ` [PATCHv2 21/25] meta: repository: simplify the matching of packages Henning Schild
2019-12-16 11:50 ` [PATCHv2 22/25] meta: deb-dl-dir: do not cache debs from isar-apt Henning Schild
2019-12-16 11:50 ` [PATCHv2 23/25] base-apt: pull base-apt population to the front of the build chain Henning Schild
2020-01-21 8:55 ` vijai kumar
2020-01-21 9:54 ` Henning Schild
2020-01-21 13:01 ` vijai kumar
2020-01-27 7:32 ` Henning Schild
2019-12-16 11:50 ` [PATCHv2 24/25] meta: base-apt: remove isar-apt check from population Henning Schild
2019-12-16 11:50 ` [PATCHv2 25/25] meta: deb-dl-dir: only export newly downloaded files Henning Schild
2019-12-19 9:21 ` [PATCHv2 00/25] base-apt-rework Henning Schild
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8bb53ef9-64ee-449e-a3b2-58eda2dab41e@googlegroups.com \
--to=vijaikumar.kanagarajan@gmail.com \
--cc=isar-users@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox