From: Henning Schild <henning.schild@siemens.com>
To: vijai kumar <vijaikumar.kanagarajan@gmail.com>
Cc: isar-users <isar-users@googlegroups.com>
Subject: Re: [PATCHv2 09/25] meta: create DL_DIR support for all apt-get downloaded .debs
Date: Tue, 28 Jan 2020 15:43:12 +0100 [thread overview]
Message-ID: <20200128154312.337e61f6@md1za8fc.ad001.siemens.net> (raw)
In-Reply-To: <8bb53ef9-64ee-449e-a3b2-58eda2dab41e@googlegroups.com>
Hi,
thanks for the report. Since you already have a patch ready, can you
please share it? I would suggest to include it into the series, or
apply it on top. Or squash it in.
Henning
On Sat, 25 Jan 2020 06:06:59 -0800
vijai kumar <vijaikumar.kanagarajan@gmail.com> wrote:
> On Monday, December 16, 2019 at 5:20:15 PM UTC+5:30, Henning Schild
> wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>>
> >
> > Hook in between the "apt-get --download-only"s and the "apt-get"s
> > and copy out all the debs for later.
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>>
> > ---
> > meta/classes/deb-dl-dir.bbclass | 15 +++++++++++++++
> > meta/classes/dpkg-base.bbclass | 1 +
> > meta/classes/dpkg-gbp.bbclass | 1 +
> > meta/classes/dpkg.bbclass | 1 +
> > meta/classes/image-locales-extension.bbclass | 2 +-
> > meta/classes/image-tools-extension.bbclass | 1 +
> > meta/classes/rootfs.bbclass | 12 ++++++++++--
> > meta/conf/bitbake.conf | 1 +
> > 8 files changed, 31 insertions(+), 3 deletions(-)
> > create mode 100644 meta/classes/deb-dl-dir.bbclass
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass
> > new file mode 100644
> > index 0000000..e41e981
> > --- /dev/null
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -0,0 +1,15 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2019 Siemens AG
> > +#
> > +# SPDX-License-Identifier: MIT
> > +
> > +deb_dl_dir_export() {
> > + export pc="${DEBDIR}/${DISTRO}"
> > + export rootfs="${1}"
> > + mkdir -p "${pc}"
> > + flock "${pc}".lock -c ' \
> > + sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > -iname '*\.deb' \
> > + -exec cp -f '{}' "${pc}" \;
> > + sudo chown -R $(id -u):$(id -g) "${pc}"
> > + '
> > +}
> > diff --git a/meta/classes/dpkg-base.bbclass
> > b/meta/classes/dpkg-base.bbclass
> > index aa11660..fd0b07f 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -8,6 +8,7 @@ inherit buildchroot
> > inherit debianize
> > inherit terminal
> > inherit repository
> > +inherit deb-dl-dir
> >
> > DEPENDS ?= ""
> >
> > diff --git a/meta/classes/dpkg-gbp.bbclass
> > b/meta/classes/dpkg-gbp.bbclass index 2afe9db..7e908eb 100644
> > --- a/meta/classes/dpkg-gbp.bbclass
> > +++ b/meta/classes/dpkg-gbp.bbclass
> > @@ -15,6 +15,7 @@ do_install_builddeps_append() {
> > sudo -E chroot ${BUILDCHROOT_DIR} \
> > apt-get install -y -o Debug::pkgProblemResolver=yes \
> > --no-install-recommends --download-only
> > ${GBP_DEPENDS}
> > + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> > sudo -E chroot ${BUILDCHROOT_DIR} \
> > apt-get install -y -o Debug::pkgProblemResolver=yes \
> > --no-install-recommends ${GBP_DEPENDS}
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index b26c645..7e2c86d 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -9,6 +9,7 @@ do_install_builddeps() {
>
>
>
> While testing this series in a downstream project, I do not see a
> part of KBUILD_DEPENDS cached.
>
> We don't have the "linux-custom recipe rework" changes by Cedric
> there. The old Linux recipe overrides
> this task with its own variant.
>
> I have added deb_dl_* there as well to fix the issue.
>
> This might become a problem when more such recipes pop-up. Unlikely,
> but possible.
> Recipe authors should know they have to take care of this if they
> decide to override
> do_install_builddeps.
>
>
> Thanks,
> Vijai Kumar K
>
>
> E="${@ isar_export_proxies(d)}"
> > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> > ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> > + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> > ${PP}/${PPS} ${DISTRO_ARCH}
> > dpkg_undo_mounts
> > diff --git a/meta/classes/image-locales-extension.bbclass
> > b/meta/classes/image-locales-extension.bbclass
> > index 3c0758f..0f0d0ca 100644
> > --- a/meta/classes/image-locales-extension.bbclass
> > +++ b/meta/classes/image-locales-extension.bbclass
> > @@ -25,7 +25,7 @@ def get_nopurge(d):
> > j.split()[0].split(".")[0],
> > j.split()[0]))))
> >
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN +=
> > "image_install_localepurge_download"
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT +=
> > "image_install_localepurge_download"
> > image_install_localepurge_download[weight] = "40"
> > image_install_localepurge_download() {
> > sudo -E chroot '${ROOTFSDIR}' \
> > diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass
> > index b8531cb..ec67d94 100644
> > --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -34,6 +34,7 @@ do_install_imager_deps() {
> > --allow-unauthenticated --allow-downgrades
> > --download-only install \
> > ${IMAGER_INSTALL}'
> >
> > + deb_dl_dir_export ${BUILDCHROOT_DIR}
> > sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> > apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \
> > --allow-unauthenticated --allow-downgrades install \
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index b4dbfa3..a6b6110 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -1,6 +1,8 @@
> > # This software is a part of ISAR.
> > # Copyright (c) Siemens AG, 2019
> >
> > +inherit deb-dl-dir
> > +
> > ROOTFS_ARCH ?= "${DISTRO_ARCH}"
> > ROOTFS_DISTRO ?= "${DISTRO}"
> > ROOTFS_PACKAGES ?= ""
> > @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() {
> > /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> > ${ROOTFS_PACKAGES}
> > }
> >
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
> > -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> > +ROOTFS_INSTALL_COMMAND +=
> > "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
> > +
> > +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
> > +rootfs_export_package_cache[weight] = "5"
> > +rootfs_export_package_cache() {
> > + deb_dl_dir_export ${ROOTFSDIR}
> > +}
> >
> > ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if
> > (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
> > rootfs_install_clean_files[weight] = "2"
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index ab608da..d86c5b9 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
> > FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> > FILESEXTRAPATHS ?= "__default:"
> > GITDIR = "${DL_DIR}/git"
> > +DEBDIR = "${DL_DIR}/deb"
> > DEBSRCDIR = "${DL_DIR}/deb-src"
> > P = "${PN}-${PV}"
> > PF = "${PN}-${PV}-${PR}"
> > --
> > 2.24.1
> >
> >
>
next prev parent reply other threads:[~2020-01-28 14:43 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-16 11:49 [PATCHv2 00/25] base-apt-rework Henning Schild
2019-12-16 11:49 ` [PATCHv2 01/25] repository: new class to deal with repos Henning Schild
2019-12-16 11:49 ` [PATCHv2 02/25] dpkg-base: add download caching of apt:// downloads Henning Schild
2020-01-09 11:27 ` vijai kumar
2020-01-10 9:51 ` Henning Schild
2020-01-10 9:58 ` vijai kumar
2019-12-16 11:49 ` [PATCHv2 03/25] meta: dpkg-base: convert "exit" into "return 0" Henning Schild
2019-12-16 11:49 ` [PATCHv2 04/25] base-apt: change the sources.list to also offer deb-src Henning Schild
2019-12-16 11:49 ` [PATCHv2 05/25] base-apt: add deb-src packages as well Henning Schild
2019-12-16 11:49 ` [PATCHv2 06/25] base-apt: do not skip gpg check when it is signed Henning Schild
2019-12-16 11:49 ` [PATCHv2 07/25] ci: conf: add "hello" to the sample config and every build Henning Schild
2019-12-16 11:49 ` [PATCHv2 08/25] meta: split all apt-get invocations into download and execution Henning Schild
2019-12-16 11:49 ` [PATCHv2 09/25] meta: create DL_DIR support for all apt-get downloaded .debs Henning Schild
2020-01-25 14:06 ` vijai kumar
2020-01-28 14:43 ` Henning Schild [this message]
2020-01-28 15:41 ` Base-apt rework v3 Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 01/26] repository: new class to deal with repos Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 02/26] dpkg-base: add download caching of apt:// downloads Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 03/26] meta: dpkg-base: convert "exit" into "return 0" Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 04/26] base-apt: change the sources.list to also offer deb-src Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 05/26] base-apt: add deb-src packages as well Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 06/26] base-apt: do not skip gpg check when it is signed Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 07/26] ci: conf: add "hello" to the sample config and every build Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 08/26] meta: split all apt-get invocations into download and execution Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 09/26] meta: create DL_DIR support for all apt-get downloaded .debs Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 10/26] meta: import DL_DIR debs before apt-get download steps Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 11/26] meta: include DL_DIR deb import/export into bootstrap Vijai Kumar K
2020-01-28 15:41 ` [PATCH v3 12/26] base-apt: populate from DEBDIR as well Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 13/26] base-apt: drop the "apt_cache" feature Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 14/26] base-apt: do not copy debs directly out of rootfss anymore Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 15/26] base-apt: rework base-apt population Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 16/26] base-apt: move class "base-apt-helper" into only user Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 17/26] CI: include "isar-disable-apt-cache" into all CI images Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 18/26] CI: include "cowsay" into default build to test dpkg-gbp Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 19/26] CI: set BB_NO_NETWORK for cached rebuild Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 20/26] meta: repository: implement repo_contains_package and use it in base_apt Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 21/26] meta: repository: simplify the matching of packages Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 22/26] meta: deb-dl-dir: do not cache debs from isar-apt Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 23/26] base-apt: pull base-apt population to the front of the build chain Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 24/26] meta: base-apt: remove isar-apt check from population Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 25/26] meta: deb-dl-dir: only export newly downloaded files Vijai Kumar K
2020-01-28 15:42 ` [PATCH v3 26/26] meta/recipes-kernel: Export KBUILD_DEPENDS to cache Vijai Kumar K
2020-01-28 16:52 ` Henning Schild
2020-01-28 19:45 ` vijai kumar
2020-01-28 21:22 ` Henning Schild
2020-01-29 5:29 ` Vijai Kumar K
2020-01-31 9:27 ` Henning Schild
2019-12-16 11:49 ` [PATCHv2 10/25] meta: import DL_DIR debs before apt-get download steps Henning Schild
2019-12-16 11:49 ` [PATCHv2 11/25] meta: include DL_DIR deb import/export into bootstrap Henning Schild
2019-12-16 11:49 ` [PATCHv2 12/25] base-apt: populate from DEBDIR as well Henning Schild
2019-12-16 11:49 ` [PATCHv2 13/25] base-apt: drop the "apt_cache" feature Henning Schild
2019-12-16 11:50 ` [PATCHv2 14/25] base-apt: do not copy debs directly out of rootfss anymore Henning Schild
2019-12-16 11:50 ` [PATCHv2 15/25] base-apt: rework base-apt population Henning Schild
2019-12-16 11:50 ` [PATCHv2 16/25] base-apt: move class "base-apt-helper" into only user Henning Schild
2019-12-16 11:50 ` [PATCHv2 17/25] CI: include "isar-disable-apt-cache" into all CI images Henning Schild
2019-12-16 11:50 ` [PATCHv2 18/25] CI: include "cowsay" into default build to test dpkg-gbp Henning Schild
2019-12-16 11:50 ` [PATCHv2 19/25] CI: set BB_NO_NETWORK for cached rebuild Henning Schild
2019-12-16 11:50 ` [PATCHv2 20/25] meta: repository: implement repo_contains_package and use it in base_apt Henning Schild
2019-12-16 11:50 ` [PATCHv2 21/25] meta: repository: simplify the matching of packages Henning Schild
2019-12-16 11:50 ` [PATCHv2 22/25] meta: deb-dl-dir: do not cache debs from isar-apt Henning Schild
2019-12-16 11:50 ` [PATCHv2 23/25] base-apt: pull base-apt population to the front of the build chain Henning Schild
2020-01-21 8:55 ` vijai kumar
2020-01-21 9:54 ` Henning Schild
2020-01-21 13:01 ` vijai kumar
2020-01-27 7:32 ` Henning Schild
2019-12-16 11:50 ` [PATCHv2 24/25] meta: base-apt: remove isar-apt check from population Henning Schild
2019-12-16 11:50 ` [PATCHv2 25/25] meta: deb-dl-dir: only export newly downloaded files Henning Schild
2019-12-19 9:21 ` [PATCHv2 00/25] base-apt-rework Henning Schild
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200128154312.337e61f6@md1za8fc.ad001.siemens.net \
--to=henning.schild@siemens.com \
--cc=isar-users@googlegroups.com \
--cc=vijaikumar.kanagarajan@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox