From: Henning Schild <henning.schild@siemens.com>
To: <chombourger@gmail.com>
Cc: isar-users <isar-users@googlegroups.com>
Subject: Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs
Date: Tue, 29 Oct 2019 17:02:11 +0100 [thread overview]
Message-ID: <20191029170211.3d2d1272@md1za8fc.ad001.siemens.net> (raw)
In-Reply-To: <5c2908a6-97df-46dd-bb9b-21af77202918@googlegroups.com>
On Tue, 29 Oct 2019 05:48:01 -0700
<chombourger@gmail.com> wrote:
> On Monday, October 28, 2019 at 6:01:36 PM UTC+1, Henning Schild wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>>
> >
> > Hook in between the "apt-get --download-only"s and the "apt-get"s
> > and copy out all the debs for later.
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>>
> > ---
> > meta/classes/deb-dl-dir.bbclass | 14 ++++++++++++++
> > meta/classes/dpkg-base.bbclass | 1 +
> > meta/classes/dpkg-gbp.bbclass | 1 +
> > meta/classes/dpkg.bbclass | 1 +
> > meta/classes/image-locales-extension.bbclass | 2 +-
> > meta/classes/image-tools-extension.bbclass | 1 +
> > meta/classes/rootfs.bbclass | 12 ++++++++++--
> > meta/conf/bitbake.conf | 1 +
> > 8 files changed, 30 insertions(+), 3 deletions(-)
> > create mode 100644 meta/classes/deb-dl-dir.bbclass
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass
> > new file mode 100644
> > index 0000000..e58df7a
> > --- /dev/null
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -0,0 +1,14 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2019 Siemens AG
> > +#
> > +# SPDX-License-Identifier: MIT
> > +
> > +deb_dl_dir_export() {
> > + export pc="${DEBDIR}/${DISTRO}"
> > + export rootfs="${1}"
> > + mkdir -p "${pc}"
> > + flock "${pc}".lock -c ' \
> > + sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > -iname '*\.deb' \
> > + -exec cp -f '{}' "${pc}" \;
> >
>
> use -exec command {} + to reduce the number of fork/execvp pairs?
> note: you would need to use cp -t "${pc}" since the "+" modifier
> appends matches to the end of the command
Thanks, this was actually on my mind when i wrote it but eventually
overlooked.
I will look into it.
Henning
>
> > + '
> > +}
> > diff --git a/meta/classes/dpkg-base.bbclass
> > b/meta/classes/dpkg-base.bbclass
> > index b39df5c..8815478 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -8,6 +8,7 @@ inherit buildchroot
> > inherit debianize
> > inherit terminal
> > inherit repository
> > +inherit deb-dl-dir
> >
> > DEPENDS ?= ""
> >
> > diff --git a/meta/classes/dpkg-gbp.bbclass
> > b/meta/classes/dpkg-gbp.bbclass index d9c3af0..9e6945b 100644
> > --- a/meta/classes/dpkg-gbp.bbclass
> > +++ b/meta/classes/dpkg-gbp.bbclass
> > @@ -16,6 +16,7 @@ do_install_builddeps_append() {
> > sudo -E chroot ${BUILDCHROOT_DIR} \
> > apt-get install -y -o Debug::pkgProblemResolver=yes \
> > --no-install-recommends --download-only
> > ${GBP_DEPENDS}
> > + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> > sudo -E chroot ${BUILDCHROOT_DIR} \
> > apt-get install -y -o Debug::pkgProblemResolver=yes \
> > --no-install-recommends ${GBP_DEPENDS}
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > index d809cca..b6844e9 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -9,6 +9,7 @@ do_install_builddeps() {
> > E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> > ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> > + deb_dl_dir_export "${BUILDCHROOT_DIR}"
> > sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> > ${PP}/${PPS} ${DISTRO_ARCH}
> > dpkg_undo_mounts
> > diff --git a/meta/classes/image-locales-extension.bbclass
> > b/meta/classes/image-locales-extension.bbclass
> > index 3c0758f..0f0d0ca 100644
> > --- a/meta/classes/image-locales-extension.bbclass
> > +++ b/meta/classes/image-locales-extension.bbclass
> > @@ -25,7 +25,7 @@ def get_nopurge(d):
> > j.split()[0].split(".")[0],
> > j.split()[0]))))
> >
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN +=
> > "image_install_localepurge_download"
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT +=
> > "image_install_localepurge_download"
> > image_install_localepurge_download[weight] = "40"
> > image_install_localepurge_download() {
> > sudo -E chroot '${ROOTFSDIR}' \
> > diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass
> > index 243fa11..3906585 100644
> > --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -34,6 +34,7 @@ do_install_imager_deps() {
> > --allow-unauthenticated --allow-downgrades
> > --download-only install \
> > ${IMAGER_INSTALL}'
> >
> > + deb_dl_dir_export ${BUILDCHROOT_DIR}
> > sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> > apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \
> > --allow-unauthenticated --allow-downgrades install \
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 095ebc1..841aa89 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -1,6 +1,8 @@
> > # This software is a part of ISAR.
> > # Copyright (c) Siemens AG, 2019
> >
> > +inherit deb-dl-dir
> > +
> > ROOTFS_ARCH ?= "${DISTRO_ARCH}"
> > ROOTFS_DISTRO ?= "${DISTRO}"
> > ROOTFS_PACKAGES ?= ""
> > @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() {
> > /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only
> > ${ROOTFS_PACKAGES}
> > }
> >
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
> > -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
> > +ROOTFS_INSTALL_COMMAND +=
> > "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
> > +
> > +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
> > +rootfs_export_package_cache[weight] = "5"
> > +rootfs_export_package_cache() {
> > + deb_dl_dir_export ${ROOTFSDIR}
> > +}
> >
> > ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if
> > (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
> > rootfs_install_clean_files[weight] = "2"
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 4c0a809..56f4fbf 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
> > FILESPATH =
> > "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
> >
> > FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> > GITDIR = "${DL_DIR}/git"
> > +DEBDIR = "${DL_DIR}/deb"
> > DEBSRCDIR = "${DL_DIR}/deb-src"
> > P = "${PN}-${PV}"
> > PF = "${PN}-${PV}-${PR}"
> > --
> > 2.23.0
> >
> >
>
next prev parent reply other threads:[~2019-10-29 16:02 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-28 17:01 [PATCH 00/17] base-apt rework Henning Schild
2019-10-28 17:01 ` [PATCH 01/17] repository: new class to deal with repos Henning Schild
2019-11-24 15:29 ` Baurzhan Ismagulov
2019-11-25 12:03 ` Henning Schild
2019-10-28 17:01 ` [PATCH 02/17] dpkg-base: add download caching of apt:// downloads Henning Schild
2019-10-29 12:57 ` chombourger
2019-10-29 16:03 ` Henning Schild
2019-11-08 11:39 ` Henning Schild
2019-11-24 15:30 ` Baurzhan Ismagulov
2019-11-25 10:32 ` Henning Schild
2019-10-28 17:01 ` [PATCH 03/17] base-apt: change the sources.list to also offer deb-src Henning Schild
2019-10-28 17:01 ` [PATCH 04/17] base-apt: add deb-src packages as well Henning Schild
2019-10-30 19:23 ` Henning Schild
2019-10-28 17:01 ` [PATCH 05/17] base-apt: do not skip gpg check when it is signed Henning Schild
2019-11-24 15:35 ` Baurzhan Ismagulov
2019-11-25 10:30 ` Henning Schild
2019-10-28 17:01 ` [PATCH 06/17] ci: conf: add "hello" to the sample config and every build Henning Schild
2019-10-28 17:01 ` [PATCH 07/17] meta: split all apt-get invocations into download and execution Henning Schild
2019-11-24 15:41 ` Baurzhan Ismagulov
2019-11-25 10:20 ` Henning Schild
2019-10-28 17:01 ` [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs Henning Schild
2019-10-29 12:48 ` chombourger
2019-10-29 16:02 ` Henning Schild [this message]
2019-11-06 13:19 ` Henning Schild
2019-11-24 15:57 ` Baurzhan Ismagulov
2019-11-25 10:15 ` Henning Schild
2019-10-28 17:01 ` [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps Henning Schild
2019-10-30 19:21 ` Henning Schild
2019-11-06 13:17 ` Henning Schild
2019-11-24 20:20 ` Baurzhan Ismagulov
2019-11-25 10:13 ` Henning Schild
2019-10-28 17:01 ` [PATCH 10/17] base-apt: populate from DEBDIR as well Henning Schild
2019-10-28 17:01 ` [PATCH 11/17] base-apt: drop the "apt_cache" feature Henning Schild
2019-10-28 17:01 ` [PATCH 12/17] base-apt: do not copy debs directly out of rootfss anymore Henning Schild
2019-10-28 17:01 ` [PATCH 13/17] base-apt: rework base-apt population Henning Schild
2019-10-28 17:01 ` [PATCH 14/17] base-apt: move class "base-apt-helper" into only user Henning Schild
2019-10-28 17:01 ` [PATCH 15/17] CI: include "isar-disable-apt-cache" into all CI images Henning Schild
2019-10-28 17:01 ` [PATCH 16/17] CI: include "cowsay" into default build to test dpkg-gbp Henning Schild
2019-10-28 17:01 ` [PATCH 17/17] CI: set BB_NO_NETWORK for cached rebuild Henning Schild
2019-11-06 16:00 ` [PATCH 00/17] base-apt rework Jan Kiszka
2019-11-24 13:21 ` Baurzhan Ismagulov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191029170211.3d2d1272@md1za8fc.ad001.siemens.net \
--to=henning.schild@siemens.com \
--cc=chombourger@gmail.com \
--cc=isar-users@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox