* [PATCH v6 01/13] rootfs: Make rootfs finalize a separate task
2020-09-30 6:02 [PATCH v6 00/13] Deb-src caching Vijai Kumar K
@ 2020-09-30 6:02 ` Vijai Kumar K
2020-11-24 19:47 ` Baurzhan Ismagulov
2020-09-30 6:02 ` [PATCH v6 02/13] deb-dl-dir: Cache host distro debs separately Vijai Kumar K
` (3 subsequent siblings)
4 siblings, 1 reply; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:02 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
With the current implementation it is difficult to append a
postprocess function which requires a chroot environment.
For example, to add a postprocess function which runs apt-get to
download all source of packages installed in the target.
rootfs_postprocess_finalize is not actually an optional feature
but instead a necessary cleanup function for image class.
So, move the implementation to image class and make it as a task.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/image.bbclass | 41 ++++++++++++++++++++++++++++++++++++-
meta/classes/rootfs.bbclass | 39 -----------------------------------
2 files changed, 40 insertions(+), 40 deletions(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a296cc0..2391529 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}
ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
@@ -173,5 +173,44 @@ python do_deploy() {
}
addtask deploy before do_build after do_image
+do_rootfs_finalize() {
+ sudo -s <<'EOSUDO'
+ test -e "${ROOTFSDIR}/chroot-setup.sh" && \
+ "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
+ rm -f "${ROOTFSDIR}/chroot-setup.sh"
+
+ test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
+ find "${ROOTFSDIR}/usr/bin" \
+ -maxdepth 1 -name 'qemu-*-static' -type f -delete
+
+ mountpoint -q '${ROOTFSDIR}/isar-apt' && \
+ umount -l ${ROOTFSDIR}/isar-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
+
+ mountpoint -q '${ROOTFSDIR}/base-apt' && \
+ umount -l ${ROOTFSDIR}/base-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
+
+ mountpoint -q '${ROOTFSDIR}/dev' && \
+ umount -l ${ROOTFSDIR}/dev
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/proc
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/sys
+
+ rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
+ rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+
+ mv "${ROOTFSDIR}/etc/apt/sources-list" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources-list"
+EOSUDO
+}
+addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
+
# Last so that the image type can overwrite tasks if needed
inherit ${IMAGE_TYPE}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cb..eae14d5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -201,45 +201,6 @@ rootfs_generate_manifest () {
${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
}
-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
-rootfs_postprocess_finalize() {
- sudo -s <<'EOSUDO'
- test -e "${ROOTFSDIR}/chroot-setup.sh" && \
- "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
- rm -f "${ROOTFSDIR}/chroot-setup.sh"
-
- test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
- find "${ROOTFSDIR}/usr/bin" \
- -maxdepth 1 -name 'qemu-*-static' -type f -delete
-
- mountpoint -q '${ROOTFSDIR}/isar-apt' && \
- umount -l ${ROOTFSDIR}/isar-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
-
- mountpoint -q '${ROOTFSDIR}/base-apt' && \
- umount -l ${ROOTFSDIR}/base-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
-
- mountpoint -q '${ROOTFSDIR}/dev' && \
- umount -l ${ROOTFSDIR}/dev
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/proc
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/sys
-
- rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
- rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
- mv "${ROOTFSDIR}/etc/apt/sources-list" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources-list"
-EOSUDO
-}
-
do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
python do_rootfs_postprocess() {
# Take care that its correctly mounted:
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 01/13] rootfs: Make rootfs finalize a separate task
2020-09-30 6:02 ` [PATCH v6 01/13] rootfs: Make rootfs finalize a separate task Vijai Kumar K
@ 2020-11-24 19:47 ` Baurzhan Ismagulov
0 siblings, 0 replies; 31+ messages in thread
From: Baurzhan Ismagulov @ 2020-11-24 19:47 UTC (permalink / raw)
To: isar-users
On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote:
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index afec1cb..eae14d5 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -201,45 +201,6 @@ rootfs_generate_manifest () {
> ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> }
>
> -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
Suggest also removing the 'finalize-rootfs' comment before ROOTFS_FEATURES ?=
"".
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH v6 02/13] deb-dl-dir: Cache host distro debs separately
2020-09-30 6:02 [PATCH v6 00/13] Deb-src caching Vijai Kumar K
2020-09-30 6:02 ` [PATCH v6 01/13] rootfs: Make rootfs finalize a separate task Vijai Kumar K
@ 2020-09-30 6:02 ` Vijai Kumar K
2020-09-30 6:02 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Vijai Kumar K
` (2 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:02 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
we were still caching the debs from both the distros into the same
DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be
cached in the relevant subdirectory and avoids mixing of debs from
two separate distros.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 4 ++--
meta/classes/dpkg-gbp.bbclass | 8 ++++++--
meta/classes/dpkg.bbclass | 8 ++++++--
meta/classes/image-tools-extension.bbclass | 4 ++--
meta/classes/rootfs.bbclass | 4 ++--
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++--
6 files changed, 28 insertions(+), 12 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e996703..29a3d67 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -6,7 +6,7 @@
inherit repository
deb_dl_dir_import() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
[ ! -d "${pc}" ] && return 0
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -20,7 +20,7 @@ deb_dl_dir_import() {
}
deb_dl_dir_export() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
mkdir -p "${pc}"
flock "${pc}".lock -c '
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index afa1e19..ba5c3eb 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
do_install_builddeps_append() {
dpkg_do_mounts
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a24386d..4e7c2f7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,10 +9,14 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
do_install_builddeps() {
dpkg_do_mounts
E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH} --download-only
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH}
dpkg_undo_mounts
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 6590ee7..0b067ff 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,7 +25,7 @@ do_install_imager_deps() {
buildchroot_do_mounts
E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import ${BUILDCHROOT_DIR}
+ deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get update \
-o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
@@ -35,7 +35,7 @@ do_install_imager_deps() {
--allow-unauthenticated --allow-downgrades --download-only install \
${IMAGER_INSTALL}'
- deb_dl_dir_export ${BUILDCHROOT_DIR}
+ deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
--allow-unauthenticated --allow-downgrades install \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index eae14d5..6316321 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -115,7 +115,7 @@ rootfs_install_resolvconf() {
ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
rootfs_import_package_cache[weight] = "5"
rootfs_import_package_cache() {
- deb_dl_dir_import ${ROOTFSDIR}
+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
}
ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
@@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
rootfs_export_package_cache[weight] = "5"
rootfs_export_package_cache() {
- deb_dl_dir_export ${ROOTFSDIR}
+ deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO}
}
ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index fbfe669..3b19914 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -255,7 +255,11 @@ isar_bootstrap() {
export IS_HOST debootstrap_args E
if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
sudo rm -rf --one-file-system "${ROOTFSDIR}"
- deb_dl_dir_import "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+ fi
sudo -E -s <<'EOSUDO'
set -e
@@ -356,7 +360,11 @@ isar_bootstrap() {
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
fi
- deb_dl_dir_export "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
+ fi
}
CLEANFUNCS = "clean_deploy"
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing
2020-09-30 6:02 [PATCH v6 00/13] Deb-src caching Vijai Kumar K
2020-09-30 6:02 ` [PATCH v6 01/13] rootfs: Make rootfs finalize a separate task Vijai Kumar K
2020-09-30 6:02 ` [PATCH v6 02/13] deb-dl-dir: Cache host distro debs separately Vijai Kumar K
@ 2020-09-30 6:02 ` Vijai Kumar K
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
2020-11-24 19:49 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Baurzhan Ismagulov
2020-11-23 15:08 ` [PATCH v6 00/13] Deb-src caching vijaikumar....@gmail.com
2020-11-24 19:45 ` Baurzhan Ismagulov
4 siblings, 2 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:02 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Collect the deb sources of the corresponding deb binaries cached
in DEBDIR as part of image postprocess.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 27 +++++++++++++++++++++++++++
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 8 ++++++++
3 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 29a3d67..659fe4b 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,33 @@
inherit repository
+debsrc_download() {
+ export rootfs="$1"
+ export rootfs_distro="$2"
+ mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+ sudo -E -s <<'EOSUDO'
+ mkdir -p "${rootfs}/deb-src"
+ mountpoint -q "${rootfs}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
+EOSUDO
+ ( flock 9
+ set -e
+ printenv | grep -q BB_VERBOSE_LOGS && set -x
+ find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ done
+ ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
+ sudo -E -s <<'EOSUDO'
+ mountpoint -q "${rootfs}/deb-src" && \
+ umount -l "${rootfs}/deb-src"
+ rm -rf "${rootfs}/deb-src"
+EOSUDO
+}
+
deb_dl_dir_import() {
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2391529..6ca5759 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}
ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 6316321..cfabeae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,6 +185,14 @@ python do_rootfs_install() {
}
addtask rootfs_install before do_rootfs_postprocess after do_unpack
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
+cache_deb_src() {
+ rootfs_install_resolvconf
+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
+ debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
+ rootfs_install_clean_files
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
rootfs_postprocess_clean_package_cache() {
sudo -E chroot '${ROOTFSDIR}' \
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster
2020-09-30 6:02 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Vijai Kumar K
@ 2020-09-30 6:06 ` Vijai Kumar K
2020-09-30 6:06 ` [PATCH v6 05/13] deb-dl-dir: Download files only belonging to the current image Vijai Kumar K
` (3 more replies)
2020-11-24 19:49 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Baurzhan Ismagulov
1 sibling, 4 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:06 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Eventhough apt-get source skips redownloading of files, it is still
slow and takes a lot of time. Instead, lookup if the dsc file is already
present in the cache and skip based on it.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 659fe4b..79a9a88 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -20,6 +20,8 @@ EOSUDO
find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
+ [ -n "$dscfile" ] && continue
sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 05/13] deb-dl-dir: Download files only belonging to the current image
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
@ 2020-09-30 6:06 ` Vijai Kumar K
2020-09-30 6:07 ` [PATCH v6 06/13] deb-dl-dir: Factor out the mounting part Vijai Kumar K
` (2 subsequent siblings)
3 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:06 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Avoid downloading deb-srcs for debs cached from other image builds.
One way to ensure that is to see if the package is present in the dpkg
status file.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 79a9a88..ce691cf 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,15 @@
inherit repository
+is_not_part_of_current_build() {
+ local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
+ local output="$( grep -hs "^Package: ${package}" \
+ "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ [ -z "${output}" ]
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
@@ -18,6 +27,7 @@ EOSUDO
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+ is_not_part_of_current_build "${package}" && continue
local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 06/13] deb-dl-dir: Factor out the mounting part
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
2020-09-30 6:06 ` [PATCH v6 05/13] deb-dl-dir: Download files only belonging to the current image Vijai Kumar K
@ 2020-09-30 6:07 ` Vijai Kumar K
2020-09-30 6:07 ` [PATCH v6 07/13] deb-dl-dir: Fix skipping of removed files Vijai Kumar K
2020-09-30 6:07 ` [PATCH v6 08/13] repository: Add a sanity test to check missing sources Vijai Kumar K
3 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:07 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Factor out the mount and unmount section to separate function for
easy readability. No functional change intended.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 32 ++++++++++++++++++++++----------
1 file changed, 22 insertions(+), 10 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ce691cf..b2d94e7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,15 +14,30 @@ is_not_part_of_current_build() {
[ -z "${output}" ]
}
+debsrc_do_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+EOSUDO
+}
+
+debsrc_undo_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" && \
+ umount -l "${1}/deb-src"
+ rm -rf "${1}/deb-src"
+EOSUDO
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
- sudo -E -s <<'EOSUDO'
- mkdir -p "${rootfs}/deb-src"
- mountpoint -q "${rootfs}/deb-src" || \
- mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_do_mounts "${rootfs}"
+
( flock 9
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -37,11 +52,8 @@ EOSUDO
sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
done
) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
- sudo -E -s <<'EOSUDO'
- mountpoint -q "${rootfs}/deb-src" && \
- umount -l "${rootfs}/deb-src"
- rm -rf "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_undo_mounts "${rootfs}"
}
deb_dl_dir_import() {
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 07/13] deb-dl-dir: Fix skipping of removed files
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
2020-09-30 6:06 ` [PATCH v6 05/13] deb-dl-dir: Download files only belonging to the current image Vijai Kumar K
2020-09-30 6:07 ` [PATCH v6 06/13] deb-dl-dir: Factor out the mounting part Vijai Kumar K
@ 2020-09-30 6:07 ` Vijai Kumar K
2020-09-30 6:07 ` [PATCH v6 08/13] repository: Add a sanity test to check missing sources Vijai Kumar K
3 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:07 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Some packages are installed and then removed, like localepurge from
image-locales-extension.bbclass. Those information would not be
available in dpkg status file. Use dpkg log instead to see if the
package has been used on the target.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b2d94e7..a94fb10 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -7,10 +7,16 @@ inherit repository
is_not_part_of_current_build() {
local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
- local output="$( grep -hs "^Package: ${package}" \
- "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
- "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
- "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
+ local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
+ # Since we are parsing all the debs in DEBDIR, we can to some extend
+ # try to eliminate some debs that are not part of the current multiconfig
+ # build using the below method.
+ local output="$( grep -hs "status installed ${package}:${arch} ${version}" \
+ "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+ "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
+ "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
+
[ -z "${output}" ]
}
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 08/13] repository: Add a sanity test to check missing sources
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
` (2 preceding siblings ...)
2020-09-30 6:07 ` [PATCH v6 07/13] deb-dl-dir: Fix skipping of removed files Vijai Kumar K
@ 2020-09-30 6:07 ` Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 09/13] base-apt: Introduce BASE_REPO_FEATURES Vijai Kumar K
3 siblings, 1 reply; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:07 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Add a sanity test routine to test if sources for all packages
are present in the repo.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/repository.bbclass | 10 ++++++++++
meta/recipes-devtools/base-apt/base-apt.bb | 2 ++
2 files changed, 12 insertions(+)
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index c70138f..7b6e47b 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -96,3 +96,13 @@ repo_contains_package() {
# no
return 2
}
+
+repo_sanity_test() {
+ local dir="$1"
+ local dbdir="$2"
+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then
+ bbwarn "One or more sources are missing in base-apt"
+ bbwarn "${output}"
+ fi
+}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index da3e32e..8940ae8 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -66,6 +66,8 @@ repo() {
fi
populate_base_apt
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
}
python do_cache() {
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 09/13] base-apt: Introduce BASE_REPO_FEATURES
2020-09-30 6:07 ` [PATCH v6 08/13] repository: Add a sanity test to check missing sources Vijai Kumar K
@ 2020-09-30 6:18 ` Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 10/13] repository: Fix failures due to missing section Vijai Kumar K
` (2 more replies)
0 siblings, 3 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:18 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Eventhough we are collecting the debsrcs as part of postprocess,
it could not be considered a ROOTFS_FEATURE, instead a base-apt one.
Introduce BASE_REPO_FEATURES, to provide user with control to enable
or disable cache-deb-src. Disabled by default, since it is not required
for normal offline build to work.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 4 +++-
meta/classes/repository.bbclass | 9 +++++----
meta/classes/rootfs.bbclass | 1 -
meta/recipes-devtools/base-apt/base-apt.bb | 1 +
5 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 07a7781..ec0a384 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -181,6 +181,10 @@ ISAR_CROSS_COMPILE ?= "0"
# NOTE: this works for amd64 and arm64 targets so far
#ISAR_ENABLE_COMPAT_ARCH ?= "1"
+# Uncomment this to enable caching of all source packages.
+# Without this feature, only sources of packages downloaded with apt:// are downloaded.
+#BASE_REPO_FEATURES ?= "cache-deb-src"
+
#
# Uncomment this to enable use of cached base repository
#ISAR_USE_CACHED_BASE_REPO ?= "1"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6ca5759..5c2c88c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,10 +63,12 @@ image_do_mounts() {
}
ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)} "
+
inherit rootfs
inherit image-sdk-extension
inherit image-tools-extension
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 7b6e47b..1f475dc 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -100,9 +100,10 @@ repo_contains_package() {
repo_sanity_test() {
local dir="$1"
local dbdir="$2"
- local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
- if [ -n "${output}" ]; then
- bbwarn "One or more sources are missing in base-apt"
- bbwarn "${output}"
+ if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ];then
+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then
+ bbfatal "One or more sources are missing in repo. ${output}"
+ fi
fi
}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cfabeae..0aa5502 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,7 +185,6 @@ python do_rootfs_install() {
}
addtask rootfs_install before do_rootfs_postprocess after do_unpack
-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
cache_deb_src() {
rootfs_install_resolvconf
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 8940ae8..506a28f 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -10,6 +10,7 @@ SRC_URI = "file://distributions.in"
BASE_REPO_KEY ?= ""
KEYFILES ?= ""
+BASE_REPO_FEATURES ?= ""
populate_base_apt() {
find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 10/13] repository: Fix failures due to missing section
2020-09-30 6:18 ` [PATCH v6 09/13] base-apt: Introduce BASE_REPO_FEATURES Vijai Kumar K
@ 2020-09-30 6:18 ` Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 11/13] scripts/ci_build.sh: Enable deb-src caching Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 12/13] rootfs: Fix possible overwrite of existing resolv.conf Vijai Kumar K
2 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:18 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
reprepro's includedsc failed for certain packages like makedev while
complaining about missing Section information. This information is
optional according to Debian[1]. Use a dummy value to avoid such
failures.
[1] https://wiki.debian.org/DebianRepository/Format
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/repository.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 1f475dc..ac395db 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -42,7 +42,7 @@ repo_add_srcpackage() {
if [ -n "${GNUPGHOME}" ]; then
export GNUPGHOME="${GNUPGHOME}"
fi
- reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
+ reprepro -b "${dir}" --dbdir "${dbdir}" -C main -S - -P source \
includedsc "${codename}" \
"$@"
}
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 11/13] scripts/ci_build.sh: Enable deb-src caching
2020-09-30 6:18 ` [PATCH v6 09/13] base-apt: Introduce BASE_REPO_FEATURES Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 10/13] repository: Fix failures due to missing section Vijai Kumar K
@ 2020-09-30 6:18 ` Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 12/13] rootfs: Fix possible overwrite of existing resolv.conf Vijai Kumar K
2 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:18 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
Enable debsrc caching for CI builds.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
scripts/ci_build.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a5eee82..09a0b4b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -184,6 +184,7 @@ if [ -n "$REPRO_BUILD" ]; then
sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
fi
+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
# Start cross build for the defined set of configurations
sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
bitbake $BB_ARGS $CROSS_TARGETS_SET
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 12/13] rootfs: Fix possible overwrite of existing resolv.conf
2020-09-30 6:18 ` [PATCH v6 09/13] base-apt: Introduce BASE_REPO_FEATURES Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 10/13] repository: Fix failures due to missing section Vijai Kumar K
2020-09-30 6:18 ` [PATCH v6 11/13] scripts/ci_build.sh: Enable deb-src caching Vijai Kumar K
@ 2020-09-30 6:18 ` Vijai Kumar K
2020-09-30 6:21 ` [PATCH v6 13/13] rootfs: Handle failures when postprocess is rerun Vijai Kumar K
2 siblings, 1 reply; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:18 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
There is a possiblilty that one of the packages installed in the
rootfs provides /etc/resolv.conf and we might accidentally remove
it. Fix it by taking a backup of any existing resolv conf files
and restoring it later. This is needed since we could not effectively
move caching before rootfs_install_clean_files since we need the
latest dpkg log.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/rootfs.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0aa5502..ed43fb9 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -186,10 +186,18 @@ python do_rootfs_install() {
addtask rootfs_install before do_rootfs_postprocess after do_unpack
cache_deb_src() {
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
+ fi
rootfs_install_resolvconf
+
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
- rootfs_install_clean_files
+
+ sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf.isar "${ROOTFSDIR}"/etc/resolv.conf
+ fi
}
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread* [PATCH v6 13/13] rootfs: Handle failures when postprocess is rerun
2020-09-30 6:18 ` [PATCH v6 12/13] rootfs: Fix possible overwrite of existing resolv.conf Vijai Kumar K
@ 2020-09-30 6:21 ` Vijai Kumar K
0 siblings, 0 replies; 31+ messages in thread
From: Vijai Kumar K @ 2020-09-30 6:21 UTC (permalink / raw)
To: isar-users; +Cc: Vijai Kumar K
The apt state information in var/lib/apt/lists is cleared as
part of postprocessing. This makes apt-get calls in cache_deb_src
fail when rerunning the postprocess task.
Since we cannot run apt-get update again to refresh the state
information, copy the apt state information from the initial
bootstrapped image.
Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
meta/classes/rootfs.bbclass | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ed43fb9..bebc0c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -190,6 +190,10 @@ cache_deb_src() {
sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
fi
rootfs_install_resolvconf
+ # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
+ # relies on that through out the build. Copy that state information instead of apt-get update
+ # which generates a new state from upstream.
+ sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
--
2.17.1
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing
2020-09-30 6:02 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Vijai Kumar K
2020-09-30 6:06 ` [PATCH v6 04/13] deb-dl-dir: Make debsrc_download faster Vijai Kumar K
@ 2020-11-24 19:49 ` Baurzhan Ismagulov
2020-11-26 10:52 ` vijaikumar....@gmail.com
1 sibling, 1 reply; 31+ messages in thread
From: Baurzhan Ismagulov @ 2020-11-24 19:49 UTC (permalink / raw)
To: isar-users
On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 2391529..6ca5759 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -63,7 +63,7 @@ image_do_mounts() {
> }
>
> ROOTFSDIR = "${IMAGE_ROOTFS}"
> -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
Should it be "cache-deb-src"? The same for p9.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing
2020-11-24 19:49 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Baurzhan Ismagulov
@ 2020-11-26 10:52 ` vijaikumar....@gmail.com
2020-11-26 11:03 ` Baurzhan Ismagulov
0 siblings, 1 reply; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-26 10:52 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 828 bytes --]
On Wednesday, November 25, 2020 at 1:19:36 AM UTC+5:30 i...@radix50.net
wrote:
> On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> > index 2391529..6ca5759 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -63,7 +63,7 @@ image_do_mounts() {
> > }
> >
> > ROOTFSDIR = "${IMAGE_ROOTFS}"
> > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
>
> Should it be "cache-deb-src"? The same for p9.
>
Yes. It is cache-deb-src with hypens(-). The original function is with
underscore.
If you see a better name let me know. We can change it.
Thanks,
Vijai Kumar K
>
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 1324 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing
2020-11-26 10:52 ` vijaikumar....@gmail.com
@ 2020-11-26 11:03 ` Baurzhan Ismagulov
2020-11-26 11:07 ` vijaikumar....@gmail.com
0 siblings, 1 reply; 31+ messages in thread
From: Baurzhan Ismagulov @ 2020-11-26 11:03 UTC (permalink / raw)
To: isar-users
On Thu, Nov 26, 2020 at 02:52:22AM -0800, vijaikumar....@gmail.com wrote:
> > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
> >
> > Should it be "cache-deb-src"? The same for p9.
>
> Yes. It is cache-deb-src with hypens(-). The original function is with
> underscore.
>
> If you see a better name let me know. We can change it.
Sorry for the confusion, I meant that ROOTFS_FEATURES mentions cach-deb-src,
"cach" without "e".
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing
2020-11-26 11:03 ` Baurzhan Ismagulov
@ 2020-11-26 11:07 ` vijaikumar....@gmail.com
0 siblings, 0 replies; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-26 11:07 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 759 bytes --]
On Thursday, November 26, 2020 at 4:33:13 PM UTC+5:30 i...@radix50.net
wrote:
> On Thu, Nov 26, 2020 at 02:52:22AM -0800, vijaikumar....@gmail.com wrote:
> > > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest
> cach-deb-src"
> > >
> > > Should it be "cache-deb-src"? The same for p9.
> >
> > Yes. It is cache-deb-src with hypens(-). The original function is with
> > underscore.
> >
> > If you see a better name let me know. We can change it.
>
> Sorry for the confusion, I meant that ROOTFS_FEATURES mentions
> cach-deb-src,
> "cach" without "e".
>
Aha. Indeed. Good catch. will fix that in v9.
Thanks,
Vijai Kumar
>
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 1299 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-09-30 6:02 [PATCH v6 00/13] Deb-src caching Vijai Kumar K
` (2 preceding siblings ...)
2020-09-30 6:02 ` [PATCH v6 03/13] meta: cache deb srcs as part of postprocessing Vijai Kumar K
@ 2020-11-23 15:08 ` vijaikumar....@gmail.com
2020-11-23 15:38 ` Jan Kiszka
2020-11-24 19:45 ` Baurzhan Ismagulov
4 siblings, 1 reply; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-23 15:08 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 2554 bytes --]
Any review comment on this series?
Thanks,
Vijai Kumar K
On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30
vijaikumar_...@mentor.com wrote:
> Changes in v6:
> - Rebase against latest next
> - Address review comments from Henning
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
>
> Changes in v5:
> The major changes in this series are as below,
> - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
> of debs from different distro which inturn helps the src caching logic.
> - (P5) Avoid downloading package from other builds.
> - (P8) Add a reprepro based sanity test to check if the repo contains
> the sources for all the debs.
> - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
> to enable or disable various base-apt features. (cache-deb-src for now)
> - Some fixes in (P12 & P13)
> Also, addressed some review comments from Henning.
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5
>
> Changes in v4:
> - Use <source package>=<version> format instead of just using <packagename>
> to download the right version of source package.
>
> Changes in v3:
> - Take care of non-existent downloads/deb-src directory.
>
> Changes in v2:
> - Introduced additional patch to cache deb src
> - Rebased on top of henning/staging4 tree
>
> Vijai Kumar K (13):
> rootfs: Make rootfs finalize a separate task
> deb-dl-dir: Cache host distro debs separately
> meta: cache deb srcs as part of postprocessing
> deb-dl-dir: Make debsrc_download faster
> deb-dl-dir: Download files only belonging to the current image
> deb-dl-dir: Factor out the mounting part
> deb-dl-dir: Fix skipping of removed files
> repository: Add a sanity test to check missing sources
> base-apt: Introduce BASE_REPO_FEATURES
> repository: Fix failures due to missing section
> scripts/ci_build.sh: Enable deb-src caching
> rootfs: Fix possible overwrite of existing resolv.conf
> rootfs: Handle failures when postprocess is rerun
>
> meta-isar/conf/local.conf.sample | 4 ++
> meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++-
> meta/classes/dpkg-gbp.bbclass | 8 ++-
> meta/classes/dpkg.bbclass | 8 ++-
> meta/classes/image-tools-extension.bbclass | 4 +-
> meta/classes/image.bbclass | 43 ++++++++++++-
> meta/classes/repository.bbclass | 13 +++-
> meta/classes/rootfs.bbclass | 62 +++++++------------
> .../isar-bootstrap/isar-bootstrap.inc | 12 +++-
> meta/recipes-devtools/base-apt/base-apt.bb | 3 +
> scripts/ci_build.sh | 1 +
> 11 files changed, 166 insertions(+), 53 deletions(-)
>
> --
> 2.17.1
>
>
[-- Attachment #1.2: Type: text/html, Size: 3931 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 00/13] Deb-src caching
2020-11-23 15:08 ` [PATCH v6 00/13] Deb-src caching vijaikumar....@gmail.com
@ 2020-11-23 15:38 ` Jan Kiszka
2020-11-23 15:47 ` vijaikumar....@gmail.com
0 siblings, 1 reply; 31+ messages in thread
From: Jan Kiszka @ 2020-11-23 15:38 UTC (permalink / raw)
To: vijaikumar....@gmail.com, isar-users
On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
>
> Any review comment on this series?
>
Do we have this anchored in CI via some tests already?
Jan
> Thanks,
> Vijai Kumar K
>
> On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30
> vijaikumar_...@mentor.com wrote:
>
> Changes in v6:
> - Rebase against latest next
> - Address review comments from Henning
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
> <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3>
>
> Changes in v5:
> The major changes in this series are as below,
> - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
> of debs from different distro which inturn helps the src caching logic.
> - (P5) Avoid downloading package from other builds.
> - (P8) Add a reprepro based sanity test to check if the repo contains
> the sources for all the debs.
> - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
> to enable or disable various base-apt features. (cache-deb-src for now)
> - Some fixes in (P12 & P13)
> Also, addressed some review comments from Henning.
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5
> <https://github.com/vj-kumar/isar/tree/vijai/debsrc5>
>
> Changes in v4:
> - Use <source package>=<version> format instead of just using
> <packagename>
> to download the right version of source package.
>
> Changes in v3:
> - Take care of non-existent downloads/deb-src directory.
>
> Changes in v2:
> - Introduced additional patch to cache deb src
> - Rebased on top of henning/staging4 tree
>
> Vijai Kumar K (13):
> rootfs: Make rootfs finalize a separate task
> deb-dl-dir: Cache host distro debs separately
> meta: cache deb srcs as part of postprocessing
> deb-dl-dir: Make debsrc_download faster
> deb-dl-dir: Download files only belonging to the current image
> deb-dl-dir: Factor out the mounting part
> deb-dl-dir: Fix skipping of removed files
> repository: Add a sanity test to check missing sources
> base-apt: Introduce BASE_REPO_FEATURES
> repository: Fix failures due to missing section
> scripts/ci_build.sh: Enable deb-src caching
> rootfs: Fix possible overwrite of existing resolv.conf
> rootfs: Handle failures when postprocess is rerun
>
> meta-isar/conf/local.conf.sample | 4 ++
> meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++-
> meta/classes/dpkg-gbp.bbclass | 8 ++-
> meta/classes/dpkg.bbclass | 8 ++-
> meta/classes/image-tools-extension.bbclass | 4 +-
> meta/classes/image.bbclass | 43 ++++++++++++-
> meta/classes/repository.bbclass | 13 +++-
> meta/classes/rootfs.bbclass | 62 +++++++------------
> .../isar-bootstrap/isar-bootstrap.inc | 12 +++-
> meta/recipes-devtools/base-apt/base-apt.bb <http://base-apt.bb> | 3 +
> scripts/ci_build.sh | 1 +
> 11 files changed, 166 insertions(+), 53 deletions(-)
>
> --
> 2.17.1
>
> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+unsubscribe@googlegroups.com
> <mailto:isar-users+unsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com
> <https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-23 15:38 ` Jan Kiszka
@ 2020-11-23 15:47 ` vijaikumar....@gmail.com
2020-11-24 5:41 ` Jan Kiszka
0 siblings, 1 reply; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-23 15:47 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 3929 bytes --]
On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:
> On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
> >
> > Any review comment on this series?
> >
>
> Do we have this anchored in CI via some tests already?
>
Hi Jan,
Yes. We have it enabled in CI[1].
[1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ
Thanks,
Vijai Kumar K
> Jan
>
> > Thanks,
> > Vijai Kumar K
> >
> > On Wednesday, September 30, 2020 at 11:33:54 AM UTC+5:30
> > vijaikumar_...@mentor.com wrote:
> >
> > Changes in v6:
> > - Rebase against latest next
> > - Address review comments from Henning
> > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
> > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3>
> >
> > Changes in v5:
> > The major changes in this series are as below,
> > - (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
> > of debs from different distro which inturn helps the src caching logic.
> > - (P5) Avoid downloading package from other builds.
> > - (P8) Add a reprepro based sanity test to check if the repo contains
> > the sources for all the debs.
> > - (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
> > to enable or disable various base-apt features. (cache-deb-src for now)
> > - Some fixes in (P12 & P13)
> > Also, addressed some review comments from Henning.
> > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5
> > <https://github.com/vj-kumar/isar/tree/vijai/debsrc5>
> >
> > Changes in v4:
> > - Use <source package>=<version> format instead of just using
> > <packagename>
> > to download the right version of source package.
> >
> > Changes in v3:
> > - Take care of non-existent downloads/deb-src directory.
> >
> > Changes in v2:
> > - Introduced additional patch to cache deb src
> > - Rebased on top of henning/staging4 tree
> >
> > Vijai Kumar K (13):
> > rootfs: Make rootfs finalize a separate task
> > deb-dl-dir: Cache host distro debs separately
> > meta: cache deb srcs as part of postprocessing
> > deb-dl-dir: Make debsrc_download faster
> > deb-dl-dir: Download files only belonging to the current image
> > deb-dl-dir: Factor out the mounting part
> > deb-dl-dir: Fix skipping of removed files
> > repository: Add a sanity test to check missing sources
> > base-apt: Introduce BASE_REPO_FEATURES
> > repository: Fix failures due to missing section
> > scripts/ci_build.sh: Enable deb-src caching
> > rootfs: Fix possible overwrite of existing resolv.conf
> > rootfs: Handle failures when postprocess is rerun
> >
> > meta-isar/conf/local.conf.sample | 4 ++
> > meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++-
> > meta/classes/dpkg-gbp.bbclass | 8 ++-
> > meta/classes/dpkg.bbclass | 8 ++-
> > meta/classes/image-tools-extension.bbclass | 4 +-
> > meta/classes/image.bbclass | 43 ++++++++++++-
> > meta/classes/repository.bbclass | 13 +++-
> > meta/classes/rootfs.bbclass | 62 +++++++------------
> > .../isar-bootstrap/isar-bootstrap.inc | 12 +++-
> > meta/recipes-devtools/base-apt/base-apt.bb <http://base-apt.bb> | 3 +
> > scripts/ci_build.sh | 1 +
> > 11 files changed, 166 insertions(+), 53 deletions(-)
> >
> > --
> > 2.17.1
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to isar-users+...@googlegroups.com
> > <mailto:isar-users+...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com
> > <
> https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
>
[-- Attachment #1.2: Type: text/html, Size: 7760 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-23 15:47 ` vijaikumar....@gmail.com
@ 2020-11-24 5:41 ` Jan Kiszka
2020-11-24 6:30 ` vijaikumar....@gmail.com
0 siblings, 1 reply; 31+ messages in thread
From: Jan Kiszka @ 2020-11-24 5:41 UTC (permalink / raw)
To: vijaikumar....@gmail.com, isar-users
On 23.11.20 16:47, vijaikumar....@gmail.com wrote:
>
>
> On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:
>
> On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
> >
> > Any review comment on this series?
> >
>
> Do we have this anchored in CI via some tests already?
>
>
> Hi Jan,
>
> Yes. We have it enabled in CI[1].
>
> [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ
>
OK, it's executed, and my one runs in our nightly builds didn't fail.
But is caching effectively checked this way?
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-24 5:41 ` Jan Kiszka
@ 2020-11-24 6:30 ` vijaikumar....@gmail.com
0 siblings, 0 replies; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-24 6:30 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 986 bytes --]
On Tuesday, November 24, 2020 at 11:11:52 AM UTC+5:30 Jan Kiszka wrote:
> On 23.11.20 16:47, vijaikumar....@gmail.com wrote:
> >
> >
> > On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:
> >
> > On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
> > >
> > > Any review comment on this series?
> > >
> >
> > Do we have this anchored in CI via some tests already?
> >
> >
> > Hi Jan,
> >
> > Yes. We have it enabled in CI[1].
> >
> > [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ
> >
>
> OK, it's executed, and my one runs in our nightly builds didn't fail.
> But is caching effectively checked this way?
>
Yes. There is a sanity test[1] after base-apt creation which doesnot throw
a warning. It should when some sources
are missing.
[1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/LeacMMNhBAAJ
Thanks,
Vijai Kumar K
Jan
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
>
[-- Attachment #1.2: Type: text/html, Size: 1991 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-09-30 6:02 [PATCH v6 00/13] Deb-src caching Vijai Kumar K
` (3 preceding siblings ...)
2020-11-23 15:08 ` [PATCH v6 00/13] Deb-src caching vijaikumar....@gmail.com
@ 2020-11-24 19:45 ` Baurzhan Ismagulov
2020-11-25 5:51 ` Jan Kiszka
2020-11-25 6:04 ` vijaikumar....@gmail.com
4 siblings, 2 replies; 31+ messages in thread
From: Baurzhan Ismagulov @ 2020-11-24 19:45 UTC (permalink / raw)
To: isar-users
Hello Vijai Kumar,
On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
> Changes in v6:
> - Rebase against latest next
> - Address review comments from Henning
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
The code looks good to me. There is a couple of small issues, I'll reply to the
respective patches.
How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see
downloads/deb-src/debian-buster, which contains only hello. Is this intended?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 00/13] Deb-src caching
2020-11-24 19:45 ` Baurzhan Ismagulov
@ 2020-11-25 5:51 ` Jan Kiszka
2020-11-25 6:04 ` vijaikumar....@gmail.com
1 sibling, 0 replies; 31+ messages in thread
From: Jan Kiszka @ 2020-11-25 5:51 UTC (permalink / raw)
To: isar-users
On 24.11.20 20:45, Baurzhan Ismagulov wrote:
> Hello Vijai Kumar,
>
> On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
>> Changes in v6:
>> - Rebase against latest next
>> - Address review comments from Henning
>> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
>
> The code looks good to me. There is a couple of small issues, I'll reply to the
> respective patches.
>
> How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see
> downloads/deb-src/debian-buster, which contains only hello. Is this intended?
I would expect that everything a build pulled from deb-src goes into the
cache. IIRC, only hello is rebuilt from sources in these runs.
I wonder, though, and that might be a topic for some user manual
extension, how this series can also be used to fetch all sources
corresponding to all deployed binaries (compliance use case). Or does
this require extra logic?
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-24 19:45 ` Baurzhan Ismagulov
2020-11-25 5:51 ` Jan Kiszka
@ 2020-11-25 6:04 ` vijaikumar....@gmail.com
2020-11-25 6:48 ` Jan Kiszka
2020-11-26 10:21 ` Baurzhan Ismagulov
1 sibling, 2 replies; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-25 6:04 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 934 bytes --]
On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net
wrote:
> Hello Vijai Kumar,
>
> On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
> > Changes in v6:
> > - Rebase against latest next
> > - Address review comments from Henning
> > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
>
> The code looks good to me. There is a couple of small issues, I'll reply
> to the
> respective patches.
>
> How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see
> downloads/deb-src/debian-buster, which contains only hello. Is this
> intended?
>
Hi Baurzhan,
You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf.
By default its disabled. Not everyone
want to cache the deb sources. It takes a lot of time.
[1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ
Thanks,
Vijai Kumar K
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 1746 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-25 6:04 ` vijaikumar....@gmail.com
@ 2020-11-25 6:48 ` Jan Kiszka
2020-11-25 7:16 ` vijaikumar....@gmail.com
2020-11-26 10:21 ` Baurzhan Ismagulov
1 sibling, 1 reply; 31+ messages in thread
From: Jan Kiszka @ 2020-11-25 6:48 UTC (permalink / raw)
To: vijaikumar....@gmail.com, isar-users
On 25.11.20 07:04, vijaikumar....@gmail.com wrote:
>
>
> On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net
> wrote:
>
> Hello Vijai Kumar,
>
> On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
> > Changes in v6:
> > - Rebase against latest next
> > - Address review comments from Henning
> > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
> <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3>
>
> The code looks good to me. There is a couple of small issues, I'll
> reply to the
> respective patches.
>
> How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base,
> see
> downloads/deb-src/debian-buster, which contains only hello. Is this
> intended?
>
>
> Hi Baurzhan,
>
> You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local
> conf. By default its disabled. Not everyone
> want to cache the deb sources. It takes a lot of time.
>
> [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ
>
Please add a documentation patch to your series to explain all use cases
and knobs of it. See also my other reply.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-25 6:48 ` Jan Kiszka
@ 2020-11-25 7:16 ` vijaikumar....@gmail.com
0 siblings, 0 replies; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-25 7:16 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 1485 bytes --]
On Wednesday, November 25, 2020 at 12:18:40 PM UTC+5:30 Jan Kiszka wrote:
> On 25.11.20 07:04, vijaikumar....@gmail.com wrote:
> >
> >
> > On Wednesday, November 25, 2020 at 1:15:30 AM UTC+5:30 i...@radix50.net
> > wrote:
> >
> > Hello Vijai Kumar,
> >
> > On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
> > > Changes in v6:
> > > - Rebase against latest next
> > > - Address review comments from Henning
> > > Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3
> > <https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3>
> >
> > The code looks good to me. There is a couple of small issues, I'll
> > reply to the
> > respective patches.
> >
> > How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base,
> > see
> > downloads/deb-src/debian-buster, which contains only hello. Is this
> > intended?
> >
> >
> > Hi Baurzhan,
> >
> > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local
> > conf. By default its disabled. Not everyone
> > want to cache the deb sources. It takes a lot of time.
> >
> > [1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ
> >
>
> Please add a documentation patch to your series to explain all use cases
> and knobs of it. See also my other reply.
>
> Jan
>
Yes Jan. I am currently adding that in usermanual. Will send the patch soon.
Thanks,
Vijai Kumar K
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux
>
[-- Attachment #1.2: Type: text/html, Size: 3167 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH v6 00/13] Deb-src caching
2020-11-25 6:04 ` vijaikumar....@gmail.com
2020-11-25 6:48 ` Jan Kiszka
@ 2020-11-26 10:21 ` Baurzhan Ismagulov
2020-11-26 10:43 ` vijaikumar....@gmail.com
1 sibling, 1 reply; 31+ messages in thread
From: Baurzhan Ismagulov @ 2020-11-26 10:21 UTC (permalink / raw)
To: isar-users
On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote:
> You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf.
> By default its disabled. Not everyone
> want to cache the deb sources. It takes a lot of time.
Thanks Vijai Kumar, will play with v8. It's good to have this in the manual.
What about the issues below? Should I include them myself?
p1:
On Tue, Nov 24, 2020 at 08:47:11PM +0100, Baurzhan Ismagulov wrote:
> On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote:
> > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> > index afec1cb..eae14d5 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -201,45 +201,6 @@ rootfs_generate_manifest () {
> > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> > }
> >
> > -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
>
> Suggest also removing the 'finalize-rootfs' comment before ROOTFS_FEATURES ?=
> "".
p3, p9:
On Tue, Nov 24, 2020 at 08:49:19PM +0100, Baurzhan Ismagulov wrote:
> On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> > index 2391529..6ca5759 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -63,7 +63,7 @@ image_do_mounts() {
> > }
> >
> > ROOTFSDIR = "${IMAGE_ROOTFS}"
> > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
>
> Should it be "cache-deb-src"? The same for p9.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH v6 00/13] Deb-src caching
2020-11-26 10:21 ` Baurzhan Ismagulov
@ 2020-11-26 10:43 ` vijaikumar....@gmail.com
0 siblings, 0 replies; 31+ messages in thread
From: vijaikumar....@gmail.com @ 2020-11-26 10:43 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 2032 bytes --]
On Thursday, November 26, 2020 at 3:51:34 PM UTC+5:30 i...@radix50.net
wrote:
> On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote:
> > You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local
> conf.
> > By default its disabled. Not everyone
> > want to cache the deb sources. It takes a lot of time.
>
> Thanks Vijai Kumar, will play with v8. It's good to have this in the
> manual.
>
> What about the issues below? Should I include them myself?
>
Oops. Sorry about that. I somehow missed to see these comments. I will
address these in v9.
Thanks,
Vijai Kumar K
>
>
> p1:
>
> On Tue, Nov 24, 2020 at 08:47:11PM +0100, Baurzhan Ismagulov wrote:
> > On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote:
> > > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> > > index afec1cb..eae14d5 100644
> > > --- a/meta/classes/rootfs.bbclass
> > > +++ b/meta/classes/rootfs.bbclass
> > > @@ -201,45 +201,6 @@ rootfs_generate_manifest () {
> > > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> > > }
> > >
> > > -ROOTFS_POSTPROCESS_COMMAND +=
> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs',
> 'rootfs_postprocess_finalize', '', d)}"
> >
> > Suggest also removing the 'finalize-rootfs' comment before
> ROOTFS_FEATURES ?=
> > "".
>
>
> p3, p9:
>
> On Tue, Nov 24, 2020 at 08:49:19PM +0100, Baurzhan Ismagulov wrote:
> > On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> > > index 2391529..6ca5759 100644
> > > --- a/meta/classes/image.bbclass
> > > +++ b/meta/classes/image.bbclass
> > > @@ -63,7 +63,7 @@ image_do_mounts() {
> > > }
> > >
> > > ROOTFSDIR = "${IMAGE_ROOTFS}"
> > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest
> cach-deb-src"
> >
> > Should it be "cache-deb-src"? The same for p9.
>
>
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 2714 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread