[PATCH v7 00/14] Deb-src caching

[PATCH v7 00/14] Deb-src caching

[Vijai Kumar K]

Changes in v7:
- Document details in user manual.

Changes in v6:
- Rebase against latest next
- Address review comments from Henning
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3

Changes in v5:
The major changes in this series are as below,
- (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
of debs from different distro which inturn helps the src caching logic.
- (P5) Avoid downloading package from other builds.
- (P8) Add a reprepro based sanity test to check if the repo contains
the sources for all the debs.
- (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
to enable or disable various base-apt features. (cache-deb-src for now)
- Some fixes in (P12 & P13)
Also, addressed some review comments from Henning.
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5

Changes in v4:
- Use <source package>=<version> format instead of just using <packagename>
to download the right version of source package.

Changes in v3:
- Take care of non-existent downloads/deb-src directory.

Changes in v2:
- Introduced additional patch to cache deb src
- Rebased on top of henning/staging4 tree


Vijai Kumar K (14):
  rootfs: Make rootfs finalize a separate task
  deb-dl-dir: Cache host distro debs separately
  meta: cache deb srcs as part of postprocessing
  deb-dl-dir: Make debsrc_download faster
  deb-dl-dir: Download files only belonging to the current image
  deb-dl-dir: Factor out the mounting part
  deb-dl-dir: Fix skipping of removed files
  repository: Add a sanity test to check missing sources
  base-apt: Introduce BASE_REPO_FEATURES
  repository: Fix failures due to missing section
  scripts/ci_build.sh: Enable deb-src caching
  rootfs: Fix possible overwrite of existing resolv.conf
  rootfs: Handle failures when postprocess is rerun
  doc/user_manual.md: Document details about deb-src caching

 doc/user_manual.md                            | 25 ++++++++
 meta-isar/conf/local.conf.sample              |  4 ++
 meta/classes/deb-dl-dir.bbclass               | 61 +++++++++++++++++-
 meta/classes/dpkg-gbp.bbclass                 |  8 ++-
 meta/classes/dpkg.bbclass                     |  8 ++-
 meta/classes/image-tools-extension.bbclass    |  4 +-
 meta/classes/image.bbclass                    | 43 ++++++++++++-
 meta/classes/repository.bbclass               | 13 +++-
 meta/classes/rootfs.bbclass                   | 62 +++++++------------
 .../isar-bootstrap/isar-bootstrap.inc         | 12 +++-
 meta/recipes-devtools/base-apt/base-apt.bb    |  3 +
 scripts/ci_build.sh                           |  1 +
 12 files changed, 191 insertions(+), 53 deletions(-)

-- 
2.17.1

[PATCH v7 01/14] rootfs: Make rootfs finalize a separate task

[Vijai Kumar K]

With the current implementation it is difficult to append a
postprocess function which requires a chroot environment.
For example, to add a postprocess function which runs apt-get to
download all source of packages installed in the target.

rootfs_postprocess_finalize is not actually an optional feature
but instead a necessary cleanup function for image class.
So, move the implementation to image class and make it as a task.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/image.bbclass  | 41 ++++++++++++++++++++++++++++++++++++-
 meta/classes/rootfs.bbclass | 39 -----------------------------------
 2 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a296cc0..2391529 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
 }
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
 ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
 
@@ -173,5 +173,44 @@ python do_deploy() {
 }
 addtask deploy before do_build after do_image
 
+do_rootfs_finalize() {
+    sudo -s <<'EOSUDO'
+        test -e "${ROOTFSDIR}/chroot-setup.sh" && \
+            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
+        rm -f "${ROOTFSDIR}/chroot-setup.sh"
+
+        test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
+            find "${ROOTFSDIR}/usr/bin" \
+                -maxdepth 1 -name 'qemu-*-static' -type f -delete
+
+        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
+            umount -l ${ROOTFSDIR}/isar-apt
+        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
+
+        mountpoint -q '${ROOTFSDIR}/base-apt' && \
+            umount -l ${ROOTFSDIR}/base-apt
+        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
+
+        mountpoint -q '${ROOTFSDIR}/dev' && \
+            umount -l ${ROOTFSDIR}/dev
+        mountpoint -q '${ROOTFSDIR}/sys' && \
+            umount -l ${ROOTFSDIR}/proc
+        mountpoint -q '${ROOTFSDIR}/sys' && \
+            umount -l ${ROOTFSDIR}/sys
+
+        rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+
+        rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
+        rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
+        rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+
+        mv "${ROOTFSDIR}/etc/apt/sources-list" \
+            "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+
+        rm -f "${ROOTFSDIR}/etc/apt/sources-list"
+EOSUDO
+}
+addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
+
 # Last so that the image type can overwrite tasks if needed
 inherit ${IMAGE_TYPE}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cb..eae14d5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -201,45 +201,6 @@ rootfs_generate_manifest () {
         ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
 }
 
-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
-rootfs_postprocess_finalize() {
-    sudo -s <<'EOSUDO'
-        test -e "${ROOTFSDIR}/chroot-setup.sh" && \
-            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
-        rm -f "${ROOTFSDIR}/chroot-setup.sh"
-
-        test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
-            find "${ROOTFSDIR}/usr/bin" \
-                -maxdepth 1 -name 'qemu-*-static' -type f -delete
-
-        mountpoint -q '${ROOTFSDIR}/isar-apt' && \
-            umount -l ${ROOTFSDIR}/isar-apt
-        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
-
-        mountpoint -q '${ROOTFSDIR}/base-apt' && \
-            umount -l ${ROOTFSDIR}/base-apt
-        rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
-
-        mountpoint -q '${ROOTFSDIR}/dev' && \
-            umount -l ${ROOTFSDIR}/dev
-        mountpoint -q '${ROOTFSDIR}/sys' && \
-            umount -l ${ROOTFSDIR}/proc
-        mountpoint -q '${ROOTFSDIR}/sys' && \
-            umount -l ${ROOTFSDIR}/sys
-
-        rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
-
-        rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
-        rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
-        rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
-        mv "${ROOTFSDIR}/etc/apt/sources-list" \
-            "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-
-        rm -f "${ROOTFSDIR}/etc/apt/sources-list"
-EOSUDO
-}
-
 do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
 python do_rootfs_postprocess() {
     # Take care that its correctly mounted:
-- 
2.17.1

[PATCH v7 02/14] deb-dl-dir: Cache host distro debs separately

[Vijai Kumar K]

In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
we were still caching the debs from both the distros into the same
DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be
cached in the relevant subdirectory and avoids mixing of debs from
two separate distros.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass                     |  4 ++--
 meta/classes/dpkg-gbp.bbclass                       |  8 ++++++--
 meta/classes/dpkg.bbclass                           |  8 ++++++--
 meta/classes/image-tools-extension.bbclass          |  4 ++--
 meta/classes/rootfs.bbclass                         |  4 ++--
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++--
 6 files changed, 28 insertions(+), 12 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e996703..29a3d67 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -6,7 +6,7 @@
 inherit repository
 
 deb_dl_dir_import() {
-    export pc="${DEBDIR}/${DISTRO}/"
+    export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     [ ! -d "${pc}" ] && return 0
     sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -20,7 +20,7 @@ deb_dl_dir_import() {
 }
 
 deb_dl_dir_export() {
-    export pc="${DEBDIR}/${DISTRO}/"
+    export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
     mkdir -p "${pc}"
     flock "${pc}".lock -c '
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index afa1e19..ba5c3eb 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
 do_install_builddeps_append() {
     dpkg_do_mounts
-    deb_dl_dir_import "${BUILDCHROOT_DIR}"
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends --download-only ${GBP_DEPENDS}
-    deb_dl_dir_export "${BUILDCHROOT_DIR}"
+    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a24386d..4e7c2f7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,10 +9,14 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 do_install_builddeps() {
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
-    deb_dl_dir_import "${BUILDCHROOT_DIR}"
+    distro="${DISTRO}"
+    if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+       distro="${HOST_DISTRO}"
+    fi
+    deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
-    deb_dl_dir_export "${BUILDCHROOT_DIR}"
+    deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${PACKAGE_ARCH}
     dpkg_undo_mounts
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 6590ee7..0b067ff 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,7 +25,7 @@ do_install_imager_deps() {
     buildchroot_do_mounts
 
     E="${@ isar_export_proxies(d)}"
-    deb_dl_dir_import ${BUILDCHROOT_DIR}
+    deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
@@ -35,7 +35,7 @@ do_install_imager_deps() {
             --allow-unauthenticated --allow-downgrades --download-only install \
             ${IMAGER_INSTALL}'
 
-    deb_dl_dir_export ${BUILDCHROOT_DIR}
+    deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index eae14d5..6316321 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -115,7 +115,7 @@ rootfs_install_resolvconf() {
 ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
 rootfs_import_package_cache[weight] = "5"
 rootfs_import_package_cache() {
-    deb_dl_dir_import ${ROOTFSDIR}
+    deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
 }
 
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
@@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
 ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
 rootfs_export_package_cache[weight] = "5"
 rootfs_export_package_cache() {
-    deb_dl_dir_export ${ROOTFSDIR}
+    deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO}
 }
 
 ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index fbfe669..3b19914 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -255,7 +255,11 @@ isar_bootstrap() {
     export IS_HOST debootstrap_args E
     if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
         sudo rm -rf --one-file-system "${ROOTFSDIR}"
-        deb_dl_dir_import "${ROOTFSDIR}"
+        if [ "${IS_HOST}" ];then
+            deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+        else
+            deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+        fi
 
         sudo -E -s <<'EOSUDO'
             set -e
@@ -356,7 +360,11 @@ isar_bootstrap() {
             ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
 EOSUDO
     fi
-    deb_dl_dir_export "${ROOTFSDIR}"
+    if [ "${IS_HOST}" ];then
+        deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
+    else
+        deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
+    fi
 }
 
 CLEANFUNCS = "clean_deploy"
-- 
2.17.1

[PATCH v7 03/14] meta: cache deb srcs as part of postprocessing

[Vijai Kumar K]

Collect the deb sources of the corresponding deb binaries cached
in DEBDIR as part of image postprocess.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass | 27 +++++++++++++++++++++++++++
 meta/classes/image.bbclass      |  2 +-
 meta/classes/rootfs.bbclass     |  8 ++++++++
 3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 29a3d67..659fe4b 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,33 @@
 
 inherit repository
 
+debsrc_download() {
+    export rootfs="$1"
+    export rootfs_distro="$2"
+    mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+    sudo -E -s <<'EOSUDO'
+    mkdir -p "${rootfs}/deb-src"
+    mountpoint -q "${rootfs}/deb-src" || \
+    mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
+EOSUDO
+    ( flock 9
+    set -e
+    printenv | grep -q BB_VERBOSE_LOGS && set -x
+    find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+        local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+        local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+
+        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+            sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+    done
+    ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
+    sudo -E -s <<'EOSUDO'
+    mountpoint -q "${rootfs}/deb-src" && \
+    umount -l "${rootfs}/deb-src"
+    rm -rf "${rootfs}/deb-src"
+EOSUDO
+}
+
 deb_dl_dir_import() {
     export pc="${DEBDIR}/${2}"
     export rootfs="${1}"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2391529..6ca5759 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
 }
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
 ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
 
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 6316321..cfabeae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,6 +185,14 @@ python do_rootfs_install() {
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
 
+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
+cache_deb_src() {
+    rootfs_install_resolvconf
+    deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
+    debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
+    rootfs_install_clean_files
+}
+
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
 rootfs_postprocess_clean_package_cache() {
     sudo -E chroot '${ROOTFSDIR}' \
-- 
2.17.1

[PATCH v7 04/14] deb-dl-dir: Make debsrc_download faster

[Vijai Kumar K]

Eventhough apt-get source skips redownloading of files, it is still
slow and takes a lot of time. Instead, lookup if the dsc file is already
present in the cache and skip based on it.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 659fe4b..79a9a88 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -20,6 +20,8 @@ EOSUDO
     find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
         local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
         local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+        local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
+        [ -n "$dscfile" ] && continue
 
         sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
             sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
-- 
2.17.1

[PATCH v7 05/14] deb-dl-dir: Download files only belonging to the current image

[Vijai Kumar K]

Avoid downloading deb-srcs for debs cached from other image builds.
One way to ensure that is to see if the package is present in the dpkg
status file.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 79a9a88..ce691cf 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,15 @@
 
 inherit repository
 
+is_not_part_of_current_build() {
+    local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
+    local output="$( grep -hs "^Package: ${package}" \
+            "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
+            "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
+            "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+    [ -z "${output}" ]
+}
+
 debsrc_download() {
     export rootfs="$1"
     export rootfs_distro="$2"
@@ -18,6 +27,7 @@ EOSUDO
     set -e
     printenv | grep -q BB_VERBOSE_LOGS && set -x
     find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+        is_not_part_of_current_build "${package}" && continue
         local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
         local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
         local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
-- 
2.17.1

[PATCH v7 06/14] deb-dl-dir: Factor out the mounting part

[Vijai Kumar K]

Factor out the mount and unmount section to separate function for
easy readability. No functional change intended.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass | 32 ++++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ce691cf..b2d94e7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,15 +14,30 @@ is_not_part_of_current_build() {
     [ -z "${output}" ]
 }
 
+debsrc_do_mounts() {
+    sudo -s <<EOSUDO
+    mkdir -p "${1}/deb-src"
+    mountpoint -q "${1}/deb-src" || \
+    mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+EOSUDO
+}
+
+debsrc_undo_mounts() {
+    sudo -s <<EOSUDO
+    mkdir -p "${1}/deb-src"
+    mountpoint -q "${1}/deb-src" && \
+    umount -l "${1}/deb-src"
+    rm -rf "${1}/deb-src"
+EOSUDO
+}
+
 debsrc_download() {
     export rootfs="$1"
     export rootfs_distro="$2"
     mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
-    sudo -E -s <<'EOSUDO'
-    mkdir -p "${rootfs}/deb-src"
-    mountpoint -q "${rootfs}/deb-src" || \
-    mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
-EOSUDO
+
+    debsrc_do_mounts "${rootfs}"
+
     ( flock 9
     set -e
     printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -37,11 +52,8 @@ EOSUDO
             sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
     done
     ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
-    sudo -E -s <<'EOSUDO'
-    mountpoint -q "${rootfs}/deb-src" && \
-    umount -l "${rootfs}/deb-src"
-    rm -rf "${rootfs}/deb-src"
-EOSUDO
+
+    debsrc_undo_mounts "${rootfs}"
 }
 
 deb_dl_dir_import() {
-- 
2.17.1

[PATCH v7 07/14] deb-dl-dir: Fix skipping of removed files

[Vijai Kumar K]

Some packages are installed and then removed, like localepurge from
image-locales-extension.bbclass. Those information would not be
available in dpkg status file. Use dpkg log instead to see if the
package has been used on the target.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b2d94e7..a94fb10 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -7,10 +7,16 @@ inherit repository
 
 is_not_part_of_current_build() {
     local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
-    local output="$( grep -hs "^Package: ${package}" \
-            "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
-            "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
-            "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+    local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
+    local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
+    # Since we are parsing all the debs in DEBDIR, we can to some extend
+    # try to eliminate some debs that are not part of the current multiconfig
+    # build using the below method.
+    local output="$( grep -hs "status installed ${package}:${arch} ${version}" \
+            "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+            "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
+            "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
+
     [ -z "${output}" ]
 }
 
-- 
2.17.1

[PATCH v7 08/14] repository: Add a sanity test to check missing sources

[Vijai Kumar K]

Add a sanity test routine to test if sources for all packages
are present in the repo.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/repository.bbclass            | 10 ++++++++++
 meta/recipes-devtools/base-apt/base-apt.bb |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index c70138f..7b6e47b 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -96,3 +96,13 @@ repo_contains_package() {
     # no
     return 2
 }
+
+repo_sanity_test() {
+    local dir="$1"
+    local dbdir="$2"
+    local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+    if [ -n "${output}" ]; then
+        bbwarn "One or more sources are missing in base-apt"
+        bbwarn "${output}"
+    fi
+}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index da3e32e..8940ae8 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -66,6 +66,8 @@ repo() {
     fi
 
     populate_base_apt
+    repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+        "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
 }
 
 python do_cache() {
-- 
2.17.1

[PATCH v7 09/14] base-apt: Introduce BASE_REPO_FEATURES

[Vijai Kumar K]

Eventhough we are collecting the debsrcs as part of postprocess,
it could not be considered a ROOTFS_FEATURE, instead a base-apt one.
Introduce BASE_REPO_FEATURES, to provide user with control to enable
or disable cache-deb-src. Disabled by default, since it is not required
for normal offline build to work.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta-isar/conf/local.conf.sample           | 4 ++++
 meta/classes/image.bbclass                 | 4 +++-
 meta/classes/repository.bbclass            | 9 +++++----
 meta/classes/rootfs.bbclass                | 1 -
 meta/recipes-devtools/base-apt/base-apt.bb | 1 +
 5 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 07a7781..ec0a384 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -181,6 +181,10 @@ ISAR_CROSS_COMPILE ?= "0"
 # NOTE: this works for amd64 and arm64 targets so far
 #ISAR_ENABLE_COMPAT_ARCH ?= "1"
 
+# Uncomment this to enable caching of all source packages.
+# Without this feature, only sources of packages downloaded with apt:// are downloaded.
+#BASE_REPO_FEATURES ?= "cache-deb-src"
+
 #
 # Uncomment this to enable use of cached base repository
 #ISAR_USE_CACHED_BASE_REPO ?= "1"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6ca5759..5c2c88c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,10 +63,12 @@ image_do_mounts() {
 }
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
 ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
 
+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)} "
+
 inherit rootfs
 inherit image-sdk-extension
 inherit image-tools-extension
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 7b6e47b..1f475dc 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -100,9 +100,10 @@ repo_contains_package() {
 repo_sanity_test() {
     local dir="$1"
     local dbdir="$2"
-    local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
-    if [ -n "${output}" ]; then
-        bbwarn "One or more sources are missing in base-apt"
-        bbwarn "${output}"
+    if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ];then
+        local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+        if [ -n "${output}" ]; then
+            bbfatal "One or more sources are missing in repo. ${output}"
+        fi
     fi
 }
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cfabeae..0aa5502 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,7 +185,6 @@ python do_rootfs_install() {
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
 
-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
 cache_deb_src() {
     rootfs_install_resolvconf
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 8940ae8..506a28f 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -10,6 +10,7 @@ SRC_URI = "file://distributions.in"
 
 BASE_REPO_KEY ?= ""
 KEYFILES ?= ""
+BASE_REPO_FEATURES ?= ""
 
 populate_base_apt() {
     find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
-- 
2.17.1

[PATCH v7 10/14] repository: Fix failures due to missing section

[Vijai Kumar K]

reprepro's includedsc failed for certain packages like makedev while
complaining about missing Section information. This information is
optional according to Debian[1]. Use a dummy value to avoid such
failures.

[1] https://wiki.debian.org/DebianRepository/Format

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/repository.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 1f475dc..ac395db 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -42,7 +42,7 @@ repo_add_srcpackage() {
     if [ -n "${GNUPGHOME}" ]; then
         export GNUPGHOME="${GNUPGHOME}"
     fi
-    reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
+    reprepro -b "${dir}" --dbdir "${dbdir}" -C main -S - -P source \
         includedsc "${codename}" \
         "$@"
 }
-- 
2.17.1

[PATCH v7 11/14] scripts/ci_build.sh: Enable deb-src caching

[Vijai Kumar K]

Enable debsrc caching for CI builds.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 scripts/ci_build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a5eee82..09a0b4b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -184,6 +184,7 @@ if [ -n "$REPRO_BUILD" ]; then
     sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
 fi
 
+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
 # Start cross build for the defined set of configurations
 sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
 bitbake $BB_ARGS $CROSS_TARGETS_SET
-- 
2.17.1

[PATCH v7 12/14] rootfs: Fix possible overwrite of existing resolv.conf

[Vijai Kumar K]

There is a possiblilty that one of the packages installed in the
rootfs provides /etc/resolv.conf and we might accidentally remove
it. Fix it by taking a backup of any existing resolv conf files
and restoring it later. This is needed since we could not effectively
move caching before rootfs_install_clean_files since we need the
latest dpkg log.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/rootfs.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0aa5502..ed43fb9 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -186,10 +186,18 @@ python do_rootfs_install() {
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
 
 cache_deb_src() {
+    if [ -e "${ROOTFSDIR}"/etc/resolv.conf ]; then
+        sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
+    fi
     rootfs_install_resolvconf
+
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
     debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
-    rootfs_install_clean_files
+
+    sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
+    if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
+        sudo mv "${ROOTFSDIR}"/etc/resolv.conf.isar "${ROOTFSDIR}"/etc/resolv.conf
+    fi
 }
 
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
-- 
2.17.1

[PATCH v7 13/14] rootfs: Handle failures when postprocess is rerun

[Vijai Kumar K]

The apt state information in var/lib/apt/lists is cleared as
part of postprocessing. This makes apt-get calls in cache_deb_src
fail when rerunning the postprocess task.

Since we cannot run apt-get update again to refresh the state
information, copy the apt state information from the initial
bootstrapped image.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/rootfs.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ed43fb9..bebc0c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -190,6 +190,10 @@ cache_deb_src() {
         sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
     fi
     rootfs_install_resolvconf
+    # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
+    # relies on that through out the build. Copy that state information instead of apt-get update
+    # which generates a new state from upstream.
+    sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
 
     deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
     debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
-- 
2.17.1

[PATCH v7 14/14] doc/user_manual.md: Document details about deb-src caching

[Vijai Kumar K]

Add details about deb-src caching.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 doc/user_manual.md | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 8d04cd2..c0f842a 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -918,3 +918,28 @@ And build the corresponding image target:
 ```
 bitbake mc:qemuarm64-buster:isar-image-base
 ```
+## Cache all upstream Debian source packages in local apt
+
+### Motivation
+
+For OSS license clearance, we might need to have the sources of all Debian packages we
+ship with the particular image.
+
+Currently the local-apt generated has only Debian binary packages. Extend the local-apt
+to have Debian source packages as well, for OSS clearing purposes.
+
+### Solution
+
+ - Trigger download of Debian source packages as part of rootfs postprocess.
+
+With the current base-apt implementation, we already cache all the binary packages and
+generate a local-apt for offline build.
+
+Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
+download the corresponding Debian source file using "apt-get source" command.
+
+By default, the Debian source caching is not enabled. If you want to cache the Debian
+sources as well, as part of your image build, add the below line to your local.conf file.
+```
+BASE_REPO_FEATURES = "cache-deb-src"
+```
-- 
2.17.1

Re: [PATCH v7 14/14] doc/user_manual.md: Document details about deb-src caching

[Jan Kiszka]

On 25.11.20 09:06, Vijai Kumar K wrote:
> Add details about deb-src caching.
> 
> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> ---
>  doc/user_manual.md | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 8d04cd2..c0f842a 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -918,3 +918,28 @@ And build the corresponding image target:
>  ```
>  bitbake mc:qemuarm64-buster:isar-image-base
>  ```
> +## Cache all upstream Debian source packages in local apt
> +
> +### Motivation
> +
> +For OSS license clearance, we might need to have the sources of all Debian packages we
> +ship with the particular image.

OSS license *compliance*: Some licenses require to provide the
corresponding sources code, other require copyright attributions that
may be best provided via the source code. In addition, you may want to
archive the code locally in order to ensure reproducibility (and
modifiability) in the future.

> +
> +Currently the local-apt generated has only Debian binary packages. Extend the local-apt
> +to have Debian source packages as well, for OSS clearing purposes.
> +
> +### Solution
> +
> + - Trigger download of Debian source packages as part of rootfs postprocess.
> +
> +With the current base-apt implementation, we already cache all the binary packages and
> +generate a local-apt for offline build.
> +
> +Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
> +download the corresponding Debian source file using "apt-get source" command.
> +
> +By default, the Debian source caching is not enabled. If you want to cache the Debian
> +sources as well, as part of your image build, add the below line to your local.conf file.
> +```
> +BASE_REPO_FEATURES = "cache-deb-src"
> +```
> 

So this will fetch and and store ALL sources corresponding to the binary
packages that the target image installs? Or rather that any rootfs
installs, thus also buildchroots and sdkrootfs? Please make this clearer.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH v7 14/14] doc/user_manual.md: Document details about deb-src caching

[vijaikumar....@gmail.com]

[-- Attachment #1.1: Type: text/plain, Size: 2491 bytes --]



On Wednesday, November 25, 2020 at 5:05:45 PM UTC+5:30 Jan Kiszka wrote:

> On 25.11.20 09:06, Vijai Kumar K wrote: 
> > Add details about deb-src caching. 
> > 
> > Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com> 
> > --- 
> > doc/user_manual.md | 25 +++++++++++++++++++++++++ 
> > 1 file changed, 25 insertions(+) 
> > 
> > diff --git a/doc/user_manual.md b/doc/user_manual.md 
> > index 8d04cd2..c0f842a 100644 
> > --- a/doc/user_manual.md 
> > +++ b/doc/user_manual.md 
> > @@ -918,3 +918,28 @@ And build the corresponding image target: 
> > ``` 
> > bitbake mc:qemuarm64-buster:isar-image-base 
> > ``` 
> > +## Cache all upstream Debian source packages in local apt 
> > + 
> > +### Motivation 
> > + 
> > +For OSS license clearance, we might need to have the sources of all 
> Debian packages we 
> > +ship with the particular image. 
>
> OSS license *compliance*: Some licenses require to provide the 
> corresponding sources code, other require copyright attributions that 
> may be best provided via the source code. In addition, you may want to 
> archive the code locally in order to ensure reproducibility (and 
> modifiability) in the future. 
>

Will include this line.
 

>
> > + 
> > +Currently the local-apt generated has only Debian binary packages. 
> Extend the local-apt 
> > +to have Debian source packages as well, for OSS clearing purposes. 
> > + 
> > +### Solution 
> > + 
> > + - Trigger download of Debian source packages as part of rootfs 
> postprocess. 
> > + 
> > +With the current base-apt implementation, we already cache all the 
> binary packages and 
> > +generate a local-apt for offline build. 
> > + 
> > +Use rootfs postprocessing to parse through the the list fo deb files in 
> ${DEDDIR} and 
> > +download the corresponding Debian source file using "apt-get source" 
> command. 
> > + 
> > +By default, the Debian source caching is not enabled. If you want to 
> cache the Debian 
> > +sources as well, as part of your image build, add the below line to 
> your local.conf file. 
> > +``` 
> > +BASE_REPO_FEATURES = "cache-deb-src" 
> > +``` 
> > 
>
> So this will fetch and and store ALL sources corresponding to the binary 
> packages that the target image installs? Or rather that any rootfs 
> installs, thus also buildchroots and sdkrootfs? Please make this clearer. 
>

Yes. Exactly. I will update the document.

Thanks,
Vijai Kumar K
 

>
> Jan 
>
> -- 
> Siemens AG, T RDA IOT 
> Corporate Competence Center Embedded Linux 
>

[-- Attachment #1.2: Type: text/html, Size: 3317 bytes --]