[PATCH 00/17] base-apt rework

[PATCH 00/17] base-apt rework

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This series build upon "[PATCHv2 0/9] a few cleanups a bug and a
feature".

base-apt has two main issues that this series fixes:
 - it relied on the rootfs /var/cache/apt to contain all *.debs
  - this is only true of nobody cleaned up, could happen with hooks
    like used in "isar-disable-apt-cache" or similar mechanisms
 - it did not handle "dsc"s (source packages)

It had other minor issues that are described in the individual patches.

The series fixes those issues and extends CI to fully test base-apt.

The general mechanism used in the end is:
 - split all apt-gets into a download and an execution phase
 - copy out all download inbetween the two phases
 - execution phase can now go ahead and possibly delete cached files
   from the rootfs
 - actually copy in everything "old" before the download phase
 - that means the download phase potentially becomes a noop and we speed
   up by using already downloaded files (new feature)

There are a few open points that i will mention later.

Henning Schild (17):
  repository: new class to deal with repos
  dpkg-base: add download caching of apt:// downloads
  base-apt: change the sources.list to also offer deb-src
  base-apt: add deb-src packages as well
  base-apt: do not skip gpg check when it is signed
  ci: conf: add "hello" to the sample config and every build
  meta: split all apt-get invocations into download and execution
  meta: create DL_DIR support for all apt-get downloaded .debs
  meta: import DL_DIR debs before apt-get download steps
  base-apt: populate from DEBDIR as well
  base-apt: drop the "apt_cache" feature
  base-apt: do not copy debs directly out of rootfss anymore
  base-apt: rework base-apt population
  base-apt: move class "base-apt-helper" into only user
  CI: include "isar-disable-apt-cache" into all CI images
  CI: include "cowsay" into default build to test dpkg-gbp
  CI: set BB_NO_NETWORK for cached rebuild

 doc/user_manual.md                            |  1 +
 meta-isar/conf/local.conf.sample              | 10 +--
 meta/classes/base-apt-helper.bbclass          | 57 --------------
 meta/classes/deb-dl-dir.bbclass               | 24 ++++++
 meta/classes/dpkg-base.bbclass                | 45 ++++++-----
 meta/classes/dpkg-gbp.bbclass                 |  5 ++
 meta/classes/dpkg.bbclass                     |  7 +-
 meta/classes/image-cache-extension.bbclass    | 67 +++++++++++++---
 meta/classes/image-locales-extension.bbclass  |  2 +-
 meta/classes/image-tools-extension.bbclass    |  7 ++
 meta/classes/image.bbclass                    |  2 +-
 meta/classes/repository.bbclass               | 78 +++++++++++++++++++
 meta/classes/rootfs.bbclass                   | 28 ++++---
 meta/conf/bitbake.conf                        |  2 +
 .../isar-bootstrap/files/base-apt-sources     |  1 -
 .../isar-bootstrap/isar-bootstrap.inc         | 16 ++--
 meta/recipes-devtools/base-apt/base-apt.bb    | 31 ++------
 .../base-apt/files/distributions.in           |  3 -
 .../buildchroot/files/deps.sh                 | 37 +++++----
 .../isar-apt/files/distributions.in           |  3 -
 meta/recipes-devtools/isar-apt/isar-apt.bb    | 26 ++-----
 scripts/ci_build.sh                           |  7 +-
 22 files changed, 272 insertions(+), 187 deletions(-)
 delete mode 100644 meta/classes/base-apt-helper.bbclass
 create mode 100644 meta/classes/deb-dl-dir.bbclass
 create mode 100644 meta/classes/repository.bbclass
 delete mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
 delete mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
 delete mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in

-- 
2.23.0

[PATCH 01/17] repository: new class to deal with repos

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Factor out all the "reprepro" code into a common class. There are slight
functional changes since the two copies of the code got out of sync.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/base-apt-helper.bbclass          | 28 ++++----
 meta/classes/dpkg-base.bbclass                | 24 ++-----
 meta/classes/repository.bbclass               | 64 +++++++++++++++++++
 meta/recipes-devtools/base-apt/base-apt.bb    | 31 +++------
 .../base-apt/files/distributions.in           |  3 -
 .../isar-apt/files/distributions.in           |  3 -
 meta/recipes-devtools/isar-apt/isar-apt.bb    | 26 ++------
 7 files changed, 97 insertions(+), 82 deletions(-)
 create mode 100644 meta/classes/repository.bbclass
 delete mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
 delete mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in

diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
index d5674c9..567eaea 100644
--- a/meta/classes/base-apt-helper.bbclass
+++ b/meta/classes/base-apt-helper.bbclass
@@ -1,5 +1,10 @@
 # This software is a part of ISAR.
 # Copyright (C) 2018 ilbers GmbH
+# Copyright (C) 2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
+
+inherit repository
 
 compare_pkg_md5sums() {
    pkg1=$1
@@ -14,10 +19,6 @@ compare_pkg_md5sums() {
 populate_base_apt() {
     search_dir=$1
 
-    if [ -n "${GNUPGHOME}" ]; then
-        export GNUPGHOME="${GNUPGHOME}"
-    fi
-
     find $search_dir -name '*.deb' | while read package; do
         # NOTE: due to packages stored by reprepro are not modified, we can
         # use search by filename to check if package is already in repo. In
@@ -40,18 +41,15 @@ populate_base_apt() {
             compare_pkg_md5sums "$package" "$base_apt_p" && continue
 
             # md5sum differs, so remove the package from base-apt
-            name=$(echo $base_name | cut -d '_' -f 1)
-            reprepro -b ${REPO_BASE_DIR}/${BASE_DISTRO} \
-                     --dbdir ${REPO_BASE_DB_DIR}/${BASE_DISTRO} \
-                     -C main -A ${DISTRO_ARCH} \
-                     remove ${BASE_DISTRO_CODENAME} \
-                     $name
+            repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+                "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+                "${BASE_DISTRO_CODENAME}" \
+                "${base_apt_p}"
         fi
 
-        reprepro -b ${REPO_BASE_DIR}/${BASE_DISTRO} \
-                 --dbdir ${REPO_BASE_DB_DIR}/${BASE_DISTRO} \
-                 -C main \
-                 includedeb ${BASE_DISTRO_CODENAME} \
-                 $package
+        repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+            "${BASE_DISTRO_CODENAME}" \
+            "${package}"
     done
 }
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index fae0a2b..3f1e99f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -7,6 +7,7 @@
 inherit buildchroot
 inherit debianize
 inherit terminal
+inherit repository
 
 DEPENDS ?= ""
 
@@ -114,31 +115,16 @@ repo_clean() {
     DEBS=$( find ${S}/.. -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
     if [ -n "${DEBS}" ]; then
         for d in ${DEBS}; do
-            p=$( dpkg-deb --show --showformat '${Package}' ${d} )
-            a=$( dpkg-deb --show --showformat '${Architecture}' ${d} )
-            # removing "all" means no arch
-            aarg="-A ${a}"
-            [ "${a}" = "all" ] && aarg=""
-            reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
-                     --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \
-                     -C main ${aarg} \
-                     remove ${DEBDISTRONAME} \
-                     ${p}
+            repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
+                "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
         done
     fi
 }
 
-# Install package to Isar-apt
 do_deploy_deb() {
-    if [ -n "${GNUPGHOME}" ]; then
-        export GNUPGHOME="${GNUPGHOME}"
-    fi
     repo_clean
-    reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
-             --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \
-             -C main \
-             includedeb ${DEBDISTRONAME} \
-             ${S}/../*.deb
+    repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
+        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
 }
 
 addtask deploy_deb after do_dpkg_build before do_build
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
new file mode 100644
index 0000000..afd3aed
--- /dev/null
+++ b/meta/classes/repository.bbclass
@@ -0,0 +1,64 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+# Copyright (C) 2019 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+repo_create() {
+    dir="$1"
+    dbdir="$2"
+    codename="$3"
+    gpgkey="$4"
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+
+    if [ ! -f "${dir}"/conf/distributions ]; then
+        mkdir -p "${dir}"/conf/
+        cat <<EOF > "${dir}"/conf/distributions
+Codename: ${codename}
+Architectures: i386 armhf arm64 amd64 mipsel source
+Components: main
+EOF
+        if [ -n "${gpgkey}" ] ; then
+            echo "SignWith: yes" >> "${dir}"/conf/distributions
+        fi
+    fi
+    if [ ! -d "${dbdir}" ]; then
+        reprepro -b "${dir}" --dbdir "${dbdir}" export "${codename}"
+    fi
+}
+
+repo_add_packages() {
+    dir="$1"
+    dbdir="$2"
+    codename="$3"
+    shift; shift; shift
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+    reprepro -b "${dir}" --dbdir "${dbdir}" -C main \
+        includedeb "${codename}" \
+        "$@"
+}
+
+repo_del_package() {
+    dir="$1"
+    dbdir="$2"
+    codename="$3"
+    file="$4"
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+    p=$( dpkg-deb --show --showformat '${Package}' "${file}" )
+    a=$( dpkg-deb --show --showformat '${Architecture}' "${file}" )
+    # removing "all" means no arch
+    aarg="-A ${a}"
+    [ "${a}" = "all" ] && aarg=""
+    reprepro -b "${dir}" --dbdir "${dbdir}" -C main ${aarg} \
+        remove "${codename}" \
+        "${p}"
+}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 6acd6e7..765bb1b 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -1,38 +1,23 @@
 # This software is a part of ISAR.
 # Copyright (C) 2018 ilbers GmbH
+# Copyright (C) 2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
-SRC_URI = "file://distributions.in"
+inherit repository
 
 BASE_REPO_KEY ?= ""
 
-CACHE_CONF_DIR = "${REPO_BASE_DIR}/${BASE_DISTRO}/conf"
-do_cache_config[dirs] = "${CACHE_CONF_DIR}"
 do_cache_config[stamp-extra-info] = "${DISTRO}"
 do_cache_config[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
 
 # Generate reprepro config for current distro if it doesn't exist. Once it's
 # generated, this task should do nothing.
 do_cache_config() {
-    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
-        sed -e "s#{CODENAME}#"${BASE_DISTRO_CODENAME}"#g" \
-            ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
-        if [ "${BASE_REPO_KEY}" ] ; then
-            # To generate Release.gpg
-            echo "SignWith: yes" >> ${CACHE_CONF_DIR}/distributions
-        fi
-    fi
-
-    path_cache="${REPO_BASE_DIR}/${BASE_DISTRO}"
-    path_databases="${REPO_BASE_DB_DIR}/${BASE_DISTRO}"
-
-    if [ ! -d "${path_databases}" ]; then
-        if [ -n "${GNUPGHOME}" ]; then
-            export GNUPGHOME="${GNUPGHOME}"
-        fi
-        reprepro -b ${path_cache} \
-                 --dbdir ${path_databases} \
-                 export ${BASE_DISTRO_CODENAME}
-    fi
+    repo_create "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+        "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+        "${BASE_DISTRO_CODENAME}" \
+        "${BASE_REPO_KEY}"
 }
 
 addtask cache_config after do_unpack before do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
deleted file mode 100644
index fd7de0b..0000000
--- a/meta/recipes-devtools/base-apt/files/distributions.in
+++ /dev/null
@@ -1,3 +0,0 @@
-Codename: {CODENAME}
-Architectures: i386 armhf arm64 amd64 mipsel source
-Components: main
diff --git a/meta/recipes-devtools/isar-apt/files/distributions.in b/meta/recipes-devtools/isar-apt/files/distributions.in
deleted file mode 100644
index a24a700..0000000
--- a/meta/recipes-devtools/isar-apt/files/distributions.in
+++ /dev/null
@@ -1,3 +0,0 @@
-Codename: {DISTRO_NAME}
-Architectures: i386 armhf arm64 amd64 mipsel source
-Components: main
diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb
index e6f1753..0227c1b 100644
--- a/meta/recipes-devtools/isar-apt/isar-apt.bb
+++ b/meta/recipes-devtools/isar-apt/isar-apt.bb
@@ -1,32 +1,20 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2017 ilbers GmbH
+# Copyright (C) 2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
-SRC_URI = "file://distributions.in"
+inherit repository
 
-CACHE_CONF_DIR = "${REPO_ISAR_DIR}/${DISTRO}/conf"
-do_cache_config[dirs] = "${CACHE_CONF_DIR}"
 do_cache_config[stamp-extra-info] = "${DISTRO}"
 do_cache_config[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
 
 # Generate reprepro config for current distro if it doesn't exist. Once it's
 # generated, this task should do nothing.
 do_cache_config() {
-    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
-        sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
-            ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
-    fi
-
-    path_cache="${REPO_ISAR_DIR}/${DISTRO}"
-    path_databases="${REPO_ISAR_DB_DIR}/${DISTRO}"
-
-    if [ ! -d "${path_databases}" ]; then
-        if [ -n "${GNUPGHOME}" ]; then
-            export GNUPGHOME="${GNUPGHOME}"
-        fi
-        reprepro -b ${path_cache} \
-                 --dbdir ${path_databases} \
-                 export ${DEBDISTRONAME}
-    fi
+    repo_create "${REPO_ISAR_DIR}"/"${DISTRO}" \
+        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
+        "${DEBDISTRONAME}"
 }
 
 addtask cache_config after do_unpack before do_build
-- 
2.23.0

[PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Store the results of "apt-get source" in the DL_DIR. This means we
really only need to fetch once on incemental rebuilds or if multiple
recipes use the same SRC_URI.
We can also later collect the results and put them in a repository.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++--
 meta/conf/bitbake.conf         |  1 +
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 3f1e99f..b39df5c 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -37,14 +37,30 @@ do_apt_fetch() {
         -o Dir::Etc::SourceParts="-" \
         -o APT::Get::List-Cleanup="0"
 
-    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-        sh -c 'cd ${PP} && apt-get -y --only-source source ${SRC_APT}'
+    for uri in "${SRC_APT}"; do
+        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+            sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
+        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd ${PP} && apt-get -y --only-source source "$2"' my_script "${DISTRO}" "${uri}"
+    done
+
     dpkg_undo_mounts
 }
 
 addtask apt_fetch after do_unpack before do_patch
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
+addtask cleanall_apt before do_cleanall
+do_cleanall_apt[nostamp] = "1"
+do_cleanall_apt() {
+    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
+        exit
+    fi
+    for uri in "${SRC_APT}"; do
+        rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri"
+    done
+}
+
 def get_package_srcdir(d):
     s = os.path.abspath(d.getVar("S", True))
     workdir = os.path.abspath(d.getVar("WORKDIR", True))
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index bca1114..4c0a809 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
 FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
 GITDIR = "${DL_DIR}/git"
+DEBSRCDIR = "${DL_DIR}/deb-src"
 P = "${PN}-${PV}"
 PF = "${PN}-${PV}-${PR}"
 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
-- 
2.23.0

[PATCH 03/17] base-apt: change the sources.list to also offer deb-src

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The repository is ready for it already. For the moment there will be
nothing there.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/recipes-core/isar-bootstrap/files/base-apt-sources | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
index ddc4509..2a4ab2c 100644
--- a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
+++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
@@ -1 +1,2 @@
 deb [trusted=yes] file:///base-apt/{BASE_DISTRO} {BASE_DISTRO_CODENAME} main
+deb-src [trusted=yes] file:///base-apt/{BASE_DISTRO} {BASE_DISTRO_CODENAME} main
-- 
2.23.0

[PATCH 04/17] base-apt: add deb-src packages as well

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

So far the deb-src packages downloaded via "apt://" where not cached.
This commit fixes that and includes those packages into the cache.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/base-apt-helper.bbclass       |  7 +++++++
 meta/classes/image-cache-extension.bbclass |  4 ++++
 meta/classes/repository.bbclass            | 14 ++++++++++++++
 3 files changed, 25 insertions(+)

diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
index 567eaea..3013edf 100644
--- a/meta/classes/base-apt-helper.bbclass
+++ b/meta/classes/base-apt-helper.bbclass
@@ -52,4 +52,11 @@ populate_base_apt() {
             "${BASE_DISTRO_CODENAME}" \
             "${package}"
     done
+
+    find $search_dir -name '*.dsc' | while read package; do
+        repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+            "${BASE_DISTRO_CODENAME}" \
+            "${package}"
+    done
 }
diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index 30db549..f8b9ba3 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -32,5 +32,9 @@ do_cache_base_repo() {
     if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
         populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
     fi
+
+    if [ -d "${DEBSRCDIR}"/"${DISTRO}" ]; then
+        populate_base_apt "${DEBSRCDIR}"/"${DISTRO}"
+    fi
 }
 addtask cache_base_repo after do_rootfs do_install_imager_deps
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index afd3aed..22d1f93 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -30,6 +30,20 @@ EOF
     fi
 }
 
+repo_add_srcpackage() {
+    dir="$1"
+    dbdir="$2"
+    codename="$3"
+    shift; shift; shift
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+    reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
+        includedsc "${codename}" \
+        "$@"
+}
+
 repo_add_packages() {
     dir="$1"
     dbdir="$2"
-- 
2.23.0

[PATCH 05/17] base-apt: do not skip gpg check when it is signed

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

We used to have "trusted=yes" hardcoded even for a signed base-apt. Make
that flag depend on whether it is signed and generate that file with two
echos, it is simple enough.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../isar-bootstrap/files/base-apt-sources        |  2 --
 .../isar-bootstrap/isar-bootstrap.inc            | 16 ++++++++--------
 2 files changed, 8 insertions(+), 10 deletions(-)
 delete mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources

diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
deleted file mode 100644
index 2a4ab2c..0000000
--- a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
+++ /dev/null
@@ -1,2 +0,0 @@
-deb [trusted=yes] file:///base-apt/{BASE_DISTRO} {BASE_DISTRO_CODENAME} main
-deb-src [trusted=yes] file:///base-apt/{BASE_DISTRO} {BASE_DISTRO_CODENAME} main
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 5aaecd1..cc12378 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -12,8 +12,7 @@ SRC_URI = " \
     file://isar-apt.conf \
     file://isar-apt-fallback.conf \
     file://locale \
-    file://chroot-setup.sh \
-    file://base-apt-sources"
+    file://chroot-setup.sh"
 PV = "1.0"
 
 DEBOOTSTRAP ?= "qemu-debootstrap"
@@ -21,7 +20,6 @@ ROOTFSDIR = "${WORKDIR}/rootfs"
 APTPREFS = "${WORKDIR}/apt-preferences"
 APTSRCS = "${WORKDIR}/apt-sources"
 APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
-BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
 DISTRO_BOOTSTRAP_KEYFILES = ""
 THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
@@ -265,13 +263,15 @@ isar_bootstrap() {
                              "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
             mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
             if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
-                sed -e "s#{BASE_DISTRO}#"${BASE_DISTRO}"#g" \
-                    -e "s#{BASE_DISTRO_CODENAME}#"${BASE_DISTRO_CODENAME}"#g" \
-	            -i ${BASEAPTSRCS}
+                line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+                if [ -z "${BASE_REPO_KEY}" ]; then
+                    line="[trusted=yes] ${line}"
+                fi
+                echo "deb ${line}" >  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+                echo "deb-src ${line}" >>  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+
                 mkdir -p ${ROOTFSDIR}/base-apt
                 sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
-                install -v -m644 "${BASEAPTSRCS}" \
-                                 "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
             else
                 install -v -m644 "${APTSRCS}" \
                                  "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-- 
2.23.0

[PATCH 06/17] ci: conf: add "hello" to the sample config and every build

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Now that base-apt supports deb-src we can include hello in a standard
build.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 4 +---
 scripts/ci_build.sh              | 1 -
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 7715ce8..eee8da3 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -166,11 +166,9 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs hello"
 ## "cache_base_repo" relies on the cache ...
 #IMAGE_INSTALL += "isar-disable-apt-cache"
-## uses apt:// SRC_URI, which does not work with the cache
-#IMAGE_INSTALL += "hello"
 
 #
 # Enable cross-compilation support
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index d88f46b..713d1c2 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -154,7 +154,6 @@ if [ -n "$REPRO_BUILD" ]; then
 fi
 
 sed -i -e 's/#IMAGE_INSTALL += "isar-disable-apt-cache"/IMAGE_INSTALL += "isar-disable-apt-cache"/g' conf/local.conf
-sed -i -e 's/#IMAGE_INSTALL += "hello"/IMAGE_INSTALL += "hello"/g' conf/local.conf
 
 echo 'IMAGE_INSTALL += "cowsay"' >> conf/local.conf
 
-- 
2.23.0

[PATCH 07/17] meta: split all apt-get invocations into download and execution

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit does not contain a functional change. We just split all
calls of "apt-get" into a download and an execution phase, so we can
later copy out the downloaded files.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-gbp.bbclass                 |  3 ++
 meta/classes/dpkg.bbclass                     |  5 ++-
 meta/classes/image-tools-extension.bbclass    |  5 +++
 .../buildchroot/files/deps.sh                 | 37 +++++++++++--------
 4 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 3977e63..d9c3af0 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -13,6 +13,9 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 do_install_builddeps_append() {
     dpkg_do_mounts
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
+    sudo -E chroot ${BUILDCHROOT_DIR} \
+        apt-get install -y -o Debug::pkgProblemResolver=yes \
+                        --no-install-recommends --download-only ${GBP_DEPENDS}
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 8b6ef21..d809cca 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -7,7 +7,10 @@ inherit dpkg-base
 do_install_builddeps() {
     dpkg_do_mounts
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh ${PP}/${PPS} ${DISTRO_ARCH}
+    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
+        ${PP}/${PPS} ${DISTRO_ARCH} --download-only
+    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
+        ${PP}/${PPS} ${DISTRO_ARCH}
     dpkg_undo_mounts
 }
 
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 7ea7557..243fa11 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -30,6 +30,11 @@ do_install_imager_deps() {
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
             -o Dir::Etc::SourceParts="-" \
             -o APT::Get::List-Cleanup="0"
+        apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
+            --allow-unauthenticated --allow-downgrades --download-only install \
+            ${IMAGER_INSTALL}'
+
+    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
             ${IMAGER_INSTALL}'
diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 002085b..8a6db8d 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -12,27 +12,34 @@ source /isar/common.sh
 #   2) we add -y to go non-interactive
 #   3) downgrades shall be allowed in case a package recipe was changed
 install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends \
-    -y --allow-downgrades"
+    -y --allow-downgrades $3"
 
-# Make sure that we have latest isar-apt content.
-# Options meaning:
-#   Dir::Etc::SourceList - specifies which source to be used
-#   Dir::Etc::SourceParts - disables looking for the other sources
-#   APT::Get::List-Cleanup - do not erase obsolete packages list for
-#                            upstream in '/var/lib/apt/lists'
-apt-get update \
-    -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-    -o Dir::Etc::SourceParts="-" \
-    -o APT::Get::List-Cleanup="0"
+if [ "$3" != "--download-only" ]; then
+    # Make sure that we have latest isar-apt content.
+    # Options meaning:
+    #   Dir::Etc::SourceList - specifies which source to be used
+    #   Dir::Etc::SourceParts - disables looking for the other sources
+    #   APT::Get::List-Cleanup - do not erase obsolete packages list for
+    #                            upstream in '/var/lib/apt/lists'
+    apt-get update \
+        -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
+        -o Dir::Etc::SourceParts="-" \
+        -o APT::Get::List-Cleanup="0"
+fi
 
 # Do not set an architecture when building only 'all' (generic) packages.
 # This can avoid unneeded cross-build issues.
 if ! grep "^Architecture:" debian/control | grep -qv "all"; then
-	set_arch=""
+    set_arch=""
 fi
 
 # Install all build deps
-mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
+if [ "$3" == "--download-only" ]; then
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control 2>&1 \
+        | grep "mk-build-deps: Unable to install all build-dep packages"
+else
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
 
-# Upgrade any already installed packages in case we are partially rebuilding
-apt-get upgrade -y --allow-downgrades
+    # Upgrade any already installed packages in case we are partially rebuilding
+    apt-get upgrade -y --allow-downgrades
+fi
-- 
2.23.0

[PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Hook in between the "apt-get --download-only"s and the "apt-get"s and
copy out all the debs for later.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass              | 14 ++++++++++++++
 meta/classes/dpkg-base.bbclass               |  1 +
 meta/classes/dpkg-gbp.bbclass                |  1 +
 meta/classes/dpkg.bbclass                    |  1 +
 meta/classes/image-locales-extension.bbclass |  2 +-
 meta/classes/image-tools-extension.bbclass   |  1 +
 meta/classes/rootfs.bbclass                  | 12 ++++++++++--
 meta/conf/bitbake.conf                       |  1 +
 8 files changed, 30 insertions(+), 3 deletions(-)
 create mode 100644 meta/classes/deb-dl-dir.bbclass

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
new file mode 100644
index 0000000..e58df7a
--- /dev/null
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -0,0 +1,14 @@
+# This software is a part of ISAR.
+# Copyright (C) 2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
+
+deb_dl_dir_export() {
+    export pc="${DEBDIR}/${DISTRO}"
+    export rootfs="${1}"
+    mkdir -p "${pc}"
+    flock "${pc}".lock -c ' \
+        sudo find "${rootfs}"/var/cache/apt/archives/ -type f -iname '*\.deb' \
+            -exec cp -f '{}' "${pc}" \;
+    '
+}
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index b39df5c..8815478 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -8,6 +8,7 @@ inherit buildchroot
 inherit debianize
 inherit terminal
 inherit repository
+inherit deb-dl-dir
 
 DEPENDS ?= ""
 
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d9c3af0..9e6945b 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -16,6 +16,7 @@ do_install_builddeps_append() {
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends --download-only ${GBP_DEPENDS}
+    deb_dl_dir_export "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d809cca..b6844e9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,6 +9,7 @@ do_install_builddeps() {
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${DISTRO_ARCH} --download-only
+    deb_dl_dir_export "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${DISTRO_ARCH}
     dpkg_undo_mounts
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index 3c0758f..0f0d0ca 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -25,7 +25,7 @@ def get_nopurge(d):
                                           j.split()[0].split(".")[0],
                                           j.split()[0]))))
 
-ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN += "image_install_localepurge_download"
+ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
 image_install_localepurge_download[weight] = "40"
 image_install_localepurge_download() {
     sudo -E chroot '${ROOTFSDIR}' \
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 243fa11..3906585 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -34,6 +34,7 @@ do_install_imager_deps() {
             --allow-unauthenticated --allow-downgrades --download-only install \
             ${IMAGER_INSTALL}'
 
+    deb_dl_dir_export ${BUILDCHROOT_DIR}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 095ebc1..841aa89 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -1,6 +1,8 @@
 # This software is a part of ISAR.
 # Copyright (c) Siemens AG, 2019
 
+inherit deb-dl-dir
+
 ROOTFS_ARCH ?= "${DISTRO_ARCH}"
 ROOTFS_DISTRO ?= "${DISTRO}"
 ROOTFS_PACKAGES ?= ""
@@ -119,8 +121,14 @@ rootfs_install_pkgs_download() {
         /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
 }
 
-ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= ""
-ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}"
+ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= ""
+ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
+
+ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
+rootfs_export_package_cache[weight] = "5"
+rootfs_export_package_cache() {
+    deb_dl_dir_export ${ROOTFSDIR}
+}
 
 ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
 rootfs_install_clean_files[weight] = "2"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 4c0a809..56f4fbf 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
 FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
 GITDIR = "${DL_DIR}/git"
+DEBDIR = "${DL_DIR}/deb"
 DEBSRCDIR = "${DL_DIR}/deb-src"
 P = "${PN}-${PV}"
 PF = "${PN}-${PV}-${PR}"
-- 
2.23.0

[PATCH 09/17] meta: import DL_DIR debs before apt-get download steps

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

If we ever downloaded a file there is no need to do that again. We can
not only use that DL_DIR to later construct a repository but also to
speed up our rebuilds.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass            | 10 ++++++++++
 meta/classes/dpkg-gbp.bbclass              |  1 +
 meta/classes/dpkg.bbclass                  |  1 +
 meta/classes/image-tools-extension.bbclass |  1 +
 meta/classes/rootfs.bbclass                |  6 ++++++
 5 files changed, 19 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e58df7a..b137eb9 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -3,6 +3,16 @@
 #
 # SPDX-License-Identifier: MIT
 
+deb_dl_dir_import() {
+    export pc="${DEBDIR}/${DISTRO}"
+    export rootfs="${1}"
+    [ ! -d "${pc}" ] && exit 0
+    flock -s "${pc}".lock -c ' \
+        sudo find "${pc}" -type f -iname '*\.deb' -exec \
+            cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \;
+    '
+}
+
 deb_dl_dir_export() {
     export pc="${DEBDIR}/${DISTRO}"
     export rootfs="${1}"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 9e6945b..be0769c 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -13,6 +13,7 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 do_install_builddeps_append() {
     dpkg_do_mounts
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
+    deb_dl_dir_import "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} \
         apt-get install -y -o Debug::pkgProblemResolver=yes \
                         --no-install-recommends --download-only ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index b6844e9..9e10ad8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -7,6 +7,7 @@ inherit dpkg-base
 do_install_builddeps() {
     dpkg_do_mounts
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
+    deb_dl_dir_import "${BUILDCHROOT_DIR}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${DISTRO_ARCH} --download-only
     deb_dl_dir_export "${BUILDCHROOT_DIR}"
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 3906585..08b426d 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,6 +25,7 @@ do_install_imager_deps() {
     buildchroot_do_mounts
 
     E="${@ bb.build.exec_func('isar_export_proxies', d)}"
+    deb_dl_dir_import ${BUILDCHROOT_DIR}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get update \
             -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 841aa89..845304c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -113,6 +113,12 @@ rootfs_install_resolvconf() {
     fi
 }
 
+ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
+rootfs_import_package_cache[weight] = "5"
+rootfs_import_package_cache() {
+    deb_dl_dir_import ${ROOTFSDIR}
+}
+
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
 rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
-- 
2.23.0

[PATCH 10/17] base-apt: populate from DEBDIR as well

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

We have all the .debs cached in DEBDIR now, take them from there because
the rootfss might not actually have all of them.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/image-cache-extension.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index f8b9ba3..07e9e7d 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -33,6 +33,10 @@ do_cache_base_repo() {
         populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
     fi
 
+    if [ -d "${DEBDIR}"/"${DISTRO}" ]; then
+        populate_base_apt "${DEBDIR}"/"${DISTRO}"
+    fi
+
     if [ -d "${DEBSRCDIR}"/"${DISTRO}" ]; then
         populate_base_apt "${DEBSRCDIR}"/"${DISTRO}"
     fi
-- 
2.23.0

[PATCH 11/17] base-apt: drop the "apt_cache" feature

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

We now have all debs in DEBDIR anyways, no need to conditionally copy
them out of the rootfss we create.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/image-cache-extension.bbclass |  4 ----
 meta/classes/image.bbclass                 |  2 +-
 meta/classes/rootfs.bbclass                | 10 ----------
 3 files changed, 1 insertion(+), 15 deletions(-)

diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index 07e9e7d..19c9c59 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -21,10 +21,6 @@ do_cache_base_repo() {
                 "Try it without cross-build."
     fi
 
-    if [ -d ${WORKDIR}/apt_cache ]; then
-        populate_base_apt ${WORKDIR}/apt_cache
-    fi
-
     if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
         populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
     fi
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 8384b71..eec36e5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,7 +61,7 @@ image_do_mounts() {
 }
 
 ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "copy-package-cache clean-package-cache finalize-rootfs generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"
 ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
 ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
 
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 845304c..b8629d8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -9,7 +9,6 @@ ROOTFS_PACKAGES ?= ""
 
 # Features of the rootfs creation:
 # available features are:
-# 'copy-package-cache' - copy the package cache ${WORKDIR}/apt_cache
 # 'clean-package-cache' - delete package cache from rootfs
 # 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR}
 # 'finalize-rootfs' - delete files needed to chroot into the rootfs
@@ -186,15 +185,6 @@ python do_rootfs_install() {
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
 
-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', 'rootfs_postprocess_copy_package_cache', '', d)}"
-rootfs_postprocess_copy_package_cache() {
-    mkdir -p '${WORKDIR}/apt_cache'
-    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \
-        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t '${WORKDIR}/apt_cache' '{}' '+'
-    me="$(id -u):$(id -g)"
-    sudo chown -R "$me" '${WORKDIR}/apt_cache'
-}
-
 ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
 rootfs_postprocess_clean_package_cache() {
     sudo -E chroot '${ROOTFSDIR}' \
-- 
2.23.0

[PATCH 12/17] base-apt: do not copy debs directly out of rootfss anymore

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The earlier commits make sure we already have all .debs in the repo, no
need to get them from these locations. In fact these locations are not
reliable anyways, see hook in "isar-disable-apt-cache".

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/image-cache-extension.bbclass | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index 19c9c59..cff2a8b 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -21,14 +21,6 @@ do_cache_base_repo() {
                 "Try it without cross-build."
     fi
 
-    if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
-        populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
-    fi
-
-    if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
-        populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
-    fi
-
     if [ -d "${DEBDIR}"/"${DISTRO}" ]; then
         populate_base_apt "${DEBDIR}"/"${DISTRO}"
     fi
-- 
2.23.0

[PATCH 13/17] base-apt: rework base-apt population

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Just call that function once and make it find the debs and dscs in one
run.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/base-apt-helper.bbclass       | 6 ++----
 meta/classes/image-cache-extension.bbclass | 8 +-------
 2 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
index 3013edf..fb4f0c8 100644
--- a/meta/classes/base-apt-helper.bbclass
+++ b/meta/classes/base-apt-helper.bbclass
@@ -17,9 +17,7 @@ compare_pkg_md5sums() {
 }
 
 populate_base_apt() {
-    search_dir=$1
-
-    find $search_dir -name '*.deb' | while read package; do
+    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
         # NOTE: due to packages stored by reprepro are not modified, we can
         # use search by filename to check if package is already in repo. In
         # addition, m5sums could be compared to ensure, that package is the
@@ -53,7 +51,7 @@ populate_base_apt() {
             "${package}"
     done
 
-    find $search_dir -name '*.dsc' | while read package; do
+    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
         repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
             "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
             "${BASE_DISTRO_CODENAME}" \
diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index cff2a8b..7945e8b 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -21,12 +21,6 @@ do_cache_base_repo() {
                 "Try it without cross-build."
     fi
 
-    if [ -d "${DEBDIR}"/"${DISTRO}" ]; then
-        populate_base_apt "${DEBDIR}"/"${DISTRO}"
-    fi
-
-    if [ -d "${DEBSRCDIR}"/"${DISTRO}" ]; then
-        populate_base_apt "${DEBSRCDIR}"/"${DISTRO}"
-    fi
+    populate_base_apt
 }
 addtask cache_base_repo after do_rootfs do_install_imager_deps
-- 
2.23.0

[PATCH 14/17] base-apt: move class "base-apt-helper" into only user

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

There is only one place we use that code, move it there instead of
having an extra class.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/base-apt-helper.bbclass       | 60 ----------------------
 meta/classes/image-cache-extension.bbclass | 55 +++++++++++++++++++-
 2 files changed, 54 insertions(+), 61 deletions(-)
 delete mode 100644 meta/classes/base-apt-helper.bbclass

diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
deleted file mode 100644
index fb4f0c8..0000000
--- a/meta/classes/base-apt-helper.bbclass
+++ /dev/null
@@ -1,60 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2018 ilbers GmbH
-# Copyright (C) 2019 Siemens AG
-#
-# SPDX-License-Identifier: MIT
-
-inherit repository
-
-compare_pkg_md5sums() {
-   pkg1=$1
-   pkg2=$2
-
-   md1=$(md5sum $pkg1 | cut -d ' ' -f 1)
-   md2=$(md5sum $pkg2 | cut -d ' ' -f 1)
-
-   [ "$md1" = "$md2" ]
-}
-
-populate_base_apt() {
-    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
-        # NOTE: due to packages stored by reprepro are not modified, we can
-        # use search by filename to check if package is already in repo. In
-        # addition, m5sums could be compared to ensure, that package is the
-        # same and should not be overwritten. This method is easier and more
-        # robust than querying reprepro by name.
-
-        # Check if this package is taken from Isar-apt, if so - ingore it.
-        base_name=${package##*/}
-        isar_apt_p=$(find ${REPO_ISAR_DIR}/${DISTRO} -name $base_name)
-        if [ -n "$isar_apt_p" ]; then
-            # Check if MD5 sums are identical. This helps to avoid the case
-            # when packages is overridden from another repo.
-            compare_pkg_md5sums "$package" "$isar_apt_p" && continue
-        fi
-
-        # Check if this package is already in base-apt
-        base_apt_p=$(find ${REPO_BASE_DIR}/${BASE_DISTRO} -name $base_name)
-        if [ -n "$base_apt_p" ]; then
-            compare_pkg_md5sums "$package" "$base_apt_p" && continue
-
-            # md5sum differs, so remove the package from base-apt
-            repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
-                "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
-                "${BASE_DISTRO_CODENAME}" \
-                "${base_apt_p}"
-        fi
-
-        repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
-            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
-            "${BASE_DISTRO_CODENAME}" \
-            "${package}"
-    done
-
-    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
-        repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
-            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
-            "${BASE_DISTRO_CODENAME}" \
-            "${package}"
-    done
-}
diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index 7945e8b..9dd7430 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -5,7 +5,60 @@
 #
 # This class extends the image.bbclass to supply the creation of cache repositories
 
-inherit base-apt-helper
+inherit repository
+
+compare_pkg_md5sums() {
+   pkg1=$1
+   pkg2=$2
+
+   md1=$(md5sum $pkg1 | cut -d ' ' -f 1)
+   md2=$(md5sum $pkg2 | cut -d ' ' -f 1)
+
+   [ "$md1" = "$md2" ]
+}
+
+populate_base_apt() {
+    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
+        # NOTE: due to packages stored by reprepro are not modified, we can
+        # use search by filename to check if package is already in repo. In
+        # addition, m5sums could be compared to ensure, that package is the
+        # same and should not be overwritten. This method is easier and more
+        # robust than querying reprepro by name.
+
+        # Check if this package is taken from Isar-apt, if so - ingore it.
+        base_name=${package##*/}
+        isar_apt_p=$(find ${REPO_ISAR_DIR}/${DISTRO} -name $base_name)
+        if [ -n "$isar_apt_p" ]; then
+            # Check if MD5 sums are identical. This helps to avoid the case
+            # when packages is overridden from another repo.
+            compare_pkg_md5sums "$package" "$isar_apt_p" && continue
+        fi
+
+        # Check if this package is already in base-apt
+        base_apt_p=$(find ${REPO_BASE_DIR}/${BASE_DISTRO} -name $base_name)
+        if [ -n "$base_apt_p" ]; then
+            compare_pkg_md5sums "$package" "$base_apt_p" && continue
+
+            # md5sum differs, so remove the package from base-apt
+            repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+                "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+                "${BASE_DISTRO_CODENAME}" \
+                "${base_apt_p}"
+        fi
+
+        repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+            "${BASE_DISTRO_CODENAME}" \
+            "${package}"
+    done
+
+    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
+        repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
+            "${BASE_DISTRO_CODENAME}" \
+            "${package}"
+    done
+}
 
 do_cache_base_repo[depends] = "base-apt:do_cache_config"
 do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
-- 
2.23.0

[PATCH 15/17] CI: include "isar-disable-apt-cache" into all CI images

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The base-apt does not rely on the rootfs-cache anymore, so we can test
our cleanup package in all images.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 4 +---
 scripts/ci_build.sh              | 2 --
 2 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index eee8da3..abed01e 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -166,9 +166,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs hello"
-## "cache_base_repo" relies on the cache ...
-#IMAGE_INSTALL += "isar-disable-apt-cache"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs isar-disable-apt-cache hello"
 
 #
 # Enable cross-compilation support
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 713d1c2..285f4d8 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -153,8 +153,6 @@ if [ -n "$REPRO_BUILD" ]; then
     sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
 fi
 
-sed -i -e 's/#IMAGE_INSTALL += "isar-disable-apt-cache"/IMAGE_INSTALL += "isar-disable-apt-cache"/g' conf/local.conf
-
 echo 'IMAGE_INSTALL += "cowsay"' >> conf/local.conf
 
 # Start cross build for the defined set of configurations
-- 
2.23.0

[PATCH 16/17] CI: include "cowsay" into default build to test dpkg-gbp

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 2 +-
 scripts/ci_build.sh              | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index abed01e..7c1338e 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -166,7 +166,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs isar-disable-apt-cache hello"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs isar-disable-apt-cache hello cowsay"
 
 #
 # Enable cross-compilation support
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 285f4d8..4f101c1 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -153,8 +153,6 @@ if [ -n "$REPRO_BUILD" ]; then
     sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
 fi
 
-echo 'IMAGE_INSTALL += "cowsay"' >> conf/local.conf
-
 # Start cross build for the defined set of configurations
 sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
 bitbake $BB_ARGS $CROSS_TARGETS_SET
-- 
2.23.0

[PATCH 17/17] CI: set BB_NO_NETWORK for cached rebuild

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Also add it to our sample config and comment on its use in conjuction
with base-apt.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 doc/user_manual.md               | 1 +
 meta-isar/conf/local.conf.sample | 4 ++++
 scripts/ci_build.sh              | 2 ++
 3 files changed, 7 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 34f50df..81fac43 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -809,6 +809,7 @@ bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
 ```
 # Uncomment this to enable use of cached base repository
 #ISAR_USE_CACHED_BASE_REPO ?= "1"
+#BB_NO_NETWORK ?= "1"
 ```
  - Remove build artifacts to use only local base-apt:
 
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 7c1338e..f90b6a6 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -176,6 +176,10 @@ ISAR_CROSS_COMPILE ?= "0"
 #
 # Uncomment this to enable use of cached base repository
 #ISAR_USE_CACHED_BASE_REPO ?= "1"
+#
+# You probably want to uncomment this as well to make sure the build
+# does not access the network
+#BB_NO_NETWORK ?= "1"
 
 # Set root password to 'root'
 # Password was encrypted using following command:
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 4f101c1..ae57a02 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -145,12 +145,14 @@ if [ -n "$REPRO_BUILD" ]; then
     while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    sed -i -e 's/^#BB_NO_NETWORK/BB_NO_NETWORK/g' conf/local.conf
     bitbake $BB_ARGS $REPRO_TARGETS_SET
     while [ -e bitbake.sock ]; do sleep 1; done
     # Cleanup and disable use of cached base repository
     sudo rm -rf tmp
     sed -i -e 's/ISAR_USE_CACHED_BASE_REPO ?= "1"/#ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
     sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf
+    sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
 fi
 
 # Start cross build for the defined set of configurations
-- 
2.23.0

Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[chombourger]

[-- Attachment #1.1: Type: text/plain, Size: 6449 bytes --]



On Monday, October 28, 2019 at 6:01:36 PM UTC+1, Henning Schild wrote:
>
> From: Henning Schild <hennin...@siemens.com <javascript:>> 
>
> Hook in between the "apt-get --download-only"s and the "apt-get"s and 
> copy out all the debs for later. 
>
> Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> --- 
>  meta/classes/deb-dl-dir.bbclass              | 14 ++++++++++++++ 
>  meta/classes/dpkg-base.bbclass               |  1 + 
>  meta/classes/dpkg-gbp.bbclass                |  1 + 
>  meta/classes/dpkg.bbclass                    |  1 + 
>  meta/classes/image-locales-extension.bbclass |  2 +- 
>  meta/classes/image-tools-extension.bbclass   |  1 + 
>  meta/classes/rootfs.bbclass                  | 12 ++++++++++-- 
>  meta/conf/bitbake.conf                       |  1 + 
>  8 files changed, 30 insertions(+), 3 deletions(-) 
>  create mode 100644 meta/classes/deb-dl-dir.bbclass 
>
> diff --git a/meta/classes/deb-dl-dir.bbclass 
> b/meta/classes/deb-dl-dir.bbclass 
> new file mode 100644 
> index 0000000..e58df7a 
> --- /dev/null 
> +++ b/meta/classes/deb-dl-dir.bbclass 
> @@ -0,0 +1,14 @@ 
> +# This software is a part of ISAR. 
> +# Copyright (C) 2019 Siemens AG 
> +# 
> +# SPDX-License-Identifier: MIT 
> + 
> +deb_dl_dir_export() { 
> +    export pc="${DEBDIR}/${DISTRO}" 
> +    export rootfs="${1}" 
> +    mkdir -p "${pc}" 
> +    flock "${pc}".lock -c ' \ 
> +        sudo find "${rootfs}"/var/cache/apt/archives/ -type f -iname 
> '*\.deb' \ 
> +            -exec cp -f '{}' "${pc}" \; 
>

use -exec command {} + to reduce the number of fork/execvp pairs?
note: you would need to use cp -t "${pc}" since the "+" modifier appends 
matches to the end of the command
 

> +    ' 
> +} 
> diff --git a/meta/classes/dpkg-base.bbclass 
> b/meta/classes/dpkg-base.bbclass 
> index b39df5c..8815478 100644 
> --- a/meta/classes/dpkg-base.bbclass 
> +++ b/meta/classes/dpkg-base.bbclass 
> @@ -8,6 +8,7 @@ inherit buildchroot 
>  inherit debianize 
>  inherit terminal 
>  inherit repository 
> +inherit deb-dl-dir 
>   
>  DEPENDS ?= "" 
>   
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass 
> index d9c3af0..9e6945b 100644 
> --- a/meta/classes/dpkg-gbp.bbclass 
> +++ b/meta/classes/dpkg-gbp.bbclass 
> @@ -16,6 +16,7 @@ do_install_builddeps_append() { 
>      sudo -E chroot ${BUILDCHROOT_DIR} \ 
>          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
>                          --no-install-recommends --download-only 
> ${GBP_DEPENDS} 
> +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
>      sudo -E chroot ${BUILDCHROOT_DIR} \ 
>          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
>                          --no-install-recommends ${GBP_DEPENDS} 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass 
> index d809cca..b6844e9 100644 
> --- a/meta/classes/dpkg.bbclass 
> +++ b/meta/classes/dpkg.bbclass 
> @@ -9,6 +9,7 @@ do_install_builddeps() { 
>      E="${@ bb.build.exec_func('isar_export_proxies', d)}" 
>      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
>          ${PP}/${PPS} ${DISTRO_ARCH} --download-only 
> +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
>      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
>          ${PP}/${PPS} ${DISTRO_ARCH} 
>      dpkg_undo_mounts 
> diff --git a/meta/classes/image-locales-extension.bbclass 
> b/meta/classes/image-locales-extension.bbclass 
> index 3c0758f..0f0d0ca 100644 
> --- a/meta/classes/image-locales-extension.bbclass 
> +++ b/meta/classes/image-locales-extension.bbclass 
> @@ -25,7 +25,7 @@ def get_nopurge(d): 
>                                            j.split()[0].split(".")[0], 
>                                            j.split()[0])))) 
>   
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN += 
> "image_install_localepurge_download" 
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += 
> "image_install_localepurge_download" 
>  image_install_localepurge_download[weight] = "40" 
>  image_install_localepurge_download() { 
>      sudo -E chroot '${ROOTFSDIR}' \ 
> diff --git a/meta/classes/image-tools-extension.bbclass 
> b/meta/classes/image-tools-extension.bbclass 
> index 243fa11..3906585 100644 
> --- a/meta/classes/image-tools-extension.bbclass 
> +++ b/meta/classes/image-tools-extension.bbclass 
> @@ -34,6 +34,7 @@ do_install_imager_deps() { 
>              --allow-unauthenticated --allow-downgrades --download-only 
> install \ 
>              ${IMAGER_INSTALL}' 
>   
> +    deb_dl_dir_export ${BUILDCHROOT_DIR} 
>      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ 
>          apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends 
> -y \ 
>              --allow-unauthenticated --allow-downgrades install \ 
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass 
> index 095ebc1..841aa89 100644 
> --- a/meta/classes/rootfs.bbclass 
> +++ b/meta/classes/rootfs.bbclass 
> @@ -1,6 +1,8 @@ 
>  # This software is a part of ISAR. 
>  # Copyright (c) Siemens AG, 2019 
>   
> +inherit deb-dl-dir 
> + 
>  ROOTFS_ARCH ?= "${DISTRO_ARCH}" 
>  ROOTFS_DISTRO ?= "${DISTRO}" 
>  ROOTFS_PACKAGES ?= "" 
> @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() { 
>          /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only 
> ${ROOTFS_PACKAGES} 
>  } 
>   
> -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= "" 
> -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}" 
> +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= "" 
> +ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}" 
> + 
> +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache" 
> +rootfs_export_package_cache[weight] = "5" 
> +rootfs_export_package_cache() { 
> +    deb_dl_dir_export ${ROOTFSDIR} 
> +} 
>   
>  ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if 
> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" 
>  rootfs_install_clean_files[weight] = "2" 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> index 4c0a809..56f4fbf 100644 
> --- a/meta/conf/bitbake.conf 
> +++ b/meta/conf/bitbake.conf 
> @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
>  FILESPATH = 
> "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
>
>  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
>  GITDIR = "${DL_DIR}/git" 
> +DEBDIR = "${DL_DIR}/deb" 
>  DEBSRCDIR = "${DL_DIR}/deb-src" 
>  P = "${PN}-${PV}" 
>  PF = "${PN}-${PV}-${PR}" 
> -- 
> 2.23.0 
>
>

[-- Attachment #1.2: Type: text/html, Size: 8373 bytes --]

Re: [PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[chombourger]

[-- Attachment #1.1: Type: text/plain, Size: 3015 bytes --]



On Monday, October 28, 2019 at 6:01:35 PM UTC+1, Henning Schild wrote:
>
> From: Henning Schild <hennin...@siemens.com <javascript:>> 
>
> Store the results of "apt-get source" in the DL_DIR. This means we 
> really only need to fetch once on incemental rebuilds or if multiple 
> recipes use the same SRC_URI. 
> We can also later collect the results and put them in a repository. 
>
> Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> --- 
>  meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++-- 
>  meta/conf/bitbake.conf         |  1 + 
>  2 files changed, 19 insertions(+), 2 deletions(-) 
>
> diff --git a/meta/classes/dpkg-base.bbclass 
> b/meta/classes/dpkg-base.bbclass 
> index 3f1e99f..b39df5c 100644 
> --- a/meta/classes/dpkg-base.bbclass 
> +++ b/meta/classes/dpkg-base.bbclass 
> @@ -37,14 +37,30 @@ do_apt_fetch() { 
>          -o Dir::Etc::SourceParts="-" \ 
>          -o APT::Get::List-Cleanup="0" 
>   
> -    sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ 
> -        sh -c 'cd ${PP} && apt-get -y --only-source source ${SRC_APT}' 
> +    for uri in "${SRC_APT}"; do 
> +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> ${BUILDCHROOT_DIR} \ 
> +            sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd 
> /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source 
> source "$2"' my_script "${DISTRO}" "${uri}" 
> +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> ${BUILDCHROOT_DIR} \ 
> +            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd ${PP} && 
> apt-get -y --only-source source "$2"' my_script "${DISTRO}" "${uri}" 
> +    done 
> + 
>      dpkg_undo_mounts 
>  } 
>   
>  addtask apt_fetch after do_unpack before do_patch 
>  do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" 
>   
> +addtask cleanall_apt before do_cleanall 
> +do_cleanall_apt[nostamp] = "1" 
> +do_cleanall_apt() { 
> +    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then 
> +        exit 
>

shouldn't this be "return 0" instead?
(or are we really wanting to exit the bitbake process here?
 

> +    fi 
> +    for uri in "${SRC_APT}"; do 
> +        rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri" 
> +    done 
> +} 
> + 
>  def get_package_srcdir(d): 
>      s = os.path.abspath(d.getVar("S", True)) 
>      workdir = os.path.abspath(d.getVar("WORKDIR", True)) 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> index bca1114..4c0a809 100644 
> --- a/meta/conf/bitbake.conf 
> +++ b/meta/conf/bitbake.conf 
> @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
>  FILESPATH = 
> "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
>
>  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
>  GITDIR = "${DL_DIR}/git" 
> +DEBSRCDIR = "${DL_DIR}/deb-src" 
>  P = "${PN}-${PV}" 
>  PF = "${PN}-${PV}-${PR}" 
>  PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] 
> or 'defaultpkgname'}" 
> -- 
> 2.23.0 
>
>

[-- Attachment #1.2: Type: text/html, Size: 4439 bytes --]

Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[Henning Schild]

On Tue, 29 Oct 2019 05:48:01 -0700
<chombourger@gmail.com> wrote:

> On Monday, October 28, 2019 at 6:01:36 PM UTC+1, Henning Schild wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>> 
> >
> > Hook in between the "apt-get --download-only"s and the "apt-get"s
> > and copy out all the debs for later. 
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> > --- 
> >  meta/classes/deb-dl-dir.bbclass              | 14 ++++++++++++++ 
> >  meta/classes/dpkg-base.bbclass               |  1 + 
> >  meta/classes/dpkg-gbp.bbclass                |  1 + 
> >  meta/classes/dpkg.bbclass                    |  1 + 
> >  meta/classes/image-locales-extension.bbclass |  2 +- 
> >  meta/classes/image-tools-extension.bbclass   |  1 + 
> >  meta/classes/rootfs.bbclass                  | 12 ++++++++++-- 
> >  meta/conf/bitbake.conf                       |  1 + 
> >  8 files changed, 30 insertions(+), 3 deletions(-) 
> >  create mode 100644 meta/classes/deb-dl-dir.bbclass 
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass 
> > b/meta/classes/deb-dl-dir.bbclass 
> > new file mode 100644 
> > index 0000000..e58df7a 
> > --- /dev/null 
> > +++ b/meta/classes/deb-dl-dir.bbclass 
> > @@ -0,0 +1,14 @@ 
> > +# This software is a part of ISAR. 
> > +# Copyright (C) 2019 Siemens AG 
> > +# 
> > +# SPDX-License-Identifier: MIT 
> > + 
> > +deb_dl_dir_export() { 
> > +    export pc="${DEBDIR}/${DISTRO}" 
> > +    export rootfs="${1}" 
> > +    mkdir -p "${pc}" 
> > +    flock "${pc}".lock -c ' \ 
> > +        sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > -iname '*\.deb' \ 
> > +            -exec cp -f '{}' "${pc}" \; 
> >  
> 
> use -exec command {} + to reduce the number of fork/execvp pairs?
> note: you would need to use cp -t "${pc}" since the "+" modifier
> appends matches to the end of the command

Thanks, this was actually on my mind when i wrote it but eventually
overlooked.
I will look into it.

Henning

> 
> > +    ' 
> > +} 
> > diff --git a/meta/classes/dpkg-base.bbclass 
> > b/meta/classes/dpkg-base.bbclass 
> > index b39df5c..8815478 100644 
> > --- a/meta/classes/dpkg-base.bbclass 
> > +++ b/meta/classes/dpkg-base.bbclass 
> > @@ -8,6 +8,7 @@ inherit buildchroot 
> >  inherit debianize 
> >  inherit terminal 
> >  inherit repository 
> > +inherit deb-dl-dir 
> >   
> >  DEPENDS ?= "" 
> >   
> > diff --git a/meta/classes/dpkg-gbp.bbclass
> > b/meta/classes/dpkg-gbp.bbclass index d9c3af0..9e6945b 100644 
> > --- a/meta/classes/dpkg-gbp.bbclass 
> > +++ b/meta/classes/dpkg-gbp.bbclass 
> > @@ -16,6 +16,7 @@ do_install_builddeps_append() { 
> >      sudo -E chroot ${BUILDCHROOT_DIR} \ 
> >          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
> >                          --no-install-recommends --download-only 
> > ${GBP_DEPENDS} 
> > +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} \ 
> >          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
> >                          --no-install-recommends ${GBP_DEPENDS} 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass 
> > index d809cca..b6844e9 100644 
> > --- a/meta/classes/dpkg.bbclass 
> > +++ b/meta/classes/dpkg.bbclass 
> > @@ -9,6 +9,7 @@ do_install_builddeps() { 
> >      E="${@ bb.build.exec_func('isar_export_proxies', d)}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
> >          ${PP}/${PPS} ${DISTRO_ARCH} --download-only 
> > +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
> >          ${PP}/${PPS} ${DISTRO_ARCH} 
> >      dpkg_undo_mounts 
> > diff --git a/meta/classes/image-locales-extension.bbclass 
> > b/meta/classes/image-locales-extension.bbclass 
> > index 3c0758f..0f0d0ca 100644 
> > --- a/meta/classes/image-locales-extension.bbclass 
> > +++ b/meta/classes/image-locales-extension.bbclass 
> > @@ -25,7 +25,7 @@ def get_nopurge(d): 
> >                                            j.split()[0].split(".")[0], 
> >                                            j.split()[0])))) 
> >   
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN += 
> > "image_install_localepurge_download" 
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += 
> > "image_install_localepurge_download" 
> >  image_install_localepurge_download[weight] = "40" 
> >  image_install_localepurge_download() { 
> >      sudo -E chroot '${ROOTFSDIR}' \ 
> > diff --git a/meta/classes/image-tools-extension.bbclass 
> > b/meta/classes/image-tools-extension.bbclass 
> > index 243fa11..3906585 100644 
> > --- a/meta/classes/image-tools-extension.bbclass 
> > +++ b/meta/classes/image-tools-extension.bbclass 
> > @@ -34,6 +34,7 @@ do_install_imager_deps() { 
> >              --allow-unauthenticated --allow-downgrades
> > --download-only install \ 
> >              ${IMAGER_INSTALL}' 
> >   
> > +    deb_dl_dir_export ${BUILDCHROOT_DIR} 
> >      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ 
> >          apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \ 
> >              --allow-unauthenticated --allow-downgrades install \ 
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 095ebc1..841aa89 100644 
> > --- a/meta/classes/rootfs.bbclass 
> > +++ b/meta/classes/rootfs.bbclass 
> > @@ -1,6 +1,8 @@ 
> >  # This software is a part of ISAR. 
> >  # Copyright (c) Siemens AG, 2019 
> >   
> > +inherit deb-dl-dir 
> > + 
> >  ROOTFS_ARCH ?= "${DISTRO_ARCH}" 
> >  ROOTFS_DISTRO ?= "${DISTRO}" 
> >  ROOTFS_PACKAGES ?= "" 
> > @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() { 
> >          /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only 
> > ${ROOTFS_PACKAGES} 
> >  } 
> >   
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= "" 
> > -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}" 
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= "" 
> > +ROOTFS_INSTALL_COMMAND +=
> > "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}" 
> > + 
> > +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache" 
> > +rootfs_export_package_cache[weight] = "5" 
> > +rootfs_export_package_cache() { 
> > +    deb_dl_dir_export ${ROOTFSDIR} 
> > +} 
> >   
> >  ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if 
> > (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" 
> >  rootfs_install_clean_files[weight] = "2" 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> > index 4c0a809..56f4fbf 100644 
> > --- a/meta/conf/bitbake.conf 
> > +++ b/meta/conf/bitbake.conf 
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
> >  FILESPATH = 
> > "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
> >
> >  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
> >  GITDIR = "${DL_DIR}/git" 
> > +DEBDIR = "${DL_DIR}/deb" 
> >  DEBSRCDIR = "${DL_DIR}/deb-src" 
> >  P = "${PN}-${PV}" 
> >  PF = "${PN}-${PV}-${PR}" 
> > -- 
> > 2.23.0 
> >
> >  
> 

Re: [PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[Henning Schild]

On Tue, 29 Oct 2019 05:57:49 -0700
<chombourger@gmail.com> wrote:

> On Monday, October 28, 2019 at 6:01:35 PM UTC+1, Henning Schild wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>> 
> >
> > Store the results of "apt-get source" in the DL_DIR. This means we 
> > really only need to fetch once on incemental rebuilds or if
> > multiple recipes use the same SRC_URI. 
> > We can also later collect the results and put them in a repository. 
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> > --- 
> >  meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++-- 
> >  meta/conf/bitbake.conf         |  1 + 
> >  2 files changed, 19 insertions(+), 2 deletions(-) 
> >
> > diff --git a/meta/classes/dpkg-base.bbclass 
> > b/meta/classes/dpkg-base.bbclass 
> > index 3f1e99f..b39df5c 100644 
> > --- a/meta/classes/dpkg-base.bbclass 
> > +++ b/meta/classes/dpkg-base.bbclass 
> > @@ -37,14 +37,30 @@ do_apt_fetch() { 
> >          -o Dir::Etc::SourceParts="-" \ 
> >          -o APT::Get::List-Cleanup="0" 
> >   
> > -    sudo -E chroot --userspec=$( id -u ):$( id -g )
> > ${BUILDCHROOT_DIR} \ 
> > -        sh -c 'cd ${PP} && apt-get -y --only-source source
> > ${SRC_APT}' 
> > +    for uri in "${SRC_APT}"; do 
> > +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > ${BUILDCHROOT_DIR} \ 
> > +            sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd 
> > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only
> > --only-source source "$2"' my_script "${DISTRO}" "${uri}" 
> > +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > ${BUILDCHROOT_DIR} \ 
> > +            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> > ${PP} && apt-get -y --only-source source "$2"' my_script
> > "${DISTRO}" "${uri}" 
> > +    done 
> > + 
> >      dpkg_undo_mounts 
> >  } 
> >   
> >  addtask apt_fetch after do_unpack before do_patch 
> >  do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" 
> >   
> > +addtask cleanall_apt before do_cleanall 
> > +do_cleanall_apt[nostamp] = "1" 
> > +do_cleanall_apt() { 
> > +    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then 
> > +        exit 
> >  
> 
> shouldn't this be "return 0" instead?
> (or are we really wanting to exit the bitbake process here?

I am pretty sure i tested it and bitbake survived. But i will
double-check or maybe just return 0 anyways.

Henning

> 
> > +    fi 
> > +    for uri in "${SRC_APT}"; do 
> > +        rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri" 
> > +    done 
> > +} 
> > + 
> >  def get_package_srcdir(d): 
> >      s = os.path.abspath(d.getVar("S", True)) 
> >      workdir = os.path.abspath(d.getVar("WORKDIR", True)) 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> > index bca1114..4c0a809 100644 
> > --- a/meta/conf/bitbake.conf 
> > +++ b/meta/conf/bitbake.conf 
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
> >  FILESPATH = 
> > "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
> >
> >  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
> >  GITDIR = "${DL_DIR}/git" 
> > +DEBSRCDIR = "${DL_DIR}/deb-src" 
> >  P = "${PN}-${PV}" 
> >  PF = "${PN}-${PV}-${PR}" 
> >  PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
> > False),d)[0] or 'defaultpkgname'}" 
> > -- 
> > 2.23.0 
> >
> >  
> 

Re: [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps

[Henning Schild]

A future patch will actually populate base-apt out of the files in
DL_DIR, i did not sent that yet.

Once the creation+population is done purely from the download cache, we
can not exclude packages from isar-apt anymore.

At the moment i think that would not be a problem because the rebuild
from base-apt will just work. isar-apt will always win and just rebuild
the "same" stuff.

The only problem i see is that base-apt will potentially contain
product packages, which means you can not share it as widely as you can
a pure "mirror", which base-apt is not anyways.

Henning

On Mon, 28 Oct 2019 18:01:24 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> If we ever downloaded a file there is no need to do that again. We can
> not only use that DL_DIR to later construct a repository but also to
> speed up our rebuilds.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/deb-dl-dir.bbclass            | 10 ++++++++++
>  meta/classes/dpkg-gbp.bbclass              |  1 +
>  meta/classes/dpkg.bbclass                  |  1 +
>  meta/classes/image-tools-extension.bbclass |  1 +
>  meta/classes/rootfs.bbclass                |  6 ++++++
>  5 files changed, 19 insertions(+)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index e58df7a..b137eb9 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -3,6 +3,16 @@
>  #
>  # SPDX-License-Identifier: MIT
>  
> +deb_dl_dir_import() {
> +    export pc="${DEBDIR}/${DISTRO}"
> +    export rootfs="${1}"
> +    [ ! -d "${pc}" ] && exit 0
> +    flock -s "${pc}".lock -c ' \
> +        sudo find "${pc}" -type f -iname '*\.deb' -exec \
> +            cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \;
> +    '
> +}
> +
>  deb_dl_dir_export() {
>      export pc="${DEBDIR}/${DISTRO}"
>      export rootfs="${1}"
> diff --git a/meta/classes/dpkg-gbp.bbclass
> b/meta/classes/dpkg-gbp.bbclass index 9e6945b..be0769c 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -13,6 +13,7 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>  do_install_builddeps_append() {
>      dpkg_do_mounts
>      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> +    deb_dl_dir_import "${BUILDCHROOT_DIR}"
>      sudo -E chroot ${BUILDCHROOT_DIR} \
>          apt-get install -y -o Debug::pkgProblemResolver=yes \
>                          --no-install-recommends --download-only
> ${GBP_DEPENDS} diff --git a/meta/classes/dpkg.bbclass
> b/meta/classes/dpkg.bbclass index b6844e9..9e10ad8 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -7,6 +7,7 @@ inherit dpkg-base
>  do_install_builddeps() {
>      dpkg_do_mounts
>      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> +    deb_dl_dir_import "${BUILDCHROOT_DIR}"
>      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
>          ${PP}/${PPS} ${DISTRO_ARCH} --download-only
>      deb_dl_dir_export "${BUILDCHROOT_DIR}"
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass index 3906585..08b426d
> 100644 --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -25,6 +25,7 @@ do_install_imager_deps() {
>      buildchroot_do_mounts
>  
>      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> +    deb_dl_dir_import ${BUILDCHROOT_DIR}
>      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
>          apt-get update \
>              -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 841aa89..845304c 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -113,6 +113,12 @@ rootfs_install_resolvconf() {
>      fi
>  }
>  
> +ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
> +rootfs_import_package_cache[weight] = "5"
> +rootfs_import_package_cache() {
> +    deb_dl_dir_import ${ROOTFSDIR}
> +}
> +
>  ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
>  rootfs_install_pkgs_download[weight] = "600"
>  rootfs_install_pkgs_download[isar-apt-lock] = "release-after"

Re: [PATCH 04/17] base-apt: add deb-src packages as well

[Henning Schild]

There is a small section in the user-manual claiming that this does not
work, it needs to be removed in the next version of this commit.

Henning

On Mon, 28 Oct 2019 18:01:19 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> So far the deb-src packages downloaded via "apt://" where not cached.
> This commit fixes that and includes those packages into the cache.
> 14230
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/base-apt-helper.bbclass       |  7 +++++++
>  meta/classes/image-cache-extension.bbclass |  4 ++++
>  meta/classes/repository.bbclass            | 14 ++++++++++++++
>  3 files changed, 25 insertions(+)
> 
> diff --git a/meta/classes/base-apt-helper.bbclass
> b/meta/classes/base-apt-helper.bbclass index 567eaea..3013edf 100644
> --- a/meta/classes/base-apt-helper.bbclass
> +++ b/meta/classes/base-apt-helper.bbclass
> @@ -52,4 +52,11 @@ populate_base_apt() {
>              "${BASE_DISTRO_CODENAME}" \
>              "${package}"
>      done
> +
> +    find $search_dir -name '*.dsc' | while read package; do
> +        repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> +            "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> +            "${BASE_DISTRO_CODENAME}" \
> +            "${package}"
> +    done
>  }
> diff --git a/meta/classes/image-cache-extension.bbclass
> b/meta/classes/image-cache-extension.bbclass index 30db549..f8b9ba3
> 100644 --- a/meta/classes/image-cache-extension.bbclass
> +++ b/meta/classes/image-cache-extension.bbclass
> @@ -32,5 +32,9 @@ do_cache_base_repo() {
>      if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
>          populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
>      fi
> +
> +    if [ -d "${DEBSRCDIR}"/"${DISTRO}" ]; then
> +        populate_base_apt "${DEBSRCDIR}"/"${DISTRO}"
> +    fi
>  }
>  addtask cache_base_repo after do_rootfs do_install_imager_deps
> diff --git a/meta/classes/repository.bbclass
> b/meta/classes/repository.bbclass index afd3aed..22d1f93 100644
> --- a/meta/classes/repository.bbclass
> +++ b/meta/classes/repository.bbclass
> @@ -30,6 +30,20 @@ EOF
>      fi
>  }
>  
> +repo_add_srcpackage() {
> +    dir="$1"
> +    dbdir="$2"
> +    codename="$3"
> +    shift; shift; shift
> +
> +    if [ -n "${GNUPGHOME}" ]; then
> +        export GNUPGHOME="${GNUPGHOME}"
> +    fi
> +    reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
> +        includedsc "${codename}" \
> +        "$@"
> +}
> +
>  repo_add_packages() {
>      dir="$1"
>      dbdir="$2"

Re: [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps

[Henning Schild]

Am Wed, 30 Oct 2019 20:21:38 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> A future patch will actually populate base-apt out of the files in
> DL_DIR, i did not sent that yet.
> 
> Once the creation+population is done purely from the download cache,
> we can not exclude packages from isar-apt anymore.
> 
> At the moment i think that would not be a problem because the rebuild
> from base-apt will just work. isar-apt will always win and just
> rebuild the "same" stuff.

In fact the debs coming from isar-apt should never be extracted into
the download cache, because from an isar point of view they have never
been downloaded.

Henning

> The only problem i see is that base-apt will potentially contain
> product packages, which means you can not share it as widely as you
> can a pure "mirror", which base-apt is not anyways.
> 
> Henning
> 
> On Mon, 28 Oct 2019 18:01:24 +0100
> Henning Schild <henning.schild@siemens.com> wrote:
> 
> > From: Henning Schild <henning.schild@siemens.com>
> > 
> > If we ever downloaded a file there is no need to do that again. We
> > can not only use that DL_DIR to later construct a repository but
> > also to speed up our rebuilds.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  meta/classes/deb-dl-dir.bbclass            | 10 ++++++++++
> >  meta/classes/dpkg-gbp.bbclass              |  1 +
> >  meta/classes/dpkg.bbclass                  |  1 +
> >  meta/classes/image-tools-extension.bbclass |  1 +
> >  meta/classes/rootfs.bbclass                |  6 ++++++
> >  5 files changed, 19 insertions(+)
> > 
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass index e58df7a..b137eb9 100644
> > --- a/meta/classes/deb-dl-dir.bbclass
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -3,6 +3,16 @@
> >  #
> >  # SPDX-License-Identifier: MIT
> >  
> > +deb_dl_dir_import() {
> > +    export pc="${DEBDIR}/${DISTRO}"
> > +    export rootfs="${1}"
> > +    [ ! -d "${pc}" ] && exit 0
> > +    flock -s "${pc}".lock -c ' \
> > +        sudo find "${pc}" -type f -iname '*\.deb' -exec \
> > +            cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \;
> > +    '
> > +}
> > +
> >  deb_dl_dir_export() {
> >      export pc="${DEBDIR}/${DISTRO}"
> >      export rootfs="${1}"
> > diff --git a/meta/classes/dpkg-gbp.bbclass
> > b/meta/classes/dpkg-gbp.bbclass index 9e6945b..be0769c 100644
> > --- a/meta/classes/dpkg-gbp.bbclass
> > +++ b/meta/classes/dpkg-gbp.bbclass
> > @@ -13,6 +13,7 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
> >  do_install_builddeps_append() {
> >      dpkg_do_mounts
> >      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> > +    deb_dl_dir_import "${BUILDCHROOT_DIR}"
> >      sudo -E chroot ${BUILDCHROOT_DIR} \
> >          apt-get install -y -o Debug::pkgProblemResolver=yes \
> >                          --no-install-recommends --download-only
> > ${GBP_DEPENDS} diff --git a/meta/classes/dpkg.bbclass
> > b/meta/classes/dpkg.bbclass index b6844e9..9e10ad8 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -7,6 +7,7 @@ inherit dpkg-base
> >  do_install_builddeps() {
> >      dpkg_do_mounts
> >      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> > +    deb_dl_dir_import "${BUILDCHROOT_DIR}"
> >      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
> >          ${PP}/${PPS} ${DISTRO_ARCH} --download-only
> >      deb_dl_dir_export "${BUILDCHROOT_DIR}"
> > diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass index 3906585..08b426d
> > 100644 --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -25,6 +25,7 @@ do_install_imager_deps() {
> >      buildchroot_do_mounts
> >  
> >      E="${@ bb.build.exec_func('isar_export_proxies', d)}"
> > +    deb_dl_dir_import ${BUILDCHROOT_DIR}
> >      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> >          apt-get update \
> >              -o Dir::Etc::SourceList="sources.list.d/isar-apt.list"
> > \ diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 841aa89..845304c 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -113,6 +113,12 @@ rootfs_install_resolvconf() {
> >      fi
> >  }
> >  
> > +ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
> > +rootfs_import_package_cache[weight] = "5"
> > +rootfs_import_package_cache() {
> > +    deb_dl_dir_import ${ROOTFSDIR}
> > +}
> > +
> >  ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
> >  rootfs_install_pkgs_download[weight] = "600"
> >  rootfs_install_pkgs_download[isar-apt-lock] = "release-after"  
> 

Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[Henning Schild]

Am Tue, 29 Oct 2019 05:48:01 -0700
schrieb <chombourger@gmail.com>:

> On Monday, October 28, 2019 at 6:01:36 PM UTC+1, Henning Schild wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>> 
> >
> > Hook in between the "apt-get --download-only"s and the "apt-get"s
> > and copy out all the debs for later. 
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> > --- 
> >  meta/classes/deb-dl-dir.bbclass              | 14 ++++++++++++++ 
> >  meta/classes/dpkg-base.bbclass               |  1 + 
> >  meta/classes/dpkg-gbp.bbclass                |  1 + 
> >  meta/classes/dpkg.bbclass                    |  1 + 
> >  meta/classes/image-locales-extension.bbclass |  2 +- 
> >  meta/classes/image-tools-extension.bbclass   |  1 + 
> >  meta/classes/rootfs.bbclass                  | 12 ++++++++++-- 
> >  meta/conf/bitbake.conf                       |  1 + 
> >  8 files changed, 30 insertions(+), 3 deletions(-) 
> >  create mode 100644 meta/classes/deb-dl-dir.bbclass 
> >
> > diff --git a/meta/classes/deb-dl-dir.bbclass 
> > b/meta/classes/deb-dl-dir.bbclass 
> > new file mode 100644 
> > index 0000000..e58df7a 
> > --- /dev/null 
> > +++ b/meta/classes/deb-dl-dir.bbclass 
> > @@ -0,0 +1,14 @@ 
> > +# This software is a part of ISAR. 
> > +# Copyright (C) 2019 Siemens AG 
> > +# 
> > +# SPDX-License-Identifier: MIT 
> > + 
> > +deb_dl_dir_export() { 
> > +    export pc="${DEBDIR}/${DISTRO}" 
> > +    export rootfs="${1}" 
> > +    mkdir -p "${pc}" 
> > +    flock "${pc}".lock -c ' \ 
> > +        sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > -iname '*\.deb' \ 
> > +            -exec cp -f '{}' "${pc}" \; 
> >  
> 
> use -exec command {} + to reduce the number of fork/execvp pairs?
> note: you would need to use cp -t "${pc}" since the "+" modifier
> appends matches to the end of the command

In fact this guy will need to get more complicated, since it should
skip all files that came from isar-apt. They are not "downloads" in the
overall Isar picture and they should also not become part of a base-apt
we later generate from the downloads.

So i think each iteration will become more costly and we can not do
many at once.

Henning

> 
> > +    ' 
> > +} 
> > diff --git a/meta/classes/dpkg-base.bbclass 
> > b/meta/classes/dpkg-base.bbclass 
> > index b39df5c..8815478 100644 
> > --- a/meta/classes/dpkg-base.bbclass 
> > +++ b/meta/classes/dpkg-base.bbclass 
> > @@ -8,6 +8,7 @@ inherit buildchroot 
> >  inherit debianize 
> >  inherit terminal 
> >  inherit repository 
> > +inherit deb-dl-dir 
> >   
> >  DEPENDS ?= "" 
> >   
> > diff --git a/meta/classes/dpkg-gbp.bbclass
> > b/meta/classes/dpkg-gbp.bbclass index d9c3af0..9e6945b 100644 
> > --- a/meta/classes/dpkg-gbp.bbclass 
> > +++ b/meta/classes/dpkg-gbp.bbclass 
> > @@ -16,6 +16,7 @@ do_install_builddeps_append() { 
> >      sudo -E chroot ${BUILDCHROOT_DIR} \ 
> >          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
> >                          --no-install-recommends --download-only 
> > ${GBP_DEPENDS} 
> > +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} \ 
> >          apt-get install -y -o Debug::pkgProblemResolver=yes \ 
> >                          --no-install-recommends ${GBP_DEPENDS} 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass 
> > index d809cca..b6844e9 100644 
> > --- a/meta/classes/dpkg.bbclass 
> > +++ b/meta/classes/dpkg.bbclass 
> > @@ -9,6 +9,7 @@ do_install_builddeps() { 
> >      E="${@ bb.build.exec_func('isar_export_proxies', d)}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
> >          ${PP}/${PPS} ${DISTRO_ARCH} --download-only 
> > +    deb_dl_dir_export "${BUILDCHROOT_DIR}" 
> >      sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \ 
> >          ${PP}/${PPS} ${DISTRO_ARCH} 
> >      dpkg_undo_mounts 
> > diff --git a/meta/classes/image-locales-extension.bbclass 
> > b/meta/classes/image-locales-extension.bbclass 
> > index 3c0758f..0f0d0ca 100644 
> > --- a/meta/classes/image-locales-extension.bbclass 
> > +++ b/meta/classes/image-locales-extension.bbclass 
> > @@ -25,7 +25,7 @@ def get_nopurge(d): 
> >                                            j.split()[0].split(".")[0], 
> >                                            j.split()[0])))) 
> >   
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN += 
> > "image_install_localepurge_download" 
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += 
> > "image_install_localepurge_download" 
> >  image_install_localepurge_download[weight] = "40" 
> >  image_install_localepurge_download() { 
> >      sudo -E chroot '${ROOTFSDIR}' \ 
> > diff --git a/meta/classes/image-tools-extension.bbclass 
> > b/meta/classes/image-tools-extension.bbclass 
> > index 243fa11..3906585 100644 
> > --- a/meta/classes/image-tools-extension.bbclass 
> > +++ b/meta/classes/image-tools-extension.bbclass 
> > @@ -34,6 +34,7 @@ do_install_imager_deps() { 
> >              --allow-unauthenticated --allow-downgrades
> > --download-only install \ 
> >              ${IMAGER_INSTALL}' 
> >   
> > +    deb_dl_dir_export ${BUILDCHROOT_DIR} 
> >      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \ 
> >          apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \ 
> >              --allow-unauthenticated --allow-downgrades install \ 
> > diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 095ebc1..841aa89 100644 
> > --- a/meta/classes/rootfs.bbclass 
> > +++ b/meta/classes/rootfs.bbclass 
> > @@ -1,6 +1,8 @@ 
> >  # This software is a part of ISAR. 
> >  # Copyright (c) Siemens AG, 2019 
> >   
> > +inherit deb-dl-dir 
> > + 
> >  ROOTFS_ARCH ?= "${DISTRO_ARCH}" 
> >  ROOTFS_DISTRO ?= "${DISTRO}" 
> >  ROOTFS_PACKAGES ?= "" 
> > @@ -119,8 +121,14 @@ rootfs_install_pkgs_download() { 
> >          /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only 
> > ${ROOTFS_PACKAGES} 
> >  } 
> >   
> > -ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN ??= "" 
> > -ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN}" 
> > +ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT ??= "" 
> > +ROOTFS_INSTALL_COMMAND +=
> > "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}" 
> > + 
> > +ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache" 
> > +rootfs_export_package_cache[weight] = "5" 
> > +rootfs_export_package_cache() { 
> > +    deb_dl_dir_export ${ROOTFSDIR} 
> > +} 
> >   
> >  ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if 
> > (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" 
> >  rootfs_install_clean_files[weight] = "2" 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> > index 4c0a809..56f4fbf 100644 
> > --- a/meta/conf/bitbake.conf 
> > +++ b/meta/conf/bitbake.conf 
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
> >  FILESPATH = 
> > "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
> >
> >  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
> >  GITDIR = "${DL_DIR}/git" 
> > +DEBDIR = "${DL_DIR}/deb" 
> >  DEBSRCDIR = "${DL_DIR}/deb-src" 
> >  P = "${PN}-${PV}" 
> >  PF = "${PN}-${PV}-${PR}" 
> > -- 
> > 2.23.0 
> >
> >  
> 

Re: [PATCH 00/17] base-apt rework

[Jan Kiszka]

On 28.10.19 18:01, [ext] Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> This series build upon "[PATCHv2 0/9] a few cleanups a bug and a
> feature".
> 
> base-apt has two main issues that this series fixes:
>  - it relied on the rootfs /var/cache/apt to contain all *.debs
>   - this is only true of nobody cleaned up, could happen with hooks
>     like used in "isar-disable-apt-cache" or similar mechanisms
>  - it did not handle "dsc"s (source packages)
> 
> It had other minor issues that are described in the individual patches.
> 
> The series fixes those issues and extends CI to fully test base-apt.
> 
> The general mechanism used in the end is:
>  - split all apt-gets into a download and an execution phase
>  - copy out all download inbetween the two phases
>  - execution phase can now go ahead and possibly delete cached files
>    from the rootfs
>  - actually copy in everything "old" before the download phase
>  - that means the download phase potentially becomes a noop and we speed
>    up by using already downloaded files (new feature)
> 
> There are a few open points that i will mention later.
> 

Looks good from a higher perspective, but I guess we will have to play
with it. Are you preparing a v2 with all pending issues already? I'd
like to feed this into a could of scenarios, besides CI of course.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[Henning Schild]

Am Tue, 29 Oct 2019 05:57:49 -0700
schrieb <chombourger@gmail.com>:

> On Monday, October 28, 2019 at 6:01:35 PM UTC+1, Henning Schild wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascript:>> 
> >
> > Store the results of "apt-get source" in the DL_DIR. This means we 
> > really only need to fetch once on incemental rebuilds or if
> > multiple recipes use the same SRC_URI. 
> > We can also later collect the results and put them in a repository. 
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com <javascript:>> 
> > --- 
> >  meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++-- 
> >  meta/conf/bitbake.conf         |  1 + 
> >  2 files changed, 19 insertions(+), 2 deletions(-) 
> >
> > diff --git a/meta/classes/dpkg-base.bbclass 
> > b/meta/classes/dpkg-base.bbclass 
> > index 3f1e99f..b39df5c 100644 
> > --- a/meta/classes/dpkg-base.bbclass 
> > +++ b/meta/classes/dpkg-base.bbclass 
> > @@ -37,14 +37,30 @@ do_apt_fetch() { 
> >          -o Dir::Etc::SourceParts="-" \ 
> >          -o APT::Get::List-Cleanup="0" 
> >   
> > -    sudo -E chroot --userspec=$( id -u ):$( id -g )
> > ${BUILDCHROOT_DIR} \ 
> > -        sh -c 'cd ${PP} && apt-get -y --only-source source
> > ${SRC_APT}' 
> > +    for uri in "${SRC_APT}"; do 
> > +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > ${BUILDCHROOT_DIR} \ 
> > +            sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd 
> > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only
> > --only-source source "$2"' my_script "${DISTRO}" "${uri}" 
> > +        sudo -E chroot --userspec=$( id -u ):$( id -g ) 
> > ${BUILDCHROOT_DIR} \ 
> > +            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd
> > ${PP} && apt-get -y --only-source source "$2"' my_script
> > "${DISTRO}" "${uri}" 
> > +    done 
> > + 
> >      dpkg_undo_mounts 
> >  } 
> >   
> >  addtask apt_fetch after do_unpack before do_patch 
> >  do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" 
> >   
> > +addtask cleanall_apt before do_cleanall 
> > +do_cleanall_apt[nostamp] = "1" 
> > +do_cleanall_apt() { 
> > +    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then 
> > +        exit 
> >  
> 
> shouldn't this be "return 0" instead?
> (or are we really wanting to exit the bitbake process here?

This is just a copy of the same return condition in the fetching
function. I will "return 0" both implementation in another patch.

Henning

> 
> > +    fi 
> > +    for uri in "${SRC_APT}"; do 
> > +        rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri" 
> > +    done 
> > +} 
> > + 
> >  def get_package_srcdir(d): 
> >      s = os.path.abspath(d.getVar("S", True)) 
> >      workdir = os.path.abspath(d.getVar("WORKDIR", True)) 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf 
> > index bca1114..4c0a809 100644 
> > --- a/meta/conf/bitbake.conf 
> > +++ b/meta/conf/bitbake.conf 
> > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" 
> >  FILESPATH = 
> > "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}" 
> >
> >  FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" 
> >  GITDIR = "${DL_DIR}/git" 
> > +DEBSRCDIR = "${DL_DIR}/deb-src" 
> >  P = "${PN}-${PV}" 
> >  PF = "${PN}-${PV}-${PR}" 
> >  PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
> > False),d)[0] or 'defaultpkgname'}" 
> > -- 
> > 2.23.0 
> >
> >  
> 

Re: [PATCH 00/17] base-apt rework

[Baurzhan Ismagulov]

Hello Henning,

On Mon, Oct 28, 2019 at 06:01:15PM +0100, Henning Schild wrote:
> base-apt has two main issues that this series fixes:
>  - it relied on the rootfs /var/cache/apt to contain all *.debs
>   - this is only true of nobody cleaned up, could happen with hooks
>     like used in "isar-disable-apt-cache" or similar mechanisms
>  - it did not handle "dsc"s (source packages)
> 
> It had other minor issues that are described in the individual patches.
> 
> The series fixes those issues and extends CI to fully test base-apt.
> 
> The general mechanism used in the end is:
>  - split all apt-gets into a download and an execution phase
>  - copy out all download inbetween the two phases
>  - execution phase can now go ahead and possibly delete cached files
>    from the rootfs
>  - actually copy in everything "old" before the download phase
>  - that means the download phase potentially becomes a noop and we speed
>    up by using already downloaded files (new feature)

Thanks for the series and the clear annotation. While I haven't yet checked
every detail, overall, I think this is a good step. I think proper separation
of base-apt and isar-apt is important; I hope we manage to find a way to handle
that. I'll be sending other comments to the respective patches.

With kind regards,
Baurzhan.

Re: [PATCH 01/17] repository: new class to deal with repos

[Baurzhan Ismagulov]

On Mon, Oct 28, 2019 at 06:01:16PM +0100, Henning Schild wrote:
> diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
...
> +repo_create() {
...
> +    if [ ! -f "${dir}"/conf/distributions ]; then
> +        mkdir -p "${dir}"/conf/
> +        cat <<EOF > "${dir}"/conf/distributions
> +Codename: ${codename}
> +Architectures: i386 armhf arm64 amd64 mipsel source
> +Components: main
> +EOF

I suggest to keep modifying distributions.in and avoid hard-coding
architectures and components that are specific to the current Debian.

With kind regards,
Baurzhan.

Re: [PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[Baurzhan Ismagulov]

On Mon, Oct 28, 2019 at 06:01:17PM +0100, Henning Schild wrote:
> Store the results of "apt-get source" in the DL_DIR. This means we
> really only need to fetch once on incemental rebuilds or if multiple
> recipes use the same SRC_URI.
> We can also later collect the results and put them in a repository.

Suggest "incemental" -> "incremental".

With kind regards,
Baurzhan.

Re: [PATCH 05/17] base-apt: do not skip gpg check when it is signed

[Baurzhan Ismagulov]

On Mon, Oct 28, 2019 at 06:01:20PM +0100, Henning Schild wrote:
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
...
> -BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
...
>              if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> -                sed -e "s#{BASE_DISTRO}#"${BASE_DISTRO}"#g" \
> -                    -e "s#{BASE_DISTRO_CODENAME}#"${BASE_DISTRO_CODENAME}"#g" \
> -	            -i ${BASEAPTSRCS}
> +                line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
> +                if [ -z "${BASE_REPO_KEY}" ]; then
> +                    line="[trusted=yes] ${line}"
> +                fi
> +                echo "deb ${line}" >  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> +                echo "deb-src ${line}" >>  "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"

The same question w.r.t. hard-coding vs. operating on data depending on the
config. Suggest keeping base-apt-sources, renaming it to base-apt-sources.in on
this occasion.

With kind regards,
Baurzhan.

Re: [PATCH 07/17] meta: split all apt-get invocations into download and execution

[Baurzhan Ismagulov]

On Mon, Oct 28, 2019 at 06:01:22PM +0100, Henning Schild wrote:
> diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
> index 7ea7557..243fa11 100644
> --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -30,6 +30,11 @@ do_install_imager_deps() {
>              -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
>              -o Dir::Etc::SourceParts="-" \
>              -o APT::Get::List-Cleanup="0"
> +        apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
> +            --allow-unauthenticated --allow-downgrades --download-only install \
> +            ${IMAGER_INSTALL}'
> +
> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \

Is the first apt-get (and the preceding statement) indented one level deeper
than necessary? Suggest fixing the original while at it.


> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
...
> +if [ "$3" == "--download-only" ]; then

"==" isn't POSIX and fails e.g. in dash. Suggest "=".


With kind regards,
Baurzhan.

Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[Baurzhan Ismagulov]

On Mon, Oct 28, 2019 at 06:01:23PM +0100, Henning Schild wrote:
> diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
...
> +    flock "${pc}".lock -c ' \
> +        sudo find "${rootfs}"/var/cache/apt/archives/ -type f -iname '*\.deb' \
> +            -exec cp -f '{}' "${pc}" \;
> +    '

If this stays in the code, some thoughts:

* The trailing escapes seem to be unnecessary with single quotes.

* Escaping dot in a glob should be unnecessary.

* The single quotes around "{}" seem to close and re-open the flock '...'
  argument. So, the "{}" seems to be not quoted and seems to work fine. find(1)
  does recommend quoting "{}", but I failed to identify a case where it would
  be interpreted by the shell. I'd suggest to drop quoting unless I overlook
  anything.

With kind regards,
Baurzhan.

Re: [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps

[Baurzhan Ismagulov]

On Wed, Nov 06, 2019 at 02:17:09PM +0100, Henning Schild wrote:
> In fact the debs coming from isar-apt should never be extracted into
> the download cache, because from an isar point of view they have never
> been downloaded.

Does this mean you have a solution for that?


> > > diff --git a/meta/classes/deb-dl-dir.bbclass
...
> > > +deb_dl_dir_import() {
> > > +    export pc="${DEBDIR}/${DISTRO}"
> > > +    export rootfs="${1}"
> > > +    [ ! -d "${pc}" ] && exit 0
> > > +    flock -s "${pc}".lock -c ' \
> > > +        sudo find "${pc}" -type f -iname '*\.deb' -exec \
> > > +            cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \;
> > > +    '
> > > +}

Does this honor ISAR_USE_CACHED_BASE_REPO?


With kind regards,
Baurzhan.

Re: [PATCH 09/17] meta: import DL_DIR debs before apt-get download steps

[Henning Schild]

On Sun, 24 Nov 2019 21:20:05 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Wed, Nov 06, 2019 at 02:17:09PM +0100, Henning Schild wrote:
> > In fact the debs coming from isar-apt should never be extracted into
> > the download cache, because from an isar point of view they have
> > never been downloaded.  
> 
> Does this mean you have a solution for that?

I have it in mind but did not implement it yet. The idea is to not
extract debs that are coming from isar-apt into the DL cache. Match by
name+md5 just like the current detection mechanism does.

> 
> > > > diff --git a/meta/classes/deb-dl-dir.bbclass  
> ...
> > > > +deb_dl_dir_import() {
> > > > +    export pc="${DEBDIR}/${DISTRO}"
> > > > +    export rootfs="${1}"
> > > > +    [ ! -d "${pc}" ] && exit 0
> > > > +    flock -s "${pc}".lock -c ' \
> > > > +        sudo find "${pc}" -type f -iname '*\.deb' -exec \
> > > > +            cp -f '{}' "${rootfs}"/var/cache/apt/archives/ \;
> > > > +    '
> > > > +}  
> 
> Does this honor ISAR_USE_CACHED_BASE_REPO?

No and i do not see how to involve that variable here. We just copy all
debs we know into the cache of the rootfs, which ones it will actually
install depends on which repos it uses.
So even if a ton of debs become potentially available, only the one
from the base-apt repo Packages.gz can be installed in practice.

Henning

> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH 08/17] meta: create DL_DIR support for all apt-get downloaded .debs

[Henning Schild]

On Sun, 24 Nov 2019 16:57:42 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Oct 28, 2019 at 06:01:23PM +0100, Henning Schild wrote:
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass  
> ...
> > +    flock "${pc}".lock -c ' \
> > +        sudo find "${rootfs}"/var/cache/apt/archives/ -type f
> > -iname '*\.deb' \
> > +            -exec cp -f '{}' "${pc}" \;
> > +    '  
> 
> If this stays in the code, some thoughts:

It will not stay. This loop is where i plan to detect whether the debs
came from isar-apt and exclude them from extraction. They are not
downloaded and should not be cached as such.

Henning

> * The trailing escapes seem to be unnecessary with single quotes.
> 
> * Escaping dot in a glob should be unnecessary.
> 
> * The single quotes around "{}" seem to close and re-open the flock
> '...' argument. So, the "{}" seems to be not quoted and seems to work
> fine. find(1) does recommend quoting "{}", but I failed to identify a
> case where it would be interpreted by the shell. I'd suggest to drop
> quoting unless I overlook anything.
> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH 07/17] meta: split all apt-get invocations into download and execution

[Henning Schild]

On Sun, 24 Nov 2019 16:41:24 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Oct 28, 2019 at 06:01:22PM +0100, Henning Schild wrote:
> > diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass index 7ea7557..243fa11
> > 100644 --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -30,6 +30,11 @@ do_install_imager_deps() {
> >              -o Dir::Etc::SourceList="sources.list.d/isar-apt.list"
> > \ -o Dir::Etc::SourceParts="-" \
> >              -o APT::Get::List-Cleanup="0"
> > +        apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \
> > +            --allow-unauthenticated --allow-downgrades
> > --download-only install \
> > +            ${IMAGER_INSTALL}'
> > +
> > +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \  
> 
> Is the first apt-get (and the preceding statement) indented one level
> deeper than necessary? Suggest fixing the original while at it.

Yes, they are both under the same "sudo".

> 
> > diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> > b/meta/recipes-devtools/buildchroot/files/deps.sh  
> ...
> > +if [ "$3" == "--download-only" ]; then  
> 
> "==" isn't POSIX and fails e.g. in dash. Suggest "=".

Right, thanks.

Henning

> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH 05/17] base-apt: do not skip gpg check when it is signed

[Henning Schild]

On Sun, 24 Nov 2019 16:35:04 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Oct 28, 2019 at 06:01:20PM +0100, Henning Schild wrote:
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc  
> ...
> > -BASEAPTSRCS = "${WORKDIR}/base-apt-sources"  
> ...
> >              if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> > -                sed -e "s#{BASE_DISTRO}#"${BASE_DISTRO}"#g" \
> > -                    -e
> > "s#{BASE_DISTRO_CODENAME}#"${BASE_DISTRO_CODENAME}"#g" \
> > -	            -i ${BASEAPTSRCS}
> > +                line="file:///base-apt/${BASE_DISTRO}
> > ${BASE_DISTRO_CODENAME} main"
> > +                if [ -z "${BASE_REPO_KEY}" ]; then
> > +                    line="[trusted=yes] ${line}"
> > +                fi
> > +                echo "deb ${line}" >
> > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> > +                echo "deb-src ${line}" >>
> > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"  
> 
> The same question w.r.t. hard-coding vs. operating on data depending
> on the config. Suggest keeping base-apt-sources, renaming it to
> base-apt-sources.in on this occasion.

I guess you are talking about the template file vs. the echos? I am not
sure the file can actually be overridden, because BASEAPTSRCS is
assigned with "=". But a lot of things are possible within bitbake ...

As far as i understand it, the template file was never supposed to be
an interface for users to change things, but just an implementation of
"hard-coding". But in an imho not very readable way. Since there is
just one user i decided to pull it inside that one.

If it is an interface to tweak, renaming the template to .in would also
break expectations.

Henning

> With kind regards,
> Baurzhan.
> 

Re: [PATCH 02/17] dpkg-base: add download caching of apt:// downloads

[Henning Schild]

On Sun, 24 Nov 2019 16:30:07 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Oct 28, 2019 at 06:01:17PM +0100, Henning Schild wrote:
> > Store the results of "apt-get source" in the DL_DIR. This means we
> > really only need to fetch once on incemental rebuilds or if multiple
> > recipes use the same SRC_URI.
> > We can also later collect the results and put them in a repository.
> >  
> 
> Suggest "incemental" -> "incremental".

Thanks.

Henning

> With kind regards,
> Baurzhan.
> 

Re: [PATCH 01/17] repository: new class to deal with repos

[Henning Schild]

On Sun, 24 Nov 2019 16:29:18 +0100
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Mon, Oct 28, 2019 at 06:01:16PM +0100, Henning Schild wrote:
> > diff --git a/meta/classes/repository.bbclass
> > b/meta/classes/repository.bbclass  
> ...
> > +repo_create() {  
> ...
> > +    if [ ! -f "${dir}"/conf/distributions ]; then
> > +        mkdir -p "${dir}"/conf/
> > +        cat <<EOF > "${dir}"/conf/distributions
> > +Codename: ${codename}
> > +Architectures: i386 armhf arm64 amd64 mipsel source
> > +Components: main
> > +EOF  
> 
> I suggest to keep modifying distributions.in and avoid hard-coding
> architectures and components that are specific to the current Debian.

Same comment as before. As far as i know this is not a visible
interface and was never meant to be changed by i.e. a layer.
Distributing the code over multiple files, with just one place that
uses it, is a layer of abstraction that just makes the code less
readable.

That is why i pulled that file in.

Will keep it like that for the next series. Let us discuss further on
how to proceed.

Henning

> With kind regards,
> Baurzhan.
>