[PATCH v2 0/2] Improve apt-fetch related tasks

[PATCH v2 0/2] Improve apt-fetch related tasks

[Jan Kiszka]

Changes in v2:
 - rebased over next
 - fix typo in patch subject

Cleans up the scope and avoids execution (and locking) via noexec when
not needed.

Jan

Jan Kiszka (2):
  base: Move SRC_APT extraction/filtering to dpkg-base class
  dpkg: Use noexec to disable unneeded apt tasks

 meta/classes/base.bbclass      | 19 -------------------
 meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++++++++---------
 2 files changed, 22 insertions(+), 28 deletions(-)

-- 
2.31.1

[PATCH v2 1/2] base: Move SRC_APT extraction/filtering to dpkg-base class

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

apt:// SRC_URI entries only make sense for dpkg recipes. So move the
filtering over so that any attempt to use it for other types or recipes
is detected early and that related code is located in the same classes.s

Drop redundant expand=True from getVar at this chance. It's default.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass      | 19 -------------------
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 2 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b7de921..72d4cc0 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -138,25 +138,6 @@ def isar_export_proxies(d):
 
     return bb.utils.export_proxies(d)
 
-# filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
-python() {
-    src_uri = (d.getVar('SRC_URI', True) or "").split()
-    if len(src_uri) == 0:
-        return
-
-    prefix = "apt://"
-    new_src_uri = []
-    src_apt = []
-    for u in src_uri:
-        if u.startswith(prefix):
-            src_apt.append(u[len(prefix) :])
-        else:
-            new_src_uri.append(u)
-
-    d.setVar('SRC_URI', ' '.join(new_src_uri))
-    d.prependVar('SRC_APT', ' '.join(src_apt))
-}
-
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[vardeps] += "SRCREV"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index d8c0995..dfa31ff 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -73,6 +73,25 @@ addtask patch after do_adjust_git before do_dpkg_build
 
 SRC_APT ?= ""
 
+# filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
+python() {
+    src_uri = (d.getVar('SRC_URI') or "").split()
+    if len(src_uri) == 0:
+        return
+
+    prefix = "apt://"
+    new_src_uri = []
+    src_apt = []
+    for u in src_uri:
+        if u.startswith(prefix):
+            src_apt.append(u[len(prefix) :])
+        else:
+            new_src_uri.append(u)
+
+    d.setVar('SRC_URI', ' '.join(new_src_uri))
+    d.prependVar('SRC_APT', ' '.join(src_apt))
+}
+
 do_apt_fetch() {
     if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
         return 0
-- 
2.31.1

[PATCH v2 2/2] dpkg: Use noexec to disable unneeded apt tasks

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids needless waiting for isar.lock in do_apt_fetch.

Unfortunately, using an even simpler inline python function for setting
task[noexec] does not work, even when setting expand to false for
SRC_APT. So move this to the anonymous python function as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index dfa31ff..83500da 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -76,8 +76,6 @@ SRC_APT ?= ""
 # filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
 python() {
     src_uri = (d.getVar('SRC_URI') or "").split()
-    if len(src_uri) == 0:
-        return
 
     prefix = "apt://"
     new_src_uri = []
@@ -90,12 +88,14 @@ python() {
 
     d.setVar('SRC_URI', ' '.join(new_src_uri))
     d.prependVar('SRC_APT', ' '.join(src_apt))
+
+    if d.getVar('SRC_APT').strip() == '':
+        d.setVarFlag('do_apt_fetch', 'noexec', '1')
+        d.setVarFlag('do_apt_unpack', 'noexec', '1')
+        d.setVarFlag('do_cleanall_apt', 'noexec', '1')
 }
 
 do_apt_fetch() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
     sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
@@ -118,9 +118,6 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
 
 do_apt_unpack() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
     rm -rf ${S}
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
@@ -144,9 +141,6 @@ addtask apt_unpack after do_apt_fetch before do_patch
 addtask cleanall_apt before do_cleanall
 do_cleanall_apt[nostamp] = "1"
 do_cleanall_apt() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
     for uri in "${SRC_APT}"; do
         rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri"
     done
-- 
2.31.1

Re: [PATCH v2 0/2] Improve apt-fetch related tasks

[Anton Mikanovich]

06.09.2021 21:45, Jan Kiszka wrote:
> Changes in v2:
>   - rebased over next
>   - fix typo in patch subject
>
> Cleans up the scope and avoids execution (and locking) via noexec when
> not needed.
>
> Jan
>
> Jan Kiszka (2):
>    base: Move SRC_APT extraction/filtering to dpkg-base class
>    dpkg: Use noexec to disable unneeded apt tasks
>
>   meta/classes/base.bbclass      | 19 -------------------
>   meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++++++++---------
>   2 files changed, 22 insertions(+), 28 deletions(-)
>
Applied to next, thanks.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov