[PATCH 0/2] Improve apt-fetch related tasks

[PATCH 0/2] Improve apt-fetch related tasks

[Jan Kiszka]

Cleans up the scope and avoids execution (and locking) via noexec when
not needed.

Jan

Jan Kiszka (2):
  base: Move SRC_ATP extraction/filtering to dpkg-base class
  dpkg: Use noexec to disable unneeded apt tasks

 meta/classes/base.bbclass      | 19 -------------------
 meta/classes/dpkg-base.bbclass | 33 ++++++++++++++++++++++-----------
 2 files changed, 22 insertions(+), 30 deletions(-)

-- 
2.31.1

[PATCH 1/2] base: Move SRC_ATP extraction/filtering to dpkg-base class

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

apt:// SRC_URI entries only make sense for dpkg recipes. So move the
filtering over so that any attempt to use it for other types or recipes
is detected early and that related code is located in the same classes.s

Drop redundant expand=True from getVar at this chance. It's default.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/base.bbclass      | 19 -------------------
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 2 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b7de921..72d4cc0 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -138,25 +138,6 @@ def isar_export_proxies(d):
 
     return bb.utils.export_proxies(d)
 
-# filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
-python() {
-    src_uri = (d.getVar('SRC_URI', True) or "").split()
-    if len(src_uri) == 0:
-        return
-
-    prefix = "apt://"
-    new_src_uri = []
-    src_apt = []
-    for u in src_uri:
-        if u.startswith(prefix):
-            src_apt.append(u[len(prefix) :])
-        else:
-            new_src_uri.append(u)
-
-    d.setVar('SRC_URI', ' '.join(new_src_uri))
-    d.prependVar('SRC_APT', ' '.join(src_apt))
-}
-
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[vardeps] += "SRCREV"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 8286168..1cac68b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -73,6 +73,25 @@ addtask patch after do_adjust_git before do_dpkg_build
 
 SRC_APT ?= ""
 
+# filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
+python() {
+    src_uri = (d.getVar('SRC_URI') or "").split()
+    if len(src_uri) == 0:
+        return
+
+    prefix = "apt://"
+    new_src_uri = []
+    src_apt = []
+    for u in src_uri:
+        if u.startswith(prefix):
+            src_apt.append(u[len(prefix) :])
+        else:
+            new_src_uri.append(u)
+
+    d.setVar('SRC_URI', ' '.join(new_src_uri))
+    d.prependVar('SRC_APT', ' '.join(src_apt))
+}
+
 fetch_apt() {
     sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
         -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
-- 
2.31.1

[PATCH 2/2] dpkg: Use noexec to disable unneeded apt tasks

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This avoids needless waiting for isar.lock in do_apt_fetch.

Unfortunately, using an even simpler inline python function for setting
task[noexec] does not work, even when setting expand to false for
SRC_APT. So move this to the anonymous python function as well.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 1cac68b..1f2bebf 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -76,8 +76,6 @@ SRC_APT ?= ""
 # filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
 python() {
     src_uri = (d.getVar('SRC_URI') or "").split()
-    if len(src_uri) == 0:
-        return
 
     prefix = "apt://"
     new_src_uri = []
@@ -90,6 +88,11 @@ python() {
 
     d.setVar('SRC_URI', ' '.join(new_src_uri))
     d.prependVar('SRC_APT', ' '.join(src_apt))
+
+    if d.getVar('SRC_APT').strip() == '':
+        d.setVarFlag('do_apt_fetch', 'noexec', '1')
+        d.setVarFlag('do_apt_unpack', 'noexec', '1')
+        d.setVarFlag('do_cleanall_apt', 'noexec', '1')
 }
 
 fetch_apt() {
@@ -105,10 +108,6 @@ fetch_apt() {
 }
 
 python do_apt_fetch() {
-    src_apt = d.getVar("SRC_APT", True)
-    if not src_apt:
-        return 0
-
     dpkg_do_mounts(d)
     try:
         isar_export_proxies(d)
@@ -139,10 +138,6 @@ unpack_apt() {
 python do_apt_unpack() {
     import shutil
 
-    src_apt = d.getVar("SRC_APT", True)
-    if not src_apt:
-        return 0
-
     srcsubdir = d.getVar('S', True)
     if os.path.exists(srcsubdir):
         shutil.rmtree(srcsubdir)
@@ -160,9 +155,6 @@ addtask apt_unpack after do_apt_fetch before do_patch
 addtask cleanall_apt before do_cleanall
 do_cleanall_apt[nostamp] = "1"
 do_cleanall_apt() {
-    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
-        return 0
-    fi
     for uri in "${SRC_APT}"; do
         rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri"
     done
-- 
2.31.1

Re: [PATCH 1/2] base: Move SRC_ATP extraction/filtering to dpkg-base class

[Henning Schild]

Subject typo

SRC_ATP/SRC_APT

I guess i thought of it as a generic feature that people might want to
use also outside of "building packages".

It could be used by layers to fetch all sources listed in a manifest.
Or something alike. But as long as such code does not exist or is not
in Isar, i guess that move can be done.

Henning

Am Thu, 26 Aug 2021 11:23:28 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> apt:// SRC_URI entries only make sense for dpkg recipes. So move the
> filtering over so that any attempt to use it for other types or
> recipes is detected early and that related code is located in the
> same classes.s
> 
> Drop redundant expand=True from getVar at this chance. It's default.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/base.bbclass      | 19 -------------------
>  meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
>  2 files changed, 19 insertions(+), 19 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index b7de921..72d4cc0 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -138,25 +138,6 @@ def isar_export_proxies(d):
>  
>      return bb.utils.export_proxies(d)
>  
> -# filter out all "apt://" URIs out of SRC_URI and stick them into
> SRC_APT -python() {
> -    src_uri = (d.getVar('SRC_URI', True) or "").split()
> -    if len(src_uri) == 0:
> -        return
> -
> -    prefix = "apt://"
> -    new_src_uri = []
> -    src_apt = []
> -    for u in src_uri:
> -        if u.startswith(prefix):
> -            src_apt.append(u[len(prefix) :])
> -        else:
> -            new_src_uri.append(u)
> -
> -    d.setVar('SRC_URI', ' '.join(new_src_uri))
> -    d.prependVar('SRC_APT', ' '.join(src_apt))
> -}
> -
>  do_fetch[dirs] = "${DL_DIR}"
>  do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
>  do_fetch[vardeps] += "SRCREV"
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 8286168..1cac68b 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -73,6 +73,25 @@ addtask patch after do_adjust_git before
> do_dpkg_build 
>  SRC_APT ?= ""
>  
> +# filter out all "apt://" URIs out of SRC_URI and stick them into
> SRC_APT +python() {
> +    src_uri = (d.getVar('SRC_URI') or "").split()
> +    if len(src_uri) == 0:
> +        return
> +
> +    prefix = "apt://"
> +    new_src_uri = []
> +    src_apt = []
> +    for u in src_uri:
> +        if u.startswith(prefix):
> +            src_apt.append(u[len(prefix) :])
> +        else:
> +            new_src_uri.append(u)
> +
> +    d.setVar('SRC_URI', ' '.join(new_src_uri))
> +    d.prependVar('SRC_APT', ' '.join(src_apt))
> +}
> +
>  fetch_apt() {
>      sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
>          -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \