[PATCH v2 0/4] Fix SRC_URI:remove ignoring

[PATCH v2 0/4] Fix SRC_URI:remove ignoring

[Anton Mikanovich]

SRC_URI:remove override now got lost due to manual SRC_URI manipulations.
Add example for CI and few fixes to make this test case pass.

Changes since v1:
 - Move test case to meta-test.
 - Fix leading space on append.
 - Improve wording.

Anton Mikanovich (4):
  conf: Enable bbappends
  meta-test: Add examples of SRC_URI overrides usage
  dpkg-base: Fix SRC_URI:remove ignoring
  multiarch: Fix SRC_URI:remove ignoring

 meta-isar/conf/layer.conf                            | 2 +-
 meta-test/conf/layer.conf                            | 2 +-
 meta-test/recipes-app/hello-isar/hello-isar.bbappend | 9 +++++++++
 meta/classes/dpkg-base.bbclass                       | 5 +----
 meta/classes/multiarch.bbclass                       | 5 +++--
 meta/conf/layer.conf                                 | 2 +-
 6 files changed, 16 insertions(+), 9 deletions(-)
 create mode 100644 meta-test/recipes-app/hello-isar/hello-isar.bbappend

-- 
2.34.1

[PATCH v2 1/4] conf: Enable bbappends

[Anton Mikanovich]

Enable usage of bbappend files in all the layers.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-isar/conf/layer.conf | 2 +-
 meta-test/conf/layer.conf | 2 +-
 meta/conf/layer.conf      | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index dec2658f..2209b18b 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -5,7 +5,7 @@
 BBPATH .= ":${LAYERDIR}"
 
 # We have recipes-* directories, add to BBFILES
-BBFILES +=  "${LAYERDIR}/recipes-*/*/*.bb"
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
 
 BBFILE_COLLECTIONS += "isar"
 BBFILE_PATTERN_isar = "^${LAYERDIR}/"
diff --git a/meta-test/conf/layer.conf b/meta-test/conf/layer.conf
index 07b5c77d..0f15119d 100644
--- a/meta-test/conf/layer.conf
+++ b/meta-test/conf/layer.conf
@@ -5,7 +5,7 @@
 BBPATH .= ":${LAYERDIR}"
 
 # We have recipes-* directories, add to BBFILES
-BBFILES +=  "${LAYERDIR}/recipes-*/*/*.bb"
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
 
 BBFILE_COLLECTIONS += "test"
 BBFILE_PATTERN_test = "^${LAYERDIR}/"
diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
index a38ab97d..b622ce69 100644
--- a/meta/conf/layer.conf
+++ b/meta/conf/layer.conf
@@ -5,7 +5,7 @@
 BBPATH .= ":${LAYERDIR}"
 
 # We have recipes-* directories, add to BBFILES
-BBFILES +=  "${LAYERDIR}/recipes-*/*/*.bb"
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"
 
 BBFILE_COLLECTIONS += "core"
 BBFILE_PATTERN_core = "^${LAYERDIR}/"
-- 
2.34.1

[PATCH v2 2/4] meta-test: Add examples of SRC_URI overrides usage

[Anton Mikanovich]

To check SRC_URI variable manipulations add some append and remove
override examples.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-test/recipes-app/hello-isar/hello-isar.bbappend | 9 +++++++++
 1 file changed, 9 insertions(+)
 create mode 100644 meta-test/recipes-app/hello-isar/hello-isar.bbappend

diff --git a/meta-test/recipes-app/hello-isar/hello-isar.bbappend b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
new file mode 100644
index 00000000..d567599f
--- /dev/null
+++ b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
@@ -0,0 +1,9 @@
+# Examples for SRC_URI parser testing
+SRC_URI += " \
+    file://nonexist-file \
+"
+SRC_URI:append = " \
+    git://nonexist-git \
+"
+SRC_URI:remove = "file://nonexist-file"
+SRC_URI:remove = "git://nonexist-git"
-- 
2.34.1

[PATCH v2 3/4] dpkg-base: Fix SRC_URI:remove ignoring

[Anton Mikanovich]

Current SRC_APT filling code results in losing SRC_URI:remove values.
Remove setting SRC_URI value in prior to remove apt:// items with
additional variables.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a8263046..5f88c805 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -87,15 +87,12 @@ python() {
     src_uri = (d.getVar('SRC_URI', False) or "").split()
 
     prefix = "apt://"
-    new_src_uri = []
     src_apt = []
     for u in src_uri:
         if u.startswith(prefix):
             src_apt.append(u[len(prefix) :])
-        else:
-            new_src_uri.append(u)
+            d.setVar('SRC_URI:remove', u)
 
-    d.setVar('SRC_URI', ' '.join(new_src_uri))
     d.prependVar('SRC_APT', ' '.join(src_apt))
 
     if len(d.getVar('SRC_APT').strip()) > 0:
-- 
2.34.1

[PATCH v2 4/4] multiarch: Fix SRC_URI:remove ignoring

[Anton Mikanovich]

Setting SRC_URI right after getVar with expand=False results in loosing
SRC_URI:remove values. Rebuild logic to use remove-append approach.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/multiarch.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 48cec9ab..b3fc9441 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -49,8 +49,9 @@ python multiarch_virtclass_handler() {
         vars = 'SRC_URI FILESPATH'.split()
         for var in vars:
             v = d.getVar(var, expand=False)
-            if v is not None:
-                d.setVar(var, v.replace('${PN}', '${BPN}'))
+            if v is not None and '${PN}' in v:
+                d.setVar(var + ':remove', v)
+                d.appendVar(var, v.replace('${PN}', '${BPN}'))
 
     # When building compat/native, the corresponding suffix needs to be
     # propagated to all bitbake dependency definitions.
-- 
2.34.1

Re: [PATCH v2 0/4] Fix SRC_URI:remove ignoring

[Jan Kiszka]

On 16.02.24 15:10, Anton Mikanovich wrote:
> SRC_URI:remove override now got lost due to manual SRC_URI manipulations.
> Add example for CI and few fixes to make this test case pass.
> 
> Changes since v1:
>  - Move test case to meta-test.
>  - Fix leading space on append.
>  - Improve wording.
> 
> Anton Mikanovich (4):
>   conf: Enable bbappends
>   meta-test: Add examples of SRC_URI overrides usage
>   dpkg-base: Fix SRC_URI:remove ignoring
>   multiarch: Fix SRC_URI:remove ignoring
> 
>  meta-isar/conf/layer.conf                            | 2 +-
>  meta-test/conf/layer.conf                            | 2 +-
>  meta-test/recipes-app/hello-isar/hello-isar.bbappend | 9 +++++++++
>  meta/classes/dpkg-base.bbclass                       | 5 +----
>  meta/classes/multiarch.bbclass                       | 5 +++--
>  meta/conf/layer.conf                                 | 2 +-
>  6 files changed, 16 insertions(+), 9 deletions(-)
>  create mode 100644 meta-test/recipes-app/hello-isar/hello-isar.bbappend
> 

Good catch! That would have been nasty to understand in downstream layers.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH v2 0/4] Fix SRC_URI:remove ignoring

[Uladzimir Bely]

On Fri, 2024-02-16 at 16:10 +0200, Anton Mikanovich wrote:
> SRC_URI:remove override now got lost due to manual SRC_URI
> manipulations.
> Add example for CI and few fixes to make this test case pass.
> 
> Changes since v1:
>  - Move test case to meta-test.
>  - Fix leading space on append.
>  - Improve wording.
> 
> Anton Mikanovich (4):
>   conf: Enable bbappends
>   meta-test: Add examples of SRC_URI overrides usage
>   dpkg-base: Fix SRC_URI:remove ignoring
>   multiarch: Fix SRC_URI:remove ignoring
> 
>  meta-isar/conf/layer.conf                            | 2 +-
>  meta-test/conf/layer.conf                            | 2 +-
>  meta-test/recipes-app/hello-isar/hello-isar.bbappend | 9 +++++++++
>  meta/classes/dpkg-base.bbclass                       | 5 +----
>  meta/classes/multiarch.bbclass                       | 5 +++--
>  meta/conf/layer.conf                                 | 2 +-
>  6 files changed, 16 insertions(+), 9 deletions(-)
>  create mode 100644 meta-test/recipes-app/hello-isar/hello-
> isar.bbappend
> 
> -- 
> 2.34.1
> 

Applied to next.

RE: [PATCH v2 4/4] multiarch: Fix SRC_URI:remove ignoring

[Schmidt, Adriaan]

Anton Mikanovich, Friday, February 16, 2024 3:11 PM:
> Setting SRC_URI right after getVar with expand=False results in loosing
> SRC_URI:remove values. Rebuild logic to use remove-append approach.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/multiarch.bbclass | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
> index 48cec9ab..b3fc9441 100644
> --- a/meta/classes/multiarch.bbclass
> +++ b/meta/classes/multiarch.bbclass
> @@ -49,8 +49,9 @@ python multiarch_virtclass_handler() {
>          vars = 'SRC_URI FILESPATH'.split()
>          for var in vars:
>              v = d.getVar(var, expand=False)
> -            if v is not None:
> -                d.setVar(var, v.replace('${PN}', '${BPN}'))
> +            if v is not None and '${PN}' in v:
> +                d.setVar(var + ':remove', v)
> +                d.appendVar(var, v.replace('${PN}', '${BPN}'))

Hi Anton,

Just stumbled across this when using current next:
There is a space missing in the append:
d.appendVar(var, ' ' + v.replace('${PN}', '${BPN}'))

Also, setting variables "V:remove" (potentially multiple times) from
Python code smells a little funny. I'm not sure about Bitbake parsing,
and the order of execution-of-Python-blocks vs. variable-assignments vs.
expansions vs. :remove-operators, but I'm a worried that this might
be relying on some undocumented implementation details and only work
by accident.
But I admit that I didn't have a closer look at the actual problem
you're solving here, so I also don't have a good alternative proposal.

So for now, I'd suggest to just fix the missing space.

Thanks,
Adriaan

> 
>      # When building compat/native, the corresponding suffix needs to be
>      # propagated to all bitbake dependency definitions.
> --
> 2.34.1
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/20240216141051.3490109-5-
> amikan%40ilbers.de.

Re: [PATCH v2 4/4] multiarch: Fix SRC_URI:remove ignoring

[Anton Mikanovich]

14/03/2024 10:28, Schmidt, Adriaan wrote:
> Hi Anton,
>
> Just stumbled across this when using current next:
> There is a space missing in the append:
> d.appendVar(var, ' ' + v.replace('${PN}', '${BPN}'))
>
> Also, setting variables "V:remove" (potentially multiple times) from
> Python code smells a little funny. I'm not sure about Bitbake parsing,
> and the order of execution-of-Python-blocks vs. variable-assignments vs.
> expansions vs. :remove-operators, but I'm a worried that this might
> be relying on some undocumented implementation details and only work
> by accident.
> But I admit that I didn't have a closer look at the actual problem
> you're solving here, so I also don't have a good alternative proposal.
>
> So for now, I'd suggest to just fix the missing space.
>
> Thanks,
> Adriaan

Hello Adriaan,

Thanks for notice, I will send a fix for missing space.

I'm also not satisfied with ":remove" setting but don't see better solution.
In general, it's all about Bitbake ignoring :remove overrides during getVar
with expand=False (but taking care about :append overrides), which makes us
use such a hack.