[PATCH 0/2] Fix u-boot-script packages conflict

[PATCH 0/2] Fix u-boot-script packages conflict

[Anton Mikanovich]

When trying to execute do_prepare_build task of u-boot-script package
for some targets 'multiple execution' occurs:

$ bitbake mc:bananapi-buster:u-boot-script:do_prepare_build mc:de0-nano-soc-buster:u-boot-script:do_prepare_build
ERROR: Detect multiple executions of do_prepare_build in /build/tmp/work/debian-buster-armhf/u-boot-script/1.1-r0
ERROR: Rerun a clean build with empty STAMPCLEAN and compare the sigdata files

This is caused by different u-boot-script packages building in the same
directory. The difference between those two output debs is dependency on
linux-image-${KERNEL_NAME}, so we need to separate them and fix Isar to
allow parallel building.

Anton Mikanovich (2):
  dpkg: Allow parallel build of different package versions
  u-boot-script: Split packages by kernel

 meta/classes/dpkg-base.bbclass                      | 7 +++++--
 meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb | 3 +++
 2 files changed, 8 insertions(+), 2 deletions(-)

-- 
2.25.1

[PATCH 1/2] dpkg: Allow parallel build of different package versions

[Anton Mikanovich]

To be able to use different versions of the same package for the
different machines/multiconfigs we need to split workdir mount points
and sstate manifests based on package version.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index cb5ce4a..8850544 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -158,8 +158,8 @@ def get_package_srcdir(d):
     return s
 
 # Each package should have its own unique build folder, so use
-# recipe name as identifier
-PP = "/home/builder/${PN}"
+# recipe name and version as identifier
+PP = "/home/builder/${P}"
 PPS ?= "${@get_package_srcdir(d)}"
 
 # Empty do_prepare_build() implementation, to be overwritten if needed
@@ -221,6 +221,9 @@ do_dpkg_build[cleandirs] += "${DPKG_SSTATE}"
 do_dpkg_build[sstate-plaindirs] = "${DPKG_SSTATE}"
 do_dpkg_build[sstate-interceptfuncs] = "dpkg_build_sstate_prepare"
 
+# Allow parallel build of different versions
+SSTATE_MANFILEPREFIX .= "-${PV}"
+
 dpkg_build_sstate_prepare() {
     test -n "$(find ${S}/.. -maxdepth 1 -name '*.deb' -print -quit)" &&
         ln -f ${S}/../*.deb -t ${DPKG_SSTATE}
-- 
2.25.1

Re: [PATCH 1/2] dpkg: Allow parallel build of different package versions

[Henning Schild]

Am Tue, 11 Jan 2022 12:36:55 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> To be able to use different versions of the same package for the
> different machines/multiconfigs we need to split workdir mount points
> and sstate manifests based on package version.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index cb5ce4a..8850544 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -158,8 +158,8 @@ def get_package_srcdir(d):
>      return s
>  
>  # Each package should have its own unique build folder, so use
> -# recipe name as identifier
> -PP = "/home/builder/${PN}"
> +# recipe name and version as identifier
> +PP = "/home/builder/${P}"

Better go directly for PF to also cover PR

>  PPS ?= "${@get_package_srcdir(d)}"
>  
>  # Empty do_prepare_build() implementation, to be overwritten if
> needed @@ -221,6 +221,9 @@ do_dpkg_build[cleandirs] +=
> "${DPKG_SSTATE}" do_dpkg_build[sstate-plaindirs] = "${DPKG_SSTATE}"
>  do_dpkg_build[sstate-interceptfuncs] = "dpkg_build_sstate_prepare"
>  
> +# Allow parallel build of different versions
> +SSTATE_MANFILEPREFIX .= "-${PV}"

I guess it should be
SSTATE_MANFILEPREFIX = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-${PF}
or .= "${PV}-${PR}"

Henning

>  dpkg_build_sstate_prepare() {
>      test -n "$(find ${S}/.. -maxdepth 1 -name '*.deb' -print -quit)"
> && ln -f ${S}/../*.deb -t ${DPKG_SSTATE}

[PATCH 2/2] u-boot-script: Split packages by kernel

[Anton Mikanovich]

u-boot-script package depends on linux-image which is different for the
machines with different KERNEL_NAME. In case of the same distro and
arch those u-boot-script packages will conflict, so we need to set
different versions (based on machine) to allow building in parallel.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
index e07deee..33ed70f 100644
--- a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
+++ b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
@@ -16,6 +16,9 @@ SRC_URI = " \
 
 DEBIAN_DEPENDS = "u-boot-tools, linux-image-${KERNEL_NAME}"
 
+# Package version should be machine specific to separate kernel deps
+PV .= "-${MACHINE}"
+
 do_install() {
 	install -v -d ${D}/usr/sbin
 	install -v -m 755 ${WORKDIR}/update-u-boot-script ${D}/usr/sbin/
-- 
2.25.1

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Henning Schild]

Am Tue, 11 Jan 2022 12:36:56 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:

> u-boot-script package depends on linux-image which is different for
> the machines with different KERNEL_NAME. In case of the same distro
> and arch those u-boot-script packages will conflict, so we need to set
> different versions (based on machine) to allow building in parallel.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
> b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb index
> e07deee..33ed70f 100644 ---
> a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb +++
> b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb @@ -16,6 +16,9
> @@ SRC_URI = " \ 
>  DEBIAN_DEPENDS = "u-boot-tools, linux-image-${KERNEL_NAME}"
>  
> +# Package version should be machine specific to separate kernel deps
> +PV .= "-${MACHINE}"

Doing this in PV is strange, it should be done in PN .. maybe with
KERNEL_NAME and not MACHINE

Henning

>  do_install() {
>  	install -v -d ${D}/usr/sbin
>  	install -v -m 755 ${WORKDIR}/update-u-boot-script
> ${D}/usr/sbin/

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Jan Kiszka]

On 11.01.22 10:36, Anton Mikanovich wrote:
> u-boot-script package depends on linux-image which is different for the
> machines with different KERNEL_NAME. In case of the same distro and
> arch those u-boot-script packages will conflict, so we need to set
> different versions (based on machine) to allow building in parallel.
> 

It makes no sense to build the same package with the same content 
multiple times. We likely rather need to make it depend on a generic 
kernel package / create a generic kernel package name.

Jan

> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>   meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
> index e07deee..33ed70f 100644
> --- a/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
> +++ b/meta/recipes-bsp/u-boot-script/u-boot-script_1.1.bb
> @@ -16,6 +16,9 @@ SRC_URI = " \
>   
>   DEBIAN_DEPENDS = "u-boot-tools, linux-image-${KERNEL_NAME}"
>   
> +# Package version should be machine specific to separate kernel deps
> +PV .= "-${MACHINE}"
> +
>   do_install() {
>   	install -v -d ${D}/usr/sbin
>   	install -v -m 755 ${WORKDIR}/update-u-boot-script ${D}/usr/sbin/

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Anton Mikanovich]

11.01.2022 15:03, Jan Kiszka wrote:
> It makes no sense to build the same package with the same content 
> multiple times. We likely rather need to make it depend on a generic 
> kernel package / create a generic kernel package name.
>
> Jan

I'm afraid it will not help - moving this dependency into another 
package will also move the issue into this new package.

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Jan Kiszka]

On 11.01.22 16:05, Anton Mikanovich wrote:
> 11.01.2022 15:03, Jan Kiszka wrote:
>> It makes no sense to build the same package with the same content 
>> multiple times. We likely rather need to make it depend on a generic 
>> kernel package / create a generic kernel package name.
>>
>> Jan
> 
> I'm afraid it will not help - moving this dependency into another 
> package will also move the issue into this new package.
> 

It will. We just have to make u-boot-script depend on a generic kernel 
so that this package is only installed after some kernel - no matter 
which one - is already installed on the target image. There is NO 
dependency on the concrete kernel in this packages, just study it.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Anton Mikanovich]

11.01.2022 18:56, Jan Kiszka wrote:
> On 11.01.22 16:05, Anton Mikanovich wrote:
>> 11.01.2022 15:03, Jan Kiszka wrote:
>>> It makes no sense to build the same package with the same content 
>>> multiple times. We likely rather need to make it depend on a generic 
>>> kernel package / create a generic kernel package name.
>>>
>>> Jan
>>
>> I'm afraid it will not help - moving this dependency into another 
>> package will also move the issue into this new package.
>>
>
> It will. We just have to make u-boot-script depend on a generic kernel 
> so that this package is only installed after some kernel - no matter 
> which one - is already installed on the target image. There is NO 
> dependency on the concrete kernel in this packages, just study it.
>
> Jan

Can you please provide any details about how it can be implemented?
We have linux-image-generic package in Ubuntu, but not in Debian. And we 
can't just create it, because of the same deps conflict issue in Isar.
I understood u-boot-script depends on any kernel, but we need to declare 
this 'any' somehow.

RE: [PATCH 2/2] u-boot-script: Split packages by kernel

[Schmidt, Adriaan]

Anton Mikanovich, Mittwoch, 12. Januar 2022 14:35:
> 11.01.2022 18:56, Jan Kiszka wrote:
> > On 11.01.22 16:05, Anton Mikanovich wrote:
> >> 11.01.2022 15:03, Jan Kiszka wrote:
> >>> It makes no sense to build the same package with the same content
> >>> multiple times. We likely rather need to make it depend on a generic
> >>> kernel package / create a generic kernel package name.
> >>>
> >>> Jan
> >>
> >> I'm afraid it will not help - moving this dependency into another
> >> package will also move the issue into this new package.
> >>
> >
> > It will. We just have to make u-boot-script depend on a generic kernel
> > so that this package is only installed after some kernel - no matter
> > which one - is already installed on the target image. There is NO
> > dependency on the concrete kernel in this packages, just study it.
> >
> > Jan
> 
> Can you please provide any details about how it can be implemented?
> We have linux-image-generic package in Ubuntu, but not in Debian. And we
> can't just create it, because of the same deps conflict issue in Isar.
> I understood u-boot-script depends on any kernel, but we need to declare
> this 'any' somehow.

OE has the concept of virtual providers:
https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#metadata-virtual-providers
And it looks like the PREFERRED_PROVIDER_* variables are a bitbake feature, so it shouldn't be too hard to do something like this in Isar.

@Jan: I assume this is what you had in mind?

Adriaan

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Jan Kiszka]

On 12.01.22 14:51, Schmidt, Adriaan (T CED SES-DE) wrote:
> Anton Mikanovich, Mittwoch, 12. Januar 2022 14:35:
>> 11.01.2022 18:56, Jan Kiszka wrote:
>>> On 11.01.22 16:05, Anton Mikanovich wrote:
>>>> 11.01.2022 15:03, Jan Kiszka wrote:
>>>>> It makes no sense to build the same package with the same content
>>>>> multiple times. We likely rather need to make it depend on a generic
>>>>> kernel package / create a generic kernel package name.
>>>>>
>>>>> Jan
>>>>
>>>> I'm afraid it will not help - moving this dependency into another
>>>> package will also move the issue into this new package.
>>>>
>>>
>>> It will. We just have to make u-boot-script depend on a generic kernel
>>> so that this package is only installed after some kernel - no matter
>>> which one - is already installed on the target image. There is NO
>>> dependency on the concrete kernel in this packages, just study it.
>>>
>>> Jan
>>
>> Can you please provide any details about how it can be implemented?
>> We have linux-image-generic package in Ubuntu, but not in Debian. And we
>> can't just create it, because of the same deps conflict issue in Isar.
>> I understood u-boot-script depends on any kernel, but we need to declare
>> this 'any' somehow.
> 
> OE has the concept of virtual providers:
> https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#metadata-virtual-providers
> And it looks like the PREFERRED_PROVIDER_* variables are a bitbake feature, so it shouldn't be too hard to do something like this in Isar.
> 
> @Jan: I assume this is what you had in mind?

Yes and no: Yes, we would need just a provider, no, his won't help us as 
we need it as runtime dependency, and Debian kernel do not expose that. 
See my other reply.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 2/2] u-boot-script: Split packages by kernel

[Jan Kiszka]

On 12.01.22 14:34, Anton Mikanovich wrote:
> 11.01.2022 18:56, Jan Kiszka wrote:
>> On 11.01.22 16:05, Anton Mikanovich wrote:
>>> 11.01.2022 15:03, Jan Kiszka wrote:
>>>> It makes no sense to build the same package with the same content 
>>>> multiple times. We likely rather need to make it depend on a generic 
>>>> kernel package / create a generic kernel package name.
>>>>
>>>> Jan
>>>
>>> I'm afraid it will not help - moving this dependency into another 
>>> package will also move the issue into this new package.
>>>
>>
>> It will. We just have to make u-boot-script depend on a generic kernel 
>> so that this package is only installed after some kernel - no matter 
>> which one - is already installed on the target image. There is NO 
>> dependency on the concrete kernel in this packages, just study it.
>>
>> Jan
> 
> Can you please provide any details about how it can be implemented?
> We have linux-image-generic package in Ubuntu, but not in Debian. And we 
> can't just create it, because of the same deps conflict issue in Isar.
> I understood u-boot-script depends on any kernel, but we need to declare 
> this 'any' somehow.
> 

Two directions to resolve that:

1. make it depend on some linux-image-custom (to-be-added "Provides:" of
    our kernels) | linux-image-<list of distro kernels>

2. clarify if a kernel actually has to be installed prior to
    u-boot-script or if the kernel script hook isn't enough (or can be
    made enough)

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux