[PATCH 0/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

[PATCH 0/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Cedric Hombourger' via isar-users]

With sources being built only once, do_prepare_build (and its dependencies)
are skipped when building -compat or -native packages. This unfortunately
means that do_install is not executed and is causing the package build
to fail as the install step is not finding any files in ${PP}/image. To
avoid changing how do_install is scheduled in the normal case, it is simply
added as an extra dependency to do_dpkg_build to make sure it gets executed
prior to the build of the binary package.

This patch depends on https://groups.google.com/g/isar-users/c/Ys54mk6_bRM

Cedric Hombourger (1):
  dpkg-raw: make sure do_install gets called for -compat/-native builds

 meta/classes/dpkg-raw.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240912151121.212959-1-cedric.hombourger%40siemens.com.

[PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Cedric Hombourger' via isar-users]

With sources being built only once, do_prepare_build (and its dependencies)
are skipped when building -compat or -native packages. This unfortunately
means that do_install is not executed and is causing the package build
to fail as the install step is not finding any files in ${PP}/image. To
avoid changing how do_install is scheduled in the normal case, it is simply
added as an extra dependency to do_dpkg_build to make sure it gets executed
prior to the build of the binary package.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..94aba1aa 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -15,6 +15,12 @@ do_install() {
 do_install[cleandirs] = "${D}"
 addtask install after do_patch do_transform_template before do_prepare_build
 
+# make sure do_install gets executed when building -compat / -native packages
+# are built since do_prepare_build would be skipped for those (sources are
+# built only once). To avoid changing how do_install is scheduled in the normal
+# case, it is simply added as an extra dependency to do_dpkg_build
+do_dpkg_build[depends] += "${PN}:do_install"
+
 do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
 	cd ${D}
-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240912151121.212959-2-cedric.hombourger%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Jan Kiszka' via isar-users]

On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> With sources being built only once, do_prepare_build (and its dependencies)
> are skipped when building -compat or -native packages. This unfortunately
> means that do_install is not executed and is causing the package build
> to fail as the install step is not finding any files in ${PP}/image. To
> avoid changing how do_install is scheduled in the normal case, it is simply
> added as an extra dependency to do_dpkg_build to make sure it gets executed
> prior to the build of the binary package.

What is the use case for raw-compat and -native? Are you actually having
a case with different binaries then?

> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  meta/classes/dpkg-raw.bbclass | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index dd7b761f..94aba1aa 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -15,6 +15,12 @@ do_install() {
>  do_install[cleandirs] = "${D}"
>  addtask install after do_patch do_transform_template before do_prepare_build
>  
> +# make sure do_install gets executed when building -compat / -native packages
> +# are built since do_prepare_build would be skipped for those (sources are
> +# built only once). To avoid changing how do_install is scheduled in the normal
> +# case, it is simply added as an extra dependency to do_dpkg_build
> +do_dpkg_build[depends] += "${PN}:do_install"
> +

So, this class would then violate the rule of shared, identical "source"
packages? Does not feel correct.

>  do_prepare_build[cleandirs] += "${S}/debian"
>  do_prepare_build() {
>  	cd ${D}

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ed6c6869-9c77-4ede-89fe-882cbdc4e520%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> > With sources being built only once, do_prepare_build (and its
> > dependencies)
> > are skipped when building -compat or -native packages. This
> > unfortunately
> > means that do_install is not executed and is causing the package
> > build
> > to fail as the install step is not finding any files in
> > ${PP}/image. To
> > avoid changing how do_install is scheduled in the normal case, it
> > is simply
> > added as an extra dependency to do_dpkg_build to make sure it gets
> > executed
> > prior to the build of the binary package.
> 
> What is the use case for raw-compat and -native? Are you actually
> having
> a case with different binaries then?

For my specific case, I had wished -native would not be built because
my dpkg-raw package has DPKG_ARCH set to "all". In that case, we should
not have a separate build for -native

The issue will occur for any "dpkg" package being built for -native
when it DEPENDS on a "dpkg-raw" package because our native class
automatically changes all dependencies to -native

While most "dpkg-raw" packages should have DPKG_ARCH = "all", many
recipes did not bother setting this (in most downstream layers that I
have seen) and will end up with different binary packages in their -
native builds when HOST_ARCH != DISTRO_ARCH

To reproduce the issue that I have seen (and which should be fixed
IMO):

1. bitbake mc:qemuarm64-bookworm:example-raw
2. bitbake mc:qemuarm64-bookworm:example-raw-native

PS: I am planning to investigate how to suppress the duplicate build of
"dpkg-raw" package that has DPKG_ARCH set to "all" when a dependent
"dpkg" package is built for both variants

I believe the issue demonstrated with the above scenario needs fixing
(the error message you otherwise get as a user is rather cryptic)

> 
> > 
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >  meta/classes/dpkg-raw.bbclass | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-
> > raw.bbclass
> > index dd7b761f..94aba1aa 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -15,6 +15,12 @@ do_install() {
> >  do_install[cleandirs] = "${D}"
> >  addtask install after do_patch do_transform_template before
> > do_prepare_build
> >  
> > +# make sure do_install gets executed when building -compat / -
> > native packages
> > +# are built since do_prepare_build would be skipped for those
> > (sources are
> > +# built only once). To avoid changing how do_install is scheduled
> > in the normal
> > +# case, it is simply added as an extra dependency to do_dpkg_build
> > +do_dpkg_build[depends] += "${PN}:do_install"
> > +
> 
> So, this class would then violate the rule of shared, identical
> "source"
> packages? Does not feel correct.

No same sources are used between non-native and -native. See below

Tasks for example-raw:

$ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example-
raw/0.3-r0/temp/log.task_order 
20240912-143135.942912 do_fetch (171631): log.do_fetch.171631
20240912-143135.967641 do_get_reference_env (171636):
log.do_get_reference_env.171636
20240912-143144.778512 do_unpack (171697): log.do_unpack.171697
20240912-143145.280183 do_adjust_git (171767): log.do_adjust_git.171767
20240912-143145.369652 do_transform_template (171780):
log.do_transform_template.171780
20240912-143145.477501 do_patch (171793): log.do_patch.171793
20240912-143145.964197 do_install (171849): log.do_install.171849
20240912-143146.108066 do_prepare_build (171883):
log.do_prepare_build.171883
20240912-143146.247279 do_dpkg_source (172015):
log.do_dpkg_source.172015
20240912-143146.303464 do_local_isarapt (172169):
log.do_local_isarapt.172169
20240912-143146.517836 do_deploy_source (172513):
log.do_deploy_source.172513
20240912-143457.684464 do_dpkg_build (195129): log.do_dpkg_build.195129
20240912-143527.279878 do_deploy_deb (197150): log.do_deploy_deb.197150


Tasks for example-raw-native (with my changes):

$ cat build-test-raw-native/tmp/work/debian-bookworm-arm64/example-raw-
native/0.3-r0/temp/log.task_order 
20240912-143632.541485 do_fetch (197337): log.do_fetch.197337
20240912-143632.548091 do_get_reference_env (197338):
log.do_get_reference_env.197338
20240912-143632.555547 do_local_isarapt (197341):
log.do_local_isarapt.197341
20240912-143639.819570 do_unpack (197499): log.do_unpack.197499
20240912-143640.047376 do_adjust_git (197743): log.do_adjust_git.197743
20240912-143640.160769 do_transform_template (198038):
log.do_transform_template.198038
20240912-143640.267071 do_patch (198156): log.do_patch.198156
20240912-143640.593967 do_install (198222): log.do_install.198222
20240912-143742.934586 do_fetch_common_source (210951):
log.do_fetch_common_source.210951
20240912-143743.720108 do_dpkg_build (211215): log.do_dpkg_build.211215
20240912-143748.362105 do_deploy_deb (212659): log.do_deploy_deb.212659

Without my changes, do_install would not be executed and do_dpkg_build
would fail miserably.

run.do_dpkg_build will find the source package (more specifically the
.dsc file) in the work tree and use it

I therefore believe that identical sources are used between -native and
non-native

Am I missing something?


> 
> >  do_prepare_build[cleandirs] += "${S}/debian"
> >  do_prepare_build() {
> >         cd ${D}
> 
> Jan
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/c147d4428dc8ad963af75ad1c9f392d5787d0f1f.camel%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Jan Kiszka' via isar-users]

On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
>>> With sources being built only once, do_prepare_build (and its
>>> dependencies)
>>> are skipped when building -compat or -native packages. This
>>> unfortunately
>>> means that do_install is not executed and is causing the package
>>> build
>>> to fail as the install step is not finding any files in
>>> ${PP}/image. To
>>> avoid changing how do_install is scheduled in the normal case, it
>>> is simply
>>> added as an extra dependency to do_dpkg_build to make sure it gets
>>> executed
>>> prior to the build of the binary package.
>>
>> What is the use case for raw-compat and -native? Are you actually
>> having
>> a case with different binaries then?
> 
> For my specific case, I had wished -native would not be built because
> my dpkg-raw package has DPKG_ARCH set to "all". In that case, we should
> not have a separate build for -native
> 
> The issue will occur for any "dpkg" package being built for -native
> when it DEPENDS on a "dpkg-raw" package because our native class
> automatically changes all dependencies to -native
> 

Then stub-out the other variants, make them depend on the BPN package,
but do not further foster their build.

Jan

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/bdaab90c-3b1c-407b-a272-937bcc05d6d1%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> > > > With sources being built only once, do_prepare_build (and its
> > > > dependencies)
> > > > are skipped when building -compat or -native packages. This
> > > > unfortunately
> > > > means that do_install is not executed and is causing the
> > > > package
> > > > build
> > > > to fail as the install step is not finding any files in
> > > > ${PP}/image. To
> > > > avoid changing how do_install is scheduled in the normal case,
> > > > it
> > > > is simply
> > > > added as an extra dependency to do_dpkg_build to make sure it
> > > > gets
> > > > executed
> > > > prior to the build of the binary package.
> > > 
> > > What is the use case for raw-compat and -native? Are you actually
> > > having
> > > a case with different binaries then?
> > 
> > For my specific case, I had wished -native would not be built
> > because
> > my dpkg-raw package has DPKG_ARCH set to "all". In that case, we
> > should
> > not have a separate build for -native
> > 
> > The issue will occur for any "dpkg" package being built for -native
> > when it DEPENDS on a "dpkg-raw" package because our native class
> > automatically changes all dependencies to -native
> > 
> 
> Then stub-out the other variants, make them depend on the BPN
> package,
> but do not further foster their build.

My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN names
But we have multiarch [1] suffixing each of our DEPENDS with -native
when built for -native

Or maybe I am not understanding your proposal? I am pretty sure we have
a bug in Isar. I can build a test case if the illustration with
example-raw is not sufficient / convincing

[1]
https://github.com/ilbers/isar/blob/master/meta/classes/multiarch.bbclass#L73

> 
> Jan
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3bab618191d7b2c72c80a7186e0767d5d1635073.camel%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Jan Kiszka' via isar-users]

On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
>> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
>>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
>>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
>>>>> With sources being built only once, do_prepare_build (and its
>>>>> dependencies)
>>>>> are skipped when building -compat or -native packages. This
>>>>> unfortunately
>>>>> means that do_install is not executed and is causing the
>>>>> package
>>>>> build
>>>>> to fail as the install step is not finding any files in
>>>>> ${PP}/image. To
>>>>> avoid changing how do_install is scheduled in the normal case,
>>>>> it
>>>>> is simply
>>>>> added as an extra dependency to do_dpkg_build to make sure it
>>>>> gets
>>>>> executed
>>>>> prior to the build of the binary package.
>>>>
>>>> What is the use case for raw-compat and -native? Are you actually
>>>> having
>>>> a case with different binaries then?
>>>
>>> For my specific case, I had wished -native would not be built
>>> because
>>> my dpkg-raw package has DPKG_ARCH set to "all". In that case, we
>>> should
>>> not have a separate build for -native
>>>
>>> The issue will occur for any "dpkg" package being built for -native
>>> when it DEPENDS on a "dpkg-raw" package because our native class
>>> automatically changes all dependencies to -native
>>>
>>
>> Then stub-out the other variants, make them depend on the BPN
>> package,
>> but do not further foster their build.
> 
> My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN names
> But we have multiarch [1] suffixing each of our DEPENDS with -native
> when built for -native
> 
> Or maybe I am not understanding your proposal? I am pretty sure we have
> a bug in Isar. I can build a test case if the illustration with
> example-raw is not sufficient / convincing

Yes, this is an issue in isar. But I would address it there by not doing
anything in <raw>-native/compat targets, only making them depend on
their base target. IIRC, we already have such a case deep in the kernel
build.

Rebuilding the same thing is wrong.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/fc4dd743-2415-4076-85eb-8c6b73086bab%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
> On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
> > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
> > > > > > With sources being built only once, do_prepare_build (and
> > > > > > its
> > > > > > dependencies)
> > > > > > are skipped when building -compat or -native packages. This
> > > > > > unfortunately
> > > > > > means that do_install is not executed and is causing the
> > > > > > package
> > > > > > build
> > > > > > to fail as the install step is not finding any files in
> > > > > > ${PP}/image. To
> > > > > > avoid changing how do_install is scheduled in the normal
> > > > > > case,
> > > > > > it
> > > > > > is simply
> > > > > > added as an extra dependency to do_dpkg_build to make sure
> > > > > > it
> > > > > > gets
> > > > > > executed
> > > > > > prior to the build of the binary package.
> > > > > 
> > > > > What is the use case for raw-compat and -native? Are you
> > > > > actually
> > > > > having
> > > > > a case with different binaries then?
> > > > 
> > > > For my specific case, I had wished -native would not be built
> > > > because
> > > > my dpkg-raw package has DPKG_ARCH set to "all". In that case,
> > > > we
> > > > should
> > > > not have a separate build for -native
> > > > 
> > > > The issue will occur for any "dpkg" package being built for -
> > > > native
> > > > when it DEPENDS on a "dpkg-raw" package because our native
> > > > class
> > > > automatically changes all dependencies to -native
> > > > 
> > > 
> > > Then stub-out the other variants, make them depend on the BPN
> > > package,
> > > but do not further foster their build.
> > 
> > My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN
> > names
> > But we have multiarch [1] suffixing each of our DEPENDS with -
> > native
> > when built for -native
> > 
> > Or maybe I am not understanding your proposal? I am pretty sure we
> > have
> > a bug in Isar. I can build a test case if the illustration with
> > example-raw is not sufficient / convincing
> 
> Yes, this is an issue in isar. But I would address it there by not
> doing
> anything in <raw>-native/compat targets, only making them depend on
> their base target. IIRC, we already have such a case deep in the
> kernel
> build.
> 
> Rebuilding the same thing is wrong.

Agreed but :)

There are two cases to consider

1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for -
native

2) dpkg-raw produces an architecture-dependent package (Architecture:
any) => we would need to build both -native and non-native

I hit 1) and I am investigating further (getting myself familiar with
the multiarch code) and 2) is IMO addressed with this patch

Actually, when I first hit the issue, my dpkg-raw package did not have
DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
the extra build

> 
> Jan
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/5d4f3a853602cd72e49d2672ae70ba6051ceaa82.camel%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote:
> On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
> > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
> > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX)
> > > > wrote:
> > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users
> > > > > > wrote:
> > > > > > > With sources being built only once, do_prepare_build (and
> > > > > > > its
> > > > > > > dependencies)
> > > > > > > are skipped when building -compat or -native packages.
> > > > > > > This
> > > > > > > unfortunately
> > > > > > > means that do_install is not executed and is causing the
> > > > > > > package
> > > > > > > build
> > > > > > > to fail as the install step is not finding any files in
> > > > > > > ${PP}/image. To
> > > > > > > avoid changing how do_install is scheduled in the normal
> > > > > > > case,
> > > > > > > it
> > > > > > > is simply
> > > > > > > added as an extra dependency to do_dpkg_build to make
> > > > > > > sure
> > > > > > > it
> > > > > > > gets
> > > > > > > executed
> > > > > > > prior to the build of the binary package.
> > > > > > 
> > > > > > What is the use case for raw-compat and -native? Are you
> > > > > > actually
> > > > > > having
> > > > > > a case with different binaries then?
> > > > > 
> > > > > For my specific case, I had wished -native would not be built
> > > > > because
> > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that case,
> > > > > we
> > > > > should
> > > > > not have a separate build for -native
> > > > > 
> > > > > The issue will occur for any "dpkg" package being built for -
> > > > > native
> > > > > when it DEPENDS on a "dpkg-raw" package because our native
> > > > > class
> > > > > automatically changes all dependencies to -native
> > > > > 
> > > > 
> > > > Then stub-out the other variants, make them depend on the BPN
> > > > package,
> > > > but do not further foster their build.
> > > 
> > > My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN
> > > names
> > > But we have multiarch [1] suffixing each of our DEPENDS with -
> > > native
> > > when built for -native
> > > 
> > > Or maybe I am not understanding your proposal? I am pretty sure
> > > we
> > > have
> > > a bug in Isar. I can build a test case if the illustration with
> > > example-raw is not sufficient / convincing
> > 
> > Yes, this is an issue in isar. But I would address it there by not
> > doing
> > anything in <raw>-native/compat targets, only making them depend on
> > their base target. IIRC, we already have such a case deep in the
> > kernel
> > build.
> > 
> > Rebuilding the same thing is wrong.
> 
> Agreed but :)
> 
> There are two cases to consider
> 
> 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for -
> native
> 
> 2) dpkg-raw produces an architecture-dependent package (Architecture:
> any) => we would need to build both -native and non-native
> 
> I hit 1) and I am investigating further (getting myself familiar with
> the multiarch code) and 2) is IMO addressed with this patch
> 
> Actually, when I first hit the issue, my dpkg-raw package did not
> have
> DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
> the extra build
> 

For 1) and if my reading of multiarch is correct, the following should
do the trick:

diff --git a/meta/classes/multiarch.bbclass
b/meta/classes/multiarch.bbclass
index bb0f7983..8aa2de64 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -10,7 +10,7 @@ python() {
         d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target
variant
-    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
+    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or
d.getVar('DPKG_ARCH') == 'all':
         pn = d.getVar('PN')
         if not pn.endswith('-native') and not pn.endswith('-compat'):
             provides = (d.getVar('PROVIDES') or '').split()


This appears to work in the quick test I did but will perform more
tests

> > 
> > Jan
> > 
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/28b9b7b484f96a583b8a5efa2d8e961a8be8c752.camel%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Jan Kiszka' via isar-users]

On 13.09.24 13:12, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
>> On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
>>> On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
>>>> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
>>>>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
>>>>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users wrote:
>>>>>>> With sources being built only once, do_prepare_build (and
>>>>>>> its
>>>>>>> dependencies)
>>>>>>> are skipped when building -compat or -native packages. This
>>>>>>> unfortunately
>>>>>>> means that do_install is not executed and is causing the
>>>>>>> package
>>>>>>> build
>>>>>>> to fail as the install step is not finding any files in
>>>>>>> ${PP}/image. To
>>>>>>> avoid changing how do_install is scheduled in the normal
>>>>>>> case,
>>>>>>> it
>>>>>>> is simply
>>>>>>> added as an extra dependency to do_dpkg_build to make sure
>>>>>>> it
>>>>>>> gets
>>>>>>> executed
>>>>>>> prior to the build of the binary package.
>>>>>>
>>>>>> What is the use case for raw-compat and -native? Are you
>>>>>> actually
>>>>>> having
>>>>>> a case with different binaries then?
>>>>>
>>>>> For my specific case, I had wished -native would not be built
>>>>> because
>>>>> my dpkg-raw package has DPKG_ARCH set to "all". In that case,
>>>>> we
>>>>> should
>>>>> not have a separate build for -native
>>>>>
>>>>> The issue will occur for any "dpkg" package being built for -
>>>>> native
>>>>> when it DEPENDS on a "dpkg-raw" package because our native
>>>>> class
>>>>> automatically changes all dependencies to -native
>>>>>
>>>>
>>>> Then stub-out the other variants, make them depend on the BPN
>>>> package,
>>>> but do not further foster their build.
>>>
>>> My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN
>>> names
>>> But we have multiarch [1] suffixing each of our DEPENDS with -
>>> native
>>> when built for -native
>>>
>>> Or maybe I am not understanding your proposal? I am pretty sure we
>>> have
>>> a bug in Isar. I can build a test case if the illustration with
>>> example-raw is not sufficient / convincing
>>
>> Yes, this is an issue in isar. But I would address it there by not
>> doing
>> anything in <raw>-native/compat targets, only making them depend on
>> their base target. IIRC, we already have such a case deep in the
>> kernel
>> build.
>>
>> Rebuilding the same thing is wrong.
> 
> Agreed but :)
> 
> There are two cases to consider
> 
> 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for -
> native
> 
> 2) dpkg-raw produces an architecture-dependent package (Architecture:
> any) => we would need to build both -native and non-native
> 
> I hit 1) and I am investigating further (getting myself familiar with
> the multiarch code) and 2) is IMO addressed with this patch

No, your patch is even for that case wrong because it enforces source
package rebuilds that are no longer expected and supported - same base
target, same source package.

Jan

> 
> Actually, when I first hit the issue, my dpkg-raw package did not have
> DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
> the extra build
> 
>>
>> Jan
>>
> 

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/cd38dd82-e504-4119-a501-8049baedfc5c%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['Jan Kiszka' via isar-users]

On 13.09.24 13:33, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote:
>> On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
>>> On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
>>>> On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
>>>>> On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX)
>>>>> wrote:
>>>>>> On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
>>>>>>> On 12.09.24 17:11, 'Cedric Hombourger' via isar-users
>>>>>>> wrote:
>>>>>>>> With sources being built only once, do_prepare_build (and
>>>>>>>> its
>>>>>>>> dependencies)
>>>>>>>> are skipped when building -compat or -native packages.
>>>>>>>> This
>>>>>>>> unfortunately
>>>>>>>> means that do_install is not executed and is causing the
>>>>>>>> package
>>>>>>>> build
>>>>>>>> to fail as the install step is not finding any files in
>>>>>>>> ${PP}/image. To
>>>>>>>> avoid changing how do_install is scheduled in the normal
>>>>>>>> case,
>>>>>>>> it
>>>>>>>> is simply
>>>>>>>> added as an extra dependency to do_dpkg_build to make
>>>>>>>> sure
>>>>>>>> it
>>>>>>>> gets
>>>>>>>> executed
>>>>>>>> prior to the build of the binary package.
>>>>>>>
>>>>>>> What is the use case for raw-compat and -native? Are you
>>>>>>> actually
>>>>>>> having
>>>>>>> a case with different binaries then?
>>>>>>
>>>>>> For my specific case, I had wished -native would not be built
>>>>>> because
>>>>>> my dpkg-raw package has DPKG_ARCH set to "all". In that case,
>>>>>> we
>>>>>> should
>>>>>> not have a separate build for -native
>>>>>>
>>>>>> The issue will occur for any "dpkg" package being built for -
>>>>>> native
>>>>>> when it DEPENDS on a "dpkg-raw" package because our native
>>>>>> class
>>>>>> automatically changes all dependencies to -native
>>>>>>
>>>>>
>>>>> Then stub-out the other variants, make them depend on the BPN
>>>>> package,
>>>>> but do not further foster their build.
>>>>
>>>> My "dpkg" recipe is clean, it has a DEPENDS clause with only BPN
>>>> names
>>>> But we have multiarch [1] suffixing each of our DEPENDS with -
>>>> native
>>>> when built for -native
>>>>
>>>> Or maybe I am not understanding your proposal? I am pretty sure
>>>> we
>>>> have
>>>> a bug in Isar. I can build a test case if the illustration with
>>>> example-raw is not sufficient / convincing
>>>
>>> Yes, this is an issue in isar. But I would address it there by not
>>> doing
>>> anything in <raw>-native/compat targets, only making them depend on
>>> their base target. IIRC, we already have such a case deep in the
>>> kernel
>>> build.
>>>
>>> Rebuilding the same thing is wrong.
>>
>> Agreed but :)
>>
>> There are two cases to consider
>>
>> 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for -
>> native
>>
>> 2) dpkg-raw produces an architecture-dependent package (Architecture:
>> any) => we would need to build both -native and non-native
>>
>> I hit 1) and I am investigating further (getting myself familiar with
>> the multiarch code) and 2) is IMO addressed with this patch
>>
>> Actually, when I first hit the issue, my dpkg-raw package did not
>> have
>> DPKG_ARCH set to "all" hence my first instinct was to set it to avoid
>> the extra build
>>
> 
> For 1) and if my reading of multiarch is correct, the following should
> do the trick:
> 
> diff --git a/meta/classes/multiarch.bbclass
> b/meta/classes/multiarch.bbclass
> index bb0f7983..8aa2de64 100644
> --- a/meta/classes/multiarch.bbclass
> +++ b/meta/classes/multiarch.bbclass
> @@ -10,7 +10,7 @@ python() {
>          d.appendVar('BBCLASSEXTEND', ' compat')
>  
>      # build native separately only when it differs from the target
> variant
> -    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
> +    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or
> d.getVar('DPKG_ARCH') == 'all':
>          pn = d.getVar('PN')
>          if not pn.endswith('-native') and not pn.endswith('-compat'):
>              provides = (d.getVar('PROVIDES') or '').split()
> 
> 
> This appears to work in the quick test I did but will perform more
> tests
> 

At least for debianized packages (like dpkg-raw) and for -native. Will
probably not address -compat dependencies. Yeah, now it's getting
complicated.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/6a4e0583-0466-47a3-a1a6-3ca08d982373%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 14:15 +0200, Jan Kiszka wrote:
> On 13.09.24 13:12, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
> > > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
> > > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX)
> > > > > wrote:
> > > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> > > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users
> > > > > > > wrote:
> > > > > > > > With sources being built only once, do_prepare_build
> > > > > > > > (and
> > > > > > > > its
> > > > > > > > dependencies)
> > > > > > > > are skipped when building -compat or -native packages.
> > > > > > > > This
> > > > > > > > unfortunately
> > > > > > > > means that do_install is not executed and is causing
> > > > > > > > the
> > > > > > > > package
> > > > > > > > build
> > > > > > > > to fail as the install step is not finding any files in
> > > > > > > > ${PP}/image. To
> > > > > > > > avoid changing how do_install is scheduled in the
> > > > > > > > normal
> > > > > > > > case,
> > > > > > > > it
> > > > > > > > is simply
> > > > > > > > added as an extra dependency to do_dpkg_build to make
> > > > > > > > sure
> > > > > > > > it
> > > > > > > > gets
> > > > > > > > executed
> > > > > > > > prior to the build of the binary package.
> > > > > > > 
> > > > > > > What is the use case for raw-compat and -native? Are you
> > > > > > > actually
> > > > > > > having
> > > > > > > a case with different binaries then?
> > > > > > 
> > > > > > For my specific case, I had wished -native would not be
> > > > > > built
> > > > > > because
> > > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that
> > > > > > case,
> > > > > > we
> > > > > > should
> > > > > > not have a separate build for -native
> > > > > > 
> > > > > > The issue will occur for any "dpkg" package being built for
> > > > > > -
> > > > > > native
> > > > > > when it DEPENDS on a "dpkg-raw" package because our native
> > > > > > class
> > > > > > automatically changes all dependencies to -native
> > > > > > 
> > > > > 
> > > > > Then stub-out the other variants, make them depend on the BPN
> > > > > package,
> > > > > but do not further foster their build.
> > > > 
> > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only
> > > > BPN
> > > > names
> > > > But we have multiarch [1] suffixing each of our DEPENDS with -
> > > > native
> > > > when built for -native
> > > > 
> > > > Or maybe I am not understanding your proposal? I am pretty sure
> > > > we
> > > > have
> > > > a bug in Isar. I can build a test case if the illustration with
> > > > example-raw is not sufficient / convincing
> > > 
> > > Yes, this is an issue in isar. But I would address it there by
> > > not
> > > doing
> > > anything in <raw>-native/compat targets, only making them depend
> > > on
> > > their base target. IIRC, we already have such a case deep in the
> > > kernel
> > > build.
> > > 
> > > Rebuilding the same thing is wrong.
> > 
> > Agreed but :)
> > 
> > There are two cases to consider
> > 
> > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild for
> > -
> > native
> > 
> > 2) dpkg-raw produces an architecture-dependent package
> > (Architecture:
> > any) => we would need to build both -native and non-native
> > 
> > I hit 1) and I am investigating further (getting myself familiar
> > with
> > the multiarch code) and 2) is IMO addressed with this patch
> 
> No, your patch is even for that case wrong because it enforces source
> package rebuilds that are no longer expected and supported - same
> base
> target, same source package.

Are you sure? My understanding of the code and testing (with changes)
is showing that do_dpkg_source is only called once as as expected for
the non-native variant, for the -native case, do_fetch_common_source is
called: the source package to build -native is pulled from isar-apt

It is in fact one of the reason this patch is dependent on [2]. Without
this, the build will try to get its inputs from /home/builder/example-
raw/image because sources used to build -native were created when the
non-native package was built but would have been put under
/home/builder/example-raw-native/ without [2]

[2] https://groups.google.com/g/isar-users/c/md3gmDvhEPQ
> 
> Jan
> 
> > 
> > Actually, when I first hit the issue, my dpkg-raw package did not
> > have
> > DPKG_ARCH set to "all" hence my first instinct was to set it to
> > avoid
> > the extra build
> > 
> > > 
> > > Jan
> > > 
> > 
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3c44f4dda1228b5038708f448a16828da06fbc9c.camel%40siemens.com.

Re: [PATCH 1/1] dpkg-raw: make sure do_install gets called for -compat/-native builds

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-09-13 at 14:28 +0200, Jan Kiszka wrote:
> On 13.09.24 13:33, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > On Fri, 2024-09-13 at 13:12 +0200, Cedric Hombourger wrote:
> > > On Fri, 2024-09-13 at 13:04 +0200, Jan Kiszka wrote:
> > > > On 13.09.24 12:52, Hombourger, Cedric (DI CTO FDS CES LX)
> > > > wrote:
> > > > > On Fri, 2024-09-13 at 12:45 +0200, Jan Kiszka wrote:
> > > > > > On 13.09.24 10:29, Hombourger, Cedric (DI CTO FDS CES LX)
> > > > > > wrote:
> > > > > > > On Fri, 2024-09-13 at 10:08 +0200, Jan Kiszka wrote:
> > > > > > > > On 12.09.24 17:11, 'Cedric Hombourger' via isar-users
> > > > > > > > wrote:
> > > > > > > > > With sources being built only once, do_prepare_build
> > > > > > > > > (and
> > > > > > > > > its
> > > > > > > > > dependencies)
> > > > > > > > > are skipped when building -compat or -native
> > > > > > > > > packages.
> > > > > > > > > This
> > > > > > > > > unfortunately
> > > > > > > > > means that do_install is not executed and is causing
> > > > > > > > > the
> > > > > > > > > package
> > > > > > > > > build
> > > > > > > > > to fail as the install step is not finding any files
> > > > > > > > > in
> > > > > > > > > ${PP}/image. To
> > > > > > > > > avoid changing how do_install is scheduled in the
> > > > > > > > > normal
> > > > > > > > > case,
> > > > > > > > > it
> > > > > > > > > is simply
> > > > > > > > > added as an extra dependency to do_dpkg_build to make
> > > > > > > > > sure
> > > > > > > > > it
> > > > > > > > > gets
> > > > > > > > > executed
> > > > > > > > > prior to the build of the binary package.
> > > > > > > > 
> > > > > > > > What is the use case for raw-compat and -native? Are
> > > > > > > > you
> > > > > > > > actually
> > > > > > > > having
> > > > > > > > a case with different binaries then?
> > > > > > > 
> > > > > > > For my specific case, I had wished -native would not be
> > > > > > > built
> > > > > > > because
> > > > > > > my dpkg-raw package has DPKG_ARCH set to "all". In that
> > > > > > > case,
> > > > > > > we
> > > > > > > should
> > > > > > > not have a separate build for -native
> > > > > > > 
> > > > > > > The issue will occur for any "dpkg" package being built
> > > > > > > for -
> > > > > > > native
> > > > > > > when it DEPENDS on a "dpkg-raw" package because our
> > > > > > > native
> > > > > > > class
> > > > > > > automatically changes all dependencies to -native
> > > > > > > 
> > > > > > 
> > > > > > Then stub-out the other variants, make them depend on the
> > > > > > BPN
> > > > > > package,
> > > > > > but do not further foster their build.
> > > > > 
> > > > > My "dpkg" recipe is clean, it has a DEPENDS clause with only
> > > > > BPN
> > > > > names
> > > > > But we have multiarch [1] suffixing each of our DEPENDS with
> > > > > -
> > > > > native
> > > > > when built for -native
> > > > > 
> > > > > Or maybe I am not understanding your proposal? I am pretty
> > > > > sure
> > > > > we
> > > > > have
> > > > > a bug in Isar. I can build a test case if the illustration
> > > > > with
> > > > > example-raw is not sufficient / convincing
> > > > 
> > > > Yes, this is an issue in isar. But I would address it there by
> > > > not
> > > > doing
> > > > anything in <raw>-native/compat targets, only making them
> > > > depend on
> > > > their base target. IIRC, we already have such a case deep in
> > > > the
> > > > kernel
> > > > build.
> > > > 
> > > > Rebuilding the same thing is wrong.
> > > 
> > > Agreed but :)
> > > 
> > > There are two cases to consider
> > > 
> > > 1) dpkg-raw package has DPKG_ARCH = "all" => shall not rebuild
> > > for -
> > > native
> > > 
> > > 2) dpkg-raw produces an architecture-dependent package
> > > (Architecture:
> > > any) => we would need to build both -native and non-native
> > > 
> > > I hit 1) and I am investigating further (getting myself familiar
> > > with
> > > the multiarch code) and 2) is IMO addressed with this patch
> > > 
> > > Actually, when I first hit the issue, my dpkg-raw package did not
> > > have
> > > DPKG_ARCH set to "all" hence my first instinct was to set it to
> > > avoid
> > > the extra build
> > > 
> > 
> > For 1) and if my reading of multiarch is correct, the following
> > should
> > do the trick:
> > 
> > diff --git a/meta/classes/multiarch.bbclass
> > b/meta/classes/multiarch.bbclass
> > index bb0f7983..8aa2de64 100644
> > --- a/meta/classes/multiarch.bbclass
> > +++ b/meta/classes/multiarch.bbclass
> > @@ -10,7 +10,7 @@ python() {
> >          d.appendVar('BBCLASSEXTEND', ' compat')
> >  
> >      # build native separately only when it differs from the target
> > variant
> > -    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
> > +    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH') or
> > d.getVar('DPKG_ARCH') == 'all':
> >          pn = d.getVar('PN')
> >          if not pn.endswith('-native') and not pn.endswith('-
> > compat'):
> >              provides = (d.getVar('PROVIDES') or '').split()
> > 
> > 
> > This appears to work in the quick test I did but will perform more
> > tests
> > 
> 
> At least for debianized packages (like dpkg-raw) and for -native.
> Will
> probably not address -compat dependencies. Yeah, now it's getting
> complicated.

if DPKG_ARCH == all, wouldn't we expect the recipe to have PROVIDES set
to "example-raw-compat example-raw-native" since whatever flavor would
result in an _all.deb package being generated?

> 
> Jan
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/edc99898c9b98ed255a59b9170d64242d708e8cd.camel%40siemens.com.

[PATCH v2 0/2] do not build -compat/-native for dpkg-raw packages

['Cedric Hombourger' via isar-users]

The intent of the dpkg-raw class is to easily package configuration and data
files into a Debian package. Packages to be compiled should really use other
dpkg classes where support for cross-compilation and multiarch is provided
and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes and a
deprecation warning will be raised if overriden.

Despite this change, Isar may still build dpkg-raw packages a second time
when pulled by a -compat or -native package. multiarch is changed to check
if DPKG_ARCH was changed to all and will add -compat and -native to the
PROVIDES of the package (this condition will be checked for all dpkg variants)

It shall be noted that setting DPKG_ARCH in a recipe is a contract, the
user declares that generated package(s) are either architecture dependent
or independent.

Changes since v2:
 * v1 used a completely different approach where we were ensuring that
   do_install was called whether sources were built or pulled from
   isar_apt (do_fetch_common_source) since override_dh_install is
   pulling contents of dpkg-raw packages outside of the source tree
   (${PP}/image)

Cedric Hombourger (2):
  multiarch: avoid separate builds when DPKG_ARCH is all
  dpkg-raw: change DPKG_ARCH to all

 RECIPE-API-CHANGELOG.md        | 11 +++++++++++
 meta/classes/dpkg-raw.bbclass  |  8 ++++++++
 meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
 3 files changed, 43 insertions(+), 10 deletions(-)

-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240917070048.577-1-cedric.hombourger%40siemens.com.

[PATCH v2 1/2] multiarch: avoid separate builds when DPKG_ARCH is all

['Cedric Hombourger' via isar-users]

Binary-independent packages (DPKG_ARCH == all) do not require separate
builds for our special -compat and -native targets: extend PROVIDES for
these cases. It should be noted that DPKG_ARCH is always set for all
dpkg variants and defaults to "any" (only recipes setting DPKG_ARCH to
all explicitly will be impacted).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index bb0f7983..9df8eefd 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,25 +5,39 @@
 
 inherit compat
 python() {
+    pn = d.getVar('PN')
+    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
+    archIsAll = d.getVar('DPKG_ARCH') == 'all'
+
+    def pn_multiarch_target(pn):
+        return pn.endswith('-native') or pn.endswith('-compat')
+
+    def extend_provides(pn, provides, d):
+        if not pn_multiarch_target(pn):
+            all_provides = (d.getVar('PROVIDES') or '').split()
+            for p in all_provides:
+                if not pn_multiarch_target(p):
+                    d.appendVar('PROVIDES', f' {p}-{provides}')
+            d.appendVar('PROVIDES', f' {pn}-{provides}')
+
     # provide compat only when we can build it
     if isar_can_build_compat(d):
-        d.appendVar('BBCLASSEXTEND', ' compat')
+        # but do not build separately if architecture-independent
+        if archIsAll:
+            extend_provides(pn, 'compat', d)
+        else:
+            d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target variant
-    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
-        pn = d.getVar('PN')
-        if not pn.endswith('-native') and not pn.endswith('-compat'):
-            provides = (d.getVar('PROVIDES') or '').split()
-            for p in provides:
-                d.appendVar('PROVIDES', f' {p}-native')
-            d.appendVar('PROVIDES', f' {pn}-native')
-    else:
+    if archIsAll is False and archDiffers:
         d.appendVar('BBCLASSEXTEND', ' native')
+    else:
+        extend_provides(pn, 'native', d)
 
     # drop own -native build dependencies at recipe level if building natively
     # and not for the builder architecture
     depends = d.getVar('DEPENDS')
-    if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
+    if depends is not None and archDiffers \
        and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
         new_deps = []
         for dep in depends.split():
-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240917070048.577-2-cedric.hombourger%40siemens.com.

[PATCH v2 2/2] dpkg-raw: change DPKG_ARCH to all

['Cedric Hombourger' via isar-users]

Packages created by dpkg-raw are really for configuration and data files
and therefore architecture independent. DPKG_ARCH is now set to all and
a warning will be produced if changed by the recipe (this will later be
turned into an error).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md       | 11 +++++++++++
 meta/classes/dpkg-raw.bbclass |  8 ++++++++
 2 files changed, 19 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 21c558d2..d40827d0 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -646,3 +646,14 @@ recipe would use the following setting:
 ```
 HEADERS_INSTALL_EXTRA += "nvidia"
 ```
+
+### Architecture for dpkg-raw packages
+
+The intent of the dpkg-raw class is to easily package configuration and data
+files into a Debian package. Packages to be compiled should really use other
+dpkg classes where support for cross-compilation and multiarch is provided
+and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes and a
+deprecation warning will be raised if overriden.
+
+This change fixes an issue where a `dpkg` package is built for `-compat`
+or `-native` and `DEPENDS` on a `dpkg-raw` package.
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..cbaf298c 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,6 +7,9 @@ inherit dpkg
 
 D = "${WORKDIR}/image"
 
+# Create a binary-indep package
+DPKG_ARCH = "all"
+
 # Populate folder that will be picked up as package
 do_install() {
 	bbnote "Put your files for this package in $""{D}"
@@ -17,6 +20,11 @@ addtask install after do_patch do_transform_template before do_prepare_build
 
 do_prepare_build[cleandirs] += "${S}/debian"
 do_prepare_build() {
+	if [ "${DPKG_ARCH}" != "all" ]; then
+		bbwarn "support for building binary-arch packages with " \
+		       "dpkg-raw is deprecated and will be removed"
+	fi
+
 	cd ${D}
 	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
 		sed 's:^./::' > ${S}/debian/${PN}.install
-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240917070048.577-3-cedric.hombourger%40siemens.com.

Re: [PATCH v2 1/2] multiarch: avoid separate builds when DPKG_ARCH is all

['Jan Kiszka' via isar-users]

On 17.09.24 09:00, 'Cedric Hombourger' via isar-users wrote:
> Binary-independent packages (DPKG_ARCH == all) do not require separate
> builds for our special -compat and -native targets: extend PROVIDES for
> these cases. It should be noted that DPKG_ARCH is always set for all
> dpkg variants and defaults to "any" (only recipes setting DPKG_ARCH to
> all explicitly will be impacted).
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
>  1 file changed, 24 insertions(+), 10 deletions(-)
> 
> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
> index bb0f7983..9df8eefd 100644
> --- a/meta/classes/multiarch.bbclass
> +++ b/meta/classes/multiarch.bbclass
> @@ -5,25 +5,39 @@
>  
>  inherit compat
>  python() {
> +    pn = d.getVar('PN')
> +    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
> +    archIsAll = d.getVar('DPKG_ARCH') == 'all'
> +
> +    def pn_multiarch_target(pn):
> +        return pn.endswith('-native') or pn.endswith('-compat')
> +
> +    def extend_provides(pn, provides, d):
> +        if not pn_multiarch_target(pn):
> +            all_provides = (d.getVar('PROVIDES') or '').split()
> +            for p in all_provides:
> +                if not pn_multiarch_target(p):
> +                    d.appendVar('PROVIDES', f' {p}-{provides}')
> +            d.appendVar('PROVIDES', f' {pn}-{provides}')
> +

Could split this patch into 2, the first introducing these helpers
without changing the current logic, and as second one adding the actual
logic of this patch? All this is complex, and it would greatly help with
reviewing when changes are finer-grained.

>      # provide compat only when we can build it
>      if isar_can_build_compat(d):
> -        d.appendVar('BBCLASSEXTEND', ' compat')
> +        # but do not build separately if architecture-independent
> +        if archIsAll:
> +            extend_provides(pn, 'compat', d)
> +        else:
> +            d.appendVar('BBCLASSEXTEND', ' compat')
>  
>      # build native separately only when it differs from the target variant
> -    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
> -        pn = d.getVar('PN')
> -        if not pn.endswith('-native') and not pn.endswith('-compat'):
> -            provides = (d.getVar('PROVIDES') or '').split()
> -            for p in provides:
> -                d.appendVar('PROVIDES', f' {p}-native')
> -            d.appendVar('PROVIDES', f' {pn}-native')
> -    else:
> +    if archIsAll is False and archDiffers:
>          d.appendVar('BBCLASSEXTEND', ' native')
> +    else:
> +        extend_provides(pn, 'native', d)
>  
>      # drop own -native build dependencies at recipe level if building natively
>      # and not for the builder architecture
>      depends = d.getVar('DEPENDS')
> -    if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
> +    if depends is not None and archDiffers \
>         and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
>          new_deps = []
>          for dep in depends.split():

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/202964ce-96ec-47e2-b20c-3cf571717a97%40siemens.com.

Re: [PATCH v2 2/2] dpkg-raw: change DPKG_ARCH to all

['Jan Kiszka' via isar-users]

On 17.09.24 09:00, 'Cedric Hombourger' via isar-users wrote:
> Packages created by dpkg-raw are really for configuration and data files
> and therefore architecture independent. DPKG_ARCH is now set to all and
> a warning will be produced if changed by the recipe (this will later be
> turned into an error).

Well, this is generally correct, but not always. We may also package a
prebuilt arch-dependent thing here, and then the result would no longer
be "all". Such cases should remain expressible correctly.

That said, changing the default is not a bad move.

Jan

> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md       | 11 +++++++++++
>  meta/classes/dpkg-raw.bbclass |  8 ++++++++
>  2 files changed, 19 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 21c558d2..d40827d0 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -646,3 +646,14 @@ recipe would use the following setting:
>  ```
>  HEADERS_INSTALL_EXTRA += "nvidia"
>  ```
> +
> +### Architecture for dpkg-raw packages
> +
> +The intent of the dpkg-raw class is to easily package configuration and data
> +files into a Debian package. Packages to be compiled should really use other
> +dpkg classes where support for cross-compilation and multiarch is provided
> +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes and a
> +deprecation warning will be raised if overriden.
> +
> +This change fixes an issue where a `dpkg` package is built for `-compat`
> +or `-native` and `DEPENDS` on a `dpkg-raw` package.
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index dd7b761f..cbaf298c 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -7,6 +7,9 @@ inherit dpkg
>  
>  D = "${WORKDIR}/image"
>  
> +# Create a binary-indep package
> +DPKG_ARCH = "all"
> +
>  # Populate folder that will be picked up as package
>  do_install() {
>  	bbnote "Put your files for this package in $""{D}"
> @@ -17,6 +20,11 @@ addtask install after do_patch do_transform_template before do_prepare_build
>  
>  do_prepare_build[cleandirs] += "${S}/debian"
>  do_prepare_build() {
> +	if [ "${DPKG_ARCH}" != "all" ]; then
> +		bbwarn "support for building binary-arch packages with " \
> +		       "dpkg-raw is deprecated and will be removed"
> +	fi
> +
>  	cd ${D}
>  	find . -maxdepth 1 ! -name .. -and ! -name . -and ! -name debian | \
>  		sed 's:^./::' > ${S}/debian/${PN}.install

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/e1fb1549-7275-4a33-b5bd-4db9ab1baaf0%40siemens.com.

[PATCH v3 0/3] do not build -compat/-native for dpkg-raw packages

['Cedric Hombourger' via isar-users]

The intent of the dpkg-raw class is to easily package configuration and data
files into a Debian package. Packages to be compiled should really use other
dpkg classes where support for cross-compilation and multiarch is provided
and tested. `DPKG_ARCH` is now defaulting to `all` for `dpkg-raw` recipes.

Despite this change, Isar may still build dpkg-raw packages a second time
when pulled by a -compat or -native package. multiarch is changed to check
if DPKG_ARCH is "all" and will add -compat and -native to the PROVIDES of
the package (this condition will be checked for all dpkg variants)

It shall be noted that setting DPKG_ARCH in a recipe is a contract, the
user declares that generated package(s) are either architecture dependent
or independent.

Changes in v3:
 * Do not warn if DPKG_ARCH is not "all" as users may use this class
   to create a Debian package from a prebuilt binaries (shared libraries,
   applications, etc.)
 * Set DEBIAN_MULTI_ARCH to "foreign" in dpkg-raw if DPKG_ARCH is "all"
 * Introduce helpers in a separate commit to ease reviews (a "fast"
   ci_build run was used to validate this intermediate step).

Changes in v2:
 * v1 used a completely different approach where we were ensuring that
   do_install was called whether sources were built or pulled from
   isar_apt (do_fetch_common_source) since override_dh_install is
   pulling contents of dpkg-raw packages outside of the source tree
   (${PP}/image)

Cedric Hombourger (3):
  multiarch: introduce some helpers to later handle corner cases
  multiarch: avoid separate builds when DPKG_ARCH is all
  dpkg-raw: change DPKG_ARCH to all

 RECIPE-API-CHANGELOG.md        | 10 ++++++++++
 meta/classes/dpkg-raw.bbclass  |  4 ++++
 meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++----------
 3 files changed, 38 insertions(+), 10 deletions(-)

-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240926033935.2955085-1-cedric.hombourger%40siemens.com.

[PATCH v3 1/3] multiarch: introduce some helpers to later handle corner cases

['Cedric Hombourger' via isar-users]

-native / -compat do not always apply and some corner cases were
found. Introduce some helper functions to refactor the existing
code without changing its semantics for now.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/multiarch.bbclass | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index bb0f7983..802686d5 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,25 +5,35 @@
 
 inherit compat
 python() {
+    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
+
+    def pn_multiarch_target(pn):
+        return pn.endswith('-native') or pn.endswith('-compat')
+
+    def extend_provides(pn, provides, d):
+        if not pn_multiarch_target(pn):
+            all_provides = (d.getVar('PROVIDES') or '').split()
+            for p in all_provides:
+                if not pn_multiarch_target(p):
+                    d.appendVar('PROVIDES', f' {p}-{provides}')
+            d.appendVar('PROVIDES', f' {pn}-{provides}')
+
     # provide compat only when we can build it
     if isar_can_build_compat(d):
         d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target variant
-    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
+    if archDiffers is False:
         pn = d.getVar('PN')
-        if not pn.endswith('-native') and not pn.endswith('-compat'):
-            provides = (d.getVar('PROVIDES') or '').split()
-            for p in provides:
-                d.appendVar('PROVIDES', f' {p}-native')
-            d.appendVar('PROVIDES', f' {pn}-native')
+        if pn_multiarch_target(pn) is False:
+            extend_provides(pn, 'native', d)
     else:
         d.appendVar('BBCLASSEXTEND', ' native')
 
     # drop own -native build dependencies at recipe level if building natively
     # and not for the builder architecture
     depends = d.getVar('DEPENDS')
-    if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
+    if depends is not None and archDiffers \
        and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
         new_deps = []
         for dep in depends.split():
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240926033935.2955085-2-cedric.hombourger%40siemens.com.

[PATCH v3 2/3] multiarch: avoid separate builds when DPKG_ARCH is all

['Cedric Hombourger' via isar-users]

Binary-independent packages (DPKG_ARCH == all) do not require separate
builds for our special -compat and -native targets: extend PROVIDES for
these cases. It should be noted that DPKG_ARCH is always set for all
dpkg variants and defaults to "any" (only recipes setting DPKG_ARCH to
"all" explicitly will be impacted).

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/multiarch.bbclass | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 802686d5..9df8eefd 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -5,7 +5,9 @@
 
 inherit compat
 python() {
+    pn = d.getVar('PN')
     archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
+    archIsAll = d.getVar('DPKG_ARCH') == 'all'
 
     def pn_multiarch_target(pn):
         return pn.endswith('-native') or pn.endswith('-compat')
@@ -20,15 +22,17 @@ python() {
 
     # provide compat only when we can build it
     if isar_can_build_compat(d):
-        d.appendVar('BBCLASSEXTEND', ' compat')
+        # but do not build separately if architecture-independent
+        if archIsAll:
+            extend_provides(pn, 'compat', d)
+        else:
+            d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target variant
-    if archDiffers is False:
-        pn = d.getVar('PN')
-        if pn_multiarch_target(pn) is False:
-            extend_provides(pn, 'native', d)
-    else:
+    if archIsAll is False and archDiffers:
         d.appendVar('BBCLASSEXTEND', ' native')
+    else:
+        extend_provides(pn, 'native', d)
 
     # drop own -native build dependencies at recipe level if building natively
     # and not for the builder architecture
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240926033935.2955085-3-cedric.hombourger%40siemens.com.

[PATCH v3 3/3] dpkg-raw: change DPKG_ARCH to all

['Cedric Hombourger' via isar-users]

Packages created by dpkg-raw are really for configuration and data files
and therefore architecture independent. DPKG_ARCH is now set to "all" and
DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is changed
(it would then be reset to "no").

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md       | 10 ++++++++++
 meta/classes/dpkg-raw.bbclass |  4 ++++
 2 files changed, 14 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 21c558d2..57b2205a 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -646,3 +646,13 @@ recipe would use the following setting:
 ```
 HEADERS_INSTALL_EXTRA += "nvidia"
 ```
+
+### Architecture for dpkg-raw packages
+
+The intent of the dpkg-raw class is to easily package configuration and data
+files into a Debian package. Packages to be compiled should really use other
+dpkg classes where support for cross-compilation and multiarch is provided
+and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+
+This change fixes an issue where a `dpkg` package is built for `-compat` or
+`-native` and `DEPENDS` on a `dpkg-raw` package.
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..ac70d5cc 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,6 +7,10 @@ inherit dpkg
 
 D = "${WORKDIR}/image"
 
+# Create a binary-indep package
+DPKG_ARCH = "all"
+DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
+
 # Populate folder that will be picked up as package
 do_install() {
 	bbnote "Put your files for this package in $""{D}"
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240926033935.2955085-4-cedric.hombourger%40siemens.com.

Re: [PATCH v3 0/3] do not build -compat/-native for dpkg-raw packages

[Uladzimir Bely]

On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
wrote:
> The intent of the dpkg-raw class is to easily package configuration
> and data
> files into a Debian package. Packages to be compiled should really
> use other
> dpkg classes where support for cross-compilation and multiarch is
> provided
> and tested. `DPKG_ARCH` is now defaulting to `all` for `dpkg-raw`
> recipes.
> 
> Despite this change, Isar may still build dpkg-raw packages a second
> time
> when pulled by a -compat or -native package. multiarch is changed to
> check
> if DPKG_ARCH is "all" and will add -compat and -native to the
> PROVIDES of
> the package (this condition will be checked for all dpkg variants)
> 
> It shall be noted that setting DPKG_ARCH in a recipe is a contract,
> the
> user declares that generated package(s) are either architecture
> dependent
> or independent.
> 
> Changes in v3:
>  * Do not warn if DPKG_ARCH is not "all" as users may use this class
>    to create a Debian package from a prebuilt binaries (shared
> libraries,
>    applications, etc.)
>  * Set DEBIAN_MULTI_ARCH to "foreign" in dpkg-raw if DPKG_ARCH is
> "all"
>  * Introduce helpers in a separate commit to ease reviews (a "fast"
>    ci_build run was used to validate this intermediate step).
> 
> Changes in v2:
>  * v1 used a completely different approach where we were ensuring
> that
>    do_install was called whether sources were built or pulled from
>    isar_apt (do_fetch_common_source) since override_dh_install is
>    pulling contents of dpkg-raw packages outside of the source tree
>    (${PP}/image)
> 
> Cedric Hombourger (3):
>   multiarch: introduce some helpers to later handle corner cases
>   multiarch: avoid separate builds when DPKG_ARCH is all
>   dpkg-raw: change DPKG_ARCH to all
> 
>  RECIPE-API-CHANGELOG.md        | 10 ++++++++++
>  meta/classes/dpkg-raw.bbclass  |  4 ++++
>  meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++--------
> --
>  3 files changed, 38 insertions(+), 10 deletions(-)
> 
> -- 
> 2.34.1
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/9927afac2c263d856ede2040a6ef19040a507d52.camel%40ilbers.de.

Re: [PATCH v3 0/3] do not build -compat/-native for dpkg-raw packages

['Jan Kiszka' via isar-users]

On 04.10.24 13:54, Uladzimir Bely wrote:
> On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
> wrote:
>> The intent of the dpkg-raw class is to easily package configuration
>> and data
>> files into a Debian package. Packages to be compiled should really
>> use other
>> dpkg classes where support for cross-compilation and multiarch is
>> provided
>> and tested. `DPKG_ARCH` is now defaulting to `all` for `dpkg-raw`
>> recipes.
>>
>> Despite this change, Isar may still build dpkg-raw packages a second
>> time
>> when pulled by a -compat or -native package. multiarch is changed to
>> check
>> if DPKG_ARCH is "all" and will add -compat and -native to the
>> PROVIDES of
>> the package (this condition will be checked for all dpkg variants)
>>
>> It shall be noted that setting DPKG_ARCH in a recipe is a contract,
>> the
>> user declares that generated package(s) are either architecture
>> dependent
>> or independent.
>>
>> Changes in v3:
>>  * Do not warn if DPKG_ARCH is not "all" as users may use this class
>>    to create a Debian package from a prebuilt binaries (shared
>> libraries,
>>    applications, etc.)
>>  * Set DEBIAN_MULTI_ARCH to "foreign" in dpkg-raw if DPKG_ARCH is
>> "all"
>>  * Introduce helpers in a separate commit to ease reviews (a "fast"
>>    ci_build run was used to validate this intermediate step).
>>
>> Changes in v2:
>>  * v1 used a completely different approach where we were ensuring
>> that
>>    do_install was called whether sources were built or pulled from
>>    isar_apt (do_fetch_common_source) since override_dh_install is
>>    pulling contents of dpkg-raw packages outside of the source tree
>>    (${PP}/image)
>>
>> Cedric Hombourger (3):
>>   multiarch: introduce some helpers to later handle corner cases
>>   multiarch: avoid separate builds when DPKG_ARCH is all
>>   dpkg-raw: change DPKG_ARCH to all
>>
>>  RECIPE-API-CHANGELOG.md        | 10 ++++++++++
>>  meta/classes/dpkg-raw.bbclass  |  4 ++++
>>  meta/classes/multiarch.bbclass | 34 ++++++++++++++++++++++++--------
>> --
>>  3 files changed, 38 insertions(+), 10 deletions(-)
>>
>> -- 
>> 2.34.1
>>
> 
> Applied to next, thanks.
> 

Sorry, forgotten to review. We will need fixes on top now.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ae9229bc-3b1c-4161-bf4e-d121ae22ca4a%40siemens.com.

Re: [PATCH v3 1/3] multiarch: introduce some helpers to later handle corner cases

['Jan Kiszka' via isar-users]

On 26.09.24 05:39, 'Cedric Hombourger' via isar-users wrote:
> -native / -compat do not always apply and some corner cases were
> found. Introduce some helper functions to refactor the existing
> code without changing its semantics for now.
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  meta/classes/multiarch.bbclass | 24 +++++++++++++++++-------
>  1 file changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
> index bb0f7983..802686d5 100644
> --- a/meta/classes/multiarch.bbclass
> +++ b/meta/classes/multiarch.bbclass
> @@ -5,25 +5,35 @@
>  
>  inherit compat
>  python() {
> +    archDiffers = d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH')
> +
> +    def pn_multiarch_target(pn):
> +        return pn.endswith('-native') or pn.endswith('-compat')
> +
> +    def extend_provides(pn, provides, d):
> +        if not pn_multiarch_target(pn):

Redundant? You are already testing this...

> +            all_provides = (d.getVar('PROVIDES') or '').split()
> +            for p in all_provides:
> +                if not pn_multiarch_target(p):
> +                    d.appendVar('PROVIDES', f' {p}-{provides}')
> +            d.appendVar('PROVIDES', f' {pn}-{provides}')
> +
>      # provide compat only when we can build it
>      if isar_can_build_compat(d):
>          d.appendVar('BBCLASSEXTEND', ' compat')
>  
>      # build native separately only when it differs from the target variant
> -    if d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
> +    if archDiffers is False:
>          pn = d.getVar('PN')
> -        if not pn.endswith('-native') and not pn.endswith('-compat'):
> -            provides = (d.getVar('PROVIDES') or '').split()
> -            for p in provides:
> -                d.appendVar('PROVIDES', f' {p}-native')
> -            d.appendVar('PROVIDES', f' {pn}-native')
> +        if pn_multiarch_target(pn) is False:

...here. And you are using different styles ("not" vs "is False").

But these lines get reworked again with patch 2 which at least resolves
the duplicate test. But check again if you are using consistent styles.

> +            extend_provides(pn, 'native', d)
>      else:
>          d.appendVar('BBCLASSEXTEND', ' native')
>  
>      # drop own -native build dependencies at recipe level if building natively
>      # and not for the builder architecture
>      depends = d.getVar('DEPENDS')
> -    if depends is not None and d.getVar('HOST_ARCH') != d.getVar('DISTRO_ARCH') \
> +    if depends is not None and archDiffers \
>         and not bb.utils.to_boolean(d.getVar('ISAR_CROSS_COMPILE')):
>          new_deps = []
>          for dep in depends.split():

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/507dfa6c-8fc7-415c-8f45-4f96ebfb57ad%40siemens.com.

Re: [PATCH v3 3/3] dpkg-raw: change DPKG_ARCH to all

['Jan Kiszka' via isar-users]

On 26.09.24 05:39, 'Cedric Hombourger' via isar-users wrote:
> Packages created by dpkg-raw are really for configuration and data files
> and therefore architecture independent. DPKG_ARCH is now set to "all" and
> DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is changed
> (it would then be reset to "no").
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md       | 10 ++++++++++
>  meta/classes/dpkg-raw.bbclass |  4 ++++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 21c558d2..57b2205a 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -646,3 +646,13 @@ recipe would use the following setting:
>  ```
>  HEADERS_INSTALL_EXTRA += "nvidia"
>  ```
> +
> +### Architecture for dpkg-raw packages
> +
> +The intent of the dpkg-raw class is to easily package configuration and data
> +files into a Debian package. Packages to be compiled should really use other
> +dpkg classes where support for cross-compilation and multiarch is provided
> +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.

Not fully true. In rare cases, it can also be used for pre-compiled
binaries. This sections should state that the /common/ case is 'all' and
therefore the /default/ is set to 'all'.

> +
> +This change fixes an issue where a `dpkg` package is built for `-compat` or
> +`-native` and `DEPENDS` on a `dpkg-raw` package.

This is misleading. One can still tune DPKG_ARCH and run into such
cases. Warn about them, don't claim they are fixed.

> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index dd7b761f..ac70d5cc 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -7,6 +7,10 @@ inherit dpkg
>  
>  D = "${WORKDIR}/image"
>  
> +# Create a binary-indep package
> +DPKG_ARCH = "all"

Weak assignment, please.

> +DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
> +
>  # Populate folder that will be picked up as package
>  do_install() {
>  	bbnote "Put your files for this package in $""{D}"

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/1ecf87ea-beb3-4e5c-bc1e-3f4b404b5421%40siemens.com.

[PATCH 1/3] dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH

['Cedric Hombourger' via isar-users]

Fixes: 0816ae6e
Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index ac70d5cc..a7bf204a 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,9 +7,9 @@ inherit dpkg
 
 D = "${WORKDIR}/image"
 
-# Create a binary-indep package
-DPKG_ARCH = "all"
-DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
+# Default to creating a binary-indep package
+DPKG_ARCH ??= "all"
+DEBIAN_MULTI_ARCH ??= "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
 
 # Populate folder that will be picked up as package
 do_install() {
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004131124.163444-1-cedric.hombourger%40siemens.com.

[PATCH 2/3] multiarch: use "if not" vs "if cond is False" for consistency

['Cedric Hombourger' via isar-users]

Fixes: 13ef1d48
Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/multiarch.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 9df8eefd..babdfbd4 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -29,7 +29,7 @@ python() {
             d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target variant
-    if archIsAll is False and archDiffers:
+    if not archIsAll and archDiffers:
         d.appendVar('BBCLASSEXTEND', ' native')
     else:
         extend_provides(pn, 'native', d)
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004131124.163444-2-cedric.hombourger%40siemens.com.

[PATCH 3/3] RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-arch changes

['Cedric Hombourger' via isar-users]

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 57b2205a..8c1e8fc8 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -649,10 +649,14 @@ HEADERS_INSTALL_EXTRA += "nvidia"
 
 ### Architecture for dpkg-raw packages
 
-The intent of the dpkg-raw class is to easily package configuration and data
-files into a Debian package. Packages to be compiled should really use other
-dpkg classes where support for cross-compilation and multiarch is provided
-and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+The primary use-case of the dpkg-raw class is to easily package configuration
+and data files into a Debian package: the target architecture will now default
+to "all". It may also be used to package binaries that were built outside of
+Isar: such recipes may still override `DPKG_ARCH` to `"any"`.
 
 This change fixes an issue where a `dpkg` package is built for `-compat` or
-`-native` and `DEPENDS` on a `dpkg-raw` package.
+`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.
+Some issues remain with `dpkg-raw` packages: Isar will advertise -native and
+-compat variants even though such recipes can only produce packages for
+`${DISTRO_ARCH}` or `"all"`. That corner case needs to be further discussed
+and will require further changes.
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004131124.163444-3-cedric.hombourger%40siemens.com.

Re: [PATCH 3/3] RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-arch changes

['Jan Kiszka' via isar-users]

On 04.10.24 15:11, 'Cedric Hombourger' via isar-users wrote:
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 57b2205a..8c1e8fc8 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -649,10 +649,14 @@ HEADERS_INSTALL_EXTRA += "nvidia"
>  
>  ### Architecture for dpkg-raw packages
>  
> -The intent of the dpkg-raw class is to easily package configuration and data
> -files into a Debian package. Packages to be compiled should really use other
> -dpkg classes where support for cross-compilation and multiarch is provided
> -and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
> +The primary use-case of the dpkg-raw class is to easily package configuration
> +and data files into a Debian package: the target architecture will now default
> +to "all". It may also be used to package binaries that were built outside of
> +Isar: such recipes may still override `DPKG_ARCH` to `"any"`.
>  

...to 'any' or whatever reflects the package content accurately.

>  This change fixes an issue where a `dpkg` package is built for `-compat` or
> -`-native` and `DEPENDS` on a `dpkg-raw` package.
> +`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.
> +Some issues remain with `dpkg-raw` packages: Isar will advertise -native and
> +-compat variants even though such recipes can only produce packages for
> +`${DISTRO_ARCH}` or `"all"`. That corner case needs to be further discussed
> +and will require further changes.

The fact that -native/-compat stub targets are provided for 'all'
packages is not the issue. The issue is limited to DPKG_ARCH != all.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/62b2d7bb-25cf-4c42-9bee-fa95d18db6c8%40siemens.com.

[PATCH v2 0/3] address review comments for dpkg-raw/multiarch

['Cedric Hombourger' via isar-users]

Changes in v2:
  Attempt to address feedback [1] from Jan in the RECIPE-API-CHANGELOG
  First two patches (code changes) are the same.

[1] https://lists.isar-build.org/isar-users/62b2d7bb-25cf-4c42-9bee-fa95d18db6c8@siemens.com/T/#u

Cedric Hombourger (3):
  dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH
  multiarch: use "if not" vs "if cond is False" for consistency
  RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-arch
    changes

 RECIPE-API-CHANGELOG.md        | 19 ++++++++++++++-----
 meta/classes/dpkg-raw.bbclass  |  6 +++---
 meta/classes/multiarch.bbclass |  2 +-
 3 files changed, 18 insertions(+), 9 deletions(-)

-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004141731.172009-1-cedric.hombourger%40siemens.com.

[PATCH v2 1/3] dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH

['Cedric Hombourger' via isar-users]

Fixes: 0816ae6e
Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/dpkg-raw.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index ac70d5cc..a7bf204a 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -7,9 +7,9 @@ inherit dpkg
 
 D = "${WORKDIR}/image"
 
-# Create a binary-indep package
-DPKG_ARCH = "all"
-DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
+# Default to creating a binary-indep package
+DPKG_ARCH ??= "all"
+DEBIAN_MULTI_ARCH ??= "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else 'no' }"
 
 # Populate folder that will be picked up as package
 do_install() {
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004141731.172009-2-cedric.hombourger%40siemens.com.

[PATCH v2 2/3] multiarch: use "if not" vs "if cond is False" for consistency

['Cedric Hombourger' via isar-users]

Fixes: 13ef1d48
Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/multiarch.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 9df8eefd..babdfbd4 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -29,7 +29,7 @@ python() {
             d.appendVar('BBCLASSEXTEND', ' compat')
 
     # build native separately only when it differs from the target variant
-    if archIsAll is False and archDiffers:
+    if not archIsAll and archDiffers:
         d.appendVar('BBCLASSEXTEND', ' native')
     else:
         extend_provides(pn, 'native', d)
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004141731.172009-3-cedric.hombourger%40siemens.com.

[PATCH v2 3/3] RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-arch changes

['Cedric Hombourger' via isar-users]

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 RECIPE-API-CHANGELOG.md | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 57b2205a..8eeaf325 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -649,10 +649,19 @@ HEADERS_INSTALL_EXTRA += "nvidia"
 
 ### Architecture for dpkg-raw packages
 
-The intent of the dpkg-raw class is to easily package configuration and data
-files into a Debian package. Packages to be compiled should really use other
-dpkg classes where support for cross-compilation and multiarch is provided
-and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
+The primary use-case of the dpkg-raw class is to easily package configuration
+and data files into a Debian package: the target architecture will now default
+to "all". It may also be used to package binaries that were built outside of
+Isar: such recipes may still override `DPKG_ARCH` to `"any"` or a specific
+architecture matching binaries to be included in the payload of the package.
 
 This change fixes an issue where a `dpkg` package is built for `-compat` or
-`-native` and `DEPENDS` on a `dpkg-raw` package.
+`-native` and `DEPENDS` on a `dpkg-raw` package with `DPKG_ARCH` set to `"all"`.
+Some issues remain with `dpkg-raw` packages targetting a specific architecture:
+Isar will advertise -native and -compat variants even though such recipes can
+only produce packages for that architecture and not what could possibly expect
+for -native or -compat. If we consider a dpkg-raw recipe generating an `arm64`
+package on an `amd64` host: you would expect the -native variant to produce
+an `amd64` package and -compat an 'armhf` package: it will however remain
+`arm64` and build of dependent recipes (image or dpkg) may fail because of
+the architecture mismatch.
-- 
2.39.5

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20241004141731.172009-4-cedric.hombourger%40siemens.com.

Re: [PATCH v2 0/3] address review comments for dpkg-raw/multiarch

[Uladzimir Bely]

On Fri, 2024-10-04 at 16:17 +0200, 'Cedric Hombourger' via isar-users
wrote:
> Changes in v2:
>   Attempt to address feedback [1] from Jan in the RECIPE-API-
> CHANGELOG
>   First two patches (code changes) are the same.
> 
> [1]
> https://lists.isar-build.org/isar-users/62b2d7bb-25cf-4c42-9bee-fa95d18db6c8@siemens.com/T/#u
> 
> Cedric Hombourger (3):
>   dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH
>   multiarch: use "if not" vs "if cond is False" for consistency
>   RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-
> arch
>     changes
> 
>  RECIPE-API-CHANGELOG.md        | 19 ++++++++++++++-----
>  meta/classes/dpkg-raw.bbclass  |  6 +++---
>  meta/classes/multiarch.bbclass |  2 +-
>  3 files changed, 18 insertions(+), 9 deletions(-)
> 
> -- 
> 2.39.5
> 

Hello all.

The patchset is tested and ready to be merged. Since it's technically a
set of fixes for already merged things, we'd like to apply it to next
as soon as possible, if there are no objections.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/810faecbbf589b041aba026cd38fdef3e522ba98.camel%40ilbers.de.

Re: [PATCH v2 0/3] address review comments for dpkg-raw/multiarch

[Uladzimir Bely]

On Fri, 2024-10-04 at 16:17 +0200, 'Cedric Hombourger' via isar-users
wrote:
> Changes in v2:
>   Attempt to address feedback [1] from Jan in the RECIPE-API-
> CHANGELOG
>   First two patches (code changes) are the same.
> 
> [1]
> https://lists.isar-build.org/isar-users/62b2d7bb-25cf-4c42-9bee-fa95d18db6c8@siemens.com/T/#u
> 
> Cedric Hombourger (3):
>   dpkg-raw: use weak assignments for DPKG_ARCH and DEBIAN_MULTI_ARCH
>   multiarch: use "if not" vs "if cond is False" for consistency
>   RECIPE-API-CHANGELOG.md: clarify scope of recent multiarch/dpkg-
> arch
>     changes
> 
>  RECIPE-API-CHANGELOG.md        | 19 ++++++++++++++-----
>  meta/classes/dpkg-raw.bbclass  |  6 +++---
>  meta/classes/multiarch.bbclass |  2 +-
>  3 files changed, 18 insertions(+), 9 deletions(-)
> 
> -- 
> 2.39.5
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/4d48fbd24d10a4e4408521655f8df302e8de5985.camel%40ilbers.de.

Re: [PATCH v3 3/3] dpkg-raw: change DPKG_ARCH to all

[Uladzimir Bely]

On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
wrote:
> Packages created by dpkg-raw are really for configuration and data
> files
> and therefore architecture independent. DPKG_ARCH is now set to "all"
> and
> DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is
> changed
> (it would then be reset to "no").
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  RECIPE-API-CHANGELOG.md       | 10 ++++++++++
>  meta/classes/dpkg-raw.bbclass |  4 ++++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index 21c558d2..57b2205a 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -646,3 +646,13 @@ recipe would use the following setting:
>  ```
>  HEADERS_INSTALL_EXTRA += "nvidia"
>  ```
> +
> +### Architecture for dpkg-raw packages
> +
> +The intent of the dpkg-raw class is to easily package configuration
> and data
> +files into a Debian package. Packages to be compiled should really
> use other
> +dpkg classes where support for cross-compilation and multiarch is
> provided
> +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw` recipes.
> +
> +This change fixes an issue where a `dpkg` package is built for `-
> compat` or
> +`-native` and `DEPENDS` on a `dpkg-raw` package.
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-
> raw.bbclass
> index dd7b761f..ac70d5cc 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -7,6 +7,10 @@ inherit dpkg
>  
>  D = "${WORKDIR}/image"
>  
> +# Create a binary-indep package
> +DPKG_ARCH = "all"
> +DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else
> 'no' }"
> +

Playing with isar-apt I've found an issue that existed for a long time
but with this patch it became easier to find.

For the packages with "all" architecture there are no source packages
found in isar-apt.

Steps to reproduce:

1. Run `bitbake libhello hello example-raw sshd-regen-keys`.
2. Check isar-apt contents (e.g. `build/tmp/deploy/isar-apt/debian-
bookworm-amd64/apt/debian-bookworm/pool/main/`)

With current `next` we have in isar-apt:

```
libhello            dsc + deb
hello               dsc + deb
example-raw         only deb
sshd-regen-keys     only deb
```

With this patch reverted (and some followed patches):

```
libhello            dsc + deb
hello               dsc + deb
example-raw         dsc + deb
sshd-regen-keys     only deb
```

Package `sshd-regen-keys` is still affected since it explicitly sets
DPKG_ARCH = "all" in the recipe.

Debugging shows that source package goes to the isar-apt but later
is wrongly removed by `repo_del_package` from `repository.bbclass` due
to "all" architecture.

>  # Populate folder that will be picked up as package
>  do_install() {
>  	bbnote "Put your files for this package in $""{D}"
> -- 
> 2.34.1
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/5e56d7f698c9b25eea59373e904e24f2c9c91746.camel%40ilbers.de.

Re: [PATCH v3 3/3] dpkg-raw: change DPKG_ARCH to all

[Uladzimir Bely]

On Fri, 2024-11-22 at 09:07 +0300, Uladzimir Bely wrote:
> On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
> wrote:
> > Packages created by dpkg-raw are really for configuration and data
> > files
> > and therefore architecture independent. DPKG_ARCH is now set to
> > "all"
> > and
> > DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is
> > changed
> > (it would then be reset to "no").
> > 
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md       | 10 ++++++++++
> >  meta/classes/dpkg-raw.bbclass |  4 ++++
> >  2 files changed, 14 insertions(+)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 21c558d2..57b2205a 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -646,3 +646,13 @@ recipe would use the following setting:
> >  ```
> >  HEADERS_INSTALL_EXTRA += "nvidia"
> >  ```
> > +
> > +### Architecture for dpkg-raw packages
> > +
> > +The intent of the dpkg-raw class is to easily package
> > configuration
> > and data
> > +files into a Debian package. Packages to be compiled should really
> > use other
> > +dpkg classes where support for cross-compilation and multiarch is
> > provided
> > +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw`
> > recipes.
> > +
> > +This change fixes an issue where a `dpkg` package is built for `-
> > compat` or
> > +`-native` and `DEPENDS` on a `dpkg-raw` package.
> > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-
> > raw.bbclass
> > index dd7b761f..ac70d5cc 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -7,6 +7,10 @@ inherit dpkg
> >  
> >  D = "${WORKDIR}/image"
> >  
> > +# Create a binary-indep package
> > +DPKG_ARCH = "all"
> > +DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else
> > 'no' }"
> > +
> 
> Playing with isar-apt I've found an issue that existed for a long
> time
> but with this patch it became easier to find.
> 
> For the packages with "all" architecture there are no source packages
> found in isar-apt.
> 
> Steps to reproduce:
> 
> 1. Run `bitbake libhello hello example-raw sshd-regen-keys`.
> 2. Check isar-apt contents (e.g. `build/tmp/deploy/isar-apt/debian-
> bookworm-amd64/apt/debian-bookworm/pool/main/`)
> 
> With current `next` we have in isar-apt:
> 
> ```
> libhello            dsc + deb
> hello               dsc + deb
> example-raw         only deb
> sshd-regen-keys     only deb
> ```
> 
> With this patch reverted (and some followed patches):
> 
> ```
> libhello            dsc + deb
> hello               dsc + deb
> example-raw         dsc + deb
> sshd-regen-keys     only deb
> ```
> 
> Package `sshd-regen-keys` is still affected since it explicitly sets
> DPKG_ARCH = "all" in the recipe.
> 
> Debugging shows that source package goes to the isar-apt but later
> is wrongly removed by `repo_del_package` from `repository.bbclass`
> due
> to "all" architecture.
> 

@Cedric

Oh, while I was writing this email you seem to have sent a patchset
that fixes the issue. Thanks, will check it.

> >  # Populate folder that will be picked up as package
> >  do_install() {
> >  	bbnote "Put your files for this package in $""{D}"
> > -- 
> > 2.34.1
> > 
> 
> -- 
> Best regards,
> Uladzimir.
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/1e9a5535961e2d2a597b3bf6c5141d88ab26e4d3.camel%40ilbers.de.

Re: [PATCH v3 3/3] dpkg-raw: change DPKG_ARCH to all

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-11-22 at 09:07 +0300, Uladzimir Bely wrote:
> On Thu, 2024-09-26 at 05:39 +0200, 'Cedric Hombourger' via isar-users
> wrote:
> > Packages created by dpkg-raw are really for configuration and data
> > files
> > and therefore architecture independent. DPKG_ARCH is now set to
> > "all"
> > and
> > DEBIAN_MULTI_ARCH will default to "foreign" unless DPKG_ARCH is
> > changed
> > (it would then be reset to "no").
> > 
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >  RECIPE-API-CHANGELOG.md       | 10 ++++++++++
> >  meta/classes/dpkg-raw.bbclass |  4 ++++
> >  2 files changed, 14 insertions(+)
> > 
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 21c558d2..57b2205a 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -646,3 +646,13 @@ recipe would use the following setting:
> >  ```
> >  HEADERS_INSTALL_EXTRA += "nvidia"
> >  ```
> > +
> > +### Architecture for dpkg-raw packages
> > +
> > +The intent of the dpkg-raw class is to easily package
> > configuration
> > and data
> > +files into a Debian package. Packages to be compiled should really
> > use other
> > +dpkg classes where support for cross-compilation and multiarch is
> > provided
> > +and tested. `DPKG_ARCH` is now set to `all` for `dpkg-raw`
> > recipes.
> > +
> > +This change fixes an issue where a `dpkg` package is built for `-
> > compat` or
> > +`-native` and `DEPENDS` on a `dpkg-raw` package.
> > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-
> > raw.bbclass
> > index dd7b761f..ac70d5cc 100644
> > --- a/meta/classes/dpkg-raw.bbclass
> > +++ b/meta/classes/dpkg-raw.bbclass
> > @@ -7,6 +7,10 @@ inherit dpkg
> >  
> >  D = "${WORKDIR}/image"
> >  
> > +# Create a binary-indep package
> > +DPKG_ARCH = "all"
> > +DEBIAN_MULTI_ARCH = "${@ 'foreign' if '${DPKG_ARCH}' == 'all' else
> > 'no' }"
> > +
> 
> Playing with isar-apt I've found an issue that existed for a long
> time
> but with this patch it became easier to find.

See https://groups.google.com/g/isar-users/c/NrVHusKyNTA for a fix
Looks like good timing :)

> 
> For the packages with "all" architecture there are no source packages
> found in isar-apt.
> 
> Steps to reproduce:
> 
> 1. Run `bitbake libhello hello example-raw sshd-regen-keys`.
> 2. Check isar-apt contents (e.g. `build/tmp/deploy/isar-apt/debian-
> bookworm-amd64/apt/debian-bookworm/pool/main/`)
> 
> With current `next` we have in isar-apt:
> 
> ```
> libhello            dsc + deb
> hello               dsc + deb
> example-raw         only deb
> sshd-regen-keys     only deb
> ```
> 
> With this patch reverted (and some followed patches):
> 
> ```
> libhello            dsc + deb
> hello               dsc + deb
> example-raw         dsc + deb
> sshd-regen-keys     only deb
> ```
> 
> Package `sshd-regen-keys` is still affected since it explicitly sets
> DPKG_ARCH = "all" in the recipe.
> 
> Debugging shows that source package goes to the isar-apt but later
> is wrongly removed by `repo_del_package` from `repository.bbclass`
> due
> to "all" architecture.
> 
> >  # Populate folder that will be picked up as package
> >  do_install() {
> >         bbnote "Put your files for this package in $""{D}"
> > -- 
> > 2.34.1
> > 
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/777fdbbacc9aa34dee2c5766916f4b6874f084e0.camel%40siemens.com.