[PATCH] image: allow multiple image recipes to deploy DTB_FILES

[PATCH] image: allow multiple image recipes to deploy DTB_FILES

['Cedric Hombourger' via isar-users]

sstate is checking for overlapping files in DEPLOY_DIR and will
raise an error when building a second image for a machine that
uses DTB_FILES.

Reproducer:
  bitbake mc:phyboard-mira-bookworm:isar-image-base
  bitbake mc:phyboard-mira-bookworm:isar-image-debug

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 meta/classes/image.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c29d9e26..8b316c83 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
 INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or '${IMAGE_FULLNAME}-initrd.img'}"
 
 # This defines the deployed dtbs for reuse by imagers
+# Since we may be building several images with the same set of DTB_FILES, silent sstate
+# overlap checks
 DTB_FILES ?= ""
+DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' + os.path.basename(dtb) for dtb in d.getVar('DTB_FILES').split()]) }"
+SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
 
 # Useful variables for imager implementations:
 PP = "/home/builder/${PN}-${MACHINE}"
-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240705071856.904-1-cedric.hombourger%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['Jan Kiszka' via isar-users]

On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
> sstate is checking for overlapping files in DEPLOY_DIR and will
> raise an error when building a second image for a machine that
> uses DTB_FILES.
> 
> Reproducer:
>   bitbake mc:phyboard-mira-bookworm:isar-image-base
>   bitbake mc:phyboard-mira-bookworm:isar-image-debug
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  meta/classes/image.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..8b316c83 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
>  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or '${IMAGE_FULLNAME}-initrd.img'}"
>  
>  # This defines the deployed dtbs for reuse by imagers
> +# Since we may be building several images with the same set of DTB_FILES, silent sstate
> +# overlap checks
>  DTB_FILES ?= ""
> +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' + os.path.basename(dtb) for dtb in d.getVar('DTB_FILES').split()]) }"
> +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
>  
>  # Useful variables for imager implementations:
>  PP = "/home/builder/${PN}-${MACHINE}"

There is still
https://patchwork.isar-build.org/project/isar/list/?series=1209 pending
which looks much less risky to paper over a real conflict.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2f1bad7a-5e81-468b-8c75-804e7cee1a49%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
> On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
> > sstate is checking for overlapping files in DEPLOY_DIR and will
> > raise an error when building a second image for a machine that
> > uses DTB_FILES.
> >
> > Reproducer:
> >   bitbake mc:phyboard-mira-bookworm:isar-image-base
> >   bitbake mc:phyboard-mira-bookworm:isar-image-debug
> >
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >  meta/classes/image.bbclass | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/meta/classes/image.bbclass
> > b/meta/classes/image.bbclass
> > index c29d9e26..8b316c83 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
> >  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
> > '${IMAGE_FULLNAME}-initrd.img'}"
> >
> >  # This defines the deployed dtbs for reuse by imagers
> > +# Since we may be building several images with the same set of
> > DTB_FILES, silent sstate
> > +# overlap checks
> >  DTB_FILES ?= ""
> > +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
> > os.path.basename(dtb) for dtb in d.getVar('DTB_FILES').split()]) }"
> > +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
> >
> >  # Useful variables for imager implementations:
> >  PP = "/home/builder/${PN}-${MACHINE}"
>
> There is still
> https://patchwork.isar-build.org/project/isar/list/?series=1209
>  pending
> which looks much less risky to paper over a real conflict.

Thanks Jan. I had missed this RFC patch series.

That said, while it will solve the problem we are having with building
multiple images with MACHINEs using DTB_FILES, it does not seem to
solve the problem of a build where we are building multiple kernels
(e.g. -rt and non-rt) for a given MACHINE: both kernels will be
shipping the same DTB_FILES.

On our side, we happen to have that use-case on our side as well.

Cedric

>
> Jan
>

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/4c2171a5857d5c22ff1810f237235b561a13654c.camel%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

[Uladzimir Bely]

On Fri, 2024-07-05 at 07:47 +0000, 'cedric.hombourger@siemens.com' via
isar-users wrote:
> On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
> > On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
> > > sstate is checking for overlapping files in DEPLOY_DIR and will
> > > raise an error when building a second image for a machine that
> > > uses DTB_FILES.
> > > 
> > > Reproducer:
> > >   bitbake mc:phyboard-mira-bookworm:isar-image-base
> > >   bitbake mc:phyboard-mira-bookworm:isar-image-debug
> > > 
> > > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > > ---
> > >  meta/classes/image.bbclass | 4 ++++
> > >  1 file changed, 4 insertions(+)
> > > 
> > > diff --git a/meta/classes/image.bbclass
> > > b/meta/classes/image.bbclass
> > > index c29d9e26..8b316c83 100644
> > > --- a/meta/classes/image.bbclass
> > > +++ b/meta/classes/image.bbclass
> > > @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
> > >  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
> > > '${IMAGE_FULLNAME}-initrd.img'}"
> > > 
> > >  # This defines the deployed dtbs for reuse by imagers
> > > +# Since we may be building several images with the same set of
> > > DTB_FILES, silent sstate
> > > +# overlap checks
> > >  DTB_FILES ?= ""
> > > +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
> > > os.path.basename(dtb) for dtb in d.getVar('DTB_FILES').split()])
> > > }"
> > > +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
> > > 
> > >  # Useful variables for imager implementations:
> > >  PP = "/home/builder/${PN}-${MACHINE}"
> > 
> > There is still
> > https://patchwork.isar-build.org/project/isar/list/?series=1209
> >  pending
> > which looks much less risky to paper over a real conflict.
> 
> Thanks Jan. I had missed this RFC patch series.
> 
> That said, while it will solve the problem we are having with
> building
> multiple images with MACHINEs using DTB_FILES, it does not seem to
> solve the problem of a build where we are building multiple kernels
> (e.g. -rt and non-rt) for a given MACHINE: both kernels will be
> shipping the same DTB_FILES.
> 
> On our side, we happen to have that use-case on our side as well.
> 
> Cedric


Hello all.

Yes, patchset v2 (non-RFC) for this is still in "work-in-progress"
state. There are some issues we faced with:
 - Parallell building issue. If some machines of same arch use the same
linux recipe, but have different "DTB_FILES" value, we have sstate-
related conflict for the linux recipe ("multiple execution
detected..."). This is currently solved by splitting workdirs by adding
"-${MACHINE}" prefix here.
 - Since "linux-distro.bb" now inherits dpkg-raw bbclass, it builds
something. And in case of empty DTB_FILES it looks a bit useless (we
build empty deb in fact).


> 
> > 
> > Jan
> > 
> 
> --
> Cedric Hombourger
> Siemens AG
> http://www.siemens.com/
> 

-- 
Best regards,
Uladzimir.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2d99f3d0a9f222be54e4685c6a28ec0022fe9c5c.camel%40ilbers.de.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['Jan Kiszka' via isar-users]

On 05.07.24 09:47, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
>> On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
>>> sstate is checking for overlapping files in DEPLOY_DIR and will
>>> raise an error when building a second image for a machine that
>>> uses DTB_FILES.
>>>
>>> Reproducer:
>>>   bitbake mc:phyboard-mira-bookworm:isar-image-base
>>>   bitbake mc:phyboard-mira-bookworm:isar-image-debug
>>>
>>> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
>>> ---
>>>  meta/classes/image.bbclass | 4 ++++
>>>  1 file changed, 4 insertions(+)
>>>
>>> diff --git a/meta/classes/image.bbclass
>>> b/meta/classes/image.bbclass
>>> index c29d9e26..8b316c83 100644
>>> --- a/meta/classes/image.bbclass
>>> +++ b/meta/classes/image.bbclass
>>> @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
>>>  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
>>> '${IMAGE_FULLNAME}-initrd.img'}"
>>>
>>>  # This defines the deployed dtbs for reuse by imagers
>>> +# Since we may be building several images with the same set of
>>> DTB_FILES, silent sstate
>>> +# overlap checks
>>>  DTB_FILES ?= ""
>>> +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
>>> os.path.basename(dtb) for dtb in d.getVar('DTB_FILES').split()]) }"
>>> +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
>>>
>>>  # Useful variables for imager implementations:
>>>  PP = "/home/builder/${PN}-${MACHINE}"
>>
>> There is still
>> https://patchwork.isar-build.org/project/isar/list/?series=1209
>>  pending
>> which looks much less risky to paper over a real conflict.
> 
> Thanks Jan. I had missed this RFC patch series.
> 
> That said, while it will solve the problem we are having with building
> multiple images with MACHINEs using DTB_FILES, it does not seem to
> solve the problem of a build where we are building multiple kernels
> (e.g. -rt and non-rt) for a given MACHINE: both kernels will be
> shipping the same DTB_FILES.

Adding Uladzimir: The risk of SSTATE_ALLOW_OVERLAP_FILES, also with your
scenario, remains that those kernels are not coming with identical
sources and, thus, with potentially different DTBs.

Jan

> 
> On our side, we happen to have that use-case on our side as well.
> 
> Cedric
> 
>>
>> Jan
>>
> 
> --
> Cedric Hombourger
> Siemens AG
> http://www.siemens.com/

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/08e9d380-9cd3-4ce4-a156-7d08e444248b%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-07-05 at 09:55 +0200, Jan Kiszka wrote:
> On 05.07.24 09:47, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
> > > On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
> > > > sstate is checking for overlapping files in DEPLOY_DIR and will
> > > > raise an error when building a second image for a machine that
> > > > uses DTB_FILES.
> > > >
> > > > Reproducer:
> > > >   bitbake mc:phyboard-mira-bookworm:isar-image-base
> > > >   bitbake mc:phyboard-mira-bookworm:isar-image-debug
> > > >
> > > > Signed-off-by: Cedric Hombourger
> > > > <cedric.hombourger@siemens.com>
> > > > ---
> > > >  meta/classes/image.bbclass | 4 ++++
> > > >  1 file changed, 4 insertions(+)
> > > >
> > > > diff --git a/meta/classes/image.bbclass
> > > > b/meta/classes/image.bbclass
> > > > index c29d9e26..8b316c83 100644
> > > > --- a/meta/classes/image.bbclass
> > > > +++ b/meta/classes/image.bbclass
> > > > @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
> > > >  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
> > > > '${IMAGE_FULLNAME}-initrd.img'}"
> > > >
> > > >  # This defines the deployed dtbs for reuse by imagers
> > > > +# Since we may be building several images with the same set of
> > > > DTB_FILES, silent sstate
> > > > +# overlap checks
> > > >  DTB_FILES ?= ""
> > > > +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
> > > > os.path.basename(dtb) for dtb in
> > > > d.getVar('DTB_FILES').split()]) }"
> > > > +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
> > > >
> > > >  # Useful variables for imager implementations:
> > > >  PP = "/home/builder/${PN}-${MACHINE}"
> > >
> > > There is still
> > > https://patchwork.isar-build.org/project/isar/list/?series=1209
> > >  pending
> > > which looks much less risky to paper over a real conflict.
> >
> > Thanks Jan. I had missed this RFC patch series.
> >
> > That said, while it will solve the problem we are having with
> > building
> > multiple images with MACHINEs using DTB_FILES, it does not seem to
> > solve the problem of a build where we are building multiple kernels
> > (e.g. -rt and non-rt) for a given MACHINE: both kernels will be
> > shipping the same DTB_FILES.
>
> Adding Uladzimir: The risk of SSTATE_ALLOW_OVERLAP_FILES, also with
> your
> scenario, remains that those kernels are not coming with identical
> sources and, thus, with potentially different DTBs.

Yes I cannot agree more. We probably need to look at the consumers of
${DEPLOY_DIR_IMAGE}/*.dtb.

For instance, if the kernel was to produce a package with dtb files,
that package could be added to the list of packages to be installed in
the IMAGER sbuild environment

I am happy to look into this.

Once concern though is that the tree is currently broken for the
multiple-image / multiple-kernel use-cases we talked about.

If we agree that the deployment of dtb files into DEPLOY_DIR_IMAGE
should go away (and I think it should), I would then believe that a
work-around such as proposed herein would be acceptable (as long as it
does not stay in the tree for too long).

If we cannot fix or work-around this issue soon then I am afraid that
we will see all those work-arounds in downstream layers. Thoughts?

>
> Jan
>
> >
> > On our side, we happen to have that use-case on our side as well.
> >
> > Cedric
> >
> > >
> > > Jan
> > >
> >
> > --
> > Cedric Hombourger
> > Siemens AG
> > http://www.siemens.com/
>

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/2e7b629ea4b27798f7429d0ee49b20365cb93f9b.camel%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['cedric.hombourger@siemens.com' via isar-users]

On Fri, 2024-07-05 at 11:55 +0200, Cedric Hombourger wrote:
> On Fri, 2024-07-05 at 09:55 +0200, Jan Kiszka wrote:
> > On 05.07.24 09:47, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> > > On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
> > > > On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
> > > > > sstate is checking for overlapping files in DEPLOY_DIR and
> > > > > will
> > > > > raise an error when building a second image for a machine
> > > > > that
> > > > > uses DTB_FILES.
> > > > >
> > > > > Reproducer:
> > > > >   bitbake mc:phyboard-mira-bookworm:isar-image-base
> > > > >   bitbake mc:phyboard-mira-bookworm:isar-image-debug
> > > > >
> > > > > Signed-off-by: Cedric Hombourger
> > > > > <cedric.hombourger@siemens.com>
> > > > > ---
> > > > >  meta/classes/image.bbclass | 4 ++++
> > > > >  1 file changed, 4 insertions(+)
> > > > >
> > > > > diff --git a/meta/classes/image.bbclass
> > > > > b/meta/classes/image.bbclass
> > > > > index c29d9e26..8b316c83 100644
> > > > > --- a/meta/classes/image.bbclass
> > > > > +++ b/meta/classes/image.bbclass
> > > > > @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
> > > > >  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
> > > > > '${IMAGE_FULLNAME}-initrd.img'}"
> > > > >
> > > > >  # This defines the deployed dtbs for reuse by imagers
> > > > > +# Since we may be building several images with the same set
> > > > > of
> > > > > DTB_FILES, silent sstate
> > > > > +# overlap checks
> > > > >  DTB_FILES ?= ""
> > > > > +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
> > > > > os.path.basename(dtb) for dtb in
> > > > > d.getVar('DTB_FILES').split()]) }"
> > > > > +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
> > > > >
> > > > >  # Useful variables for imager implementations:
> > > > >  PP = "/home/builder/${PN}-${MACHINE}"
> > > >
> > > > There is still
> > > > https://patchwork.isar-build.org/project/isar/list/?series=1209
> > > >  pending
> > > > which looks much less risky to paper over a real conflict.
> > >
> > > Thanks Jan. I had missed this RFC patch series.
> > >
> > > That said, while it will solve the problem we are having with
> > > building
> > > multiple images with MACHINEs using DTB_FILES, it does not seem
> > > to
> > > solve the problem of a build where we are building multiple
> > > kernels
> > > (e.g. -rt and non-rt) for a given MACHINE: both kernels will be
> > > shipping the same DTB_FILES.
> >
> > Adding Uladzimir: The risk of SSTATE_ALLOW_OVERLAP_FILES, also with
> > your
> > scenario, remains that those kernels are not coming with identical
> > sources and, thus, with potentially different DTBs.
>
> Yes I cannot agree more. We probably need to look at the consumers of
> ${DEPLOY_DIR_IMAGE}/*.dtb.
>
> For instance, if the kernel was to produce a package with dtb files,
> that package could be added to the list of packages to be installed
> in
> the IMAGER sbuild environment
>
> I am happy to look into this.
>
> Once concern though is that the tree is currently broken for the
> multiple-image / multiple-kernel use-cases we talked about.
>
> If we agree that the deployment of dtb files into DEPLOY_DIR_IMAGE
> should go away (and I think it should), I would then believe that a
> work-around such as proposed herein would be acceptable (as long as
> it
> does not stay in the tree for too long).


I realize that the above patch isn't ideal but as I said our tree is
currently broken and I fear it will take some time before the solution
from Uladzimir is finalized and addresses uses-cases such as multiple
kernels for the same MACHINE (shipping the same DTB files and being
built as part of e.g. multiconfig build) and multiple images.

On my side, I won't be able to work on a different approach (avoid
using DEPLOY_DIR_IMAGE as a share) until next week (currently working
on a different patch series for linux-custom)

Should we instead use the proposed work-around in our own layers for
now?

>
> If we cannot fix or work-around this issue soon then I am afraid that
> we will see all those work-arounds in downstream layers. Thoughts?
>
> >
> > Jan
> >
> > >
> > > On our side, we happen to have that use-case on our side as well.
> > >
> > > Cedric
> > >
> > > >
> > > > Jan
> > > >
> > >
> > > --
> > > Cedric Hombourger
> > > Siemens AG
> > > http://www.siemens.com/
> >
>

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d1112c0eca142565dc0389796bf89df09d9a2707.camel%40siemens.com.

Re: [PATCH] image: allow multiple image recipes to deploy DTB_FILES

['Jan Kiszka' via isar-users]

On 11.07.24 10:52, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
> On Fri, 2024-07-05 at 11:55 +0200, Cedric Hombourger wrote:
>> On Fri, 2024-07-05 at 09:55 +0200, Jan Kiszka wrote:
>>> On 05.07.24 09:47, Hombourger, Cedric (DI CTO FDS CES LX) wrote:
>>>> On Fri, 2024-07-05 at 09:37 +0200, Jan Kiszka wrote:
>>>>> On 05.07.24 09:18, 'Cedric Hombourger' via isar-users wrote:
>>>>>> sstate is checking for overlapping files in DEPLOY_DIR and
>>>>>> will
>>>>>> raise an error when building a second image for a machine
>>>>>> that
>>>>>> uses DTB_FILES.
>>>>>>
>>>>>> Reproducer:
>>>>>>   bitbake mc:phyboard-mira-bookworm:isar-image-base
>>>>>>   bitbake mc:phyboard-mira-bookworm:isar-image-debug
>>>>>>
>>>>>> Signed-off-by: Cedric Hombourger
>>>>>> <cedric.hombourger@siemens.com>
>>>>>> ---
>>>>>>  meta/classes/image.bbclass | 4 ++++
>>>>>>  1 file changed, 4 insertions(+)
>>>>>>
>>>>>> diff --git a/meta/classes/image.bbclass
>>>>>> b/meta/classes/image.bbclass
>>>>>> index c29d9e26..8b316c83 100644
>>>>>> --- a/meta/classes/image.bbclass
>>>>>> +++ b/meta/classes/image.bbclass
>>>>>> @@ -27,7 +27,11 @@ INITRD_IMAGE ?= ""
>>>>>>  INITRD_DEPLOY_FILE = "${@ d.getVar('INITRD_IMAGE') or
>>>>>> '${IMAGE_FULLNAME}-initrd.img'}"
>>>>>>
>>>>>>  # This defines the deployed dtbs for reuse by imagers
>>>>>> +# Since we may be building several images with the same set
>>>>>> of
>>>>>> DTB_FILES, silent sstate
>>>>>> +# overlap checks
>>>>>>  DTB_FILES ?= ""
>>>>>> +DEPLOY_DTB_FILES = "${@ ' '.join(['${DEPLOY_DIR_IMAGE}/' +
>>>>>> os.path.basename(dtb) for dtb in
>>>>>> d.getVar('DTB_FILES').split()]) }"
>>>>>> +SSTATE_ALLOW_OVERLAP_FILES += "${DEPLOY_DTB_FILES}"
>>>>>>
>>>>>>  # Useful variables for imager implementations:
>>>>>>  PP = "/home/builder/${PN}-${MACHINE}"
>>>>>
>>>>> There is still
>>>>> https://patchwork.isar-build.org/project/isar/list/?series=1209
>>>>>  pending
>>>>> which looks much less risky to paper over a real conflict.
>>>>
>>>> Thanks Jan. I had missed this RFC patch series.
>>>>
>>>> That said, while it will solve the problem we are having with
>>>> building
>>>> multiple images with MACHINEs using DTB_FILES, it does not seem
>>>> to
>>>> solve the problem of a build where we are building multiple
>>>> kernels
>>>> (e.g. -rt and non-rt) for a given MACHINE: both kernels will be
>>>> shipping the same DTB_FILES.
>>>
>>> Adding Uladzimir: The risk of SSTATE_ALLOW_OVERLAP_FILES, also with
>>> your
>>> scenario, remains that those kernels are not coming with identical
>>> sources and, thus, with potentially different DTBs.
>>
>> Yes I cannot agree more. We probably need to look at the consumers of
>> ${DEPLOY_DIR_IMAGE}/*.dtb.
>>
>> For instance, if the kernel was to produce a package with dtb files,
>> that package could be added to the list of packages to be installed
>> in
>> the IMAGER sbuild environment
>>
>> I am happy to look into this.
>>
>> Once concern though is that the tree is currently broken for the
>> multiple-image / multiple-kernel use-cases we talked about.
>>
>> If we agree that the deployment of dtb files into DEPLOY_DIR_IMAGE
>> should go away (and I think it should), I would then believe that a
>> work-around such as proposed herein would be acceptable (as long as
>> it
>> does not stay in the tree for too long).
> 
> 
> I realize that the above patch isn't ideal but as I said our tree is
> currently broken and I fear it will take some time before the solution
> from Uladzimir is finalized and addresses uses-cases such as multiple
> kernels for the same MACHINE (shipping the same DTB files and being
> built as part of e.g. multiconfig build) and multiple images.
> 
> On my side, I won't be able to work on a different approach (avoid
> using DEPLOY_DIR_IMAGE as a share) until next week (currently working
> on a different patch series for linux-custom)
> 
> Should we instead use the proposed work-around in our own layers for
> now?

Downstream, it is much easier to asses the risk of using
SSTATE_ALLOW_OVERLAP_FILES, even more when targeting only a single
device. Therefore, you can carry such a workaround there for now and a
final solution (TM) in upstream a bit more time.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/bf490579-0cbf-4e93-8e69-992ffa69589f%40siemens.com.