Incremental target images

Incremental target images

[Jan Kiszka]

Hi all,

I was wondering if / how we could model the increasingly common case of 
building very similar target images more efficiently.

A devel image, e.g., likely consists of almost the same base package set 
as the release image. It may only add further packages and maybe replace 
very few (like customization packages). When building both, we could 
save time - specifically when doing cross - by bootstrapping the 
baseline only once. We already do that for the debootstrap step, but not 
yet for further packages.

What do you think? And how could that be modeled from user perspective?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Incremental target images

[Claudius Heine]

Hi Jan,

On 2018-08-27 08:41, [ext] Jan Kiszka wrote:
> Hi all,
> 
> I was wondering if / how we could model the increasingly common case of 
> building very similar target images more efficiently.
> 
> A devel image, e.g., likely consists of almost the same base package set 
> as the release image. It may only add further packages and maybe replace 
> very few (like customization packages). When building both, we could 
> save time - specifically when doing cross - by bootstrapping the 
> baseline only once. We already do that for the debootstrap step, but not 
> yet for further packages.
> 
> What do you think? And how could that be modeled from user perspective?

That is not very easy.

A complex way this could be done by having a `common image` recipe that 
depends on other image recipes to deploy their required package list, 
then figure out the common dependencies of all of them and build the 
custom image. Then those other image recipes depend of the `common 
image` recipe in turn to create it, copy it and base their own 
customizations on top.

 From a user perspective they would have to add their image recipes into 
a global variable or bbappend, so that is available in the `common 
image` recipe. The rest could be done in the image classes.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Incremental target images

[Claudius Heine]

On 2018-08-27 09:25, [ext] Claudius Heine wrote:
> Hi Jan,
> 
> On 2018-08-27 08:41, [ext] Jan Kiszka wrote:
>> Hi all,
>>
>> I was wondering if / how we could model the increasingly common case 
>> of building very similar target images more efficiently.
>>
>> A devel image, e.g., likely consists of almost the same base package 
>> set as the release image. It may only add further packages and maybe 
>> replace very few (like customization packages). When building both, we 
>> could save time - specifically when doing cross - by bootstrapping the 
>> baseline only once. We already do that for the debootstrap step, but 
>> not yet for further packages.
>>
>> What do you think? And how could that be modeled from user perspective?
> 
> That is not very easy.
> 
> A complex way this could be done by having a `common image` recipe that 
> depends on other image recipes to deploy their required package list,

What I meant by that is each image recipe writes its package list into 
its own text file in the 'deploy' directory.

> then figure out the common dependencies of all of them and build the 
> custom image.

s/custom/common/

Problem scenario here is:

  Image A: depends on A which depends on D
  Image B: depends on B which depends on D

Since D is not named in any recipes, but should be installed in the 
`common image` as automatically selected package (not as manual). This 
scripting is a bit tricky.

> Then those other image recipes depend of the `common 
> image` recipe in turn to create it, copy it and base their own 
> customizations on top.
> 
>  From a user perspective they would have to add their image recipes into 
> a global variable or bbappend, so that is available in the `common 
> image` recipe. The rest could be done in the image classes.
> 
> Cheers,
> Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Incremental target images

[Jan Kiszka]

On 2018-08-27 09:44, Claudius Heine wrote:
> On 2018-08-27 09:25, [ext] Claudius Heine wrote:
>> Hi Jan,
>>
>> On 2018-08-27 08:41, [ext] Jan Kiszka wrote:
>>> Hi all,
>>>
>>> I was wondering if / how we could model the increasingly common case 
>>> of building very similar target images more efficiently.
>>>
>>> A devel image, e.g., likely consists of almost the same base package 
>>> set as the release image. It may only add further packages and maybe 
>>> replace very few (like customization packages). When building both, 
>>> we could save time - specifically when doing cross - by bootstrapping 
>>> the baseline only once. We already do that for the debootstrap step, 
>>> but not yet for further packages.
>>>
>>> What do you think? And how could that be modeled from user perspective?
>>
>> That is not very easy.
>>
>> A complex way this could be done by having a `common image` recipe 
>> that depends on other image recipes to deploy their required package 
>> list,
> 
> What I meant by that is each image recipe writes its package list into 
> its own text file in the 'deploy' directory.
> 
>> then figure out the common dependencies of all of them and build the 
>> custom image.
> 
> s/custom/common/
> 
> Problem scenario here is:
> 
>   Image A: depends on A which depends on D
>   Image B: depends on B which depends on D
> 
> Since D is not named in any recipes, but should be installed in the 
> `common image` as automatically selected package (not as manual). This 
> scripting is a bit tricky.
> 
>> Then those other image recipes depend of the `common image` recipe in 
>> turn to create it, copy it and base their own customizations on top.
>>
>>  From a user perspective they would have to add their image recipes 
>> into a global variable or bbappend, so that is available in the 
>> `common image` recipe. The rest could be done in the image classes.
>>
>> Cheers,
>> Claudius
> 

Hmm, I'm not seeing yet where the complication could come from. I would 
have expected the following:

- write a new image recipe include that takes a preexisting image in
   form of a rootfs as input (rather than using setup_root_file_system)

- create specialized images on top of that which depend on some base
   image recipe as well as additional IMAGE_INSTALL recipes and may come
   with own IMAGE_PREINSTALLs (always on top of base images)

- push all common IMAGE_PREINSTALLs into base image recipes, same for
   all IMAGE_INSTALLs

And done. What am I missing?

Jan

Re: Incremental target images

[Claudius Heine]

On 2018-08-27 10:29, Jan Kiszka wrote:
> On 2018-08-27 09:44, Claudius Heine wrote:
>> On 2018-08-27 09:25, [ext] Claudius Heine wrote:
>>> Hi Jan,
>>>
>>> On 2018-08-27 08:41, [ext] Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> I was wondering if / how we could model the increasingly common case 
>>>> of building very similar target images more efficiently.
>>>>
>>>> A devel image, e.g., likely consists of almost the same base package 
>>>> set as the release image. It may only add further packages and maybe 
>>>> replace very few (like customization packages). When building both, 
>>>> we could save time - specifically when doing cross - by 
>>>> bootstrapping the baseline only once. We already do that for the 
>>>> debootstrap step, but not yet for further packages.
>>>>
>>>> What do you think? And how could that be modeled from user perspective?
>>>
>>> That is not very easy.
>>>
>>> A complex way this could be done by having a `common image` recipe 
>>> that depends on other image recipes to deploy their required package 
>>> list,
>>
>> What I meant by that is each image recipe writes its package list into 
>> its own text file in the 'deploy' directory.
>>
>>> then figure out the common dependencies of all of them and build the 
>>> custom image.
>>
>> s/custom/common/
>>
>> Problem scenario here is:
>>
>>   Image A: depends on A which depends on D
>>   Image B: depends on B which depends on D
>>
>> Since D is not named in any recipes, but should be installed in the 
>> `common image` as automatically selected package (not as manual). This 
>> scripting is a bit tricky.
>>
>>> Then those other image recipes depend of the `common image` recipe in 
>>> turn to create it, copy it and base their own customizations on top.
>>>
>>>  From a user perspective they would have to add their image recipes 
>>> into a global variable or bbappend, so that is available in the 
>>> `common image` recipe. The rest could be done in the image classes.
>>>
>>> Cheers,
>>> Claudius
>>
> 
> Hmm, I'm not seeing yet where the complication could come from. I would 
> have expected the following:
> 
> - write a new image recipe include that takes a preexisting image in
>    form of a rootfs as input (rather than using setup_root_file_system)
> 
> - create specialized images on top of that which depend on some base
>    image recipe as well as additional IMAGE_INSTALL recipes and may come
>    with own IMAGE_PREINSTALLs (always on top of base images)
> 
> - push all common IMAGE_PREINSTALLs into base image recipes, same for
>    all IMAGE_INSTALLs
> 
> And done. What am I missing?

Ok, then I misunderstood you. I thought you wanted a mostly transparent 
mechanism in the background and not just an inheritance scheme for images.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Incremental target images

[chombourger]

[-- Attachment #1.1: Type: text/plain, Size: 2923 bytes --]

This would have been my thinking as. Add something like IMAGE_CREATE_FROM = 
"production-image" in your development-image recipe would go a long way.

+1 for Jan's suggestion.

Cedric

On Monday, August 27, 2018 at 10:29:35 AM UTC+2, Jan Kiszka wrote:
>
> On 2018-08-27 09:44, Claudius Heine wrote: 
> > On 2018-08-27 09:25, [ext] Claudius Heine wrote: 
> >> Hi Jan, 
> >> 
> >> On 2018-08-27 08:41, [ext] Jan Kiszka wrote: 
> >>> Hi all, 
> >>> 
> >>> I was wondering if / how we could model the increasingly common case 
> >>> of building very similar target images more efficiently. 
> >>> 
> >>> A devel image, e.g., likely consists of almost the same base package 
> >>> set as the release image. It may only add further packages and maybe 
> >>> replace very few (like customization packages). When building both, 
> >>> we could save time - specifically when doing cross - by bootstrapping 
> >>> the baseline only once. We already do that for the debootstrap step, 
> >>> but not yet for further packages. 
> >>> 
> >>> What do you think? And how could that be modeled from user 
> perspective? 
> >> 
> >> That is not very easy. 
> >> 
> >> A complex way this could be done by having a `common image` recipe 
> >> that depends on other image recipes to deploy their required package 
> >> list, 
> > 
> > What I meant by that is each image recipe writes its package list into 
> > its own text file in the 'deploy' directory. 
> > 
> >> then figure out the common dependencies of all of them and build the 
> >> custom image. 
> > 
> > s/custom/common/ 
> > 
> > Problem scenario here is: 
> > 
> >   Image A: depends on A which depends on D 
> >   Image B: depends on B which depends on D 
> > 
> > Since D is not named in any recipes, but should be installed in the 
> > `common image` as automatically selected package (not as manual). This 
> > scripting is a bit tricky. 
> > 
> >> Then those other image recipes depend of the `common image` recipe in 
> >> turn to create it, copy it and base their own customizations on top. 
> >> 
> >>  From a user perspective they would have to add their image recipes 
> >> into a global variable or bbappend, so that is available in the 
> >> `common image` recipe. The rest could be done in the image classes. 
> >> 
> >> Cheers, 
> >> Claudius 
> > 
>
> Hmm, I'm not seeing yet where the complication could come from. I would 
> have expected the following: 
>
> - write a new image recipe include that takes a preexisting image in 
>    form of a rootfs as input (rather than using setup_root_file_system) 
>
> - create specialized images on top of that which depend on some base 
>    image recipe as well as additional IMAGE_INSTALL recipes and may come 
>    with own IMAGE_PREINSTALLs (always on top of base images) 
>
> - push all common IMAGE_PREINSTALLs into base image recipes, same for 
>    all IMAGE_INSTALLs 
>
> And done. What am I missing? 
>
> Jan 
>

[-- Attachment #1.2: Type: text/html, Size: 3622 bytes --]