multistrap support

multistrap support

[kazuhiro3.hayashi]

Hi,

I would like to know if isar still has some plans to use multistrap again
(or similar existing tools if exists) to generate target images.

Multistrap for isar-image-base has been removed by the following commit:
https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
as the base tree instead of running the debootstrap process.

I guess that the main purposes of quitting multistrap is that
there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
Are there any other reasons?
(Some bugs difficult to be fixed, mismatches with isar specification, etc.)

The current isar-bootstrap based approach would work fine for isar system,
but in that case, it might be difficult to share efforts for
developing and maintaining the functionality with non isar users.
I'm just interested in the future plan of isar.

Kind regards,
Kazu

 Kazuhiro Hayashi
 Corporate Software Engineering & Technology Center
 Toshiba Corporation
 Tel: +81-44-549-2476
 E-mail: kazuhiro3.hayashi@toshiba.co.jp

Re: multistrap support

[Claudius Heine]

Hi Kazu,

I don't speak for the whole Isar community, but just from my point of 
view. For context I implemented the switch from multistrap to debootstrap.

On 13/02/2019 14.14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi,
> 
> I would like to know if isar still has some plans to use multistrap again
> (or similar existing tools if exists) to generate target images.

No, only debootstrap is supported and we will most probably stick with that.

> Multistrap for isar-image-base has been removed by the following commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> as the base tree instead of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> Are there any other reasons?

That is one of them. Debootstrap is better supported by the main Debian 
community than any other bootstrapping tools. And since it is out of 
scope or the Isar Project to implement its own bootstrapping mechanism, 
we will stick with the second most official one 'debootstrap'. The 
Debian installer being the most official way to install Debian, which is 
a bit to bulky and integrated to be easily used in a scripted environment.

Both multistrap and debootstrap had or still have some inconveniences 
with our usage and it would of course be great to have one that fits 
exactly our purposes and is used and supported by the broader 
Debian/Embedded Debian and possible Debian server community. But as I 
said Isar does not have the resources to implement that.

> (Some bugs difficult to be fixed, mismatches with isar specification, etc.)
> 
> The current isar-bootstrap based approach would work fine for isar system,
> but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.

Maybe you can give an example about which feature you think could be 
moved outside of the isar project and be used separately.

One issue here is that Isar uses bitbake as the build task scheduler, so 
shell scripts etc. are difficult to extract from it to be used outside 
of Isar. And if they are extracted from the Isar recipes, then using 
them in Isar again is disadvantageous for the Isar build process.

So sharing code between a bitbake/isar project and other build systems 
is not really possible.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: multistrap support

[Henning Schild]

Hi,

in fact Isar does not really care too much about how exactly
setup_root_file_system does what it does. Just the interface has to be
stable at some point. So you take a list of repos and packages and off
you go ... to put it overly-simple.

rootfs = strap(repos, package-names)

That is why the switch from multistrap to debootstrap was relatively
simple. So you could even think about switching back or switching to
yet another <funny>-strap.

What i would like to understand is why you care about multistrap and
why you think the strap-tool has an implication on collaboration with
other projects? Where do you see the risk of not being able to share
efforts with other projects that use multistrap?

Henning

Am Wed, 13 Feb 2019 13:14:57 +0000
schrieb <kazuhiro3.hayashi@toshiba.co.jp>:

> Hi,
> 
> I would like to know if isar still has some plans to use multistrap
> again (or similar existing tools if exists) to generate target images.
> 
> Multistrap for isar-image-base has been removed by the following
> commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in
> isar-bootstrap-helper.bbclass, which seems to use (pre-built?)
> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH as the base tree instead
> of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is
> 2.2.10 on Nov. 2018) Are there any other reasons?
> (Some bugs difficult to be fixed, mismatches with isar specification,
> etc.)
> 
> The current isar-bootstrap based approach would work fine for isar
> system, but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.
> 
> Kind regards,
> Kazu
> 
>  Kazuhiro Hayashi
>  Corporate Software Engineering & Technology Center
>  Toshiba Corporation
>  Tel: +81-44-549-2476
>  E-mail: kazuhiro3.hayashi@toshiba.co.jp
> 
> 

Re: multistrap support

[Jan Kiszka]

On 13.02.19 14:14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi,
> 
> I would like to know if isar still has some plans to use multistrap again
> (or similar existing tools if exists) to generate target images.
> 
> Multistrap for isar-image-base has been removed by the following commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> as the base tree instead of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> Are there any other reasons?
> (Some bugs difficult to be fixed, mismatches with isar specification, etc.)

Indeed, the key reason (beside some technical issues) for no longer using 
multistrap in Isar was that the multistrap creator himself explained to us that 
it's dead and we should rather sooner than later move away from it.

We then decided to pick up the most commonly used bootstrapping mechanism for 
Debian today, simply to reduce the risk of running into problems no one else is 
seeing - and fixing.

> 
> The current isar-bootstrap based approach would work fine for isar system,
> but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.

This takes concrete ideas what you consider worth sharing and what interfaces 
this may imply. Debootstrap is not married to Isar but it would take some use 
case to make that shareable or replaceable with something else.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: [meta-eid] Re: multistrap support

[kazuhiro3.hayashi]

Hi Claudius, (and Isar developers)

Thank you for your replies,
I would like to reply one by one.

> -----Original Message-----
> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
> Behalf Of Claudius Heine
> Sent: Wednesday, February 13, 2019 10:56 PM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: [meta-eid] Re: multistrap support
> 
> Hi Kazu,
> 
> I don't speak for the whole Isar community, but just from my point of
> view. For context I implemented the switch from multistrap to debootstrap.
> 
> On 13/02/2019 14.14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> > Hi,
> >
> > I would like to know if isar still has some plans to use multistrap again
> > (or similar existing tools if exists) to generate target images.
> 
> No, only debootstrap is supported and we will most probably stick with that.
> 
> > Multistrap for isar-image-base has been removed by the following commit:
> >
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
> ca6c8884
> > and replaced by setup_root_file_system() in
> isar-bootstrap-helper.bbclass,
> > which seems to use (pre-built?)
> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> > as the base tree instead of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is 2.2.10
> on Nov. 2018)
> > Are there any other reasons?
> 
> That is one of them. Debootstrap is better supported by the main Debian
> community than any other bootstrapping tools. And since it is out of
> scope or the Isar Project to implement its own bootstrapping mechanism,
> we will stick with the second most official one 'debootstrap'. The
> Debian installer being the most official way to install Debian, which is
> a bit to bulky and integrated to be easily used in a scripted environment.

I'm fully agree that using debootstrap, one of the core package in Debian
for a long time, as the base tool to generate images from multiple apt repos
if we decided not to use multistrap.
Regarding Debian installer, base-installer(udeb) also uses debootstrap
to install the base system before using apt and tasksel, but as you said,
it's integrated to d-i scripts so difficult to use in non-installer environment.

> 
> Both multistrap and debootstrap had or still have some inconveniences
> with our usage and it would of course be great to have one that fits
> exactly our purposes and is used and supported by the broader
> Debian/Embedded Debian and possible Debian server community. But as I
> said Isar does not have the resources to implement that.

Thank you for your opinion.
I didn't mean that multistrap should be used in Isar again.
Also, my opinion is that any other full 'replacement' of debootstrap
should not be implemented if bootstrapping from multiple repositories
can be achieved by pure debootstrap + additional scripts.

(Sorry, I was misunderstanding multistrap; I thought it uses
 debootstrap as back-end, but that is a replacement actually.)

The current bitbake function in Isar is one of the better solutions.
I should check more detailed implementation of that.

> 
> > (Some bugs difficult to be fixed, mismatches with isar specification,
> etc.)
> >
> > The current isar-bootstrap based approach would work fine for isar system,
> > but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> 
> Maybe you can give an example about which feature you think could be
> moved outside of the isar project and be used separately.

I should learn the mechanism of setup_root_file_system() more,
but at least I'm originally considering how I can provide the following API:

mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> [<script>]]

I guess that the most important part for me is isar_bootstrap() in isar-bootstrap.inc,
which is shared by isar-bootstrap-*.bb and copied by setup_root_file_system as
a base tree ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).
Other remaining commands in setup_root_file_system() seems to be almost based on apt,
so too easy to control multiple repositories.
In the following commands, only one debootstrap command is called with
one repository argument given from get_distro_primary_source_entry().
https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc#L175
Does it work with multiple apt repositories?
Or, installing required packages from other repos with apt after debootstrapping finishes?

> 
> One issue here is that Isar uses bitbake as the build task scheduler, so
> shell scripts etc. are difficult to extract from it to be used outside
> of Isar. And if they are extracted from the Isar recipes, then using
> them in Isar again is disadvantageous for the Isar build process.
> 
> So sharing code between a bitbake/isar project and other build systems
> is not really possible.

I usually have the same issues in my bitbake recipes development.
It might be possible to share the concrete processes to create
a Debian base system from multiple repositories,
even if it's difficult to share the actual codes.

Background of my question:
I guess that people who are developing Debian based system
with their own custom packages (I'm one of them) may want to use
a command (or a wrapper script) to create a base image from multiple apt repos,
especially if they need to replace a part of existing Debian essential packages
by their custom packages (coreutils, util-linux, dpkg, etc.).
I'm investigating existing approaches that provide the API above or similar one.

Regards,
Kazu

RE: multistrap support

[kazuhiro3.hayashi]

Hi Henning,

> -----Original Message-----
> From: Henning Schild [mailto:henning.schild@siemens.com]
> Sent: Thursday, February 14, 2019 12:19 AM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>
> Cc: isar-users@googlegroups.com; meta-eid@googlegroups.com
> Subject: Re: multistrap support
> 
> Hi,
> 
> in fact Isar does not really care too much about how exactly
> setup_root_file_system does what it does. Just the interface has to be
> stable at some point. So you take a list of repos and packages and off
> you go ... to put it overly-simple.
> 
> rootfs = strap(repos, package-names)
> 
> That is why the switch from multistrap to debootstrap was relatively
> simple. So you could even think about switching back or switching to
> yet another <funny>-strap.

Thanks, I got it.

I'm now watching isar_bootstrap() which creates base tree with debootstrap.
It seems to run one debootstrap command with one primary apt repository,
then merge available updates from other apt repositories by `apt-get dist-upgrade`.
I understood that these steps could replace calling multistrap command.

> 
> What i would like to understand is why you care about multistrap and
> why you think the strap-tool has an implication on collaboration with
> other projects? Where do you see the risk of not being able to share
> efforts with other projects that use multistrap?

I should explain that I'm not heavy user of Isar at the moment actually,
and I'm prototyping another build tool:
https://events.linuxfoundation.org/wp-content/uploads/2017/12/ELCE2018_Debian-Yocto-State-of-the-Art_r6_Kazuhiro-Hayashi.pdf
Many important visions of that project are from Isar development policy,
and tooling is one of the most important topic.

I WAS thinking that bootstrapping with multiple apt repositories
are complicated and existing tools like multistrap should be used
because of the fact (history) that multistrap has been developed as an independent package
from debootstrap to support multiple repositories.
However, if we can replace multistrap functionality by the above
(simple and easy to understand) steps with debootstrap,
the strap-tool might not affect collaboration of Isar and other projects so much.

Regards,
Kazu

> 
> Henning
> 
> Am Wed, 13 Feb 2019 13:14:57 +0000
> schrieb <kazuhiro3.hayashi@toshiba.co.jp>:
> 
> > Hi,
> >
> > I would like to know if isar still has some plans to use multistrap
> > again (or similar existing tools if exists) to generate target images.
> >
> > Multistrap for isar-image-base has been removed by the following
> > commit:
> >
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
> ca6c8884
> > and replaced by setup_root_file_system() in
> > isar-bootstrap-helper.bbclass, which seems to use (pre-built?)
> > isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH as the base tree instead
> > of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is
> > 2.2.10 on Nov. 2018) Are there any other reasons?
> > (Some bugs difficult to be fixed, mismatches with isar specification,
> > etc.)
> >
> > The current isar-bootstrap based approach would work fine for isar
> > system, but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> >
> > Kind regards,
> > Kazu
> >
> >  Kazuhiro Hayashi
> >  Corporate Software Engineering & Technology Center
> >  Toshiba Corporation
> >  Tel: +81-44-549-2476
> >  E-mail: kazuhiro3.hayashi@toshiba.co.jp
> >
> >

Re: [meta-eid] Re: multistrap support

[Jan Kiszka]

On 14.02.19 04:53, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Background of my question:
> I guess that people who are developing Debian based system
> with their own custom packages (I'm one of them) may want to use
> a command (or a wrapper script) to create a base image from multiple apt repos,
> especially if they need to replace a part of existing Debian essential packages
> by their custom packages (coreutils, util-linux, dpkg, etc.).
> I'm investigating existing approaches that provide the API above or similar one.

Isar gained support for debian package rebuilding (with modifications) just 
recently. You may want to study the existing (rather straightforward) 
implementation and usage of that (meta-isar/recipes-app/hello/).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [meta-eid] multistrap support

[Baurzhan Ismagulov]

On Wed, Feb 13, 2019 at 01:14:57PM +0000, kazuhiro3.hayashi@toshiba.co.jp wrote:
> I would like to know if isar still has some plans to use multistrap again
> (or similar existing tools if exists) to generate target images.
> 
> Multistrap for isar-image-base has been removed by the following commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> as the base tree instead of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> Are there any other reasons?
> (Some bugs difficult to be fixed, mismatches with isar specification, etc.)
> 
> The current isar-bootstrap based approach would work fine for isar system,
> but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.

Neil told us that he doesn't maintain multistrap anymore. We also encountered
issues (e.g., https://github.com/ilbers/isar/issues/16). For me personally,
Perl is an obstacle (I go through Perl tutorials every couple of years :) ). So
switching to debootstrap was easier at the time.

That is not to say that debootstrap is the tool we want to have. The focus of
debootstrap is to create Debian rootfss on Debian and non-Debian systems with
the least possible dependencies. Given that, it is developed in shell, which is
also not maintainable.

In the long run, we'll need more and more introspection into apt & Co. for more
advanced features. One way could be multistrap re-implementation in Python, as
suggested by Wookey. ELBE has good experiences with libapt. aptly also promises
some features we would like to use (though probably not multistrapping OOTB).

Bootstrapping hybrid repos is also a useful use case. It can be achieved by
making the "secondary" distribution upgradable from Debian, but if one has a
lot of customized packages, using multistrap would require less effort and be
less error-prone as well. In Isar, we try to stay away from massive
modifications, but there are projects ending up tuning significant parts of
Debian.

The specific tools don't have to be mutually exclusive; Python offers enough
polymorphism to include both as plugins (bitbake support has to be checked).
With quite diverse requirements and opinions we already have today, we'll need
pluggability, be it for multistrap or other features; the earlier we start
playing with it, the better. For me, it is the question of enough itch rather
than resources.

For your specific use case, I don't see a problem using multistrap for meta-eid
till we have a better alternative.

With kind regards,
Baurzhan.

Re: [meta-eid] Re: multistrap support

[Claudius Heine]

Hi Kazu,

On 14/02/2019 04.53, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi Claudius, (and Isar developers)
> 
> Thank you for your replies,
> I would like to reply one by one.
> 
>> -----Original Message-----
>> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
>> Behalf Of Claudius Heine
>> Sent: Wednesday, February 13, 2019 10:56 PM
>> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
>> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
>> Cc: meta-eid@googlegroups.com
>> Subject: [meta-eid] Re: multistrap support
>>
>> Hi Kazu,
>>
>> I don't speak for the whole Isar community, but just from my point of
>> view. For context I implemented the switch from multistrap to debootstrap.
>>
>> On 13/02/2019 14.14, kazuhiro3.hayashi@toshiba.co.jp wrote:
>>> Hi,
>>>
>>> I would like to know if isar still has some plans to use multistrap again
>>> (or similar existing tools if exists) to generate target images.
>>
>> No, only debootstrap is supported and we will most probably stick with that.
>>
>>> Multistrap for isar-image-base has been removed by the following commit:
>>>
>> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
>> ca6c8884
>>> and replaced by setup_root_file_system() in
>> isar-bootstrap-helper.bbclass,
>>> which seems to use (pre-built?)
>> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
>>> as the base tree instead of running the debootstrap process.
>>>
>>> I guess that the main purposes of quitting multistrap is that
>>> there is no big update for a few years (though the last update is 2.2.10
>> on Nov. 2018)
>>> Are there any other reasons?
>>
>> That is one of them. Debootstrap is better supported by the main Debian
>> community than any other bootstrapping tools. And since it is out of
>> scope or the Isar Project to implement its own bootstrapping mechanism,
>> we will stick with the second most official one 'debootstrap'. The
>> Debian installer being the most official way to install Debian, which is
>> a bit to bulky and integrated to be easily used in a scripted environment.
> 
> I'm fully agree that using debootstrap, one of the core package in Debian
> for a long time, as the base tool to generate images from multiple apt repos
> if we decided not to use multistrap.
> Regarding Debian installer, base-installer(udeb) also uses debootstrap
> to install the base system before using apt and tasksel, but as you said,
> it's integrated to d-i scripts so difficult to use in non-installer environment.

I didn't know that the Debian installer uses debootstrap somewhere 
inside as well. Interesting.

>>
>> Both multistrap and debootstrap had or still have some inconveniences
>> with our usage and it would of course be great to have one that fits
>> exactly our purposes and is used and supported by the broader
>> Debian/Embedded Debian and possible Debian server community. But as I
>> said Isar does not have the resources to implement that.
> 
> Thank you for your opinion.
> I didn't mean that multistrap should be used in Isar again.
> Also, my opinion is that any other full 'replacement' of debootstrap
> should not be implemented if bootstrapping from multiple repositories
> can be achieved by pure debootstrap + additional scripts.
> 
> (Sorry, I was misunderstanding multistrap; I thought it uses
>   debootstrap as back-end, but that is a replacement actually.)
> 
> The current bitbake function in Isar is one of the better solutions.
> I should check more detailed implementation of that.

Thanks!

> 
>>
>>> (Some bugs difficult to be fixed, mismatches with isar specification,
>> etc.)
>>>
>>> The current isar-bootstrap based approach would work fine for isar system,
>>> but in that case, it might be difficult to share efforts for
>>> developing and maintaining the functionality with non isar users.
>>> I'm just interested in the future plan of isar.
>>
>> Maybe you can give an example about which feature you think could be
>> moved outside of the isar project and be used separately.
> 
> I should learn the mechanism of setup_root_file_system() more,
> but at least I'm originally considering how I can provide the following API:
> 
> mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> [<script>]]

To further consider is the case of raspbian where we need an additional 
public repo key and the possibility of employing package pinning and 
priorities as well as apt configuration to change the package selection 
of the bootstrapped image.

It would be great to have temporary apt settings (repo, package pinning) 
to allow offline building from a local repo, but the system will use the 
official one in production later.

IMO it would be great if the bootstrap process itself would populate a 
local repo with all used packages + source packages while it is fetching 
them from upstream and using that repo at the same time with high 
priority. This way calling the bootstrap process multiple times in a row 
(with removal of bootstrapped system, but not local repo) would result 
in exactly the same file system.
   -> idempotent bootstrap
   -> caching of installation data
   -> reproducible bootstraping

The bootstrap process should also not just be one big step, but split 
into smaller idempotent ones. So that customization should be possible 
at every level.

IMO the 'script' option should not be used, since that seems to be a 
hacky way because the bootstrap process was not split up enough to call 
a script yourself at the right moment.

So my suggestion would be something like this:

mybootstrap init <suite> <arch> <target directory> # Setups a directory 
with some configuration files (.mybootstrap directory?)

mybootstrap -C <target directory> addrepo 
<static|temporary|reproducible> <url> <prio> # Add a repo, should work 
in with a un-generated as well as a generated system, static for repos 
that should remain after finalization, temporary ones should be removed 
and reproducible ones should be used and populated with used packages as 
mybootstrap does its thing, maybe a package filter here might be useful 
as well

mybootstrap -C <target directory> generate <package selection> # 
generates initial bootstrap environment with only required packages + 
apt -> afterwards it should be a chrootable system

mybootstrap -C <target directory> install <package> # Installs packages, 
since this might be more involved that just calling 'apt-get install', 
because of the reproducible repository, it would make send to wrap 
around that

mybootstrap -C <target directory> execute <script> # starts script 
inside the system with the right chroot

mybootstrap -C <target directory> finalize # removes mybootstrap stuff 
and temporary+reproducible repos from the apt configuration

I have more ideas for some additional subcommands, like executing repo 
updates, controlling the 'reproducible' repo better (removing packages, 
selectively updating packages, etc.) and some more. But that interface 
in general would be great for scripting custom Debian installations.

> I guess that the most important part for me is isar_bootstrap() in isar-bootstrap.inc,
> which is shared by isar-bootstrap-*.bb and copied by setup_root_file_system as
> a base tree ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).
> Other remaining commands in setup_root_file_system() seems to be almost based on apt,
> so too easy to control multiple repositories.
> In the following commands, only one debootstrap command is called with
> one repository argument given from get_distro_primary_source_entry().
> https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc#L175
> Does it work with multiple apt repositories?
> Or, installing required packages from other repos with apt after debootstrapping finishes?

The isar-bootstrap process has changed alot after I initially 
implemented it (cross-build support, etc.), and I haven't worked on that 
since.

But the general idea when I implemented it was that just one repo will 
be used for debootstrap (the first one listed), and afterwards the other 
repos are added and an update+dist-upgrade is performed on all of them.

After that step the upstream apt index should not be changed. No global 
apt-get update, only apt-get update of local repos where newly created 
packages are placed.

That way the buildchroot and the resulting image should contain the same 
versions of the packages which are installed in their respective usages 
of setup_root_file_system function in a copy of that initial 
bootstrapped system.

> 
>>
>> One issue here is that Isar uses bitbake as the build task scheduler, so
>> shell scripts etc. are difficult to extract from it to be used outside
>> of Isar. And if they are extracted from the Isar recipes, then using
>> them in Isar again is disadvantageous for the Isar build process.
>>
>> So sharing code between a bitbake/isar project and other build systems
>> is not really possible.
> 
> I usually have the same issues in my bitbake recipes development.
> It might be possible to share the concrete processes to create
> a Debian base system from multiple repositories,
> even if it's difficult to share the actual codes.
> 
> Background of my question:
> I guess that people who are developing Debian based system
> with their own custom packages (I'm one of them) may want to use
> a command (or a wrapper script) to create a base image from multiple apt repos,
> especially if they need to replace a part of existing Debian essential packages
> by their custom packages (coreutils, util-linux, dpkg, etc.).
> I'm investigating existing approaches that provide the API above or similar one.

We would currently deal with that by having the local repo that contains 
our build packages added to the apt with a high priority.

Installing or updating now would remove the upstream package and install 
our own versions. Not the best way, but it works in principle.

I don't know how tested in Isar that currently is though.

regards,
Claudius


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

RE: multistrap support

[kazuhiro3.hayashi]

Hi Jan,

> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Thursday, February 14, 2019 1:15 AM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: Re: multistrap support
> 
> On 13.02.19 14:14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> > Hi,
> >
> > I would like to know if isar still has some plans to use multistrap again
> > (or similar existing tools if exists) to generate target images.
> >
> > Multistrap for isar-image-base has been removed by the following commit:
> >
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
> ca6c8884
> > and replaced by setup_root_file_system() in
> isar-bootstrap-helper.bbclass,
> > which seems to use (pre-built?)
> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> > as the base tree instead of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is 2.2.10
> on Nov. 2018)
> > Are there any other reasons?
> > (Some bugs difficult to be fixed, mismatches with isar specification,
> etc.)
> 
> Indeed, the key reason (beside some technical issues) for no longer using
> multistrap in Isar was that the multistrap creator himself explained to
> us that
> it's dead and we should rather sooner than later move away from it.

Thank you for the detailed information.

> 
> We then decided to pick up the most commonly used bootstrapping mechanism
> for
> Debian today, simply to reduce the risk of running into problems no one
> else is
> seeing - and fixing.

Reasonable approach.

> 
> >
> > The current isar-bootstrap based approach would work fine for isar system,
> > but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> 
> This takes concrete ideas what you consider worth sharing and what
> interfaces
> this may imply. Debootstrap is not married to Isar but it would take some
> use
> case to make that shareable or replaceable with something else.

I was thinking a kind of the following things, mentioned in the reply to Claudius:
> mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> [<script>]]
For examples, I assumes that the following apt repositories could be listed in the <mirror_list>
1. The official Debian apt repository (or a (local) mirror of it)
2. An apt repository for specific custom distro (like Debian deliverable)
3. An apt repository for specific project/product (includes apps, etc.)
If a command like the above provided, we can always generate the rootfs tree
only by one command (+ chroot apt install) without setting any tools up
(probably for experiment purposes).
This kind of technical discussion would be kept in other replies...

Regards,
Kazu

RE: [meta-eid] Re: multistrap support

[kazuhiro3.hayashi]

> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Thursday, February 14, 2019 3:19 PM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; claudius.heine.ext@siemens.com;
> isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: Re: [meta-eid] Re: multistrap support
> 
> On 14.02.19 04:53, kazuhiro3.hayashi@toshiba.co.jp wrote:
> > Background of my question:
> > I guess that people who are developing Debian based system
> > with their own custom packages (I'm one of them) may want to use
> > a command (or a wrapper script) to create a base image from multiple apt
> repos,
> > especially if they need to replace a part of existing Debian essential
> packages
> > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > I'm investigating existing approaches that provide the API above or
> similar one.
> 
> Isar gained support for debian package rebuilding (with modifications) just
> recently. You may want to study the existing (rather straightforward)
> implementation and usage of that (meta-isar/recipes-app/hello/).

Great news.
I should look at that, thanks.

> 
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux