Re: multistrap support

Re: multistrap support

[kazuhiro3.hayashi]

Hi Claudius,

Sorry for my late reply.
Thank you very much for sharing detailed information of Isar and your ideas!

> -----Original Message-----
> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
> Behalf Of Claudius Heine
> Sent: Thursday, February 14, 2019 6:08 PM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: Re: [meta-eid] Re: multistrap support
> 
> Hi Kazu,
> 
(snipped)
> >>
> >>> (Some bugs difficult to be fixed, mismatches with isar specification,
> >> etc.)
> >>>
> >>> The current isar-bootstrap based approach would work fine for isar
> system,
> >>> but in that case, it might be difficult to share efforts for
> >>> developing and maintaining the functionality with non isar users.
> >>> I'm just interested in the future plan of isar.
> >>
> >> Maybe you can give an example about which feature you think could be
> >> moved outside of the isar project and be used separately.
> >
> > I should learn the mechanism of setup_root_file_system() more,
> > but at least I'm originally considering how I can provide the following
> API:
> >
> > mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list>
> [<script>]]
> 
> To further consider is the case of raspbian where we need an additional
> public repo key and the possibility of employing package pinning and
> priorities as well as apt configuration to change the package selection
> of the bootstrapped image.
> 
> It would be great to have temporary apt settings (repo, package pinning)
> to allow offline building from a local repo, but the system will use the
> official one in production later.
> 
> IMO it would be great if the bootstrap process itself would populate a
> local repo with all used packages + source packages while it is fetching
> them from upstream and using that repo at the same time with high
> priority. This way calling the bootstrap process multiple times in a row
> (with removal of bootstrapped system, but not local repo) would result
> in exactly the same file system.
>    -> idempotent bootstrap
>    -> caching of installation data
>    -> reproducible bootstrapping

Yeah, that's required feature in product development.
I don't consider this issue deeply, but should in near future.

I usually create Debian based rootfs using only Debian official packages
(without other custom repos) and need to make it reproducible
in product development.
The way to do that is just keeping all package + version list in one file.

1st build:
	1. Download required packages from Debian mirror
	2. Create a local repository with the packages downloaded in 1
	3. Generate package + version list registered in the local repo
	4. debootstrap with the local repo
	5. chroot apt install with the local repo
Rebuild:
	1. Download required packages listed in the list from Debian mirror
	   (created in 3 in the 1st build)
	2. Same as 2, 4, 5 in the 1st build.

This might work when we use multiple apt repos.
Disadvantages are the time to build the local repository and
the time to create the package list. 
Also we need to keep the package list for each build somewhere.

> 
> The bootstrap process should also not just be one big step, but split
> into smaller idempotent ones. So that customization should be possible
> at every level.
> 
> IMO the 'script' option should not be used, since that seems to be a
> hacky way because the bootstrap process was not split up enough to call
> a script yourself at the right moment.

I think so too.

> 
> So my suggestion would be something like this:
> 
> mybootstrap init <suite> <arch> <target directory> # Setups a directory
> with some configuration files (.mybootstrap directory?)
> 
> mybootstrap -C <target directory> addrepo
> <static|temporary|reproducible> <url> <prio> # Add a repo, should work
> in with a un-generated as well as a generated system, static for repos
> that should remain after finalization, temporary ones should be removed
> and reproducible ones should be used and populated with used packages as
> mybootstrap does its thing, maybe a package filter here might be useful
> as well
> 
> mybootstrap -C <target directory> generate <package selection> #
> generates initial bootstrap environment with only required packages +
> apt -> afterwards it should be a chrootable system
> 
> mybootstrap -C <target directory> install <package> # Installs packages,
> since this might be more involved that just calling 'apt-get install',
> because of the reproducible repository, it would make send to wrap
> around that
> 
> mybootstrap -C <target directory> execute <script> # starts script
> inside the system with the right chroot
> 
> mybootstrap -C <target directory> finalize # removes mybootstrap stuff
> and temporary+reproducible repos from the apt configuration
> 
> I have more ideas for some additional subcommands, like executing repo
> updates, controlling the 'reproducible' repo better (removing packages,
> selectively updating packages, etc.) and some more. But that interface
> in general would be great for scripting custom Debian installations.

Thank you for your many ideas!
I guess the most important feature is "addrepo",
especially arguments to control repository types and priorities.
It would be great if mybootstrap above has features for reproducibility in itself.

Required features to control repositories above might depend on
what kind of apt repositories we need to handle.
(e.g. Debian mirrors managed by Debian project,
 Third-party repositories managed by its community,
 Local repositories (including apps, custom packages) fully managed by users,
 Temporal repositories for debugging, etc.)
It makes sense to provide options to flexibly handle such kind of differences.

> 
> > I guess that the most important part for me is isar_bootstrap() in
> isar-bootstrap.inc,
> > which is shared by isar-bootstrap-*.bb and copied by
> setup_root_file_system as
> > a base tree
> ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).
> > Other remaining commands in setup_root_file_system() seems to be almost
> based on apt,
> > so too easy to control multiple repositories.
> > In the following commands, only one debootstrap command is called with
> > one repository argument given from get_distro_primary_source_entry().
> >
> https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-boot
> strap/isar-bootstrap.inc#L175
> > Does it work with multiple apt repositories?
> > Or, installing required packages from other repos with apt after
> debootstrapping finishes?
> 
> The isar-bootstrap process has changed alot after I initially
> implemented it (cross-build support, etc.), and I haven't worked on that
> since.
> 
> But the general idea when I implemented it was that just one repo will
> be used for debootstrap (the first one listed), and afterwards the other
> repos are added and an update+dist-upgrade is performed on all of them.

Thanks. The current implementation seems to be basically same as previous.

> 
> After that step the upstream apt index should not be changed. No global
> apt-get update, only apt-get update of local repos where newly created
> packages are placed.
> 
> That way the buildchroot and the resulting image should contain the same
> versions of the packages which are installed in their respective usages
> of setup_root_file_system function in a copy of that initial
> bootstrapped system.
> 
> >
> >>
> >> One issue here is that Isar uses bitbake as the build task scheduler,
> so
> >> shell scripts etc. are difficult to extract from it to be used outside
> >> of Isar. And if they are extracted from the Isar recipes, then using
> >> them in Isar again is disadvantageous for the Isar build process.
> >>
> >> So sharing code between a bitbake/isar project and other build systems
> >> is not really possible.
> >
> > I usually have the same issues in my bitbake recipes development.
> > It might be possible to share the concrete processes to create
> > a Debian base system from multiple repositories,
> > even if it's difficult to share the actual codes.
> >
> > Background of my question:
> > I guess that people who are developing Debian based system
> > with their own custom packages (I'm one of them) may want to use
> > a command (or a wrapper script) to create a base image from multiple apt
> repos,
> > especially if they need to replace a part of existing Debian essential
> packages
> > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > I'm investigating existing approaches that provide the API above or
> similar one.
> 
> We would currently deal with that by having the local repo that contains
> our build packages added to the apt with a high priority.
> 
> Installing or updating now would remove the upstream package and install
> our own versions. Not the best way, but it works in principle.
> 
> I don't know how tested in Isar that currently is though.

Thank you for the information. I also think that this is the simplest way
and easy-to-understand if we don't need to install significantly customized
"essential" packages installed by debootstrap. (e.g. minimal perl)
I guess such kind of situations are not in the main focus of Isar and
should not be supported from maintenance PoV.


Regards,
Kazu

Re: multistrap support

[Henning Schild]

Am Mon, 25 Feb 2019 17:08:19 +0000
schrieb <kazuhiro3.hayashi@toshiba.co.jp>:

> Hi Claudius,
> 
> Sorry for my late reply.
> Thank you very much for sharing detailed information of Isar and your
> ideas!
> 
> > -----Original Message-----
> > From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com]
> > On Behalf Of Claudius Heine
> > Sent: Thursday, February 14, 2019 6:08 PM
> > To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> > <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> > Cc: meta-eid@googlegroups.com
> > Subject: Re: [meta-eid] Re: multistrap support
> > 
> > Hi Kazu,
> >   
> (snipped)
> > >>  
> > >>> (Some bugs difficult to be fixed, mismatches with isar
> > >>> specification,  
> > >> etc.)  
> > >>>
> > >>> The current isar-bootstrap based approach would work fine for
> > >>> isar  
> > system,  
> > >>> but in that case, it might be difficult to share efforts for
> > >>> developing and maintaining the functionality with non isar
> > >>> users. I'm just interested in the future plan of isar.  
> > >>
> > >> Maybe you can give an example about which feature you think
> > >> could be moved outside of the isar project and be used
> > >> separately.  
> > >
> > > I should learn the mechanism of setup_root_file_system() more,
> > > but at least I'm originally considering how I can provide the
> > > following  
> > API:  
> > >
> > > mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list>  
> > [<script>]]
> > 
> > To further consider is the case of raspbian where we need an
> > additional public repo key and the possibility of employing package
> > pinning and priorities as well as apt configuration to change the
> > package selection of the bootstrapped image.
> > 
> > It would be great to have temporary apt settings (repo, package
> > pinning) to allow offline building from a local repo, but the
> > system will use the official one in production later.
> > 
> > IMO it would be great if the bootstrap process itself would
> > populate a local repo with all used packages + source packages
> > while it is fetching them from upstream and using that repo at the
> > same time with high priority. This way calling the bootstrap
> > process multiple times in a row (with removal of bootstrapped
> > system, but not local repo) would result in exactly the same file
> > system. -> idempotent bootstrap
> >    -> caching of installation data
> >    -> reproducible bootstrapping  
> 
> Yeah, that's required feature in product development.
> I don't consider this issue deeply, but should in near future.
> 
> I usually create Debian based rootfs using only Debian official
> packages (without other custom repos) and need to make it reproducible
> in product development.
> The way to do that is just keeping all package + version list in one
> file.
> 
> 1st build:
> 	1. Download required packages from Debian mirror
> 	2. Create a local repository with the packages downloaded in 1
> 	3. Generate package + version list registered in the local
> repo 4. debootstrap with the local repo
> 	5. chroot apt install with the local repo
> Rebuild:
> 	1. Download required packages listed in the list from Debian
> mirror (created in 3 in the 1st build)
> 	2. Same as 2, 4, 5 in the 1st build.
> 
> This might work when we use multiple apt repos.
> Disadvantages are the time to build the local repository and
> the time to create the package list. 
> Also we need to keep the package list for each build somewhere.

Isar has support for something like that. Check out
ISAR_USE_CACHED_BASE_REPO

https://github.com/ilbers/isar/blob/next/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages

That is not the final solution and has several open points.

Henning

> > 
> > The bootstrap process should also not just be one big step, but
> > split into smaller idempotent ones. So that customization should be
> > possible at every level.
> > 
> > IMO the 'script' option should not be used, since that seems to be a
> > hacky way because the bootstrap process was not split up enough to
> > call a script yourself at the right moment.  
> 
> I think so too.
> 
> > 
> > So my suggestion would be something like this:
> > 
> > mybootstrap init <suite> <arch> <target directory> # Setups a
> > directory with some configuration files (.mybootstrap directory?)
> > 
> > mybootstrap -C <target directory> addrepo
> > <static|temporary|reproducible> <url> <prio> # Add a repo, should
> > work in with a un-generated as well as a generated system, static
> > for repos that should remain after finalization, temporary ones
> > should be removed and reproducible ones should be used and
> > populated with used packages as mybootstrap does its thing, maybe a
> > package filter here might be useful as well
> > 
> > mybootstrap -C <target directory> generate <package selection> #
> > generates initial bootstrap environment with only required packages
> > + apt -> afterwards it should be a chrootable system
> > 
> > mybootstrap -C <target directory> install <package> # Installs
> > packages, since this might be more involved that just calling
> > 'apt-get install', because of the reproducible repository, it would
> > make send to wrap around that
> > 
> > mybootstrap -C <target directory> execute <script> # starts script
> > inside the system with the right chroot
> > 
> > mybootstrap -C <target directory> finalize # removes mybootstrap
> > stuff and temporary+reproducible repos from the apt configuration
> > 
> > I have more ideas for some additional subcommands, like executing
> > repo updates, controlling the 'reproducible' repo better (removing
> > packages, selectively updating packages, etc.) and some more. But
> > that interface in general would be great for scripting custom
> > Debian installations.  
> 
> Thank you for your many ideas!
> I guess the most important feature is "addrepo",
> especially arguments to control repository types and priorities.
> It would be great if mybootstrap above has features for
> reproducibility in itself.
> 
> Required features to control repositories above might depend on
> what kind of apt repositories we need to handle.
> (e.g. Debian mirrors managed by Debian project,
>  Third-party repositories managed by its community,
>  Local repositories (including apps, custom packages) fully managed
> by users, Temporal repositories for debugging, etc.)
> It makes sense to provide options to flexibly handle such kind of
> differences.
> 
> >   
> > > I guess that the most important part for me is isar_bootstrap()
> > > in  
> > isar-bootstrap.inc,  
> > > which is shared by isar-bootstrap-*.bb and copied by  
> > setup_root_file_system as  
> > > a base tree  
> > ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).  
> > > Other remaining commands in setup_root_file_system() seems to be
> > > almost  
> > based on apt,  
> > > so too easy to control multiple repositories.
> > > In the following commands, only one debootstrap command is called
> > > with one repository argument given from
> > > get_distro_primary_source_entry(). 
> > https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-boot
> > strap/isar-bootstrap.inc#L175  
> > > Does it work with multiple apt repositories?
> > > Or, installing required packages from other repos with apt after  
> > debootstrapping finishes?
> > 
> > The isar-bootstrap process has changed alot after I initially
> > implemented it (cross-build support, etc.), and I haven't worked on
> > that since.
> > 
> > But the general idea when I implemented it was that just one repo
> > will be used for debootstrap (the first one listed), and afterwards
> > the other repos are added and an update+dist-upgrade is performed
> > on all of them.  
> 
> Thanks. The current implementation seems to be basically same as
> previous.
> 
> > 
> > After that step the upstream apt index should not be changed. No
> > global apt-get update, only apt-get update of local repos where
> > newly created packages are placed.
> > 
> > That way the buildchroot and the resulting image should contain the
> > same versions of the packages which are installed in their
> > respective usages of setup_root_file_system function in a copy of
> > that initial bootstrapped system.
> >   
> > >  
> > >>
> > >> One issue here is that Isar uses bitbake as the build task
> > >> scheduler,  
> > so  
> > >> shell scripts etc. are difficult to extract from it to be used
> > >> outside of Isar. And if they are extracted from the Isar
> > >> recipes, then using them in Isar again is disadvantageous for
> > >> the Isar build process.
> > >>
> > >> So sharing code between a bitbake/isar project and other build
> > >> systems is not really possible.  
> > >
> > > I usually have the same issues in my bitbake recipes development.
> > > It might be possible to share the concrete processes to create
> > > a Debian base system from multiple repositories,
> > > even if it's difficult to share the actual codes.
> > >
> > > Background of my question:
> > > I guess that people who are developing Debian based system
> > > with their own custom packages (I'm one of them) may want to use
> > > a command (or a wrapper script) to create a base image from
> > > multiple apt  
> > repos,  
> > > especially if they need to replace a part of existing Debian
> > > essential  
> > packages  
> > > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > > I'm investigating existing approaches that provide the API above
> > > or  
> > similar one.
> > 
> > We would currently deal with that by having the local repo that
> > contains our build packages added to the apt with a high priority.
> > 
> > Installing or updating now would remove the upstream package and
> > install our own versions. Not the best way, but it works in
> > principle.
> > 
> > I don't know how tested in Isar that currently is though.  
> 
> Thank you for the information. I also think that this is the simplest
> way and easy-to-understand if we don't need to install significantly
> customized "essential" packages installed by debootstrap. (e.g.
> minimal perl) I guess such kind of situations are not in the main
> focus of Isar and should not be supported from maintenance PoV.
> 
> 
> Regards,
> Kazu
> 
> 

Re: multistrap support

[kazuhiro3.hayashi]

Hi Henning,

> > (snipped)
> > I usually create Debian based rootfs using only Debian official
> > packages (without other custom repos) and need to make it reproducible
> > in product development.
> > The way to do that is just keeping all package + version list in one
> > file.
> >
> > 1st build:
> > 	1. Download required packages from Debian mirror
> > 	2. Create a local repository with the packages downloaded in 1
> > 	3. Generate package + version list registered in the local
> > repo 4. debootstrap with the local repo
> > 	5. chroot apt install with the local repo
> > Rebuild:
> > 	1. Download required packages listed in the list from Debian
> > mirror (created in 3 in the 1st build)
> > 	2. Same as 2, 4, 5 in the 1st build.
> >
> > This might work when we use multiple apt repos.
> > Disadvantages are the time to build the local repository and
> > the time to create the package list.
> > Also we need to keep the package list for each build somewhere.
> 
> Isar has support for something like that. Check out
> ISAR_USE_CACHED_BASE_REPO
> 
> https://github.com/ilbers/isar/blob/next/doc/user_manual.md#creation-o
> f-local-apt-repo-caching-upstream-debian-packages
> 
> That is not the final solution and has several open points.
> 
> Henning

Thank you for the information.
I talked with Maxim yesterday about this topic and understand
how to generate "downloads/base-apt".

Kind regards,
Kazu

RE: multistrap support

[kazuhiro3.hayashi]

Hi Baurzhan,

> -----Original Message-----
> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
> Behalf Of Baurzhan Ismagulov
> Sent: Thursday, February 14, 2019 4:24 PM
> To: isar-users@googlegroups.com
> Cc: wookey@wookware.org; meta-eid@googlegroups.com
> Subject: Re: [meta-eid] multistrap support
> 
> On Wed, Feb 13, 2019 at 01:14:57PM +0000, kazuhiro...@toshiba.co.jp wrote:
> > I would like to know if isar still has some plans to use multistrap again
> > (or similar existing tools if exists) to generate target images.
> >
> > Multistrap for isar-image-base has been removed by the following commit:
> > https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> > and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> > which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> > as the base tree instead of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> > Are there any other reasons?
> > (Some bugs difficult to be fixed, mismatches with isar specification, etc.)
> >
> > The current isar-bootstrap based approach would work fine for isar system,
> > but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> 
> Neil told us that he doesn't maintain multistrap anymore. We also encountered
> issues (e.g., https://github.com/ilbers/isar/issues/16). For me personally,
> Perl is an obstacle (I go through Perl tutorials every couple of years :) ). So
> switching to debootstrap was easier at the time.

Thanks. At the moment, I think that the debootstrap based approach might be safer
using than multistrap from future maintenance point of view.

> 
> That is not to say that debootstrap is the tool we want to have. The focus of
> debootstrap is to create Debian rootfss on Debian and non-Debian systems with
> the least possible dependencies. Given that, it is developed in shell, which is
> also not maintainable.
> 
> In the long run, we'll need more and more introspection into apt & Co. for more
> advanced features. One way could be multistrap re-implementation in Python, as
> suggested by Wookey. ELBE has good experiences with libapt. aptly also promises
> some features we would like to use (though probably not multistrapping OOTB).

I'm not sure how the multistrap re-implementation is demanded in Debian community,
though we are interested in that. In this discussion, I recognized that
there are several approaches to bootstrap Debian images including customized
'essential' packages. For examples,

1. multistrap (Debian mirror + Custom repos)
2. debootstrap (Debian mirror) + apt (Custom repos)
3. debootstrap (Merged repo = Debian mirror + Custom repos)
and more...?

aptly probably helps creating the merged repo in 3 as well as snapshotting
and maintaining required repos effectively, I should look at this.

> 
> Bootstrapping hybrid repos is also a useful use case. It can be achieved by
> making the "secondary" distribution upgradable from Debian, but if one has a
> lot of customized packages, using multistrap would require less effort and be
> less error-prone as well. In Isar, we try to stay away from massive
> modifications, but there are projects ending up tuning significant parts of
> Debian.

Thank you for telling the current situation in Isar.
IIUC, this is comparison of the 1 and 3 above.
I agree that multistrap effectively works in case where there are a lot of
customized packages.

> 
> The specific tools don't have to be mutually exclusive; Python offers enough
> polymorphism to include both as plugins (bitbake support has to be checked).
> With quite diverse requirements and opinions we already have today, we'll need
> pluggability, be it for multistrap or other features; the earlier we start
> playing with it, the better. For me, it is the question of enough itch rather
> than resources.
> 
> For your specific use case, I don't see a problem using multistrap for meta-eid
> till we have a better alternative.

I agree.
I've just support two solutions (1)multistrap and (2)debootstrap+apt in
debian-image.bb (Very simple example of the image generator)
https://github.com/eid-project/meta-eid/blob/master/recipes-image/debian-image/debian-image.bb

Kind regards,
Kazu

multistrap support

[kazuhiro3.hayashi]

Hi,

I would like to know if isar still has some plans to use multistrap again
(or similar existing tools if exists) to generate target images.

Multistrap for isar-image-base has been removed by the following commit:
https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
as the base tree instead of running the debootstrap process.

I guess that the main purposes of quitting multistrap is that
there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
Are there any other reasons?
(Some bugs difficult to be fixed, mismatches with isar specification, etc.)

The current isar-bootstrap based approach would work fine for isar system,
but in that case, it might be difficult to share efforts for
developing and maintaining the functionality with non isar users.
I'm just interested in the future plan of isar.

Kind regards,
Kazu

 Kazuhiro Hayashi
 Corporate Software Engineering & Technology Center
 Toshiba Corporation
 Tel: +81-44-549-2476
 E-mail: kazuhiro3.hayashi@toshiba.co.jp

Re: multistrap support

[Claudius Heine]

Hi Kazu,

I don't speak for the whole Isar community, but just from my point of 
view. For context I implemented the switch from multistrap to debootstrap.

On 13/02/2019 14.14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi,
> 
> I would like to know if isar still has some plans to use multistrap again
> (or similar existing tools if exists) to generate target images.

No, only debootstrap is supported and we will most probably stick with that.

> Multistrap for isar-image-base has been removed by the following commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> as the base tree instead of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> Are there any other reasons?

That is one of them. Debootstrap is better supported by the main Debian 
community than any other bootstrapping tools. And since it is out of 
scope or the Isar Project to implement its own bootstrapping mechanism, 
we will stick with the second most official one 'debootstrap'. The 
Debian installer being the most official way to install Debian, which is 
a bit to bulky and integrated to be easily used in a scripted environment.

Both multistrap and debootstrap had or still have some inconveniences 
with our usage and it would of course be great to have one that fits 
exactly our purposes and is used and supported by the broader 
Debian/Embedded Debian and possible Debian server community. But as I 
said Isar does not have the resources to implement that.

> (Some bugs difficult to be fixed, mismatches with isar specification, etc.)
> 
> The current isar-bootstrap based approach would work fine for isar system,
> but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.

Maybe you can give an example about which feature you think could be 
moved outside of the isar project and be used separately.

One issue here is that Isar uses bitbake as the build task scheduler, so 
shell scripts etc. are difficult to extract from it to be used outside 
of Isar. And if they are extracted from the Isar recipes, then using 
them in Isar again is disadvantageous for the Isar build process.

So sharing code between a bitbake/isar project and other build systems 
is not really possible.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: multistrap support

[Henning Schild]

Hi,

in fact Isar does not really care too much about how exactly
setup_root_file_system does what it does. Just the interface has to be
stable at some point. So you take a list of repos and packages and off
you go ... to put it overly-simple.

rootfs = strap(repos, package-names)

That is why the switch from multistrap to debootstrap was relatively
simple. So you could even think about switching back or switching to
yet another <funny>-strap.

What i would like to understand is why you care about multistrap and
why you think the strap-tool has an implication on collaboration with
other projects? Where do you see the risk of not being able to share
efforts with other projects that use multistrap?

Henning

Am Wed, 13 Feb 2019 13:14:57 +0000
schrieb <kazuhiro3.hayashi@toshiba.co.jp>:

> Hi,
> 
> I would like to know if isar still has some plans to use multistrap
> again (or similar existing tools if exists) to generate target images.
> 
> Multistrap for isar-image-base has been removed by the following
> commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in
> isar-bootstrap-helper.bbclass, which seems to use (pre-built?)
> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH as the base tree instead
> of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is
> 2.2.10 on Nov. 2018) Are there any other reasons?
> (Some bugs difficult to be fixed, mismatches with isar specification,
> etc.)
> 
> The current isar-bootstrap based approach would work fine for isar
> system, but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.
> 
> Kind regards,
> Kazu
> 
>  Kazuhiro Hayashi
>  Corporate Software Engineering & Technology Center
>  Toshiba Corporation
>  Tel: +81-44-549-2476
>  E-mail: kazuhiro3.hayashi@toshiba.co.jp
> 
> 

RE: multistrap support

[kazuhiro3.hayashi]

Hi Henning,

> -----Original Message-----
> From: Henning Schild [mailto:henning.schild@siemens.com]
> Sent: Thursday, February 14, 2019 12:19 AM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>
> Cc: isar-users@googlegroups.com; meta-eid@googlegroups.com
> Subject: Re: multistrap support
> 
> Hi,
> 
> in fact Isar does not really care too much about how exactly
> setup_root_file_system does what it does. Just the interface has to be
> stable at some point. So you take a list of repos and packages and off
> you go ... to put it overly-simple.
> 
> rootfs = strap(repos, package-names)
> 
> That is why the switch from multistrap to debootstrap was relatively
> simple. So you could even think about switching back or switching to
> yet another <funny>-strap.

Thanks, I got it.

I'm now watching isar_bootstrap() which creates base tree with debootstrap.
It seems to run one debootstrap command with one primary apt repository,
then merge available updates from other apt repositories by `apt-get dist-upgrade`.
I understood that these steps could replace calling multistrap command.

> 
> What i would like to understand is why you care about multistrap and
> why you think the strap-tool has an implication on collaboration with
> other projects? Where do you see the risk of not being able to share
> efforts with other projects that use multistrap?

I should explain that I'm not heavy user of Isar at the moment actually,
and I'm prototyping another build tool:
https://events.linuxfoundation.org/wp-content/uploads/2017/12/ELCE2018_Debian-Yocto-State-of-the-Art_r6_Kazuhiro-Hayashi.pdf
Many important visions of that project are from Isar development policy,
and tooling is one of the most important topic.

I WAS thinking that bootstrapping with multiple apt repositories
are complicated and existing tools like multistrap should be used
because of the fact (history) that multistrap has been developed as an independent package
from debootstrap to support multiple repositories.
However, if we can replace multistrap functionality by the above
(simple and easy to understand) steps with debootstrap,
the strap-tool might not affect collaboration of Isar and other projects so much.

Regards,
Kazu

> 
> Henning
> 
> Am Wed, 13 Feb 2019 13:14:57 +0000
> schrieb <kazuhiro3.hayashi@toshiba.co.jp>:
> 
> > Hi,
> >
> > I would like to know if isar still has some plans to use multistrap
> > again (or similar existing tools if exists) to generate target images.
> >
> > Multistrap for isar-image-base has been removed by the following
> > commit:
> >
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
> ca6c8884
> > and replaced by setup_root_file_system() in
> > isar-bootstrap-helper.bbclass, which seems to use (pre-built?)
> > isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH as the base tree instead
> > of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is
> > 2.2.10 on Nov. 2018) Are there any other reasons?
> > (Some bugs difficult to be fixed, mismatches with isar specification,
> > etc.)
> >
> > The current isar-bootstrap based approach would work fine for isar
> > system, but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> >
> > Kind regards,
> > Kazu
> >
> >  Kazuhiro Hayashi
> >  Corporate Software Engineering & Technology Center
> >  Toshiba Corporation
> >  Tel: +81-44-549-2476
> >  E-mail: kazuhiro3.hayashi@toshiba.co.jp
> >
> >

Re: multistrap support

[Jan Kiszka]

On 13.02.19 14:14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi,
> 
> I would like to know if isar still has some plans to use multistrap again
> (or similar existing tools if exists) to generate target images.
> 
> Multistrap for isar-image-base has been removed by the following commit:
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647ca6c8884
> and replaced by setup_root_file_system() in isar-bootstrap-helper.bbclass,
> which seems to use (pre-built?) isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> as the base tree instead of running the debootstrap process.
> 
> I guess that the main purposes of quitting multistrap is that
> there is no big update for a few years (though the last update is 2.2.10 on Nov. 2018)
> Are there any other reasons?
> (Some bugs difficult to be fixed, mismatches with isar specification, etc.)

Indeed, the key reason (beside some technical issues) for no longer using 
multistrap in Isar was that the multistrap creator himself explained to us that 
it's dead and we should rather sooner than later move away from it.

We then decided to pick up the most commonly used bootstrapping mechanism for 
Debian today, simply to reduce the risk of running into problems no one else is 
seeing - and fixing.

> 
> The current isar-bootstrap based approach would work fine for isar system,
> but in that case, it might be difficult to share efforts for
> developing and maintaining the functionality with non isar users.
> I'm just interested in the future plan of isar.

This takes concrete ideas what you consider worth sharing and what interfaces 
this may imply. Debootstrap is not married to Isar but it would take some use 
case to make that shareable or replaceable with something else.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: multistrap support

[kazuhiro3.hayashi]

Hi Jan,

> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Thursday, February 14, 2019 1:15 AM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: Re: multistrap support
> 
> On 13.02.19 14:14, kazuhiro3.hayashi@toshiba.co.jp wrote:
> > Hi,
> >
> > I would like to know if isar still has some plans to use multistrap again
> > (or similar existing tools if exists) to generate target images.
> >
> > Multistrap for isar-image-base has been removed by the following commit:
> >
> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
> ca6c8884
> > and replaced by setup_root_file_system() in
> isar-bootstrap-helper.bbclass,
> > which seems to use (pre-built?)
> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
> > as the base tree instead of running the debootstrap process.
> >
> > I guess that the main purposes of quitting multistrap is that
> > there is no big update for a few years (though the last update is 2.2.10
> on Nov. 2018)
> > Are there any other reasons?
> > (Some bugs difficult to be fixed, mismatches with isar specification,
> etc.)
> 
> Indeed, the key reason (beside some technical issues) for no longer using
> multistrap in Isar was that the multistrap creator himself explained to
> us that
> it's dead and we should rather sooner than later move away from it.

Thank you for the detailed information.

> 
> We then decided to pick up the most commonly used bootstrapping mechanism
> for
> Debian today, simply to reduce the risk of running into problems no one
> else is
> seeing - and fixing.

Reasonable approach.

> 
> >
> > The current isar-bootstrap based approach would work fine for isar system,
> > but in that case, it might be difficult to share efforts for
> > developing and maintaining the functionality with non isar users.
> > I'm just interested in the future plan of isar.
> 
> This takes concrete ideas what you consider worth sharing and what
> interfaces
> this may imply. Debootstrap is not married to Isar but it would take some
> use
> case to make that shareable or replaceable with something else.

I was thinking a kind of the following things, mentioned in the reply to Claudius:
> mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> [<script>]]
For examples, I assumes that the following apt repositories could be listed in the <mirror_list>
1. The official Debian apt repository (or a (local) mirror of it)
2. An apt repository for specific custom distro (like Debian deliverable)
3. An apt repository for specific project/product (includes apps, etc.)
If a command like the above provided, we can always generate the rootfs tree
only by one command (+ chroot apt install) without setting any tools up
(probably for experiment purposes).
This kind of technical discussion would be kept in other replies...

Regards,
Kazu