Re: multistrap support

[Henning Schild <henning.schild@siemens.com>]

Am Mon, 25 Feb 2019 17:08:19 +0000
schrieb <kazuhiro3.hayashi@toshiba.co.jp>:

> Hi Claudius,
> 
> Sorry for my late reply.
> Thank you very much for sharing detailed information of Isar and your
> ideas!
> 
> > -----Original Message-----
> > From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com]
> > On Behalf Of Claudius Heine
> > Sent: Thursday, February 14, 2019 6:08 PM
> > To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> > <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> > Cc: meta-eid@googlegroups.com
> > Subject: Re: [meta-eid] Re: multistrap support
> > 
> > Hi Kazu,
> >   
> (snipped)
> > >>  
> > >>> (Some bugs difficult to be fixed, mismatches with isar
> > >>> specification,  
> > >> etc.)  
> > >>>
> > >>> The current isar-bootstrap based approach would work fine for
> > >>> isar  
> > system,  
> > >>> but in that case, it might be difficult to share efforts for
> > >>> developing and maintaining the functionality with non isar
> > >>> users. I'm just interested in the future plan of isar.  
> > >>
> > >> Maybe you can give an example about which feature you think
> > >> could be moved outside of the isar project and be used
> > >> separately.  
> > >
> > > I should learn the mechanism of setup_root_file_system() more,
> > > but at least I'm originally considering how I can provide the
> > > following  
> > API:  
> > >
> > > mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list>  
> > [<script>]]
> > 
> > To further consider is the case of raspbian where we need an
> > additional public repo key and the possibility of employing package
> > pinning and priorities as well as apt configuration to change the
> > package selection of the bootstrapped image.
> > 
> > It would be great to have temporary apt settings (repo, package
> > pinning) to allow offline building from a local repo, but the
> > system will use the official one in production later.
> > 
> > IMO it would be great if the bootstrap process itself would
> > populate a local repo with all used packages + source packages
> > while it is fetching them from upstream and using that repo at the
> > same time with high priority. This way calling the bootstrap
> > process multiple times in a row (with removal of bootstrapped
> > system, but not local repo) would result in exactly the same file
> > system. -> idempotent bootstrap
> >    -> caching of installation data
> >    -> reproducible bootstrapping  
> 
> Yeah, that's required feature in product development.
> I don't consider this issue deeply, but should in near future.
> 
> I usually create Debian based rootfs using only Debian official
> packages (without other custom repos) and need to make it reproducible
> in product development.
> The way to do that is just keeping all package + version list in one
> file.
> 
> 1st build:
> 	1. Download required packages from Debian mirror
> 	2. Create a local repository with the packages downloaded in 1
> 	3. Generate package + version list registered in the local
> repo 4. debootstrap with the local repo
> 	5. chroot apt install with the local repo
> Rebuild:
> 	1. Download required packages listed in the list from Debian
> mirror (created in 3 in the 1st build)
> 	2. Same as 2, 4, 5 in the 1st build.
> 
> This might work when we use multiple apt repos.
> Disadvantages are the time to build the local repository and
> the time to create the package list. 
> Also we need to keep the package list for each build somewhere.

Isar has support for something like that. Check out
ISAR_USE_CACHED_BASE_REPO

https://github.com/ilbers/isar/blob/next/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages

That is not the final solution and has several open points.

Henning

> > 
> > The bootstrap process should also not just be one big step, but
> > split into smaller idempotent ones. So that customization should be
> > possible at every level.
> > 
> > IMO the 'script' option should not be used, since that seems to be a
> > hacky way because the bootstrap process was not split up enough to
> > call a script yourself at the right moment.  
> 
> I think so too.
> 
> > 
> > So my suggestion would be something like this:
> > 
> > mybootstrap init <suite> <arch> <target directory> # Setups a
> > directory with some configuration files (.mybootstrap directory?)
> > 
> > mybootstrap -C <target directory> addrepo
> > <static|temporary|reproducible> <url> <prio> # Add a repo, should
> > work in with a un-generated as well as a generated system, static
> > for repos that should remain after finalization, temporary ones
> > should be removed and reproducible ones should be used and
> > populated with used packages as mybootstrap does its thing, maybe a
> > package filter here might be useful as well
> > 
> > mybootstrap -C <target directory> generate <package selection> #
> > generates initial bootstrap environment with only required packages
> > + apt -> afterwards it should be a chrootable system
> > 
> > mybootstrap -C <target directory> install <package> # Installs
> > packages, since this might be more involved that just calling
> > 'apt-get install', because of the reproducible repository, it would
> > make send to wrap around that
> > 
> > mybootstrap -C <target directory> execute <script> # starts script
> > inside the system with the right chroot
> > 
> > mybootstrap -C <target directory> finalize # removes mybootstrap
> > stuff and temporary+reproducible repos from the apt configuration
> > 
> > I have more ideas for some additional subcommands, like executing
> > repo updates, controlling the 'reproducible' repo better (removing
> > packages, selectively updating packages, etc.) and some more. But
> > that interface in general would be great for scripting custom
> > Debian installations.  
> 
> Thank you for your many ideas!
> I guess the most important feature is "addrepo",
> especially arguments to control repository types and priorities.
> It would be great if mybootstrap above has features for
> reproducibility in itself.
> 
> Required features to control repositories above might depend on
> what kind of apt repositories we need to handle.
> (e.g. Debian mirrors managed by Debian project,
>  Third-party repositories managed by its community,
>  Local repositories (including apps, custom packages) fully managed
> by users, Temporal repositories for debugging, etc.)
> It makes sense to provide options to flexibly handle such kind of
> differences.
> 
> >   
> > > I guess that the most important part for me is isar_bootstrap()
> > > in  
> > isar-bootstrap.inc,  
> > > which is shared by isar-bootstrap-*.bb and copied by  
> > setup_root_file_system as  
> > > a base tree  
> > ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).  
> > > Other remaining commands in setup_root_file_system() seems to be
> > > almost  
> > based on apt,  
> > > so too easy to control multiple repositories.
> > > In the following commands, only one debootstrap command is called
> > > with one repository argument given from
> > > get_distro_primary_source_entry(). 
> > https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-boot
> > strap/isar-bootstrap.inc#L175  
> > > Does it work with multiple apt repositories?
> > > Or, installing required packages from other repos with apt after  
> > debootstrapping finishes?
> > 
> > The isar-bootstrap process has changed alot after I initially
> > implemented it (cross-build support, etc.), and I haven't worked on
> > that since.
> > 
> > But the general idea when I implemented it was that just one repo
> > will be used for debootstrap (the first one listed), and afterwards
> > the other repos are added and an update+dist-upgrade is performed
> > on all of them.  
> 
> Thanks. The current implementation seems to be basically same as
> previous.
> 
> > 
> > After that step the upstream apt index should not be changed. No
> > global apt-get update, only apt-get update of local repos where
> > newly created packages are placed.
> > 
> > That way the buildchroot and the resulting image should contain the
> > same versions of the packages which are installed in their
> > respective usages of setup_root_file_system function in a copy of
> > that initial bootstrapped system.
> >   
> > >  
> > >>
> > >> One issue here is that Isar uses bitbake as the build task
> > >> scheduler,  
> > so  
> > >> shell scripts etc. are difficult to extract from it to be used
> > >> outside of Isar. And if they are extracted from the Isar
> > >> recipes, then using them in Isar again is disadvantageous for
> > >> the Isar build process.
> > >>
> > >> So sharing code between a bitbake/isar project and other build
> > >> systems is not really possible.  
> > >
> > > I usually have the same issues in my bitbake recipes development.
> > > It might be possible to share the concrete processes to create
> > > a Debian base system from multiple repositories,
> > > even if it's difficult to share the actual codes.
> > >
> > > Background of my question:
> > > I guess that people who are developing Debian based system
> > > with their own custom packages (I'm one of them) may want to use
> > > a command (or a wrapper script) to create a base image from
> > > multiple apt  
> > repos,  
> > > especially if they need to replace a part of existing Debian
> > > essential  
> > packages  
> > > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > > I'm investigating existing approaches that provide the API above
> > > or  
> > similar one.
> > 
> > We would currently deal with that by having the local repo that
> > contains our build packages added to the apt with a high priority.
> > 
> > Installing or updating now would remove the upstream package and
> > install our own versions. Not the best way, but it works in
> > principle.
> > 
> > I don't know how tested in Isar that currently is though.  
> 
> Thank you for the information. I also think that this is the simplest
> way and easy-to-understand if we don't need to install significantly
> customized "essential" packages installed by debootstrap. (e.g.
> minimal perl) I guess such kind of situations are not in the main
> focus of Isar and should not be supported from maintenance PoV.
> 
> 
> Regards,
> Kazu
> 
>