Re: multistrap support

[<kazuhiro3.hayashi@toshiba.co.jp>]

Hi Claudius,

Sorry for my late reply.
Thank you very much for sharing detailed information of Isar and your ideas!

> -----Original Message-----
> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
> Behalf Of Claudius Heine
> Sent: Thursday, February 14, 2019 6:08 PM
> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> Cc: meta-eid@googlegroups.com
> Subject: Re: [meta-eid] Re: multistrap support
> 
> Hi Kazu,
> 
(snipped)
> >>
> >>> (Some bugs difficult to be fixed, mismatches with isar specification,
> >> etc.)
> >>>
> >>> The current isar-bootstrap based approach would work fine for isar
> system,
> >>> but in that case, it might be difficult to share efforts for
> >>> developing and maintaining the functionality with non isar users.
> >>> I'm just interested in the future plan of isar.
> >>
> >> Maybe you can give an example about which feature you think could be
> >> moved outside of the isar project and be used separately.
> >
> > I should learn the mechanism of setup_root_file_system() more,
> > but at least I'm originally considering how I can provide the following
> API:
> >
> > mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list>
> [<script>]]
> 
> To further consider is the case of raspbian where we need an additional
> public repo key and the possibility of employing package pinning and
> priorities as well as apt configuration to change the package selection
> of the bootstrapped image.
> 
> It would be great to have temporary apt settings (repo, package pinning)
> to allow offline building from a local repo, but the system will use the
> official one in production later.
> 
> IMO it would be great if the bootstrap process itself would populate a
> local repo with all used packages + source packages while it is fetching
> them from upstream and using that repo at the same time with high
> priority. This way calling the bootstrap process multiple times in a row
> (with removal of bootstrapped system, but not local repo) would result
> in exactly the same file system.
>    -> idempotent bootstrap
>    -> caching of installation data
>    -> reproducible bootstrapping

Yeah, that's required feature in product development.
I don't consider this issue deeply, but should in near future.

I usually create Debian based rootfs using only Debian official packages
(without other custom repos) and need to make it reproducible
in product development.
The way to do that is just keeping all package + version list in one file.

1st build:
	1. Download required packages from Debian mirror
	2. Create a local repository with the packages downloaded in 1
	3. Generate package + version list registered in the local repo
	4. debootstrap with the local repo
	5. chroot apt install with the local repo
Rebuild:
	1. Download required packages listed in the list from Debian mirror
	   (created in 3 in the 1st build)
	2. Same as 2, 4, 5 in the 1st build.

This might work when we use multiple apt repos.
Disadvantages are the time to build the local repository and
the time to create the package list. 
Also we need to keep the package list for each build somewhere.

> 
> The bootstrap process should also not just be one big step, but split
> into smaller idempotent ones. So that customization should be possible
> at every level.
> 
> IMO the 'script' option should not be used, since that seems to be a
> hacky way because the bootstrap process was not split up enough to call
> a script yourself at the right moment.

I think so too.

> 
> So my suggestion would be something like this:
> 
> mybootstrap init <suite> <arch> <target directory> # Setups a directory
> with some configuration files (.mybootstrap directory?)
> 
> mybootstrap -C <target directory> addrepo
> <static|temporary|reproducible> <url> <prio> # Add a repo, should work
> in with a un-generated as well as a generated system, static for repos
> that should remain after finalization, temporary ones should be removed
> and reproducible ones should be used and populated with used packages as
> mybootstrap does its thing, maybe a package filter here might be useful
> as well
> 
> mybootstrap -C <target directory> generate <package selection> #
> generates initial bootstrap environment with only required packages +
> apt -> afterwards it should be a chrootable system
> 
> mybootstrap -C <target directory> install <package> # Installs packages,
> since this might be more involved that just calling 'apt-get install',
> because of the reproducible repository, it would make send to wrap
> around that
> 
> mybootstrap -C <target directory> execute <script> # starts script
> inside the system with the right chroot
> 
> mybootstrap -C <target directory> finalize # removes mybootstrap stuff
> and temporary+reproducible repos from the apt configuration
> 
> I have more ideas for some additional subcommands, like executing repo
> updates, controlling the 'reproducible' repo better (removing packages,
> selectively updating packages, etc.) and some more. But that interface
> in general would be great for scripting custom Debian installations.

Thank you for your many ideas!
I guess the most important feature is "addrepo",
especially arguments to control repository types and priorities.
It would be great if mybootstrap above has features for reproducibility in itself.

Required features to control repositories above might depend on
what kind of apt repositories we need to handle.
(e.g. Debian mirrors managed by Debian project,
 Third-party repositories managed by its community,
 Local repositories (including apps, custom packages) fully managed by users,
 Temporal repositories for debugging, etc.)
It makes sense to provide options to flexibly handle such kind of differences.

> 
> > I guess that the most important part for me is isar_bootstrap() in
> isar-bootstrap.inc,
> > which is shared by isar-bootstrap-*.bb and copied by
> setup_root_file_system as
> > a base tree
> ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).
> > Other remaining commands in setup_root_file_system() seems to be almost
> based on apt,
> > so too easy to control multiple repositories.
> > In the following commands, only one debootstrap command is called with
> > one repository argument given from get_distro_primary_source_entry().
> >
> https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-boot
> strap/isar-bootstrap.inc#L175
> > Does it work with multiple apt repositories?
> > Or, installing required packages from other repos with apt after
> debootstrapping finishes?
> 
> The isar-bootstrap process has changed alot after I initially
> implemented it (cross-build support, etc.), and I haven't worked on that
> since.
> 
> But the general idea when I implemented it was that just one repo will
> be used for debootstrap (the first one listed), and afterwards the other
> repos are added and an update+dist-upgrade is performed on all of them.

Thanks. The current implementation seems to be basically same as previous.

> 
> After that step the upstream apt index should not be changed. No global
> apt-get update, only apt-get update of local repos where newly created
> packages are placed.
> 
> That way the buildchroot and the resulting image should contain the same
> versions of the packages which are installed in their respective usages
> of setup_root_file_system function in a copy of that initial
> bootstrapped system.
> 
> >
> >>
> >> One issue here is that Isar uses bitbake as the build task scheduler,
> so
> >> shell scripts etc. are difficult to extract from it to be used outside
> >> of Isar. And if they are extracted from the Isar recipes, then using
> >> them in Isar again is disadvantageous for the Isar build process.
> >>
> >> So sharing code between a bitbake/isar project and other build systems
> >> is not really possible.
> >
> > I usually have the same issues in my bitbake recipes development.
> > It might be possible to share the concrete processes to create
> > a Debian base system from multiple repositories,
> > even if it's difficult to share the actual codes.
> >
> > Background of my question:
> > I guess that people who are developing Debian based system
> > with their own custom packages (I'm one of them) may want to use
> > a command (or a wrapper script) to create a base image from multiple apt
> repos,
> > especially if they need to replace a part of existing Debian essential
> packages
> > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > I'm investigating existing approaches that provide the API above or
> similar one.
> 
> We would currently deal with that by having the local repo that contains
> our build packages added to the apt with a high priority.
> 
> Installing or updating now would remove the upstream package and install
> our own versions. Not the best way, but it works in principle.
> 
> I don't know how tested in Isar that currently is though.

Thank you for the information. I also think that this is the simplest way
and easy-to-understand if we don't need to install significantly customized
"essential" packages installed by debootstrap. (e.g. minimal perl)
I guess such kind of situations are not in the main focus of Isar and
should not be supported from maintenance PoV.


Regards,
Kazu