Re: [meta-eid] Re: multistrap support

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Kazu,

On 14/02/2019 04.53, kazuhiro3.hayashi@toshiba.co.jp wrote:
> Hi Claudius, (and Isar developers)
> 
> Thank you for your replies,
> I would like to reply one by one.
> 
>> -----Original Message-----
>> From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com] On
>> Behalf Of Claudius Heine
>> Sent: Wednesday, February 13, 2019 10:56 PM
>> To: hayashi kazuhiro(林 和宏 ○ＳＷＣ□ＯＳＴ)
>> <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
>> Cc: meta-eid@googlegroups.com
>> Subject: [meta-eid] Re: multistrap support
>>
>> Hi Kazu,
>>
>> I don't speak for the whole Isar community, but just from my point of
>> view. For context I implemented the switch from multistrap to debootstrap.
>>
>> On 13/02/2019 14.14, kazuhiro3.hayashi@toshiba.co.jp wrote:
>>> Hi,
>>>
>>> I would like to know if isar still has some plans to use multistrap again
>>> (or similar existing tools if exists) to generate target images.
>>
>> No, only debootstrap is supported and we will most probably stick with that.
>>
>>> Multistrap for isar-image-base has been removed by the following commit:
>>>
>> https://github.com/ilbers/isar/commit/19a314559178f7afd93ce3dafe8c8647
>> ca6c8884
>>> and replaced by setup_root_file_system() in
>> isar-bootstrap-helper.bbclass,
>>> which seems to use (pre-built?)
>> isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH
>>> as the base tree instead of running the debootstrap process.
>>>
>>> I guess that the main purposes of quitting multistrap is that
>>> there is no big update for a few years (though the last update is 2.2.10
>> on Nov. 2018)
>>> Are there any other reasons?
>>
>> That is one of them. Debootstrap is better supported by the main Debian
>> community than any other bootstrapping tools. And since it is out of
>> scope or the Isar Project to implement its own bootstrapping mechanism,
>> we will stick with the second most official one 'debootstrap'. The
>> Debian installer being the most official way to install Debian, which is
>> a bit to bulky and integrated to be easily used in a scripted environment.
> 
> I'm fully agree that using debootstrap, one of the core package in Debian
> for a long time, as the base tool to generate images from multiple apt repos
> if we decided not to use multistrap.
> Regarding Debian installer, base-installer(udeb) also uses debootstrap
> to install the base system before using apt and tasksel, but as you said,
> it's integrated to d-i scripts so difficult to use in non-installer environment.

I didn't know that the Debian installer uses debootstrap somewhere 
inside as well. Interesting.

>>
>> Both multistrap and debootstrap had or still have some inconveniences
>> with our usage and it would of course be great to have one that fits
>> exactly our purposes and is used and supported by the broader
>> Debian/Embedded Debian and possible Debian server community. But as I
>> said Isar does not have the resources to implement that.
> 
> Thank you for your opinion.
> I didn't mean that multistrap should be used in Isar again.
> Also, my opinion is that any other full 'replacement' of debootstrap
> should not be implemented if bootstrapping from multiple repositories
> can be achieved by pure debootstrap + additional scripts.
> 
> (Sorry, I was misunderstanding multistrap; I thought it uses
>   debootstrap as back-end, but that is a replacement actually.)
> 
> The current bitbake function in Isar is one of the better solutions.
> I should check more detailed implementation of that.

Thanks!

> 
>>
>>> (Some bugs difficult to be fixed, mismatches with isar specification,
>> etc.)
>>>
>>> The current isar-bootstrap based approach would work fine for isar system,
>>> but in that case, it might be difficult to share efforts for
>>> developing and maintaining the functionality with non isar users.
>>> I'm just interested in the future plan of isar.
>>
>> Maybe you can give an example about which feature you think could be
>> moved outside of the isar project and be used separately.
> 
> I should learn the mechanism of setup_root_file_system() more,
> but at least I'm originally considering how I can provide the following API:
> 
> mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> [<script>]]

To further consider is the case of raspbian where we need an additional 
public repo key and the possibility of employing package pinning and 
priorities as well as apt configuration to change the package selection 
of the bootstrapped image.

It would be great to have temporary apt settings (repo, package pinning) 
to allow offline building from a local repo, but the system will use the 
official one in production later.

IMO it would be great if the bootstrap process itself would populate a 
local repo with all used packages + source packages while it is fetching 
them from upstream and using that repo at the same time with high 
priority. This way calling the bootstrap process multiple times in a row 
(with removal of bootstrapped system, but not local repo) would result 
in exactly the same file system.
   -> idempotent bootstrap
   -> caching of installation data
   -> reproducible bootstraping

The bootstrap process should also not just be one big step, but split 
into smaller idempotent ones. So that customization should be possible 
at every level.

IMO the 'script' option should not be used, since that seems to be a 
hacky way because the bootstrap process was not split up enough to call 
a script yourself at the right moment.

So my suggestion would be something like this:

mybootstrap init <suite> <arch> <target directory> # Setups a directory 
with some configuration files (.mybootstrap directory?)

mybootstrap -C <target directory> addrepo 
<static|temporary|reproducible> <url> <prio> # Add a repo, should work 
in with a un-generated as well as a generated system, static for repos 
that should remain after finalization, temporary ones should be removed 
and reproducible ones should be used and populated with used packages as 
mybootstrap does its thing, maybe a package filter here might be useful 
as well

mybootstrap -C <target directory> generate <package selection> # 
generates initial bootstrap environment with only required packages + 
apt -> afterwards it should be a chrootable system

mybootstrap -C <target directory> install <package> # Installs packages, 
since this might be more involved that just calling 'apt-get install', 
because of the reproducible repository, it would make send to wrap 
around that

mybootstrap -C <target directory> execute <script> # starts script 
inside the system with the right chroot

mybootstrap -C <target directory> finalize # removes mybootstrap stuff 
and temporary+reproducible repos from the apt configuration

I have more ideas for some additional subcommands, like executing repo 
updates, controlling the 'reproducible' repo better (removing packages, 
selectively updating packages, etc.) and some more. But that interface 
in general would be great for scripting custom Debian installations.

> I guess that the most important part for me is isar_bootstrap() in isar-bootstrap.inc,
> which is shared by isar-bootstrap-*.bb and copied by setup_root_file_system as
> a base tree ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/).
> Other remaining commands in setup_root_file_system() seems to be almost based on apt,
> so too easy to control multiple repositories.
> In the following commands, only one debootstrap command is called with
> one repository argument given from get_distro_primary_source_entry().
> https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc#L175
> Does it work with multiple apt repositories?
> Or, installing required packages from other repos with apt after debootstrapping finishes?

The isar-bootstrap process has changed alot after I initially 
implemented it (cross-build support, etc.), and I haven't worked on that 
since.

But the general idea when I implemented it was that just one repo will 
be used for debootstrap (the first one listed), and afterwards the other 
repos are added and an update+dist-upgrade is performed on all of them.

After that step the upstream apt index should not be changed. No global 
apt-get update, only apt-get update of local repos where newly created 
packages are placed.

That way the buildchroot and the resulting image should contain the same 
versions of the packages which are installed in their respective usages 
of setup_root_file_system function in a copy of that initial 
bootstrapped system.

> 
>>
>> One issue here is that Isar uses bitbake as the build task scheduler, so
>> shell scripts etc. are difficult to extract from it to be used outside
>> of Isar. And if they are extracted from the Isar recipes, then using
>> them in Isar again is disadvantageous for the Isar build process.
>>
>> So sharing code between a bitbake/isar project and other build systems
>> is not really possible.
> 
> I usually have the same issues in my bitbake recipes development.
> It might be possible to share the concrete processes to create
> a Debian base system from multiple repositories,
> even if it's difficult to share the actual codes.
> 
> Background of my question:
> I guess that people who are developing Debian based system
> with their own custom packages (I'm one of them) may want to use
> a command (or a wrapper script) to create a base image from multiple apt repos,
> especially if they need to replace a part of existing Debian essential packages
> by their custom packages (coreutils, util-linux, dpkg, etc.).
> I'm investigating existing approaches that provide the API above or similar one.

We would currently deal with that by having the local repo that contains 
our build packages added to the apt with a high priority.

Installing or updating now would remove the upstream package and install 
our own versions. Not the best way, but it works in principle.

I don't know how tested in Isar that currently is though.

regards,
Claudius


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de