From mboxrd@z Thu Jan  1 00:00:00 1970
X-GM-THRID: 6657472919531159552
X-Received: by 2002:a1c:4087:: with SMTP id n129mr2863wma.13.1551117560530;
        Mon, 25 Feb 2019 09:59:20 -0800 (PST)
X-BeenThere: isar-users@googlegroups.com
Received: by 2002:a5d:4488:: with SMTP id j8ls1370950wrq.1.gmail; Mon, 25 Feb
 2019 09:59:20 -0800 (PST)
X-Google-Smtp-Source: AHgI3IagfqP+WUK+mh5sxqNIaKM25Nvi2EIspIgzvFtLYo7PKI1WQAIA/+R2vjnOwVKL7bQXlMrJ
X-Received: by 2002:adf:822f:: with SMTP id 44mr1018071wrb.2.1551117560099;
        Mon, 25 Feb 2019 09:59:20 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1551117560; cv=none;
        d=google.com; s=arc-20160816;
        b=sbso8yhpHDfouBBVsapVPhenoee9zcsCHUpp/NAZyo45masbz1MFYHdzNvoFWMgfaU
         T7afnlrmWRSyPAgGQiSIkj4SpO3q/8hzVakxfRRtb875FF+hnFzTsyMeyP5Zb/7XmGR5
         bS8ThGHySy2OJxKj3fwe1R9bJ+3Gg4uAx08vHR61M6vV1v/WXupfREF6MzQpVEQqAinc
         Ad+f2Su5QJ2lAqpbW8j2mnhVFMP7m+GzDkVnooiM1IEzcImMc2NmgAOl2SaJ8pMotZ4n
         bMFqX1QkSlHOtqfd4s3m6YP23YB7DJGeU64NtLdYebG9VSCK5ALhM563+JgxxbKm68br
         2M7g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=content-transfer-encoding:mime-version:references:in-reply-to
         :message-id:subject:cc:to:from:date;
        bh=YO1XGZ31+tssjmI01i1TqRfHRo2tfi2s0OYBF7poQNM=;
        b=POc34e05JpWvhvLj4LZx9W7wFV5SONUu3LKu4WJBRLhQb1km2nkD+4va9PGi7UAy63
         XKchSy7PuEKfYZy5GgPBsSWb4fVL8tU0eif/ETz4p1uQ/5YJYfJOpZ9PJRpnREmgNb2p
         t88Wk8LgWKw/Yep/NAnrIHRSCCYZbaYL31KdMtnn7dAunG+bO7wMhPc4UyG4+aF40c8O
         gNTcEymRYPQDNFoVkCn+KH0InhtoPGcwcx4HGA9a+JGXML/Zb7t+m+NpVofmdSePjPL7
         6+Aow7je3bQ0Mh2TV7KiPcMH/n0MkMXvI9EjJbOuX8L8IwRuIB4/h6XPxeDwy265dPqn
         3Kag==
ARC-Authentication-Results: i=1; gmr-mx.google.com;
       spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com
Return-Path: <henning.schild@siemens.com>
Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28])
        by gmr-mx.google.com with ESMTPS id b134si160075wmd.2.2019.02.25.09.59.20
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Mon, 25 Feb 2019 09:59:20 -0800 (PST)
Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28;
Authentication-Results: gmr-mx.google.com;
       spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com
Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35])
	by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x1PHxJoR026593
	(version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Mon, 25 Feb 2019 18:59:19 +0100
Received: from md1za8fc.ad001.siemens.net ([139.25.68.188])
	by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x1PHxI9a015602;
	Mon, 25 Feb 2019 18:59:19 +0100
Date: Mon, 25 Feb 2019 18:59:18 +0100
From: Henning Schild <henning.schild@siemens.com>
To: <kazuhiro3.hayashi@toshiba.co.jp>
Cc: <claudius.heine.ext@siemens.com>, <isar-users@googlegroups.com>,
        <meta-eid@googlegroups.com>
Subject: Re: multistrap support
Message-ID: <20190225185918.0fc7edb8@md1za8fc.ad001.siemens.net>
In-Reply-To: <TYXPR01MB182254013C0FABA27476595DE17A0@TYXPR01MB1822.jpnprd01.prod.outlook.com>
References: <TYXPR01MB182254013C0FABA27476595DE17A0@TYXPR01MB1822.jpnprd01.prod.outlook.com>
X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-TUID: AzTYTR3kyy3O

Am Mon, 25 Feb 2019 17:08:19 +0000
schrieb <kazuhiro3.hayashi@toshiba.co.jp>:

> Hi Claudius,
>=20
> Sorry for my late reply.
> Thank you very much for sharing detailed information of Isar and your
> ideas!
>=20
> > -----Original Message-----
> > From: meta-eid@googlegroups.com [mailto:meta-eid@googlegroups.com]
> > On Behalf Of Claudius Heine
> > Sent: Thursday, February 14, 2019 6:08 PM
> > To: hayashi kazuhiro(=E6=9E=97 =E5=92=8C=E5=AE=8F =E2=97=8B=EF=BC=B3=EF=
=BC=B7=EF=BC=A3=E2=96=A1=EF=BC=AF=EF=BC=B3=EF=BC=B4)
> > <kazuhiro3.hayashi@toshiba.co.jp>; isar-users@googlegroups.com
> > Cc: meta-eid@googlegroups.com
> > Subject: Re: [meta-eid] Re: multistrap support
> >=20
> > Hi Kazu,
> >  =20
> (snipped)
> > >> =20
> > >>> (Some bugs difficult to be fixed, mismatches with isar
> > >>> specification, =20
> > >> etc.) =20
> > >>>
> > >>> The current isar-bootstrap based approach would work fine for
> > >>> isar =20
> > system, =20
> > >>> but in that case, it might be difficult to share efforts for
> > >>> developing and maintaining the functionality with non isar
> > >>> users. I'm just interested in the future plan of isar. =20
> > >>
> > >> Maybe you can give an example about which feature you think
> > >> could be moved outside of the isar project and be used
> > >> separately. =20
> > >
> > > I should learn the mechanism of setup_root_file_system() more,
> > > but at least I'm originally considering how I can provide the
> > > following =20
> > API: =20
> > >
> > > mybootstrap [debootstrap_OPTION] <suite> <target> [<mirror_list> =20
> > [<script>]]
> >=20
> > To further consider is the case of raspbian where we need an
> > additional public repo key and the possibility of employing package
> > pinning and priorities as well as apt configuration to change the
> > package selection of the bootstrapped image.
> >=20
> > It would be great to have temporary apt settings (repo, package
> > pinning) to allow offline building from a local repo, but the
> > system will use the official one in production later.
> >=20
> > IMO it would be great if the bootstrap process itself would
> > populate a local repo with all used packages + source packages
> > while it is fetching them from upstream and using that repo at the
> > same time with high priority. This way calling the bootstrap
> > process multiple times in a row (with removal of bootstrapped
> > system, but not local repo) would result in exactly the same file
> > system. -> idempotent bootstrap
> >    -> caching of installation data
> >    -> reproducible bootstrapping =20
>=20
> Yeah, that's required feature in product development.
> I don't consider this issue deeply, but should in near future.
>=20
> I usually create Debian based rootfs using only Debian official
> packages (without other custom repos) and need to make it reproducible
> in product development.
> The way to do that is just keeping all package + version list in one
> file.
>=20
> 1st build:
> 	1. Download required packages from Debian mirror
> 	2. Create a local repository with the packages downloaded in 1
> 	3. Generate package + version list registered in the local
> repo 4. debootstrap with the local repo
> 	5. chroot apt install with the local repo
> Rebuild:
> 	1. Download required packages listed in the list from Debian
> mirror (created in 3 in the 1st build)
> 	2. Same as 2, 4, 5 in the 1st build.
>=20
> This might work when we use multiple apt repos.
> Disadvantages are the time to build the local repository and
> the time to create the package list.=20
> Also we need to keep the package list for each build somewhere.

Isar has support for something like that. Check out
ISAR_USE_CACHED_BASE_REPO

https://github.com/ilbers/isar/blob/next/doc/user_manual.md#creation-of-loc=
al-apt-repo-caching-upstream-debian-packages

That is not the final solution and has several open points.

Henning

> >=20
> > The bootstrap process should also not just be one big step, but
> > split into smaller idempotent ones. So that customization should be
> > possible at every level.
> >=20
> > IMO the 'script' option should not be used, since that seems to be a
> > hacky way because the bootstrap process was not split up enough to
> > call a script yourself at the right moment. =20
>=20
> I think so too.
>=20
> >=20
> > So my suggestion would be something like this:
> >=20
> > mybootstrap init <suite> <arch> <target directory> # Setups a
> > directory with some configuration files (.mybootstrap directory?)
> >=20
> > mybootstrap -C <target directory> addrepo
> > <static|temporary|reproducible> <url> <prio> # Add a repo, should
> > work in with a un-generated as well as a generated system, static
> > for repos that should remain after finalization, temporary ones
> > should be removed and reproducible ones should be used and
> > populated with used packages as mybootstrap does its thing, maybe a
> > package filter here might be useful as well
> >=20
> > mybootstrap -C <target directory> generate <package selection> #
> > generates initial bootstrap environment with only required packages
> > + apt -> afterwards it should be a chrootable system
> >=20
> > mybootstrap -C <target directory> install <package> # Installs
> > packages, since this might be more involved that just calling
> > 'apt-get install', because of the reproducible repository, it would
> > make send to wrap around that
> >=20
> > mybootstrap -C <target directory> execute <script> # starts script
> > inside the system with the right chroot
> >=20
> > mybootstrap -C <target directory> finalize # removes mybootstrap
> > stuff and temporary+reproducible repos from the apt configuration
> >=20
> > I have more ideas for some additional subcommands, like executing
> > repo updates, controlling the 'reproducible' repo better (removing
> > packages, selectively updating packages, etc.) and some more. But
> > that interface in general would be great for scripting custom
> > Debian installations. =20
>=20
> Thank you for your many ideas!
> I guess the most important feature is "addrepo",
> especially arguments to control repository types and priorities.
> It would be great if mybootstrap above has features for
> reproducibility in itself.
>=20
> Required features to control repositories above might depend on
> what kind of apt repositories we need to handle.
> (e.g. Debian mirrors managed by Debian project,
>  Third-party repositories managed by its community,
>  Local repositories (including apps, custom packages) fully managed
> by users, Temporal repositories for debugging, etc.)
> It makes sense to provide options to flexibly handle such kind of
> differences.
>=20
> >  =20
> > > I guess that the most important part for me is isar_bootstrap()
> > > in =20
> > isar-bootstrap.inc, =20
> > > which is shared by isar-bootstrap-*.bb and copied by =20
> > setup_root_file_system as =20
> > > a base tree =20
> > ("${DEPLOY_DIR_IMAGE}/isar-bootstrap-$ROOTFS_DISTRO-$ROOTFS_ARCH/). =20
> > > Other remaining commands in setup_root_file_system() seems to be
> > > almost =20
> > based on apt, =20
> > > so too easy to control multiple repositories.
> > > In the following commands, only one debootstrap command is called
> > > with one repository argument given from
> > > get_distro_primary_source_entry().=20
> > https://github.com/ilbers/isar/blob/master/meta/recipes-core/isar-boot
> > strap/isar-bootstrap.inc#L175 =20
> > > Does it work with multiple apt repositories?
> > > Or, installing required packages from other repos with apt after =20
> > debootstrapping finishes?
> >=20
> > The isar-bootstrap process has changed alot after I initially
> > implemented it (cross-build support, etc.), and I haven't worked on
> > that since.
> >=20
> > But the general idea when I implemented it was that just one repo
> > will be used for debootstrap (the first one listed), and afterwards
> > the other repos are added and an update+dist-upgrade is performed
> > on all of them. =20
>=20
> Thanks. The current implementation seems to be basically same as
> previous.
>=20
> >=20
> > After that step the upstream apt index should not be changed. No
> > global apt-get update, only apt-get update of local repos where
> > newly created packages are placed.
> >=20
> > That way the buildchroot and the resulting image should contain the
> > same versions of the packages which are installed in their
> > respective usages of setup_root_file_system function in a copy of
> > that initial bootstrapped system.
> >  =20
> > > =20
> > >>
> > >> One issue here is that Isar uses bitbake as the build task
> > >> scheduler, =20
> > so =20
> > >> shell scripts etc. are difficult to extract from it to be used
> > >> outside of Isar. And if they are extracted from the Isar
> > >> recipes, then using them in Isar again is disadvantageous for
> > >> the Isar build process.
> > >>
> > >> So sharing code between a bitbake/isar project and other build
> > >> systems is not really possible. =20
> > >
> > > I usually have the same issues in my bitbake recipes development.
> > > It might be possible to share the concrete processes to create
> > > a Debian base system from multiple repositories,
> > > even if it's difficult to share the actual codes.
> > >
> > > Background of my question:
> > > I guess that people who are developing Debian based system
> > > with their own custom packages (I'm one of them) may want to use
> > > a command (or a wrapper script) to create a base image from
> > > multiple apt =20
> > repos, =20
> > > especially if they need to replace a part of existing Debian
> > > essential =20
> > packages =20
> > > by their custom packages (coreutils, util-linux, dpkg, etc.).
> > > I'm investigating existing approaches that provide the API above
> > > or =20
> > similar one.
> >=20
> > We would currently deal with that by having the local repo that
> > contains our build packages added to the apt with a high priority.
> >=20
> > Installing or updating now would remove the upstream package and
> > install our own versions. Not the best way, but it works in
> > principle.
> >=20
> > I don't know how tested in Isar that currently is though. =20
>=20
> Thank you for the information. I also think that this is the simplest
> way and easy-to-understand if we don't need to install significantly
> customized "essential" packages installed by debootstrap. (e.g.
> minimal perl) I guess such kind of situations are not in the main
> focus of Isar and should not be supported from maintenance PoV.
>=20
>=20
> Regards,
> Kazu
>=20
>=20