[PATCH 0/1] Documentation update for debootstrapisation

[PATCH 0/1] Documentation update for debootstrapisation

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

this should update the documentation to contain the information about
isar-bootstrap. All references to multistrap should now be removed from
the documentation.

It also contains some documentation about `DISTRO_KERNELS`. I am not
sure if everything of the documentation is now up to date though.

Cheers,
Claudius

Claudius Heine (1):
  doc: updated doc to contain the changes by the isar-bootstrap patches

 doc/technical_overview.md | 93 ++++++++++++++++++++++++++++++++---------------
 doc/user_manual.md        | 24 ++++++------
 2 files changed, 74 insertions(+), 43 deletions(-)

-- 
2.16.3

[PATCH 1/1] doc: updated doc to contain the changes by the isar-bootstrap patches

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 doc/technical_overview.md | 93 ++++++++++++++++++++++++++++++++---------------
 doc/user_manual.md        | 24 ++++++------
 2 files changed, 74 insertions(+), 43 deletions(-)

diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index bdb758e..83624a5 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -34,8 +34,7 @@ There are two types of dependencies in Isar:
 
 **NOTE:** Isar doesn't manage dependencies in Debian file systems. User can
 only list specific Debian dependencies in recipe, so they eventually will be
-passed to apt-get or multistrap. Dependency installation is managed by Debian
-tools.
+passed to apt-get. Dependency installation is managed by Debian tools.
 
 ## 2.2 Stamps
 
@@ -68,16 +67,14 @@ One of the key aspect of Debian philosophy claims the fact, that everything in
 Debian should be built within Debian environment. Moreover native compilation
 is more preferable than cross-compilation. To follow this rules, Isar
 introduces the new component - buildchroot. Bulidchroot is typical Debian
-filesystem that is created using standard Debian tools: multistrap, apt. The
+filesystem that is created using standard Debian tools: debootstrap, apt. The
 source of packages can be either official Debian repositories or custom
 repositories created by user.
 
 Buildchroot lifecycle can be described as following:
 
- - Buildchroot has initial configuration file which is passed to multistrap
-   tool. This configuration file is generated by bitbake recipe from patterns
-   and values defined by user. Based on this configuration file, multistrap
-   generates initial filesystem.
+ - Buildchroot starts out with a minimal base system that was generated by the
+   isar-bootstrap recipe that uses debootstrap and apt.
 
  - During building custom Debian package, list of its build dependencies is
    installed to buildchroot.
@@ -92,10 +89,8 @@ it doesn't have development packages installed.
 
 Target filesystem lifecycle can be described as following:
 
- - Target filesystem has initial configuration file which is passed to
-   multistrap tool. This configuration file is generated by bitbake recipe from
-   patterns and values defined by user. Based on this configuration file,
-   multistrap generates initial filesystem.
+ - Target filesystem starts, like the buildchroot, with the same minimal base
+   system generated by the isar-bootstrap recipe.
 
  - According to the list of custom packages in bitbake recipes, the initial
    filesystem will be populated by successfully built packages.
@@ -106,8 +101,11 @@ Target filesystem lifecycle can be described as following:
 
 Whole Isar build process can be split into the following steps:
 
- - Generation of initial buildchroots for each configuration (Debian distro,
-   machine and architecture) requested by user.
+ - Generation of a minimal base system by isar-bootstrap using debootstrap and
+   apt for each configuration requested by the user (Debian distro, machine and
+   architecture).
+
+ - Generation of initial buildchroots for each configuration.
 
  - Generation of initial target filesystems for each configuration.
 
@@ -119,29 +117,64 @@ Whole Isar build process can be split into the following steps:
 
 All these steps are described in details below.
 
-## 3.2 Initial Buildchroot Generation
+## 3.2 Minimal base system
 
-As mentioned above, initial buildchroot is generated using multistrap. The
-bitbake recipe which is responsible for buildchroot can be found here:
-`meta/recipes-devtools/buildchroot/buildchroot.bb`
+A minimal base system is generated by the isar-bootstrap recipe found here:
+`meta/recipes-core/isar-bootstrap/isar-bootstrap.bb`.
 
-This recipe implementes `do_build` task which performs the following:
+This recipe implements several tasks which are executed in this order:
+
+1. `do_generate_keyring`: Generates a new gpg keyring from the gpg keys
+   specified in the `APTKEYFILES` variable. This keyring is later used by
+   debootstrap and apt to verify the apt repositories.
+
+2. `do_apt_config_prepare`: Generates aggregated apt preferences and source
+   files based on the `DISTRO_APT_PREFERENCES` and `DISTRO_APT_SOURCES`
+   variables. That are later deployed to the minimal base system to be used by
+   `apt` itself.
+
+3. `do_bootstrap`: Bootstraps a minimal debian base system using debootstrap.
+
+4. `do_apt_config_install`: Installs the apt configuration generated by
+   `do_apt_config_prepare` to the base system as well as an apt base
+   configuration that ensures no recommended or suggested package are installed
+   automatically. This setting keeps the system minimal for all further steps.
 
-1. Generate multistrap config from template:
-   `meta/recipes-devtools/buildchroot/files/multistrap.conf.in`.
+5. `do_apt_update`: Performs an initial `apt-get update` and
+   `apt-get dist-upgrade` on the minimal system. This is necessary because
+   debootstrap itself only allows using one repository. This task ensures that
+   more up to date packages, for instance from security repos are installed if
+   they exist.
 
-2. Install pre/post scripts for multistrap:
-   `meta/recipes-devtools/buildchroot/files/configscript.sh` and
-   `meta/recipes-devtools/buildchroot/files/setup.sh`.
+6. `do_deploy`: Creates a symlink to the directory containing the generated
+   minimal base system, that can be used by the other recipes, that depend on
+   isar-bootstrap.
 
-3. Run multistrap.
+## 3.3 Initial Buildchroot Generation
 
-4. Install script for building custom Debian packages:
+As mentioned above, initial buildchroot is generated using the minimal base
+system generated by isar-bootstrap. The bitbake recipe which is responsible for
+buildchroot can be found here:
+`meta/recipes-devtools/buildchroot/buildchroot.bb`
+
+This recipe implementes `do_build` task which performs the following:
+
+1. Calls the `setup_root_file_system` helper function defined in
+   `meta/classes/isar-bootstrap-helper.bbclass` that copies the minimal base
+   system to the buildchroot working directory, sets up the isar-apt repository
+   which will later contain all isar-build packages, calls update on just this
+   repository and installs all packages given via a parameter to this system.
+   Those are, in buildchroots case, the basic package building toolchain.
+
+2. Install script for building custom Debian packages:
    `meta/recipes-devtools/buildchroot/files/build.sh`.
 
+3. Installs and executes a configuration script, that further customizes the
+   buildchroot. For instance it sets the locale settings.
+
 The single stamp is created for each user buildchroot configuration.
 
-## 3.3 Initial Target Filesystem Generation
+## 3.4 Initial Target Filesystem Generation
 
 Initial target filesystem generation process is very similar to buildchroot
 creating, the difference is only in initial packages list.
@@ -150,10 +183,10 @@ Target image recipes are the part of Isar core. There is a sample of typical
 Isar image that can be customized according to the user requirements:
 `meta-isar/recipes-core/images/isar-image-base.bb`.
 
-Like for buildchroot, the multistrap configuration files for image can be found
+Like for buildchroot, the configuration files for image can be found
 here: `meta-isar/recipes-core/images/files`, and it implements `do_build` task.
 
-## 3.4 Building Custom Packages
+## 3.5 Building Custom Packages
 
 Isar provides the possibility to build Debian packages from sources. This
 feature works with Debian-like source packages, i.e. the source code tree
@@ -193,7 +226,7 @@ Both consist of the following steps:
 5. Task `do_deploy_deb`: install successfully built packages
    `${WORKDIR}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}`
 
-## 3.5 Populate Target Filesystem
+## 3.6 Populate Target Filesystem
 
 Each target image can be extended by custom packages listed in IMAGE_INSTALL
 variable. Task `do_populate` performs the following:
@@ -206,7 +239,7 @@ variable. Task `do_populate` performs the following:
 
 4. Execute dpkg command in chroot for all the copied packages.
 
-## 3.6 Generate Bootable Image
+## 3.7 Generate Bootable Image
 
 This process contains the following steps:
 
diff --git a/doc/user_manual.md b/doc/user_manual.md
index 0771d1e..058f7bd 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -58,7 +58,7 @@ git
 grub-efi-amd64-bin          # wic UEFI: /usr/lib/grub/x86_64-efi/moddep.lst
 grub-efi-ia32-bin           # wic UEFI: /usr/lib/grub/i386-efi/moddep.lst
 mtools                      # wic FAT: mcopy
-multistrap
+debootstrap
 parted
 python
 python3                     # wic
@@ -79,7 +79,7 @@ Notes:
 
 ### Setup Sudo
 
-Isar requires `sudo` rights without password to work with `chroot` and `multistrap`. To add them, use the following steps:
+Isar requires `sudo` rights without password to work with `chroot` and `debootstrap`. To add them, use the following steps:
 ```
  # visudo
 ```
@@ -366,17 +366,17 @@ Isar can generate various images types for specific machine. The `IMAGE_TYPE` va
 
 The distro is defined by the set of the following variables:
 
- - `DISTRO_SUITE` - Repository suite like stable, jessie, wheezy etc.
- - `DISTRO_COMPONENTS` - Repository components like main, contrib, non-free etc.
- - `DISTRO_APT_SOURCE` - Repository URL.
- - `DISTRO_CONFIG_SCRIPT` - Target filesystem finalization script. This script is called after `multistrap` has unpacked the base system packages. It is designed to finalize filesystem, for example to add `fstab` according to machine hardware configuration. The script should be placed to `files` folder in image recipe folder.
+ - `DISTRO_APT_SOURCES` - List of apt source files
+ - `DISTRO_APT_KEYS` - List of gpg key URIs used to verify apt repos
+ - `DISTRO_APT_PREFERENCES` - List of apt preference files
+ - `DISTRO_KERNELS` - List of supported kernel suffixes
 
 Below is an example for Raspbian Jessie:
 ```
-DISTRO_SUITE = "jessie"
-DISTRO_COMPONENTS = "main contrib non-free firmware"
-DISTRO_APT_SOURCE = "http://archive.raspbian.org/raspbian"
-DISTRO_CONFIG_SCRIPT = "raspbian-configscript.sh"
+DISTRO_APT_SOURCES += "conf/distro/raspbian-jessie.list"
+DISTRO_APT_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
+DISTRO_CONFIG_SCRIPT?= "raspbian-configscript.sh"
+DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
 ```
 
 To add new distro, user should perform the following steps:
@@ -439,9 +439,7 @@ To add new machine user should perform the following steps:
 Image in Isar contains the following artifacts:
 
  - Image recipe - Describes set of rules how to generate target image.
- - `Multistrap` configuration file - Contains information about distro, suite, `apt` source etc.
- - `Multistrap` setup script - Performs pre-install filesystem configuration.
- - `Multistrap` config script - Performs post-install filesystem configuration.
+ - Config script - Performs some general base system configuration after all packages were installed. (locale, fstab, cleanup, etc.)
 
 In image recipe, the following variable defines the list of packages that will be included to target image: `IMAGE_PREINSTALL`. These packages will be taken from `apt` source.
 
-- 
2.16.3

Re: [PATCH 0/1] Documentation update for debootstrapisation

[Baurzhan Ismagulov]

On Thu, Apr 05, 2018 at 12:48:23PM +0200, claudius.heine.ext@siemens.com wrote:
> this should update the documentation to contain the information about
> isar-bootstrap. All references to multistrap should now be removed from
> the documentation.
> 
> It also contains some documentation about `DISTRO_KERNELS`. I am not
> sure if everything of the documentation is now up to date though.

Thanks, applied to next.

With kind regards,
Baurzhan.