Analysis of isar: Current work of efibootguard / swupdate integration

Analysis of isar: Current work of efibootguard / swupdate integration

[Andreas Reichel]

# isar - integration of efibootguard and swupdate #

## My analysis of the current status of isar ##

Consider the following situation:

efibootguard is compiled from sources, swupdate is compiled from sources, both are integrated into an isar image.

problem: there was no final/usable build pipeline for this, i.e. architecture of isar still under construction/discussion,
thus I introduced the following pipeline-steps to be independent of the current work:

* do_clean
* do_install_build_deps
* do_configure
* do_compile
* do_build
* do_deploy_deb
* do_package_deb
* do_temp_deploy
* do_deploy_wic_files

in the following order:

addtask do_install_build_deps before do_configure after do_unpack
addtask do_compile before do_build
addtask do_configure before do_compile after do_unpack
addtask do_fetch before do_unpack
addtask do_temp_deploy before do_build after do_compile
addtask do_deploy_deb after do_temp_deploy before do_package_deb
addtask do_package_deb after do_deploy_deb before do_build
addtask do_deploy_wic_files after do_package_deb after do_build

According to problems I encountered, I will explain workaround-fixes I applied
to integrate efibootguard with swupdate using my special build pipeline steps:

* do_clean: It is advantegous that the tmp folder doesn't need to be cleaned
            everytime to save time during development.  do_clean just removes
            all the stamps in the stamp directory that bitbake creates and the
            content of the working directory. do_clean itself has a [nostamp]
            flag set so that it is always executed.

* do_install_build_deps: Problem was, that swupdate and efibootguard needed
            additional packages in the buildchroot, that was already finished.
            Thus, this task bind-mounts `/proc`, `/sys`, `/dev`, `/dev/pts` and
            takes care of proxy settings so that apt works inside buildchroot
            with chroot. This way, additional debian packages could be
            installed on requirement basis.

            ```
            apt-get update > ${PP}/isar_install_builddeps.log
            echo "Yes, do as I say!" | apt-get install --force-yes ${CUSTOM_BUILD_DEPS} > ${PP}/isar_install_builddeps.log
            ```

            Some deps required the ugly "Yes, do as I say!" override because
            apt refuses to alter the init system (which is not needed in
            buildchroot anyway.)

* do_configure: changes into ${WORKDIR}/git and configures the software
            according to its needs and copies a .config if needed.

* do_compile: changes into ${WORKDIR}/git and compiles by calling make

* do_build: EMPTY - since it was not clear what this step should do, however,
            this is the default bitbake target and I wanted software to build
            when I type 'bitbake efibootguard'. This should also be regarded
            like an 'all' make target. So every build specific task is executed
            before do_build.

* do_deploy_deb: copies files that are going to be in a binary debian package

* do_package_deb: builds the actual .deb package

* do_temp_deploy: This is an important part for cross-package dependencies...
             for example swupdate requires a library of efibootguard. This
             means, efibootguard must have executed this task before swupdate
             starts do_install_build_deps.

             in swupdate.bb:

             ```
             DEPENDS += "efibootguard"
             do_install_build_deps[deptask] = "do_temp_deploy"
             ```

             Now this task will put the needed library files into a temporary
             location, where swupdate can pick them up later.

* do_install and do_package both only return true

The following problem is unsolved:

* When a .deb package is created from swupdate, dependencies are set.
  do_populate_sysroot then fails, because dpkg cannot download aything.


Next problem is about communication between bitbake and wic. For efibootguard,
a special partition layout must be used and a .wks file is created for this.
However, the boot loader configuration is written by a wic source plugin,
because image specific options should be set in the .wks file.

wic:

* commit dd109de433577 in isar - "wic: Remove sysroot support" breaks wic such that:
        * command line option to specify location of native tools is not
          accepted anymore, although it is still mentioned in the help text of
          wic,
        * all native tools become host dependencies
        * wic has no possibility anymore to install native tools, since bitbake
          cannot or should not change host software (usually, if a tool
          is not found, bitbake tries to execute a recipe according to a
          hard coded map, which fails)
        * different wic source plugins may require different native tools and
          thus for different wic images different build environments must exist
          due to host dependencies.

Fact: there is no native-sysroot within isar build system, only a buildchroot
      that has the same architecture as the target platform.

Consider the following situation:

wic is used to create the bootable image.  The .wks file specifies the
parameters for the bootloader.  These parameters are used to write the
environment files for the config partitions (efibootguard).  These environment
files are generated by a wic plugin, that calls `bg_setenv`.

The problem here is, that `bg_setenv` must be a native utility. However, the
build-chroot cannot be used to install native utilities into the outer
environment. A special native root doesn't exist either. Thus, wic can never
call custom tools without emulation like binfmt.

I have patched wic so that it looks for binaries in /sbin and /bin of
the built root file system and the host.

*Note*: I regard my current implementation as a meta-step towards real
integration and will rebase my commits if the pipeline in isar can handle

* clearly defined names for the build pipeline tasks
* Cross dependencies between .deb packages that are compiled from sources, i.e.
  one source needs libs from the other source tools needed by wic source
  plugins (where are they, host / buildchroot)
* wic native_sysroot according to the point above
* build dependencies installable by every recipe according to its needs

Current state of my work is in:

https://github.com/andi8086/isar
branch `andreas/isar-patchset`

Kind regards
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: Analysis of isar: Current work of efibootguard / swupdate integration

[Alexander Smirnov]

Hi Andreas,

On 09/21/2017 05:18 PM, Andreas Reichel wrote:
> # isar - integration of efibootguard and swupdate #
> 
> ## My analysis of the current status of isar ##
> 
> Consider the following situation:
> 
> efibootguard is compiled from sources, swupdate is compiled from sources, both are integrated into an isar image.
> 
> problem: there was no final/usable build pipeline for this, i.e. architecture of isar still under construction/discussion,
> thus I introduced the following pipeline-steps to be independent of the current work:
> 

I'd like to split Isar pipeline into the parts:
  - image creation
  - dpkg-buildpackage package creation
  - dpkg-deb package creation
  - filesystem packaging

These sets of tasks are currently used to build Isar pipeline.

Please correct me if I'm wrong, but what I see above:

  - You have several packages, each has its own recipe.
  - You need to compile this packages from source code.
  - The expected outcome is a set of binary *.deb packages.
  - This packages should be installed to target root filesystem.

I'm asking, because there is already hello application, that is built 
from sources, so you only need to create 'debian/*' files in your source 
code. This will be true way to integrate custom package to Isar, that is 
built from sources using auto-tools :-)

> * do_clean
> * do_install_build_deps
> * do_configure
> * do_compile
> * do_build
> * do_deploy_deb
> * do_package_deb
> * do_temp_deploy
> * do_deploy_wic_files
> 
> in the following order:
> 
> addtask do_install_build_deps before do_configure after do_unpack
> addtask do_compile before do_build
> addtask do_configure before do_compile after do_unpack
> addtask do_fetch before do_unpack
> addtask do_temp_deploy before do_build after do_compile
> addtask do_deploy_deb after do_temp_deploy before do_package_deb
> addtask do_package_deb after do_deploy_deb before do_build
> addtask do_deploy_wic_files after do_package_deb after do_build
> 
> According to problems I encountered, I will explain workaround-fixes I applied
> to integrate efibootguard with swupdate using my special build pipeline steps:
> 
> * do_clean: It is advantegous that the tmp folder doesn't need to be cleaned
>              everytime to save time during development.  do_clean just removes
>              all the stamps in the stamp directory that bitbake creates and the
>              content of the working directory. do_clean itself has a [nostamp]
>              flag set so that it is always executed.
> 
> * do_install_build_deps: Problem was, that swupdate and efibootguard needed
>              additional packages in the buildchroot, that was already finished.
>              Thus, this task bind-mounts `/proc`, `/sys`, `/dev`, `/dev/pts` and
>              takes care of proxy settings so that apt works inside buildchroot
>              with chroot. This way, additional debian packages could be
>              installed on requirement basis.
> 
>              ```
>              apt-get update > ${PP}/isar_install_builddeps.log
>              echo "Yes, do as I say!" | apt-get install --force-yes ${CUSTOM_BUILD_DEPS} > ${PP}/isar_install_builddeps.log
>              ```
> 
>              Some deps required the ugly "Yes, do as I say!" override because
>              apt refuses to alter the init system (which is not needed in
>              buildchroot anyway.)
> 
> * do_configure: changes into ${WORKDIR}/git and configures the software
>              according to its needs and copies a .config if needed.
> 
> * do_compile: changes into ${WORKDIR}/git and compiles by calling make
> 
> * do_build: EMPTY - since it was not clear what this step should do, however,
>              this is the default bitbake target and I wanted software to build
>              when I type 'bitbake efibootguard'. This should also be regarded
>              like an 'all' make target. So every build specific task is executed
>              before do_build.
> 
> * do_deploy_deb: copies files that are going to be in a binary debian package
> 
> * do_package_deb: builds the actual .deb package
> 
> * do_temp_deploy: This is an important part for cross-package dependencies...
>               for example swupdate requires a library of efibootguard. This
>               means, efibootguard must have executed this task before swupdate
>               starts do_install_build_deps.
> 
>               in swupdate.bb:
> 
>               ```
>               DEPENDS += "efibootguard"
>               do_install_build_deps[deptask] = "do_temp_deploy"
>               ```
> 
>               Now this task will put the needed library files into a temporary
>               location, where swupdate can pick them up later.
> 
> * do_install and do_package both only return true
> 
> The following problem is unsolved:
> 
> * When a .deb package is created from swupdate, dependencies are set.
>    do_populate_sysroot then fails, because dpkg cannot download aything.
> 
> 
> Next problem is about communication between bitbake and wic. For efibootguard,
> a special partition layout must be used and a .wks file is created for this.
> However, the boot loader configuration is written by a wic source plugin,
> because image specific options should be set in the .wks file.
> 
> wic:
> 
> * commit dd109de433577 in isar - "wic: Remove sysroot support" breaks wic such that:
>          * command line option to specify location of native tools is not
>            accepted anymore, although it is still mentioned in the help text of
>            wic,
>          * all native tools become host dependencies
>          * wic has no possibility anymore to install native tools, since bitbake
>            cannot or should not change host software (usually, if a tool
>            is not found, bitbake tries to execute a recipe according to a
>            hard coded map, which fails)
>          * different wic source plugins may require different native tools and
>            thus for different wic images different build environments must exist
>            due to host dependencies.
> 
> Fact: there is no native-sysroot within isar build system, only a buildchroot
>        that has the same architecture as the target platform.
> 
> Consider the following situation:
> 
> wic is used to create the bootable image.  The .wks file specifies the
> parameters for the bootloader.  These parameters are used to write the
> environment files for the config partitions (efibootguard).  These environment
> files are generated by a wic plugin, that calls `bg_setenv`.
> 

Is it generic tools that could be installed to host via apt-get?

> The problem here is, that `bg_setenv` must be a native utility. However, the
> build-chroot cannot be used to install native utilities into the outer
> environment. A special native root doesn't exist either. Thus, wic can never
> call custom tools without emulation like binfmt.
> 
> I have patched wic so that it looks for binaries in /sbin and /bin of
> the built root file system and the host.
> 
> *Note*: I regard my current implementation as a meta-step towards real
> integration and will rebase my commits if the pipeline in isar can handle
> 
> * clearly defined names for the build pipeline tasks
> * Cross dependencies between .deb packages that are compiled from sources, i.e.
>    one source needs libs from the other source tools needed by wic source
>    plugins (where are they, host / buildchroot)
> * wic native_sysroot according to the point above
> * build dependencies installable by every recipe according to its needs
> 
> Current state of my work is in:
> 
> https://github.com/andi8086/isar
> branch `andreas/isar-patchset`
> 
> Kind regards
> Andreas
> 
With best regards,
Alex