Re: Analysis of isar: Current work of efibootguard / swupdate integration

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi Andreas,

On 09/21/2017 05:18 PM, Andreas Reichel wrote:
> # isar - integration of efibootguard and swupdate #
> 
> ## My analysis of the current status of isar ##
> 
> Consider the following situation:
> 
> efibootguard is compiled from sources, swupdate is compiled from sources, both are integrated into an isar image.
> 
> problem: there was no final/usable build pipeline for this, i.e. architecture of isar still under construction/discussion,
> thus I introduced the following pipeline-steps to be independent of the current work:
> 

I'd like to split Isar pipeline into the parts:
  - image creation
  - dpkg-buildpackage package creation
  - dpkg-deb package creation
  - filesystem packaging

These sets of tasks are currently used to build Isar pipeline.

Please correct me if I'm wrong, but what I see above:

  - You have several packages, each has its own recipe.
  - You need to compile this packages from source code.
  - The expected outcome is a set of binary *.deb packages.
  - This packages should be installed to target root filesystem.

I'm asking, because there is already hello application, that is built 
from sources, so you only need to create 'debian/*' files in your source 
code. This will be true way to integrate custom package to Isar, that is 
built from sources using auto-tools :-)

> * do_clean
> * do_install_build_deps
> * do_configure
> * do_compile
> * do_build
> * do_deploy_deb
> * do_package_deb
> * do_temp_deploy
> * do_deploy_wic_files
> 
> in the following order:
> 
> addtask do_install_build_deps before do_configure after do_unpack
> addtask do_compile before do_build
> addtask do_configure before do_compile after do_unpack
> addtask do_fetch before do_unpack
> addtask do_temp_deploy before do_build after do_compile
> addtask do_deploy_deb after do_temp_deploy before do_package_deb
> addtask do_package_deb after do_deploy_deb before do_build
> addtask do_deploy_wic_files after do_package_deb after do_build
> 
> According to problems I encountered, I will explain workaround-fixes I applied
> to integrate efibootguard with swupdate using my special build pipeline steps:
> 
> * do_clean: It is advantegous that the tmp folder doesn't need to be cleaned
>              everytime to save time during development.  do_clean just removes
>              all the stamps in the stamp directory that bitbake creates and the
>              content of the working directory. do_clean itself has a [nostamp]
>              flag set so that it is always executed.
> 
> * do_install_build_deps: Problem was, that swupdate and efibootguard needed
>              additional packages in the buildchroot, that was already finished.
>              Thus, this task bind-mounts `/proc`, `/sys`, `/dev`, `/dev/pts` and
>              takes care of proxy settings so that apt works inside buildchroot
>              with chroot. This way, additional debian packages could be
>              installed on requirement basis.
> 
>              ```
>              apt-get update > ${PP}/isar_install_builddeps.log
>              echo "Yes, do as I say!" | apt-get install --force-yes ${CUSTOM_BUILD_DEPS} > ${PP}/isar_install_builddeps.log
>              ```
> 
>              Some deps required the ugly "Yes, do as I say!" override because
>              apt refuses to alter the init system (which is not needed in
>              buildchroot anyway.)
> 
> * do_configure: changes into ${WORKDIR}/git and configures the software
>              according to its needs and copies a .config if needed.
> 
> * do_compile: changes into ${WORKDIR}/git and compiles by calling make
> 
> * do_build: EMPTY - since it was not clear what this step should do, however,
>              this is the default bitbake target and I wanted software to build
>              when I type 'bitbake efibootguard'. This should also be regarded
>              like an 'all' make target. So every build specific task is executed
>              before do_build.
> 
> * do_deploy_deb: copies files that are going to be in a binary debian package
> 
> * do_package_deb: builds the actual .deb package
> 
> * do_temp_deploy: This is an important part for cross-package dependencies...
>               for example swupdate requires a library of efibootguard. This
>               means, efibootguard must have executed this task before swupdate
>               starts do_install_build_deps.
> 
>               in swupdate.bb:
> 
>               ```
>               DEPENDS += "efibootguard"
>               do_install_build_deps[deptask] = "do_temp_deploy"
>               ```
> 
>               Now this task will put the needed library files into a temporary
>               location, where swupdate can pick them up later.
> 
> * do_install and do_package both only return true
> 
> The following problem is unsolved:
> 
> * When a .deb package is created from swupdate, dependencies are set.
>    do_populate_sysroot then fails, because dpkg cannot download aything.
> 
> 
> Next problem is about communication between bitbake and wic. For efibootguard,
> a special partition layout must be used and a .wks file is created for this.
> However, the boot loader configuration is written by a wic source plugin,
> because image specific options should be set in the .wks file.
> 
> wic:
> 
> * commit dd109de433577 in isar - "wic: Remove sysroot support" breaks wic such that:
>          * command line option to specify location of native tools is not
>            accepted anymore, although it is still mentioned in the help text of
>            wic,
>          * all native tools become host dependencies
>          * wic has no possibility anymore to install native tools, since bitbake
>            cannot or should not change host software (usually, if a tool
>            is not found, bitbake tries to execute a recipe according to a
>            hard coded map, which fails)
>          * different wic source plugins may require different native tools and
>            thus for different wic images different build environments must exist
>            due to host dependencies.
> 
> Fact: there is no native-sysroot within isar build system, only a buildchroot
>        that has the same architecture as the target platform.
> 
> Consider the following situation:
> 
> wic is used to create the bootable image.  The .wks file specifies the
> parameters for the bootloader.  These parameters are used to write the
> environment files for the config partitions (efibootguard).  These environment
> files are generated by a wic plugin, that calls `bg_setenv`.
> 

Is it generic tools that could be installed to host via apt-get?

> The problem here is, that `bg_setenv` must be a native utility. However, the
> build-chroot cannot be used to install native utilities into the outer
> environment. A special native root doesn't exist either. Thus, wic can never
> call custom tools without emulation like binfmt.
> 
> I have patched wic so that it looks for binaries in /sbin and /bin of
> the built root file system and the host.
> 
> *Note*: I regard my current implementation as a meta-step towards real
> integration and will rebase my commits if the pipeline in isar can handle
> 
> * clearly defined names for the build pipeline tasks
> * Cross dependencies between .deb packages that are compiled from sources, i.e.
>    one source needs libs from the other source tools needed by wic source
>    plugins (where are they, host / buildchroot)
> * wic native_sysroot according to the point above
> * build dependencies installable by every recipe according to its needs
> 
> Current state of my work is in:
> 
> https://github.com/andi8086/isar
> branch `andreas/isar-patchset`
> 
> Kind regards
> Andreas
> 
With best regards,
Alex