From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6468235886978924544 X-Received: by 10.223.166.39 with SMTP id k36mr650845wrc.19.1506003524114; Thu, 21 Sep 2017 07:18:44 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.195.204 with SMTP id d12ls770172wrg.14.gmail; Thu, 21 Sep 2017 07:18:43 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDB51qbH4zUX7y1c3of/fkhkvBpAPEs7SHxZw9pe2ZEzSdiI7b1+etTxBbDtfDy1ybxkIx3 X-Received: by 10.28.230.79 with SMTP id d76mr133451wmh.12.1506003523757; Thu, 21 Sep 2017 07:18:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506003523; cv=none; d=google.com; s=arc-20160816; b=edgnsNy4xWcbbDuDQ+mZ9c0B9J9ZSbd+WLGp8L8ARzITVRLdNReQ0X2yFLapNE16G7 pDy1NaGmdNTxwu3LGuTBCTkWI4R10Qqq1s1OQafHZqpWhWcb1cdnkvfNPnHJonlv/aDS rNqrgTm2SzrcvhsjthkSnAkztgourY1fK2LK8+Ae8ZI92V7Yix/S0VjLCsip2/kTqxLm 5Fb5QPYfRgmA4QEwhrpFsPZFqTxymMSXAjiipE82x4+cyV4UJGZGkQu0e5yjmntfMC/j 8+RrKe/IWJdSZ9bT86FOwrPppBsVW1NBbPEoFTGep/i67GBuxKPOFztgoK1Zg26GnLol y6eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:content-disposition:content-description:mime-version :message-id:subject:to:from:date:arc-authentication-results; bh=ThsU2O+OqOAK1Q3IMpSB+jiXboiGyf6BlHzqb1wmfUo=; b=RUEhIvbzn4Jj12izFiq9yc5uoMfHhNQPFhgm6DFkzkdo/EVVCxH6bBwzw0fXXEWYOh 9si9I45yl5byLCn+8x/rE3yP/AcosOm+ywMwqhoC1S8y4OQ3RSp/p5tQWpDVW3iAo6xj rVd63NTBILSjIeDSezP5Pr/pjU++LzCSTfMatPHTj/FscTNNNQ7RKvGgXb0qd7CmBL69 jo3cCbTKURMT9DqnM1+r7RiigIevUIedLLmtgEQG36TYVenMhTIYgyJQHc7csNAO+cV7 FAdSpw+oe0d4EutxxVegX2JY3UQ2utTcnpK6qCWcmiW8WuegCJlt/oMRnTkF48o7PTNq VkLw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of andreas.reichel.ext@siemens.com) smtp.mailfrom=andreas.reichel.ext@siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id q145si427045wme.4.2017.09.21.07.18.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Sep 2017 07:18:43 -0700 (PDT) Received-SPF: neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of andreas.reichel.ext@siemens.com) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of andreas.reichel.ext@siemens.com) smtp.mailfrom=andreas.reichel.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id v8LEIh1e020955 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 21 Sep 2017 16:18:43 +0200 Received: from iiotirae (golem.ppmd.siemens.net [139.25.69.162]) by mail2.sbs.de (8.15.2/8.15.2) with SMTP id v8LEIh35027087; Thu, 21 Sep 2017 16:18:43 +0200 Date: Thu, 21 Sep 2017 16:18:01 +0200 From: Andreas Reichel To: isar-users@googlegroups.com Subject: Analysis of isar: Current work of efibootguard / swupdate integration Message-ID: <20170921141759.GA29477@iiotirae> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Description: message Content-Disposition: inline User-Agent: Mutt/1.8.3 (2017-05-23) X-TUID: Gp81CRbCruEM # isar - integration of efibootguard and swupdate # ## My analysis of the current status of isar ## Consider the following situation: efibootguard is compiled from sources, swupdate is compiled from sources, both are integrated into an isar image. problem: there was no final/usable build pipeline for this, i.e. architecture of isar still under construction/discussion, thus I introduced the following pipeline-steps to be independent of the current work: * do_clean * do_install_build_deps * do_configure * do_compile * do_build * do_deploy_deb * do_package_deb * do_temp_deploy * do_deploy_wic_files in the following order: addtask do_install_build_deps before do_configure after do_unpack addtask do_compile before do_build addtask do_configure before do_compile after do_unpack addtask do_fetch before do_unpack addtask do_temp_deploy before do_build after do_compile addtask do_deploy_deb after do_temp_deploy before do_package_deb addtask do_package_deb after do_deploy_deb before do_build addtask do_deploy_wic_files after do_package_deb after do_build According to problems I encountered, I will explain workaround-fixes I applied to integrate efibootguard with swupdate using my special build pipeline steps: * do_clean: It is advantegous that the tmp folder doesn't need to be cleaned everytime to save time during development. do_clean just removes all the stamps in the stamp directory that bitbake creates and the content of the working directory. do_clean itself has a [nostamp] flag set so that it is always executed. * do_install_build_deps: Problem was, that swupdate and efibootguard needed additional packages in the buildchroot, that was already finished. Thus, this task bind-mounts `/proc`, `/sys`, `/dev`, `/dev/pts` and takes care of proxy settings so that apt works inside buildchroot with chroot. This way, additional debian packages could be installed on requirement basis. ``` apt-get update > ${PP}/isar_install_builddeps.log echo "Yes, do as I say!" | apt-get install --force-yes ${CUSTOM_BUILD_DEPS} > ${PP}/isar_install_builddeps.log ``` Some deps required the ugly "Yes, do as I say!" override because apt refuses to alter the init system (which is not needed in buildchroot anyway.) * do_configure: changes into ${WORKDIR}/git and configures the software according to its needs and copies a .config if needed. * do_compile: changes into ${WORKDIR}/git and compiles by calling make * do_build: EMPTY - since it was not clear what this step should do, however, this is the default bitbake target and I wanted software to build when I type 'bitbake efibootguard'. This should also be regarded like an 'all' make target. So every build specific task is executed before do_build. * do_deploy_deb: copies files that are going to be in a binary debian package * do_package_deb: builds the actual .deb package * do_temp_deploy: This is an important part for cross-package dependencies... for example swupdate requires a library of efibootguard. This means, efibootguard must have executed this task before swupdate starts do_install_build_deps. in swupdate.bb: ``` DEPENDS += "efibootguard" do_install_build_deps[deptask] = "do_temp_deploy" ``` Now this task will put the needed library files into a temporary location, where swupdate can pick them up later. * do_install and do_package both only return true The following problem is unsolved: * When a .deb package is created from swupdate, dependencies are set. do_populate_sysroot then fails, because dpkg cannot download aything. Next problem is about communication between bitbake and wic. For efibootguard, a special partition layout must be used and a .wks file is created for this. However, the boot loader configuration is written by a wic source plugin, because image specific options should be set in the .wks file. wic: * commit dd109de433577 in isar - "wic: Remove sysroot support" breaks wic such that: * command line option to specify location of native tools is not accepted anymore, although it is still mentioned in the help text of wic, * all native tools become host dependencies * wic has no possibility anymore to install native tools, since bitbake cannot or should not change host software (usually, if a tool is not found, bitbake tries to execute a recipe according to a hard coded map, which fails) * different wic source plugins may require different native tools and thus for different wic images different build environments must exist due to host dependencies. Fact: there is no native-sysroot within isar build system, only a buildchroot that has the same architecture as the target platform. Consider the following situation: wic is used to create the bootable image. The .wks file specifies the parameters for the bootloader. These parameters are used to write the environment files for the config partitions (efibootguard). These environment files are generated by a wic plugin, that calls `bg_setenv`. The problem here is, that `bg_setenv` must be a native utility. However, the build-chroot cannot be used to install native utilities into the outer environment. A special native root doesn't exist either. Thus, wic can never call custom tools without emulation like binfmt. I have patched wic so that it looks for binaries in /sbin and /bin of the built root file system and the host. *Note*: I regard my current implementation as a meta-step towards real integration and will rebase my commits if the pipeline in isar can handle * clearly defined names for the build pipeline tasks * Cross dependencies between .deb packages that are compiled from sources, i.e. one source needs libs from the other source tools needed by wic source plugins (where are they, host / buildchroot) * wic native_sysroot according to the point above * build dependencies installable by every recipe according to its needs Current state of my work is in: https://github.com/andi8086/isar branch `andreas/isar-patchset` Kind regards Andreas -- Andreas Reichel Dipl.-Phys. (Univ.) Software Consultant Andreas.Reichel@tngtech.com, +49-174-3180074 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082