From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6468235886978924544 X-Received: by 10.28.229.82 with SMTP id c79mr795718wmh.13.1508326777506; Wed, 18 Oct 2017 04:39:37 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.142.79 with SMTP id n73ls1162177wrb.3.gmail; Wed, 18 Oct 2017 04:39:37 -0700 (PDT) X-Google-Smtp-Source: ABhQp+Rw94T3Si7JSB7fmjd1enuPOevkHxkfKRscSJLw3vfQMFvA+DY4stfdAPwShOaOd/j2UR7m X-Received: by 10.28.150.130 with SMTP id y124mr808452wmd.22.1508326777187; Wed, 18 Oct 2017 04:39:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508326777; cv=none; d=google.com; s=arc-20160816; b=xQVCJrv4JYflvL2o8mgdQ7sZ9y+3vy/il984DqIFzkrSOzX0G0fZJQ3GLs6D6zJ3GB tlqEP8bKVN8Ly51Ctx1V49c9MyOF460hdloOYptaPdl+j+PsH7wwniM9KsoHf4t3oqIg Qo4iwaGTsRQmL2s91bs67AYMq6EgPFBCZljvIIj5nRmRdf04rf7loPWfgn7HrM8TtNvf 07+KqTyhzV1toVvh5zEpyzGrkMejJJoAAksK9lDSQE9bzVe643mjHwZ7n98lDToZFkBX BN+tdNiOffzQGJ2Tp+xs2baauaTRWLzNiqToEK+VzJ2yzAEAnA1Cgv5OwAphh5MWWdlN QlCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:cc:from:references:to:subject :arc-authentication-results; bh=oXa+5pR8OeNS4rBNfbgkckB5dUy2odAhIteKHEKafmE=; b=Lyzlgj4aKTfV/BtHZT9Pa8yjfhlRnGSQOFDcXXRTbSeMHjhmNFgfV21bT7JBRmoCFq aISj0xEIdLScNY2oYpgvATYZ8CXzUDtmkb8fe9wiLDF2hwRMCDnptj64bWusrKyp9y8q 7AtLROPYG3jmSO8U4vxjECNkho4dFrd2JARqpOBMNwwt8kWBZGKJZkt4+D4bVaRIvVWf BxgbPksEjvkiZrTfnpirH2ncB/qAesnDgoQuz1t5l9y7NSl30jEyOyyDzYGBc1fnxbeB CXLR6QCr7rED0GaYu6pCCvjF0RDbYSalJmHVIgSZgCZl9VJEPKAf4ZDspZZ/N2uXG1K4 QZsA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id r15si711956wrc.1.2017.10.18.04.39.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Oct 2017 04:39:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v9IBdYhF026638 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 18 Oct 2017 13:39:35 +0200 Subject: Re: Analysis of isar: Current work of efibootguard / swupdate integration To: Andreas Reichel References: <20170921141759.GA29477@iiotirae> From: Alexander Smirnov Cc: isar-users@googlegroups.com Message-ID: <334ce737-ed84-82a9-d425-e22763a3885a@ilbers.de> Date: Wed, 18 Oct 2017 14:39:29 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170921141759.GA29477@iiotirae> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: bZd6/WVaUTX+ Hi Andreas, On 09/21/2017 05:18 PM, Andreas Reichel wrote: > # isar - integration of efibootguard and swupdate # > > ## My analysis of the current status of isar ## > > Consider the following situation: > > efibootguard is compiled from sources, swupdate is compiled from sources, both are integrated into an isar image. > > problem: there was no final/usable build pipeline for this, i.e. architecture of isar still under construction/discussion, > thus I introduced the following pipeline-steps to be independent of the current work: > I'd like to split Isar pipeline into the parts: - image creation - dpkg-buildpackage package creation - dpkg-deb package creation - filesystem packaging These sets of tasks are currently used to build Isar pipeline. Please correct me if I'm wrong, but what I see above: - You have several packages, each has its own recipe. - You need to compile this packages from source code. - The expected outcome is a set of binary *.deb packages. - This packages should be installed to target root filesystem. I'm asking, because there is already hello application, that is built from sources, so you only need to create 'debian/*' files in your source code. This will be true way to integrate custom package to Isar, that is built from sources using auto-tools :-) > * do_clean > * do_install_build_deps > * do_configure > * do_compile > * do_build > * do_deploy_deb > * do_package_deb > * do_temp_deploy > * do_deploy_wic_files > > in the following order: > > addtask do_install_build_deps before do_configure after do_unpack > addtask do_compile before do_build > addtask do_configure before do_compile after do_unpack > addtask do_fetch before do_unpack > addtask do_temp_deploy before do_build after do_compile > addtask do_deploy_deb after do_temp_deploy before do_package_deb > addtask do_package_deb after do_deploy_deb before do_build > addtask do_deploy_wic_files after do_package_deb after do_build > > According to problems I encountered, I will explain workaround-fixes I applied > to integrate efibootguard with swupdate using my special build pipeline steps: > > * do_clean: It is advantegous that the tmp folder doesn't need to be cleaned > everytime to save time during development. do_clean just removes > all the stamps in the stamp directory that bitbake creates and the > content of the working directory. do_clean itself has a [nostamp] > flag set so that it is always executed. > > * do_install_build_deps: Problem was, that swupdate and efibootguard needed > additional packages in the buildchroot, that was already finished. > Thus, this task bind-mounts `/proc`, `/sys`, `/dev`, `/dev/pts` and > takes care of proxy settings so that apt works inside buildchroot > with chroot. This way, additional debian packages could be > installed on requirement basis. > > ``` > apt-get update > ${PP}/isar_install_builddeps.log > echo "Yes, do as I say!" | apt-get install --force-yes ${CUSTOM_BUILD_DEPS} > ${PP}/isar_install_builddeps.log > ``` > > Some deps required the ugly "Yes, do as I say!" override because > apt refuses to alter the init system (which is not needed in > buildchroot anyway.) > > * do_configure: changes into ${WORKDIR}/git and configures the software > according to its needs and copies a .config if needed. > > * do_compile: changes into ${WORKDIR}/git and compiles by calling make > > * do_build: EMPTY - since it was not clear what this step should do, however, > this is the default bitbake target and I wanted software to build > when I type 'bitbake efibootguard'. This should also be regarded > like an 'all' make target. So every build specific task is executed > before do_build. > > * do_deploy_deb: copies files that are going to be in a binary debian package > > * do_package_deb: builds the actual .deb package > > * do_temp_deploy: This is an important part for cross-package dependencies... > for example swupdate requires a library of efibootguard. This > means, efibootguard must have executed this task before swupdate > starts do_install_build_deps. > > in swupdate.bb: > > ``` > DEPENDS += "efibootguard" > do_install_build_deps[deptask] = "do_temp_deploy" > ``` > > Now this task will put the needed library files into a temporary > location, where swupdate can pick them up later. > > * do_install and do_package both only return true > > The following problem is unsolved: > > * When a .deb package is created from swupdate, dependencies are set. > do_populate_sysroot then fails, because dpkg cannot download aything. > > > Next problem is about communication between bitbake and wic. For efibootguard, > a special partition layout must be used and a .wks file is created for this. > However, the boot loader configuration is written by a wic source plugin, > because image specific options should be set in the .wks file. > > wic: > > * commit dd109de433577 in isar - "wic: Remove sysroot support" breaks wic such that: > * command line option to specify location of native tools is not > accepted anymore, although it is still mentioned in the help text of > wic, > * all native tools become host dependencies > * wic has no possibility anymore to install native tools, since bitbake > cannot or should not change host software (usually, if a tool > is not found, bitbake tries to execute a recipe according to a > hard coded map, which fails) > * different wic source plugins may require different native tools and > thus for different wic images different build environments must exist > due to host dependencies. > > Fact: there is no native-sysroot within isar build system, only a buildchroot > that has the same architecture as the target platform. > > Consider the following situation: > > wic is used to create the bootable image. The .wks file specifies the > parameters for the bootloader. These parameters are used to write the > environment files for the config partitions (efibootguard). These environment > files are generated by a wic plugin, that calls `bg_setenv`. > Is it generic tools that could be installed to host via apt-get? > The problem here is, that `bg_setenv` must be a native utility. However, the > build-chroot cannot be used to install native utilities into the outer > environment. A special native root doesn't exist either. Thus, wic can never > call custom tools without emulation like binfmt. > > I have patched wic so that it looks for binaries in /sbin and /bin of > the built root file system and the host. > > *Note*: I regard my current implementation as a meta-step towards real > integration and will rebase my commits if the pipeline in isar can handle > > * clearly defined names for the build pipeline tasks > * Cross dependencies between .deb packages that are compiled from sources, i.e. > one source needs libs from the other source tools needed by wic source > plugins (where are they, host / buildchroot) > * wic native_sysroot according to the point above > * build dependencies installable by every recipe according to its needs > > Current state of my work is in: > > https://github.com/andi8086/isar > branch `andreas/isar-patchset` > > Kind regards > Andreas > With best regards, Alex