From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6592507174472122368 X-Received: by 2002:a2e:7819:: with SMTP id t25-v6mr679415ljc.35.1535703045076; Fri, 31 Aug 2018 01:10:45 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:1656:: with SMTP id m83-v6ls629236lfi.25.gmail; Fri, 31 Aug 2018 01:10:44 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbNqHUehcUWpkb3gQW56tSj7E3BFvCyj/uGsG2htGbTu3mZd9g2//hB5R2v6oxlAIuWnIyt X-Received: by 2002:a19:d8ea:: with SMTP id r103-v6mr262222lfi.11.1535703044437; Fri, 31 Aug 2018 01:10:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535703044; cv=none; d=google.com; s=arc-20160816; b=u867ExbzvkHYS59+kH1LGOkVyi70sSEOt8cEp5GMEofuaWmXGW/uaJPLV0KP4cUg7t ffU2H35eMWlFtHGQKCy14Xz8r4DqJ+2I4czngz6qYNMbNLkBSakJdAqn53HXyrGmalyT GzvZvdTofLHaD/hehT7OqwnytPLYOYYzeFx95a84FLhJTxY9JLJ6BIhEu74O6Qrxcibw y/1ATKZWP78O1a+VrElz54bIrMki17SpHf2d/I6t7sgaGmzcD4qDxyn4kH7708iJuZYE CwaORBLNgJDQJnDvAhMGdfeU+x7UiKrYj4l28c6qS3fhSwOmrtwMidoBP35QNSXIQKx4 4xVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=7+tf8n7L7/iD2XQgTdKdKuBOAi3oCv8ZFWGv99pDFKk=; b=i6na6ZHUzaRs0riK9eTB9tQGZ04ERuGQwfeIqxMrxlOJxoXn8aCqGCvNlB4QUUBbVi mjEhoQLJgGumxb5b7NUvLxA21s9MTy8FZA6HGCrXgOfHZeYjPGaoebe2Cx36XVuSI7yy h9p83+wcpQ+j+9sRW74aaMoDrNQ1z9kb0ZBT/2nLeA6h4FmF9mpn5TmzpPAaqy9Jx4jA l6XbE5kt2hJoJ94nTHZrAvQ/w67NLeGn5NBtNVDFEuWhapJrQvMw4pe05QJz5D8qSYW/ ACtAZEksvJhiVVNiD7XkaaLcmRFdi0DG4oZn0r/erUDysqwUDMP9UYHbcOF86YMs8hzq LvmA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id m2-v6si277289lfi.4.2018.08.31.01.10.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Aug 2018 01:10:44 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w7V8Ah7n015954 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 Aug 2018 10:10:43 +0200 Received: from [139.22.120.205] ([139.22.120.205]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id w7V8AgiL026814; Fri, 31 Aug 2018 10:10:43 +0200 Subject: Re: [PATCH v3 1/3] dpkg: move installation of dependencies into dpkg_prepare To: Cedric Hombourger , isar-users@googlegroups.com References: <20180822154145.07d4a2ca@md1pvb1c.ad001.siemens.net> <1535264488-128-1-git-send-email-Cedric_Hombourger@mentor.com> From: Jan Kiszka Message-ID: Date: Fri, 31 Aug 2018 10:10:42 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <1535264488-128-1-git-send-email-Cedric_Hombourger@mentor.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: gky2Swcpxt9q On 2018-08-26 08:21, Cedric Hombourger wrote: > In preparation for a rework of locking mechanisms around apt > operations (installation of packages or inclusion of new > packages into the isar-apt database), move the code for > installing build dependencies out of dpkg_runbuild and into > a new function: dpkg_prepare. > > Signed-off-by: Cedric Hombourger > --- > doc/technical_overview.md | 29 ++++++++++---- > meta/classes/dpkg-base.bbclass | 30 +++++++++++--- > meta/classes/dpkg-raw.bbclass | 15 ++++--- > meta/classes/dpkg.bbclass | 8 +++- > meta/recipes-devtools/buildchroot/buildchroot.inc | 9 ++++- > meta/recipes-devtools/buildchroot/files/build.sh | 48 +---------------------- > meta/recipes-devtools/buildchroot/files/common.sh | 28 +++++++++++++ > meta/recipes-devtools/buildchroot/files/deps.sh | 32 +++++++++++++++ > meta/recipes-kernel/linux-module/module.inc | 2 +- > 9 files changed, 128 insertions(+), 73 deletions(-) > create mode 100644 meta/recipes-devtools/buildchroot/files/common.sh > create mode 100644 meta/recipes-devtools/buildchroot/files/deps.sh > > diff --git a/doc/technical_overview.md b/doc/technical_overview.md > index ddd2c79..4f6d954 100644 > --- a/doc/technical_overview.md > +++ b/doc/technical_overview.md > @@ -217,24 +217,37 @@ Both consist of the following steps: > 3. Task `do_install` _only_ for `dpkg-raw`: copy all you want in your > debian package to `${D}`, install hooks in `${D}/DEBIAN` > > -4. Task `do_build`: mount folder with unpacked files to buildchroot, execute > - the actual build function `dpkg_runbuild`, and finally umount again > +4. Task `do_prepare': perform any preparation steps to the unpacked/patched > + sources before the build. This task calls the dpkg_prepare shell function > + with the buildchroot mounts in place (`dpkg_do_mounts') > > - 4.1. the `dpkg_runbuild` function of `dpkg.bbclass` runs `build.sh` in the > - buildchroot. That performs the following: > + 4.1. the `dpkg_prepare` function of `dpkg.bbclass` runs `/isar/deps.sh` in > + the buildchroot. That performs the following: > > 1. Go to `/home/build/${PN}` > > 2. Get list of dependencies from debian/control and install them > > - 3. Run dpkg-buildpackage > + 4.2. the `dpkg_prepare` function of `dpkg-raw.bbclass` translate the > + recipe meta-data into a debian/control file suitable for packaging > + with dpkg-deb > + > +5. Task `do_build`: mount folder (`dpkg_do_mounts') with unpacked files to buildchroot, > + execute the actual build function `dpkg_runbuild`, and finally umount again > + (`dpkg_undo_mounts') > + > + 5.1. the `dpkg_runbuild` function of `dpkg.bbclass` runs `build.sh` in the > + buildchroot. That performs the following: > + > + 1. Go to `/home/build/${PN}` > + > + 2. Run dpkg-buildpackage > > - 4.2. the `dpkg_runbuild` function of `dpkg-raw.bbclass` basically runs > + 5.2. the `dpkg_runbuild` function of `dpkg-raw.bbclass` basically runs > `dpkg-deb` to construct a Debian package from a folder of files, > without compiling anything > > - > -5. Task `do_deploy_deb`: install successfully built packages > +6. Task `do_deploy_deb`: install successfully built packages > `${WORKDIR}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}` > > ## 3.6 Populate Target Filesystem > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index a41df2c..5826357 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -13,7 +13,7 @@ python __anonymous() { > dep = "buildchroot-host:do_build" > rootfs = d.getVar('BUILDCHROOT_HOST_DIR', True) > > - d.setVarFlag('do_build', 'depends', dep) > + d.setVarFlag('do_prepare', 'depends', dep) > d.setVar('BUILDCHROOT_DIR', rootfs) > } > > @@ -32,7 +32,7 @@ addtask patch after do_adjust_git before do_build > > # Add dependency between Isar recipes > DEPENDS ?= "" > -do_build[deptask] = "do_deploy_deb" > +do_prepare[deptask] = "do_deploy_deb" > > def get_package_srcdir(d): > s = d.getVar("S", True) > @@ -50,7 +50,10 @@ PPS ?= "${@get_package_srcdir(d)}" > BUILDROOT = "${BUILDCHROOT_DIR}/${PP}" > do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > > -# default to "emtpy" implementation > +# default to "emtpy" implementation for dpkg_prepare() and dpkg_runbuild() > +dpkg_prepare() { > + true > +} > dpkg_runbuild() { > die "This should never be called, overwrite it in your derived class" > } > @@ -58,7 +61,7 @@ dpkg_runbuild() { > MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock" > > # Wrap the function dpkg_runbuild with the bind mount for buildroot > -do_build() { > +dpkg_do_mounts() { > mkdir -p ${BUILDROOT} > sudo mount --bind ${WORKDIR} ${BUILDROOT} > > @@ -70,13 +73,28 @@ do_build() { > mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev > mount -t proc none ${BUILDCHROOT_DIR}/proc > fi' > +} > > - dpkg_runbuild > - > +dpkg_undo_mounts() { > sudo umount ${BUILDROOT} 2>/dev/null || true > sudo rmdir ${BUILDROOT} 2>/dev/null || true > } > > +do_prepare() { > + dpkg_do_mounts > + dpkg_prepare > + dpkg_undo_mounts > +} > + > +addtask prepare after do_patch before do_build > +do_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > + > +do_build() { > + dpkg_do_mounts > + dpkg_runbuild > + dpkg_undo_mounts > +} > + > CLEANFUNCS += "repo_clean" > > repo_clean() { > diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass > index 28233ac..c5f1c88 100644 > --- a/meta/classes/dpkg-raw.bbclass > +++ b/meta/classes/dpkg-raw.bbclass > @@ -14,9 +14,9 @@ do_install() { > } > > do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > -addtask install after do_unpack before do_deb_package_prepare > +addtask install after do_unpack before do_prepare > > -do_deb_package_prepare() { > +deb_package_prepare() { > sudo rm -rf ${D}/DEBIAN > mkdir -p ${D}/DEBIAN > cat<<-__EOF__ > ${D}/DEBIAN/control > @@ -42,17 +42,16 @@ do_deb_package_prepare() { > done > } > > -do_deb_package_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > -addtask deb_package_prepare after do_install before do_deb_package_conffiles > - > -do_deb_package_conffiles() { > +deb_package_conffiles() { > CONFFILES=${D}/DEBIAN/conffiles > find ${D} -type f -path '${D}/etc/*' | sed -e 's|^${D}|/|' >> $CONFFILES > test -s $CONFFILES || rm $CONFFILES > } > > -do_deb_package_conffiles[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > -addtask deb_package_conffiles after do_deb_package_prepare before do_build > +dpkg_prepare() { > + deb_package_prepare > + deb_package_conffiles > +} > > dpkg_runbuild() { > sudo chown -R root:root ${D}/DEBIAN/ > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index ab70645..e8bd6ba 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -3,8 +3,14 @@ > > inherit dpkg-base > > +# Install build dependencies for package > +dpkg_prepare() { > + E="${@ bb.utils.export_proxies(d)}" > + sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh ${PP}/${PPS} ${DISTRO_ARCH} > +} > + > # Build package from sources using build script > dpkg_runbuild() { > E="${@ bb.utils.export_proxies(d)}" > - sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${PPS} ${DISTRO_ARCH} > + sudo -E chroot ${BUILDCHROOT_DIR} /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH} > } > diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc > index 43e3cd6..7dd909e 100644 > --- a/meta/recipes-devtools/buildchroot/buildchroot.inc > +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc > @@ -8,7 +8,9 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260 > > FILESPATH_prepend := "${THISDIR}/files:" > SRC_URI = "file://configscript.sh \ > - file://build.sh" > + file://build.sh \ > + file://common.sh \ > + file://deps.sh" > PV = "1.0" > > inherit isar-bootstrap-helper > @@ -27,7 +29,10 @@ do_build() { > > # Install package builder script > sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" > - sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR} > + sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar > + sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}/isar/ > + sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/ > + sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/ > > # Configure root filesystem > sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR} > diff --git a/meta/recipes-devtools/buildchroot/files/build.sh b/meta/recipes-devtools/buildchroot/files/build.sh > index e2dabab..3c80bd7 100644 > --- a/meta/recipes-devtools/buildchroot/files/build.sh > +++ b/meta/recipes-devtools/buildchroot/files/build.sh > @@ -4,53 +4,7 @@ > # Copyright (C) 2015-2017 ilbers GmbH > # Copyright (c) 2018 Siemens AG > > -set -e > - > -# Create human-readable names > -target_arch=$2 > - > -# Notes: > -# mk-build-deps for jessie and jtretch has different parameter name to specify > -# host architecture. > -debian_version=$(cut -c1 /etc/debian_version) > -if [ $(($debian_version)) -ge 9 ]; then > - set_arch="--host-arch $target_arch" > -else > - set_arch="-a $target_arch" > -fi > - > -# Go to build directory > -cd $1 > - > -# To avoid Perl locale warnings: > -export LC_ALL=C > -export LANG=C > -export LANGUAGE=C > - > -# Install command to be used by mk-build-deps > -# Notes: > -# 1) everything before the -y switch is unchanged from the defaults > -# 2) we add -y to go non-interactive > -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y" > - > -( > - # Lock-protected because apt and dpkg do not wait in case of contention > - flock 42 || exit 1 > - > - # Make sure that we have latest isar-apt content. > - # Options meaning: > - # Dir::Etc::sourcelist - specifies which source to be used > - # Dir::Etc::sourceparts - disables looking for the other sources > - # APT::Get::List-Cleanup - do not erase obsolete packages list for > - # upstream in '/var/lib/apt/lists' > - apt-get update \ > - -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ > - -o Dir::Etc::sourceparts="-" \ > - -o APT::Get::List-Cleanup="0" > - > - # Install all build deps > - mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control > -) 42>/dpkg.lock > +source /isar/common.sh > > # If autotools files have been created, update their timestamp to > # prevent them from being regenerated > diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh > new file mode 100644 > index 0000000..b7551eb > --- /dev/null > +++ b/meta/recipes-devtools/buildchroot/files/common.sh > @@ -0,0 +1,28 @@ > +#!/bin/bash > +# > +# This software is a part of ISAR. > +# Copyright (C) 2015-2017 ilbers GmbH > +# Copyright (c) 2018 Siemens AG > + > +set -e > + > +# Create human-readable names > +target_arch=$2 > + > +# Notes: > +# mk-build-deps for jessie and jtretch has different parameter name to specify > +# host architecture. > +debian_version=$(cut -c1 /etc/debian_version) > +if [ $(($debian_version)) -ge 9 ]; then > + set_arch="--host-arch $target_arch" > +else > + set_arch="-a $target_arch" > +fi > + > +# Go to build directory > +cd $1 > + > +# To avoid Perl locale warnings: > +export LC_ALL=C > +export LANG=C > +export LANGUAGE=C > diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh > new file mode 100644 > index 0000000..854a4d5 > --- /dev/null > +++ b/meta/recipes-devtools/buildchroot/files/deps.sh > @@ -0,0 +1,32 @@ > +#!/bin/bash > +# > +# This software is a part of ISAR. > +# Copyright (C) 2015-2017 ilbers GmbH > +# Copyright (c) 2018 Siemens AG > + > +source /isar/common.sh > + > +# Install command to be used by mk-build-deps > +# Notes: > +# 1) everything before the -y switch is unchanged from the defaults > +# 2) we add -y to go non-interactive > +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y" > + > +( > + # Lock-protected because apt and dpkg do not wait in case of contention > + flock 42 || exit 1 > + > + # Make sure that we have latest isar-apt content. > + # Options meaning: > + # Dir::Etc::sourcelist - specifies which source to be used > + # Dir::Etc::sourceparts - disables looking for the other sources > + # APT::Get::List-Cleanup - do not erase obsolete packages list for > + # upstream in '/var/lib/apt/lists' > + apt-get update \ > + -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ > + -o Dir::Etc::sourceparts="-" \ > + -o APT::Get::List-Cleanup="0" > + > + # Install all build deps > + mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control > +) 42>/dpkg.lock > diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc > index 3075f44..3a3cab1 100644 > --- a/meta/recipes-kernel/linux-module/module.inc > +++ b/meta/recipes-kernel/linux-module/module.inc > @@ -17,7 +17,7 @@ AUTOLOAD ?= "0" > > inherit dpkg > > -dpkg_runbuild_prepend() { > +do_prepare_prepend() { > cp -r ${WORKDIR}/debian ${S}/ > sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ > -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \ > This patches causes regression with jailhouse-images which I do not understand yet: 2018-08-31 07:53:54 - ERROR - ERROR: mc:espressobin-jailhouse:non-root-initramfs-2018.05.1-r0 do_prepare: Function failed: do_prepare (log file is located at /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928) 2018-08-31 07:53:54 - ERROR - ERROR: Logfile of failure stored in: /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928 2018-08-31 07:53:54 - INFO - Log data follows: 2018-08-31 07:53:54 - INFO - | DEBUG: Executing shell function do_prepare 2018-08-31 07:53:54 - INFO - | Get:1 file:/isar-apt isar InRelease 2018-08-31 07:53:54 - INFO - | Ign:1 file:/isar-apt isar InRelease 2018-08-31 07:53:54 - INFO - | Get:2 file:/isar-apt isar Release [3554 B] 2018-08-31 07:53:54 - INFO - | Get:2 file:/isar-apt isar Release [3554 B] 2018-08-31 07:53:54 - INFO - | Get:3 file:/isar-apt isar Release.gpg 2018-08-31 07:53:54 - INFO - | Ign:3 file:/isar-apt isar Release.gpg 2018-08-31 07:53:54 - INFO - | Reading package lists... 2018-08-31 07:53:54 - INFO - | E: You must put some 'source' URIs in your sources.list 2018-08-31 07:53:54 - INFO - | mk-build-deps: Unable to find package name in `apt-cache showsrc debian/control' 2018-08-31 07:53:54 - INFO - | WARNING: exit code 29 from a shell command. 2018-08-31 07:53:54 - INFO - | ERROR: Function failed: do_prepare (log file is located at /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928) 2018-08-31 07:53:54 - INFO - NOTE: recipe non-root-initramfs-2018.05.1-r0: task do_prepare: Failed 2018-08-31 07:53:54 - ERROR - ERROR: Task (multiconfig:espressobin-jailhouse:/jailhouse-images/recipes-core/non-root-initramfs/non-root-initramfs_2018.05.1.bb:do_prepare) failed with exit code '1' I also wonder why this didn't trigger with the isar core tests. Any idea? Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux