Hi Jan,

if your recipe is editing files in debian/, you would then want them to be 
done before we install build dependencies
this can be done by using a "do_prepare_prepend" to make the tweaks you need

maybe we could add a do_amend_debian_meta (defaulting to a no-op) to make 
it easy for people to tweak meta-data and not having to know about 
do_prepare?

Cedric

On Friday, August 31, 2018 at 10:10:45 AM UTC+2, Jan Kiszka wrote:
>
> On 2018-08-26 08:21, Cedric Hombourger wrote: 
> > In preparation for a rework of locking mechanisms around apt 
> > operations (installation of packages or inclusion of new 
> > packages into the isar-apt database), move the code for 
> > installing build dependencies out of dpkg_runbuild and into 
> > a new function: dpkg_prepare. 
> > 
> > Signed-off-by: Cedric Hombourger <Cedric_H...@mentor.com <javascript:>> 
> > --- 
> >   doc/technical_overview.md                         | 29 ++++++++++---- 
> >   meta/classes/dpkg-base.bbclass                    | 30 +++++++++++--- 
> >   meta/classes/dpkg-raw.bbclass                     | 15 ++++--- 
> >   meta/classes/dpkg.bbclass                         |  8 +++- 
> >   meta/recipes-devtools/buildchroot/buildchroot.inc |  9 ++++- 
> >   meta/recipes-devtools/buildchroot/files/build.sh  | 48 
> +---------------------- 
> >   meta/recipes-devtools/buildchroot/files/common.sh | 28 +++++++++++++ 
> >   meta/recipes-devtools/buildchroot/files/deps.sh   | 32 +++++++++++++++ 
> >   meta/recipes-kernel/linux-module/module.inc       |  2 +- 
> >   9 files changed, 128 insertions(+), 73 deletions(-) 
> >   create mode 100644 meta/recipes-devtools/buildchroot/files/common.sh 
> >   create mode 100644 meta/recipes-devtools/buildchroot/files/deps.sh 
> > 
> > diff --git a/doc/technical_overview.md b/doc/technical_overview.md 
> > index ddd2c79..4f6d954 100644 
> > --- a/doc/technical_overview.md 
> > +++ b/doc/technical_overview.md 
> > @@ -217,24 +217,37 @@ Both consist of the following steps: 
> >   3. Task `do_install` _only_ for `dpkg-raw`: copy all you want in your 
> >      debian package to `${D}`, install hooks in `${D}/DEBIAN` 
> >   
> > -4. Task `do_build`: mount folder with unpacked files to buildchroot, 
> execute 
> > -   the actual build function `dpkg_runbuild`, and finally umount again 
> > +4. Task `do_prepare': perform any preparation steps to the 
> unpacked/patched 
> > +   sources before the build. This task calls the dpkg_prepare shell 
> function 
> > +   with the buildchroot mounts in place (`dpkg_do_mounts') 
> >   
> > -   4.1. the `dpkg_runbuild` function of `dpkg.bbclass` runs `build.sh` 
> in the 
> > -        buildchroot. That performs the following: 
> > +   4.1. the `dpkg_prepare` function of `dpkg.bbclass` runs 
> `/isar/deps.sh` in 
> > +        the buildchroot. That performs the following: 
> >   
> >           1. Go to `/home/build/${PN}` 
> >   
> >           2. Get list of dependencies from debian/control and install 
> them 
> >   
> > -        3. Run dpkg-buildpackage 
> > +   4.2. the `dpkg_prepare` function of `dpkg-raw.bbclass` translate the 
> > +        recipe meta-data into a debian/control file suitable for 
> packaging 
> > +        with dpkg-deb 
> > + 
> > +5. Task `do_build`: mount folder (`dpkg_do_mounts') with unpacked files 
> to buildchroot, 
> > +   execute the actual build function `dpkg_runbuild`, and finally 
> umount again 
> > +   (`dpkg_undo_mounts') 
> > + 
> > +   5.1. the `dpkg_runbuild` function of `dpkg.bbclass` runs `build.sh` 
> in the 
> > +        buildchroot. That performs the following: 
> > + 
> > +        1. Go to `/home/build/${PN}` 
> > + 
> > +        2. Run dpkg-buildpackage 
> >   
> > -   4.2. the `dpkg_runbuild` function of `dpkg-raw.bbclass` basically 
> runs 
> > +   5.2. the `dpkg_runbuild` function of `dpkg-raw.bbclass` basically 
> runs 
> >           `dpkg-deb` to construct a Debian package from a folder of 
> files, 
> >           without compiling anything 
> >   
> > - 
> > -5. Task `do_deploy_deb`: install successfully built packages 
> > +6. Task `do_deploy_deb`: install successfully built packages 
> >      `${WORKDIR}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}` 
> >   
> >   ## 3.6 Populate Target Filesystem 
> > diff --git a/meta/classes/dpkg-base.bbclass 
> b/meta/classes/dpkg-base.bbclass 
> > index a41df2c..5826357 100644 
> > --- a/meta/classes/dpkg-base.bbclass 
> > +++ b/meta/classes/dpkg-base.bbclass 
> > @@ -13,7 +13,7 @@ python __anonymous() { 
> >           dep = "buildchroot-host:do_build" 
> >           rootfs = d.getVar('BUILDCHROOT_HOST_DIR', True) 
> >   
> > -    d.setVarFlag('do_build', 'depends', dep) 
> > +    d.setVarFlag('do_prepare', 'depends', dep) 
> >       d.setVar('BUILDCHROOT_DIR', rootfs) 
> >   } 
> >   
> > @@ -32,7 +32,7 @@ addtask patch after do_adjust_git before do_build 
> >   
> >   # Add dependency between Isar recipes 
> >   DEPENDS ?= "" 
> > -do_build[deptask] = "do_deploy_deb" 
> > +do_prepare[deptask] = "do_deploy_deb" 
> >   
> >   def get_package_srcdir(d): 
> >       s = d.getVar("S", True) 
> > @@ -50,7 +50,10 @@ PPS ?= "${@get_package_srcdir(d)}" 
> >   BUILDROOT = "${BUILDCHROOT_DIR}/${PP}" 
> >   do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" 
> >   
> > -# default to "emtpy" implementation 
> > +# default to "emtpy" implementation for dpkg_prepare() and 
> dpkg_runbuild() 
> > +dpkg_prepare() { 
> > +    true 
> > +} 
> >   dpkg_runbuild() { 
> >       die "This should never be called, overwrite it in your derived 
> class" 
> >   } 
> > @@ -58,7 +61,7 @@ dpkg_runbuild() { 
> >   MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock" 
> >   
> >   # Wrap the function dpkg_runbuild with the bind mount for buildroot 
> > -do_build() { 
> > +dpkg_do_mounts() { 
> >       mkdir -p ${BUILDROOT} 
> >       sudo mount --bind ${WORKDIR} ${BUILDROOT} 
> >   
> > @@ -70,13 +73,28 @@ do_build() { 
> >               mount -t devtmpfs -o mode=0755,nosuid devtmpfs 
> ${BUILDCHROOT_DIR}/dev 
> >               mount -t proc none ${BUILDCHROOT_DIR}/proc 
> >           fi' 
> > +} 
> >   
> > -    dpkg_runbuild 
> > - 
> > +dpkg_undo_mounts() { 
> >       sudo umount ${BUILDROOT} 2>/dev/null || true 
> >       sudo rmdir ${BUILDROOT} 2>/dev/null || true 
> >   } 
> >   
> > +do_prepare() { 
> > +    dpkg_do_mounts 
> > +    dpkg_prepare 
> > +    dpkg_undo_mounts 
> > +} 
> > + 
> > +addtask prepare after do_patch before do_build 
> > +do_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" 
> > + 
> > +do_build() { 
> > +    dpkg_do_mounts 
> > +    dpkg_runbuild 
> > +    dpkg_undo_mounts 
> > +} 
> > + 
> >   CLEANFUNCS += "repo_clean" 
> >   
> >   repo_clean() { 
> > diff --git a/meta/classes/dpkg-raw.bbclass 
> b/meta/classes/dpkg-raw.bbclass 
> > index 28233ac..c5f1c88 100644 
> > --- a/meta/classes/dpkg-raw.bbclass 
> > +++ b/meta/classes/dpkg-raw.bbclass 
> > @@ -14,9 +14,9 @@ do_install() { 
> >   } 
> >   
> >   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" 
> > -addtask install after do_unpack before do_deb_package_prepare 
> > +addtask install after do_unpack before do_prepare 
> >   
> > -do_deb_package_prepare() { 
> > +deb_package_prepare() { 
> >           sudo rm -rf ${D}/DEBIAN 
> >           mkdir -p ${D}/DEBIAN 
> >           cat<<-__EOF__ > ${D}/DEBIAN/control 
> > @@ -42,17 +42,16 @@ do_deb_package_prepare() { 
> >           done 
> >   } 
> >   
> > -do_deb_package_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" 
> > -addtask deb_package_prepare after do_install before 
> do_deb_package_conffiles 
> > - 
> > -do_deb_package_conffiles() { 
> > +deb_package_conffiles() { 
> >           CONFFILES=${D}/DEBIAN/conffiles 
> >           find ${D} -type f -path '${D}/etc/*' | sed -e 's|^${D}|/|' >> 
> $CONFFILES 
> >           test -s $CONFFILES || rm $CONFFILES 
> >   } 
> >   
> > -do_deb_package_conffiles[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" 
> > -addtask deb_package_conffiles after do_deb_package_prepare before 
> do_build 
> > +dpkg_prepare() { 
> > +        deb_package_prepare 
> > +        deb_package_conffiles 
> > +} 
> >   
> >   dpkg_runbuild() { 
> >           sudo chown -R root:root ${D}/DEBIAN/ 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass 
> > index ab70645..e8bd6ba 100644 
> > --- a/meta/classes/dpkg.bbclass 
> > +++ b/meta/classes/dpkg.bbclass 
> > @@ -3,8 +3,14 @@ 
> >   
> >   inherit dpkg-base 
> >   
> > +# Install build dependencies for package 
> > +dpkg_prepare() { 
> > +    E="${@ bb.utils.export_proxies(d)}" 
> > +    sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh ${PP}/${PPS} 
> ${DISTRO_ARCH} 
> > +} 
> > + 
> >   # Build package from sources using build script 
> >   dpkg_runbuild() { 
> >       E="${@ bb.utils.export_proxies(d)}" 
> > -    sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${PPS} 
> ${DISTRO_ARCH} 
> > +    sudo -E chroot ${BUILDCHROOT_DIR} /isar/build.sh ${PP}/${PPS} 
> ${DISTRO_ARCH} 
> >   } 
> > diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc 
> b/meta/recipes-devtools/buildchroot/buildchroot.inc 
> > index 43e3cd6..7dd909e 100644 
> > --- a/meta/recipes-devtools/buildchroot/buildchroot.inc 
> > +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc 
> > @@ -8,7 +8,9 @@ LIC_FILES_CHKSUM = 
> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260 
> >   
> >   FILESPATH_prepend := "${THISDIR}/files:" 
> >   SRC_URI = "file://configscript.sh \ 
> > -           file://build.sh" 
> > +           file://build.sh \ 
> > +           file://common.sh \ 
> > +           file://deps.sh" 
> >   PV = "1.0" 
> >   
> >   inherit isar-bootstrap-helper 
> > @@ -27,7 +29,10 @@ do_build() { 
> >   
> >       # Install package builder script 
> >       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" 
> > -    sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR} 
> > +    sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar 
> > +    sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}/isar/ 
> > +    sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/ 
> > +    sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/ 
> >   
> >       # Configure root filesystem 
> >       sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR} 
> > diff --git a/meta/recipes-devtools/buildchroot/files/build.sh 
> b/meta/recipes-devtools/buildchroot/files/build.sh 
> > index e2dabab..3c80bd7 100644 
> > --- a/meta/recipes-devtools/buildchroot/files/build.sh 
> > +++ b/meta/recipes-devtools/buildchroot/files/build.sh 
> > @@ -4,53 +4,7 @@ 
> >   # Copyright (C) 2015-2017 ilbers GmbH 
> >   # Copyright (c) 2018 Siemens AG 
> >   
> > -set -e 
> > - 
> > -# Create human-readable names 
> > -target_arch=$2 
> > - 
> > -# Notes: 
> > -#   mk-build-deps for jessie and jtretch has different parameter name 
> to specify 
> > -#   host architecture. 
> > -debian_version=$(cut -c1 /etc/debian_version) 
> > -if [ $(($debian_version)) -ge 9 ]; then 
> > -    set_arch="--host-arch $target_arch" 
> > -else 
> > -    set_arch="-a $target_arch" 
> > -fi 
> > - 
> > -# Go to build directory 
> > -cd $1 
> > - 
> > -# To avoid Perl locale warnings: 
> > -export LC_ALL=C 
> > -export LANG=C 
> > -export LANGUAGE=C 
> > - 
> > -# Install command to be used by mk-build-deps 
> > -# Notes: 
> > -#   1) everything before the -y switch is unchanged from the defaults 
> > -#   2) we add -y to go non-interactive 
> > -install_cmd="apt-get -o Debug::pkgProblemResolver=yes 
> --no-install-recommends -y" 
> > - 
> > -( 
> > -    # Lock-protected because apt and dpkg do not wait in case of 
> contention 
> > -    flock 42 || exit 1 
> > - 
> > -    # Make sure that we have latest isar-apt content. 
> > -    # Options meaning: 
> > -    #   Dir::Etc::sourcelist - specifies which source to be used 
> > -    #   Dir::Etc::sourceparts - disables looking for the other sources 
> > -    #   APT::Get::List-Cleanup - do not erase obsolete packages list 
> for 
> > -    #                            upstream in '/var/lib/apt/lists' 
> > -    apt-get update \ 
> > -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ 
> > -        -o Dir::Etc::sourceparts="-" \ 
> > -        -o APT::Get::List-Cleanup="0" 
> > - 
> > -    # Install all build deps 
> > -    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control 
> > -) 42>/dpkg.lock 
> > +source /isar/common.sh 
> >   
> >   # If autotools files have been created, update their timestamp to 
> >   # prevent them from being regenerated 
> > diff --git a/meta/recipes-devtools/buildchroot/files/common.sh 
> b/meta/recipes-devtools/buildchroot/files/common.sh 
> > new file mode 100644 
> > index 0000000..b7551eb 
> > --- /dev/null 
> > +++ b/meta/recipes-devtools/buildchroot/files/common.sh 
> > @@ -0,0 +1,28 @@ 
> > +#!/bin/bash 
> > +# 
> > +# This software is a part of ISAR. 
> > +# Copyright (C) 2015-2017 ilbers GmbH 
> > +# Copyright (c) 2018 Siemens AG 
> > + 
> > +set -e 
> > + 
> > +# Create human-readable names 
> > +target_arch=$2 
> > + 
> > +# Notes: 
> > +#   mk-build-deps for jessie and jtretch has different parameter name 
> to specify 
> > +#   host architecture. 
> > +debian_version=$(cut -c1 /etc/debian_version) 
> > +if [ $(($debian_version)) -ge 9 ]; then 
> > +    set_arch="--host-arch $target_arch" 
> > +else 
> > +    set_arch="-a $target_arch" 
> > +fi 
> > + 
> > +# Go to build directory 
> > +cd $1 
> > + 
> > +# To avoid Perl locale warnings: 
> > +export LC_ALL=C 
> > +export LANG=C 
> > +export LANGUAGE=C 
> > diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh 
> b/meta/recipes-devtools/buildchroot/files/deps.sh 
> > new file mode 100644 
> > index 0000000..854a4d5 
> > --- /dev/null 
> > +++ b/meta/recipes-devtools/buildchroot/files/deps.sh 
> > @@ -0,0 +1,32 @@ 
> > +#!/bin/bash 
> > +# 
> > +# This software is a part of ISAR. 
> > +# Copyright (C) 2015-2017 ilbers GmbH 
> > +# Copyright (c) 2018 Siemens AG 
> > + 
> > +source /isar/common.sh 
> > + 
> > +# Install command to be used by mk-build-deps 
> > +# Notes: 
> > +#   1) everything before the -y switch is unchanged from the defaults 
> > +#   2) we add -y to go non-interactive 
> > +install_cmd="apt-get -o Debug::pkgProblemResolver=yes 
> --no-install-recommends -y" 
> > + 
> > +( 
> > +    # Lock-protected because apt and dpkg do not wait in case of 
> contention 
> > +    flock 42 || exit 1 
> > + 
> > +    # Make sure that we have latest isar-apt content. 
> > +    # Options meaning: 
> > +    #   Dir::Etc::sourcelist - specifies which source to be used 
> > +    #   Dir::Etc::sourceparts - disables looking for the other sources 
> > +    #   APT::Get::List-Cleanup - do not erase obsolete packages list 
> for 
> > +    #                            upstream in '/var/lib/apt/lists' 
> > +    apt-get update \ 
> > +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ 
> > +        -o Dir::Etc::sourceparts="-" \ 
> > +        -o APT::Get::List-Cleanup="0" 
> > + 
> > +    # Install all build deps 
> > +    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control 
> > +) 42>/dpkg.lock 
> > diff --git a/meta/recipes-kernel/linux-module/module.inc 
> b/meta/recipes-kernel/linux-module/module.inc 
> > index 3075f44..3a3cab1 100644 
> > --- a/meta/recipes-kernel/linux-module/module.inc 
> > +++ b/meta/recipes-kernel/linux-module/module.inc 
> > @@ -17,7 +17,7 @@ AUTOLOAD ?= "0" 
> >   
> >   inherit dpkg 
> >   
> > -dpkg_runbuild_prepend() { 
> > +do_prepare_prepend() { 
> >       cp -r ${WORKDIR}/debian ${S}/ 
> >       sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \ 
> >              -e 's/@KERNEL_NAME@/${KERNEL_NAME}/g' \ 
> > 
>
> This patches causes regression with jailhouse-images which I do not 
> understand yet: 
>
> 2018-08-31 07:53:54 - ERROR    - ERROR: 
> mc:espressobin-jailhouse:non-root-initramfs-2018.05.1-r0 do_prepare: 
> Function failed: do_prepare (log file is located at 
> /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928) 
>
> 2018-08-31 07:53:54 - ERROR    - ERROR: Logfile of failure stored in: 
> /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928 
>
> 2018-08-31 07:53:54 - INFO     - Log data follows: 
> 2018-08-31 07:53:54 - INFO     - | DEBUG: Executing shell function 
> do_prepare 
> 2018-08-31 07:53:54 - INFO     - | Get:1 file:/isar-apt isar InRelease 
> 2018-08-31 07:53:54 - INFO     - | Ign:1 file:/isar-apt isar InRelease 
> 2018-08-31 07:53:54 - INFO     - | Get:2 file:/isar-apt isar Release [3554 
> B] 
> 2018-08-31 07:53:54 - INFO     - | Get:2 file:/isar-apt isar Release [3554 
> B] 
> 2018-08-31 07:53:54 - INFO     - | Get:3 file:/isar-apt isar Release.gpg 
> 2018-08-31 07:53:54 - INFO     - | Ign:3 file:/isar-apt isar Release.gpg 
> 2018-08-31 07:53:54 - INFO     - | Reading package lists... 
> 2018-08-31 07:53:54 - INFO     - | E: You must put some 'source' URIs in 
> your sources.list 
> 2018-08-31 07:53:54 - INFO     - | mk-build-deps: Unable to find package 
> name in `apt-cache showsrc debian/control' 
> 2018-08-31 07:53:54 - INFO     - | WARNING: exit code 29 from a shell 
> command. 
> 2018-08-31 07:53:54 - INFO     - | ERROR: Function failed: do_prepare (log 
> file is located at 
> /out/build/tmp/work/debian-stretch-arm64/non-root-initramfs-2018.05.1-r0/temp/log.do_prepare.28928) 
>
> 2018-08-31 07:53:54 - INFO     - NOTE: recipe 
> non-root-initramfs-2018.05.1-r0: task do_prepare: Failed 
> 2018-08-31 07:53:54 - ERROR    - ERROR: Task 
> (multiconfig:espressobin-jailhouse:/jailhouse-images/recipes-core/non-root-initramfs/non-root-initramfs_2018.05.1.bb:do_prepare) 
> failed with exit code '1' 
>
>
> I also wonder why this didn't trigger with the isar core tests. Any idea? 
>
> Jan 
>
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE 
> Corporate Competence Center Embedded Linux 
>