From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6691586504498610176 X-Received: by 2002:a1c:7e08:: with SMTP id z8mr8434421wmc.36.1558947806719; Mon, 27 May 2019 02:03:26 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:54c2:: with SMTP id x2ls1726538wrv.10.gmail; Mon, 27 May 2019 02:03:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqy2VPx8OBx3bbOUIKzWr3s4ui0w7FY8EYu/Z5A3Mft6P3rh3/DeufviKgIvXH+GoMeF5KAB X-Received: by 2002:a5d:528b:: with SMTP id c11mr12432260wrv.25.1558947806227; Mon, 27 May 2019 02:03:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558947806; cv=none; d=google.com; s=arc-20160816; b=DD+aO4Hm007okKNOoDvkFBEb9MEJX2/by5IpCJANTireN0+VuJUBueLZzHa51dOZ2f rTQTNAn4cyJaTQkPp9eefmOVko7GHuLlMpEgLlGpvV6/lmz+7oW3/kPBoz+1n3w2IAtw R3jf2129Bkk6f8tRsYfx6PuCaUJvBcVmj65B0jXtTE6mKCfRlaDRQuuKY4G3GjpiJUR0 ICmGLfsBUS1IiaLHMKLRkmaDOJt/CCv6WoE6fyyEnKorfo/AXWS2BSVNhcwVl3hMgVpX xjUlW2Tr/EsBGLqHdcpeep1JMwCSH0UI/00ywCOmjvH4q33oSl5P5OrXKB/t7Hp4appk rXow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=lbmAyNaspB1gT3bBSv98ZaMDWt4NR3avkcqLVEgfZfE=; b=bbL8B952nz9Wk2wH1qrI5QA9wOrhJLt3UapBt4D/yyVlTN+pTeDY6UQneiQ5J8yuRg 9F/32D4+WTSB+seIXTu2rdModtffx+flGepFpb/uHmUQQOgT/a5Jgj9ZtnOPGlJOw7DX lllZ80cZHifnyyVoqFvQxlSG6t0E01LonXgIWHG/7EVYJ+1vWl2cnt7g1xd0DQ61xFLf ipcbQNmoPpMvllP4geOGBkBdwYH4l+skjvOnXyfZX9Z1w1LY6VwSYl48Tgr9JSHI6MqP TYrysQ61qXe6YNWaaUCBC4CwuohDfu2J8rKd9KRfJyi4bWXOJGNML4V5+I51fIwhLbxL TkXA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id q11si406113wrj.2.2019.05.27.02.03.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 May 2019 02:03:26 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id x4R93Pvi009743 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 May 2019 11:03:25 +0200 Received: from [139.25.69.232] (linux-ses-ext02.ppmd.siemens.net [139.25.69.232]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x4R93Nuc005393; Mon, 27 May 2019 11:03:23 +0200 Subject: Re: [PATCH v4 2/8] split up isar-bootstrap helper and implement pre-process pipeline To: "Maxim Yu. Osipov" , isar-users@googlegroups.com Cc: Claudius Heine References: <20190523145521.23050-1-claudius.heine.ext@siemens.com> <20190523145521.23050-3-claudius.heine.ext@siemens.com> <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> <18fb7e43-c5ed-f007-30c1-4be54a39e7ad@ilbers.de> From: Claudius Heine Message-ID: <2507d110-feb9-95c9-11ab-a914afbc0547@siemens.com> Date: Mon, 27 May 2019 11:03:23 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <18fb7e43-c5ed-f007-30c1-4be54a39e7ad@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: GSYe7EtYdggW Hi, On 27/05/2019 09.20, Maxim Yu. Osipov wrote: > Hi Claudius, > > On 5/27/19 8:55 AM, Claudius Heine wrote: >> Hi Maxim, >> >> On 24/05/2019 14.49, Maxim Yu. Osipov wrote: >>> Hi Claudius, >>> >>> One may agree that setup_root_file_system is too "fat" >>> and should be split to logical parts. >> >> There are multiple issues with `setup_root_file_system`, not just it >> being 'fat'. For instance the amount of parameters this function >> accepts just screams to me that it does not scale necessary >> customization. >> >>> My concern is that one have to carefully keep track for every >>> ROOTFS_CONFIGURE/ROOTFS_INSTALL command's  'weight' - order of >>> execution - possible source of errors. >> >> The weight parameter does not specify the order or execution, but just >> the percentage of the progress bar shown when building. Since >> do_rootfs_install is a pretty big task, I found it useful to show at >> least some progress. > > Ah, I see now. Please add this comment into the code (to avoid confusion). I think that could also be done later in an additional patchset, if that is the only reason for a new series. > >> >> The order of commands is the order in which they are listed in the >> ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND array. >> >> Currently that is not an interface for the user, but that might happen >> if more variables are added there to signify different stages of the >> rootfs build process, for instance how the >> `ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN` was done. > > I think that it's worth to add description of > ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND in RECIPE_API_CHANGELOG. I can do that, but why? No user API has changed AFAIK and ROOTFS_*_COMMAND aren't yet user API. They might become in the future, but currently I would like to threat them as internal API since adding to them from outside of isar is not very comfortable. regards, Claudius > > Regards, > Maxim. > > >> regards, >> Claudius >> >>> >>> Just a case: one my not define weight value (in such case default >>> value is 20) as its is done for >>> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" - so if >>> we have several such default weights the order of commands could be >>> wrong. >>> >>> Regards, >>> Maxim. >>> >>> On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote: >>>> From: Claudius Heine >>>> >>>> This removes the isar-bootstrap-helper.bbclass and integrates its >>>> functionality into the rootfs.bbclass. >>>> >>>> It introduces the rootfs_install task that runs through multiple >>>> functions from command lists in order to allow easy integration of >>>> custom functions. >>>> >>>> The introduced functions are, in general order of execution: >>>> >>>>    rootfs_prepare: copy the isar-bootstrap rootfs base to workdir >>>>    rootfs_configure_*: basic configuration of the root file system, >>>>      before any custom packages where installed. (`/dev`, `/proc`, etc. >>>>      are not mounted). Functions are added to >>>> `ROOTFS_CONFIGURE_COMMAND`. >>>>    rootfs_do_mount: mount standard directories and repositories into >>>> the >>>>      rootfs >>>>    do_rootfs_install_*: downloading and installing packages. Functions >>>>    are added to `ROOTFS_INSTALL_COMMAND`. >>>> >>>> Signed-off-by: Claudius Heine >>>> --- >>>>   meta/classes/image.bbclass                    |  24 +-- >>>>   meta/classes/isar-bootstrap-helper.bbclass    | 132 -------------- >>>>   meta/classes/rootfs.bbclass                   | 172 >>>> +++++++++++++++++- >>>>   .../isar-bootstrap/isar-bootstrap.inc         |   5 + >>>>   .../buildchroot/buildchroot-host.bb           |  16 +- >>>>   .../buildchroot/buildchroot-target.bb         |   2 - >>>>   .../buildchroot/buildchroot.inc               |  26 ++- >>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--- >>>>   8 files changed, 228 insertions(+), 197 deletions(-) >>>>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass >>>> >>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >>>> index d77626c..99eea92 100644 >>>> --- a/meta/classes/image.bbclass >>>> +++ b/meta/classes/image.bbclass >>>> @@ -51,8 +51,10 @@ image_do_mounts() { >>>>       buildchroot_do_mounts >>>>   } >>>> -inherit isar-bootstrap-helper >>>> -ROOTFS_FEATURES += "finalize-rootfs" >>>> +ROOTFSDIR = "${IMAGE_ROOTFS}" >>>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >>>> finalize-rootfs" >>>> +ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >>>> + >>>>   inherit rootfs >>>>   inherit image-sdk-extension >>>>   inherit image-cache-extension >>>> @@ -119,8 +121,10 @@ python set_image_size () { >>>>       d.setVarFlag('ROOTFS_SIZE', 'export', '1') >>>>   } >>>> -do_image_gen_fstab() { >>>> -    cat > ${WORKDIR}/fstab << EOF >>>> +ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab" >>>> +image_configure_fstab[weight] = "2" >>>> +image_configure_fstab() { >>>> +    sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF >>>>   # Begin /etc/fstab >>>>   /dev/root    /        auto        defaults        0    0 >>>>   proc        /proc        proc        nosuid,noexec,nodev    0    0 >>>> @@ -132,18 +136,6 @@ devtmpfs    /dev        devtmpfs >>>> mode=0755,nosuid    0    0 >>>>   # End /etc/fstab >>>>   EOF >>>>   } >>>> -addtask image_gen_fstab before do_rootfs_install >>>> - >>>> -do_rootfs_install[depends] = "isar-apt:do_cache_config >>>> isar-bootstrap-target:do_bootstrap" >>>> -do_rootfs_install[deptask] = "do_deploy_deb" >>>> -do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \ >>>> -                             ${IMAGE_ROOTFS}/isar-apt" >>>> -do_rootfs_install() { >>>> -    setup_root_file_system --clean --keep-apt-cache \ >>>> -        --fstab "${WORKDIR}/fstab" \ >>>> -        "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL} >>>> -} >>>> -addtask rootfs_install before do_build after do_unpack >>>>   do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" >>>>   do_copy_boot_files() { >>>> diff --git a/meta/classes/isar-bootstrap-helper.bbclass >>>> b/meta/classes/isar-bootstrap-helper.bbclass >>>> deleted file mode 100644 >>>> index bbc6322..0000000 >>>> --- a/meta/classes/isar-bootstrap-helper.bbclass >>>> +++ /dev/null >>>> @@ -1,132 +0,0 @@ >>>> -# Helper functions for using isar-bootstrap >>>> -# >>>> -# This software is a part of ISAR. >>>> -# Copyright (c) Siemens AG, 2018 >>>> -# >>>> -# SPDX-License-Identifier: MIT >>>> - >>>> -def reverse_bb_array(d, varname): >>>> -    array = d.getVar(varname, True) >>>> -    if array is None: >>>> -        return None >>>> -    array = reversed(array.split()) >>>> -    return " ".join(i for i in array) >>>> - >>>> - >>>> -update_etc_os_release() { >>>> -    OS_RELEASE_BUILD_ID="" >>>> -    OS_RELEASE_VARIANT="" >>>> -    while true; do >>>> -        case "$1" in >>>> -    --build-id) OS_RELEASE_BUILD_ID=$2; shift ;; >>>> -    --variant) OS_RELEASE_VARIANT=$2; shift ;; >>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>> -        *) break ;; >>>> -        esac >>>> -        shift >>>> -    done >>>> -    ROOTFSDIR="$1" >>>> - >>>> -    if [ -n "${OS_RELEASE_BUILD_ID}" ]; then >>>> -        sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release >>>> -        echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \ >>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>> -    fi >>>> -    if [ -n "${OS_RELEASE_VARIANT}" ]; then >>>> -        sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release >>>> -        echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \ >>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>> -    fi >>>> -} >>>> - >>>> -setup_root_file_system() { >>>> -    CLEAN="" >>>> -    COPYISARAPT="" >>>> -    FSTAB="" >>>> -    ROOTFS_ARCH="${DISTRO_ARCH}" >>>> -    ROOTFS_DISTRO="${DISTRO}" >>>> -    while true; do >>>> -        case "$1" in >>>> -        --clean) CLEAN=1 ;; >>>> -        --copyisarapt) COPYISARAPT=1 ;; >>>> -        --fstab) FSTAB=$2; shift ;; >>>> -        --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;; >>>> -        --host-distro) ROOTFS_DISTRO="$2"; shift ;; >>>> -        --keep-apt-cache) KEEP_APT_CACHE=1 ;; >>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>> -        *) break ;; >>>> -        esac >>>> -        shift >>>> -    done >>>> -    ROOTFSDIR="$1" >>>> -    shift >>>> -    PACKAGES="$@" >>>> -    APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>> -    CLEAN_FILES="${ROOTFSDIR}/etc/hostname >>>> ${ROOTFSDIR}/etc/resolv.conf" >>>> - >>>> -    sudo cp -Trpfx \ >>>> -        "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \ >>>> -        "$ROOTFSDIR" >>>> -    [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee >>>> "$ROOTFSDIR/etc/fstab" >>>> - >>>> -    echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main" >>>> | \ >>>> -        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >>>> >/dev/null >>>> - >>>> -    echo "Package: *\nPin: release >>>> n=${DEBDISTRONAME}\nPin-Priority: 1000" | \ >>>> -        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null >>>> - >>>> -    if [ ${COPYISARAPT} ]; then >>>> -        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >>>> -    else >>>> -        sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} >>>> $ROOTFSDIR/isar-apt >>>> -    fi >>>> - >>>> -    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>> -        sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt >>>> -    fi >>>> - >>>> -    sudo mount --rbind /dev ${ROOTFSDIR}/dev >>>> -    sudo mount --make-rslave ${ROOTFSDIR}/dev >>>> -    sudo mount -t proc none $ROOTFSDIR/proc >>>> -    sudo mount --rbind /sys ${ROOTFSDIR}/sys >>>> -    sudo mount --make-rslave ${ROOTFSDIR}/sys >>>> - >>>> -    # Refresh /etc/resolv.conf >>>> -    sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc >>>> - >>>> -    # Install packages: >>>> -    E="${@ bb.utils.export_proxies(d)}" >>>> -    export DEBIAN_FRONTEND=noninteractive >>>> -    # To avoid Perl locale warnings: >>>> -    export LANG=C >>>> -    export LANGUAGE=C >>>> -    export LC_ALL=C >>>> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \ >>>> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>> -        -o Dir::Etc::sourceparts="-" \ >>>> -        -o APT::Get::List-Cleanup="0" >>>> -    # Add multiarch for cross-target >>>> -    if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then >>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg >>>> --add-architecture ${DISTRO_ARCH} >>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update >>>> -    fi >>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>> -        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES >>>> -    [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES} >>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>> -        /usr/bin/apt-get ${APT_ARGS} $PACKAGES >>>> -    if [ ${CLEAN} ]; then >>>> -        if [ ${KEEP_APT_CACHE} -eq 1 ]; then >>>> -            mkdir -p ${WORKDIR}/apt_cache >>>> -            sudo find ${ROOTFSDIR}/var/cache/apt/archives \ >>>> -                -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>> ${WORKDIR}/apt_cache '{}' '+' >>>> -            sudo chown -R $(whoami) ${WORKDIR}/apt_cache >>>> -        fi >>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>> -            /usr/bin/apt-get autoremove --purge --yes >>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>> -            /usr/bin/apt-get clean >>>> -        sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR" >>>> -        sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" >>>> "$ROOTFSDIR/var/lib/apt/lists/"* >>>> -    fi >>>> -} >>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >>>> index 45b0350..0fc8ba8 100644 >>>> --- a/meta/classes/rootfs.bbclass >>>> +++ b/meta/classes/rootfs.bbclass >>>> @@ -1,14 +1,178 @@ >>>>   # This software is a part of ISAR. >>>>   # Copyright (c) Siemens AG, 2019 >>>> +ROOTFS_ARCH ?= "${DISTRO_ARCH}" >>>> +ROOTFS_DISTRO ?= "${DISTRO}" >>>> +ROOTFS_PACKAGES ?= "" >>>> + >>>>   # Features of the rootfs creation: >>>>   # available features are: >>>> +# 'deploy-package-cache' - copy the package cache ${WORKDIR}/apt_cache >>>> +# 'clean-package-cache' - delete package cache from rootfs >>>>   # 'finalize-rootfs' - delete files needed to chroot into the rootfs >>>>   ROOTFS_FEATURES ?= "" >>>> +ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>> + >>>> +ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" >>>> + >>>> +# Useful environment variables: >>>> +export E = "${@ bb.utils.export_proxies(d)}" >>>> +export DEBIAN_FRONTEND = "noninteractive" >>>> +# To avoid Perl locale warnings: >>>> +export LANG = "C" >>>> +export LANGUAGE = "C" >>>> +export LC_ALL = "C" >>>> + >>>> +rootfs_do_mounts[weight] = "3" >>>> +rootfs_do_mounts() { >>>> +    sudo -s <<'EOSUDO' >>>> +        mountpoint -q '${ROOTFSDIR}/dev' || \ >>>> +            mount --rbind /dev '${ROOTFSDIR}/dev' >>>> +        mount --make-rslave '${ROOTFSDIR}/dev' >>>> +        mountpoint -q '${ROOTFSDIR}/proc' || \ >>>> +            mount -t proc none '${ROOTFSDIR}/proc' >>>> +        mountpoint -q '${ROOTFSDIR}/sys' || \ >>>> +            mount --rbind /sys '${ROOTFSDIR}/sys' >>>> +        mount --make-rslave '${ROOTFSDIR}/sys' >>>> + >>>> +        # Mount isar-apt if the directory does not exist or if it >>>> is empty >>>> +        # This prevents overwriting something that was copied there >>>> +        if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ >>>> +           [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth >>>> 1 | wc -l)" = "0" ] >>>> +        then >>>> +            mkdir -p '${ROOTFSDIR}/isar-apt' >>>> +            mountpoint -q '${ROOTFSDIR}/isar-apt' || \ >>>> +                mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >>>> '${ROOTFSDIR}/isar-apt' >>>> +        fi >>>> + >>>> +        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set >>>> +        if [ >>>> "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" >>>> = 'True' ] >>>> +        then >>>> +            mkdir -p '${ROOTFSDIR}/base-apt' >>>> +            mountpoint -q '${ROOTFSDIR}/base-apt' || \ >>>> +                mount --bind '${REPO_BASE_DIR}' >>>> '${ROOTFSDIR}/base-apt' >>>> +        fi >>>> + >>>> +EOSUDO >>>> +} >>>> + >>>> +rootfs_do_qemu() { >>>> +    if [ '${@repr(d.getVar('ROOTFS_ARCH') == >>>> d.getVar('HOST_ARCH'))}' = 'False' ] >>>> +    then >>>> +        test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \ >>>> +            sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' >>>> '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' >>>> +    fi >>>> +} >>>> + >>>> +BOOTSTRAP_SRC = >>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}" >>>> >>>> +BOOTSTRAP_SRC_${ROOTFS_ARCH} = >>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" >>>> + >>>> +rootfs_prepare[weight] = "25" >>>> +rootfs_prepare(){ >>>> +    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' >>>> +} >>>> + >>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" >>>> +rootfs_configure_isar_apt[weight] = "2" >>>> +rootfs_configure_isar_apt() { >>>> +    sudo -s <<'EOSUDO' >>>> + >>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' >>>> +    echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >>>> > \ >>>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' >>>> + >>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' >>>> +    cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar' >>>> +Package: * >>>> +Pin: release n=${DEBDISTRONAME} >>>> +Pin-Priority: 1000 >>>> +EOF >>>> +EOSUDO >>>> +} >>>> + >>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" >>>> +rootfs_install_pkgs_update[weight] = "5" >>>> +rootfs_install_pkgs_update() { >>>> +    sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ >>>> +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>> +        -o Dir::Etc::sourceparts="-" \ >>>> +        -o APT::Get::List-Cleanup="0" >>>> +} >>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf" >>>> +rootfs_install_resolvconf[weight] = "1" >>>> +rootfs_install_resolvconf() { >>>> +    sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc' >>>> +} >>>> + >>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" >>>> +rootfs_install_pkgs_download[weight] = "600" >>>> +rootfs_install_pkgs_download() { >>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only >>>> ${ROOTFS_PACKAGES} >>>> +} >>>> + >>>> +ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if >>>> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" >>>> +rootfs_install_clean_files[weight] = "2" >>>> +rootfs_install_clean_files() { >>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>> +        /bin/rm -f ${ROOTFS_CLEAN_FILES} >>>> +} >>>> + >>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install" >>>> +rootfs_install_pkgs_install[weight] = "8000" >>>> +rootfs_install_pkgs_install() { >>>> +    sudo -E chroot "${ROOTFSDIR}" \ >>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} >>>> +} >>> >>> >>> Honestly I don't see benefits in such multiple >>> configure_cmds/rootfs_install_cmds - it doesn't increase code >>> readability as one have to additionally keep track for every command >>> 'weight' value and it's easy to miss the correct order of commands >>> execution. >>> >>> >>>> +do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" >>>> +do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} >>>> ${ROOTFS_INSTALL_COMMAND}" >>>> +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if >>>> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else >>>> 'host'}:do_build isar-apt:do_cache_config" >>>> +do_rootfs_install[deptask] = "do_deploy_deb" >>>> +python do_rootfs_install() { >>>> +    configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or >>>> "").split() >>>> +    install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or >>>> "").split() >>>> + >>>> +    # Mount after configure commands, so that they have time to copy >>>> +    # 'isar-apt' (sdkchroot): >>>> +    cmds = ['rootfs_prepare'] + configure_cmds + >>>> ['rootfs_do_mounts'] + install_cmds >>>> + >>>> +    stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20") >>>> +                     for i in cmds] >>>> + >>>> +    progress_reporter = bb.progress.MultiStageProgressReporter(d, >>>> stage_weights) >>>> + >>>> +    for cmd in cmds: >>>> +        progress_reporter.next_stage() >>>> +        bb.build.exec_func(cmd, d) >>>> +    progress_reporter.finish() >>>> +} >>>> +addtask rootfs_install before do_rootfs_postprocess after do_unpack >>>> + >>>> +ROOTFS_POSTPROCESS_COMMAND += >>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', >>>> 'rootfs_postprocess_copy_package_cache', '', d)}" >>>> +rootfs_postprocess_copy_package_cache() { >>>> +    mkdir -p '${WORKDIR}/apt_cache' >>>> +    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \ >>>> +        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>> '${WORKDIR}/apt_cache' '{}' '+' >>>> +    me="$(id -u):$(id -g)" >>>> +    sudo chown -R "$me" '${WORKDIR}/apt_cache' >>>> +} >>>> + >>>> +ROOTFS_POSTPROCESS_COMMAND += >>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', >>>> 'rootfs_postprocess_clean_package_cache', '', d)}" >>>> +rootfs_postprocess_clean_package_cache() { >>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>> +        /usr/bin/apt-get clean >>>> +    sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >>>> +} >>>> + >>>>   ROOTFS_POSTPROCESS_COMMAND += >>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', >>>> 'rootfs_postprocess_finalize', '', d)}" >>>>   rootfs_postprocess_finalize() { >>>>       sudo -s <<'EOSUDO' >>>> +        test -e "${ROOTFSDIR}/chroot-setup.sh" && \ >>>> +            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}" >>>> +        rm -f "${ROOTFSDIR}/chroot-setup.sh" >>>> + >>>>           test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \ >>>>               find "${ROOTFSDIR}/usr/bin" \ >>>>                   -maxdepth 1 -name 'qemu-*-static' -type f -delete >>>> @@ -42,6 +206,12 @@ EOSUDO >>>>   do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >>>>   python do_rootfs_postprocess() { >>>> +    # Take care that its correctly mounted: >>>> +    bb.build.exec_func('rootfs_do_mounts', d) >>>> +    # Take care that qemu-*-static is available, since it could >>>> have been >>>> +    # removed on a previous execution of this task: >>>> +    bb.build.exec_func('rootfs_do_qemu', d) >>>> + >>>>       cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND") >>>>       if cmds is None or not cmds.strip(): >>>>           return >>>> @@ -49,7 +219,7 @@ python do_rootfs_postprocess() { >>>>       for cmd in cmds: >>>>           bb.build.exec_func(cmd, d) >>>>   } >>>> -addtask rootfs_postprocess before do_rootfs after do_rootfs_install >>>> +addtask rootfs_postprocess before do_rootfs >>>>   python do_rootfs() { >>>>       """Virtual task""" >>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>> index b740c57..f8741c2 100644 >>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>> @@ -327,6 +327,11 @@ isar_bootstrap() { >>>>               mount --make-rslave ${ROOTFSDIR}/sys >>>>               export DEBIAN_FRONTEND=noninteractive >>>> + >>>> +            if [ ${IS_HOST} ]; then >>>> +                chroot "${ROOTFSDIR}" /usr/bin/dpkg >>>> --add-architecture ${DISTRO_ARCH} >>>> +            fi >>>> + >>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ >>>>                                     -o Debug::pkgProblemResolver=yes >>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>> b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>> index 73cd548..121992c 100644 >>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>> @@ -7,6 +7,8 @@ DESCRIPTION = "Isar development filesystem for host" >>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>>   require buildchroot.inc >>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>>   BUILDCHROOT_PREINSTALL ?= "make \ >>>>                              debhelper \ >>>> @@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \ >>>>                              automake \ >>>>                              devscripts \ >>>>                              equivs \ >>>> -                           libc6:${DISTRO_ARCH}" >>>> - >>>> -# According to the wiki page: >>>> -#     https://wiki.debian.org/CrossToolchains >>>> -BUILDCHROOT_PREINSTALL_append_armhf += "binutils-arm-linux-gnueabihf \ >>>> -                                        crossbuild-essential-armhf" >>>> -BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \ >>>> -                                        crossbuild-essential-arm64" >>>> - >>>> - >>>> -PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'" >>>> -do_build[depends] = "isar-apt:do_cache_config >>>> isar-bootstrap-host:do_bootstrap" >>>> +                           libc6:${DISTRO_ARCH} \ >>>> +                           crossbuild-essential-${DISTRO_ARCH}" >>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>> b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>> index 20f4d23..5a01258 100644 >>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>> @@ -21,5 +21,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>>>                              devscripts \ >>>>                              equivs \ >>>>                              adduser" >>>> - >>>> -do_build[depends] = "isar-apt:do_cache_config >>>> isar-bootstrap-target:do_bootstrap" >>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>> b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>> index ca9ce51..40fc8b0 100644 >>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>> @@ -13,18 +13,25 @@ SRC_URI = "file://configscript.sh \ >>>>              file://deps.sh" >>>>   PV = "1.0" >>>> -inherit isar-bootstrap-helper >>>> +inherit rootfs >>>>   BUILDCHROOT_DIR = "${WORKDIR}/rootfs" >>>> +ROOTFSDIR = "${BUILDCHROOT_DIR}" >>>> +ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}" >>>> +ROOTFS_CLEAN_FILES = "" >>>> + >>>> +rootfs_do_mounts_append() { >>>> +    sudo -s <<'EOSUDO' >>>> +    mkdir -p '${BUILDCHROOT_DIR}/downloads' >>>> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ >>>> +        mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' >>>> +EOSUDO >>>> +} >>>> -do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \ >>>> -                            ${BUILDCHROOT_DIR}/isar-apt \ >>>> -                            ${BUILDCHROOT_DIR}/downloads \ >>>> -                            ${BUILDCHROOT_DIR}/home/builder" >>>> - >>>> -do_build() { >>>> -    setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} >>>> ${BUILDCHROOT_PREINSTALL} >>>> - >>>> +ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" >>>> +buildchroot_install_files() { >>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder" >>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt" >>>>       # Install package builder script >>>>       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" >>>>       sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar >>>> @@ -38,5 +45,4 @@ do_build() { >>>>       GROUP_ID=$(id -g) >>>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID >>>> $GROUP_ID >>>> -    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads >>>>   } >>>> diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>> b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>> index 49aeb47..57d52ba 100644 >>>> --- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>> +++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>> @@ -12,10 +12,25 @@ SRC_URI = " \ >>>>       file://configscript.sh \ >>>>       file://README.sdk" >>>>   PV = "0.1" >>>> - >>>> -inherit isar-bootstrap-helper >>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>> +TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}" >>>> +TOOLCHAIN_${HOST_ARCH} = "build-essential" >>>> +TOOLCHAIN_i386 = "build-essential" >>>> + >>>> +inherit rootfs >>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>> +ROOTFSDIR = "${S}" >>>> +ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}" >>>> +ROOTFS_FEATURES += "copy-package-cache" >>>> + >>>> +python() { >>>> +    if d.getVar("HOST_ARCH") not in ['i386', 'amd64']: >>>> +        raise bb.parse.SkipRecipe("SDK doesn't support {} as >>>> host".format( >>>> +            d.getVar("ROOTFS_ARCH"))) >>>> +} >>>> + >>>>   SDKCHROOT_PREINSTALL := "debhelper \ >>>>                              autotools-dev \ >>>>                              dpkg \ >>>> @@ -28,29 +43,14 @@ SDKCHROOT_PREINSTALL := "debhelper \ >>>>   S = "${WORKDIR}/rootfs" >>>> -do_build[dirs] = "${DEPLOY_DIR_IMAGE}" >>>> -do_build[root_cleandirs] = "${S} \ >>>> -                            ${S}/isar-apt" >>>> - >>>> -do_build[depends] = "isar-apt:do_cache_config >>>> isar-bootstrap-host:do_bootstrap" >>>> - >>>> -do_build() { >>>> - >>>> -    if [ ${HOST_DISTRO} != "debian-stretch" ]; then >>>> -        bbfatal "SDK doesn't support ${HOST_DISTRO}" >>>> -    fi >>>> -    if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then >>>> -        bbfatal "SDK doesn't support ${HOST_ARCH} as host" >>>> -    fi >>>> - >>>> -    if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" >>>> ]; then >>>> -        packages="${SDKCHROOT_PREINSTALL} build-essential" >>>> -    else >>>> -        packages="${SDKCHROOT_PREINSTALL} >>>> crossbuild-essential-${DISTRO_ARCH}" >>>> -    fi >>>> - >>>> -    setup_root_file_system --copyisarapt --host-arch '${HOST_ARCH}' >>>> --host-distro '${HOST_DISTRO}' "${S}" $packages >>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" >>>> +rootfs_configure_isar_apt_dir() { >>>> +    # Copy isar-apt instead of mounting: >>>> +    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt >>>> +} >>>> +ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files" >>>> +sdkchroot_install_files() { >>>>       # Configure root filesystem >>>>       sudo install -m 644 ${WORKDIR}/README.sdk ${S} >>>>       sudo install -m 755 ${WORKDIR}/configscript.sh ${S} >>>> >>> >>> >> > > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de