From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6691586504498610176 X-Received: by 2002:a17:906:d557:: with SMTP id gk23mr13000491ejb.285.1558940161577; Sun, 26 May 2019 23:56:01 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:4946:: with SMTP id f6ls3629155ejt.4.gmail; Sun, 26 May 2019 23:56:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqza5SPPWFKDx2sDT1d5/bz5hI5dBbuzbGErv865cD5wGNYNQu9PRkYfbkIMKApEWqoBzXAA X-Received: by 2002:a17:906:f84a:: with SMTP id ks10mr77488999ejb.65.1558940161144; Sun, 26 May 2019 23:56:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558940161; cv=none; d=google.com; s=arc-20160816; b=VUkCIcjMJ3Nbft0Vboxjmk78oYejVBlbbIixdaLlxSoqHahGKXOPNt5GokCqs/y5+B dDyuMNYlQ598A/qVXxJUKY28okLSOM03djefQ8MFePYyrqfVifAiZ8DAKsgjbfmValkp u8YtF6CJh605jMIfZfZUIyJIX5oIOl/wNGxi0audQAWFdW7QS7IY+wiL5Y5/2JP2EUeK 2hLTcy9jzUDYKTShlQjA5uSLoUQsWMHDGNe3DY9I25uFWZqSH9jaKZlaV4gJODSKlRLD douRqZUJJNoPm+CTdbte6zXEaVd0MJxIMaKZcc4OOXOgKeJpw3QS4suFY3yZDzJ4fz/b yZrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=QqsBd6o87fBuoxQOldikiOGFQIM+g6m0kMFqh3jNV64=; b=TksWeucqPGbkiimcPXUzxkBJRI4f3g3k3+wOyriV/54WI41BPsC3PjvDrvtsuBGKgq Bx5FD2Alq95ff/7Lh6yEYjUpjmXO0liIX70sqmTpuTG83xurt+SQ+xUUciZQ5fWO1XFY UOhuZYGTjwWGvpjHnaEIRz4pzu/GJBZDivuEE/eY15sqnBtChAh520zrQnP1++fTIDGG oja5TjtPTnuOLc5lLUir5PCCT6jJVxOrtZMBmceMJh+WSFKcBSKJ/jrxivWGhNQOrVWi GHOchXABppOjzq5h3F4/ZTbOMoCV3sGNPseWSPEQ3I8jM2hLTMvxuL5R5/pe2hMHufZ5 dFuw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id h25si353931ejz.1.2019.05.26.23.56.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 26 May 2019 23:56:01 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x4R6u0ad010220 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 May 2019 08:56:00 +0200 Received: from [139.25.69.232] (linux-ses-ext02.ppmd.siemens.net [139.25.69.232]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x4R6txjR032601; Mon, 27 May 2019 08:55:59 +0200 Subject: Re: [PATCH v4 2/8] split up isar-bootstrap helper and implement pre-process pipeline To: "Maxim Yu. Osipov" , isar-users@googlegroups.com Cc: Claudius Heine References: <20190523145521.23050-1-claudius.heine.ext@siemens.com> <20190523145521.23050-3-claudius.heine.ext@siemens.com> <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> From: Claudius Heine Message-ID: Date: Mon, 27 May 2019 08:55:59 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: dC8quHc31Um1 Hi Maxim, On 24/05/2019 14.49, Maxim Yu. Osipov wrote: > Hi Claudius, > > One may agree that setup_root_file_system is too "fat" > and should be split to logical parts. There are multiple issues with `setup_root_file_system`, not just it being 'fat'. For instance the amount of parameters this function accepts just screams to me that it does not scale necessary customization. > My concern is that one have to carefully keep track for every > ROOTFS_CONFIGURE/ROOTFS_INSTALL command's  'weight' - order of execution > - possible source of errors. The weight parameter does not specify the order or execution, but just the percentage of the progress bar shown when building. Since do_rootfs_install is a pretty big task, I found it useful to show at least some progress. The order of commands is the order in which they are listed in the ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND array. Currently that is not an interface for the user, but that might happen if more variables are added there to signify different stages of the rootfs build process, for instance how the `ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN` was done. regards, Claudius > > Just a case: one my not define weight value (in such case default value > is 20) as its is done for > ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" - so if we > have several such default weights the order of commands could be wrong. > > Regards, > Maxim. > > On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote: >> From: Claudius Heine >> >> This removes the isar-bootstrap-helper.bbclass and integrates its >> functionality into the rootfs.bbclass. >> >> It introduces the rootfs_install task that runs through multiple >> functions from command lists in order to allow easy integration of >> custom functions. >> >> The introduced functions are, in general order of execution: >> >>    rootfs_prepare: copy the isar-bootstrap rootfs base to workdir >>    rootfs_configure_*: basic configuration of the root file system, >>      before any custom packages where installed. (`/dev`, `/proc`, etc. >>      are not mounted). Functions are added to `ROOTFS_CONFIGURE_COMMAND`. >>    rootfs_do_mount: mount standard directories and repositories into the >>      rootfs >>    do_rootfs_install_*: downloading and installing packages. Functions >>    are added to `ROOTFS_INSTALL_COMMAND`. >> >> Signed-off-by: Claudius Heine >> --- >>   meta/classes/image.bbclass                    |  24 +-- >>   meta/classes/isar-bootstrap-helper.bbclass    | 132 -------------- >>   meta/classes/rootfs.bbclass                   | 172 +++++++++++++++++- >>   .../isar-bootstrap/isar-bootstrap.inc         |   5 + >>   .../buildchroot/buildchroot-host.bb           |  16 +- >>   .../buildchroot/buildchroot-target.bb         |   2 - >>   .../buildchroot/buildchroot.inc               |  26 ++- >>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--- >>   8 files changed, 228 insertions(+), 197 deletions(-) >>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass >> >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index d77626c..99eea92 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -51,8 +51,10 @@ image_do_mounts() { >>       buildchroot_do_mounts >>   } >> -inherit isar-bootstrap-helper >> -ROOTFS_FEATURES += "finalize-rootfs" >> +ROOTFSDIR = "${IMAGE_ROOTFS}" >> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >> finalize-rootfs" >> +ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >> + >>   inherit rootfs >>   inherit image-sdk-extension >>   inherit image-cache-extension >> @@ -119,8 +121,10 @@ python set_image_size () { >>       d.setVarFlag('ROOTFS_SIZE', 'export', '1') >>   } >> -do_image_gen_fstab() { >> -    cat > ${WORKDIR}/fstab << EOF >> +ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab" >> +image_configure_fstab[weight] = "2" >> +image_configure_fstab() { >> +    sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF >>   # Begin /etc/fstab >>   /dev/root    /        auto        defaults        0    0 >>   proc        /proc        proc        nosuid,noexec,nodev    0    0 >> @@ -132,18 +136,6 @@ devtmpfs    /dev        devtmpfs >> mode=0755,nosuid    0    0 >>   # End /etc/fstab >>   EOF >>   } >> -addtask image_gen_fstab before do_rootfs_install >> - >> -do_rootfs_install[depends] = "isar-apt:do_cache_config >> isar-bootstrap-target:do_bootstrap" >> -do_rootfs_install[deptask] = "do_deploy_deb" >> -do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \ >> -                             ${IMAGE_ROOTFS}/isar-apt" >> -do_rootfs_install() { >> -    setup_root_file_system --clean --keep-apt-cache \ >> -        --fstab "${WORKDIR}/fstab" \ >> -        "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL} >> -} >> -addtask rootfs_install before do_build after do_unpack >>   do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" >>   do_copy_boot_files() { >> diff --git a/meta/classes/isar-bootstrap-helper.bbclass >> b/meta/classes/isar-bootstrap-helper.bbclass >> deleted file mode 100644 >> index bbc6322..0000000 >> --- a/meta/classes/isar-bootstrap-helper.bbclass >> +++ /dev/null >> @@ -1,132 +0,0 @@ >> -# Helper functions for using isar-bootstrap >> -# >> -# This software is a part of ISAR. >> -# Copyright (c) Siemens AG, 2018 >> -# >> -# SPDX-License-Identifier: MIT >> - >> -def reverse_bb_array(d, varname): >> -    array = d.getVar(varname, True) >> -    if array is None: >> -        return None >> -    array = reversed(array.split()) >> -    return " ".join(i for i in array) >> - >> - >> -update_etc_os_release() { >> -    OS_RELEASE_BUILD_ID="" >> -    OS_RELEASE_VARIANT="" >> -    while true; do >> -        case "$1" in >> -    --build-id) OS_RELEASE_BUILD_ID=$2; shift ;; >> -    --variant) OS_RELEASE_VARIANT=$2; shift ;; >> -        -*) bbfatal "$0: invalid option specified: $1" ;; >> -        *) break ;; >> -        esac >> -        shift >> -    done >> -    ROOTFSDIR="$1" >> - >> -    if [ -n "${OS_RELEASE_BUILD_ID}" ]; then >> -        sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release >> -        echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \ >> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >> -    fi >> -    if [ -n "${OS_RELEASE_VARIANT}" ]; then >> -        sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release >> -        echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \ >> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >> -    fi >> -} >> - >> -setup_root_file_system() { >> -    CLEAN="" >> -    COPYISARAPT="" >> -    FSTAB="" >> -    ROOTFS_ARCH="${DISTRO_ARCH}" >> -    ROOTFS_DISTRO="${DISTRO}" >> -    while true; do >> -        case "$1" in >> -        --clean) CLEAN=1 ;; >> -        --copyisarapt) COPYISARAPT=1 ;; >> -        --fstab) FSTAB=$2; shift ;; >> -        --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;; >> -        --host-distro) ROOTFS_DISTRO="$2"; shift ;; >> -        --keep-apt-cache) KEEP_APT_CACHE=1 ;; >> -        -*) bbfatal "$0: invalid option specified: $1" ;; >> -        *) break ;; >> -        esac >> -        shift >> -    done >> -    ROOTFSDIR="$1" >> -    shift >> -    PACKAGES="$@" >> -    APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >> -    CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf" >> - >> -    sudo cp -Trpfx \ >> -        "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \ >> -        "$ROOTFSDIR" >> -    [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee "$ROOTFSDIR/etc/fstab" >> - >> -    echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main" | \ >> -        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >> >/dev/null >> - >> -    echo "Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: >> 1000" | \ >> -        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null >> - >> -    if [ ${COPYISARAPT} ]; then >> -        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >> -    else >> -        sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >> -    fi >> - >> -    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >> -        sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt >> -    fi >> - >> -    sudo mount --rbind /dev ${ROOTFSDIR}/dev >> -    sudo mount --make-rslave ${ROOTFSDIR}/dev >> -    sudo mount -t proc none $ROOTFSDIR/proc >> -    sudo mount --rbind /sys ${ROOTFSDIR}/sys >> -    sudo mount --make-rslave ${ROOTFSDIR}/sys >> - >> -    # Refresh /etc/resolv.conf >> -    sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc >> - >> -    # Install packages: >> -    E="${@ bb.utils.export_proxies(d)}" >> -    export DEBIAN_FRONTEND=noninteractive >> -    # To avoid Perl locale warnings: >> -    export LANG=C >> -    export LANGUAGE=C >> -    export LC_ALL=C >> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \ >> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >> -        -o Dir::Etc::sourceparts="-" \ >> -        -o APT::Get::List-Cleanup="0" >> -    # Add multiarch for cross-target >> -    if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then >> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture >> ${DISTRO_ARCH} >> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update >> -    fi >> -    sudo -E chroot "$ROOTFSDIR" \ >> -        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES >> -    [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES} >> -    sudo -E chroot "$ROOTFSDIR" \ >> -        /usr/bin/apt-get ${APT_ARGS} $PACKAGES >> -    if [ ${CLEAN} ]; then >> -        if [ ${KEEP_APT_CACHE} -eq 1 ]; then >> -            mkdir -p ${WORKDIR}/apt_cache >> -            sudo find ${ROOTFSDIR}/var/cache/apt/archives \ >> -                -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >> ${WORKDIR}/apt_cache '{}' '+' >> -            sudo chown -R $(whoami) ${WORKDIR}/apt_cache >> -        fi >> -        sudo -E chroot "$ROOTFSDIR" \ >> -            /usr/bin/apt-get autoremove --purge --yes >> -        sudo -E chroot "$ROOTFSDIR" \ >> -            /usr/bin/apt-get clean >> -        sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR" >> -        sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" >> "$ROOTFSDIR/var/lib/apt/lists/"* >> -    fi >> -} >> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >> index 45b0350..0fc8ba8 100644 >> --- a/meta/classes/rootfs.bbclass >> +++ b/meta/classes/rootfs.bbclass >> @@ -1,14 +1,178 @@ >>   # This software is a part of ISAR. >>   # Copyright (c) Siemens AG, 2019 >> +ROOTFS_ARCH ?= "${DISTRO_ARCH}" >> +ROOTFS_DISTRO ?= "${DISTRO}" >> +ROOTFS_PACKAGES ?= "" >> + >>   # Features of the rootfs creation: >>   # available features are: >> +# 'deploy-package-cache' - copy the package cache ${WORKDIR}/apt_cache >> +# 'clean-package-cache' - delete package cache from rootfs >>   # 'finalize-rootfs' - delete files needed to chroot into the rootfs >>   ROOTFS_FEATURES ?= "" >> +ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >> + >> +ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" >> + >> +# Useful environment variables: >> +export E = "${@ bb.utils.export_proxies(d)}" >> +export DEBIAN_FRONTEND = "noninteractive" >> +# To avoid Perl locale warnings: >> +export LANG = "C" >> +export LANGUAGE = "C" >> +export LC_ALL = "C" >> + >> +rootfs_do_mounts[weight] = "3" >> +rootfs_do_mounts() { >> +    sudo -s <<'EOSUDO' >> +        mountpoint -q '${ROOTFSDIR}/dev' || \ >> +            mount --rbind /dev '${ROOTFSDIR}/dev' >> +        mount --make-rslave '${ROOTFSDIR}/dev' >> +        mountpoint -q '${ROOTFSDIR}/proc' || \ >> +            mount -t proc none '${ROOTFSDIR}/proc' >> +        mountpoint -q '${ROOTFSDIR}/sys' || \ >> +            mount --rbind /sys '${ROOTFSDIR}/sys' >> +        mount --make-rslave '${ROOTFSDIR}/sys' >> + >> +        # Mount isar-apt if the directory does not exist or if it is >> empty >> +        # This prevents overwriting something that was copied there >> +        if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ >> +           [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth 1 >> | wc -l)" = "0" ] >> +        then >> +            mkdir -p '${ROOTFSDIR}/isar-apt' >> +            mountpoint -q '${ROOTFSDIR}/isar-apt' || \ >> +                mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >> '${ROOTFSDIR}/isar-apt' >> +        fi >> + >> +        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set >> +        if [ >> "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" >> = 'True' ] >> +        then >> +            mkdir -p '${ROOTFSDIR}/base-apt' >> +            mountpoint -q '${ROOTFSDIR}/base-apt' || \ >> +                mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt' >> +        fi >> + >> +EOSUDO >> +} >> + >> +rootfs_do_qemu() { >> +    if [ '${@repr(d.getVar('ROOTFS_ARCH') == d.getVar('HOST_ARCH'))}' >> = 'False' ] >> +    then >> +        test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \ >> +            sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' >> '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' >> +    fi >> +} >> + >> +BOOTSTRAP_SRC = >> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}" >> +BOOTSTRAP_SRC_${ROOTFS_ARCH} = >> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" >> + >> +rootfs_prepare[weight] = "25" >> +rootfs_prepare(){ >> +    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' >> +} >> + >> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" >> +rootfs_configure_isar_apt[weight] = "2" >> +rootfs_configure_isar_apt() { >> +    sudo -s <<'EOSUDO' >> + >> +    mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' >> +    echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \ >> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' >> + >> +    mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' >> +    cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar' >> +Package: * >> +Pin: release n=${DEBDISTRONAME} >> +Pin-Priority: 1000 >> +EOF >> +EOSUDO >> +} >> + >> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" >> +rootfs_install_pkgs_update[weight] = "5" >> +rootfs_install_pkgs_update() { >> +    sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ >> +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >> +        -o Dir::Etc::sourceparts="-" \ >> +        -o APT::Get::List-Cleanup="0" >> +} >> +ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf" >> +rootfs_install_resolvconf[weight] = "1" >> +rootfs_install_resolvconf() { >> +    sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc' >> +} >> + >> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" >> +rootfs_install_pkgs_download[weight] = "600" >> +rootfs_install_pkgs_download() { >> +    sudo -E chroot '${ROOTFSDIR}' \ >> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only >> ${ROOTFS_PACKAGES} >> +} >> + >> +ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if >> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" >> +rootfs_install_clean_files[weight] = "2" >> +rootfs_install_clean_files() { >> +    sudo -E chroot '${ROOTFSDIR}' \ >> +        /bin/rm -f ${ROOTFS_CLEAN_FILES} >> +} >> + >> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install" >> +rootfs_install_pkgs_install[weight] = "8000" >> +rootfs_install_pkgs_install() { >> +    sudo -E chroot "${ROOTFSDIR}" \ >> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} >> +} > > > Honestly I don't see benefits in such multiple > configure_cmds/rootfs_install_cmds - it doesn't increase code > readability as one have to additionally keep track for every command > 'weight' value and it's easy to miss the correct order of commands > execution. > > >> +do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" >> +do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} >> ${ROOTFS_INSTALL_COMMAND}" >> +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if >> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else >> 'host'}:do_build isar-apt:do_cache_config" >> +do_rootfs_install[deptask] = "do_deploy_deb" >> +python do_rootfs_install() { >> +    configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or >> "").split() >> +    install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or >> "").split() >> + >> +    # Mount after configure commands, so that they have time to copy >> +    # 'isar-apt' (sdkchroot): >> +    cmds = ['rootfs_prepare'] + configure_cmds + ['rootfs_do_mounts'] >> + install_cmds >> + >> +    stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20") >> +                     for i in cmds] >> + >> +    progress_reporter = bb.progress.MultiStageProgressReporter(d, >> stage_weights) >> + >> +    for cmd in cmds: >> +        progress_reporter.next_stage() >> +        bb.build.exec_func(cmd, d) >> +    progress_reporter.finish() >> +} >> +addtask rootfs_install before do_rootfs_postprocess after do_unpack >> + >> +ROOTFS_POSTPROCESS_COMMAND += >> "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', >> 'rootfs_postprocess_copy_package_cache', '', d)}" >> +rootfs_postprocess_copy_package_cache() { >> +    mkdir -p '${WORKDIR}/apt_cache' >> +    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \ >> +        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >> '${WORKDIR}/apt_cache' '{}' '+' >> +    me="$(id -u):$(id -g)" >> +    sudo chown -R "$me" '${WORKDIR}/apt_cache' >> +} >> + >> +ROOTFS_POSTPROCESS_COMMAND += >> "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', >> 'rootfs_postprocess_clean_package_cache', '', d)}" >> +rootfs_postprocess_clean_package_cache() { >> +    sudo -E chroot '${ROOTFSDIR}' \ >> +        /usr/bin/apt-get clean >> +    sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >> +} >> + >>   ROOTFS_POSTPROCESS_COMMAND += >> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', >> 'rootfs_postprocess_finalize', '', d)}" >>   rootfs_postprocess_finalize() { >>       sudo -s <<'EOSUDO' >> +        test -e "${ROOTFSDIR}/chroot-setup.sh" && \ >> +            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}" >> +        rm -f "${ROOTFSDIR}/chroot-setup.sh" >> + >>           test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \ >>               find "${ROOTFSDIR}/usr/bin" \ >>                   -maxdepth 1 -name 'qemu-*-static' -type f -delete >> @@ -42,6 +206,12 @@ EOSUDO >>   do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >>   python do_rootfs_postprocess() { >> +    # Take care that its correctly mounted: >> +    bb.build.exec_func('rootfs_do_mounts', d) >> +    # Take care that qemu-*-static is available, since it could have >> been >> +    # removed on a previous execution of this task: >> +    bb.build.exec_func('rootfs_do_qemu', d) >> + >>       cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND") >>       if cmds is None or not cmds.strip(): >>           return >> @@ -49,7 +219,7 @@ python do_rootfs_postprocess() { >>       for cmd in cmds: >>           bb.build.exec_func(cmd, d) >>   } >> -addtask rootfs_postprocess before do_rootfs after do_rootfs_install >> +addtask rootfs_postprocess before do_rootfs >>   python do_rootfs() { >>       """Virtual task""" >> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> index b740c57..f8741c2 100644 >> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> @@ -327,6 +327,11 @@ isar_bootstrap() { >>               mount --make-rslave ${ROOTFSDIR}/sys >>               export DEBIAN_FRONTEND=noninteractive >> + >> +            if [ ${IS_HOST} ]; then >> +                chroot "${ROOTFSDIR}" /usr/bin/dpkg >> --add-architecture ${DISTRO_ARCH} >> +            fi >> + >>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ >>                                     -o Debug::pkgProblemResolver=yes >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >> b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >> index 73cd548..121992c 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >> @@ -7,6 +7,8 @@ DESCRIPTION = "Isar development filesystem for host" >>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>   require buildchroot.inc >> +ROOTFS_ARCH = "${HOST_ARCH}" >> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>   BUILDCHROOT_PREINSTALL ?= "make \ >>                              debhelper \ >> @@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \ >>                              automake \ >>                              devscripts \ >>                              equivs \ >> -                           libc6:${DISTRO_ARCH}" >> - >> -# According to the wiki page: >> -#     https://wiki.debian.org/CrossToolchains >> -BUILDCHROOT_PREINSTALL_append_armhf += "binutils-arm-linux-gnueabihf \ >> -                                        crossbuild-essential-armhf" >> -BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \ >> -                                        crossbuild-essential-arm64" >> - >> - >> -PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'" >> -do_build[depends] = "isar-apt:do_cache_config >> isar-bootstrap-host:do_bootstrap" >> +                           libc6:${DISTRO_ARCH} \ >> +                           crossbuild-essential-${DISTRO_ARCH}" >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >> b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >> index 20f4d23..5a01258 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >> @@ -21,5 +21,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>                              devscripts \ >>                              equivs \ >>                              adduser" >> - >> -do_build[depends] = "isar-apt:do_cache_config >> isar-bootstrap-target:do_bootstrap" >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc >> b/meta/recipes-devtools/buildchroot/buildchroot.inc >> index ca9ce51..40fc8b0 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc >> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc >> @@ -13,18 +13,25 @@ SRC_URI = "file://configscript.sh \ >>              file://deps.sh" >>   PV = "1.0" >> -inherit isar-bootstrap-helper >> +inherit rootfs >>   BUILDCHROOT_DIR = "${WORKDIR}/rootfs" >> +ROOTFSDIR = "${BUILDCHROOT_DIR}" >> +ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}" >> +ROOTFS_CLEAN_FILES = "" >> + >> +rootfs_do_mounts_append() { >> +    sudo -s <<'EOSUDO' >> +    mkdir -p '${BUILDCHROOT_DIR}/downloads' >> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ >> +        mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' >> +EOSUDO >> +} >> -do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \ >> -                            ${BUILDCHROOT_DIR}/isar-apt \ >> -                            ${BUILDCHROOT_DIR}/downloads \ >> -                            ${BUILDCHROOT_DIR}/home/builder" >> - >> -do_build() { >> -    setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} >> ${BUILDCHROOT_PREINSTALL} >> - >> +ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" >> +buildchroot_install_files() { >> +    sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder" >> +    sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt" >>       # Install package builder script >>       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" >>       sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar >> @@ -38,5 +45,4 @@ do_build() { >>       GROUP_ID=$(id -g) >>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID >> -    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads >>   } >> diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >> b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >> index 49aeb47..57d52ba 100644 >> --- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >> +++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >> @@ -12,10 +12,25 @@ SRC_URI = " \ >>       file://configscript.sh \ >>       file://README.sdk" >>   PV = "0.1" >> - >> -inherit isar-bootstrap-helper >>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >> +TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}" >> +TOOLCHAIN_${HOST_ARCH} = "build-essential" >> +TOOLCHAIN_i386 = "build-essential" >> + >> +inherit rootfs >> +ROOTFS_ARCH = "${HOST_ARCH}" >> +ROOTFS_DISTRO = "${HOST_DISTRO}" >> +ROOTFSDIR = "${S}" >> +ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}" >> +ROOTFS_FEATURES += "copy-package-cache" >> + >> +python() { >> +    if d.getVar("HOST_ARCH") not in ['i386', 'amd64']: >> +        raise bb.parse.SkipRecipe("SDK doesn't support {} as >> host".format( >> +            d.getVar("ROOTFS_ARCH"))) >> +} >> + >>   SDKCHROOT_PREINSTALL := "debhelper \ >>                              autotools-dev \ >>                              dpkg \ >> @@ -28,29 +43,14 @@ SDKCHROOT_PREINSTALL := "debhelper \ >>   S = "${WORKDIR}/rootfs" >> -do_build[dirs] = "${DEPLOY_DIR_IMAGE}" >> -do_build[root_cleandirs] = "${S} \ >> -                            ${S}/isar-apt" >> - >> -do_build[depends] = "isar-apt:do_cache_config >> isar-bootstrap-host:do_bootstrap" >> - >> -do_build() { >> - >> -    if [ ${HOST_DISTRO} != "debian-stretch" ]; then >> -        bbfatal "SDK doesn't support ${HOST_DISTRO}" >> -    fi >> -    if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then >> -        bbfatal "SDK doesn't support ${HOST_ARCH} as host" >> -    fi >> - >> -    if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" ]; >> then >> -        packages="${SDKCHROOT_PREINSTALL} build-essential" >> -    else >> -        packages="${SDKCHROOT_PREINSTALL} >> crossbuild-essential-${DISTRO_ARCH}" >> -    fi >> - >> -    setup_root_file_system --copyisarapt --host-arch '${HOST_ARCH}' >> --host-distro '${HOST_DISTRO}' "${S}" $packages >> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" >> +rootfs_configure_isar_apt_dir() { >> +    # Copy isar-apt instead of mounting: >> +    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt >> +} >> +ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files" >> +sdkchroot_install_files() { >>       # Configure root filesystem >>       sudo install -m 644 ${WORKDIR}/README.sdk ${S} >>       sudo install -m 755 ${WORKDIR}/configscript.sh ${S} >> > > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de