From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6691586504498610176 X-Received: by 2002:a19:e308:: with SMTP id a8mr7642207lfh.69.1558945852086; Mon, 27 May 2019 01:30:52 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:9e89:: with SMTP id h131ls1402002lfe.10.gmail; Mon, 27 May 2019 01:30:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqxc0wf42nUsbWbygGhn8u8wbCiah7R5F8KjeaL8y3d5WAah/+XFy9KcGYDwkGancGozce9+ X-Received: by 2002:ac2:410a:: with SMTP id b10mr7085133lfi.175.1558945851604; Mon, 27 May 2019 01:30:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558945851; cv=none; d=google.com; s=arc-20160816; b=ioEHmoHKFItneKro7FDU4FB+mb5KEJ4G4vReZ9P2PpEsH8goSI8vVggakN/EGwMei4 TYdOM+pO1fEEKDA3GI6fUt6cKpwa3pbN+3qdPBvVbgVf1AcQxgpwgQlcEHzh1jqZ2Yuj ULp+7rEsVEmAD0dvZn8xJLRaFhZhKWTvvbsxwhhc/UU2pyvPN20DXFMg32WOP779mALG XOgGa88okLuskukM4QRuQqUdVvKmU9EBjmhbywwfJr38IXZZG58HmltF8WbMaYm2V6Mb p4AVmKHb/OiTVQ7uW20LLv9lolfH0Fj8lbkjLRL+N4ECtkczdj2bdDJt5w2TIKvgW5mh VGaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=s3HofJHZoiDt+ZyIjS1zzeYI46CDDkA5kUhPRTSrQNs=; b=daXmXnEpC2WmkaNbtiuoAyathCJNDt3HupxXWZJlJtFfj1vXh+wYr8tQ3XtwVZc7MD 6ZQ0zHD3WpmU9YVuxuKJlPRl7JhheMUN0qPfUHtkdxuNpw0qYYgGFI6RaNMhh4wJwhgL Isipeu7ECl0dKekj0qPU0VGm4/sbbBELhCu9OMjyItKVtFkSYEuOOngC3zLsx6sD8rpT UPo+ifVicMIIXgIWDokjklaVlXosHm6vnBUW3nXfEqQoe4tV0KBlLNpZYIdKvYfBeejn 9nKHVUaX2/FgOZPlCfZ/BMAczALBU3gB6KvZvKDaGYEsjRYuk6IKynYH7yi2py+VhxQ7 U+WQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id s187si493712lfe.4.2019.05.27.01.30.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 May 2019 01:30:51 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x4R8UnTr029664 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 May 2019 10:30:49 +0200 Received: from [139.25.69.232] (linux-ses-ext02.ppmd.siemens.net [139.25.69.232]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x4R8UlS8004995; Mon, 27 May 2019 10:30:48 +0200 Subject: Re: [PATCH v4 2/8] split up isar-bootstrap helper and implement pre-process pipeline To: "Maxim Yu. Osipov" , isar-users@googlegroups.com Cc: Claudius Heine References: <20190523145521.23050-1-claudius.heine.ext@siemens.com> <20190523145521.23050-3-claudius.heine.ext@siemens.com> <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> <18fb7e43-c5ed-f007-30c1-4be54a39e7ad@ilbers.de> <5492a25b-db1f-342f-dc9f-413dd7535981@ilbers.de> From: Claudius Heine Message-ID: <827d2d4a-9905-2853-fcb3-a6fff71b66df@siemens.com> Date: Mon, 27 May 2019 10:30:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <5492a25b-db1f-342f-dc9f-413dd7535981@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: /Fb0H7mvvzyc On 27/05/2019 09.36, Maxim Yu. Osipov wrote: > Hi again Claudius, > > I think that my question in first email is still valid: > > Honestly I don't see benefits in such multiple > configure_cmds/rootfs_install_cmds in rootfs.class - > it doesn't increase code readability (as for me I was confused regarding > order of execution). > Why not collapse them into the single rootfs_install_* function? Well, it is a bit late to have that argument. The right point would have been when I posted the RFC commit over a month ago or better over two months ago when I described my design the first time. It also increase code readability, because the `setup_root_file_system` had so many special cases and parameters that it is very difficult to figure out what is exactly happening and why. And it is not much more difficult than figuring out in which order bitbake tasks are executed. It might be even easier, since those task have a deterministic order and can be easily inspected using 'bitbake -e ...'. The number of parameters and special cases of `setup_root_file_system` also shows pretty clearly that there is a need to add custom code for new features or for downstream usage in between certain steps of the root file system creation. Just putting those in multiple tasks is not possible, because they would not be idempotent. This design was inspired by openembedded [1]. So I do expect that it is battle tested. [1] https://git.openembedded.org/openembedded-core/tree/meta/classes/rootfs-postcommands.bbclass > > Regards, > Maxim. > > On 5/27/19 9:20 AM, Maxim Yu. Osipov wrote: >> Hi Claudius, >> >> On 5/27/19 8:55 AM, Claudius Heine wrote: >>> Hi Maxim, >>> >>> On 24/05/2019 14.49, Maxim Yu. Osipov wrote: >>>> Hi Claudius, >>>> >>>> One may agree that setup_root_file_system is too "fat" >>>> and should be split to logical parts. >>> >>> There are multiple issues with `setup_root_file_system`, not just it >>> being 'fat'. For instance the amount of parameters this function >>> accepts just screams to me that it does not scale necessary >>> customization. >>> >>>> My concern is that one have to carefully keep track for every >>>> ROOTFS_CONFIGURE/ROOTFS_INSTALL command's  'weight' - order of >>>> execution - possible source of errors. >>> >>> The weight parameter does not specify the order or execution, but >>> just the percentage of the progress bar shown when building. Since >>> do_rootfs_install is a pretty big task, I found it useful to show at >>> least some progress. >> >> Ah, I see now. Please add this comment into the code (to avoid >> confusion). >> >>> >>> The order of commands is the order in which they are listed in the >>> ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND array. >>> >>> Currently that is not an interface for the user, but that might >>> happen if more variables are added there to signify different stages >>> of the rootfs build process, for instance how the >>> `ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN` was done. >> >> I think that it's worth to add description of >> ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND in RECIPE_API_CHANGELOG. >> >> Regards, >> Maxim. >> >> >>> regards, >>> Claudius >>> >>>> >>>> Just a case: one my not define weight value (in such case default >>>> value is 20) as its is done for >>>> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" - so if >>>> we have several such default weights the order of commands could be >>>> wrong. >>>> >>>> Regards, >>>> Maxim. >>>> >>>> On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote: >>>>> From: Claudius Heine >>>>> >>>>> This removes the isar-bootstrap-helper.bbclass and integrates its >>>>> functionality into the rootfs.bbclass. >>>>> >>>>> It introduces the rootfs_install task that runs through multiple >>>>> functions from command lists in order to allow easy integration of >>>>> custom functions. >>>>> >>>>> The introduced functions are, in general order of execution: >>>>> >>>>>    rootfs_prepare: copy the isar-bootstrap rootfs base to workdir >>>>>    rootfs_configure_*: basic configuration of the root file system, >>>>>      before any custom packages where installed. (`/dev`, `/proc`, >>>>> etc. >>>>>      are not mounted). Functions are added to >>>>> `ROOTFS_CONFIGURE_COMMAND`. >>>>>    rootfs_do_mount: mount standard directories and repositories >>>>> into the >>>>>      rootfs >>>>>    do_rootfs_install_*: downloading and installing packages. Functions >>>>>    are added to `ROOTFS_INSTALL_COMMAND`. >>>>> >>>>> Signed-off-by: Claudius Heine >>>>> --- >>>>>   meta/classes/image.bbclass                    |  24 +-- >>>>>   meta/classes/isar-bootstrap-helper.bbclass    | 132 -------------- >>>>>   meta/classes/rootfs.bbclass                   | 172 >>>>> +++++++++++++++++- >>>>>   .../isar-bootstrap/isar-bootstrap.inc         |   5 + >>>>>   .../buildchroot/buildchroot-host.bb           |  16 +- >>>>>   .../buildchroot/buildchroot-target.bb         |   2 - >>>>>   .../buildchroot/buildchroot.inc               |  26 ++- >>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--- >>>>>   8 files changed, 228 insertions(+), 197 deletions(-) >>>>>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass >>>>> >>>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >>>>> index d77626c..99eea92 100644 >>>>> --- a/meta/classes/image.bbclass >>>>> +++ b/meta/classes/image.bbclass >>>>> @@ -51,8 +51,10 @@ image_do_mounts() { >>>>>       buildchroot_do_mounts >>>>>   } >>>>> -inherit isar-bootstrap-helper >>>>> -ROOTFS_FEATURES += "finalize-rootfs" >>>>> +ROOTFSDIR = "${IMAGE_ROOTFS}" >>>>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >>>>> finalize-rootfs" >>>>> +ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >>>>> + >>>>>   inherit rootfs >>>>>   inherit image-sdk-extension >>>>>   inherit image-cache-extension >>>>> @@ -119,8 +121,10 @@ python set_image_size () { >>>>>       d.setVarFlag('ROOTFS_SIZE', 'export', '1') >>>>>   } >>>>> -do_image_gen_fstab() { >>>>> -    cat > ${WORKDIR}/fstab << EOF >>>>> +ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab" >>>>> +image_configure_fstab[weight] = "2" >>>>> +image_configure_fstab() { >>>>> +    sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF >>>>>   # Begin /etc/fstab >>>>>   /dev/root    /        auto        defaults        0    0 >>>>>   proc        /proc        proc        nosuid,noexec,nodev    0    0 >>>>> @@ -132,18 +136,6 @@ devtmpfs    /dev        devtmpfs >>>>> mode=0755,nosuid    0    0 >>>>>   # End /etc/fstab >>>>>   EOF >>>>>   } >>>>> -addtask image_gen_fstab before do_rootfs_install >>>>> - >>>>> -do_rootfs_install[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-target:do_bootstrap" >>>>> -do_rootfs_install[deptask] = "do_deploy_deb" >>>>> -do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \ >>>>> -                             ${IMAGE_ROOTFS}/isar-apt" >>>>> -do_rootfs_install() { >>>>> -    setup_root_file_system --clean --keep-apt-cache \ >>>>> -        --fstab "${WORKDIR}/fstab" \ >>>>> -        "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL} >>>>> -} >>>>> -addtask rootfs_install before do_build after do_unpack >>>>>   do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" >>>>>   do_copy_boot_files() { >>>>> diff --git a/meta/classes/isar-bootstrap-helper.bbclass >>>>> b/meta/classes/isar-bootstrap-helper.bbclass >>>>> deleted file mode 100644 >>>>> index bbc6322..0000000 >>>>> --- a/meta/classes/isar-bootstrap-helper.bbclass >>>>> +++ /dev/null >>>>> @@ -1,132 +0,0 @@ >>>>> -# Helper functions for using isar-bootstrap >>>>> -# >>>>> -# This software is a part of ISAR. >>>>> -# Copyright (c) Siemens AG, 2018 >>>>> -# >>>>> -# SPDX-License-Identifier: MIT >>>>> - >>>>> -def reverse_bb_array(d, varname): >>>>> -    array = d.getVar(varname, True) >>>>> -    if array is None: >>>>> -        return None >>>>> -    array = reversed(array.split()) >>>>> -    return " ".join(i for i in array) >>>>> - >>>>> - >>>>> -update_etc_os_release() { >>>>> -    OS_RELEASE_BUILD_ID="" >>>>> -    OS_RELEASE_VARIANT="" >>>>> -    while true; do >>>>> -        case "$1" in >>>>> -    --build-id) OS_RELEASE_BUILD_ID=$2; shift ;; >>>>> -    --variant) OS_RELEASE_VARIANT=$2; shift ;; >>>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>>> -        *) break ;; >>>>> -        esac >>>>> -        shift >>>>> -    done >>>>> -    ROOTFSDIR="$1" >>>>> - >>>>> -    if [ -n "${OS_RELEASE_BUILD_ID}" ]; then >>>>> -        sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release >>>>> -        echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \ >>>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>>> -    fi >>>>> -    if [ -n "${OS_RELEASE_VARIANT}" ]; then >>>>> -        sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release >>>>> -        echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \ >>>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>>> -    fi >>>>> -} >>>>> - >>>>> -setup_root_file_system() { >>>>> -    CLEAN="" >>>>> -    COPYISARAPT="" >>>>> -    FSTAB="" >>>>> -    ROOTFS_ARCH="${DISTRO_ARCH}" >>>>> -    ROOTFS_DISTRO="${DISTRO}" >>>>> -    while true; do >>>>> -        case "$1" in >>>>> -        --clean) CLEAN=1 ;; >>>>> -        --copyisarapt) COPYISARAPT=1 ;; >>>>> -        --fstab) FSTAB=$2; shift ;; >>>>> -        --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;; >>>>> -        --host-distro) ROOTFS_DISTRO="$2"; shift ;; >>>>> -        --keep-apt-cache) KEEP_APT_CACHE=1 ;; >>>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>>> -        *) break ;; >>>>> -        esac >>>>> -        shift >>>>> -    done >>>>> -    ROOTFSDIR="$1" >>>>> -    shift >>>>> -    PACKAGES="$@" >>>>> -    APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>>> -    CLEAN_FILES="${ROOTFSDIR}/etc/hostname >>>>> ${ROOTFSDIR}/etc/resolv.conf" >>>>> - >>>>> -    sudo cp -Trpfx \ >>>>> -        "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \ >>>>> -        "$ROOTFSDIR" >>>>> -    [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee >>>>> "$ROOTFSDIR/etc/fstab" >>>>> - >>>>> -    echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} >>>>> main" | \ >>>>> -        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >>>>> >/dev/null >>>>> - >>>>> -    echo "Package: *\nPin: release >>>>> n=${DEBDISTRONAME}\nPin-Priority: 1000" | \ >>>>> -        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null >>>>> - >>>>> -    if [ ${COPYISARAPT} ]; then >>>>> -        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >>>>> -    else >>>>> -        sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} >>>>> $ROOTFSDIR/isar-apt >>>>> -    fi >>>>> - >>>>> -    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>> -        sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt >>>>> -    fi >>>>> - >>>>> -    sudo mount --rbind /dev ${ROOTFSDIR}/dev >>>>> -    sudo mount --make-rslave ${ROOTFSDIR}/dev >>>>> -    sudo mount -t proc none $ROOTFSDIR/proc >>>>> -    sudo mount --rbind /sys ${ROOTFSDIR}/sys >>>>> -    sudo mount --make-rslave ${ROOTFSDIR}/sys >>>>> - >>>>> -    # Refresh /etc/resolv.conf >>>>> -    sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc >>>>> - >>>>> -    # Install packages: >>>>> -    E="${@ bb.utils.export_proxies(d)}" >>>>> -    export DEBIAN_FRONTEND=noninteractive >>>>> -    # To avoid Perl locale warnings: >>>>> -    export LANG=C >>>>> -    export LANGUAGE=C >>>>> -    export LC_ALL=C >>>>> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \ >>>>> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>>> -        -o Dir::Etc::sourceparts="-" \ >>>>> -        -o APT::Get::List-Cleanup="0" >>>>> -    # Add multiarch for cross-target >>>>> -    if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then >>>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg >>>>> --add-architecture ${DISTRO_ARCH} >>>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update >>>>> -    fi >>>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>>> -        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES >>>>> -    [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES} >>>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>>> -        /usr/bin/apt-get ${APT_ARGS} $PACKAGES >>>>> -    if [ ${CLEAN} ]; then >>>>> -        if [ ${KEEP_APT_CACHE} -eq 1 ]; then >>>>> -            mkdir -p ${WORKDIR}/apt_cache >>>>> -            sudo find ${ROOTFSDIR}/var/cache/apt/archives \ >>>>> -                -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>>> ${WORKDIR}/apt_cache '{}' '+' >>>>> -            sudo chown -R $(whoami) ${WORKDIR}/apt_cache >>>>> -        fi >>>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>>> -            /usr/bin/apt-get autoremove --purge --yes >>>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>>> -            /usr/bin/apt-get clean >>>>> -        sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR" >>>>> -        sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" >>>>> "$ROOTFSDIR/var/lib/apt/lists/"* >>>>> -    fi >>>>> -} >>>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >>>>> index 45b0350..0fc8ba8 100644 >>>>> --- a/meta/classes/rootfs.bbclass >>>>> +++ b/meta/classes/rootfs.bbclass >>>>> @@ -1,14 +1,178 @@ >>>>>   # This software is a part of ISAR. >>>>>   # Copyright (c) Siemens AG, 2019 >>>>> +ROOTFS_ARCH ?= "${DISTRO_ARCH}" >>>>> +ROOTFS_DISTRO ?= "${DISTRO}" >>>>> +ROOTFS_PACKAGES ?= "" >>>>> + >>>>>   # Features of the rootfs creation: >>>>>   # available features are: >>>>> +# 'deploy-package-cache' - copy the package cache >>>>> ${WORKDIR}/apt_cache >>>>> +# 'clean-package-cache' - delete package cache from rootfs >>>>>   # 'finalize-rootfs' - delete files needed to chroot into the rootfs >>>>>   ROOTFS_FEATURES ?= "" >>>>> +ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>>> + >>>>> +ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" >>>>> + >>>>> +# Useful environment variables: >>>>> +export E = "${@ bb.utils.export_proxies(d)}" >>>>> +export DEBIAN_FRONTEND = "noninteractive" >>>>> +# To avoid Perl locale warnings: >>>>> +export LANG = "C" >>>>> +export LANGUAGE = "C" >>>>> +export LC_ALL = "C" >>>>> + >>>>> +rootfs_do_mounts[weight] = "3" >>>>> +rootfs_do_mounts() { >>>>> +    sudo -s <<'EOSUDO' >>>>> +        mountpoint -q '${ROOTFSDIR}/dev' || \ >>>>> +            mount --rbind /dev '${ROOTFSDIR}/dev' >>>>> +        mount --make-rslave '${ROOTFSDIR}/dev' >>>>> +        mountpoint -q '${ROOTFSDIR}/proc' || \ >>>>> +            mount -t proc none '${ROOTFSDIR}/proc' >>>>> +        mountpoint -q '${ROOTFSDIR}/sys' || \ >>>>> +            mount --rbind /sys '${ROOTFSDIR}/sys' >>>>> +        mount --make-rslave '${ROOTFSDIR}/sys' >>>>> + >>>>> +        # Mount isar-apt if the directory does not exist or if it >>>>> is empty >>>>> +        # This prevents overwriting something that was copied there >>>>> +        if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ >>>>> +           [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth >>>>> 1 | wc -l)" = "0" ] >>>>> +        then >>>>> +            mkdir -p '${ROOTFSDIR}/isar-apt' >>>>> +            mountpoint -q '${ROOTFSDIR}/isar-apt' || \ >>>>> +                mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >>>>> '${ROOTFSDIR}/isar-apt' >>>>> +        fi >>>>> + >>>>> +        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set >>>>> +        if [ >>>>> "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" >>>>> = 'True' ] >>>>> +        then >>>>> +            mkdir -p '${ROOTFSDIR}/base-apt' >>>>> +            mountpoint -q '${ROOTFSDIR}/base-apt' || \ >>>>> +                mount --bind '${REPO_BASE_DIR}' >>>>> '${ROOTFSDIR}/base-apt' >>>>> +        fi >>>>> + >>>>> +EOSUDO >>>>> +} >>>>> + >>>>> +rootfs_do_qemu() { >>>>> +    if [ '${@repr(d.getVar('ROOTFS_ARCH') == >>>>> d.getVar('HOST_ARCH'))}' = 'False' ] >>>>> +    then >>>>> +        test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \ >>>>> +            sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' >>>>> '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' >>>>> +    fi >>>>> +} >>>>> + >>>>> +BOOTSTRAP_SRC = >>>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}" >>>>> >>>>> +BOOTSTRAP_SRC_${ROOTFS_ARCH} = >>>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" >>>>> + >>>>> +rootfs_prepare[weight] = "25" >>>>> +rootfs_prepare(){ >>>>> +    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' >>>>> +} >>>>> + >>>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" >>>>> +rootfs_configure_isar_apt[weight] = "2" >>>>> +rootfs_configure_isar_apt() { >>>>> +    sudo -s <<'EOSUDO' >>>>> + >>>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' >>>>> +    echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} >>>>> main' > \ >>>>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' >>>>> + >>>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' >>>>> +    cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar' >>>>> +Package: * >>>>> +Pin: release n=${DEBDISTRONAME} >>>>> +Pin-Priority: 1000 >>>>> +EOF >>>>> +EOSUDO >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" >>>>> +rootfs_install_pkgs_update[weight] = "5" >>>>> +rootfs_install_pkgs_update() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ >>>>> +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>>> +        -o Dir::Etc::sourceparts="-" \ >>>>> +        -o APT::Get::List-Cleanup="0" >>>>> +} >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf" >>>>> +rootfs_install_resolvconf[weight] = "1" >>>>> +rootfs_install_resolvconf() { >>>>> +    sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc' >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" >>>>> +rootfs_install_pkgs_download[weight] = "600" >>>>> +rootfs_install_pkgs_download() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only >>>>> ${ROOTFS_PACKAGES} >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if >>>>> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" >>>>> +rootfs_install_clean_files[weight] = "2" >>>>> +rootfs_install_clean_files() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /bin/rm -f ${ROOTFS_CLEAN_FILES} >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install" >>>>> +rootfs_install_pkgs_install[weight] = "8000" >>>>> +rootfs_install_pkgs_install() { >>>>> +    sudo -E chroot "${ROOTFSDIR}" \ >>>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} >>>>> +} >>>> >>>> >>>> Honestly I don't see benefits in such multiple >>>> configure_cmds/rootfs_install_cmds - it doesn't increase code >>>> readability as one have to additionally keep track for every command >>>> 'weight' value and it's easy to miss the correct order of commands >>>> execution. >>>> >>>> >>>>> +do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" >>>>> +do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} >>>>> ${ROOTFS_INSTALL_COMMAND}" >>>>> +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if >>>>> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else >>>>> 'host'}:do_build isar-apt:do_cache_config" >>>>> +do_rootfs_install[deptask] = "do_deploy_deb" >>>>> +python do_rootfs_install() { >>>>> +    configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) >>>>> or "").split() >>>>> +    install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or >>>>> "").split() >>>>> + >>>>> +    # Mount after configure commands, so that they have time to copy >>>>> +    # 'isar-apt' (sdkchroot): >>>>> +    cmds = ['rootfs_prepare'] + configure_cmds + >>>>> ['rootfs_do_mounts'] + install_cmds >>>>> + >>>>> +    stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20") >>>>> +                     for i in cmds] >>>>> + >>>>> +    progress_reporter = bb.progress.MultiStageProgressReporter(d, >>>>> stage_weights) >>>>> + >>>>> +    for cmd in cmds: >>>>> +        progress_reporter.next_stage() >>>>> +        bb.build.exec_func(cmd, d) >>>>> +    progress_reporter.finish() >>>>> +} >>>>> +addtask rootfs_install before do_rootfs_postprocess after do_unpack >>>>> + >>>>> +ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', >>>>> 'rootfs_postprocess_copy_package_cache', '', d)}" >>>>> +rootfs_postprocess_copy_package_cache() { >>>>> +    mkdir -p '${WORKDIR}/apt_cache' >>>>> +    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \ >>>>> +        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>>> '${WORKDIR}/apt_cache' '{}' '+' >>>>> +    me="$(id -u):$(id -g)" >>>>> +    sudo chown -R "$me" '${WORKDIR}/apt_cache' >>>>> +} >>>>> + >>>>> +ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', >>>>> 'rootfs_postprocess_clean_package_cache', '', d)}" >>>>> +rootfs_postprocess_clean_package_cache() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /usr/bin/apt-get clean >>>>> +    sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >>>>> +} >>>>> + >>>>>   ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', >>>>> 'rootfs_postprocess_finalize', '', d)}" >>>>>   rootfs_postprocess_finalize() { >>>>>       sudo -s <<'EOSUDO' >>>>> +        test -e "${ROOTFSDIR}/chroot-setup.sh" && \ >>>>> +            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}" >>>>> +        rm -f "${ROOTFSDIR}/chroot-setup.sh" >>>>> + >>>>>           test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \ >>>>>               find "${ROOTFSDIR}/usr/bin" \ >>>>>                   -maxdepth 1 -name 'qemu-*-static' -type f -delete >>>>> @@ -42,6 +206,12 @@ EOSUDO >>>>>   do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >>>>>   python do_rootfs_postprocess() { >>>>> +    # Take care that its correctly mounted: >>>>> +    bb.build.exec_func('rootfs_do_mounts', d) >>>>> +    # Take care that qemu-*-static is available, since it could >>>>> have been >>>>> +    # removed on a previous execution of this task: >>>>> +    bb.build.exec_func('rootfs_do_qemu', d) >>>>> + >>>>>       cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND") >>>>>       if cmds is None or not cmds.strip(): >>>>>           return >>>>> @@ -49,7 +219,7 @@ python do_rootfs_postprocess() { >>>>>       for cmd in cmds: >>>>>           bb.build.exec_func(cmd, d) >>>>>   } >>>>> -addtask rootfs_postprocess before do_rootfs after do_rootfs_install >>>>> +addtask rootfs_postprocess before do_rootfs >>>>>   python do_rootfs() { >>>>>       """Virtual task""" >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> index b740c57..f8741c2 100644 >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> @@ -327,6 +327,11 @@ isar_bootstrap() { >>>>>               mount --make-rslave ${ROOTFSDIR}/sys >>>>>               export DEBIAN_FRONTEND=noninteractive >>>>> + >>>>> +            if [ ${IS_HOST} ]; then >>>>> +                chroot "${ROOTFSDIR}" /usr/bin/dpkg >>>>> --add-architecture ${DISTRO_ARCH} >>>>> +            fi >>>>> + >>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ >>>>>                                     -o Debug::pkgProblemResolver=yes >>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> index 73cd548..121992c 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> @@ -7,6 +7,8 @@ DESCRIPTION = "Isar development filesystem for host" >>>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>>>   require buildchroot.inc >>>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>>>   BUILDCHROOT_PREINSTALL ?= "make \ >>>>>                              debhelper \ >>>>> @@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \ >>>>>                              automake \ >>>>>                              devscripts \ >>>>>                              equivs \ >>>>> -                           libc6:${DISTRO_ARCH}" >>>>> - >>>>> -# According to the wiki page: >>>>> -#     https://wiki.debian.org/CrossToolchains >>>>> -BUILDCHROOT_PREINSTALL_append_armhf += >>>>> "binutils-arm-linux-gnueabihf \ >>>>> -                                        crossbuild-essential-armhf" >>>>> -BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \ >>>>> -                                        crossbuild-essential-arm64" >>>>> - >>>>> - >>>>> -PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'" >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-host:do_bootstrap" >>>>> +                           libc6:${DISTRO_ARCH} \ >>>>> +                           crossbuild-essential-${DISTRO_ARCH}" >>>>> diff --git >>>>> a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> index 20f4d23..5a01258 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> @@ -21,5 +21,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>>>>                              devscripts \ >>>>>                              equivs \ >>>>>                              adduser" >>>>> - >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-target:do_bootstrap" >>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> index ca9ce51..40fc8b0 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> @@ -13,18 +13,25 @@ SRC_URI = "file://configscript.sh \ >>>>>              file://deps.sh" >>>>>   PV = "1.0" >>>>> -inherit isar-bootstrap-helper >>>>> +inherit rootfs >>>>>   BUILDCHROOT_DIR = "${WORKDIR}/rootfs" >>>>> +ROOTFSDIR = "${BUILDCHROOT_DIR}" >>>>> +ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}" >>>>> +ROOTFS_CLEAN_FILES = "" >>>>> + >>>>> +rootfs_do_mounts_append() { >>>>> +    sudo -s <<'EOSUDO' >>>>> +    mkdir -p '${BUILDCHROOT_DIR}/downloads' >>>>> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ >>>>> +        mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' >>>>> +EOSUDO >>>>> +} >>>>> -do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \ >>>>> -                            ${BUILDCHROOT_DIR}/isar-apt \ >>>>> -                            ${BUILDCHROOT_DIR}/downloads \ >>>>> -                            ${BUILDCHROOT_DIR}/home/builder" >>>>> - >>>>> -do_build() { >>>>> -    setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} >>>>> ${BUILDCHROOT_PREINSTALL} >>>>> - >>>>> +ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" >>>>> +buildchroot_install_files() { >>>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder" >>>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt" >>>>>       # Install package builder script >>>>>       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" >>>>>       sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar >>>>> @@ -38,5 +45,4 @@ do_build() { >>>>>       GROUP_ID=$(id -g) >>>>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID >>>>> $GROUP_ID >>>>> -    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads >>>>>   } >>>>> diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> index 49aeb47..57d52ba 100644 >>>>> --- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> +++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> @@ -12,10 +12,25 @@ SRC_URI = " \ >>>>>       file://configscript.sh \ >>>>>       file://README.sdk" >>>>>   PV = "0.1" >>>>> - >>>>> -inherit isar-bootstrap-helper >>>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>>> +TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}" >>>>> +TOOLCHAIN_${HOST_ARCH} = "build-essential" >>>>> +TOOLCHAIN_i386 = "build-essential" >>>>> + >>>>> +inherit rootfs >>>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>>> +ROOTFSDIR = "${S}" >>>>> +ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}" >>>>> +ROOTFS_FEATURES += "copy-package-cache" >>>>> + >>>>> +python() { >>>>> +    if d.getVar("HOST_ARCH") not in ['i386', 'amd64']: >>>>> +        raise bb.parse.SkipRecipe("SDK doesn't support {} as >>>>> host".format( >>>>> +            d.getVar("ROOTFS_ARCH"))) >>>>> +} >>>>> + >>>>>   SDKCHROOT_PREINSTALL := "debhelper \ >>>>>                              autotools-dev \ >>>>>                              dpkg \ >>>>> @@ -28,29 +43,14 @@ SDKCHROOT_PREINSTALL := "debhelper \ >>>>>   S = "${WORKDIR}/rootfs" >>>>> -do_build[dirs] = "${DEPLOY_DIR_IMAGE}" >>>>> -do_build[root_cleandirs] = "${S} \ >>>>> -                            ${S}/isar-apt" >>>>> - >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-host:do_bootstrap" >>>>> - >>>>> -do_build() { >>>>> - >>>>> -    if [ ${HOST_DISTRO} != "debian-stretch" ]; then >>>>> -        bbfatal "SDK doesn't support ${HOST_DISTRO}" >>>>> -    fi >>>>> -    if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then >>>>> -        bbfatal "SDK doesn't support ${HOST_ARCH} as host" >>>>> -    fi >>>>> - >>>>> -    if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" >>>>> ]; then >>>>> -        packages="${SDKCHROOT_PREINSTALL} build-essential" >>>>> -    else >>>>> -        packages="${SDKCHROOT_PREINSTALL} >>>>> crossbuild-essential-${DISTRO_ARCH}" >>>>> -    fi >>>>> - >>>>> -    setup_root_file_system --copyisarapt --host-arch >>>>> '${HOST_ARCH}' --host-distro '${HOST_DISTRO}' "${S}" $packages >>>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" >>>>> +rootfs_configure_isar_apt_dir() { >>>>> +    # Copy isar-apt instead of mounting: >>>>> +    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt >>>>> +} >>>>> +ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files" >>>>> +sdkchroot_install_files() { >>>>>       # Configure root filesystem >>>>>       sudo install -m 644 ${WORKDIR}/README.sdk ${S} >>>>>       sudo install -m 755 ${WORKDIR}/configscript.sh ${S} >>>>> >>>> >>>> >>> >> >> > > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de