From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6691586504498610176 X-Received: by 2002:a5d:4692:: with SMTP id u18mr61238435wrq.285.1558941640737; Mon, 27 May 2019 00:20:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:edce:: with SMTP id v14ls160270wro.4.gmail; Mon, 27 May 2019 00:20:40 -0700 (PDT) X-Google-Smtp-Source: APXvYqytn7MlxbEh2VRcj0jA6BTc3bH83bpRlE9yTXbO6veQUHUGZzf1a9towDKt+IZ0qP11pCDi X-Received: by 2002:adf:aa09:: with SMTP id p9mr20148392wrd.59.1558941640191; Mon, 27 May 2019 00:20:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558941640; cv=none; d=google.com; s=arc-20160816; b=JJ1jKghcu1XkNP/i5+I0d9FnM1nnvd9Sh2VRq6QIi02NT9ACatKOh8748adxdKPm4k UM99f7yaWIh8DdvWN/V3H1xlpuKn+85gNbSyKiJnVyYtMkdzb4z8aMjYh/0+RIKdKeo7 /psgvgT9MvHqnyKsCBU4qjaK+c22cRz4jSWr08T4prltKpwFDHnLG0QEevxOo1ZGrYe+ doWrtmZKwpRY1QYs/Zf9bhZ9rEUEm7d4oxyVjUwhkXtcfAOB0yoyb3uPP30UWsUjujkj ODI0qMMDsB3ZosDYKKnn99hZAkyKXI+buPupAbXzJ8WLLNONVx2oBnIBW1wJZ0v5rQHg acpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:cc:to :subject; bh=p9UP85sld2XcII6ilesu0RQ5isaxkL2susN+WObZ2+M=; b=s2MbMbdkPzsBsI9V8o97DiczAcgTot9LZ/g7b88w5bme/P8WI10uvpTju0MFNs/Oot Y1ErxRcc9AvOr2H1R2ytIL+6XEVRQzjnr8c3Ve3WsbyQWAYuHC9Se7V515cxCuQH6nUs nP06MJDkCOst22KbgQtc5W65gsrzju4CvOx7FahacidxGYKRNS+fYTkDYGXZNcyp8TGu BZHHLrlRQyRnn1//BSdilpgwGzDvGTQEnGjRB61CMbPaEcA7eaD4uEqzGRFsVOrA+fO5 mru3rfvZ4eb6QkvcUu2nd/FlHB8dMG2iCMzLaY3U8+uwc7bZ+mEDqV0U1Rcu35HUqSqg UsWg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id y139si409299wmd.0.2019.05.27.00.20.40 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 May 2019 00:20:40 -0700 (PDT) Received-SPF: pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.1.29] (195.165-131-109.adsl-dyn.isp.belgacom.be [109.131.165.195] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x4R7KYNh024896 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 May 2019 09:20:36 +0200 Subject: Re: [PATCH v4 2/8] split up isar-bootstrap helper and implement pre-process pipeline To: Claudius Heine , isar-users@googlegroups.com Cc: Claudius Heine References: <20190523145521.23050-1-claudius.heine.ext@siemens.com> <20190523145521.23050-3-claudius.heine.ext@siemens.com> <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: <18fb7e43-c5ed-f007-30c1-4be54a39e7ad@ilbers.de> Date: Mon, 27 May 2019 09:20:30 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: NVVOC2407UYq Hi Claudius, On 5/27/19 8:55 AM, Claudius Heine wrote: > Hi Maxim, > > On 24/05/2019 14.49, Maxim Yu. Osipov wrote: >> Hi Claudius, >> >> One may agree that setup_root_file_system is too "fat" >> and should be split to logical parts. > > There are multiple issues with `setup_root_file_system`, not just it > being 'fat'. For instance the amount of parameters this function accepts > just screams to me that it does not scale necessary customization. > >> My concern is that one have to carefully keep track for every >> ROOTFS_CONFIGURE/ROOTFS_INSTALL command's  'weight' - order of >> execution - possible source of errors. > > The weight parameter does not specify the order or execution, but just > the percentage of the progress bar shown when building. Since > do_rootfs_install is a pretty big task, I found it useful to show at > least some progress. Ah, I see now. Please add this comment into the code (to avoid confusion). > > The order of commands is the order in which they are listed in the > ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND array. > > Currently that is not an interface for the user, but that might happen > if more variables are added there to signify different stages of the > rootfs build process, for instance how the > `ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN` was done. I think that it's worth to add description of ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND in RECIPE_API_CHANGELOG. Regards, Maxim. > regards, > Claudius > >> >> Just a case: one my not define weight value (in such case default >> value is 20) as its is done for >> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" - so if we >> have several such default weights the order of commands could be wrong. >> >> Regards, >> Maxim. >> >> On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote: >>> From: Claudius Heine >>> >>> This removes the isar-bootstrap-helper.bbclass and integrates its >>> functionality into the rootfs.bbclass. >>> >>> It introduces the rootfs_install task that runs through multiple >>> functions from command lists in order to allow easy integration of >>> custom functions. >>> >>> The introduced functions are, in general order of execution: >>> >>>    rootfs_prepare: copy the isar-bootstrap rootfs base to workdir >>>    rootfs_configure_*: basic configuration of the root file system, >>>      before any custom packages where installed. (`/dev`, `/proc`, etc. >>>      are not mounted). Functions are added to >>> `ROOTFS_CONFIGURE_COMMAND`. >>>    rootfs_do_mount: mount standard directories and repositories into the >>>      rootfs >>>    do_rootfs_install_*: downloading and installing packages. Functions >>>    are added to `ROOTFS_INSTALL_COMMAND`. >>> >>> Signed-off-by: Claudius Heine >>> --- >>>   meta/classes/image.bbclass                    |  24 +-- >>>   meta/classes/isar-bootstrap-helper.bbclass    | 132 -------------- >>>   meta/classes/rootfs.bbclass                   | 172 +++++++++++++++++- >>>   .../isar-bootstrap/isar-bootstrap.inc         |   5 + >>>   .../buildchroot/buildchroot-host.bb           |  16 +- >>>   .../buildchroot/buildchroot-target.bb         |   2 - >>>   .../buildchroot/buildchroot.inc               |  26 ++- >>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--- >>>   8 files changed, 228 insertions(+), 197 deletions(-) >>>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass >>> >>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >>> index d77626c..99eea92 100644 >>> --- a/meta/classes/image.bbclass >>> +++ b/meta/classes/image.bbclass >>> @@ -51,8 +51,10 @@ image_do_mounts() { >>>       buildchroot_do_mounts >>>   } >>> -inherit isar-bootstrap-helper >>> -ROOTFS_FEATURES += "finalize-rootfs" >>> +ROOTFSDIR = "${IMAGE_ROOTFS}" >>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >>> finalize-rootfs" >>> +ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >>> + >>>   inherit rootfs >>>   inherit image-sdk-extension >>>   inherit image-cache-extension >>> @@ -119,8 +121,10 @@ python set_image_size () { >>>       d.setVarFlag('ROOTFS_SIZE', 'export', '1') >>>   } >>> -do_image_gen_fstab() { >>> -    cat > ${WORKDIR}/fstab << EOF >>> +ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab" >>> +image_configure_fstab[weight] = "2" >>> +image_configure_fstab() { >>> +    sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF >>>   # Begin /etc/fstab >>>   /dev/root    /        auto        defaults        0    0 >>>   proc        /proc        proc        nosuid,noexec,nodev    0    0 >>> @@ -132,18 +136,6 @@ devtmpfs    /dev        devtmpfs >>> mode=0755,nosuid    0    0 >>>   # End /etc/fstab >>>   EOF >>>   } >>> -addtask image_gen_fstab before do_rootfs_install >>> - >>> -do_rootfs_install[depends] = "isar-apt:do_cache_config >>> isar-bootstrap-target:do_bootstrap" >>> -do_rootfs_install[deptask] = "do_deploy_deb" >>> -do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \ >>> -                             ${IMAGE_ROOTFS}/isar-apt" >>> -do_rootfs_install() { >>> -    setup_root_file_system --clean --keep-apt-cache \ >>> -        --fstab "${WORKDIR}/fstab" \ >>> -        "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL} >>> -} >>> -addtask rootfs_install before do_build after do_unpack >>>   do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" >>>   do_copy_boot_files() { >>> diff --git a/meta/classes/isar-bootstrap-helper.bbclass >>> b/meta/classes/isar-bootstrap-helper.bbclass >>> deleted file mode 100644 >>> index bbc6322..0000000 >>> --- a/meta/classes/isar-bootstrap-helper.bbclass >>> +++ /dev/null >>> @@ -1,132 +0,0 @@ >>> -# Helper functions for using isar-bootstrap >>> -# >>> -# This software is a part of ISAR. >>> -# Copyright (c) Siemens AG, 2018 >>> -# >>> -# SPDX-License-Identifier: MIT >>> - >>> -def reverse_bb_array(d, varname): >>> -    array = d.getVar(varname, True) >>> -    if array is None: >>> -        return None >>> -    array = reversed(array.split()) >>> -    return " ".join(i for i in array) >>> - >>> - >>> -update_etc_os_release() { >>> -    OS_RELEASE_BUILD_ID="" >>> -    OS_RELEASE_VARIANT="" >>> -    while true; do >>> -        case "$1" in >>> -    --build-id) OS_RELEASE_BUILD_ID=$2; shift ;; >>> -    --variant) OS_RELEASE_VARIANT=$2; shift ;; >>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>> -        *) break ;; >>> -        esac >>> -        shift >>> -    done >>> -    ROOTFSDIR="$1" >>> - >>> -    if [ -n "${OS_RELEASE_BUILD_ID}" ]; then >>> -        sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release >>> -        echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \ >>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>> -    fi >>> -    if [ -n "${OS_RELEASE_VARIANT}" ]; then >>> -        sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release >>> -        echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \ >>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>> -    fi >>> -} >>> - >>> -setup_root_file_system() { >>> -    CLEAN="" >>> -    COPYISARAPT="" >>> -    FSTAB="" >>> -    ROOTFS_ARCH="${DISTRO_ARCH}" >>> -    ROOTFS_DISTRO="${DISTRO}" >>> -    while true; do >>> -        case "$1" in >>> -        --clean) CLEAN=1 ;; >>> -        --copyisarapt) COPYISARAPT=1 ;; >>> -        --fstab) FSTAB=$2; shift ;; >>> -        --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;; >>> -        --host-distro) ROOTFS_DISTRO="$2"; shift ;; >>> -        --keep-apt-cache) KEEP_APT_CACHE=1 ;; >>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>> -        *) break ;; >>> -        esac >>> -        shift >>> -    done >>> -    ROOTFSDIR="$1" >>> -    shift >>> -    PACKAGES="$@" >>> -    APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>> -    CLEAN_FILES="${ROOTFSDIR}/etc/hostname >>> ${ROOTFSDIR}/etc/resolv.conf" >>> - >>> -    sudo cp -Trpfx \ >>> -        "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \ >>> -        "$ROOTFSDIR" >>> -    [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee "$ROOTFSDIR/etc/fstab" >>> - >>> -    echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main" | \ >>> -        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >>> >/dev/null >>> - >>> -    echo "Package: *\nPin: release n=${DEBDISTRONAME}\nPin-Priority: >>> 1000" | \ >>> -        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null >>> - >>> -    if [ ${COPYISARAPT} ]; then >>> -        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >>> -    else >>> -        sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} >>> $ROOTFSDIR/isar-apt >>> -    fi >>> - >>> -    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>> -        sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt >>> -    fi >>> - >>> -    sudo mount --rbind /dev ${ROOTFSDIR}/dev >>> -    sudo mount --make-rslave ${ROOTFSDIR}/dev >>> -    sudo mount -t proc none $ROOTFSDIR/proc >>> -    sudo mount --rbind /sys ${ROOTFSDIR}/sys >>> -    sudo mount --make-rslave ${ROOTFSDIR}/sys >>> - >>> -    # Refresh /etc/resolv.conf >>> -    sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc >>> - >>> -    # Install packages: >>> -    E="${@ bb.utils.export_proxies(d)}" >>> -    export DEBIAN_FRONTEND=noninteractive >>> -    # To avoid Perl locale warnings: >>> -    export LANG=C >>> -    export LANGUAGE=C >>> -    export LC_ALL=C >>> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \ >>> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>> -        -o Dir::Etc::sourceparts="-" \ >>> -        -o APT::Get::List-Cleanup="0" >>> -    # Add multiarch for cross-target >>> -    if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then >>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture >>> ${DISTRO_ARCH} >>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update >>> -    fi >>> -    sudo -E chroot "$ROOTFSDIR" \ >>> -        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES >>> -    [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES} >>> -    sudo -E chroot "$ROOTFSDIR" \ >>> -        /usr/bin/apt-get ${APT_ARGS} $PACKAGES >>> -    if [ ${CLEAN} ]; then >>> -        if [ ${KEEP_APT_CACHE} -eq 1 ]; then >>> -            mkdir -p ${WORKDIR}/apt_cache >>> -            sudo find ${ROOTFSDIR}/var/cache/apt/archives \ >>> -                -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>> ${WORKDIR}/apt_cache '{}' '+' >>> -            sudo chown -R $(whoami) ${WORKDIR}/apt_cache >>> -        fi >>> -        sudo -E chroot "$ROOTFSDIR" \ >>> -            /usr/bin/apt-get autoremove --purge --yes >>> -        sudo -E chroot "$ROOTFSDIR" \ >>> -            /usr/bin/apt-get clean >>> -        sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR" >>> -        sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" >>> "$ROOTFSDIR/var/lib/apt/lists/"* >>> -    fi >>> -} >>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >>> index 45b0350..0fc8ba8 100644 >>> --- a/meta/classes/rootfs.bbclass >>> +++ b/meta/classes/rootfs.bbclass >>> @@ -1,14 +1,178 @@ >>>   # This software is a part of ISAR. >>>   # Copyright (c) Siemens AG, 2019 >>> +ROOTFS_ARCH ?= "${DISTRO_ARCH}" >>> +ROOTFS_DISTRO ?= "${DISTRO}" >>> +ROOTFS_PACKAGES ?= "" >>> + >>>   # Features of the rootfs creation: >>>   # available features are: >>> +# 'deploy-package-cache' - copy the package cache ${WORKDIR}/apt_cache >>> +# 'clean-package-cache' - delete package cache from rootfs >>>   # 'finalize-rootfs' - delete files needed to chroot into the rootfs >>>   ROOTFS_FEATURES ?= "" >>> +ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>> + >>> +ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" >>> + >>> +# Useful environment variables: >>> +export E = "${@ bb.utils.export_proxies(d)}" >>> +export DEBIAN_FRONTEND = "noninteractive" >>> +# To avoid Perl locale warnings: >>> +export LANG = "C" >>> +export LANGUAGE = "C" >>> +export LC_ALL = "C" >>> + >>> +rootfs_do_mounts[weight] = "3" >>> +rootfs_do_mounts() { >>> +    sudo -s <<'EOSUDO' >>> +        mountpoint -q '${ROOTFSDIR}/dev' || \ >>> +            mount --rbind /dev '${ROOTFSDIR}/dev' >>> +        mount --make-rslave '${ROOTFSDIR}/dev' >>> +        mountpoint -q '${ROOTFSDIR}/proc' || \ >>> +            mount -t proc none '${ROOTFSDIR}/proc' >>> +        mountpoint -q '${ROOTFSDIR}/sys' || \ >>> +            mount --rbind /sys '${ROOTFSDIR}/sys' >>> +        mount --make-rslave '${ROOTFSDIR}/sys' >>> + >>> +        # Mount isar-apt if the directory does not exist or if it is >>> empty >>> +        # This prevents overwriting something that was copied there >>> +        if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ >>> +           [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth 1 >>> | wc -l)" = "0" ] >>> +        then >>> +            mkdir -p '${ROOTFSDIR}/isar-apt' >>> +            mountpoint -q '${ROOTFSDIR}/isar-apt' || \ >>> +                mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >>> '${ROOTFSDIR}/isar-apt' >>> +        fi >>> + >>> +        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set >>> +        if [ >>> "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = >>> 'True' ] >>> +        then >>> +            mkdir -p '${ROOTFSDIR}/base-apt' >>> +            mountpoint -q '${ROOTFSDIR}/base-apt' || \ >>> +                mount --bind '${REPO_BASE_DIR}' '${ROOTFSDIR}/base-apt' >>> +        fi >>> + >>> +EOSUDO >>> +} >>> + >>> +rootfs_do_qemu() { >>> +    if [ '${@repr(d.getVar('ROOTFS_ARCH') == >>> d.getVar('HOST_ARCH'))}' = 'False' ] >>> +    then >>> +        test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \ >>> +            sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' >>> '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' >>> +    fi >>> +} >>> + >>> +BOOTSTRAP_SRC = >>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}" >>> +BOOTSTRAP_SRC_${ROOTFS_ARCH} = >>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" >>> + >>> +rootfs_prepare[weight] = "25" >>> +rootfs_prepare(){ >>> +    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' >>> +} >>> + >>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" >>> +rootfs_configure_isar_apt[weight] = "2" >>> +rootfs_configure_isar_apt() { >>> +    sudo -s <<'EOSUDO' >>> + >>> +    mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' >>> +    echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \ >>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' >>> + >>> +    mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' >>> +    cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar' >>> +Package: * >>> +Pin: release n=${DEBDISTRONAME} >>> +Pin-Priority: 1000 >>> +EOF >>> +EOSUDO >>> +} >>> + >>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" >>> +rootfs_install_pkgs_update[weight] = "5" >>> +rootfs_install_pkgs_update() { >>> +    sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ >>> +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>> +        -o Dir::Etc::sourceparts="-" \ >>> +        -o APT::Get::List-Cleanup="0" >>> +} >>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf" >>> +rootfs_install_resolvconf[weight] = "1" >>> +rootfs_install_resolvconf() { >>> +    sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc' >>> +} >>> + >>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" >>> +rootfs_install_pkgs_download[weight] = "600" >>> +rootfs_install_pkgs_download() { >>> +    sudo -E chroot '${ROOTFSDIR}' \ >>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only >>> ${ROOTFS_PACKAGES} >>> +} >>> + >>> +ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if >>> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" >>> +rootfs_install_clean_files[weight] = "2" >>> +rootfs_install_clean_files() { >>> +    sudo -E chroot '${ROOTFSDIR}' \ >>> +        /bin/rm -f ${ROOTFS_CLEAN_FILES} >>> +} >>> + >>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install" >>> +rootfs_install_pkgs_install[weight] = "8000" >>> +rootfs_install_pkgs_install() { >>> +    sudo -E chroot "${ROOTFSDIR}" \ >>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} >>> +} >> >> >> Honestly I don't see benefits in such multiple >> configure_cmds/rootfs_install_cmds - it doesn't increase code >> readability as one have to additionally keep track for every command >> 'weight' value and it's easy to miss the correct order of commands >> execution. >> >> >>> +do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" >>> +do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} >>> ${ROOTFS_INSTALL_COMMAND}" >>> +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if >>> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else >>> 'host'}:do_build isar-apt:do_cache_config" >>> +do_rootfs_install[deptask] = "do_deploy_deb" >>> +python do_rootfs_install() { >>> +    configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or >>> "").split() >>> +    install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or >>> "").split() >>> + >>> +    # Mount after configure commands, so that they have time to copy >>> +    # 'isar-apt' (sdkchroot): >>> +    cmds = ['rootfs_prepare'] + configure_cmds + >>> ['rootfs_do_mounts'] + install_cmds >>> + >>> +    stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20") >>> +                     for i in cmds] >>> + >>> +    progress_reporter = bb.progress.MultiStageProgressReporter(d, >>> stage_weights) >>> + >>> +    for cmd in cmds: >>> +        progress_reporter.next_stage() >>> +        bb.build.exec_func(cmd, d) >>> +    progress_reporter.finish() >>> +} >>> +addtask rootfs_install before do_rootfs_postprocess after do_unpack >>> + >>> +ROOTFS_POSTPROCESS_COMMAND += >>> "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', >>> 'rootfs_postprocess_copy_package_cache', '', d)}" >>> +rootfs_postprocess_copy_package_cache() { >>> +    mkdir -p '${WORKDIR}/apt_cache' >>> +    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \ >>> +        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>> '${WORKDIR}/apt_cache' '{}' '+' >>> +    me="$(id -u):$(id -g)" >>> +    sudo chown -R "$me" '${WORKDIR}/apt_cache' >>> +} >>> + >>> +ROOTFS_POSTPROCESS_COMMAND += >>> "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', >>> 'rootfs_postprocess_clean_package_cache', '', d)}" >>> +rootfs_postprocess_clean_package_cache() { >>> +    sudo -E chroot '${ROOTFSDIR}' \ >>> +        /usr/bin/apt-get clean >>> +    sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >>> +} >>> + >>>   ROOTFS_POSTPROCESS_COMMAND += >>> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', >>> 'rootfs_postprocess_finalize', '', d)}" >>>   rootfs_postprocess_finalize() { >>>       sudo -s <<'EOSUDO' >>> +        test -e "${ROOTFSDIR}/chroot-setup.sh" && \ >>> +            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}" >>> +        rm -f "${ROOTFSDIR}/chroot-setup.sh" >>> + >>>           test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \ >>>               find "${ROOTFSDIR}/usr/bin" \ >>>                   -maxdepth 1 -name 'qemu-*-static' -type f -delete >>> @@ -42,6 +206,12 @@ EOSUDO >>>   do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >>>   python do_rootfs_postprocess() { >>> +    # Take care that its correctly mounted: >>> +    bb.build.exec_func('rootfs_do_mounts', d) >>> +    # Take care that qemu-*-static is available, since it could have >>> been >>> +    # removed on a previous execution of this task: >>> +    bb.build.exec_func('rootfs_do_qemu', d) >>> + >>>       cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND") >>>       if cmds is None or not cmds.strip(): >>>           return >>> @@ -49,7 +219,7 @@ python do_rootfs_postprocess() { >>>       for cmd in cmds: >>>           bb.build.exec_func(cmd, d) >>>   } >>> -addtask rootfs_postprocess before do_rootfs after do_rootfs_install >>> +addtask rootfs_postprocess before do_rootfs >>>   python do_rootfs() { >>>       """Virtual task""" >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> index b740c57..f8741c2 100644 >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> @@ -327,6 +327,11 @@ isar_bootstrap() { >>>               mount --make-rslave ${ROOTFSDIR}/sys >>>               export DEBIAN_FRONTEND=noninteractive >>> + >>> +            if [ ${IS_HOST} ]; then >>> +                chroot "${ROOTFSDIR}" /usr/bin/dpkg >>> --add-architecture ${DISTRO_ARCH} >>> +            fi >>> + >>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ >>>                                     -o Debug::pkgProblemResolver=yes >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>> b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>> index 73cd548..121992c 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>> @@ -7,6 +7,8 @@ DESCRIPTION = "Isar development filesystem for host" >>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>   require buildchroot.inc >>> +ROOTFS_ARCH = "${HOST_ARCH}" >>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>   BUILDCHROOT_PREINSTALL ?= "make \ >>>                              debhelper \ >>> @@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \ >>>                              automake \ >>>                              devscripts \ >>>                              equivs \ >>> -                           libc6:${DISTRO_ARCH}" >>> - >>> -# According to the wiki page: >>> -#     https://wiki.debian.org/CrossToolchains >>> -BUILDCHROOT_PREINSTALL_append_armhf += "binutils-arm-linux-gnueabihf \ >>> -                                        crossbuild-essential-armhf" >>> -BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \ >>> -                                        crossbuild-essential-arm64" >>> - >>> - >>> -PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'" >>> -do_build[depends] = "isar-apt:do_cache_config >>> isar-bootstrap-host:do_bootstrap" >>> +                           libc6:${DISTRO_ARCH} \ >>> +                           crossbuild-essential-${DISTRO_ARCH}" >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>> b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>> index 20f4d23..5a01258 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>> @@ -21,5 +21,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>>                              devscripts \ >>>                              equivs \ >>>                              adduser" >>> - >>> -do_build[depends] = "isar-apt:do_cache_config >>> isar-bootstrap-target:do_bootstrap" >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc >>> b/meta/recipes-devtools/buildchroot/buildchroot.inc >>> index ca9ce51..40fc8b0 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc >>> @@ -13,18 +13,25 @@ SRC_URI = "file://configscript.sh \ >>>              file://deps.sh" >>>   PV = "1.0" >>> -inherit isar-bootstrap-helper >>> +inherit rootfs >>>   BUILDCHROOT_DIR = "${WORKDIR}/rootfs" >>> +ROOTFSDIR = "${BUILDCHROOT_DIR}" >>> +ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}" >>> +ROOTFS_CLEAN_FILES = "" >>> + >>> +rootfs_do_mounts_append() { >>> +    sudo -s <<'EOSUDO' >>> +    mkdir -p '${BUILDCHROOT_DIR}/downloads' >>> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ >>> +        mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' >>> +EOSUDO >>> +} >>> -do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \ >>> -                            ${BUILDCHROOT_DIR}/isar-apt \ >>> -                            ${BUILDCHROOT_DIR}/downloads \ >>> -                            ${BUILDCHROOT_DIR}/home/builder" >>> - >>> -do_build() { >>> -    setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} >>> ${BUILDCHROOT_PREINSTALL} >>> - >>> +ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" >>> +buildchroot_install_files() { >>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder" >>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt" >>>       # Install package builder script >>>       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" >>>       sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar >>> @@ -38,5 +45,4 @@ do_build() { >>>       GROUP_ID=$(id -g) >>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID >>> -    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads >>>   } >>> diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>> b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>> index 49aeb47..57d52ba 100644 >>> --- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>> +++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>> @@ -12,10 +12,25 @@ SRC_URI = " \ >>>       file://configscript.sh \ >>>       file://README.sdk" >>>   PV = "0.1" >>> - >>> -inherit isar-bootstrap-helper >>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>> +TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}" >>> +TOOLCHAIN_${HOST_ARCH} = "build-essential" >>> +TOOLCHAIN_i386 = "build-essential" >>> + >>> +inherit rootfs >>> +ROOTFS_ARCH = "${HOST_ARCH}" >>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>> +ROOTFSDIR = "${S}" >>> +ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}" >>> +ROOTFS_FEATURES += "copy-package-cache" >>> + >>> +python() { >>> +    if d.getVar("HOST_ARCH") not in ['i386', 'amd64']: >>> +        raise bb.parse.SkipRecipe("SDK doesn't support {} as >>> host".format( >>> +            d.getVar("ROOTFS_ARCH"))) >>> +} >>> + >>>   SDKCHROOT_PREINSTALL := "debhelper \ >>>                              autotools-dev \ >>>                              dpkg \ >>> @@ -28,29 +43,14 @@ SDKCHROOT_PREINSTALL := "debhelper \ >>>   S = "${WORKDIR}/rootfs" >>> -do_build[dirs] = "${DEPLOY_DIR_IMAGE}" >>> -do_build[root_cleandirs] = "${S} \ >>> -                            ${S}/isar-apt" >>> - >>> -do_build[depends] = "isar-apt:do_cache_config >>> isar-bootstrap-host:do_bootstrap" >>> - >>> -do_build() { >>> - >>> -    if [ ${HOST_DISTRO} != "debian-stretch" ]; then >>> -        bbfatal "SDK doesn't support ${HOST_DISTRO}" >>> -    fi >>> -    if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then >>> -        bbfatal "SDK doesn't support ${HOST_ARCH} as host" >>> -    fi >>> - >>> -    if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" ]; >>> then >>> -        packages="${SDKCHROOT_PREINSTALL} build-essential" >>> -    else >>> -        packages="${SDKCHROOT_PREINSTALL} >>> crossbuild-essential-${DISTRO_ARCH}" >>> -    fi >>> - >>> -    setup_root_file_system --copyisarapt --host-arch '${HOST_ARCH}' >>> --host-distro '${HOST_DISTRO}' "${S}" $packages >>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" >>> +rootfs_configure_isar_apt_dir() { >>> +    # Copy isar-apt instead of mounting: >>> +    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt >>> +} >>> +ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files" >>> +sdkchroot_install_files() { >>>       # Configure root filesystem >>>       sudo install -m 644 ${WORKDIR}/README.sdk ${S} >>>       sudo install -m 755 ${WORKDIR}/configscript.sh ${S} >>> >> >> > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov