From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6691586504498610176 X-Received: by 2002:a17:906:5d12:: with SMTP id g18mr82947174ejt.286.1558950551260; Mon, 27 May 2019 02:49:11 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:4946:: with SMTP id f6ls3745849ejt.4.gmail; Mon, 27 May 2019 02:49:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqyQ/6fM0Bi9wKhu3Lle6ssAKKvvpMvWSMvTk4XVUEJkzyo1R5pXozXJH81DyHmUs2KDv5cR X-Received: by 2002:a17:906:1483:: with SMTP id x3mr65065765ejc.90.1558950550905; Mon, 27 May 2019 02:49:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558950550; cv=none; d=google.com; s=arc-20160816; b=lx6LRLLMYIFcPTmC98dNzOqmyW1yTm0s6aDejdZRw82zW1NG0hYrljfY5CbsJP2D/K TAVp6t8+Ucjhbr5i/bJDGavAriHY226H1cEz2wV9lBtdycS8JJ1fJk6FsTRNnpQ0Mxbq RYgc4slAadol69j68SCqiNdTfcT5K0GbGH8fZwTBwBhRt9XhT8c2nN0nUQ9eyEzSq39P 903oY1GIH+nt45icVltdl7IJtnl/w6al6quRBmrB4vhbkSzZHZAyhEXfvC2HDpUALB9F GT2Ny9A3DwC/1jFW0qKCCds4Zd9gc0KOn83Ajg4oGFYhlcaJ7WTzM6DTNxhXXNYKK8Vc tsKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:cc:to :subject; bh=Ygjsj/0wZ0ZLYATyL8AUlR+R/jOnfKY7io9X0aGipqc=; b=DsIJMpMEZLJi4BbEjHF7dUc6TEXwaJ5nA9yf6qFKvrMclfSYn5R3cCzaVQC+EWFrPg M+/AnHJtdmX9dj7w3Ux6rEMWT9kBmJy945WpLi3Cii01jDOaVZChbpq7kg+MPxJkESm2 TJBQT0Qywo7ZfXeVzgscr/9+o4sfHDbaqPyvk2ewGHJjtCXXobbjF2u71EAIdCc4C0CO 2bdKO18j6F/UgiB8BXE/SlpCG3knuOm2XuL9rwyvQ1d7c2hDgjudWWArnjGwnmBCOWf1 9i/v+vPEXZ7kEfe/KKKSIESjxDqFtqFwXuZsqfA7tb8E3MnvxNl+k4iPxgXlyeI4loz6 fojQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id l30si620056edd.4.2019.05.27.02.49.10 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 27 May 2019 02:49:10 -0700 (PDT) Received-SPF: pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.1.29] ([109.131.165.195]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x4R9n7Kq025409 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 May 2019 11:49:09 +0200 Subject: Re: [PATCH v4 2/8] split up isar-bootstrap helper and implement pre-process pipeline To: Claudius Heine , isar-users@googlegroups.com Cc: Claudius Heine References: <20190523145521.23050-1-claudius.heine.ext@siemens.com> <20190523145521.23050-3-claudius.heine.ext@siemens.com> <1867c412-713b-5dfb-fca8-15f9ffc99499@ilbers.de> <18fb7e43-c5ed-f007-30c1-4be54a39e7ad@ilbers.de> <2507d110-feb9-95c9-11ab-a914afbc0547@siemens.com> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: Date: Mon, 27 May 2019 11:49:03 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <2507d110-feb9-95c9-11ab-a914afbc0547@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: XjFHT/AFcTPv On 5/27/19 11:03 AM, Claudius Heine wrote: > Hi, > > On 27/05/2019 09.20, Maxim Yu. Osipov wrote: >> Hi Claudius, >> >> On 5/27/19 8:55 AM, Claudius Heine wrote: >>> Hi Maxim, >>> >>> On 24/05/2019 14.49, Maxim Yu. Osipov wrote: >>>> Hi Claudius, >>>> >>>> One may agree that setup_root_file_system is too "fat" >>>> and should be split to logical parts. >>> >>> There are multiple issues with `setup_root_file_system`, not just it >>> being 'fat'. For instance the amount of parameters this function >>> accepts just screams to me that it does not scale necessary >>> customization. >>> >>>> My concern is that one have to carefully keep track for every >>>> ROOTFS_CONFIGURE/ROOTFS_INSTALL command's  'weight' - order of >>>> execution - possible source of errors. >>> >>> The weight parameter does not specify the order or execution, but >>> just the percentage of the progress bar shown when building. Since >>> do_rootfs_install is a pretty big task, I found it useful to show at >>> least some progress. >> >> Ah, I see now. Please add this comment into the code (to avoid >> confusion). > > I think that could also be done later in an additional patchset, if that > is the only reason for a new series. Sure - new series are not required. - just add the more comments in the appropriate place in the code. >> >>> >>> The order of commands is the order in which they are listed in the >>> ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND array. >>> >>> Currently that is not an interface for the user, but that might >>> happen if more variables are added there to signify different stages >>> of the rootfs build process, for instance how the >>> `ROOTFS_INSTALL_COMMAND_BEFORE_CLEAN` was done. >> >> I think that it's worth to add description of >> ROOTFS_CONFIGURE_COMMAND/ROOTFS_INSTALL_COMMAND in RECIPE_API_CHANGELOG. > > I can do that, but why? No user API has changed AFAIK and > ROOTFS_*_COMMAND aren't yet user API. They might become in the future, > but currently I would like to threat them as internal API since adding > to them from outside of isar is not very comfortable. I hope treat and not threat ;) Regards, Maxim. > regards, > Claudius > >> >> Regards, >> Maxim. >> >> >>> regards, >>> Claudius >>> >>>> >>>> Just a case: one my not define weight value (in such case default >>>> value is 20) as its is done for >>>> ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" - so if >>>> we have several such default weights the order of commands could be >>>> wrong. >>>> >>>> Regards, >>>> Maxim. >>>> >>>> On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote: >>>>> From: Claudius Heine >>>>> >>>>> This removes the isar-bootstrap-helper.bbclass and integrates its >>>>> functionality into the rootfs.bbclass. >>>>> >>>>> It introduces the rootfs_install task that runs through multiple >>>>> functions from command lists in order to allow easy integration of >>>>> custom functions. >>>>> >>>>> The introduced functions are, in general order of execution: >>>>> >>>>>    rootfs_prepare: copy the isar-bootstrap rootfs base to workdir >>>>>    rootfs_configure_*: basic configuration of the root file system, >>>>>      before any custom packages where installed. (`/dev`, `/proc`, >>>>> etc. >>>>>      are not mounted). Functions are added to >>>>> `ROOTFS_CONFIGURE_COMMAND`. >>>>>    rootfs_do_mount: mount standard directories and repositories >>>>> into the >>>>>      rootfs >>>>>    do_rootfs_install_*: downloading and installing packages. Functions >>>>>    are added to `ROOTFS_INSTALL_COMMAND`. >>>>> >>>>> Signed-off-by: Claudius Heine >>>>> --- >>>>>   meta/classes/image.bbclass                    |  24 +-- >>>>>   meta/classes/isar-bootstrap-helper.bbclass    | 132 -------------- >>>>>   meta/classes/rootfs.bbclass                   | 172 >>>>> +++++++++++++++++- >>>>>   .../isar-bootstrap/isar-bootstrap.inc         |   5 + >>>>>   .../buildchroot/buildchroot-host.bb           |  16 +- >>>>>   .../buildchroot/buildchroot-target.bb         |   2 - >>>>>   .../buildchroot/buildchroot.inc               |  26 ++- >>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--- >>>>>   8 files changed, 228 insertions(+), 197 deletions(-) >>>>>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass >>>>> >>>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >>>>> index d77626c..99eea92 100644 >>>>> --- a/meta/classes/image.bbclass >>>>> +++ b/meta/classes/image.bbclass >>>>> @@ -51,8 +51,10 @@ image_do_mounts() { >>>>>       buildchroot_do_mounts >>>>>   } >>>>> -inherit isar-bootstrap-helper >>>>> -ROOTFS_FEATURES += "finalize-rootfs" >>>>> +ROOTFSDIR = "${IMAGE_ROOTFS}" >>>>> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >>>>> finalize-rootfs" >>>>> +ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >>>>> + >>>>>   inherit rootfs >>>>>   inherit image-sdk-extension >>>>>   inherit image-cache-extension >>>>> @@ -119,8 +121,10 @@ python set_image_size () { >>>>>       d.setVarFlag('ROOTFS_SIZE', 'export', '1') >>>>>   } >>>>> -do_image_gen_fstab() { >>>>> -    cat > ${WORKDIR}/fstab << EOF >>>>> +ROOTFS_CONFIGURE_COMMAND += "image_configure_fstab" >>>>> +image_configure_fstab[weight] = "2" >>>>> +image_configure_fstab() { >>>>> +    sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF >>>>>   # Begin /etc/fstab >>>>>   /dev/root    /        auto        defaults        0    0 >>>>>   proc        /proc        proc        nosuid,noexec,nodev    0    0 >>>>> @@ -132,18 +136,6 @@ devtmpfs    /dev        devtmpfs >>>>> mode=0755,nosuid    0    0 >>>>>   # End /etc/fstab >>>>>   EOF >>>>>   } >>>>> -addtask image_gen_fstab before do_rootfs_install >>>>> - >>>>> -do_rootfs_install[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-target:do_bootstrap" >>>>> -do_rootfs_install[deptask] = "do_deploy_deb" >>>>> -do_rootfs_install[root_cleandirs] = "${IMAGE_ROOTFS} \ >>>>> -                             ${IMAGE_ROOTFS}/isar-apt" >>>>> -do_rootfs_install() { >>>>> -    setup_root_file_system --clean --keep-apt-cache \ >>>>> -        --fstab "${WORKDIR}/fstab" \ >>>>> -        "${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL} >>>>> -} >>>>> -addtask rootfs_install before do_build after do_unpack >>>>>   do_copy_boot_files[dirs] = "${DEPLOY_DIR_IMAGE}" >>>>>   do_copy_boot_files() { >>>>> diff --git a/meta/classes/isar-bootstrap-helper.bbclass >>>>> b/meta/classes/isar-bootstrap-helper.bbclass >>>>> deleted file mode 100644 >>>>> index bbc6322..0000000 >>>>> --- a/meta/classes/isar-bootstrap-helper.bbclass >>>>> +++ /dev/null >>>>> @@ -1,132 +0,0 @@ >>>>> -# Helper functions for using isar-bootstrap >>>>> -# >>>>> -# This software is a part of ISAR. >>>>> -# Copyright (c) Siemens AG, 2018 >>>>> -# >>>>> -# SPDX-License-Identifier: MIT >>>>> - >>>>> -def reverse_bb_array(d, varname): >>>>> -    array = d.getVar(varname, True) >>>>> -    if array is None: >>>>> -        return None >>>>> -    array = reversed(array.split()) >>>>> -    return " ".join(i for i in array) >>>>> - >>>>> - >>>>> -update_etc_os_release() { >>>>> -    OS_RELEASE_BUILD_ID="" >>>>> -    OS_RELEASE_VARIANT="" >>>>> -    while true; do >>>>> -        case "$1" in >>>>> -    --build-id) OS_RELEASE_BUILD_ID=$2; shift ;; >>>>> -    --variant) OS_RELEASE_VARIANT=$2; shift ;; >>>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>>> -        *) break ;; >>>>> -        esac >>>>> -        shift >>>>> -    done >>>>> -    ROOTFSDIR="$1" >>>>> - >>>>> -    if [ -n "${OS_RELEASE_BUILD_ID}" ]; then >>>>> -        sudo sed -i '/^BUILD_ID=.*/d' ${ROOTFSDIR}/etc/os-release >>>>> -        echo "BUILD_ID=\"${OS_RELEASE_BUILD_ID}\"" | \ >>>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>>> -    fi >>>>> -    if [ -n "${OS_RELEASE_VARIANT}" ]; then >>>>> -        sudo sed -i '/^VARIANT=.*/d' ${ROOTFSDIR}/etc/os-release >>>>> -        echo "VARIANT=\"${OS_RELEASE_VARIANT}\"" | \ >>>>> -            sudo tee -a ${ROOTFSDIR}/etc/os-release >>>>> -    fi >>>>> -} >>>>> - >>>>> -setup_root_file_system() { >>>>> -    CLEAN="" >>>>> -    COPYISARAPT="" >>>>> -    FSTAB="" >>>>> -    ROOTFS_ARCH="${DISTRO_ARCH}" >>>>> -    ROOTFS_DISTRO="${DISTRO}" >>>>> -    while true; do >>>>> -        case "$1" in >>>>> -        --clean) CLEAN=1 ;; >>>>> -        --copyisarapt) COPYISARAPT=1 ;; >>>>> -        --fstab) FSTAB=$2; shift ;; >>>>> -        --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;; >>>>> -        --host-distro) ROOTFS_DISTRO="$2"; shift ;; >>>>> -        --keep-apt-cache) KEEP_APT_CACHE=1 ;; >>>>> -        -*) bbfatal "$0: invalid option specified: $1" ;; >>>>> -        *) break ;; >>>>> -        esac >>>>> -        shift >>>>> -    done >>>>> -    ROOTFSDIR="$1" >>>>> -    shift >>>>> -    PACKAGES="$@" >>>>> -    APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>>> -    CLEAN_FILES="${ROOTFSDIR}/etc/hostname >>>>> ${ROOTFSDIR}/etc/resolv.conf" >>>>> - >>>>> -    sudo cp -Trpfx \ >>>>> -        "${DEPLOY_DIR_BOOTSTRAP}/$ROOTFS_DISTRO-$ROOTFS_ARCH/" \ >>>>> -        "$ROOTFSDIR" >>>>> -    [ -n "${FSTAB}" ] && cat ${FSTAB} | sudo tee >>>>> "$ROOTFSDIR/etc/fstab" >>>>> - >>>>> -    echo "deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} >>>>> main" | \ >>>>> -        sudo tee "$ROOTFSDIR/etc/apt/sources.list.d/isar-apt.list" >>>>> >/dev/null >>>>> - >>>>> -    echo "Package: *\nPin: release >>>>> n=${DEBDISTRONAME}\nPin-Priority: 1000" | \ >>>>> -        sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null >>>>> - >>>>> -    if [ ${COPYISARAPT} ]; then >>>>> -        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt >>>>> -    else >>>>> -        sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} >>>>> $ROOTFSDIR/isar-apt >>>>> -    fi >>>>> - >>>>> -    if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then >>>>> -        sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt >>>>> -    fi >>>>> - >>>>> -    sudo mount --rbind /dev ${ROOTFSDIR}/dev >>>>> -    sudo mount --make-rslave ${ROOTFSDIR}/dev >>>>> -    sudo mount -t proc none $ROOTFSDIR/proc >>>>> -    sudo mount --rbind /sys ${ROOTFSDIR}/sys >>>>> -    sudo mount --make-rslave ${ROOTFSDIR}/sys >>>>> - >>>>> -    # Refresh /etc/resolv.conf >>>>> -    sudo cp -L /etc/resolv.conf ${ROOTFSDIR}/etc >>>>> - >>>>> -    # Install packages: >>>>> -    E="${@ bb.utils.export_proxies(d)}" >>>>> -    export DEBIAN_FRONTEND=noninteractive >>>>> -    # To avoid Perl locale warnings: >>>>> -    export LANG=C >>>>> -    export LANGUAGE=C >>>>> -    export LC_ALL=C >>>>> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \ >>>>> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>>> -        -o Dir::Etc::sourceparts="-" \ >>>>> -        -o APT::Get::List-Cleanup="0" >>>>> -    # Add multiarch for cross-target >>>>> -    if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then >>>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg >>>>> --add-architecture ${DISTRO_ARCH} >>>>> -        sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update >>>>> -    fi >>>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>>> -        /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES >>>>> -    [ ${CLEAN} ] && sudo rm -f ${CLEAN_FILES} >>>>> -    sudo -E chroot "$ROOTFSDIR" \ >>>>> -        /usr/bin/apt-get ${APT_ARGS} $PACKAGES >>>>> -    if [ ${CLEAN} ]; then >>>>> -        if [ ${KEEP_APT_CACHE} -eq 1 ]; then >>>>> -            mkdir -p ${WORKDIR}/apt_cache >>>>> -            sudo find ${ROOTFSDIR}/var/cache/apt/archives \ >>>>> -                -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>>> ${WORKDIR}/apt_cache '{}' '+' >>>>> -            sudo chown -R $(whoami) ${WORKDIR}/apt_cache >>>>> -        fi >>>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>>> -            /usr/bin/apt-get autoremove --purge --yes >>>>> -        sudo -E chroot "$ROOTFSDIR" \ >>>>> -            /usr/bin/apt-get clean >>>>> -        sudo "$ROOTFSDIR/chroot-setup.sh" "cleanup" "$ROOTFSDIR" >>>>> -        sudo rm -rf "$ROOTFSDIR/chroot-setup.sh" >>>>> "$ROOTFSDIR/var/lib/apt/lists/"* >>>>> -    fi >>>>> -} >>>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass >>>>> index 45b0350..0fc8ba8 100644 >>>>> --- a/meta/classes/rootfs.bbclass >>>>> +++ b/meta/classes/rootfs.bbclass >>>>> @@ -1,14 +1,178 @@ >>>>>   # This software is a part of ISAR. >>>>>   # Copyright (c) Siemens AG, 2019 >>>>> +ROOTFS_ARCH ?= "${DISTRO_ARCH}" >>>>> +ROOTFS_DISTRO ?= "${DISTRO}" >>>>> +ROOTFS_PACKAGES ?= "" >>>>> + >>>>>   # Features of the rootfs creation: >>>>>   # available features are: >>>>> +# 'deploy-package-cache' - copy the package cache >>>>> ${WORKDIR}/apt_cache >>>>> +# 'clean-package-cache' - delete package cache from rootfs >>>>>   # 'finalize-rootfs' - delete files needed to chroot into the rootfs >>>>>   ROOTFS_FEATURES ?= "" >>>>> +ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes" >>>>> + >>>>> +ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf" >>>>> + >>>>> +# Useful environment variables: >>>>> +export E = "${@ bb.utils.export_proxies(d)}" >>>>> +export DEBIAN_FRONTEND = "noninteractive" >>>>> +# To avoid Perl locale warnings: >>>>> +export LANG = "C" >>>>> +export LANGUAGE = "C" >>>>> +export LC_ALL = "C" >>>>> + >>>>> +rootfs_do_mounts[weight] = "3" >>>>> +rootfs_do_mounts() { >>>>> +    sudo -s <<'EOSUDO' >>>>> +        mountpoint -q '${ROOTFSDIR}/dev' || \ >>>>> +            mount --rbind /dev '${ROOTFSDIR}/dev' >>>>> +        mount --make-rslave '${ROOTFSDIR}/dev' >>>>> +        mountpoint -q '${ROOTFSDIR}/proc' || \ >>>>> +            mount -t proc none '${ROOTFSDIR}/proc' >>>>> +        mountpoint -q '${ROOTFSDIR}/sys' || \ >>>>> +            mount --rbind /sys '${ROOTFSDIR}/sys' >>>>> +        mount --make-rslave '${ROOTFSDIR}/sys' >>>>> + >>>>> +        # Mount isar-apt if the directory does not exist or if it >>>>> is empty >>>>> +        # This prevents overwriting something that was copied there >>>>> +        if [ ! -e '${ROOTFSDIR}/isar-apt' ] || \ >>>>> +           [ "$(find '${ROOTFSDIR}/isar-apt' -maxdepth 1 -mindepth >>>>> 1 | wc -l)" = "0" ] >>>>> +        then >>>>> +            mkdir -p '${ROOTFSDIR}/isar-apt' >>>>> +            mountpoint -q '${ROOTFSDIR}/isar-apt' || \ >>>>> +                mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >>>>> '${ROOTFSDIR}/isar-apt' >>>>> +        fi >>>>> + >>>>> +        # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set >>>>> +        if [ >>>>> "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" >>>>> = 'True' ] >>>>> +        then >>>>> +            mkdir -p '${ROOTFSDIR}/base-apt' >>>>> +            mountpoint -q '${ROOTFSDIR}/base-apt' || \ >>>>> +                mount --bind '${REPO_BASE_DIR}' >>>>> '${ROOTFSDIR}/base-apt' >>>>> +        fi >>>>> + >>>>> +EOSUDO >>>>> +} >>>>> + >>>>> +rootfs_do_qemu() { >>>>> +    if [ '${@repr(d.getVar('ROOTFS_ARCH') == >>>>> d.getVar('HOST_ARCH'))}' = 'False' ] >>>>> +    then >>>>> +        test -e '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' || \ >>>>> +            sudo cp '/usr/bin/qemu-${QEMU_ARCH}-static' >>>>> '${ROOTFSDIR}/usr/bin/qemu-${QEMU_ARCH}-static' >>>>> +    fi >>>>> +} >>>>> + >>>>> +BOOTSTRAP_SRC = >>>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}-${DISTRO_ARCH}" >>>>> >>>>> +BOOTSTRAP_SRC_${ROOTFS_ARCH} = >>>>> "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}" >>>>> + >>>>> +rootfs_prepare[weight] = "25" >>>>> +rootfs_prepare(){ >>>>> +    sudo cp -Trpfx '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}' >>>>> +} >>>>> + >>>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt" >>>>> +rootfs_configure_isar_apt[weight] = "2" >>>>> +rootfs_configure_isar_apt() { >>>>> +    sudo -s <<'EOSUDO' >>>>> + >>>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d' >>>>> +    echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} >>>>> main' > \ >>>>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list' >>>>> + >>>>> +    mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d' >>>>> +    cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar' >>>>> +Package: * >>>>> +Pin: release n=${DEBDISTRONAME} >>>>> +Pin-Priority: 1000 >>>>> +EOF >>>>> +EOSUDO >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update" >>>>> +rootfs_install_pkgs_update[weight] = "5" >>>>> +rootfs_install_pkgs_update() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \ >>>>> +        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \ >>>>> +        -o Dir::Etc::sourceparts="-" \ >>>>> +        -o APT::Get::List-Cleanup="0" >>>>> +} >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_resolvconf" >>>>> +rootfs_install_resolvconf[weight] = "1" >>>>> +rootfs_install_resolvconf() { >>>>> +    sudo cp -rL /etc/resolv.conf '${ROOTFSDIR}/etc' >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download" >>>>> +rootfs_install_pkgs_download[weight] = "600" >>>>> +rootfs_install_pkgs_download() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only >>>>> ${ROOTFS_PACKAGES} >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if >>>>> (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}" >>>>> +rootfs_install_clean_files[weight] = "2" >>>>> +rootfs_install_clean_files() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /bin/rm -f ${ROOTFS_CLEAN_FILES} >>>>> +} >>>>> + >>>>> +ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_install" >>>>> +rootfs_install_pkgs_install[weight] = "8000" >>>>> +rootfs_install_pkgs_install() { >>>>> +    sudo -E chroot "${ROOTFSDIR}" \ >>>>> +        /usr/bin/apt-get ${ROOTFS_APT_ARGS} ${ROOTFS_PACKAGES} >>>>> +} >>>> >>>> >>>> Honestly I don't see benefits in such multiple >>>> configure_cmds/rootfs_install_cmds - it doesn't increase code >>>> readability as one have to additionally keep track for every command >>>> 'weight' value and it's easy to miss the correct order of commands >>>> execution. >>>> >>>> >>>>> +do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" >>>>> +do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} >>>>> ${ROOTFS_INSTALL_COMMAND}" >>>>> +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if >>>>> d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else >>>>> 'host'}:do_build isar-apt:do_cache_config" >>>>> +do_rootfs_install[deptask] = "do_deploy_deb" >>>>> +python do_rootfs_install() { >>>>> +    configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) >>>>> or "").split() >>>>> +    install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or >>>>> "").split() >>>>> + >>>>> +    # Mount after configure commands, so that they have time to copy >>>>> +    # 'isar-apt' (sdkchroot): >>>>> +    cmds = ['rootfs_prepare'] + configure_cmds + >>>>> ['rootfs_do_mounts'] + install_cmds >>>>> + >>>>> +    stage_weights = [int(d.getVarFlag(i, 'weight', True) or "20") >>>>> +                     for i in cmds] >>>>> + >>>>> +    progress_reporter = bb.progress.MultiStageProgressReporter(d, >>>>> stage_weights) >>>>> + >>>>> +    for cmd in cmds: >>>>> +        progress_reporter.next_stage() >>>>> +        bb.build.exec_func(cmd, d) >>>>> +    progress_reporter.finish() >>>>> +} >>>>> +addtask rootfs_install before do_rootfs_postprocess after do_unpack >>>>> + >>>>> +ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'copy-package-cache', >>>>> 'rootfs_postprocess_copy_package_cache', '', d)}" >>>>> +rootfs_postprocess_copy_package_cache() { >>>>> +    mkdir -p '${WORKDIR}/apt_cache' >>>>> +    sudo find '${ROOTFSDIR}/var/cache/apt/archives' \ >>>>> +        -maxdepth 1 -name '*.deb' -execdir /bin/mv -t >>>>> '${WORKDIR}/apt_cache' '{}' '+' >>>>> +    me="$(id -u):$(id -g)" >>>>> +    sudo chown -R "$me" '${WORKDIR}/apt_cache' >>>>> +} >>>>> + >>>>> +ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', >>>>> 'rootfs_postprocess_clean_package_cache', '', d)}" >>>>> +rootfs_postprocess_clean_package_cache() { >>>>> +    sudo -E chroot '${ROOTFSDIR}' \ >>>>> +        /usr/bin/apt-get clean >>>>> +    sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >>>>> +} >>>>> + >>>>>   ROOTFS_POSTPROCESS_COMMAND += >>>>> "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', >>>>> 'rootfs_postprocess_finalize', '', d)}" >>>>>   rootfs_postprocess_finalize() { >>>>>       sudo -s <<'EOSUDO' >>>>> +        test -e "${ROOTFSDIR}/chroot-setup.sh" && \ >>>>> +            "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}" >>>>> +        rm -f "${ROOTFSDIR}/chroot-setup.sh" >>>>> + >>>>>           test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \ >>>>>               find "${ROOTFSDIR}/usr/bin" \ >>>>>                   -maxdepth 1 -name 'qemu-*-static' -type f -delete >>>>> @@ -42,6 +206,12 @@ EOSUDO >>>>>   do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" >>>>>   python do_rootfs_postprocess() { >>>>> +    # Take care that its correctly mounted: >>>>> +    bb.build.exec_func('rootfs_do_mounts', d) >>>>> +    # Take care that qemu-*-static is available, since it could >>>>> have been >>>>> +    # removed on a previous execution of this task: >>>>> +    bb.build.exec_func('rootfs_do_qemu', d) >>>>> + >>>>>       cmds = d.getVar("ROOTFS_POSTPROCESS_COMMAND") >>>>>       if cmds is None or not cmds.strip(): >>>>>           return >>>>> @@ -49,7 +219,7 @@ python do_rootfs_postprocess() { >>>>>       for cmd in cmds: >>>>>           bb.build.exec_func(cmd, d) >>>>>   } >>>>> -addtask rootfs_postprocess before do_rootfs after do_rootfs_install >>>>> +addtask rootfs_postprocess before do_rootfs >>>>>   python do_rootfs() { >>>>>       """Virtual task""" >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> index b740c57..f8741c2 100644 >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>>>> @@ -327,6 +327,11 @@ isar_bootstrap() { >>>>>               mount --make-rslave ${ROOTFSDIR}/sys >>>>>               export DEBIAN_FRONTEND=noninteractive >>>>> + >>>>> +            if [ ${IS_HOST} ]; then >>>>> +                chroot "${ROOTFSDIR}" /usr/bin/dpkg >>>>> --add-architecture ${DISTRO_ARCH} >>>>> +            fi >>>>> + >>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y >>>>>               chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ >>>>>                                     -o Debug::pkgProblemResolver=yes >>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> index 73cd548..121992c 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb >>>>> @@ -7,6 +7,8 @@ DESCRIPTION = "Isar development filesystem for host" >>>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>>>   require buildchroot.inc >>>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>>>   BUILDCHROOT_PREINSTALL ?= "make \ >>>>>                              debhelper \ >>>>> @@ -18,15 +20,5 @@ BUILDCHROOT_PREINSTALL ?= "make \ >>>>>                              automake \ >>>>>                              devscripts \ >>>>>                              equivs \ >>>>> -                           libc6:${DISTRO_ARCH}" >>>>> - >>>>> -# According to the wiki page: >>>>> -#     https://wiki.debian.org/CrossToolchains >>>>> -BUILDCHROOT_PREINSTALL_append_armhf += >>>>> "binutils-arm-linux-gnueabihf \ >>>>> -                                        crossbuild-essential-armhf" >>>>> -BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \ >>>>> -                                        crossbuild-essential-arm64" >>>>> - >>>>> - >>>>> -PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'" >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-host:do_bootstrap" >>>>> +                           libc6:${DISTRO_ARCH} \ >>>>> +                           crossbuild-essential-${DISTRO_ARCH}" >>>>> diff --git >>>>> a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> index 20f4d23..5a01258 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb >>>>> @@ -21,5 +21,3 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>>>>                              devscripts \ >>>>>                              equivs \ >>>>>                              adduser" >>>>> - >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-target:do_bootstrap" >>>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> index ca9ce51..40fc8b0 100644 >>>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc >>>>> @@ -13,18 +13,25 @@ SRC_URI = "file://configscript.sh \ >>>>>              file://deps.sh" >>>>>   PV = "1.0" >>>>> -inherit isar-bootstrap-helper >>>>> +inherit rootfs >>>>>   BUILDCHROOT_DIR = "${WORKDIR}/rootfs" >>>>> +ROOTFSDIR = "${BUILDCHROOT_DIR}" >>>>> +ROOTFS_PACKAGES = "${BUILDCHROOT_PREINSTALL}" >>>>> +ROOTFS_CLEAN_FILES = "" >>>>> + >>>>> +rootfs_do_mounts_append() { >>>>> +    sudo -s <<'EOSUDO' >>>>> +    mkdir -p '${BUILDCHROOT_DIR}/downloads' >>>>> +    mountpoint -q '${BUILDCHROOT_DIR}/downloads' || \ >>>>> +        mount --bind '${DL_DIR}' '${BUILDCHROOT_DIR}/downloads' >>>>> +EOSUDO >>>>> +} >>>>> -do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \ >>>>> -                            ${BUILDCHROOT_DIR}/isar-apt \ >>>>> -                            ${BUILDCHROOT_DIR}/downloads \ >>>>> -                            ${BUILDCHROOT_DIR}/home/builder" >>>>> - >>>>> -do_build() { >>>>> -    setup_root_file_system ${PARAMS} ${BUILDCHROOT_DIR} >>>>> ${BUILDCHROOT_PREINSTALL} >>>>> - >>>>> +ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files" >>>>> +buildchroot_install_files() { >>>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder" >>>>> +    sudo mkdir -p "${BUILDCHROOT_DIR}/isar-apt" >>>>>       # Install package builder script >>>>>       sudo chmod -R a+rw "${BUILDCHROOT_DIR}/home/builder" >>>>>       sudo install -m 755 -d ${BUILDCHROOT_DIR}/isar >>>>> @@ -38,5 +45,4 @@ do_build() { >>>>>       GROUP_ID=$(id -g) >>>>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID >>>>> $GROUP_ID >>>>> -    sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads >>>>>   } >>>>> diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> index 49aeb47..57d52ba 100644 >>>>> --- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> +++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb >>>>> @@ -12,10 +12,25 @@ SRC_URI = " \ >>>>>       file://configscript.sh \ >>>>>       file://README.sdk" >>>>>   PV = "0.1" >>>>> - >>>>> -inherit isar-bootstrap-helper >>>>>   PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}" >>>>> +TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}" >>>>> +TOOLCHAIN_${HOST_ARCH} = "build-essential" >>>>> +TOOLCHAIN_i386 = "build-essential" >>>>> + >>>>> +inherit rootfs >>>>> +ROOTFS_ARCH = "${HOST_ARCH}" >>>>> +ROOTFS_DISTRO = "${HOST_DISTRO}" >>>>> +ROOTFSDIR = "${S}" >>>>> +ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}" >>>>> +ROOTFS_FEATURES += "copy-package-cache" >>>>> + >>>>> +python() { >>>>> +    if d.getVar("HOST_ARCH") not in ['i386', 'amd64']: >>>>> +        raise bb.parse.SkipRecipe("SDK doesn't support {} as >>>>> host".format( >>>>> +            d.getVar("ROOTFS_ARCH"))) >>>>> +} >>>>> + >>>>>   SDKCHROOT_PREINSTALL := "debhelper \ >>>>>                              autotools-dev \ >>>>>                              dpkg \ >>>>> @@ -28,29 +43,14 @@ SDKCHROOT_PREINSTALL := "debhelper \ >>>>>   S = "${WORKDIR}/rootfs" >>>>> -do_build[dirs] = "${DEPLOY_DIR_IMAGE}" >>>>> -do_build[root_cleandirs] = "${S} \ >>>>> -                            ${S}/isar-apt" >>>>> - >>>>> -do_build[depends] = "isar-apt:do_cache_config >>>>> isar-bootstrap-host:do_bootstrap" >>>>> - >>>>> -do_build() { >>>>> - >>>>> -    if [ ${HOST_DISTRO} != "debian-stretch" ]; then >>>>> -        bbfatal "SDK doesn't support ${HOST_DISTRO}" >>>>> -    fi >>>>> -    if [ ${HOST_ARCH} != "i386" -a ${HOST_ARCH} != "amd64" ]; then >>>>> -        bbfatal "SDK doesn't support ${HOST_ARCH} as host" >>>>> -    fi >>>>> - >>>>> -    if [ ${HOST_ARCH} = ${DISTRO_ARCH} -o ${DISTRO_ARCH} = "i386" >>>>> ]; then >>>>> -        packages="${SDKCHROOT_PREINSTALL} build-essential" >>>>> -    else >>>>> -        packages="${SDKCHROOT_PREINSTALL} >>>>> crossbuild-essential-${DISTRO_ARCH}" >>>>> -    fi >>>>> - >>>>> -    setup_root_file_system --copyisarapt --host-arch >>>>> '${HOST_ARCH}' --host-distro '${HOST_DISTRO}' "${S}" $packages >>>>> +ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir" >>>>> +rootfs_configure_isar_apt_dir() { >>>>> +    # Copy isar-apt instead of mounting: >>>>> +    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt >>>>> +} >>>>> +ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files" >>>>> +sdkchroot_install_files() { >>>>>       # Configure root filesystem >>>>>       sudo install -m 644 ${WORKDIR}/README.sdk ${S} >>>>>       sudo install -m 755 ${WORKDIR}/configscript.sh ${S} >>>>> >>>> >>>> >>> >> >> > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov