From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449326914083487744 X-Received: by 10.28.184.69 with SMTP id i66mr742819wmf.18.1502259554635; Tue, 08 Aug 2017 23:19:14 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.2.139 with SMTP id y11ls218553lje.4.gmail; Tue, 08 Aug 2017 23:19:14 -0700 (PDT) X-Received: by 10.25.160.135 with SMTP id j129mr991730lfe.26.1502259554163; Tue, 08 Aug 2017 23:19:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502259554; cv=none; d=google.com; s=arc-20160816; b=f7VlWs9h85k4skWKORHUtMJckT/DIvE3JNkuRB0Co2CKMh24JmiQPG6wmHICNbdqpE G1okLGr7l9dYSwNauIn9Y3h88cRIxwEgMn5lLg5Ds8radhuRaEIl6YhYmUw3zPqPvTCo Yk9DPiOwu1mT6vR4CJTzxZ1Ss+LJ/qS9YbbFH4ua/30+7U7YmMpjBlKmnLLOJbVWeTCD /n2iWU3CoRVwTcoARgvOAxUhoXUvxSy8/UN701KPdLHHsB2UadDxWieYOw9uohxsnDUp O3w+siDR5NYFd2t7AoaFszPRDCjuDU8ZChHDinbWCe2nX6afBXdFfixMQzYBZtahW184 JOgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date:arc-authentication-results; bh=inahhnktLBLsl+Mig2akKjXctqA5XigN4Ef2L8vMhTk=; b=zgXvZMRJKYnKkqT3VQ4ixiNwYD7DnxG9dEC04nCwSlkPRwlk/LUI7XwXG54MaUsbtM 8UOaBZeG82gdI/TYyLyEClXzT2TbLaQmyF/2Jlbt+33lZ+OKSD59BbinxmyP0a/P2wsW TDEl5Ttkc7TkeQfDZqUTzfIWS1MrtgDUTMZky+2U6PhuFY++osUjA5Ngxl4cE+HM87VD YpSRKH0LgtWhzs8n5RVjHxgoAH8yb3AcEcbgVjEHoSqawvU7Bg7lNtcYV0BWWvCgvezu wjf8hleGOIJT+soHt38dWOJAY4IuAEmzWx+UeqDEagqNuvGqQem6Y51soFxFOGcOrKLd vFrA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id 2si971341wms.6.2017.08.08.23.19.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Aug 2017 23:19:14 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.28 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id v796JDtc022434 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 9 Aug 2017 08:19:13 +0200 Received: from md1em3qc (DNBOB7LC.ad001.siemens.net [139.22.119.198] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id v796JDki027476; Wed, 9 Aug 2017 08:19:13 +0200 Date: Wed, 9 Aug 2017 08:21:09 +0200 From: Henning Schild To: , "PFEFFERL, JOHANN" Subject: Re: [PATCH 3/3] classes: image: remove populate and replace it with a custom repo Message-ID: <20170809082109.549bf091@md1em3qc> In-Reply-To: References: X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: 9d6J/DuNVQ3G Hey, Hans pointed out that this one effectively doubles build-times. That is because we do multistrap two times. Once for the buildchroot and one for the rootfs. Without this patch these two run in parallel, with it they are serialized because the rootfs can only start once all .debs are ready. He suggested building a repo, very much like mine, but replace "dpdk -i" in do_populate with "apt-get install". This way only do_populate of rootfs has to wait for buildchroot doing its stuff and multistrap can already run in the background. But i still think we should feed the debs into multistrap - we introduce an important policy "the only way into the rootfs is multistrap" - that is very likely to help with the reproducible builds - that reduces all fetching/internet access to multistrap and the recipes build in chroot - build times will still be very short compared to Yocto To address the performance problem we should first look at what is taking multistrap so long, the downloading or the installing. And then look into caching: - make the second multistrap use the files downloaded by the first - keep the buildchroot across builds - keep all remote .debs acroos builds Henning Am Tue, 1 Aug 2017 17:24:08 +0200 schrieb Henning Schild : > Create a custom Debian repository and give that to multistrap. That > way multistrap can take care of our dependencies and we do not force > in packages with "dpdk -i" anymore. > The only way stuff gets into the rootfs is debian packages installed > by multistrap. Plus whatever the configscripts do. > > Signed-off-by: Henning Schild > --- > .../recipes-core/images/files/multistrap.conf.in | 10 +++++- > meta/classes/ext4-img.bbclass | 2 +- > meta/classes/image.bbclass | 37 > ++++++++++------------ 3 files changed, 26 insertions(+), 23 > deletions(-) > > diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in > b/meta-isar/recipes-core/images/files/multistrap.conf.in index > 2d08c36..fd24baa 100644 --- > a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ > b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7 > @@ [General] > noauth=true > unpack=true > -bootstrap=##DISTRO## > +bootstrap=##DISTRO## Isar > aptsources=##DISTRO## > configscript=##CONFIG_SCRIPT## > setupscript=##SETUP_SCRIPT## > @@ -16,3 +16,11 @@ suite=##DISTRO_SUITE## > components=##DISTRO_COMPONENTS## > packages=##IMAGE_PREINSTALL## > omitdebsrc=true > + > +[Isar] > +packages=##IMAGE_INSTALL## > +source=##DEPLOY_DIR_DEB## > +suite=./ > +components= > +omitdebsrc=true > + > diff --git a/meta/classes/ext4-img.bbclass > b/meta/classes/ext4-img.bbclass index 61699b8..430032c 100644 > --- a/meta/classes/ext4-img.bbclass > +++ b/meta/classes/ext4-img.bbclass > @@ -36,4 +36,4 @@ do_ext4_image() { > fi > } > > -addtask ext4_image before do_build after do_populate > +addtask ext4_image before do_build after do_configure > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ab3d1ff..d9bd879 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -9,6 +9,18 @@ IMAGE_INSTALL ?= "" > WORKDIR = "${TMPDIR}/work/${PN}/${MACHINE}" > S = "${WORKDIR}/rootfs" > > +do_repository() { > + if [ -n "${IMAGE_INSTALL}" ]; then > + cd ${DEPLOY_DIR_DEB} > + sudo dpkg-scanpackages . > Packages > + sudo gzip Packages > + fi > +} > + > +addtask repository before do_build > +do_repository[deptask] = "do_build" > +do_repository[stamp-extra-info] = "${MACHINE}" > + > do_multistrap_conf() { > # Copy config file > install -m 644 ${FILESDIR}/multistrap.conf.in > ${WORKDIR}/multistrap.conf @@ -33,9 +45,11 @@ do_multistrap_conf() { > ${WORKDIR}/multistrap.conf > sed -i > 's|##DIR_HOOKS##|./tmp/work/${PN}/${MACHINE}/hooks_multistrap|' \ > ${WORKDIR}/multistrap.conf > + sed -i 's|##DEPLOY_DIR_DEB##|copy:///${DEPLOY_DIR_DEB}|' > ${WORKDIR}/multistrap.conf > + sed -i 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|' > ${WORKDIR}/multistrap.conf } > > -addtask multistrap_conf before do_build > +addtask multistrap_conf after do_repository before do_build > do_multistrap_conf[stamp-extra-info] = "${MACHINE}" > do_multistrap_conf[dirs] = "${WORKDIR}/hooks_multistrap" > > @@ -44,7 +58,7 @@ do_multistrap() { > > # Create root filesystem > sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f \ > - "${WORKDIR}/multistrap.conf" || true > + "${WORKDIR}/multistrap.conf" > } > > addtask multistrap after do_multistrap_conf before do_build > @@ -60,22 +74,3 @@ do_configure() { > > addtask configure after do_multistrap before do_build > do_configure[stamp-extra-info] = "${MACHINE}" > - > -# Install Debian packages, that were built from sources > -do_populate() { > - if [ -n "${IMAGE_INSTALL}" ]; then > - sudo mkdir -p ${S}/deb > - > - for p in ${IMAGE_INSTALL}; do > - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb > - done > - > - sudo chroot ${S} /usr/bin/dpkg -i -R /deb > - > - sudo rm -rf ${S}/deb > - fi > -} > - > -addtask populate after do_configure before do_build > -do_populate[deptask] = "do_build" > -do_populate[stamp-extra-info] = "${MACHINE}"