From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.223.176.180 with SMTP id i49mr4139727wra.12.1514912555920; Tue, 02 Jan 2018 09:02:35 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.48.16 with SMTP id w16ls1560320wmw.1.gmail; Tue, 02 Jan 2018 09:02:35 -0800 (PST) X-Google-Smtp-Source: ACJfBota3D/1eo1eAwZ6Kx/mxxA3+AU/jb12OyWt6A7B3gguYrL3hMOIyaw+Q+qtt17TzomeqT3f X-Received: by 10.28.5.202 with SMTP id 193mr4316304wmf.22.1514912555601; Tue, 02 Jan 2018 09:02:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514912555; cv=none; d=google.com; s=arc-20160816; b=yNYpbJ9b3SzgOk0Ws6g0t1zR/pSUQSOcFDgS9lPDPy5073PQYcTaMNgZXtieoYVzaR Ef6QoQx4ww30KQbaPtjACHSBVIMSfVfHbbxvqFyeR/qEMeE6FeGiw8z6CZ2xkigMwQBG y/Br0IDrCz1vcFbCwOAeKVD3U7BP5L/b6GRSC7O3kqISGbm/JpEMjHW6vndXgSGAsmpX PqfTMm6oR6ANwS8jTXjFSymx2dJQlQiKyHcciLgh14OdPEBKrLgvkWHsTRGTNLgA8A/M 3DJ6OUORjG51/y+wya8zW7dVlRW64HKzy+dVq3nuFPDNUB6ZEiibfpeUqDX5mcqlH/D5 46Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=zM0zmAA+OxiaPdUkxr7S0XEs16IZl/PLHe2Sr+7Z0mc=; b=t/UDblPmd8VL+7KQyUeyJj+LoPjKS3M8xPnXIe5Nwxvie4Q4uedb6e2vC4X1YA8d48 A104aUTgN/QpkBHG3IcuWnOdZYDwRvuOOzefNM9DoY5AXxW9mBWeQ7R+liYkeZUj7mRk I+qNEEMvCF6FmZXcyMwrcrfXmqmv/wLBz88ZWBvQhNoH7VNfQVA1CtfMI2Detyyrtyb2 pVkMNDHBk0zvT5rjf62O9vEdpYF0atZdIlYfRas+EirkcbzLEiCAnhTGdsh+InfRgVWB akBKmTzI3rGkRf/1SeomRKGqDFQAFG45OXm0IA5HEYOURhbyg3nuXdDM+sNHC++jMvQg CE3g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id i17si1070854wmh.4.2018.01.02.09.02.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 09:02:35 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w02H2WWI017128 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 2 Jan 2018 18:02:34 +0100 Subject: Re: [RFC][PATCH 1/6] base-apt: Introduce fetching upstream apt To: Jan Kiszka , isar-users@googlegroups.com References: <20180102145744.21814-1-asmirnov@ilbers.de> <20180102145744.21814-2-asmirnov@ilbers.de> <29246a20-e829-310d-eddf-ad955267cd2a@siemens.com> From: Alexander Smirnov Message-ID: Date: Tue, 2 Jan 2018 20:02:26 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <29246a20-e829-310d-eddf-ad955267cd2a@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: eur+wZtYYfud Hi, On 01/02/2018 07:15 PM, Jan Kiszka wrote: > On 2018-01-02 15:57, Alexander Smirnov wrote: >> This patch introduces mechanism how to fetch deb packages from dedicated >> upstream apt repos and store them localy. Local repository is called >> 'base-apt' and it will be used to generate buildchroot and image root >> filesystems. Using 'base-apt' will guarantee build reproducibility between >> builds. >> >> Signed-off-by: Alexander Smirnov >> --- >> meta/conf/isar-bitbake.conf | 2 + >> meta/recipes-devtools/base-apt/base-apt.bb | 79 ++++++++++++++++++++++ >> .../base-apt/files/distributions.in | 3 + >> .../base-apt/files/multistrap.conf.in | 28 ++++++++ >> 4 files changed, 112 insertions(+) >> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb >> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in >> create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in >> >> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf >> index 5a26743..df54399 100644 >> --- a/meta/conf/isar-bitbake.conf >> +++ b/meta/conf/isar-bitbake.conf >> @@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}" >> SSTATE_DIR ?= "${TMPDIR}/sstate-cache" >> BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" >> >> +BASE_APT_DIR ?= "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt" > > That should also go into TOPDIR, analogously to my patch for DL_DIR. > No problem, have included this to fix-list for next series version. >> + >> # Setup our default hash policy >> BB_SIGNATURE_HANDLER ?= "noop" >> >> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb >> new file mode 100644 >> index 0000000..b292d89 >> --- /dev/null >> +++ b/meta/recipes-devtools/base-apt/base-apt.bb >> @@ -0,0 +1,79 @@ >> +# Caching upstream apt repository to local one. >> +# >> +# This software is a part of ISAR. >> +# Copyright (C) 2015-2017 ilbers GmbH >> + >> +DESCRIPTION = "Upstream apt caching" >> + >> +LICENSE = "gpl-2.0" >> +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> + >> +FILESPATH =. "${LAYERDIR_core}/recipes-devtools/base-apt/files:" >> +SRC_URI = "file://distributions.in \ >> + file://multistrap.conf.in \ >> + " >> + >> +BASE_PREINSTALL ?= "" >> + >> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> + >> +BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf" >> +do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}" >> +do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> + >> +do_get_base_apt() { >> + # Adjust multistrap config >> + sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \ >> + -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> + -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> + -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> + -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> + -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> + -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> + -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ >> + -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> + -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> + "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> + >> + # Fetch deb packages >> + sudo -E multistrap \ >> + -a ${DISTRO_ARCH} \ >> + -d "${WORKDIR}/download" \ >> + -f "${WORKDIR}/multistrap.conf" \ >> + > ${WORKDIR}/multistrap.log 2>&1 >> + >> + if [ ! -e "${BASE_APT_CONF_DIR}/distributions" ]; then >> + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \ >> + ${WORKDIR}/distributions.in > ${BASE_APT_CONF_DIR}/distributions >> + fi >> + >> + # Create reprepro cache >> + if [ ! -d "${BASE_APT_DIR}/apt" ]; then >> + reprepro -b ${BASE_APT_DIR}/apt \ >> + --dbdir ${BASE_APT_DIR}/db \ >> + export ${DISTRO_SUITE} >> + fi >> + >> + # Process all the packages fetched by multistrap >> + for deb in $(ls ${WORKDIR}/download/var/cache/apt/archives/*.deb); >> + do >> + pn=$(dpkg-deb -I $deb | grep 'Package:' | cut -d ' ' -f 3) >> + pv=$(dpkg-deb -I $deb | grep 'Version:' | cut -d ' ' -f 3) >> + line=$(cat ${WORKDIR}/multistrap.log | grep -E '^(Get:)' | grep " $pn ") >> + url=$(echo $line | cut -d ' ' -f 2) >> + component=$(basename $(echo $line | cut -d ' ' -f 3)) >> + >> + # Store download history >> + echo $pn $pv $component $url >> ${WORKDIR}/deb.list >> + >> + reprepro -b ${BASE_APT_DIR}/apt \ >> + --dbdir ${BASE_APT_DIR}/db \ >> + -C $component \ >> + includedeb ${DISTRO_SUITE} \ >> + $deb >> + done >> + >> + sudo rm -rf ${WORKDIR}/download >> +} >> + >> +addtask get_base_apt after do_unpack before do_build >> diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in >> new file mode 100644 >> index 0000000..44e9513 >> --- /dev/null >> +++ b/meta/recipes-devtools/base-apt/files/distributions.in >> @@ -0,0 +1,3 @@ >> +Codename: {DISTRO_NAME} >> +Architectures: i386 armhf amd64 source >> +Components: main contrib non-free firmware >> diff --git a/meta/recipes-devtools/base-apt/files/multistrap.conf.in b/meta/recipes-devtools/base-apt/files/multistrap.conf.in >> new file mode 100644 >> index 0000000..27bf985 >> --- /dev/null >> +++ b/meta/recipes-devtools/base-apt/files/multistrap.conf.in >> @@ -0,0 +1,28 @@ >> +# This software is a part of ISAR. >> +# Copyright (C) 2015-2017 ilbers GmbH >> + >> +[General] >> +noauth=true >> +unpack=false >> +ignorenativearch=true >> +bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >> +aptsources=##DISTRO_MULTICONF_APTSOURCES## >> + >> +[base] >> +source=##DISTRO_APT_SOURCE## >> +suite=##DISTRO_SUITE## >> +components=##DISTRO_COMPONENTS## >> +packages=##BASE_PREINSTALL## >> +omitdebsrc=true >> + >> +[updates] >> +source=##DISTRO_APT_SOURCE## >> +suite=##DISTRO_SUITE##-updates >> +components=##DISTRO_COMPONENTS## >> +omitdebsrc=true >> + >> +[security] >> +source=##DISTRO_APT_SOURCE_SEC## >> +suite=##DISTRO_SUITE##/updates >> +components=##DISTRO_COMPONENTS## >> +omitdebsrc=true >> > > Hmm, is there really no way to share the common bits with > meta/recipes-devtools/buildchroot/files/multistrap.conf.in? I suspect > the want to be kept in sync, except for the General section. multistrap config for buildchroot and image will not use these sections anymore, they are dropped later in this series. Alex