From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.80.203.138 with SMTP id k10mr3557962edi.5.1514909734360; Tue, 02 Jan 2018 08:15:34 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.80.212.18 with SMTP id t18ls11838980edh.6.gmail; Tue, 02 Jan 2018 08:15:33 -0800 (PST) X-Google-Smtp-Source: ACJfBos1Ws6VHPnAKeGogPRST91FE7kiYu/D+v7tju0gRg3uIjelZ+5D6rglFIq5DW7WtBnNiYCG X-Received: by 10.80.164.203 with SMTP id x11mr12177548edb.7.1514909733948; Tue, 02 Jan 2018 08:15:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514909733; cv=none; d=google.com; s=arc-20160816; b=EKRH5dtxRWdllwih3jg2umtyW0Dy99i4OShYpj8+AZa4g5HZx3zgjX5Z6A4d2bHh51 U7vscgWwubqTFg2K1KzKHEmjPsNfOFBsHGcKM/WjspIi0DUV/+4CIZKmZwoPxO0YUG1g UozbZsofcfdwsx+Tbs9ZUAG2jsjkI0RDeOFe1psoCkPZLR/N8ZUIUWnmxz1dKKaD8oIM OYZIhgJpkULiN0ZnbaBO2b2qg+pBs9RoETa1ufjYsiz5C3y/yq4X9Kt3FmMXS9eTO2r+ GPKK/dTN6WNgkYeElGjxM0N0Snioiod9161SpeUkxl0CNLsNt5/6B9/SccFFngdWdheF svXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=hwNJ2kUDq7vtum9qbHNMx7Z9dp5zQa+8pgAq4DvkRBc=; b=MHjOEIcq6cO6B/nF3R+FUUZwBG+tNTTUjncSkMynTp1P8TWy3pU0u7UtSeEgRxgfys 3Xx8HAtuuVP3N067QbIJC5NCEvCtaMyJYRXgMJFhd+U+AsFgv+eQ8PGowQ5Yi2NydqEG KKRUw56nIIDFEZoxKA6veDtCJT7oIMJfSf/Q1dyDLrWNpXfW6y1Pw/XMk1lvL9dFrAGu 91OK6zWRP1JZlem7aulqcN7fIC0e3UCn7iFWH73Js2GxjgVgmR2K5Wtjz6DjZzjQ6Zq1 n+jN5KAEAtJolk/ZwbJuEQU5TJJqMNd2ceIDN5ler2LUWihjHsqakInw/b3qZnmdIdvS y8rw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id r58si2676171eda.3.2018.01.02.08.15.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 08:15:33 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w02GFXlL011530 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Jan 2018 17:15:33 +0100 Received: from md1f2u6c.ww002.siemens.net (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w02GFXbW012734; Tue, 2 Jan 2018 17:15:33 +0100 Subject: Re: [RFC][PATCH 1/6] base-apt: Introduce fetching upstream apt To: Alexander Smirnov , isar-users@googlegroups.com References: <20180102145744.21814-1-asmirnov@ilbers.de> <20180102145744.21814-2-asmirnov@ilbers.de> From: Jan Kiszka Message-ID: <29246a20-e829-310d-eddf-ad955267cd2a@siemens.com> Date: Tue, 2 Jan 2018 17:15:33 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20180102145744.21814-2-asmirnov@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: cvm4Fu9tLvWZ On 2018-01-02 15:57, Alexander Smirnov wrote: > This patch introduces mechanism how to fetch deb packages from dedicated > upstream apt repos and store them localy. Local repository is called > 'base-apt' and it will be used to generate buildchroot and image root > filesystems. Using 'base-apt' will guarantee build reproducibility between > builds. > > Signed-off-by: Alexander Smirnov > --- > meta/conf/isar-bitbake.conf | 2 + > meta/recipes-devtools/base-apt/base-apt.bb | 79 ++++++++++++++++++++++ > .../base-apt/files/distributions.in | 3 + > .../base-apt/files/multistrap.conf.in | 28 ++++++++ > 4 files changed, 112 insertions(+) > create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb > create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in > create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in > > diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf > index 5a26743..df54399 100644 > --- a/meta/conf/isar-bitbake.conf > +++ b/meta/conf/isar-bitbake.conf > @@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}" > SSTATE_DIR ?= "${TMPDIR}/sstate-cache" > BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" > > +BASE_APT_DIR ?= "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt" That should also go into TOPDIR, analogously to my patch for DL_DIR. > + > # Setup our default hash policy > BB_SIGNATURE_HANDLER ?= "noop" > > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb > new file mode 100644 > index 0000000..b292d89 > --- /dev/null > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > @@ -0,0 +1,79 @@ > +# Caching upstream apt repository to local one. > +# > +# This software is a part of ISAR. > +# Copyright (C) 2015-2017 ilbers GmbH > + > +DESCRIPTION = "Upstream apt caching" > + > +LICENSE = "gpl-2.0" > +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > + > +FILESPATH =. "${LAYERDIR_core}/recipes-devtools/base-apt/files:" > +SRC_URI = "file://distributions.in \ > + file://multistrap.conf.in \ > + " > + > +BASE_PREINSTALL ?= "" > + > +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" > + > +BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf" > +do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}" > +do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > + > +do_get_base_apt() { > + # Adjust multistrap config > + sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \ > + -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ > + -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ > + -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ > + -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ > + -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ > + -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ > + -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ > + -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ > + -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > + "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" > + > + # Fetch deb packages > + sudo -E multistrap \ > + -a ${DISTRO_ARCH} \ > + -d "${WORKDIR}/download" \ > + -f "${WORKDIR}/multistrap.conf" \ > + > ${WORKDIR}/multistrap.log 2>&1 > + > + if [ ! -e "${BASE_APT_CONF_DIR}/distributions" ]; then > + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \ > + ${WORKDIR}/distributions.in > ${BASE_APT_CONF_DIR}/distributions > + fi > + > + # Create reprepro cache > + if [ ! -d "${BASE_APT_DIR}/apt" ]; then > + reprepro -b ${BASE_APT_DIR}/apt \ > + --dbdir ${BASE_APT_DIR}/db \ > + export ${DISTRO_SUITE} > + fi > + > + # Process all the packages fetched by multistrap > + for deb in $(ls ${WORKDIR}/download/var/cache/apt/archives/*.deb); > + do > + pn=$(dpkg-deb -I $deb | grep 'Package:' | cut -d ' ' -f 3) > + pv=$(dpkg-deb -I $deb | grep 'Version:' | cut -d ' ' -f 3) > + line=$(cat ${WORKDIR}/multistrap.log | grep -E '^(Get:)' | grep " $pn ") > + url=$(echo $line | cut -d ' ' -f 2) > + component=$(basename $(echo $line | cut -d ' ' -f 3)) > + > + # Store download history > + echo $pn $pv $component $url >> ${WORKDIR}/deb.list > + > + reprepro -b ${BASE_APT_DIR}/apt \ > + --dbdir ${BASE_APT_DIR}/db \ > + -C $component \ > + includedeb ${DISTRO_SUITE} \ > + $deb > + done > + > + sudo rm -rf ${WORKDIR}/download > +} > + > +addtask get_base_apt after do_unpack before do_build > diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in > new file mode 100644 > index 0000000..44e9513 > --- /dev/null > +++ b/meta/recipes-devtools/base-apt/files/distributions.in > @@ -0,0 +1,3 @@ > +Codename: {DISTRO_NAME} > +Architectures: i386 armhf amd64 source > +Components: main contrib non-free firmware > diff --git a/meta/recipes-devtools/base-apt/files/multistrap.conf.in b/meta/recipes-devtools/base-apt/files/multistrap.conf.in > new file mode 100644 > index 0000000..27bf985 > --- /dev/null > +++ b/meta/recipes-devtools/base-apt/files/multistrap.conf.in > @@ -0,0 +1,28 @@ > +# This software is a part of ISAR. > +# Copyright (C) 2015-2017 ilbers GmbH > + > +[General] > +noauth=true > +unpack=false > +ignorenativearch=true > +bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## > +aptsources=##DISTRO_MULTICONF_APTSOURCES## > + > +[base] > +source=##DISTRO_APT_SOURCE## > +suite=##DISTRO_SUITE## > +components=##DISTRO_COMPONENTS## > +packages=##BASE_PREINSTALL## > +omitdebsrc=true > + > +[updates] > +source=##DISTRO_APT_SOURCE## > +suite=##DISTRO_SUITE##-updates > +components=##DISTRO_COMPONENTS## > +omitdebsrc=true > + > +[security] > +source=##DISTRO_APT_SOURCE_SEC## > +suite=##DISTRO_SUITE##/updates > +components=##DISTRO_COMPONENTS## > +omitdebsrc=true > Hmm, is there really no way to share the common bits with meta/recipes-devtools/buildchroot/files/multistrap.conf.in? I suspect the want to be kept in sync, except for the General section. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux