From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.28.212.2 with SMTP id l2mr157456wmg.6.1514985339124; Wed, 03 Jan 2018 05:15:39 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.193.142 with SMTP id r136ls309765wmf.0.canary-gmail; Wed, 03 Jan 2018 05:15:38 -0800 (PST) X-Google-Smtp-Source: ACJfBotx/O05S2OPQHS+RyYWA1b1foFG/pE9kFuBdAyykdhEeR/GWlQRBx4VflNmch+DlJu4fX98 X-Received: by 10.223.196.221 with SMTP id o29mr173109wrf.9.1514985338703; Wed, 03 Jan 2018 05:15:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514985338; cv=none; d=google.com; s=arc-20160816; b=lVI4IxLyzlF+y1oJZtKbNdoHEeLYxuaADWUCNLAHiD+SFMqs2yvN7dMBZFV3zAcp4W UkOzViGaM+rZ3qBoJOaVZh2OVLuEiRi0r7C+fUiUXi7JgRJ8Lums3TX4OMhiVjo3Tpe0 25L8u6oY9eBtWOdwXboM4abCd3o6eyQx5m+xzn+6gVDEXWL5q+bTd0ruQTrbVCx3Kjvx bE9NToIQNhnrpo2ZGzE+G0+VqwPazLlbjYCkUxlBjk8H7y44BWpYaY6LwTL1OXBbARTs ELLb/7D3AnTH+oCTleU4I9ZaraTNMnJ7TMohR0QFymezieotfHOK9vCupsPI9XG6SrRf a2Cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=HMbMIFpIExZ8qy8TKqV1rJUS4Asy28UMR5ec89KQXu4=; b=gJmgezVM97mBVhC0QTC69qs7vO1u9Jb+Eib1uFZn9CU2ljqhyqW7J9vEXUkYa+gsNQ OtQCI4HBiWWEF/OOYg7j72ArLHJmWyF0/2H5XylJ2lB4MW1Y3rqswTTcdthg/ihM4QAu mtmV+ztQzMAJu1bNxft2teSovv1UVmUr4TrgdoOl4c35ahfxAxhzRbjoYogGHbiZQw3Z GoqixH/LfpdiSQ/QjuMIvK/qcMlikPl77sjjbn5C8r6nSGbUWiIRl/J6BSaHgvqhH3ak dISWe9ygIjpKBqyBkj92XmraPOK6FZuGRwPdqhcbpwUM5ViPgXikzI9B2D9yf35zzKjx KX2w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id j81si1537479wmd.1.2018.01.03.05.15.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Jan 2018 05:15:38 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w03DFcTh026155 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 3 Jan 2018 14:15:38 +0100 Received: from mmd1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w03DFc2C031496; Wed, 3 Jan 2018 14:15:38 +0100 Date: Wed, 3 Jan 2018 14:15:37 +0100 From: Henning Schild To: Alexander Smirnov Cc: Jan Kiszka , Subject: Re: [RFC][PATCH 1/6] base-apt: Introduce fetching upstream apt Message-ID: <20180103141537.7123e332@mmd1pvb1c.ad001.siemens.net> In-Reply-To: References: <20180102145744.21814-1-asmirnov@ilbers.de> <20180102145744.21814-2-asmirnov@ilbers.de> <29246a20-e829-310d-eddf-ad955267cd2a@siemens.com> X-Mailer: Claws Mail 3.15.0-dirty (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: DV/SSWZ21giY Am Tue, 2 Jan 2018 20:02:26 +0300 schrieb Alexander Smirnov : > Hi, > > On 01/02/2018 07:15 PM, Jan Kiszka wrote: > > On 2018-01-02 15:57, Alexander Smirnov wrote: > >> This patch introduces mechanism how to fetch deb packages from > >> dedicated upstream apt repos and store them localy. Local > >> repository is called 'base-apt' and it will be used to generate > >> buildchroot and image root filesystems. Using 'base-apt' will > >> guarantee build reproducibility between builds. > >> > >> Signed-off-by: Alexander Smirnov > >> --- > >> meta/conf/isar-bitbake.conf | 2 + > >> meta/recipes-devtools/base-apt/base-apt.bb | 79 > >> ++++++++++++++++++++++ .../base-apt/files/distributions.in > >> | 3 + .../base-apt/files/multistrap.conf.in | 28 > >> ++++++++ 4 files changed, 112 insertions(+) > >> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb > >> create mode 100644 > >> meta/recipes-devtools/base-apt/files/distributions.in create mode > >> 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in > >> > >> diff --git a/meta/conf/isar-bitbake.conf > >> b/meta/conf/isar-bitbake.conf index 5a26743..df54399 100644 > >> --- a/meta/conf/isar-bitbake.conf > >> +++ b/meta/conf/isar-bitbake.conf > >> @@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = > >> "${TMPDIR}/deploy/deb/${MACHINE}" SSTATE_DIR ?= > >> "${TMPDIR}/sstate-cache" BUILDCHROOT_DIR = > >> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" > >> +BASE_APT_DIR ?= > >> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt" > > > > That should also go into TOPDIR, analogously to my patch for DL_DIR. > > > > No problem, have included this to fix-list for next series version. > > >> + > >> # Setup our default hash policy > >> BB_SIGNATURE_HANDLER ?= "noop" > >> > >> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb > >> b/meta/recipes-devtools/base-apt/base-apt.bb new file mode 100644 > >> index 0000000..b292d89 > >> --- /dev/null > >> +++ b/meta/recipes-devtools/base-apt/base-apt.bb > >> @@ -0,0 +1,79 @@ > >> +# Caching upstream apt repository to local one. > >> +# > >> +# This software is a part of ISAR. > >> +# Copyright (C) 2015-2017 ilbers GmbH > >> + > >> +DESCRIPTION = "Upstream apt caching" > >> + > >> +LICENSE = "gpl-2.0" > >> +LIC_FILES_CHKSUM = > >> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > >> + +FILESPATH =. "${LAYERDIR_core}/recipes-devtools/base-apt/files:" > >> +SRC_URI = "file://distributions.in \ > >> + file://multistrap.conf.in \ > >> + " > >> + > >> +BASE_PREINSTALL ?= "" > >> + > >> +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" > >> + > >> +BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf" > >> +do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}" > >> +do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > >> + > >> +do_get_base_apt() { > >> + # Adjust multistrap config > >> + sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \ > >> + -e > >> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' > >> \ > >> + -e > >> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' > >> \ > >> + -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ > >> + -e > >> 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ > >> + -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ > >> + -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ > >> + -e > >> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ > >> + -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ > >> + -e > >> 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > >> + "${WORKDIR}/multistrap.conf.in" > > >> "${WORKDIR}/multistrap.conf" + > >> + # Fetch deb packages > >> + sudo -E multistrap \ > >> + -a ${DISTRO_ARCH} \ > >> + -d "${WORKDIR}/download" \ > >> + -f "${WORKDIR}/multistrap.conf" \ > >> + > ${WORKDIR}/multistrap.log 2>&1 > >> + > >> + if [ ! -e "${BASE_APT_CONF_DIR}/distributions" ]; then > >> + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \ > >> + ${WORKDIR}/distributions.in > > >> ${BASE_APT_CONF_DIR}/distributions > >> + fi > >> + > >> + # Create reprepro cache > >> + if [ ! -d "${BASE_APT_DIR}/apt" ]; then > >> + reprepro -b ${BASE_APT_DIR}/apt \ > >> + --dbdir ${BASE_APT_DIR}/db \ > >> + export ${DISTRO_SUITE} > >> + fi > >> + > >> + # Process all the packages fetched by multistrap > >> + for deb in $(ls > >> ${WORKDIR}/download/var/cache/apt/archives/*.deb); > >> + do > >> + pn=$(dpkg-deb -I $deb | grep 'Package:' | cut -d ' ' -f 3) > >> + pv=$(dpkg-deb -I $deb | grep 'Version:' | cut -d ' ' -f 3) > >> + line=$(cat ${WORKDIR}/multistrap.log | grep -E '^(Get:)' > >> | grep " $pn ") > >> + url=$(echo $line | cut -d ' ' -f 2) > >> + component=$(basename $(echo $line | cut -d ' ' -f 3)) > >> + > >> + # Store download history > >> + echo $pn $pv $component $url >> ${WORKDIR}/deb.list > >> + > >> + reprepro -b ${BASE_APT_DIR}/apt \ > >> + --dbdir ${BASE_APT_DIR}/db \ > >> + -C $component \ > >> + includedeb ${DISTRO_SUITE} \ > >> + $deb > >> + done > >> + > >> + sudo rm -rf ${WORKDIR}/download > >> +} > >> + > >> +addtask get_base_apt after do_unpack before do_build > >> diff --git a/meta/recipes-devtools/base-apt/files/distributions.in > >> b/meta/recipes-devtools/base-apt/files/distributions.in new file > >> mode 100644 index 0000000..44e9513 > >> --- /dev/null > >> +++ b/meta/recipes-devtools/base-apt/files/distributions.in > >> @@ -0,0 +1,3 @@ > >> +Codename: {DISTRO_NAME} > >> +Architectures: i386 armhf amd64 source > >> +Components: main contrib non-free firmware > >> diff --git > >> a/meta/recipes-devtools/base-apt/files/multistrap.conf.in > >> b/meta/recipes-devtools/base-apt/files/multistrap.conf.in new file > >> mode 100644 index 0000000..27bf985 --- /dev/null > >> +++ b/meta/recipes-devtools/base-apt/files/multistrap.conf.in > >> @@ -0,0 +1,28 @@ > >> +# This software is a part of ISAR. > >> +# Copyright (C) 2015-2017 ilbers GmbH > >> + > >> +[General] > >> +noauth=true > >> +unpack=false > >> +ignorenativearch=true > >> +bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## > >> +aptsources=##DISTRO_MULTICONF_APTSOURCES## > >> + > >> +[base] > >> +source=##DISTRO_APT_SOURCE## > >> +suite=##DISTRO_SUITE## > >> +components=##DISTRO_COMPONENTS## > >> +packages=##BASE_PREINSTALL## > >> +omitdebsrc=true > >> + > >> +[updates] > >> +source=##DISTRO_APT_SOURCE## > >> +suite=##DISTRO_SUITE##-updates > >> +components=##DISTRO_COMPONENTS## > >> +omitdebsrc=true > >> + > >> +[security] > >> +source=##DISTRO_APT_SOURCE_SEC## > >> +suite=##DISTRO_SUITE##/updates > >> +components=##DISTRO_COMPONENTS## > >> +omitdebsrc=true > >> > > > > Hmm, is there really no way to share the common bits with > > meta/recipes-devtools/buildchroot/files/multistrap.conf.in? I > > suspect the want to be kept in sync, except for the General > > section. > > multistrap config for buildchroot and image will not use these > sections anymore, they are dropped later in this series. I think Jan was not just talking about the config. This would be the third copy of a multistrap-task in Isar. Consolidating the present two is on my whishlist for some time, i would not like to see yet another similar task being introduced. The task get_base_apt() should probably be split into at least two seperate tasks. multistrap and reprepro + fill But that will happen naturally if all multistrap-users use a multistrap base-class. Henning > Alex >