From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6506467811408216064 X-Received: by 10.28.0.78 with SMTP id 75mr4087769wma.31.1514905090697; Tue, 02 Jan 2018 06:58:10 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.135.241 with SMTP id c46ls9355549wrc.8.gmail; Tue, 02 Jan 2018 06:58:10 -0800 (PST) X-Google-Smtp-Source: ACJfBovmhXliQj3RzkQvbeRJI1YtsQ2yJjTgMPrbR7axXqsHFzX7I2CS+VFl3nQEbagMu8Oy44RC X-Received: by 10.28.139.211 with SMTP id n202mr4701797wmd.1.1514905090296; Tue, 02 Jan 2018 06:58:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1514905090; cv=none; d=google.com; s=arc-20160816; b=xC+S7Ajjuo7kQ/i7P/3GTC7ZgpdX5QK0b2k6DCi52Rb+T3EVswOa4QpOYRiHHd3aVq dUomh4BXTjXqRhxcJinw8ATsW4rUuIL49yGJ1BLbqPh36z/gG2ic1hAVw49an3GjDhvu ue+yGJYhhnPupMn1bYXWWKqZp/uhyvvWL2MdYKKkCT2d57onAKRTXk9pJ/tVz2HoaldB mYff7GOKKGuuDQzZXQja7B6kk1zsUSwhkbMGnw1hkymfwprrlYMReSLRwoyC7gXr3nHg uuw15PD9jnD/FpC0ukqCSGjfP0Q37kGwqOTNS7YKEhGf3U7lR/o/DmwN9cRITY4nR7HX ZCTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=AAjhxgdyBLb5mUMeXdfWSa8q9/xK4Vc6XQ3GyAuR4Fw=; b=EnEEefd0x7DtJvSIpvukisGcVu+BAdgVPiatKgjnkhFAcy5JSYT1YMGnRKd5PnuM7+ ZMLNxFJMUUNvXSNfYo9yzsvz1/s9k/8aI3Xh2NdZaaOICNBjxSOx6O9Yw4sjEatwD6Ol Fu5+DN/JLj5FY+u0Vqm5to+BePotmWLZGDtGJdLMx59+IWCuPnKdorAMOvQc7F9QtWO2 rxsTBRfFiOr/RVtOgB+gCDxbNvk0UVqHdhmavY0RY57KrwI3p7gCresRxG7TCAhZCKgs n1KBBwKpWoLYocgMpkdDOCCV27UcFkGOfTi4hGZM5v+QvwzUSO0LhcFiJw1otLmI2SPR tw7Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id r70si1177249wmg.2.2018.01.02.06.58.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 06:58:10 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from localhost.localdomain ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w02Evpwg016822 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 2 Jan 2018 15:57:59 +0100 From: Alexander Smirnov To: isar-users@googlegroups.com Cc: Alexander Smirnov Subject: [RFC][PATCH 1/6] base-apt: Introduce fetching upstream apt Date: Tue, 2 Jan 2018 17:57:39 +0300 Message-Id: <20180102145744.21814-2-asmirnov@ilbers.de> X-Mailer: git-send-email 2.9.5 In-Reply-To: <20180102145744.21814-1-asmirnov@ilbers.de> References: <20180102145744.21814-1-asmirnov@ilbers.de> X-TUID: JaYSXzDDsTtX This patch introduces mechanism how to fetch deb packages from dedicated upstream apt repos and store them localy. Local repository is called 'base-apt' and it will be used to generate buildchroot and image root filesystems. Using 'base-apt' will guarantee build reproducibility between builds. Signed-off-by: Alexander Smirnov --- meta/conf/isar-bitbake.conf | 2 + meta/recipes-devtools/base-apt/base-apt.bb | 79 ++++++++++++++++++++++ .../base-apt/files/distributions.in | 3 + .../base-apt/files/multistrap.conf.in | 28 ++++++++ 4 files changed, 112 insertions(+) create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf index 5a26743..df54399 100644 --- a/meta/conf/isar-bitbake.conf +++ b/meta/conf/isar-bitbake.conf @@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}" SSTATE_DIR ?= "${TMPDIR}/sstate-cache" BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" +BASE_APT_DIR ?= "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt" + # Setup our default hash policy BB_SIGNATURE_HANDLER ?= "noop" diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb new file mode 100644 index 0000000..b292d89 --- /dev/null +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -0,0 +1,79 @@ +# Caching upstream apt repository to local one. +# +# This software is a part of ISAR. +# Copyright (C) 2015-2017 ilbers GmbH + +DESCRIPTION = "Upstream apt caching" + +LICENSE = "gpl-2.0" +LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" + +FILESPATH =. "${LAYERDIR_core}/recipes-devtools/base-apt/files:" +SRC_URI = "file://distributions.in \ + file://multistrap.conf.in \ + " + +BASE_PREINSTALL ?= "" + +WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" + +BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf" +do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}" +do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" + +do_get_base_apt() { + # Adjust multistrap config + sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \ + -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ + -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ + -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ + -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ + -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ + -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ + -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ + -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ + -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ + "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" + + # Fetch deb packages + sudo -E multistrap \ + -a ${DISTRO_ARCH} \ + -d "${WORKDIR}/download" \ + -f "${WORKDIR}/multistrap.conf" \ + > ${WORKDIR}/multistrap.log 2>&1 + + if [ ! -e "${BASE_APT_CONF_DIR}/distributions" ]; then + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \ + ${WORKDIR}/distributions.in > ${BASE_APT_CONF_DIR}/distributions + fi + + # Create reprepro cache + if [ ! -d "${BASE_APT_DIR}/apt" ]; then + reprepro -b ${BASE_APT_DIR}/apt \ + --dbdir ${BASE_APT_DIR}/db \ + export ${DISTRO_SUITE} + fi + + # Process all the packages fetched by multistrap + for deb in $(ls ${WORKDIR}/download/var/cache/apt/archives/*.deb); + do + pn=$(dpkg-deb -I $deb | grep 'Package:' | cut -d ' ' -f 3) + pv=$(dpkg-deb -I $deb | grep 'Version:' | cut -d ' ' -f 3) + line=$(cat ${WORKDIR}/multistrap.log | grep -E '^(Get:)' | grep " $pn ") + url=$(echo $line | cut -d ' ' -f 2) + component=$(basename $(echo $line | cut -d ' ' -f 3)) + + # Store download history + echo $pn $pv $component $url >> ${WORKDIR}/deb.list + + reprepro -b ${BASE_APT_DIR}/apt \ + --dbdir ${BASE_APT_DIR}/db \ + -C $component \ + includedeb ${DISTRO_SUITE} \ + $deb + done + + sudo rm -rf ${WORKDIR}/download +} + +addtask get_base_apt after do_unpack before do_build diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in new file mode 100644 index 0000000..44e9513 --- /dev/null +++ b/meta/recipes-devtools/base-apt/files/distributions.in @@ -0,0 +1,3 @@ +Codename: {DISTRO_NAME} +Architectures: i386 armhf amd64 source +Components: main contrib non-free firmware diff --git a/meta/recipes-devtools/base-apt/files/multistrap.conf.in b/meta/recipes-devtools/base-apt/files/multistrap.conf.in new file mode 100644 index 0000000..27bf985 --- /dev/null +++ b/meta/recipes-devtools/base-apt/files/multistrap.conf.in @@ -0,0 +1,28 @@ +# This software is a part of ISAR. +# Copyright (C) 2015-2017 ilbers GmbH + +[General] +noauth=true +unpack=false +ignorenativearch=true +bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## +aptsources=##DISTRO_MULTICONF_APTSOURCES## + +[base] +source=##DISTRO_APT_SOURCE## +suite=##DISTRO_SUITE## +components=##DISTRO_COMPONENTS## +packages=##BASE_PREINSTALL## +omitdebsrc=true + +[updates] +source=##DISTRO_APT_SOURCE## +suite=##DISTRO_SUITE##-updates +components=##DISTRO_COMPONENTS## +omitdebsrc=true + +[security] +source=##DISTRO_APT_SOURCE_SEC## +suite=##DISTRO_SUITE##/updates +components=##DISTRO_COMPONENTS## +omitdebsrc=true -- 2.1.4