From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6518759238035046400 X-Received: by 10.28.0.11 with SMTP id 11mr4390060wma.28.1517825184693; Mon, 05 Feb 2018 02:06:24 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.47.71 with SMTP id v68ls1428340wmv.6.gmail; Mon, 05 Feb 2018 02:06:24 -0800 (PST) X-Google-Smtp-Source: AH8x224zZgADSa3JsD+YfDbRzsi1FaXgqo8t6jlZBfapVnFyeK5SZWltZY5mgXEMWtLM6mAmQibc X-Received: by 10.28.116.24 with SMTP id p24mr4864944wmc.6.1517825184140; Mon, 05 Feb 2018 02:06:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517825184; cv=none; d=google.com; s=arc-20160816; b=TrnFQadFPaJMoeR84D3rpicuJ5jXr0TE/IxFbO853djjItwVqqZxRpEW/r2j3aKSmH VjelatnOyqgYhJrm85Amw1oGD5zLThcxQK48hDacwaKLUpmUpEG04BsZfcVPdpf38Q/l fRk629tilVRtaxcIdKxKCoCYj/shVTqZPGmrd5WU9QMR2tytXEedTtOTYuS8h/Vy95bo qh/TNpgpeCrNJfOqXg2sQS6hhcmlmx/PJ8ld0nVuk3b9FK+vnPaCMA4KSBXqk0dlDbkh G+XUS1Syqau7AAcvsVD5XTh2y3DULUWjeKjGFSmr8AeDD184X8myp5JaqpHjHbTddSCw 0pGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=nbndffYt3lIsm9ShQbJrQmu2rUZ2+kEICzvMCrWAMac=; b=gBayBlU62Q72o+5tYfcBk1wt7R6eWabTYPwHdUlaF4Ns2jEBeh6E8akAqbjyqgIoMn vv55VvO+1byjA606J/xySsCabvMyNXRWkkZaaLK2M/0vceVAvlrDL9ufN948AsQ/WTRB EXtKb2cIa126VcKiShVMzCyLfsDfCMDtYdgYhfHDcITMuM7N1JcoX84sji4YRRyEfusi KwtVYAnGzUKhKHB4FFjU1dP0zglAmsI+daxmxN0ou/MBRK8GdgwpmbmSBWfy/qIej2Kt Gl1N6SF+EAuZTiHe2so6RkxdDW5VG39vwqgS25IM5mWvb6Q04V56u3Wz4Fib4fy84Fb2 iQeQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 200si861244wml.2.2018.02.05.02.06.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Feb 2018 02:06:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w15A6K9r024456 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 5 Feb 2018 11:06:21 +0100 Subject: Re: [PATCH 1/5] base: add populate_repo task to include distro packages to the repo To: Cedric_Hombourger@mentor.com, isar-users@googlegroups.com References: <20180204175454.220-1-Cedric_Hombourger@mentor.com> <20180204175454.220-2-Cedric_Hombourger@mentor.com> From: Alexander Smirnov Message-ID: <5e18edf4-a6b0-c85c-d50e-67a9aaeab0cb@ilbers.de> Date: Mon, 5 Feb 2018 13:06:15 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20180204175454.220-2-Cedric_Hombourger@mentor.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: CuKA173CBOWW Hi, n 02/04/2018 08:54 PM, Cedric_Hombourger@mentor.com wrote: > From: Cedric Hombourger > > A package repository is already created for custom packages (built by > Isar). Provide a populate_repo task to also include any packages > included into images built by Isar. This mechanism will add both binary > and source packages. The augmented repository may be used later for > reproducible builds (technically, packages pulled for the buildchroots > we build should also be added; this will be the subject of another > changeset). Isar-apt should never be used to mirror upstream repositories. Isar-apt intended to be an overlay for pure Debian. Regarding reproducibility, there are already two series discusses in this list which introduce dedicated base-apt repository for caching upstream: https://groups.google.com/d/msg/isar-users/9b0f6LP4X6k/LJ3mLxE1EAAJ https://groups.google.com/d/msg/isar-users/QQUsVmSaAGk/B6FGnVWhAQAJ Alex > > Signed-off-by: Cedric Hombourger > --- > .../recipes-core/images/files/multistrap.conf.in | 6 +- > meta-isar/recipes-core/images/isar-image-base.bb | 78 ++++++++++++++++++++++ > 2 files changed, 81 insertions(+), 3 deletions(-) > > diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in > index 432b6af..8e4e276 100644 > --- a/meta-isar/recipes-core/images/files/multistrap.conf.in > +++ b/meta-isar/recipes-core/images/files/multistrap.conf.in > @@ -16,19 +16,19 @@ source=##DISTRO_APT_SOURCE## > suite=##DISTRO_SUITE## > components=##DISTRO_COMPONENTS## > packages=##IMAGE_PREINSTALL## > -omitdebsrc=true > +omitdebsrc=false > > [updates] > source=##DISTRO_APT_SOURCE## > suite=##DISTRO_SUITE##-updates > components=##DISTRO_COMPONENTS## > -omitdebsrc=true > +omitdebsrc=false > > [security] > source=##DISTRO_APT_SOURCE_SEC## > suite=##DISTRO_SUITE##/updates > components=##DISTRO_COMPONENTS## > -omitdebsrc=true > +omitdebsrc=false > > [Isar] > packages=##IMAGE_INSTALL## > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb > index dea2cdb..6ce3d6d 100644 > --- a/meta-isar/recipes-core/images/isar-image-base.bb > +++ b/meta-isar/recipes-core/images/isar-image-base.bb > @@ -52,6 +52,79 @@ do_rootfs_populate_dev() { > _mknod_cdev 666 ptmx 5 2 > } > > +do_populate_repo() { > + > + APT_CACHEDIR=${IMAGE_ROOTFS}/var/cache/apt # Home of the apt cache (for the target image) > + APT_DEBDIR=${APT_CACHEDIR}/archives # Downloaded packages can be found there > + APT_SRCDIR=${WORKDIR}/apt-sources # Put source packages outside of the rootfs > + > + sudo mkdir -p ${APT_SRCDIR} > + > + apt_status_file=$(mktemp) > + for p in ${DEPLOY_DIR_DEB}/${PN}/*.deb; do > + # Only process packages that were NOT built by Isar (already added to the repo) > + deb=$(basename ${p}) > + if [ ! -f ${DEPLOY_DIR_DEB}/${deb} ]; then > + # Extract package name > + pkg=$(dpkg-deb -I ${p}|grep -e '^ Package:'|sed -e 's,^ Package: ,,g') > + > + # Extract package section and priority so we can tell rerepro where to place this package > + sect=$(dpkg-deb -I ${p}|grep -e '^ Section:'|sed -e 's,^ Section: ,,g') > + prio=$(dpkg-deb -I ${p}|grep -e '^ Priority:'|sed -e 's,^ Priority: ,,g') > + > + # Determine name of the source package (same as the binary package if not specified) > + src=$(dpkg-deb -I ${p}|grep -e '^ Source:'|sed -e 's,^ Source: ,,g'|awk '{ print $1; }') > + src=${src:-${pkg}} > + > + # Add the binary package to the repo > + call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > + --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ > + -C main -S ${sect} -P ${prio} \ > + includedeb ${DEBDISTRONAME} ${p} > + > + # Get source package and add it to the repo > + # Notes: > + # - multistrap --source-dir is failing to fetch sources hence this manual process > + # (the culprit appears to be the following apt-get option: APT::Default-Release='*') > + # - the apt-get command-line was copied from multistrap and tweaked for sources > + # (a separate/temporary apt status file is used to avoid polluting the existing one > + # with source transactions) > + # - as we assume sources were already fetched if the directory exists, quietly > + # attempt to delete it first; if sources are indeed there, it will fetch. If > + # they aren't, the directory will be deleted and we will have another go at it > + rmdir ${APT_SRCDIR}/${src} 2>/dev/null || true > + if [ ! -d "${APT_SRCDIR}/${src}" ]; then > + sudo mkdir -p ${APT_SRCDIR}/${src} > + cd ${APT_SRCDIR}/${src} > + bbnote "fetching source package '${src}' (binary: ${pkg})" > + sudo -E apt-get \ > + -o Dir::Etc::TrustedParts=${IMAGE_ROOTFS}/etc/apt/trusted.gpg.d \ > + -o Dir::Etc::Trusted=${IMAGE_ROOTFS}/etc/apt/trusted.gpg \ > + -o Apt::Get::AllowUnauthenticated=true \ > + -o Apt::Get::Download-Only=true \ > + -o Apt::Install-Recommends=false \ > + -o Dir=${IMAGE_ROOTFS} \ > + -o Dir::Etc=${IMAGE_ROOTFS}/etc/apt/ \ > + -o Dir::Etc::Parts=${IMAGE_ROOTFS}/etc/apt/apt.conf.d/ \ > + -o Dir::Etc::PreferencesParts=${IMAGE_ROOTFS}/etc/apt/preferences.d/ \ > + -o Dir::State=${IMAGE_ROOTFS}/var/lib/apt/ \ > + -o Dir::State::Status=${apt_status_file} \ > + -o Dir::Cache=${IMAGE_ROOTFS}/var/cache/apt/ -y source ${src} > + call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ > + --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ > + -C main -S ${sect} -P ${prio} \ > + includedsc ${DEBDISTRONAME} \ > + ${APT_SRCDIR}/${src}/*.dsc > + else > + bbnote "already fetched source package '${src}' (binary: ${pkg})" > + fi > + fi > + done > + rm -f ${apt_status_file} > +} > + > +addtask populate_repo > + > do_rootfs() { > E="${@ bb.utils.export_proxies(d)}" > > @@ -94,6 +167,11 @@ do_rootfs() { > # because of proxy settings > sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" > > + # Copy packages before they vanish when the root file-systems get configured > + # (apt purge is being run) > + install -m 755 -d ${DEPLOY_DIR_DEB}/${PN} > + sudo cp -l ${IMAGE_ROOTFS}/var/cache/apt/archives/*.deb ${DEPLOY_DIR_DEB}/${PN}/ > + > # Configure root filesystem > sudo chroot ${IMAGE_ROOTFS} /${DISTRO_CONFIG_SCRIPT} ${MACHINE_SERIAL} ${BAUDRATE_TTY} \ > ${ROOTFS_DEV} > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov