From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6518759238035046400 X-Received: by 10.107.173.69 with SMTP id w66mr33124598ioe.19.1517766911921; Sun, 04 Feb 2018 09:55:11 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.36.73.215 with SMTP id e84ls1434894itd.8.canary-gmail; Sun, 04 Feb 2018 09:55:11 -0800 (PST) X-Google-Smtp-Source: AH8x224e+g00WuSdQcbDEJqnFiCusYo5+patXpWrEWqjRcLOHRLvzmeP/eLaBzQ+aff4vgy0FP0s X-Received: by 10.36.51.141 with SMTP id k135mr35878775itk.19.1517766911679; Sun, 04 Feb 2018 09:55:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517766911; cv=none; d=google.com; s=arc-20160816; b=lZ5M7emNwx9qkpXjKoomMn3ysk1GODnI9rl43Cu24tFUXknxv8hbzOYuNOsiOV46El L9HCnpuihxJIqVhaVAAdPs5WQYCdWiXs60Alz0nboLM2+n9PcgPg3jrsmKBZCIN+SLMy vw2MLm7oC2fJHGbzdP3mZPuU6o7eu216u+wCKlO9+PSiAm/ZALFuGc8kz4Vm9bd+r5Dg hFARhxTfVbF7j5LbLB6LjDi977fEPWDhAsrVn6a+qzasVpTerpRgU0TKUqSuBH8v8ldQ h2pQsmjX2Qg0e8U9KMconybb91TWG7BKAzwEzVL2i2qmIjN/UtgrEoO1sF1opiwR6ZfJ eiWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:arc-authentication-results; bh=V0zmPqEzjafTAfRChfVnJxZaZDWexAV7BFgas95F10k=; b=MihtSv7kufQ1budHreYeNjLL9fzqlFiqzzIa6+oCxBuz592SqzRRQaTui7x4iq+FIY lwzY3MYKzTjKKvKZBrcMqpkbxG8rCA0hYh0xj8188CeZSqPAO0WnPviXm3TFHGpgylLW FkEJmSJEcjhZNy1QY3cUdj7P8/DcZwmbvwHnaBmIakVBfjKJC8AbsOQfasfznor9XqED UY2JdMpzQsANEx7G8RUAzwgayzKoGukghadyDhTSqvmKI0LTw5IVhPiOZ48xupr0WUTK KC/mvqxwrPzC7eNFQCvczFFfxMa9kzrhB3ml+9x976Kly2lV5X90to361XIBmYHIIF/8 aK7g== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from relay1.mentorg.com (relay1.mentorg.com. [192.94.38.131]) by gmr-mx.google.com with ESMTPS id q12si436355ioh.5.2018.02.04.09.55.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 04 Feb 2018 09:55:11 -0800 (PST) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) client-ip=192.94.38.131; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 192.94.38.131 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Received: from nat-ies.mentorg.com ([192.94.31.2] helo=svr-ies-mbx-02.mgc.mentorg.com) by relay1.mentorg.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-SHA384:256) id 1eiOVS-0005c6-Mm from Cedric_Hombourger@mentor.com for isar-users@googlegroups.com; Sun, 04 Feb 2018 09:55:10 -0800 Received: from 3a23c91bf2ab.anacadf.mentorg.com (137.202.0.87) by svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) with Microsoft SMTP Server (TLS) id 15.0.1320.4; Sun, 4 Feb 2018 17:55:06 +0000 From: To: CC: Cedric Hombourger Subject: [PATCH 1/5] base: add populate_repo task to include distro packages to the repo Date: Sun, 4 Feb 2018 17:54:50 +0000 Message-ID: <20180204175454.220-2-Cedric_Hombourger@mentor.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180204175454.220-1-Cedric_Hombourger@mentor.com> References: <20180204175454.220-1-Cedric_Hombourger@mentor.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [137.202.0.87] X-ClientProxiedBy: svr-ies-mbx-01.mgc.mentorg.com (139.181.222.1) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: 6UuUC1Dtw/mB From: Cedric Hombourger A package repository is already created for custom packages (built by Isar). Provide a populate_repo task to also include any packages included into images built by Isar. This mechanism will add both binary and source packages. The augmented repository may be used later for reproducible builds (technically, packages pulled for the buildchroots we build should also be added; this will be the subject of another changeset). Signed-off-by: Cedric Hombourger --- .../recipes-core/images/files/multistrap.conf.in | 6 +- meta-isar/recipes-core/images/isar-image-base.bb | 78 ++++++++++++++++++++++ 2 files changed, 81 insertions(+), 3 deletions(-) diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in index 432b6af..8e4e276 100644 --- a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -16,19 +16,19 @@ source=##DISTRO_APT_SOURCE## suite=##DISTRO_SUITE## components=##DISTRO_COMPONENTS## packages=##IMAGE_PREINSTALL## -omitdebsrc=true +omitdebsrc=false [updates] source=##DISTRO_APT_SOURCE## suite=##DISTRO_SUITE##-updates components=##DISTRO_COMPONENTS## -omitdebsrc=true +omitdebsrc=false [security] source=##DISTRO_APT_SOURCE_SEC## suite=##DISTRO_SUITE##/updates components=##DISTRO_COMPONENTS## -omitdebsrc=true +omitdebsrc=false [Isar] packages=##IMAGE_INSTALL## diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb index dea2cdb..6ce3d6d 100644 --- a/meta-isar/recipes-core/images/isar-image-base.bb +++ b/meta-isar/recipes-core/images/isar-image-base.bb @@ -52,6 +52,79 @@ do_rootfs_populate_dev() { _mknod_cdev 666 ptmx 5 2 } +do_populate_repo() { + + APT_CACHEDIR=${IMAGE_ROOTFS}/var/cache/apt # Home of the apt cache (for the target image) + APT_DEBDIR=${APT_CACHEDIR}/archives # Downloaded packages can be found there + APT_SRCDIR=${WORKDIR}/apt-sources # Put source packages outside of the rootfs + + sudo mkdir -p ${APT_SRCDIR} + + apt_status_file=$(mktemp) + for p in ${DEPLOY_DIR_DEB}/${PN}/*.deb; do + # Only process packages that were NOT built by Isar (already added to the repo) + deb=$(basename ${p}) + if [ ! -f ${DEPLOY_DIR_DEB}/${deb} ]; then + # Extract package name + pkg=$(dpkg-deb -I ${p}|grep -e '^ Package:'|sed -e 's,^ Package: ,,g') + + # Extract package section and priority so we can tell rerepro where to place this package + sect=$(dpkg-deb -I ${p}|grep -e '^ Section:'|sed -e 's,^ Section: ,,g') + prio=$(dpkg-deb -I ${p}|grep -e '^ Priority:'|sed -e 's,^ Priority: ,,g') + + # Determine name of the source package (same as the binary package if not specified) + src=$(dpkg-deb -I ${p}|grep -e '^ Source:'|sed -e 's,^ Source: ,,g'|awk '{ print $1; }') + src=${src:-${pkg}} + + # Add the binary package to the repo + call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ + --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ + -C main -S ${sect} -P ${prio} \ + includedeb ${DEBDISTRONAME} ${p} + + # Get source package and add it to the repo + # Notes: + # - multistrap --source-dir is failing to fetch sources hence this manual process + # (the culprit appears to be the following apt-get option: APT::Default-Release='*') + # - the apt-get command-line was copied from multistrap and tweaked for sources + # (a separate/temporary apt status file is used to avoid polluting the existing one + # with source transactions) + # - as we assume sources were already fetched if the directory exists, quietly + # attempt to delete it first; if sources are indeed there, it will fetch. If + # they aren't, the directory will be deleted and we will have another go at it + rmdir ${APT_SRCDIR}/${src} 2>/dev/null || true + if [ ! -d "${APT_SRCDIR}/${src}" ]; then + sudo mkdir -p ${APT_SRCDIR}/${src} + cd ${APT_SRCDIR}/${src} + bbnote "fetching source package '${src}' (binary: ${pkg})" + sudo -E apt-get \ + -o Dir::Etc::TrustedParts=${IMAGE_ROOTFS}/etc/apt/trusted.gpg.d \ + -o Dir::Etc::Trusted=${IMAGE_ROOTFS}/etc/apt/trusted.gpg \ + -o Apt::Get::AllowUnauthenticated=true \ + -o Apt::Get::Download-Only=true \ + -o Apt::Install-Recommends=false \ + -o Dir=${IMAGE_ROOTFS} \ + -o Dir::Etc=${IMAGE_ROOTFS}/etc/apt/ \ + -o Dir::Etc::Parts=${IMAGE_ROOTFS}/etc/apt/apt.conf.d/ \ + -o Dir::Etc::PreferencesParts=${IMAGE_ROOTFS}/etc/apt/preferences.d/ \ + -o Dir::State=${IMAGE_ROOTFS}/var/lib/apt/ \ + -o Dir::State::Status=${apt_status_file} \ + -o Dir::Cache=${IMAGE_ROOTFS}/var/cache/apt/ -y source ${src} + call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \ + --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \ + -C main -S ${sect} -P ${prio} \ + includedsc ${DEBDISTRONAME} \ + ${APT_SRCDIR}/${src}/*.dsc + else + bbnote "already fetched source package '${src}' (binary: ${pkg})" + fi + fi + done + rm -f ${apt_status_file} +} + +addtask populate_repo + do_rootfs() { E="${@ bb.utils.export_proxies(d)}" @@ -94,6 +167,11 @@ do_rootfs() { # because of proxy settings sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" + # Copy packages before they vanish when the root file-systems get configured + # (apt purge is being run) + install -m 755 -d ${DEPLOY_DIR_DEB}/${PN} + sudo cp -l ${IMAGE_ROOTFS}/var/cache/apt/archives/*.deb ${DEPLOY_DIR_DEB}/${PN}/ + # Configure root filesystem sudo chroot ${IMAGE_ROOTFS} /${DISTRO_CONFIG_SCRIPT} ${MACHINE_SERIAL} ${BAUDRATE_TTY} \ ${ROOTFS_DEV} -- 2.11.0