From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6771003123014041600 X-Received: by 2002:adf:a285:: with SMTP id s5mr30753282wra.118.1580226212513; Tue, 28 Jan 2020 07:43:32 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a7b:c4d0:: with SMTP id g16ls2706858wmk.2.canary-gmail; Tue, 28 Jan 2020 07:43:31 -0800 (PST) X-Google-Smtp-Source: APXvYqwWe2U79+bl2TJXHDh1dwVE3Clh60KzJfjW5QIwIyGCvznJkob/ZnOWDdXml5FhtSDODKBb X-Received: by 2002:a1c:5ac3:: with SMTP id o186mr6027747wmb.144.1580226211851; Tue, 28 Jan 2020 07:43:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580226211; cv=none; d=google.com; s=arc-20160816; b=jnvwdSPZ4OLgh1Zevj2W6sxv8AHWMS7Rajoj64bNogCZWEi5TNKgiIW439xODkzZfX UmBSZzQSm9xe9/p6pA2bsX/fplVlIFf8/jSO/3w1prv7X8cYAYvNP6xGzCEf18Soybm4 HDoRiuC5syxuG3oszKE3AUjAaCcErxuf536S6oQmAZkTMZLYcqhWAzz+CvQthrHR8+by XUf/g5caYxzt8nn34w4eccXRA1YQx8dUSKe5IpwpiGLfkSBvVy1194Q63zTCO0HNOzCS IyVYBWaTbgTTbjF/QYukolHp4d8mcbFeVOrXf4u1e9vNlN/Yyy1xN5bGXBVEx9MEsb+A UzKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KyMgc1mKSOwb20wx6826RmZZdsiwGLCb28k58VF3WxQ=; b=KXrx6DDmlIjjK4fPTt6/jfiIWyVDvT+xTbgbp5V5W1UaTVXeEkq0phex7mxLB2aZvm Z3QAq0VQRDMTWdipV/LWPo5TVrDn9pQ4BnYW88QIpRovYyYAqIdnspAPfhuWRWA1Rh4u ugtY0djENhF9N6IvqrdBkPgHZK76O4AqlKMnUM46V//+MTeuQ3p+vARLpu/7DShS7qS1 Al9bacHtszT0ekPuhuBRpEkJtW99u5tGwix1WWljKNPXdTHgJjMDGdodPot0IgOaZ6zq y+E7EKwArU85hc9DF/mDpDTROqL3VNdIuaB4sfyWk1ve0cyk14jZ+jlStQDcrzw5JW79 CUFA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=f42o1rUA; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::444 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com. [2a00:1450:4864:20::444]) by gmr-mx.google.com with ESMTPS id p23si128994wma.1.2020.01.28.07.43.31 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2020 07:43:31 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::444 as permitted sender) client-ip=2a00:1450:4864:20::444; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=f42o1rUA; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::444 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wr1-x444.google.com with SMTP id a6so1951611wrx.12 for ; Tue, 28 Jan 2020 07:43:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KyMgc1mKSOwb20wx6826RmZZdsiwGLCb28k58VF3WxQ=; b=f42o1rUAQQquJnjx0kvNmOlSEkDiLO7QHCVCq7tpLSrUTBPDODhNfJaQRCjwmhU/SP spCILd53rgNmg1gCEpdjkGzY2moKc/koUsAdcfw9QQgS8h3ttEjnAd/U6Kz+txAi+8wG YLROoxNCoZH5uUqlKSqhkeI0p2Sr+welhA9eFK8SCaKSRmM3cjkiD7/wPNl2BGuiOoW0 cEeC+I7udAR1r1nY/eIfp/7bq4Tq3JkpIQHddlidwSS1oThn2A3xn/2nbszD4mBP2yx4 kBVIAoSphVIElYd+mTyLIQDtgJJHHYHMEsUmo5LgsXt7NNkBvw53SKbq9STtJCHAKfp+ iobQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KyMgc1mKSOwb20wx6826RmZZdsiwGLCb28k58VF3WxQ=; b=BekDDTzdmzilZCP2R6ydfi4zuYPL2VNx/NkSJY8zfotmYy3BxTEpLEgngMm4UGCD/Y GzetcBw+Ng7L4SA3jRmAfJjDnWEbJyWsZTxN/gigxMmOeKQEo4+Icvlw/Zcm1bUXuQ0D e1HVBSHhr2+NlTYmijqYrgWTjsja2jJCPhxg4mx7SBzOOZ5fJjoq6X6Dh0fAcp00tAAU UBjUJ14iIpuYomRyu+AabZDsZQhZY/TUfabJx7BOFhdFDnXbhmh3aF4IrCN3SIPDFxsX 4TJNrfJiFm+DH5bDG9b2vPmphZVKYONvxNjP43Z3CMOdjik4G5Pd+Of8cpd9y1S1WQuj znag== X-Gm-Message-State: APjAAAXbn0nrVqnqgiEmWKqxROiMeH3wZ9lLcYRYV+GupZA9ujp4xwYI 7beAIe0k9DWWzylfzdgsosbv4+Dcik8= X-Received: by 2002:a5d:438c:: with SMTP id i12mr28873335wrq.196.1580226211168; Tue, 28 Jan 2020 07:43:31 -0800 (PST) Return-Path: Received: from oxygen.mgc.mentorg.com (nat-sch.mentorg.com. [139.181.36.34]) by smtp.gmail.com with ESMTPSA id u8sm3601200wmm.15.2020.01.28.07.43.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jan 2020 07:43:30 -0800 (PST) From: Vijai Kumar K To: isar-users@googlegroups.com Cc: Henning Schild Subject: [PATCH v3 23/26] base-apt: pull base-apt population to the front of the build chain Date: Tue, 28 Jan 2020 21:12:10 +0530 Message-Id: <20200128154213.22353-24-vijaikumar.kanagarajan@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200128154213.22353-1-vijaikumar.kanagarajan@gmail.com> References: <20200128154312.337e61f6@md1za8fc.ad001.siemens.net> <20200128154213.22353-1-vijaikumar.kanagarajan@gmail.com> X-TUID: 7FCIwTGsj0DJ From: Henning Schild Users had to call a special task before switching to using base-apt. With this commit you flip one switch and the repo will be created as one of the first steps. The base-apt repo gets moved from DL_DIR to DEPLOY_DIR. It is "processed" files and not just a download-cache. Signed-off-by: Henning Schild --- doc/user_manual.md | 8 +-- meta/classes/image-cache-extension.bbclass | 64 ------------------ meta/classes/image.bbclass | 1 - meta/classes/rootfs.bbclass | 2 +- meta/conf/bitbake.conf | 4 +- .../isar-bootstrap/isar-bootstrap.inc | 1 + meta/recipes-devtools/base-apt/base-apt.bb | 67 ++++++++++++++++--- scripts/ci_build.sh | 4 +- 8 files changed, 69 insertions(+), 82 deletions(-) delete mode 100644 meta/classes/image-cache-extension.bbclass diff --git a/doc/user_manual.md b/doc/user_manual.md index 4d699bd..aaf9638 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -822,10 +822,10 @@ path to the public key in `conf/local.conf`, e.g.: BASE_REPO_KEY = "file://"' ``` - - Trigger creation of local apt caching Debian packages during image generation. + - Trigger the download and caching of all required files by doing a warm-up build. ``` -bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base +bitbake mc:qemuarm-buster:isar-image-base ``` - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`: @@ -835,14 +835,14 @@ bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base #ISAR_USE_CACHED_BASE_REPO ?= "1" #BB_NO_NETWORK ?= "1" ``` - - Remove build artifacts to use only local base-apt: + - Remove build artifacts to use only local base-apt, in fact toggling ISAR_USE_CACHED_BASE_REPO should trigger a full rebuild as well. This is just the way to be extra sure that only the download cache is used. ``` sudo rm -rf tmp ``` - - Trigger again generation of image (now using local caching repo): + - Trigger the generation of your image again (now a local repo will be created out of the download cache from the last run): ``` bitbake mc:qemuarm-buster:isar-image-base diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass deleted file mode 100644 index 4123326..0000000 --- a/meta/classes/image-cache-extension.bbclass +++ /dev/null @@ -1,64 +0,0 @@ -# This software is a part of ISAR. -# Copyright (C) Siemens AG, 2019 -# -# SPDX-License-Identifier: MIT -# -# This class extends the image.bbclass to supply the creation of cache repositories - -inherit repository - -populate_base_apt() { - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do - # NOTE: due to packages stored by reprepro are not modified, we can - # use search by filename to check if package is already in repo. In - # addition, md5sums are compared to ensure that the package is the - # same and should not be overwritten. This method is easier and more - # robust than querying reprepro by name. - - # Check if this package is taken from Isar-apt, if so - ingore it. - repo_contains_package "${REPO_ISAR_DIR}/${DISTRO}" "${package}" && \ - continue - - # Check if this package is already in base-apt - ret=0 - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" || - ret=$? - [ "${ret}" = "0" ] && continue - if [ "${ret}" = "1" ]; then - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ - "${BASE_DISTRO_CODENAME}" \ - "${base_apt_p}" - fi - - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ - "${BASE_DISTRO_CODENAME}" \ - "${package}" - done - - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ - "${BASE_DISTRO_CODENAME}" \ - "${package}" - done -} - -do_cache_base_repo[depends] = "base-apt:do_cache_config" -do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock" -do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}" -do_cache_base_repo() { - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then - # We would need two separate repository paths for that. - # Otherwise packages (especially the 'all' arch ones) from one - # distribution can influence the package versions of the other - # distribution. - bbfatal "Different host and target distributions are currently not supported." \ - "Try it without cross-build." - fi - - populate_base_apt -} -addtask cache_base_repo after do_rootfs do_install_imager_deps diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index b687fcf..cfd617a 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -66,7 +66,6 @@ ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" inherit rootfs inherit image-sdk-extension -inherit image-cache-extension inherit image-tools-extension inherit image-postproc-extension inherit image-locales-extension diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 6d4d42c..a10e7cc 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -151,7 +151,7 @@ rootfs_install_pkgs_install() { do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}" -do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config" +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config base-apt:do_cache" do_rootfs_install[deptask] = "do_deploy_deb" python do_rootfs_install() { configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split() diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index d86c5b9..e46142d 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -80,8 +80,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db" THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg" # Base apt repository paths -REPO_BASE_DIR = "${DL_DIR}/base-apt/${DISTRO}/apt" -REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/${DISTRO}/db" +REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt" +REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/db" BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \ SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index 5b14cd3..57cdea9 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -219,6 +219,7 @@ def get_host_release(): do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS" do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" +do_bootstrap[depends] = "base-apt:do_cache" isar_bootstrap() { deb_dl_dir_import "${ROOTFSDIR}" diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb index c9a7dad..2dba779 100644 --- a/meta/recipes-devtools/base-apt/base-apt.bb +++ b/meta/recipes-devtools/base-apt/base-apt.bb @@ -9,26 +9,77 @@ inherit repository BASE_REPO_KEY ?= "" KEYFILES ?= "" -do_cache_config[stamp-extra-info] = "${DISTRO}" -do_cache_config[lockfiles] = "${REPO_BASE_DIR}/isar.lock" +populate_base_apt() { + find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do + # NOTE: due to packages stored by reprepro are not modified, we can + # use search by filename to check if package is already in repo. In + # addition, md5sums are compared to ensure that the package is the + # same and should not be overwritten. This method is easier and more + # robust than querying reprepro by name. -# Generate reprepro config for current distro if it doesn't exist. Once it's -# generated, this task should do nothing. -repo_config() { + # Check if this package is taken from Isar-apt, if so - ingore it. + repo_contains_package "${REPO_ISAR_DIR}/${DISTRO}" "${package}" && \ + continue + + # Check if this package is already in base-apt + ret=0 + repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" "${package}" || + ret=$? + [ "${ret}" = "0" ] && continue + if [ "${ret}" = "1" ]; then + repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + "${BASE_DISTRO_CODENAME}" \ + "${base_apt_p}" + fi + + repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + "${BASE_DISTRO_CODENAME}" \ + "${package}" + done + + find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do + repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ + "${BASE_DISTRO_CODENAME}" \ + "${package}" + done +} + +do_cache[stamp-extra-info] = "${DISTRO}" +do_cache[lockfiles] = "${REPO_BASE_DIR}/isar.lock" + +repo() { repo_create "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ "${BASE_DISTRO_CODENAME}" \ "${KEYFILES}" + + if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && + [ '${DISTRO}' != '${HOST_DISTRO}' ]; then + # We would need two separate repository paths for that. + # Otherwise packages (especially the 'all' arch ones) from one + # distribution can influence the package versions of the other + # distribution. + bbfatal "Different host and target distributions are currently not supported." \ + "Try it without cross-build." + fi + + populate_base_apt } -python do_cache_config() { +python do_cache() { + if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): + return 0 + for key in d.getVar('BASE_REPO_KEY').split(): d.appendVar("SRC_URI", " %s" % key) fetcher = bb.fetch2.Fetch([key], d) filename = fetcher.localpath(key) d.appendVar("KEYFILES", " %s" % filename) - bb.build.exec_func('repo_config', d) + bb.build.exec_func('repo', d) } -addtask cache_config after do_unpack before do_build +addtask cache after do_unpack before do_build diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index 7da3ee9..733ba86 100755 --- a/scripts/ci_build.sh +++ b/scripts/ci_build.sh @@ -145,7 +145,7 @@ if [ -n "$REPRO_BUILD" ]; then # Enable use of signed cached base repository echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" >> conf/local.conf - bitbake $BB_ARGS -c cache_base_repo $REPRO_TARGETS_SET_SIGNED + bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED while [ -e bitbake.sock ]; do sleep 1; done sudo rm -rf tmp sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf @@ -159,7 +159,7 @@ if [ -n "$REPRO_BUILD" ]; then sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf # Enable use of unsigned cached base repository - bitbake $BB_ARGS -c cache_base_repo $REPRO_TARGETS_SET + bitbake $BB_ARGS $REPRO_TARGETS_SET while [ -e bitbake.sock ]; do sleep 1; done sudo rm -rf tmp sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf -- 2.17.1