From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6771003121820762112 X-Received: by 2002:a05:6402:22d4:: with SMTP id dm20mr3116572edb.353.1579600460685; Tue, 21 Jan 2020 01:54:20 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:c0d1:: with SMTP id j17ls8783386edp.4.gmail; Tue, 21 Jan 2020 01:54:20 -0800 (PST) X-Google-Smtp-Source: APXvYqydf793Gjy8TCAePCKtgRyKcnDsgZxAMPPUc7RmlrOerr9XF/WEBTyZZAnqMcGvhE2gi4+d X-Received: by 2002:aa7:cd05:: with SMTP id b5mr3010363edw.315.1579600460214; Tue, 21 Jan 2020 01:54:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1579600460; cv=none; d=google.com; s=arc-20160816; b=X6OMgvJ3U/lv0ejkN3c0HSyJ/6B9ptTeco/67egmMGYdkfORGs+7h1R7cpuzDpHG4u VAOHzpDBeGAdStlPqpLtSyO84zZugtLXu3lQxLCz4Semjhapo7TpMtNym0B4qWL+TeV7 Fb2r1Ia7cgRyuWslpIYotEeMx0mSnHnOpqlhX3YXPrHwaUrNU9RuhcKg4uSBIx7H5Jkn 40upRdzRsDa4VbfhVWwfrgYnavz/0xvECvzwf72LNtBWIEskUu9E8dY6ePHSIh7j6xyV a2sZnVWManCfXIJVG7yHywsqBKGdTAbWlne/RCtLQ9g+iu4B3iKuGV8jNi8G3hSQ3O21 EPjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=Bke2f9nZBBDPw2FYG7J7Fi3y7dZejaWdS+oMXc8QVxE=; b=CiLzvggUlixMgdxpnA/Vt1x5P0KsdoDmV66/9B2FNA8K33QAewyQO9jVQ74WHNXA4m IoBxpT5eisuVTmQh6he07BMJsmtLwN6FJpNKjKe/TD/DwGM1sikN5txNeQoeE9tbkjCQ 0L/7bO/EgB1Ap4t2pMSzBxBVkHZXfSWn279hX8LmZeGikwkJ7vHLaDF9dFCZDOsOdu6x 9Hq67+wvr75YqAfgsBhzayM+wSjllVCLKNf20ZUAWskf5Qs7k87yj0oYykejXRB4yqk9 WDd+iDk36gkgVbSwjVsVI6QXkhM/bWaPYApwWideed4XM+xg2iYw89hC6yybDBhvDXG4 DsoQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id d29si1620589edj.0.2020.01.21.01.54.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Jan 2020 01:54:20 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 00L9sJYP013799 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jan 2020 10:54:19 +0100 Received: from md1za8fc.ad001.siemens.net ([167.87.47.119]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 00L9sIH0030154; Tue, 21 Jan 2020 10:54:19 +0100 Date: Tue, 21 Jan 2020 10:54:17 +0100 From: Henning Schild To: vijai kumar Cc: isar-users Subject: Re: [PATCHv2 23/25] base-apt: pull base-apt population to the front of the build chain Message-ID: <20200121105417.37bb6ad4@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20191216115011.17664-1-henning.schild@siemens.com> <20191216115011.17664-24-henning.schild@siemens.com> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: fzkMEdJ9b7Iy Hi Vijai, On Tue, 21 Jan 2020 00:55:43 -0800 vijai kumar wrote: > Hi Henning, > > Can't we have the base-apt repo generation towards the end of the > build chain? Could you go into detail why you think the point in time matters? It was at the end because the old implementation relied on the rootfs hopefully keeping all .debs in its cache. While that worked it was never really robust. And it was optional, now it just "comes for free along the way". Whether you use it or not is still optional. > Right now if you need to generate a base-apt then you need to build > twice. Once without > ISAR_USE_CACHED_BASE_REPO and once with ISAR_USE_CACHED_BASE_REPO set. Still the case. You will need a second run where you tell Isar to use that repo. Only difference its creation during the first run was implicit instead of explicit. > What if we could just generate the repo towards the end nevertheless > and let user decide > whether to use the repo in the subsequent build by setting > ISAR_USE_CACHED_BASE_REPO? Again, i do not see where the "point in time of the first build" makes a difference. And the user still has to decide using that interface you mention. Henning > Thanks, > Vijai Kumar K > > > On Monday, December 16, 2019 at 5:20:16 PM UTC+5:30, Henning Schild > wrote: > > > > From: Henning Schild > > > > > Users had to call a special task before switching to using > > base-apt. With this commit you flip one switch and the repo will be > > created as one of the first steps. > > > > The base-apt repo gets moved from DL_DIR to DEPLOY_DIR. It is > > "processed" files and not just a download-cache. > > > > Signed-off-by: Henning Schild > > > --- > > doc/user_manual.md | 8 +-- > > meta/classes/image-cache-extension.bbclass | 64 > > ------------------ meta/classes/image.bbclass | > > 1 - meta/classes/rootfs.bbclass | 2 +- > > meta/conf/bitbake.conf | 4 +- > > .../isar-bootstrap/isar-bootstrap.inc | 1 + > > meta/recipes-devtools/base-apt/base-apt.bb | 67 > > ++++++++++++++++--- scripts/ci_build.sh | > > 4 +- 8 files changed, 69 insertions(+), 82 deletions(-) > > delete mode 100644 meta/classes/image-cache-extension.bbclass > > > > diff --git a/doc/user_manual.md b/doc/user_manual.md > > index 4d699bd..aaf9638 100644 > > --- a/doc/user_manual.md > > +++ b/doc/user_manual.md > > @@ -822,10 +822,10 @@ path to the public key in `conf/local.conf`, > > e.g.: BASE_REPO_KEY = > > "file://"' ``` > > > > - - Trigger creation of local apt caching Debian packages during > > image generation. > > + - Trigger the download and caching of all required files by doing > > a warm-up build. > > > > ``` > > -bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base > > +bitbake mc:qemuarm-buster:isar-image-base > > ``` > > > > - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`: > > @@ -835,14 +835,14 @@ bitbake -c cache_base_repo > > mc:qemuarm-buster:isar-image-base > > #ISAR_USE_CACHED_BASE_REPO ?= "1" > > #BB_NO_NETWORK ?= "1" > > ``` > > - - Remove build artifacts to use only local base-apt: > > + - Remove build artifacts to use only local base-apt, in fact > > toggling ISAR_USE_CACHED_BASE_REPO should trigger a full rebuild as > > well. This is just the way to be extra sure that only the download > > cache is used. > > ``` > > sudo rm -rf tmp > > > > ``` > > > > - - Trigger again generation of image (now using local caching > > repo): > > + - Trigger the generation of your image again (now a local repo > > will be created out of the download cache from the last run): > > > > ``` > > bitbake mc:qemuarm-buster:isar-image-base > > diff --git a/meta/classes/image-cache-extension.bbclass > > b/meta/classes/image-cache-extension.bbclass > > deleted file mode 100644 > > index 4123326..0000000 > > --- a/meta/classes/image-cache-extension.bbclass > > +++ /dev/null > > @@ -1,64 +0,0 @@ > > -# This software is a part of ISAR. > > -# Copyright (C) Siemens AG, 2019 > > -# > > -# SPDX-License-Identifier: MIT > > -# > > -# This class extends the image.bbclass to supply the creation of > > cache repositories > > - > > -inherit repository > > - > > -populate_base_apt() { > > - find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read > > package; do > > - # NOTE: due to packages stored by reprepro are not > > modified, we can > > - # use search by filename to check if package is already in > > repo. In > > - # addition, md5sums are compared to ensure that the > > package is the > > - # same and should not be overwritten. This method is > > easier and more > > - # robust than querying reprepro by name. > > - > > - # Check if this package is taken from Isar-apt, if so - > > ingore it. > > - repo_contains_package "${REPO_ISAR_DIR}/${DISTRO}" > > "${package}" && \ > > - continue > > - > > - # Check if this package is already in base-apt > > - ret=0 > > - repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" > > "${package}" || > > - ret=$? > > - [ "${ret}" = "0" ] && continue > > - if [ "${ret}" = "1" ]; then > > - repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > - "${BASE_DISTRO_CODENAME}" \ > > - "${base_apt_p}" > > - fi > > - > > - repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > - "${BASE_DISTRO_CODENAME}" \ > > - "${package}" > > - done > > - > > - find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read > > package; do > > - repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > - "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > - "${BASE_DISTRO_CODENAME}" \ > > - "${package}" > > - done > > -} > > - > > -do_cache_base_repo[depends] = "base-apt:do_cache_config" > > -do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock" > > -do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}" > > -do_cache_base_repo() { > > - if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > > - [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > - # We would need two separate repository paths for that. > > - # Otherwise packages (especially the 'all' arch ones) from > > one > > - # distribution can influence the package versions of the > > other > > - # distribution. > > - bbfatal "Different host and target distributions are > > currently not supported." \ > > - "Try it without cross-build." > > - fi > > - > > - populate_base_apt > > -} > > -addtask cache_base_repo after do_rootfs do_install_imager_deps > > diff --git a/meta/classes/image.bbclass > > b/meta/classes/image.bbclass index b687fcf..cfd617a 100644 > > --- a/meta/classes/image.bbclass > > +++ b/meta/classes/image.bbclass > > @@ -66,7 +66,6 @@ ROOTFS_MANIFEST_DEPLOY_DIR ?= > > "${DEPLOY_DIR_IMAGE}" > > inherit rootfs > > inherit image-sdk-extension > > -inherit image-cache-extension > > inherit image-tools-extension > > inherit image-postproc-extension > > inherit image-locales-extension > > diff --git a/meta/classes/rootfs.bbclass > > b/meta/classes/rootfs.bbclass index 6d4d42c..a10e7cc 100644 > > --- a/meta/classes/rootfs.bbclass > > +++ b/meta/classes/rootfs.bbclass > > @@ -151,7 +151,7 @@ rootfs_install_pkgs_install() { > > > > do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}" > > do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} > > ${ROOTFS_INSTALL_COMMAND}" > > -do_rootfs_install[depends] = "isar-bootstrap-${@'target' if > > d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else > > 'host'}:do_build isar-apt:do_cache_config" > > +do_rootfs_install[depends] = "isar-bootstrap-${@'target' if > > d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else > > 'host'}:do_build isar-apt:do_cache_config base-apt:do_cache" > > do_rootfs_install[deptask] = "do_deploy_deb" > > python do_rootfs_install() { > > configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) > > or "").split() > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > > index d86c5b9..e46142d 100644 > > --- a/meta/conf/bitbake.conf > > +++ b/meta/conf/bitbake.conf > > @@ -80,8 +80,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db" > > THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg" > > > > # Base apt repository paths > > -REPO_BASE_DIR = "${DL_DIR}/base-apt/${DISTRO}/apt" > > -REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/${DISTRO}/db" > > +REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt" > > +REPO_BASE_DB_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/db" > > > > BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH > > BBSERVER DL_DIR \ > > SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME > > SHELL TERM \ > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index 5b14cd3..57cdea9 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -219,6 +219,7 @@ def get_host_release(): > > > > do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS" > > do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" > > +do_bootstrap[depends] = "base-apt:do_cache" > > > > isar_bootstrap() { > > deb_dl_dir_import "${ROOTFSDIR}" > > diff --git a/meta/recipes-devtools/base-apt/base-apt.bb > > b/meta/recipes-devtools/base-apt/base-apt.bb > > index c9a7dad..2dba779 100644 > > --- a/meta/recipes-devtools/base-apt/base-apt.bb > > +++ b/meta/recipes-devtools/base-apt/base-apt.bb > > @@ -9,26 +9,77 @@ inherit repository > > BASE_REPO_KEY ?= "" > > KEYFILES ?= "" > > > > -do_cache_config[stamp-extra-info] = "${DISTRO}" > > -do_cache_config[lockfiles] = "${REPO_BASE_DIR}/isar.lock" > > +populate_base_apt() { > > + find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read > > package; do > > + # NOTE: due to packages stored by reprepro are not > > modified, we can > > + # use search by filename to check if package is already in > > repo. In > > + # addition, md5sums are compared to ensure that the > > package is the > > + # same and should not be overwritten. This method is > > easier and more > > + # robust than querying reprepro by name. > > > > -# Generate reprepro config for current distro if it doesn't exist. > > Once it's > > -# generated, this task should do nothing. > > -repo_config() { > > + # Check if this package is taken from Isar-apt, if so - > > ingore it. > > + repo_contains_package "${REPO_ISAR_DIR}/${DISTRO}" > > "${package}" && \ > > + continue > > + > > + # Check if this package is already in base-apt > > + ret=0 > > + repo_contains_package "${REPO_BASE_DIR}/${BASE_DISTRO}" > > "${package}" || > > + ret=$? > > + [ "${ret}" = "0" ] && continue > > + if [ "${ret}" = "1" ]; then > > + repo_del_package "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${base_apt_p}" > > + fi > > + > > + repo_add_packages "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${package}" > > + done > > + > > + find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read > > package; do > > + repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > + "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > + "${BASE_DISTRO_CODENAME}" \ > > + "${package}" > > + done > > +} > > + > > +do_cache[stamp-extra-info] = "${DISTRO}" > > +do_cache[lockfiles] = "${REPO_BASE_DIR}/isar.lock" > > + > > +repo() { > > repo_create "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \ > > "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \ > > "${BASE_DISTRO_CODENAME}" \ > > "${KEYFILES}" > > + > > + if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] && > > + [ '${DISTRO}' != '${HOST_DISTRO}' ]; then > > + # We would need two separate repository paths for that. > > + # Otherwise packages (especially the 'all' arch ones) from > > one > > + # distribution can influence the package versions of the > > other > > + # distribution. > > + bbfatal "Different host and target distributions are > > currently not supported." \ > > + "Try it without cross-build." > > + fi > > + > > + populate_base_apt > > } > > > > -python do_cache_config() { > > +python do_cache() { > > + if not > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')): > > + return 0 > > + > > for key in d.getVar('BASE_REPO_KEY').split(): > > d.appendVar("SRC_URI", " %s" % key) > > fetcher = bb.fetch2.Fetch([key], d) > > filename = fetcher.localpath(key) > > d.appendVar("KEYFILES", " %s" % filename) > > > > - bb.build.exec_func('repo_config', d) > > + bb.build.exec_func('repo', d) > > } > > > > -addtask cache_config after do_unpack before do_build > > +addtask cache after do_unpack before do_build > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh > > index 7da3ee9..733ba86 100755 > > --- a/scripts/ci_build.sh > > +++ b/scripts/ci_build.sh > > @@ -145,7 +145,7 @@ if [ -n "$REPRO_BUILD" ]; then > > > > # Enable use of signed cached base repository > > echo BASE_REPO_KEY=\"file://$ISAR_TESTSUITE_GPG_PUB_KEY_FILE\" > > >> conf/local.conf > > - bitbake $BB_ARGS -c cache_base_repo $REPRO_TARGETS_SET_SIGNED > > + bitbake $BB_ARGS $REPRO_TARGETS_SET_SIGNED > > while [ -e bitbake.sock ]; do sleep 1; done > > sudo rm -rf tmp > > sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= > > "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf > > @@ -159,7 +159,7 @@ if [ -n "$REPRO_BUILD" ]; then > > sed -i -e 's/^BASE_REPO_KEY/#BASE_REPO_KEY/g' conf/local.conf > > > > # Enable use of unsigned cached base repository > > - bitbake $BB_ARGS -c cache_base_repo $REPRO_TARGETS_SET > > + bitbake $BB_ARGS $REPRO_TARGETS_SET > > while [ -e bitbake.sock ]; do sleep 1; done > > sudo rm -rf tmp > > sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= > > "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf > > -- > > 2.24.1 > > > > >