From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6458972999552860160 X-Received: by 10.46.83.27 with SMTP id h27mr352399ljb.35.1503907246958; Mon, 28 Aug 2017 01:00:46 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.206.200 with SMTP id e191ls1105415wmg.2.canary-gmail; Mon, 28 Aug 2017 01:00:46 -0700 (PDT) X-Received: by 10.223.135.110 with SMTP id 43mr275446wrz.10.1503907246621; Mon, 28 Aug 2017 01:00:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1503907246; cv=none; d=google.com; s=arc-20160816; b=X1pDSblulO3to09n2gDjXUXAcGv74u5QfUk6UMxGJz/PjN1/pbo+qSdvoAQSImAhaQ mEhcxG2Z2ZlJNSHTWGHdL2QEIQdPN1SKB4gNrzV2Z0uGN2FXzJ6nnicFb/00EMyrftTD jbvERcYfizyHKw/0upODsAQYSmID/dTmLh9hao3VTjc8oQrQXVL3UYZPl0ywNDLCNhoh ft4vO22A19GLO2gmLjdlJdSnq5ZhBqwwwH/n+j6rpncVrvjM7WUZRsoqmBgPFspoMduU DB83S2EvLpR9biWYXyeoPdzh+4rA4N3lRLdIyoYn1agDUQY4wQX5m9+wsPGhjleOyaPk sRMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=pWi9d4XDFgDXNBNoFfI1iT8jbNfXE4MfOYYl+wI+bNQ=; b=ZrEy6pK7h7OTzEA1b4bKe+Jka0W6m3oSQKmnt+KNJ7V7zT8qajcBRQDjL1S5QcdNsy AlcF7v81lJwxi6c73lYgHF9+/uqTI0U3ojAcS3R1WDgkq9STjRZS8TjmVjyNlZzCpPnc xJgRztvs+jTebliGpH1xi63ZPl0sNqPFNZt99EmQJOCbRi564JafQtyolMeFln1KOnwu PXq500tlewXym72Cr0qwkJkTyCcxRj2xSyU+tVBwR22OcFavSjAUTYL1UmLIOPyJPTK5 YxMfJmLruf7ZLVwrenTAlLQErkV3uzKbxpgLdEliIuPZ601cXUJV7QPoKUcR52fC1IOp ACbw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id v202si350346wmv.3.2017.08.28.01.00.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 28 Aug 2017 01:00:46 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v7S80kV5023170 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 28 Aug 2017 10:00:46 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id v7S80jwl000885; Mon, 28 Aug 2017 10:00:46 +0200 Subject: Re: [PATCH 3/6] classes/dpkg: Split install for cache To: Alexander Smirnov , isar-users@googlegroups.com References: <20170827151339.12806-1-asmirnov@ilbers.de> <20170827151339.12806-4-asmirnov@ilbers.de> From: Claudius Heine Message-ID: <8e793c26-12cc-636a-6ea6-9cfadccf5f09@siemens.com> Date: Mon, 28 Aug 2017 10:00:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20170827151339.12806-4-asmirnov@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 0BRu4T4GeM57 Hi, On 08/27/2017 05:13 PM, Alexander Smirnov wrote: > Split install function into two parts: > - install_to_cache: if caching is enabled > - install_to_deploy: if caching is disabled > > This patch brings flexibility to the implementation and makes it possible > to move all the caching implementation code to dedicated class. The magic > behavior depending on the value of DEBCACHE_ENABLED is now transparent in > the bitbake pipeline (can be inspected via "bitbake -g"). > > Signed-off-by: Alexander Smirnov > --- > meta/classes/dpkg.bbclass | 66 ++++++++++++++++++++++++++-------------------- > meta/classes/image.bbclass | 2 +- > 2 files changed, 38 insertions(+), 30 deletions(-) > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index b1e201d..118ba2f 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -56,46 +56,53 @@ do_build() { > } > > # Install package to dedicated deploy directory > -do_binary_deb_install() { > +do_install_to_deploy() { > + # Deb caching is disabled, simply copy all binary packages to the deploy > + # directory > + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > +} > + > +addtask install_to_deploy after do_build > +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" > +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_deploy[noexec] = "1" > + > +# Install package to dedicated apt repo > +do_install_to_cache() { > readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > > - if [ "${DEBCACHE_ENABLED}" != "0" ]; then > - # If `bitbake` is running for the first time, the cache doesn't exist > - # yet and needs to be configured using a `distributions` file. > - # A template stored in the layer directory is pre-processed to > - # generate the configuration file, which is then placed in the > - # appropriate directory. > - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > - mkdir -p "${DIR_CACHE}/conf" > - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > - "${DEBFILESDIR}/distributions.in" \ > - > "${DIR_CACHE}/conf/distributions" > - fi > - > - # Add binary and source packages to the deb cache > - # If the cache doesn't exist yet, it will be created using the > - # `distributions` file generated above. > - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do > - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ > - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > - done > - else > - # Deb caching is disabled, simply copy all binary packages to the > - # deploy directory > - mkdir -p "${DEPLOY_DIR_DEB}" > - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > + # If `bitbake` is running for the first time, the cache doesn't exist > + # yet and needs to be configured using a `distributions` file. > + # A template stored in the layer directory is pre-processed to > + # generate the configuration file, which is then placed in the > + # appropriate directory. > + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > + mkdir -p "${DIR_CACHE}/conf" > + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > + "${DEBFILESDIR}/distributions.in" \ > + > "${DIR_CACHE}/conf/distributions" > fi > + > + # Add binary and source packages to the deb cache > + # If the cache doesn't exist yet, it will be created using the > + # `distributions` file generated above. > + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ > + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + done > } > > -addtask binary_deb_install after do_build > -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +addtask install_to_cache after do_build What was the reason why we put stuff after the 'build' task? In OE the 'build' task is the default task triggering everything necessary for a recipe. In Isar that changed. Why? > +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_cache[noexec] = "1" > > # Deb caching lambda run during the parsing phase that checks whether the > # current package has to be rebuilt, or taken from the cache > python __anonymous () { > if d.getVar("DEBCACHE_ENABLED", True) == "0": > - # Deb caching is disabled, do nothing > + # Deb caching is disabled > + d.delVarFlag('do_install_to_deploy', 'noexec') > return True > > PN = d.getVar("PN", True) > @@ -108,6 +115,7 @@ python __anonymous () { > path_cache = os.path.join(DEBCACHEDIR, DISTRO) > path_databases = os.path.join(DEBDBDIR, DISTRO) > path_distributions = os.path.join(path_cache, "conf", "distributions") > + d.delVarFlag('do_install_to_cache', 'noexec') > > # The distributions file is needed by `reprepro` to know what types > # of packages are supported, what the distribution name is, etc. > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c2ff453..6b1b5eb 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -46,4 +46,4 @@ do_populate() { > } > > addtask populate before do_build > -do_populate[deptask] = "do_install" > +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" I would rather still have a, maybe empty, 'do_install' step that just depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. I don't see why ever small change in one class should change the complete pipeline for every recipe. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de