From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6816610675337461760 X-Received: by 2002:a05:6402:31b1:: with SMTP id dj17mr18068468edb.146.1587539202716; Wed, 22 Apr 2020 00:06:42 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:3da:: with SMTP id c26ls526119eja.7.gmail; Wed, 22 Apr 2020 00:06:42 -0700 (PDT) X-Google-Smtp-Source: APiQypK3SMlkA6bjxInFFwekG/AxVzgs7o7togOz3ceAzwkePtplj9dscU5KaxUgAFenNdza44hw X-Received: by 2002:a17:907:2142:: with SMTP id rk2mr25298858ejb.356.1587539202231; Wed, 22 Apr 2020 00:06:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587539202; cv=none; d=google.com; s=arc-20160816; b=VCzl8golhfHrx7/8w3+FnL2wbMZMn7lGr7laafQXNMQo79GTQt/wwYIHGEESbph3xl oNlFi9ZIOmZ2teKAEc1KI2eWtXOx0waoeK2NAGV7szbNW/5fexxcO6vPXTmuFop4BtiB ZmWpjf+YnqHmvbZ5csto/Isqa+8ieD2BFf3VKtt856a6eKStrVuB32lzJEOPAEbuP0Uc 1w8ynZmruQIWLOPPviLgGktpTEzwUqJ4bOGECGYBPC6sYCyIPcRmNmJEYDMVLEalCKfg aAoBljHS6urcYzGWdoCBYmSEwRA3ya97k2pulsj1ljhA+X8RkurfE8W+CWLJExVxJsh4 wbUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=mEiK6TjZsK+d9iUdbYySDEI3BR1lq03flJ5ZOnWSQeg=; b=0bYLDmsbXEnn4RX2ZnQ0shd36Zk+eOOIZI3ZULawbTKIZCqBooGJKVtkxKDJHhHYC1 1vN8rZ5yK5YaanL/4kTeqFjjYoDwt49OP4yPun3ytJzNQsnQTq0RCXIPHJD9/tCs9HUk BoQCc0lnWFAgAKXymAFabII+cdO92XpSl0RIT57MdjAJ3gweyhDkS2eKQOiOrqRuJxMU sYRLrpQsFegg3/V595+tDSanraSvc45WFuQGRUOZxqt15L00D/G9KjAPqUz3w4nT29xc iCb8nGjQgmZTFiagR4ox+D1t1nXCs0nKTTn6Q8JS36XByBKsnaGcI5W82fIxSitl1GKP EEJQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id i2si285539edn.2.2020.04.22.00.06.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Apr 2020 00:06:42 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id 03M76fEB021385 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 22 Apr 2020 09:06:41 +0200 Received: from md1za8fc.ad001.siemens.net ([167.87.23.203]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 03M76e34031489; Wed, 22 Apr 2020 09:06:40 +0200 Date: Wed, 22 Apr 2020 09:06:39 +0200 From: Henning Schild To: Vijai Kumar K Cc: , , , Vijai Kumar K Subject: Re: [PATCH v5 03/13] meta: cache deb srcs as part of postprocessing Message-ID: <20200422090639.645bf93b@md1za8fc.ad001.siemens.net> In-Reply-To: <20200417093040.15130-4-Vijaikumar_Kanagarajan@mentor.com> References: <20200417093040.15130-1-Vijaikumar_Kanagarajan@mentor.com> <20200417093040.15130-4-Vijaikumar_Kanagarajan@mentor.com> X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: fS/Wk5gW9TIF On Fri, 17 Apr 2020 15:00:30 +0530 Vijai Kumar K wrote: > Collect the deb sources of the corresponding deb binaries cached > in DEBDIR as part of image postprocess. > > Signed-off-by: Vijai Kumar K > --- > meta/classes/deb-dl-dir.bbclass | 39 > +++++++++++++++++++++++++++++++++ meta/classes/image.bbclass | > 2 +- meta/classes/rootfs.bbclass | 8 +++++++ > 3 files changed, 48 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/deb-dl-dir.bbclass > b/meta/classes/deb-dl-dir.bbclass index 29a3d67..472b9fe 100644 > --- a/meta/classes/deb-dl-dir.bbclass > +++ b/meta/classes/deb-dl-dir.bbclass > @@ -5,6 +5,45 @@ > > inherit repository > > +debsrc_download() { > + export rootfs="$1" > + export rootfs_distro="$2" > + mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}" > + ( flock 9 I think you can grab that lock only for the actual writes, and keep the generation of the list out of the critical section. Note that i played with this "flock 9" syntax instead of what is used in deb-dl-dir, it did not work as expected. Probably because it will be many shells and 9 is a different fd in all of them. > + set -e > + printenv | grep -q BB_VERBOSE_LOGS && set -x > + sudo -E -s <<'EOSUDO' > + mkdir -p "${rootfs}/deb-src" > + mountpoint -q "${rootfs}/deb-src" || \ > + mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src" > +EOSUDO > + find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f > -iname '*\.deb' | while read package; do > + local src="$( dpkg-deb --show --showformat '${Source}' > "${package}" )" > + # If the binary package version and source package version > are different, then the > + # source package version will be present inside "()" of the > Source field. dpkg-query(1) dpkg-deb --show --showformat '${source:Version}' dpkg-deb --show --showformat '${source:Upstream-Version}' might help to write this cleaner. > + local version="$( echo "$src" | cut -sd "(" -f2 | cut -sd > ")" -f1 )" > + if [ -z ${version} ]; then > + version="$( dpkg-deb --show --showformat '${Version}' > "${package}" )" > + fi > + # Now strip any version information that might be available. > + src="$( echo "$src" | cut -d' ' -f1 )" > + # If there is no source field, then the source package has > the same name as the > + # binary package. > + if [ -z "${src}" ];then > + src="$( dpkg-deb --show --showformat '${Package}' > "${package}" )" > + fi I still could not find those proxies that all downloading functions need in their env. Henning > + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \ > + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd > "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source > source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" > "${version}" > + done > + sudo -E -s <<'EOSUDO' > + mountpoint -q "${rootfs}/deb-src" && \ > + umount -l "${rootfs}/deb-src" > + rm -rf "${rootfs}/deb-src" > +EOSUDO > + ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock" > +} > + > deb_dl_dir_import() { > export pc="${DEBDIR}/${2}" > export rootfs="${1}" > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 6b04c0a..fcaebd6 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -63,7 +63,7 @@ image_do_mounts() { > } > > ROOTFSDIR = "${IMAGE_ROOTFS}" > -ROOTFS_FEATURES += "clean-package-cache generate-manifest" > +ROOTFS_FEATURES += "clean-package-cache generate-manifest > cach-deb-src" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} > ${IMAGE_INSTALL}" ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" > > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass > index cee358c..ee57989 100644 > --- a/meta/classes/rootfs.bbclass > +++ b/meta/classes/rootfs.bbclass > @@ -185,6 +185,14 @@ python do_rootfs_install() { > } > addtask rootfs_install before do_rootfs_postprocess after do_unpack > > +ROOTFS_POSTPROCESS_COMMAND += > "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', > 'cache_deb_src', '', d)}" +cache_deb_src() { > + rootfs_install_resolvconf > + deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO} > + debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO} > + rootfs_install_clean_files > +} > + > ROOTFS_POSTPROCESS_COMMAND += > "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', > 'rootfs_postprocess_clean_package_cache', '', d)}" > rootfs_postprocess_clean_package_cache() { sudo -E chroot > '${ROOTFSDIR}' \