From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6816610675337461760 X-Received: by 2002:a2e:5746:: with SMTP id r6mr19469491ljd.15.1588141181895; Tue, 28 Apr 2020 23:19:41 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:74f:: with SMTP id 76ls5810782lfh.0.gmail; Tue, 28 Apr 2020 23:19:41 -0700 (PDT) X-Google-Smtp-Source: APiQypIrQ9KwzqRVdCdKUyw0suU4FSoh9lCEpM6/bfaLwVhts72gdsONJZKnj43sITx4U5e0kvy1 X-Received: by 2002:a05:6512:54c:: with SMTP id h12mr21067968lfl.120.1588141181146; Tue, 28 Apr 2020 23:19:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1588141181; cv=none; d=google.com; s=arc-20160816; b=t4WTzXNhWnR0hY2lhyjG0q61LoObJEtNFw0+8E0Re9HPXbHkIYkmd94K/BtKRhfUR5 tanTnYNWab1ssHY4ijShUkk0WEVS12Y+rT5ifTyD+D5ZacH+MGE0y1Rf+XaBsZEHwMZ+ PnDIwnDHMB3ZhGNdSG4KA/4BOwfoguQqi83NIbgWB5tgwWZvH/5ZCUcf7mPjC1Uecpcc 9NUuVta70/AoxtS4G/Eo9zBJApAFLoXElvUaeBYzvUhya/WY8cTloboyp7uwuIa74d1K gSbEgAKkyFUc3T1S5Bdp7M6I607ZNy99UFZnpYhn5L6bu9kV6yDg62ufo1PfNlj+c1HF kK8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=/BsJ4Tl4AvTvi3LGsoSE6NTkK9k7BBUnXAxEB1FyRNk=; b=ybvdYU+6+Akq4wekmKsDlIrT3+Q309tcb7JKg1kTNd8hvOTAqQ4zCqIsoR+sbigdrH wnTRmL1jWVSq+ZsSVCm5KS+iNEJcuBUMgmUQOvZYA1As3i4pDAlYhoWjD1OFoLf1LEef fHR/9Et0DbQnEAlENs6PVo8mTZNEKW1DNZFMhLd8ImufSDuMgb4J08Kq1jv6hWQSUdOA qyZnNFCzQUtWVc+CKpy98sFVfw/pwzSFNiAidaZ4gUedsD5kHfj17sfojxTvgwIezfhQ uOEnK0hq3jjyEHNfzkp5qOZdWTkZ+EW+4DHRZuLA7TMmswA0+9PzdVBpoVLYlx6WtaCy BVQQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id d19si72221lji.3.2020.04.28.23.19.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Apr 2020 23:19:41 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 03T6Jdki002418 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 29 Apr 2020 08:19:40 +0200 Received: from md1za8fc.ad001.siemens.net ([167.87.51.203]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 03T6JdhK016198; Wed, 29 Apr 2020 08:19:39 +0200 Date: Wed, 29 Apr 2020 08:19:37 +0200 From: Henning Schild To: vijai kumar Cc: isar-users Subject: Re: [PATCH v5 03/13] meta: cache deb srcs as part of postprocessing Message-ID: <20200429081937.7e6a059d@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20200417093040.15130-1-Vijaikumar_Kanagarajan@mentor.com> <20200417093040.15130-4-Vijaikumar_Kanagarajan@mentor.com> <20200422090639.645bf93b@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.5 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: PurLQ4X20vxV Am Wed, 22 Apr 2020 02:57:13 -0700 (PDT) schrieb vijai kumar : > On Wednesday, April 22, 2020 at 12:36:42 PM UTC+5:30, Henning Schild > wrote: > > > > On Fri, 17 Apr 2020 15:00:30 +0530 > > Vijai Kumar K > wrote: > > > > > Collect the deb sources of the corresponding deb binaries cached > > > in DEBDIR as part of image postprocess. > > > > > > Signed-off-by: Vijai Kumar K > > > --- > > > meta/classes/deb-dl-dir.bbclass | 39 > > > +++++++++++++++++++++++++++++++++ meta/classes/image.bbclass > > > | 2 +- meta/classes/rootfs.bbclass | 8 +++++++ > > > 3 files changed, 48 insertions(+), 1 deletion(-) > > > > > > diff --git a/meta/classes/deb-dl-dir.bbclass > > > b/meta/classes/deb-dl-dir.bbclass index 29a3d67..472b9fe 100644 > > > --- a/meta/classes/deb-dl-dir.bbclass > > > +++ b/meta/classes/deb-dl-dir.bbclass > > > @@ -5,6 +5,45 @@ > > > > > > inherit repository > > > > > > +debsrc_download() { > > > + export rootfs="$1" > > > + export rootfs_distro="$2" > > > + mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}" > > > + ( flock 9 > > > > I think you can grab that lock only for the actual writes, and keep > > the generation of the list out of the critical section. > > > > To note, this lock also guards the mount part. Any why does that need to be under the lock? What is the essence of the lock anyways? As far as i understand there are multiple writers potentially creating the same files in DEBSRCDIR. If that is a problem we also need locking in do_apt_fetch. While one multiconfig image is in your postprocess, another might still be fetching that same sources. > > Note that i played with this "flock 9" syntax instead of what is > > used in deb-dl-dir, it did not work as expected. Probably because > > it will be many shells and 9 is a different fd in all of them. > > > > Interesting. Works as expected here. If we still need to switch the > syntax to be sure, we could. So you did try multiconfig and two or more writers never ran at the same time? In deb-dl-dir there is exclusive writer locking and shared reader locking, maybe that is why i decided against "flock 9". > > + set -e > > > + printenv | grep -q BB_VERBOSE_LOGS && set -x > > > + sudo -E -s <<'EOSUDO' > > > + mkdir -p "${rootfs}/deb-src" > > > + mountpoint -q "${rootfs}/deb-src" || \ > > > + mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src" > > > +EOSUDO > > > + find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f > > > -iname '*\.deb' | while read package; do > > > + local src="$( dpkg-deb --show --showformat '${Source}' > > > "${package}" )" > > > + # If the binary package version and source package > > > version are different, then the > > > + # source package version will be present inside "()" of > > > the Source field. > > > > dpkg-query(1) > > > > dpkg-deb --show --showformat '${source:Version}' > > dpkg-deb --show --showformat '${source:Upstream-Version}' > > > > might help to write this cleaner. > > > > Thanks. Will use this. > > > > > > > + local version="$( echo "$src" | cut -sd "(" -f2 | cut > > > -sd ")" -f1 )" > > > + if [ -z ${version} ]; then > > > + version="$( dpkg-deb --show --showformat > > > '${Version}' "${package}" )" > > > + fi > > > + # Now strip any version information that might be > > > available. > > > + src="$( echo "$src" | cut -d' ' -f1 )" > > > + # If there is no source field, then the source package > > > has the same name as the > > > + # binary package. > > > + if [ -z "${src}" ];then > > > + src="$( dpkg-deb --show --showformat '${Package}' > > > "${package}" )" > > > + fi > > > > I still could not find those proxies that all downloading functions > > need in their env. > > > > From what I see, the rootfs class from where this is called, already > takes care of this > > https://github.com/ilbers/isar/blob/next/meta/classes/rootfs.bbclass#L22 I see. It might be a good idea to run a local test with BB_NO_NETWORK. That will show whether the proxies really "arrive" and you will learn how you feature and that switch work together. Henning > > > > > Henning > > > > > + sudo -E chroot --userspec=$( id -u ):$( id -g ) > > > ${rootfs} \ > > > + sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd > > > "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source > > > source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" > > > "${version}" > > > + done > > > + sudo -E -s <<'EOSUDO' > > > + mountpoint -q "${rootfs}/deb-src" && \ > > > + umount -l "${rootfs}/deb-src" > > > + rm -rf "${rootfs}/deb-src" > > > +EOSUDO > > > + ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock" > > > +} > > > + > > > deb_dl_dir_import() { > > > export pc="${DEBDIR}/${2}" > > > export rootfs="${1}" > > > diff --git a/meta/classes/image.bbclass > > > b/meta/classes/image.bbclass index 6b04c0a..fcaebd6 100644 > > > --- a/meta/classes/image.bbclass > > > +++ b/meta/classes/image.bbclass > > > @@ -63,7 +63,7 @@ image_do_mounts() { > > > } > > > > > > ROOTFSDIR = "${IMAGE_ROOTFS}" > > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest" > > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest > > > cach-deb-src" ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} > > > ${IMAGE_INSTALL}" ROOTFS_MANIFEST_DEPLOY_DIR ?= > > > "${DEPLOY_DIR_IMAGE}" > > > diff --git a/meta/classes/rootfs.bbclass > > > b/meta/classes/rootfs.bbclass index cee358c..ee57989 100644 > > > --- a/meta/classes/rootfs.bbclass > > > +++ b/meta/classes/rootfs.bbclass > > > @@ -185,6 +185,14 @@ python do_rootfs_install() { > > > } > > > addtask rootfs_install before do_rootfs_postprocess after > > > do_unpack > > > +ROOTFS_POSTPROCESS_COMMAND += > > > "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', > > > 'cache_deb_src', '', d)}" +cache_deb_src() { > > > + rootfs_install_resolvconf > > > + deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO} > > > + debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO} > > > + rootfs_install_clean_files > > > +} > > > + > > > ROOTFS_POSTPROCESS_COMMAND += > > > "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', > > > 'rootfs_postprocess_clean_package_cache', '', d)}" > > > rootfs_postprocess_clean_package_cache() { sudo -E chroot > > > '${ROOTFSDIR}' \ > > > > >