From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6790334981638979584 X-Received: by 2002:a6b:6e08:: with SMTP id d8mr4640977ioh.167.1586953764267; Wed, 15 Apr 2020 05:29:24 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:9ac1:: with SMTP id x1ls1983451ion.11.gmail; Wed, 15 Apr 2020 05:29:23 -0700 (PDT) X-Google-Smtp-Source: APiQypKNvGSTm5WxcZ2Az7zZ+FtwGYY3mKmYGH+AS4si62gi/Ihz2fSBO+x9I5HA04sc9GiSraiH X-Received: by 2002:a5d:96c5:: with SMTP id r5mr25668811iol.41.1586953763944; Wed, 15 Apr 2020 05:29:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1586953763; cv=none; d=google.com; s=arc-20160816; b=Bk7snVZH4mLVXcRKKFYK3uoaA6NRTb5r4yf4kRtqcAjebjoIUR2ljc8voAUZX41mDt /30/NksgWC1oKaGcg061OCpNNXIWxOgDapuUSQ2VY6/7lvGJqquZdeSWE+aw1YZH33ol bU5/jN522L/4FRdZ3WsqQd1eAmmuuA5b+5tyREi0d5P+DrTWO3QU0WPHJIxYgj22LkzN blJH4gSS7bXSddOC7meNa652rOSuOgvA/JJquIwY5g4W/e1tqHjha88jihrA6qZPh89u gk0IBWiOJuu89xu28zKhAiBsie/zQ8EunSahrw7niUzQf0CcAym6gqilUd2ElK4jCNnd pDRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=QUprvuYO6TU+yROXp4XtjJTMjMay8aawKoAxMUaPIcU=; b=lae2MSDGko0gM7PSoLQYq7jxsXXw1ttUu7P1PHee9mni54pC6aNWEnF1Yz/JrKbX14 Yep6HJSI1WLPNU2XxCcqeYLwkHubcv867VbRC0AQJFyTzQgwdYSrns7R4fTwKes7khyL SFeKs7MF16LzpdvLlixiq0+oePfPdy89J8rFEbMp5AErdoYjeGcHys0eFKdg7/spLcIL iCjHENfNVGP/Cez2+G9cK0r17fmbhRJuya2ydCBjvZzQShuV2YMJ+sz1izwJxRgAITIe wdLaeTudK74dLzNleWv+X3VJQW9UjkaEbSkuI13FUwyQv5/4hoksh9nFqV8SC6XZMQ0z NZ6w== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=uO2Xukm9; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::743 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com. [2607:f8b0:4864:20::743]) by gmr-mx.google.com with ESMTPS id x16si242690iov.1.2020.04.15.05.29.23 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Apr 2020 05:29:23 -0700 (PDT) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::743 as permitted sender) client-ip=2607:f8b0:4864:20::743; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=uO2Xukm9; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::743 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qk1-x743.google.com with SMTP id 20so8845284qkl.10 for ; Wed, 15 Apr 2020 05:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QUprvuYO6TU+yROXp4XtjJTMjMay8aawKoAxMUaPIcU=; b=uO2Xukm9sNu1f30LrY5sQ4rYXqCrTISTwE8U8PjWuK02NOmLlU6EP3bT0xlqzHe8O+ VAAjQF5o0grXRnlBLZ1N8OoyF+ur9hA+igT8daZIuBnrlt9HybGBVl4+5uLu4OUIOHIr hf1S0x88qg+A2bXE1h/MOid9z7FYpUqN1szddYkkDssZczvkq7QffIVqEbiqA9j9FcAC yMDT1HgbKdRXo7U8bRfcbI2/wrXg7W6VZcd2+AYOas4P0f7EDACeT3jHPZi0yzIOD8hr T0ap2AJdmnDDvx2S3Nrzw9MH0M7NZqr0cg2PBz5mK59BT6CvZ3iX13+V/YBOX9uLhb7w YA/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QUprvuYO6TU+yROXp4XtjJTMjMay8aawKoAxMUaPIcU=; b=TgtIISG/EGqIRLSeP5QSwie7VWYJo/R6TMl2TLVzMHs1WfQ/XNvbQXwFLDYl3LABpL eQ7ppHYRRDc6QneLWKo8RKmchnrFVuv1lCcrmcHYzNYckxPiu4Wpl+go6+uZUT7dxFO5 K3gfDZEF846GvHrkUGUdlMa8g+6eSYJ2COASUnv+y3PT/JniZKtlFFAB04vgWJWvPjPr JNzGQgEkG2ScAMIu83TvfpudtgWqEZd/b0bXY6Lvtb8pDwl2tzbZkJ109shbfYilLE0c w53OhaVWdnG08WjNb7ICAbxmK+E7wrjyerTSpxa7bF0pMENHXiXARJwZ/4l8ZbVVUGuM iJ1A== X-Gm-Message-State: AGi0PuYvm/REgudRl8v+n4zdnwaW3F+H7TEJ/7NmZhsuMNv2bupz1WDg noNOIpI07uwrTWmIEJgMONdzY4Q/NC/t5BhEDZE= X-Received: by 2002:a05:620a:1084:: with SMTP id g4mr25344266qkk.52.1586953763217; Wed, 15 Apr 2020 05:29:23 -0700 (PDT) MIME-Version: 1.0 References: <20200403130551.2158-1-Vijaikumar_Kanagarajan@mentor.com> <20200403130551.2158-2-Vijaikumar_Kanagarajan@mentor.com> <20200408120427.366b3968@md1za8fc.ad001.siemens.net> <20200408143008.6e2caa90@md1za8fc.ad001.siemens.net> In-Reply-To: <20200408143008.6e2caa90@md1za8fc.ad001.siemens.net> From: vijai kumar Date: Wed, 15 Apr 2020 17:59:12 +0530 Message-ID: Subject: Re: [PATCH v4 2/2] meta: cache deb srcs as part of postprocessing To: Henning Schild Cc: isar-users , Vijai Kumar K Content-Type: text/plain; charset="UTF-8" X-TUID: 3gXHbQO8wt4p On Wed, Apr 8, 2020 at 6:00 PM Henning Schild wrote: > > Am Wed, 8 Apr 2020 16:07:15 +0530 > schrieb vijai kumar : > > > On Wed, Apr 8, 2020 at 3:34 PM Henning Schild > > wrote: > > > > > > Am Fri, 3 Apr 2020 18:35:51 +0530 > > > schrieb Vijai Kumar K : > > > > > > > Collect the deb sources of the corresponding deb binaries cached > > > > in DEBDIR as part of postprocess for those to be later included > > > > into the final base-apt by do_cache. > > > > > > > > Signed-off-by: Vijai Kumar K > > > > --- > > > > meta/classes/image.bbclass | 2 +- > > > > meta/classes/rootfs.bbclass | 46 > > > > +++++++++++++++++++++++++++++++++++++ 2 files changed, 47 > > > > insertions(+), 1 deletion(-) > > > > > > > > diff --git a/meta/classes/image.bbclass > > > > b/meta/classes/image.bbclass index 9fa58f8..1c7a527 100644 > > > > --- a/meta/classes/image.bbclass > > > > +++ b/meta/classes/image.bbclass > > > > @@ -60,7 +60,7 @@ image_do_mounts() { > > > > } > > > > > > > > ROOTFSDIR = "${IMAGE_ROOTFS}" > > > > -ROOTFS_FEATURES += "copy-package-cache clean-package-cache > > > > generate-manifest" +ROOTFS_FEATURES += "copy-package-cache > > > > clean-package-cache generate-manifest cache-deb-src" > > > > ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" > > > > ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}" > > > > diff --git a/meta/classes/rootfs.bbclass > > > > b/meta/classes/rootfs.bbclass index 8bb003d..7bfdfc9 100644 > > > > --- a/meta/classes/rootfs.bbclass > > > > +++ b/meta/classes/rootfs.bbclass > > > > @@ -201,6 +201,52 @@ rootfs_generate_manifest () { > > > > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest > > > > } > > > > > > > > +ROOTFS_POSTPROCESS_COMMAND += > > > > "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', > > > > 'cache_deb_src', '', d)}" +cache_deb_src() { > > > > + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then > > > > + return 0 > > > > + fi > > > > > > Should the source packages not all end up in the cache, so they can > > > and probably should be fetched from there. > > > > Sorry. But I am not able to understand this. Can you please explain > > it again? > > A first build without the cache will fetch all sources and drop them > into "${DEBSRCDIR}"/"${DISTRO}", just the the apt:// fetcher does. > A second build with an enabled cache will place all those src-pkgs in > base-apt (see populate_base_apt repo_add_srcpackage loop). > > So a second run of this function here should be able to fetch all those > srcs-pkgs from base-apt. And it would be a good idea to actually do > that to prove that everything is available offline. > > Note that for real offline BB_NO_NETWORK would be required. And that > "guard" should still be able to download from base-apt. Thinking about > it again ... i think you do not need the guard. If all src-pkgs are > available offline this function will never access the internet, if it > still tries the invalid proxy "guard" from isar_export_proxies will > trigger. > > I think it boils down to removing the > [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && exit 0 > and passing the ci offline/cache test > > > > > > > > + mkdir -p "${DEBSRCDIR}"/"${DISTRO}" > > > > + > > > > + sudo -s <<'EOSUDO' > > > > + cp -L /etc/resolv.conf '${ROOTFSDIR}/etc' > > > > + mkdir -p '${ROOTFSDIR}/deb-src' > > > > + mountpoint -q '${ROOTFSDIR}/deb-src' || \ > > > > + mount --bind '${DEBSRCDIR}' '${ROOTFSDIR}/deb-src' > > > > +EOSUDO > > > > + > > > > + sudo -E chroot ${ROOTFSDIR} /usr/bin/apt-get update > > Why is that in here? Doing this in the image is not allowed, only for > isar-apt! Hi Henning, I am sorry. But why is it not allowed? Am I missing any side effects of this call? Thanks, Vijai Kumar K > > > > > + find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read > > > > package; do > > You are reading this without grabbing the lock. In multiconfig other > images might be filling that directory as you read it. And you might be > calling dpkg-deb on half copied files. > > Try deb_dl_dir_import and looping over /var/cache/apt/archives/ ... in > which case you will find yourself dealing with isar-apt packages that > you need to skip. > In fact you should use the manifest as input to not download packages > installed in other images with the same distro but without the feature. > > Yeahh multiconfig! > > Henning > > > > > + local src="$( dpkg-deb --show --showformat '${Source}' > > > > "${package}" )" > > > > + # If the binary package version and source package > > > > version are different, then the > > > > + # source package version will be present inside "()" of > > > > the Source field. > > > > + local version="$( echo "$src" | cut -sd "(" -f2 | cut -sd > > > > ")" -f1 )" > > > > + if [ -z ${version} ]; then > > > > + version="$( dpkg-deb --show --showformat '${Version}' > > > > "${package}" )" > > > > + fi > > > > + # Now strip any version information that might be > > > > available. > > > > + src="$( echo "$src" | cut -d' ' -f1 )" > > > > + # If there is no source field, then the source package > > > > has the same name as the > > > > + # binary package. > > > > + if [ -z "${src}" ];then > > > > + src="$( dpkg-deb --show --showformat '${Package}' > > > > "${package}" )" > > > > + fi > > > > + > > > > + sudo -E chroot --userspec=$( id -u ):$( id -g ) > > > > ${ROOTFSDIR} \ > > > > + sh -c 'mkdir -p "/deb-src/${1}/${2}" && cd > > > > "/deb-src/${1}/${2}" && \ > > > > + apt-get -y --download-only --only-source source > > > > "$2"="$3"' \ > > > > + download-src "${DISTRO}" "${src}" "${version}" > > > > + done > > > > > > Looks like we are going online without proxy configuration here. It > > > also needs a BB_NO_NETWORK guard. > > > > Will take care of that. > > > > > > > > And i would suggest to generate the list of things you want to > > > fetch, factor out the fetcher from dpkg-base and reuse is instead > > > of copying it. > > > > Sure. I will have a look into how I can reuse that part. > > > > > > > > And i would personally like a new series of patches to be sent > > > without "in-reply-to". Maybe its my client but i find these deeply > > > nested threads very hard to follow. > > > > No Problem. Will send the next series separately. > > > > Thanks, > > Vijai Kumar K > > > > > > > > Henning > > > > > > > + sudo -s <<'EOSUDO' > > > > + mountpoint -q '${ROOTFSDIR}/deb-src' && \ > > > > + umount -l ${ROOTFSDIR}/deb-src > > > > + rm -rf '${ROOTFSDIR}/etc/resolv.conf' > > > > +EOSUDO > > > > +} > > > > + > > > > do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" > > > > python do_rootfs_postprocess() { > > > > # Take care that its correctly mounted: > > > >