From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6771003121820762112 X-Received: by 2002:a81:7a8e:: with SMTP id v136mr1933691ywc.438.1578650342796; Fri, 10 Jan 2020 01:59:02 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a81:844f:: with SMTP id u76ls753070ywf.15.gmail; Fri, 10 Jan 2020 01:59:02 -0800 (PST) X-Google-Smtp-Source: APXvYqw9OiB/aCEHRdOnIivMUNAZaDuCrW24IwGCfE68q2QhbchNAMEDcWmSDcVjeLOrSYYrJnPi X-Received: by 2002:a81:5258:: with SMTP id g85mr2140867ywb.116.1578650342295; Fri, 10 Jan 2020 01:59:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1578650342; cv=none; d=google.com; s=arc-20160816; b=jAhc6QxNKafP8gsNYotTVXCFG3dfU1e9QhyaoDUi86K97xNUANNzAGFO+AccA/7Eqi Ty5RmT5OYdWNWBozE5/+A2WS6LCUyDs6GaRuwDQHg1m/q10jpmuqdZnReFrulsG1MrRG ecldL5RjQD1jiNsa2wrO4v+0fkKTBGu6fTEV9O2OK+A0m5+4TEGvVvtJ9PcdbLR6ZCsv thljZlUsiZVmRZTgQn9rFk2M0ckc/fqvingJqU0dFXG4IbZMhN87MNKklYJVdEIHAHVN M1cLxd5w6EFhBUGMX22x9zmQQPkl4ErrfDzWdkxfGuoVFde8y07HUCM0sZ7nqby6nagn 4X/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=OipW9/6yCCJwZL/L5BvTlYquPgkX5jSUin7yjrH4onU=; b=vgi27+yr2Tai/sfBRnatm7i3eeF5LtdCGl6v6RpmRvzfH/s5UGn6++NVIKavtyc82x jZKCiCWuUDqIbSuXVJmgUzJ0ijtGqp5jtLAukYSuJup9rb1+GG2qQZk6ZfyAmt7td0cV JFH2LUP3nmp5UhPAopXnMzvrI/GiJs1hzeESG//V561LJ5Zm7Q/v+hQgbh0H4Yjv28zj PdGf+5NadsPwa+Ans2ZXbbGJCs9NkZcA+mLl2Cj3+19UKhEMPbRaf5CwY/5CtGH8OBuo ruikWwQJs5chETkjZsq1XQ6fl5IPLfQ5ePjEnhJ6B3RgrboL8VU7h/bB6hWNzfVezCDL LgIQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ukNhJ79X; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::734 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com. [2607:f8b0:4864:20::734]) by gmr-mx.google.com with ESMTPS id z5si99775ywg.5.2020.01.10.01.59.02 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 10 Jan 2020 01:59:02 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::734 as permitted sender) client-ip=2607:f8b0:4864:20::734; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ukNhJ79X; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2607:f8b0:4864:20::734 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qk1-x734.google.com with SMTP id a203so1298094qkc.3 for ; Fri, 10 Jan 2020 01:59:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OipW9/6yCCJwZL/L5BvTlYquPgkX5jSUin7yjrH4onU=; b=ukNhJ79XU1RIn9uKbtgNekrRZf85YL0f7QwVhFTNHl8xt829CCxxl3TxufAzadHiia o/Ra0uMNdGyGUyb/ItHEE1vRREn9uu4c6ft8NB00vpl4YjXzRGARo1ZKxehRABQhaecJ Q31mSYehB3DoOJSzLwyvGXGLczlRNcSRfYQYhRR4ey+cCSaxKWvpb29nFiMv354hwPb6 TLAj+6LYDvXuWOCRko6uAlortr7UEMN9NAA5LQny453Nfrb6QZOl9gaK+/DLugNKQUUe jp53sig+EBZ9v+BYjfpv52S6yF4js5t1qHekROAY576xXD4lVRGm091F5c8zQe71DVCX 7mNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OipW9/6yCCJwZL/L5BvTlYquPgkX5jSUin7yjrH4onU=; b=nCLrCJ5EMAOGPx2tRXqzDVk+Vmod3f4VzcLrpJ+MD2aZzoxFNuWybfe63HundW+33i 8QjZmVurANCA/OplXA5kgZRC5WSvAlOw9qtrrR8WK9eERGjDQWb87DgZ3lMOgzV26IBl GtUtGJFiNMRts/rYhTVqs+tiA6SJgkDyALagBve4Q4AxwBxN8yTk7otR4pauGYUDTSQY NkN8hyqpUzNfg3rxldiSdJ1+S7f62FHPPn52vF42dwocPv+RD+34ByZqwgLbiijwi+Br lImqNML19kFY9PE4wAtnys4UBZUpDOd9kHvct4Vk09Mn9jJATryopeQ90nM5zjU+GF4Z 6/yw== X-Gm-Message-State: APjAAAU1DQSaTZegzAKdkeikIvmRjoWOj+bQ4wpy1ZwkvJk9VBQUTkPz eJNzF24IPKo/1PsrGbqocuKKcssJFWuk02eAdXI= X-Received: by 2002:a05:620a:15d8:: with SMTP id o24mr2252334qkm.52.1578650341842; Fri, 10 Jan 2020 01:59:01 -0800 (PST) MIME-Version: 1.0 References: <20191216115011.17664-1-henning.schild@siemens.com> <20191216115011.17664-3-henning.schild@siemens.com> <590c1f63-c923-47c1-8c48-9ed3edc766a0@googlegroups.com> <20200110105111.3d935274@md1za8fc.ad001.siemens.net> In-Reply-To: <20200110105111.3d935274@md1za8fc.ad001.siemens.net> From: vijai kumar Date: Fri, 10 Jan 2020 15:28:39 +0530 Message-ID: Subject: Re: [PATCHv2 02/25] dpkg-base: add download caching of apt:// downloads To: Henning Schild Cc: isar-users Content-Type: multipart/alternative; boundary="000000000000d9a179059bc62c64" X-TUID: Cmb7JFAmKIvu --000000000000d9a179059bc62c64 Content-Type: text/plain; charset="UTF-8" On Fri 10 Jan, 2020, 3:21 PM Henning Schild, wrote: > Am Thu, 9 Jan 2020 03:27:25 -0800 > schrieb vijai kumar : > > > On Monday, December 16, 2019 at 5:20:15 PM UTC+5:30, Henning Schild > > wrote: > > > > > > From: Henning Schild > > > > > > > Store the results of "apt-get source" in the DL_DIR. This means we > > > really only need to fetch once on incremental rebuilds or if > > > multiple recipes use the same SRC_URI. > > > We can also later collect the results and put them in a repository. > > > > > > Signed-off-by: Henning Schild > > > > --- > > > meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++-- > > > meta/conf/bitbake.conf | 1 + > > > 2 files changed, 19 insertions(+), 2 deletions(-) > > > > > > diff --git a/meta/classes/dpkg-base.bbclass > > > b/meta/classes/dpkg-base.bbclass > > > index f38168c..7a97d91 100644 > > > --- a/meta/classes/dpkg-base.bbclass > > > +++ b/meta/classes/dpkg-base.bbclass > > > @@ -63,14 +63,30 @@ do_apt_fetch() { > > > -o Dir::Etc::SourceParts="-" \ > > > -o APT::Get::List-Cleanup="0" > > > > > > - sudo -E chroot --userspec=$( id -u ):$( id -g ) > > > ${BUILDCHROOT_DIR} \ > > > - sh -c 'cd ${PP} && apt-get -y --only-source source > > > ${SRC_APT}' > > > + for uri in "${SRC_APT}"; do > > > + sudo -E chroot --userspec=$( id -u ):$( id -g ) > > > ${BUILDCHROOT_DIR} \ > > > + sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd > > > /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only > > > --only-source source "$2"' my_script "${DISTRO}" "${uri}" > > > + sudo -E chroot --userspec=$( id -u ):$( id -g ) > > > ${BUILDCHROOT_DIR} \ > > > + sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd > > > ${PP} && apt-get -y --only-source source "$2"' my_script > > > "${DISTRO}" "${uri}" > > > > Hi Henning, > > > > Thank you for the base-apt rework. > > BTW. What does my_script do here? > > It fills $0 of the embedded shell script with a "program name". > > Henning > Ah. I see. I should have known :) my bad. Thanks, Vijai Kumar K > Thanks, > > Vijai Kumar K > > > > + done > > > + > > > dpkg_undo_mounts > > > } > > > > > > addtask apt_fetch after do_unpack before do_patch > > > do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" > > > > > > +addtask cleanall_apt before do_cleanall > > > +do_cleanall_apt[nostamp] = "1" > > > +do_cleanall_apt() { > > > + if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then > > > + exit > > > + fi > > > + for uri in "${SRC_APT}"; do > > > + rm -rf "${DEBSRCDIR}"/"${DISTRO}"/"$uri" > > > + done > > > +} > > > + > > > def get_package_srcdir(d): > > > s = os.path.abspath(d.getVar("S", True)) > > > workdir = os.path.abspath(d.getVar("WORKDIR", True)) > > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > > > index 0eb3b54..ab608da 100644 > > > --- a/meta/conf/bitbake.conf > > > +++ b/meta/conf/bitbake.conf > > > @@ -26,6 +26,7 @@ DEPLOY_DIR = "${TMPDIR}/deploy" > > > FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}" > > > FILESEXTRAPATHS ?= "__default:" > > > GITDIR = "${DL_DIR}/git" > > > +DEBSRCDIR = "${DL_DIR}/deb-src" > > > P = "${PN}-${PV}" > > > PF = "${PN}-${PV}-${PR}" > > > PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', > > > False),d)[0] or 'defaultpkgname'}" > > > -- > > > 2.24.1 > > > > > > > > > > --000000000000d9a179059bc62c64 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri 10 Jan, 2020, 3:21 PM Henning Schild, <henning.schild@siemens.com>= wrote:
Am Thu, 9 Jan 2020 03:27:25= -0800
schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com&= gt;:

> On Monday, December 16, 2019 at 5:20:15 PM UTC+5:30, Henning Schild > wrote:
> >
> > From: Henning Schild <hennin...@siemens.com <javascr= ipt:>>
> >
> > Store the results of "apt-get source" in the DL_DIR. Th= is means we
> > really only need to fetch once on incremental rebuilds or if
> > multiple recipes use the same SRC_URI.
> > We can also later collect the results and put them in a repositor= y.
> >
> > Signed-off-by: Henning Schild <hennin...@siemens.com &l= t;javascript:>>
> > ---
> >=C2=A0 meta/classes/dpkg-base.bbclass | 20 ++++++++++++++++++-- > >=C2=A0 meta/conf/bitbake.conf=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 1 +
> >=C2=A0 2 files changed, 19 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/dpkg-base.bbclass
> > b/meta/classes/dpkg-base.bbclass
> > index f38168c..7a97d91 100644
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -63,14 +63,30 @@ do_apt_fetch() {
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -o Dir::Etc::SourceParts=3D&quo= t;-" \
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -o APT::Get::List-Cleanup=3D&qu= ot;0"
> >=C2=A0 =C2=A0
> > -=C2=A0 =C2=A0 sudo -E chroot --userspec=3D$( id -u ):$( id -g )<= br> > > ${BUILDCHROOT_DIR} \
> > -=C2=A0 =C2=A0 =C2=A0 =C2=A0 sh -c 'cd ${PP} && apt-g= et -y --only-source source
> > ${SRC_APT}'
> > +=C2=A0 =C2=A0 for uri in "${SRC_APT}"; do
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 sudo -E chroot --userspec=3D$( id -u= ):$( id -g )
> > ${BUILDCHROOT_DIR} \
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sh -c 'mkdir -p /d= ownloads/deb-src/"$1"/"$2" && cd
> > /downloads/deb-src/"$1"/"$2" && apt-g= et -y --download-only
> > --only-source source "$2"' my_script "${DISTRO= }" "${uri}"
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 sudo -E chroot --userspec=3D$( id -u= ):$( id -g )
> > ${BUILDCHROOT_DIR} \
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sh -c 'cp /downloa= ds/deb-src/"$1"/"$2"/* ${PP} && cd
> > ${PP} && apt-get -y --only-source source "$2"&#= 39; my_script
> > "${DISTRO}" "${uri}"
>
> Hi Henning,
>
> Thank you for the base-apt rework.
> BTW. What does my_script do here?

It fills $0 of the embedded shell script with a "program name".
Henning

Ah. I see. I should have known :) my bad.
Thanks,
Vijai Kumar K
=

> Thanks,
> Vijai Kumar K
>
> +=C2=A0 =C2=A0 done
> > +
> >=C2=A0 =C2=A0 =C2=A0 dpkg_undo_mounts
> >=C2=A0 }
> >=C2=A0 =C2=A0
> >=C2=A0 addtask apt_fetch after do_unpack before do_patch
> >=C2=A0 do_apt_fetch[lockfiles] +=3D "${REPO_ISAR_DIR}/isar.lo= ck"
> >=C2=A0 =C2=A0
> > +addtask cleanall_apt before do_cleanall
> > +do_cleanall_apt[nostamp] =3D "1"
> > +do_cleanall_apt() {
> > +=C2=A0 =C2=A0 if [ -z "${@d.getVar("SRC_APT", Tru= e).strip()}" ]; then
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 exit
> > +=C2=A0 =C2=A0 fi
> > +=C2=A0 =C2=A0 for uri in "${SRC_APT}"; do
> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 rm -rf "${DEBSRCDIR}"/&quo= t;${DISTRO}"/"$uri"
> > +=C2=A0 =C2=A0 done
> > +}
> > +
> >=C2=A0 def get_package_srcdir(d):
> >=C2=A0 =C2=A0 =C2=A0 s =3D os.path.abspath(d.getVar("S",= True))
> >=C2=A0 =C2=A0 =C2=A0 workdir =3D os.path.abspath(d.getVar("WO= RKDIR", True))
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 0eb3b54..ab608da 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -26,6 +26,7 @@ DEPLOY_DIR =3D "${TMPDIR}/deploy" > >=C2=A0 FILE_DIRNAME =3D "${@os.path.dirname(d.getVar('FIL= E', False))}"
> >=C2=A0 FILESEXTRAPATHS ?=3D "__default:"
> >=C2=A0 GITDIR =3D "${DL_DIR}/git"
> > +DEBSRCDIR =3D "${DL_DIR}/deb-src"
> >=C2=A0 P =3D "${PN}-${PV}"
> >=C2=A0 PF =3D "${PN}-${PV}-${PR}"
> >=C2=A0 PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar(= 'FILE',
> > False),d)[0] or 'defaultpkgname'}"
> > --
> > 2.24.1
> >
> >=C2=A0
>

--000000000000d9a179059bc62c64--