* [PATCH v4 0/2] Custom ${S} series
@ 2021-03-31 6:32 Vijai Kumar K
2021-03-31 6:32 ` [PATCH v4 1/2] dpkg-base: Introduce do_apt_unpack Vijai Kumar K
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Vijai Kumar K @ 2021-03-31 6:32 UTC (permalink / raw)
To: isar-users, henning.schild, ibr; +Cc: Vijai Kumar K
Changes since V3:
- Fix build issue with qemumipsel-stretch.
(Need to pass --only source with --print-uris, since we are dealing
with source package)
CI build succeeded: http://ci.isar-build.org:8080/job/isar_vkk_devel/95/
Changes since V2:
- Introduce P2 to demonstrate apt:// without ${PV} set.
- Address review comment from Henning.
- Fix issue where do_apt_unpack is triggered for non apt://
SRC_URIs.
CI build going on: http://ci.isar-build.org:8080/job/isar_vkk_devel/93/
Changes since V1:
- Introduce a separate do_apt_unpack task
Vijai Kumar K (2):
dpkg-base: Introduce do_apt_unpack
Modify hello to demonstrate apt:// without ${PV}
.../recipes-app/hello/{hello.inc => hello.bb} | 0
meta-isar/recipes-app/hello/hello_2.10.bb | 11 -------
meta-isar/recipes-app/hello/hello_2.9.bb | 8 -----
meta/classes/dpkg-base.bbclass | 29 ++++++++++++++++---
4 files changed, 25 insertions(+), 23 deletions(-)
rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%)
delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
--
2.17.1
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH v4 1/2] dpkg-base: Introduce do_apt_unpack 2021-03-31 6:32 [PATCH v4 0/2] Custom ${S} series Vijai Kumar K @ 2021-03-31 6:32 ` Vijai Kumar K 2021-03-31 6:32 ` [PATCH v4 2/2] Modify hello to demonstrate apt:// without ${PV} Vijai Kumar K 2021-03-31 7:37 ` [PATCH v4 0/2] Custom ${S} series vijai kumar 2 siblings, 0 replies; 6+ messages in thread From: Vijai Kumar K @ 2021-03-31 6:32 UTC (permalink / raw) To: isar-users, henning.schild, ibr; +Cc: Vijai Kumar K The Debian source package fetch and unpacking happens now inside the do_apt_fetch task. With the current do_apt_fetch implementation, it is not possible to use a custom source directory(${S}). apt-get source by default extracts the contents of the debian source into folder with name <pkg>_<version>. Add provision for specifying a custom source directory. Add a new task called do_apt_unpack and move unpacking logic there. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/classes/dpkg-base.bbclass | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index 5c7bddc..9f17906 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -58,7 +58,6 @@ do_apt_fetch() { if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then return 0 fi - rm -rf ${S} dpkg_do_mounts E="${@ isar_export_proxies(d)}" sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \ @@ -69,16 +68,38 @@ do_apt_fetch() { for uri in "${SRC_APT}"; do sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}" - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ - sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd ${PP} && apt-get -y --only-source source "$2"' my_script "${DISTRO}" "${uri}" done dpkg_undo_mounts } -addtask apt_fetch after do_unpack before do_patch +addtask apt_fetch after do_unpack before do_apt_unpack do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" +do_apt_unpack() { + if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then + return 0 + fi + rm -rf ${S} + dpkg_do_mounts + E="${@ isar_export_proxies(d)}" + + for uri in "${SRC_APT}"; do + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \ + sh -c ' \ + set -e + dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")" + cd ${PP} + cp /downloads/deb-src/"${1}"/"${2}"/* ${PP} + dpkg-source -x "${dscfile}" "${PPS}"' \ + my_script "${DISTRO}" "${uri}" + done + + dpkg_undo_mounts +} + +addtask apt_unpack after do_apt_fetch before do_patch + addtask cleanall_apt before do_cleanall do_cleanall_apt[nostamp] = "1" do_cleanall_apt() { -- 2.17.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v4 2/2] Modify hello to demonstrate apt:// without ${PV} 2021-03-31 6:32 [PATCH v4 0/2] Custom ${S} series Vijai Kumar K 2021-03-31 6:32 ` [PATCH v4 1/2] dpkg-base: Introduce do_apt_unpack Vijai Kumar K @ 2021-03-31 6:32 ` Vijai Kumar K 2021-03-31 7:37 ` [PATCH v4 0/2] Custom ${S} series vijai kumar 2 siblings, 0 replies; 6+ messages in thread From: Vijai Kumar K @ 2021-03-31 6:32 UTC (permalink / raw) To: isar-users, henning.schild, ibr; +Cc: Vijai Kumar K With the introduction of do_apt_unpack, we can set custom ${S}. With that feature in place it is no longer mandatory to specify the ${PV} of the package. ${PV} was mandatory before because do_apt_fetch was unpacking the contents into a folder named "<debian_source>-<debian-source-version>" hardcoding ${S} to it. So we needed to know the correct ${PV}(debian-source-version) beforehand. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta-isar/recipes-app/hello/{hello.inc => hello.bb} | 0 meta-isar/recipes-app/hello/hello_2.10.bb | 11 ----------- meta-isar/recipes-app/hello/hello_2.9.bb | 8 -------- 3 files changed, 19 deletions(-) rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%) delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.bb similarity index 100% rename from meta-isar/recipes-app/hello/hello.inc rename to meta-isar/recipes-app/hello/hello.bb diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb b/meta-isar/recipes-app/hello/hello_2.10.bb deleted file mode 100644 index bfb8722..0000000 --- a/meta-isar/recipes-app/hello/hello_2.10.bb +++ /dev/null @@ -1,11 +0,0 @@ -# Example recipe to rebuild a debian source package -# -# This software is a part of ISAR. -# Copyright (c) Siemens AG, 2019 -# -# SPDX-License-Identifier: MIT - -require hello.inc - -DEFAULT_PREFERENCE_debian-buster = "1" -DEFAULT_PREFERENCE_debian-stretch = "1" diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb b/meta-isar/recipes-app/hello/hello_2.9.bb deleted file mode 100644 index 2fe59d1..0000000 --- a/meta-isar/recipes-app/hello/hello_2.9.bb +++ /dev/null @@ -1,8 +0,0 @@ -# Example recipe to rebuild a debian source package -# -# This software is a part of ISAR. -# Copyright (c) Siemens AG, 2019 -# -# SPDX-License-Identifier: MIT - -require hello.inc -- 2.17.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4 0/2] Custom ${S} series 2021-03-31 6:32 [PATCH v4 0/2] Custom ${S} series Vijai Kumar K 2021-03-31 6:32 ` [PATCH v4 1/2] dpkg-base: Introduce do_apt_unpack Vijai Kumar K 2021-03-31 6:32 ` [PATCH v4 2/2] Modify hello to demonstrate apt:// without ${PV} Vijai Kumar K @ 2021-03-31 7:37 ` vijai kumar 2021-03-31 8:34 ` Henning Schild 2 siblings, 1 reply; 6+ messages in thread From: vijai kumar @ 2021-03-31 7:37 UTC (permalink / raw) To: Vijai Kumar K, Henning Schild, Baurzhan Ismagulov, Jan Kiszka; +Cc: isar-users Hi, On Wed, Mar 31, 2021 at 12:03 PM Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> wrote: > > Changes since V3: > - Fix build issue with qemumipsel-stretch. > (Need to pass --only source with --print-uris, since we are dealing > with source package) This case is something interesting. If you see the hello package in [1], There are 2 versions of the binary package. All x86 and ARM architectures have version "2.10-1+deb9u1" Rest of the architectures like mips, s390 and ppc have "2.10-1+b1" Since we are using --only-source option to download, apt skips the binary to source translation and directly fetches the latest source. Which is "2.10-1+deb9u1". That is how it could be done, since the recipe here is just the debian source file name. But if you see by apt-cache policy the installation candidate for mipsel is 2.10-1+b1 and the corresponding source version is "2.10-1". Ideally we should pull 2.10-1 instead of 2.10-1+deb9u1 for mipsel. That is what the binary is available for? I know a non-maintainer upload will have a different binary package version (+bX), but will point to the same source. Here there are 2 different sources for different archs. I am not sure if there are other such packages which have a similar scenario. Just wanted to gather info on it here. [1] https://packages.debian.org/stretch/hello Thanks, Vijai Kumar K > > CI build succeeded: http://ci.isar-build.org:8080/job/isar_vkk_devel/95/ > > Changes since V2: > - Introduce P2 to demonstrate apt:// without ${PV} set. > - Address review comment from Henning. > - Fix issue where do_apt_unpack is triggered for non apt:// > SRC_URIs. > > CI build going on: http://ci.isar-build.org:8080/job/isar_vkk_devel/93/ > > Changes since V1: > > - Introduce a separate do_apt_unpack task > > Vijai Kumar K (2): > dpkg-base: Introduce do_apt_unpack > Modify hello to demonstrate apt:// without ${PV} > > .../recipes-app/hello/{hello.inc => hello.bb} | 0 > meta-isar/recipes-app/hello/hello_2.10.bb | 11 ------- > meta-isar/recipes-app/hello/hello_2.9.bb | 8 ----- > meta/classes/dpkg-base.bbclass | 29 ++++++++++++++++--- > 4 files changed, 25 insertions(+), 23 deletions(-) > rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%) > delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb > delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb > > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20210331063235.6877-1-Vijaikumar_Kanagarajan%40mentor.com. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4 0/2] Custom ${S} series 2021-03-31 7:37 ` [PATCH v4 0/2] Custom ${S} series vijai kumar @ 2021-03-31 8:34 ` Henning Schild 2021-03-31 18:53 ` vijai kumar 0 siblings, 1 reply; 6+ messages in thread From: Henning Schild @ 2021-03-31 8:34 UTC (permalink / raw) To: vijai kumar; +Cc: Vijai Kumar K, Baurzhan Ismagulov, Jan Kiszka, isar-users Am Wed, 31 Mar 2021 13:07:17 +0530 schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>: > Hi, > > On Wed, Mar 31, 2021 at 12:03 PM Vijai Kumar K > <Vijaikumar_Kanagarajan@mentor.com> wrote: > > > > Changes since V3: > > - Fix build issue with qemumipsel-stretch. > > (Need to pass --only source with --print-uris, since we are dealing > > with source package) > > This case is something interesting. If you see the hello package in > [1], There are 2 versions of the binary package. > > All x86 and ARM architectures have version "2.10-1+deb9u1" > Rest of the architectures like mips, s390 and ppc have "2.10-1+b1" That is weird on its own. But we probably had this problem even before this series. Do those new sources produce working packages for those other arches or do we really need to find the other sources? > Since we are using --only-source option to download, apt skips the > binary to source translation and directly fetches the latest source. > Which is "2.10-1+deb9u1". That is how it could be done, since the > recipe here is just the debian source file name. > But if you see by apt-cache policy the installation candidate for > mipsel is 2.10-1+b1 and the corresponding source version is "2.10-1". Maybe we can also download that binary, maybe not store it, to make apt double check against the binary. Jan found preferences not working for apt-get source, maybe his case would also benefit from that. What i understand is that "--only-source" can maybe be skipped to download binary and source and get a cross check. > Ideally we should pull 2.10-1 instead of 2.10-1+deb9u1 for mipsel. > That is what the binary is available for? We modify the package anyways, i guess it is important to produce a working package, not too much "taking old sources" because upstream did not build new ones for that arch. Henning > I know a non-maintainer upload will have a different binary package > version (+bX), but will point to the same source. > Here there are 2 different sources for different archs. > I am not sure if there are other such packages which have a similar > scenario. Just wanted to gather info on it here. > > [1] https://packages.debian.org/stretch/hello > > Thanks, > Vijai Kumar K > > > > > CI build succeeded: > > http://ci.isar-build.org:8080/job/isar_vkk_devel/95/ > > > > Changes since V2: > > - Introduce P2 to demonstrate apt:// without ${PV} set. > > - Address review comment from Henning. > > - Fix issue where do_apt_unpack is triggered for non apt:// > > SRC_URIs. > > > > CI build going on: > > http://ci.isar-build.org:8080/job/isar_vkk_devel/93/ > > > > Changes since V1: > > > > - Introduce a separate do_apt_unpack task > > > > Vijai Kumar K (2): > > dpkg-base: Introduce do_apt_unpack > > Modify hello to demonstrate apt:// without ${PV} > > > > .../recipes-app/hello/{hello.inc => hello.bb} | 0 > > meta-isar/recipes-app/hello/hello_2.10.bb | 11 ------- > > meta-isar/recipes-app/hello/hello_2.9.bb | 8 ----- > > meta/classes/dpkg-base.bbclass | 29 > > ++++++++++++++++--- 4 files changed, 25 insertions(+), 23 > > deletions(-) rename meta-isar/recipes-app/hello/{hello.inc => > > hello.bb} (100%) delete mode 100644 > > meta-isar/recipes-app/hello/hello_2.10.bb delete mode 100644 > > meta-isar/recipes-app/hello/hello_2.9.bb > > > > -- > > 2.17.1 > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. To unsubscribe from this group and stop > > receiving emails from it, send an email to > > isar-users+unsubscribe@googlegroups.com. To view this discussion on > > the web visit > > https://groups.google.com/d/msgid/isar-users/20210331063235.6877-1-Vijaikumar_Kanagarajan%40mentor.com. > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4 0/2] Custom ${S} series 2021-03-31 8:34 ` Henning Schild @ 2021-03-31 18:53 ` vijai kumar 0 siblings, 0 replies; 6+ messages in thread From: vijai kumar @ 2021-03-31 18:53 UTC (permalink / raw) To: Henning Schild; +Cc: Vijai Kumar K, Baurzhan Ismagulov, Jan Kiszka, isar-users On Wed, Mar 31, 2021 at 2:04 PM Henning Schild <henning.schild@siemens.com> wrote: > > Am Wed, 31 Mar 2021 13:07:17 +0530 > schrieb vijai kumar <vijaikumar.kanagarajan@gmail.com>: > > > Hi, > > > > On Wed, Mar 31, 2021 at 12:03 PM Vijai Kumar K > > <Vijaikumar_Kanagarajan@mentor.com> wrote: > > > > > > Changes since V3: > > > - Fix build issue with qemumipsel-stretch. > > > (Need to pass --only source with --print-uris, since we are dealing > > > with source package) > > > > This case is something interesting. If you see the hello package in > > [1], There are 2 versions of the binary package. > > > > All x86 and ARM architectures have version "2.10-1+deb9u1" > > Rest of the architectures like mips, s390 and ppc have "2.10-1+b1" > > That is weird on its own. But we probably had this problem even before > this series. > Do those new sources produce working packages for those other arches or > do we really need to find the other sources? It does produce a working package. At least for hello. > > > Since we are using --only-source option to download, apt skips the > > binary to source translation and directly fetches the latest source. > > Which is "2.10-1+deb9u1". That is how it could be done, since the > > recipe here is just the debian source file name. > > But if you see by apt-cache policy the installation candidate for > > mipsel is 2.10-1+b1 and the corresponding source version is "2.10-1". > > Maybe we can also download that binary, maybe not store it, to make apt > double check against the binary. Jan found preferences not working for > apt-get source, maybe his case would also benefit from that. What i > understand is that "--only-source" can maybe be skipped to download > binary and source and get a cross check. For that we need to know the binary package it generates. Not all source packages produce binary packages with the same name. I still believe using --only-source is right, if the SRC_URI of the recipe is expected to point to the source package name. Which is by the way is the case. This also acts as a guard, in case the user incorrectly specifies the binary package name in SRC_URI. In such a case do_apt_fetch would simply fail. > > > Ideally we should pull 2.10-1 instead of 2.10-1+deb9u1 for mipsel. > > That is what the binary is available for? > > We modify the package anyways, i guess it is important to produce a > working package, not too much "taking old sources" because upstream did > not build new ones for that arch. I agree here. Some other packages with similar scenarios are mentioned below(not a complete list). 1. https://packages.debian.org/stretch/zstd 2. https://packages.debian.org/stretch/zsh-static 3. https://packages.debian.org/stretch/apt 4. https://packages.debian.org/stretch/busybox The updated version comes in via the security feed. Where only x86 and arm based binaries are uploaded. This FAQ[1], explains why it is missing for certain archs. I guess it should be okay to use the new source to generate the debs. Maybe we could capture this entire scenario in a doc. I could amend the PR to add some info in one of the docs. [1] https://www.debian.org/security/faq#archismissing Thanks, Vijai Kumar K > > Henning > > > I know a non-maintainer upload will have a different binary package > > version (+bX), but will point to the same source. > > Here there are 2 different sources for different archs. > > I am not sure if there are other such packages which have a similar > > scenario. Just wanted to gather info on it here. > > > > [1] https://packages.debian.org/stretch/hello > > > > Thanks, > > Vijai Kumar K > > > > > > > > CI build succeeded: > > > http://ci.isar-build.org:8080/job/isar_vkk_devel/95/ > > > > > > Changes since V2: > > > - Introduce P2 to demonstrate apt:// without ${PV} set. > > > - Address review comment from Henning. > > > - Fix issue where do_apt_unpack is triggered for non apt:// > > > SRC_URIs. > > > > > > CI build going on: > > > http://ci.isar-build.org:8080/job/isar_vkk_devel/93/ > > > > > > Changes since V1: > > > > > > - Introduce a separate do_apt_unpack task > > > > > > Vijai Kumar K (2): > > > dpkg-base: Introduce do_apt_unpack > > > Modify hello to demonstrate apt:// without ${PV} > > > > > > .../recipes-app/hello/{hello.inc => hello.bb} | 0 > > > meta-isar/recipes-app/hello/hello_2.10.bb | 11 ------- > > > meta-isar/recipes-app/hello/hello_2.9.bb | 8 ----- > > > meta/classes/dpkg-base.bbclass | 29 > > > ++++++++++++++++--- 4 files changed, 25 insertions(+), 23 > > > deletions(-) rename meta-isar/recipes-app/hello/{hello.inc => > > > hello.bb} (100%) delete mode 100644 > > > meta-isar/recipes-app/hello/hello_2.10.bb delete mode 100644 > > > meta-isar/recipes-app/hello/hello_2.9.bb > > > > > > -- > > > 2.17.1 > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "isar-users" group. To unsubscribe from this group and stop > > > receiving emails from it, send an email to > > > isar-users+unsubscribe@googlegroups.com. To view this discussion on > > > the web visit > > > https://groups.google.com/d/msgid/isar-users/20210331063235.6877-1-Vijaikumar_Kanagarajan%40mentor.com. > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-03-31 18:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-31 6:32 [PATCH v4 0/2] Custom ${S} series Vijai Kumar K
2021-03-31 6:32 ` [PATCH v4 1/2] dpkg-base: Introduce do_apt_unpack Vijai Kumar K
2021-03-31 6:32 ` [PATCH v4 2/2] Modify hello to demonstrate apt:// without ${PV} Vijai Kumar K
2021-03-31 7:37 ` [PATCH v4 0/2] Custom ${S} series vijai kumar
2021-03-31 8:34 ` Henning Schild
2021-03-31 18:53 ` vijai kumar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox