[PATCH v5 0/3] Custom ${S} series

[PATCH v5 0/3] Custom ${S} series

[Vijai Kumar K]

Changes since V4:
- Added P3 containing detailed documentation of apt:// in user manual.

Changes since V3:
- Fix build issue with qemumipsel-stretch.
(Need to pass --only source with --print-uris, since we are dealing
with source package)

CI build succeeded: http://ci.isar-build.org:8080/job/isar_vkk_devel/95/

Changes since V2:
- Introduce P2 to demonstrate apt:// without ${PV} set.
- Address review comment from Henning.
- Fix issue where do_apt_unpack is triggered for non apt://
SRC_URIs.

CI build going on: http://ci.isar-build.org:8080/job/isar_vkk_devel/93/

Changes since V1:

- Introduce a separate do_apt_unpack task

Vijai Kumar K (3):
  dpkg-base: Introduce do_apt_unpack
  Modify hello to demonstrate apt:// without ${PV}
  docs: Add detailed documentation about apt://

 doc/user_manual.md                            | 61 +++++++++++++++++--
 .../recipes-app/hello/{hello.inc => hello.bb} |  0
 meta-isar/recipes-app/hello/hello_2.10.bb     | 11 ----
 meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ---
 meta/classes/dpkg-base.bbclass                | 29 +++++++--
 5 files changed, 82 insertions(+), 27 deletions(-)
 rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%)
 delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb

-- 
2.17.1

[PATCH v5 1/3] dpkg-base: Introduce do_apt_unpack

[Vijai Kumar K]

The Debian source package fetch and unpacking happens now inside the
do_apt_fetch task.

With the current do_apt_fetch implementation, it is not possible to use
a custom source directory(${S}).
apt-get source by default extracts the contents of the debian source
into folder with name <pkg>_<version>.

Add provision for specifying a custom source directory.

Add a new task called do_apt_unpack and move unpacking logic there.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/dpkg-base.bbclass | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 5c7bddc..9f17906 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -58,7 +58,6 @@ do_apt_fetch() {
     if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
         return 0
     fi
-    rm -rf ${S}
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
     sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
@@ -69,16 +68,38 @@ do_apt_fetch() {
     for uri in "${SRC_APT}"; do
         sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
             sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
-        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
-            sh -c 'cp /downloads/deb-src/"$1"/"$2"/* ${PP} && cd ${PP} && apt-get -y --only-source source "$2"' my_script "${DISTRO}" "${uri}"
     done
 
     dpkg_undo_mounts
 }
 
-addtask apt_fetch after do_unpack before do_patch
+addtask apt_fetch after do_unpack before do_apt_unpack
 do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
 
+do_apt_unpack() {
+    if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
+        return 0
+    fi
+    rm -rf ${S}
+    dpkg_do_mounts
+    E="${@ isar_export_proxies(d)}"
+
+    for uri in "${SRC_APT}"; do
+        sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+            sh -c ' \
+                set -e
+                dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
+                cd ${PP}
+                cp /downloads/deb-src/"${1}"/"${2}"/* ${PP}
+                dpkg-source -x "${dscfile}" "${PPS}"' \
+                    my_script "${DISTRO}" "${uri}"
+    done
+
+    dpkg_undo_mounts
+}
+
+addtask apt_unpack after do_apt_fetch before do_patch
+
 addtask cleanall_apt before do_cleanall
 do_cleanall_apt[nostamp] = "1"
 do_cleanall_apt() {
-- 
2.17.1

[PATCH v5 2/3] Modify hello to demonstrate apt:// without ${PV}

[Vijai Kumar K]

With the introduction of do_apt_unpack, we can set custom ${S}.

With that feature in place it is no longer mandatory to specify
the ${PV} of the package.

${PV} was mandatory before because do_apt_fetch was unpacking
the contents into a folder named "<debian_source>-<debian-source-version>"
hardcoding ${S} to it. So we needed to know the correct
${PV}(debian-source-version) beforehand.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta-isar/recipes-app/hello/{hello.inc => hello.bb} |  0
 meta-isar/recipes-app/hello/hello_2.10.bb           | 11 -----------
 meta-isar/recipes-app/hello/hello_2.9.bb            |  8 --------
 3 files changed, 19 deletions(-)
 rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%)
 delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.bb
similarity index 100%
rename from meta-isar/recipes-app/hello/hello.inc
rename to meta-isar/recipes-app/hello/hello.bb
diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb b/meta-isar/recipes-app/hello/hello_2.10.bb
deleted file mode 100644
index bfb8722..0000000
--- a/meta-isar/recipes-app/hello/hello_2.10.bb
+++ /dev/null
@@ -1,11 +0,0 @@
-# Example recipe to rebuild a debian source package
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2019
-#
-# SPDX-License-Identifier: MIT
-
-require hello.inc
-
-DEFAULT_PREFERENCE_debian-buster = "1"
-DEFAULT_PREFERENCE_debian-stretch = "1"
diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb b/meta-isar/recipes-app/hello/hello_2.9.bb
deleted file mode 100644
index 2fe59d1..0000000
--- a/meta-isar/recipes-app/hello/hello_2.9.bb
+++ /dev/null
@@ -1,8 +0,0 @@
-# Example recipe to rebuild a debian source package
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2019
-#
-# SPDX-License-Identifier: MIT
-
-require hello.inc
-- 
2.17.1

[PATCH v5 3/3] docs: Add detailed documentation about apt://

[Vijai Kumar K]

Add detailed documentation about apt:// in user manual.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 doc/user_manual.md | 61 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 57 insertions(+), 4 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 9ca8e01..530f4c6 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -676,16 +676,69 @@ Before creating a new recipe it's highly recommended to take a look into the Bit
 
 Isar currently supports two ways of creating custom packages.
 
+### Compilation of upstream sources
+
+Isar does understand `SRC_URI` entries starting with "apt://". For an example
+of a customized upstream package have a look at `meta-isar/recipes-app/hello`.
+This is what you do if you want to rebuild/modify an upstream package.
+
+### apt:// options
+With apt:// you can specify the version of package you want to fetch by one of the below methods.
+
+ - Specify the right ${PV} in the recipe name or inside the recipe.
+```
+inherit dpkg
+
+PV=2.10
+
+SRC_URI = "apt://${PN}"
+```
+ - You could also specify the version in SRC_URI as below
+```
+inherit dpkg
+
+SRC_URI="apt://hello=2.10"
+```
+ - You can also specify the distribution instead of the package version.
+```
+inherit dpkg
+
+SRC_URI="apt://hello/buster"
+```
+ - You can also ignore the ${PV} or distribution name and let apt resolve the version at build time.
+
+Recipe filename: hello.bb
+```
+inherit dpkg
+
+SRC_URI="apt://hello"
+```
+
+When you use the last two methods, apt will pull the latest source package available for that particular
+distribution. This might be different than the latest binary package version available for that particular
+architecture.
+
+This happens when new source package is available via the debian security feeds, but builds are only available
+for the major architectures like amd64, i386 and arm.
+
+Please see https://www.debian.org/security/faq#archismissing for details.
+
+If the user wants to make sure that he builds the right binary package available for their architecture,
+please set ${PV}, so that the right source package is pulled for that architecture.
+
+Below are some of the packages with this scenario at the time of writing this.
+
+1. https://packages.debian.org/stretch/zstd
+2. https://packages.debian.org/stretch/hello
+3. https://packages.debian.org/stretch/apt
+4. https://packages.debian.org/stretch/busybox
+
 ### Compilation of debianized-sources
 
 The `deb` packages are built using `dpkg-buildpackage`, so the sources should contain the `debian` directory with necessary meta information. This way is the default way of adding software that needs to be compiled from source. The bbclass for this approach is called `dpkg`.
 
 **NOTE:** If the sources do not contain a `debian` directory your recipe can fetch, create, or ship that. You might want to read the the next section before returning here.
 
-This is also what you do if you want to rebuild/modify an upstream package.
-Isar does understand `SRC_URI` entries starting with "apt://". For an example
-of a customized upstream package have a look at `meta-isar/recipes-app/hello`.
-
 #### Example
 ```
 DESCRIPTION = "Sample application for ISAR"
-- 
2.17.1

Re: [PATCH v5 0/3] Custom ${S} series

[vijaikumar....@gmail.com]

[-- Attachment #1.1: Type: text/plain, Size: 1542 bytes --]

Can this be merged? If there are no other review comments?

Thanks,
Vijai Kumar K

On Monday, April 5, 2021 at 2:05:22 PM UTC+5:30 vijaikumar_...@mentor.com 
wrote:

> Changes since V4: 
> - Added P3 containing detailed documentation of apt:// in user manual. 
>
> Changes since V3: 
> - Fix build issue with qemumipsel-stretch. 
> (Need to pass --only source with --print-uris, since we are dealing 
> with source package) 
>
> CI build succeeded: http://ci.isar-build.org:8080/job/isar_vkk_devel/95/ 
>
> Changes since V2: 
> - Introduce P2 to demonstrate apt:// without ${PV} set. 
> - Address review comment from Henning. 
> - Fix issue where do_apt_unpack is triggered for non apt:// 
> SRC_URIs. 
>
> CI build going on: http://ci.isar-build.org:8080/job/isar_vkk_devel/93/ 
>
> Changes since V1: 
>
> - Introduce a separate do_apt_unpack task 
>
> Vijai Kumar K (3): 
> dpkg-base: Introduce do_apt_unpack 
> Modify hello to demonstrate apt:// without ${PV} 
> docs: Add detailed documentation about apt:// 
>
> doc/user_manual.md | 61 +++++++++++++++++-- 
> .../recipes-app/hello/{hello.inc => hello.bb} | 0 
> meta-isar/recipes-app/hello/hello_2.10.bb | 11 ---- 
> meta-isar/recipes-app/hello/hello_2.9.bb | 8 --- 
> meta/classes/dpkg-base.bbclass | 29 +++++++-- 
> 5 files changed, 82 insertions(+), 27 deletions(-) 
> rename meta-isar/recipes-app/hello/{hello.inc => hello.bb} (100%) 
> delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb 
> delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb 
>
> -- 
> 2.17.1 
>
>

[-- Attachment #1.2: Type: text/html, Size: 3890 bytes --]

Re: [PATCH v5 0/3] Custom ${S} series

[Anton Mikanovich]

15.04.2021 12:01, vijaikumar....@gmail.com wrote:
> Can this be merged? If there are no other review comments?
>
> Thanks,
> Vijai Kumar K

Merged to next, thank you.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v5 0/3] Custom ${S} series

[Jan Kiszka]

On 15.04.21 11:01, vijaikumar....@gmail.com wrote:
> Can this be merged? If there are no other review comments?
> 

LGTM!

Thanks,
Jan

> Thanks,
> Vijai Kumar K
> 
> On Monday, April 5, 2021 at 2:05:22 PM UTC+5:30
> vijaikumar_...@mentor.com wrote:
> 
>     Changes since V4:
>     - Added P3 containing detailed documentation of apt:// in user manual.
> 
>     Changes since V3:
>     - Fix build issue with qemumipsel-stretch.
>     (Need to pass --only source with --print-uris, since we are dealing
>     with source package)
> 
>     CI build succeeded:
>     http://ci.isar-build.org:8080/job/isar_vkk_devel/95/
>     <http://ci.isar-build.org:8080/job/isar_vkk_devel/95/>
> 
>     Changes since V2:
>     - Introduce P2 to demonstrate apt:// without ${PV} set.
>     - Address review comment from Henning.
>     - Fix issue where do_apt_unpack is triggered for non apt://
>     SRC_URIs.
> 
>     CI build going on:
>     http://ci.isar-build.org:8080/job/isar_vkk_devel/93/
>     <http://ci.isar-build.org:8080/job/isar_vkk_devel/93/>
> 
>     Changes since V1:
> 
>     - Introduce a separate do_apt_unpack task
> 
>     Vijai Kumar K (3):
>     dpkg-base: Introduce do_apt_unpack
>     Modify hello to demonstrate apt:// without ${PV}
>     docs: Add detailed documentation about apt://
> 
>     doc/user_manual.md | 61 +++++++++++++++++--
>     .../recipes-app/hello/{hello.inc => hello.bb <http://hello.bb>} | 0
>     meta-isar/recipes-app/hello/hello_2.10.bb <http://hello_2.10.bb> |
>     11 ----
>     meta-isar/recipes-app/hello/hello_2.9.bb <http://hello_2.9.bb> | 8 ---
>     meta/classes/dpkg-base.bbclass | 29 +++++++--
>     5 files changed, 82 insertions(+), 27 deletions(-)
>     rename meta-isar/recipes-app/hello/{hello.inc => hello.bb
>     <http://hello.bb>} (100%)
>     delete mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>     <http://hello_2.10.bb>
>     delete mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>     <http://hello_2.9.bb>
> 
>     -- 
>     2.17.1
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+unsubscribe@googlegroups.com
> <mailto:isar-users+unsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/a768b142-75a6-4dbd-893d-0289d604a982n%40googlegroups.com
> <https://groups.google.com/d/msgid/isar-users/a768b142-75a6-4dbd-893d-0289d604a982n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux