[PATCH 0/6] v3 of getting the custom debian package in

[PATCH 0/6] v3 of getting the custom debian package in

[Henning Schild]

This series is another repost of patches we have discussed as
"[PATCH 11-16 of 16 v2 0/6]" before. This time with a strong focus in doing
nothing but what is absolutely required to get the feature merged.

Patch 5 is _the_ patch that matters. Anything done before it is somehow
required to work towards it. That is at least the idea. Please consider
that during review.

You can consider the following threads dead from now on:
 "[PATCH 11-16 of 16 v2 0/6] and now for the second half"
 "[PATCH 0-10 of 16 v2 0/8] splitting the previous q"
 "[PATCH 00/16] fixes and introducing dpdk-bin support"

I did not test whether all individual commits actually work, i would like
to assign that task to CI and repair whatever is broken.

I decided to start without a tag for the series, consider it a fresh v0.
Here some history on the patches:

Patches 1 and 2 are basically 
 "[PATCH 0-10 of 16 v2 6/8] classes: move fetch and unpack into isar-base"
but this time with a split where the change does not happen with the move.

Patch 3: similar to
 "[PATCH 11-16 of 16 v2 4/6] meta: classes: rename dpkg to dpkg-src"
I decided on another name to reflect that you hace sources with a "debian"
directory.
And because i realized that dpkg-bin would be an incorrect name, because
that class could some day provide a cross-compile step as well. One that
is not based on /debian and the tools around that.

Patch 4: similar to
 "[PATCH 11-16 of 16 v2 3/6] meta: classes: move package_write_deb to new class isar-base-dpkg"

Patch 5: last seen under the name
 "[PATCH 11-16 of 16 v2 5/6] meta: add dpkg-bin class"
 This is _the_ patch that matters, what is before is preparation, what comes
 after is documentation.

Patch 6: last seen under the name
 "[PATCH 11-16 of 16 v2 6/6] recipes-app/hello-bin: add example on how to use dpkg-bin"

Henning Schild (6):
  meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out
  meta: Move 'do_fetch' and 'do_unpack' to base-class
  meta/dpkg: rename to dpkg-debian and add comment
  meta/dpkg-debian: Make 'do_install' more generic, prepare for pulling
    out
  meta/dpkg: add dpkg-custom class
  recipes-app/example-custom: add example on how to use dpkg-custom

 .../example-custom/example-custom_0.1.bb           | 35 ++++++++++++
 meta-isar/recipes-app/example-custom/files/README  |  1 +
 .../recipes-app/example-custom/files/postinst      | 16 ++++++
 meta-isar/recipes-app/hello/hello.bb               |  2 +-
 meta/classes/base.bbclass                          | 36 +++++++++++++
 meta/classes/dpkg-custom.bbclass                   | 57 ++++++++++++++++++++
 meta/classes/dpkg-debian.bbclass                   | 30 +++++++++++
 meta/classes/dpkg.bbclass                          | 62 ++--------------------
 8 files changed, 181 insertions(+), 58 deletions(-)
 create mode 100644 meta-isar/recipes-app/example-custom/example-custom_0.1.bb
 create mode 100644 meta-isar/recipes-app/example-custom/files/README
 create mode 100644 meta-isar/recipes-app/example-custom/files/postinst
 create mode 100644 meta/classes/dpkg-custom.bbclass
 create mode 100644 meta/classes/dpkg-debian.bbclass

-- 
2.13.5

[PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Henning Schild]

Issue:
The unpacker in dpkg is not generic since it does not unpack to
WORKDIR, so it can not be pulled into a more generic class to make it
available for other users.

Change:
Make the unpacker generic by making it unpack to WORKDIR and keep the
old semantic by overwriting this variable for the task do_unpack. Change
the comment as well, it would be wrong outside this context.

Impact:
This patch does not change the behaviour of Isar. It addresses the issue
and prepares for another patch pulling the code to another class.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 23d5e6c..81e21ce 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -31,13 +31,15 @@ do_unpack[dirs] = "${BUILDROOT}"
 do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 S ?= "${BUILDROOT}"
 
-# Unpack package and put it into working directory in buildchroot
+WORKDIR_task-unpack = "${BUILDROOT}"
+
+# Unpack package and put it into working directory
 python do_unpack() {
     src_uri = (d.getVar('SRC_URI', True) or "").split()
     if len(src_uri) == 0:
         return
 
-    rootdir = d.getVar('BUILDROOT', True)
+    rootdir = d.getVar('WORKDIR', True)
 
     try:
         fetcher = bb.fetch2.Fetch(src_uri, d)
-- 
2.13.5

[PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Henning Schild]

Issues:
unpack and fetch can only be used if you inherit dpkg, they should be
available to others i.e. classes and images

Change:
Pull the code out of the dpdk-class and stick it into the base-class. It is the
same code, no changes to it.

Impact:
This patch does not change the behaviour of Isar. It addresses the
issue, preparing for future patches.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/base.bbclass | 36 ++++++++++++++++++++++++++++++++++++
 meta/classes/dpkg.bbclass | 35 -----------------------------------
 2 files changed, 36 insertions(+), 35 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 2179ba9..3cace37 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -58,6 +58,42 @@ python do_listtasks() {
 			sys.__stdout__.write("%s\n" % e)
 }
 
+do_fetch[dirs] = "${DL_DIR}"
+
+# Fetch package from the source link
+python do_fetch() {
+    src_uri = (d.getVar('SRC_URI', True) or "").split()
+    if len(src_uri) == 0:
+        return
+
+    try:
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+        fetcher.download()
+    except bb.fetch2.BBFetchException as e:
+        raise bb.build.FuncFailed(e)
+}
+
+addtask fetch before do_build
+
+do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
+# Unpack package and put it into working directory
+python do_unpack() {
+    src_uri = (d.getVar('SRC_URI', True) or "").split()
+    if len(src_uri) == 0:
+        return
+
+    rootdir = d.getVar('WORKDIR', True)
+
+    try:
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+        fetcher.unpack(rootdir)
+    except bb.fetch2.BBFetchException as e:
+        raise bb.build.FuncFailed(e)
+}
+
+addtask unpack after do_fetch before do_build
+
 addtask build
 do_build[dirs] = "${TOPDIR}"
 python base_do_build () {
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 81e21ce..0b8c7c4 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -10,46 +10,11 @@ do_unpack[deptask] = "do_build"
 PP = "/home/builder/${PN}"
 BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
 
-do_fetch[dirs] = "${DL_DIR}"
-
-# Fetch package from the source link
-python do_fetch() {
-    src_uri = (d.getVar('SRC_URI', True) or "").split()
-    if len(src_uri) == 0:
-        return
-
-    try:
-        fetcher = bb.fetch2.Fetch(src_uri, d)
-        fetcher.download()
-    except bb.fetch2.BBFetchException as e:
-        raise bb.build.FuncFailed(e)
-}
-
-addtask fetch before do_build
-
 do_unpack[dirs] = "${BUILDROOT}"
-do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 S ?= "${BUILDROOT}"
 
 WORKDIR_task-unpack = "${BUILDROOT}"
 
-# Unpack package and put it into working directory
-python do_unpack() {
-    src_uri = (d.getVar('SRC_URI', True) or "").split()
-    if len(src_uri) == 0:
-        return
-
-    rootdir = d.getVar('WORKDIR', True)
-
-    try:
-        fetcher = bb.fetch2.Fetch(src_uri, d)
-        fetcher.unpack(rootdir)
-    except bb.fetch2.BBFetchException as e:
-        raise bb.build.FuncFailed(e)
-}
-
-addtask unpack after do_fetch before do_build
-
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 # Build package from sources using build script
-- 
2.13.5

[PATCH 3/6] meta/dpkg: rename to dpkg-debian and add comment

[Henning Schild]

Issues:
In order to have multiple classes that create debian packages we need to
come up with different names for them.

Change:
Give the class a name that is supposed to reflect that this class should
be used if you have sources and a /debian directory for them. Also add a
small comment to the header to make that more clear.
Update the one user in the Isar repository as well.

Impact:
This patch does not change the behaviour of Isar. It addresses the
issue. However, all forks of Isar will have to update all recipes that
inherited the class.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-app/hello/hello.bb               | 2 +-
 meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)
 rename meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} (84%)

diff --git a/meta-isar/recipes-app/hello/hello.bb b/meta-isar/recipes-app/hello/hello.bb
index 56424fb..30cf8a8 100644
--- a/meta-isar/recipes-app/hello/hello.bb
+++ b/meta-isar/recipes-app/hello/hello.bb
@@ -15,4 +15,4 @@ SRCREV = "ad7065ecc4840cc436bfcdac427386dbba4ea719"
 
 SRC_DIR = "git"
 
-inherit dpkg
+inherit dpkg-debian
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg-debian.bbclass
similarity index 84%
rename from meta/classes/dpkg.bbclass
rename to meta/classes/dpkg-debian.bbclass
index 0b8c7c4..7466254 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg-debian.bbclass
@@ -1,5 +1,9 @@
 # This software is a part of ISAR.
 # Copyright (C) 2015-2016 ilbers GmbH
+#
+# This class allows you to build debian packages from sources.
+# These sources need to be enabled for that, they need to have the
+# debian/ folder.
 
 # Add dependency from buildchroot creation
 DEPENDS += "buildchroot"
-- 
2.13.5

[PATCH 4/6] meta/dpkg-debian: Make 'do_install' more generic, prepare for pulling out

[Henning Schild]

Issue:
The 'do_install' in dpkg-debian is not generic since it wants to find the
debian package in BUILDROOT. And this variable is only meaningful in
this class.

Change:
Make do_install generic by making it look into WORKDIR and keep the
old semantic by overwriting this variable for the task do_install.

Impact:
This patch does not change the behaviour of Isar. It addresses the issue
and prepares for another patch pulling code to another class.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-debian.bbclass | 13 +++----------
 meta/classes/dpkg.bbclass        | 14 ++++++++++++++
 2 files changed, 17 insertions(+), 10 deletions(-)
 create mode 100644 meta/classes/dpkg.bbclass

diff --git a/meta/classes/dpkg-debian.bbclass b/meta/classes/dpkg-debian.bbclass
index 7466254..5df3678 100644
--- a/meta/classes/dpkg-debian.bbclass
+++ b/meta/classes/dpkg-debian.bbclass
@@ -5,6 +5,8 @@
 # These sources need to be enabled for that, they need to have the
 # debian/ folder.
 
+inherit dpkg
+
 # Add dependency from buildchroot creation
 DEPENDS += "buildchroot"
 do_unpack[deptask] = "do_build"
@@ -18,6 +20,7 @@ do_unpack[dirs] = "${BUILDROOT}"
 S ?= "${BUILDROOT}"
 
 WORKDIR_task-unpack = "${BUILDROOT}"
+WORKDIR_task-install = "${BUILDROOT}"
 
 do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
@@ -25,13 +28,3 @@ do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_build() {
     sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
 }
-
-
-# Install package to dedicated deploy directory
-do_install() {
-    install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
-}
-
-addtask install after do_build
-do_install[dirs] = "${DEPLOY_DIR_DEB}"
-do_install[stamp-extra-info] = "${MACHINE}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
new file mode 100644
index 0000000..4fb2757
--- /dev/null
+++ b/meta/classes/dpkg.bbclass
@@ -0,0 +1,14 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017 Siemens AG
+#
+# This class should contain common functionality of all classes or recipes
+# that create debian packages.
+
+# Install package to dedicated deploy directory
+do_install() {
+    install -m 644 ${WORKDIR}/*.deb ${DEPLOY_DIR_DEB}/
+}
+
+addtask install after do_build
+do_install[dirs] = "${DEPLOY_DIR_DEB}"
+do_install[stamp-extra-info] = "${MACHINE}"
-- 
2.13.5

[PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Issues:
1. full customizations of the images is hard to impossible to
realize in a layer without touching Isar
1.1. there is no easy way to just copy a file into the image
1.2. configuration (passwords, groups, cfg-files changes) can not be
done in a layer, there is no way too hook into multistrap or the
configure-script

Change:
Introduce a class that lets users create custom debian packages on the
fly, without having to create a /debian directory and actually building.
That allows you to pull in debian-dependencies, you could have a
package that has no content and is just there to install what you need
for a feature of your product.
Using package hooks (preinst, postinst ..) you can configure pretty much
all you want when installing the package.
The package can contain actual payload as well, basically any files that
come from "somewhere else". Say binary data like wallpapers, sound files
or application binaries.

Impact:
This patch addresses the metioned issue in a way that uses debian
mechanism. All the customizations will enjoy features like
- collission protection (multiple packages providing the same file)
- config file protection
- versioning and the ability to deploy your changes in an updateable way

This patch introduces a major new feature to Isar.

This class introduces a new class for building debian packages on the
fly. They can basically contain anything from random sources, where
building happens outside of Isar. It also allows to create meta-packages
that contain nothing but pull in dependencies, once all our packets come
in via multistrap that will come in handy.
For rootfs configuration you would use post- and pre- scripts just like
regular debian packages do.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-custom.bbclass | 57 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 meta/classes/dpkg-custom.bbclass

diff --git a/meta/classes/dpkg-custom.bbclass b/meta/classes/dpkg-custom.bbclass
new file mode 100644
index 0000000..e4e743f
--- /dev/null
+++ b/meta/classes/dpkg-custom.bbclass
@@ -0,0 +1,57 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017 Siemens AG
+
+inherit dpkg
+
+DEBIAN_DEPENDS ?= ""
+MAINTAINER ?= "FIXME Unknown maintainer"
+
+D = "${WORKDIR}/image/"
+
+# Populate folder that will be picked up as package
+# TODO this should be called 'do_install' 
+do_populate_package() {
+	bbnote "Put your files for this package in ${D}"
+}
+
+addtask populate_package after do_unpack before do_deb_package_prepare
+
+# so we can put hooks etc. in there already
+do_populate_package[dirs] = "${D}/DEBIAN"
+
+do_deb_package_prepare() {
+	cat<<-__EOF__ > ${D}/DEBIAN/control
+		Package: ${PN}
+		Architecture: `dpkg --print-architecture`
+		Section: misc
+		Priority: optional
+		Maintainer: ${MAINTAINER}
+		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
+		Version: ${PV}+isar
+		Description: ${DESCRIPTION}
+	__EOF__
+	for t in pre post
+	do
+		for a in inst rm
+		do
+			chmod -f +x ${D}/DEBIAN/${t}${a} || true
+		done
+	done
+}
+
+addtask deb_package_prepare after do_populate_package before do_deb_package_conffiles
+
+do_deb_package_conffiles() {
+	CONFFILES=${D}/DEBIAN/conffiles
+	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> $CONFFILES
+	test -s $CONFFILES || rm $CONFFILES
+}
+
+addtask deb_package_conffiles after do_deb_package_prepare before do_deb_package
+
+do_deb_package() {
+	sudo chown -R root:root ${D}/DEBIAN/
+	sudo dpkg-deb --build ${D} ${WORKDIR}
+}
+
+addtask deb_package after do_deb_package_conffiles before do_install
-- 
2.13.5

[PATCH 6/6] recipes-app/example-custom: add example on how to use dpkg-custom

[Henning Schild]

Issue:
It might not be clear to a user how to make use of the class
dpkg-custom.

Change:
Add an example that demonstrates some features.

Impact:
This patch does not change the behaviour of Isar. It addresses the issue
partially, documentation would also be useful.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .../example-custom/example-custom_0.1.bb           | 35 ++++++++++++++++++++++
 meta-isar/recipes-app/example-custom/files/README  |  1 +
 .../recipes-app/example-custom/files/postinst      | 16 ++++++++++
 3 files changed, 52 insertions(+)
 create mode 100644 meta-isar/recipes-app/example-custom/example-custom_0.1.bb
 create mode 100644 meta-isar/recipes-app/example-custom/files/README
 create mode 100644 meta-isar/recipes-app/example-custom/files/postinst

diff --git a/meta-isar/recipes-app/example-custom/example-custom_0.1.bb b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
new file mode 100644
index 0000000..efa813e
--- /dev/null
+++ b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
@@ -0,0 +1,35 @@
+# Sample application using dpkg-custom, which turns a folder (${D}) of
+# files into a .deb 
+#
+# This software is a part of ISAR.
+
+DESCRIPTION = "Sample application for ISAR"
+MAINTAINER = "Your name here <you@domain.com>"
+DEBIAN_DEPENDS = "apt"
+
+SRC_URI = "file://README \
+	   file://postinst"
+
+inherit dpkg-custom
+
+do_populate_package() {
+	bbnote "Creating ${PN} binary"
+	echo "#!/bin/sh" > ${WORKDIR}/${PN}
+	echo "echo Hello ISAR! ${PN}_${PV}" >> ${WORKDIR}/${PN}
+
+	bbnote "Putting ${PN} into package"
+	install -v -d ${D}/usr/local/bin/
+	install -v -m 755 ${WORKDIR}/${PN} ${D}/usr/local/bin/${PN}
+
+	bbnote "Now copy ${FILESDIR}/README into package"
+	install -v -d ${D}/usr/local/doc/
+	install -v -m 644 ${WORKDIR}/README ${D}/usr/local/doc/README-${P}
+
+	bbnote "Now for a debian hook, see dpkg-deb"
+	install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
+
+	bbnote "Now for a fake config file"
+	echo "# empty config file" > ${WORKDIR}/${PN}.conf
+	install -v -d ${D}/usr/local/etc/
+	install -v -m 644 ${WORKDIR}/${PN}.conf ${D}/usr/local/etc/${PN}.conf
+}
diff --git a/meta-isar/recipes-app/example-custom/files/README b/meta-isar/recipes-app/example-custom/files/README
new file mode 100644
index 0000000..6e2ce0f
--- /dev/null
+++ b/meta-isar/recipes-app/example-custom/files/README
@@ -0,0 +1 @@
+This is an example file that we get from FILESDIR in recipe.
diff --git a/meta-isar/recipes-app/example-custom/files/postinst b/meta-isar/recipes-app/example-custom/files/postinst
new file mode 100644
index 0000000..a28afa3
--- /dev/null
+++ b/meta-isar/recipes-app/example-custom/files/postinst
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+set -e
+
+if ! getent group isar >/dev/null; then
+	addgroup --quiet --system isar
+fi
+
+if ! getent passwd isar >/dev/null; then
+	useradd --system --gid isar --create-home \
+		--home /var/lib/isar --no-user-group \
+		--comment "My isar user" \
+		isar
+fi
+
+chown -R isar:isar /var/lib/isar
-- 
2.13.5

Re: [PATCH 4/6] meta/dpkg-debian: Make 'do_install' more generic, prepare for pulling out

[Henning Schild]

This was supposed to be a two step thing, first make it generic later
pull it out. Some mistake in rebasing ... will fix.

Henning

Am Wed, 30 Aug 2017 21:03:55 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> Issue:
> The 'do_install' in dpkg-debian is not generic since it wants to find
> the debian package in BUILDROOT. And this variable is only meaningful
> in this class.
> 
> Change:
> Make do_install generic by making it look into WORKDIR and keep the
> old semantic by overwriting this variable for the task do_install.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the
> issue and prepares for another patch pulling code to another class.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/dpkg-debian.bbclass | 13 +++----------
>  meta/classes/dpkg.bbclass        | 14 ++++++++++++++
>  2 files changed, 17 insertions(+), 10 deletions(-)
>  create mode 100644 meta/classes/dpkg.bbclass
> 
> diff --git a/meta/classes/dpkg-debian.bbclass
> b/meta/classes/dpkg-debian.bbclass index 7466254..5df3678 100644
> --- a/meta/classes/dpkg-debian.bbclass
> +++ b/meta/classes/dpkg-debian.bbclass
> @@ -5,6 +5,8 @@
>  # These sources need to be enabled for that, they need to have the
>  # debian/ folder.
>  
> +inherit dpkg
> +
>  # Add dependency from buildchroot creation
>  DEPENDS += "buildchroot"
>  do_unpack[deptask] = "do_build"
> @@ -18,6 +20,7 @@ do_unpack[dirs] = "${BUILDROOT}"
>  S ?= "${BUILDROOT}"
>  
>  WORKDIR_task-unpack = "${BUILDROOT}"
> +WORKDIR_task-install = "${BUILDROOT}"
>  
>  do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>  
> @@ -25,13 +28,3 @@ do_build[stamp-extra-info] =
> "${DISTRO}-${DISTRO_ARCH}" do_build() {
>      sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>  }
> -
> -
> -# Install package to dedicated deploy directory
> -do_install() {
> -    install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
> -}
> -
> -addtask install after do_build
> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
> -do_install[stamp-extra-info] = "${MACHINE}"
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> new file mode 100644
> index 0000000..4fb2757
> --- /dev/null
> +++ b/meta/classes/dpkg.bbclass
> @@ -0,0 +1,14 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2017 Siemens AG
> +#
> +# This class should contain common functionality of all classes or
> recipes +# that create debian packages.
> +
> +# Install package to dedicated deploy directory
> +do_install() {
> +    install -m 644 ${WORKDIR}/*.deb ${DEPLOY_DIR_DEB}/
> +}
> +
> +addtask install after do_build
> +do_install[dirs] = "${DEPLOY_DIR_DEB}"
> +do_install[stamp-extra-info] = "${MACHINE}"

Re: [PATCH 3/6] meta/dpkg: rename to dpkg-debian and add comment

[Claudius Heine]

Hi,

On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
> Issues:
> In order to have multiple classes that create debian packages we need to
> come up with different names for them.
> 
> Change:
> Give the class a name that is supposed to reflect that this class should
> be used if you have sources and a /debian directory for them. Also add a
> small comment to the header to make that more clear.
> Update the one user in the Isar repository as well.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the
> issue. However, all forks of Isar will have to update all recipes that
> inherited the class.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta-isar/recipes-app/hello/hello.bb               | 2 +-
>   meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} | 4 ++++
>   2 files changed, 5 insertions(+), 1 deletion(-)
>   rename meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} (84%)

I am not 100% behind the new 'dpkg-debian' name, because its redundant.
One might think: "Of course a dpkg is for debian, because dpkg stands
for debian package."
I know that you are referencing the /debian directory with the '-debian' 
prefix, but I think this is not intuitive.

Here are some other/better/worse suggestions:

   - dpkg-src-dir
     Its a directory that contains prepared debian package source files.
   - dpkg-src
     Simpler but then we should later expand this to support all source
     package formats from debian [1] in a custom unpack routine (if
     bitbake does not support all these different source package formats
     already)

[1] https://people.debian.org/~hertzog/dpkg-source.html#lbAH

Cheers,
Claudius

> 
> diff --git a/meta-isar/recipes-app/hello/hello.bb b/meta-isar/recipes-app/hello/hello.bb
> index 56424fb..30cf8a8 100644
> --- a/meta-isar/recipes-app/hello/hello.bb
> +++ b/meta-isar/recipes-app/hello/hello.bb
> @@ -15,4 +15,4 @@ SRCREV = "ad7065ecc4840cc436bfcdac427386dbba4ea719"
>   
>   SRC_DIR = "git"
>   
> -inherit dpkg
> +inherit dpkg-debian
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg-debian.bbclass
> similarity index 84%
> rename from meta/classes/dpkg.bbclass
> rename to meta/classes/dpkg-debian.bbclass
> index 0b8c7c4..7466254 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg-debian.bbclass
> @@ -1,5 +1,9 @@
>   # This software is a part of ISAR.
>   # Copyright (C) 2015-2016 ilbers GmbH
> +#
> +# This class allows you to build debian packages from sources.
> +# These sources need to be enabled for that, they need to have the
> +# debian/ folder.
>   
>   # Add dependency from buildchroot creation
>   DEPENDS += "buildchroot"
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi

On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
> Issues:
> 1. full customizations of the images is hard to impossible to
> realize in a layer without touching Isar
> 1.1. there is no easy way to just copy a file into the image
> 1.2. configuration (passwords, groups, cfg-files changes) can not be
> done in a layer, there is no way too hook into multistrap or the
> configure-script
> 
> Change:
> Introduce a class that lets users create custom debian packages on the
> fly, without having to create a /debian directory and actually building.
> That allows you to pull in debian-dependencies, you could have a
> package that has no content and is just there to install what you need
> for a feature of your product.
> Using package hooks (preinst, postinst ..) you can configure pretty much
> all you want when installing the package.
> The package can contain actual payload as well, basically any files that
> come from "somewhere else". Say binary data like wallpapers, sound files
> or application binaries.
> 
> Impact:
> This patch addresses the metioned issue in a way that uses debian
> mechanism. All the customizations will enjoy features like
> - collission protection (multiple packages providing the same file)
> - config file protection
> - versioning and the ability to deploy your changes in an updateable way
> 
> This patch introduces a major new feature to Isar.
> 
> This class introduces a new class for building debian packages on the
> fly. They can basically contain anything from random sources, where
> building happens outside of Isar. It also allows to create meta-packages
> that contain nothing but pull in dependencies, once all our packets come
> in via multistrap that will come in handy.
> For rootfs configuration you would use post- and pre- scripts just like
> regular debian packages do.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/dpkg-custom.bbclass | 57 ++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 57 insertions(+)
>   create mode 100644 meta/classes/dpkg-custom.bbclass

Also not a big fan of this name. 'custom' is a bit too unspecific for 
what it does. I get that your idea that those packages are *custom* 
made, but at the same time, so are all the packages that are directly 
created within isar.

I liked the old 'dpkg-bin' name more, but maybe even this is a bit to 
unspecific. How about 'dpkg-plain'? That describes better that those 
packages are very simple, created impromptu without much to it, IMO.

Cheers,
Claudius

> 
> diff --git a/meta/classes/dpkg-custom.bbclass b/meta/classes/dpkg-custom.bbclass
> new file mode 100644
> index 0000000..e4e743f
> --- /dev/null
> +++ b/meta/classes/dpkg-custom.bbclass
> @@ -0,0 +1,57 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2017 Siemens AG
> +
> +inherit dpkg
> +
> +DEBIAN_DEPENDS ?= ""
> +MAINTAINER ?= "FIXME Unknown maintainer"
> +
> +D = "${WORKDIR}/image/"
> +
> +# Populate folder that will be picked up as package
> +# TODO this should be called 'do_install'
> +do_populate_package() {
> +	bbnote "Put your files for this package in ${D}"
> +}
> +
> +addtask populate_package after do_unpack before do_deb_package_prepare
> +
> +# so we can put hooks etc. in there already
> +do_populate_package[dirs] = "${D}/DEBIAN"
> +
> +do_deb_package_prepare() {
> +	cat<<-__EOF__ > ${D}/DEBIAN/control
> +		Package: ${PN}
> +		Architecture: `dpkg --print-architecture`
> +		Section: misc
> +		Priority: optional
> +		Maintainer: ${MAINTAINER}
> +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
> +		Version: ${PV}+isar
> +		Description: ${DESCRIPTION}
> +	__EOF__
> +	for t in pre post
> +	do
> +		for a in inst rm
> +		do
> +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
> +		done
> +	done
> +}
> +
> +addtask deb_package_prepare after do_populate_package before do_deb_package_conffiles
> +
> +do_deb_package_conffiles() {
> +	CONFFILES=${D}/DEBIAN/conffiles
> +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> $CONFFILES
> +	test -s $CONFFILES || rm $CONFFILES
> +}
> +
> +addtask deb_package_conffiles after do_deb_package_prepare before do_deb_package
> +
> +do_deb_package() {
> +	sudo chown -R root:root ${D}/DEBIAN/
> +	sudo dpkg-deb --build ${D} ${WORKDIR}
> +}
> +
> +addtask deb_package after do_deb_package_conffiles before do_install
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Jan Kiszka]

On 2017-08-31 10:38, [ext] Claudius Heine wrote:
> Hi
> 
> On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
>> Issues:
>> 1. full customizations of the images is hard to impossible to
>> realize in a layer without touching Isar
>> 1.1. there is no easy way to just copy a file into the image
>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>> done in a layer, there is no way too hook into multistrap or the
>> configure-script
>>
>> Change:
>> Introduce a class that lets users create custom debian packages on the
>> fly, without having to create a /debian directory and actually building.
>> That allows you to pull in debian-dependencies, you could have a
>> package that has no content and is just there to install what you need
>> for a feature of your product.
>> Using package hooks (preinst, postinst ..) you can configure pretty much
>> all you want when installing the package.
>> The package can contain actual payload as well, basically any files that
>> come from "somewhere else". Say binary data like wallpapers, sound files
>> or application binaries.
>>
>> Impact:
>> This patch addresses the metioned issue in a way that uses debian
>> mechanism. All the customizations will enjoy features like
>> - collission protection (multiple packages providing the same file)
>> - config file protection
>> - versioning and the ability to deploy your changes in an updateable way
>>
>> This patch introduces a major new feature to Isar.
>>
>> This class introduces a new class for building debian packages on the
>> fly. They can basically contain anything from random sources, where
>> building happens outside of Isar. It also allows to create meta-packages
>> that contain nothing but pull in dependencies, once all our packets come
>> in via multistrap that will come in handy.
>> For rootfs configuration you would use post- and pre- scripts just like
>> regular debian packages do.
>>
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   meta/classes/dpkg-custom.bbclass | 57
>> ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 meta/classes/dpkg-custom.bbclass
> 
> Also not a big fan of this name. 'custom' is a bit too unspecific for
> what it does. I get that your idea that those packages are *custom*
> made, but at the same time, so are all the packages that are directly
> created within isar.
> 
> I liked the old 'dpkg-bin' name more, but maybe even this is a bit to
> unspecific. How about 'dpkg-plain'? That describes better that those
> packages are very simple, created impromptu without much to it, IMO.

How about dpkg-wrap, because this wraps existing stuff as-is into debian
package?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi agani, was a bit to fast sending the mail before,

On 08/31/2017 10:38 AM, [ext] Claudius Heine wrote:
> Hi
> 
> On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
>> Issues:
>> 1. full customizations of the images is hard to impossible to
>> realize in a layer without touching Isar
>> 1.1. there is no easy way to just copy a file into the image
>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>> done in a layer, there is no way too hook into multistrap or the
>> configure-script
>>
>> Change:
>> Introduce a class that lets users create custom debian packages on the
>> fly, without having to create a /debian directory and actually building.
>> That allows you to pull in debian-dependencies, you could have a
>> package that has no content and is just there to install what you need
>> for a feature of your product.
>> Using package hooks (preinst, postinst ..) you can configure pretty much
>> all you want when installing the package.
>> The package can contain actual payload as well, basically any files that
>> come from "somewhere else". Say binary data like wallpapers, sound files
>> or application binaries.
>>
>> Impact:
>> This patch addresses the metioned issue in a way that uses debian
>> mechanism. All the customizations will enjoy features like
>> - collission protection (multiple packages providing the same file)
>> - config file protection
>> - versioning and the ability to deploy your changes in an updateable way
>>
>> This patch introduces a major new feature to Isar.
>>
>> This class introduces a new class for building debian packages on the
>> fly. They can basically contain anything from random sources, where
>> building happens outside of Isar. It also allows to create meta-packages
>> that contain nothing but pull in dependencies, once all our packets come
>> in via multistrap that will come in handy.
>> For rootfs configuration you would use post- and pre- scripts just like
>> regular debian packages do.
>>
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   meta/classes/dpkg-custom.bbclass | 57 
>> ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 meta/classes/dpkg-custom.bbclass
> 
> Also not a big fan of this name. 'custom' is a bit too unspecific for 
> what it does. I get that your idea that those packages are *custom* 
> made, but at the same time, so are all the packages that are directly 
> created within isar.
> 
> I liked the old 'dpkg-bin' name more, but maybe even this is a bit to 
> unspecific. How about 'dpkg-plain'? That describes better that those 
> packages are very simple, created impromptu without much to it, IMO.
> 
> Cheers,
> Claudius
> 
>>
>> diff --git a/meta/classes/dpkg-custom.bbclass 
>> b/meta/classes/dpkg-custom.bbclass
>> new file mode 100644
>> index 0000000..e4e743f
>> --- /dev/null
>> +++ b/meta/classes/dpkg-custom.bbclass
>> @@ -0,0 +1,57 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2017 Siemens AG
>> +
>> +inherit dpkg
>> +
>> +DEBIAN_DEPENDS ?= ""
>> +MAINTAINER ?= "FIXME Unknown maintainer"
>> +
>> +D = "${WORKDIR}/image/"
>> +
>> +# Populate folder that will be picked up as package
>> +# TODO this should be called 'do_install'
>> +do_populate_package() {
>> +    bbnote "Put your files for this package in ${D}"
>> +}
>> +
>> +addtask populate_package after do_unpack before do_deb_package_prepare
>> +
>> +# so we can put hooks etc. in there already
>> +do_populate_package[dirs] = "${D}/DEBIAN"
>> +
>> +do_deb_package_prepare() {

OE has a slightly different naming scheme for these tasks. They would 
call them 'do_package_deb_prepare' or similar [1]. Might cause confusion 
if we have different naming schemes between those two projects.

>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
>> +        Package: ${PN}
>> +        Architecture: `dpkg --print-architecture`

Are you sure that this works when cross building? Wouldn't dpkg print 
the architecture of the host system instead of the target system? I 
think this kind of information has to come from the machine variables.

Cheers,
Claudius

>> +        Section: misc
>> +        Priority: optional
>> +        Maintainer: ${MAINTAINER}
>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
>> +        Version: ${PV}+isar
>> +        Description: ${DESCRIPTION}
>> +    __EOF__
>> +    for t in pre post
>> +    do
>> +        for a in inst rm
>> +        do
>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
>> +        done
>> +    done
>> +}
>> +
>> +addtask deb_package_prepare after do_populate_package before 
>> do_deb_package_conffiles
>> +
>> +do_deb_package_conffiles() {
>> +    CONFFILES=${D}/DEBIAN/conffiles
>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> 
>> $CONFFILES
>> +    test -s $CONFFILES || rm $CONFFILES
>> +}
>> +
>> +addtask deb_package_conffiles after do_deb_package_prepare before 
>> do_deb_package
>> +
>> +do_deb_package() {
>> +    sudo chown -R root:root ${D}/DEBIAN/
>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
>> +}
>> +
>> +addtask deb_package after do_deb_package_conffiles before do_install
>>
> 

[1] 
http://git.openembedded.org/openembedded-core/tree/meta/classes/package_deb.bbclass#n43

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi Jan,

On 08/31/2017 10:42 AM, Jan Kiszka wrote:
> On 2017-08-31 10:38, [ext] Claudius Heine wrote:
>> Hi
>>
>> On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
>>> Issues:
>>> 1. full customizations of the images is hard to impossible to
>>> realize in a layer without touching Isar
>>> 1.1. there is no easy way to just copy a file into the image
>>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>>> done in a layer, there is no way too hook into multistrap or the
>>> configure-script
>>>
>>> Change:
>>> Introduce a class that lets users create custom debian packages on the
>>> fly, without having to create a /debian directory and actually building.
>>> That allows you to pull in debian-dependencies, you could have a
>>> package that has no content and is just there to install what you need
>>> for a feature of your product.
>>> Using package hooks (preinst, postinst ..) you can configure pretty much
>>> all you want when installing the package.
>>> The package can contain actual payload as well, basically any files that
>>> come from "somewhere else". Say binary data like wallpapers, sound files
>>> or application binaries.
>>>
>>> Impact:
>>> This patch addresses the metioned issue in a way that uses debian
>>> mechanism. All the customizations will enjoy features like
>>> - collission protection (multiple packages providing the same file)
>>> - config file protection
>>> - versioning and the ability to deploy your changes in an updateable way
>>>
>>> This patch introduces a major new feature to Isar.
>>>
>>> This class introduces a new class for building debian packages on the
>>> fly. They can basically contain anything from random sources, where
>>> building happens outside of Isar. It also allows to create meta-packages
>>> that contain nothing but pull in dependencies, once all our packets come
>>> in via multistrap that will come in handy.
>>> For rootfs configuration you would use post- and pre- scripts just like
>>> regular debian packages do.
>>>
>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>> ---
>>>    meta/classes/dpkg-custom.bbclass | 57
>>> ++++++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 57 insertions(+)
>>>    create mode 100644 meta/classes/dpkg-custom.bbclass
>>
>> Also not a big fan of this name. 'custom' is a bit too unspecific for
>> what it does. I get that your idea that those packages are *custom*
>> made, but at the same time, so are all the packages that are directly
>> created within isar.
>>
>> I liked the old 'dpkg-bin' name more, but maybe even this is a bit to
>> unspecific. How about 'dpkg-plain'? That describes better that those
>> packages are very simple, created impromptu without much to it, IMO.
> 
> How about dpkg-wrap, because this wraps existing stuff as-is into debian
> package?

I would still be slightly in favor of 'dpkg-plain', because as a recipe 
writer you inherit from dpkg-plain, because you want to create a very 
simple debian package. With 'dpkg-wrap' I would think about a wrapper 
around dpkg packages of around the dpkg tool.

But here we are in the subjective area.

So generally its about describing what it does 'dpkg-wrap' comparing 
with describing what it provides 'dpkg-plain'.

I would still prefer the latter. But would be ok with both.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Thu, 31 Aug 2017 10:42:53 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 2017-08-31 10:38, [ext] Claudius Heine wrote:
> > Hi
> > 
> > On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:  
> >> Issues:
> >> 1. full customizations of the images is hard to impossible to
> >> realize in a layer without touching Isar
> >> 1.1. there is no easy way to just copy a file into the image
> >> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >> be done in a layer, there is no way too hook into multistrap or the
> >> configure-script
> >>
> >> Change:
> >> Introduce a class that lets users create custom debian packages on
> >> the fly, without having to create a /debian directory and actually
> >> building. That allows you to pull in debian-dependencies, you
> >> could have a package that has no content and is just there to
> >> install what you need for a feature of your product.
> >> Using package hooks (preinst, postinst ..) you can configure
> >> pretty much all you want when installing the package.
> >> The package can contain actual payload as well, basically any
> >> files that come from "somewhere else". Say binary data like
> >> wallpapers, sound files or application binaries.
> >>
> >> Impact:
> >> This patch addresses the metioned issue in a way that uses debian
> >> mechanism. All the customizations will enjoy features like
> >> - collission protection (multiple packages providing the same file)
> >> - config file protection
> >> - versioning and the ability to deploy your changes in an
> >> updateable way
> >>
> >> This patch introduces a major new feature to Isar.
> >>
> >> This class introduces a new class for building debian packages on
> >> the fly. They can basically contain anything from random sources,
> >> where building happens outside of Isar. It also allows to create
> >> meta-packages that contain nothing but pull in dependencies, once
> >> all our packets come in via multistrap that will come in handy.
> >> For rootfs configuration you would use post- and pre- scripts just
> >> like regular debian packages do.
> >>
> >> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >> ---
> >>   meta/classes/dpkg-custom.bbclass | 57
> >> ++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 meta/classes/dpkg-custom.bbclass  
> > 
> > Also not a big fan of this name. 'custom' is a bit too unspecific
> > for what it does. I get that your idea that those packages are
> > *custom* made, but at the same time, so are all the packages that
> > are directly created within isar.
> > 
> > I liked the old 'dpkg-bin' name more, but maybe even this is a bit
> > to unspecific. How about 'dpkg-plain'? That describes better that
> > those packages are very simple, created impromptu without much to
> > it, IMO.  
> 
> How about dpkg-wrap, because this wraps existing stuff as-is into
> debian package?

I explained the new names in the cover letter. dpkg-bin can not be used
because that suggests that we do not compile in this class. As of today
we do not but i can imagine a future where we do, just without the
debian/ directory.
So now dpkg-src also does not match anymore, i think.

Any suggestions for both names taking the custom-compile into account?

Henning

> Jan
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Thu, 31 Aug 2017 10:53:10 +0200
schrieb Claudius Heine <claudius.heine.ext@siemens.com>:

> Hi agani, was a bit to fast sending the mail before,
> 
> On 08/31/2017 10:38 AM, [ext] Claudius Heine wrote:
> > Hi
> > 
> > On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:  
> >> Issues:
> >> 1. full customizations of the images is hard to impossible to
> >> realize in a layer without touching Isar
> >> 1.1. there is no easy way to just copy a file into the image
> >> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >> be done in a layer, there is no way too hook into multistrap or the
> >> configure-script
> >>
> >> Change:
> >> Introduce a class that lets users create custom debian packages on
> >> the fly, without having to create a /debian directory and actually
> >> building. That allows you to pull in debian-dependencies, you
> >> could have a package that has no content and is just there to
> >> install what you need for a feature of your product.
> >> Using package hooks (preinst, postinst ..) you can configure
> >> pretty much all you want when installing the package.
> >> The package can contain actual payload as well, basically any
> >> files that come from "somewhere else". Say binary data like
> >> wallpapers, sound files or application binaries.
> >>
> >> Impact:
> >> This patch addresses the metioned issue in a way that uses debian
> >> mechanism. All the customizations will enjoy features like
> >> - collission protection (multiple packages providing the same file)
> >> - config file protection
> >> - versioning and the ability to deploy your changes in an
> >> updateable way
> >>
> >> This patch introduces a major new feature to Isar.
> >>
> >> This class introduces a new class for building debian packages on
> >> the fly. They can basically contain anything from random sources,
> >> where building happens outside of Isar. It also allows to create
> >> meta-packages that contain nothing but pull in dependencies, once
> >> all our packets come in via multistrap that will come in handy.
> >> For rootfs configuration you would use post- and pre- scripts just
> >> like regular debian packages do.
> >>
> >> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >> ---
> >>   meta/classes/dpkg-custom.bbclass | 57 
> >> ++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 meta/classes/dpkg-custom.bbclass  
> > 
> > Also not a big fan of this name. 'custom' is a bit too unspecific
> > for what it does. I get that your idea that those packages are
> > *custom* made, but at the same time, so are all the packages that
> > are directly created within isar.
> > 
> > I liked the old 'dpkg-bin' name more, but maybe even this is a bit
> > to unspecific. How about 'dpkg-plain'? That describes better that
> > those packages are very simple, created impromptu without much to
> > it, IMO.
> > 
> > Cheers,
> > Claudius
> >   
> >>
> >> diff --git a/meta/classes/dpkg-custom.bbclass 
> >> b/meta/classes/dpkg-custom.bbclass
> >> new file mode 100644
> >> index 0000000..e4e743f
> >> --- /dev/null
> >> +++ b/meta/classes/dpkg-custom.bbclass
> >> @@ -0,0 +1,57 @@
> >> +# This software is a part of ISAR.
> >> +# Copyright (C) 2017 Siemens AG
> >> +
> >> +inherit dpkg
> >> +
> >> +DEBIAN_DEPENDS ?= ""
> >> +MAINTAINER ?= "FIXME Unknown maintainer"
> >> +
> >> +D = "${WORKDIR}/image/"
> >> +
> >> +# Populate folder that will be picked up as package
> >> +# TODO this should be called 'do_install'
> >> +do_populate_package() {
> >> +    bbnote "Put your files for this package in ${D}"
> >> +}
> >> +
> >> +addtask populate_package after do_unpack before
> >> do_deb_package_prepare +
> >> +# so we can put hooks etc. in there already
> >> +do_populate_package[dirs] = "${D}/DEBIAN"
> >> +
> >> +do_deb_package_prepare() {  
> 
> OE has a slightly different naming scheme for these tasks. They would 
> call them 'do_package_deb_prepare' or similar [1]. Might cause
> confusion if we have different naming schemes between those two
> projects.

I have given up the naming discussions for now, especially comparison
to OE. These discsussions have held up important features for way too
long. I will probably accept any name chosen by others, unless it
contains implementation details and other things that do not belong
there.

> >> +    cat<<-__EOF__ > ${D}/DEBIAN/control
> >> +        Package: ${PN}
> >> +        Architecture: `dpkg --print-architecture`  
> 
> Are you sure that this works when cross building? Wouldn't dpkg print 
> the architecture of the host system instead of the target system? I 
> think this kind of information has to come from the machine variables.

Yes, will look into that.

Henning

> Cheers,
> Claudius
> 
> >> +        Section: misc
> >> +        Priority: optional
> >> +        Maintainer: ${MAINTAINER}
> >> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
> >> +        Version: ${PV}+isar
> >> +        Description: ${DESCRIPTION}
> >> +    __EOF__
> >> +    for t in pre post
> >> +    do
> >> +        for a in inst rm
> >> +        do
> >> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
> >> +        done
> >> +    done
> >> +}
> >> +
> >> +addtask deb_package_prepare after do_populate_package before 
> >> do_deb_package_conffiles
> >> +
> >> +do_deb_package_conffiles() {
> >> +    CONFFILES=${D}/DEBIAN/conffiles
> >> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> 
> >> $CONFFILES
> >> +    test -s $CONFFILES || rm $CONFFILES
> >> +}
> >> +
> >> +addtask deb_package_conffiles after do_deb_package_prepare before 
> >> do_deb_package
> >> +
> >> +do_deb_package() {
> >> +    sudo chown -R root:root ${D}/DEBIAN/
> >> +    sudo dpkg-deb --build ${D} ${WORKDIR}
> >> +}
> >> +
> >> +addtask deb_package after do_deb_package_conffiles before
> >> do_install 
> >   
> 
> [1] 
> http://git.openembedded.org/openembedded-core/tree/meta/classes/package_deb.bbclass#n43
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

On 08/31/2017 11:32 AM, Henning Schild wrote:
> Am Thu, 31 Aug 2017 10:42:53 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 2017-08-31 10:38, [ext] Claudius Heine wrote:
>>> Hi
>>>
>>> On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:
>>>> Issues:
>>>> 1. full customizations of the images is hard to impossible to
>>>> realize in a layer without touching Isar
>>>> 1.1. there is no easy way to just copy a file into the image
>>>> 1.2. configuration (passwords, groups, cfg-files changes) can not
>>>> be done in a layer, there is no way too hook into multistrap or the
>>>> configure-script
>>>>
>>>> Change:
>>>> Introduce a class that lets users create custom debian packages on
>>>> the fly, without having to create a /debian directory and actually
>>>> building. That allows you to pull in debian-dependencies, you
>>>> could have a package that has no content and is just there to
>>>> install what you need for a feature of your product.
>>>> Using package hooks (preinst, postinst ..) you can configure
>>>> pretty much all you want when installing the package.
>>>> The package can contain actual payload as well, basically any
>>>> files that come from "somewhere else". Say binary data like
>>>> wallpapers, sound files or application binaries.
>>>>
>>>> Impact:
>>>> This patch addresses the metioned issue in a way that uses debian
>>>> mechanism. All the customizations will enjoy features like
>>>> - collission protection (multiple packages providing the same file)
>>>> - config file protection
>>>> - versioning and the ability to deploy your changes in an
>>>> updateable way
>>>>
>>>> This patch introduces a major new feature to Isar.
>>>>
>>>> This class introduces a new class for building debian packages on
>>>> the fly. They can basically contain anything from random sources,
>>>> where building happens outside of Isar. It also allows to create
>>>> meta-packages that contain nothing but pull in dependencies, once
>>>> all our packets come in via multistrap that will come in handy.
>>>> For rootfs configuration you would use post- and pre- scripts just
>>>> like regular debian packages do.
>>>>
>>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>>> ---
>>>>    meta/classes/dpkg-custom.bbclass | 57
>>>> ++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 57 insertions(+)
>>>>    create mode 100644 meta/classes/dpkg-custom.bbclass
>>>
>>> Also not a big fan of this name. 'custom' is a bit too unspecific
>>> for what it does. I get that your idea that those packages are
>>> *custom* made, but at the same time, so are all the packages that
>>> are directly created within isar.
>>>
>>> I liked the old 'dpkg-bin' name more, but maybe even this is a bit
>>> to unspecific. How about 'dpkg-plain'? That describes better that
>>> those packages are very simple, created impromptu without much to
>>> it, IMO.
>>
>> How about dpkg-wrap, because this wraps existing stuff as-is into
>> debian package?
> 
> I explained the new names in the cover letter. dpkg-bin can not be used
> because that suggests that we do not compile in this class. As of today
> we do not but i can imagine a future where we do, just without the
> debian/ directory.
> So now dpkg-src also does not match anymore, i think.
> 
> Any suggestions for both names taking the custom-compile into account?

I would still use 'dpkg-plain' for 'dpkg-custom' and 'dpkg-src' or 
'dpkg-src-dir' for 'dpkg-debian'.

Customization of the current implementation of the 'dpkg-custom' class 
is a bit unusual, since it adds tasks before the do_install step. Maybe 
it needs to be refactored a bit so that packaging is done after the 
'do_install' task and then it packages just everything that is in '${D}' 
while using the preinst and the other hooks from the WORKDIR. Maybe have 
an additional directory for them, just so that its a bit cleaner.

As I pointed out in message 
<60d3f1ca-67a1-9c43-07f6-d2d89e4c2e51@siemens.com>, we should fix the 
general task pipeline for this. I would propose something like this:

fetch, unpack, configure, compile, install, package, populate_repo

So 'dpkg-custom'/'dpkg-plain' should only contain the 'do_package' and 
their subordinate tasks.

Why would the 'dpkg-debian' class need a custom compile step? Since the 
/debian directory already contains instructions to build the software.
If something needs to be done before or after compilation there, then 
just modifiy the configure or install task.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Thu, 31 Aug 2017 14:14:10 +0200
schrieb Claudius Heine <claudius.heine.ext@siemens.com>:

> On 08/31/2017 11:32 AM, Henning Schild wrote:
> > Am Thu, 31 Aug 2017 10:42:53 +0200
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> On 2017-08-31 10:38, [ext] Claudius Heine wrote:  
> >>> Hi
> >>>
> >>> On 08/30/2017 09:03 PM, [ext] Henning Schild wrote:  
> >>>> Issues:
> >>>> 1. full customizations of the images is hard to impossible to
> >>>> realize in a layer without touching Isar
> >>>> 1.1. there is no easy way to just copy a file into the image
> >>>> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >>>> be done in a layer, there is no way too hook into multistrap or
> >>>> the configure-script
> >>>>
> >>>> Change:
> >>>> Introduce a class that lets users create custom debian packages
> >>>> on the fly, without having to create a /debian directory and
> >>>> actually building. That allows you to pull in
> >>>> debian-dependencies, you could have a package that has no
> >>>> content and is just there to install what you need for a feature
> >>>> of your product. Using package hooks (preinst, postinst ..) you
> >>>> can configure pretty much all you want when installing the
> >>>> package. The package can contain actual payload as well,
> >>>> basically any files that come from "somewhere else". Say binary
> >>>> data like wallpapers, sound files or application binaries.
> >>>>
> >>>> Impact:
> >>>> This patch addresses the metioned issue in a way that uses debian
> >>>> mechanism. All the customizations will enjoy features like
> >>>> - collission protection (multiple packages providing the same
> >>>> file)
> >>>> - config file protection
> >>>> - versioning and the ability to deploy your changes in an
> >>>> updateable way
> >>>>
> >>>> This patch introduces a major new feature to Isar.
> >>>>
> >>>> This class introduces a new class for building debian packages on
> >>>> the fly. They can basically contain anything from random sources,
> >>>> where building happens outside of Isar. It also allows to create
> >>>> meta-packages that contain nothing but pull in dependencies, once
> >>>> all our packets come in via multistrap that will come in handy.
> >>>> For rootfs configuration you would use post- and pre- scripts
> >>>> just like regular debian packages do.
> >>>>
> >>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >>>> ---
> >>>>    meta/classes/dpkg-custom.bbclass | 57
> >>>> ++++++++++++++++++++++++++++++++++++++++
> >>>>    1 file changed, 57 insertions(+)
> >>>>    create mode 100644 meta/classes/dpkg-custom.bbclass  
> >>>
> >>> Also not a big fan of this name. 'custom' is a bit too unspecific
> >>> for what it does. I get that your idea that those packages are
> >>> *custom* made, but at the same time, so are all the packages that
> >>> are directly created within isar.
> >>>
> >>> I liked the old 'dpkg-bin' name more, but maybe even this is a bit
> >>> to unspecific. How about 'dpkg-plain'? That describes better that
> >>> those packages are very simple, created impromptu without much to
> >>> it, IMO.  
> >>
> >> How about dpkg-wrap, because this wraps existing stuff as-is into
> >> debian package?  
> > 
> > I explained the new names in the cover letter. dpkg-bin can not be
> > used because that suggests that we do not compile in this class. As
> > of today we do not but i can imagine a future where we do, just
> > without the debian/ directory.
> > So now dpkg-src also does not match anymore, i think.
> > 
> > Any suggestions for both names taking the custom-compile into
> > account?  
> 
> I would still use 'dpkg-plain' for 'dpkg-custom' and 'dpkg-src' or 
> 'dpkg-src-dir' for 'dpkg-debian'.

Ok, how about -raw and -src ?

> Customization of the current implementation of the 'dpkg-custom'
> class is a bit unusual, since it adds tasks before the do_install
> step. Maybe it needs to be refactored a bit so that packaging is done
> after the 'do_install' task and then it packages just everything that
> is in '${D}' while using the preinst and the other hooks from the
> WORKDIR. Maybe have an additional directory for them, just so that
> its a bit cleaner.
>
> As I pointed out in message 
> <60d3f1ca-67a1-9c43-07f6-d2d89e4c2e51@siemens.com>, we should fix the 
> general task pipeline for this. I would propose something like this:
> 
> fetch, unpack, configure, compile, install, package, populate_repo

That is another discussion about names. The patch has a note that the
task where D is populated should be called do_install, but do_install is
taken and can not be used.
do_install currently is what you would like to call populate_repo

About the "make all" and do_build please continue the following
discussion if you think it is still relevant.
https://groups.google.com/d/msg/isar-users/UyEQmcTlHtw/UnrOh3JnBQAJ
 
> So 'dpkg-custom'/'dpkg-plain' should only contain the 'do_package'
> and their subordinate tasks.

I think logically that is the case right now, the names are just
different. If i am wrong please point me to the logical problem, not
the name.
 
> Why would the 'dpkg-debian' class need a custom compile step? Since
> the /debian directory already contains instructions to build the
> software. If something needs to be done before or after compilation
> there, then just modifiy the configure or install task.

It is dpkg-custom/-raw that could gain a custom compile step. That is
why i do not want to call it -bin today.

Henning
 
> Cheers,
> Claudius
> 

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Alexander Smirnov]

On 08/30/2017 10:03 PM, Henning Schild wrote:
> Issue:
> The unpacker in dpkg is not generic since it does not unpack to
> WORKDIR, so it can not be pulled into a more generic class to make it
> available for other users.
> 

As I already wrote to mail-list, WORKDIR and BUILDROOT are different 
folders, and there was the reason to do so.

In Yocto, WORKDIR is the place which keeps the following artifacts:

  - Source code tree
  - Patches, configs
  - Binaries
  - Yocto meta information (logs and build commands)

WORKDIR is usually something like: ./tmp/work/gcc-arch/package-name

Due to Yocto performs cross-compilation, it is able to keep source core 
out of cross filesystem. But this doesn't work for Isar, which uses 
'native' compilation in chroot. That's why BUILDROOT variable was 
introduced.

So in Isar, WORKDIR stays the same and keeps logs, build commands, 
patches and other stuff (for example configs and scripts for 
buildchroot). The BUILDROOT is the path to dedicated buildchroot where 
to unpack sources and build them.

If I build multiconfig for N targets with this patch, I will have N 
distributed locations with bitbake meta information for the same package.

If I cleanall buildchroot, I will lose all the bitbake meta information 
for current distro-arch.

So at the moment I don't see any benefit of this change.

Also it's not clear, what do you mean by 'not generic', this unpack 
implementation designed for building deb packages, and at the moment no 
other users are expected.

-- 
With best regards,
Alexander Smirnov

> Change:
> Make the unpacker generic by making it unpack to WORKDIR and keep the
> old semantic by overwriting this variable for the task do_unpack. Change
> the comment as well, it would be wrong outside this context.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the issue
> and prepares for another patch pulling the code to another class.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/dpkg.bbclass | 6 ++++--
>   1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 23d5e6c..81e21ce 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -31,13 +31,15 @@ do_unpack[dirs] = "${BUILDROOT}"
>   do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   S ?= "${BUILDROOT}"
>   
> -# Unpack package and put it into working directory in buildchroot
> +WORKDIR_task-unpack = "${BUILDROOT}"
> +
> +# Unpack package and put it into working directory
>   python do_unpack() {
>       src_uri = (d.getVar('SRC_URI', True) or "").split()
>       if len(src_uri) == 0:
>           return
>   
> -    rootdir = d.getVar('BUILDROOT', True)
> +    rootdir = d.getVar('WORKDIR', True)
>   
>       try:
>           fetcher = bb.fetch2.Fetch(src_uri, d)
> 

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Alexander Smirnov]

On 08/30/2017 10:03 PM, Henning Schild wrote:
> Issues:
> unpack and fetch can only be used if you inherit dpkg, they should be
> available to others i.e. classes and images

For me it's ok to move fetch to base class. Unpack has dependency from 
dpkg class.

-- 
With best regards,
Alexander Smirnov

> 
> Change:
> Pull the code out of the dpdk-class and stick it into the base-class. It is the
> same code, no changes to it.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the
> issue, preparing for future patches.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/base.bbclass | 36 ++++++++++++++++++++++++++++++++++++
>   meta/classes/dpkg.bbclass | 35 -----------------------------------
>   2 files changed, 36 insertions(+), 35 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 2179ba9..3cace37 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -58,6 +58,42 @@ python do_listtasks() {
>   			sys.__stdout__.write("%s\n" % e)
>   }
>   
> +do_fetch[dirs] = "${DL_DIR}"
> +
> +# Fetch package from the source link
> +python do_fetch() {
> +    src_uri = (d.getVar('SRC_URI', True) or "").split()
> +    if len(src_uri) == 0:
> +        return
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(src_uri, d)
> +        fetcher.download()
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}
> +
> +addtask fetch before do_build
> +
> +do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
> +# Unpack package and put it into working directory
> +python do_unpack() {
> +    src_uri = (d.getVar('SRC_URI', True) or "").split()
> +    if len(src_uri) == 0:
> +        return
> +
> +    rootdir = d.getVar('WORKDIR', True)
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(src_uri, d)
> +        fetcher.unpack(rootdir)
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}
> +
> +addtask unpack after do_fetch before do_build
> +
>   addtask build
>   do_build[dirs] = "${TOPDIR}"
>   python base_do_build () {
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 81e21ce..0b8c7c4 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -10,46 +10,11 @@ do_unpack[deptask] = "do_build"
>   PP = "/home/builder/${PN}"
>   BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
>   
> -do_fetch[dirs] = "${DL_DIR}"
> -
> -# Fetch package from the source link
> -python do_fetch() {
> -    src_uri = (d.getVar('SRC_URI', True) or "").split()
> -    if len(src_uri) == 0:
> -        return
> -
> -    try:
> -        fetcher = bb.fetch2.Fetch(src_uri, d)
> -        fetcher.download()
> -    except bb.fetch2.BBFetchException as e:
> -        raise bb.build.FuncFailed(e)
> -}
> -
> -addtask fetch before do_build
> -
>   do_unpack[dirs] = "${BUILDROOT}"
> -do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   S ?= "${BUILDROOT}"
>   
>   WORKDIR_task-unpack = "${BUILDROOT}"
>   
> -# Unpack package and put it into working directory
> -python do_unpack() {
> -    src_uri = (d.getVar('SRC_URI', True) or "").split()
> -    if len(src_uri) == 0:
> -        return
> -
> -    rootdir = d.getVar('WORKDIR', True)
> -
> -    try:
> -        fetcher = bb.fetch2.Fetch(src_uri, d)
> -        fetcher.unpack(rootdir)
> -    except bb.fetch2.BBFetchException as e:
> -        raise bb.build.FuncFailed(e)
> -}
> -
> -addtask unpack after do_fetch before do_build
> -
>   do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   
>   # Build package from sources using build script
>

Re: [PATCH 3/6] meta/dpkg: rename to dpkg-debian and add comment

[Alexander Smirnov]

On 08/30/2017 10:03 PM, Henning Schild wrote:
> Issues:
> In order to have multiple classes that create debian packages we need to
> come up with different names for them.
> 

Please describe, what do you mean by multiple classes to create debian 
packages? Why single class is not enough? What should do the other classes?

-- 
With best regards,
Alexander Smirnov

> Change:
> Give the class a name that is supposed to reflect that this class should
> be used if you have sources and a /debian directory for them. Also add a
> small comment to the header to make that more clear.
> Update the one user in the Isar repository as well.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the
> issue. However, all forks of Isar will have to update all recipes that
> inherited the class.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta-isar/recipes-app/hello/hello.bb               | 2 +-
>   meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} | 4 ++++
>   2 files changed, 5 insertions(+), 1 deletion(-)
>   rename meta/classes/{dpkg.bbclass => dpkg-debian.bbclass} (84%)
> 
> diff --git a/meta-isar/recipes-app/hello/hello.bb b/meta-isar/recipes-app/hello/hello.bb
> index 56424fb..30cf8a8 100644
> --- a/meta-isar/recipes-app/hello/hello.bb
> +++ b/meta-isar/recipes-app/hello/hello.bb
> @@ -15,4 +15,4 @@ SRCREV = "ad7065ecc4840cc436bfcdac427386dbba4ea719"
>   
>   SRC_DIR = "git"
>   
> -inherit dpkg
> +inherit dpkg-debian
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg-debian.bbclass
> similarity index 84%
> rename from meta/classes/dpkg.bbclass
> rename to meta/classes/dpkg-debian.bbclass
> index 0b8c7c4..7466254 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg-debian.bbclass
> @@ -1,5 +1,9 @@
>   # This software is a part of ISAR.
>   # Copyright (C) 2015-2016 ilbers GmbH
> +#
> +# This class allows you to build debian packages from sources.
> +# These sources need to be enabled for that, they need to have the
> +# debian/ folder.
>   
>   # Add dependency from buildchroot creation
>   DEPENDS += "buildchroot"
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Alexander Smirnov]

On 08/30/2017 10:03 PM, Henning Schild wrote:
> Issues:
> 1. full customizations of the images is hard to impossible to
> realize in a layer without touching Isar
> 1.1. there is no easy way to just copy a file into the image
> 1.2. configuration (passwords, groups, cfg-files changes) can not be
> done in a layer, there is no way too hook into multistrap or the
> configure-script
> 
> Change:
> Introduce a class that lets users create custom debian packages on the
> fly, without having to create a /debian directory and actually building.
> That allows you to pull in debian-dependencies, you could have a
> package that has no content and is just there to install what you need
> for a feature of your product.
> Using package hooks (preinst, postinst ..) you can configure pretty much
> all you want when installing the package.
> The package can contain actual payload as well, basically any files that
> come from "somewhere else". Say binary data like wallpapers, sound files
> or application binaries.
> 
> Impact:
> This patch addresses the metioned issue in a way that uses debian
> mechanism. All the customizations will enjoy features like
> - collission protection (multiple packages providing the same file)
> - config file protection
> - versioning and the ability to deploy your changes in an updateable way
> 
> This patch introduces a major new feature to Isar.
> 
> This class introduces a new class for building debian packages on the
> fly. They can basically contain anything from random sources, where
> building happens outside of Isar. It also allows to create meta-packages
> that contain nothing but pull in dependencies, once all our packets come
> in via multistrap that will come in handy.
> For rootfs configuration you would use post- and pre- scripts just like
> regular debian packages do.
> 

Good idea.

> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/dpkg-custom.bbclass | 57 ++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 57 insertions(+)
>   create mode 100644 meta/classes/dpkg-custom.bbclass
> 
> diff --git a/meta/classes/dpkg-custom.bbclass b/meta/classes/dpkg-custom.bbclass
> new file mode 100644
> index 0000000..e4e743f
> --- /dev/null
> +++ b/meta/classes/dpkg-custom.bbclass
> @@ -0,0 +1,57 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2017 Siemens AG
> +
> +inherit dpkg
> +
> +DEBIAN_DEPENDS ?= ""
> +MAINTAINER ?= "FIXME Unknown maintainer"
> +
> +D = "${WORKDIR}/image/"
> +
> +# Populate folder that will be picked up as package
> +# TODO this should be called 'do_install'
> +do_populate_package() {
> +	bbnote "Put your files for this package in ${D}"
> +}
> +
> +addtask populate_package after do_unpack before do_deb_package_prepare
> +
> +# so we can put hooks etc. in there already
> +do_populate_package[dirs] = "${D}/DEBIAN"
> +
> +do_deb_package_prepare() {
> +	cat<<-__EOF__ > ${D}/DEBIAN/control
> +		Package: ${PN}
> +		Architecture: `dpkg --print-architecture`
> +		Section: misc
> +		Priority: optional
> +		Maintainer: ${MAINTAINER}
> +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
> +		Version: ${PV}+isar
> +		Description: ${DESCRIPTION}
> +	__EOF__
> +	for t in pre post
> +	do
> +		for a in inst rm
> +		do
> +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
> +		done
> +	done
> +}
> +
> +addtask deb_package_prepare after do_populate_package before do_deb_package_conffiles
> +
> +do_deb_package_conffiles() {
> +	CONFFILES=${D}/DEBIAN/conffiles
> +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> $CONFFILES
> +	test -s $CONFFILES || rm $CONFFILES
> +}
> +
> +addtask deb_package_conffiles after do_deb_package_prepare before do_deb_package
> +
> +do_deb_package() {
> +	sudo chown -R root:root ${D}/DEBIAN/
> +	sudo dpkg-deb --build ${D} ${WORKDIR}
> +}
> +
> +addtask deb_package after do_deb_package_conffiles before do_install
> 

If I got this correctly, the difference from dpkg-debian flow is that 
you generate 'debian' folder on the fly. So:

1. Why we need another do_deb_package, while do_build from dpkg class 
does the same?

2. Can we just add optional task to generic pipeline which generates 
debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian 
]; then blablabla.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 6/6] recipes-app/example-custom: add example on how to use dpkg-custom

[Alexander Smirnov]

On 08/30/2017 10:03 PM, Henning Schild wrote:
> Issue:
> It might not be clear to a user how to make use of the class
> dpkg-custom.
> 
> Change:
> Add an example that demonstrates some features.
> 
> Impact:
> This patch does not change the behaviour of Isar. It addresses the issue
> partially, documentation would also be useful.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   .../example-custom/example-custom_0.1.bb           | 35 ++++++++++++++++++++++
>   meta-isar/recipes-app/example-custom/files/README  |  1 +
>   .../recipes-app/example-custom/files/postinst      | 16 ++++++++++
>   3 files changed, 52 insertions(+)
>   create mode 100644 meta-isar/recipes-app/example-custom/example-custom_0.1.bb
>   create mode 100644 meta-isar/recipes-app/example-custom/files/README
>   create mode 100644 meta-isar/recipes-app/example-custom/files/postinst
> 
> diff --git a/meta-isar/recipes-app/example-custom/example-custom_0.1.bb b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
> new file mode 100644
> index 0000000..efa813e
> --- /dev/null
> +++ b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
> @@ -0,0 +1,35 @@
> +# Sample application using dpkg-custom, which turns a folder (${D}) of
> +# files into a .deb
> +#
> +# This software is a part of ISAR.
> +
> +DESCRIPTION = "Sample application for ISAR"
> +MAINTAINER = "Your name here <you@domain.com>"
> +DEBIAN_DEPENDS = "apt"
> +
> +SRC_URI = "file://README \
> +	   file://postinst"
> +
> +inherit dpkg-custom
> +
> +do_populate_package() {
> +	bbnote "Creating ${PN} binary"
> +	echo "#!/bin/sh" > ${WORKDIR}/${PN}
> +	echo "echo Hello ISAR! ${PN}_${PV}" >> ${WORKDIR}/${PN}
> +
> +	bbnote "Putting ${PN} into package"
> +	install -v -d ${D}/usr/local/bin/
> +	install -v -m 755 ${WORKDIR}/${PN} ${D}/usr/local/bin/${PN}
> +
> +	bbnote "Now copy ${FILESDIR}/README into package"
> +	install -v -d ${D}/usr/local/doc/
> +	install -v -m 644 ${WORKDIR}/README ${D}/usr/local/doc/README-${P}
> +
> +	bbnote "Now for a debian hook, see dpkg-deb"
> +	install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
> +
> +	bbnote "Now for a fake config file"
> +	echo "# empty config file" > ${WORKDIR}/${PN}.conf
> +	install -v -d ${D}/usr/local/etc/
> +	install -v -m 644 ${WORKDIR}/${PN}.conf ${D}/usr/local/etc/${PN}.conf
> +}
> diff --git a/meta-isar/recipes-app/example-custom/files/README b/meta-isar/recipes-app/example-custom/files/README
> new file mode 100644
> index 0000000..6e2ce0f
> --- /dev/null
> +++ b/meta-isar/recipes-app/example-custom/files/README
> @@ -0,0 +1 @@
> +This is an example file that we get from FILESDIR in recipe.
> diff --git a/meta-isar/recipes-app/example-custom/files/postinst b/meta-isar/recipes-app/example-custom/files/postinst
> new file mode 100644
> index 0000000..a28afa3
> --- /dev/null
> +++ b/meta-isar/recipes-app/example-custom/files/postinst
> @@ -0,0 +1,16 @@
> +#!/bin/sh
> +
> +set -e
> +
> +if ! getent group isar >/dev/null; then
> +	addgroup --quiet --system isar
> +fi
> +
> +if ! getent passwd isar >/dev/null; then
> +	useradd --system --gid isar --create-home \
> +		--home /var/lib/isar --no-user-group \
> +		--comment "My isar user" \
> +		isar
> +fi
> +
> +chown -R isar:isar /var/lib/isar
> 

That's a good demo scenario. But probably it makes sense to demonstrate 
it on real use case, for example set default password for root. This 
anyway should be done, probably it's a good time :-)

-- 
With best regards,
Alexander Smirnov

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Jan Kiszka]

On 2017-09-04 17:14, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issues:
>> unpack and fetch can only be used if you inherit dpkg, they should be
>> available to others i.e. classes and images
> 
> For me it's ok to move fetch to base class. Unpack has dependency from
> dpkg class.
> 

What is that dependency?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 3/6] meta/dpkg: rename to dpkg-debian and add comment

[Jan Kiszka]

On 2017-09-04 17:17, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issues:
>> In order to have multiple classes that create debian packages we need to
>> come up with different names for them.
>>
> 
> Please describe, what do you mean by multiple classes to create debian
> packages? Why single class is not enough? What should do the other classes?

Because one class is supposed to work against a debianized repo, the
other to generate the debian meta files ad-hoc according to the given
set of files to package. Patch 5. Maybe read the whole series first, and
the cover letter.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Jan Kiszka]

On 2017-09-04 17:36, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issues:
>> 1. full customizations of the images is hard to impossible to
>> realize in a layer without touching Isar
>> 1.1. there is no easy way to just copy a file into the image
>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>> done in a layer, there is no way too hook into multistrap or the
>> configure-script
>>
>> Change:
>> Introduce a class that lets users create custom debian packages on the
>> fly, without having to create a /debian directory and actually building.
>> That allows you to pull in debian-dependencies, you could have a
>> package that has no content and is just there to install what you need
>> for a feature of your product.
>> Using package hooks (preinst, postinst ..) you can configure pretty much
>> all you want when installing the package.
>> The package can contain actual payload as well, basically any files that
>> come from "somewhere else". Say binary data like wallpapers, sound files
>> or application binaries.
>>
>> Impact:
>> This patch addresses the metioned issue in a way that uses debian
>> mechanism. All the customizations will enjoy features like
>> - collission protection (multiple packages providing the same file)
>> - config file protection
>> - versioning and the ability to deploy your changes in an updateable way
>>
>> This patch introduces a major new feature to Isar.
>>
>> This class introduces a new class for building debian packages on the
>> fly. They can basically contain anything from random sources, where
>> building happens outside of Isar. It also allows to create meta-packages
>> that contain nothing but pull in dependencies, once all our packets come
>> in via multistrap that will come in handy.
>> For rootfs configuration you would use post- and pre- scripts just like
>> regular debian packages do.
>>
> 
> Good idea.
> 
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   meta/classes/dpkg-custom.bbclass | 57
>> ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 meta/classes/dpkg-custom.bbclass
>>
>> diff --git a/meta/classes/dpkg-custom.bbclass
>> b/meta/classes/dpkg-custom.bbclass
>> new file mode 100644
>> index 0000000..e4e743f
>> --- /dev/null
>> +++ b/meta/classes/dpkg-custom.bbclass
>> @@ -0,0 +1,57 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2017 Siemens AG
>> +
>> +inherit dpkg
>> +
>> +DEBIAN_DEPENDS ?= ""
>> +MAINTAINER ?= "FIXME Unknown maintainer"
>> +
>> +D = "${WORKDIR}/image/"
>> +
>> +# Populate folder that will be picked up as package
>> +# TODO this should be called 'do_install'
>> +do_populate_package() {
>> +    bbnote "Put your files for this package in ${D}"
>> +}
>> +
>> +addtask populate_package after do_unpack before do_deb_package_prepare
>> +
>> +# so we can put hooks etc. in there already
>> +do_populate_package[dirs] = "${D}/DEBIAN"
>> +
>> +do_deb_package_prepare() {
>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
>> +        Package: ${PN}
>> +        Architecture: `dpkg --print-architecture`
>> +        Section: misc
>> +        Priority: optional
>> +        Maintainer: ${MAINTAINER}
>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
>> +        Version: ${PV}+isar
>> +        Description: ${DESCRIPTION}
>> +    __EOF__
>> +    for t in pre post
>> +    do
>> +        for a in inst rm
>> +        do
>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
>> +        done
>> +    done
>> +}
>> +
>> +addtask deb_package_prepare after do_populate_package before
>> do_deb_package_conffiles
>> +
>> +do_deb_package_conffiles() {
>> +    CONFFILES=${D}/DEBIAN/conffiles
>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
>> $CONFFILES
>> +    test -s $CONFFILES || rm $CONFFILES
>> +}
>> +
>> +addtask deb_package_conffiles after do_deb_package_prepare before
>> do_deb_package
>> +
>> +do_deb_package() {
>> +    sudo chown -R root:root ${D}/DEBIAN/
>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
>> +}
>> +
>> +addtask deb_package after do_deb_package_conffiles before do_install
>>
> 
> If I got this correctly, the difference from dpkg-debian flow is that
> you generate 'debian' folder on the fly. So:
> 
> 1. Why we need another do_deb_package, while do_build from dpkg class
> does the same?
> 
> 2. Can we just add optional task to generic pipeline which generates
> debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian
> ]; then blablabla.
> 

I'm not too deep into the details here, but the general considerations
should be here:

- Do both classes share enough code to unify them?
- Is it easy enough for the user to specify and the class to derive the
  mode of operation when there is only one class?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Alexander Smirnov]

On 09/04/2017 07:03 PM, Jan Kiszka wrote:
> On 2017-09-04 17:14, Alexander Smirnov wrote:
>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>> Issues:
>>> unpack and fetch can only be used if you inherit dpkg, they should be
>>> available to others i.e. classes and images
>>
>> For me it's ok to move fetch to base class. Unpack has dependency from
>> dpkg class.
>>
> 
> What is that dependency?
> 

Commented in previous patch.

-- 
With best regards,
Alexander Smirnov

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Jan Kiszka]

On 2017-09-04 18:10, Alexander Smirnov wrote:
> On 09/04/2017 07:03 PM, Jan Kiszka wrote:
>> On 2017-09-04 17:14, Alexander Smirnov wrote:
>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>> Issues:
>>>> unpack and fetch can only be used if you inherit dpkg, they should be
>>>> available to others i.e. classes and images
>>>
>>> For me it's ok to move fetch to base class. Unpack has dependency from
>>> dpkg class.
>>>
>>
>> What is that dependency?
>>
> 
> Commented in previous patch.

The previous patch makes it generic, thus independent of dpkg. I cannot
follow you, sorry.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Alexander Smirnov]

On 09/04/2017 07:08 PM, Jan Kiszka wrote:
> On 2017-09-04 17:36, Alexander Smirnov wrote:
>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>> Issues:
>>> 1. full customizations of the images is hard to impossible to
>>> realize in a layer without touching Isar
>>> 1.1. there is no easy way to just copy a file into the image
>>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>>> done in a layer, there is no way too hook into multistrap or the
>>> configure-script
>>>
>>> Change:
>>> Introduce a class that lets users create custom debian packages on the
>>> fly, without having to create a /debian directory and actually building.
>>> That allows you to pull in debian-dependencies, you could have a
>>> package that has no content and is just there to install what you need
>>> for a feature of your product.
>>> Using package hooks (preinst, postinst ..) you can configure pretty much
>>> all you want when installing the package.
>>> The package can contain actual payload as well, basically any files that
>>> come from "somewhere else". Say binary data like wallpapers, sound files
>>> or application binaries.
>>>
>>> Impact:
>>> This patch addresses the metioned issue in a way that uses debian
>>> mechanism. All the customizations will enjoy features like
>>> - collission protection (multiple packages providing the same file)
>>> - config file protection
>>> - versioning and the ability to deploy your changes in an updateable way
>>>
>>> This patch introduces a major new feature to Isar.
>>>
>>> This class introduces a new class for building debian packages on the
>>> fly. They can basically contain anything from random sources, where
>>> building happens outside of Isar. It also allows to create meta-packages
>>> that contain nothing but pull in dependencies, once all our packets come
>>> in via multistrap that will come in handy.
>>> For rootfs configuration you would use post- and pre- scripts just like
>>> regular debian packages do.
>>>
>>
>> Good idea.
>>
>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>> ---
>>>    meta/classes/dpkg-custom.bbclass | 57
>>> ++++++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 57 insertions(+)
>>>    create mode 100644 meta/classes/dpkg-custom.bbclass
>>>
>>> diff --git a/meta/classes/dpkg-custom.bbclass
>>> b/meta/classes/dpkg-custom.bbclass
>>> new file mode 100644
>>> index 0000000..e4e743f
>>> --- /dev/null
>>> +++ b/meta/classes/dpkg-custom.bbclass
>>> @@ -0,0 +1,57 @@
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2017 Siemens AG
>>> +
>>> +inherit dpkg
>>> +
>>> +DEBIAN_DEPENDS ?= ""
>>> +MAINTAINER ?= "FIXME Unknown maintainer"
>>> +
>>> +D = "${WORKDIR}/image/"
>>> +
>>> +# Populate folder that will be picked up as package
>>> +# TODO this should be called 'do_install'
>>> +do_populate_package() {
>>> +    bbnote "Put your files for this package in ${D}"
>>> +}
>>> +
>>> +addtask populate_package after do_unpack before do_deb_package_prepare
>>> +
>>> +# so we can put hooks etc. in there already
>>> +do_populate_package[dirs] = "${D}/DEBIAN"
>>> +
>>> +do_deb_package_prepare() {
>>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
>>> +        Package: ${PN}
>>> +        Architecture: `dpkg --print-architecture`
>>> +        Section: misc
>>> +        Priority: optional
>>> +        Maintainer: ${MAINTAINER}
>>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
>>> +        Version: ${PV}+isar
>>> +        Description: ${DESCRIPTION}
>>> +    __EOF__
>>> +    for t in pre post
>>> +    do
>>> +        for a in inst rm
>>> +        do
>>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
>>> +        done
>>> +    done
>>> +}
>>> +
>>> +addtask deb_package_prepare after do_populate_package before
>>> do_deb_package_conffiles
>>> +
>>> +do_deb_package_conffiles() {
>>> +    CONFFILES=${D}/DEBIAN/conffiles
>>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
>>> $CONFFILES
>>> +    test -s $CONFFILES || rm $CONFFILES
>>> +}
>>> +
>>> +addtask deb_package_conffiles after do_deb_package_prepare before
>>> do_deb_package
>>> +
>>> +do_deb_package() {
>>> +    sudo chown -R root:root ${D}/DEBIAN/
>>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
>>> +}
>>> +
>>> +addtask deb_package after do_deb_package_conffiles before do_install
>>>
>>
>> If I got this correctly, the difference from dpkg-debian flow is that
>> you generate 'debian' folder on the fly. So:
>>
>> 1. Why we need another do_deb_package, while do_build from dpkg class
>> does the same?
>>
>> 2. Can we just add optional task to generic pipeline which generates
>> debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian
>> ]; then blablabla.
>>
> 
> I'm not too deep into the details here, but the general considerations
> should be here:
> 
> - Do both classes share enough code to unify them?

 From my POV - yes.

> - Is it easy enough for the user to specify and the class to derive the
>    mode of operation when there is only one class?

That's, for example, how autotools work in Yocto. If you need to run 
autoconf/configure in your recipe, you just:

inherit autotools

and respective tasks are added to your pipeline. So in current case, 
user could inherit some 'debian-gen' class, which adds task that 
generates debian folder for your project.

In my opinion, the pipeline should stay the same, but some tasks could 
be included/excluded on demand.

Alex

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Alexander Smirnov]

On 09/04/2017 07:11 PM, Jan Kiszka wrote:
> On 2017-09-04 18:10, Alexander Smirnov wrote:
>> On 09/04/2017 07:03 PM, Jan Kiszka wrote:
>>> On 2017-09-04 17:14, Alexander Smirnov wrote:
>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>>> Issues:
>>>>> unpack and fetch can only be used if you inherit dpkg, they should be
>>>>> available to others i.e. classes and images
>>>>
>>>> For me it's ok to move fetch to base class. Unpack has dependency from
>>>> dpkg class.
>>>>
>>>
>>> What is that dependency?
>>>
>>
>> Commented in previous patch.
> 
> The previous patch makes it generic, thus independent of dpkg. I cannot
> follow you, sorry.

Ok. Currently task unpack depends on dpkg class, because it uses 
BUILDROOT variable which is defined locally in dpkg.class. So the 
current task is also generic, and can be moved to base class but with 
BUILDROOT variable definition. Or BUILDROOT could be defined in 
bitbake.conf.

My concern about previous patch that it doesn't make task more generic - 
the content stays the same, but it changes the WORKDIR, what affects 
whole Isar design.

Alex

Re: [PATCH 2/6] meta: Move 'do_fetch' and 'do_unpack' to base-class

[Jan Kiszka]

On 2017-09-04 18:39, Alexander Smirnov wrote:
> On 09/04/2017 07:11 PM, Jan Kiszka wrote:
>> On 2017-09-04 18:10, Alexander Smirnov wrote:
>>> On 09/04/2017 07:03 PM, Jan Kiszka wrote:
>>>> On 2017-09-04 17:14, Alexander Smirnov wrote:
>>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>>>> Issues:
>>>>>> unpack and fetch can only be used if you inherit dpkg, they should be
>>>>>> available to others i.e. classes and images
>>>>>
>>>>> For me it's ok to move fetch to base class. Unpack has dependency from
>>>>> dpkg class.
>>>>>
>>>>
>>>> What is that dependency?
>>>>
>>>
>>> Commented in previous patch.
>>
>> The previous patch makes it generic, thus independent of dpkg. I cannot
>> follow you, sorry.
> 
> Ok. Currently task unpack depends on dpkg class, because it uses
> BUILDROOT variable which is defined locally in dpkg.class. So the
> current task is also generic, and can be moved to base class but with
> BUILDROOT variable definition. Or BUILDROOT could be defined in
> bitbake.conf.
> 
> My concern about previous patch that it doesn't make task more generic -
> the content stays the same, but it changes the WORKDIR, what affects
> whole Isar design.

Your comment on patch 1 was delayed in my inbox, that destroyed the
context for me.

Well if both WORKDIR AND BUILDROOT are well defined variables, just make
them omnipresent and be fine, agreed.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Jan Kiszka]

On 2017-09-04 17:11, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issue:
>> The unpacker in dpkg is not generic since it does not unpack to
>> WORKDIR, so it can not be pulled into a more generic class to make it
>> available for other users.
>>
> 
> As I already wrote to mail-list, WORKDIR and BUILDROOT are different
> folders, and there was the reason to do so.
> 
> In Yocto, WORKDIR is the place which keeps the following artifacts:
> 
>  - Source code tree
>  - Patches, configs
>  - Binaries
>  - Yocto meta information (logs and build commands)
> 
> WORKDIR is usually something like: ./tmp/work/gcc-arch/package-name
> 
> Due to Yocto performs cross-compilation, it is able to keep source core
> out of cross filesystem. But this doesn't work for Isar, which uses
> 'native' compilation in chroot. That's why BUILDROOT variable was
> introduced.
> 
> So in Isar, WORKDIR stays the same and keeps logs, build commands,
> patches and other stuff (for example configs and scripts for
> buildchroot). The BUILDROOT is the path to dedicated buildchroot where
> to unpack sources and build them.

That should probably go into your design doc, even more as it deviates
from OE practice.

> 
> If I build multiconfig for N targets with this patch, I will have N
> distributed locations with bitbake meta information for the same package.
> 
> If I cleanall buildchroot, I will lose all the bitbake meta information
> for current distro-arch.
> 
> So at the moment I don't see any benefit of this change.
> 
> Also it's not clear, what do you mean by 'not generic', this unpack
> implementation designed for building deb packages, and at the moment no
> other users are expected.
> 

I suppose many misunderstandings come from the fuzziness that
multiconfig introduces. That requires documentation, even more as it is
easy to break and then fairly hard to diagnose.

How does OE deal with multiconfig requirements on recipes? Or don't they
have the complication?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Jan Kiszka]

On 2017-09-04 18:30, Alexander Smirnov wrote:
> 
> 
> On 09/04/2017 07:08 PM, Jan Kiszka wrote:
>> On 2017-09-04 17:36, Alexander Smirnov wrote:
>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>> Issues:
>>>> 1. full customizations of the images is hard to impossible to
>>>> realize in a layer without touching Isar
>>>> 1.1. there is no easy way to just copy a file into the image
>>>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>>>> done in a layer, there is no way too hook into multistrap or the
>>>> configure-script
>>>>
>>>> Change:
>>>> Introduce a class that lets users create custom debian packages on the
>>>> fly, without having to create a /debian directory and actually
>>>> building.
>>>> That allows you to pull in debian-dependencies, you could have a
>>>> package that has no content and is just there to install what you need
>>>> for a feature of your product.
>>>> Using package hooks (preinst, postinst ..) you can configure pretty
>>>> much
>>>> all you want when installing the package.
>>>> The package can contain actual payload as well, basically any files
>>>> that
>>>> come from "somewhere else". Say binary data like wallpapers, sound
>>>> files
>>>> or application binaries.
>>>>
>>>> Impact:
>>>> This patch addresses the metioned issue in a way that uses debian
>>>> mechanism. All the customizations will enjoy features like
>>>> - collission protection (multiple packages providing the same file)
>>>> - config file protection
>>>> - versioning and the ability to deploy your changes in an updateable
>>>> way
>>>>
>>>> This patch introduces a major new feature to Isar.
>>>>
>>>> This class introduces a new class for building debian packages on the
>>>> fly. They can basically contain anything from random sources, where
>>>> building happens outside of Isar. It also allows to create
>>>> meta-packages
>>>> that contain nothing but pull in dependencies, once all our packets
>>>> come
>>>> in via multistrap that will come in handy.
>>>> For rootfs configuration you would use post- and pre- scripts just like
>>>> regular debian packages do.
>>>>
>>>
>>> Good idea.
>>>
>>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>>> ---
>>>>    meta/classes/dpkg-custom.bbclass | 57
>>>> ++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 57 insertions(+)
>>>>    create mode 100644 meta/classes/dpkg-custom.bbclass
>>>>
>>>> diff --git a/meta/classes/dpkg-custom.bbclass
>>>> b/meta/classes/dpkg-custom.bbclass
>>>> new file mode 100644
>>>> index 0000000..e4e743f
>>>> --- /dev/null
>>>> +++ b/meta/classes/dpkg-custom.bbclass
>>>> @@ -0,0 +1,57 @@
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (C) 2017 Siemens AG
>>>> +
>>>> +inherit dpkg
>>>> +
>>>> +DEBIAN_DEPENDS ?= ""
>>>> +MAINTAINER ?= "FIXME Unknown maintainer"
>>>> +
>>>> +D = "${WORKDIR}/image/"
>>>> +
>>>> +# Populate folder that will be picked up as package
>>>> +# TODO this should be called 'do_install'
>>>> +do_populate_package() {
>>>> +    bbnote "Put your files for this package in ${D}"
>>>> +}
>>>> +
>>>> +addtask populate_package after do_unpack before do_deb_package_prepare
>>>> +
>>>> +# so we can put hooks etc. in there already
>>>> +do_populate_package[dirs] = "${D}/DEBIAN"
>>>> +
>>>> +do_deb_package_prepare() {
>>>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
>>>> +        Package: ${PN}
>>>> +        Architecture: `dpkg --print-architecture`
>>>> +        Section: misc
>>>> +        Priority: optional
>>>> +        Maintainer: ${MAINTAINER}
>>>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
>>>> +        Version: ${PV}+isar
>>>> +        Description: ${DESCRIPTION}
>>>> +    __EOF__
>>>> +    for t in pre post
>>>> +    do
>>>> +        for a in inst rm
>>>> +        do
>>>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
>>>> +        done
>>>> +    done
>>>> +}
>>>> +
>>>> +addtask deb_package_prepare after do_populate_package before
>>>> do_deb_package_conffiles
>>>> +
>>>> +do_deb_package_conffiles() {
>>>> +    CONFFILES=${D}/DEBIAN/conffiles
>>>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
>>>> $CONFFILES
>>>> +    test -s $CONFFILES || rm $CONFFILES
>>>> +}
>>>> +
>>>> +addtask deb_package_conffiles after do_deb_package_prepare before
>>>> do_deb_package
>>>> +
>>>> +do_deb_package() {
>>>> +    sudo chown -R root:root ${D}/DEBIAN/
>>>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
>>>> +}
>>>> +
>>>> +addtask deb_package after do_deb_package_conffiles before do_install
>>>>
>>>
>>> If I got this correctly, the difference from dpkg-debian flow is that
>>> you generate 'debian' folder on the fly. So:
>>>
>>> 1. Why we need another do_deb_package, while do_build from dpkg class
>>> does the same?
>>>
>>> 2. Can we just add optional task to generic pipeline which generates
>>> debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian
>>> ]; then blablabla.
>>>
>>
>> I'm not too deep into the details here, but the general considerations
>> should be here:
>>
>> - Do both classes share enough code to unify them?
> 
> From my POV - yes.
> 
>> - Is it easy enough for the user to specify and the class to derive the
>>    mode of operation when there is only one class?
> 
> That's, for example, how autotools work in Yocto. If you need to run
> autoconf/configure in your recipe, you just:
> 
> inherit autotools
> 
> and respective tasks are added to your pipeline. So in current case,
> user could inherit some 'debian-gen' class, which adds task that
> generates debian folder for your project.
> 
> In my opinion, the pipeline should stay the same, but some tasks could
> be included/excluded on demand.

IOW: keep dpkg.bbclass, derive dpgk-gen.bbclass from it in a way that it
overwrites/expands the former to generate the meta data. I suppose one
has to look at the resulting code to validate if that is beneficial, but
the reuse may indeed increase once we start building from source with
the generator approach as well - something Henning was already talking
about.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Claudius Heine]

Hi,

On 09/04/2017 05:11 PM, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issue:
>> The unpacker in dpkg is not generic since it does not unpack to
>> WORKDIR, so it can not be pulled into a more generic class to make it
>> available for other users.
>>
> 
> As I already wrote to mail-list, WORKDIR and BUILDROOT are different 
> folders, and there was the reason to do so.

Yes we (Henning and I) get that. That is the reason why the patch 
includes the line:

+WORKDIR_task-unpack = "${BUILDROOT}"

That means that only for the unpack task in the dpkg.bbclass the 
variable WORKDIR is set to the BUILDROOT variable. This makes the unpack 
step more generic so that it can be moved to the base class. Essentially 
removing the dependency of the unpack task to the dpkg class.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Alexander Smirnov]

Hi,

> 
> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:
>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>> Issue:
>>> The unpacker in dpkg is not generic since it does not unpack to
>>> WORKDIR, so it can not be pulled into a more generic class to make it
>>> available for other users.
>>>
>>
>> As I already wrote to mail-list, WORKDIR and BUILDROOT are different 
>> folders, and there was the reason to do so.
> 
> Yes we (Henning and I) get that. That is the reason why the patch 
> includes the line:
> 
> +WORKDIR_task-unpack = "${BUILDROOT}"
> 
> That means that only for the unpack task in the dpkg.bbclass the 
> variable WORKDIR is set to the BUILDROOT variable. This makes the unpack 
> step more generic so that it can be moved to the base class. Essentially 
> removing the dependency of the unpack task to the dpkg class.

Does this mean that "log.do_unpack" and "run.do_unpack" will be stored 
in buildchroot? Let me check this.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Alexander Smirnov]

Hi,

>> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:
>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>> Issue:
>>>> The unpacker in dpkg is not generic since it does not unpack to
>>>> WORKDIR, so it can not be pulled into a more generic class to make it
>>>> available for other users.
>>>>
>>>
>>> As I already wrote to mail-list, WORKDIR and BUILDROOT are different 
>>> folders, and there was the reason to do so.
>>
>> Yes we (Henning and I) get that. That is the reason why the patch 
>> includes the line:
>>
>> +WORKDIR_task-unpack = "${BUILDROOT}"
>>
>> That means that only for the unpack task in the dpkg.bbclass the 
>> variable WORKDIR is set to the BUILDROOT variable. This makes the 
>> unpack step more generic so that it can be moved to the base class. 
>> Essentially removing the dependency of the unpack task to the dpkg class.
> 
> Does this mean that "log.do_unpack" and "run.do_unpack" will be stored 
> in buildchroot? Let me check this.
> 

Yes it does:

ls 
isar/build/tmp/work/buildchroot/debian-wheezy-armhf/rootfs/home/builder/hello/temp/

log.do_unpack
log.do_unpack.19557
log.task_order
run.do_unpack
run.do_unpack.19557

While it's assumed that they should be stored in: 
isar/build/tmp/work/hello-0.1+g7f35942-1-r0/temp/

-- 
With best regards,
Alexander Smirnov

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Claudius Heine]

Hi,

On 09/05/2017 09:37 AM, Alexander Smirnov wrote:
> Hi,
> 
>>> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:
>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>>> Issue:
>>>>> The unpacker in dpkg is not generic since it does not unpack to
>>>>> WORKDIR, so it can not be pulled into a more generic class to make it
>>>>> available for other users.
>>>>>
>>>>
>>>> As I already wrote to mail-list, WORKDIR and BUILDROOT are different 
>>>> folders, and there was the reason to do so.
>>>
>>> Yes we (Henning and I) get that. That is the reason why the patch 
>>> includes the line:
>>>
>>> +WORKDIR_task-unpack = "${BUILDROOT}"
>>>
>>> That means that only for the unpack task in the dpkg.bbclass the 
>>> variable WORKDIR is set to the BUILDROOT variable. This makes the 
>>> unpack step more generic so that it can be moved to the base class. 
>>> Essentially removing the dependency of the unpack task to the dpkg 
>>> class.
>>
>> Does this mean that "log.do_unpack" and "run.do_unpack" will be stored 
>> in buildchroot? Let me check this.
>>
> 
> Yes it does:
> 
> ls 
> isar/build/tmp/work/buildchroot/debian-wheezy-armhf/rootfs/home/builder/hello/temp/ 
> 
> 
> log.do_unpack
> log.do_unpack.19557
> log.task_order
> run.do_unpack
> run.do_unpack.19557
> 
> While it's assumed that they should be stored in: 
> isar/build/tmp/work/hello-0.1+g7f35942-1-r0/temp/

Ok then the solution would be to overwrite all there variables that use 
the WORKDIR variable as well (could be done automatically via a python 
script), or to use for example a "UNPACKDIR" variable, that if set, is 
used instead of the "WORKDIR" variable. The first approach might be more 
compatible with OE, the second approach might be easier to do.

Claudius


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 6/6] recipes-app/example-custom: add example on how to use dpkg-custom

[Henning Schild]

Am Mon, 4 Sep 2017 18:40:20 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/30/2017 10:03 PM, Henning Schild wrote:
> > Issue:
> > It might not be clear to a user how to make use of the class
> > dpkg-custom.
> > 
> > Change:
> > Add an example that demonstrates some features.
> > 
> > Impact:
> > This patch does not change the behaviour of Isar. It addresses the
> > issue partially, documentation would also be useful.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >   .../example-custom/example-custom_0.1.bb           | 35
> > ++++++++++++++++++++++
> > meta-isar/recipes-app/example-custom/files/README  |  1
> > + .../recipes-app/example-custom/files/postinst      | 16
> > ++++++++++ 3 files changed, 52 insertions(+) create mode 100644
> > meta-isar/recipes-app/example-custom/example-custom_0.1.bb create
> > mode 100644 meta-isar/recipes-app/example-custom/files/README
> > create mode 100644
> > meta-isar/recipes-app/example-custom/files/postinst
> > 
> > diff --git
> > a/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
> > b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb new
> > file mode 100644 index 0000000..efa813e --- /dev/null
> > +++ b/meta-isar/recipes-app/example-custom/example-custom_0.1.bb
> > @@ -0,0 +1,35 @@
> > +# Sample application using dpkg-custom, which turns a folder
> > (${D}) of +# files into a .deb
> > +#
> > +# This software is a part of ISAR.
> > +
> > +DESCRIPTION = "Sample application for ISAR"
> > +MAINTAINER = "Your name here <you@domain.com>"
> > +DEBIAN_DEPENDS = "apt"
> > +
> > +SRC_URI = "file://README \
> > +	   file://postinst"
> > +
> > +inherit dpkg-custom
> > +
> > +do_populate_package() {
> > +	bbnote "Creating ${PN} binary"
> > +	echo "#!/bin/sh" > ${WORKDIR}/${PN}
> > +	echo "echo Hello ISAR! ${PN}_${PV}" >> ${WORKDIR}/${PN}
> > +
> > +	bbnote "Putting ${PN} into package"
> > +	install -v -d ${D}/usr/local/bin/
> > +	install -v -m 755 ${WORKDIR}/${PN} ${D}/usr/local/bin/${PN}
> > +
> > +	bbnote "Now copy ${FILESDIR}/README into package"
> > +	install -v -d ${D}/usr/local/doc/
> > +	install -v -m 644 ${WORKDIR}/README
> > ${D}/usr/local/doc/README-${P} +
> > +	bbnote "Now for a debian hook, see dpkg-deb"
> > +	install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
> > +
> > +	bbnote "Now for a fake config file"
> > +	echo "# empty config file" > ${WORKDIR}/${PN}.conf
> > +	install -v -d ${D}/usr/local/etc/
> > +	install -v -m 644 ${WORKDIR}/${PN}.conf
> > ${D}/usr/local/etc/${PN}.conf +}
> > diff --git a/meta-isar/recipes-app/example-custom/files/README
> > b/meta-isar/recipes-app/example-custom/files/README new file mode
> > 100644 index 0000000..6e2ce0f
> > --- /dev/null
> > +++ b/meta-isar/recipes-app/example-custom/files/README
> > @@ -0,0 +1 @@
> > +This is an example file that we get from FILESDIR in recipe.
> > diff --git a/meta-isar/recipes-app/example-custom/files/postinst
> > b/meta-isar/recipes-app/example-custom/files/postinst new file mode
> > 100644 index 0000000..a28afa3
> > --- /dev/null
> > +++ b/meta-isar/recipes-app/example-custom/files/postinst
> > @@ -0,0 +1,16 @@
> > +#!/bin/sh
> > +
> > +set -e
> > +
> > +if ! getent group isar >/dev/null; then
> > +	addgroup --quiet --system isar
> > +fi
> > +
> > +if ! getent passwd isar >/dev/null; then
> > +	useradd --system --gid isar --create-home \
> > +		--home /var/lib/isar --no-user-group \
> > +		--comment "My isar user" \
> > +		isar
> > +fi
> > +
> > +chown -R isar:isar /var/lib/isar
> >   
> 
> That's a good demo scenario. But probably it makes sense to
> demonstrate it on real use case, for example set default password for
> root. This anyway should be done, probably it's a good time :-)

Well unfortunately the root password is special. It is special because
everyone probably wants to set it, and i do not want people to start
abusing that example to do so.
And it is special because the configure-script sets it as well and the
question "how and when do custom packages get into the rootfs" has not
been decided yet. Once we have answered that question i would be happy
to provide patches that move all of the configure-script into hooks.

Henning

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Mon, 4 Sep 2017 18:36:23 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 08/30/2017 10:03 PM, Henning Schild wrote:
> > Issues:
> > 1. full customizations of the images is hard to impossible to
> > realize in a layer without touching Isar
> > 1.1. there is no easy way to just copy a file into the image
> > 1.2. configuration (passwords, groups, cfg-files changes) can not be
> > done in a layer, there is no way too hook into multistrap or the
> > configure-script
> > 
> > Change:
> > Introduce a class that lets users create custom debian packages on
> > the fly, without having to create a /debian directory and actually
> > building. That allows you to pull in debian-dependencies, you could
> > have a package that has no content and is just there to install
> > what you need for a feature of your product.
> > Using package hooks (preinst, postinst ..) you can configure pretty
> > much all you want when installing the package.
> > The package can contain actual payload as well, basically any files
> > that come from "somewhere else". Say binary data like wallpapers,
> > sound files or application binaries.
> > 
> > Impact:
> > This patch addresses the metioned issue in a way that uses debian
> > mechanism. All the customizations will enjoy features like
> > - collission protection (multiple packages providing the same file)
> > - config file protection
> > - versioning and the ability to deploy your changes in an
> > updateable way
> > 
> > This patch introduces a major new feature to Isar.
> > 
> > This class introduces a new class for building debian packages on
> > the fly. They can basically contain anything from random sources,
> > where building happens outside of Isar. It also allows to create
> > meta-packages that contain nothing but pull in dependencies, once
> > all our packets come in via multistrap that will come in handy.
> > For rootfs configuration you would use post- and pre- scripts just
> > like regular debian packages do.
> >   
> 
> Good idea.
> 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >   meta/classes/dpkg-custom.bbclass | 57
> > ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57
> > insertions(+) create mode 100644 meta/classes/dpkg-custom.bbclass
> > 
> > diff --git a/meta/classes/dpkg-custom.bbclass
> > b/meta/classes/dpkg-custom.bbclass new file mode 100644
> > index 0000000..e4e743f
> > --- /dev/null
> > +++ b/meta/classes/dpkg-custom.bbclass
> > @@ -0,0 +1,57 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2017 Siemens AG
> > +
> > +inherit dpkg
> > +
> > +DEBIAN_DEPENDS ?= ""
> > +MAINTAINER ?= "FIXME Unknown maintainer"
> > +
> > +D = "${WORKDIR}/image/"
> > +
> > +# Populate folder that will be picked up as package
> > +# TODO this should be called 'do_install'
> > +do_populate_package() {
> > +	bbnote "Put your files for this package in ${D}"
> > +}
> > +
> > +addtask populate_package after do_unpack before
> > do_deb_package_prepare +
> > +# so we can put hooks etc. in there already
> > +do_populate_package[dirs] = "${D}/DEBIAN"
> > +
> > +do_deb_package_prepare() {
> > +	cat<<-__EOF__ > ${D}/DEBIAN/control
> > +		Package: ${PN}
> > +		Architecture: `dpkg --print-architecture`
> > +		Section: misc
> > +		Priority: optional
> > +		Maintainer: ${MAINTAINER}
> > +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]'
> > ','`
> > +		Version: ${PV}+isar
> > +		Description: ${DESCRIPTION}
> > +	__EOF__
> > +	for t in pre post
> > +	do
> > +		for a in inst rm
> > +		do
> > +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
> > +		done
> > +	done
> > +}
> > +
> > +addtask deb_package_prepare after do_populate_package before
> > do_deb_package_conffiles +
> > +do_deb_package_conffiles() {
> > +	CONFFILES=${D}/DEBIAN/conffiles
> > +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
> > $CONFFILES
> > +	test -s $CONFFILES || rm $CONFFILES
> > +}
> > +
> > +addtask deb_package_conffiles after do_deb_package_prepare before
> > do_deb_package +
> > +do_deb_package() {
> > +	sudo chown -R root:root ${D}/DEBIAN/
> > +	sudo dpkg-deb --build ${D} ${WORKDIR}
> > +}
> > +
> > +addtask deb_package after do_deb_package_conffiles before
> > do_install 
> 
> If I got this correctly, the difference from dpkg-debian flow is that 
> you generate 'debian' folder on the fly. So:

Well it is a capital "DEBIAN" folder that contains the package metadata
and no build instructions like 'debian'.

> 1. Why we need another do_deb_package, while do_build from dpkg class 
> does the same?

do_build does not do the same. It eventually calls 'dpkg-buildpackage'
which does "configure; make; make install; make deb"

Using the same language this step is just "make deb" and it uses the
debian tool 'dpkg-deb'.

Custom steps for "configure; make;" could be added in the future. "make
install" is do_populate_package.

> 2. Can we just add optional task to generic pipeline which generates 
> debian folder? Or make it as a part of do_build: if [ ! -d
> ${D}/debian ]; then blablabla.

That would be adding support for 'debian'-izing a source package.

This patch is more about packaging binaries and creating meta-packages.

Henning

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Mon, 4 Sep 2017 19:30:43 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 09/04/2017 07:08 PM, Jan Kiszka wrote:
> > On 2017-09-04 17:36, Alexander Smirnov wrote:  
> >> On 08/30/2017 10:03 PM, Henning Schild wrote:  
> >>> Issues:
> >>> 1. full customizations of the images is hard to impossible to
> >>> realize in a layer without touching Isar
> >>> 1.1. there is no easy way to just copy a file into the image
> >>> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >>> be done in a layer, there is no way too hook into multistrap or
> >>> the configure-script
> >>>
> >>> Change:
> >>> Introduce a class that lets users create custom debian packages
> >>> on the fly, without having to create a /debian directory and
> >>> actually building. That allows you to pull in
> >>> debian-dependencies, you could have a package that has no content
> >>> and is just there to install what you need for a feature of your
> >>> product. Using package hooks (preinst, postinst ..) you can
> >>> configure pretty much all you want when installing the package.
> >>> The package can contain actual payload as well, basically any
> >>> files that come from "somewhere else". Say binary data like
> >>> wallpapers, sound files or application binaries.
> >>>
> >>> Impact:
> >>> This patch addresses the metioned issue in a way that uses debian
> >>> mechanism. All the customizations will enjoy features like
> >>> - collission protection (multiple packages providing the same
> >>> file)
> >>> - config file protection
> >>> - versioning and the ability to deploy your changes in an
> >>> updateable way
> >>>
> >>> This patch introduces a major new feature to Isar.
> >>>
> >>> This class introduces a new class for building debian packages on
> >>> the fly. They can basically contain anything from random sources,
> >>> where building happens outside of Isar. It also allows to create
> >>> meta-packages that contain nothing but pull in dependencies, once
> >>> all our packets come in via multistrap that will come in handy.
> >>> For rootfs configuration you would use post- and pre- scripts
> >>> just like regular debian packages do.
> >>>  
> >>
> >> Good idea.
> >>  
> >>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >>> ---
> >>>    meta/classes/dpkg-custom.bbclass | 57
> >>> ++++++++++++++++++++++++++++++++++++++++
> >>>    1 file changed, 57 insertions(+)
> >>>    create mode 100644 meta/classes/dpkg-custom.bbclass
> >>>
> >>> diff --git a/meta/classes/dpkg-custom.bbclass
> >>> b/meta/classes/dpkg-custom.bbclass
> >>> new file mode 100644
> >>> index 0000000..e4e743f
> >>> --- /dev/null
> >>> +++ b/meta/classes/dpkg-custom.bbclass
> >>> @@ -0,0 +1,57 @@
> >>> +# This software is a part of ISAR.
> >>> +# Copyright (C) 2017 Siemens AG
> >>> +
> >>> +inherit dpkg
> >>> +
> >>> +DEBIAN_DEPENDS ?= ""
> >>> +MAINTAINER ?= "FIXME Unknown maintainer"
> >>> +
> >>> +D = "${WORKDIR}/image/"
> >>> +
> >>> +# Populate folder that will be picked up as package
> >>> +# TODO this should be called 'do_install'
> >>> +do_populate_package() {
> >>> +    bbnote "Put your files for this package in ${D}"
> >>> +}
> >>> +
> >>> +addtask populate_package after do_unpack before
> >>> do_deb_package_prepare +
> >>> +# so we can put hooks etc. in there already
> >>> +do_populate_package[dirs] = "${D}/DEBIAN"
> >>> +
> >>> +do_deb_package_prepare() {
> >>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
> >>> +        Package: ${PN}
> >>> +        Architecture: `dpkg --print-architecture`
> >>> +        Section: misc
> >>> +        Priority: optional
> >>> +        Maintainer: ${MAINTAINER}
> >>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
> >>> +        Version: ${PV}+isar
> >>> +        Description: ${DESCRIPTION}
> >>> +    __EOF__
> >>> +    for t in pre post
> >>> +    do
> >>> +        for a in inst rm
> >>> +        do
> >>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
> >>> +        done
> >>> +    done
> >>> +}
> >>> +
> >>> +addtask deb_package_prepare after do_populate_package before
> >>> do_deb_package_conffiles
> >>> +
> >>> +do_deb_package_conffiles() {
> >>> +    CONFFILES=${D}/DEBIAN/conffiles
> >>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
> >>> $CONFFILES
> >>> +    test -s $CONFFILES || rm $CONFFILES
> >>> +}
> >>> +
> >>> +addtask deb_package_conffiles after do_deb_package_prepare before
> >>> do_deb_package
> >>> +
> >>> +do_deb_package() {
> >>> +    sudo chown -R root:root ${D}/DEBIAN/
> >>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
> >>> +}
> >>> +
> >>> +addtask deb_package after do_deb_package_conffiles before
> >>> do_install 
> >>
> >> If I got this correctly, the difference from dpkg-debian flow is
> >> that you generate 'debian' folder on the fly. So:
> >>
> >> 1. Why we need another do_deb_package, while do_build from dpkg
> >> class does the same?
> >>
> >> 2. Can we just add optional task to generic pipeline which
> >> generates debian folder? Or make it as a part of do_build: if [ !
> >> -d ${D}/debian ]; then blablabla.
> >>  
> > 
> > I'm not too deep into the details here, but the general
> > considerations should be here:
> > 
> > - Do both classes share enough code to unify them?  
> 
>  From my POV - yes.

I do not think so. Because it is not about 'debian'-izing the sources
before do_build but rather replacing do_build with a series of tasks
that use other debian tools. So it is an orthogonal workflow from the
source to the debian package.

Henning

> 
> > - Is it easy enough for the user to specify and the class to derive
> > the mode of operation when there is only one class?  
> 
> That's, for example, how autotools work in Yocto. If you need to run 
> autoconf/configure in your recipe, you just:
> 
> inherit autotools
> 
> and respective tasks are added to your pipeline. So in current case, 
> user could inherit some 'debian-gen' class, which adds task that 
> generates debian folder for your project.
> 
> In my opinion, the pipeline should stay the same, but some tasks
> could be included/excluded on demand.
> 
> Alex
> 

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Henning Schild]

Am Tue, 5 Sep 2017 10:37:48 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> >> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:  
> >>> On 08/30/2017 10:03 PM, Henning Schild wrote:  
> >>>> Issue:
> >>>> The unpacker in dpkg is not generic since it does not unpack to
> >>>> WORKDIR, so it can not be pulled into a more generic class to
> >>>> make it available for other users.
> >>>>  
> >>>
> >>> As I already wrote to mail-list, WORKDIR and BUILDROOT are
> >>> different folders, and there was the reason to do so.  
> >>
> >> Yes we (Henning and I) get that. That is the reason why the patch 
> >> includes the line:
> >>
> >> +WORKDIR_task-unpack = "${BUILDROOT}"
> >>
> >> That means that only for the unpack task in the dpkg.bbclass the 
> >> variable WORKDIR is set to the BUILDROOT variable. This makes the 
> >> unpack step more generic so that it can be moved to the base
> >> class. Essentially removing the dependency of the unpack task to
> >> the dpkg class.  
> > 
> > Does this mean that "log.do_unpack" and "run.do_unpack" will be
> > stored in buildchroot? Let me check this.
> >   
> 
> Yes it does:
> 
> ls 
> isar/build/tmp/work/buildchroot/debian-wheezy-armhf/rootfs/home/builder/hello/temp/
> 
> log.do_unpack
> log.do_unpack.19557
> log.task_order
> run.do_unpack
> run.do_unpack.19557
> 
> While it's assumed that they should be stored in: 
> isar/build/tmp/work/hello-0.1+g7f35942-1-r0/temp/

I think i am starting to understand the whole problem with the WORKDIR
and the BUILDROOT better. Instead of the unpacking to BUILDROOT or
messing with WORKDIR i think we should have a new task after do_unpack

So instead of do_unpack -> BUILDROOT

do_unpack archive -> WORKDIR
do_copy_to_buildroot WORKDIR -> BUILDROOT

That should keep all the logs in an unmodified WORKDIR and the unpacker
would always extract to there. Classes that need the files to live in
BUILDROOT would have to include the task do_copy_to_buildroot.

Henning

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Alexander Smirnov]

On 09/08/2017 11:15 AM, Henning Schild wrote:
> Am Mon, 4 Sep 2017 18:36:23 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>> Issues:
>>> 1. full customizations of the images is hard to impossible to
>>> realize in a layer without touching Isar
>>> 1.1. there is no easy way to just copy a file into the image
>>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>>> done in a layer, there is no way too hook into multistrap or the
>>> configure-script
>>>
>>> Change:
>>> Introduce a class that lets users create custom debian packages on
>>> the fly, without having to create a /debian directory and actually
>>> building. That allows you to pull in debian-dependencies, you could
>>> have a package that has no content and is just there to install
>>> what you need for a feature of your product.
>>> Using package hooks (preinst, postinst ..) you can configure pretty
>>> much all you want when installing the package.
>>> The package can contain actual payload as well, basically any files
>>> that come from "somewhere else". Say binary data like wallpapers,
>>> sound files or application binaries.
>>>
>>> Impact:
>>> This patch addresses the metioned issue in a way that uses debian
>>> mechanism. All the customizations will enjoy features like
>>> - collission protection (multiple packages providing the same file)
>>> - config file protection
>>> - versioning and the ability to deploy your changes in an
>>> updateable way
>>>
>>> This patch introduces a major new feature to Isar.
>>>
>>> This class introduces a new class for building debian packages on
>>> the fly. They can basically contain anything from random sources,
>>> where building happens outside of Isar. It also allows to create
>>> meta-packages that contain nothing but pull in dependencies, once
>>> all our packets come in via multistrap that will come in handy.
>>> For rootfs configuration you would use post- and pre- scripts just
>>> like regular debian packages do.
>>>    
>>
>> Good idea.
>>
>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>> ---
>>>    meta/classes/dpkg-custom.bbclass | 57
>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57
>>> insertions(+) create mode 100644 meta/classes/dpkg-custom.bbclass
>>>
>>> diff --git a/meta/classes/dpkg-custom.bbclass
>>> b/meta/classes/dpkg-custom.bbclass new file mode 100644
>>> index 0000000..e4e743f
>>> --- /dev/null
>>> +++ b/meta/classes/dpkg-custom.bbclass
>>> @@ -0,0 +1,57 @@
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2017 Siemens AG
>>> +
>>> +inherit dpkg
>>> +
>>> +DEBIAN_DEPENDS ?= ""
>>> +MAINTAINER ?= "FIXME Unknown maintainer"
>>> +
>>> +D = "${WORKDIR}/image/"
>>> +
>>> +# Populate folder that will be picked up as package
>>> +# TODO this should be called 'do_install'
>>> +do_populate_package() {
>>> +	bbnote "Put your files for this package in ${D}"
>>> +}
>>> +
>>> +addtask populate_package after do_unpack before
>>> do_deb_package_prepare +
>>> +# so we can put hooks etc. in there already
>>> +do_populate_package[dirs] = "${D}/DEBIAN"
>>> +
>>> +do_deb_package_prepare() {
>>> +	cat<<-__EOF__ > ${D}/DEBIAN/control
>>> +		Package: ${PN}
>>> +		Architecture: `dpkg --print-architecture`
>>> +		Section: misc
>>> +		Priority: optional
>>> +		Maintainer: ${MAINTAINER}
>>> +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]'
>>> ','`
>>> +		Version: ${PV}+isar
>>> +		Description: ${DESCRIPTION}
>>> +	__EOF__
>>> +	for t in pre post
>>> +	do
>>> +		for a in inst rm
>>> +		do
>>> +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
>>> +		done
>>> +	done
>>> +}
>>> +
>>> +addtask deb_package_prepare after do_populate_package before
>>> do_deb_package_conffiles +
>>> +do_deb_package_conffiles() {
>>> +	CONFFILES=${D}/DEBIAN/conffiles
>>> +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >>
>>> $CONFFILES
>>> +	test -s $CONFFILES || rm $CONFFILES
>>> +}
>>> +
>>> +addtask deb_package_conffiles after do_deb_package_prepare before
>>> do_deb_package +
>>> +do_deb_package() {
>>> +	sudo chown -R root:root ${D}/DEBIAN/
>>> +	sudo dpkg-deb --build ${D} ${WORKDIR}
>>> +}
>>> +
>>> +addtask deb_package after do_deb_package_conffiles before
>>> do_install
>>
>> If I got this correctly, the difference from dpkg-debian flow is that
>> you generate 'debian' folder on the fly. So:
> 
> Well it is a capital "DEBIAN" folder that contains the package metadata
> and no build instructions like 'debian'.
> 
>> 1. Why we need another do_deb_package, while do_build from dpkg class
>> does the same?
> 
> do_build does not do the same. It eventually calls 'dpkg-buildpackage'
> which does "configure; make; make install; make deb"
> 
> Using the same language this step is just "make deb" and it uses the
> debian tool 'dpkg-deb'.
> 
> Custom steps for "configure; make;" could be added in the future. "make
> install" is do_populate_package.

Ok, I see now. But why it's done by host dpkg-deb tool? This should be 
done inside buildchroot to keep version compatibility. If you use host 
stretch and build for wheezy, there will be big gap in versions.

> 
>> 2. Can we just add optional task to generic pipeline which generates
>> debian folder? Or make it as a part of do_build: if [ ! -d
>> ${D}/debian ]; then blablabla.
> 
> That would be adding support for 'debian'-izing a source package.
> 
> This patch is more about packaging binaries and creating meta-packages.
> 

Ok.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 4/6] meta/dpkg-debian: Make 'do_install' more generic, prepare for pulling out

[Henning Schild]

Am Wed, 30 Aug 2017 21:09:17 +0200
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> This was supposed to be a two step thing, first make it generic later
> pull it out. Some mistake in rebasing ... will fix.

Please still review this patch as required for this series. It is
similar to 1 and 2 and the same disussions about WORKDIR vs BUILDROOT
and splitting apply.

Henning

> Henning
> 
> Am Wed, 30 Aug 2017 21:03:55 +0200
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
> > Issue:
> > The 'do_install' in dpkg-debian is not generic since it wants to
> > find the debian package in BUILDROOT. And this variable is only
> > meaningful in this class.
> > 
> > Change:
> > Make do_install generic by making it look into WORKDIR and keep the
> > old semantic by overwriting this variable for the task do_install.
> > 
> > Impact:
> > This patch does not change the behaviour of Isar. It addresses the
> > issue and prepares for another patch pulling code to another class.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  meta/classes/dpkg-debian.bbclass | 13 +++----------
> >  meta/classes/dpkg.bbclass        | 14 ++++++++++++++
> >  2 files changed, 17 insertions(+), 10 deletions(-)
> >  create mode 100644 meta/classes/dpkg.bbclass
> > 
> > diff --git a/meta/classes/dpkg-debian.bbclass
> > b/meta/classes/dpkg-debian.bbclass index 7466254..5df3678 100644
> > --- a/meta/classes/dpkg-debian.bbclass
> > +++ b/meta/classes/dpkg-debian.bbclass
> > @@ -5,6 +5,8 @@
> >  # These sources need to be enabled for that, they need to have the
> >  # debian/ folder.
> >  
> > +inherit dpkg
> > +
> >  # Add dependency from buildchroot creation
> >  DEPENDS += "buildchroot"
> >  do_unpack[deptask] = "do_build"
> > @@ -18,6 +20,7 @@ do_unpack[dirs] = "${BUILDROOT}"
> >  S ?= "${BUILDROOT}"
> >  
> >  WORKDIR_task-unpack = "${BUILDROOT}"
> > +WORKDIR_task-install = "${BUILDROOT}"
> >  
> >  do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> >  
> > @@ -25,13 +28,3 @@ do_build[stamp-extra-info] =
> > "${DISTRO}-${DISTRO_ARCH}" do_build() {
> >      sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> >  }
> > -
> > -
> > -# Install package to dedicated deploy directory
> > -do_install() {
> > -    install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
> > -}
> > -
> > -addtask install after do_build
> > -do_install[dirs] = "${DEPLOY_DIR_DEB}"
> > -do_install[stamp-extra-info] = "${MACHINE}"
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> > new file mode 100644
> > index 0000000..4fb2757
> > --- /dev/null
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -0,0 +1,14 @@
> > +# This software is a part of ISAR.
> > +# Copyright (C) 2017 Siemens AG
> > +#
> > +# This class should contain common functionality of all classes or
> > recipes +# that create debian packages.
> > +
> > +# Install package to dedicated deploy directory
> > +do_install() {
> > +    install -m 644 ${WORKDIR}/*.deb ${DEPLOY_DIR_DEB}/
> > +}
> > +
> > +addtask install after do_build
> > +do_install[dirs] = "${DEPLOY_DIR_DEB}"
> > +do_install[stamp-extra-info] = "${MACHINE}"  
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Fri, 8 Sep 2017 11:31:23 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 09/08/2017 11:15 AM, Henning Schild wrote:
> > Am Mon, 4 Sep 2017 18:36:23 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 08/30/2017 10:03 PM, Henning Schild wrote:  
> >>> Issues:
> >>> 1. full customizations of the images is hard to impossible to
> >>> realize in a layer without touching Isar
> >>> 1.1. there is no easy way to just copy a file into the image
> >>> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >>> be done in a layer, there is no way too hook into multistrap or
> >>> the configure-script
> >>>
> >>> Change:
> >>> Introduce a class that lets users create custom debian packages on
> >>> the fly, without having to create a /debian directory and actually
> >>> building. That allows you to pull in debian-dependencies, you
> >>> could have a package that has no content and is just there to
> >>> install what you need for a feature of your product.
> >>> Using package hooks (preinst, postinst ..) you can configure
> >>> pretty much all you want when installing the package.
> >>> The package can contain actual payload as well, basically any
> >>> files that come from "somewhere else". Say binary data like
> >>> wallpapers, sound files or application binaries.
> >>>
> >>> Impact:
> >>> This patch addresses the metioned issue in a way that uses debian
> >>> mechanism. All the customizations will enjoy features like
> >>> - collission protection (multiple packages providing the same
> >>> file)
> >>> - config file protection
> >>> - versioning and the ability to deploy your changes in an
> >>> updateable way
> >>>
> >>> This patch introduces a major new feature to Isar.
> >>>
> >>> This class introduces a new class for building debian packages on
> >>> the fly. They can basically contain anything from random sources,
> >>> where building happens outside of Isar. It also allows to create
> >>> meta-packages that contain nothing but pull in dependencies, once
> >>> all our packets come in via multistrap that will come in handy.
> >>> For rootfs configuration you would use post- and pre- scripts just
> >>> like regular debian packages do.
> >>>      
> >>
> >> Good idea.
> >>  
> >>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >>> ---
> >>>    meta/classes/dpkg-custom.bbclass | 57
> >>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57
> >>> insertions(+) create mode 100644 meta/classes/dpkg-custom.bbclass
> >>>
> >>> diff --git a/meta/classes/dpkg-custom.bbclass
> >>> b/meta/classes/dpkg-custom.bbclass new file mode 100644
> >>> index 0000000..e4e743f
> >>> --- /dev/null
> >>> +++ b/meta/classes/dpkg-custom.bbclass
> >>> @@ -0,0 +1,57 @@
> >>> +# This software is a part of ISAR.
> >>> +# Copyright (C) 2017 Siemens AG
> >>> +
> >>> +inherit dpkg
> >>> +
> >>> +DEBIAN_DEPENDS ?= ""
> >>> +MAINTAINER ?= "FIXME Unknown maintainer"
> >>> +
> >>> +D = "${WORKDIR}/image/"
> >>> +
> >>> +# Populate folder that will be picked up as package
> >>> +# TODO this should be called 'do_install'
> >>> +do_populate_package() {
> >>> +	bbnote "Put your files for this package in ${D}"
> >>> +}
> >>> +
> >>> +addtask populate_package after do_unpack before
> >>> do_deb_package_prepare +
> >>> +# so we can put hooks etc. in there already
> >>> +do_populate_package[dirs] = "${D}/DEBIAN"
> >>> +
> >>> +do_deb_package_prepare() {
> >>> +	cat<<-__EOF__ > ${D}/DEBIAN/control
> >>> +		Package: ${PN}
> >>> +		Architecture: `dpkg --print-architecture`
> >>> +		Section: misc
> >>> +		Priority: optional
> >>> +		Maintainer: ${MAINTAINER}
> >>> +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]'
> >>> ','`
> >>> +		Version: ${PV}+isar
> >>> +		Description: ${DESCRIPTION}
> >>> +	__EOF__
> >>> +	for t in pre post
> >>> +	do
> >>> +		for a in inst rm
> >>> +		do
> >>> +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
> >>> +		done
> >>> +	done
> >>> +}
> >>> +
> >>> +addtask deb_package_prepare after do_populate_package before
> >>> do_deb_package_conffiles +
> >>> +do_deb_package_conffiles() {
> >>> +	CONFFILES=${D}/DEBIAN/conffiles
> >>> +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|'
> >>> >> $CONFFILES
> >>> +	test -s $CONFFILES || rm $CONFFILES
> >>> +}
> >>> +
> >>> +addtask deb_package_conffiles after do_deb_package_prepare before
> >>> do_deb_package +
> >>> +do_deb_package() {
> >>> +	sudo chown -R root:root ${D}/DEBIAN/
> >>> +	sudo dpkg-deb --build ${D} ${WORKDIR}
> >>> +}
> >>> +
> >>> +addtask deb_package after do_deb_package_conffiles before
> >>> do_install  
> >>
> >> If I got this correctly, the difference from dpkg-debian flow is
> >> that you generate 'debian' folder on the fly. So:  
> > 
> > Well it is a capital "DEBIAN" folder that contains the package
> > metadata and no build instructions like 'debian'.
> >   
> >> 1. Why we need another do_deb_package, while do_build from dpkg
> >> class does the same?  
> > 
> > do_build does not do the same. It eventually calls
> > 'dpkg-buildpackage' which does "configure; make; make install; make
> > deb"
> > 
> > Using the same language this step is just "make deb" and it uses the
> > debian tool 'dpkg-deb'.
> > 
> > Custom steps for "configure; make;" could be added in the future.
> > "make install" is do_populate_package.  
> 
> Ok, I see now. But why it's done by host dpkg-deb tool? This should
> be done inside buildchroot to keep version compatibility. If you use
> host stretch and build for wheezy, there will be big gap in versions.

Thanks for pointing that out. That is a mistake just like the ARCH
thing Claudius pointed out. I will look into that.
I kindly ask you to review the patch again and provide such feedback
inline.

Henning

> >   
> >> 2. Can we just add optional task to generic pipeline which
> >> generates debian folder? Or make it as a part of do_build: if [ !
> >> -d ${D}/debian ]; then blablabla.  
> > 
> > That would be adding support for 'debian'-izing a source package.
> > 
> > This patch is more about packaging binaries and creating
> > meta-packages. 
> 
> Ok.
> 

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Claudius Heine]

[-- Attachment #1.1: Type: text/plain, Size: 2880 bytes --]

Hi,

On 08.09.2017 10:30, Henning Schild wrote:
> Am Tue, 5 Sep 2017 10:37:48 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi,
>>
>>>> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:  
>>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:  
>>>>>> Issue:
>>>>>> The unpacker in dpkg is not generic since it does not unpack to
>>>>>> WORKDIR, so it can not be pulled into a more generic class to
>>>>>> make it available for other users.
>>>>>>  
>>>>>
>>>>> As I already wrote to mail-list, WORKDIR and BUILDROOT are
>>>>> different folders, and there was the reason to do so.  
>>>>
>>>> Yes we (Henning and I) get that. That is the reason why the patch 
>>>> includes the line:
>>>>
>>>> +WORKDIR_task-unpack = "${BUILDROOT}"
>>>>
>>>> That means that only for the unpack task in the dpkg.bbclass the 
>>>> variable WORKDIR is set to the BUILDROOT variable. This makes the 
>>>> unpack step more generic so that it can be moved to the base
>>>> class. Essentially removing the dependency of the unpack task to
>>>> the dpkg class.  
>>>
>>> Does this mean that "log.do_unpack" and "run.do_unpack" will be
>>> stored in buildchroot? Let me check this.
>>>   
>>
>> Yes it does:
>>
>> ls 
>> isar/build/tmp/work/buildchroot/debian-wheezy-armhf/rootfs/home/builder/hello/temp/
>>
>> log.do_unpack
>> log.do_unpack.19557
>> log.task_order
>> run.do_unpack
>> run.do_unpack.19557
>>
>> While it's assumed that they should be stored in: 
>> isar/build/tmp/work/hello-0.1+g7f35942-1-r0/temp/
> 
> I think i am starting to understand the whole problem with the WORKDIR
> and the BUILDROOT better. Instead of the unpacking to BUILDROOT or
> messing with WORKDIR i think we should have a new task after do_unpack
> 
> So instead of do_unpack -> BUILDROOT
> 
> do_unpack archive -> WORKDIR
> do_copy_to_buildroot WORKDIR -> BUILDROOT
> 
> That should keep all the logs in an unmodified WORKDIR and the unpacker
> would always extract to there. Classes that need the files to live in
> BUILDROOT would have to include the task do_copy_to_buildroot.

I agree with that separating those into two tasks is the better
solution, but I don't like the 'copy_to_buildroot' name, because it
limits possible implementations of this task by that name. For instance
you could, and maybe even should, use a bind mount instead of copying
the files around.

Something like "populate" or "prepare" as a prefix sounds a bit better IMO.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Alexander Smirnov]

On 09/08/2017 11:42 AM, Henning Schild wrote:
> Am Fri, 8 Sep 2017 11:31:23 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 09/08/2017 11:15 AM, Henning Schild wrote:
>>> Am Mon, 4 Sep 2017 18:36:23 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:
>>>>> Issues:
>>>>> 1. full customizations of the images is hard to impossible to
>>>>> realize in a layer without touching Isar
>>>>> 1.1. there is no easy way to just copy a file into the image
>>>>> 1.2. configuration (passwords, groups, cfg-files changes) can not
>>>>> be done in a layer, there is no way too hook into multistrap or
>>>>> the configure-script
>>>>>
>>>>> Change:
>>>>> Introduce a class that lets users create custom debian packages on
>>>>> the fly, without having to create a /debian directory and actually
>>>>> building. That allows you to pull in debian-dependencies, you
>>>>> could have a package that has no content and is just there to
>>>>> install what you need for a feature of your product.
>>>>> Using package hooks (preinst, postinst ..) you can configure
>>>>> pretty much all you want when installing the package.
>>>>> The package can contain actual payload as well, basically any
>>>>> files that come from "somewhere else". Say binary data like
>>>>> wallpapers, sound files or application binaries.
>>>>>
>>>>> Impact:
>>>>> This patch addresses the metioned issue in a way that uses debian
>>>>> mechanism. All the customizations will enjoy features like
>>>>> - collission protection (multiple packages providing the same
>>>>> file)
>>>>> - config file protection
>>>>> - versioning and the ability to deploy your changes in an
>>>>> updateable way
>>>>>
>>>>> This patch introduces a major new feature to Isar.
>>>>>
>>>>> This class introduces a new class for building debian packages on
>>>>> the fly. They can basically contain anything from random sources,
>>>>> where building happens outside of Isar. It also allows to create
>>>>> meta-packages that contain nothing but pull in dependencies, once
>>>>> all our packets come in via multistrap that will come in handy.
>>>>> For rootfs configuration you would use post- and pre- scripts just
>>>>> like regular debian packages do.
>>>>>       
>>>>
>>>> Good idea.
>>>>   
>>>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>>>> ---
>>>>>     meta/classes/dpkg-custom.bbclass | 57
>>>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57
>>>>> insertions(+) create mode 100644 meta/classes/dpkg-custom.bbclass
>>>>>
>>>>> diff --git a/meta/classes/dpkg-custom.bbclass
>>>>> b/meta/classes/dpkg-custom.bbclass new file mode 100644
>>>>> index 0000000..e4e743f
>>>>> --- /dev/null
>>>>> +++ b/meta/classes/dpkg-custom.bbclass
>>>>> @@ -0,0 +1,57 @@
>>>>> +# This software is a part of ISAR.
>>>>> +# Copyright (C) 2017 Siemens AG
>>>>> +
>>>>> +inherit dpkg
>>>>> +
>>>>> +DEBIAN_DEPENDS ?= ""
>>>>> +MAINTAINER ?= "FIXME Unknown maintainer"
>>>>> +
>>>>> +D = "${WORKDIR}/image/"
>>>>> +
>>>>> +# Populate folder that will be picked up as package
>>>>> +# TODO this should be called 'do_install'
>>>>> +do_populate_package() {
>>>>> +	bbnote "Put your files for this package in ${D}"
>>>>> +}
>>>>> +
>>>>> +addtask populate_package after do_unpack before
>>>>> do_deb_package_prepare +
>>>>> +# so we can put hooks etc. in there already
>>>>> +do_populate_package[dirs] = "${D}/DEBIAN"
>>>>> +
>>>>> +do_deb_package_prepare() {
>>>>> +	cat<<-__EOF__ > ${D}/DEBIAN/control
>>>>> +		Package: ${PN}
>>>>> +		Architecture: `dpkg --print-architecture`
>>>>> +		Section: misc
>>>>> +		Priority: optional
>>>>> +		Maintainer: ${MAINTAINER}
>>>>> +		Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]'
>>>>> ','`
>>>>> +		Version: ${PV}+isar
>>>>> +		Description: ${DESCRIPTION}
>>>>> +	__EOF__
>>>>> +	for t in pre post
>>>>> +	do
>>>>> +		for a in inst rm
>>>>> +		do
>>>>> +			chmod -f +x ${D}/DEBIAN/${t}${a} || true
>>>>> +		done
>>>>> +	done
>>>>> +}
>>>>> +
>>>>> +addtask deb_package_prepare after do_populate_package before
>>>>> do_deb_package_conffiles +
>>>>> +do_deb_package_conffiles() {
>>>>> +	CONFFILES=${D}/DEBIAN/conffiles
>>>>> +	find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|'
>>>>>>> $CONFFILES
>>>>> +	test -s $CONFFILES || rm $CONFFILES
>>>>> +}
>>>>> +
>>>>> +addtask deb_package_conffiles after do_deb_package_prepare before
>>>>> do_deb_package +
>>>>> +do_deb_package() {
>>>>> +	sudo chown -R root:root ${D}/DEBIAN/
>>>>> +	sudo dpkg-deb --build ${D} ${WORKDIR}
>>>>> +}
>>>>> +
>>>>> +addtask deb_package after do_deb_package_conffiles before
>>>>> do_install
>>>>
>>>> If I got this correctly, the difference from dpkg-debian flow is
>>>> that you generate 'debian' folder on the fly. So:
>>>
>>> Well it is a capital "DEBIAN" folder that contains the package
>>> metadata and no build instructions like 'debian'.
>>>    
>>>> 1. Why we need another do_deb_package, while do_build from dpkg
>>>> class does the same?
>>>
>>> do_build does not do the same. It eventually calls
>>> 'dpkg-buildpackage' which does "configure; make; make install; make
>>> deb"
>>>
>>> Using the same language this step is just "make deb" and it uses the
>>> debian tool 'dpkg-deb'.
>>>
>>> Custom steps for "configure; make;" could be added in the future.
>>> "make install" is do_populate_package.
>>
>> Ok, I see now. But why it's done by host dpkg-deb tool? This should
>> be done inside buildchroot to keep version compatibility. If you use
>> host stretch and build for wheezy, there will be big gap in versions.
> 
> Thanks for pointing that out. That is a mistake just like the ARCH
> thing Claudius pointed out. I will look into that.
> I kindly ask you to review the patch again and provide such feedback
> inline.

Sure, will do.

>>>    
>>>> 2. Can we just add optional task to generic pipeline which
>>>> generates debian folder? Or make it as a part of do_build: if [ !
>>>> -d ${D}/debian ]; then blablabla.
>>>
>>> That would be adding support for 'debian'-izing a source package.
>>>
>>> This patch is more about packaging binaries and creating
>>> meta-packages.
>>
>> Ok.
>>
> 

Re: [PATCH 1/6] meta/dpkg: Make 'do_unpack' more generic, prepare for pulling out

[Henning Schild]

Am Fri, 8 Sep 2017 10:44:29 +0200
schrieb Claudius Heine <ch@denx.de>:

> Hi,
> 
> On 08.09.2017 10:30, Henning Schild wrote:
> > Am Tue, 5 Sep 2017 10:37:48 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi,
> >>  
> >>>> On 09/04/2017 05:11 PM, Alexander Smirnov wrote:    
> >>>>> On 08/30/2017 10:03 PM, Henning Schild wrote:    
> >>>>>> Issue:
> >>>>>> The unpacker in dpkg is not generic since it does not unpack to
> >>>>>> WORKDIR, so it can not be pulled into a more generic class to
> >>>>>> make it available for other users.
> >>>>>>    
> >>>>>
> >>>>> As I already wrote to mail-list, WORKDIR and BUILDROOT are
> >>>>> different folders, and there was the reason to do so.    
> >>>>
> >>>> Yes we (Henning and I) get that. That is the reason why the
> >>>> patch includes the line:
> >>>>
> >>>> +WORKDIR_task-unpack = "${BUILDROOT}"
> >>>>
> >>>> That means that only for the unpack task in the dpkg.bbclass the 
> >>>> variable WORKDIR is set to the BUILDROOT variable. This makes
> >>>> the unpack step more generic so that it can be moved to the base
> >>>> class. Essentially removing the dependency of the unpack task to
> >>>> the dpkg class.    
> >>>
> >>> Does this mean that "log.do_unpack" and "run.do_unpack" will be
> >>> stored in buildchroot? Let me check this.
> >>>     
> >>
> >> Yes it does:
> >>
> >> ls 
> >> isar/build/tmp/work/buildchroot/debian-wheezy-armhf/rootfs/home/builder/hello/temp/
> >>
> >> log.do_unpack
> >> log.do_unpack.19557
> >> log.task_order
> >> run.do_unpack
> >> run.do_unpack.19557
> >>
> >> While it's assumed that they should be stored in: 
> >> isar/build/tmp/work/hello-0.1+g7f35942-1-r0/temp/  
> > 
> > I think i am starting to understand the whole problem with the
> > WORKDIR and the BUILDROOT better. Instead of the unpacking to
> > BUILDROOT or messing with WORKDIR i think we should have a new task
> > after do_unpack
> > 
> > So instead of do_unpack -> BUILDROOT
> > 
> > do_unpack archive -> WORKDIR
> > do_copy_to_buildroot WORKDIR -> BUILDROOT
> > 
> > That should keep all the logs in an unmodified WORKDIR and the
> > unpacker would always extract to there. Classes that need the files
> > to live in BUILDROOT would have to include the task
> > do_copy_to_buildroot.  
> 
> I agree with that separating those into two tasks is the better
> solution, but I don't like the 'copy_to_buildroot' name, because it
> limits possible implementations of this task by that name. For
> instance you could, and maybe even should, use a bind mount instead
> of copying the files around.
> 
> Something like "populate" or "prepare" as a prefix sounds a bit
> better IMO.

Agreed, i will think of a better name.

Henning

> Cheers,
> Claudius
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi,

On 09/04/2017 05:36 PM, Alexander Smirnov wrote:
> On 08/30/2017 10:03 PM, Henning Schild wrote:
>> Issues:
>> 1. full customizations of the images is hard to impossible to
>> realize in a layer without touching Isar
>> 1.1. there is no easy way to just copy a file into the image
>> 1.2. configuration (passwords, groups, cfg-files changes) can not be
>> done in a layer, there is no way too hook into multistrap or the
>> configure-script
>>
>> Change:
>> Introduce a class that lets users create custom debian packages on the
>> fly, without having to create a /debian directory and actually building.
>> That allows you to pull in debian-dependencies, you could have a
>> package that has no content and is just there to install what you need
>> for a feature of your product.
>> Using package hooks (preinst, postinst ..) you can configure pretty much
>> all you want when installing the package.
>> The package can contain actual payload as well, basically any files that
>> come from "somewhere else". Say binary data like wallpapers, sound files
>> or application binaries.
>>
>> Impact:
>> This patch addresses the metioned issue in a way that uses debian
>> mechanism. All the customizations will enjoy features like
>> - collission protection (multiple packages providing the same file)
>> - config file protection
>> - versioning and the ability to deploy your changes in an updateable way
>>
>> This patch introduces a major new feature to Isar.
>>
>> This class introduces a new class for building debian packages on the
>> fly. They can basically contain anything from random sources, where
>> building happens outside of Isar. It also allows to create meta-packages
>> that contain nothing but pull in dependencies, once all our packets come
>> in via multistrap that will come in handy.
>> For rootfs configuration you would use post- and pre- scripts just like
>> regular debian packages do.
>>
> 
> Good idea.
> 
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   meta/classes/dpkg-custom.bbclass | 57 
>> ++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 57 insertions(+)
>>   create mode 100644 meta/classes/dpkg-custom.bbclass
>>
>> diff --git a/meta/classes/dpkg-custom.bbclass 
>> b/meta/classes/dpkg-custom.bbclass
>> new file mode 100644
>> index 0000000..e4e743f
>> --- /dev/null
>> +++ b/meta/classes/dpkg-custom.bbclass
>> @@ -0,0 +1,57 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2017 Siemens AG
>> +
>> +inherit dpkg
>> +
>> +DEBIAN_DEPENDS ?= ""
>> +MAINTAINER ?= "FIXME Unknown maintainer"
>> +
>> +D = "${WORKDIR}/image/"
>> +
>> +# Populate folder that will be picked up as package
>> +# TODO this should be called 'do_install'
>> +do_populate_package() {
>> +    bbnote "Put your files for this package in ${D}"
>> +}
>> +
>> +addtask populate_package after do_unpack before do_deb_package_prepare
>> +
>> +# so we can put hooks etc. in there already
>> +do_populate_package[dirs] = "${D}/DEBIAN"
>> +
>> +do_deb_package_prepare() {
>> +    cat<<-__EOF__ > ${D}/DEBIAN/control
>> +        Package: ${PN}
>> +        Architecture: `dpkg --print-architecture`
>> +        Section: misc
>> +        Priority: optional
>> +        Maintainer: ${MAINTAINER}
>> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
>> +        Version: ${PV}+isar
>> +        Description: ${DESCRIPTION}
>> +    __EOF__
>> +    for t in pre post
>> +    do
>> +        for a in inst rm
>> +        do
>> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
>> +        done
>> +    done
>> +}
>> +
>> +addtask deb_package_prepare after do_populate_package before 
>> do_deb_package_conffiles
>> +
>> +do_deb_package_conffiles() {
>> +    CONFFILES=${D}/DEBIAN/conffiles
>> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> 
>> $CONFFILES
>> +    test -s $CONFFILES || rm $CONFFILES
>> +}
>> +
>> +addtask deb_package_conffiles after do_deb_package_prepare before 
>> do_deb_package
>> +
>> +do_deb_package() {
>> +    sudo chown -R root:root ${D}/DEBIAN/
>> +    sudo dpkg-deb --build ${D} ${WORKDIR}
>> +}
>> +
>> +addtask deb_package after do_deb_package_conffiles before do_install
>>
> 
> If I got this correctly, the difference from dpkg-debian flow is that 
> you generate 'debian' folder on the fly. So:
> 
> 1. Why we need another do_deb_package, while do_build from dpkg class 
> does the same?

Why not? 'do_build' is a very bad name, because it not really describes 
anything specific and if you have multiple tasks with good names like 
'deb_package' (while I still would prefer 'package_deb') that do just 
one job each, then its much easier to recognize the tasks when calling 
'bitbake -c liststasks ..' and its simpler to extend the pipeline in 
each recipe as needed.
> 2. Can we just add optional task to generic pipeline which generates 
> debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian 
> ]; then blablabla.

No, please not part of the 'do_build' task! Thats exactly the reason why 
'build' is not a good name, because it describes everything and nothing 
and now developers are ensnared by its name to put everything in it.
And you agree that one big task that rules the world is just very bad 
design. Every task should be very simple and do just one thing.

About the optional task inserted into the generic pipeline: I don't 
really see the benefit. I currently tend to prefer them to be separated 
and I see different arguments on both sides. My current view of those 
packages where the 'DEBIAN' directory is created on the fly are that 
they aren't clean packages. The 'right' way to do this is to have 
debianized sources or a patch that adds the right 'DEBIAN' directory to 
the sources. This here is just a useful shortcut for debianizing the 
sources.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Henning Schild]

Am Mon, 11 Sep 2017 10:13:02 +0200
schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:

> Hi,
> 
> On 09/04/2017 05:36 PM, Alexander Smirnov wrote:
> > On 08/30/2017 10:03 PM, Henning Schild wrote:  
> >> Issues:
> >> 1. full customizations of the images is hard to impossible to
> >> realize in a layer without touching Isar
> >> 1.1. there is no easy way to just copy a file into the image
> >> 1.2. configuration (passwords, groups, cfg-files changes) can not
> >> be done in a layer, there is no way too hook into multistrap or the
> >> configure-script
> >>
> >> Change:
> >> Introduce a class that lets users create custom debian packages on
> >> the fly, without having to create a /debian directory and actually
> >> building. That allows you to pull in debian-dependencies, you
> >> could have a package that has no content and is just there to
> >> install what you need for a feature of your product.
> >> Using package hooks (preinst, postinst ..) you can configure
> >> pretty much all you want when installing the package.
> >> The package can contain actual payload as well, basically any
> >> files that come from "somewhere else". Say binary data like
> >> wallpapers, sound files or application binaries.
> >>
> >> Impact:
> >> This patch addresses the metioned issue in a way that uses debian
> >> mechanism. All the customizations will enjoy features like
> >> - collission protection (multiple packages providing the same file)
> >> - config file protection
> >> - versioning and the ability to deploy your changes in an
> >> updateable way
> >>
> >> This patch introduces a major new feature to Isar.
> >>
> >> This class introduces a new class for building debian packages on
> >> the fly. They can basically contain anything from random sources,
> >> where building happens outside of Isar. It also allows to create
> >> meta-packages that contain nothing but pull in dependencies, once
> >> all our packets come in via multistrap that will come in handy.
> >> For rootfs configuration you would use post- and pre- scripts just
> >> like regular debian packages do.
> >>  
> > 
> > Good idea.
> >   
> >> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >> ---
> >>   meta/classes/dpkg-custom.bbclass | 57 
> >> ++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 57 insertions(+)
> >>   create mode 100644 meta/classes/dpkg-custom.bbclass
> >>
> >> diff --git a/meta/classes/dpkg-custom.bbclass 
> >> b/meta/classes/dpkg-custom.bbclass
> >> new file mode 100644
> >> index 0000000..e4e743f
> >> --- /dev/null
> >> +++ b/meta/classes/dpkg-custom.bbclass
> >> @@ -0,0 +1,57 @@
> >> +# This software is a part of ISAR.
> >> +# Copyright (C) 2017 Siemens AG
> >> +
> >> +inherit dpkg
> >> +
> >> +DEBIAN_DEPENDS ?= ""
> >> +MAINTAINER ?= "FIXME Unknown maintainer"
> >> +
> >> +D = "${WORKDIR}/image/"
> >> +
> >> +# Populate folder that will be picked up as package
> >> +# TODO this should be called 'do_install'
> >> +do_populate_package() {
> >> +    bbnote "Put your files for this package in ${D}"
> >> +}
> >> +
> >> +addtask populate_package after do_unpack before
> >> do_deb_package_prepare +
> >> +# so we can put hooks etc. in there already
> >> +do_populate_package[dirs] = "${D}/DEBIAN"
> >> +
> >> +do_deb_package_prepare() {
> >> +    cat<<-__EOF__ > ${D}/DEBIAN/control
> >> +        Package: ${PN}
> >> +        Architecture: `dpkg --print-architecture`
> >> +        Section: misc
> >> +        Priority: optional
> >> +        Maintainer: ${MAINTAINER}
> >> +        Depends: `echo ${DEBIAN_DEPENDS} | tr '[:blank:]' ','`
> >> +        Version: ${PV}+isar
> >> +        Description: ${DESCRIPTION}
> >> +    __EOF__
> >> +    for t in pre post
> >> +    do
> >> +        for a in inst rm
> >> +        do
> >> +            chmod -f +x ${D}/DEBIAN/${t}${a} || true
> >> +        done
> >> +    done
> >> +}
> >> +
> >> +addtask deb_package_prepare after do_populate_package before 
> >> do_deb_package_conffiles
> >> +
> >> +do_deb_package_conffiles() {
> >> +    CONFFILES=${D}/DEBIAN/conffiles
> >> +    find ${D} -type f -path '*/etc/*' | sed -e 's|^${D}|/|' >> 
> >> $CONFFILES
> >> +    test -s $CONFFILES || rm $CONFFILES
> >> +}
> >> +
> >> +addtask deb_package_conffiles after do_deb_package_prepare before 
> >> do_deb_package
> >> +
> >> +do_deb_package() {
> >> +    sudo chown -R root:root ${D}/DEBIAN/
> >> +    sudo dpkg-deb --build ${D} ${WORKDIR}
> >> +}
> >> +
> >> +addtask deb_package after do_deb_package_conffiles before
> >> do_install 
> > 
> > If I got this correctly, the difference from dpkg-debian flow is
> > that you generate 'debian' folder on the fly. So:
> > 
> > 1. Why we need another do_deb_package, while do_build from dpkg
> > class does the same?  
> 
> Why not? 'do_build' is a very bad name, because it not really
> describes anything specific and if you have multiple tasks with good
> names like 'deb_package' (while I still would prefer 'package_deb')
> that do just one job each, then its much easier to recognize the
> tasks when calling 'bitbake -c liststasks ..' and its simpler to
> extend the pipeline in each recipe as needed.
> > 2. Can we just add optional task to generic pipeline which
> > generates debian folder? Or make it as a part of do_build: if [ !
> > -d ${D}/debian ]; then blablabla.  
> 
> No, please not part of the 'do_build' task! Thats exactly the reason
> why 'build' is not a good name, because it describes everything and
> nothing and now developers are ensnared by its name to put everything
> in it. And you agree that one big task that rules the world is just
> very bad design. Every task should be very simple and do just one
> thing.
> 
> About the optional task inserted into the generic pipeline: I don't 
> really see the benefit. I currently tend to prefer them to be
> separated and I see different arguments on both sides. My current
> view of those packages where the 'DEBIAN' directory is created on the
> fly are that they aren't clean packages. The 'right' way to do this
> is to have debianized sources or a patch that adds the right 'DEBIAN'
> directory to the sources. This here is just a useful shortcut for
> debianizing the sources.

I guess you should read the whole thread and prepare your reply based
on all information.

Henning

> Cheers,
> Claudius
> 

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Alexander Smirnov]

Drop the unrelated stuff...

>> Also not a big fan of this name. 'custom' is a bit too unspecific for
>> what it does. I get that your idea that those packages are *custom*
>> made, but at the same time, so are all the packages that are directly
>> created within isar.
>>
>> I liked the old 'dpkg-bin' name more, but maybe even this is a bit to
>> unspecific. How about 'dpkg-plain'? That describes better that those
>> packages are very simple, created impromptu without much to it, IMO.
> 
> How about dpkg-wrap, because this wraps existing stuff as-is into debian
> package?
> 

Eventually Isar will have three ways to create Debian package:

  - Build it in chroot (current dpkg.bbclass)
  - Build package using cross-compilation (planned to be dpkg-cross.bbclass)
  - Pack existing files (current dpkg-custom.bbclass)

I agree with Jan, that custom is very wide term. Customization usually 
means modifying something existing according to your needs. In current 
patch you create your own package from the scratch, so there is no 
reference package for them from which you have branched and which you 
have customized.

Due to we have several classes to create package, I'd suggest to use the 
words that describe *how* this class build package. So, dpkg-wrap, 
dpkg-pack etc...

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi again,

On 09/11/2017 10:13 AM, Claudius Heine wrote:
>> If I got this correctly, the difference from dpkg-debian flow is that 
>> you generate 'debian' folder on the fly. So:
>>
>> 1. Why we need another do_deb_package, while do_build from dpkg class 
>> does the same?
> 
> Why not? 'do_build' is a very bad name, because it not really describes 
> anything specific and if you have multiple tasks with good names like 
> 'deb_package' (while I still would prefer 'package_deb') that do just 
> one job each, then its much easier to recognize the tasks when calling 
> 'bitbake -c liststasks ..' and its simpler to extend the pipeline in 
> each recipe as needed.

>> 2. Can we just add optional task to generic pipeline which generates 
>> debian folder? Or make it as a part of do_build: if [ ! -d ${D}/debian 
>> ]; then blablabla.
> 
> No, please not part of the 'do_build' task! Thats exactly the reason why 
> 'build' is not a good name, because it describes everything and nothing 
> and now developers are ensnared by its name to put everything in it.
> And you agree that one big task that rules the world is just very bad 
> design. Every task should be very simple and do just one thing.
> 
> About the optional task inserted into the generic pipeline: I don't 
> really see the benefit. I currently tend to prefer them to be separated 
> and I see different arguments on both sides. My current view of those 
> packages where the 'DEBIAN' directory is created on the fly are that 
> they aren't clean packages. The 'right' way to do this is to have 
> debianized sources or a patch that adds the right 'DEBIAN' directory to 
> the sources. This here is just a useful shortcut for debianizing the 
> sources.

I missed the context here that this thread is about building the 
packages without the debianized source and your 'do_build' is about 
creating packages from debian source packages.

Sorry for this confusion.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 5/6] meta/dpkg: add dpkg-custom class

[Claudius Heine]

Hi Henning,

On 09/11/2017 10:19 AM, Henning Schild wrote:
> I guess you should read the whole thread and prepare your reply based
> on all information.

Yes, sorry.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de