[PATCH v4 0/8] Provide infrastructure and examples for custom kernels and modules

[PATCH v4 0/8] Provide infrastructure and examples for custom kernels and modules

[Jan Kiszka]

This is now finally the promised patches to provide simple patterns for
building custom kernels for Isar images that replace the distro kernels.
It also provids the same pattern for custom out-of-tree kernel modules.

Along come two fixes, one to respect proxies during dpkg builds, the
other to adjust isar-apt repo prio.

Furthermore, I'm proposing to rename "SRC_DIR" to OE-standard "S".

Changes in v3 (only resent patch 5):
 - allow central overriding of kernel dependencies

Changes in v4
 - simplified changes to isar-image-base.bb by reordering code
 - modeled custom kernel build as class

Jan

Jan Kiszka (8):
  Forward proxy settings to dpkg build
  Prioritize isar-apt repo over all others
  Replace SRC_DIR with S
  Install kernel via replaceable recipe
  Provide class for easy custom kernel builds
  Add custom kernel examples
  Provide include file for easy custom module builds
  Add exemplary kernel module

 doc/user_manual.md                                 |   6 +-
 meta-isar/conf/distro/debian-jessie.conf           |   2 +
 meta-isar/conf/distro/debian-stretch.conf          |   2 +
 meta-isar/conf/distro/debian-wheezy.conf           |   2 +
 meta-isar/conf/distro/raspbian-jessie.conf         |   2 +
 meta-isar/conf/local.conf.sample                   |   2 +-
 meta-isar/conf/multiconfig/qemuamd64-jessie.conf   |   2 +-
 meta-isar/conf/multiconfig/qemuamd64-stretch.conf  |   3 +-
 meta-isar/conf/multiconfig/qemuarm-jessie.conf     |   2 +-
 meta-isar/conf/multiconfig/qemuarm-stretch.conf    |   3 +-
 meta-isar/conf/multiconfig/qemuarm-wheezy.conf     |   2 +-
 meta-isar/conf/multiconfig/qemui386-jessie.conf    |   2 +-
 meta-isar/conf/multiconfig/qemui386-stretch.conf   |   3 +-
 meta-isar/conf/multiconfig/rpi-jessie.conf         |   3 +-
 .../recipes-app/example-hello/example-hello.bb     |   2 +-
 meta-isar/recipes-app/libhello/libhello.bb         |   2 +-
 .../recipes-core/images/files/multistrap.conf.in   |   1 +
 meta-isar/recipes-core/images/isar-image-base.bb   |   3 +
 .../example-module/example-module.bb               |  12 +
 .../example-module/files/src/Makefile              |  18 ++
 .../example-module/files/src/example-module.c      |  19 ++
 meta-isar/recipes-kernel/linux/files/defconfig     | 308 +++++++++++++++++++++
 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
 .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
 meta/classes/dpkg.bbclass                          |   3 +-
 meta/classes/linux-kernel.bbclass                  |  98 +++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   |   4 +-
 .../buildchroot/files/isar-apt-prefs               |   3 +
 .../buildchroot/files/multistrap.conf.in           |   1 +
 .../linux-module/files/debian/changelog            |   5 +
 .../linux-module/files/debian/compat               |   1 +
 .../linux-module/files/debian/control              |  11 +
 .../recipes-kernel/linux-module/files/debian/rules |  15 +
 meta/recipes-kernel/linux-module/module.inc        |  24 ++
 meta/recipes-kernel/linux/linux-distro.bb          |  16 ++
 35 files changed, 598 insertions(+), 16 deletions(-)
 create mode 100644 meta-isar/recipes-kernel/example-module/example-module.bb
 create mode 100644 meta-isar/recipes-kernel/example-module/files/src/Makefile
 create mode 100644 meta-isar/recipes-kernel/example-module/files/src/example-module.c
 create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
 create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
 create mode 100644 meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
 create mode 100644 meta/classes/linux-kernel.bbclass
 create mode 100644 meta/recipes-devtools/buildchroot/files/isar-apt-prefs
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/changelog
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/compat
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/control
 create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
 create mode 100644 meta/recipes-kernel/linux-module/module.inc
 create mode 100644 meta/recipes-kernel/linux/linux-distro.bb

-- 
2.13.6

[PATCH v4 1/8] Forward proxy settings to dpkg build

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Required when the package needs to install build dependencies.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c10f5ba..9cd4403 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,5 +5,6 @@ inherit dpkg-base
 
 # Build package from sources using build script
 dpkg_runbuild() {
-    sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
+    E="${@ bb.utils.export_proxies(d)}"
+    sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
 }
-- 
2.13.6

[PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This ensures that we can override packages from upstream Debian or other
external sources with our self-built versions. We achieve this for now
by asking multistrap to drop a preferences file for the buildchroot so
that dependency installations use the right priority. For the image
build, this does not work because all packages are pull during the
bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
our repo gets the higher priority.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-core/images/files/multistrap.conf.in     | 1 +
 meta-isar/recipes-core/images/isar-image-base.bb           | 1 +
 meta/recipes-devtools/buildchroot/buildchroot.bb           | 4 +++-
 meta/recipes-devtools/buildchroot/files/isar-apt-prefs     | 3 +++
 meta/recipes-devtools/buildchroot/files/multistrap.conf.in | 1 +
 5 files changed, 9 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/buildchroot/files/isar-apt-prefs

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 432b6af..2ab7eab 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -10,6 +10,7 @@ aptsources=##DISTRO_MULTICONF_APTSOURCES##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
 hookdir=##DIR_HOOKS##
+aptdefaultrelease=isar
 
 [base]
 source=##DISTRO_APT_SOURCE##
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 8ddbabb..e54b9a6 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -53,6 +53,7 @@ do_rootfs() {
         -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
         -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
         -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
+        -e 's|##APT_PREFS##|./'"$WORKDIR_REL"'/isar-apt-prefs|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
     # Do not use bitbake flag [dirs] here because this folder should have
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..eb6a744 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -12,7 +12,8 @@ FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
 SRC_URI = "file://multistrap.conf.in \
            file://configscript.sh \
            file://setup.sh \
-           file://build.sh"
+           file://build.sh \
+           file://isar-apt-prefs"
 PV = "1.0"
 
 BUILDCHROOT_PREINSTALL ?= "gcc \
@@ -58,6 +59,7 @@ do_build() {
         -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
         -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+        -e 's|##APT_PREFS##|./'"$WORKDIR_REL"'/isar-apt-prefs|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
     do_setup_mounts
diff --git a/meta/recipes-devtools/buildchroot/files/isar-apt-prefs b/meta/recipes-devtools/buildchroot/files/isar-apt-prefs
new file mode 100644
index 0000000..2db40e5
--- /dev/null
+++ b/meta/recipes-devtools/buildchroot/files/isar-apt-prefs
@@ -0,0 +1,3 @@
+Package: *
+Pin: release n=isar,c=main
+Pin-Priority: 1001
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index 480a4b8..89c4968 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -10,6 +10,7 @@ aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
 hookdir=##DIR_HOOKS##
+aptpreferences=##APT_PREFS##
 
 [base]
 source=##DISTRO_APT_SOURCE##
-- 
2.13.6

[PATCH v4 3/8] Replace SRC_DIR with S

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This aligns us with OE/bitbake style.

Along updating the manual, also fix SRC_REV -> SRCREV.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 doc/user_manual.md                                   | 6 +++---
 meta-isar/recipes-app/example-hello/example-hello.bb | 2 +-
 meta-isar/recipes-app/libhello/libhello.bb           | 2 +-
 meta/classes/dpkg.bbclass                            | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 969f6d2..0771d1e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -513,7 +513,7 @@ PV = "1.0"
 SRC_URI = "git://github.com/ilbers/hello.git"
 SRCREV = "ad7065ecc4840cc436bfcdac427386dbba4ea719"
 
-SRC_DIR = "git"
+S = "git"
 
 inherit dpkg
 ```
@@ -531,8 +531,8 @@ This approach prevents duplication of the license files in different packages.
 
  - `PV` - Package version.
  - `SRC_URI` - The link where to fetch application source. Please check the BitBake user manual for supported download formats.
- - `SRC_DIR` - The directory name where application sources will be unpacked. For `git` repositories, it should be set to `git`. Please check the BitBake user manual for supported download formats.
- - `SRC_REV` - Source code revision to fetch. Please check the BitBake user manual for supported download formats.
+ - `S` - The directory name where application sources will be unpacked. For `git` repositories, it should be set to `git`. Please check the BitBake user manual for supported download formats.
+ - `SRCREV` - Source code revision to fetch. Please check the BitBake user manual for supported download formats.
 
 The last line in the example above adds recipe to the Isar work chain.
 
diff --git a/meta-isar/recipes-app/example-hello/example-hello.bb b/meta-isar/recipes-app/example-hello/example-hello.bb
index 9d83e4c..73cca0a 100644
--- a/meta-isar/recipes-app/example-hello/example-hello.bb
+++ b/meta-isar/recipes-app/example-hello/example-hello.bb
@@ -17,6 +17,6 @@ DEPENDS += "libhello"
 SRC_URI = "git://github.com/ilbers/hello.git;protocol=https"
 SRCREV = "86cc719b3359adc3c4e243387feba50360a860f3"
 
-SRC_DIR = "git"
+S = "git"
 
 inherit dpkg
diff --git a/meta-isar/recipes-app/libhello/libhello.bb b/meta-isar/recipes-app/libhello/libhello.bb
index df5793e..1875831 100644
--- a/meta-isar/recipes-app/libhello/libhello.bb
+++ b/meta-isar/recipes-app/libhello/libhello.bb
@@ -13,6 +13,6 @@ PV = "0.1-98f2e41"
 SRC_URI = "git://github.com/ilbers/libhello.git;protocol=https"
 SRCREV = "98f2e41e7d05ab8d19b0c5d160b104b725c8fd93"
 
-SRC_DIR = "git"
+S = "git"
 
 inherit dpkg
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 9cd4403..06f0579 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -6,5 +6,5 @@ inherit dpkg-base
 # Build package from sources using build script
 dpkg_runbuild() {
     E="${@ bb.utils.export_proxies(d)}"
-    sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
+    sudo -E chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${S}
 }
-- 
2.13.6

[PATCH v4 4/8] Install kernel via replaceable recipe

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This simplifies the common task of using a custom kernel instead of the
pre-selected disto variant: Move the kernel installation into a dummy
dpkg-raw recipe that only has the kernel package as dependency.

Which recipe is used for providing the kernel can now be selected via
the well-know PREFERRED_PROVIDER_virtual/kernel, just like in OE.

The kernel package name is derived from the KERNEL_ARCH variable which
is communicated from the target multiconfig file to the linux-distro
recipe.

Based on an idea of Henning Schild.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/conf/distro/debian-jessie.conf          |  2 ++
 meta-isar/conf/distro/debian-stretch.conf         |  2 ++
 meta-isar/conf/distro/debian-wheezy.conf          |  2 ++
 meta-isar/conf/distro/raspbian-jessie.conf        |  2 ++
 meta-isar/conf/multiconfig/qemuamd64-jessie.conf  |  2 +-
 meta-isar/conf/multiconfig/qemuamd64-stretch.conf |  3 ++-
 meta-isar/conf/multiconfig/qemuarm-jessie.conf    |  2 +-
 meta-isar/conf/multiconfig/qemuarm-stretch.conf   |  3 ++-
 meta-isar/conf/multiconfig/qemuarm-wheezy.conf    |  2 +-
 meta-isar/conf/multiconfig/qemui386-jessie.conf   |  2 +-
 meta-isar/conf/multiconfig/qemui386-stretch.conf  |  3 ++-
 meta-isar/conf/multiconfig/rpi-jessie.conf        |  3 ++-
 meta-isar/recipes-core/images/isar-image-base.bb  |  2 ++
 meta/recipes-kernel/linux/linux-distro.bb         | 16 ++++++++++++++++
 14 files changed, 38 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-kernel/linux/linux-distro.bb

diff --git a/meta-isar/conf/distro/debian-jessie.conf b/meta-isar/conf/distro/debian-jessie.conf
index d4bc4b1..4494c91 100644
--- a/meta-isar/conf/distro/debian-jessie.conf
+++ b/meta-isar/conf/distro/debian-jessie.conf
@@ -10,3 +10,5 @@ DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
+
+PREFERRED_PROVIDER_virtual/kernel ?= "linux-distro"
diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf
index 74177fd..1375c1d 100644
--- a/meta-isar/conf/distro/debian-stretch.conf
+++ b/meta-isar/conf/distro/debian-stretch.conf
@@ -8,3 +8,5 @@ DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
+
+PREFERRED_PROVIDER_virtual/kernel ?= "linux-distro"
diff --git a/meta-isar/conf/distro/debian-wheezy.conf b/meta-isar/conf/distro/debian-wheezy.conf
index 017316e..1e99ddd 100644
--- a/meta-isar/conf/distro/debian-wheezy.conf
+++ b/meta-isar/conf/distro/debian-wheezy.conf
@@ -10,3 +10,5 @@ DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
 DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
+
+PREFERRED_PROVIDER_virtual/kernel ?= "linux-distro"
diff --git a/meta-isar/conf/distro/raspbian-jessie.conf b/meta-isar/conf/distro/raspbian-jessie.conf
index bd3c280..6a6c568 100644
--- a/meta-isar/conf/distro/raspbian-jessie.conf
+++ b/meta-isar/conf/distro/raspbian-jessie.conf
@@ -10,3 +10,5 @@ DISTRO_APT_SOURCE_SEC ?= ""
 DISTRO_CONFIG_SCRIPT ?= "raspbian-configscript.sh"
 DISTRO_MULTICONF_BOOTSTRAP ?= "base"
 DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
+
+PREFERRED_PROVIDER_virtual/kernel ?= "linux-distro"
diff --git a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
index 51d39b6..c80a936 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
@@ -6,7 +6,7 @@ MACHINE ?= "qemuamd64"
 DISTRO ?= "debian-jessie"
 DISTRO_ARCH ?= "amd64"
 
-IMAGE_PREINSTALL += "linux-image-amd64"
+KERNEL_ARCH ?= "amd64"
 
 ROOTFS_DEV ?= "sda"
 
diff --git a/meta-isar/conf/multiconfig/qemuamd64-stretch.conf b/meta-isar/conf/multiconfig/qemuamd64-stretch.conf
index c59876a..28ab3ae 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-stretch.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-stretch.conf
@@ -6,9 +6,10 @@ MACHINE ?= "qemuamd64"
 DISTRO ?= "debian-stretch"
 DISTRO_ARCH ?= "amd64"
 
+KERNEL_ARCH ?= "amd64"
+
 IMAGE_PREINSTALL += " \
     init \
-    linux-image-amd64 \
 "
 
 ROOTFS_DEV ?= "sda"
diff --git a/meta-isar/conf/multiconfig/qemuarm-jessie.conf b/meta-isar/conf/multiconfig/qemuarm-jessie.conf
index ed84c6c..48244eb 100644
--- a/meta-isar/conf/multiconfig/qemuarm-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-jessie.conf
@@ -13,7 +13,7 @@ MACHINE ?= "qemuarm"
 DISTRO ?= "debian-jessie"
 DISTRO_ARCH ?= "armhf"
 
-IMAGE_PREINSTALL += "linux-image-armmp"
+KERNEL_ARCH ?= "armmp"
 
 ROOTFS_DEV ?= "vda"
 
diff --git a/meta-isar/conf/multiconfig/qemuarm-stretch.conf b/meta-isar/conf/multiconfig/qemuarm-stretch.conf
index 1cce97a..5cf24f2 100644
--- a/meta-isar/conf/multiconfig/qemuarm-stretch.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-stretch.conf
@@ -6,9 +6,10 @@ MACHINE ?= "qemuarm"
 DISTRO ?= "debian-stretch"
 DISTRO_ARCH ?= "armhf"
 
+KERNEL_ARCH ?= "armmp"
+
 IMAGE_PREINSTALL += " \
     init \
-    linux-image-armmp \
 "
 
 ROOTFS_DEV ?= "vda"
diff --git a/meta-isar/conf/multiconfig/qemuarm-wheezy.conf b/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
index 38ea191..695cdce 100644
--- a/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
+++ b/meta-isar/conf/multiconfig/qemuarm-wheezy.conf
@@ -13,7 +13,7 @@ MACHINE ?= "qemuarm"
 DISTRO ?= "debian-wheezy"
 DISTRO_ARCH ?= "armhf"
 
-IMAGE_PREINSTALL += "linux-image-vexpress"
+KERNEL_ARCH ?= "vexpress"
 
 ROOTFS_DEV ?= "mmcblk0"
 
diff --git a/meta-isar/conf/multiconfig/qemui386-jessie.conf b/meta-isar/conf/multiconfig/qemui386-jessie.conf
index d589661..f73ad98 100644
--- a/meta-isar/conf/multiconfig/qemui386-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemui386-jessie.conf
@@ -6,7 +6,7 @@ MACHINE ?= "qemui386"
 DISTRO ?= "debian-jessie"
 DISTRO_ARCH ?= "i386"
 
-IMAGE_PREINSTALL += "linux-image-686-pae"
+KERNEL_ARCH ?= "686-pae"
 
 ROOTFS_DEV ?= "sda"
 
diff --git a/meta-isar/conf/multiconfig/qemui386-stretch.conf b/meta-isar/conf/multiconfig/qemui386-stretch.conf
index 40239e4..60f8ccb 100644
--- a/meta-isar/conf/multiconfig/qemui386-stretch.conf
+++ b/meta-isar/conf/multiconfig/qemui386-stretch.conf
@@ -6,9 +6,10 @@ MACHINE ?= "qemui386"
 DISTRO ?= "debian-stretch"
 DISTRO_ARCH ?= "i386"
 
+KERNEL_ARCH ?= "686-pae"
+
 IMAGE_PREINSTALL += " \
     init \
-    linux-image-686-pae \
 "
 
 ROOTFS_DEV ?= "sda"
diff --git a/meta-isar/conf/multiconfig/rpi-jessie.conf b/meta-isar/conf/multiconfig/rpi-jessie.conf
index d1dd9d9..c7c7fae 100644
--- a/meta-isar/conf/multiconfig/rpi-jessie.conf
+++ b/meta-isar/conf/multiconfig/rpi-jessie.conf
@@ -13,13 +13,14 @@ MACHINE ?= "rpi"
 DISTRO ?= "raspbian-jessie"
 DISTRO_ARCH ?= "armhf"
 
+KERNEL_ARCH ?= "rpi-rpfv"
+
 IMAGE_PREINSTALL += " \
     bind9-host \
     dnsutils \
     iproute2 \
     iputils-ping \
     isc-dhcp-client \
-    linux-image-rpi-rpfv \
     lrzsz \
     net-tools \
     netcat-openbsd \
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index e54b9a6..2d16f8f 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -18,6 +18,8 @@ PV = "1.0"
 
 inherit image
 
+IMAGE_INSTALL += "${@d.getVar("PREFERRED_PROVIDER_virtual/kernel", True)}"
+
 DEPENDS += "${IMAGE_INSTALL}"
 
 IMAGE_PREINSTALL += "apt \
diff --git a/meta/recipes-kernel/linux/linux-distro.bb b/meta/recipes-kernel/linux/linux-distro.bb
new file mode 100644
index 0000000..1af93ed
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-distro.bb
@@ -0,0 +1,16 @@
+# Debian kernel pseudo package
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+DESCRIPTION = "Meta package for distribution kernel"
+MAINTAINER = "ISAR project <isar-users@googlegroups.com>"
+PROVIDES = "virtual/kernel"
+
+KERNEL_ARCH ?= "${DISTRO_ARCH}"
+
+DEBIAN_DEPENDS = "linux-image-${KERNEL_ARCH}"
+
+inherit dpkg-raw
-- 
2.13.6

[PATCH v4 5/8] Provide class for easy custom kernel builds

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

With this class, it becomes almost trivial to replace the distro kernel
with a custom build. You just need to inherit linux-kernel, specify the
source URI, define via S where the source is unpacked to and provide a
defconfig. To switch to a custom kernel recipe,
PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf or
the distro conf.

The approach works internally by first running "make deb-pkg" on the
kernel and the repackages the output to make the binary linux-image and
linux-header debs act as replacement of their distro packages. This
results in a suboptimal technical implementation which may eventually be
replaced by an isar-implemented deb-pkg build process. However, this is
not expected to affect the user-visible interface of this class.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/linux-kernel.bbclass | 98 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 meta/classes/linux-kernel.bbclass

diff --git a/meta/classes/linux-kernel.bbclass b/meta/classes/linux-kernel.bbclass
new file mode 100644
index 0000000..5f4df3f
--- /dev/null
+++ b/meta/classes/linux-kernel.bbclass
@@ -0,0 +1,98 @@
+# Custom kernel build
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+DESCRIPTION ?= "Custom kernel"
+PROVIDES = "virtual/kernel"
+
+inherit dpkg-base
+
+KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-base (>= 4.3~)"
+KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
+KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
+
+REPACK_DIR = "${PP}/repack"
+REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
+REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
+
+dpkg_runbuild() {
+	E="${@ bb.utils.export_proxies(d)}"
+	sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
+		set -e
+
+		apt-get install -y -o Debug::pkgProblemResolver=yes \
+			--no-install-recommends ${KBUILD_DEPENDS}
+
+		cd ${PP}/${S}
+		cp ../defconfig .config
+		make olddefconfig
+
+		make -j ${@bb.utils.cpu_count() * 2} deb-pkg
+
+		rm -rf ${REPACK_DIR}
+		mkdir -p ${REPACK_DIR}
+		mkdir -p ${REPACK_LINUX_IMAGE_DIR}
+		mkdir -p ${REPACK_LINUX_HEADERS_DIR}
+
+		cd ${PP}
+		tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
+		dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
+			${REPACK_LINUX_IMAGE_DIR}
+		dpkg-deb -R linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
+			${REPACK_LINUX_HEADERS_DIR}
+
+		dpkg-gencontrol -crepack/debian/control \
+			-lrepack/debian/changelog \
+			-frepack/debian/files \
+			-plinux-image-${PV} \
+			-P${REPACK_LINUX_IMAGE_DIR} \
+			-DPackage="linux-image-${KERNEL_ARCH}" \
+			-DSection=kernel \
+			-DPriority=required \
+			-DProvides="${PN}" \
+			-DDepends="${KERNEL_DEBIAN_DEPENDS}"
+
+		# Add Debian-like link installation to postinst
+		touch ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
+		sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
+		    -e "/^set -e$/a\\
+\\
+if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
+	change=install\\
+else\\
+	change=upgrade\\
+fi\\
+linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
+rm -f /lib/modules/${PV}/.fresh-install"
+
+		# Add Debian-like link removal to postrm
+		sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
+		    -e "/^set -e$/a\\
+\\
+rm -f /lib/modules/${PV}/.fresh-install\\
+\\
+if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks >/dev/null; then\\
+	linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
+fi"
+
+		dpkg-gencontrol -crepack/debian/control \
+			-lrepack/debian/changelog \
+			-frepack/debian/files \
+			-plinux-headers-${PV} \
+			-P${REPACK_LINUX_HEADERS_DIR} \
+			-Vkernel:debarch="${KERNEL_ARCH}" \
+			-DPackage="linux-headers-${KERNEL_ARCH}" \
+			-DSection=kernel \
+			-DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
+
+		dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
+			linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
+		rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
+		dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
+			linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
+		rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
+	'
+}
-- 
2.13.6

[PATCH v4 6/8] Add custom kernel examples

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Provide two examples for describing custom kernel builds. Both are for
x86_64, simply using upstream x86_64_defconfig to define the required
defconfig. One example is demoing a build from a tarball, the other
pulling from git.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/recipes-kernel/linux/files/defconfig     | 308 +++++++++++++++++++++
 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
 .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
 3 files changed, 340 insertions(+)
 create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
 create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
 create mode 100644 meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb

diff --git a/meta-isar/recipes-kernel/linux/files/defconfig b/meta-isar/recipes-kernel/linux/files/defconfig
new file mode 100644
index 0000000..e32fc1f
--- /dev/null
+++ b/meta-isar/recipes-kernel/linux/files/defconfig
@@ -0,0 +1,308 @@
+# CONFIG_LOCALVERSION_AUTO is not set
+CONFIG_SYSVIPC=y
+CONFIG_POSIX_MQUEUE=y
+CONFIG_BSD_PROCESS_ACCT=y
+CONFIG_TASKSTATS=y
+CONFIG_TASK_DELAY_ACCT=y
+CONFIG_TASK_XACCT=y
+CONFIG_TASK_IO_ACCOUNTING=y
+CONFIG_FHANDLE=y
+CONFIG_AUDIT=y
+CONFIG_NO_HZ=y
+CONFIG_HIGH_RES_TIMERS=y
+CONFIG_LOG_BUF_SHIFT=18
+CONFIG_CGROUPS=y
+CONFIG_CGROUP_FREEZER=y
+CONFIG_CPUSETS=y
+CONFIG_CGROUP_CPUACCT=y
+CONFIG_CGROUP_SCHED=y
+CONFIG_BLK_DEV_INITRD=y
+# CONFIG_COMPAT_BRK is not set
+CONFIG_PROFILING=y
+CONFIG_KPROBES=y
+CONFIG_JUMP_LABEL=y
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODULE_FORCE_UNLOAD=y
+CONFIG_PARTITION_ADVANCED=y
+CONFIG_OSF_PARTITION=y
+CONFIG_AMIGA_PARTITION=y
+CONFIG_MAC_PARTITION=y
+CONFIG_BSD_DISKLABEL=y
+CONFIG_MINIX_SUBPARTITION=y
+CONFIG_SOLARIS_X86_PARTITION=y
+CONFIG_UNIXWARE_DISKLABEL=y
+CONFIG_SGI_PARTITION=y
+CONFIG_SUN_PARTITION=y
+CONFIG_KARMA_PARTITION=y
+CONFIG_EFI_PARTITION=y
+CONFIG_SMP=y
+CONFIG_CALGARY_IOMMU=y
+CONFIG_NR_CPUS=64
+CONFIG_SCHED_SMT=y
+CONFIG_PREEMPT_VOLUNTARY=y
+CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
+CONFIG_X86_MCE=y
+CONFIG_MICROCODE=y
+CONFIG_MICROCODE_AMD=y
+CONFIG_X86_MSR=y
+CONFIG_X86_CPUID=y
+CONFIG_NUMA=y
+CONFIG_X86_CHECK_BIOS_CORRUPTION=y
+# CONFIG_MTRR_SANITIZER is not set
+CONFIG_EFI=y
+CONFIG_HZ_1000=y
+CONFIG_KEXEC=y
+CONFIG_CRASH_DUMP=y
+CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_MEMORY=y
+# CONFIG_COMPAT_VDSO is not set
+CONFIG_HIBERNATION=y
+CONFIG_PM_DEBUG=y
+CONFIG_PM_TRACE_RTC=y
+CONFIG_ACPI_DOCK=y
+CONFIG_CPU_FREQ=y
+# CONFIG_CPU_FREQ_STAT is not set
+CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
+CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
+CONFIG_CPU_FREQ_GOV_ONDEMAND=y
+CONFIG_X86_ACPI_CPUFREQ=y
+CONFIG_PCI_MMCONFIG=y
+CONFIG_PCIEPORTBUS=y
+CONFIG_PCCARD=y
+CONFIG_YENTA=y
+CONFIG_HOTPLUG_PCI=y
+CONFIG_BINFMT_MISC=y
+CONFIG_IA32_EMULATION=y
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_UNIX=y
+CONFIG_XFRM_USER=y
+CONFIG_INET=y
+CONFIG_IP_MULTICAST=y
+CONFIG_IP_ADVANCED_ROUTER=y
+CONFIG_IP_MULTIPLE_TABLES=y
+CONFIG_IP_ROUTE_MULTIPATH=y
+CONFIG_IP_ROUTE_VERBOSE=y
+CONFIG_IP_PNP=y
+CONFIG_IP_PNP_DHCP=y
+CONFIG_IP_PNP_BOOTP=y
+CONFIG_IP_PNP_RARP=y
+CONFIG_IP_MROUTE=y
+CONFIG_IP_PIMSM_V1=y
+CONFIG_IP_PIMSM_V2=y
+CONFIG_SYN_COOKIES=y
+# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
+# CONFIG_INET_XFRM_MODE_TUNNEL is not set
+# CONFIG_INET_XFRM_MODE_BEET is not set
+# CONFIG_INET_DIAG is not set
+CONFIG_TCP_CONG_ADVANCED=y
+# CONFIG_TCP_CONG_BIC is not set
+# CONFIG_TCP_CONG_WESTWOOD is not set
+# CONFIG_TCP_CONG_HTCP is not set
+CONFIG_TCP_MD5SIG=y
+CONFIG_IPV6=y
+CONFIG_INET6_AH=y
+CONFIG_INET6_ESP=y
+CONFIG_NETLABEL=y
+CONFIG_NETFILTER=y
+# CONFIG_NETFILTER_ADVANCED is not set
+CONFIG_NF_CONNTRACK=y
+CONFIG_NF_CONNTRACK_FTP=y
+CONFIG_NF_CONNTRACK_IRC=y
+CONFIG_NF_CONNTRACK_SIP=y
+CONFIG_NF_CT_NETLINK=y
+CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
+CONFIG_NETFILTER_XT_TARGET_NFLOG=y
+CONFIG_NETFILTER_XT_TARGET_SECMARK=y
+CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
+CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
+CONFIG_NETFILTER_XT_MATCH_POLICY=y
+CONFIG_NETFILTER_XT_MATCH_STATE=y
+CONFIG_NF_CONNTRACK_IPV4=y
+CONFIG_IP_NF_IPTABLES=y
+CONFIG_IP_NF_FILTER=y
+CONFIG_IP_NF_TARGET_REJECT=y
+CONFIG_NF_NAT=y
+CONFIG_IP_NF_TARGET_MASQUERADE=y
+CONFIG_IP_NF_MANGLE=y
+CONFIG_NF_CONNTRACK_IPV6=y
+CONFIG_IP6_NF_IPTABLES=y
+CONFIG_IP6_NF_MATCH_IPV6HEADER=y
+CONFIG_IP6_NF_FILTER=y
+CONFIG_IP6_NF_TARGET_REJECT=y
+CONFIG_IP6_NF_MANGLE=y
+CONFIG_NET_SCHED=y
+CONFIG_NET_EMATCH=y
+CONFIG_NET_CLS_ACT=y
+CONFIG_HAMRADIO=y
+CONFIG_CFG80211=y
+CONFIG_MAC80211=y
+CONFIG_MAC80211_LEDS=y
+CONFIG_RFKILL=y
+CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+CONFIG_DEBUG_DEVRES=y
+CONFIG_CONNECTOR=y
+CONFIG_BLK_DEV_LOOP=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_BLK_DEV_SR=y
+CONFIG_BLK_DEV_SR_VENDOR=y
+CONFIG_CHR_DEV_SG=y
+CONFIG_SCSI_CONSTANTS=y
+CONFIG_SCSI_SPI_ATTRS=y
+# CONFIG_SCSI_LOWLEVEL is not set
+CONFIG_ATA=y
+CONFIG_SATA_AHCI=y
+CONFIG_ATA_PIIX=y
+CONFIG_PATA_AMD=y
+CONFIG_PATA_OLDPIIX=y
+CONFIG_PATA_SCH=y
+CONFIG_MD=y
+CONFIG_BLK_DEV_MD=y
+CONFIG_BLK_DEV_DM=y
+CONFIG_DM_MIRROR=y
+CONFIG_DM_ZERO=y
+CONFIG_MACINTOSH_DRIVERS=y
+CONFIG_MAC_EMUMOUSEBTN=y
+CONFIG_NETDEVICES=y
+CONFIG_NETCONSOLE=y
+CONFIG_TIGON3=y
+CONFIG_NET_TULIP=y
+CONFIG_E100=y
+CONFIG_E1000=y
+CONFIG_E1000E=y
+CONFIG_SKY2=y
+CONFIG_FORCEDETH=y
+CONFIG_8139TOO=y
+CONFIG_R8169=y
+CONFIG_FDDI=y
+CONFIG_INPUT_POLLDEV=y
+# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
+CONFIG_INPUT_EVDEV=y
+CONFIG_INPUT_JOYSTICK=y
+CONFIG_INPUT_TABLET=y
+CONFIG_INPUT_TOUCHSCREEN=y
+CONFIG_INPUT_MISC=y
+CONFIG_VT_HW_CONSOLE_BINDING=y
+# CONFIG_LEGACY_PTYS is not set
+CONFIG_SERIAL_NONSTANDARD=y
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_8250_NR_UARTS=32
+CONFIG_SERIAL_8250_EXTENDED=y
+CONFIG_SERIAL_8250_MANY_PORTS=y
+CONFIG_SERIAL_8250_SHARE_IRQ=y
+CONFIG_SERIAL_8250_DETECT_IRQ=y
+CONFIG_SERIAL_8250_RSA=y
+CONFIG_HW_RANDOM=y
+# CONFIG_HW_RANDOM_INTEL is not set
+# CONFIG_HW_RANDOM_AMD is not set
+CONFIG_NVRAM=y
+CONFIG_HPET=y
+# CONFIG_HPET_MMAP is not set
+CONFIG_I2C_I801=y
+CONFIG_WATCHDOG=y
+CONFIG_AGP=y
+CONFIG_AGP_AMD64=y
+CONFIG_AGP_INTEL=y
+CONFIG_DRM=y
+CONFIG_DRM_I915=y
+CONFIG_FB_MODE_HELPERS=y
+CONFIG_FB_TILEBLITTING=y
+CONFIG_FB_EFI=y
+# CONFIG_LCD_CLASS_DEVICE is not set
+CONFIG_VGACON_SOFT_SCROLLBACK=y
+CONFIG_LOGO=y
+# CONFIG_LOGO_LINUX_MONO is not set
+# CONFIG_LOGO_LINUX_VGA16 is not set
+CONFIG_SOUND=y
+CONFIG_SND=y
+CONFIG_SND_SEQUENCER=y
+CONFIG_SND_SEQ_DUMMY=y
+CONFIG_SND_MIXER_OSS=y
+CONFIG_SND_PCM_OSS=y
+CONFIG_SND_SEQUENCER_OSS=y
+CONFIG_SND_HRTIMER=y
+CONFIG_SND_HDA_INTEL=y
+CONFIG_SND_HDA_HWDEP=y
+CONFIG_HIDRAW=y
+CONFIG_HID_GYRATION=y
+CONFIG_LOGITECH_FF=y
+CONFIG_HID_NTRIG=y
+CONFIG_HID_PANTHERLORD=y
+CONFIG_PANTHERLORD_FF=y
+CONFIG_HID_PETALYNX=y
+CONFIG_HID_SAMSUNG=y
+CONFIG_HID_SONY=y
+CONFIG_HID_SUNPLUS=y
+CONFIG_HID_TOPSEED=y
+CONFIG_HID_PID=y
+CONFIG_USB_HIDDEV=y
+CONFIG_USB=y
+CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
+CONFIG_USB_MON=y
+CONFIG_USB_EHCI_HCD=y
+CONFIG_USB_EHCI_TT_NEWSCHED=y
+CONFIG_USB_OHCI_HCD=y
+CONFIG_USB_UHCI_HCD=y
+CONFIG_USB_PRINTER=y
+CONFIG_USB_STORAGE=y
+CONFIG_EDAC=y
+CONFIG_RTC_CLASS=y
+# CONFIG_RTC_HCTOSYS is not set
+CONFIG_DMADEVICES=y
+CONFIG_EEEPC_LAPTOP=y
+CONFIG_AMD_IOMMU=y
+CONFIG_INTEL_IOMMU=y
+# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
+CONFIG_EFI_VARS=y
+CONFIG_EXT4_FS=y
+CONFIG_EXT4_FS_POSIX_ACL=y
+CONFIG_EXT4_FS_SECURITY=y
+CONFIG_QUOTA=y
+CONFIG_QUOTA_NETLINK_INTERFACE=y
+# CONFIG_PRINT_QUOTA_WARNING is not set
+CONFIG_QFMT_V2=y
+CONFIG_AUTOFS4_FS=y
+CONFIG_ISO9660_FS=y
+CONFIG_JOLIET=y
+CONFIG_ZISOFS=y
+CONFIG_MSDOS_FS=y
+CONFIG_VFAT_FS=y
+CONFIG_PROC_KCORE=y
+CONFIG_TMPFS_POSIX_ACL=y
+CONFIG_HUGETLBFS=y
+CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
+CONFIG_NFS_V4=y
+CONFIG_ROOT_NFS=y
+CONFIG_NLS_DEFAULT="utf8"
+CONFIG_NLS_CODEPAGE_437=y
+CONFIG_NLS_ASCII=y
+CONFIG_NLS_ISO8859_1=y
+CONFIG_NLS_UTF8=y
+CONFIG_PRINTK_TIME=y
+# CONFIG_ENABLE_WARN_DEPRECATED is not set
+CONFIG_MAGIC_SYSRQ=y
+# CONFIG_UNUSED_SYMBOLS is not set
+CONFIG_DEBUG_KERNEL=y
+# CONFIG_SCHED_DEBUG is not set
+CONFIG_SCHEDSTATS=y
+CONFIG_TIMER_STATS=y
+CONFIG_DEBUG_STACK_USAGE=y
+CONFIG_BLK_DEV_IO_TRACE=y
+CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
+CONFIG_EARLY_PRINTK_DBGP=y
+CONFIG_DEBUG_STACKOVERFLOW=y
+# CONFIG_DEBUG_RODATA_TEST is not set
+CONFIG_DEBUG_BOOT_PARAMS=y
+CONFIG_OPTIMIZE_INLINING=y
+CONFIG_UNWINDER_ORC=y
+CONFIG_SECURITY=y
+CONFIG_SECURITY_NETWORK=y
+CONFIG_SECURITY_SELINUX=y
+CONFIG_SECURITY_SELINUX_BOOTPARAM=y
+CONFIG_SECURITY_SELINUX_DISABLE=y
+# CONFIG_CRYPTO_ANSI_CPRNG is not set
diff --git a/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb b/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
new file mode 100644
index 0000000..dc04a02
--- /dev/null
+++ b/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
@@ -0,0 +1,17 @@
+# Example recipe for building the CIP 4.4 kernel
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+inherit linux-kernel
+
+SRC_URI += " \
+    git://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-cip.git;branch=linux-4.4.y-cip \
+    file://defconfig"
+
+SRCREV = "4e52cc5f668c4666e31a8485725b5f4e897b3baf"
+PV = "4.4.112-cip18"
+
+S = "git"
diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb b/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
new file mode 100644
index 0000000..f6b820f
--- /dev/null
+++ b/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
@@ -0,0 +1,15 @@
+# Example recipe for building the mainline kernel
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+inherit linux-kernel
+
+SRC_URI += " \
+    https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-${PV}.tar.xz \
+    file://defconfig"
+SRC_URI[sha256sum] = "866a94c1c38d923ae18e74b683d7a8a79b674ebdfe7f40f1a3be9a27d39fe354"
+
+S = "linux-${PV}"
-- 
2.13.6

[PATCH v4 7/8] Provide include file for easy custom module builds

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This provides an include for custom kernel modules. We choose an include
over a class as we carry a couple of files to debianize the module. The
user just needs to specify the module SRC_URI, and those sources must be
then reside in a separate directory which is - as usual - defined via S.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 .../linux-module/files/debian/changelog            |  5 +++++
 .../linux-module/files/debian/compat               |  1 +
 .../linux-module/files/debian/control              | 11 ++++++++++
 .../recipes-kernel/linux-module/files/debian/rules | 15 ++++++++++++++
 meta/recipes-kernel/linux-module/module.inc        | 24 ++++++++++++++++++++++
 5 files changed, 56 insertions(+)
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/changelog
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/compat
 create mode 100644 meta/recipes-kernel/linux-module/files/debian/control
 create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
 create mode 100644 meta/recipes-kernel/linux-module/module.inc

diff --git a/meta/recipes-kernel/linux-module/files/debian/changelog b/meta/recipes-kernel/linux-module/files/debian/changelog
new file mode 100644
index 0000000..c1c3516
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/changelog
@@ -0,0 +1,5 @@
+@PN@ (@PV@) unstable; urgency=low
+
+  * Generated package.
+
+ -- ISAR project <isar-users@googlegroups.com>  Tue, 6 Feb 2018 00:00:00 +0000
diff --git a/meta/recipes-kernel/linux-module/files/debian/compat b/meta/recipes-kernel/linux-module/files/debian/compat
new file mode 100644
index 0000000..ec63514
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/meta/recipes-kernel/linux-module/files/debian/control b/meta/recipes-kernel/linux-module/files/debian/control
new file mode 100644
index 0000000..d8fbcaa
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/control
@@ -0,0 +1,11 @@
+Source: @PN@
+Section: kernel
+Priority: optional
+Standards-Version: 3.9.6
+Build-Depends: linux-headers-@KERNEL_ARCH@
+Maintainer: ISAR project <isar-users@googlegroups.com>
+
+Package: @PN@
+Architecture: any
+Depends: linux-image-@KERNEL_ARCH@
+Description: @DESCRIPTION@
diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules
new file mode 100755
index 0000000..4723cd8
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/rules
@@ -0,0 +1,15 @@
+#!/usr/bin/make -f
+
+# Debian rules for custom kernel module build
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+export KDIR=$(shell ls -d /lib/modules/*/build)
+
+export DEB_BUILD_OPTIONS=parallel=$(shell nproc)
+
+%:
+	dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
new file mode 100644
index 0000000..ecac75f
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -0,0 +1,24 @@
+# Custom kernel module recipe include
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH =. "${LAYERDIR_core}/recipes-kernel/linux-module/files:"
+
+DESCRIPTION ?= "Custom kernel module ${PN}"
+
+DEPENDS = "virtual/kernel"
+
+inherit dpkg
+
+SRC_URI = "file://debian/"
+
+dpkg_runbuild_prepend() {
+    cp -r ${WORKDIR}/debian ${WORKDIR}/${S}/
+    sed -i -e 's/@PN@/${PN}/g' -e 's/@PV@/${PV}/g' \
+           -e 's/@KERNEL_ARCH@/${KERNEL_ARCH}/g' \
+           -e 's/@DESCRIPTION@/${DESCRIPTION}/g' \
+        ${WORKDIR}/${S}/debian/changelog ${WORKDIR}/${S}/debian/control
+}
-- 
2.13.6

[PATCH v4 8/8] Add exemplary kernel module

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This demonstrates the usage of the new module.inc via a primitive
hello world module.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta-isar/conf/local.conf.sample                      |  2 +-
 .../recipes-kernel/example-module/example-module.bb   | 12 ++++++++++++
 .../recipes-kernel/example-module/files/src/Makefile  | 18 ++++++++++++++++++
 .../example-module/files/src/example-module.c         | 19 +++++++++++++++++++
 4 files changed, 50 insertions(+), 1 deletion(-)
 create mode 100644 meta-isar/recipes-kernel/example-module/example-module.bb
 create mode 100644 meta-isar/recipes-kernel/example-module/files/src/Makefile
 create mode 100644 meta-isar/recipes-kernel/example-module/files/src/example-module.c

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2ae43e7..92dea46 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -153,7 +153,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "example-hello example-raw"
+IMAGE_INSTALL = "example-hello example-raw example-module"
 
 #
 # Default parallel jobs for bitbake:
diff --git a/meta-isar/recipes-kernel/example-module/example-module.bb b/meta-isar/recipes-kernel/example-module/example-module.bb
new file mode 100644
index 0000000..c7042cf
--- /dev/null
+++ b/meta-isar/recipes-kernel/example-module/example-module.bb
@@ -0,0 +1,12 @@
+# Example recipe for building a custom module
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+include recipes-kernel/linux-module/module.inc
+
+SRC_URI += "file://src"
+
+S = "src"
diff --git a/meta-isar/recipes-kernel/example-module/files/src/Makefile b/meta-isar/recipes-kernel/example-module/files/src/Makefile
new file mode 100644
index 0000000..1625b28
--- /dev/null
+++ b/meta-isar/recipes-kernel/example-module/files/src/Makefile
@@ -0,0 +1,18 @@
+# Example module
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: GPL-2.0
+
+obj-m := example-module.o
+
+INSTALL_MOD_PATH ?= $(DESTDIR)
+export INSTALL_MOD_PATH
+
+modules modules_install clean:
+	$(MAKE) -C $(KDIR) M=$(PWD) $@
+
+install: modules_install
+
+.PHONY: modules modules_install install clean
diff --git a/meta-isar/recipes-kernel/example-module/files/src/example-module.c b/meta-isar/recipes-kernel/example-module/files/src/example-module.c
new file mode 100644
index 0000000..07cb214
--- /dev/null
+++ b/meta-isar/recipes-kernel/example-module/files/src/example-module.c
@@ -0,0 +1,19 @@
+/*
+ * Example modules
+ *
+ * Copyright (c) Siemens AG, 2018
+ *
+ * SPDX-License-Identifier: GPL-2.0
+ */
+
+#include <linux/module.h>
+
+static int __init example_module_init(void)
+{
+	printk("Just an example\n");
+	return -ENOANO;
+}
+
+module_init(example_module_init);
+
+MODULE_LICENSE("GPL");
-- 
2.13.6

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/11/2018 06:25 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This ensures that we can override packages from upstream Debian or other
> external sources with our self-built versions. We achieve this for now
> by asking multistrap to drop a preferences file for the buildchroot so
> that dependency installations use the right priority. For the image
> build, this does not work because all packages are pull during the
> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
> our repo gets the higher priority.
> 

I'm not sure that I completely understand this. What is the use-case for 
this? If you build your own copy of upstream package, this 
prioritization could be resolved by specifying suffix to version.

For me this approach looks like a good way to have unexpected troubles 
like the following:

1. You pick current glibc from upstream distro XYZ, let's say v1.0.
2. Patch it and put glibc-v1.0-isar to isar-apt.
3. Wait for some time until glibc is updated in upstream, let's say to v1.1.
4. Due to glibc-v1.0-isar is set to be preferred, it's going to be 
installed. But glibc is the core library in distro, and lots of packages 
in XYZ now require glibc-v1.1. So even if you will be able to generate 
image, you can't guarantee it works correctly...

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 10:33, Alexander Smirnov wrote:
> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This ensures that we can override packages from upstream Debian or other
>> external sources with our self-built versions. We achieve this for now
>> by asking multistrap to drop a preferences file for the buildchroot so
>> that dependency installations use the right priority. For the image
>> build, this does not work because all packages are pull during the
>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
>> our repo gets the higher priority.
>>
> 
> I'm not sure that I completely understand this. What is the use-case for
> this? If you build your own copy of upstream package, this
> prioritization could be resolved by specifying suffix to version.

Simple example: custom kernel build of linux-cip

This generates a package called linux-image-amd64 (for amd64 as target),
and that may have a version < debian's kernel. Now, if we do not
prioritize isar-apt over upstream, the upstream kernel is taken, even
when you switched PREFERRED_PROVIDER (in fact, the build will break when
that virtual package linux-cip is selected but can't be installed due to
the version preference "newest").

> 
> For me this approach looks like a good way to have unexpected troubles
> like the following:
> 
> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
> 2. Patch it and put glibc-v1.0-isar to isar-apt.
> 3. Wait for some time until glibc is updated in upstream, let's say to
> v1.1.
> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be
> installed. But glibc is the core library in distro, and lots of packages
> in XYZ now require glibc-v1.1. So even if you will be able to generate
> image, you can't guarantee it works correctly...

Well, sure, no one takes away the task from you of properly maintaining
the recipes when upstream updates happen (and you pull them into your
reproducible build).

Jan

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/12/2018 12:41 PM, Jan Kiszka wrote:
> On 2018-02-12 10:33, Alexander Smirnov wrote:
>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> This ensures that we can override packages from upstream Debian or other
>>> external sources with our self-built versions. We achieve this for now
>>> by asking multistrap to drop a preferences file for the buildchroot so
>>> that dependency installations use the right priority. For the image
>>> build, this does not work because all packages are pull during the
>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
>>> our repo gets the higher priority.
>>>
>>
>> I'm not sure that I completely understand this. What is the use-case for
>> this? If you build your own copy of upstream package, this
>> prioritization could be resolved by specifying suffix to version.
> 
> Simple example: custom kernel build of linux-cip
> 
> This generates a package called linux-image-amd64 (for amd64 as target),
> and that may have a version < debian's kernel. Now, if we do not
> prioritize isar-apt over upstream, the upstream kernel is taken, even
> when you switched PREFERRED_PROVIDER (in fact, the build will break when
> that virtual package linux-cip is selected but can't be installed due to
> the version preference "newest").

Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having 
the same name as with original Debian one could be confusing.


Moreover 'man 1 multistrap' states the same:
...
The combination of default release, explicit suite and apt preferences 
can quickly become complex and bugs can be very hard to identify.
...

So, from two options:

  - Rename custom kernel package
  - Prioritize apt

the first one looks more easy and robust.

> 
>>
>> For me this approach looks like a good way to have unexpected troubles
>> like the following:
>>
>> 1. You pick current glibc from upstream distro XYZ, let's say v1.0.
>> 2. Patch it and put glibc-v1.0-isar to isar-apt.
>> 3. Wait for some time until glibc is updated in upstream, let's say to
>> v1.1.
>> 4. Due to glibc-v1.0-isar is set to be preferred, it's going to be
>> installed. But glibc is the core library in distro, and lots of packages
>> in XYZ now require glibc-v1.1. So even if you will be able to generate
>> image, you can't guarantee it works correctly...
> 
> Well, sure, no one takes away the task from you of properly maintaining
> the recipes when upstream updates happen (and you pull them into your
> reproducible build).

If this is only about specific package[s], I'd suggest to propose this 
feature as an option to *overwrite* the specific package and do not do 
this by default for the whole isar-apt.

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 11:08, Alexander Smirnov wrote:
> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> This ensures that we can override packages from upstream Debian or
>>>> other
>>>> external sources with our self-built versions. We achieve this for now
>>>> by asking multistrap to drop a preferences file for the buildchroot so
>>>> that dependency installations use the right priority. For the image
>>>> build, this does not work because all packages are pull during the
>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
>>>> our repo gets the higher priority.
>>>>
>>>
>>> I'm not sure that I completely understand this. What is the use-case for
>>> this? If you build your own copy of upstream package, this
>>> prioritization could be resolved by specifying suffix to version.
>>
>> Simple example: custom kernel build of linux-cip
>>
>> This generates a package called linux-image-amd64 (for amd64 as target),
>> and that may have a version < debian's kernel. Now, if we do not
>> prioritize isar-apt over upstream, the upstream kernel is taken, even
>> when you switched PREFERRED_PROVIDER (in fact, the build will break when
>> that virtual package linux-cip is selected but can't be installed due to
>> the version preference "newest").
> 
> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
> the same name as with original Debian one could be confusing.
> 

That is mandatory, in order to write packages that depend on THE kernel,
instead of "my-custom-kernel-in-version-4.4.112".

Jan

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 11:11, Jan Kiszka wrote:
> On 2018-02-12 11:08, Alexander Smirnov wrote:
>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> This ensures that we can override packages from upstream Debian or
>>>>> other
>>>>> external sources with our self-built versions. We achieve this for now
>>>>> by asking multistrap to drop a preferences file for the buildchroot so
>>>>> that dependency installations use the right priority. For the image
>>>>> build, this does not work because all packages are pull during the
>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
>>>>> our repo gets the higher priority.
>>>>>
>>>>
>>>> I'm not sure that I completely understand this. What is the use-case for
>>>> this? If you build your own copy of upstream package, this
>>>> prioritization could be resolved by specifying suffix to version.
>>>
>>> Simple example: custom kernel build of linux-cip
>>>
>>> This generates a package called linux-image-amd64 (for amd64 as target),
>>> and that may have a version < debian's kernel. Now, if we do not
>>> prioritize isar-apt over upstream, the upstream kernel is taken, even
>>> when you switched PREFERRED_PROVIDER (in fact, the build will break when
>>> that virtual package linux-cip is selected but can't be installed due to
>>> the version preference "newest").
>>
>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>> the same name as with original Debian one could be confusing.
>>
> 
> That is mandatory, in order to write packages that depend on THE kernel,
> instead of "my-custom-kernel-in-version-4.4.112".

To make it clearer, just look at the example-module case: The recipe can
build both against the distro as well as all custom kernels, without
touching it. And that is a pattern not limited to kernel<->module
relationships.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/12/2018 01:27 PM, Jan Kiszka wrote:
> On 2018-02-12 11:11, Jan Kiszka wrote:
>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>
>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>> other
>>>>>> external sources with our self-built versions. We achieve this for now
>>>>>> by asking multistrap to drop a preferences file for the buildchroot so
>>>>>> that dependency installations use the right priority. For the image
>>>>>> build, this does not work because all packages are pull during the
>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure that
>>>>>> our repo gets the higher priority.
>>>>>>
>>>>>
>>>>> I'm not sure that I completely understand this. What is the use-case for
>>>>> this? If you build your own copy of upstream package, this
>>>>> prioritization could be resolved by specifying suffix to version.
>>>>
>>>> Simple example: custom kernel build of linux-cip
>>>>
>>>> This generates a package called linux-image-amd64 (for amd64 as target),
>>>> and that may have a version < debian's kernel. Now, if we do not
>>>> prioritize isar-apt over upstream, the upstream kernel is taken, even
>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break when
>>>> that virtual package linux-cip is selected but can't be installed due to
>>>> the version preference "newest").
>>>
>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>> the same name as with original Debian one could be confusing.
>>>
>>
>> That is mandatory, in order to write packages that depend on THE kernel,
>> instead of "my-custom-kernel-in-version-4.4.112".
> 
> To make it clearer, just look at the example-module case: The recipe can
> build both against the distro as well as all custom kernels, without
> touching it. And that is a pattern not limited to kernel<->module
> relationships.

kernel<->module pair are built from the same source tree, so here no 
problem with ${PN}-${PV} substitution as dependency. Also I've parsed 
pure Debian packages, nobody depends from 'linux-image-*', so no 
correlation with upstream packages.

What are the other use-cases for this pattern? As I mentioned above with 
'glibc' example, this is unclean approach, distro could become broken at 
anytime. This could become robust *only* after build reproducibility 
integration, when you cache the upstream content. But with build-rep you 
could manually manage the content of the cache, so prioritization is not 
needed, you could just drop duplications from the cache.

In general having packages with the same name in 'source.list' and 
overriding version policies in apt looks very dangerous for me.

If the current series requires prioritization for kernel only, it would 
be nice to specify only this package in apt preferences.

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 14:09, Alexander Smirnov wrote:
> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>
>>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>>> other
>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>> for now
>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>> buildchroot so
>>>>>>> that dependency installations use the right priority. For the image
>>>>>>> build, this does not work because all packages are pull during the
>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>> that
>>>>>>> our repo gets the higher priority.
>>>>>>>
>>>>>>
>>>>>> I'm not sure that I completely understand this. What is the
>>>>>> use-case for
>>>>>> this? If you build your own copy of upstream package, this
>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>
>>>>> Simple example: custom kernel build of linux-cip
>>>>>
>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>> target),
>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>> prioritize isar-apt over upstream, the upstream kernel is taken, even
>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break
>>>>> when
>>>>> that virtual package linux-cip is selected but can't be installed
>>>>> due to
>>>>> the version preference "newest").
>>>>
>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>>> the same name as with original Debian one could be confusing.
>>>>
>>>
>>> That is mandatory, in order to write packages that depend on THE kernel,
>>> instead of "my-custom-kernel-in-version-4.4.112".
>>
>> To make it clearer, just look at the example-module case: The recipe can
>> build both against the distro as well as all custom kernels, without
>> touching it. And that is a pattern not limited to kernel<->module
>> relationships.
> 
> kernel<->module pair are built from the same source tree, so here no
> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
> pure Debian packages, nobody depends from 'linux-image-*', so no
> correlation with upstream packages.

No, they can be independent, just like in the example I submitted. It's
an external kernel module vs. some kernel. Please re-read.

For Debian, this specific problem does not exist because Debian only
ships in-tree modules or those that are built on the target on
installation. But you will find the exact same pattern when browsing
external projects that come with modules.

> 
> What are the other use-cases for this pattern? As I mentioned above with
> 'glibc' example, this is unclean approach, distro could become broken at
> anytime. This could become robust *only* after build reproducibility
> integration, when you cache the upstream content. But with build-rep you
> could manually manage the content of the cache, so prioritization is not
> needed, you could just drop duplications from the cache.

Every lib-dependency is another pattern for the special kernel/module
case: Every distro package that come with a dependencies that you want
to replace with a custom build AND you have good reason to downgrade
that package, then you want your own packages to have higher prior.
That's why this hard-coding is the right thing to do, in general: If you
are building a package that competes with upstream, there must be a
reason...

> 
> In general having packages with the same name in 'source.list' and
> overriding version policies in apt looks very dangerous for me.
> 
> If the current series requires prioritization for kernel only, it would
> be nice to specify only this package in apt preferences.

See above, the kernel is just the first case. There will be more, I'm
absolutely sure.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/12/2018 04:45 PM, Jan Kiszka wrote:
> On 2018-02-12 14:09, Alexander Smirnov wrote:
>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>
>>>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>>>> other
>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>> for now
>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>> buildchroot so
>>>>>>>> that dependency installations use the right priority. For the image
>>>>>>>> build, this does not work because all packages are pull during the
>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>> that
>>>>>>>> our repo gets the higher priority.
>>>>>>>>
>>>>>>>
>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>> use-case for
>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>
>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>
>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>> target),
>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken, even
>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break
>>>>>> when
>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>> due to
>>>>>> the version preference "newest").
>>>>>
>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>>>> the same name as with original Debian one could be confusing.
>>>>>
>>>>
>>>> That is mandatory, in order to write packages that depend on THE kernel,
>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>
>>> To make it clearer, just look at the example-module case: The recipe can
>>> build both against the distro as well as all custom kernels, without
>>> touching it. And that is a pattern not limited to kernel<->module
>>> relationships.
>>
>> kernel<->module pair are built from the same source tree, so here no
>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>> pure Debian packages, nobody depends from 'linux-image-*', so no
>> correlation with upstream packages.
> 
> No, they can be independent, just like in the example I submitted. It's
> an external kernel module vs. some kernel. Please re-read.
> 
> For Debian, this specific problem does not exist because Debian only
> ships in-tree modules or those that are built on the target on
> installation. But you will find the exact same pattern when browsing
> external projects that come with modules.

Ok, I see that you define KERNEL_ARCH variable in machine config and 
this variable is used to bind kernel<->module build process. Sometimes 
KERNEL_ARCH has to be changed according to your kernel configuration. 
It's not enough to change preferred_provider from "linux-distro" to 
"linux-cip" if you want to use LPAE for your ARM kernel. You also need 
to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.

How you see the overall procedure to switch to custom kernel? Why 
KERNEL_ARCH can't be "cip-armmp" in this case?

For example armbian uses branch names and machines to distinguish images:

https://www.armbian.com/kernel/

  - linux-image-dev-sun8i
  - linux-image-next-odroidc2

So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get 
conflicts with the upstream. Also this doesn't affect modules building 
at all.

> 
>>
>> What are the other use-cases for this pattern? As I mentioned above with
>> 'glibc' example, this is unclean approach, distro could become broken at
>> anytime. This could become robust *only* after build reproducibility
>> integration, when you cache the upstream content. But with build-rep you
>> could manually manage the content of the cache, so prioritization is not
>> needed, you could just drop duplications from the cache.
> 
> Every lib-dependency is another pattern for the special kernel/module
> case: Every distro package that come with a dependencies that you want
> to replace with a custom build AND you have good reason to downgrade
> that package, then you want your own packages to have higher prior.
> That's why this hard-coding is the right thing to do, in general: If you
> are building a package that competes with upstream, there must be a
> reason...
> 
>>
>> In general having packages with the same name in 'source.list' and
>> overriding version policies in apt looks very dangerous for me.
>>
>> If the current series requires prioritization for kernel only, it would
>> be nice to specify only this package in apt preferences.
> 
> See above, the kernel is just the first case. There will be more, I'm
> absolutely sure.

I've asked especially about these cases. :-( At the moment there is only 
kernel case, which could be solved without this prioritization.

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 17:31, Alexander Smirnov wrote:
> 
> 
> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>
>>>>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>>>>> other
>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>> for now
>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>> buildchroot so
>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>> image
>>>>>>>>> build, this does not work because all packages are pull during the
>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>>> that
>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>> use-case for
>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>>
>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>
>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>> target),
>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>> even
>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break
>>>>>>> when
>>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>>> due to
>>>>>>> the version preference "newest").
>>>>>>
>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>>>>> the same name as with original Debian one could be confusing.
>>>>>>
>>>>>
>>>>> That is mandatory, in order to write packages that depend on THE
>>>>> kernel,
>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>
>>>> To make it clearer, just look at the example-module case: The recipe
>>>> can
>>>> build both against the distro as well as all custom kernels, without
>>>> touching it. And that is a pattern not limited to kernel<->module
>>>> relationships.
>>>
>>> kernel<->module pair are built from the same source tree, so here no
>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>> correlation with upstream packages.
>>
>> No, they can be independent, just like in the example I submitted. It's
>> an external kernel module vs. some kernel. Please re-read.
>>
>> For Debian, this specific problem does not exist because Debian only
>> ships in-tree modules or those that are built on the target on
>> installation. But you will find the exact same pattern when browsing
>> external projects that come with modules.
> 
> Ok, I see that you define KERNEL_ARCH variable in machine config and
> this variable is used to bind kernel<->module build process. Sometimes
> KERNEL_ARCH has to be changed according to your kernel configuration.
> It's not enough to change preferred_provider from "linux-distro" to
> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.

OK, but this cannot be defined by the kernel recipe. It has to be
defined by the machine setting. A kernel recipe could then select a
config that matches or reject the KERNEL_ARCH. Currently, we do nothing
in this direction, that is a valid point for future improvements.

> 
> How you see the overall procedure to switch to custom kernel? Why

PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"

See also
https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf

> KERNEL_ARCH can't be "cip-armmp" in this case?

Because the KERNEL_ARCH remains amd64, armmp, or whatever you
configured. Changing the kernel source doesn't change that in any way.

> 
> For example armbian uses branch names and machines to distinguish images:
> 
> https://www.armbian.com/kernel/
> 
>  - linux-image-dev-sun8i
>  - linux-image-next-odroidc2
> 
> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
> conflicts with the upstream. Also this doesn't affect modules building
> at all.

This is about conflict: We are *replacing* the distro kernel. But when
doing this, we do not want to change all its consumers. Armbian likely
never ran into the use case of having to design such a full replacement,
also regarding dependencies (I do not see them shipping any out-of-tree
module packages).

And the same is true for all the non-kernel cases we will face when
replacing library-like packages. Please do not focus all your thoughts
on the kernel. This issue is absolutely generic.

> 
>>
>>>
>>> What are the other use-cases for this pattern? As I mentioned above with
>>> 'glibc' example, this is unclean approach, distro could become broken at
>>> anytime. This could become robust *only* after build reproducibility
>>> integration, when you cache the upstream content. But with build-rep you
>>> could manually manage the content of the cache, so prioritization is not
>>> needed, you could just drop duplications from the cache.
>>
>> Every lib-dependency is another pattern for the special kernel/module
>> case: Every distro package that come with a dependencies that you want
>> to replace with a custom build AND you have good reason to downgrade
>> that package, then you want your own packages to have higher prior.
>> That's why this hard-coding is the right thing to do, in general: If you
>> are building a package that competes with upstream, there must be a
>> reason...
>>
>>>
>>> In general having packages with the same name in 'source.list' and
>>> overriding version policies in apt looks very dangerous for me.
>>>
>>> If the current series requires prioritization for kernel only, it would
>>> be nice to specify only this package in apt preferences.
>>
>> See above, the kernel is just the first case. There will be more, I'm
>> absolutely sure.
> 
> I've asked especially about these cases. :-( At the moment there is only
> kernel case, which could be solved without this prioritization.

Use your imagination: enforced replacement of ANY package we rebuild,
but with a lower version. We rebuild packages for a reason. That implies
we want them to be actually used. And that means our packages require a
higher priority.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 18:00, Jan Kiszka wrote:
>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>> this variable is used to bind kernel<->module build process. Sometimes
>> KERNEL_ARCH has to be changed according to your kernel configuration.
>> It's not enough to change preferred_provider from "linux-distro" to
>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
> 
> OK, but this cannot be defined by the kernel recipe. It has to be
> defined by the machine setting. A kernel recipe could then select a
> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
> in this direction, that is a valid point for future improvements.

Actually, this is rather simple to address in the own recipes that want
multi-arch kernel support, and I will likely use this pattern in
jailhouse-images:

FILESPATH =. "config-${KERNEL_ARCH}:"

and then put config-amd64/defconfig, config-armmp/defconfig etc. into
your recipe folder.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/12/2018 08:00 PM, Jan Kiszka wrote:
> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>
>>
>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>
>>>>>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>>>>>> other
>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>> for now
>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>> buildchroot so
>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>> image
>>>>>>>>>> build, this does not work because all packages are pull during the
>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>>>> that
>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>> use-case for
>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>>>
>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>
>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>> target),
>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>> even
>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break
>>>>>>>> when
>>>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>>>> due to
>>>>>>>> the version preference "newest").
>>>>>>>
>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>
>>>>>>
>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>> kernel,
>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>
>>>>> To make it clearer, just look at the example-module case: The recipe
>>>>> can
>>>>> build both against the distro as well as all custom kernels, without
>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>> relationships.
>>>>
>>>> kernel<->module pair are built from the same source tree, so here no
>>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>> correlation with upstream packages.
>>>
>>> No, they can be independent, just like in the example I submitted. It's
>>> an external kernel module vs. some kernel. Please re-read.
>>>
>>> For Debian, this specific problem does not exist because Debian only
>>> ships in-tree modules or those that are built on the target on
>>> installation. But you will find the exact same pattern when browsing
>>> external projects that come with modules.
>>
>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>> this variable is used to bind kernel<->module build process. Sometimes
>> KERNEL_ARCH has to be changed according to your kernel configuration.
>> It's not enough to change preferred_provider from "linux-distro" to
>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
> 
> OK, but this cannot be defined by the kernel recipe. It has to be
> defined by the machine setting. A kernel recipe could then select a
> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
> in this direction, that is a valid point for future improvements.
> 
>>
>> How you see the overall procedure to switch to custom kernel? Why
> 
> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
> 
> See also
> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
> 
>> KERNEL_ARCH can't be "cip-armmp" in this case?
> 
> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
> configured. Changing the kernel source doesn't change that in any way.

It doesn't remain for all the cases. ARM debian provides two kernel images:

  - linux-image-armmp
  - linux-image-armmp-lpae

Current Isar doesn't handle this, but if you build your kernel with 
LPAE, I assume you should change KERNEL_ARCH respectively, otherwise 
there will be some inaccuracy. So the same machine could have several 
KERNEL_ARCHs.

Then, what should I write in KERNEL_ARCH if I add new machine, let's say 
Cubieboard?

In general, KERNEL_ARCH is not exactly what it's used here for. 
According to the:

https://kernel-handbook.alioth.debian.org/ch-packaging.html

the linux image name looks like the following:

  - linux-image[-featureset]-flavour

Also chapter 3.3 stays:

8<--
Each architecture usually has multiple flavours of the binary kernel 
images. Different flavours correspond to different kernel configuration 
files, used to build the binary images from the same kernel tree.
8<--

So having KERNEL_ARCH as the constant is deviation from Debian flow.

In addition:

8<--
Again, multiple flavours of binary images may be built from the 
featureset tree. For example, the i386 architecture has a number of 
different flavours, such as 686 and 686-pae, built from the common 
Debian kernel source. It also contains the rt featureset. The source 
tree for building the kernels for each of these featuresets is obtained 
by applying additional patches to the Debian kernel source. It may be 
used to build the rt-686-pae binary image flavours.
8<--

So in my opinion, CIP is a feature set.

I don't really think, that replacing upstream Debian kernel package by 
custom one and claiming that is the same will make Debian community happy...


Ok, I see that we have different vision on this topic, I could stay with 
this implementation for now, but in general in my opinion this looks 
like a big hack and should be reworked.

> 
>>
>> For example armbian uses branch names and machines to distinguish images:
>>
>> https://www.armbian.com/kernel/
>>
>>   - linux-image-dev-sun8i
>>   - linux-image-next-odroidc2
>>
>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>> conflicts with the upstream. Also this doesn't affect modules building
>> at all.
> 
> This is about conflict: We are *replacing* the distro kernel. But when
> doing this, we do not want to change all its consumers. Armbian likely
> never ran into the use case of having to design such a full replacement,
> also regarding dependencies (I do not see them shipping any out-of-tree
> module packages).
> 
> And the same is true for all the non-kernel cases we will face when
> replacing library-like packages. Please do not focus all your thoughts
> on the kernel. This issue is absolutely generic.
> 
>>
>>>
>>>>
>>>> What are the other use-cases for this pattern? As I mentioned above with
>>>> 'glibc' example, this is unclean approach, distro could become broken at
>>>> anytime. This could become robust *only* after build reproducibility
>>>> integration, when you cache the upstream content. But with build-rep you
>>>> could manually manage the content of the cache, so prioritization is not
>>>> needed, you could just drop duplications from the cache.
>>>
>>> Every lib-dependency is another pattern for the special kernel/module
>>> case: Every distro package that come with a dependencies that you want
>>> to replace with a custom build AND you have good reason to downgrade
>>> that package, then you want your own packages to have higher prior.
>>> That's why this hard-coding is the right thing to do, in general: If you
>>> are building a package that competes with upstream, there must be a
>>> reason...
>>>
>>>>
>>>> In general having packages with the same name in 'source.list' and
>>>> overriding version policies in apt looks very dangerous for me.
>>>>
>>>> If the current series requires prioritization for kernel only, it would
>>>> be nice to specify only this package in apt preferences.
>>>
>>> See above, the kernel is just the first case. There will be more, I'm
>>> absolutely sure.
>>
>> I've asked especially about these cases. :-( At the moment there is only
>> kernel case, which could be solved without this prioritization.
> 
> Use your imagination: enforced replacement of ANY package we rebuild,
> but with a lower version. We rebuild packages for a reason. That implies
> we want them to be actually used. And that means our packages require a
> higher priority.
> 

My imagination told me how it will be difficult to debug segfaults on 
the target, because I pinned old libA in Isar-apt, while upstream 
dependents already have been compiled according the new version of libA. 
It wouldn't be so funny if nobody had faced with such problems in the 
past...

Alex

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Alexander Smirnov]

On 02/11/2018 06:25 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> With this class, it becomes almost trivial to replace the distro kernel
> with a custom build. You just need to inherit linux-kernel, specify the
> source URI, define via S where the source is unpacked to and provide a
> defconfig. To switch to a custom kernel recipe,
> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf or
> the distro conf.
> 
> The approach works internally by first running "make deb-pkg" on the
> kernel and the repackages the output to make the binary linux-image and
> linux-header debs act as replacement of their distro packages. This
> results in a suboptimal technical implementation which may eventually be
> replaced by an isar-implemented deb-pkg build process. However, this is
> not expected to affect the user-visible interface of this class.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/linux-kernel.bbclass | 98 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 98 insertions(+)
>   create mode 100644 meta/classes/linux-kernel.bbclass
> 
> diff --git a/meta/classes/linux-kernel.bbclass b/meta/classes/linux-kernel.bbclass
> new file mode 100644
> index 0000000..5f4df3f
> --- /dev/null
> +++ b/meta/classes/linux-kernel.bbclass
> @@ -0,0 +1,98 @@
> +# Custom kernel build
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +DESCRIPTION ?= "Custom kernel"
> +PROVIDES = "virtual/kernel"
> +
> +inherit dpkg-base
> +
> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool, kmod, linux-base (>= 4.3~)"
> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
> +
> +REPACK_DIR = "${PP}/repack"
> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
> +
> +dpkg_runbuild() {
> +	E="${@ bb.utils.export_proxies(d)}"
> +	sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
> +		set -e
> +
> +		apt-get install -y -o Debug::pkgProblemResolver=yes \
> +			--no-install-recommends ${KBUILD_DEPENDS}
> +
> +		cd ${PP}/${S}
> +		cp ../defconfig .config
> +		make olddefconfig
> +
> +		make -j ${@bb.utils.cpu_count() * 2} deb-pkg
> +
> +		rm -rf ${REPACK_DIR}
> +		mkdir -p ${REPACK_DIR}
> +		mkdir -p ${REPACK_LINUX_IMAGE_DIR}
> +		mkdir -p ${REPACK_LINUX_HEADERS_DIR}
> +
> +		cd ${PP}
> +		tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
> +		dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
> +			${REPACK_LINUX_IMAGE_DIR}
> +		dpkg-deb -R linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
> +			${REPACK_LINUX_HEADERS_DIR}
> +
> +		dpkg-gencontrol -crepack/debian/control \
> +			-lrepack/debian/changelog \
> +			-frepack/debian/files \
> +			-plinux-image-${PV} \
> +			-P${REPACK_LINUX_IMAGE_DIR} \
> +			-DPackage="linux-image-${KERNEL_ARCH}" \
> +			-DSection=kernel \
> +			-DPriority=required \
> +			-DProvides="${PN}" \
> +			-DDepends="${KERNEL_DEBIAN_DEPENDS}"
> +
> +		# Add Debian-like link installation to postinst
> +		touch ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
> +		sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
> +		    -e "/^set -e$/a\\
> +\\
> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
> +	change=install\\
> +else\\
> +	change=upgrade\\
> +fi\\
> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
> +rm -f /lib/modules/${PV}/.fresh-install"
> +
> +		# Add Debian-like link removal to postrm
> +		sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
> +		    -e "/^set -e$/a\\
> +\\
> +rm -f /lib/modules/${PV}/.fresh-install\\
> +\\
> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks >/dev/null; then\\
> +	linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
> +fi"
> +
> +		dpkg-gencontrol -crepack/debian/control \
> +			-lrepack/debian/changelog \
> +			-frepack/debian/files \
> +			-plinux-headers-${PV} \
> +			-P${REPACK_LINUX_HEADERS_DIR} \
> +			-Vkernel:debarch="${KERNEL_ARCH}" \
> +			-DPackage="linux-headers-${KERNEL_ARCH}" \
> +			-DSection=kernel \
> +			-DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
> +
> +		dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
> +			linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
> +		rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
> +		dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
> +			linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
> +		rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
> +	'
> +}
> 

What is the benefit of having this script in the class? I see that we 
have separate files for:

  - buildchroot
  - kernel modules

Why this chroot script should be a part of bitbake class? This makes 
debugging much more complicated, because I have no possibility to run 
this script manually from buildchroot (like I could do with build.sh)

Also formatting style looks nasty here.

Alex

Re: [PATCH v4 6/8] Add custom kernel examples

[Alexander Smirnov]

On 02/11/2018 06:25 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Provide two examples for describing custom kernel builds. Both are for
> x86_64, simply using upstream x86_64_defconfig to define the required
> defconfig. One example is demoing a build from a tarball, the other
> pulling from git.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta-isar/recipes-kernel/linux/files/defconfig     | 308 +++++++++++++++++++++
>   meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
>   .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
>   3 files changed, 340 insertions(+)
>   create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
>   create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
>   create mode 100644 meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
> 
> diff --git a/meta-isar/recipes-kernel/linux/files/defconfig b/meta-isar/recipes-kernel/linux/files/defconfig
> new file mode 100644
> index 0000000..e32fc1f
> --- /dev/null
> +++ b/meta-isar/recipes-kernel/linux/files/defconfig
> @@ -0,0 +1,308 @@

Do you have any concerns if I add a comment here from where this config 
was taken? Sometimes it would be very helpful to understand what this 
config contains, at least which arch.

> +# CONFIG_LOCALVERSION_AUTO is not set
> +CONFIG_SYSVIPC=y
> +CONFIG_POSIX_MQUEUE=y
> +CONFIG_BSD_PROCESS_ACCT=y
> +CONFIG_TASKSTATS=y
> +CONFIG_TASK_DELAY_ACCT=y
> +CONFIG_TASK_XACCT=y
> +CONFIG_TASK_IO_ACCOUNTING=y
> +CONFIG_FHANDLE=y
> +CONFIG_AUDIT=y
> +CONFIG_NO_HZ=y
> +CONFIG_HIGH_RES_TIMERS=y
> +CONFIG_LOG_BUF_SHIFT=18
> +CONFIG_CGROUPS=y
> +CONFIG_CGROUP_FREEZER=y
> +CONFIG_CPUSETS=y
> +CONFIG_CGROUP_CPUACCT=y
> +CONFIG_CGROUP_SCHED=y
> +CONFIG_BLK_DEV_INITRD=y
> +# CONFIG_COMPAT_BRK is not set
> +CONFIG_PROFILING=y
> +CONFIG_KPROBES=y
> +CONFIG_JUMP_LABEL=y
> +CONFIG_MODULES=y
> +CONFIG_MODULE_UNLOAD=y
> +CONFIG_MODULE_FORCE_UNLOAD=y
> +CONFIG_PARTITION_ADVANCED=y
> +CONFIG_OSF_PARTITION=y
> +CONFIG_AMIGA_PARTITION=y
> +CONFIG_MAC_PARTITION=y
> +CONFIG_BSD_DISKLABEL=y
> +CONFIG_MINIX_SUBPARTITION=y
> +CONFIG_SOLARIS_X86_PARTITION=y
> +CONFIG_UNIXWARE_DISKLABEL=y
> +CONFIG_SGI_PARTITION=y
> +CONFIG_SUN_PARTITION=y
> +CONFIG_KARMA_PARTITION=y
> +CONFIG_EFI_PARTITION=y
> +CONFIG_SMP=y
> +CONFIG_CALGARY_IOMMU=y
> +CONFIG_NR_CPUS=64
> +CONFIG_SCHED_SMT=y
> +CONFIG_PREEMPT_VOLUNTARY=y
> +CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
> +CONFIG_X86_MCE=y
> +CONFIG_MICROCODE=y
> +CONFIG_MICROCODE_AMD=y
> +CONFIG_X86_MSR=y
> +CONFIG_X86_CPUID=y
> +CONFIG_NUMA=y
> +CONFIG_X86_CHECK_BIOS_CORRUPTION=y
> +# CONFIG_MTRR_SANITIZER is not set
> +CONFIG_EFI=y
> +CONFIG_HZ_1000=y
> +CONFIG_KEXEC=y
> +CONFIG_CRASH_DUMP=y
> +CONFIG_RANDOMIZE_BASE=y
> +CONFIG_RANDOMIZE_MEMORY=y
> +# CONFIG_COMPAT_VDSO is not set
> +CONFIG_HIBERNATION=y
> +CONFIG_PM_DEBUG=y
> +CONFIG_PM_TRACE_RTC=y
> +CONFIG_ACPI_DOCK=y
> +CONFIG_CPU_FREQ=y
> +# CONFIG_CPU_FREQ_STAT is not set
> +CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
> +CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> +CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> +CONFIG_X86_ACPI_CPUFREQ=y
> +CONFIG_PCI_MMCONFIG=y
> +CONFIG_PCIEPORTBUS=y
> +CONFIG_PCCARD=y
> +CONFIG_YENTA=y
> +CONFIG_HOTPLUG_PCI=y
> +CONFIG_BINFMT_MISC=y
> +CONFIG_IA32_EMULATION=y
> +CONFIG_NET=y
> +CONFIG_PACKET=y
> +CONFIG_UNIX=y
> +CONFIG_XFRM_USER=y
> +CONFIG_INET=y
> +CONFIG_IP_MULTICAST=y
> +CONFIG_IP_ADVANCED_ROUTER=y
> +CONFIG_IP_MULTIPLE_TABLES=y
> +CONFIG_IP_ROUTE_MULTIPATH=y
> +CONFIG_IP_ROUTE_VERBOSE=y
> +CONFIG_IP_PNP=y
> +CONFIG_IP_PNP_DHCP=y
> +CONFIG_IP_PNP_BOOTP=y
> +CONFIG_IP_PNP_RARP=y
> +CONFIG_IP_MROUTE=y
> +CONFIG_IP_PIMSM_V1=y
> +CONFIG_IP_PIMSM_V2=y
> +CONFIG_SYN_COOKIES=y
> +# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
> +# CONFIG_INET_XFRM_MODE_TUNNEL is not set
> +# CONFIG_INET_XFRM_MODE_BEET is not set
> +# CONFIG_INET_DIAG is not set
> +CONFIG_TCP_CONG_ADVANCED=y
> +# CONFIG_TCP_CONG_BIC is not set
> +# CONFIG_TCP_CONG_WESTWOOD is not set
> +# CONFIG_TCP_CONG_HTCP is not set
> +CONFIG_TCP_MD5SIG=y
> +CONFIG_IPV6=y
> +CONFIG_INET6_AH=y
> +CONFIG_INET6_ESP=y
> +CONFIG_NETLABEL=y
> +CONFIG_NETFILTER=y
> +# CONFIG_NETFILTER_ADVANCED is not set
> +CONFIG_NF_CONNTRACK=y
> +CONFIG_NF_CONNTRACK_FTP=y
> +CONFIG_NF_CONNTRACK_IRC=y
> +CONFIG_NF_CONNTRACK_SIP=y
> +CONFIG_NF_CT_NETLINK=y
> +CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
> +CONFIG_NETFILTER_XT_TARGET_NFLOG=y
> +CONFIG_NETFILTER_XT_TARGET_SECMARK=y
> +CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
> +CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
> +CONFIG_NETFILTER_XT_MATCH_POLICY=y
> +CONFIG_NETFILTER_XT_MATCH_STATE=y
> +CONFIG_NF_CONNTRACK_IPV4=y
> +CONFIG_IP_NF_IPTABLES=y
> +CONFIG_IP_NF_FILTER=y
> +CONFIG_IP_NF_TARGET_REJECT=y
> +CONFIG_NF_NAT=y
> +CONFIG_IP_NF_TARGET_MASQUERADE=y
> +CONFIG_IP_NF_MANGLE=y
> +CONFIG_NF_CONNTRACK_IPV6=y
> +CONFIG_IP6_NF_IPTABLES=y
> +CONFIG_IP6_NF_MATCH_IPV6HEADER=y
> +CONFIG_IP6_NF_FILTER=y
> +CONFIG_IP6_NF_TARGET_REJECT=y
> +CONFIG_IP6_NF_MANGLE=y
> +CONFIG_NET_SCHED=y
> +CONFIG_NET_EMATCH=y
> +CONFIG_NET_CLS_ACT=y
> +CONFIG_HAMRADIO=y
> +CONFIG_CFG80211=y
> +CONFIG_MAC80211=y
> +CONFIG_MAC80211_LEDS=y
> +CONFIG_RFKILL=y
> +CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> +CONFIG_DEVTMPFS=y
> +CONFIG_DEVTMPFS_MOUNT=y
> +CONFIG_DEBUG_DEVRES=y
> +CONFIG_CONNECTOR=y
> +CONFIG_BLK_DEV_LOOP=y
> +CONFIG_BLK_DEV_SD=y
> +CONFIG_BLK_DEV_SR=y
> +CONFIG_BLK_DEV_SR_VENDOR=y
> +CONFIG_CHR_DEV_SG=y
> +CONFIG_SCSI_CONSTANTS=y
> +CONFIG_SCSI_SPI_ATTRS=y
> +# CONFIG_SCSI_LOWLEVEL is not set
> +CONFIG_ATA=y
> +CONFIG_SATA_AHCI=y
> +CONFIG_ATA_PIIX=y
> +CONFIG_PATA_AMD=y
> +CONFIG_PATA_OLDPIIX=y
> +CONFIG_PATA_SCH=y
> +CONFIG_MD=y
> +CONFIG_BLK_DEV_MD=y
> +CONFIG_BLK_DEV_DM=y
> +CONFIG_DM_MIRROR=y
> +CONFIG_DM_ZERO=y
> +CONFIG_MACINTOSH_DRIVERS=y
> +CONFIG_MAC_EMUMOUSEBTN=y
> +CONFIG_NETDEVICES=y
> +CONFIG_NETCONSOLE=y
> +CONFIG_TIGON3=y
> +CONFIG_NET_TULIP=y
> +CONFIG_E100=y
> +CONFIG_E1000=y
> +CONFIG_E1000E=y
> +CONFIG_SKY2=y
> +CONFIG_FORCEDETH=y
> +CONFIG_8139TOO=y
> +CONFIG_R8169=y
> +CONFIG_FDDI=y
> +CONFIG_INPUT_POLLDEV=y
> +# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
> +CONFIG_INPUT_EVDEV=y
> +CONFIG_INPUT_JOYSTICK=y
> +CONFIG_INPUT_TABLET=y
> +CONFIG_INPUT_TOUCHSCREEN=y
> +CONFIG_INPUT_MISC=y
> +CONFIG_VT_HW_CONSOLE_BINDING=y
> +# CONFIG_LEGACY_PTYS is not set
> +CONFIG_SERIAL_NONSTANDARD=y
> +CONFIG_SERIAL_8250=y
> +CONFIG_SERIAL_8250_CONSOLE=y
> +CONFIG_SERIAL_8250_NR_UARTS=32
> +CONFIG_SERIAL_8250_EXTENDED=y
> +CONFIG_SERIAL_8250_MANY_PORTS=y
> +CONFIG_SERIAL_8250_SHARE_IRQ=y
> +CONFIG_SERIAL_8250_DETECT_IRQ=y
> +CONFIG_SERIAL_8250_RSA=y
> +CONFIG_HW_RANDOM=y
> +# CONFIG_HW_RANDOM_INTEL is not set
> +# CONFIG_HW_RANDOM_AMD is not set
> +CONFIG_NVRAM=y
> +CONFIG_HPET=y
> +# CONFIG_HPET_MMAP is not set
> +CONFIG_I2C_I801=y
> +CONFIG_WATCHDOG=y
> +CONFIG_AGP=y
> +CONFIG_AGP_AMD64=y
> +CONFIG_AGP_INTEL=y
> +CONFIG_DRM=y
> +CONFIG_DRM_I915=y
> +CONFIG_FB_MODE_HELPERS=y
> +CONFIG_FB_TILEBLITTING=y
> +CONFIG_FB_EFI=y
> +# CONFIG_LCD_CLASS_DEVICE is not set
> +CONFIG_VGACON_SOFT_SCROLLBACK=y
> +CONFIG_LOGO=y
> +# CONFIG_LOGO_LINUX_MONO is not set
> +# CONFIG_LOGO_LINUX_VGA16 is not set
> +CONFIG_SOUND=y
> +CONFIG_SND=y
> +CONFIG_SND_SEQUENCER=y
> +CONFIG_SND_SEQ_DUMMY=y
> +CONFIG_SND_MIXER_OSS=y
> +CONFIG_SND_PCM_OSS=y
> +CONFIG_SND_SEQUENCER_OSS=y
> +CONFIG_SND_HRTIMER=y
> +CONFIG_SND_HDA_INTEL=y
> +CONFIG_SND_HDA_HWDEP=y
> +CONFIG_HIDRAW=y
> +CONFIG_HID_GYRATION=y
> +CONFIG_LOGITECH_FF=y
> +CONFIG_HID_NTRIG=y
> +CONFIG_HID_PANTHERLORD=y
> +CONFIG_PANTHERLORD_FF=y
> +CONFIG_HID_PETALYNX=y
> +CONFIG_HID_SAMSUNG=y
> +CONFIG_HID_SONY=y
> +CONFIG_HID_SUNPLUS=y
> +CONFIG_HID_TOPSEED=y
> +CONFIG_HID_PID=y
> +CONFIG_USB_HIDDEV=y
> +CONFIG_USB=y
> +CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
> +CONFIG_USB_MON=y
> +CONFIG_USB_EHCI_HCD=y
> +CONFIG_USB_EHCI_TT_NEWSCHED=y
> +CONFIG_USB_OHCI_HCD=y
> +CONFIG_USB_UHCI_HCD=y
> +CONFIG_USB_PRINTER=y
> +CONFIG_USB_STORAGE=y
> +CONFIG_EDAC=y
> +CONFIG_RTC_CLASS=y
> +# CONFIG_RTC_HCTOSYS is not set
> +CONFIG_DMADEVICES=y
> +CONFIG_EEEPC_LAPTOP=y
> +CONFIG_AMD_IOMMU=y
> +CONFIG_INTEL_IOMMU=y
> +# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
> +CONFIG_EFI_VARS=y
> +CONFIG_EXT4_FS=y
> +CONFIG_EXT4_FS_POSIX_ACL=y
> +CONFIG_EXT4_FS_SECURITY=y
> +CONFIG_QUOTA=y
> +CONFIG_QUOTA_NETLINK_INTERFACE=y
> +# CONFIG_PRINT_QUOTA_WARNING is not set
> +CONFIG_QFMT_V2=y
> +CONFIG_AUTOFS4_FS=y
> +CONFIG_ISO9660_FS=y
> +CONFIG_JOLIET=y
> +CONFIG_ZISOFS=y
> +CONFIG_MSDOS_FS=y
> +CONFIG_VFAT_FS=y
> +CONFIG_PROC_KCORE=y
> +CONFIG_TMPFS_POSIX_ACL=y
> +CONFIG_HUGETLBFS=y
> +CONFIG_NFS_FS=y
> +CONFIG_NFS_V3_ACL=y
> +CONFIG_NFS_V4=y
> +CONFIG_ROOT_NFS=y
> +CONFIG_NLS_DEFAULT="utf8"
> +CONFIG_NLS_CODEPAGE_437=y
> +CONFIG_NLS_ASCII=y
> +CONFIG_NLS_ISO8859_1=y
> +CONFIG_NLS_UTF8=y
> +CONFIG_PRINTK_TIME=y
> +# CONFIG_ENABLE_WARN_DEPRECATED is not set
> +CONFIG_MAGIC_SYSRQ=y
> +# CONFIG_UNUSED_SYMBOLS is not set
> +CONFIG_DEBUG_KERNEL=y
> +# CONFIG_SCHED_DEBUG is not set
> +CONFIG_SCHEDSTATS=y
> +CONFIG_TIMER_STATS=y
> +CONFIG_DEBUG_STACK_USAGE=y
> +CONFIG_BLK_DEV_IO_TRACE=y
> +CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
> +CONFIG_EARLY_PRINTK_DBGP=y
> +CONFIG_DEBUG_STACKOVERFLOW=y
> +# CONFIG_DEBUG_RODATA_TEST is not set
> +CONFIG_DEBUG_BOOT_PARAMS=y
> +CONFIG_OPTIMIZE_INLINING=y
> +CONFIG_UNWINDER_ORC=y
> +CONFIG_SECURITY=y
> +CONFIG_SECURITY_NETWORK=y
> +CONFIG_SECURITY_SELINUX=y
> +CONFIG_SECURITY_SELINUX_BOOTPARAM=y
> +CONFIG_SECURITY_SELINUX_DISABLE=y
> +# CONFIG_CRYPTO_ANSI_CPRNG is not set
> diff --git a/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb b/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
> new file mode 100644
> index 0000000..dc04a02
> --- /dev/null
> +++ b/meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
> @@ -0,0 +1,17 @@
> +# Example recipe for building the CIP 4.4 kernel
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +inherit linux-kernel
> +
> +SRC_URI += " \
> +    git://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-cip.git;branch=linux-4.4.y-cip \
> +    file://defconfig"
> +
> +SRCREV = "4e52cc5f668c4666e31a8485725b5f4e897b3baf"
> +PV = "4.4.112-cip18"
> +
> +S = "git"
> diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb b/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
> new file mode 100644
> index 0000000..f6b820f
> --- /dev/null
> +++ b/meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
> @@ -0,0 +1,15 @@
> +# Example recipe for building the mainline kernel
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +inherit linux-kernel
> +
> +SRC_URI += " \
> +    https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-${PV}.tar.xz \
> +    file://defconfig"
> +SRC_URI[sha256sum] = "866a94c1c38d923ae18e74b683d7a8a79b674ebdfe7f40f1a3be9a27d39fe354"
> +
> +S = "linux-${PV}"
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Jan Kiszka]

On 2018-02-12 19:56, Alexander Smirnov wrote:
> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> With this class, it becomes almost trivial to replace the distro kernel
>> with a custom build. You just need to inherit linux-kernel, specify the
>> source URI, define via S where the source is unpacked to and provide a
>> defconfig. To switch to a custom kernel recipe,
>> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf or
>> the distro conf.
>>
>> The approach works internally by first running "make deb-pkg" on the
>> kernel and the repackages the output to make the binary linux-image and
>> linux-header debs act as replacement of their distro packages. This
>> results in a suboptimal technical implementation which may eventually be
>> replaced by an isar-implemented deb-pkg build process. However, this is
>> not expected to affect the user-visible interface of this class.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/linux-kernel.bbclass | 98
>> +++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 98 insertions(+)
>>   create mode 100644 meta/classes/linux-kernel.bbclass
>>
>> diff --git a/meta/classes/linux-kernel.bbclass
>> b/meta/classes/linux-kernel.bbclass
>> new file mode 100644
>> index 0000000..5f4df3f
>> --- /dev/null
>> +++ b/meta/classes/linux-kernel.bbclass
>> @@ -0,0 +1,98 @@
>> +# Custom kernel build
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +DESCRIPTION ?= "Custom kernel"
>> +PROVIDES = "virtual/kernel"
>> +
>> +inherit dpkg-base
>> +
>> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
>> kmod, linux-base (>= 4.3~)"
>> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
>> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
>> +
>> +REPACK_DIR = "${PP}/repack"
>> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
>> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
>> +
>> +dpkg_runbuild() {
>> +    E="${@ bb.utils.export_proxies(d)}"
>> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
>> +        set -e
>> +
>> +        apt-get install -y -o Debug::pkgProblemResolver=yes \
>> +            --no-install-recommends ${KBUILD_DEPENDS}
>> +
>> +        cd ${PP}/${S}
>> +        cp ../defconfig .config
>> +        make olddefconfig
>> +
>> +        make -j ${@bb.utils.cpu_count() * 2} deb-pkg
>> +
>> +        rm -rf ${REPACK_DIR}
>> +        mkdir -p ${REPACK_DIR}
>> +        mkdir -p ${REPACK_LINUX_IMAGE_DIR}
>> +        mkdir -p ${REPACK_LINUX_HEADERS_DIR}
>> +
>> +        cd ${PP}
>> +        tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
>> +        dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>> +            ${REPACK_LINUX_IMAGE_DIR}
>> +        dpkg-deb -R linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>> +            ${REPACK_LINUX_HEADERS_DIR}
>> +
>> +        dpkg-gencontrol -crepack/debian/control \
>> +            -lrepack/debian/changelog \
>> +            -frepack/debian/files \
>> +            -plinux-image-${PV} \
>> +            -P${REPACK_LINUX_IMAGE_DIR} \
>> +            -DPackage="linux-image-${KERNEL_ARCH}" \
>> +            -DSection=kernel \
>> +            -DPriority=required \
>> +            -DProvides="${PN}" \
>> +            -DDepends="${KERNEL_DEBIAN_DEPENDS}"
>> +
>> +        # Add Debian-like link installation to postinst
>> +        touch ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
>> +            -e "/^set -e$/a\\
>> +\\
>> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
>> +    change=install\\
>> +else\\
>> +    change=upgrade\\
>> +fi\\
>> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
>> +rm -f /lib/modules/${PV}/.fresh-install"
>> +
>> +        # Add Debian-like link removal to postrm
>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
>> +            -e "/^set -e$/a\\
>> +\\
>> +rm -f /lib/modules/${PV}/.fresh-install\\
>> +\\
>> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks
>> >/dev/null; then\\
>> +    linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
>> +fi"
>> +
>> +        dpkg-gencontrol -crepack/debian/control \
>> +            -lrepack/debian/changelog \
>> +            -frepack/debian/files \
>> +            -plinux-headers-${PV} \
>> +            -P${REPACK_LINUX_HEADERS_DIR} \
>> +            -Vkernel:debarch="${KERNEL_ARCH}" \
>> +            -DPackage="linux-headers-${KERNEL_ARCH}" \
>> +            -DSection=kernel \
>> +            -DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
>> +
>> +        dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
>> +            linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>> +        rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>> +        dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
>> +            linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>> +        rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>> +    '
>> +}
>>
> 
> What is the benefit of having this script in the class? I see that we
> have separate files for:
> 
>  - buildchroot
>  - kernel modules
> 
> Why this chroot script should be a part of bitbake class? This makes
> debugging much more complicated, because I have no possibility to run
> this script manually from buildchroot (like I could do with build.sh)

I played with it, and it "just" took 6 exports to set up the interface
to script-based build. That will likely not be convenient for debugging
as well (and I never had to call things manually that way, I got all
information from logs), but I can keep that pattern. Will also mean:
back to an include, rather than using a class (no big issue).

> 
> Also formatting style looks nasty here.

That's not going to change unless you suggest some alternative for the
sed-based code injection into postinst/postrm.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 6/8] Add custom kernel examples

[Jan Kiszka]

On 2018-02-12 20:02, Alexander Smirnov wrote:
> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Provide two examples for describing custom kernel builds. Both are for
>> x86_64, simply using upstream x86_64_defconfig to define the required
>> defconfig. One example is demoing a build from a tarball, the other
>> pulling from git.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta-isar/recipes-kernel/linux/files/defconfig     | 308
>> +++++++++++++++++++++
>>   meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
>>   .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
>>   3 files changed, 340 insertions(+)
>>   create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
>>   create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
>>   create mode 100644
>> meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
>>
>> diff --git a/meta-isar/recipes-kernel/linux/files/defconfig
>> b/meta-isar/recipes-kernel/linux/files/defconfig
>> new file mode 100644
>> index 0000000..e32fc1f
>> --- /dev/null
>> +++ b/meta-isar/recipes-kernel/linux/files/defconfig
>> @@ -0,0 +1,308 @@
> 
> Do you have any concerns if I add a comment here from where this config
> was taken? Sometimes it would be very helpful to understand what this
> config contains, at least which arch.

Done.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-12 19:51, Alexander Smirnov wrote:
> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>
>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>> Debian or
>>>>>>>>>>> other
>>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>>> for now
>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>> buildchroot so
>>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>>> image
>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>> during the
>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>>>>> that
>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>> use-case for
>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>>>>
>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>
>>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>>> target),
>>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>>> even
>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>> break
>>>>>>>>> when
>>>>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>>>>> due to
>>>>>>>>> the version preference "newest").
>>>>>>>>
>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>> Having
>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>
>>>>>>>
>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>> kernel,
>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>
>>>>>> To make it clearer, just look at the example-module case: The recipe
>>>>>> can
>>>>>> build both against the distro as well as all custom kernels, without
>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>> relationships.
>>>>>
>>>>> kernel<->module pair are built from the same source tree, so here no
>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>> correlation with upstream packages.
>>>>
>>>> No, they can be independent, just like in the example I submitted. It's
>>>> an external kernel module vs. some kernel. Please re-read.
>>>>
>>>> For Debian, this specific problem does not exist because Debian only
>>>> ships in-tree modules or those that are built on the target on
>>>> installation. But you will find the exact same pattern when browsing
>>>> external projects that come with modules.
>>>
>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>> this variable is used to bind kernel<->module build process. Sometimes
>>> KERNEL_ARCH has to be changed according to your kernel configuration.
>>> It's not enough to change preferred_provider from "linux-distro" to
>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>
>> OK, but this cannot be defined by the kernel recipe. It has to be
>> defined by the machine setting. A kernel recipe could then select a
>> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
>> in this direction, that is a valid point for future improvements.
>>
>>>
>>> How you see the overall procedure to switch to custom kernel? Why
>>
>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>
>> See also
>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>
>>
>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>
>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>> configured. Changing the kernel source doesn't change that in any way.
> 
> It doesn't remain for all the cases. ARM debian provides two kernel images:
> 
>  - linux-image-armmp
>  - linux-image-armmp-lpae
> 
> Current Isar doesn't handle this, but if you build your kernel with
> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
> there will be some inaccuracy. So the same machine could have several
> KERNEL_ARCHs.

In fact, one machine would have to have multiple configs, selecting the
kernel flavor. In OE terms, that would make different machines, though
targeting the same hardware.

> 
> Then, what should I write in KERNEL_ARCH if I add new machine, let's say
> Cubieboard?
> 
> In general, KERNEL_ARCH is not exactly what it's used here for.
> According to the:
> 
> https://kernel-handbook.alioth.debian.org/ch-packaging.html
> 
> the linux image name looks like the following:
> 
>  - linux-image[-featureset]-flavour
> 
> Also chapter 3.3 stays:
> 
> 8<--
> Each architecture usually has multiple flavours of the binary kernel
> images. Different flavours correspond to different kernel configuration
> files, used to build the binary images from the same kernel tree.
> 8<--
> 
> So having KERNEL_ARCH as the constant is deviation from Debian flow.
> 
> In addition:
> 
> 8<--
> Again, multiple flavours of binary images may be built from the
> featureset tree. For example, the i386 architecture has a number of
> different flavours, such as 686 and 686-pae, built from the common
> Debian kernel source. It also contains the rt featureset. The source
> tree for building the kernels for each of these featuresets is obtained
> by applying additional patches to the Debian kernel source. It may be
> used to build the rt-686-pae binary image flavours.
> 8<--
> 
> So in my opinion, CIP is a feature set.
> 
> I don't really think, that replacing upstream Debian kernel package by
> custom one and claiming that is the same will make Debian community
> happy...
> 
> 
> Ok, I see that we have different vision on this topic, I could stay with
> this implementation for now, but in general in my opinion this looks
> like a big hack and should be reworked.

Debian has a strong word here, though it complicates things
significantly. But let me look into expressing the kernel selection via
KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
used, though. And we will have to generate / parameterize the control
files of all module packages. Maybe that's Debian intention as well.

> 
>>
>>>
>>> For example armbian uses branch names and machines to distinguish
>>> images:
>>>
>>> https://www.armbian.com/kernel/
>>>
>>>   - linux-image-dev-sun8i
>>>   - linux-image-next-odroidc2
>>>
>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>> conflicts with the upstream. Also this doesn't affect modules building
>>> at all.
>>
>> This is about conflict: We are *replacing* the distro kernel. But when
>> doing this, we do not want to change all its consumers. Armbian likely
>> never ran into the use case of having to design such a full replacement,
>> also regarding dependencies (I do not see them shipping any out-of-tree
>> module packages).
>>
>> And the same is true for all the non-kernel cases we will face when
>> replacing library-like packages. Please do not focus all your thoughts
>> on the kernel. This issue is absolutely generic.
>>
>>>
>>>>
>>>>>
>>>>> What are the other use-cases for this pattern? As I mentioned above
>>>>> with
>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>> broken at
>>>>> anytime. This could become robust *only* after build reproducibility
>>>>> integration, when you cache the upstream content. But with
>>>>> build-rep you
>>>>> could manually manage the content of the cache, so prioritization
>>>>> is not
>>>>> needed, you could just drop duplications from the cache.
>>>>
>>>> Every lib-dependency is another pattern for the special kernel/module
>>>> case: Every distro package that come with a dependencies that you want
>>>> to replace with a custom build AND you have good reason to downgrade
>>>> that package, then you want your own packages to have higher prior.
>>>> That's why this hard-coding is the right thing to do, in general: If
>>>> you
>>>> are building a package that competes with upstream, there must be a
>>>> reason...
>>>>
>>>>>
>>>>> In general having packages with the same name in 'source.list' and
>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>
>>>>> If the current series requires prioritization for kernel only, it
>>>>> would
>>>>> be nice to specify only this package in apt preferences.
>>>>
>>>> See above, the kernel is just the first case. There will be more, I'm
>>>> absolutely sure.
>>>
>>> I've asked especially about these cases. :-( At the moment there is only
>>> kernel case, which could be solved without this prioritization.
>>
>> Use your imagination: enforced replacement of ANY package we rebuild,
>> but with a lower version. We rebuild packages for a reason. That implies
>> we want them to be actually used. And that means our packages require a
>> higher priority.
>>
> 
> My imagination told me how it will be difficult to debug segfaults on
> the target, because I pinned old libA in Isar-apt, while upstream
> dependents already have been compiled according the new version of libA.
> It wouldn't be so funny if nobody had faced with such problems in the
> past...

Again: One guy shooting himself into the foot by explicitly stating this
incorrect setup doesn't mean all others users have to give up on this
valid and required default property of self-build packages. It remains a
fact that self-built packages are there to customize and overwrite
upstream packages, independent of how the kernel will look like. That's
one key purpose of Isar.

If I manage to model the kernel differently, I will probably be able to
drop this patch from /this/ series. But the topic will only come up
again when Claudius proposes his multistrap rework which will include
shipping aptpreferences as well - and prioritizing our own repo properly.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/13/2018 10:22 AM, Jan Kiszka wrote:
> On 2018-02-12 19:51, Alexander Smirnov wrote:
>> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>>
>>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>>> Debian or
>>>>>>>>>>>> other
>>>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>>>> for now
>>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>>> buildchroot so
>>>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>>>> image
>>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>>> during the
>>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>>>>>> that
>>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>>> use-case for
>>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>>>>>
>>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>>
>>>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>>>> target),
>>>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>>>> even
>>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>>> break
>>>>>>>>>> when
>>>>>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>>>>>> due to
>>>>>>>>>> the version preference "newest").
>>>>>>>>>
>>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>>> Having
>>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>>
>>>>>>>>
>>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>>> kernel,
>>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>>
>>>>>>> To make it clearer, just look at the example-module case: The recipe
>>>>>>> can
>>>>>>> build both against the distro as well as all custom kernels, without
>>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>>> relationships.
>>>>>>
>>>>>> kernel<->module pair are built from the same source tree, so here no
>>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>>> correlation with upstream packages.
>>>>>
>>>>> No, they can be independent, just like in the example I submitted. It's
>>>>> an external kernel module vs. some kernel. Please re-read.
>>>>>
>>>>> For Debian, this specific problem does not exist because Debian only
>>>>> ships in-tree modules or those that are built on the target on
>>>>> installation. But you will find the exact same pattern when browsing
>>>>> external projects that come with modules.
>>>>
>>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>>> this variable is used to bind kernel<->module build process. Sometimes
>>>> KERNEL_ARCH has to be changed according to your kernel configuration.
>>>> It's not enough to change preferred_provider from "linux-distro" to
>>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>>
>>> OK, but this cannot be defined by the kernel recipe. It has to be
>>> defined by the machine setting. A kernel recipe could then select a
>>> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
>>> in this direction, that is a valid point for future improvements.
>>>
>>>>
>>>> How you see the overall procedure to switch to custom kernel? Why
>>>
>>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>>
>>> See also
>>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>>
>>>
>>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>>
>>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>>> configured. Changing the kernel source doesn't change that in any way.
>>
>> It doesn't remain for all the cases. ARM debian provides two kernel images:
>>
>>   - linux-image-armmp
>>   - linux-image-armmp-lpae
>>
>> Current Isar doesn't handle this, but if you build your kernel with
>> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
>> there will be some inaccuracy. So the same machine could have several
>> KERNEL_ARCHs.
> 
> In fact, one machine would have to have multiple configs, selecting the
> kernel flavor. In OE terms, that would make different machines, though
> targeting the same hardware.
> 
>>
>> Then, what should I write in KERNEL_ARCH if I add new machine, let's say
>> Cubieboard?
>>
>> In general, KERNEL_ARCH is not exactly what it's used here for.
>> According to the:
>>
>> https://kernel-handbook.alioth.debian.org/ch-packaging.html
>>
>> the linux image name looks like the following:
>>
>>   - linux-image[-featureset]-flavour
>>
>> Also chapter 3.3 stays:
>>
>> 8<--
>> Each architecture usually has multiple flavours of the binary kernel
>> images. Different flavours correspond to different kernel configuration
>> files, used to build the binary images from the same kernel tree.
>> 8<--
>>
>> So having KERNEL_ARCH as the constant is deviation from Debian flow.
>>
>> In addition:
>>
>> 8<--
>> Again, multiple flavours of binary images may be built from the
>> featureset tree. For example, the i386 architecture has a number of
>> different flavours, such as 686 and 686-pae, built from the common
>> Debian kernel source. It also contains the rt featureset. The source
>> tree for building the kernels for each of these featuresets is obtained
>> by applying additional patches to the Debian kernel source. It may be
>> used to build the rt-686-pae binary image flavours.
>> 8<--
>>
>> So in my opinion, CIP is a feature set.
>>
>> I don't really think, that replacing upstream Debian kernel package by
>> custom one and claiming that is the same will make Debian community
>> happy...
>>
>>
>> Ok, I see that we have different vision on this topic, I could stay with
>> this implementation for now, but in general in my opinion this looks
>> like a big hack and should be reworked.
> 
> Debian has a strong word here, though it complicates things
> significantly. But let me look into expressing the kernel selection via
> KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
> used, though. And we will have to generate / parameterize the control
> files of all module packages. Maybe that's Debian intention as well.
> 
>>
>>>
>>>>
>>>> For example armbian uses branch names and machines to distinguish
>>>> images:
>>>>
>>>> https://www.armbian.com/kernel/
>>>>
>>>>    - linux-image-dev-sun8i
>>>>    - linux-image-next-odroidc2
>>>>
>>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>>> conflicts with the upstream. Also this doesn't affect modules building
>>>> at all.
>>>
>>> This is about conflict: We are *replacing* the distro kernel. But when
>>> doing this, we do not want to change all its consumers. Armbian likely
>>> never ran into the use case of having to design such a full replacement,
>>> also regarding dependencies (I do not see them shipping any out-of-tree
>>> module packages).
>>>
>>> And the same is true for all the non-kernel cases we will face when
>>> replacing library-like packages. Please do not focus all your thoughts
>>> on the kernel. This issue is absolutely generic.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> What are the other use-cases for this pattern? As I mentioned above
>>>>>> with
>>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>>> broken at
>>>>>> anytime. This could become robust *only* after build reproducibility
>>>>>> integration, when you cache the upstream content. But with
>>>>>> build-rep you
>>>>>> could manually manage the content of the cache, so prioritization
>>>>>> is not
>>>>>> needed, you could just drop duplications from the cache.
>>>>>
>>>>> Every lib-dependency is another pattern for the special kernel/module
>>>>> case: Every distro package that come with a dependencies that you want
>>>>> to replace with a custom build AND you have good reason to downgrade
>>>>> that package, then you want your own packages to have higher prior.
>>>>> That's why this hard-coding is the right thing to do, in general: If
>>>>> you
>>>>> are building a package that competes with upstream, there must be a
>>>>> reason...
>>>>>
>>>>>>
>>>>>> In general having packages with the same name in 'source.list' and
>>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>>
>>>>>> If the current series requires prioritization for kernel only, it
>>>>>> would
>>>>>> be nice to specify only this package in apt preferences.
>>>>>
>>>>> See above, the kernel is just the first case. There will be more, I'm
>>>>> absolutely sure.
>>>>
>>>> I've asked especially about these cases. :-( At the moment there is only
>>>> kernel case, which could be solved without this prioritization.
>>>
>>> Use your imagination: enforced replacement of ANY package we rebuild,
>>> but with a lower version. We rebuild packages for a reason. That implies
>>> we want them to be actually used. And that means our packages require a
>>> higher priority.
>>>
>>
>> My imagination told me how it will be difficult to debug segfaults on
>> the target, because I pinned old libA in Isar-apt, while upstream
>> dependents already have been compiled according the new version of libA.
>> It wouldn't be so funny if nobody had faced with such problems in the
>> past...
> 
> Again: One guy shooting himself into the foot by explicitly stating this
> incorrect setup doesn't mean all others users have to give up on this
> valid and required default property of self-build packages. It remains a
> fact that self-built packages are there to customize and overwrite
> upstream packages, independent of how the kernel will look like. That's
> one key purpose of Isar.
> 

I'm not against package overwriting. I mean that there is *not the only 
one way* to do this. Apt provides way to overwrite packages via versions.

How I see the creation of robust derivatives:

1. Create local copy for upstream apt (let's name it base-apt) with 
packages you are going to use for your build.
2. Choose package you want to modify, let's say libA.
3. Create recipe for it and update version to be ">" than original one.
4. Work with this environment.

This approach at least somehow guarantees that you will have 
reproducible build. The packages in your base-apt are built according to 
the libA version you've customized. So my opinion: user could overwrite 
only package in local mirrors, i.e. mirrors that could not be updated by 
3rd parties.


What could happen bad with pinning over upstream (libA is from upstream):

1. libA.h v1.0 had:

#define BUFFER_SIZE 32

2. libA.h v1.1 has:

#define BUFFER_SIZE 16

3. Upstream contains packages that were already rebuilt according to the 
libA v1.1

4. You are still using libA v1.0 derivative.

5. Buffer overrun could occur now in any libA consumer from the upstream.

So, there is *no* robust way to overwrite package from live upstream, 
you are always in the risk.

I could imaging that some business units may require this apt 
prioritization for their specific product's needs, so lets have this 
feature in Isar. But I think it shouldn't be enabled by default. Also 
the user documentation for this feature should describe all the risks 
that user takes if this option is enabled.

> If I manage to model the kernel differently, I will probably be able to
> drop this patch from /this/ series. But the topic will only come up
> again when Claudius proposes his multistrap rework which will include
> shipping aptpreferences as well - and prioritizing our own repo properly.
> 

As I said, I could stay with this now to move forward. So the things 
discussed above could be fixed later.

Alex

Re: [PATCH v4 6/8] Add custom kernel examples

[Alexander Smirnov]

On 02/13/2018 10:03 AM, Jan Kiszka wrote:
> On 2018-02-12 20:02, Alexander Smirnov wrote:
>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> Provide two examples for describing custom kernel builds. Both are for
>>> x86_64, simply using upstream x86_64_defconfig to define the required
>>> defconfig. One example is demoing a build from a tarball, the other
>>> pulling from git.
>>>
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>    meta-isar/recipes-kernel/linux/files/defconfig     | 308
>>> +++++++++++++++++++++
>>>    meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
>>>    .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
>>>    3 files changed, 340 insertions(+)
>>>    create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
>>>    create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
>>>    create mode 100644
>>> meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
>>>
>>> diff --git a/meta-isar/recipes-kernel/linux/files/defconfig
>>> b/meta-isar/recipes-kernel/linux/files/defconfig
>>> new file mode 100644
>>> index 0000000..e32fc1f
>>> --- /dev/null
>>> +++ b/meta-isar/recipes-kernel/linux/files/defconfig
>>> @@ -0,0 +1,308 @@
>>
>> Do you have any concerns if I add a comment here from where this config
>> was taken? Sometimes it would be very helpful to understand what this
>> config contains, at least which arch.
> 
> Done.

What do you mean by "done"? :-)

BTW: in future it would nice to implement configurable defconfig like 
it's done in Yocto via variable, for example:

https://github.com/Freescale/meta-freescale/blob/master/conf/machine/t1042d4rdb-64b.conf#L22

otherwise this recipe could be used for single machine only.

Alex

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Alexander Smirnov]

On 02/13/2018 10:03 AM, Jan Kiszka wrote:
> On 2018-02-12 19:56, Alexander Smirnov wrote:
>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> With this class, it becomes almost trivial to replace the distro kernel
>>> with a custom build. You just need to inherit linux-kernel, specify the
>>> source URI, define via S where the source is unpacked to and provide a
>>> defconfig. To switch to a custom kernel recipe,
>>> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf or
>>> the distro conf.
>>>
>>> The approach works internally by first running "make deb-pkg" on the
>>> kernel and the repackages the output to make the binary linux-image and
>>> linux-header debs act as replacement of their distro packages. This
>>> results in a suboptimal technical implementation which may eventually be
>>> replaced by an isar-implemented deb-pkg build process. However, this is
>>> not expected to affect the user-visible interface of this class.
>>>
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>    meta/classes/linux-kernel.bbclass | 98
>>> +++++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 98 insertions(+)
>>>    create mode 100644 meta/classes/linux-kernel.bbclass
>>>
>>> diff --git a/meta/classes/linux-kernel.bbclass
>>> b/meta/classes/linux-kernel.bbclass
>>> new file mode 100644
>>> index 0000000..5f4df3f
>>> --- /dev/null
>>> +++ b/meta/classes/linux-kernel.bbclass
>>> @@ -0,0 +1,98 @@
>>> +# Custom kernel build
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2018
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +DESCRIPTION ?= "Custom kernel"
>>> +PROVIDES = "virtual/kernel"
>>> +
>>> +inherit dpkg-base
>>> +
>>> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
>>> kmod, linux-base (>= 4.3~)"
>>> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
>>> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
>>> +
>>> +REPACK_DIR = "${PP}/repack"
>>> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
>>> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
>>> +
>>> +dpkg_runbuild() {
>>> +    E="${@ bb.utils.export_proxies(d)}"
>>> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
>>> +        set -e
>>> +
>>> +        apt-get install -y -o Debug::pkgProblemResolver=yes \
>>> +            --no-install-recommends ${KBUILD_DEPENDS}
>>> +
>>> +        cd ${PP}/${S}
>>> +        cp ../defconfig .config
>>> +        make olddefconfig
>>> +
>>> +        make -j ${@bb.utils.cpu_count() * 2} deb-pkg
>>> +
>>> +        rm -rf ${REPACK_DIR}
>>> +        mkdir -p ${REPACK_DIR}
>>> +        mkdir -p ${REPACK_LINUX_IMAGE_DIR}
>>> +        mkdir -p ${REPACK_LINUX_HEADERS_DIR}
>>> +
>>> +        cd ${PP}
>>> +        tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
>>> +        dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>> +            ${REPACK_LINUX_IMAGE_DIR}
>>> +        dpkg-deb -R linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>> +            ${REPACK_LINUX_HEADERS_DIR}
>>> +
>>> +        dpkg-gencontrol -crepack/debian/control \
>>> +            -lrepack/debian/changelog \
>>> +            -frepack/debian/files \
>>> +            -plinux-image-${PV} \
>>> +            -P${REPACK_LINUX_IMAGE_DIR} \
>>> +            -DPackage="linux-image-${KERNEL_ARCH}" \
>>> +            -DSection=kernel \
>>> +            -DPriority=required \
>>> +            -DProvides="${PN}" \
>>> +            -DDepends="${KERNEL_DEBIAN_DEPENDS}"
>>> +
>>> +        # Add Debian-like link installation to postinst
>>> +        touch ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
>>> +            -e "/^set -e$/a\\
>>> +\\
>>> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
>>> +    change=install\\
>>> +else\\
>>> +    change=upgrade\\
>>> +fi\\
>>> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
>>> +rm -f /lib/modules/${PV}/.fresh-install"
>>> +
>>> +        # Add Debian-like link removal to postrm
>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
>>> +            -e "/^set -e$/a\\
>>> +\\
>>> +rm -f /lib/modules/${PV}/.fresh-install\\
>>> +\\
>>> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks
>>>> /dev/null; then\\
>>> +    linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
>>> +fi"
>>> +
>>> +        dpkg-gencontrol -crepack/debian/control \
>>> +            -lrepack/debian/changelog \
>>> +            -frepack/debian/files \
>>> +            -plinux-headers-${PV} \
>>> +            -P${REPACK_LINUX_HEADERS_DIR} \
>>> +            -Vkernel:debarch="${KERNEL_ARCH}" \
>>> +            -DPackage="linux-headers-${KERNEL_ARCH}" \
>>> +            -DSection=kernel \
>>> +            -DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
>>> +
>>> +        dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
>>> +            linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>> +        rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>> +        dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
>>> +            linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>> +        rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>> +    '
>>> +}
>>>
>>
>> What is the benefit of having this script in the class? I see that we
>> have separate files for:
>>
>>   - buildchroot
>>   - kernel modules
>>
>> Why this chroot script should be a part of bitbake class? This makes
>> debugging much more complicated, because I have no possibility to run
>> this script manually from buildchroot (like I could do with build.sh)
> 
> I played with it, and it "just" took 6 exports to set up the interface
> to script-based build. That will likely not be convenient for debugging
> as well (and I never had to call things manually that way, I got all
> information from logs), but I can keep that pattern. Will also mean:
> back to an include, rather than using a class (no big issue).
> 
>>
>> Also formatting style looks nasty here.
> 
> That's not going to change unless you suggest some alternative for the
> sed-based code injection into postinst/postrm.

Just for sure that I understand the issue, why no sed injections? You 
already do this for kernel module by "sedding" control file.

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-13 09:09, Alexander Smirnov wrote:
> 
> 
> On 02/13/2018 10:22 AM, Jan Kiszka wrote:
>> On 2018-02-12 19:51, Alexander Smirnov wrote:
>>> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>>>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>>>
>>>>>
>>>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>>>> Debian or
>>>>>>>>>>>>> other
>>>>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>>>>> for now
>>>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>>>> buildchroot so
>>>>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>>>>> image
>>>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>>>> during the
>>>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to
>>>>>>>>>>>>> ensure
>>>>>>>>>>>>> that
>>>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>>>> use-case for
>>>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>>>> prioritization could be resolved by specifying suffix to
>>>>>>>>>>>> version.
>>>>>>>>>>>
>>>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>>>
>>>>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>>>>> target),
>>>>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>>>>> even
>>>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>>>> break
>>>>>>>>>>> when
>>>>>>>>>>> that virtual package linux-cip is selected but can't be
>>>>>>>>>>> installed
>>>>>>>>>>> due to
>>>>>>>>>>> the version preference "newest").
>>>>>>>>>>
>>>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>>>> Having
>>>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>>>> kernel,
>>>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>>>
>>>>>>>> To make it clearer, just look at the example-module case: The
>>>>>>>> recipe
>>>>>>>> can
>>>>>>>> build both against the distro as well as all custom kernels,
>>>>>>>> without
>>>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>>>> relationships.
>>>>>>>
>>>>>>> kernel<->module pair are built from the same source tree, so here no
>>>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've
>>>>>>> parsed
>>>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>>>> correlation with upstream packages.
>>>>>>
>>>>>> No, they can be independent, just like in the example I submitted.
>>>>>> It's
>>>>>> an external kernel module vs. some kernel. Please re-read.
>>>>>>
>>>>>> For Debian, this specific problem does not exist because Debian only
>>>>>> ships in-tree modules or those that are built on the target on
>>>>>> installation. But you will find the exact same pattern when browsing
>>>>>> external projects that come with modules.
>>>>>
>>>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>>>> this variable is used to bind kernel<->module build process. Sometimes
>>>>> KERNEL_ARCH has to be changed according to your kernel configuration.
>>>>> It's not enough to change preferred_provider from "linux-distro" to
>>>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>>>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>>>
>>>> OK, but this cannot be defined by the kernel recipe. It has to be
>>>> defined by the machine setting. A kernel recipe could then select a
>>>> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
>>>> in this direction, that is a valid point for future improvements.
>>>>
>>>>>
>>>>> How you see the overall procedure to switch to custom kernel? Why
>>>>
>>>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>>>
>>>> See also
>>>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>>>
>>>>
>>>>
>>>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>>>
>>>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>>>> configured. Changing the kernel source doesn't change that in any way.
>>>
>>> It doesn't remain for all the cases. ARM debian provides two kernel
>>> images:
>>>
>>>   - linux-image-armmp
>>>   - linux-image-armmp-lpae
>>>
>>> Current Isar doesn't handle this, but if you build your kernel with
>>> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
>>> there will be some inaccuracy. So the same machine could have several
>>> KERNEL_ARCHs.
>>
>> In fact, one machine would have to have multiple configs, selecting the
>> kernel flavor. In OE terms, that would make different machines, though
>> targeting the same hardware.
>>
>>>
>>> Then, what should I write in KERNEL_ARCH if I add new machine, let's say
>>> Cubieboard?
>>>
>>> In general, KERNEL_ARCH is not exactly what it's used here for.
>>> According to the:
>>>
>>> https://kernel-handbook.alioth.debian.org/ch-packaging.html
>>>
>>> the linux image name looks like the following:
>>>
>>>   - linux-image[-featureset]-flavour
>>>
>>> Also chapter 3.3 stays:
>>>
>>> 8<--
>>> Each architecture usually has multiple flavours of the binary kernel
>>> images. Different flavours correspond to different kernel configuration
>>> files, used to build the binary images from the same kernel tree.
>>> 8<--
>>>
>>> So having KERNEL_ARCH as the constant is deviation from Debian flow.
>>>
>>> In addition:
>>>
>>> 8<--
>>> Again, multiple flavours of binary images may be built from the
>>> featureset tree. For example, the i386 architecture has a number of
>>> different flavours, such as 686 and 686-pae, built from the common
>>> Debian kernel source. It also contains the rt featureset. The source
>>> tree for building the kernels for each of these featuresets is obtained
>>> by applying additional patches to the Debian kernel source. It may be
>>> used to build the rt-686-pae binary image flavours.
>>> 8<--
>>>
>>> So in my opinion, CIP is a feature set.
>>>
>>> I don't really think, that replacing upstream Debian kernel package by
>>> custom one and claiming that is the same will make Debian community
>>> happy...
>>>
>>>
>>> Ok, I see that we have different vision on this topic, I could stay with
>>> this implementation for now, but in general in my opinion this looks
>>> like a big hack and should be reworked.
>>
>> Debian has a strong word here, though it complicates things
>> significantly. But let me look into expressing the kernel selection via
>> KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
>> used, though. And we will have to generate / parameterize the control
>> files of all module packages. Maybe that's Debian intention as well.
>>
>>>
>>>>
>>>>>
>>>>> For example armbian uses branch names and machines to distinguish
>>>>> images:
>>>>>
>>>>> https://www.armbian.com/kernel/
>>>>>
>>>>>    - linux-image-dev-sun8i
>>>>>    - linux-image-next-odroidc2
>>>>>
>>>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>>>> conflicts with the upstream. Also this doesn't affect modules building
>>>>> at all.
>>>>
>>>> This is about conflict: We are *replacing* the distro kernel. But when
>>>> doing this, we do not want to change all its consumers. Armbian likely
>>>> never ran into the use case of having to design such a full
>>>> replacement,
>>>> also regarding dependencies (I do not see them shipping any out-of-tree
>>>> module packages).
>>>>
>>>> And the same is true for all the non-kernel cases we will face when
>>>> replacing library-like packages. Please do not focus all your thoughts
>>>> on the kernel. This issue is absolutely generic.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> What are the other use-cases for this pattern? As I mentioned above
>>>>>>> with
>>>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>>>> broken at
>>>>>>> anytime. This could become robust *only* after build reproducibility
>>>>>>> integration, when you cache the upstream content. But with
>>>>>>> build-rep you
>>>>>>> could manually manage the content of the cache, so prioritization
>>>>>>> is not
>>>>>>> needed, you could just drop duplications from the cache.
>>>>>>
>>>>>> Every lib-dependency is another pattern for the special kernel/module
>>>>>> case: Every distro package that come with a dependencies that you
>>>>>> want
>>>>>> to replace with a custom build AND you have good reason to downgrade
>>>>>> that package, then you want your own packages to have higher prior.
>>>>>> That's why this hard-coding is the right thing to do, in general: If
>>>>>> you
>>>>>> are building a package that competes with upstream, there must be a
>>>>>> reason...
>>>>>>
>>>>>>>
>>>>>>> In general having packages with the same name in 'source.list' and
>>>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>>>
>>>>>>> If the current series requires prioritization for kernel only, it
>>>>>>> would
>>>>>>> be nice to specify only this package in apt preferences.
>>>>>>
>>>>>> See above, the kernel is just the first case. There will be more, I'm
>>>>>> absolutely sure.
>>>>>
>>>>> I've asked especially about these cases. :-( At the moment there is
>>>>> only
>>>>> kernel case, which could be solved without this prioritization.
>>>>
>>>> Use your imagination: enforced replacement of ANY package we rebuild,
>>>> but with a lower version. We rebuild packages for a reason. That
>>>> implies
>>>> we want them to be actually used. And that means our packages require a
>>>> higher priority.
>>>>
>>>
>>> My imagination told me how it will be difficult to debug segfaults on
>>> the target, because I pinned old libA in Isar-apt, while upstream
>>> dependents already have been compiled according the new version of libA.
>>> It wouldn't be so funny if nobody had faced with such problems in the
>>> past...
>>
>> Again: One guy shooting himself into the foot by explicitly stating this
>> incorrect setup doesn't mean all others users have to give up on this
>> valid and required default property of self-build packages. It remains a
>> fact that self-built packages are there to customize and overwrite
>> upstream packages, independent of how the kernel will look like. That's
>> one key purpose of Isar.
>>
> 
> I'm not against package overwriting. I mean that there is *not the only
> one way* to do this. Apt provides way to overwrite packages via versions.
> 
> How I see the creation of robust derivatives:
> 
> 1. Create local copy for upstream apt (let's name it base-apt) with
> packages you are going to use for your build.
> 2. Choose package you want to modify, let's say libA.
> 3. Create recipe for it and update version to be ">" than original one.
> 4. Work with this environment.

This doesn't solve intentional downgrades, like the kernel example would
have been one.

> 
> This approach at least somehow guarantees that you will have
> reproducible build. The packages in your base-apt are built according to
> the libA version you've customized. So my opinion: user could overwrite
> only package in local mirrors, i.e. mirrors that could not be updated by
> 3rd parties.
> 
> 
> What could happen bad with pinning over upstream (libA is from upstream):
> 
> 1. libA.h v1.0 had:
> 
> #define BUFFER_SIZE 32
> 
> 2. libA.h v1.1 has:
> 
> #define BUFFER_SIZE 16
> 
> 3. Upstream contains packages that were already rebuilt according to the
> libA v1.1
> 
> 4. You are still using libA v1.0 derivative.
> 
> 5. Buffer overrun could occur now in any libA consumer from the upstream.
> 
> So, there is *no* robust way to overwrite package from live upstream,
> you are always in the risk.

Of course, if you overwrite something in an incompatible way, you have
to ensure updating all dependencies as well. That is the mistake done in
your example.

> 
> I could imaging that some business units may require this apt
> prioritization for their specific product's needs, so lets have this
> feature in Isar. But I think it shouldn't be enabled by default. Also
> the user documentation for this feature should describe all the risks
> that user takes if this option is enabled.

Let's see how flexible the repo and package pinning configuration will
become with Claudius approach, and then we may switch to an explicit per
package enabling as well. More intuitive is the default I propose.

> 
>> If I manage to model the kernel differently, I will probably be able to
>> drop this patch from /this/ series. But the topic will only come up
>> again when Claudius proposes his multistrap rework which will include
>> shipping aptpreferences as well - and prioritizing our own repo properly.
>>
> 
> As I said, I could stay with this now to move forward. So the things
> discussed above could be fixed later.

User-visible changes should not be lightly merged. So this discussion is
valuable when it helps to come up with consistent and sustainable
interfaces. So far, the user was not yet enough in our focus, and that
is hopefully changing now.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Jan Kiszka]

On 2018-02-13 09:19, Alexander Smirnov wrote:
> 
> 
> On 02/13/2018 10:03 AM, Jan Kiszka wrote:
>> On 2018-02-12 19:56, Alexander Smirnov wrote:
>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> With this class, it becomes almost trivial to replace the distro kernel
>>>> with a custom build. You just need to inherit linux-kernel, specify the
>>>> source URI, define via S where the source is unpacked to and provide a
>>>> defconfig. To switch to a custom kernel recipe,
>>>> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf or
>>>> the distro conf.
>>>>
>>>> The approach works internally by first running "make deb-pkg" on the
>>>> kernel and the repackages the output to make the binary linux-image and
>>>> linux-header debs act as replacement of their distro packages. This
>>>> results in a suboptimal technical implementation which may
>>>> eventually be
>>>> replaced by an isar-implemented deb-pkg build process. However, this is
>>>> not expected to affect the user-visible interface of this class.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>    meta/classes/linux-kernel.bbclass | 98
>>>> +++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 98 insertions(+)
>>>>    create mode 100644 meta/classes/linux-kernel.bbclass
>>>>
>>>> diff --git a/meta/classes/linux-kernel.bbclass
>>>> b/meta/classes/linux-kernel.bbclass
>>>> new file mode 100644
>>>> index 0000000..5f4df3f
>>>> --- /dev/null
>>>> +++ b/meta/classes/linux-kernel.bbclass
>>>> @@ -0,0 +1,98 @@
>>>> +# Custom kernel build
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (c) Siemens AG, 2018
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +
>>>> +DESCRIPTION ?= "Custom kernel"
>>>> +PROVIDES = "virtual/kernel"
>>>> +
>>>> +inherit dpkg-base
>>>> +
>>>> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
>>>> kmod, linux-base (>= 4.3~)"
>>>> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
>>>> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
>>>> +
>>>> +REPACK_DIR = "${PP}/repack"
>>>> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
>>>> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
>>>> +
>>>> +dpkg_runbuild() {
>>>> +    E="${@ bb.utils.export_proxies(d)}"
>>>> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
>>>> +        set -e
>>>> +
>>>> +        apt-get install -y -o Debug::pkgProblemResolver=yes \
>>>> +            --no-install-recommends ${KBUILD_DEPENDS}
>>>> +
>>>> +        cd ${PP}/${S}
>>>> +        cp ../defconfig .config
>>>> +        make olddefconfig
>>>> +
>>>> +        make -j ${@bb.utils.cpu_count() * 2} deb-pkg
>>>> +
>>>> +        rm -rf ${REPACK_DIR}
>>>> +        mkdir -p ${REPACK_DIR}
>>>> +        mkdir -p ${REPACK_LINUX_IMAGE_DIR}
>>>> +        mkdir -p ${REPACK_LINUX_HEADERS_DIR}
>>>> +
>>>> +        cd ${PP}
>>>> +        tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
>>>> +        dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>>> +            ${REPACK_LINUX_IMAGE_DIR}
>>>> +        dpkg-deb -R linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>>> +            ${REPACK_LINUX_HEADERS_DIR}
>>>> +
>>>> +        dpkg-gencontrol -crepack/debian/control \
>>>> +            -lrepack/debian/changelog \
>>>> +            -frepack/debian/files \
>>>> +            -plinux-image-${PV} \
>>>> +            -P${REPACK_LINUX_IMAGE_DIR} \
>>>> +            -DPackage="linux-image-${KERNEL_ARCH}" \
>>>> +            -DSection=kernel \
>>>> +            -DPriority=required \
>>>> +            -DProvides="${PN}" \
>>>> +            -DDepends="${KERNEL_DEBIAN_DEPENDS}"
>>>> +
>>>> +        # Add Debian-like link installation to postinst
>>>> +        touch
>>>> ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
>>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
>>>> +            -e "/^set -e$/a\\
>>>> +\\
>>>> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
>>>> +    change=install\\
>>>> +else\\
>>>> +    change=upgrade\\
>>>> +fi\\
>>>> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
>>>> +rm -f /lib/modules/${PV}/.fresh-install"
>>>> +
>>>> +        # Add Debian-like link removal to postrm
>>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
>>>> +            -e "/^set -e$/a\\
>>>> +\\
>>>> +rm -f /lib/modules/${PV}/.fresh-install\\
>>>> +\\
>>>> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks
>>>>> /dev/null; then\\
>>>> +    linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
>>>> +fi"
>>>> +
>>>> +        dpkg-gencontrol -crepack/debian/control \
>>>> +            -lrepack/debian/changelog \
>>>> +            -frepack/debian/files \
>>>> +            -plinux-headers-${PV} \
>>>> +            -P${REPACK_LINUX_HEADERS_DIR} \
>>>> +            -Vkernel:debarch="${KERNEL_ARCH}" \
>>>> +            -DPackage="linux-headers-${KERNEL_ARCH}" \
>>>> +            -DSection=kernel \
>>>> +            -DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
>>>> +
>>>> +        dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
>>>> +            linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
>>>> +            linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +    '
>>>> +}
>>>>
>>>
>>> What is the benefit of having this script in the class? I see that we
>>> have separate files for:
>>>
>>>   - buildchroot
>>>   - kernel modules
>>>
>>> Why this chroot script should be a part of bitbake class? This makes
>>> debugging much more complicated, because I have no possibility to run
>>> this script manually from buildchroot (like I could do with build.sh)
>>
>> I played with it, and it "just" took 6 exports to set up the interface
>> to script-based build. That will likely not be convenient for debugging
>> as well (and I never had to call things manually that way, I got all
>> information from logs), but I can keep that pattern. Will also mean:
>> back to an include, rather than using a class (no big issue).
>>
>>>
>>> Also formatting style looks nasty here.
>>
>> That's not going to change unless you suggest some alternative for the
>> sed-based code injection into postinst/postrm.
> 
> Just for sure that I understand the issue, why no sed injections? You
> already do this for kernel module by "sedding" control file.

The sed code is visually the most ugliest part of my build script.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 6/8] Add custom kernel examples

[Jan Kiszka]

On 2018-02-13 09:16, Alexander Smirnov wrote:
> On 02/13/2018 10:03 AM, Jan Kiszka wrote:
>> On 2018-02-12 20:02, Alexander Smirnov wrote:
>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> Provide two examples for describing custom kernel builds. Both are for
>>>> x86_64, simply using upstream x86_64_defconfig to define the required
>>>> defconfig. One example is demoing a build from a tarball, the other
>>>> pulling from git.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>    meta-isar/recipes-kernel/linux/files/defconfig     | 308
>>>> +++++++++++++++++++++
>>>>    meta-isar/recipes-kernel/linux/linux-cip_4.4.bb    |  17 ++
>>>>    .../recipes-kernel/linux/linux-mainline_4.14.18.bb |  15 +
>>>>    3 files changed, 340 insertions(+)
>>>>    create mode 100644 meta-isar/recipes-kernel/linux/files/defconfig
>>>>    create mode 100644 meta-isar/recipes-kernel/linux/linux-cip_4.4.bb
>>>>    create mode 100644
>>>> meta-isar/recipes-kernel/linux/linux-mainline_4.14.18.bb
>>>>
>>>> diff --git a/meta-isar/recipes-kernel/linux/files/defconfig
>>>> b/meta-isar/recipes-kernel/linux/files/defconfig
>>>> new file mode 100644
>>>> index 0000000..e32fc1f
>>>> --- /dev/null
>>>> +++ b/meta-isar/recipes-kernel/linux/files/defconfig
>>>> @@ -0,0 +1,308 @@
>>>
>>> Do you have any concerns if I add a comment here from where this config
>>> was taken? Sometimes it would be very helpful to understand what this
>>> config contains, at least which arch.
>>
>> Done.
> 
> What do you mean by "done"? :-)
> 
> BTW: in future it would nice to implement configurable defconfig like
> it's done in Yocto via variable, for example:
> 
> https://github.com/Freescale/meta-freescale/blob/master/conf/machine/t1042d4rdb-64b.conf#L22
> 
> 
> otherwise this recipe could be used for single machine only.

See
https://groups.google.com/d/msgid/isar-users/8485f265-30ea-41c0-e19d-d807c6005925%40siemens.com

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov]

On 02/13/2018 11:22 AM, Jan Kiszka wrote:
> On 2018-02-13 09:09, Alexander Smirnov wrote:
>>
>>
>> On 02/13/2018 10:22 AM, Jan Kiszka wrote:
>>> On 2018-02-12 19:51, Alexander Smirnov wrote:
>>>> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>>>>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>>>>
>>>>>>
>>>>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>>>>> Debian or
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>>>>>> for now
>>>>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>>>>> buildchroot so
>>>>>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>>>>>> image
>>>>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>>>>> during the
>>>>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to
>>>>>>>>>>>>>> ensure
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>>>>> use-case for
>>>>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>>>>> prioritization could be resolved by specifying suffix to
>>>>>>>>>>>>> version.
>>>>>>>>>>>>
>>>>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>>>>
>>>>>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>>>>>> target),
>>>>>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>>>>>> even
>>>>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>>>>> break
>>>>>>>>>>>> when
>>>>>>>>>>>> that virtual package linux-cip is selected but can't be
>>>>>>>>>>>> installed
>>>>>>>>>>>> due to
>>>>>>>>>>>> the version preference "newest").
>>>>>>>>>>>
>>>>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>>>>> Having
>>>>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>>>>> kernel,
>>>>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>>>>
>>>>>>>>> To make it clearer, just look at the example-module case: The
>>>>>>>>> recipe
>>>>>>>>> can
>>>>>>>>> build both against the distro as well as all custom kernels,
>>>>>>>>> without
>>>>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>>>>> relationships.
>>>>>>>>
>>>>>>>> kernel<->module pair are built from the same source tree, so here no
>>>>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've
>>>>>>>> parsed
>>>>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>>>>> correlation with upstream packages.
>>>>>>>
>>>>>>> No, they can be independent, just like in the example I submitted.
>>>>>>> It's
>>>>>>> an external kernel module vs. some kernel. Please re-read.
>>>>>>>
>>>>>>> For Debian, this specific problem does not exist because Debian only
>>>>>>> ships in-tree modules or those that are built on the target on
>>>>>>> installation. But you will find the exact same pattern when browsing
>>>>>>> external projects that come with modules.
>>>>>>
>>>>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>>>>> this variable is used to bind kernel<->module build process. Sometimes
>>>>>> KERNEL_ARCH has to be changed according to your kernel configuration.
>>>>>> It's not enough to change preferred_provider from "linux-distro" to
>>>>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>>>>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>>>>
>>>>> OK, but this cannot be defined by the kernel recipe. It has to be
>>>>> defined by the machine setting. A kernel recipe could then select a
>>>>> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
>>>>> in this direction, that is a valid point for future improvements.
>>>>>
>>>>>>
>>>>>> How you see the overall procedure to switch to custom kernel? Why
>>>>>
>>>>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>>>>
>>>>> See also
>>>>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>>>>
>>>>>
>>>>>
>>>>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>>>>
>>>>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>>>>> configured. Changing the kernel source doesn't change that in any way.
>>>>
>>>> It doesn't remain for all the cases. ARM debian provides two kernel
>>>> images:
>>>>
>>>>    - linux-image-armmp
>>>>    - linux-image-armmp-lpae
>>>>
>>>> Current Isar doesn't handle this, but if you build your kernel with
>>>> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
>>>> there will be some inaccuracy. So the same machine could have several
>>>> KERNEL_ARCHs.
>>>
>>> In fact, one machine would have to have multiple configs, selecting the
>>> kernel flavor. In OE terms, that would make different machines, though
>>> targeting the same hardware.
>>>
>>>>
>>>> Then, what should I write in KERNEL_ARCH if I add new machine, let's say
>>>> Cubieboard?
>>>>
>>>> In general, KERNEL_ARCH is not exactly what it's used here for.
>>>> According to the:
>>>>
>>>> https://kernel-handbook.alioth.debian.org/ch-packaging.html
>>>>
>>>> the linux image name looks like the following:
>>>>
>>>>    - linux-image[-featureset]-flavour
>>>>
>>>> Also chapter 3.3 stays:
>>>>
>>>> 8<--
>>>> Each architecture usually has multiple flavours of the binary kernel
>>>> images. Different flavours correspond to different kernel configuration
>>>> files, used to build the binary images from the same kernel tree.
>>>> 8<--
>>>>
>>>> So having KERNEL_ARCH as the constant is deviation from Debian flow.
>>>>
>>>> In addition:
>>>>
>>>> 8<--
>>>> Again, multiple flavours of binary images may be built from the
>>>> featureset tree. For example, the i386 architecture has a number of
>>>> different flavours, such as 686 and 686-pae, built from the common
>>>> Debian kernel source. It also contains the rt featureset. The source
>>>> tree for building the kernels for each of these featuresets is obtained
>>>> by applying additional patches to the Debian kernel source. It may be
>>>> used to build the rt-686-pae binary image flavours.
>>>> 8<--
>>>>
>>>> So in my opinion, CIP is a feature set.
>>>>
>>>> I don't really think, that replacing upstream Debian kernel package by
>>>> custom one and claiming that is the same will make Debian community
>>>> happy...
>>>>
>>>>
>>>> Ok, I see that we have different vision on this topic, I could stay with
>>>> this implementation for now, but in general in my opinion this looks
>>>> like a big hack and should be reworked.
>>>
>>> Debian has a strong word here, though it complicates things
>>> significantly. But let me look into expressing the kernel selection via
>>> KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
>>> used, though. And we will have to generate / parameterize the control
>>> files of all module packages. Maybe that's Debian intention as well.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> For example armbian uses branch names and machines to distinguish
>>>>>> images:
>>>>>>
>>>>>> https://www.armbian.com/kernel/
>>>>>>
>>>>>>     - linux-image-dev-sun8i
>>>>>>     - linux-image-next-odroidc2
>>>>>>
>>>>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>>>>> conflicts with the upstream. Also this doesn't affect modules building
>>>>>> at all.
>>>>>
>>>>> This is about conflict: We are *replacing* the distro kernel. But when
>>>>> doing this, we do not want to change all its consumers. Armbian likely
>>>>> never ran into the use case of having to design such a full
>>>>> replacement,
>>>>> also regarding dependencies (I do not see them shipping any out-of-tree
>>>>> module packages).
>>>>>
>>>>> And the same is true for all the non-kernel cases we will face when
>>>>> replacing library-like packages. Please do not focus all your thoughts
>>>>> on the kernel. This issue is absolutely generic.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> What are the other use-cases for this pattern? As I mentioned above
>>>>>>>> with
>>>>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>>>>> broken at
>>>>>>>> anytime. This could become robust *only* after build reproducibility
>>>>>>>> integration, when you cache the upstream content. But with
>>>>>>>> build-rep you
>>>>>>>> could manually manage the content of the cache, so prioritization
>>>>>>>> is not
>>>>>>>> needed, you could just drop duplications from the cache.
>>>>>>>
>>>>>>> Every lib-dependency is another pattern for the special kernel/module
>>>>>>> case: Every distro package that come with a dependencies that you
>>>>>>> want
>>>>>>> to replace with a custom build AND you have good reason to downgrade
>>>>>>> that package, then you want your own packages to have higher prior.
>>>>>>> That's why this hard-coding is the right thing to do, in general: If
>>>>>>> you
>>>>>>> are building a package that competes with upstream, there must be a
>>>>>>> reason...
>>>>>>>
>>>>>>>>
>>>>>>>> In general having packages with the same name in 'source.list' and
>>>>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>>>>
>>>>>>>> If the current series requires prioritization for kernel only, it
>>>>>>>> would
>>>>>>>> be nice to specify only this package in apt preferences.
>>>>>>>
>>>>>>> See above, the kernel is just the first case. There will be more, I'm
>>>>>>> absolutely sure.
>>>>>>
>>>>>> I've asked especially about these cases. :-( At the moment there is
>>>>>> only
>>>>>> kernel case, which could be solved without this prioritization.
>>>>>
>>>>> Use your imagination: enforced replacement of ANY package we rebuild,
>>>>> but with a lower version. We rebuild packages for a reason. That
>>>>> implies
>>>>> we want them to be actually used. And that means our packages require a
>>>>> higher priority.
>>>>>
>>>>
>>>> My imagination told me how it will be difficult to debug segfaults on
>>>> the target, because I pinned old libA in Isar-apt, while upstream
>>>> dependents already have been compiled according the new version of libA.
>>>> It wouldn't be so funny if nobody had faced with such problems in the
>>>> past...
>>>
>>> Again: One guy shooting himself into the foot by explicitly stating this
>>> incorrect setup doesn't mean all others users have to give up on this
>>> valid and required default property of self-build packages. It remains a
>>> fact that self-built packages are there to customize and overwrite
>>> upstream packages, independent of how the kernel will look like. That's
>>> one key purpose of Isar.
>>>
>>
>> I'm not against package overwriting. I mean that there is *not the only
>> one way* to do this. Apt provides way to overwrite packages via versions.
>>
>> How I see the creation of robust derivatives:
>>
>> 1. Create local copy for upstream apt (let's name it base-apt) with
>> packages you are going to use for your build.
>> 2. Choose package you want to modify, let's say libA.
>> 3. Create recipe for it and update version to be ">" than original one.
>> 4. Work with this environment.
> 
> This doesn't solve intentional downgrades, like the kernel example would
> have been one.
> 
>>
>> This approach at least somehow guarantees that you will have
>> reproducible build. The packages in your base-apt are built according to
>> the libA version you've customized. So my opinion: user could overwrite
>> only package in local mirrors, i.e. mirrors that could not be updated by
>> 3rd parties.
>>
>>
>> What could happen bad with pinning over upstream (libA is from upstream):
>>
>> 1. libA.h v1.0 had:
>>
>> #define BUFFER_SIZE 32
>>
>> 2. libA.h v1.1 has:
>>
>> #define BUFFER_SIZE 16
>>
>> 3. Upstream contains packages that were already rebuilt according to the
>> libA v1.1
>>
>> 4. You are still using libA v1.0 derivative.
>>
>> 5. Buffer overrun could occur now in any libA consumer from the upstream.
>>
>> So, there is *no* robust way to overwrite package from live upstream,
>> you are always in the risk.
> 
> Of course, if you overwrite something in an incompatible way, you have
> to ensure updating all dependencies as well. That is the mistake done in
> your example.
> 
>>
>> I could imaging that some business units may require this apt
>> prioritization for their specific product's needs, so lets have this
>> feature in Isar. But I think it shouldn't be enabled by default. Also
>> the user documentation for this feature should describe all the risks
>> that user takes if this option is enabled.
> 
> Let's see how flexible the repo and package pinning configuration will
> become with Claudius approach, and then we may switch to an explicit per
> package enabling as well. More intuitive is the default I propose.
> 
>>
>>> If I manage to model the kernel differently, I will probably be able to
>>> drop this patch from /this/ series. But the topic will only come up
>>> again when Claudius proposes his multistrap rework which will include
>>> shipping aptpreferences as well - and prioritizing our own repo properly.
>>>
>>
>> As I said, I could stay with this now to move forward. So the things
>> discussed above could be fixed later.
> 
> User-visible changes should not be lightly merged. So this discussion is
> valuable when it helps to come up with consistent and sustainable
> interfaces. So far, the user was not yet enough in our focus, and that
> is hopefully changing now.
> 

What are the next steps with this series?
Anyway I want at least patches 1 and 3 from it.

Alex

Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka]

On 2018-02-13 09:45, Alexander Smirnov wrote:
> 
> 
> On 02/13/2018 11:22 AM, Jan Kiszka wrote:
>> On 2018-02-13 09:09, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/13/2018 10:22 AM, Jan Kiszka wrote:
>>>> On 2018-02-12 19:51, Alexander Smirnov wrote:
>>>>> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>>>>>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>>>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>>>>>> Debian or
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>> external sources with our self-built versions. We achieve
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> for now
>>>>>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>>>>>> buildchroot so
>>>>>>>>>>>>>>> that dependency installations use the right priority. For
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> image
>>>>>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>>>>>> during the
>>>>>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to
>>>>>>>>>>>>>>> ensure
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>>>>>> use-case for
>>>>>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>>>>>> prioritization could be resolved by specifying suffix to
>>>>>>>>>>>>>> version.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>>>>>
>>>>>>>>>>>>> This generates a package called linux-image-amd64 (for
>>>>>>>>>>>>> amd64 as
>>>>>>>>>>>>> target),
>>>>>>>>>>>>> and that may have a version < debian's kernel. Now, if we
>>>>>>>>>>>>> do not
>>>>>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is
>>>>>>>>>>>>> taken,
>>>>>>>>>>>>> even
>>>>>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>>>>>> break
>>>>>>>>>>>>> when
>>>>>>>>>>>>> that virtual package linux-cip is selected but can't be
>>>>>>>>>>>>> installed
>>>>>>>>>>>>> due to
>>>>>>>>>>>>> the version preference "newest").
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>>>>>> Having
>>>>>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>>>>>> kernel,
>>>>>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>>>>>
>>>>>>>>>> To make it clearer, just look at the example-module case: The
>>>>>>>>>> recipe
>>>>>>>>>> can
>>>>>>>>>> build both against the distro as well as all custom kernels,
>>>>>>>>>> without
>>>>>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>>>>>> relationships.
>>>>>>>>>
>>>>>>>>> kernel<->module pair are built from the same source tree, so
>>>>>>>>> here no
>>>>>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've
>>>>>>>>> parsed
>>>>>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>>>>>> correlation with upstream packages.
>>>>>>>>
>>>>>>>> No, they can be independent, just like in the example I submitted.
>>>>>>>> It's
>>>>>>>> an external kernel module vs. some kernel. Please re-read.
>>>>>>>>
>>>>>>>> For Debian, this specific problem does not exist because Debian
>>>>>>>> only
>>>>>>>> ships in-tree modules or those that are built on the target on
>>>>>>>> installation. But you will find the exact same pattern when
>>>>>>>> browsing
>>>>>>>> external projects that come with modules.
>>>>>>>
>>>>>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>>>>>> this variable is used to bind kernel<->module build process.
>>>>>>> Sometimes
>>>>>>> KERNEL_ARCH has to be changed according to your kernel
>>>>>>> configuration.
>>>>>>> It's not enough to change preferred_provider from "linux-distro" to
>>>>>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also
>>>>>>> need
>>>>>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>>>>>
>>>>>> OK, but this cannot be defined by the kernel recipe. It has to be
>>>>>> defined by the machine setting. A kernel recipe could then select a
>>>>>> config that matches or reject the KERNEL_ARCH. Currently, we do
>>>>>> nothing
>>>>>> in this direction, that is a valid point for future improvements.
>>>>>>
>>>>>>>
>>>>>>> How you see the overall procedure to switch to custom kernel? Why
>>>>>>
>>>>>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>>>>>
>>>>>> See also
>>>>>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>>>>>
>>>>>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>>>>>> configured. Changing the kernel source doesn't change that in any
>>>>>> way.
>>>>>
>>>>> It doesn't remain for all the cases. ARM debian provides two kernel
>>>>> images:
>>>>>
>>>>>    - linux-image-armmp
>>>>>    - linux-image-armmp-lpae
>>>>>
>>>>> Current Isar doesn't handle this, but if you build your kernel with
>>>>> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
>>>>> there will be some inaccuracy. So the same machine could have several
>>>>> KERNEL_ARCHs.
>>>>
>>>> In fact, one machine would have to have multiple configs, selecting the
>>>> kernel flavor. In OE terms, that would make different machines, though
>>>> targeting the same hardware.
>>>>
>>>>>
>>>>> Then, what should I write in KERNEL_ARCH if I add new machine,
>>>>> let's say
>>>>> Cubieboard?
>>>>>
>>>>> In general, KERNEL_ARCH is not exactly what it's used here for.
>>>>> According to the:
>>>>>
>>>>> https://kernel-handbook.alioth.debian.org/ch-packaging.html
>>>>>
>>>>> the linux image name looks like the following:
>>>>>
>>>>>    - linux-image[-featureset]-flavour
>>>>>
>>>>> Also chapter 3.3 stays:
>>>>>
>>>>> 8<--
>>>>> Each architecture usually has multiple flavours of the binary kernel
>>>>> images. Different flavours correspond to different kernel
>>>>> configuration
>>>>> files, used to build the binary images from the same kernel tree.
>>>>> 8<--
>>>>>
>>>>> So having KERNEL_ARCH as the constant is deviation from Debian flow.
>>>>>
>>>>> In addition:
>>>>>
>>>>> 8<--
>>>>> Again, multiple flavours of binary images may be built from the
>>>>> featureset tree. For example, the i386 architecture has a number of
>>>>> different flavours, such as 686 and 686-pae, built from the common
>>>>> Debian kernel source. It also contains the rt featureset. The source
>>>>> tree for building the kernels for each of these featuresets is
>>>>> obtained
>>>>> by applying additional patches to the Debian kernel source. It may be
>>>>> used to build the rt-686-pae binary image flavours.
>>>>> 8<--
>>>>>
>>>>> So in my opinion, CIP is a feature set.
>>>>>
>>>>> I don't really think, that replacing upstream Debian kernel package by
>>>>> custom one and claiming that is the same will make Debian community
>>>>> happy...
>>>>>
>>>>>
>>>>> Ok, I see that we have different vision on this topic, I could stay
>>>>> with
>>>>> this implementation for now, but in general in my opinion this looks
>>>>> like a big hack and should be reworked.
>>>>
>>>> Debian has a strong word here, though it complicates things
>>>> significantly. But let me look into expressing the kernel selection via
>>>> KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
>>>> used, though. And we will have to generate / parameterize the control
>>>> files of all module packages. Maybe that's Debian intention as well.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> For example armbian uses branch names and machines to distinguish
>>>>>>> images:
>>>>>>>
>>>>>>> https://www.armbian.com/kernel/
>>>>>>>
>>>>>>>     - linux-image-dev-sun8i
>>>>>>>     - linux-image-next-odroidc2
>>>>>>>
>>>>>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>>>>>> conflicts with the upstream. Also this doesn't affect modules
>>>>>>> building
>>>>>>> at all.
>>>>>>
>>>>>> This is about conflict: We are *replacing* the distro kernel. But
>>>>>> when
>>>>>> doing this, we do not want to change all its consumers. Armbian
>>>>>> likely
>>>>>> never ran into the use case of having to design such a full
>>>>>> replacement,
>>>>>> also regarding dependencies (I do not see them shipping any
>>>>>> out-of-tree
>>>>>> module packages).
>>>>>>
>>>>>> And the same is true for all the non-kernel cases we will face when
>>>>>> replacing library-like packages. Please do not focus all your
>>>>>> thoughts
>>>>>> on the kernel. This issue is absolutely generic.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What are the other use-cases for this pattern? As I mentioned
>>>>>>>>> above
>>>>>>>>> with
>>>>>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>>>>>> broken at
>>>>>>>>> anytime. This could become robust *only* after build
>>>>>>>>> reproducibility
>>>>>>>>> integration, when you cache the upstream content. But with
>>>>>>>>> build-rep you
>>>>>>>>> could manually manage the content of the cache, so prioritization
>>>>>>>>> is not
>>>>>>>>> needed, you could just drop duplications from the cache.
>>>>>>>>
>>>>>>>> Every lib-dependency is another pattern for the special
>>>>>>>> kernel/module
>>>>>>>> case: Every distro package that come with a dependencies that you
>>>>>>>> want
>>>>>>>> to replace with a custom build AND you have good reason to
>>>>>>>> downgrade
>>>>>>>> that package, then you want your own packages to have higher prior.
>>>>>>>> That's why this hard-coding is the right thing to do, in
>>>>>>>> general: If
>>>>>>>> you
>>>>>>>> are building a package that competes with upstream, there must be a
>>>>>>>> reason...
>>>>>>>>
>>>>>>>>>
>>>>>>>>> In general having packages with the same name in 'source.list' and
>>>>>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>>>>>
>>>>>>>>> If the current series requires prioritization for kernel only, it
>>>>>>>>> would
>>>>>>>>> be nice to specify only this package in apt preferences.
>>>>>>>>
>>>>>>>> See above, the kernel is just the first case. There will be
>>>>>>>> more, I'm
>>>>>>>> absolutely sure.
>>>>>>>
>>>>>>> I've asked especially about these cases. :-( At the moment there is
>>>>>>> only
>>>>>>> kernel case, which could be solved without this prioritization.
>>>>>>
>>>>>> Use your imagination: enforced replacement of ANY package we rebuild,
>>>>>> but with a lower version. We rebuild packages for a reason. That
>>>>>> implies
>>>>>> we want them to be actually used. And that means our packages
>>>>>> require a
>>>>>> higher priority.
>>>>>>
>>>>>
>>>>> My imagination told me how it will be difficult to debug segfaults on
>>>>> the target, because I pinned old libA in Isar-apt, while upstream
>>>>> dependents already have been compiled according the new version of
>>>>> libA.
>>>>> It wouldn't be so funny if nobody had faced with such problems in the
>>>>> past...
>>>>
>>>> Again: One guy shooting himself into the foot by explicitly stating
>>>> this
>>>> incorrect setup doesn't mean all others users have to give up on this
>>>> valid and required default property of self-build packages. It
>>>> remains a
>>>> fact that self-built packages are there to customize and overwrite
>>>> upstream packages, independent of how the kernel will look like. That's
>>>> one key purpose of Isar.
>>>>
>>>
>>> I'm not against package overwriting. I mean that there is *not the only
>>> one way* to do this. Apt provides way to overwrite packages via
>>> versions.
>>>
>>> How I see the creation of robust derivatives:
>>>
>>> 1. Create local copy for upstream apt (let's name it base-apt) with
>>> packages you are going to use for your build.
>>> 2. Choose package you want to modify, let's say libA.
>>> 3. Create recipe for it and update version to be ">" than original one.
>>> 4. Work with this environment.
>>
>> This doesn't solve intentional downgrades, like the kernel example would
>> have been one.
>>
>>>
>>> This approach at least somehow guarantees that you will have
>>> reproducible build. The packages in your base-apt are built according to
>>> the libA version you've customized. So my opinion: user could overwrite
>>> only package in local mirrors, i.e. mirrors that could not be updated by
>>> 3rd parties.
>>>
>>>
>>> What could happen bad with pinning over upstream (libA is from
>>> upstream):
>>>
>>> 1. libA.h v1.0 had:
>>>
>>> #define BUFFER_SIZE 32
>>>
>>> 2. libA.h v1.1 has:
>>>
>>> #define BUFFER_SIZE 16
>>>
>>> 3. Upstream contains packages that were already rebuilt according to the
>>> libA v1.1
>>>
>>> 4. You are still using libA v1.0 derivative.
>>>
>>> 5. Buffer overrun could occur now in any libA consumer from the
>>> upstream.
>>>
>>> So, there is *no* robust way to overwrite package from live upstream,
>>> you are always in the risk.
>>
>> Of course, if you overwrite something in an incompatible way, you have
>> to ensure updating all dependencies as well. That is the mistake done in
>> your example.
>>
>>>
>>> I could imaging that some business units may require this apt
>>> prioritization for their specific product's needs, so lets have this
>>> feature in Isar. But I think it shouldn't be enabled by default. Also
>>> the user documentation for this feature should describe all the risks
>>> that user takes if this option is enabled.
>>
>> Let's see how flexible the repo and package pinning configuration will
>> become with Claudius approach, and then we may switch to an explicit per
>> package enabling as well. More intuitive is the default I propose.
>>
>>>
>>>> If I manage to model the kernel differently, I will probably be able to
>>>> drop this patch from /this/ series. But the topic will only come up
>>>> again when Claudius proposes his multistrap rework which will include
>>>> shipping aptpreferences as well - and prioritizing our own repo
>>>> properly.
>>>>
>>>
>>> As I said, I could stay with this now to move forward. So the things
>>> discussed above could be fixed later.
>>
>> User-visible changes should not be lightly merged. So this discussion is
>> valuable when it helps to come up with consistent and sustainable
>> interfaces. So far, the user was not yet enough in our focus, and that
>> is hopefully changing now.
>>
> 
> What are the next steps with this series?
> Anyway I want at least patches 1 and 3 from it.

Yes, feel free to pick them ahead of the rest! I will work on the others
and rebase on next when ready.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v4 0/8] Provide infrastructure and examples for custom kernels and modules

[Alexander Smirnov]

On 02/11/2018 06:25 PM, Jan Kiszka wrote:
> This is now finally the promised patches to provide simple patterns for
> building custom kernels for Isar images that replace the distro kernels.
> It also provids the same pattern for custom out-of-tree kernel modules.
> 
> Along come two fixes, one to respect proxies during dpkg builds, the
> other to adjust isar-apt repo prio.
> 
> Furthermore, I'm proposing to rename "SRC_DIR" to OE-standard "S".
> 
> Changes in v3 (only resent patch 5):
>   - allow central overriding of kernel dependencies
> 
> Changes in v4
>   - simplified changes to isar-image-base.bb by reordering code
>   - modeled custom kernel build as class
> 

Applied 1 and 3. Thanks!

Alex

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Henning Schild]

Am Tue, 13 Feb 2018 08:03:12 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 2018-02-12 19:56, Alexander Smirnov wrote:
> > On 02/11/2018 06:25 PM, Jan Kiszka wrote:  
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> With this class, it becomes almost trivial to replace the distro
> >> kernel with a custom build. You just need to inherit linux-kernel,
> >> specify the source URI, define via S where the source is unpacked
> >> to and provide a defconfig. To switch to a custom kernel recipe,
> >> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf
> >> or the distro conf.
> >>
> >> The approach works internally by first running "make deb-pkg" on
> >> the kernel and the repackages the output to make the binary
> >> linux-image and linux-header debs act as replacement of their
> >> distro packages. This results in a suboptimal technical
> >> implementation which may eventually be replaced by an
> >> isar-implemented deb-pkg build process. However, this is not
> >> expected to affect the user-visible interface of this class.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>   meta/classes/linux-kernel.bbclass | 98
> >> +++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 98 insertions(+)
> >>   create mode 100644 meta/classes/linux-kernel.bbclass
> >>
> >> diff --git a/meta/classes/linux-kernel.bbclass
> >> b/meta/classes/linux-kernel.bbclass
> >> new file mode 100644
> >> index 0000000..5f4df3f
> >> --- /dev/null
> >> +++ b/meta/classes/linux-kernel.bbclass
> >> @@ -0,0 +1,98 @@
> >> +# Custom kernel build
> >> +#
> >> +# This software is a part of ISAR.
> >> +# Copyright (c) Siemens AG, 2018
> >> +#
> >> +# SPDX-License-Identifier: MIT
> >> +
> >> +DESCRIPTION ?= "Custom kernel"
> >> +PROVIDES = "virtual/kernel"
> >> +
> >> +inherit dpkg-base
> >> +
> >> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
> >> kmod, linux-base (>= 4.3~)"
> >> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
> >> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
> >> +
> >> +REPACK_DIR = "${PP}/repack"
> >> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
> >> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
> >> +
> >> +dpkg_runbuild() {
> >> +    E="${@ bb.utils.export_proxies(d)}"
> >> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
> >> +        set -e
> >> +
> >> +        apt-get install -y -o Debug::pkgProblemResolver=yes \
> >> +            --no-install-recommends ${KBUILD_DEPENDS}
> >> +
> >> +        cd ${PP}/${S}
> >> +        cp ../defconfig .config
> >> +        make olddefconfig
> >> +
> >> +        make -j ${@bb.utils.cpu_count() * 2} deb-pkg
> >> +
> >> +        rm -rf ${REPACK_DIR}
> >> +        mkdir -p ${REPACK_DIR}
> >> +        mkdir -p ${REPACK_LINUX_IMAGE_DIR}
> >> +        mkdir -p ${REPACK_LINUX_HEADERS_DIR}
> >> +
> >> +        cd ${PP}
> >> +        tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
> >> +        dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
> >> +            ${REPACK_LINUX_IMAGE_DIR}
> >> +        dpkg-deb -R
> >> linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
> >> +            ${REPACK_LINUX_HEADERS_DIR}
> >> +
> >> +        dpkg-gencontrol -crepack/debian/control \
> >> +            -lrepack/debian/changelog \
> >> +            -frepack/debian/files \
> >> +            -plinux-image-${PV} \
> >> +            -P${REPACK_LINUX_IMAGE_DIR} \
> >> +            -DPackage="linux-image-${KERNEL_ARCH}" \
> >> +            -DSection=kernel \
> >> +            -DPriority=required \
> >> +            -DProvides="${PN}" \
> >> +            -DDepends="${KERNEL_DEBIAN_DEPENDS}"
> >> +
> >> +        # Add Debian-like link installation to postinst
> >> +        touch
> >> ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
> >> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
> >> +            -e "/^set -e$/a\\
> >> +\\
> >> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
> >> +    change=install\\
> >> +else\\
> >> +    change=upgrade\\
> >> +fi\\
> >> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
> >> +rm -f /lib/modules/${PV}/.fresh-install"
> >> +
> >> +        # Add Debian-like link removal to postrm
> >> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
> >> +            -e "/^set -e$/a\\
> >> +\\
> >> +rm -f /lib/modules/${PV}/.fresh-install\\
> >> +\\
> >> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks  
> >> >/dev/null; then\\  
> >> +    linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
> >> +fi"
> >> +
> >> +        dpkg-gencontrol -crepack/debian/control \
> >> +            -lrepack/debian/changelog \
> >> +            -frepack/debian/files \
> >> +            -plinux-headers-${PV} \
> >> +            -P${REPACK_LINUX_HEADERS_DIR} \
> >> +            -Vkernel:debarch="${KERNEL_ARCH}" \
> >> +            -DPackage="linux-headers-${KERNEL_ARCH}" \
> >> +            -DSection=kernel \
> >> +            -DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
> >> +
> >> +        dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
> >> +            linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
> >> +        rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
> >> +        dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
> >> +
> >> linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
> >> +        rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
> >> +    '
> >> +}
> >>  
> > 
> > What is the benefit of having this script in the class? I see that
> > we have separate files for:
> > 
> >  - buildchroot
> >  - kernel modules
> > 
> > Why this chroot script should be a part of bitbake class? This makes
> > debugging much more complicated, because I have no possibility to
> > run this script manually from buildchroot (like I could do with
> > build.sh)  
> 
> I played with it, and it "just" took 6 exports to set up the interface
> to script-based build. That will likely not be convenient for
> debugging as well (and I never had to call things manually that way,
> I got all information from logs), but I can keep that pattern. Will
> also mean: back to an include, rather than using a class (no big
> issue).

You could write all these exports into a file that the scripts sources.
Given that file exists, one could run manually and the recipe would not
contain all that logic.

> > 
> > Also formatting style looks nasty here.  
> 
> That's not going to change unless you suggest some alternative for the
> sed-based code injection into postinst/postrm.

Can that code be appended with echo >> ? The patching looks pretty
unconditional already.

Henning
 
> Jan
> 

Re: [PATCH v4 5/8] Provide class for easy custom kernel builds

[Jan Kiszka]

On 2018-02-13 10:53, Henning Schild wrote:
> Am Tue, 13 Feb 2018 08:03:12 +0100
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> On 2018-02-12 19:56, Alexander Smirnov wrote:
>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:  
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> With this class, it becomes almost trivial to replace the distro
>>>> kernel with a custom build. You just need to inherit linux-kernel,
>>>> specify the source URI, define via S where the source is unpacked
>>>> to and provide a defconfig. To switch to a custom kernel recipe,
>>>> PREFERRED_PROVIDER_virtual/kernel has to be adjusted in local.conf
>>>> or the distro conf.
>>>>
>>>> The approach works internally by first running "make deb-pkg" on
>>>> the kernel and the repackages the output to make the binary
>>>> linux-image and linux-header debs act as replacement of their
>>>> distro packages. This results in a suboptimal technical
>>>> implementation which may eventually be replaced by an
>>>> isar-implemented deb-pkg build process. However, this is not
>>>> expected to affect the user-visible interface of this class.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>   meta/classes/linux-kernel.bbclass | 98
>>>> +++++++++++++++++++++++++++++++++++++++
>>>>   1 file changed, 98 insertions(+)
>>>>   create mode 100644 meta/classes/linux-kernel.bbclass
>>>>
>>>> diff --git a/meta/classes/linux-kernel.bbclass
>>>> b/meta/classes/linux-kernel.bbclass
>>>> new file mode 100644
>>>> index 0000000..5f4df3f
>>>> --- /dev/null
>>>> +++ b/meta/classes/linux-kernel.bbclass
>>>> @@ -0,0 +1,98 @@
>>>> +# Custom kernel build
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (c) Siemens AG, 2018
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +
>>>> +DESCRIPTION ?= "Custom kernel"
>>>> +PROVIDES = "virtual/kernel"
>>>> +
>>>> +inherit dpkg-base
>>>> +
>>>> +KERNEL_DEBIAN_DEPENDS ?= "initramfs-tools | linux-initramfs-tool,
>>>> kmod, linux-base (>= 4.3~)"
>>>> +KERNEL_HEADERS_DEBIAN_DEPENDS ?= "libc6, libssl1.1, gcc"
>>>> +KBUILD_DEPENDS ?= "libssl-dev libelf-dev bc"
>>>> +
>>>> +REPACK_DIR = "${PP}/repack"
>>>> +REPACK_LINUX_IMAGE_DIR = "${REPACK_DIR}/linux-image"
>>>> +REPACK_LINUX_HEADERS_DIR = "${REPACK_DIR}/linux-headers"
>>>> +
>>>> +dpkg_runbuild() {
>>>> +    E="${@ bb.utils.export_proxies(d)}"
>>>> +    sudo -E chroot ${BUILDCHROOT_DIR} sh -c '
>>>> +        set -e
>>>> +
>>>> +        apt-get install -y -o Debug::pkgProblemResolver=yes \
>>>> +            --no-install-recommends ${KBUILD_DEPENDS}
>>>> +
>>>> +        cd ${PP}/${S}
>>>> +        cp ../defconfig .config
>>>> +        make olddefconfig
>>>> +
>>>> +        make -j ${@bb.utils.cpu_count() * 2} deb-pkg
>>>> +
>>>> +        rm -rf ${REPACK_DIR}
>>>> +        mkdir -p ${REPACK_DIR}
>>>> +        mkdir -p ${REPACK_LINUX_IMAGE_DIR}
>>>> +        mkdir -p ${REPACK_LINUX_HEADERS_DIR}
>>>> +
>>>> +        cd ${PP}
>>>> +        tar xzf linux-${PV}_${PV}-1.debian.tar.gz -C ${REPACK_DIR}
>>>> +        dpkg-deb -R linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>>> +            ${REPACK_LINUX_IMAGE_DIR}
>>>> +        dpkg-deb -R
>>>> linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb \
>>>> +            ${REPACK_LINUX_HEADERS_DIR}
>>>> +
>>>> +        dpkg-gencontrol -crepack/debian/control \
>>>> +            -lrepack/debian/changelog \
>>>> +            -frepack/debian/files \
>>>> +            -plinux-image-${PV} \
>>>> +            -P${REPACK_LINUX_IMAGE_DIR} \
>>>> +            -DPackage="linux-image-${KERNEL_ARCH}" \
>>>> +            -DSection=kernel \
>>>> +            -DPriority=required \
>>>> +            -DProvides="${PN}" \
>>>> +            -DDepends="${KERNEL_DEBIAN_DEPENDS}"
>>>> +
>>>> +        # Add Debian-like link installation to postinst
>>>> +        touch
>>>> ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install
>>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst \
>>>> +            -e "/^set -e$/a\\
>>>> +\\
>>>> +if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
>>>> +    change=install\\
>>>> +else\\
>>>> +    change=upgrade\\
>>>> +fi\\
>>>> +linux-update-symlinks \$change ${PV} /boot/vmlinuz-${PV}\\
>>>> +rm -f /lib/modules/${PV}/.fresh-install"
>>>> +
>>>> +        # Add Debian-like link removal to postrm
>>>> +        sed -i ${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm \
>>>> +            -e "/^set -e$/a\\
>>>> +\\
>>>> +rm -f /lib/modules/${PV}/.fresh-install\\
>>>> +\\
>>>> +if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks  
>>>>> /dev/null; then\\  
>>>> +    linux-update-symlinks remove ${PV}  /boot/vmlinuz-${PV}\\
>>>> +fi"
>>>> +
>>>> +        dpkg-gencontrol -crepack/debian/control \
>>>> +            -lrepack/debian/changelog \
>>>> +            -frepack/debian/files \
>>>> +            -plinux-headers-${PV} \
>>>> +            -P${REPACK_LINUX_HEADERS_DIR} \
>>>> +            -Vkernel:debarch="${KERNEL_ARCH}" \
>>>> +            -DPackage="linux-headers-${KERNEL_ARCH}" \
>>>> +            -DSection=kernel \
>>>> +            -DDepends="${KERNEL_HEADERS_DEBIAN_DEPENDS}"
>>>> +
>>>> +        dpkg-deb -b ${REPACK_LINUX_IMAGE_DIR} \
>>>> +            linux-image-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        rm -f linux-image-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        dpkg-deb -b ${REPACK_LINUX_HEADERS_DIR} \
>>>> +
>>>> linux-headers-${KERNEL_ARCH}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +        rm -f linux-headers-${PV}_${PV}-1_${KERNEL_ARCH}.deb
>>>> +    '
>>>> +}
>>>>  
>>>
>>> What is the benefit of having this script in the class? I see that
>>> we have separate files for:
>>>
>>>  - buildchroot
>>>  - kernel modules
>>>
>>> Why this chroot script should be a part of bitbake class? This makes
>>> debugging much more complicated, because I have no possibility to
>>> run this script manually from buildchroot (like I could do with
>>> build.sh)  
>>
>> I played with it, and it "just" took 6 exports to set up the interface
>> to script-based build. That will likely not be convenient for
>> debugging as well (and I never had to call things manually that way,
>> I got all information from logs), but I can keep that pattern. Will
>> also mean: back to an include, rather than using a class (no big
>> issue).
> 
> You could write all these exports into a file that the scripts sources.
> Given that file exists, one could run manually and the recipe would not
> contain all that logic.

That sounds like overkill. And there is already a file containing that
information (run.do_build).

I'm still looking for a concrete case where local invocation is needed,
even more as we are now able to restart builds easily.

> 
>>>
>>> Also formatting style looks nasty here.  
>>
>> That's not going to change unless you suggest some alternative for the
>> sed-based code injection into postinst/postrm.
> 
> Can that code be appended with echo >> ? The patching looks pretty
> unconditional already.

I need to inject the code at a specific place in to the script, so it's
not about appending or prepending. At the same time, I'm concerned that
a real patch would break too often.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux