On 07.11.19 10:23, Cedric Hombourger wrote:
> Regain control over the packaging of the Linux kernel by providing our
> own debian recipes (debian,{control,rules} files and friends) instead
> of using the kernel's builddeb script. This will allow generation of
> packages for "perf" (not included in this changeset) with per distro
> scheme (Debian and Ubuntu have differences) and maintain compatibility
> with upstream packages generated from different source packages but
> dependent on a certain deployment scheme (e.g. linux-base for Debian)
>
> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> ---
> doc/custom_kernel.md | 181 ++++++++++++++++
> .../linux/files/build-kernel.sh | 128 -----------
> meta/recipes-kernel/linux/files/debian/compat | 1 +
> .../linux/files/debian/control.tmpl | 37 ++++
> .../linux/files/debian/isar/build.tmpl | 36 ++++
> .../linux/files/debian/isar/clean.tmpl | 20 ++
> .../linux/files/debian/isar/common.tmpl | 48 +++++
> .../linux/files/debian/isar/install.tmpl | 204 ++++++++++++++++++
> .../files/debian/linux-image.postinst.tmpl | 29 +++
> .../files/debian/linux-image.postrm.tmpl | 35 +++
> .../files/debian/linux-image.preinst.tmpl | 25 +++
> .../linux/files/debian/linux-image.prerm.tmpl | 21 ++
> .../linux/files/debian/rules.tmpl | 41 ++++
> meta/recipes-kernel/linux/linux-custom.inc | 179 ++++++++++-----
> 14 files changed, 800 insertions(+), 185 deletions(-)
> create mode 100644 doc/custom_kernel.md
> delete mode 100644 meta/recipes-kernel/linux/files/build-kernel.sh
> create mode 100644 meta/recipes-kernel/linux/files/debian/compat
> create mode 100644 meta/recipes-kernel/linux/files/debian/control.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/isar/build.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/isar/clean.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/isar/common.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/isar/install.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/linux-image.postinst.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/linux-image.postrm.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/linux-image.preinst.tmpl
> create mode 100644 meta/recipes-kernel/linux/files/debian/linux-image.prerm.tmpl
> create mode 100755 meta/recipes-kernel/linux/files/debian/rules.tmpl
>
> diff --git a/doc/custom_kernel.md b/doc/custom_kernel.md
> new file mode 100644
> index 0000000..934325a
> --- /dev/null
> +++ b/doc/custom_kernel.md
> @@ -0,0 +1,181 @@
> +# Custom kernel recipe for Isar
> +
> +## Contents
> +
> + - [Summary](#summary)
> + - [Proposal owners](#proposal-owners)
> + - [Current status](#current-status)
> + - [Detailed description](#detailed-description)
> + - [Benefit to Isar](#benefit-to-isar)
> + - [How can I help?](#how-can-i-help)
> + - [Scope](#scope)
> + - [How to test](#how-to-test)
> + - [Dependencies](#dependencies)
> + - [Documentation](#documentation)
> + - [Questions and answers](#questions-and-answers)
> +
> +## Summary
> +
> +Isar currently uses the Linux kernel's builddeb script to generate linux-image,
> +linux-headers and linux-libc-dev packages for your custom kernel sources. The
> +main benefit of this approach was that the low level details of how to package
> +the Linux kernel image, modules and headers were left in the kernel. There are
> +however some drawbacks: Isar had to repack generated packages to introduce some
> +of the changes it needs such as suffixing the kernel packages with the name of
> +the custom kernel (e.g. with "mainline" for the "linux-mainline" recipe), lack
> +of automatic dependencies (the builddeb script from the kernel does not use
> +debhelper), it was difficult to add packages especially distribution specific
> +packages (maintain version specific patch in Isar vs being able to upstream
> +distribution specific changes upstream), etc. This change is about being less
> +dependent on the kernel packaging scripts (which aren't used by major distros
> +anyway) and regain control on how we want or need the Linux kernel packaged.
> +
> +## Proposal owners
> +
> + * name: [Cedric Hombourger](https://github.com/chombourger)
> +
> +## Current status
> +
> +### Tests
> +
> +The following were recently checked:
> +
> + * Custom kernels in meta-isar build without (known) failures
> + * initrd image gets generated when the kernel is added to the rootfs
> + * `ISAR_CROSS_COMPILE` is supported
> + * `KERNEL_FILE` is honored
> + * `KERNEL_DEFCONFIG` may either be a file (specified via `SRC_URI`) or a
> + kernel-provided defconfig
> + * Config fragments get merged with the user-specified kernel configuration
> + * Path to installed DTBs may be configured (new)
> + * Configuration and build from a `devshell`
> + * Support for the `KERNEL_LIBC_DEV_DEPLOY` directive
> +
> +while the following needs work:
> +
> + * Check if custom-kernel builds are reproducible
> +
> +### Remaining work items
> +
> + * Check if there is a better way to cross compile kernel "scripts" such as
> + `fixdep` or `modpost` than manually compiling them (they are compiled for
> + the build machine (amd64) and not for the target (e.g. armhf) even though
> + we are cross-compiling - yet the linux-headers package should be shipping
> + "scripts" for the target machine)
> +
> +## Detailed description
> +
> +### Requirements
> +
> +#### Current
> +
> +Isar has had a `linux-custom` recipe for a long time and counts many users. It is therefore
> +paramount to maintain features that it has today with no or little changes to the APIs. The
> +following requirements were identified and will be maintained:
> +
> + 1. Sources to the custom Linux kernel may be specified via `SRC_URI`
> +
> + 2. Configure the kernel via an in-tree or an external `defconfig` via `KERNEL_DEFCONFIG`
> +
> + 3. Support kernel configuration tweaks via configuration fragments (`.cfg` files)
> +
> + 4. Patches to the linux kernel may be specified via `SRC_URI`
> +
> + 5. Ensure that the Isar recipe `PV` matches the kernel release identifier (`KERNEL_RELEASE`)
> +
> + 6. Produce a `linux-image` package that ships the kernel image and modules
> +
> + 7. Allow the name of the kernel image to be changed via `KERNEL_FILE` (defaults to `vmlinuz`)
> +
> + 8. Produce a `linux-headers` package which includes kernel headers and kbuild scripts/tools
> +
> + 9. The `linux-headers` package shall support native and cross compiles of out-of-tree kernel modules
> +
> + 10. Produce a `linux-libc-dev` package to support user-land builds
> +
> + 11. Only ship the `linux-libc-dev` package if instructed to (`KERNEL_LIBC_DEV_DEPLOY` equals to `"1"`)
> +
> + 12. Support both native and cross compiles (`ISAR_CROSS_COMPILE`)
> +
> + 13. Support for the following kernel architectures:
> +
> + * arm
> + * arm64
> + * mips
> + * x86
> +
> + 14. Support `devshell` (kernel configuration shall be applied)
> +
> +#### New requirements (proposed)
> +
> +This proposal includes the following new requirements:
> +
> + 1. Support creation of additional packages
> +
> + 2. Allow per-distro packaging tweaks
> +
> +### Approach
> +
> +The new implementation will have the `builddeb` code ported from the Linux kernel included to the
> +Isar environment. It will be refactor to support pre and post hooks. Hooks may be used by external
> +layers to extend the kernel packaging instructions provided by "stock" Isar or by Isar itself to
> +implement distro-specific packages or tweaks.
> +
> +Hooks need the ability to add `build` and `install` steps which can be achieved by having the main
> +`build` and `install` steps call pre and post scripts. This will however solve only one side of the
> +problem: new packages should be declared in the `debian/control` file. The packaging process
> +(initiated by `debian/rules`) will use a baseline `control` file when started and a mechanism will
> +be provided for hooks to append `control` blocks as they are processed.
> +
> +### Future work
> +
> +In the event where this proposal is accepted, here are a few future projects that are being
> +considered:
> +
> + 1. Package `perf` as Debian does
> +
> + 2. Add `ubuntu` as an Isar supported target distro and validate `linux-custom` recipe
> +
> +## Benefit to Isar
> +
> + * Control the kernel build and packaging process
> +
> + * No longer need to repack packages
> +
> + * Provide hooks for custom kernel recipes to add build/install directives
> +
> +## How can I help?
> +
> + * Check if you are able to build your own linux-custom recipes with this change
> + * Check if packages generated by this recipe are compatible with your Debian-based distro
> + * Review and provide comments on this changeset
> + * Create templates for distro-specific packages (e.g. perf)
> +
> +## Scope
> +
> + * Proposal owners: proposal owners are to communicate the idea to isar-users and make sure
> + that all of their requirements are met (to the extent they are already supported with the
> + current solution) or may be met in the future.
> +
> + * Other developers: port your out-of-tree linux-custom recipes to this new solution and
> + report issues.
> +
> +## How to test?
> +
> +The linux-mainline recipe may be used for some basic testing. This recipe is being used by the
> +following machines:
> +
> + * de0-nano-soc
> + * qemumipsel
> +
> +## Dependencies
> +
> + * None
> +
> +## Documentation
> +
> + * Document user-visible variables added by this proposal
> +
> +## Questions and answers
> +
> + * None
> diff --git a/meta/recipes-kernel/linux/files/build-kernel.sh b/meta/recipes-kernel/linux/files/build-kernel.sh
> deleted file mode 100644
> index f56e96c..0000000
> --- a/meta/recipes-kernel/linux/files/build-kernel.sh
> +++ /dev/null
> @@ -1,128 +0,0 @@
> -#!/bin/bash
> -#
> -# Custom kernel build
> -#
> -# This software is a part of ISAR.
> -# Copyright (c) Siemens AG, 2018
> -#
> -# SPDX-License-Identifier: MIT
> -
> -source /isar/common.sh
> -
> -host_arch=$(dpkg --print-architecture)
> -
> -if [ "$host_arch" != "$target_arch" ]; then
> - case $target_arch in
> - armhf)
> - export ARCH=arm
> - export CROSS_COMPILE="arm-linux-gnueabihf-"
> - ;;
> - arm64)
> - export ARCH=arm64
> - export CROSS_COMPILE="aarch64-linux-gnu-"
> - ;;
> - mipsel)
> - export ARCH=mips
> - export CROSS_COMPILE="mipsel-linux-gnu-"
> - ;;
> - *)
> - echo "error: unsupported architecture ($target_arch)"
> - exit 1
> - ;;
> - esac
> -fi
> -
> -REPACK_DIR="$1/../repack"
> -REPACK_LINUX_IMAGE_DIR="${REPACK_DIR}/linux-image"
> -REPACK_LINUX_HEADERS_DIR="${REPACK_DIR}/linux-headers"
> -
> -make ${KERNEL_CONFIG_TARGET} || exit ${?}
> -if [ -n "${KERNEL_FRAGMENTS}" ]; then
> - scripts/kconfig/merge_config.sh -m .config ${KERNEL_FRAGMENTS}
> -fi
> -
> -KV=$( make -s kernelrelease )
> -if [ "${KV}" != "${PV}" ]; then
> - echo "ERROR: Recipe PV is \"${PV}\" but should be \"${KV}\"" 1>&2
> - echo "ERROR: Probably due to CONFIG_LOCALVERSION" 1>&2
> - exit 1
> -fi
> -
> -rm -f .version
> -KBUILD_DEBARCH=$target_arch make -j $(($(nproc) * 2)) deb-pkg
> -
> -rm -rf "${REPACK_DIR}"
> -mkdir -p "${REPACK_DIR}"
> -mkdir -p "${REPACK_LINUX_IMAGE_DIR}"
> -mkdir -p "${REPACK_LINUX_HEADERS_DIR}"
> -
> -cp -a debian "${REPACK_DIR}"
> -
> -# dpkg-gencontrol performs cross-incompatible checks on the
> -# Architecture field; trick it to accept the control file
> -sed -i "s/Architecture: .*/Architecture: any/" "${REPACK_DIR}/debian/control"
> -
> -cd ..
> -
> -dpkg-deb -R linux-image-${PV}_${PV}-1_*.deb "${REPACK_LINUX_IMAGE_DIR}"
> -dpkg-deb -R linux-headers-${PV}_${PV}-1_*.deb "${REPACK_LINUX_HEADERS_DIR}"
> -
> -dpkg-gencontrol -crepack/debian/control \
> - -lrepack/debian/changelog \
> - -frepack/debian/files \
> - -plinux-image-${PV} \
> - -P"${REPACK_LINUX_IMAGE_DIR}" \
> - -DPackage="linux-image-${KERNEL_NAME}" \
> - -DSection=kernel \
> - -DPriority=required \
> - -DDepends="${KERNEL_DEBIAN_DEPENDS}" \
> - -DArchitecture=$target_arch
> -
> -# Add Debian-like link installation to postinst
> -mkdir -p ${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}
> -touch "${REPACK_LINUX_IMAGE_DIR}/lib/modules/${PV}/.fresh-install"
> -sed -i "${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postinst" \
> - -e "/^set -e$/a\\
> -\\
> -if [ -f /lib/modules/${PV}/.fresh-install ]; then\\
> - change=install\\
> -else\\
> - change=upgrade\\
> -fi\\
> -linux-update-symlinks \$change ${PV} /boot/${KERNEL_FILE}-${PV}\\
> -rm -f /lib/modules/${PV}/.fresh-install"
> -
> -# Add Debian-like link removal to postrm
> -sed -i "${REPACK_LINUX_IMAGE_DIR}/DEBIAN/postrm" \
> - -e "/^set -e$/a\\
> -\\
> -rm -f /lib/modules/${PV}/.fresh-install\\
> -\\
> -if [ \"\$1\" != upgrade ] && command -v linux-update-symlinks >/dev/null; then\\
> - linux-update-symlinks remove ${PV} /boot/${KERNEL_FILE}-${PV}\\
> -fi"
> -
> -# Make sure arm64 kernels are decompressed
> -if [ "$target_arch" = "arm64" ]; then
> - kernel_file="${REPACK_LINUX_IMAGE_DIR}/boot/${KERNEL_FILE}-${PV}"
> - mv "${kernel_file}" "${kernel_file}.gz"
> - gunzip "${kernel_file}.gz"
> -fi
Next regression: This got lost.
Started a test build to check
Thanks for catching that subtle one
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
--
You received this message because you are subscribed to a topic in the Google Groups "isar-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/isar-users/XDeqow7oo9I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d177ccc9-c18c-15c7-a356-92dd2c9fd8e6%40siemens.com.