[PATCH v2] sdk: make SDK an image

[PATCH v2] sdk: make SDK an image

[Adriaan Schmidt]

This derives the SDK from image.bbclass (currently it only
inherits rootfs), removing the custom packaging, and instead
relying on existing features of image.bbclass.
Note that because there is not tarxz-img IMAGE_TYPE, we switch
the SDK type to targz-img. This is only temporary, until we have
the OE-inspired imagetypes.

API change: SDK_FORMATS and SDK_INCLUDE_ISAR_APT can no longer be
    set in the image recipe. This follows the (possibly buggy)
    behavior SDK_[PRE]INSTALL. Currently a good location to set
    these might be the machine.conf.
API change: to get the container-sdk, you need to set SDK_FORMAT
    to `container-img` and CONTAINER_FORMATS to the desired container
    type (just like you would for any other image).

Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
---
 doc/user_manual.md                           | 25 +++----
 meta/classes/image-sdk-extension.bbclass     | 74 +-------------------
 meta/classes/targz-img.bbclass               |  3 +-
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 58 +++++++++++++--
 4 files changed, 70 insertions(+), 90 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c81c6e4..0859667 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1010,14 +1010,14 @@ mount devtmpfs $1/dev     -t devtmpfs -o mode=0755,nosuid
 mount devpts   $1/dev/pts -t devpts   -o gid=5,mode=620
 mount tmpfs    $1/dev/shm -t tmpfs    -o rw,seclabel,nosuid,nodev
 
-$ sudo scripts/mount_chroot.sh ../build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf
+$ sudo scripts/mount_chroot.sh ../build/tmp/deploy/sdkchroot/debian-buster-armhf
 
 ```
 
  - chroot to isar SDK rootfs:
 
 ```
-$ sudo chroot build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf
+$ sudo chroot build/tmp/deploy/sdkchroot/debian-buster-armhf
 ```
  - Check that cross toolchains are installed
 
@@ -1082,18 +1082,19 @@ User specifies the variable `SDK_FORMATS` providing a space-separated list of
 SDK formats to generate.
 
 Supported formats are:
- - `tar-xz`: (default) is the non-containerized format that results from 
+ - `targz-img`: (default) is the non-containerized format that results from 
    following the instructions in 
    "[Create an ISAR SDK root filesystem](#create-an-isar-sdk-root-filesystem)"
- - `docker-archive`: an archive containing a Docker image that can be imported 
-   with 
-   [`docker load`](https://docs.docker.com/engine/reference/commandline/load)
- - `docker-daemon`: resulting container image is made available on the local 
-   Docker Daemon
- - `containers-storage`: resulting container image is made available to tools 
-   using containers/storage back-end (e.g. Podman, CRIO, buildah,...)
- - `oci-archive`: an archive containing an OCI image, mostly for archiving as 
-   seed for any of the above formats
+ - `container-img`: select the container formats by setting `CONTAINER_FORMATS`:
+   - `docker-archive`: an archive containing a Docker image that can be imported 
+     with 
+     [`docker load`](https://docs.docker.com/engine/reference/commandline/load)
+   - `docker-daemon`: resulting container image is made available on the local 
+     Docker Daemon
+   - `containers-storage`: resulting container image is made available to tools 
+     using containers/storage back-end (e.g. Podman, CRIO, buildah,...)
+   - `oci-archive`: an archive containing an OCI image, mostly for archiving as 
+     seed for any of the above formats
 
 User manually triggers creation of SDK formats for his target platform by 
 launching the task `do_populate_sdk` for target image, f.e.
diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 426b925..ec1c300 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,83 +5,11 @@
 #
 # This class extends the image.bbclass to supply the creation of a sdk
 
-SDK_INCLUDE_ISAR_APT ?= "0"
-SDK_FORMATS ?= "tar-xz"
-
-sdk_tar_xz() {
-    # Copy mount_chroot.sh for convenience
-    sudo cp ${SCRIPTSDIR}/mount_chroot.sh ${SDKCHROOT_DIR}
-
-    # Create SDK archive
-    cd -P ${SDKCHROOT_DIR}/..
-    sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
-        -c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
-    bbdebug 1 "SDK rootfs available in ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz"
-}
-
 do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_populate_sdk[depends] = "sdkchroot:do_build"
 do_populate_sdk[vardeps] += "SDK_INCLUDE_ISAR_APT SDK_FORMATS"
 do_populate_sdk() {
-    local sdk_container_formats=""
-
-    if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then
-        # Copy isar-apt with deployed Isar packages
-        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt
-    else
-        # Remove isar-apt repo entry
-        sudo rm -f ${SDKCHROOT_DIR}/etc/apt/sources.list.d/isar-apt.list
-    fi
-
-    sudo umount -R ${SDKCHROOT_DIR}/dev || true
-    sudo umount ${SDKCHROOT_DIR}/proc || true
-    sudo umount -R ${SDKCHROOT_DIR}/sys || true
-
-    # Remove setup scripts
-    sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh
-
-    # Make all links relative
-    for link in $(find ${SDKCHROOT_DIR}/ -type l); do
-        target=$(readlink $link)
-
-        if [ "${target#/}" != "${target}" ]; then
-            basedir=$(dirname $link)
-            new_target=$(realpath --no-symlinks -m --relative-to=$basedir ${SDKCHROOT_DIR}/${target})
-
-            # remove first to allow rewriting directory links
-            sudo rm $link
-            sudo ln -s $new_target $link
-        fi
-    done
-
-    # Set up sysroot wrapper
-    for tool_pattern in "gcc-[0-9]*" "g++-[0-9]*" "cpp-[0-9]*" "ld.bfd" "ld.gold"; do
-        for tool in $(find ${SDKCHROOT_DIR}/usr/bin -type f -name "*-linux-gnu*-${tool_pattern}"); do
-            sudo mv "${tool}" "${tool}.bin"
-            sudo ln -sf gcc-sysroot-wrapper.sh ${tool}
-        done
-    done
-
-    # separate SDK formats: TAR and container formats
-    for sdk_format in ${SDK_FORMATS} ; do
-        case ${sdk_format} in
-            "tar-xz")
-                sdk_tar_xz
-                ;;
-            "docker-archive" | "oci" | "oci-archive" | "docker-daemon" | "containers-storage")
-                sdk_container_formats="${sdk_container_formats} ${sdk_format}"
-                ;;
-            *)
-                die "unsupported SDK format specified: ${sdk_format}"
-                ;;
-        esac
-    done
-
-    # generate the SDK in all the desired container formats
-    if [ -n "${sdk_container_formats}" ] ; then
-        bbnote "Generating SDK container in ${sdk_container_formats} format"
-        containerize_rootfs "${SDKCHROOT_DIR}" "sdk-${DISTRO}-${DISTRO_ARCH}" "${sdk_container_formats}"
-    fi
+    :
 }
 
 addtask populate_sdk after do_rootfs
diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass
index bf94af0..f90e9fb 100644
--- a/meta/classes/targz-img.bbclass
+++ b/meta/classes/targz-img.bbclass
@@ -4,10 +4,11 @@
 # SPDX-License-Identifier: MIT
 
 TARGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz"
+TAR_OPTIIONS ?= ""
 
 do_targz_image() {
     rm -f ${TARGZ_IMAGE_FILE}
-    sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
+    sudo tar ${TAR_OPTIIONS} -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
     sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
 }
 
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index e367eae..0dacd57 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -16,6 +16,8 @@ SRC_URI = " \
 PV = "0.1"
 
 SDK_INSTALL ?= ""
+SDK_FORMATS ?= "targz-img"
+SDK_INCLUDE_ISAR_APT ?= "0"
 
 DEPENDS += "${SDK_INSTALL}"
 
@@ -26,7 +28,11 @@ TOOLCHAIN_${HOST_ARCH} = "build-essential"
 TOOLCHAIN_i386 = "build-essential"
 TOOLCHAIN_append_compat-arch = " crossbuild-essential-${COMPAT_DISTRO_ARCH}"
 
-inherit rootfs
+IMAGE_FSTYPES = "${SDK_FORMATS}"
+IMAGE_FULLNAME = "sdk-${DISTRO}-${DISTRO_ARCH}"
+TAR_OPTIONS += "--transform=\"s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|\""
+
+inherit image
 ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
@@ -35,6 +41,11 @@ ROOTFS_FEATURES += "clean-package-cache generate-manifest export-dpkg-status"
 ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_SDKCHROOT}"
 ROOTFS_DPKGSTATUS_DEPLOY_DIR = "${DEPLOY_DIR_SDKCHROOT}"
 
+# Override settings from image.bbclass: SDK is not MACHINE-specific
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"
+
 SDK_PREINSTALL += " \
     debhelper \
     autotools-dev \
@@ -50,10 +61,17 @@ SDK_INCLUDE_ISAR_APT ?= "0"
 
 S = "${WORKDIR}/rootfs"
 
-ROOTFS_CONFIGURE_COMMAND += "${@'rootfs_configure_isar_apt_dir' if d.getVar('SDK_INCLUDE_ISAR_APT') == '1' else ''}"
+do_rootfs_install[vardeps] += "SDK_INCLUDE_ISAR_APT"
+
+ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt_dir"
 rootfs_configure_isar_apt_dir() {
-    # Copy isar-apt instead of mounting:
-    sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${ROOTFSDIR}/isar-apt
+    if [ "${SDK_INCLUDE_ISAR_APT}" = "1" ]; then
+        # Copy isar-apt with deployed Isar packages
+        sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt
+    else
+        # Remove isar-apt repo entry
+        sudo rm -f ${SDKCHROOT_DIR}/etc/apt/sources.list.d/isar-apt.list
+    fi
 }
 
 ROOTFS_POSTPROCESS_COMMAND =+ "sdkchroot_install_files"
@@ -66,6 +84,38 @@ sdkchroot_install_files() {
     sudo chroot ${S} /configscript.sh  ${DISTRO_ARCH}
 }
 
+ROOTFS_POSTPROCESS_COMMAND += "sdk_finalize"
+sdk_finalize() {
+    sudo umount -R ${SDKCHROOT_DIR}/dev || true
+    sudo umount ${SDKCHROOT_DIR}/proc || true
+    sudo umount -R ${SDKCHROOT_DIR}/sys || true
+
+    # Remove setup scripts
+    sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh
+
+    # Make all links relative
+    for link in $(find ${SDKCHROOT_DIR}/ -xdev -type l); do
+        target=$(readlink $link)
+
+        if [ "${target#/}" != "${target}" ]; then
+            basedir=$(dirname $link)
+            new_target=$(realpath --no-symlinks -m --relative-to=$basedir ${SDKCHROOT_DIR}/${target})
+
+            # remove first to allow rewriting directory links
+            sudo rm $link
+            sudo ln -s $new_target $link
+        fi
+    done
+
+    # Set up sysroot wrapper
+    for tool_pattern in "gcc-[0-9]*" "g++-[0-9]*" "cpp-[0-9]*" "ld.bfd" "ld.gold"; do
+        for tool in $(find ${SDKCHROOT_DIR}/usr/bin -xdev -type f -name "*-linux-gnu*-${tool_pattern}"); do
+            sudo mv "${tool}" "${tool}.bin"
+            sudo ln -sf gcc-sysroot-wrapper.sh ${tool}
+        done
+    done
+}
+
 do_sdkchroot_deploy[dirs] = "${DEPLOY_DIR_SDKCHROOT}"
 do_sdkchroot_deploy() {
     ln -Tfsr "${ROOTFSDIR}" "${SDKCHROOT_DIR}"
-- 
2.30.2

Re: [PATCH v2] sdk: make SDK an image

[Jan Kiszka]

On 15.12.21 11:55, Adriaan Schmidt wrote:
> This derives the SDK from image.bbclass (currently it only
> inherits rootfs), removing the custom packaging, and instead
> relying on existing features of image.bbclass.
> Note that because there is not tarxz-img IMAGE_TYPE, we switch
> the SDK type to targz-img. This is only temporary, until we have
> the OE-inspired imagetypes.

Then the ordering is also wrong: First make sure we can keep the
previous output format, then refactor.

Meanwhile, we can think about how to hook the SDK image build into a
task of another image build, so that we move in the right direction /wrt
configuration.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

RE: [PATCH v2] sdk: make SDK an image

[Schmidt, Adriaan]

Kiszka, Jan, 15. Dezember 2021 12:55:
> On 15.12.21 11:55, Adriaan Schmidt wrote:
> > This derives the SDK from image.bbclass (currently it only
> > inherits rootfs), removing the custom packaging, and instead
> > relying on existing features of image.bbclass.
> > Note that because there is not tarxz-img IMAGE_TYPE, we switch
> > the SDK type to targz-img. This is only temporary, until we have
> > the OE-inspired imagetypes.
> 
> Then the ordering is also wrong: First make sure we can keep the
> previous output format, then refactor.
> 
> Meanwhile, we can think about how to hook the SDK image build into a
> task of another image build, so that we move in the right direction /wrt
> configuration.

I had a look at OE. There the SDK is something generated by the target
image, but it's something quite complex, with lots of Python code and
stuff in libraries. But except for the Isar SDK being simpler, it's
following the same idea.

I now think:
- The SDK should be generated by the target image recipe, but maybe
  more integrated than it currently is. (Put the logic currently in
  the sdkchroot recipe into the image-sdk-extension class, so we can
  easily use settings made in the target image recipe)
- The SDK should only be generated as tar[xz|gz], but not as container
- If someone wants an SDK image (container, development VM, ...),
  that would be a separate recipe further processing the SDK, and the
  user would need to create that recipe explicitly. I can imagine that
  for the default case, this SDK image recipe would only have two lines:
  include the target image recipe (to get all the SDK_* settings and the
  do_populate_sdk task) and include some new class that makes the
  adjustments to generate an "image containing the SDK" instead
  of the original target image.

Adriaan

Re: [PATCH v2] sdk: make SDK an image

[Jan Kiszka]

On 16.12.21 10:12, Schmidt, Adriaan (T RDA IOT SES-DE) wrote:
> Kiszka, Jan, 15. Dezember 2021 12:55:
>> On 15.12.21 11:55, Adriaan Schmidt wrote:
>>> This derives the SDK from image.bbclass (currently it only
>>> inherits rootfs), removing the custom packaging, and instead
>>> relying on existing features of image.bbclass.
>>> Note that because there is not tarxz-img IMAGE_TYPE, we switch
>>> the SDK type to targz-img. This is only temporary, until we have
>>> the OE-inspired imagetypes.
>>
>> Then the ordering is also wrong: First make sure we can keep the
>> previous output format, then refactor.
>>
>> Meanwhile, we can think about how to hook the SDK image build into a
>> task of another image build, so that we move in the right direction /wrt
>> configuration.
> 
> I had a look at OE. There the SDK is something generated by the target
> image, but it's something quite complex, with lots of Python code and
> stuff in libraries. But except for the Isar SDK being simpler, it's
> following the same idea.
> 
> I now think:
> - The SDK should be generated by the target image recipe, but maybe
>   more integrated than it currently is. (Put the logic currently in
>   the sdkchroot recipe into the image-sdk-extension class, so we can
>   easily use settings made in the target image recipe)
> - The SDK should only be generated as tar[xz|gz], but not as container
> - If someone wants an SDK image (container, development VM, ...),
>   that would be a separate recipe further processing the SDK, and the
>   user would need to create that recipe explicitly. I can imagine that
>   for the default case, this SDK image recipe would only have two lines:
>   include the target image recipe (to get all the SDK_* settings and the
>   do_populate_sdk task) and include some new class that makes the
>   adjustments to generate an "image containing the SDK" instead
>   of the original target image.

But how do get from the tarball to the container image? You are
suggesting to drop an existing and actually used interface
(https://github.com/siemens/meta-iot2050/blob/master/conf/distro/iot2050-debian.conf#L48),
but you do not provide an equally simple alternative yet. That can't be
"someone downstream has to write a special class".

And it's also unclear to me how to ensure that container images are
generated corresponding to the specific target images.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

RE: [PATCH v2] sdk: make SDK an image

[Schmidt, Adriaan]

Kiszka, Jan, 16. Dezember 2021 15:31:
> On 16.12.21 10:12, Schmidt, Adriaan (T RDA IOT SES-DE) wrote:
> > Kiszka, Jan, 15. Dezember 2021 12:55:
> >> On 15.12.21 11:55, Adriaan Schmidt wrote:
> >>> This derives the SDK from image.bbclass (currently it only
> >>> inherits rootfs), removing the custom packaging, and instead
> >>> relying on existing features of image.bbclass.
> >>> Note that because there is not tarxz-img IMAGE_TYPE, we switch
> >>> the SDK type to targz-img. This is only temporary, until we have
> >>> the OE-inspired imagetypes.
> >>
> >> Then the ordering is also wrong: First make sure we can keep the
> >> previous output format, then refactor.
> >>
> >> Meanwhile, we can think about how to hook the SDK image build into a
> >> task of another image build, so that we move in the right direction /wrt
> >> configuration.
> >
> > I had a look at OE. There the SDK is something generated by the target
> > image, but it's something quite complex, with lots of Python code and
> > stuff in libraries. But except for the Isar SDK being simpler, it's
> > following the same idea.
> >
> > I now think:
> > - The SDK should be generated by the target image recipe, but maybe
> >   more integrated than it currently is. (Put the logic currently in
> >   the sdkchroot recipe into the image-sdk-extension class, so we can
> >   easily use settings made in the target image recipe)
> > - The SDK should only be generated as tar[xz|gz], but not as container
> > - If someone wants an SDK image (container, development VM, ...),
> >   that would be a separate recipe further processing the SDK, and the
> >   user would need to create that recipe explicitly. I can imagine that
> >   for the default case, this SDK image recipe would only have two lines:
> >   include the target image recipe (to get all the SDK_* settings and the
> >   do_populate_sdk task) and include some new class that makes the
> >   adjustments to generate an "image containing the SDK" instead
> >   of the original target image.
> 
> But how do get from the tarball to the container image? You are
> suggesting to drop an existing and actually used interface
> (https://github.com/siemens/meta-iot2050/blob/master/conf/distro/iot2050-
> debian.conf#L48),
> but you do not provide an equally simple alternative yet. That can't be
> "someone downstream has to write a special class".

The CONTAINER_FORMATS of the current implementation introduce a partial
duplicate of IMAGE_FSTYPES (only supporting tarballs and containers), when
I think it should be using the existing imaging mechanisms. (Soon someone
will request their SDK in the form of a VM image with KDE and Eclipse
installed...)

My current draft requires a 2-line recipe for the SDK image, e.g. isar-image-sdk.bb:
---
SDK_TARGET_IMAGE = "isar-image-base"
inherit sdkimage
---
This depends on ${SDK_TARGET_IMAGE}:do_populate_sdk, so all SDK_[PRE]INSTALL
and other settings defining the SDK are in the target image recipe.
(It's still not ideal, because im my current implementation you can
ONLY customize the SDK image in the target image recipe and not in the
SDK image one...)

`bitbake isar-image-sdk` then generates SDK images as requested by SDK_FORMATS,
with the same semantics as IMAGE_FSTYPES, so in the example you linked you'd have to
request container-img and set CONTAINER_FORMATS to define which containers
you want (actually, my imagetypes-draft would later make all the container
image types first-class citizens, so that would bring you back to the current
way of defining things).

For a generic solution I see no way around having a separate image.
The current SDK-container generation calls parts of the imaging code
as a function. This sort of works (although Henning already stumbled
across some of the details when he worked on improving container imaging),
but it's not something we can do in general.

> And it's also unclear to me how to ensure that container images are
> generated corresponding to the specific target images.

I hope it's become a little clearer...

Adriaan

RE: [PATCH v2] sdk: make SDK an image

[Schmidt, Adriaan]

Schmidt, Adriaan, 16. Dezember 2021 16:30:
> For a generic solution I see no way around having a separate image.

Ok, I take that back!
By calling on some deep bitbake voodoo, I think I found a way to do this
while preserving all interfaces. However, having this as a stand-alone
improvement of the current implementation requires some ugly hacks
(that would later be cleaned up by my other imaging/imagetypes work).
Much better to have it all in one series.
Will start preparing that for review...

Adriaan