Re: [RFC PATCH 2/2] docs: document usage of sdk container images

[Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>]

On 12/01/2021 13:18, Henning Schild wrote:
> Am Tue, 12 Jan 2021 13:03:18 +0100
> schrieb Silvano Cirujano Cuesta <silvano.cirujano-cuesta@siemens.com>:
>
>> On 12/01/2021 12:40, Henning Schild wrote:
>>> Am Tue, 12 Jan 2021 11:33:38 +0100
>>> schrieb "[ext] Silvano Cirujano Cuesta"
>>> <silvano.cirujano-cuesta@siemens.com>:
>>>  
>>>> Signed-off-by: Silvano Cirujano Cuesta
>>>> <silvano.cirujano-cuesta@siemens.com> ---
>>>>  doc/user_manual.md | 70
>>>> ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70
>>>> insertions(+)
>>>>
>>>> diff --git a/doc/user_manual.md b/doc/user_manual.md
>>>> index a4f3d1d..ff779b1 100644
>>>> --- a/doc/user_manual.md
>>>> +++ b/doc/user_manual.md
>>>> @@ -834,6 +834,76 @@ ii  crossbuild-essential-armhf           12.3
>>>>                all          Inf ~#
>>>>  ```
>>>>  
>>>> +## Create a "containerized" ISAR SDK root filesystem
>>>> +
>>>> +### Motivation
>>>> +
>>>> +Distributing and using the SDK root filesystem created following
>>>> the instructions in "[Create an ISAR SDK root
>>>> filesystem](#create-an-isar-sdk-root-filesystem)" becomes easier
>>>> using container images (at least for those using containers anyway)
>>>> +A "containerized" SDK adds to those advantages of a normal SDK
>>>> root filesystem the comfort of container images. + +### Approach +
>>>> +Create container image with SDK root filesystem with installed
>>>> cross-toolchain for target architecture and ability to install
>>>> already prebuilt target binary artifacts. +Developer:
>>>> + - runs a container based on the resulting container image
>>>> mounting the source code to be built,
>>>> + - develops applications for target platform on the container and
>>>> + - leaves the container getting the results on the mounted
>>>> directory. +
>>>> +### Solution
>>>> +
>>>> +User specifies the variable `SDK_FORMAT` providing a
>>>> space-separated list of SDK formats to generate. +Supported
>>>> formats are:
>>>> + - `tar`: is the default, is the non-containerized format that
>>>> results from following the instructions in "[Create an ISAR SDK
>>>> root filesystem](#create-an-isar-sdk-root-filesystem)"
>>>> + - `docker-archive`: an archive containing a Docker image that can
>>>> be imported with [`docker
>>>> import`](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.docker.com%2Fengine%2Freference%2Fcommandline%2Fimport%2F&amp;data=04%7C01%7Csilvano.cirujano-cuesta%40siemens.com%7C2e13fa7588f34212393508d8b6f43140%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637460507195377080%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oevvzWybG30IG2D1fMNPQRcfsfuvYSj1afdrIUF1ABw%3D&amp;reserved=0)
>>>> + - `docker-daemon`: (not supported from inside of a container)
>>>> resulting container image is made available on the local Docker
>>>> Daemon
>>>> + - `containers-storage`: (not supported from inside of a
>>>> container) resulting container image is made available to tools
>>>> using containers/storage back-end (e.g. Podman, CRIO, buildah,...)
>>>> + - `oci-archive`: an archive containing an OCI image, mostly for
>>>> archiving as seed for any of the above formats +  
>>> Maybe a script to postprocess the one tarball we have would be a
>>> better option. I do not understand why skopeo can not be used
>>> inside docker or podman, but we _really_ should not motivate anyone
>>> to run isar plain. Otherwise i agree that a bitbake task is a good
>>> idea and much better than postprocessing outside of the build
>>> system.  
>> What do you mean by post-processing? I mean, a Docker image is
>> nothing else but a couple of JSON and TAR files using a fixed file
>> tree structure, with SHAs... But anyway writing something that does
>> it is like reinventing the wheel, that's what tools like Umoci,
>> Skopeo,... are for.
> I mean some sort of script that needs to be called after bitbake. Not
> nice but can be done outside of a container.

Hmmm, obviously, but seems to be so obvious for me isn't that obvious :-)

The additions DON'T NEED to run in a container, but CAN be run in a container (e.g. kas-container).

If running bitbake OUTSIDE of a container (e.g. VM), then the formats "docker-daemon" and "containers-storage" should work like a charm.

If running bitbake INSIDE of a container (e.g. kas-container), then the formats "docker-daemon" and "containers-storage" won't work.

I don't know if doc/user_manual.md is the right place for all the clarifications that appear to be missing (this one and the dependency on skopeo and umoci), but they are clearly needed.

>
>> My first PoC was in fact just a shell script post-processing the TAR
>> file. Technically feasible, but it doesn't feel like a comfortable
>> workflow...
> Doing that in bitbake is for sure the best solution, given it works.
Of course, it works! ;-)
>
>> The initial idea was from Jan Kiszka (I forgot to add a
>> "Suggested-by" to the commits) and after some discussion about the
>> convenience of integrating it, I agreed that the workflow feels much
>> better this way.
>>
>> Skopeo can be used inside of Docker or Podman, it's in fact what I'm
>> doing. What doesn't work is adding the container image to a Docker
>> Daemon (unless you pass the Docker socket to the container, what I
>> wouldn't do) or containers/storage system (mounting the storage
>> directory, what I wouldn't do). IMO the target users should know how
>> to handle docker-archives (I can add that "docker load" has to be
>> used).
> Maybe i am just confused by the two "(not supported from inside of a
> container)". Does this mean one can not create the images, or not use
> ... or not both?
> The answer should probably made more clear in the docs.
>
> If creation works and only use fails, that is all fine. If creation
> inside a container does not work ... i would refrain from writing that
> code in bitbake and go post-process (extra shell script or only doc)
> with it.

As mentioned above, running bitbake for building the SDK container image can be run from inside and outside of a container. The only difference is that running in a container the host shouldn't be reachable and therefore the formats "docker-daemon" and "containers-storage" shouldn't work.

If SDK_FORMAT = "tar", then only the currently existing format (tarballed root filesystem) gets generated. If SDK_FORMAT = "tar docker-archive", then you get both the old format and a docker image archive that can be loaded on any Docker installation.

Possible scenarios (probably too much information for the documentation):

- Project A has a build system that generates a SDK container running bitbake without a container selecting SDK_FORMAT="docker-daemon". After finalizing pushes the SDK docker image to a container image registry (docker push ...). Developers pull the SDK container image from the container image registry (docker pull ...), start a container based on the image and mounting the directory with the source code, edit the source code on the host IDE and build in the container with the provided SDK.

- Project B has a build system that generates a SDK container running bitbake with a container selecting SDK_FORMAT="tar docker-archive". Since the project doesn't have any container image registry, it distributes the container image using a network share. Team B1 do have a container image registry and distributes the image internally using it. BTW, without Team B1 project B wouldn't need anything but "tar".

Clearer now?

  Silvano

>
> Henning
>
>>   Silvano
>>
>>> Henning
>>>  
>>>> +User manually triggers creation of SDK formats for his target
>>>> platform by launching the task `do_populate_sdk` for target image,
>>>> f.e. +`bitbake -c do_populate_sdk
>>>> mc:${MACHINE}-${DISTRO}:isar-image-base`. +Packages that should be
>>>> additionally installed into the SDK can be appended to
>>>> `SDK_PREINSTALL` (external repositories) and `SDK_INSTALL`
>>>> (self-built). + +The resulting SDK formats are archived into
>>>> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}-${sdk_format}.tar.xz`
>>>> (being `sdk_format` each one of the formats specified in
>>>> `SDK_FORMATS`). +The SDK container directory `/isar-apt` contains a
>>>> copy of isar-apt repo with locally prebuilt target debian packages
>>>> (for <HOST_DISTRO>). +One may get into an SDK container and install
>>>> required target packages with the help of `apt-get install
>>>> <package_name>:<DISTRO_ARCH>` command. +The directory with the
>>>> source code to develop on should be mounted on the container (with
>>>> `--volume <host-directory>:<container-directory>`) to be able to
>>>> edit files in the host with an IDE and build in the container. +
>>>> +### Example +
>>>> + - Make the SDK formats to generate available to the task
>>>> +
>>>> +For one-shot builds (use `local.conf` otherwise):
>>>> +
>>>> +```
>>>> +export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE SDK_FORMATS"
>>>> +export SDK_FORMATS="docker-archive"
>>>> +```
>>>> +
>>>> + - Trigger creation of SDK root filesystem
>>>> +
>>>> +```
>>>> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
>>>> +```
>>>> +
>>>> + - Load the SDK container image into the Docker Daemon
>>>> +
>>>> +```
>>>> +xzcat
>>>> build/tmp/deploy/images/qemuarm/sdk-debian-buster-armhf-docker-archive.tar.xz
>>>> | docker load +``` +
>>>> + - Run a container using the SDK container image (following
>>>> commands starting with `#~:` are to be run in the container) +
>>>> +```
>>>> +docker run --rm -ti --volume "$(pwd):/build"
>>>> isar-sdk-buster-armhf:latest +```
>>>> +
>>>> + - Check that cross toolchains are installed
>>>> +
>>>> +```
>>>> +:~# dpkg -l | grep crossbuild-essential-armhf
>>>> +ii  crossbuild-essential-armhf           12.3
>>>> all Informational list of cross-build-essential packages +```
>>>> +
>>>>  ## Creation of local apt repo caching upstream Debian packages
>>>>  
>>>>  ### Motivation  

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux