Re: [PATCH v2 00/24] Sbuild/Schroot migration

[Jan Kiszka <jan.kiszka@siemens.com>]

On 23.11.21 14:05, Uladzimir Bely wrote:
> In the email from Friday, 19 Nov 2021 г. 15:13:09 +03 user Uladzimir Bely 
> wrote:
>> This is a patchset showing how sbuild/schroot tools can be integrated
>> into Isar build system.
>>
>> ...
>>
>> Current limitations:
>> - there is an unsolved problem with building foreigh architectures
>> using kas-docker.

What exactly is the problem here?

>>
>> - qemuamd64 images are buildable in kas-docker, but some changes
>> in kas are reqired (like installing additional packages and adding
>> 'builder' user to 'sbuild group). I used the following changes to
>> build ghcr.io/siemens/kas/kas-isar:2.6.2-sbuild image based on
>> current ghcr.io/siemens/kas/kas-isar:2.6.2:
>> https://github.com/WiseLord/kas/commit/5b7b02aa33
>>
>> ...
>> meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (50%)
> 
> Here I would like to add some info regarding running the patchset with `kas-
> docker` and/or using Gitlab CI.
> 
> Initially, the patchset was checked in Jenkins CI (that doesn't use docker) 
> and sent to the maillist then. But there are some docker-related issues solved 
> a bit later. So here I'll try to explain them (and this is to be added to user 
> docs in the following patchset version), so someone could play with sbuild 
> patchset.
> 
> 1. Additional packages required.
> 
> The patchset requires 'schroot' and 'sbuild' packages to be installed on the 
> host. Also, they both should have at least "bullseye" version, otherwise some 
> things won't work properly (like caching .deb via 'sbuild' or foreign 
> architectures support in 'schroot').
> 
> While current (ghcr.io/siemens/kas/kas-isar:2.6.2) image doesn't include the 
> required packages, I temporary prepared and upuloaded one based on it 
> (ghcr.io/wiselord/kas-isar:2.6.2-sbuild). Look at Dockerfile.isar.sbuild chunk 
> in the patch https://github.com/WiseLord/kas/commit/38f4f11f11 for details.
> 
> Gitlab
> 
> Gitlab uses .gitlab-ci.yml for setup. So, to use modified image in Gitlab CI 
> user should simply change the first line:
> 
>> - image: ghcr.io/siemens/kas/kas-isar:latest
>> + image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild
> 
> This is required until official kas-isar image have everything included.
> 
> 2. User should be added to 'sbuild' group.
> 
> In case of 'kas-docker' it happend to be a bit tricky, because 'builder' user 
> is created 'on the fly', when container is run. I'm not an expert in 'kas' so 
> I simply pathced container-entrypoint to add user to sbuild group. Look at
> `container.entrypoint` chunk in the patch for details.
> 
> 3. Overlayfs restrictions.
> 
> Docker uses overlayfs to mount dockerimage rootfs. Schroot uses /var/lib/
> schroot/union/{overlay,underlay} directories to keep 'basic' image and 
> temporary layers on top of it.
> 
> So we happen to have 'overlayfs over overlayfs' situation that is not 
> supported by overlayfs kernel driver.
> 
> The solutionis to use an external volume for the /var/lib/schroot/union/. Look 
> at 'kas-container' chunk in the patch for details.
> 

I've commented on some aspects of you kas patch. It generally looks
feasible, just some smaller details should be improved.

> Gitlab
> 
> It seems there is no way to say gitlab using external volume via the 
> configuration file `.gitlab-ci.yml`. But it can be done by the following 
> changes in /etc/gitlab/runner/config.toml:
> 
>> -volumes = ["/cache"]
>> +volumes = ["/m/ws-10/schroot-10a/union:/var/lib/schroot/union", "/cache"]
> 
> So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on host. 
> Also, there should be empty "overlay" and "underlay" directories created in 
> it.

That's also not unrealistic, given that we already need to provide
special runners for the purpose of granting privileges and allowing
binfmt_misc. One target would be
https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci, the backend
for many of our gitlab CI runners that are Isar-compatible.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux