Re: ISAR schroot mountpoints when running in container

["Bezdeka, Florian" <florian.bezdeka@siemens.com>]

On Fri, 2022-07-01 at 13:38 +0200, Henning Schild wrote:
> Am Fri, 1 Jul 2022 11:30:39 +0000
> schrieb "Moessbauer, Felix" <felix.moessbauer@siemens.com>:
> 
> > > -----Original Message-----
> > > From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> > > Sent: Friday, July 1, 2022 12:44 PM
> > > To: Bezdeka, Florian (T CED SES-DE) <florian.bezdeka@siemens.com>;
> > > ubely@ilbers.de; isar-users@googlegroups.com; Moessbauer, Felix (T
> > > CED SES- DE) <felix.moessbauer@siemens.com>
> > > Subject: Re: ISAR schroot mountpoints when running in container
> > > 
> > > On 01.07.22 12:30, Bezdeka, Florian wrote:  
> > > > On Fri, 2022-07-01 at 13:23 +0300, Uladzimir Bely wrote:  
> > > > > In the email from Friday, 1 July 2022 12:11:42 +03 user
> > > > > Moessbauer, Felix
> > > > > wrote:  
> > > > > > Hi,
> > > > > > 
> > > > > > as we now have sbuild in ISAR next, first users stumble upon
> > > > > > issues when running the kas docker image in the gitlab-ci. This
> > > > > > requires additional mountpoints, as the schroot itself uses
> > > > > > overlayfs and stacking two overlayfs is not possible. By that,
> > > > > > you have to mount the overlay from the host.
> > > > > > 
> > > > > > The kas-container script already has support for that, but it
> > > > > > lacks documentation on how to configure this manually. In
> > > > > > short: The following mountpoint has to be added (as RW):
> > > > > > /var/lib/schroot/union/overlay
> > > > > > 
> > > > > > As this is neither an ISAR issue, nor a KAS issue per-se, I send
> > > > > > this to both lists.
> > > > > >  
> > > > > 
> > > > > Hello.
> > > > > 
> > > > > Yes, this overlayfs-over-overlayfs issue can be solved by
> > > > > something like
> > > > > 
> > > > > `volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]`
> > > > > 
> > > > > placed to `/etc/gitlab-runner/config.toml`, as mentioned in
> > > > > 'sbuild' series cover-letter  
> > > > 
> > > > Oh no. That's going to kill a lot of gitlab-ci setups. Even kas-
> > > > container based environments (often used as local development
> > > > environment) will need adjustments. Uncool.
> > > > 
> > > > Anyway, I will try to look into the kas-container script. Let's
> > > > hope there is a simple solution for adding one more mount.
> > > >  
> > > 
> > > When designing that, we should also have an eye on the optimization
> > > of mapping build/tmp/ onto a tmpfs mount inside the container.  
> > 
> > Wait a minute: We are mixing things up here.
> > 
> > 1. kas-container (script) already provides this mountpoint when
> > enabling isar mode. By that, local deployments are not affected
> > 

My fault. Looked at the wrong sources. Fine, nothing to do here.

> >  2.
> > When running the kas container image directly (via docker, podman or
> > gitlab-ci), the mount point is missing. There is not much we can do
> > about, except to document it for the user. 
> > 

True but still a new breaking change.


> > 3. Schroot ID collisions:
> > When running two kas containers in parallel, that also use the same
> > mountpoints on the host, we likely get name collisions (as PIDs are
> > no longer unique). This scenario is common for gitlab-ci. 
> > 

That was my main concern. Even when the mount is properly propagated
into the container we might see strange build behavior.

> > 4. Using
> > tmpfs: With Schroot, we could easily put just the schroot onto a
> > tmpfs. This has the further advantage, that the upper-dir of the
> > overlayfs can be a tmpfs which itself can be mounted inside the
> > container. By that, no bind-mount from the host is required. But this
> > might only work for machines with a lot of memory.

Interesting idea, but yes I would expect OOM everywhere. Do we have
some numbers from our internal layers how much space would be required?

> 
> What if the runner had btrfs? I guess that would be a good
> suggestion which would lead to no overlay at the base, not needing a
> volume mount in the config.

We might have both. There is no guarantee that all runners have btrfs.
We need something that works FS independent.

> 
> Henning
> 
> > Required changes:
> > 
> > In ISAR, we have to make the name of the Schroot folder more unique.
> > But as BB requires recipes to be deterministic (per-build), we have
> > to inject the ID from the outside. This could happen either via
> > local.conf or via an env-var. This env-var has to be provided by KAS,
> > with an fallback in ISAR to use the PID of the bitbake process if not
> > provided.
> > 
> > A probably better strategy would be to get a per-bitbake invocation
> > constant UUID directly from Bitbake. Don't know if that already
> > exists in BB. Putting Adriaan in CC. 
> > 
> > Felix
> > 
> > > 
> > > Jan
> > > 
> > > --
> > > Siemens AG, Technology
> > > Competence Center Embedded Linux  
> > 
>