Re: [RFC] using lightweight containers instead of chroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 08.07.21 21:34, Helmut Grohne wrote:
> Hi Jan,
> 
> On Thu, Jul 08, 2021 at 07:16:05PM +0200, Jan Kiszka wrote:
>> Yeah, not cleanly separated: We need DPKG_ROOT to install build
>> dependencies unprivileged into a build environment, and then we need
>> something ideally unprivileged - e.g. namespaces - to run the build
>> inside that environment.
> 
> Still wrong. DPKG_ROOT is not meant to install build dependencies. I
> doubt that it covers sufficient packages to do so any time soon. I also
> don't see any point in doing so when you can just chroot (unprivileged
> via user namespaces) into it and install them there.
> 
> The reason to want DPKG_ROOT is to prepare the root filesystem for your
> embedded board. If that's not what you want to do, DPKG_ROOT is not your
> tool.

We want it for both the board and the build env. We have the same issues
to solve there, means installing packages in unprivileged manner, either
for target arch or the builder arch (cross and native builds need to be
supported). So I see DPKG_ROOT as a building block for both. In
addition, it would overcome qemu-user, which is speed gain.

But, as you mentioned, that it all long-term, nothing to achieve soon
with a sufficiently large set of packages. It's still important to
develop a vision of where we want to go once possible.

> 
>> We don't want to reimplement if key requirements can be fulfilled with
>> Debian's own tools. One of them is having control over how build
>> dependencies are installed and from where (use cases: isar-apt, offline
>> builds). Another one would be the ability to patch an original package
>> after unpacking its sources.
> 
> I think the goal should be making Debian's tools fulfil your key
> requirements. (Regardless of whether using an abstraction layer or not.)
> 
> Can you elaborate how you want to control your build dependencies? It
> would seem to me that you could just patch debian/control and be done,
> no? What else do you need to control? Do you want to change the
> resolver?

Yes, we prepare (ad-hoc debianization) or patch or just use
debian/control for that. Packages should then be taken from the
configured deb repos, including local isar-apt, and using a pre-defined
apt database (only updated once so that all builds use the same package
versions). Local caching of the fetched deb packages is used in
addition, both as optimization as well as a building block for offline
builds.

> 
> The "from where" part seems relatively easy to me. All of the build
> tools support one way or another to supply extra repositories. All you
> need to do here is ensure that your extra packages from extra
> repositories have a higher version than those in your base distribution.
> 
> Why does your build tool need to support patching after unpacking
> sources? I'm using that approach in rebootstrap and I consider it a
> design mistake. You can just extract your source package outside the
> builder, bump the version, apply your patches, create a new source
> package and build that. Creating actual source packages saves you from a
> lot of trouble.

One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g.
to inject patches before they are accepted upstream or to address issues
that are not compatible with upstream packaging. Recent example:
https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955
(some patches not even accepted upstream yet but needed).

This procedure should be automated and expressed with very few
statements. The current approaches we have are far from optimal, but the
vision remains that you just tell Isar the package, the patch, and then
it does whatever is best from Debian perspective. That may include the
mentioned source package patching upfront. That will have to go into our
recipes and classes.

> 
> 
>> Why can't that also support a local folder with a apt repo? Spawning
>> servers is surely doable but can become fun (unless using containers...)
>> when running multiple build in parallel.
> 
> I've considered that, but support for this is lacking at this point.
> Ideally, you'd want to bind mount the local folder into the chroot. When
> using user namespaces your mapped root user may be underprivileged to
> see that local folder already. sbuild does not provide such means
> (unless you modify the underlying schroot, which requires root).
> pbuilder has --bindmounts (but doesn't use a user namespace). mmdebstrap
> can only sync directories, which means copying them.
> 
>> If that is not going to be rebuilt for every package (cached baseline,
>> some dependency versions for all built packages in a run) and could
>> possibly also be reused for other things (running the imaging), it could
>> be a building block to replace our buildchroots. If not, we would have
>> to continue feeding it with separately bootstrapped chroots.
> 
> Where does that "not rebuilt" requirement come from? Is that for
> performance reasons?

Sure. We are not building hundreds of packages, but it also not only
about building one or two. The other reason is the mentioned identical
versions in all buildchroots. Reproducing them must provide identical
content (where common) for a single image build run.

> 
> If yes, please benchmark it. I've locally compared build times
> (sbuild/pbuilder/mmdebstrap) and the difference is small (<10 seconds).
> After all, extracting a pbuilder (or sbuild) base.tgz is not immediate
> either and mmdebstrap is quite fast.
> 
>> Most of the Isar builds (complete images, not just packages) are already
>> happening inside containers (kas-isar). Thus we rather need to be
>> careful not requiring any nesting that could become complex - at best.
>> At the same time, we want Isar to continue work on a plain Debian system.
> 
> You cannot use user namespaces in secure docker containers anyway, but
> then when you already are inside a container, you can assume root
> privileges and use chroot normally.

Currently, we have no secure container env. The container simply serves
as deployment tool of Isar's dependencies and builder-side
configurations so that you can run it on any distro, not only Debian.
And you can run it in CI identically to your host.

> 
>> The key question for us remains what value the abstraction can have for
>> Isar, today and on the long run. Switching builders is not a daily
>> business so far. Isar needs one, and that needs to plug well into the
>> image creation workflow. If interacting with mdbp can be simpler than
>> talking to sbuild directly, that would be a clear gain. If it's rather
>> more complex under the Isar constraints while we do not have the
>> compelling use case for switching builders, it is likely not the best
>> way forward.
> 
> I think the value is not to be gained by you, but by Isar users. You
> already found your preferred builder and it is easy to stick to that.
> Others may have set up a different one already. The value gained from
> the abstraction is to be able to seamlessly use the builder of the
> user's choice.

Isar users use Isar, they usually do not modify the core. And the goal
of Isar is to avoid that they have to but rather provide easy means to
configure things that must be configurable. If the builder would fall
under this, e.g. no one build would build all types of packages (hope
that is not the case, so far it wasn't), it would become an Isar
requirement to switching them without much effort.

> 
> Another aspect is the need for avoiding nesting of virtualization
> technology. When building inside kas-isar, you cannot use user
> namespaces (because secure docker containers cannot). When building on a
> plain Debian system, you want to use user namespaces to be able to build
> unprivileged. At present, the only builder that allows switching between
> user namespaces and no user namespaces is sbuild. mdbp also gives that
> flexibility by allowing you to seamlessly swap builders.
> 
> So if sbuild suits your need, then go for it rather than reinventing the
> wheel.
> 
> I caution that using sbuild in a reproducible way is hard. People do
> change its defaults to their needs. You need to pass a lot of options to
> sbuild to avoid using user's defaults. That was one of the lessons I
> learned the hard way when writing mdbp.
> 

That is a valid point we need to keep an eye on. If mdbp solved that
already, it would be one of the added values I mentioned that could make
it relevant for Isar, even if not requiring to switch builders.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux