Re: [RFC] using lightweight containers instead of chroot

[Helmut Grohne <helmut@subdivi.de>]

Hi Jan,

On Thu, Jul 08, 2021 at 07:16:05PM +0200, Jan Kiszka wrote:
> Yeah, not cleanly separated: We need DPKG_ROOT to install build
> dependencies unprivileged into a build environment, and then we need
> something ideally unprivileged - e.g. namespaces - to run the build
> inside that environment.

Still wrong. DPKG_ROOT is not meant to install build dependencies. I
doubt that it covers sufficient packages to do so any time soon. I also
don't see any point in doing so when you can just chroot (unprivileged
via user namespaces) into it and install them there.

The reason to want DPKG_ROOT is to prepare the root filesystem for your
embedded board. If that's not what you want to do, DPKG_ROOT is not your
tool.

> We don't want to reimplement if key requirements can be fulfilled with
> Debian's own tools. One of them is having control over how build
> dependencies are installed and from where (use cases: isar-apt, offline
> builds). Another one would be the ability to patch an original package
> after unpacking its sources.

I think the goal should be making Debian's tools fulfil your key
requirements. (Regardless of whether using an abstraction layer or not.)

Can you elaborate how you want to control your build dependencies? It
would seem to me that you could just patch debian/control and be done,
no? What else do you need to control? Do you want to change the
resolver?

The "from where" part seems relatively easy to me. All of the build
tools support one way or another to supply extra repositories. All you
need to do here is ensure that your extra packages from extra
repositories have a higher version than those in your base distribution.

Why does your build tool need to support patching after unpacking
sources? I'm using that approach in rebootstrap and I consider it a
design mistake. You can just extract your source package outside the
builder, bump the version, apply your patches, create a new source
package and build that. Creating actual source packages saves you from a
lot of trouble.


> Why can't that also support a local folder with a apt repo? Spawning
> servers is surely doable but can become fun (unless using containers...)
> when running multiple build in parallel.

I've considered that, but support for this is lacking at this point.
Ideally, you'd want to bind mount the local folder into the chroot. When
using user namespaces your mapped root user may be underprivileged to
see that local folder already. sbuild does not provide such means
(unless you modify the underlying schroot, which requires root).
pbuilder has --bindmounts (but doesn't use a user namespace). mmdebstrap
can only sync directories, which means copying them.

> If that is not going to be rebuilt for every package (cached baseline,
> some dependency versions for all built packages in a run) and could
> possibly also be reused for other things (running the imaging), it could
> be a building block to replace our buildchroots. If not, we would have
> to continue feeding it with separately bootstrapped chroots.

Where does that "not rebuilt" requirement come from? Is that for
performance reasons?

If yes, please benchmark it. I've locally compared build times
(sbuild/pbuilder/mmdebstrap) and the difference is small (<10 seconds).
After all, extracting a pbuilder (or sbuild) base.tgz is not immediate
either and mmdebstrap is quite fast.

> Most of the Isar builds (complete images, not just packages) are already
> happening inside containers (kas-isar). Thus we rather need to be
> careful not requiring any nesting that could become complex - at best.
> At the same time, we want Isar to continue work on a plain Debian system.

You cannot use user namespaces in secure docker containers anyway, but
then when you already are inside a container, you can assume root
privileges and use chroot normally.

> The key question for us remains what value the abstraction can have for
> Isar, today and on the long run. Switching builders is not a daily
> business so far. Isar needs one, and that needs to plug well into the
> image creation workflow. If interacting with mdbp can be simpler than
> talking to sbuild directly, that would be a clear gain. If it's rather
> more complex under the Isar constraints while we do not have the
> compelling use case for switching builders, it is likely not the best
> way forward.

I think the value is not to be gained by you, but by Isar users. You
already found your preferred builder and it is easy to stick to that.
Others may have set up a different one already. The value gained from
the abstraction is to be able to seamlessly use the builder of the
user's choice.

Another aspect is the need for avoiding nesting of virtualization
technology. When building inside kas-isar, you cannot use user
namespaces (because secure docker containers cannot). When building on a
plain Debian system, you want to use user namespaces to be able to build
unprivileged. At present, the only builder that allows switching between
user namespaces and no user namespaces is sbuild. mdbp also gives that
flexibility by allowing you to seamlessly swap builders.

So if sbuild suits your need, then go for it rather than reinventing the
wheel.

I caution that using sbuild in a reproducible way is hard. People do
change its defaults to their needs. You need to pass a lot of options to
sbuild to avoid using user's defaults. That was one of the lessons I
learned the hard way when writing mdbp.

Helmut