Re: [RFC] using lightweight containers instead of chroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 08.07.21 15:52, Helmut Grohne wrote:
> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>> On 08.07.21 11:07, Cedric Hombourger wrote:
>> ...
> 
>> Longterm, there is also the desire to include support for DPKG_ROOT as
>> chroot-less way of building packages, faster when doing it cross and
>> also without special permissions (e.g. for qemu-user). But that requires
>> per-package support from Debian upstream. Discussions only started, in
>> particular with Helmuth (added to CC).
> 
> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
> to building packages. It is only relevant to installing them (which may
> be relevant here for creating filesystem images).
> 
> Building packages without chroot in a reproducible way seems next to
> impossible to me. Even when you use user namespaces, chroot does not go
> away. It merely becomes unprivileged. Is that what you mean here?

Yeah, not cleanly separated: We need DPKG_ROOT to install build
dependencies unprivileged into a build environment, and then we need
something ideally unprivileged - e.g. namespaces - to run the build
inside that environment.

> 
>>> Proposal
>>>
>>>    We may want to use unshare(1) to create a mount namespace where we
>>>    will create our bind mounts,
>>>    chroot into the buildchroot and run the specified command/script
> 
> Are you aware that sbuild directly supports this use case? It has a
> --mode argument and one of its values is "unshare". In that case, you
> supply a tarball containing the chroot and it'll perform an unprivileged
> build inside an unshared chroot.

Possibly, the prototype on Ilbers side is already using that? I didn't
see it yet, just speculating.

> 
>>>    The immediate benefit of this approach is that all mounts
>>>    automatically disappear as the supplied
>>>    command exits (whether it aborts prematurely because of an error or
>>>    normally on completion).
>>>
>>>    Another nice benefit is that bind mounts we created within this
>>>    namespace are not (directly) visible
>>>    from the parent namespace
>>>
>>>    However, we found that running scripts within an unshare environment
>>>    may not be as easy as
>>>    chroot. We would welcome feedback on the code snippets provided
>>>    below if you happen to have
>>>    some better ideas.
> 
> All of this applies to sbuild --mode=unshare as well except that it
> makes running scripts from hooks simple.
> 
>> I suspect Helmuth can tell us if that would take us on a fragile path
>> from Debian perspective. Isar-internal implementation details we could
>> likely sort out, but if that approach has architectural limits /wrt what
>> Debian packages expect/require, it might be the wrong direction.
> 
> Reimplementing this functionality seems like a waste of time to me. If
> we ignore that for a moment, we notice that there are already ~10
> implementations and updating them all is painful. Therefore, we can
> conclude that changes to the build environment are rare and your
> reimplementation likely is maintainable with limited effort.
> 

We don't want to reimplement if key requirements can be fulfilled with
Debian's own tools. One of them is having control over how build
dependencies are installed and from where (use cases: isar-apt, offline
builds). Another one would be the ability to patch an original package
after unpacking its sources.

>>>
>>>    def isar_user_spec():
>>>         import os
>>>         return '%d:%d' % (os.getuid(), os.getgid())
>>>
>>>    ISAR_USER_SPEC    = "${@ isar_user_spec() }"
>>>    ISAR_UNSHARE_CMD  = "sudo unshare --pid --fork --ipc --mount sh -ex"
>>>    ISAR_CHROOT_SHELL = "sh -ex"
>>>    ISAR_CHROOT_ROOT  = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>    ISAR_CHROOT_USER  = "chroot --userspec='${ISAR_USER_SPEC}'
>>>    ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>
>>>    # Would be similar to buildchroot_do_mounts but will happen in a
>>>    separate mount namespace
>>>    BUILDCHROOT_DO_MOUNTS =
>>>    "                                                         \
>>>         mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>>    '${BUILDCHROOT_DIR}/isar-apt'     ; \
>>>         mount --bind '${DL_DIR}'
>>>    '${BUILDCHROOT_DIR}/downloads'                     ; \
>>>         mount -t proc none
>>>    '${BUILDCHROOT_DIR}/proc'                                ; \
>>>         mount --rbind /sys
>>>    '${BUILDCHROOT_DIR}/sys'                                 ; \
>>>         mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>>    ${BUILDCHROOT_DIR}/dev/shm  ; \
>>>         mount -o bind /dev/pts
>>>    ${BUILDCHROOT_DIR}/dev/pts                             \
>>>    "
>>>
>>>    # Would be similar to dpkg_do_mounts but will happen in a separate
>>>    mount namespace
>>>    DPKG_DO_MOUNTS = "                         \
>>>         ${BUILDCHROOT_DO_MOUNTS}             ; \
>>>         mkdir -p ${BUILDROOT}                ; \
>>>         mount --bind ${WORKDIR} ${BUILDROOT}   \
>>>    "
>>>
>>>    # Build package from sources using build script
>>>    _runbuild() {
>>>         export arch=${1}
>>>
>>>         E="${@ isar_export_proxies(d)}"
>>>         (   cat <<"        UNSHARE"
>>>                 ${DPKG_DO_MOUNTS}
>>>                 (   cat <<"                SCRIPT"
>>>                         export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>>                         export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>>                         export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>                         /isar/build.sh ${PP}/${PPS} ${arch}
>>>                     SCRIPT
>>>                 ) | ${ISAR_CHROOT_USER}
>>>             UNSHARE
>>>         ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>>    }
>>>
>>>    dpkg_runbuild() {
>>>         ( _runbuild ${PACKAGE_ARCH} )
>>>    }
>>>
>>>    PS: I am not very happy with the need to feed the script to execute
>>>    under unshare
>>>         via stdin, if there are better ways, we would be happy to
>>>    consider them!
> 
> When I started talking to Jan, I proposed adding an abstraction layer
> for package building. Work on that layer has now progressed under the
> name "mdbp" and source is available at
> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
> Raphael Hertzog on unifying the API with debusine. Let us for a moment
> consider the implications of using mdbp here.
> 
>  * Much of the complexity would go away. What you are left with is
>    writing a json file describing how you want your package built. What
>    gets a little more difficult is getting the isar-apt repository past
>    mdbp. Likely that would require a temporary http server.

Why can't that also support a local folder with a apt repo? Spawning
servers is surely doable but can become fun (unless using containers...)
when running multiple build in parallel.

>  * mdbp is not another builder, but an adapter to existing ones. It can
>    perform your builds using an existing sbuild or pbuilder
>    installation. If you want more isolation, maybe using debspawn
>    (backed by systemd-nspawn) is for you?
>  * mdbp also provides a stateless backend that uses mmdebstrap. This
>    backend performs the build in a user namespace.

If that is not going to be rebuilt for every package (cached baseline,
some dependency versions for all built packages in a run) and could
possibly also be reused for other things (running the imaging), it could
be a building block to replace our buildchroots. If not, we would have
to continue feeding it with separately bootstrapped chroots.

>  * If you decide that you prefer building in docker, we can add a
>    backend for e.g. debocker or something else.

Most of the Isar builds (complete images, not just packages) are already
happening inside containers (kas-isar). Thus we rather need to be
careful not requiring any nesting that could become complex - at best.
At the same time, we want Isar to continue work on a plain Debian system.

> 
> I'm not sure what you'd be missing by using mdbp here, but one thing
> you'd certainly miss is quite a bit of complex code.
> 
> If you want to consider this route, read the schema first:
> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
> 

The key requirements are listed above. Baurzhan looked into mdbp
already, maybe he can comment on further details.

The key question for us remains what value the abstraction can have for
Isar, today and on the long run. Switching builders is not a daily
business so far. Isar needs one, and that needs to plug well into the
image creation workflow. If interacting with mdbp can be simpler than
talking to sbuild directly, that would be a clear gain. If it's rather
more complex under the Isar constraints while we do not have the
compelling use case for switching builders, it is likely not the best
way forward.

Thanks,
Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux