Re: [RFC] using lightweight containers instead of chroot

[Jan Kiszka <jan.kiszka@siemens.com>]

On 08.07.21 11:07, Cedric Hombourger wrote:
> Hello,
> 
> This is a RFC for us to replace uses of chroot within the Isar classes
> to use Linux namespaces.
> 
> Motivation
> 
>    We will be submitting another RFC for per-recipe buildchroots but we

Good that you are sharing this: Baurzhan and his team are currently
working on sbuilder support with separate buildchroots. The idea of
using sbuilder is to get closer to how Debian builds packages as well.

Longterm, there is also the desire to include support for DPKG_ROOT as
chroot-less way of building packages, faster when doing it cross and
also without special permissions (e.g. for qemu-user). But that requires
per-package support from Debian upstream. Discussions only started, in
particular with Helmuth (added to CC).

>    should note that it would likely
>    increase the number of active bind mounts during our builds. So
>    before we discuss that, we may want to
>    discuss how we run our build scripts today using chroots.
> 
>    There are numerous challenges with our current implementation and
>    attempts to , Examples would include:
> 
>        1e31d5a events: Warn if mounted paths left
>        d21d495 dpkg: Make mount buildroot reliable
>        22c42de dpkg-base: Warn about unmounting problems
>        ...
> 
>    And you have found yourself in the situation where Isar wiped out
>    /dev or your layers (at least I did)
> 
>    chroot is a powerful tool but it is starting to show its age and we
>    see how much burden is on us to setup and
>    shutdown our chroot environment
> 
> Proposal
> 
>    We may want to use unshare(1) to create a mount namespace where we
>    will create our bind mounts,
>    chroot into the buildchroot and run the specified command/script
> 
>    The immediate benefit of this approach is that all mounts
>    automatically disappear as the supplied
>    command exits (whether it aborts prematurely because of an error or
>    normally on completion).
> 
>    Another nice benefit is that bind mounts we created within this
>    namespace are not (directly) visible
>    from the parent namespace
> 
>    However, we found that running scripts within an unshare environment
>    may not be as easy as
>    chroot. We would welcome feedback on the code snippets provided
>    below if you happen to have
>    some better ideas.

I suspect Helmuth can tell us if that would take us on a fragile path
from Debian perspective. Isar-internal implementation details we could
likely sort out, but if that approach has architectural limits /wrt what
Debian packages expect/require, it might be the wrong direction.

> 
>    def isar_user_spec():
>         import os
>         return '%d:%d' % (os.getuid(), os.getgid())
> 
>    ISAR_USER_SPEC    = "${@ isar_user_spec() }"
>    ISAR_UNSHARE_CMD  = "sudo unshare --pid --fork --ipc --mount sh -ex"
>    ISAR_CHROOT_SHELL = "sh -ex"
>    ISAR_CHROOT_ROOT  = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>    ISAR_CHROOT_USER  = "chroot --userspec='${ISAR_USER_SPEC}'
>    ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
> 
>    # Would be similar to buildchroot_do_mounts but will happen in a
>    separate mount namespace
>    BUILDCHROOT_DO_MOUNTS =
>    "                                                         \
>         mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>    '${BUILDCHROOT_DIR}/isar-apt'     ; \
>         mount --bind '${DL_DIR}'
>    '${BUILDCHROOT_DIR}/downloads'                     ; \
>         mount -t proc none
>    '${BUILDCHROOT_DIR}/proc'                                ; \
>         mount --rbind /sys
>    '${BUILDCHROOT_DIR}/sys'                                 ; \
>         mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>    ${BUILDCHROOT_DIR}/dev/shm  ; \
>         mount -o bind /dev/pts
>    ${BUILDCHROOT_DIR}/dev/pts                             \
>    "
> 
>    # Would be similar to dpkg_do_mounts but will happen in a separate
>    mount namespace
>    DPKG_DO_MOUNTS = "                         \
>         ${BUILDCHROOT_DO_MOUNTS}             ; \
>         mkdir -p ${BUILDROOT}                ; \
>         mount --bind ${WORKDIR} ${BUILDROOT}   \
>    "
> 
>    # Build package from sources using build script
>    _runbuild() {
>         export arch=${1}
> 
>         E="${@ isar_export_proxies(d)}"
>         (   cat <<"        UNSHARE"
>                 ${DPKG_DO_MOUNTS}
>                 (   cat <<"                SCRIPT"
>                         export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>                         export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>                         export PARALLEL_MAKE="${PARALLEL_MAKE}"
>                         /isar/build.sh ${PP}/${PPS} ${arch}
>                     SCRIPT
>                 ) | ${ISAR_CHROOT_USER}
>             UNSHARE
>         ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>    }
> 
>    dpkg_runbuild() {
>         ( _runbuild ${PACKAGE_ARCH} )
>    }
> 
>    PS: I am not very happy with the need to feed the script to execute
>    under unshare
>         via stdin, if there are better ways, we would be happy to
>    consider them!
> 
> Proposed Next steps
> 
>    1. Collect feedback and answer questions
>        a. is the use of unshare a good idea? (no is an OK answer!)
>        b. can we come up with a better code construct?
> 
>    2. Check compatibility with containerized-builds (e.g. from within a
>    kas build)
> 

That would obviously be a critical thing, compatibility with both
classic Docker and podman.

>    3. Check for better ways to spawn scripts
> 
>    4. Implement and submit RFC patches
> 
> Future RFCs
> 
>    - per-recipe buildchroots (mimic sbuild)
> 

Please coordinate here who could work on what and who already has
completed something so that we do not end up with two implementations
but rather one that is even better.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux