Re: [RFC] using lightweight containers instead of chroot

[Cedric Hombourger <cedric_hombourger@mentor.com>]

On 7/8/2021 3:52 PM, Helmut Grohne wrote:
> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>> On 08.07.21 11:07, Cedric Hombourger wrote:
>> ...
>> Longterm, there is also the desire to include support for DPKG_ROOT as
>> chroot-less way of building packages, faster when doing it cross and
>> also without special permissions (e.g. for qemu-user). But that requires
>> per-package support from Debian upstream. Discussions only started, in
>> particular with Helmuth (added to CC).
> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
> to building packages. It is only relevant to installing them (which may
> be relevant here for creating filesystem images).
>
> Building packages without chroot in a reproducible way seems next to
> impossible to me. Even when you use user namespaces, chroot does not go
> away. It merely becomes unprivileged. Is that what you mean here?
>
>>> Proposal
>>>
>>>     We may want to use unshare(1) to create a mount namespace where we
>>>     will create our bind mounts,
>>>     chroot into the buildchroot and run the specified command/script
> Are you aware that sbuild directly supports this use case? It has a
> --mode argument and one of its values is "unshare". In that case, you
> supply a tarball containing the chroot and it'll perform an unprivileged
> build inside an unshared chroot.

I was not and that's very promising. I am now modifying the PoC code to 
use it.

Did I read correctly that we can tell sbuild to use an existing 
directory for its chroot when using the "unshare" mode? I am asking 
because that's failing for me (my host is on Debian/testing). Here's the 
error

    copy() failed: Is a directory
    tar: This does not look like a tar archive
    tar: Exiting with failure status due to previous errors

and here's the full command I ran (with the key args being -c 
<existing-chroot-dir> --chroot-mode=unshare):

    + sbuild -c
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
    --chroot-mode=unshare -d industrial-os --no-apt-update -v
    --pre-build-commands= mkdir -p
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
    ;     mount --bind
    /home/chombourger/unshare_sbuild/experimental/mel-apt
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
    ;     mount --bind
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/deploy/isar-apt/industrial-os-amd64/apt/industrial-os'
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/isar-apt'
    ;     mount --bind
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/downloads'
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/downloads'
    ;     mount -t proc none
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/proc'
    ;     mount --rbind /sys
    '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/sys'
    ;     mount -t tmpfs -o rw,nosuid,nodev,seclabel none
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/shm
    ;     mount -o bind /dev/pts
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/pts
    ;     mkdir -p
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
    ;     mount --bind
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
    --build-path=/home/builder/base-files/base-files-10.3+deb10u10 -D

and it produced the following output:

    dh clean
    dh: warning: Compatibility levels before 10 are deprecated (level 9
    in use)
        dh_clean
    dh_clean: warning: Compatibility levels before 10 are deprecated
    (level 9 in use)
    dpkg-source: info: using source format '3.0 (native)'
    dpkg-source: info: building base-files in base-files_2.4+ind3.tar.xz
    dpkg-source: info: building base-files in base-files_2.4+ind3.dsc
    Selected distribution industrial-os
    Selected chroot
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
    D: Setting ABORT=undef
    D: Setting
    Job=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Setting Build Dir=
    D: Setting Max Lock Trys=120
    D: Setting Lock Interval=5
    D: Setting Pkg Status=pending
    D: Setting Pkg Status Trigger=undef
    D: Setting Pkg Start Time=0
    D: Setting Pkg End Time=0
    D: Setting Pkg Fail Stage=init
    D: Setting Build Start Time=0
    D: Setting Build End Time=0
    D: Setting Install Start Time=0
    D: Setting Install End Time=0
    D: Setting This Time=0
    D: Setting This Space=0
    D: Setting Sub Task=initialisation
    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
    D: Setting Session ID=
    D: Setting Chroot ID=/
    D: Setting Defaults=HASH(0x55a7fc2a7e48)
    D: Setting Split=1
    D: Setting Split=0
    D: Setting Host=Sbuild::ChrootRoot=HASH(0x55a7fc2a8010)
    D: Setting Priority=0
    D: Setting Location=/
    D: Setting Session Purged=0
    D: Setting Session=undef
    D: Setting Dependency Resolver=undef
    D: Setting Log File=undef
    D: Setting Log Stream=undef
    D: Setting Summary Stats=HASH(0x55a7fc283c70)
    D: Setting dpkg-buildpackage pid=undef
    D: Setting Dpkg Version=undef
    D: Setting DSC:
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Setting
    DSC=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Setting Source
    Dir=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
    D: Setting DSC Base=base-files_2.4+ind3.dsc
    D: DSC =
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Source Dir =
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
    D: DSC Base = base-files_2.4+ind3.dsc
    D: Setting package version:
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Parsing
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
    D: Setting Package=base-files
    D: Setting Version=1:2.4+ind3
    D: Setting Package_Version=base-files_1:2.4+ind3
    D: Setting Package_OVersion=base-files_1:2.4+ind3
    D: Setting Package_OSVersion=base-files_2.4+ind3
    D: Setting Package_SVersion=base-files_2.4+ind3
    D: Setting OVersion=1:2.4+ind3
    D: Setting OSVersion=2.4+ind3
    D: Setting SVersion=2.4+ind3
    D: Setting VersionEpoch=1
    D: Setting VersionUpstream=2.4+ind3
    D: Setting VersionDebian=
    D: Setting DSC File=base-files_2.4+ind3.dsc
    D: Setting DSC Dir=base-files-2.4+ind3
    D: Package = base-files
    D: Version = 1:2.4+ind3
    D: Package_Version = base-files_1:2.4+ind3
    D: Package_OVersion = base-files_1:2.4+ind3
    D: Package_OSVersion = base-files_2.4+ind3
    D: Package_SVersion = base-files_2.4+ind3
    D: OVersion = 1:2.4+ind3
    D: OSVersion = 2.4+ind3
    D: SVersion = 2.4+ind3
    D: VersionEpoch = 1
    D: VersionUpstream = 2.4+ind3
    D: VersionDebian =
    D: DSC File = base-files_2.4+ind3.dsc
    D: DSC Dir = base-files-2.4+ind3
    D: Setting Pkg Status Trigger=CODE(0x55a7fc2300c8)
    D: Setting Pkg Status=building
    D: Setting Pkg Start Time=1625844658
    D: Setting Pkg End Time=1625844658
    D: Setting Host Arch=amd64
    D: Setting Build Arch=amd64
    D: Setting Build Profiles=
    D: Setting Build Type=binary
    D: Setting FILTER_PREFIX=__SBUILD_FILTER_1412690:
    D: Setting COLOUR_PREFIX=__SBUILD_COLOUR_1412690:
    D: Setting Log
    File=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3_amd64-2021-07-09T15:30:58Z.build
    D: Setting Log Stream=GLOB(0x55a7fc28c788)
    sbuild (Debian sbuild) 0.81.2 (31 January 2021) on build.local

    +==============================================================================+
    | base-files 1:2.4+ind3 (amd64)                Fri, 09 Jul 2021
    15:30:58 +0000 |
    +==============================================================================+

    Package: base-files
    Version: 1:2.4+ind3
    Source Version: 1:2.4+ind3
    Distribution: industrial-os
    Machine Architecture: amd64
    Host Architecture: amd64
    Build Architecture: amd64
    Build Type: binary

    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
    D: Setting Chroots=HASH(0x55a7fc28ff88)
    I: No tarballs found in /home/builder/.cache/sbuild
    D: Setting Chroots=HASH(0x55a7fc298bd0)
    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
    D: Setting Session ID=
    D: Setting Chroot
    ID=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
    D: Setting Defaults=HASH(0x55a7fc362e08)
    D: Setting Chroots=Sbuild::ChrootInfoUnshare=HASH(0x55a7fc2880a0)
    D: Setting Uid Gid Map=ARRAY(0x55a7fbdf7608)
    running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
    = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
    {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
    receive EOF";0 == system "newuidmap $ppid  0 1000 1 1 100000 1" or
    die "newuidmap failed: $!";0 == system "newgidmap $ppid  0 1000 1 1
    100000 1" or die "newgidmap failed: $!";exit 0;}0 == syscall
    &SYS_unshare, 268435456 or die "unshare() failed: $!";close
    $wfh;$cpid == waitpid $cpid, 0 or die "waitpid() failed: $!";if ($?
    != 0) {die "child had a non-zero exit status: $?";}0 == syscall
    &SYS_setgid, 0 or die "setgid failed: $!";0 == syscall &SYS_setuid,
    0 or die "setuid failed: $!";0 == syscall &SYS_setgroups, 0, 0 or
    die "setgroups failed: $!";exec { $ARGV[0] } @ARGV or die "exec()
    failed: $!"; chown 1:1 /tmp/tmp.sbuild.l7HJK58Vy2
    Unpacking
    /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs
    to /tmp/tmp.sbuild.l7HJK58Vy2...
    running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
    = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
    {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
    receive EOF";0 == system "newuidmap $ppid  0 100000 65536" or die
    "newuidmap failed: $!";0 == system "newgidmap $ppid  0 100000 65536"
    or die "newgidmap failed: $!";exit 0;}0 == syscall &SYS_unshare,
    268435456 or die "unshare() failed: $!";close $wfh;$cpid == waitpid
    $cpid, 0 or die "waitpid() failed: $!";if ($? != 0) {die "child had
    a non-zero exit status: $?";}0 == syscall &SYS_setgid, 0 or die
    "setgid failed: $!";0 == syscall &SYS_setuid, 0 or die "setuid
    failed: $!";0 == syscall &SYS_setgroups, 0, 0 or die "setgroups
    failed: $!";exec { $ARGV[0] } @ARGV or die "exec() failed: $!"; tar
    --exclude=./dev/urandom --exclude=./dev/random --exclude=./dev/full
    --exclude=./dev/null --exclude=./dev/console --exclude=./dev/zero
    --exclude=./dev/tty --exclude=./dev/ptmx --directory
    /tmp/tmp.sbuild.l7HJK58Vy2 --extract
    copy() failed: Is a directory
    tar: This does not look like a tar archive
    tar: Exiting with failure status due to previous errors
    D: Error run_chroot_session(): Error creating chroot session:
    skipping base-filesD: Setting Session=undef
    D: Error run_chroot(): Error creating chroot session: skipping
    base-filesE: Error creating chroot session: skipping base-files
    D: Setting Pkg Status=failed
    D: Setting Pkg Fail Stage=create-session


The extra bind-mounts would be needed for the following apt source to be 
work:

    $ cat
    tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/etc/apt/sources.list.d/isar-apt.list
    deb [trusted=yes] file:///isar-apt mel main

>
>>>     The immediate benefit of this approach is that all mounts
>>>     automatically disappear as the supplied
>>>     command exits (whether it aborts prematurely because of an error or
>>>     normally on completion).
>>>
>>>     Another nice benefit is that bind mounts we created within this
>>>     namespace are not (directly) visible
>>>     from the parent namespace
>>>
>>>     However, we found that running scripts within an unshare environment
>>>     may not be as easy as
>>>     chroot. We would welcome feedback on the code snippets provided
>>>     below if you happen to have
>>>     some better ideas.
> All of this applies to sbuild --mode=unshare as well except that it
> makes running scripts from hooks simple.
it certainly would!
>
>> I suspect Helmuth can tell us if that would take us on a fragile path
>> from Debian perspective. Isar-internal implementation details we could
>> likely sort out, but if that approach has architectural limits /wrt what
>> Debian packages expect/require, it might be the wrong direction.
> Reimplementing this functionality seems like a waste of time to me. If
> we ignore that for a moment, we notice that there are already ~10
> implementations and updating them all is painful. Therefore, we can
> conclude that changes to the build environment are rare and your
> reimplementation likely is maintainable with limited effort.
>
>>>     def isar_user_spec():
>>>          import os
>>>          return '%d:%d' % (os.getuid(), os.getgid())
>>>
>>>     ISAR_USER_SPEC    = "${@ isar_user_spec() }"
>>>     ISAR_UNSHARE_CMD  = "sudo unshare --pid --fork --ipc --mount sh -ex"
>>>     ISAR_CHROOT_SHELL = "sh -ex"
>>>     ISAR_CHROOT_ROOT  = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>     ISAR_CHROOT_USER  = "chroot --userspec='${ISAR_USER_SPEC}'
>>>     ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>
>>>     # Would be similar to buildchroot_do_mounts but will happen in a
>>>     separate mount namespace
>>>     BUILDCHROOT_DO_MOUNTS =
>>>     "                                                         \
>>>          mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>>     '${BUILDCHROOT_DIR}/isar-apt'     ; \
>>>          mount --bind '${DL_DIR}'
>>>     '${BUILDCHROOT_DIR}/downloads'                     ; \
>>>          mount -t proc none
>>>     '${BUILDCHROOT_DIR}/proc'                                ; \
>>>          mount --rbind /sys
>>>     '${BUILDCHROOT_DIR}/sys'                                 ; \
>>>          mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>>     ${BUILDCHROOT_DIR}/dev/shm  ; \
>>>          mount -o bind /dev/pts
>>>     ${BUILDCHROOT_DIR}/dev/pts                             \
>>>     "
>>>
>>>     # Would be similar to dpkg_do_mounts but will happen in a separate
>>>     mount namespace
>>>     DPKG_DO_MOUNTS = "                         \
>>>          ${BUILDCHROOT_DO_MOUNTS}             ; \
>>>          mkdir -p ${BUILDROOT}                ; \
>>>          mount --bind ${WORKDIR} ${BUILDROOT}   \
>>>     "
>>>
>>>     # Build package from sources using build script
>>>     _runbuild() {
>>>          export arch=${1}
>>>
>>>          E="${@ isar_export_proxies(d)}"
>>>          (   cat <<"        UNSHARE"
>>>                  ${DPKG_DO_MOUNTS}
>>>                  (   cat <<"                SCRIPT"
>>>                          export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>>                          export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>>                          export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>                          /isar/build.sh ${PP}/${PPS} ${arch}
>>>                      SCRIPT
>>>                  ) | ${ISAR_CHROOT_USER}
>>>              UNSHARE
>>>          ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>>     }
>>>
>>>     dpkg_runbuild() {
>>>          ( _runbuild ${PACKAGE_ARCH} )
>>>     }
>>>
>>>     PS: I am not very happy with the need to feed the script to execute
>>>     under unshare
>>>          via stdin, if there are better ways, we would be happy to
>>>     consider them!
> When I started talking to Jan, I proposed adding an abstraction layer
> for package building. Work on that layer has now progressed under the
> name "mdbp" and source is available at
> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
> Raphael Hertzog on unifying the API with debusine. Let us for a moment
> consider the implications of using mdbp here.
Thanks for the pointer. Yet another thing I probably want to look at
>
>   * Much of the complexity would go away. What you are left with is
>     writing a json file describing how you want your package built. What
>     gets a little more difficult is getting the isar-apt repository past
>     mdbp. Likely that would require a temporary http server.
>   * mdbp is not another builder, but an adapter to existing ones. It can
>     perform your builds using an existing sbuild or pbuilder
>     installation. If you want more isolation, maybe using debspawn
>     (backed by systemd-nspawn) is for you?
>   * mdbp also provides a stateless backend that uses mmdebstrap. This
>     backend performs the build in a user namespace.
>   * If you decide that you prefer building in docker, we can add a
>     backend for e.g. debocker or something else.
>
> I'm not sure what you'd be missing by using mdbp here, but one thing
> you'd certainly miss is quite a bit of complex code.
>
> If you want to consider this route, read the schema first:
> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json
>
> Helmut
>