Re: [RFC] using lightweight containers instead of chroot

[Cedric Hombourger <cedric_hombourger@mentor.com>]

On 7/9/2021 5:46 PM, Cedric Hombourger wrote:
>
> On 7/8/2021 3:52 PM, Helmut Grohne wrote:
>> On Thu, Jul 08, 2021 at 01:38:01PM +0200, Jan Kiszka wrote:
>>> On 08.07.21 11:07, Cedric Hombourger wrote:
>>> ...
>>> Longterm, there is also the desire to include support for DPKG_ROOT as
>>> chroot-less way of building packages, faster when doing it cross and
>>> also without special permissions (e.g. for qemu-user). But that 
>>> requires
>>> per-package support from Debian upstream. Discussions only started, in
>>> particular with Helmuth (added to CC).
>> You appear to be confusing some aspects here. DPKG_ROOT is not relevant
>> to building packages. It is only relevant to installing them (which may
>> be relevant here for creating filesystem images).
>>
>> Building packages without chroot in a reproducible way seems next to
>> impossible to me. Even when you use user namespaces, chroot does not go
>> away. It merely becomes unprivileged. Is that what you mean here?
>>
>>>> Proposal
>>>>
>>>>     We may want to use unshare(1) to create a mount namespace where we
>>>>     will create our bind mounts,
>>>>     chroot into the buildchroot and run the specified command/script
>> Are you aware that sbuild directly supports this use case? It has a
>> --mode argument and one of its values is "unshare". In that case, you
>> supply a tarball containing the chroot and it'll perform an unprivileged
>> build inside an unshared chroot.
>
> I was not and that's very promising. I am now modifying the PoC code 
> to use it.
>
> Did I read correctly that we can tell sbuild to use an existing 
> directory for its chroot when using the "unshare" mode? I am asking 
> because that's failing for me (my host is on Debian/testing). Here's 
> the error

looks like --chroot=<dir> isn't supported yet for "unshare" as found in 
the sbuild code:

# FIXME: support directory chroots
#if (-d $path) {
# if ($file eq $chroot) {
# $tarball = $path;
# last;
# }
#} else {

>
>    copy() failed: Is a directory
>    tar: This does not look like a tar archive
>    tar: Exiting with failure status due to previous errors
>
> and here's the full command I ran (with the key args being -c 
> <existing-chroot-dir> --chroot-mode=unshare):
>
>    + sbuild -c
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
>    --chroot-mode=unshare -d industrial-os --no-apt-update -v
>    --pre-build-commands= mkdir -p
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
>    ;     mount --bind
>    /home/chombourger/unshare_sbuild/experimental/mel-apt
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/base-apt
>    ;     mount --bind
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/deploy/isar-apt/industrial-os-amd64/apt/industrial-os'
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/isar-apt'
>    ;     mount --bind
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/downloads'
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/downloads'
>    ;     mount -t proc none
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/proc'
>    ;     mount --rbind /sys
> '/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/sys'
>    ;     mount -t tmpfs -o rw,nosuid,nodev,seclabel none
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/shm
>    ;     mount -o bind /dev/pts
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/dev/pts
>    ;     mkdir -p
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
>    ;     mount --bind
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs//home/builder/base-files
>    --build-path=/home/builder/base-files/base-files-10.3+deb10u10 -D
>
> and it produced the following output:
>
>    dh clean
>    dh: warning: Compatibility levels before 10 are deprecated (level 9
>    in use)
>        dh_clean
>    dh_clean: warning: Compatibility levels before 10 are deprecated
>    (level 9 in use)
>    dpkg-source: info: using source format '3.0 (native)'
>    dpkg-source: info: building base-files in base-files_2.4+ind3.tar.xz
>    dpkg-source: info: building base-files in base-files_2.4+ind3.dsc
>    Selected distribution industrial-os
>    Selected chroot
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
>    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
>    D: Setting ABORT=undef
>    D: Setting
> Job=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Setting Build Dir=
>    D: Setting Max Lock Trys=120
>    D: Setting Lock Interval=5
>    D: Setting Pkg Status=pending
>    D: Setting Pkg Status Trigger=undef
>    D: Setting Pkg Start Time=0
>    D: Setting Pkg End Time=0
>    D: Setting Pkg Fail Stage=init
>    D: Setting Build Start Time=0
>    D: Setting Build End Time=0
>    D: Setting Install Start Time=0
>    D: Setting Install End Time=0
>    D: Setting This Time=0
>    D: Setting This Space=0
>    D: Setting Sub Task=initialisation
>    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
>    D: Setting Session ID=
>    D: Setting Chroot ID=/
>    D: Setting Defaults=HASH(0x55a7fc2a7e48)
>    D: Setting Split=1
>    D: Setting Split=0
>    D: Setting Host=Sbuild::ChrootRoot=HASH(0x55a7fc2a8010)
>    D: Setting Priority=0
>    D: Setting Location=/
>    D: Setting Session Purged=0
>    D: Setting Session=undef
>    D: Setting Dependency Resolver=undef
>    D: Setting Log File=undef
>    D: Setting Log Stream=undef
>    D: Setting Summary Stats=HASH(0x55a7fc283c70)
>    D: Setting dpkg-buildpackage pid=undef
>    D: Setting Dpkg Version=undef
>    D: Setting DSC:
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Setting
> DSC=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Setting Source
> Dir=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
>    D: Setting DSC Base=base-files_2.4+ind3.dsc
>    D: DSC =
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Source Dir =
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0
>    D: DSC Base = base-files_2.4+ind3.dsc
>    D: Setting package version:
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Parsing
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3.dsc
>    D: Setting Package=base-files
>    D: Setting Version=1:2.4+ind3
>    D: Setting Package_Version=base-files_1:2.4+ind3
>    D: Setting Package_OVersion=base-files_1:2.4+ind3
>    D: Setting Package_OSVersion=base-files_2.4+ind3
>    D: Setting Package_SVersion=base-files_2.4+ind3
>    D: Setting OVersion=1:2.4+ind3
>    D: Setting OSVersion=2.4+ind3
>    D: Setting SVersion=2.4+ind3
>    D: Setting VersionEpoch=1
>    D: Setting VersionUpstream=2.4+ind3
>    D: Setting VersionDebian=
>    D: Setting DSC File=base-files_2.4+ind3.dsc
>    D: Setting DSC Dir=base-files-2.4+ind3
>    D: Package = base-files
>    D: Version = 1:2.4+ind3
>    D: Package_Version = base-files_1:2.4+ind3
>    D: Package_OVersion = base-files_1:2.4+ind3
>    D: Package_OSVersion = base-files_2.4+ind3
>    D: Package_SVersion = base-files_2.4+ind3
>    D: OVersion = 1:2.4+ind3
>    D: OSVersion = 2.4+ind3
>    D: SVersion = 2.4+ind3
>    D: VersionEpoch = 1
>    D: VersionUpstream = 2.4+ind3
>    D: VersionDebian =
>    D: DSC File = base-files_2.4+ind3.dsc
>    D: DSC Dir = base-files-2.4+ind3
>    D: Setting Pkg Status Trigger=CODE(0x55a7fc2300c8)
>    D: Setting Pkg Status=building
>    D: Setting Pkg Start Time=1625844658
>    D: Setting Pkg End Time=1625844658
>    D: Setting Host Arch=amd64
>    D: Setting Build Arch=amd64
>    D: Setting Build Profiles=
>    D: Setting Build Type=binary
>    D: Setting FILTER_PREFIX=__SBUILD_FILTER_1412690:
>    D: Setting COLOUR_PREFIX=__SBUILD_COLOUR_1412690:
>    D: Setting Log
> File=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/2.4+ind3-r0/base-files_2.4+ind3_amd64-2021-07-09T15:30:58Z.build
>    D: Setting Log Stream=GLOB(0x55a7fc28c788)
>    sbuild (Debian sbuild) 0.81.2 (31 January 2021) on build.local
>
> +==============================================================================+
>    | base-files 1:2.4+ind3 (amd64)                Fri, 09 Jul 2021
>    15:30:58 +0000 |
> +==============================================================================+
>
>    Package: base-files
>    Version: 1:2.4+ind3
>    Source Version: 1:2.4+ind3
>    Distribution: industrial-os
>    Machine Architecture: amd64
>    Host Architecture: amd64
>    Build Architecture: amd64
>    Build Type: binary
>
>    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
>    D: Setting Chroots=HASH(0x55a7fc28ff88)
>    I: No tarballs found in /home/builder/.cache/sbuild
>    D: Setting Chroots=HASH(0x55a7fc298bd0)
>    D: Setting Config=Sbuild::ConfBase=HASH(0x55a7fa84d508)
>    D: Setting Session ID=
>    D: Setting Chroot
> ID=/home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/
>    D: Setting Defaults=HASH(0x55a7fc362e08)
>    D: Setting Chroots=Sbuild::ChrootInfoUnshare=HASH(0x55a7fc2880a0)
>    D: Setting Uid Gid Map=ARRAY(0x55a7fbdf7608)
>    running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
>    = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
>    {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
>    receive EOF";0 == system "newuidmap $ppid  0 1000 1 1 100000 1" or
>    die "newuidmap failed: $!";0 == system "newgidmap $ppid  0 1000 1 1
>    100000 1" or die "newgidmap failed: $!";exit 0;}0 == syscall
>    &SYS_unshare, 268435456 or die "unshare() failed: $!";close
>    $wfh;$cpid == waitpid $cpid, 0 or die "waitpid() failed: $!";if ($?
>    != 0) {die "child had a non-zero exit status: $?";}0 == syscall
>    &SYS_setgid, 0 or die "setgid failed: $!";0 == syscall &SYS_setuid,
>    0 or die "setuid failed: $!";0 == syscall &SYS_setgroups, 0, 0 or
>    die "setgroups failed: $!";exec { $ARGV[0] } @ARGV or die "exec()
>    failed: $!"; chown 1:1 /tmp/tmp.sbuild.l7HJK58Vy2
>    Unpacking
> /home/chombourger/unshare_sbuild/experimental/build-ipc/tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs
>    to /tmp/tmp.sbuild.l7HJK58Vy2...
>    running perl -e require 'syscall.ph';pipe my $rfh, my $wfh;my $ppid
>    = $$;my $cpid = fork() // die "fork() failed: $!";if ($cpid == 0)
>    {close $wfh;0 == sysread $rfh, my $c, 1 or die "read() did not
>    receive EOF";0 == system "newuidmap $ppid  0 100000 65536" or die
>    "newuidmap failed: $!";0 == system "newgidmap $ppid  0 100000 65536"
>    or die "newgidmap failed: $!";exit 0;}0 == syscall &SYS_unshare,
>    268435456 or die "unshare() failed: $!";close $wfh;$cpid == waitpid
>    $cpid, 0 or die "waitpid() failed: $!";if ($? != 0) {die "child had
>    a non-zero exit status: $?";}0 == syscall &SYS_setgid, 0 or die
>    "setgid failed: $!";0 == syscall &SYS_setuid, 0 or die "setuid
>    failed: $!";0 == syscall &SYS_setgroups, 0, 0 or die "setgroups
>    failed: $!";exec { $ARGV[0] } @ARGV or die "exec() failed: $!"; tar
>    --exclude=./dev/urandom --exclude=./dev/random --exclude=./dev/full
>    --exclude=./dev/null --exclude=./dev/console --exclude=./dev/zero
>    --exclude=./dev/tty --exclude=./dev/ptmx --directory
>    /tmp/tmp.sbuild.l7HJK58Vy2 --extract
>    copy() failed: Is a directory
>    tar: This does not look like a tar archive
>    tar: Exiting with failure status due to previous errors
>    D: Error run_chroot_session(): Error creating chroot session:
>    skipping base-filesD: Setting Session=undef
>    D: Error run_chroot(): Error creating chroot session: skipping
>    base-filesE: Error creating chroot session: skipping base-files
>    D: Setting Pkg Status=failed
>    D: Setting Pkg Fail Stage=create-session
>
>
> The extra bind-mounts would be needed for the following apt source to 
> be work:
>
>    $ cat
> tmp/work/industrial-os-amd64/base-files/buildchroot/rootfs/etc/apt/sources.list.d/isar-apt.list
>    deb [trusted=yes] file:///isar-apt mel main
>
>>
>>>>     The immediate benefit of this approach is that all mounts
>>>>     automatically disappear as the supplied
>>>>     command exits (whether it aborts prematurely because of an 
>>>> error or
>>>>     normally on completion).
>>>>
>>>>     Another nice benefit is that bind mounts we created within this
>>>>     namespace are not (directly) visible
>>>>     from the parent namespace
>>>>
>>>>     However, we found that running scripts within an unshare 
>>>> environment
>>>>     may not be as easy as
>>>>     chroot. We would welcome feedback on the code snippets provided
>>>>     below if you happen to have
>>>>     some better ideas.
>> All of this applies to sbuild --mode=unshare as well except that it
>> makes running scripts from hooks simple.
> it certainly would!
>>
>>> I suspect Helmuth can tell us if that would take us on a fragile path
>>> from Debian perspective. Isar-internal implementation details we could
>>> likely sort out, but if that approach has architectural limits /wrt 
>>> what
>>> Debian packages expect/require, it might be the wrong direction.
>> Reimplementing this functionality seems like a waste of time to me. If
>> we ignore that for a moment, we notice that there are already ~10
>> implementations and updating them all is painful. Therefore, we can
>> conclude that changes to the build environment are rare and your
>> reimplementation likely is maintainable with limited effort.
>>
>>>>     def isar_user_spec():
>>>>          import os
>>>>          return '%d:%d' % (os.getuid(), os.getgid())
>>>>
>>>>     ISAR_USER_SPEC    = "${@ isar_user_spec() }"
>>>>     ISAR_UNSHARE_CMD  = "sudo unshare --pid --fork --ipc --mount sh 
>>>> -ex"
>>>>     ISAR_CHROOT_SHELL = "sh -ex"
>>>>     ISAR_CHROOT_ROOT  = "chroot ${BUILDCHROOT_DIR} 
>>>> ${ISAR_CHROOT_SHELL}"
>>>>     ISAR_CHROOT_USER  = "chroot --userspec='${ISAR_USER_SPEC}'
>>>>     ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}"
>>>>
>>>>     # Would be similar to buildchroot_do_mounts but will happen in a
>>>>     separate mount namespace
>>>>     BUILDCHROOT_DO_MOUNTS =
>>>> "                                                         \
>>>>          mount --bind '${REPO_ISAR_DIR}/${DISTRO}'
>>>>     '${BUILDCHROOT_DIR}/isar-apt'     ; \
>>>>          mount --bind '${DL_DIR}'
>>>>     '${BUILDCHROOT_DIR}/downloads'                     ; \
>>>>          mount -t proc none
>>>>     '${BUILDCHROOT_DIR}/proc' ; \
>>>>          mount --rbind /sys
>>>>     '${BUILDCHROOT_DIR}/sys' ; \
>>>>          mount -t tmpfs -o rw,nosuid,nodev,seclabel none
>>>>     ${BUILDCHROOT_DIR}/dev/shm  ; \
>>>>          mount -o bind /dev/pts
>>>>     ${BUILDCHROOT_DIR}/dev/pts                             \
>>>>     "
>>>>
>>>>     # Would be similar to dpkg_do_mounts but will happen in a separate
>>>>     mount namespace
>>>>     DPKG_DO_MOUNTS = "                         \
>>>>          ${BUILDCHROOT_DO_MOUNTS}             ; \
>>>>          mkdir -p ${BUILDROOT}                ; \
>>>>          mount --bind ${WORKDIR} ${BUILDROOT}   \
>>>>     "
>>>>
>>>>     # Build package from sources using build script
>>>>     _runbuild() {
>>>>          export arch=${1}
>>>>
>>>>          E="${@ isar_export_proxies(d)}"
>>>>          (   cat <<"        UNSHARE"
>>>>                  ${DPKG_DO_MOUNTS}
>>>>                  (   cat <<"                SCRIPT"
>>>>                          export 
>>>> DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}"
>>>>                          export 
>>>> DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
>>>>                          export PARALLEL_MAKE="${PARALLEL_MAKE}"
>>>>                          /isar/build.sh ${PP}/${PPS} ${arch}
>>>>                      SCRIPT
>>>>                  ) | ${ISAR_CHROOT_USER}
>>>>              UNSHARE
>>>>          ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD}
>>>>     }
>>>>
>>>>     dpkg_runbuild() {
>>>>          ( _runbuild ${PACKAGE_ARCH} )
>>>>     }
>>>>
>>>>     PS: I am not very happy with the need to feed the script to 
>>>> execute
>>>>     under unshare
>>>>          via stdin, if there are better ways, we would be happy to
>>>>     consider them!
>> When I started talking to Jan, I proposed adding an abstraction layer
>> for package building. Work on that layer has now progressed under the
>> name "mdbp" and source is available at
>> https://git.subdivi.de/?p=~helmut/mdbp.git. I'm also working with
>> Raphael Hertzog on unifying the API with debusine. Let us for a moment
>> consider the implications of using mdbp here.
> Thanks for the pointer. Yet another thing I probably want to look at
>>
>>   * Much of the complexity would go away. What you are left with is
>>     writing a json file describing how you want your package built. What
>>     gets a little more difficult is getting the isar-apt repository past
>>     mdbp. Likely that would require a temporary http server.
>>   * mdbp is not another builder, but an adapter to existing ones. It can
>>     perform your builds using an existing sbuild or pbuilder
>>     installation. If you want more isolation, maybe using debspawn
>>     (backed by systemd-nspawn) is for you?
>>   * mdbp also provides a stateless backend that uses mmdebstrap. This
>>     backend performs the build in a user namespace.
>>   * If you decide that you prefer building in docker, we can add a
>>     backend for e.g. debocker or something else.
>>
>> I'm not sure what you'd be missing by using mdbp here, but one thing
>> you'd certainly miss is quite a bit of complex code.
>>
>> If you want to consider this route, read the schema first:
>> https://git.subdivi.de/?p=~helmut/mdbp.git;a=blob;f=mdbp/build_schema.json 
>>
>>
>> Helmut
>>