From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6982479695616933888 X-Received: by 2002:a05:6402:d7:: with SMTP id i23mr37821705edu.291.1625744283920; Thu, 08 Jul 2021 04:38:03 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:e4f:: with SMTP id q15ls1120414eji.5.gmail; Thu, 08 Jul 2021 04:38:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxrXhFkFlg/ldEZ7hvWv1P8s1Cn8DqDHcgwCArP5Bb6zdyBzy/wyJ6iagMYT1CFDM5JyP49 X-Received: by 2002:a17:906:5d13:: with SMTP id g19mr30931342ejt.90.1625744282955; Thu, 08 Jul 2021 04:38:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1625744282; cv=none; d=google.com; s=arc-20160816; b=B5MhgFd0Tn2vQwuZBthTV5PXq2KCJwPXsnVqVIwqWYIZRTQSd3wK/sLVKan1Dzp0wC SdI02FDElvuLCI8c393fsTCZ5fX/a5PNF5eWbQu96zZirvpg/vfcU6UsMhasKx5doKXI 5nMI2oUejRl+GcrpIRBpqDvnOMJPilULTRJhtuMTjaggRcJXMS/mpqk3Xf9f1m4PtnDb 49VYQLt0yPwZo+j0Xu1NKEV0k9CsK5lEynW758GXom/4G7WMju/CB/u4PE+KG9dXEHB3 U5zs6lPKX2QjMzk+xw3SThCz9q1tfcO5nIWWHbYm0muaBN+mbC3wBYgNVHfIXGh2C15G h8qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=kq+EV/BMaX74sCvj5IMOU9VQQAX+tlAkFuExP76OOJs=; b=L6vNLloRuVjNYtINlHAlEdMmGTJA6mrm/StpVsJDGUZ8Gz/qE3DdEh2O4hBa6qGOat OF1gQHgCbONuSxrxXEnyLt4Gyp94kxh6RsTUnh62P+lypRpFZWVo6xdWrcGC8id0fIro Po8kxGcAmtxFAREa9zlbUGCZAazMzpjrS4HYr9beKtp8Cw1oRw93qGoYWIoe9ai1tFIE JoFWdtSjMj33uzgaJTVKpinOKeCr7ktp3wzEmO3JwQNyoX3bYHnN9YCrUfWjfy7VSmgI JCC13K5EpkzkTtu6RGZXW3jsrFR+w/49c2ffBUWdyobrrjz4iYeI/Y+3fOM18LZ9bXFz kAeQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id s18si106240ejo.1.2021.07.08.04.38.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Jul 2021 04:38:02 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id 168Bc2WZ013667 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 8 Jul 2021 13:38:02 +0200 Received: from [167.87.42.31] ([167.87.42.31]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 168Bc1Il015189; Thu, 8 Jul 2021 13:38:02 +0200 Subject: Re: [RFC] using lightweight containers instead of chroot To: Cedric Hombourger , isar-users , Baurzhan Ismagulov , Helmut Grohne Cc: "MacDonald, Joe" References: <11b6ea24-b31a-a417-bcd9-0b32c5abe308@mentor.com> From: Jan Kiszka Message-ID: Date: Thu, 8 Jul 2021 13:38:01 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <11b6ea24-b31a-a417-bcd9-0b32c5abe308@mentor.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: L3S86brfMyDt On 08.07.21 11:07, Cedric Hombourger wrote: > Hello, > > This is a RFC for us to replace uses of chroot within the Isar classes > to use Linux namespaces. > > Motivation > >    We will be submitting another RFC for per-recipe buildchroots but we Good that you are sharing this: Baurzhan and his team are currently working on sbuilder support with separate buildchroots. The idea of using sbuilder is to get closer to how Debian builds packages as well. Longterm, there is also the desire to include support for DPKG_ROOT as chroot-less way of building packages, faster when doing it cross and also without special permissions (e.g. for qemu-user). But that requires per-package support from Debian upstream. Discussions only started, in particular with Helmuth (added to CC). >    should note that it would likely >    increase the number of active bind mounts during our builds. So >    before we discuss that, we may want to >    discuss how we run our build scripts today using chroots. > >    There are numerous challenges with our current implementation and >    attempts to , Examples would include: > >        1e31d5a events: Warn if mounted paths left >        d21d495 dpkg: Make mount buildroot reliable >        22c42de dpkg-base: Warn about unmounting problems >        ... > >    And you have found yourself in the situation where Isar wiped out >    /dev or your layers (at least I did) > >    chroot is a powerful tool but it is starting to show its age and we >    see how much burden is on us to setup and >    shutdown our chroot environment > > Proposal > >    We may want to use unshare(1) to create a mount namespace where we >    will create our bind mounts, >    chroot into the buildchroot and run the specified command/script > >    The immediate benefit of this approach is that all mounts >    automatically disappear as the supplied >    command exits (whether it aborts prematurely because of an error or >    normally on completion). > >    Another nice benefit is that bind mounts we created within this >    namespace are not (directly) visible >    from the parent namespace > >    However, we found that running scripts within an unshare environment >    may not be as easy as >    chroot. We would welcome feedback on the code snippets provided >    below if you happen to have >    some better ideas. I suspect Helmuth can tell us if that would take us on a fragile path from Debian perspective. Isar-internal implementation details we could likely sort out, but if that approach has architectural limits /wrt what Debian packages expect/require, it might be the wrong direction. > >    def isar_user_spec(): >         import os >         return '%d:%d' % (os.getuid(), os.getgid()) > >    ISAR_USER_SPEC    = "${@ isar_user_spec() }" >    ISAR_UNSHARE_CMD  = "sudo unshare --pid --fork --ipc --mount sh -ex" >    ISAR_CHROOT_SHELL = "sh -ex" >    ISAR_CHROOT_ROOT  = "chroot ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}" >    ISAR_CHROOT_USER  = "chroot --userspec='${ISAR_USER_SPEC}' >    ${BUILDCHROOT_DIR} ${ISAR_CHROOT_SHELL}" > >    # Would be similar to buildchroot_do_mounts but will happen in a >    separate mount namespace >    BUILDCHROOT_DO_MOUNTS = >    "                                                         \ >         mount --bind '${REPO_ISAR_DIR}/${DISTRO}' >    '${BUILDCHROOT_DIR}/isar-apt'     ; \ >         mount --bind '${DL_DIR}' >    '${BUILDCHROOT_DIR}/downloads'                     ; \ >         mount -t proc none >    '${BUILDCHROOT_DIR}/proc'                                ; \ >         mount --rbind /sys >    '${BUILDCHROOT_DIR}/sys'                                 ; \ >         mount -t tmpfs -o rw,nosuid,nodev,seclabel none >    ${BUILDCHROOT_DIR}/dev/shm  ; \ >         mount -o bind /dev/pts >    ${BUILDCHROOT_DIR}/dev/pts                             \ >    " > >    # Would be similar to dpkg_do_mounts but will happen in a separate >    mount namespace >    DPKG_DO_MOUNTS = "                         \ >         ${BUILDCHROOT_DO_MOUNTS}             ; \ >         mkdir -p ${BUILDROOT}                ; \ >         mount --bind ${WORKDIR} ${BUILDROOT}   \ >    " > >    # Build package from sources using build script >    _runbuild() { >         export arch=${1} > >         E="${@ isar_export_proxies(d)}" >         (   cat <<"        UNSHARE" >                 ${DPKG_DO_MOUNTS} >                 (   cat <<"                SCRIPT" >                         export DEB_BUILD_OPTIONS="${DEB_BUILD_OPTIONS}" >                         export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}" >                         export PARALLEL_MAKE="${PARALLEL_MAKE}" >                         /isar/build.sh ${PP}/${PPS} ${arch} >                     SCRIPT >                 ) | ${ISAR_CHROOT_USER} >             UNSHARE >         ) | envsubst '$arch' | ${ISAR_UNSHARE_CMD} >    } > >    dpkg_runbuild() { >         ( _runbuild ${PACKAGE_ARCH} ) >    } > >    PS: I am not very happy with the need to feed the script to execute >    under unshare >         via stdin, if there are better ways, we would be happy to >    consider them! > > Proposed Next steps > >    1. Collect feedback and answer questions >        a. is the use of unshare a good idea? (no is an OK answer!) >        b. can we come up with a better code construct? > >    2. Check compatibility with containerized-builds (e.g. from within a >    kas build) > That would obviously be a critical thing, compatibility with both classic Docker and podman. >    3. Check for better ways to spawn scripts > >    4. Implement and submit RFC patches > > Future RFCs > >    - per-recipe buildchroots (mimic sbuild) > Please coordinate here who could work on what and who already has completed something so that we do not end up with two implementations but rather one that is even better. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux