From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6982479695616933888 X-Received: by 2002:a7b:c2a2:: with SMTP id c2mr54744427wmk.89.1626078343732; Mon, 12 Jul 2021 01:25:43 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:4c50:: with SMTP id n16ls397832wrt.2.gmail; Mon, 12 Jul 2021 01:25:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwE7QxN/LYBbjTJ2qY0D+7+vhyss5eVWl3YKrH7mGl8+f8Mc6h/uuPpJLkWIWgyR0PZ3pWo X-Received: by 2002:a5d:598d:: with SMTP id n13mr41009561wri.246.1626078342779; Mon, 12 Jul 2021 01:25:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1626078342; cv=none; d=google.com; s=arc-20160816; b=HfhY5pkMno+9AKAKeMfasg+bjI6Y3dXFQSk1ovp3L5UhwB/cq5QDHSI1FIa+LuZPKz ccvpZN8fFHSmxkwmF1rbDU5O4ibD7EnWS265qd91hFJDVPLqBw0V9fTfMdK9gQSww4dU XR//9+ZE2NGM/L9j2U+S0Um/Nl6cyTwCAAEu76vMZ6Bj4QtgTBWRdK9o3peZqWzrGl8Y UW9cGSV8FFEMJc3szb4ANJ14V+vkIJjbT+tspcp5fgwx0A39QJLZOFQHOaS9IJAtvN0A 23c2TDfvYhfPIFDn77r5k2PYOgniFc03/6Q8aJhmuSzcKPHw9gURQI0hYXQv/1PqhxNe v7Qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=shzvaJryRebaGUhkW8gwB85cCA3lJsNRpYI5t0eCVHg=; b=FwdQrH7nAOgt4Scif3UWsTqsbzEy5qWmEpsbZ07hiLwUsEYLDp5P/Lu84mrzPXMl7I uKJQBrZZNqd+/A1cFm8JXlMbQT6K44TGlX492EDpDX+pnfELtPUTuP5f6WwqaVikfpjc n+4RtAwXeXEHnYypdIXqdBZilb7NW8k5FUFYTtFhuDD2e92WFGAgmKv0fh8G7UQkq5T6 JpWazCRgn4X5Z6nsH1zK7oVpgMa1T5rsYVNtVRyjkk2tPu7/OJt68JkkVEBP6HVs6DEc bwLHf3an3bLj5bfG3fGUhlXCLRW2p6uLYkHw6b5Twj7mrLf6OGlBre5mo9uVz2ubqhqh mhXA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id j10si469141wru.3.2021.07.12.01.25.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Jul 2021 01:25:42 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.40 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id 16C8PgXe029925 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Jul 2021 10:25:42 +0200 Received: from [167.87.3.139] ([167.87.3.139]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 16C8PfMM002733; Mon, 12 Jul 2021 10:25:41 +0200 Subject: Re: [RFC] using lightweight containers instead of chroot To: Helmut Grohne Cc: Cedric Hombourger , isar-users , Baurzhan Ismagulov , "MacDonald, Joe" References: <11b6ea24-b31a-a417-bcd9-0b32c5abe308@mentor.com> From: Jan Kiszka Message-ID: <009b570c-b5ba-a7ba-2db9-c3b77e14a9e3@siemens.com> Date: Mon, 12 Jul 2021 10:25:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 8Wh2SOQgrhEU On 08.07.21 21:34, Helmut Grohne wrote: > Hi Jan, > > On Thu, Jul 08, 2021 at 07:16:05PM +0200, Jan Kiszka wrote: >> Yeah, not cleanly separated: We need DPKG_ROOT to install build >> dependencies unprivileged into a build environment, and then we need >> something ideally unprivileged - e.g. namespaces - to run the build >> inside that environment. > > Still wrong. DPKG_ROOT is not meant to install build dependencies. I > doubt that it covers sufficient packages to do so any time soon. I also > don't see any point in doing so when you can just chroot (unprivileged > via user namespaces) into it and install them there. > > The reason to want DPKG_ROOT is to prepare the root filesystem for your > embedded board. If that's not what you want to do, DPKG_ROOT is not your > tool. We want it for both the board and the build env. We have the same issues to solve there, means installing packages in unprivileged manner, either for target arch or the builder arch (cross and native builds need to be supported). So I see DPKG_ROOT as a building block for both. In addition, it would overcome qemu-user, which is speed gain. But, as you mentioned, that it all long-term, nothing to achieve soon with a sufficiently large set of packages. It's still important to develop a vision of where we want to go once possible. > >> We don't want to reimplement if key requirements can be fulfilled with >> Debian's own tools. One of them is having control over how build >> dependencies are installed and from where (use cases: isar-apt, offline >> builds). Another one would be the ability to patch an original package >> after unpacking its sources. > > I think the goal should be making Debian's tools fulfil your key > requirements. (Regardless of whether using an abstraction layer or not.) > > Can you elaborate how you want to control your build dependencies? It > would seem to me that you could just patch debian/control and be done, > no? What else do you need to control? Do you want to change the > resolver? Yes, we prepare (ad-hoc debianization) or patch or just use debian/control for that. Packages should then be taken from the configured deb repos, including local isar-apt, and using a pre-defined apt database (only updated once so that all builds use the same package versions). Local caching of the fetched deb packages is used in addition, both as optimization as well as a building block for offline builds. > > The "from where" part seems relatively easy to me. All of the build > tools support one way or another to supply extra repositories. All you > need to do here is ensure that your extra packages from extra > repositories have a higher version than those in your base distribution. > > Why does your build tool need to support patching after unpacking > sources? I'm using that approach in rebootstrap and I consider it a > design mistake. You can just extract your source package outside the > builder, bump the version, apply your patches, create a new source > package and build that. Creating actual source packages saves you from a > lot of trouble. One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g. to inject patches before they are accepted upstream or to address issues that are not compatible with upstream packaging. Recent example: https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955 (some patches not even accepted upstream yet but needed). This procedure should be automated and expressed with very few statements. The current approaches we have are far from optimal, but the vision remains that you just tell Isar the package, the patch, and then it does whatever is best from Debian perspective. That may include the mentioned source package patching upfront. That will have to go into our recipes and classes. > > >> Why can't that also support a local folder with a apt repo? Spawning >> servers is surely doable but can become fun (unless using containers...) >> when running multiple build in parallel. > > I've considered that, but support for this is lacking at this point. > Ideally, you'd want to bind mount the local folder into the chroot. When > using user namespaces your mapped root user may be underprivileged to > see that local folder already. sbuild does not provide such means > (unless you modify the underlying schroot, which requires root). > pbuilder has --bindmounts (but doesn't use a user namespace). mmdebstrap > can only sync directories, which means copying them. > >> If that is not going to be rebuilt for every package (cached baseline, >> some dependency versions for all built packages in a run) and could >> possibly also be reused for other things (running the imaging), it could >> be a building block to replace our buildchroots. If not, we would have >> to continue feeding it with separately bootstrapped chroots. > > Where does that "not rebuilt" requirement come from? Is that for > performance reasons? Sure. We are not building hundreds of packages, but it also not only about building one or two. The other reason is the mentioned identical versions in all buildchroots. Reproducing them must provide identical content (where common) for a single image build run. > > If yes, please benchmark it. I've locally compared build times > (sbuild/pbuilder/mmdebstrap) and the difference is small (<10 seconds). > After all, extracting a pbuilder (or sbuild) base.tgz is not immediate > either and mmdebstrap is quite fast. > >> Most of the Isar builds (complete images, not just packages) are already >> happening inside containers (kas-isar). Thus we rather need to be >> careful not requiring any nesting that could become complex - at best. >> At the same time, we want Isar to continue work on a plain Debian system. > > You cannot use user namespaces in secure docker containers anyway, but > then when you already are inside a container, you can assume root > privileges and use chroot normally. Currently, we have no secure container env. The container simply serves as deployment tool of Isar's dependencies and builder-side configurations so that you can run it on any distro, not only Debian. And you can run it in CI identically to your host. > >> The key question for us remains what value the abstraction can have for >> Isar, today and on the long run. Switching builders is not a daily >> business so far. Isar needs one, and that needs to plug well into the >> image creation workflow. If interacting with mdbp can be simpler than >> talking to sbuild directly, that would be a clear gain. If it's rather >> more complex under the Isar constraints while we do not have the >> compelling use case for switching builders, it is likely not the best >> way forward. > > I think the value is not to be gained by you, but by Isar users. You > already found your preferred builder and it is easy to stick to that. > Others may have set up a different one already. The value gained from > the abstraction is to be able to seamlessly use the builder of the > user's choice. Isar users use Isar, they usually do not modify the core. And the goal of Isar is to avoid that they have to but rather provide easy means to configure things that must be configurable. If the builder would fall under this, e.g. no one build would build all types of packages (hope that is not the case, so far it wasn't), it would become an Isar requirement to switching them without much effort. > > Another aspect is the need for avoiding nesting of virtualization > technology. When building inside kas-isar, you cannot use user > namespaces (because secure docker containers cannot). When building on a > plain Debian system, you want to use user namespaces to be able to build > unprivileged. At present, the only builder that allows switching between > user namespaces and no user namespaces is sbuild. mdbp also gives that > flexibility by allowing you to seamlessly swap builders. > > So if sbuild suits your need, then go for it rather than reinventing the > wheel. > > I caution that using sbuild in a reproducible way is hard. People do > change its defaults to their needs. You need to pass a lot of options to > sbuild to avoid using user's defaults. That was one of the lessons I > learned the hard way when writing mdbp. > That is a valid point we need to keep an eye on. If mdbp solved that already, it would be one of the added values I mentioned that could make it relevant for Isar, even if not requiring to switch builders. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux