From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6982479695616933888 X-Received: by 2002:a5d:634c:: with SMTP id b12mr17611940wrw.238.1626087362177; Mon, 12 Jul 2021 03:56:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1d07:: with SMTP id l7ls1380896wms.3.canary-gmail; Mon, 12 Jul 2021 03:56:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJweDOoEdftzwrVb9x9ALYXjot8Va6koDs7BSPR4OiDDBq2I+T5o8+/cp3m8BC3X/iMlj5Oh X-Received: by 2002:a05:600c:4ecc:: with SMTP id g12mr3577190wmq.118.1626087361219; Mon, 12 Jul 2021 03:56:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1626087361; cv=none; d=google.com; s=arc-20160816; b=cEY4eLsOLG8kptKyy92VI2shQayNs8mT9N0Ji/j2z1jmrAANijDSakXVQGKmuVojF7 gf2OSRdoZGbzB/Ll78xd7WMFlZl/y7D/2RSG+oifYlD9PdEcKyZ/pNYWpESqEgLDs2Be CKyywF5rLZjj5Au49IOddYW+qtHpM4bpGt7uSecDIESD5CWn3Y4gIX8RxqHKtccPJvkx FLzQxTKOlYO6VMVuf9Y4rdDEPqdIuZW5FNZutnFdNgjM18yZT7A2mDMMopZU7CXcQeUX WzAzsJ3sCvg6Omn1MiD9nXgpkB2mq3iuRw0KdWKW5sdm7GMeIfq09inXMWm9Q45DhVH8 G3BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date; bh=hFce8WvMDqFn/S+npVLHIDkYdfKDK0vDI/rCNkqJmmI=; b=spuAjPe1WzUFzgN0FyIfDwhPsGWfqBzELdRVBhzScGJgDzV9XrX60aAAMgQchzkQ2s ov/spZ9QKmP1H6meG+qALipDyVhIgFU552SqmbedhmBrAMeefz/bXtOtMtejBt4Aistd jWHQOCWKkfrF87I7yYXJgLd1mMzGXBZ+8lNri4q/oo25xMhxfRvuWNNOQUIC4inF4v1B E156rxnCM9M67auPbxBV2Nt6xYTaGEw+LgaHPkc1nx1oCq9Z08Osir4UU8pHmss8hd96 n90Gbp85MqOkBG3+HlFeXAvoDTYrgRPq6XEDRiDSviB3My0FAnX12UD8vdecoBOp3xR1 pnmA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 136.243.71.142 is neither permitted nor denied by best guess record for domain of helmut@subdivi.de) smtp.mailfrom=helmut@subdivi.de Return-Path: Received: from isilmar-4.linta.de (isilmar-4.linta.de. [136.243.71.142]) by gmr-mx.google.com with ESMTPS id z14si573053wrs.0.2021.07.12.03.56.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 03:56:01 -0700 (PDT) Received-SPF: neutral (google.com: 136.243.71.142 is neither permitted nor denied by best guess record for domain of helmut@subdivi.de) client-ip=136.243.71.142; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 136.243.71.142 is neither permitted nor denied by best guess record for domain of helmut@subdivi.de) smtp.mailfrom=helmut@subdivi.de Received: from isilmar-4.linta.de (isilmar.linta [10.0.0.1]) by isilmar-4.linta.de (Postfix) with ESMTP id 5ED4A200239; Mon, 12 Jul 2021 10:55:59 +0000 (UTC) Date: Mon, 12 Jul 2021 12:54:13 +0200 From: Helmut Grohne To: Jan Kiszka Cc: Cedric Hombourger , isar-users , Baurzhan Ismagulov , "MacDonald, Joe" Subject: Re: [RFC] using lightweight containers instead of chroot Message-ID: References: <11b6ea24-b31a-a417-bcd9-0b32c5abe308@mentor.com> <009b570c-b5ba-a7ba-2db9-c3b77e14a9e3@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <009b570c-b5ba-a7ba-2db9-c3b77e14a9e3@siemens.com> X-TUID: E3z2MNE/b3GF Hi Jan, On Mon, Jul 12, 2021 at 10:25:41AM +0200, Jan Kiszka wrote: > On 08.07.21 21:34, Helmut Grohne wrote: > > The reason to want DPKG_ROOT is to prepare the root filesystem for your > > embedded board. If that's not what you want to do, DPKG_ROOT is not your > > tool. > > We want it for both the board and the build env. We have the same issues > to solve there, means installing packages in unprivileged manner, either > for target arch or the builder arch (cross and native builds need to be > supported). So I see DPKG_ROOT as a building block for both. In > addition, it would overcome qemu-user, which is speed gain. No. Let us for a moment assume that DPKG_ROOT would just work for all packages in Debian (and that is a far stretch really). Then you'd be able to install your Build-Depends into a directory without using chroot. But how do you build your package then? You need to chroot the dpkg-buildpackage call. So you are back to requiring chroot, at which point you can just install your whole build environment using chroot. DPKG_ROOT is the wrong tool for that job. The one thing that it fixes is requiring qemu for the embedded filesystem image. Cross builds already work entirely without qemu (including package installation). > Yes, we prepare (ad-hoc debianization) or patch or just use > debian/control for that. Packages should then be taken from the > configured deb repos, including local isar-apt, and using a pre-defined > apt database (only updated once so that all builds use the same package > versions). Local caching of the fetched deb packages is used in > addition, both as optimization as well as a building block for offline > builds. I'm sorry, but none of this explains why you need to patch a package after it is unpacked. Before building starts, you can just download all the packages that are to be patched, patch them and include the patched ones in isar-apt. At that point, you can run without hooks. > One purpose of Isar is ad-hoc fix-ups for upstream Debian packages. E.g. > to inject patches before they are accepted upstream or to address issues > that are not compatible with upstream packaging. Recent example: > https://github.com/siemens/meta-iot2050/commit/a1d35d5a3494236b2cdb0bd310c3ff0d4600d955 > (some patches not even accepted upstream yet but needed). I do see why you need to patch packages. I do not see why that needs to happen inside the build. Why not do it before building? > This procedure should be automated and expressed with very few > statements. The current approaches we have are far from optimal, but the > vision remains that you just tell Isar the package, the patch, and then > it does whatever is best from Debian perspective. That may include the > mentioned source package patching upfront. That will have to go into our > recipes and classes. That sounds a lot like your requirement is a consequence of your current implementation and not an actual requirement. I fully agree that such patching needs to be simple to perform and (unfortunately) is a frequent thing to do. Just the how we do it is sub-optimal both in Isar and rebootstrap. > Sure. We are not building hundreds of packages, but it also not only > about building one or two. The other reason is the mentioned identical > versions in all buildchroots. Reproducing them must provide identical > content (where common) for a single image build run. I think I already debunked the performance aspect. For reproducibility, you do want a clean build to reproduce the artifacts of a previous one. Reproducibility needs to work regardless of whether you cache the base image. Indeed, the reproducible folks are working on reproducible installations somewhat and the dpkg-root-demo explicitly verifies reproducibility (given the same inputs, but you do cache them anyway for offline builds). So again, the requirement of caching an image seems to be a consequence of your current implementation. > Currently, we have no secure container env. The container simply serves > as deployment tool of Isar's dependencies and builder-side > configurations so that you can run it on any distro, not only Debian. > And you can run it in CI identically to your host. If your docker container is not setup securely, you can make it support user namespaces and be done. :) > Isar users use Isar, they usually do not modify the core. And the goal > of Isar is to avoid that they have to but rather provide easy means to > configure things that must be configurable. If the builder would fall > under this, e.g. no one build would build all types of packages (hope > that is not the case, so far it wasn't), it would become an Isar > requirement to switching them without much effort. How would I disagree with that? Of course providing users with the flexibility to switch their builder turns out to become a requirement once users rely on it. However, the "without much effort" part is the thing that mdbp is supposed to provide. In theory, your builder becomes a shell command with a sane default and your user can change that one command to switch the builder, because the API differences are hidden behind a common mdbp API. > That is a valid point we need to keep an eye on. If mdbp solved that > already, it would be one of the added values I mentioned that could make > it relevant for Isar, even if not requiring to switch builders. Great. Given that builder-flexibility seems to be a non-requirement for you, I suggest that you use sbuild and look into the mdbp-sbuild implementation to figure what kind of settings you need to fix to make it reproducible. That's also a form of using it. ;) Helmut