From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7192512605249863680 X-Received: by 2002:a1c:4b0d:0:b0:3d9:f7c0:91ef with SMTP id y13-20020a1c4b0d000000b003d9f7c091efmr2920555wma.47.1674722889132; Thu, 26 Jan 2023 00:48:09 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:1ca4:b0:3cf:afd2:ab84 with SMTP id k36-20020a05600c1ca400b003cfafd2ab84ls658552wms.2.-pod-control-gmail; Thu, 26 Jan 2023 00:48:08 -0800 (PST) X-Google-Smtp-Source: AMrXdXt3MpP7Iyo0GL6SNOzruiSasJsxDmP+eUj1ohT6NDHANmA+RLJ7qdc4pdaYJ0GzGcYRzFA5 X-Received: by 2002:a05:600c:2d0b:b0:3da:fcf0:a31d with SMTP id x11-20020a05600c2d0b00b003dafcf0a31dmr34960608wmf.22.1674722887891; Thu, 26 Jan 2023 00:48:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674722887; cv=none; d=google.com; s=arc-20160816; b=wZlPvH1CaMQefA3s/kxXs+NRsnfDNo8RR2tOVeAlI9mk8RUeRQEJ8R19PcM+d6mLUs JAohn2kO1NLBndkRG7iCqRpr02GXgxSL1BfgkLF97ztNUCy9ZjUazcKw7SJ733NaU0Rm C1nOP9pO8V7jFqcsikSmzeBNbMUWVMatwdlj7rOvAX3XXIxL9sogw0bZSBCN0kh+PwnS kVMGI8CbMSLLkou08BYjoXLAxA7IquKekdpNrtWPhWUZw44ODYgLtXSzhW/rBoasspP5 9pa64cNswJGGD13kQ3fofaxvPtVoPyTNOrNcZlUVfQ4oDudY1ZVAScvqv6UbMQssNUq5 4Pnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:mime-version:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:dkim-signature; bh=Xx3401NBflKkMvtuRGl1up1Vp1XxdRTvTsQw2VCVCrQ=; b=ptq0RRSCXsO49pWqh/oeBxO1ScB5TkM/4hybA4F9RKIfZqy8cKCOXmS467/xyBVzXe Uu0AeHzchsf3T5l1krM89NMfk3VD9uXWgkxfjOyX+QDCd/c7NMOd6VJy1VXpEQhLEGYq 7Jl2ZrS0kYNVmLxuRslZut3ofZ2ukpGIM8/TilzlUizOnxS/eBo43utEdLIeYoo6JN0W 7MfZUzCcTipXNjOqMDDVNdO5CZs8nIJWL59CUORzXucu2xenteYl9HzIVVdrmglD7KpI hKmkjBySvXchIff9Q7lgFOKWCXuNgwr7XEKh5mQeac7BWcPsY1q6Uqe3jsxYNWDxLqZk MWCQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="aY/eXaJo"; spf=pass (google.com: domain of fm-68982-20230126084807c8fc97e088755ad9f5-cvgfr3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-68982-20230126084807c8fc97e088755ad9f5-cvGFR3@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from mta-64-228.siemens.flowmailer.net (mta-64-228.siemens.flowmailer.net. [185.136.64.228]) by gmr-mx.google.com with ESMTPS id bd24-20020a05600c1f1800b003d9c716fa3csi455971wmb.1.2023.01.26.00.48.07 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jan 2023 00:48:07 -0800 (PST) Received-SPF: pass (google.com: domain of fm-68982-20230126084807c8fc97e088755ad9f5-cvgfr3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) client-ip=185.136.64.228; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b="aY/eXaJo"; spf=pass (google.com: domain of fm-68982-20230126084807c8fc97e088755ad9f5-cvgfr3@rts-flowmailer.siemens.com designates 185.136.64.228 as permitted sender) smtp.mailfrom=fm-68982-20230126084807c8fc97e088755ad9f5-cvGFR3@rts-flowmailer.siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: by mta-64-228.siemens.flowmailer.net with ESMTPSA id 20230126084807c8fc97e088755ad9f5 for ; Thu, 26 Jan 2023 09:48:07 +0100 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=florian.bezdeka@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc:References:In-Reply-To; bh=Xx3401NBflKkMvtuRGl1up1Vp1XxdRTvTsQw2VCVCrQ=; b=aY/eXaJonL7VcgBXt/1vVFNGmmzMDGK2ujEgck7ZYDMyffHpA/knTy7KbgBWxAejtpwlYv jkLFuuMO6eZF/iK9nO6VldTmifjja608keXiooDGiZIePMuMKC0u4ia0vca4jvu8DsR5LCWd ZluaQr80DpbWpKbAxWGS2gHutTR/Y=; Message-ID: Subject: Re: [PATCH 0/5] allow creation of users/groups before rootfs creation From: Florian Bezdeka To: "Schaffner, Tobias" , "Schild, Henning" Cc: "Gylstorff, Quirin" , "isar-users@googlegroups.com" , "Adler, Michael" Date: Thu, 26 Jan 2023 09:48:06 +0100 In-Reply-To: <1b8a1db2-0390-8027-4a45-471f2385a50e@siemens.com> References: <20230125090156.284309-1-tobias.schaffner@siemens.com> <20230125142901.597613d7@md1za8fc.ad001.siemens.net> <20230125172916.0d49528f@md1za8fc.ad001.siemens.net> <73457b6d-c5cd-7e43-5142-2feab5caa54a@siemens.com> <20230125223848.6b911eb5@md1za8fc.ad001.siemens.net> <1b8a1db2-0390-8027-4a45-471f2385a50e@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-68982:519-21489:flowmailer X-TUID: tSXd/LNS/om6 On Thu, 2023-01-26 at 08:21 +0000, Schaffner, Tobias wrote: > On 25.01.23 22:38, Schild, Henning (T CED SES-DE) wrote: > > Am Wed, 25 Jan 2023 21:55:00 +0100 > > schrieb "Schaffner, Tobias (T CED SES-DE)" > > : > >=20 > > > On 25.01.23 17:29, Schild, Henning (T CED SES-DE) wrote: > > > > Am Wed, 25 Jan 2023 14:44:40 +0100 > > > > schrieb Gylstorff Quirin : > > > > =20 > > > > > On 1/25/23 14:29, Henning Schild wrote: > > > > > > Am Wed, 25 Jan 2023 10:01:51 +0100 > > > > > > schrieb "T. Schaffner" : > > > > > > =20 > > > > > > > From: Tobias Schaffner > > > > > > >=20 > > > > > > > This patch series will allow to specify a `pre` flag for the > > > > > > > USER_ and GROUP_ bitbake variables. If this flag is set to > > > > > > > `true` the given user or group will be created in the rootfs > > > > > > > configuration step instead of on rootfs postprocessing. This = is > > > > > > > helpful when a specific id should be used which would otherwi= se > > > > > > > be picked by a user or group created by one of the installed > > > > > > > packages. > > > > > >=20 > > > > > > While i do understand the reason i am not sure how relevant tha= t > > > > > > is. Why would anything only function with a fixed ID? Whoever > > > > > > provided that thing should maybe fix it. > > > > >=20 > > > > > Specific IDs are necessary for Updating an A/B rootfs system with= a > > > > > persistent partition. In this case you do not want to change any > > > > > IDs as this can lead to wrong file permissions. > > > >=20 > > > > I see, that is much more understandable. It also goes into the > > > > reproducible build direction. > > > > But if that is the case the solution does not seem to be good enoug= h > > > > because it only considers users/groups created by isar. Not the one= s > > > > created by installing debian packages. Where the order could be > > > > potentially random. Say you DEBIAN_DEPENDS or IMAGE_PREINSTALL "ftp= d > > > > wwwd" which will craete users "ftp www" where the two deamons do no= t > > > > have any dep on each other and apt-get could install them in any > > > > order. That order might in reality not change too often but it > > > > could i.e. when you move from debian11 to debian12 or when you > > > > bring the third (or 10th) user-adding package into your new > > > > firmware. > > >=20 > > > Thanks for the review! > > >=20 > > > This series is not about reproducible builds. The IDs of the users an= d > > > groups that are created by debian packages are expected to change > > > between _image_ versions. So we expect these IDs to be different afte= r > > > a swupdate. > >=20 > > But your initial argument was that you have to create a user before any > > debian package comes, or did i misunderstand? Maybe we need to > > differentiate between upstream packages and isar-created ones? >=20 > Lets make this clearer with an example: > A downstream layer creates one user X that writes to a separate partition= . > This user gets the ID 1000 as there are no packages that create any users= . >=20 > Now they want to create a image update that should be deployed. > The updated image includes a new debian package that creates a user Y. > The user Y of this package gets the ID 1000 as it is now the first user t= hat > is created. >=20 > User X will now get 1001 and it is not possible to change this. The data > created on the separate partition belongs to user Y. >=20 > The only resort at the moment is to revert the patch that moved the user > creation to the post processing. Hm... I have the strong feeling that your approach is broken in the same way... I was questioning myself how Debian itself solves this problem. The answer is: Debian has access to the "old" or "current" user database while doing updates. So it can not happen that a user gets a different UID. We would have to feed in the "current" user database during such updates to do it the same way. Why your approach is broken: Consider three image versions A, B and C. There is no guarantee that someone will update from A to B and then to C. A -> C is also possible. Image A can add a user using your "pre" logic, B can remove that user, C can introduce a new different user, but with same ID. That brings you into the same situation. As Isar can't tell if that is a dangerous scenario for all the possible use cases it's a NACK security wise. >=20 > > > For reproducible builds it is important that the ordering of installe= d > > > packages and their dependencies stay the same but that is a different > > > story. I expect the ordering algorithm for a specific set of packages > > > including dependencies to be deterministic but would have to look int= o > > > that in detail. > >=20 > > I expect the algorithm to be non-deterministic ... potentially as you > > progress using debian. I can just claim, you have to prove. >=20 > From my point of view this problem derives from the fact that the set of > installed packages will change on image updates. If the user Y is created > in the first or in the last package does not matter. Its just the fact > that an additional user gets created. >=20 > > > Still, if somebody had a requirement that a user or group of a packag= e > > > (e.g. ftp or www) stays the same, one could use this change to create > > > the user beforehand and pin its id using this change. > >=20 > > True. But we might still want to keep the old database to run > > assertions so such things do not go unnoticed. > >=20 > > > > So what you maybe really want is giving isar an /etc/passwd > > > > /etc/group pair. Every new firmware is build with the given layer > > > > code and that file-pair from the first release. Where you inject > > > > those files between bootstrap and install ... hoping that bootstrap > > > > will always be the same. Maybe one can inject before bootstrap ... > > > > or write a postproc function that will find all different ids and > > > > all files and fix up. Or at least start with an assertion in > > > > postproc that looks at the old database. > > >=20 > > > Care that just adding /etc/passwd and /etc/group might not be enough > > > but you would have to handle the side effects of a useradd/groupadd > > > call properly (E.g. creating the home if set). And I expect more > > > things to come up if you have a detailed look. > >=20 > > Yes. Not even want to think about LDAP or yp where the databases live > > in different files. > >=20 > > > Take into account that the specific patch that introduces the pre fla= g > > > is small, easy to maintain and configure. > >=20 > > It should still be motivated so that we later understand why we have > > it. And i would not call it easy to configure if one does not > > understand why ... people will "pre" randomly because they do not know > > what they are doing. We moved that stuff from pre to post once, and i > > can not help the feeling that people might want to want to mix pre and > > post one day ... at which point it might become really un-easy. > >=20 > > > > Is the problem of uid/gid depend on install order known in the > > > > debian community and how do others solve it? Gentoo has moved from > > > > such dynamic allocation to static some years ago, probably for > > > > similar reasons. > > >=20 > > > I am open to discuss this with the reproducible build guys but again > > > IMHO this discussion belongs into another thread. > >=20 > > IMHO not, the repro guys want full repro ... which we can happily leave > > out when it comes to me. You want partial repro, which we should > > discuss here. >=20 > I think you may have a valid point that a non deterministic ordering on > package install may be a problem for reproducable builds. Not only when > it comes to users but also e.g. every time two packages append to the sam= e > file. >=20 > > I do not want to hold this back, just understand it and see if there > > can be a better solution. Feel free to ignore me and merge if nobody > > else asks questions. >=20 > I am happy to discuss this, I just think we are mixing things up. But > maybe I am missing something. >=20 > Best! >=20 > > Henning > >=20 > > > > Henning > > > > =20 > > > > > >=20 > > > > > > So i am willing to say that this is super-niche! And it deserve= s a > > > > > > niche-solution in its layer, not a feature in Isar. > > > > > >=20 > > > > > > You could hook in a task between bootstrap and image_install. O= r > > > > > > you could rebuild a bootstrap package to have reserved ids. You > > > > > > could run "the thing" in namespaces ... > > > > > >=20 > > > > > > So is that really relevant? Please go into detail. > > > > > >=20 > > > > > > Whatever happens i think the python rewrite is cool. But the co= de > > > > > > may have been coming/inspired from OE ... in which case it woul= d > > > > > > not be cool, because it would fork away further. > > > > >=20 > > > > >=20 > > > > > OE uses a complete different implementation than to original: > > > > > https://git.openembedded.org/openembedded-core/tree/meta/classes/= useradd.bbclass > > > > >=20 > > > > > see also: > > > > >=20 > > > > > https://git.openembedded.org/openembedded-core/tree/meta-skeleton= /recipes-skeleton/useradd/useradd-example.bb > > > > >=20 > > > > >=20 > > > > >=20 > > > > > Quirin > > > > > >=20 > > > > > > Henning > > > > > > =20 > > > > > > > A rewrite of the image-account-extension in python was done o= n > > > > > > > the way. This allows us to drop a lot of encoding and parsing > > > > > > > code that was used to transition to shell and therefore made = it > > > > > > > easier to read and maintain. > > > > > > >=20 > > > > > > > Using python functions for more complex tasks allows us the u= sage > > > > > > > of unittests. A very basic infrastructure for unittesting usi= ng > > > > > > > the build in python unittest and the bb.parse module was adde= d. > > > > > > > This was used to test the re-implementation of the > > > > > > > image-account-extension as a first showcase. > > > > > > >=20 > > > > > > > Tobias Schaffner (5): > > > > > > > simplify image-account-extension > > > > > > > allow creation of users/groups before rootfs creation > > > > > > > create a minimal python unittest infrastructure > > > > > > > add unittests for the image-account-extension > > > > > > > set minimal python version in user_manual to 3.5 > > > > > > >=20 > > > > > > > doc/user_manual.md | 4 +- > > > > > > > meta/classes/image-account-extension.bbclass | 391 > > > > > > > +++++++----------- testsuite/unittests/README.md > > > > > > > > 28 ++ testsuite/unittests/bitbake.py | 37 += + > > > > > > > testsuite/unittests/rootfs.py | 45 ++ > > > > > > > .../unittests/test_image_account_extension.py | 175 +++++= +++ > > > > > > > 6 files changed, 434 insertions(+), 246 deletions(-) > > > > > > > create mode 100644 testsuite/unittests/README.md > > > > > > > create mode 100644 testsuite/unittests/bitbake.py > > > > > > > create mode 100644 testsuite/unittests/rootfs.py > > > > > > > create mode 100644 > > > > > > > testsuite/unittests/test_image_account_extension.py > > > > > > > =20 > > > > > > =20 > > > > >=20 > > > > >=20 > > > > >=20 > > > > > =20 > > > > =20 > >=20 >=20