From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.28.154.197 with SMTP id c188mr321706wme.16.1516725460482; Tue, 23 Jan 2018 08:37:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.26.132 with SMTP id a126ls2242588wma.1.canary-gmail; Tue, 23 Jan 2018 08:37:39 -0800 (PST) X-Google-Smtp-Source: AH8x22438jlwikm/AoKiHt96lPbM9wXKU9AlOpCwo0FwwOQiB8wOUk2JrOP5+zQ7IpBBqCGwvFDA X-Received: by 10.28.154.197 with SMTP id c188mr321702wme.16.1516725459835; Tue, 23 Jan 2018 08:37:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516725459; cv=none; d=google.com; s=arc-20160816; b=OWKwUdUDK7OUZ99zdPa/3JZX0/DhA87ytD8omRXJ+/vTyXCZbpaXk7AVVHV1bOberj PoffjysSlKToyt+aOpz4ieF6eajH/DYMfY0GWLd2+KOIwu/psOE6XOmnEmG+YsJU3zVU yX8mnaYvnQTA8Fj6Qcj7to0nKmUa9vGk2SMC6/TkGspBfL+IpoHLyAiw83fI56q23IcU ROI04s+axgd/PDYMpR1QlGEy6XAQK3DrwAY9U+/j6OEiW5d3fG7Qg6g/ZjfVapGVcLOd 7HQkMYpGzizzXJE+SjCjZiLAkKVFgSok2MvT8PW+VSsz0Pw3PA6QpDItRjBgaNqEFaui o6lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-transfer-encoding :content-disposition:mime-version:mail-followup-to:message-id :subject:to:from:date:arc-authentication-results; bh=BVtXpWj8N3cxIM5hQV6YR01FzmXLFhObk+9k1zeNxC0=; b=qoj/w4yCsuduL4eHxxrcJFzufiohZbo5096zczfYDQUfP8cFfeLwpfY2fxcnGtk6pr +lwlILXyrxzINcZUia+ZI4gWboN7UwNkNxSkBEsb3pigaC165/NU9lTe2oCoWb+ae1pD f2Oycczt4VmGGGpYnrsiHTTZj5oHslIFWB9vBM3MfMID85rKJ35fz+7FmmHxyf4E6rF7 o/i0eU2GeGpMappiGOJ/DR47/V4IsmqJ08K+gl8JZ/QIktrWeKyGOvBN4wL74W1gw7B+ Qx88FwIa3bZkvOjmNHIhFkGtPZzstIgJveayinem9snP+6wSTAFjJD5WQ+vYwcJjN2B2 P3Cg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id d8si1157130wmf.1.2018.01.23.08.37.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jan 2018 08:37:39 -0800 (PST) Received-SPF: pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of christian.storm@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=christian.storm@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w0NGbd3V002078 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 23 Jan 2018 17:37:39 +0100 Received: from localhost ([139.25.69.251]) by mail2.siemens.de (8.15.2/8.15.2) with ESMTPS id w0NGbdFf030450 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 23 Jan 2018 17:37:39 +0100 Date: Tue, 23 Jan 2018 17:34:48 +0100 From: Christian Storm To: isar-users@googlegroups.com Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class Message-ID: <20180123163448.chwg33a6tcjwrx5r@MD1KR9XC.ww002.siemens.net> Mail-Followup-To: isar-users@googlegroups.com MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180123115026.GA4542@yssyq.radix50.net> User-Agent: Mutt/20170113 (1.7.2) X-TUID: f6D5HgBG1e78 Hi all, > [...] > My original vision has been to have a separate partial mirroring tool that > understands Isar metadata (recipes and / or Debian packages). > > 1. Conceptually, I dislike the by-product solution because in the projects, > repo maintenance is often done by configuration managers. They don't build > the code, they just make the mirror available (with Isar's predecessor, > those were complete mirrors made with debmirror). Project developers switch > from one snapshot to another and adjust the code if necessary. Requiring CMs > to build the code is not optimal and, in my opinion, should not be done. We > should provide tools and people should decide whether to combine them or > not. > > Providing separate mirroring is also the right step towards offline work and > SDK as supported by Yocto. > > 2. Mirrors are maintained per image, because two products in the same repo may > have different release timelines. That is why we need to have an explicit > list of images to produce the mirror for. This could be addressed with both > bitbake-based approaches, by-product and up-front. > > 3. Projects update from time to time, not on every build. This could be done > with both approaches through turning off mirroring in local.conf. With a > separate tool, it would be more convenient to use. > > 4. Finally, there is a question of partial updates, e.g., update only glib and, > optionally, its dependencies, dependents, and conflicting packages. This is > not addressed by either of bitbake-based solutions. > > > To create such a tool, we would have to: > > 1. Understand Isar metadata. > > 2. Implement mirroring. > > To avoid messing with (1) at this time, we used bitbake for now. The goal is to > implement mirroring quickly and get experience with it, because important > details may require adjusting the vision. Then this qualifies for a proof-of-concept on which we may discuss, technically, I would say. Whether this "mirroring tool" belongs into ISAR's core or should be an external tool that's suited towards ISAR is a strategic architectural decision. I'm in favor of the former. > There are also technical issues with the by-product solution: > > 1. Complicated and error-prone workflow > 2. Unnecessary build repeat burden for the project developer > 3. Intertwined, inseparable mirror and build > 4. Unpredictable builds, esp. if sid is used (usual for early development) > > > The following technical issues have been raised regarding the two up-front > implementations: > > 1. DEBIAN_DEPENDS mixes Build-Depends and Depends. > > I agree that this is not optimal. This could be solved by separating > DEBIAN_DEPENDS and DEBIAN_BUILD_DEPENDS. This itself would be a temporary > solution till we implement debian/control backend for bitbake (as an > alternative to .bb backend). Well, that would deviate from bitbake syntax and "look-and-feel" more than it is already... > 2. We first need a working solution of two "inherit dpkg" packages actually > depending on each-other. > > An intermediate solution is available in Isar's predecessor. The approach is > to provide DEPENDS in recipes to ensure the right do_build order, and to > install the development packages into buildchroot. Of course, in the future > that should be extended with adding the built packages to apt and moving to > debian/control backend. > > That said, it's a matter of priority -- reproducibility and wic were at the > top of Jan's wishlist. FWIW, Alex already has an implementation for own > package dependencies and will post it. > > 3. Reimplementing Debian tools and constant catching up with upstream. > > Some conceptual points first. As already discussed in the context of wic > last year, we strive for solving project problems today and merging with > upstream after we get experience with the feature, as opposed to developing > huge series solving all problems of the world and pushing them upstream > first. This delayed feedback loop works well e.g. for C and C++ > standardization. You see, now you promise a better implementation with less > or no changes to wic; it may be well worth waiting with upstreaming. This is > not to say we want to leave the problems in the code; we just want to choose > what problem we tackle at which point of time. > > Another point is, we don't actually reimplement Debian tools. If a tool that > we need existed, we would have used it, no? But they don't. So, we can > extend an upstream one, or we can implement our own. For a PoC, we choose > whatever is faster. Merging with upstream is a TODO that we deliberately add > to our list. I agree that in this way, we've accumulated quite a number of > hacks; we are going to resolve them. > > That said, we shouldn't shy away from creating and maintaining new tools if > it's a better way of solving the problem. Debian describes the format of the > files, and many tools provide similar functionality. Many Linux features > have been maintained as patches for years before being accepted to the > mainline. > > Returning to the issue at hand: I agree that parsing debian/control manually > is not optimal. Isar already does that in other places; upstreaming it to > some tool is a TODO that should not block moving forward on reproducibility. > We expect changes in this area, so even the choice of the tool to merge > depends on the workflow we'd end up with. > > > Up-front mirroring was Christian's requirement that noone objected to at the > time [1]. Considering all of this: > [...] Well, recalling what I wrote back in November, I don't read a requirement, literally, on upfront mirroring. I explicitly mentioned this to be my brain dump of ideas and wrap up of the discussion so far in order to revive the discussion on the topic and to foster the implementation of this feature which I still very much like to see become upstream -master reality. I did explicitly *not* request a particular implementation -- as I tried to make this clear by using "Regardless/irrespective of the technical implementation" multiple times -- in order to find the best solution, which is contradictory to requesting some concrete implementation of the feature. Maybe this was not explicit enough though... That said, upfront mirroring may be *one* solution -- and maybe the most straightforward one -- to the Debian-inherent window of vulnerability problem I described. There are certainly others... Finding and elaborating on those was my intention, not dictating a solution. Kind regards, Christian -- Dr. Christian Storm Siemens AG, Corporate Technology, CT RDA IOT SES-DE Otto-Hahn-Ring 6, 81739 M�nchen, Germany