From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.223.130.240 with SMTP id 103mr330873wrc.13.1516712546980; Tue, 23 Jan 2018 05:02:26 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.116.12 with SMTP id p12ls2123060wmc.2.gmail; Tue, 23 Jan 2018 05:02:26 -0800 (PST) X-Google-Smtp-Source: AH8x2260fq8XceSLx8maOIZFRkgJbIWMMVVU2++ZRKyF60UXbdWuWBhpqLkO+zNfw1kxhW1hA8Te X-Received: by 10.28.106.25 with SMTP id f25mr263241wmc.16.1516712546375; Tue, 23 Jan 2018 05:02:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516712546; cv=none; d=google.com; s=arc-20160816; b=NttTUqJX2yoyHPCJ9uCLYiJ4ERxuvc/wKJ/beHiVdxN/j+mhue99lUjC7/vRRaAk0s BmEDa4VZP0M9j9CT73Oj5c4fZww9d6G5zUeua8KtrTTTQtD0VUqinIXPMha2KghDXoRo zBPXD0c7sZwlVCLVR8FvB9jKoem5+EoC0MfKqYgHLdMfRKtc63HV+QB77rT7jXnrBIi7 jUX+kGYSoax0EUePkyelT1W15/FclDJy1Cmv0jnUlfcZ81b1X4WjETOCY7mxW8A/oKUg wYKl9YG+YF1jAJ/C0RS1F2UL7JP4+ltpE82+6WUa4ltlHCEHG5l7M6GwpFKxESOM0Cw3 aDqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=HTzAvQ1VviO9PyuS1zVrlss01tTbzZFlXzmwAFerQk4=; b=XroychOotAagbsOl172l8Oh8jrsW8cTJJXy3ZyE389pHjMokece5NAgBQVFqd4sd7Z PeNNKx/rJHgvDTSOFHjwNmKzhco3JHFvHOwEuw7jf6sE20twKmLBg5GyWG9+hMBtXe5b 1udnhc+cOCEr17veMZyEpvoQYqxcAEgvuPXm+HdKMdBoodJR3zSyfwQItgH5jtAOUexw w/61TTqLUkI4+kZbntlK/ATHCbia9KG9Jdtmzx1gZGi7nJbt153SO0E8JT8QXLxnI87q ULvQ9WEJEllCIW/YFEERwJKqHdqzzlkyyxvwqaCEcUssnQdyZh3rMDNyNeZbSd53KyfB 2grQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id 44si52026wrk.0.2018.01.23.05.02.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jan 2018 05:02:26 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w0ND2Q15005933 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 23 Jan 2018 14:02:26 +0100 Received: from md1f2u6c.ww002.siemens.net (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id w0ND2P4c004416 for ; Tue, 23 Jan 2018 14:02:25 +0100 Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class To: isar-users@googlegroups.com References: <20180111111939.25667-1-asmirnov@ilbers.de> <20180111111939.25667-3-asmirnov@ilbers.de> <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> <20180112172526.19bd23a2@mmd1pvb1c.ad001.siemens.net> <20180123115026.GA4542@yssyq.radix50.net> From: Jan Kiszka Message-ID: Date: Tue, 23 Jan 2018 14:02:24 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20180123115026.GA4542@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: GWQd+DSyNaL9 On 2018-01-23 12:50, Baurzhan Ismagulov wrote: > On Sun, Jan 14, 2018 at 05:53:03PM +0100, Jan Kiszka wrote: >> I agree with Henning's view. We should exploit Debian for building the >> repo and only use bitbake recipes to control and direct Debian tools, >> definitely not reimplement them. >> >> Having one build run that both generates outputs and fills that repos >> which would then be used on succeeding runs would be ok from user >> perspective. We do not necessarily need that strict ordering of >> fetch-before-use in the first build. > > It's a trade-off between perfectly solving the by-product way vs. solving the > up-front way with TODOs for the next steps. > > > My original vision has been to have a separate partial mirroring tool that > understands Isar metadata (recipes and / or Debian packages). > > 1. Conceptually, I dislike the by-product solution because in the projects, > repo maintenance is often done by configuration managers. They don't build > the code, they just make the mirror available (with Isar's predecessor, > those were complete mirrors made with debmirror). Project developers switch > from one snapshot to another and adjust the code if necessary. Requiring CMs > to build the code is not optimal and, in my opinion, should not be done. We > should provide tools and people should decide whether to combine them or > not. Configuation managers are not our primary concern. Addressing all their "needs" (or lack of technical experience) will restrict us too much. Let's do something for today's devops first and address that special case on top - locally. > > Providing separate mirroring is also the right step towards offline work and > SDK as supported by Yocto. I don't see how having online connection for the first run and then being able to rebuild offline afterwards is a problem, or something that restricts SDK builds. > > 2. Mirrors are maintained per image, because two products in the same repo may > have different release timelines. That is why we need to have an explicit > list of images to produce the mirror for. This could be addressed with both > bitbake-based approaches, by-product and up-front. Again: user experience. The common case is that you do not mix reproducible and non-reproducible images in a single build run. So you also want to control that with a single, general knob, and not by duplicating the list that you already pass to bitbake. > > 3. Projects update from time to time, not on every build. This could be done > with both approaches through turning off mirroring in local.conf. With a > separate tool, it would be more convenient to use. Only if that tool is as robust as the build itself. I and also Henning have major concerns that this will not be the case, thus our requirement to avoid that logic duplication and fragility. > > 4. Finally, there is a question of partial updates, e.g., update only glib and, > optionally, its dependencies, dependents, and conflicting packages. This is > not addressed by either of bitbake-based solutions. That could probably be solved with an update "mask", a list of packages passed to a build that should be pulled from extern vs. from the internal mirror. > > > To create such a tool, we would have to: > > 1. Understand Isar metadata. > > 2. Implement mirroring. > > To avoid messing with (1) at this time, we used bitbake for now. The goal is to > implement mirroring quickly and get experience with it, because important > details may require adjusting the vision. > > > There are also technical issues with the by-product solution: > > 1. Complicated and error-prone workflow > 2. Unnecessary build repeat burden for the project developer There is not need to repeat the build to get a mirror. > 3. Intertwined, inseparable mirror and build > 4. Unpredictable builds, esp. if sid is used (usual for early development) > > > The following technical issues have been raised regarding the two up-front > implementations: > > 1. DEBIAN_DEPENDS mixes Build-Depends and Depends. > > I agree that this is not optimal. This could be solved by separating > DEBIAN_DEPENDS and DEBIAN_BUILD_DEPENDS. This itself would be a temporary > solution till we implement debian/control backend for bitbake (as an > alternative to .bb backend). > > 2. We first need a working solution of two "inherit dpkg" packages actually > depending on each-other. > > An intermediate solution is available in Isar's predecessor. The approach is > to provide DEPENDS in recipes to ensure the right do_build order, and to > install the development packages into buildchroot. Of course, in the future > that should be extended with adding the built packages to apt and moving to > debian/control backend. > > That said, it's a matter of priority -- reproducibility and wic were at the > top of Jan's wishlist. FWIW, Alex already has an implementation for own > package dependencies and will post it. wic was addressed by Henning, you can focus on reviews. > > 3. Reimplementing Debian tools and constant catching up with upstream. > > Some conceptual points first. As already discussed in the context of wic > last year, we strive for solving project problems today and merging with > upstream after we get experience with the feature, as opposed to developing > huge series solving all problems of the world and pushing them upstream > first. This delayed feedback loop works well e.g. for C and C++ > standardization. You see, now you promise a better implementation with less > or no changes to wic; it may be well worth waiting with upstreaming. This is > not to say we want to leave the problems in the code; we just want to choose > what problem we tackle at which point of time. > > Another point is, we don't actually reimplement Debian tools. If a tool that > we need existed, we would have used it, no? But they don't. So, we can > extend an upstream one, or we can implement our own. For a PoC, we choose > whatever is faster. Merging with upstream is a TODO that we deliberately add > to our list. I agree that in this way, we've accumulated quite a number of > hacks; we are going to resolve them. > > That said, we shouldn't shy away from creating and maintaining new tools if > it's a better way of solving the problem. Debian describes the format of the > files, and many tools provide similar functionality. Many Linux features > have been maintained as patches for years before being accepted to the > mainline. > > Returning to the issue at hand: I agree that parsing debian/control manually > is not optimal. Isar already does that in other places; upstreaming it to > some tool is a TODO that should not block moving forward on reproducibility. > We expect changes in this area, so even the choice of the tool to merge > depends on the workflow we'd end up with. > > > Up-front mirroring was Christian's requirement that noone objected to at the > time [1]. Considering all of this: > > 1. I suggest to clean up the first series with the list of images due to the > use case #2, and merge it. > > 2. I'd rather not go for the second series, to avoid opening a new can of worms > w.r.t. DEBIAN_DEPENDS, unless there are serious technical reasons in its > favor. > > 3. We have always been aware of the issues and are working on them. > This is a strategical project decision because it affect the user experience and the way recipes are written. Despite the project needs, we must not hurry quick intermediate solutions (unless the are truly orthogonal to the recipes). Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux