From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6467463440282681344 X-Received: by 10.28.212.73 with SMTP id l70mr2773778wmg.14.1507192726446; Thu, 05 Oct 2017 01:38:46 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.160.247 with SMTP id n52ls5262146wrn.6.gmail; Thu, 05 Oct 2017 01:38:46 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDGNbFzySqC/vZj5oMAI5/bo4pZbCh1LlC5QjnQ3ZSqVnzmBDkwQy/fvTIkHqEqyrtdNKPh X-Received: by 10.223.186.204 with SMTP id w12mr1322163wrg.24.1507192726124; Thu, 05 Oct 2017 01:38:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507192726; cv=none; d=google.com; s=arc-20160816; b=RpL/tvSQ/ctn9nTYVYvRgMhItRj2ajPA7LUVOBSXBXa1QeAbq9tbzSi8faFagQP1Pl mNgttzSI4I4uOF+aw7bAwSlg22KB1Gex0Ywlku72h+2poZK5vh6Ngj31tz/EBwgoS+nz eP4ZIaVF6dDoxVS5ZeQ/YfBgIg5nl5vIR6oleg5UXALemyrX46C1XE8/dgJEYvfZeZsj tfFX3K67jon8tAuCF6CGjcQRe0FJKRiU1TeseIQL9m0v9rNOYQcpN/wAaXwp/yeswYYl YcDwmbJFVg7n2WgqcHglDyrpCrCv5k/Ak6XCKj6Q8s3WedyvwtuGDU8UuIqI8dIUH6b5 j9Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=X/r2bO+QqJHrht4VXzMzFIgL5Qb3NGyJVK0ai6qXzUw=; b=M7p3tiMlyd0g4aOZA1F7MzomUj/AKzrK1hscqIHbDm7Q1KIebzwjaVhdYs4cu0vxaZ rchEg6NaKRWdkw3Wuwv20TieW6b7WfjEMOoKbWsqk7hoKIGGMVS0eA6LhTNrz+T8VhHe ZHwRmdB3fX6Vf0cSvjN1XP5VBswyGmIQMjawGU1cMFQBXgcijiBoFh5/O4fHJzqTUk+9 cB59tHrVFXeJka2PdNPLLjrcIwTx3Y6dD5XZOOVZfk41xtAc4sjSRPAyUjtKDvpWRVt9 Y1n1QA/qXMKAoeL8S4qo7YLUHCVVmfUZTkHITDGMSr7720ovOsXhUpWKvJUL4QRM95az RdsA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id 200si76775wmj.0.2017.10.05.01.38.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 05 Oct 2017 01:38:46 -0700 (PDT) Received-SPF: neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id v958cj3P026071 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 5 Oct 2017 10:38:45 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id v958ciCb007731; Thu, 5 Oct 2017 10:38:44 +0200 Subject: Re: [PATCH v4 0/4] Basic binary cache implementation To: Alexander Smirnov , Claudius Heine , Jan Kiszka , isar-users@googlegroups.com References: <20171002154531.4930-1-asmirnov@ilbers.de> <2e00bbc9-9823-850e-9451-9d66350713b5@ilbers.de> <1507138484.6743.54.camel@denx.de> <47ed69c7-a279-6f9d-511a-2a60d816ac0b@ilbers.de> From: Claudius Heine Message-ID: <09d3a91b-f1f8-9570-fc92-216673d05d96@siemens.com> Date: Thu, 5 Oct 2017 10:38:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <47ed69c7-a279-6f9d-511a-2a60d816ac0b@ilbers.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: yLhGYVRfIUTr Hi Alex, On 10/04/2017 08:47 PM, Alexander Smirnov wrote: > On 10/04/2017 08:34 PM, Claudius Heine wrote: >> Hi Alex, >> >> On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote: >>> On 10/04/2017 02:57 PM, Jan Kiszka wrote: >>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote: >>>>> Hi, >>>>> >>>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote: >>>>>> Hello everybody, >>>>>> >>>>>> this series introduces basic binary caching for Isar. >>>>>> >>>>>> There is a new layer: meta-isar-bin which is intended to be the >>>>>> binary >>>>>> cache. All the packages that are built in Isar are stored in >>>>>> this >>>>>> cache using reprepro. >>>>>> >>>>>> Having the separate layer makes possible to manage this cache >>>>>> separately >>>>>> from the Isar build tree. So once you have built your packages, >>>>>> you could >>>>>> re-use this cache for further builds. >>>>> >>>>> We still have to talk about your requirement for the cache to be >>>>> sharable and your resulting implementation of it as a meta layer. >>>>> >>>>> Since I still don't see the benefit of a sharable cache and think >>>>> that >>>>> putting binary packages into some kind of strange meta layer is >>>>> not the >>>>> right solution and could potentially create more problems that is >>>>> solves >>>>> because its very much in conflict with the fundamental idea of a >>>>> meta >>>>> layer. >>> >>> The last statement is a bit strange for me. The idea of meta layers >>> is >>> to split software stack into logical layers that could be >>> enabled/disabled on demand. Having binaries or recipes doesn't >>> matter >>> here, it's just the way how software is provided to build. >>> >>>> >>>> Indeed, state like this (disclaimer: i didn't look into all >>>> details), it >>>> sounds like a weird architecture to me as well. Thinking of an >>>> artifact >>>> cache, I would rather expect something like OE's sstate that one >>>> may >>>> preserve across builds, share between devs, or whatever. If it's >>>> there, >>>> the build system consults it, if not, it rebuilds. But sstate is >>>> not a >>>> layer, for some good reasons. >>>> >>> >>> Let me again summarize major points here. Isar is the build system >>> designed to work with binary packages. That's the key feature of >>> this >>> product and main difference from OE-like systems. The Isar >>> architecture >>> assumes to be designed around binary packages, what provides >>> absolutely >>> different approach how binary images could be generated using >>> bitbake. >>> >>> According to the Claudius's vision, that everything should always be >>> built from sources - Isar is not the best option here, Yocto and OE >>> are >>> much better, because they were initially designed for this work. >>> Attempts to apply OE design and philosophy to Isar could have >>> negative >>> impact and limit possible ways to implement custom features, because >>> OE >>> wasn't designed for this features. >>> Also I wonder why we are able to use binary Debian packages, while >>> the >>> rest of software should be built from sources. >> >> You are still not understanding me. I never said you have to build >> every package from source code in isar. Instead I mean that you have to >> build the root filesystem / images / packages ... ( every output of >> isar) from all the available input of isar (upstream debian >> repositories, meta-layers, source code repositories, etc.). > > Yes, and I want to add my custom apt repo to the Isar, even if I have > source code for packages in it. > >> >> If you create intermediate products in the form of caches, then that is >> ok. But if you put those intermediate products into repositories and >> begin to distribute them, then this causes problems as I described >> before. >> > > If you mean version-control repository, then nobody asks you to do this. > If you mean apt repository, Isar should be able to export it with your > product (application layer), like I've mentioned with examples in my mail. > >> Distributing intermediate products is always a bad idea and systems >> should not be designed with this in mind or even advertise it. That has >> nothing to do with 'OE' or 'Debian' or any other System, its just >> general software developing and maintenance rules. > > That's contradictory sentences for me. Debian distributes both: > >  - Complete iso, USB etc... images >  - Apt repository > > And both are intermediate in terms of your custom product. But thats a different feature. Yes eventually isar should export a tarball of the repository to the deploy dir or similar. But that is not what this patchset is about. > > So if you want to develop package for Debian, you do not need to build > whole Debian from the scratch, you likely use intermediate product > instead - install base system and tools from binaries. > > Isar is based on intermediate product, i.e. Debian apt repository. The Debian repository is not a intermediate product for isar! But the resulting apt repository that is created by isar is a intermediate or final product of isar because it is generated by isar, therefor it should not be used as the input of a build process! But I am ok with it being used as a local cache! The difference is that the cache is not normally distributed to the user while the input is normally distributed to the user. Cache is internal to the build process while output and input is external. > > So I still don't understand, why you consider that 'apt-get install' > from Debian repository is OK, but 'apt-get install' from custom > repository is a bad idea? This discussion is frankly moving from frustrating to slightly infuriating. Implementing the possibility to use additional apt repositories that are added to the multistrap.conf and are defined in a bitbake variable is also a totally fine feature. But that is also not what this patchset is about. I don't appreciate you putting words in my mouth and using straw men arguments and moving the goalpost is not how we can discuss these things constructively. > >> >> It is ok to to this occasionally on a case by case basis, but it should >> not be a requirement or even an advertised feature. >> >>> ... >> >> In the rest of your email or this thread I still don't see any reason >> why sharing caches is necessary. >> >> Building the output from the input is fine once to fill the caches. >> Just to be sure that you are in fact able to build the output from the >> input. And I don't get why this is bad in your opinion. >> > > Because I don't want to waste the time for the work that has been > already done and tested. > > Just for sure, the release builds should be run from the scratch. And > integration testing should be performed. > > Building everything from the scratch only guarantees that your package > could be built, nothing more. But I'd prefer to delegate this task to CI. > >> I get that we can trust that Debian can build their packages from their >> sources, so we don't need to test that, but at least test if we can >> still build our own packages in isar. > > Sorry, didn't get this. Yes I see, and I don't know how I can explain it any simpler. Here is the main bad idea of this patch: - Using a meta-layer as a place to dump isar generated output at. meta-layers should only contain input data. All my other arguments, that you don't seem to understand, are trying to explain why this is a bad idea. Also I would really like to have a package cache, the ability to export the generated repository and the ability to used additional repositories. But the way this is integrated mixes input and output of the build system and makes many things unclear. So those 3 features should be handled separately. Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de