Re: [PATCH v4 0/4] Basic binary cache implementation

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi,

On 10/04/2017 06:40 PM, Henning Schild wrote:
> Am Wed, 4 Oct 2017 17:29:08 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>>>>> Hello everybody,
>>>>>
>>>>> this series introduces basic binary caching for Isar.
>>>>>
>>>>> There is a new layer: meta-isar-bin which is intended to be the
>>>>> binary cache. All the packages that are built in Isar are stored
>>>>> in this cache using reprepro.
>>>>>
>>>>> Having the separate layer makes possible to manage this cache
>>>>> separately from the Isar build tree. So once you have built your
>>>>> packages, you could re-use this cache for further builds.
>>>>
>>>> We still have to talk about your requirement for the cache to be
>>>> sharable and your resulting implementation of it as a meta layer.
>>>>
>>>> Since I still don't see the benefit of a sharable cache and think
>>>> that putting binary packages into some kind of strange meta layer
>>>> is not the right solution and could potentially create more
>>>> problems that is solves because its very much in conflict with the
>>>> fundamental idea of a meta layer.
>>
>> The last statement is a bit strange for me. The idea of meta layers
>> is to split software stack into logical layers that could be
>> enabled/disabled on demand. Having binaries or recipes doesn't matter
>> here, it's just the way how software is provided to build.
> 
> Binaries that can be rebuild should not become part of the build
> system. The need to distribute them to the targets or other
> developers/images later is what we all agree on. But that should be
> mostly outside of isar.

Each binary could be re-built from sources (at least for Debian :-) ). 
Moreover Debian provides sources for them, so they are open for use.

But for me it looks like:

  - Installing binaries to target from 'ftp.debian.org' is allowed.
  - Installing binary dependencies for custom built package from 
'ftp.debian.org' is allowed.
  - Installing my binary application from 'my.debian.repo' is not 
allowed. I have to build it from sources.

What is the idea here? For me it's something like traditional OE 
philosophy, but without benefit for Isar.

One of the typical usecase: if I'm, for example, developing 'hello' 
application, why it's so critical for me to build, lets say, kernel from 
sources?

Alex

> 
> Building an Image will result in the Image and a repo. Both have to
> somehow make their way to the target, but that is somewhat out of scope
> of the build system.
> 
>>>
>>> Indeed, state like this (disclaimer: i didn't look into all
>>> details), it sounds like a weird architecture to me as well.
>>> Thinking of an artifact cache, I would rather expect something like
>>> OE's sstate that one may preserve across builds, share between
>>> devs, or whatever. If it's there, the build system consults it, if
>>> not, it rebuilds. But sstate is not a layer, for some good reasons.
>>>    
>>
>> Let me again summarize major points here. Isar is the build system
>> designed to work with binary packages. That's the key feature of this
>> product and main difference from OE-like systems. The Isar
>> architecture assumes to be designed around binary packages, what
>> provides absolutely different approach how binary images could be
>> generated using bitbake.
>>
>> According to the Claudius's vision, that everything should always be
>> built from sources - Isar is not the best option here, Yocto and OE
>> are much better, because they were initially designed for this work.
>> Attempts to apply OE design and philosophy to Isar could have
>> negative impact and limit possible ways to implement custom features,
>> because OE wasn't designed for this features.
>> Also I wonder why we are able to use binary Debian packages, while
>> the rest of software should be built from sources.
>>
>> This series introduces support for binary apt repositories in Isar.
>> This feature is response to the following cases:
>>
>> 1. Publish custom apt layer in web. When I need to install some
>> custom software to my Debian system, I frequently see instructions
>> like:
>>    - Add 'blablabla' to your /etc/apt/source.list
>>    - Run apt-get update
>>    - Run apt-get install
>>
>> That's for example how Jenkins and GitLab are distributed:
>>
>>     https://pkg.jenkins.io/debian/
> 
> I think we all get that and agree on it.
> 
>> The management of this public apt could require some manual effort,
>> while Isar (in plans) could update it automatically, so you update
>> source code for your product, run Isar and it provides complete apt
>> that you could upload to your server.
> 
> Exactly, those manual steps or custom scripts are outside the scope of
> Isar. Just like the flashing of the Images onto the targets.
>   
>> 2. Implement easy software update. On our host systems we usually do
>> not flash whole image if we need to update some packages. The update
>> is performed using 'apt-get install' (or by similar tool). But in
>> this case binary repo is needed, what could be again provided by Isar.
>>
>> Please do not limit Isar by deep embedded application, host systems
>> also could be managed by Isar.
> 
> If the repo ends up in the output directory all those scenarios are
> still possible, _and_ your source tree will not contain build-output.
>   
> Henning
> 
>> 3. In long term, help Debian community to manage Debian
>> infrastructure, what includes publishing apt updates.
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov