Re: [PATCH v4 0/4] Basic binary cache implementation

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 4 Oct 2017 19:50:09 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> On 10/04/2017 06:40 PM, Henning Schild wrote:
> > Am Wed, 4 Oct 2017 17:29:08 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 10/04/2017 02:57 PM, Jan Kiszka wrote:  
> >>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:  
> >>>> Hi,
> >>>>
> >>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:  
> >>>>> Hello everybody,
> >>>>>
> >>>>> this series introduces basic binary caching for Isar.
> >>>>>
> >>>>> There is a new layer: meta-isar-bin which is intended to be the
> >>>>> binary cache. All the packages that are built in Isar are stored
> >>>>> in this cache using reprepro.
> >>>>>
> >>>>> Having the separate layer makes possible to manage this cache
> >>>>> separately from the Isar build tree. So once you have built your
> >>>>> packages, you could re-use this cache for further builds.  
> >>>>
> >>>> We still have to talk about your requirement for the cache to be
> >>>> sharable and your resulting implementation of it as a meta layer.
> >>>>
> >>>> Since I still don't see the benefit of a sharable cache and think
> >>>> that putting binary packages into some kind of strange meta layer
> >>>> is not the right solution and could potentially create more
> >>>> problems that is solves because its very much in conflict with
> >>>> the fundamental idea of a meta layer.  
> >>
> >> The last statement is a bit strange for me. The idea of meta layers
> >> is to split software stack into logical layers that could be
> >> enabled/disabled on demand. Having binaries or recipes doesn't
> >> matter here, it's just the way how software is provided to build.  
> > 
> > Binaries that can be rebuild should not become part of the build
> > system. The need to distribute them to the targets or other
> > developers/images later is what we all agree on. But that should be
> > mostly outside of isar.  
> 
> Each binary could be re-built from sources (at least for Debian :-)
> ). Moreover Debian provides sources for them, so they are open for
> use.
> 
> But for me it looks like:
> 
>   - Installing binaries to target from 'ftp.debian.org' is allowed.
>   - Installing binary dependencies for custom built package from 
> 'ftp.debian.org' is allowed.
>   - Installing my binary application from 'my.debian.repo' is not 
> allowed. I have to build it from sources.

That is not only allowed it is absolutely wanted! But for that step you
would add that repo to your multistrap config or /etc/apt/, which is in
the scope of isar.
Building the repo is also in the scope of isar, but holding the
artefacts and publishing the repo outside the build-dir is not.

Henning

> What is the idea here? For me it's something like traditional OE 
> philosophy, but without benefit for Isar.
> 
> One of the typical usecase: if I'm, for example, developing 'hello' 
> application, why it's so critical for me to build, lets say, kernel
> from sources?
> 
> Alex
> 
> > 
> > Building an Image will result in the Image and a repo. Both have to
> > somehow make their way to the target, but that is somewhat out of
> > scope of the build system.
> >   
> >>>
> >>> Indeed, state like this (disclaimer: i didn't look into all
> >>> details), it sounds like a weird architecture to me as well.
> >>> Thinking of an artifact cache, I would rather expect something
> >>> like OE's sstate that one may preserve across builds, share
> >>> between devs, or whatever. If it's there, the build system
> >>> consults it, if not, it rebuilds. But sstate is not a layer, for
> >>> some good reasons. 
> >>
> >> Let me again summarize major points here. Isar is the build system
> >> designed to work with binary packages. That's the key feature of
> >> this product and main difference from OE-like systems. The Isar
> >> architecture assumes to be designed around binary packages, what
> >> provides absolutely different approach how binary images could be
> >> generated using bitbake.
> >>
> >> According to the Claudius's vision, that everything should always
> >> be built from sources - Isar is not the best option here, Yocto
> >> and OE are much better, because they were initially designed for
> >> this work. Attempts to apply OE design and philosophy to Isar
> >> could have negative impact and limit possible ways to implement
> >> custom features, because OE wasn't designed for this features.
> >> Also I wonder why we are able to use binary Debian packages, while
> >> the rest of software should be built from sources.
> >>
> >> This series introduces support for binary apt repositories in Isar.
> >> This feature is response to the following cases:
> >>
> >> 1. Publish custom apt layer in web. When I need to install some
> >> custom software to my Debian system, I frequently see instructions
> >> like:
> >>    - Add 'blablabla' to your /etc/apt/source.list
> >>    - Run apt-get update
> >>    - Run apt-get install
> >>
> >> That's for example how Jenkins and GitLab are distributed:
> >>
> >>     https://pkg.jenkins.io/debian/  
> > 
> > I think we all get that and agree on it.
> >   
> >> The management of this public apt could require some manual effort,
> >> while Isar (in plans) could update it automatically, so you update
> >> source code for your product, run Isar and it provides complete apt
> >> that you could upload to your server.  
> > 
> > Exactly, those manual steps or custom scripts are outside the scope
> > of Isar. Just like the flashing of the Images onto the targets.
> >     
> >> 2. Implement easy software update. On our host systems we usually
> >> do not flash whole image if we need to update some packages. The
> >> update is performed using 'apt-get install' (or by similar tool).
> >> But in this case binary repo is needed, what could be again
> >> provided by Isar.
> >>
> >> Please do not limit Isar by deep embedded application, host systems
> >> also could be managed by Isar.  
> > 
> > If the repo ends up in the output directory all those scenarios are
> > still possible, _and_ your source tree will not contain
> > build-output. 
> > Henning
> >   
> >> 3. In long term, help Debian community to manage Debian
> >> infrastructure, what includes publishing apt updates.
> >>  
> >   
>