Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov <asmirnov@ilbers.de>]

Hi,

On 09/25/2017 01:49 PM, Claudius Heine wrote:
> Hi Baurzhan,
> 
> On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
>> Hello Andreas and Claudius,
>>
>> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>>> This looks like a misdesign, because meta-layers should not be populated
>>> by the build process and furthermore, should contain recipes and not a
>>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>>
>>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>>> use this directory. Then you also have more flexibility and can handle
>>> multiple different caches, i.e. the caches become selectable and
>>> independent of the layer itself...
>>>
>>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>>> snippents for multiconf and then you can have separate caches for
>>> different target architectures and don't mix everything up in one pool.
>>>
>>> This way you can easily drop a pool by architecture without sorting
>>> files out.
>>
>> I share your concerns and don't like that myself. That said, it's a demo
>> implementation we ended up considering the following project 
>> requirements:
>>
>> R1. Product's binary repo is persistent across builds on the same host 
>> and
>>      across the hosts of different developers.
> 
> Why does the binary repo needs to be be 'persistent' across different 
> hosts?

In general Isar designed to work with binary repositories, that's the 
key feature.

Nobody forces you to use this cache persistent across different hosts, 
you are able to build everything from sources at any time.
But at the same time I want to have the possibility to cache some stable 
Isar part for my needs and do not rebuild it everytime.

> 
> This repo is only about the packages that are created by the recipes and 
> those should already build reproducible packages so they already are 
> 'persistent'. So why do you need a cache that can be shared with other 
> hosts?

This topic is not about reproducible builds. Each software product has 
sources, so it can be built from the scratch at anytime. But why I need 
to do this, when I have binary? Installing Debian, user usually prefers 
binary apt instead of building the system from sources. It's just more 
comfortable and takes much less time.

Also having Isar in binary repo in general I don't need the Isar at all 
(including setup development host). I'm able to generate target 
filesystem by using multistrap only.

> 
> Creating this sharable cache just raises the problem that some packages 
> aren't build all the time. So its possible that if they break at some 
> later point, it might happen unnoticed, because all devs and maybe even 
> the CI is using some binary cache. Not ideal.

This problem doesn't relate to binary cache. If the software package 
contains a bug, it doesn't matter if the package is binary or it's built 
from sources every time. If after bug-fix the respective binary package 
wasn't updated - it's an issue of engineer. If you don't want to export 
the cache - you don't need to think about this problem.

> 
> You have to think about what is the real point of this cache is.
> 
> Normally a build cache is there to make succeeding builds faster. To 
> detect if some item in the cache needs to be deleted, because the source 
> has updated is not a trivial problem. To do that with multiple hosts 
> involved it becomes much more difficult.

Actually it doesn't. The cache is built using reprepro, which provides 
possibility to query various information for specific packages. So, for 
example, it's quite easy to implement the following steps in anonymous 
function:
  - get binary package version
  - compare with the version in recipe
  - if it differs - force Isar to update the cache for this package

> 
> If you put the cache into a version control system, you have to bind the 
> cache to the exact revisions of the project repositories. Just to have 
> some knowledge about how those packages are created. But those revisions 
> hashes might be unique, but they are not persistent. The can disappear 
> with a rebase, etc. And now we get into the situations where we have a 
> cache that references some non existing repo revisions and nobody knows 
> how the packages are build, but since they do exactly what is needed, 
> nobody touches the sources anymore, but instead start to modify the 
> cached packages directly if they want small changes. That is the 
> nightmare scenario for reproducible builds.
> 
> The longer I think about this, the more bad scenarios I can think of. 
> And I am sure the reality might teach us all some more in time.
> Please don't give the developers a repository with binary packages, they 
> really should not be exposed to this kind of seduction.

I don't agree here, it's a matter of taste. Again, nobody forces you to 
export the cache. For me - I don't want to spend hours building 
everything from sources, when I know that there are stable binaries.

> 
>> R2. Building isar-image-base should be kept simple. New users should 
>> not be
>>      overwhelmed with repo setup, external tools, or manual 
>> configuration (as
>>      much as possible).
>>
>> R3. Build directory must be removable at any time without affecting 
>> cloned or
>>      newly built binary cache.
>>
>> We've translated that to the following design points:
>>
>> D1. Apt repo is versioned, e.g. in git.
>>
>> D2. Config files for this feature are put under isar/...
>>
>> D3. Given D1 and D2, binary packages go under isar/... :( .
>>
>> The idea is that in a project, the cache is a separate repo at the 
>> same level
>> as Isar:
>>
>> product (originally cloned from isar)
>> product-bin
>>
>> So, for production use, a tool like kas should be used to clone both 
>> at right
>> revisions. We didn't want to do that for the demo due to R2. Just like
>> meta-isar is currently a template for real products, meta-isar-bin is an
>> example that should be moved out of the product source repo for a real 
>> product.
> 
> Since you mention the 'meta' and 'meta-isar' layers. I would be in favor 
> of moving everything from the 'meta' layer into the 'meta-isar' layer. 
> We don't really need two layers IMO. We could put examples into 
> 'meta-isar-examples'.
> 
> Maybe we should discuss this separately.

Yes, I'd suggest to discuss this out of this thread.

> 
>>
>> Technically, meta-isar-bin is a layer that contains configuration 
>> files that
>> have to be put somewhere. As package installation via apt is a 
>> requirement
>> (dpkg doesn't install package dependencies), meta-isar-bin should 
>> become the
>> default and would thus be required in the current implementation.
>>
>> We had considered cloning meta-isar-bin into the build directory, but 
>> that
>> would mean manual configuration for new users, and (possibly modified) 
>> configs
>> could be deleted if the build directory is deleted -- an unpleasant 
>> surprise
>> for users familiar with OE.
>>
>> If it's a separate directory, the config files should go there, since 
>> they are
>> per-repo, and there could be several ones:
>>
>> product
>> company-bin
>> department-bin
>> product-bin
>>
>> For a source distribution, having binary repos as layers sounds 
>> perverse. But
>> given Isar's focus on binary packages, it's layering in the sense that 
>> more
>> specific repos could override more general ones.
>>
>> In the future, we'd like to move more stuff into the core Isar (either 
>> moving
>> files from meta-isar to meta, or merging meta into meta-isar and 
>> introducing
>> meta-template). Maybe we could introduce kas in a simple way, which 
>> would solve
>> this problem.
>>
>> So, ATM I'd suggest to document the steps regarding meta-isar-bin for 
>> creating
>> real products.
>>
>> If anyone has an elegant solution for the issues above, I would be 
>> glad to hear
>> it. I agree with a separate directory approach, but suggest to 
>> postpone it till
>> we find a good way to introduce it to new users.
> 
> I am still not convinced about the need to put binary repositories into 
> layers. And introducing just a simple binary package repository as a 
> package cache in the build directory does not need any special 
> configuration files or overwriting existing configuration AFAIK. So 
> introducing it to 'new users' would just work transparent.

reprepro requires configuration file.

> 
> Of course my solution would be a host local cache. But as I said, I 
> don't really see any reason for having some kind of project global cache 
> that is shared over multiple hosts. Ok you have a first first build, but
> at the same time you don't even test if those packages can still be 
> created from the sources/recipes. So thats not really an argument for me.
> 

But Debian users prefers to use ftp://ftp.debian.org/debian/ instead of 
building from tarballs.

-- 
With best regards,
Alexander Smirnov