Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Alex,

On 09/25/2017 01:57 PM, Alexander Smirnov wrote:
> Hi,
> 
> On 09/25/2017 01:49 PM, Claudius Heine wrote:
>> Hi Baurzhan,
>>
>> On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
>>> Hello Andreas and Claudius,
>>>
>>> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>>>> This looks like a misdesign, because meta-layers should not be 
>>>> populated
>>>> by the build process and furthermore, should contain recipes and not a
>>>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>>>
>>>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>>>> use this directory. Then you also have more flexibility and can handle
>>>> multiple different caches, i.e. the caches become selectable and
>>>> independent of the layer itself...
>>>>
>>>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>>>> snippents for multiconf and then you can have separate caches for
>>>> different target architectures and don't mix everything up in one pool.
>>>>
>>>> This way you can easily drop a pool by architecture without sorting
>>>> files out.
>>>
>>> I share your concerns and don't like that myself. That said, it's a demo
>>> implementation we ended up considering the following project 
>>> requirements:
>>>
>>> R1. Product's binary repo is persistent across builds on the same 
>>> host and
>>>      across the hosts of different developers.
>>
>> Why does the binary repo needs to be be 'persistent' across different 
>> hosts?
> 
> In general Isar designed to work with binary repositories, that's the 
> key feature.
> 
> Nobody forces you to use this cache persistent across different hosts, 
> you are able to build everything from sources at any time.
> But at the same time I want to have the possibility to cache some stable 
> Isar part for my needs and do not rebuild it everytime.

Yes, that is understandable. I don't dispute the usefulness of a cache 
in general, but the usefulness of a shareable cache.

IMO a cache is local to the build host and should not be shared.

> 
>>
>> This repo is only about the packages that are created by the recipes 
>> and those should already build reproducible packages so they already 
>> are 'persistent'. So why do you need a cache that can be shared with 
>> other hosts?
> 
> This topic is not about reproducible builds. Each software product has 
> sources, so it can be built from the scratch at anytime. But why I need 
> to do this, when I have binary? Installing Debian, user usually prefers 
> binary apt instead of building the system from sources. It's just more 
> comfortable and takes much less time.

The end user would rather have just a binary image to download and use. 
That takes even less time and is just slightly less comfortable than 
buying a micro sd card with the image already flashed on it.

> 
> Also having Isar in binary repo in general I don't need the Isar at all 
> (including setup development host). I'm able to generate target 
> filesystem by using multistrap only.

Then why make this part of isar if you intent to use it without isar?

The point is, if some user uses isar to create the image, she wants to 
build the image from, what isar considers, 'source' files. Those source 
files might be binary packages or source code that is compiled and 
packaged by isar.

What you are doing here is extracting some intermediate product of isar 
and planning to distribute it, and that is a really bad idea. Its 
against principles of reproducibility and maintainability.

As I already described, at some point in the future. Some developer will 
take this intermediate product of isar, change it and put it back into 
the isar image creation process. Then you get a desyncronisation between 
what isar considers 'source' and the intermediate product the developer 
has just modified.

>> Creating this sharable cache just raises the problem that some 
>> packages aren't build all the time. So its possible that if they break 
>> at some later point, it might happen unnoticed, because all devs and 
>> maybe even the CI is using some binary cache. Not ideal.
> 
> This problem doesn't relate to binary cache. If the software package 
> contains a bug, it doesn't matter if the package is binary or it's built 
> from sources every time. If after bug-fix the respective binary package 
> wasn't updated - it's an issue of engineer. If you don't want to export 
> the cache - you don't need to think about this problem.

My point it that exporting the cache should not be a requirement, 
because its a bad idea that might cause may problems.

> 
>>
>> You have to think about what is the real point of this cache is.
>>
>> Normally a build cache is there to make succeeding builds faster. To 
>> detect if some item in the cache needs to be deleted, because the 
>> source has updated is not a trivial problem. To do that with multiple 
>> hosts involved it becomes much more difficult.
> 
> Actually it doesn't. The cache is built using reprepro, which provides 
> possibility to query various information for specific packages. So, for 
> example, it's quite easy to implement the following steps in anonymous 
> function:
>   - get binary package version
>   - compare with the version in recipe
>   - if it differs - force Isar to update the cache for this package

And thats wrong. For instance I choose to change the install path of a 
binary from /usr/bin to /bin. I wouldn't need to change the version of 
the package, because it still does the same. With our mechanism it 
wouldn't rebuild because the version is the same. Bitbake can already 
handle these kind of issues.

> 
>>
>> If you put the cache into a version control system, you have to bind 
>> the cache to the exact revisions of the project repositories. Just to 
>> have some knowledge about how those packages are created. But those 
>> revisions hashes might be unique, but they are not persistent. The can 
>> disappear with a rebase, etc. And now we get into the situations where 
>> we have a cache that references some non existing repo revisions and 
>> nobody knows how the packages are build, but since they do exactly 
>> what is needed, nobody touches the sources anymore, but instead start 
>> to modify the cached packages directly if they want small changes. 
>> That is the nightmare scenario for reproducible builds.
>>
>> The longer I think about this, the more bad scenarios I can think of. 
>> And I am sure the reality might teach us all some more in time.
>> Please don't give the developers a repository with binary packages, 
>> they really should not be exposed to this kind of seduction.
> 
> I don't agree here, it's a matter of taste. Again, nobody forces you to 
> export the cache. For me - I don't want to spend hours building 
> everything from sources, when I know that there are stable binaries.

As I said, you are missing my point. I have nothing against caches in 
general. As long as they stay local to the build host.

>>
>>>
>>> Technically, meta-isar-bin is a layer that contains configuration 
>>> files that
>>> have to be put somewhere. As package installation via apt is a 
>>> requirement
>>> (dpkg doesn't install package dependencies), meta-isar-bin should 
>>> become the
>>> default and would thus be required in the current implementation.
>>>
>>> We had considered cloning meta-isar-bin into the build directory, but 
>>> that
>>> would mean manual configuration for new users, and (possibly 
>>> modified) configs
>>> could be deleted if the build directory is deleted -- an unpleasant 
>>> surprise
>>> for users familiar with OE.
>>>
>>> If it's a separate directory, the config files should go there, since 
>>> they are
>>> per-repo, and there could be several ones:
>>>
>>> product
>>> company-bin
>>> department-bin
>>> product-bin
>>>
>>> For a source distribution, having binary repos as layers sounds 
>>> perverse. But
>>> given Isar's focus on binary packages, it's layering in the sense 
>>> that more
>>> specific repos could override more general ones.
>>>
>>> In the future, we'd like to move more stuff into the core Isar 
>>> (either moving
>>> files from meta-isar to meta, or merging meta into meta-isar and 
>>> introducing
>>> meta-template). Maybe we could introduce kas in a simple way, which 
>>> would solve
>>> this problem.
>>>
>>> So, ATM I'd suggest to document the steps regarding meta-isar-bin for 
>>> creating
>>> real products.
>>>
>>> If anyone has an elegant solution for the issues above, I would be 
>>> glad to hear
>>> it. I agree with a separate directory approach, but suggest to 
>>> postpone it till
>>> we find a good way to introduce it to new users.
>>
>> I am still not convinced about the need to put binary repositories 
>> into layers. And introducing just a simple binary package repository 
>> as a package cache in the build directory does not need any special 
>> configuration files or overwriting existing configuration AFAIK. So 
>> introducing it to 'new users' would just work transparent.
> 
> reprepro requires configuration file.

So does wic. But wic does not need a separate layer just to contain its 
files.

> 
>>
>> Of course my solution would be a host local cache. But as I said, I 
>> don't really see any reason for having some kind of project global 
>> cache that is shared over multiple hosts. Ok you have a first first 
>> build, but
>> at the same time you don't even test if those packages can still be 
>> created from the sources/recipes. So thats not really an argument for me.
>>
> 
> But Debian users prefers to use ftp://ftp.debian.org/debian/ instead of 
> building from tarballs.

What does Debian users have to do with this? I would suggest we should 
target isar users and those are mostly developers that want to create 
Debian based images. Those developers what their own software deployed 
to a target machine and I expect that their own software is not 
available of ftp.debian.org.
So I don't get this point.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de