Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi Baurzhan,

On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
> Hello Andreas and Claudius,
> 
> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>> This looks like a misdesign, because meta-layers should not be populated
>> by the build process and furthermore, should contain recipes and not a
>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>
>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>> use this directory. Then you also have more flexibility and can handle
>> multiple different caches, i.e. the caches become selectable and
>> independent of the layer itself...
>>
>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>> snippents for multiconf and then you can have separate caches for
>> different target architectures and don't mix everything up in one pool.
>>
>> This way you can easily drop a pool by architecture without sorting
>> files out.
> 
> I share your concerns and don't like that myself. That said, it's a demo
> implementation we ended up considering the following project requirements:
> 
> R1. Product's binary repo is persistent across builds on the same host and
>      across the hosts of different developers.

Why does the binary repo needs to be be 'persistent' across different hosts?

This repo is only about the packages that are created by the recipes and 
those should already build reproducible packages so they already are 
'persistent'. So why do you need a cache that can be shared with other 
hosts?

Creating this sharable cache just raises the problem that some packages 
aren't build all the time. So its possible that if they break at some 
later point, it might happen unnoticed, because all devs and maybe even 
the CI is using some binary cache. Not ideal.

You have to think about what is the real point of this cache is.

Normally a build cache is there to make succeeding builds faster. To 
detect if some item in the cache needs to be deleted, because the source 
has updated is not a trivial problem. To do that with multiple hosts 
involved it becomes much more difficult.

If you put the cache into a version control system, you have to bind the 
cache to the exact revisions of the project repositories. Just to have 
some knowledge about how those packages are created. But those revisions 
hashes might be unique, but they are not persistent. The can disappear 
with a rebase, etc. And now we get into the situations where we have a 
cache that references some non existing repo revisions and nobody knows 
how the packages are build, but since they do exactly what is needed, 
nobody touches the sources anymore, but instead start to modify the 
cached packages directly if they want small changes. That is the 
nightmare scenario for reproducible builds.

The longer I think about this, the more bad scenarios I can think of. 
And I am sure the reality might teach us all some more in time.
Please don't give the developers a repository with binary packages, they 
really should not be exposed to this kind of seduction.

> R2. Building isar-image-base should be kept simple. New users should not be
>      overwhelmed with repo setup, external tools, or manual configuration (as
>      much as possible).
> 
> R3. Build directory must be removable at any time without affecting cloned or
>      newly built binary cache.
> 
> We've translated that to the following design points:
> 
> D1. Apt repo is versioned, e.g. in git.
> 
> D2. Config files for this feature are put under isar/...
> 
> D3. Given D1 and D2, binary packages go under isar/... :( .
> 
> The idea is that in a project, the cache is a separate repo at the same level
> as Isar:
> 
> product (originally cloned from isar)
> product-bin
> 
> So, for production use, a tool like kas should be used to clone both at right
> revisions. We didn't want to do that for the demo due to R2. Just like
> meta-isar is currently a template for real products, meta-isar-bin is an
> example that should be moved out of the product source repo for a real product.

Since you mention the 'meta' and 'meta-isar' layers. I would be in favor 
of moving everything from the 'meta' layer into the 'meta-isar' layer. 
We don't really need two layers IMO. We could put examples into 
'meta-isar-examples'.

Maybe we should discuss this separately.

> 
> Technically, meta-isar-bin is a layer that contains configuration files that
> have to be put somewhere. As package installation via apt is a requirement
> (dpkg doesn't install package dependencies), meta-isar-bin should become the
> default and would thus be required in the current implementation.
> 
> We had considered cloning meta-isar-bin into the build directory, but that
> would mean manual configuration for new users, and (possibly modified) configs
> could be deleted if the build directory is deleted -- an unpleasant surprise
> for users familiar with OE.
> 
> If it's a separate directory, the config files should go there, since they are
> per-repo, and there could be several ones:
> 
> product
> company-bin
> department-bin
> product-bin
> 
> For a source distribution, having binary repos as layers sounds perverse. But
> given Isar's focus on binary packages, it's layering in the sense that more
> specific repos could override more general ones.
> 
> In the future, we'd like to move more stuff into the core Isar (either moving
> files from meta-isar to meta, or merging meta into meta-isar and introducing
> meta-template). Maybe we could introduce kas in a simple way, which would solve
> this problem.
> 
> So, ATM I'd suggest to document the steps regarding meta-isar-bin for creating
> real products.
> 
> If anyone has an elegant solution for the issues above, I would be glad to hear
> it. I agree with a separate directory approach, but suggest to postpone it till
> we find a good way to introduce it to new users.

I am still not convinced about the need to put binary repositories into 
layers. And introducing just a simple binary package repository as a 
package cache in the build directory does not need any special 
configuration files or overwriting existing configuration AFAIK. So 
introducing it to 'new users' would just work transparent.

Of course my solution would be a host local cache. But as I said, I 
don't really see any reason for having some kind of project global cache 
that is shared over multiple hosts. Ok you have a first first build, but
at the same time you don't even test if those packages can still be 
created from the sources/recipes. So thats not really an argument for me.

Cheers,
Claudius



-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de