Re: Handling of additional python dependencies

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 09/27/2017 10:00 AM, Henning Schild wrote:
> Am Wed, 27 Sep 2017 09:44:42 +0200
> schrieb Claudius Heine <claudius.heine.ext@siemens.com>:
> 
>> Hi,
>>
>> On 09/27/2017 09:06 AM, Henning Schild wrote:
>>> Am Mon, 25 Sep 2017 14:44:13 +0200
>>> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:
>>>    
>>>> Hi,
>>>>
>>>> I am currently creating a proof of concept implementation for the
>>>> caching apt repo proxy for isar.
>>>
>>> Cant you just use some existing implementation, like apt-cacher or
>>> apt-cacher-ng?
>>
>> I would love to. But those solutions would still require some work
>> AFAIK.
>>
>> I only have some experience with apt-cacher-ng and I think that its a
>> bit of an overkill for our use and is missing some features that
>> would be useful:
>>
>>    * Ability to specify separate caching paths for the 'dists' and
>> 'pool' directory. In our case the 'dists' directory of the repo would
>> be part of the TMP_DIR while the 'pool' directory would be stored in
>> the DL_DIR. Maybe this can be done with symlinks.
>>
>>    * Just using the first available port and communicating it to the
>> calling process. I could imagine that this can be done with some
>> shell and netstat programming.
>>
>> I decided to implement my own proxy, because this is not that
>> difficult with the python stl and got the implementation down rather
>> quickly. But I had not anticipated that the python stl does not
>> provide a http implementation for asyncio. Implementing that myself
>> is a bit out of scope IMO.
> 
> As far as i understand your statements there might be ways to use an
> existing tool. The fact that we are talking about additional python
> deps and how to handle them suggests that writing your own is
> "difficult" after all. A few symlinks or upstream patches are IMHO
> much easier to maintain than yet another proxy. And a first prototype
> that seems to work might still be far from something that actually
> works.
> We are talking about a cache, so you need to think about eviction,
> consistency, time to live ...

All of those should be done with a manual task, because otherwise we 
could not build the same image again and again day after day for same 
product. Updating the package index has to be a explicit step.

> What do you do with all the .debs when
> Packages.gz changes?

Normally just leave them be, because some other project might need them. 
My idea is the debs are shared between multiple builds similar how its 
down with the DL_DIR in oe.

Deleting them without thought could lead to breakage of the reproducibility.

One problem we have, that apt-cacher-ng etc. doesn't have to deal with 
is that we want to have one package pool with multiple different package 
dists. So I don't know if it starts messing with the package pool when 
removing packages and breaking builds for other projects.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de