Re: Handling of additional python dependencies

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 09/27/2017 09:06 AM, Henning Schild wrote:
> Am Mon, 25 Sep 2017 14:44:13 +0200
> schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:
> 
>> Hi,
>>
>> I am currently creating a proof of concept implementation for the
>> caching apt repo proxy for isar.
> 
> Cant you just use some existing implementation, like apt-cacher or
> apt-cacher-ng?

I would love to. But those solutions would still require some work AFAIK.

I only have some experience with apt-cacher-ng and I think that its a 
bit of an overkill for our use and is missing some features that would 
be useful:

  * Ability to specify separate caching paths for the 'dists' and 'pool' 
directory. In our case the 'dists' directory of the repo would be part 
of the TMP_DIR while the 'pool' directory would be stored in the DL_DIR. 
Maybe this can be done with symlinks.

  * Just using the first available port and communicating it to the 
calling process. I could imagine that this can be done with some shell 
and netstat programming.

I decided to implement my own proxy, because this is not that difficult 
with the python stl and got the implementation down rather quickly. But 
I had not anticipated that the python stl does not provide a http 
implementation for asyncio. Implementing that myself is a bit out of 
scope IMO.

> 
>> My goal was to create this using asyncio, but the python std lacks a
>> async http protocol implementation. I tried using as much as I can
>> from the sync version of the http protocol that is available the
>> python std lib, but that is not that trivial to do. I am now at the
>> point where I have to decide if I just used some http asyncio library
>> outside of the std or try another route with this. Maybe just use the
>> sync version and slap more threads on it.
>>
>> How is the policy concerning external python dependencies and isar?
>> Is it possible to just copy those libraries into the scripts/lib/
>> directory, specify it as a host dependency or am I forced to only use
>> the python std?
> 
> I would go for a host dependency or a git-submodule, more copies/forks
> of stuff are not a good idea. Having bitbake and wic in there already
> seems problematic.

Host dependencies in that area are not optimal, because the APIs of 
those libraries are from my experience not stable, and updating our code 
to be compatible with a whole range of versions is a pain.

When developing other python applications I normally write a 
'requirements.txt' file that contains the name and version of the 
required python packages, then use virtualenv to install all those 
packages into an isolated directory and then start all python tools from 
within that environment. Maybe something similar could be done here as well.

Thanks,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de