From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6469695873171324928 X-Received: by 10.223.142.172 with SMTP id q41mr35707wrb.25.1506499206885; Wed, 27 Sep 2017 01:00:06 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.93.130 with SMTP id r124ls224028wmb.12.gmail; Wed, 27 Sep 2017 01:00:06 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBW7Za2gTOx9QLodOjuJH89jQDu4DwWVF/aVKxzRlv7Xa8ysRisDR+5M1dUoAwdjEeeH22z X-Received: by 10.28.150.151 with SMTP id y145mr75755wmd.22.1506499206583; Wed, 27 Sep 2017 01:00:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506499206; cv=none; d=google.com; s=arc-20160816; b=FrO9hq6VFZ5PMpjaE39kRN4DmavE9MLJmr6NOIIo92EWqouQ3t9svcBFs7odqxSjN7 zxqHY5jN/4nsaVpJR17iwqUntt6cfd28BNmUIzlx/Tknxi0raAD4nDO/++SmsGZ/z47C 1SywWp7affEXGHutmBaWg/P6ho29qg2zRB7VvNcDLKcqMDU2LWBmso22f2o3d1jqgD9L 8T8gYb1q4kWnwNBmsrHtdVrb5VCOeOh0OTDqiN/8JrKM1ZeLWCMEvvnIK3/Obl+3YvEc IcJe7UBM+7dDo0qYF0shlDyfLirkMQkCir2b56mqvowZdKNg1w937re/yVLWk0j1su// MbnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=tW2wIP3XgU0VT8/xbHmgMwRj1uK9wEscXl8C9kNvQjE=; b=qQL8caAX/5RDrt3owahuxwYNw0hmQl5AzUqaI3p9/qcjY8kOEeLRiNN2m3T+/3P6Bc sKXhG/qVZbt5mGl1zToehh4DoJvCiDeijAUS4dYwqp3lYpikea5Z7TeYqgB2dGmlHFKf ZlgO4buGIXuGIdEFuSRAYS8j5y3wHmNR1IaEUUSMJUYxHs1InRWaW0CyA/KrLbWyWmaK KD26Voo0TTDUyaaVxnnlLShx/v5XO3QAPdnu496DwMOd+iYYTtLW7QM4QWKfvUJEyxYN RMCJWDoW7p+DPbLNMc87Xk2WKuYyQpgl/E02hvRORjRt2ciLJmPwRQ+rN8Pqy+fUeTL0 LoaQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id b62si262563wmd.3.2017.09.27.01.00.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Sep 2017 01:00:06 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of henning.schild@siemens.com) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.siemens.de (mail1.siemens.de [139.23.33.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v8R8060e025955 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 27 Sep 2017 10:00:06 +0200 Received: from md1em3qc ([139.25.68.40]) by mail1.siemens.de (8.15.2/8.15.2) with ESMTP id v8R806iO015621; Wed, 27 Sep 2017 10:00:06 +0200 Date: Wed, 27 Sep 2017 10:00:06 +0200 From: Henning Schild To: Claudius Heine Cc: Subject: Re: Handling of additional python dependencies Message-ID: <20170927100006.1c4c4fbc@md1em3qc> In-Reply-To: References: <37e510ba-8e38-65d0-9980-286544bc8536@siemens.com> <20170927090641.344991c8@md1em3qc> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: QZe6lr9Y0Ggq Am Wed, 27 Sep 2017 09:44:42 +0200 schrieb Claudius Heine : > Hi, > > On 09/27/2017 09:06 AM, Henning Schild wrote: > > Am Mon, 25 Sep 2017 14:44:13 +0200 > > schrieb "[ext] Claudius Heine" : > > > >> Hi, > >> > >> I am currently creating a proof of concept implementation for the > >> caching apt repo proxy for isar. > > > > Cant you just use some existing implementation, like apt-cacher or > > apt-cacher-ng? > > I would love to. But those solutions would still require some work > AFAIK. > > I only have some experience with apt-cacher-ng and I think that its a > bit of an overkill for our use and is missing some features that > would be useful: > > * Ability to specify separate caching paths for the 'dists' and > 'pool' directory. In our case the 'dists' directory of the repo would > be part of the TMP_DIR while the 'pool' directory would be stored in > the DL_DIR. Maybe this can be done with symlinks. > > * Just using the first available port and communicating it to the > calling process. I could imagine that this can be done with some > shell and netstat programming. > > I decided to implement my own proxy, because this is not that > difficult with the python stl and got the implementation down rather > quickly. But I had not anticipated that the python stl does not > provide a http implementation for asyncio. Implementing that myself > is a bit out of scope IMO. As far as i understand your statements there might be ways to use an existing tool. The fact that we are talking about additional python deps and how to handle them suggests that writing your own is "difficult" after all. A few symlinks or upstream patches are IMHO much easier to maintain than yet another proxy. And a first prototype that seems to work might still be far from something that actually works. We are talking about a cache, so you need to think about eviction, consistency, time to live ... What do you do with all the .debs when Packages.gz changes? Henning > > > >> My goal was to create this using asyncio, but the python std lacks > >> a async http protocol implementation. I tried using as much as I > >> can from the sync version of the http protocol that is available > >> the python std lib, but that is not that trivial to do. I am now > >> at the point where I have to decide if I just used some http > >> asyncio library outside of the std or try another route with this. > >> Maybe just use the sync version and slap more threads on it. > >> > >> How is the policy concerning external python dependencies and isar? > >> Is it possible to just copy those libraries into the scripts/lib/ > >> directory, specify it as a host dependency or am I forced to only > >> use the python std? > > > > I would go for a host dependency or a git-submodule, more > > copies/forks of stuff are not a good idea. Having bitbake and wic > > in there already seems problematic. > > Host dependencies in that area are not optimal, because the APIs of > those libraries are from my experience not stable, and updating our > code to be compatible with a whole range of versions is a pain. > > When developing other python applications I normally write a > 'requirements.txt' file that contains the name and version of the > required python packages, then use virtualenv to install all those > packages into an isolated directory and then start all python tools > from within that environment. Maybe something similar could be done > here as well. > > Thanks, > Claudius >