From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6469695873171324928 X-Received: by 10.223.151.69 with SMTP id r63mr34006wrb.3.1506498284220; Wed, 27 Sep 2017 00:44:44 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.23.21 with SMTP id l21ls179760lje.3.gmail; Wed, 27 Sep 2017 00:44:43 -0700 (PDT) X-Google-Smtp-Source: AOwi7QBwYmV+luZIbxnju1aOZGo1cBaV+a2QuK6S0U63ssmwtoSL/xY4OIX3c5GayVoIMspmy71h X-Received: by 10.25.74.88 with SMTP id x85mr53870lfa.27.1506498283654; Wed, 27 Sep 2017 00:44:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506498283; cv=none; d=google.com; s=arc-20160816; b=j6TL/x9ItNgSHvX4G8z3qSVwIl1Btdsu3lOutdtNTfCmbBpow6k6vMYskMj4Z3B0rW kCsfnNq+E/qI+Alof6YNOWB2lqi10Cl/fwYjH7nCbgo8p6gTbxlSAvvw9s0J3vwh+FFF zcXO+Wn0G35/8QGcCXRPbTDGDr+kOlUTlIXFD5R/W8oXHYAFmF1I/ZebUHX5wRSe+sn9 BRRA/rEtYOQAzWuDboDRVwOQ+7GarcerhEP0CO77CVx6D0STrwHmpthJVfjNU+HvVSgp C+zIb7TecKJiZOdDQudKAOi8vnFrG8akXcdn72jekI7lbiS1u+lwo6PnY3njWWOks2h6 hyjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=oZ76dwVfi+Hrh5XRpaSmwRGGZaVjnBOEY29LyQStPV8=; b=KRDlKA2An4h940vDznQmLYSkyMibwPcmX+TxzAtJFVXOm6Fr3l+8igiiglPzLoBpz8 J9+jG/ONJrLS8u1s8VYnoZTVmTskVkTWNDf6kRkXR32SWIQfIVVEHFEj4SYGgnrCElvS mA3bZ99fTeuUcberQqzZ+bYedHegDRTQi4omeF6D9njr2H+KsiqFChr9eH6awhbqwU/C bldDampUc4z4cOb6lFQK4zG1cN27PB6Trz0ahjCU4M9otvHk7zbqF0DyyIHMc9NOVD+2 JSLoLA2dwnnhUiKO4EP3PrapLDywHxHFB/PCZp3E670T/ftOVkRaBMW42fpFb8t0nn9J 5zBw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id k18si1057556wmd.1.2017.09.27.00.44.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Sep 2017 00:44:43 -0700 (PDT) Received-SPF: neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 192.35.17.2 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id v8R7ihuW024993 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 27 Sep 2017 09:44:43 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id v8R7ig77011211; Wed, 27 Sep 2017 09:44:42 +0200 Subject: Re: Handling of additional python dependencies To: Henning Schild Cc: isar-users@googlegroups.com References: <37e510ba-8e38-65d0-9980-286544bc8536@siemens.com> <20170927090641.344991c8@md1em3qc> From: Claudius Heine Message-ID: Date: Wed, 27 Sep 2017 09:44:42 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170927090641.344991c8@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: HZtwRD7INH8R Hi, On 09/27/2017 09:06 AM, Henning Schild wrote: > Am Mon, 25 Sep 2017 14:44:13 +0200 > schrieb "[ext] Claudius Heine" : > >> Hi, >> >> I am currently creating a proof of concept implementation for the >> caching apt repo proxy for isar. > > Cant you just use some existing implementation, like apt-cacher or > apt-cacher-ng? I would love to. But those solutions would still require some work AFAIK. I only have some experience with apt-cacher-ng and I think that its a bit of an overkill for our use and is missing some features that would be useful: * Ability to specify separate caching paths for the 'dists' and 'pool' directory. In our case the 'dists' directory of the repo would be part of the TMP_DIR while the 'pool' directory would be stored in the DL_DIR. Maybe this can be done with symlinks. * Just using the first available port and communicating it to the calling process. I could imagine that this can be done with some shell and netstat programming. I decided to implement my own proxy, because this is not that difficult with the python stl and got the implementation down rather quickly. But I had not anticipated that the python stl does not provide a http implementation for asyncio. Implementing that myself is a bit out of scope IMO. > >> My goal was to create this using asyncio, but the python std lacks a >> async http protocol implementation. I tried using as much as I can >> from the sync version of the http protocol that is available the >> python std lib, but that is not that trivial to do. I am now at the >> point where I have to decide if I just used some http asyncio library >> outside of the std or try another route with this. Maybe just use the >> sync version and slap more threads on it. >> >> How is the policy concerning external python dependencies and isar? >> Is it possible to just copy those libraries into the scripts/lib/ >> directory, specify it as a host dependency or am I forced to only use >> the python std? > > I would go for a host dependency or a git-submodule, more copies/forks > of stuff are not a good idea. Having bitbake and wic in there already > seems problematic. Host dependencies in that area are not optimal, because the APIs of those libraries are from my experience not stable, and updating our code to be compatible with a whole range of versions is a pain. When developing other python applications I normally write a 'requirements.txt' file that contains the name and version of the required python packages, then use virtualenv to install all those packages into an isolated directory and then start all python tools from within that environment. Maybe something similar could be done here as well. Thanks, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de