From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6469695873171324928 X-Received: by 10.28.128.131 with SMTP id b125mr178081wmd.18.1506514404324; Wed, 27 Sep 2017 05:13:24 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.142.199 with SMTP id q190ls374300wmd.4.canary-gmail; Wed, 27 Sep 2017 05:13:24 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCZS25btiQdyw4WuyvfTWgFZ9yYDV8JUIeG+DDAY04CY+U/vxAlukCU/TeeJFpQWffCpL2l X-Received: by 10.223.138.151 with SMTP id y23mr142876wry.21.1506514404033; Wed, 27 Sep 2017 05:13:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506514404; cv=none; d=google.com; s=arc-20160816; b=k4m3+38BazmKhmTSznhlFNF8HBn80aZnYoVbzZ0YnEc5HAOJvYPHWrCMDSiXWLRerM Qr3OEK6bpWLlS3WmeZHil6AVcEHEMIXPjcW+PU1nARI4WDaiXhjIIoQSUF+RInwpsaMT 1PZDi8zISfqSjYHL/citXnsOYmafFeiGtfN04NVspxU2WnzGhSE1Oq4hKQU+yyoU/gTr 3MpCds7zSByVczCf/OJOv0FsciTJC50Vy98ORzrzFEklVsYmqf703MqoXE+y2PKCdM2T Szut/vvekVbJ9nWeij1rbaM1vXkWmIvcnzwK7cW55QOp76gozRTUQCLUF4tBYN1FTo65 TMJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=TNVZrcBvDX0Uk8ypgUxfyUb+9m4mkE3CsC6ppdfQ/6k=; b=Np2JS+zuPJqE//jIs2WfHmPfGTPflEmYDTjJxI7aMUEzQMjEggKErbJRsWjEGG12H0 Dn6oRnuCM41rtht2jr1G7sAfYETnCGU4D2BHjtEMdQidIL7PLw8yWfHs0MTxLRidrK5o u8Z1gTGOfqognvB+RmVL/R1guoHd2ZvHmgcU9g5RFzcm4s2TvT4kLGjp5MYJ7Sp6l78d o0VgorWBaVIXwDWN7/xzekPA/V5rgXSDeE3N0HN+RzsWyXq5/MAHHty4m43mGCXecfBj nNFHzB7lAa2UEEv0pDO43RlkrZnrDD8r4+xNpRJpD75me41em8EV0tBJA9geJnzpAUYO ybVw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id 191si302485wmn.2.2017.09.27.05.13.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Sep 2017 05:13:23 -0700 (PDT) Received-SPF: neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 194.138.37.39 is neither permitted nor denied by best guess record for domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id v8RCDNRj002465 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 27 Sep 2017 14:13:23 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id v8RCDNRk022670; Wed, 27 Sep 2017 14:13:23 +0200 Subject: Re: Handling of additional python dependencies To: Henning Schild Cc: isar-users@googlegroups.com References: <37e510ba-8e38-65d0-9980-286544bc8536@siemens.com> <20170927090641.344991c8@md1em3qc> <20170927100006.1c4c4fbc@md1em3qc> From: Claudius Heine Message-ID: Date: Wed, 27 Sep 2017 14:13:23 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170927100006.1c4c4fbc@md1em3qc> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: //qJMRsrvm7J Hi, On 09/27/2017 10:00 AM, Henning Schild wrote: > Am Wed, 27 Sep 2017 09:44:42 +0200 > schrieb Claudius Heine : > >> Hi, >> >> On 09/27/2017 09:06 AM, Henning Schild wrote: >>> Am Mon, 25 Sep 2017 14:44:13 +0200 >>> schrieb "[ext] Claudius Heine" : >>> >>>> Hi, >>>> >>>> I am currently creating a proof of concept implementation for the >>>> caching apt repo proxy for isar. >>> >>> Cant you just use some existing implementation, like apt-cacher or >>> apt-cacher-ng? >> >> I would love to. But those solutions would still require some work >> AFAIK. >> >> I only have some experience with apt-cacher-ng and I think that its a >> bit of an overkill for our use and is missing some features that >> would be useful: >> >> * Ability to specify separate caching paths for the 'dists' and >> 'pool' directory. In our case the 'dists' directory of the repo would >> be part of the TMP_DIR while the 'pool' directory would be stored in >> the DL_DIR. Maybe this can be done with symlinks. >> >> * Just using the first available port and communicating it to the >> calling process. I could imagine that this can be done with some >> shell and netstat programming. >> >> I decided to implement my own proxy, because this is not that >> difficult with the python stl and got the implementation down rather >> quickly. But I had not anticipated that the python stl does not >> provide a http implementation for asyncio. Implementing that myself >> is a bit out of scope IMO. > > As far as i understand your statements there might be ways to use an > existing tool. The fact that we are talking about additional python > deps and how to handle them suggests that writing your own is > "difficult" after all. A few symlinks or upstream patches are IMHO > much easier to maintain than yet another proxy. And a first prototype > that seems to work might still be far from something that actually > works. > We are talking about a cache, so you need to think about eviction, > consistency, time to live ... All of those should be done with a manual task, because otherwise we could not build the same image again and again day after day for same product. Updating the package index has to be a explicit step. > What do you do with all the .debs when > Packages.gz changes? Normally just leave them be, because some other project might need them. My idea is the debs are shared between multiple builds similar how its down with the DL_DIR in oe. Deleting them without thought could lead to breakage of the reproducibility. One problem we have, that apt-cacher-ng etc. doesn't have to deal with is that we want to have one package pool with multiple different package dists. So I don't know if it starts messing with the package pool when removing packages and breaking builds for other projects. Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de