From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6509751388101148672 X-Received: by 10.28.145.139 with SMTP id t133mr890119wmd.21.1516829502898; Wed, 24 Jan 2018 13:31:42 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.156.130 with SMTP id d2ls4110496wre.13.gmail; Wed, 24 Jan 2018 13:31:42 -0800 (PST) X-Google-Smtp-Source: AH8x2276W1SOqMgbTkAVJUeEvZQBZIbmj0fyqAturux9KCz0/ftHerPth092wsWQa8zM5UCh/koZ X-Received: by 10.223.139.86 with SMTP id v22mr1020619wra.11.1516829502313; Wed, 24 Jan 2018 13:31:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516829502; cv=none; d=google.com; s=arc-20160816; b=wbDpsQYxxRTpANo9A70LjmZw1nl9xWHhscNGq652hV0iHpI6h3cAcZQd7Om4oKwEAQ 3v7jz7w6xdX0dJtcSCXrBY4OuRVs7iJ3GyNYJYMNsUzbZhcwQNpKrbWysAgknQkEddN9 YpzDcwN8K/BDtRAv6L++7M/hHv5MBtmTv7QX93qAfm7lVLVUjnRjmMjS5Yl5JJe4Xis1 xzlO9zZSA46Aq7Rtnpl6NVlMmSb1Cvz9x5CPQjyn05HjshneUa5B7kH2q0bcvZvZJpb/ gMJ/obzY+zq9ZfWvpno/jiYH/rdXOteRontmDSMf9X5PIrce7HJXQE2QlPDoChA7qr4V jlrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:cc:references:to:subject :arc-authentication-results; bh=K5u8aXGPpz7ME+8VArYQRDNn1vTn2+BuyVUbq4RSSeQ=; b=u17iEtzGAzSPmy3LEpxtsFrw/1gZ3HekZWY1yCENSu4GDrhVo1XHRIc4Pz5d/5DDXR xZHDic4GjT7GiKMVAX3CRwcotvucUqpWDrRh3oEigRwMuiL5FOzygAuTvgFB1tHavdsK lRImE2bvnE+W5GBIbK6D7ewZnsnvyHjtC+/fRUseb7qs0a+xWcEwFo8yBPDj0XQ0pr+L GAC5XXJCMRMi/0mC2isx77lFrq+AjqYtIiKVS6TLApRKYmyCMBhotrlWGFzeMVeIeMK2 fX47i6ac5G8r7FKoQLqSfTQV4Q/LIr9IPAuMkVWN6Fl502CcwuOIY90fQRFTOdDW6L5M B2PQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id b37si448387wrd.3.2018.01.24.13.31.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jan 2018 13:31:42 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w0OLVeHP022544 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 24 Jan 2018 22:31:40 +0100 Received: from md1f2u6c.ww002.siemens.net ([167.87.53.243]) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w0OLVdD7025584; Wed, 24 Jan 2018 22:31:40 +0100 Subject: Re: [RFC v2][PATCH 2/3] build-rep: Add helper class To: Benedikt Niedermayr , isar-users References: <20180111111939.25667-1-asmirnov@ilbers.de> <20180111111939.25667-3-asmirnov@ilbers.de> <20180112133205.2d2d6615@mmd1pvb1c.ad001.siemens.net> <20180112172526.19bd23a2@mmd1pvb1c.ad001.siemens.net> <4c5b731d-d337-8be2-cd2d-025768ee6497@googlemail.com> Cc: Claudius Heine From: Jan Kiszka Message-ID: <7dff65b2-2d63-d694-6a68-69ba3f0a8ad4@siemens.com> Date: Wed, 24 Jan 2018 22:31:39 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <4c5b731d-d337-8be2-cd2d-025768ee6497@googlemail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 6WARLEdBmV7W On 2018-01-24 21:53, Benedikt Niedermayr wrote: > Am 24.01.2018 um 19:48 schrieb Jan Kiszka: >> I also started to look into aptly, which I had on my notes for a while. >> One reason is that it could act as a manual workaround for the missing >> version-freezing support in Isar (pull into aptly repo, take a snapshot, >> and then build from it with Isar). >> >> Another is that we also need some tooling to manage mirrors and >> snapshots on the long run, and reprepro, TBHO, smells strange. On first >> glance, it looks to me like the multistrap of the repo management tools: >> once a cool thing, but now unmaintained. >> >> Did you dig further into aptly? I'm currently wondering how it >> internally manages snapshots, if it does proper deduplication and how >> much of that can be preserved when publishing for consumption by, say, >> Isar... >> >> Jan >> > Yes, > > Until now I have implemented two approaches. > > So I will offer some of my results tomorrow. > > What I first did, was to use aptly as a complete mirror. Aptly creates a > mirror and each package, which is requested by Isar will be added to the > mirror and then be downloaded from there. > > Unfortunately that was not as easy as expected, since aptly cannot > download packages directly. Aptly was originally developed for creating > complete debian mirrors. > > It has now the "-filter" option, which can be used for narrowing the > packages (and also dependecies) to be mirrored. > > So for each additional package I extended the filter and updated the > mirror afterwards, but that architecture appeared to suffer under less > performance, since each extension of the filter > > and subsequent update caused a download of debians packages.gz file (~ > 10MB). > You went down the same path I was exploring the past hours. But I dropped as well, also because of such traps: https://groups.google.com/forum/#!searchin/aptly-discuss/partial$20mirroring%7Csort:date/aptly-discuss/SCMrc0w5Mq0/up4RpnMLBgAJ > > My current approach is to create only a local repo after the first Isar > build suceeded. This avoids complexity for now, since Isar and all > debian tools are resolving dependencies as usual, when > > running against upstream repos. Would be interested to see the changes you did for that. > > After the first build I create a snapshot of the local repo (both is > possible: creating snapshots of local repos and also mirrors). > > > I'm just investigating the deduplication topic and will give you response. > Deduplication seems to be done by aptly simply by creating hardlinks. That even works nicely when publishing to a local directory. Thinking the aptly path further: I quickly started to dream of a lazy mirror. What if we could teach it to create a full mirror but only fetch packages when someone requests them? That means we would first need a live publication services using the built-in webserver or something like inotify or even fuse for file-based publication. Then we could track accesses and pull from upstream on demand. After the first build, we would snapshot the result and use that from then on. The impact on Isar would be minimal, no separate dependency tracking, no Debian logic duplication, simple user interface. That's very similar to what Claudius initially proposed, but we would be building on top of an existing, mature repo management tool. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux