Re: [RFC v2][PATCH 2/3] build-rep: Add helper class

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-01-24 21:53, Benedikt Niedermayr wrote:
> Am 24.01.2018 um 19:48 schrieb Jan Kiszka:
>> I also started to look into aptly, which I had on my notes for a while.
>> One reason is that it could act as a manual workaround for the missing
>> version-freezing support in Isar (pull into aptly repo, take a snapshot,
>> and then build from it with Isar).
>>
>> Another is that we also need some tooling to manage mirrors and
>> snapshots on the long run, and reprepro, TBHO, smells strange. On first
>> glance, it looks to me like the multistrap of the repo management tools:
>> once a cool thing, but now unmaintained.
>>
>> Did you dig further into aptly? I'm currently wondering how it
>> internally manages snapshots, if it does proper deduplication and how
>> much of that can be preserved when publishing for consumption by, say,
>> Isar...
>>
>> Jan
>>
> Yes,
> 
> Until now I have implemented two approaches.
> 
> So I will offer some of my results tomorrow.
> 
> What I first did, was to use aptly as a complete mirror. Aptly creates a
> mirror and each package, which is requested by Isar will be added to the
> mirror and then be downloaded from there.
> 
> Unfortunately that was not as easy as expected, since aptly cannot
> download packages directly. Aptly was originally developed for creating
> complete debian mirrors.
> 
> It has now the "-filter" option, which can be used for narrowing the
> packages (and also dependecies) to be mirrored.
> 
> So for each additional package I extended the filter and updated the
> mirror afterwards, but that architecture appeared to suffer under less
> performance, since each extension of the filter
> 
> and subsequent update caused a download of debians packages.gz file (~
> 10MB).
> 

You went down the same path I was exploring the past hours. But I
dropped as well, also because of such traps:

https://groups.google.com/forum/#!searchin/aptly-discuss/partial$20mirroring%7Csort:date/aptly-discuss/SCMrc0w5Mq0/up4RpnMLBgAJ

> 
> My current approach is to create only a local repo after the first Isar
> build suceeded. This avoids complexity for now, since Isar and all
> debian tools are resolving dependencies as usual, when
> 
> running against upstream repos.

Would be interested to see the changes you did for that.

> 
> After the first build I create a snapshot of the local repo (both is
> possible: creating snapshots of local repos and also mirrors).
> 
> 
> I'm just investigating the deduplication topic and will give you response.
> 

Deduplication seems to be done by aptly simply by creating hardlinks.
That even works nicely when publishing to a local directory.

Thinking the aptly path further: I quickly started to dream of a lazy
mirror. What if we could teach it to create a full mirror but only fetch
packages when someone requests them?

That means we would first need a live publication services using the
built-in webserver or something like inotify or even fuse for file-based
publication. Then we could track accesses and pull from upstream on
demand. After the first build, we would snapshot the result and use that
from then on. The impact on Isar would be minimal, no separate
dependency tracking, no Debian logic duplication, simple user interface.

That's very similar to what Claudius initially proposed, but we would be
building on top of an existing, mature repo management tool.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux