Am 24.01.2018 um 22:31 schrieb Jan Kiszka:
> On 2018-01-24 21:53, Benedikt Niedermayr wrote:
>> Am 24.01.2018 um 19:48 schrieb Jan Kiszka:
>>> I also started to look into aptly, which I had on my notes for a while.
>>> One reason is that it could act as a manual workaround for the missing
>>> version-freezing support in Isar (pull into aptly repo, take a snapshot,
>>> and then build from it with Isar).
>>>
>>> Another is that we also need some tooling to manage mirrors and
>>> snapshots on the long run, and reprepro, TBHO, smells strange. On first
>>> glance, it looks to me like the multistrap of the repo management tools:
>>> once a cool thing, but now unmaintained.
>>>
>>> Did you dig further into aptly? I'm currently wondering how it
>>> internally manages snapshots, if it does proper deduplication and how
>>> much of that can be preserved when publishing for consumption by, say,
>>> Isar...
>>>
>>> Jan
>>>
>> Yes,
>>
>> Until now I have implemented two approaches.
>>
>> So I will offer some of my results tomorrow.
>>
>> What I first did, was to use aptly as a complete mirror. Aptly creates a
>> mirror and each package, which is requested by Isar will be added to the
>> mirror and then be downloaded from there.
>>
>> Unfortunately that was not as easy as expected, since aptly cannot
>> download packages directly. Aptly was originally developed for creating
>> complete debian mirrors.
>>
>> It has now the "-filter" option, which can be used for narrowing the
>> packages (and also dependecies) to be mirrored.
>>
>> So for each additional package I extended the filter and updated the
>> mirror afterwards, but that architecture appeared to suffer under less
>> performance, since each extension of the filter
>>
>> and subsequent update caused a download of debians packages.gz file (~
>> 10MB).
>>
> You went down the same path I was exploring the past hours. But I
> dropped as well, also because of such traps:
>
> https://groups.google.com/forum/#!searchin/aptly-discuss/partial$20mirroring%7Csort:date/aptly-discuss/SCMrc0w5Mq0/up4RpnMLBgAJ
>
>> My current approach is to create only a local repo after the first Isar
>> build suceeded. This avoids complexity for now, since Isar and all
>> debian tools are resolving dependencies as usual, when
>>
>> running against upstream repos.
> Would be interested to see the changes you did for that.
>
>> After the first build I create a snapshot of the local repo (both is
>> possible: creating snapshots of local repos and also mirrors).
>>
>>
>> I'm just investigating the deduplication topic and will give you response.
>>
> Deduplication seems to be done by aptly simply by creating hardlinks.
> That even works nicely when publishing to a local directory.
>
> Thinking the aptly path further: I quickly started to dream of a lazy
> mirror. What if we could teach it to create a full mirror but only fetch
> packages when someone requests them?
>
> That means we would first need a live publication services using the
> built-in webserver or something like inotify or even fuse for file-based
> publication. Then we could track accesses and pull from upstream on
> demand. After the first build, we would snapshot the result and use that
> from then on. The impact on Isar would be minimal, no separate
> dependency tracking, no Debian logic duplication, simple user interface.
>
> That's very similar to what Claudius initially proposed, but we would be
> building on top of an existing, mature repo management tool.
>
> Jan
>


Hi,


> Would be interested to see the changes you did for that.
I can only give you my current implementation of my apt-cache.bbclass 
and dpkg-cross.bbclass.

The apt-cache.bbclass contains a simple shell based API for accessing 
the aptly functionality.

I will push that results to my next branch at my isar fork tomorrow.


> Deduplication seems to be done by aptly simply by creating hardlinks.
> That even works nicely when publishing to a local directory.
>
> Thinking the aptly path further: I quickly started to dream of a lazy
> mirror. What if we could teach it to create a full mirror but only fetch
> packages when someone requests them?
>
> That means we would first need a live publication services using the
> built-in webserver or something like inotify or even fuse for file-based
> publication. Then we could track accesses and pull from upstream on
> demand. After the first build, we would snapshot the result and use that
> from then on. The impact on Isar would be minimal, no separate
> dependency tracking, no Debian logic duplication, simple user interface.
>
> That's very similar to what Claudius initially proposed, but we would be
> building on top of an existing, mature repo management tool.
That sounds very nice. I don't know how to find good solution/design for 
that. But maybe a good brain wave might change that :-)


I also thinking about combining buildfragments (debian packages) with 
Bitbake's "setscene" mechanism, in order to create a sstate cache 
functionality, like yocto does.

I also don't know how to achieve that goal, since there is very less 
(means none) documentation about setscene, and the only point to start 
from there is the sstate implementation

of yocto itself. So first of all I have to understand the masses of code 
related to the sstate cache.

It is a big goal but it is also very nice feature and may solve a lot of 
problems related to mirroring or snapshoting a repository, since then a 
direct relationship between task hashes and

build fragments (debian packages) would be given.

If any of the task changes in a recipe, a new/old build fragment would 
be the result. And checking out any commit of a meta-layer would relate 
to matching build fragments.

This would be very helpful when checking out to a sooner commit of Isar 
(or any other layer).

A relationship between your current git repository state and related 
mirror could then be guaranteed.



Benni