Re: RFC: base-apt caching improvements

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 27 Feb 2019 10:05:36 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Hi,
> 
> i did not really like the current approach to how we cache, because i
> thought we can and should do that somehow transparent and route all
> "apt" downloads through a proxy that will reprepro all files as the
> are requested.
> I did not find a proxy that can do that, maybe i did not look hard
> enough.
> 
> So i came back to the currently implemented model. Collect the files
> from the rootfs-es after we are all done. The way that is currently
> done is getting them from the apt cache. Unfortunately that is not
> guaranteed to work. Any package we install could mess with apt-config
> and therefore the caching of the rootfs-es. (i.e. something
> like /etc/apt/apt.conf.d/docker-clean you will find a container
> images)
>
> So that cache can not be trusted. And that cache does not work for
> source packages and can not fulfill our needs with regards to full
> caching/archiving.
> 
> So instead we should go and download all debs and sources of the
> currently installed packages of all rootfs-es (target and its
> buildchroots) explicitly and reprepro from there.
> 
> Here is what this could look like:
> 
> rm -rf /tmp/foo
> cd /tmp/foo
> dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y
> download ### reprepro *.deb
> rm -rf /tmp/foo-src
> cd /tmp/foo-src
> dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y
> source --download-only ### reprepro *.dsc
> 
> This still lacks filtering out all the packages from isar-apt but

Here are some snipplets to filter out isar-apt, does seem non-trivial
and i ended up parsing. Might actually require some perl to dig into
the guts of apt.

apt-cache policy sed | grep -A 1 "\*\*\* $( dpkg-query --show --showformat '${Version}' sed )" | grep -q "file:/isar-apt" 

or

grep -e "^Package: hello$" /var/lib/apt/lists/_isar-apt_dists_isar_main*_Packages -A 1 | grep "Version: $( dpkg-query --show --showformat '{Version}' hello )


The way we currently filter out isar-apt is pretty fragile and should
be improved. We find packages by matching file names. Instead we should
read the .debs with dpkg-deb and read the repo with reprepro.

Henning

> could be an improvement to our current way. We do not need to trust
> those fragile caches because we are explicit. We can do sources ...
> and should probably just do all of them anyways.
> For selective sources we could "apt-get source" into a central dldir
> and copy the sources to the recipe workdir. Would be a matter of
> mounting that shared dir and changing do_apt_fetch to use that
> staging dir to leave a copy. In fact we (Siemens) want all sources,
> and it might be a sane default for others as well.
> 
> Henning
>