[PATCH v2 0/7] Local apt cache with aptly V2

[Benedikt Niedermayr <benbrenson89@googlemail.com>]

Hi all,
this series implements a PoC for running reproducible builds.
The core is based on aptly, which is a nice mirroring tool for debian repositories.

The mirroring solution does the following:
- Aptly generates a mirrors for upstream repositories as well as snapshots and also a local repository for isar packages.
- Get a list of all packages for mirroring before rootfs and buildchroot is beeing created.
  This is done by running an own bitbake cooker for parsing the recipes by taking append files,
  configs, datastore and also selected layers(bblayers) into account.
  This solves the problem when upstream repos getting updates between rootfs and buildchroot creation.
- Aptly will recieve the list of packages and calculates all further required dependencies for the mirror.
- Rootfs and buildchroot will never pull from upstream repos but instead from the local mirrors.

- The following dependencies will be collected:
  * Depends
  * Build-Depends
  * Build-Depends-Indep

Open Questions:
- Think about a more dynamic multistrap.conf generation. Ŕunning sed to multiconfig templates is not really flexible.
- Get a better naming convention for aptly mirrors, snapshots and repos. This will help to get multiconfig compatible.
  Where are pros and cons here?
- Where to store the cache? TMPDIR might not be the best location.
- Finding Build-Depends for packages build with dpkg-raw is not supported, yet.


Howto test:
Prefetching of Build-Depends requires all debian control files to be available, therefore two things have to be done:
- Run unpackall task: bitbake isar-image-base:do_unpackall
- Provide a debian/control file at ${WORKDIR}/${SRC_DIR}/debian/control
- Overload the do_unpack_debian hook to make this possible(only for non debianized packages).


I know not everbody will like this more complicated way, and first I was with them.
But now I don't see any possibility to run real deterministic builds.

Benedikt Niedermayr (7):
  Implement support for setting up the local apt mirror and isar
    repository with aptly.
  Added API class for apt cache.
  Added apt-cache functionality for buildchroot.
  Added apt-cache functionality for image rootfs.
  Added do_finalize_image task.
  Added support for installing isar packages to local isar repository.
  Introducing fetchall, unpackall and unpack_debian tasks.

 .../recipes-core/images/files/multistrap.conf.in   |   8 +-
 meta-isar/recipes-core/images/isar-image-base.bb   |  25 ++-
 meta/classes/apt-cache.bbclass                     | 237 +++++++++++++++++++++
 meta/classes/base.bbclass                          |  20 ++
 meta/classes/dpkg-base.bbclass                     |  15 +-
 meta/classes/image.bbclass                         |  23 +-
 meta/recipes-devtools/buildchroot/buildchroot.bb   |  36 +++-
 .../buildchroot/files/configscript.sh              |   1 +
 .../buildchroot/files/multistrap.conf.in           |  17 +-
 meta/recipes-devtools/isar-apt/files/aptly.conf.in |  17 ++
 meta/recipes-devtools/isar-apt/isar-apt.bb         | 174 +++++++++++++--
 11 files changed, 524 insertions(+), 49 deletions(-)
 create mode 100644 meta/classes/apt-cache.bbclass
 create mode 100644 meta/recipes-devtools/isar-apt/files/aptly.conf.in

--
2.7.4