Re: [RFC v2][PATCH 2/3] build-rep: Add helper class

[Baurzhan Ismagulov <ibr@radix50.net>]

On Sun, Jan 14, 2018 at 05:53:03PM +0100, Jan Kiszka wrote:
> I agree with Henning's view. We should exploit Debian for building the
> repo and only use bitbake recipes to control and direct Debian tools,
> definitely not reimplement them.
> 
> Having one build run that both generates outputs and fills that repos
> which would then be used on succeeding runs would be ok from user
> perspective. We do not necessarily need that strict ordering of
> fetch-before-use in the first build.

It's a trade-off between perfectly solving the by-product way vs. solving the
up-front way with TODOs for the next steps.


My original vision has been to have a separate partial mirroring tool that
understands Isar metadata (recipes and / or Debian packages).

1. Conceptually, I dislike the by-product solution because in the projects,
   repo maintenance is often done by configuration managers. They don't build
   the code, they just make the mirror available (with Isar's predecessor,
   those were complete mirrors made with debmirror). Project developers switch
   from one snapshot to another and adjust the code if necessary. Requiring CMs
   to build the code is not optimal and, in my opinion, should not be done. We
   should provide tools and people should decide whether to combine them or
   not.

   Providing separate mirroring is also the right step towards offline work and
   SDK as supported by Yocto.

2. Mirrors are maintained per image, because two products in the same repo may
   have different release timelines. That is why we need to have an explicit
   list of images to produce the mirror for. This could be addressed with both
   bitbake-based approaches, by-product and up-front.

3. Projects update from time to time, not on every build. This could be done
   with both approaches through turning off mirroring in local.conf. With a
   separate tool, it would be more convenient to use.

4. Finally, there is a question of partial updates, e.g., update only glib and,
   optionally, its dependencies, dependents, and conflicting packages. This is
   not addressed by either of bitbake-based solutions.


To create such a tool, we would have to:

1. Understand Isar metadata.

2. Implement mirroring.

To avoid messing with (1) at this time, we used bitbake for now. The goal is to
implement mirroring quickly and get experience with it, because important
details may require adjusting the vision.


There are also technical issues with the by-product solution:

1. Complicated and error-prone workflow
2. Unnecessary build repeat burden for the project developer
3. Intertwined, inseparable mirror and build
4. Unpredictable builds, esp. if sid is used (usual for early development)


The following technical issues have been raised regarding the two up-front
implementations:

1. DEBIAN_DEPENDS mixes Build-Depends and Depends.

   I agree that this is not optimal. This could be solved by separating
   DEBIAN_DEPENDS and DEBIAN_BUILD_DEPENDS. This itself would be a temporary
   solution till we implement debian/control backend for bitbake (as an
   alternative to .bb backend).

2. We first need a working solution of two "inherit dpkg" packages actually
   depending on each-other.

   An intermediate solution is available in Isar's predecessor. The approach is
   to provide DEPENDS in recipes to ensure the right do_build order, and to
   install the development packages into buildchroot. Of course, in the future
   that should be extended with adding the built packages to apt and moving to
   debian/control backend.

   That said, it's a matter of priority -- reproducibility and wic were at the
   top of Jan's wishlist. FWIW, Alex already has an implementation for own
   package dependencies and will post it.

3. Reimplementing Debian tools and constant catching up with upstream.

   Some conceptual points first. As already discussed in the context of wic
   last year, we strive for solving project problems today and merging with
   upstream after we get experience with the feature, as opposed to developing
   huge series solving all problems of the world and pushing them upstream
   first. This delayed feedback loop works well e.g. for C and C++
   standardization. You see, now you promise a better implementation with less
   or no changes to wic; it may be well worth waiting with upstreaming. This is
   not to say we want to leave the problems in the code; we just want to choose
   what problem we tackle at which point of time.

   Another point is, we don't actually reimplement Debian tools. If a tool that
   we need existed, we would have used it, no? But they don't. So, we can
   extend an upstream one, or we can implement our own. For a PoC, we choose
   whatever is faster. Merging with upstream is a TODO that we deliberately add
   to our list. I agree that in this way, we've accumulated quite a number of
   hacks; we are going to resolve them.

   That said, we shouldn't shy away from creating and maintaining new tools if
   it's a better way of solving the problem. Debian describes the format of the
   files, and many tools provide similar functionality. Many Linux features
   have been maintained as patches for years before being accepted to the
   mainline.

   Returning to the issue at hand: I agree that parsing debian/control manually
   is not optimal. Isar already does that in other places; upstreaming it to
   some tool is a TODO that should not block moving forward on reproducibility.
   We expect changes in this area, so even the choice of the tool to merge
   depends on the workflow we'd end up with.


Up-front mirroring was Christian's requirement that noone objected to at the
time [1]. Considering all of this:

1. I suggest to clean up the first series with the list of images due to the
   use case #2, and merge it.

2. I'd rather not go for the second series, to avoid opening a new can of worms
   w.r.t. DEBIAN_DEPENDS, unless there are serious technical reasons in its
   favor.

3. We have always been aware of the issues and are working on them.


With kind regards,
Baurzhan.


References:

1. https://groups.google.com/d/msg/isar-users/4PVondzZryk/qFcYq1qqAQAJ