Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Alexander Smirnov <asmirnov@ilbers.de>]

On 02/13/2018 11:22 AM, Jan Kiszka wrote:
> On 2018-02-13 09:09, Alexander Smirnov wrote:
>>
>>
>> On 02/13/2018 10:22 AM, Jan Kiszka wrote:
>>> On 2018-02-12 19:51, Alexander Smirnov wrote:
>>>> On 02/12/2018 08:00 PM, Jan Kiszka wrote:
>>>>> On 2018-02-12 17:31, Alexander Smirnov wrote:
>>>>>>
>>>>>>
>>>>>> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>>>>>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>>>>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>>>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This ensures that we can override packages from upstream
>>>>>>>>>>>>>> Debian or
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>>>>>>> for now
>>>>>>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>>>>>>> buildchroot so
>>>>>>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>>>>>>> image
>>>>>>>>>>>>>> build, this does not work because all packages are pull
>>>>>>>>>>>>>> during the
>>>>>>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to
>>>>>>>>>>>>>> ensure
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>>>>>>> use-case for
>>>>>>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>>>>>>> prioritization could be resolved by specifying suffix to
>>>>>>>>>>>>> version.
>>>>>>>>>>>>
>>>>>>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>>>>>>
>>>>>>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>>>>>>> target),
>>>>>>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>>>>>>> even
>>>>>>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will
>>>>>>>>>>>> break
>>>>>>>>>>>> when
>>>>>>>>>>>> that virtual package linux-cip is selected but can't be
>>>>>>>>>>>> installed
>>>>>>>>>>>> due to
>>>>>>>>>>>> the version preference "newest").
>>>>>>>>>>>
>>>>>>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'?
>>>>>>>>>>> Having
>>>>>>>>>>> the same name as with original Debian one could be confusing.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That is mandatory, in order to write packages that depend on THE
>>>>>>>>>> kernel,
>>>>>>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>>>>>>
>>>>>>>>> To make it clearer, just look at the example-module case: The
>>>>>>>>> recipe
>>>>>>>>> can
>>>>>>>>> build both against the distro as well as all custom kernels,
>>>>>>>>> without
>>>>>>>>> touching it. And that is a pattern not limited to kernel<->module
>>>>>>>>> relationships.
>>>>>>>>
>>>>>>>> kernel<->module pair are built from the same source tree, so here no
>>>>>>>> problem with ${PN}-${PV} substitution as dependency. Also I've
>>>>>>>> parsed
>>>>>>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>>>>>>> correlation with upstream packages.
>>>>>>>
>>>>>>> No, they can be independent, just like in the example I submitted.
>>>>>>> It's
>>>>>>> an external kernel module vs. some kernel. Please re-read.
>>>>>>>
>>>>>>> For Debian, this specific problem does not exist because Debian only
>>>>>>> ships in-tree modules or those that are built on the target on
>>>>>>> installation. But you will find the exact same pattern when browsing
>>>>>>> external projects that come with modules.
>>>>>>
>>>>>> Ok, I see that you define KERNEL_ARCH variable in machine config and
>>>>>> this variable is used to bind kernel<->module build process. Sometimes
>>>>>> KERNEL_ARCH has to be changed according to your kernel configuration.
>>>>>> It's not enough to change preferred_provider from "linux-distro" to
>>>>>> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
>>>>>> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.
>>>>>
>>>>> OK, but this cannot be defined by the kernel recipe. It has to be
>>>>> defined by the machine setting. A kernel recipe could then select a
>>>>> config that matches or reject the KERNEL_ARCH. Currently, we do nothing
>>>>> in this direction, that is a valid point for future improvements.
>>>>>
>>>>>>
>>>>>> How you see the overall procedure to switch to custom kernel? Why
>>>>>
>>>>> PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"
>>>>>
>>>>> See also
>>>>> https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf
>>>>>
>>>>>
>>>>>
>>>>>> KERNEL_ARCH can't be "cip-armmp" in this case?
>>>>>
>>>>> Because the KERNEL_ARCH remains amd64, armmp, or whatever you
>>>>> configured. Changing the kernel source doesn't change that in any way.
>>>>
>>>> It doesn't remain for all the cases. ARM debian provides two kernel
>>>> images:
>>>>
>>>>    - linux-image-armmp
>>>>    - linux-image-armmp-lpae
>>>>
>>>> Current Isar doesn't handle this, but if you build your kernel with
>>>> LPAE, I assume you should change KERNEL_ARCH respectively, otherwise
>>>> there will be some inaccuracy. So the same machine could have several
>>>> KERNEL_ARCHs.
>>>
>>> In fact, one machine would have to have multiple configs, selecting the
>>> kernel flavor. In OE terms, that would make different machines, though
>>> targeting the same hardware.
>>>
>>>>
>>>> Then, what should I write in KERNEL_ARCH if I add new machine, let's say
>>>> Cubieboard?
>>>>
>>>> In general, KERNEL_ARCH is not exactly what it's used here for.
>>>> According to the:
>>>>
>>>> https://kernel-handbook.alioth.debian.org/ch-packaging.html
>>>>
>>>> the linux image name looks like the following:
>>>>
>>>>    - linux-image[-featureset]-flavour
>>>>
>>>> Also chapter 3.3 stays:
>>>>
>>>> 8<--
>>>> Each architecture usually has multiple flavours of the binary kernel
>>>> images. Different flavours correspond to different kernel configuration
>>>> files, used to build the binary images from the same kernel tree.
>>>> 8<--
>>>>
>>>> So having KERNEL_ARCH as the constant is deviation from Debian flow.
>>>>
>>>> In addition:
>>>>
>>>> 8<--
>>>> Again, multiple flavours of binary images may be built from the
>>>> featureset tree. For example, the i386 architecture has a number of
>>>> different flavours, such as 686 and 686-pae, built from the common
>>>> Debian kernel source. It also contains the rt featureset. The source
>>>> tree for building the kernels for each of these featuresets is obtained
>>>> by applying additional patches to the Debian kernel source. It may be
>>>> used to build the rt-686-pae binary image flavours.
>>>> 8<--
>>>>
>>>> So in my opinion, CIP is a feature set.
>>>>
>>>> I don't really think, that replacing upstream Debian kernel package by
>>>> custom one and claiming that is the same will make Debian community
>>>> happy...
>>>>
>>>>
>>>> Ok, I see that we have different vision on this topic, I could stay with
>>>> this implementation for now, but in general in my opinion this looks
>>>> like a big hack and should be reworked.
>>>
>>> Debian has a strong word here, though it complicates things
>>> significantly. But let me look into expressing the kernel selection via
>>> KERNEL_WHATEVER. It will mean that PREFERRED_PROVIDER can no longer be
>>> used, though. And we will have to generate / parameterize the control
>>> files of all module packages. Maybe that's Debian intention as well.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> For example armbian uses branch names and machines to distinguish
>>>>>> images:
>>>>>>
>>>>>> https://www.armbian.com/kernel/
>>>>>>
>>>>>>     - linux-image-dev-sun8i
>>>>>>     - linux-image-next-odroidc2
>>>>>>
>>>>>> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
>>>>>> conflicts with the upstream. Also this doesn't affect modules building
>>>>>> at all.
>>>>>
>>>>> This is about conflict: We are *replacing* the distro kernel. But when
>>>>> doing this, we do not want to change all its consumers. Armbian likely
>>>>> never ran into the use case of having to design such a full
>>>>> replacement,
>>>>> also regarding dependencies (I do not see them shipping any out-of-tree
>>>>> module packages).
>>>>>
>>>>> And the same is true for all the non-kernel cases we will face when
>>>>> replacing library-like packages. Please do not focus all your thoughts
>>>>> on the kernel. This issue is absolutely generic.
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> What are the other use-cases for this pattern? As I mentioned above
>>>>>>>> with
>>>>>>>> 'glibc' example, this is unclean approach, distro could become
>>>>>>>> broken at
>>>>>>>> anytime. This could become robust *only* after build reproducibility
>>>>>>>> integration, when you cache the upstream content. But with
>>>>>>>> build-rep you
>>>>>>>> could manually manage the content of the cache, so prioritization
>>>>>>>> is not
>>>>>>>> needed, you could just drop duplications from the cache.
>>>>>>>
>>>>>>> Every lib-dependency is another pattern for the special kernel/module
>>>>>>> case: Every distro package that come with a dependencies that you
>>>>>>> want
>>>>>>> to replace with a custom build AND you have good reason to downgrade
>>>>>>> that package, then you want your own packages to have higher prior.
>>>>>>> That's why this hard-coding is the right thing to do, in general: If
>>>>>>> you
>>>>>>> are building a package that competes with upstream, there must be a
>>>>>>> reason...
>>>>>>>
>>>>>>>>
>>>>>>>> In general having packages with the same name in 'source.list' and
>>>>>>>> overriding version policies in apt looks very dangerous for me.
>>>>>>>>
>>>>>>>> If the current series requires prioritization for kernel only, it
>>>>>>>> would
>>>>>>>> be nice to specify only this package in apt preferences.
>>>>>>>
>>>>>>> See above, the kernel is just the first case. There will be more, I'm
>>>>>>> absolutely sure.
>>>>>>
>>>>>> I've asked especially about these cases. :-( At the moment there is
>>>>>> only
>>>>>> kernel case, which could be solved without this prioritization.
>>>>>
>>>>> Use your imagination: enforced replacement of ANY package we rebuild,
>>>>> but with a lower version. We rebuild packages for a reason. That
>>>>> implies
>>>>> we want them to be actually used. And that means our packages require a
>>>>> higher priority.
>>>>>
>>>>
>>>> My imagination told me how it will be difficult to debug segfaults on
>>>> the target, because I pinned old libA in Isar-apt, while upstream
>>>> dependents already have been compiled according the new version of libA.
>>>> It wouldn't be so funny if nobody had faced with such problems in the
>>>> past...
>>>
>>> Again: One guy shooting himself into the foot by explicitly stating this
>>> incorrect setup doesn't mean all others users have to give up on this
>>> valid and required default property of self-build packages. It remains a
>>> fact that self-built packages are there to customize and overwrite
>>> upstream packages, independent of how the kernel will look like. That's
>>> one key purpose of Isar.
>>>
>>
>> I'm not against package overwriting. I mean that there is *not the only
>> one way* to do this. Apt provides way to overwrite packages via versions.
>>
>> How I see the creation of robust derivatives:
>>
>> 1. Create local copy for upstream apt (let's name it base-apt) with
>> packages you are going to use for your build.
>> 2. Choose package you want to modify, let's say libA.
>> 3. Create recipe for it and update version to be ">" than original one.
>> 4. Work with this environment.
> 
> This doesn't solve intentional downgrades, like the kernel example would
> have been one.
> 
>>
>> This approach at least somehow guarantees that you will have
>> reproducible build. The packages in your base-apt are built according to
>> the libA version you've customized. So my opinion: user could overwrite
>> only package in local mirrors, i.e. mirrors that could not be updated by
>> 3rd parties.
>>
>>
>> What could happen bad with pinning over upstream (libA is from upstream):
>>
>> 1. libA.h v1.0 had:
>>
>> #define BUFFER_SIZE 32
>>
>> 2. libA.h v1.1 has:
>>
>> #define BUFFER_SIZE 16
>>
>> 3. Upstream contains packages that were already rebuilt according to the
>> libA v1.1
>>
>> 4. You are still using libA v1.0 derivative.
>>
>> 5. Buffer overrun could occur now in any libA consumer from the upstream.
>>
>> So, there is *no* robust way to overwrite package from live upstream,
>> you are always in the risk.
> 
> Of course, if you overwrite something in an incompatible way, you have
> to ensure updating all dependencies as well. That is the mistake done in
> your example.
> 
>>
>> I could imaging that some business units may require this apt
>> prioritization for their specific product's needs, so lets have this
>> feature in Isar. But I think it shouldn't be enabled by default. Also
>> the user documentation for this feature should describe all the risks
>> that user takes if this option is enabled.
> 
> Let's see how flexible the repo and package pinning configuration will
> become with Claudius approach, and then we may switch to an explicit per
> package enabling as well. More intuitive is the default I propose.
> 
>>
>>> If I manage to model the kernel differently, I will probably be able to
>>> drop this patch from /this/ series. But the topic will only come up
>>> again when Claudius proposes his multistrap rework which will include
>>> shipping aptpreferences as well - and prioritizing our own repo properly.
>>>
>>
>> As I said, I could stay with this now to move forward. So the things
>> discussed above could be fixed later.
> 
> User-visible changes should not be lightly merged. So this discussion is
> valuable when it helps to come up with consistent and sustainable
> interfaces. So far, the user was not yet enough in our focus, and that
> is hopefully changing now.
> 

What are the next steps with this series?
Anyway I want at least patches 1 and 3 from it.

Alex