Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-12 17:31, Alexander Smirnov wrote:
> 
> 
> On 02/12/2018 04:45 PM, Jan Kiszka wrote:
>> On 2018-02-12 14:09, Alexander Smirnov wrote:
>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote:
>>>> On 2018-02-12 11:11, Jan Kiszka wrote:
>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote:
>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote:
>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote:
>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote:
>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>
>>>>>>>>> This ensures that we can override packages from upstream Debian or
>>>>>>>>> other
>>>>>>>>> external sources with our self-built versions. We achieve this
>>>>>>>>> for now
>>>>>>>>> by asking multistrap to drop a preferences file for the
>>>>>>>>> buildchroot so
>>>>>>>>> that dependency installations use the right priority. For the
>>>>>>>>> image
>>>>>>>>> build, this does not work because all packages are pull during the
>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure
>>>>>>>>> that
>>>>>>>>> our repo gets the higher priority.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not sure that I completely understand this. What is the
>>>>>>>> use-case for
>>>>>>>> this? If you build your own copy of upstream package, this
>>>>>>>> prioritization could be resolved by specifying suffix to version.
>>>>>>>
>>>>>>> Simple example: custom kernel build of linux-cip
>>>>>>>
>>>>>>> This generates a package called linux-image-amd64 (for amd64 as
>>>>>>> target),
>>>>>>> and that may have a version < debian's kernel. Now, if we do not
>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken,
>>>>>>> even
>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break
>>>>>>> when
>>>>>>> that virtual package linux-cip is selected but can't be installed
>>>>>>> due to
>>>>>>> the version preference "newest").
>>>>>>
>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having
>>>>>> the same name as with original Debian one could be confusing.
>>>>>>
>>>>>
>>>>> That is mandatory, in order to write packages that depend on THE
>>>>> kernel,
>>>>> instead of "my-custom-kernel-in-version-4.4.112".
>>>>
>>>> To make it clearer, just look at the example-module case: The recipe
>>>> can
>>>> build both against the distro as well as all custom kernels, without
>>>> touching it. And that is a pattern not limited to kernel<->module
>>>> relationships.
>>>
>>> kernel<->module pair are built from the same source tree, so here no
>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed
>>> pure Debian packages, nobody depends from 'linux-image-*', so no
>>> correlation with upstream packages.
>>
>> No, they can be independent, just like in the example I submitted. It's
>> an external kernel module vs. some kernel. Please re-read.
>>
>> For Debian, this specific problem does not exist because Debian only
>> ships in-tree modules or those that are built on the target on
>> installation. But you will find the exact same pattern when browsing
>> external projects that come with modules.
> 
> Ok, I see that you define KERNEL_ARCH variable in machine config and
> this variable is used to bind kernel<->module build process. Sometimes
> KERNEL_ARCH has to be changed according to your kernel configuration.
> It's not enough to change preferred_provider from "linux-distro" to
> "linux-cip" if you want to use LPAE for your ARM kernel. You also need
> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct.

OK, but this cannot be defined by the kernel recipe. It has to be
defined by the machine setting. A kernel recipe could then select a
config that matches or reject the KERNEL_ARCH. Currently, we do nothing
in this direction, that is a valid point for future improvements.

> 
> How you see the overall procedure to switch to custom kernel? Why

PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel"

See also
https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf

> KERNEL_ARCH can't be "cip-armmp" in this case?

Because the KERNEL_ARCH remains amd64, armmp, or whatever you
configured. Changing the kernel source doesn't change that in any way.

> 
> For example armbian uses branch names and machines to distinguish images:
> 
> https://www.armbian.com/kernel/
> 
>  - linux-image-dev-sun8i
>  - linux-image-next-odroidc2
> 
> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get
> conflicts with the upstream. Also this doesn't affect modules building
> at all.

This is about conflict: We are *replacing* the distro kernel. But when
doing this, we do not want to change all its consumers. Armbian likely
never ran into the use case of having to design such a full replacement,
also regarding dependencies (I do not see them shipping any out-of-tree
module packages).

And the same is true for all the non-kernel cases we will face when
replacing library-like packages. Please do not focus all your thoughts
on the kernel. This issue is absolutely generic.

> 
>>
>>>
>>> What are the other use-cases for this pattern? As I mentioned above with
>>> 'glibc' example, this is unclean approach, distro could become broken at
>>> anytime. This could become robust *only* after build reproducibility
>>> integration, when you cache the upstream content. But with build-rep you
>>> could manually manage the content of the cache, so prioritization is not
>>> needed, you could just drop duplications from the cache.
>>
>> Every lib-dependency is another pattern for the special kernel/module
>> case: Every distro package that come with a dependencies that you want
>> to replace with a custom build AND you have good reason to downgrade
>> that package, then you want your own packages to have higher prior.
>> That's why this hard-coding is the right thing to do, in general: If you
>> are building a package that competes with upstream, there must be a
>> reason...
>>
>>>
>>> In general having packages with the same name in 'source.list' and
>>> overriding version policies in apt looks very dangerous for me.
>>>
>>> If the current series requires prioritization for kernel only, it would
>>> be nice to specify only this package in apt preferences.
>>
>> See above, the kernel is just the first case. There will be more, I'm
>> absolutely sure.
> 
> I've asked especially about these cases. :-( At the moment there is only
> kernel case, which could be solved without this prioritization.

Use your imagination: enforced replacement of ANY package we rebuild,
but with a lower version. We rebuild packages for a reason. That implies
we want them to be actually used. And that means our packages require a
higher priority.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux