From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520199916203016192 X-Received: by 10.28.216.16 with SMTP id p16mr623008wmg.0.1518461474762; Mon, 12 Feb 2018 10:51:14 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.47.194 with SMTP id v185ls1206570wmv.4.canary-gmail; Mon, 12 Feb 2018 10:51:14 -0800 (PST) X-Google-Smtp-Source: AH8x226ZwsMT8wSnRre1izCzytELsgtOYfBYKhX04f+6S324AqhwXPamTiDlxzS/TRaxlbzvZdEG X-Received: by 10.28.87.5 with SMTP id l5mr609556wmb.23.1518461474206; Mon, 12 Feb 2018 10:51:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518461474; cv=none; d=google.com; s=arc-20160816; b=PRxdwNYzo0Iba2TUhwmZKfP4GkWvn986WoeaStwapC7Y8/OfIeTawdiZghJq9UnM/9 qBB03t4llnLhEWbr1enGal9yx+edOwGcR6ylUaGcrtN7zkR7VTS5Pc6MlreDYe83QA/N iG5Psq5foVQCUEaQO3jdO32PDv0W0XhgMkUlHoIai9Vhj6lwF6OoCpkt6clB1SjUxNGG S5fT8+peCj365HL3zr0BiARNCKakMIAaQvvYwGgfeIY1BOntQjlZ/lCq7cqm3cNQmmvM xs/nhMbuN7Ly/woZBi5iAhhhbMqqMs91TBL6ydJ6uQ5nAKyrVBk2wmRMQ4z4kmKCM75R wrJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=hsYDAo/YZLXDO6+vj+C8xTq1rHPrQMzc57s7MtcLyG4=; b=igdROnHJnzMTmcPhVCettZ2Z4Rq0kh8zRwzfR4ho5Lx/4UWQBlp29AazugoQIbaE6x wosAP6zuoJ/2FUk1dtmR7XhmxyL0ozppKdttMspvo2x5JUiVo5tJyJaArSBT4Tfzw1O/ J6S0P5Bac/Y4z/+m5AdsedsZXDpjcIRXvzPdTKEytLw/MrFuUBiMLk4KzXSbrSGAxxB4 TpeWfJOSKkVyKCAdzyDXicM0tTZ8F9bbLHUSvCRyQPYll0OJV+sQTwNjecivOVVw9YE0 zbLiiOy67lLGeVgTuij8LbahWk4srxmTHrMI0vcMBbD+cCzaAnsTQ9fkNz+PXPM95ZPi RDQQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id v8si480574wrg.2.2018.02.12.10.51.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Feb 2018 10:51:14 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w1CIpAfD002678 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 12 Feb 2018 19:51:12 +0100 Subject: Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others To: Jan Kiszka , isar-users References: <136df397-5fce-9f50-fd74-f35f038fbda2@siemens.com> <905dabd5-e9b9-9382-8c86-8e33d8559b21@siemens.com> <65f07e31-b2d7-4858-440e-f32f1a9b0832@siemens.com> <0d9609ec-cf09-a72e-d7b7-2b058cf00089@siemens.com> From: Alexander Smirnov Message-ID: Date: Mon, 12 Feb 2018 21:51:05 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <0d9609ec-cf09-a72e-d7b7-2b058cf00089@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: GPR2x0jquy0N On 02/12/2018 08:00 PM, Jan Kiszka wrote: > On 2018-02-12 17:31, Alexander Smirnov wrote: >> >> >> On 02/12/2018 04:45 PM, Jan Kiszka wrote: >>> On 2018-02-12 14:09, Alexander Smirnov wrote: >>>> On 02/12/2018 01:27 PM, Jan Kiszka wrote: >>>>> On 2018-02-12 11:11, Jan Kiszka wrote: >>>>>> On 2018-02-12 11:08, Alexander Smirnov wrote: >>>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote: >>>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote: >>>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote: >>>>>>>>>> From: Jan Kiszka >>>>>>>>>> >>>>>>>>>> This ensures that we can override packages from upstream Debian or >>>>>>>>>> other >>>>>>>>>> external sources with our self-built versions. We achieve this >>>>>>>>>> for now >>>>>>>>>> by asking multistrap to drop a preferences file for the >>>>>>>>>> buildchroot so >>>>>>>>>> that dependency installations use the right priority. For the >>>>>>>>>> image >>>>>>>>>> build, this does not work because all packages are pull during the >>>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure >>>>>>>>>> that >>>>>>>>>> our repo gets the higher priority. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'm not sure that I completely understand this. What is the >>>>>>>>> use-case for >>>>>>>>> this? If you build your own copy of upstream package, this >>>>>>>>> prioritization could be resolved by specifying suffix to version. >>>>>>>> >>>>>>>> Simple example: custom kernel build of linux-cip >>>>>>>> >>>>>>>> This generates a package called linux-image-amd64 (for amd64 as >>>>>>>> target), >>>>>>>> and that may have a version < debian's kernel. Now, if we do not >>>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken, >>>>>>>> even >>>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break >>>>>>>> when >>>>>>>> that virtual package linux-cip is selected but can't be installed >>>>>>>> due to >>>>>>>> the version preference "newest"). >>>>>>> >>>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having >>>>>>> the same name as with original Debian one could be confusing. >>>>>>> >>>>>> >>>>>> That is mandatory, in order to write packages that depend on THE >>>>>> kernel, >>>>>> instead of "my-custom-kernel-in-version-4.4.112". >>>>> >>>>> To make it clearer, just look at the example-module case: The recipe >>>>> can >>>>> build both against the distro as well as all custom kernels, without >>>>> touching it. And that is a pattern not limited to kernel<->module >>>>> relationships. >>>> >>>> kernel<->module pair are built from the same source tree, so here no >>>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed >>>> pure Debian packages, nobody depends from 'linux-image-*', so no >>>> correlation with upstream packages. >>> >>> No, they can be independent, just like in the example I submitted. It's >>> an external kernel module vs. some kernel. Please re-read. >>> >>> For Debian, this specific problem does not exist because Debian only >>> ships in-tree modules or those that are built on the target on >>> installation. But you will find the exact same pattern when browsing >>> external projects that come with modules. >> >> Ok, I see that you define KERNEL_ARCH variable in machine config and >> this variable is used to bind kernel<->module build process. Sometimes >> KERNEL_ARCH has to be changed according to your kernel configuration. >> It's not enough to change preferred_provider from "linux-distro" to >> "linux-cip" if you want to use LPAE for your ARM kernel. You also need >> to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct. > > OK, but this cannot be defined by the kernel recipe. It has to be > defined by the machine setting. A kernel recipe could then select a > config that matches or reject the KERNEL_ARCH. Currently, we do nothing > in this direction, that is a valid point for future improvements. > >> >> How you see the overall procedure to switch to custom kernel? Why > > PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel" > > See also > https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf > >> KERNEL_ARCH can't be "cip-armmp" in this case? > > Because the KERNEL_ARCH remains amd64, armmp, or whatever you > configured. Changing the kernel source doesn't change that in any way. It doesn't remain for all the cases. ARM debian provides two kernel images: - linux-image-armmp - linux-image-armmp-lpae Current Isar doesn't handle this, but if you build your kernel with LPAE, I assume you should change KERNEL_ARCH respectively, otherwise there will be some inaccuracy. So the same machine could have several KERNEL_ARCHs. Then, what should I write in KERNEL_ARCH if I add new machine, let's say Cubieboard? In general, KERNEL_ARCH is not exactly what it's used here for. According to the: https://kernel-handbook.alioth.debian.org/ch-packaging.html the linux image name looks like the following: - linux-image[-featureset]-flavour Also chapter 3.3 stays: 8<-- Each architecture usually has multiple flavours of the binary kernel images. Different flavours correspond to different kernel configuration files, used to build the binary images from the same kernel tree. 8<-- So having KERNEL_ARCH as the constant is deviation from Debian flow. In addition: 8<-- Again, multiple flavours of binary images may be built from the featureset tree. For example, the i386 architecture has a number of different flavours, such as 686 and 686-pae, built from the common Debian kernel source. It also contains the rt featureset. The source tree for building the kernels for each of these featuresets is obtained by applying additional patches to the Debian kernel source. It may be used to build the rt-686-pae binary image flavours. 8<-- So in my opinion, CIP is a feature set. I don't really think, that replacing upstream Debian kernel package by custom one and claiming that is the same will make Debian community happy... Ok, I see that we have different vision on this topic, I could stay with this implementation for now, but in general in my opinion this looks like a big hack and should be reworked. > >> >> For example armbian uses branch names and machines to distinguish images: >> >> https://www.armbian.com/kernel/ >> >>  - linux-image-dev-sun8i >>  - linux-image-next-odroidc2 >> >> So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get >> conflicts with the upstream. Also this doesn't affect modules building >> at all. > > This is about conflict: We are *replacing* the distro kernel. But when > doing this, we do not want to change all its consumers. Armbian likely > never ran into the use case of having to design such a full replacement, > also regarding dependencies (I do not see them shipping any out-of-tree > module packages). > > And the same is true for all the non-kernel cases we will face when > replacing library-like packages. Please do not focus all your thoughts > on the kernel. This issue is absolutely generic. > >> >>> >>>> >>>> What are the other use-cases for this pattern? As I mentioned above with >>>> 'glibc' example, this is unclean approach, distro could become broken at >>>> anytime. This could become robust *only* after build reproducibility >>>> integration, when you cache the upstream content. But with build-rep you >>>> could manually manage the content of the cache, so prioritization is not >>>> needed, you could just drop duplications from the cache. >>> >>> Every lib-dependency is another pattern for the special kernel/module >>> case: Every distro package that come with a dependencies that you want >>> to replace with a custom build AND you have good reason to downgrade >>> that package, then you want your own packages to have higher prior. >>> That's why this hard-coding is the right thing to do, in general: If you >>> are building a package that competes with upstream, there must be a >>> reason... >>> >>>> >>>> In general having packages with the same name in 'source.list' and >>>> overriding version policies in apt looks very dangerous for me. >>>> >>>> If the current series requires prioritization for kernel only, it would >>>> be nice to specify only this package in apt preferences. >>> >>> See above, the kernel is just the first case. There will be more, I'm >>> absolutely sure. >> >> I've asked especially about these cases. :-( At the moment there is only >> kernel case, which could be solved without this prioritization. > > Use your imagination: enforced replacement of ANY package we rebuild, > but with a lower version. We rebuild packages for a reason. That implies > we want them to be actually used. And that means our packages require a > higher priority. > My imagination told me how it will be difficult to debug segfaults on the target, because I pinned old libA in Isar-apt, while upstream dependents already have been compiled according the new version of libA. It wouldn't be so funny if nobody had faced with such problems in the past... Alex