From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520199916203016192 X-Received: by 10.28.111.148 with SMTP id c20mr529105wmi.7.1518454852746; Mon, 12 Feb 2018 09:00:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.167.75 with SMTP id q72ls1149371wme.0.gmail; Mon, 12 Feb 2018 09:00:52 -0800 (PST) X-Google-Smtp-Source: AH8x2240pUa5OTS055TaehbCqpFhpJecWuZedQNfj9SrtlcUEHpADmifpQxdH1xfQjVmn0Yrfpn6 X-Received: by 10.28.247.5 with SMTP id v5mr568939wmh.12.1518454852046; Mon, 12 Feb 2018 09:00:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518454852; cv=none; d=google.com; s=arc-20160816; b=EfxpIy3R2GohKTAjDltKIM6DVNvTQ8v1DAee6Izvc+xM36AYWE+DHOpV9orZeECFwq wVeQjhU4gFM2a1kNVmTlnZU2QLCxGfxYB9wOiNtsq3UiAIarX7AsQmuWtFJ6ZFPPH8pu TvcY1ESHCPQp6mtC6kklje053a9RC1lQw5vTxlZeqw4SDRai8wh0FZAe1z4Z/nmC6hv/ 6eYB4bTkjEvoq1G59smWJ7jorD8kcFPsT3tQVlkjVmr7zTP608rS4L/Wev/JSJsf6umX AP9lYBDAM1D4UBdlZeWDOY4t175JxXpXgqeWk2ZRFll38bHSeT9xwErJXtGyx+Ipq0vH 2Dwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=/RhdTNILwoAOH4vN0c1cHJDWjEw1namNfoRwPNHIGOw=; b=bJB9NGebhvbNOZHomYJyR0z/Tf0p77oCliOziWYqR/Khn0hbi/szHeQMKn7FWokrlM XW4GnsjaRznGpGBK17UWg0fWccmJP3IGqp4NOLxAtWmkjtAJyCcProBctWJ83IeD9a2i ebCat4VemWKTvuafoJqtJrjnv90IUdnC/4H1RSxSGDpATgptEEtgtJu7NTli9kr6kjCf aPj7IYviJgyRAthB/K8Pr72fONuMJEUW9rBou8NgNXznu+hB2xJHMRNH9TQoHucmzj39 EGhxfovSmqegK9u9p9QN/UQNIZwET0z3q5GJNS1avCuIeDOfV4cnBw3Ak9n12Pk89/1N gOAA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id s81si378538wmd.2.2018.02.12.09.00.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Feb 2018 09:00:51 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w1CH0pri016095 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Feb 2018 18:00:51 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w1CH0pGN031795; Mon, 12 Feb 2018 18:00:51 +0100 Subject: Re: [PATCH v4 2/8] Prioritize isar-apt repo over all others To: Alexander Smirnov , isar-users References: <136df397-5fce-9f50-fd74-f35f038fbda2@siemens.com> <905dabd5-e9b9-9382-8c86-8e33d8559b21@siemens.com> <65f07e31-b2d7-4858-440e-f32f1a9b0832@siemens.com> From: Jan Kiszka Message-ID: <0d9609ec-cf09-a72e-d7b7-2b058cf00089@siemens.com> Date: Mon, 12 Feb 2018 18:00:50 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: RPtMNtYTCLNM On 2018-02-12 17:31, Alexander Smirnov wrote: > > > On 02/12/2018 04:45 PM, Jan Kiszka wrote: >> On 2018-02-12 14:09, Alexander Smirnov wrote: >>> On 02/12/2018 01:27 PM, Jan Kiszka wrote: >>>> On 2018-02-12 11:11, Jan Kiszka wrote: >>>>> On 2018-02-12 11:08, Alexander Smirnov wrote: >>>>>> On 02/12/2018 12:41 PM, Jan Kiszka wrote: >>>>>>> On 2018-02-12 10:33, Alexander Smirnov wrote: >>>>>>>> On 02/11/2018 06:25 PM, Jan Kiszka wrote: >>>>>>>>> From: Jan Kiszka >>>>>>>>> >>>>>>>>> This ensures that we can override packages from upstream Debian or >>>>>>>>> other >>>>>>>>> external sources with our self-built versions. We achieve this >>>>>>>>> for now >>>>>>>>> by asking multistrap to drop a preferences file for the >>>>>>>>> buildchroot so >>>>>>>>> that dependency installations use the right priority. For the >>>>>>>>> image >>>>>>>>> build, this does not work because all packages are pull during the >>>>>>>>> bootstrap. Therefore, we set aptdefaultrelease to isar to ensure >>>>>>>>> that >>>>>>>>> our repo gets the higher priority. >>>>>>>>> >>>>>>>> >>>>>>>> I'm not sure that I completely understand this. What is the >>>>>>>> use-case for >>>>>>>> this? If you build your own copy of upstream package, this >>>>>>>> prioritization could be resolved by specifying suffix to version. >>>>>>> >>>>>>> Simple example: custom kernel build of linux-cip >>>>>>> >>>>>>> This generates a package called linux-image-amd64 (for amd64 as >>>>>>> target), >>>>>>> and that may have a version < debian's kernel. Now, if we do not >>>>>>> prioritize isar-apt over upstream, the upstream kernel is taken, >>>>>>> even >>>>>>> when you switched PREFERRED_PROVIDER (in fact, the build will break >>>>>>> when >>>>>>> that virtual package linux-cip is selected but can't be installed >>>>>>> due to >>>>>>> the version preference "newest"). >>>>>> >>>>>> Ok, but why you can't name you kernel 'linux-cip-image-amd64'? Having >>>>>> the same name as with original Debian one could be confusing. >>>>>> >>>>> >>>>> That is mandatory, in order to write packages that depend on THE >>>>> kernel, >>>>> instead of "my-custom-kernel-in-version-4.4.112". >>>> >>>> To make it clearer, just look at the example-module case: The recipe >>>> can >>>> build both against the distro as well as all custom kernels, without >>>> touching it. And that is a pattern not limited to kernel<->module >>>> relationships. >>> >>> kernel<->module pair are built from the same source tree, so here no >>> problem with ${PN}-${PV} substitution as dependency. Also I've parsed >>> pure Debian packages, nobody depends from 'linux-image-*', so no >>> correlation with upstream packages. >> >> No, they can be independent, just like in the example I submitted. It's >> an external kernel module vs. some kernel. Please re-read. >> >> For Debian, this specific problem does not exist because Debian only >> ships in-tree modules or those that are built on the target on >> installation. But you will find the exact same pattern when browsing >> external projects that come with modules. > > Ok, I see that you define KERNEL_ARCH variable in machine config and > this variable is used to bind kernel<->module build process. Sometimes > KERNEL_ARCH has to be changed according to your kernel configuration. > It's not enough to change preferred_provider from "linux-distro" to > "linux-cip" if you want to use LPAE for your ARM kernel. You also need > to change KERNEL_ARCH from "armmp" to "armmp-lpae" to be correct. OK, but this cannot be defined by the kernel recipe. It has to be defined by the machine setting. A kernel recipe could then select a config that matches or reject the KERNEL_ARCH. Currently, we do nothing in this direction, that is a valid point for future improvements. > > How you see the overall procedure to switch to custom kernel? Why PREFERRED_PROVIDER_virtual/kernel = "my-nice-kernel" See also https://github.com/siemens/jailhouse-images/blob/master/conf/multiconfig/qemuamd64-jailhouse.conf > KERNEL_ARCH can't be "cip-armmp" in this case? Because the KERNEL_ARCH remains amd64, armmp, or whatever you configured. Changing the kernel source doesn't change that in any way. > > For example armbian uses branch names and machines to distinguish images: > > https://www.armbian.com/kernel/ > >  - linux-image-dev-sun8i >  - linux-image-next-odroidc2 > > So if define KERNEL_ARCH as ${ORIGIN}-${MACHINE}, you never will get > conflicts with the upstream. Also this doesn't affect modules building > at all. This is about conflict: We are *replacing* the distro kernel. But when doing this, we do not want to change all its consumers. Armbian likely never ran into the use case of having to design such a full replacement, also regarding dependencies (I do not see them shipping any out-of-tree module packages). And the same is true for all the non-kernel cases we will face when replacing library-like packages. Please do not focus all your thoughts on the kernel. This issue is absolutely generic. > >> >>> >>> What are the other use-cases for this pattern? As I mentioned above with >>> 'glibc' example, this is unclean approach, distro could become broken at >>> anytime. This could become robust *only* after build reproducibility >>> integration, when you cache the upstream content. But with build-rep you >>> could manually manage the content of the cache, so prioritization is not >>> needed, you could just drop duplications from the cache. >> >> Every lib-dependency is another pattern for the special kernel/module >> case: Every distro package that come with a dependencies that you want >> to replace with a custom build AND you have good reason to downgrade >> that package, then you want your own packages to have higher prior. >> That's why this hard-coding is the right thing to do, in general: If you >> are building a package that competes with upstream, there must be a >> reason... >> >>> >>> In general having packages with the same name in 'source.list' and >>> overriding version policies in apt looks very dangerous for me. >>> >>> If the current series requires prioritization for kernel only, it would >>> be nice to specify only this package in apt preferences. >> >> See above, the kernel is just the first case. There will be more, I'm >> absolutely sure. > > I've asked especially about these cases. :-( At the moment there is only > kernel case, which could be solved without this prioritization. Use your imagination: enforced replacement of ANY package we rebuild, but with a lower version. We rebuild packages for a reason. That implies we want them to be actually used. And that means our packages require a higher priority. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux