From: Jan Kiszka <jan.kiszka@siemens.com>
To: Uladzimir Bely <ubely@ilbers.de>, isar-users@googlegroups.com
Subject: Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs
Date: Mon, 28 Nov 2022 12:45:40 +0100 [thread overview]
Message-ID: <e9c365ef-0638-c43c-cf6f-49f292aa8c77@siemens.com> (raw)
In-Reply-To: <9441047.eNJFYEL58v@home>
On 28.11.22 12:12, Uladzimir Bely wrote:
> In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote:
>> On 28.11.22 07:41, Uladzimir Bely wrote:
>>> In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
>>>> On 28.11.22 07:05, Uladzimir Bely wrote:
>>>>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 26.11.22 06:30, Uladzimir Bely wrote:
>>>>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan
>>>>>>> Kiszka
>>>>>
>>>>> wrote:
>>>>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>>>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
>>>>>>>>> Kiszka
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
>>>>>>>>>>> Kiszka
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>>>>>>>> and target distros to the different download subdirs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due
>>>>>>>>>>>>> to
>>>>>>>>>>>>> missing packages in base-apt.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix the issue by splitting base-apt by introducing
>>>>>>>>>>>>> BASE_HOST_DISTRO
>>>>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
>>>>>>>>>>>>> buildchroot-host,
>>>>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>
>>>>>>>>>>>>> meta-isar/conf/distro/raspbian-stretch.conf | 2 +
>>>>>>>>>>>>> meta-isar/conf/distro/raspios-bullseye.conf | 2 +
>>>>>>>>>>>>> meta-isar/conf/distro/ubuntu-focal.conf | 2 +
>>>>>>>>>>>>> meta/conf/distro/debian-common.conf | 2 +
>>>>>>>>>>>>> .../isar-bootstrap/isar-bootstrap.inc | 9 +++-
>>>>>>>>>>>>> meta/recipes-devtools/base-apt/base-apt.bb | 47
>>>>>>>>>>>>> ++++++++++---------
>>>>>>>>>>>>> 6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
>>>>>>>>>>>>> c8f523b3..66d965c9
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "raspbian"
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "stretch"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>>>>>>>
>>>>>>>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>>>>>>>
>>>>>>>>>>>> HOST_BASE_DISTRO
>>>>>>>>>>>
>>>>>>>>>>> Not a problem, if you think it would be better.
>>>>>>>>>>>
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> HOST_DISTRO ?= "debian-stretch"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
>>>>>>>>>>>>> 5e4a09c9..7bd81dd6
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "raspios"
>>>>>>>>>>>>> BASE_DISTRO_arm64 = "debian"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>>>>>>> HOST_DISTRO ?= "debian-bullseye"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index
>>>>>>>>>>>>> 379b5b30..75da79a1
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "ubuntu"
>>>>>>>>>>>>> BASE_DISTRO_CODENAME = "focal"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>>>> "conf/distro/${HOST_DISTRO}.list
>>>>>>>>>>>>> conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
>>>>>>>>>>>>> 100644
>>>>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>> BASE_DISTRO = "debian"
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>>>>>>>> +
>>>>>>>>>>>>>
>>>>>>>>>>>>> HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>>> DISTRO_APT_SOURCES ?=
>>>>>>>>>>>>> "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>>>>>>>> ",gnupg"
>>>>>>>>>>>>>
>>>>>>>>>>>>> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>>>>>>> "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
>>>>>>>>>>>>> BOOTSTRAP_DISTRO
>>>>>>>>>>>>> =
>>>>>>>>>>>>> "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
>>>>>>>>>>>>> '1'
>>>>>>>>>>>>> else
>>>>>>>>>>>>> 'DISTRO')}">
>>>>>>>>>>>>>
>>>>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>>>>>>>
>>>>>>>>>>>>> FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>>>>>>>
>>>>>>>>>>>>> inherit deb-dl-dir
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
>>>>>>>
>>>>>>> aptsources_entry_list):
>>>>>>>>>>>>> import re
>>>>>>>>>>>>>
>>>>>>>>>>>>> if
>>>
>>> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>>>>>>>> - premirrors = "\S*
>>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>>>>>>>> + premirrors = "\S*
>>>>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>>>>>>>
>>>>>>>>>>>>> else:
>>>>>>>>>>>>> premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
>>>>>>>>>>>>> ""
>>>>>>>>>>>>>
>>>>>>>>>>>>> mirror_list = [entry.split()
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/preferences.d/bo
>>>>>>>>>>>>> ot
>>>>>>>>>>>>> st
>>>>>>>>>>>>> ra
>>>>>>>>>>>>> p"
>>>>>>>>>>>>>
>>>>>>>>>>>>> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>>>>>>> if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>>>>>>>
>>>>>>>>>>>>> - line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>>>> main" +
>>>>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>>>>>>>
>>>>>>>>>>>>> if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>>>>>>>
>>>>>>>>>>>>> line="[trusted=yes] ${line}"
>>>>>>>>>>>>>
>>>>>>>>>>>>> fi
>>>>>>>>>>>>> echo "deb ${line}" >
>>>>>>>>>>>>> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>>>>>>>
>>>>>>>>>>>>> + line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>>>> main"
>>>>>>>>>>>>
>>>>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
>>>>>>>>>>>> of
>>>>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>>>>>>>
>>>>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>>>>>>>> "deb-src"
>>>>>>>>>>> line while it should point to BASE_DISTRO even for host case.
>>>>>>>>>>> Without
>>>>>>>>>>> it,
>>>>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
>>>>>>>>>>> cached
>>>>>>>>>>> sources from "host" repo, while sources were cached in "target"
>>>>>>>>>>> one.
>>>>>>>>>>>
>>>>>>>>>>> That's how it looks now:
>>>>>>>>>>> | $ cat
>>>>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>>>>>>>
>>>>>>>>>>> apt.list
>>>>>>>>>>>
>>>>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>> | $ cat
>>>>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap
>>>>>>>>>>> | t.
>>>>>>>>>>> | li
>>>>>>>>>>> | st
>>>>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>>>
>>>>>>>>>> Indeed, I misintepreted the code.
>>>>>>>>>>
>>>>>>>>>> But I think we still have an issue on the deb-src population side:
>>>>>>>>>>
>>>>>>>>>> # ls build/downloads/deb-src/
>>>>>>>>>> iot2050-debian
>>>>>>>>>>
>>>>>>>>>> And that was with this patch applied.
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>
>>>>>>>>> Actually, I checked the case with the following configuration:
>>>>>>>>>
>>>>>>>>> distro: my-debian
>>>>>>>>>
>>>>>>>>> local_conf_header:
>>>>>>>>> nodebian: |
>>>>>>>>>
>>>>>>>>> require conf/distro/debian-bullseye.conf
>>>>>>>>> DISTRO_NAME = "My Debian System"
>>>>>>>>>
>>>>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>>>>>>>
>>>>>>>>> Cached build was OK, "hello" package (that we build from sources)
>>>>>>>>> was
>>>>>>>>>
>>>>>>>>> downloaded at first build to "my-debian" directory:
>>>>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
>>>>>>>>> | hello_2.10.orig.tar.gz
>>>>>>>>>
>>>>>>>>> and second build picked the files with no problem.
>>>>>>>>>
>>>>>>>>> The same with "raspios-bullseye" target - "hello" sources are
>>>>>>>>> downloaded
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> the "target" dir:
>>>>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>>>>>>>> | hello_2.10-2.debian.tar.xz hello_2.10-2.dsc
>>>>>>>>> | hello_2.10.orig.tar.gz
>>>>>>>>>
>>>>>>>>> I even tested the case with BASE_REPO_FEATURES += "cache-deb-src"
>>>>>>>>> and
>>>>>>>>> it
>>>>>>>>> was OK (honestly, I don't know who could require cached build with
>>>>>>>>> this
>>>>>>>>> option enabled :) ).
>>>>>>>>>
>>>>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer
>>>>>>>>> also
>>>>>>>>> (despite some packages in it are currently not compatible with
>>>>>>>>> BB_NO_NETWORK option)
>>>>>>>>
>>>>>>>> Please then test without the coral packages, they have an own issue
>>>>>>>> /wrt
>>>>>>>> offline builds.
>>>>>>>
>>>>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>>>>>>
>>>>>> ...but it caches to the wrong downloads/deb-src path, no?
>>>>>>
>>>>>> Jan
>>>>>
>>>>> Why wrong?
>>>>>
>>>>> | meta-iot2050/build $ tree -d downloads/deb*
>>>>> | downloads/deb
>>>>> | ├── debian-bullseye
>>>>> | └── iot2050-debian
>>>>> | downloads/deb-src
>>>>> | └── iot2050-debian
>>>>> |
>>>>> | ├── openssl
>>>>> | └── swig
>>>>>
>>>>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
>>>>> corresponding directory.
>>>>
>>>> deb-src fetching still lacks base-distro normalization, like deb also
> has:
>>> Saying "normalization", do you mean the following: if ${DISTRO} is just
>>> renaming (e.g. "iot2050-debian") everything should be downloaded to and
>>> picked up from the directory named "debian-bullseye"?
>>
>> Yes, just like you did for binary debs: Reduce the DISTRO to its
>> BASE_DISTRO, then use that as key for the cache. The current situation
>> is inconsistent and non-optimal, even if working.
>>
>> Jan
>
> I've just pushed v4 of patchset that still doesn't include this wish. For now,
> I would apply it to the isar/next to have cached cross build working.
>
> We have some plans to improve base-apt in general (there were some PoC patches
> on list), so improvements you mention could be done a bit later - for now I
> need to switch to some other tasks.
>
The problem is in meta/classes/rootfs.bbclass. It lacks normalization of
ROOTFS_DISTRO to some "ROOTFS_BASE_DISTRO". That also explains why you
still saw same packages under deb/iot2050-debian - you ran the final
rootfs step which I didn't (due to earlier errors).
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
next prev parent reply other threads:[~2022-11-28 11:45 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-25 6:42 [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs Uladzimir Bely
2022-11-25 7:42 ` Jan Kiszka
2022-11-25 7:56 ` Uladzimir Bely
2022-11-25 14:11 ` Jan Kiszka
2022-11-25 14:48 ` Uladzimir Bely
2022-11-25 15:40 ` Jan Kiszka
2022-11-26 5:30 ` Uladzimir Bely
2022-11-27 19:05 ` Jan Kiszka
2022-11-28 6:05 ` Uladzimir Bely
2022-11-28 6:32 ` Jan Kiszka
2022-11-28 6:41 ` Uladzimir Bely
2022-11-28 6:51 ` Jan Kiszka
2022-11-28 11:12 ` Uladzimir Bely
2022-11-28 11:45 ` Jan Kiszka [this message]
2022-11-25 6:42 ` [PATCH v3 2/3] testsuite: Run signed repro test in cross mode Uladzimir Bely
2022-11-25 6:42 ` [PATCH v3 3/3] testsuite: Replace bananapi with rpi-arm-v7 in signed repro tests Uladzimir Bely
2022-11-25 7:37 ` [PATCH v3 0/3] Fix cross-build from base-apt when custom DISTRO name used Jan Kiszka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e9c365ef-0638-c43c-cf6f-49f292aa8c77@siemens.com \
--to=jan.kiszka@siemens.com \
--cc=isar-users@googlegroups.com \
--cc=ubely@ilbers.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox