Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Jan Kiszka <jan.kiszka@siemens.com>]

On 28.11.22 07:41, Uladzimir Bely wrote:
> In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
>> On 28.11.22 07:05, Uladzimir Bely wrote:
>>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka 
> wrote:
>>>> On 26.11.22 06:30, Uladzimir Bely wrote:
>>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
>>>>>>> Kiszka
>>>>>
>>>>> wrote:
>>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
>>>>>>>>> Kiszka
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>>>>>> and target distros to the different download subdirs.
>>>>>>>>>>>
>>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>>>>>>>> missing packages in base-apt.
>>>>>>>>>>>
>>>>>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>>>>>
>>>>>>>>>>> Fix the issue by splitting base-apt by introducing
>>>>>>>>>>> BASE_HOST_DISTRO
>>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
>>>>>>>>>>> buildchroot-host,
>>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>>  meta-isar/conf/distro/raspbian-stretch.conf   |  2 +
>>>>>>>>>>>  meta-isar/conf/distro/raspios-bullseye.conf   |  2 +
>>>>>>>>>>>  meta-isar/conf/distro/ubuntu-focal.conf       |  2 +
>>>>>>>>>>>  meta/conf/distro/debian-common.conf           |  2 +
>>>>>>>>>>>  .../isar-bootstrap/isar-bootstrap.inc         |  9 +++-
>>>>>>>>>>>  meta/recipes-devtools/base-apt/base-apt.bb    | 47
>>>>>>>>>>>  ++++++++++---------
>>>>>>>>>>>  6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
>>>>>>>>>>> c8f523b3..66d965c9
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>
>>>>>>>>>>>  BASE_DISTRO = "raspbian"
>>>>>>>>>>>  BASE_DISTRO_CODENAME = "stretch"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>>>>>
>>>>>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>>>>>
>>>>>>>>>> HOST_BASE_DISTRO
>>>>>>>>>
>>>>>>>>> Not a problem, if you think it would be better.
>>>>>>>>>
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>>  HOST_DISTRO ?= "debian-stretch"
>>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
>>>>>>>>>>> 5e4a09c9..7bd81dd6
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>>>>>
>>>>>>>>>>>  BASE_DISTRO = "raspios"
>>>>>>>>>>>  BASE_DISTRO_arm64 = "debian"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>>  BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>>>>>  HOST_DISTRO ?= "debian-bullseye"
>>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>>>>>
>>>>>>>>>>>  BASE_DISTRO = "ubuntu"
>>>>>>>>>>>  BASE_DISTRO_CODENAME = "focal"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>>  DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>>>>>  HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>>>>>>>>  conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
>>>>>>>>>>> 100644
>>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>>>>>
>>>>>>>>>>>  BASE_DISTRO = "debian"
>>>>>>>>>>>
>>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>>>>>> +
>>>>>>>>>>>
>>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>>>>>  DISTRO_APT_SOURCES ?=
>>>>>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>>>>>> ",gnupg"
>>>>>>>>>>>
>>>>>>>>>>>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>>>>>  "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>>>>>  d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO
>>>>>>>>>>>  =
>>>>>>>>>>>  "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
>>>>>>>>>>>  '1'
>>>>>>>>>>>  else
>>>>>>>>>>>  'DISTRO')}">
>>>>>>>>>>>
>>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>>>>>
>>>>>>>>>>>  FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>>>>>  
>>>>>>>>>>>  inherit deb-dl-dir
>>>>>>>>>>>
>>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
>>>>>
>>>>> aptsources_entry_list):
>>>>>>>>>>>      import re
>>>>>>>>>>>
>>>>>>>>>>>      if 
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>>>>>> -        premirrors = "\S*
>>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>>>>>> +        premirrors = "\S*
>>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>>>>>
>>>>>>>>>>>      else:
>>>>>>>>>>>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
>>>>>>>>>>>          ""
>>>>>>>>>>>      
>>>>>>>>>>>      mirror_list = [entry.split()
>>>>>>>>>>>
>>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>>>>>
>>>>>>>>>>>                           "${ROOTFSDIR}/etc/apt/preferences.d/boot
>>>>>>>>>>>                           st
>>>>>>>>>>>                           ra
>>>>>>>>>>>                           p"
>>>>>>>>>>>          
>>>>>>>>>>>          mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>>>>>          if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>>>>>
>>>>>>>>>>> -            line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>> main" +            line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>>>>>
>>>>>>>>>>>              if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>>>>>              
>>>>>>>>>>>                  line="[trusted=yes] ${line}"
>>>>>>>>>>>              
>>>>>>>>>>>              fi
>>>>>>>>>>>              echo "deb ${line}" >
>>>>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>>>>>
>>>>>>>>>>> +            line="file:///base-apt/${BASE_DISTRO}
>>>>>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>>>>>> main"
>>>>>>>>>>
>>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
>>>>>>>>>> of
>>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>>>>>
>>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>>>>>> "deb-src"
>>>>>>>>> line while it should point to BASE_DISTRO even for host case.
>>>>>>>>> Without
>>>>>>>>> it,
>>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
>>>>>>>>> cached
>>>>>>>>> sources from "host" repo, while sources were cached in "target" one.
>>>>>>>>>
>>>>>>>>> That's how it looks now:
>>>>>>>>> | $ cat
>>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>>>>>
>>>>>>>>> apt.list
>>>>>>>>>
>>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>> | $ cat
>>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.
>>>>>>>>> | li
>>>>>>>>> | st
>>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>>>>>
>>>>>>>> Indeed, I misintepreted the code.
>>>>>>>>
>>>>>>>> But I think we still have an issue on the deb-src population side:
>>>>>>>>
>>>>>>>> # ls build/downloads/deb-src/
>>>>>>>> iot2050-debian
>>>>>>>>
>>>>>>>> And that was with this patch applied.
>>>>>>>>
>>>>>>>> Jan
>>>>>>>
>>>>>>> Actually, I checked the case with the following configuration:
>>>>>>>
>>>>>>> distro: my-debian
>>>>>>>
>>>>>>> local_conf_header:
>>>>>>>   nodebian: |
>>>>>>>   
>>>>>>>     require conf/distro/debian-bullseye.conf
>>>>>>>     DISTRO_NAME = "My Debian System"
>>>>>>>
>>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>>>>>
>>>>>>> Cached build was OK, "hello" package (that we build from sources) was
>>>>>>>
>>>>>>> downloaded at first build to "my-debian" directory:
>>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>>>>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc  hello_2.10.orig.tar.gz
>>>>>>>
>>>>>>> and second build picked the files with no problem.
>>>>>>>
>>>>>>> The same with "raspios-bullseye" target - "hello" sources are
>>>>>>> downloaded
>>>>>>> to
>>>>>>>
>>>>>>> the "target" dir:
>>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>>>>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc  hello_2.10.orig.tar.gz
>>>>>>>
>>>>>>> I even tested the case with  BASE_REPO_FEATURES += "cache-deb-src" and
>>>>>>> it
>>>>>>> was OK (honestly, I don't know who could require cached build with
>>>>>>> this
>>>>>>> option enabled :) ).
>>>>>>>
>>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
>>>>>>> (despite some packages in it are currently not compatible with
>>>>>>> BB_NO_NETWORK option)
>>>>>>
>>>>>> Please then test without the coral packages, they have an own issue
>>>>>> /wrt
>>>>>> offline builds.
>>>>>
>>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
>>>>
>>>> ...but it caches to the wrong downloads/deb-src path, no?
>>>>
>>>> Jan
>>>
>>> Why wrong?
>>>
>>> | meta-iot2050/build $ tree -d downloads/deb*
>>> | downloads/deb
>>> | ├── debian-bullseye
>>> | └── iot2050-debian
>>> | downloads/deb-src
>>> | └── iot2050-debian
>>> | 
>>> |     ├── openssl
>>> |     └── swig
>>>
>>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
>>> corresponding directory.
>>
>> deb-src fetching still lacks base-distro normalization, like deb also has:
>>
> 
> Saying "normalization", do you mean the following: if ${DISTRO} is just 
> renaming (e.g. "iot2050-debian") everything should be downloaded to and picked 
> up from the directory named "debian-bullseye"?
> 

Yes, just like you did for binary debs: Reduce the DISTRO to its
BASE_DISTRO, then use that as key for the cache. The current situation
is inconsistent and non-optimal, even if working.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux