Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Jan Kiszka <jan.kiszka@siemens.com>]

On 26.11.22 06:30, Uladzimir Bely wrote:
> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan Kiszka wrote:
>> On 25.11.22 15:48, Uladzimir Bely wrote:
>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan Kiszka 
> wrote:
>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan Kiszka
>>>
>>> wrote:
>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
>>>>>>> Downstreams may want to change DISTRO to some custom value.
>>>>>>> When cross-building, this leads to downloading packages for host
>>>>>>> and target distros to the different download subdirs.
>>>>>>>
>>>>>>> While base-apt is populated only from DISTRO download subdir,
>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>>>>>> missing packages in base-apt.
>>>>>>>
>>>>>>> The similar issue happens with cross-building raspberry targets
>>>>>>> since they use their own mirrors and DISTRO.
>>>>>>>
>>>>>>> Fix the issue by splitting base-apt by introducing BASE_HOST_DISTRO
>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
>>>>>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>>>>>
>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>>>>>> ---
>>>>>>>
>>>>>>>  meta-isar/conf/distro/raspbian-stretch.conf   |  2 +
>>>>>>>  meta-isar/conf/distro/raspios-bullseye.conf   |  2 +
>>>>>>>  meta-isar/conf/distro/ubuntu-focal.conf       |  2 +
>>>>>>>  meta/conf/distro/debian-common.conf           |  2 +
>>>>>>>  .../isar-bootstrap/isar-bootstrap.inc         |  9 +++-
>>>>>>>  meta/recipes-devtools/base-apt/base-apt.bb    | 47
>>>>>>>  ++++++++++---------
>>>>>>>  6 files changed, 40 insertions(+), 24 deletions(-)
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..66d965c9
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>
>>>>>>>  BASE_DISTRO = "raspbian"
>>>>>>>  BASE_DISTRO_CODENAME = "stretch"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO = "debian"
>>>>>>
>>>>>> Better prefix it with HOST, to be consistent:
>>>>>>
>>>>>> HOST_BASE_DISTRO
>>>>>
>>>>> Not a problem, if you think it would be better.
>>>>>
>>>>>>> +
>>>>>>>
>>>>>>>  HOST_DISTRO ?= "debian-stretch"
>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..7bd81dd6
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>>>>>> @@ -6,6 +6,8 @@
>>>>>>>
>>>>>>>  BASE_DISTRO = "raspios"
>>>>>>>  BASE_DISTRO_arm64 = "debian"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO ?= "debian"
>>>>>>> +
>>>>>>>
>>>>>>>  BASE_DISTRO_CODENAME = "bullseye"
>>>>>>>  HOST_DISTRO ?= "debian-bullseye"
>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>
>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..75da79a1
>>>>>>> 100644
>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>>>>>
>>>>>>>  BASE_DISTRO = "ubuntu"
>>>>>>>  BASE_DISTRO_CODENAME = "focal"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
>>>>>>> +
>>>>>>>
>>>>>>>  DISTRO_APT_SOURCES_arm64 ?=
>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>>>>>  HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>>>>>  conf/distro/${HOST_DISTRO}-ports.list">
>>>>>>>
>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b 100644
>>>>>>> --- a/meta/conf/distro/debian-common.conf
>>>>>>> +++ b/meta/conf/distro/debian-common.conf
>>>>>>> @@ -5,6 +5,8 @@
>>>>>>>
>>>>>>>  BASE_DISTRO = "debian"
>>>>>>>
>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
>>>>>>> +
>>>>>>>
>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>>>>>  DISTRO_APT_SOURCES ?=
>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
>>>>>>>
>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>>>> f32d192e..2ee0d4cc 100644
>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
>>>>>>> ",gnupg"
>>>>>>>
>>>>>>>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>>>>>  "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>>>>>  d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>>>>>>  "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1'
>>>>>>>  else
>>>>>>>  'DISTRO')}">
>>>>>>>
>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
>>>>>>>
>>>>>>>  FILESEXTRAPATHS_append = ":${BBPATH}"
>>>>>>>  
>>>>>>>  inherit deb-dl-dir
>>>>>>>
>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, 
> aptsources_entry_list):
>>>>>>>      import re
>>>>>>>
>>>>>>>      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>>>> -        premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>>>>>> +        premirrors = "\S*
>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
>>>>>>>
>>>>>>>      else:
>>>>>>>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>>>>>      
>>>>>>>      mirror_list = [entry.split()
>>>>>>>
>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>>>>>
>>>>>>>                           "${ROOTFSDIR}/etc/apt/preferences.d/bootstra
>>>>>>>                           p"
>>>>>>>          
>>>>>>>          mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>>>>>          if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>>>>>
>>>>>>> -            line="file:///base-apt/${BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>> main" +            line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME} main">
>>>>>>>
>>>>>>>              if [ -z "${BASE_REPO_KEY}" ]; then
>>>>>>>              
>>>>>>>                  line="[trusted=yes] ${line}"
>>>>>>>              
>>>>>>>              fi
>>>>>>>              echo "deb ${line}" >
>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
>>>>>>>
>>>>>>> +            line="file:///base-apt/${BASE_DISTRO}
>>>>>>> ${BASE_DISTRO_CODENAME}
>>>>>>> main"
>>>>>>
>>>>>> Won't this cause duplicate lines (with complaints of apt) in case of
>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
>>>>>
>>>>> No, while I don't add new lines (no new "echo"s). I just modify
>>>>> "deb-src"
>>>>> line while it should point to BASE_DISTRO even for host case. Without
>>>>> it,
>>>>> there was a problem with "hello" (apt_fetch failed to download cached
>>>>> sources from "host" repo, while sources were cached in "target" one.
>>>>>
>>>>> That's how it looks now:
>>>>> | $ cat isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
>>>>>
>>>>> apt.list
>>>>>
>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>> | $ cat
>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-apt.list
>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
>>>>
>>>> Indeed, I misintepreted the code.
>>>>
>>>> But I think we still have an issue on the deb-src population side:
>>>>
>>>> # ls build/downloads/deb-src/
>>>> iot2050-debian
>>>>
>>>> And that was with this patch applied.
>>>>
>>>> Jan
>>>
>>> Actually, I checked the case with the following configuration:
>>>
>>> distro: my-debian
>>>
>>> local_conf_header:
>>>   nodebian: |
>>>   
>>>     require conf/distro/debian-bullseye.conf
>>>     DISTRO_NAME = "My Debian System"
>>>
>>> So, I simulated renaming distro that is done in meta-iot2050.
>>>
>>> Cached build was OK, "hello" package (that we build from sources) was
>>>
>>> downloaded at first build to "my-debian" directory:
>>> | $ ls build/downloads/deb-src/my-debian/hello/
>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc  hello_2.10.orig.tar.gz
>>>
>>> and second build picked the files with no problem.
>>>
>>> The same with "raspios-bullseye" target - "hello" sources are downloaded
>>> to
>>>
>>> the "target" dir:
>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc  hello_2.10.orig.tar.gz
>>>
>>> I even tested the case with  BASE_REPO_FEATURES += "cache-deb-src" and it
>>> was OK (honestly, I don't know who could require cached build with this
>>> option enabled :) ).
>>>
>>> Anyway, I'll also try to check my changes with meta-iot2050 layer also
>>> (despite some packages in it are currently not compatible with
>>> BB_NO_NETWORK option)
>>
>> Please then test without the coral packages, they have an own issue /wrt
>> offline builds.
> 
> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
> 

...but it caches to the wrong downloads/deb-src path, no?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux