Re: [PATCH v4 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Jan Kiszka <jan.kiszka@siemens.com>]

On 29.11.22 12:29, Uladzimir Bely wrote:
> In the email from Tuesday, 29 November 2022 13:49:11 +03 user Jan Kiszka 
> wrote:
>> On 28.11.22 12:05, Uladzimir Bely wrote:
>>> Downstreams may want to change DISTRO to some custom value.
>>> When cross-building, this leads to downloading packages for host
>>> and target distros to the different download subdirs.
>>>
>>> While base-apt is populated only from DISTRO download subdir,
>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due to
>>> missing packages in base-apt.
>>>
>>> The similar issue happens with cross-building raspberry targets
>>> since they use their own mirrors and DISTRO.
>>>
>>> Fix the issue by splitting base-apt by introducing HOST_BASE_DISTRO
>>> variable that is used instead of BASE_DISTRO for caching builds
>>> for host-related components (e.g., isar-bootstrap, buildchroot-host,
>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
>>>
>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
>>> ---
>>>
>>>  meta-isar/conf/distro/raspbian-stretch.conf   |  2 +
>>>  meta-isar/conf/distro/raspios-bullseye.conf   |  2 +
>>>  meta-isar/conf/distro/ubuntu-focal.conf       |  2 +
>>>  meta/conf/distro/debian-common.conf           |  2 +
>>>  .../isar-bootstrap/isar-bootstrap.inc         |  9 +++-
>>>  meta/recipes-devtools/base-apt/base-apt.bb    | 47 ++++++++++---------
>>>  6 files changed, 40 insertions(+), 24 deletions(-)
>>>
>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
>>> b/meta-isar/conf/distro/raspbian-stretch.conf index c8f523b3..1ad0152f
>>> 100644
>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
>>> @@ -6,6 +6,8 @@
>>>
>>>  BASE_DISTRO = "raspbian"
>>>  BASE_DISTRO_CODENAME = "stretch"
>>>
>>> +HOST_BASE_DISTRO = "debian"
>>> +
>>>
>>>  HOST_DISTRO ?= "debian-stretch"
>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>
>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
>>> b/meta-isar/conf/distro/raspios-bullseye.conf index 5e4a09c9..1f65c3cd
>>> 100644
>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
>>> @@ -6,6 +6,8 @@
>>>
>>>  BASE_DISTRO = "raspios"
>>>  BASE_DISTRO_arm64 = "debian"
>>>
>>> +HOST_BASE_DISTRO ?= "debian"
>>> +
>>>
>>>  BASE_DISTRO_CODENAME = "bullseye"
>>>  HOST_DISTRO ?= "debian-bullseye"
>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>
>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
>>> b/meta-isar/conf/distro/ubuntu-focal.conf index 379b5b30..084f8e3d 100644
>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
>>>
>>>  BASE_DISTRO = "ubuntu"
>>>  BASE_DISTRO_CODENAME = "focal"
>>>
>>> +HOST_BASE_DISTRO = "${BASE_DISTRO}"
>>> +
>>>
>>>  DISTRO_APT_SOURCES_arm64 ?=
>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
>>>  HOST_DISTRO_APT_SOURCES_arm64 ?= "conf/distro/${HOST_DISTRO}.list
>>>  conf/distro/${HOST_DISTRO}-ports.list"> 
>>> diff --git a/meta/conf/distro/debian-common.conf
>>> b/meta/conf/distro/debian-common.conf index c18eebc2..ada7dc6e 100644
>>> --- a/meta/conf/distro/debian-common.conf
>>> +++ b/meta/conf/distro/debian-common.conf
>>> @@ -5,6 +5,8 @@
>>>
>>>  BASE_DISTRO = "debian"
>>>
>>> +HOST_BASE_DISTRO ?= "${BASE_DISTRO}"
>>> +
>>>
>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
>>>  DISTRO_APT_SOURCES ?=
>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list"> 
>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>> f32d192e..14d876aa 100644
>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
>>>
>>>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
>>>  "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
>>>  d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}" BOOTSTRAP_DISTRO =
>>>  "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else
>>>  'DISTRO')}"> 
>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if
>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"> 
>>>  FILESEXTRAPATHS_append = ":${BBPATH}"
>>>  
>>>  inherit deb-dl-dir
>>>
>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d, aptsources_entry_list):
>>>      import re
>>>
>>>      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>> -        premirrors = "\S* file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
>>> +        premirrors = "\S*
>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"> 
>>>      else:
>>>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or ""
>>>      
>>>      mirror_list = [entry.split()
>>>
>>> @@ -319,11 +320,15 @@ do_bootstrap() {
>>>
>>>                           "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
>>>          
>>>          mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
>>>          if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
>>>
>>> -            line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME}
>>> main" +            line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
>>> ${BASE_DISTRO_CODENAME} main"
>> There is still a conceptual glitch here: BASE_DISTRO_CODENAME may not be
>> the same as "HOST_BASE_DISTRO_CODENAME" - which does not exist.
>>
>> Jan
>>
> 
> Currently there are no such cases (at least, I don't know any). Even if host/
> target distros differ ("debian" and "raspbian"), they both use the same 
> codename.
> 
> Yes, technically we could add this variable  "HOST_BASE_DISTRO_CODENAME" - 
> only if we want, for instance, cross-build debian-based distros using ubuntu-
> based host or vice versa. But I'm not sure we really need it.

Maybe worth documenting this assumption somewhere.

> 
> By the way, I'm close to come with v5 of patchset that normalizes download 
> directories. E.g., everything (both host and target packages) is in "download/
> deb/debian-bullseye" even if DISTRO was changed. The only case when different 
> directories are used if their BASE distros are different (raspberry case).

Looking forward.

> 
> Actually, there is still an issue with meta-iot2050 project. Deb files are 
> duplicated (and, I guess, downloaded twice) in "downloads/deb", without 
> "debian-bullseye". I didn't look at it deeply, but most probably it's related 
> with incorrect usage of "deb_dl_dir_*" functions in "npm.bbclass". Since Isar 
> commit "3afafbc" from Nov 26 2020 commit they should take 2 arguments, while 
> in "meta-iot2050" layer they are called with only one (commit 69cef06a from 
> Nov 23 2020).

Yes, was locally fixed already. Will be rolled out when the dependency
are stable.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux