Re: [PATCH v3 1/3] Populate base-apt from both DISTRO and HOST_DISTRO download dirs

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Monday, 28 November 2022 09:51:12 +03 user Jan Kiszka wrote:
> On 28.11.22 07:41, Uladzimir Bely wrote:
> > In email from Monday, 28 November 2022 09:32:22 +03 You wrote:
> >> On 28.11.22 07:05, Uladzimir Bely wrote:
> >>> In the email from Sunday, 27 November 2022 22:05:53 +03 user Jan Kiszka
> > 
> > wrote:
> >>>> On 26.11.22 06:30, Uladzimir Bely wrote:
> >>>>> In the email from Friday, 25 November 2022 18:40:39 +03 user Jan
> >>>>> Kiszka
> >>> 
> >>> wrote:
> >>>>>> On 25.11.22 15:48, Uladzimir Bely wrote:
> >>>>>>> In the email from Friday, 25 November 2022 17:11:26 +03 user Jan
> >>>>>>> Kiszka
> >>>>> 
> >>>>> wrote:
> >>>>>>>> On 25.11.22 08:56, Uladzimir Bely wrote:
> >>>>>>>>> In the email from Friday, 25 November 2022 10:42:00 +03 user Jan
> >>>>>>>>> Kiszka
> >>>>>>> 
> >>>>>>> wrote:
> >>>>>>>>>> On 25.11.22 07:42, Uladzimir Bely wrote:
> >>>>>>>>>>> Downstreams may want to change DISTRO to some custom value.
> >>>>>>>>>>> When cross-building, this leads to downloading packages for host
> >>>>>>>>>>> and target distros to the different download subdirs.
> >>>>>>>>>>> 
> >>>>>>>>>>> While base-apt is populated only from DISTRO download subdir,
> >>>>>>>>>>> second cached (ISAR_USE_CACHED_BASE_REPO = "1") build fails due
> >>>>>>>>>>> to
> >>>>>>>>>>> missing packages in base-apt.
> >>>>>>>>>>> 
> >>>>>>>>>>> The similar issue happens with cross-building raspberry targets
> >>>>>>>>>>> since they use their own mirrors and DISTRO.
> >>>>>>>>>>> 
> >>>>>>>>>>> Fix the issue by splitting base-apt by introducing
> >>>>>>>>>>> BASE_HOST_DISTRO
> >>>>>>>>>>> variable that is used instead of BASE_DISTRO for caching builds
> >>>>>>>>>>> for host-related components (e.g., isar-bootstrap,
> >>>>>>>>>>> buildchroot-host,
> >>>>>>>>>>> sbuild-chroot-host). Target components still use BASE_DISTRO.
> >>>>>>>>>>> 
> >>>>>>>>>>> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> >>>>>>>>>>> ---
> >>>>>>>>>>> 
> >>>>>>>>>>>  meta-isar/conf/distro/raspbian-stretch.conf   |  2 +
> >>>>>>>>>>>  meta-isar/conf/distro/raspios-bullseye.conf   |  2 +
> >>>>>>>>>>>  meta-isar/conf/distro/ubuntu-focal.conf       |  2 +
> >>>>>>>>>>>  meta/conf/distro/debian-common.conf           |  2 +
> >>>>>>>>>>>  .../isar-bootstrap/isar-bootstrap.inc         |  9 +++-
> >>>>>>>>>>>  meta/recipes-devtools/base-apt/base-apt.bb    | 47
> >>>>>>>>>>>  ++++++++++---------
> >>>>>>>>>>>  6 files changed, 40 insertions(+), 24 deletions(-)
> >>>>>>>>>>> 
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/raspbian-stretch.conf index
> >>>>>>>>>>> c8f523b3..66d965c9
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspbian-stretch.conf
> >>>>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>>> 
> >>>>>>>>>>>  BASE_DISTRO = "raspbian"
> >>>>>>>>>>>  BASE_DISTRO_CODENAME = "stretch"
> >>>>>>>>>>> 
> >>>>>>>>>>> +BASE_HOST_DISTRO = "debian"
> >>>>>>>>>> 
> >>>>>>>>>> Better prefix it with HOST, to be consistent:
> >>>>>>>>>> 
> >>>>>>>>>> HOST_BASE_DISTRO
> >>>>>>>>> 
> >>>>>>>>> Not a problem, if you think it would be better.
> >>>>>>>>> 
> >>>>>>>>>>> +
> >>>>>>>>>>> 
> >>>>>>>>>>>  HOST_DISTRO ?= "debian-stretch"
> >>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>> 
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/raspios-bullseye.conf index
> >>>>>>>>>>> 5e4a09c9..7bd81dd6
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/raspios-bullseye.conf
> >>>>>>>>>>> @@ -6,6 +6,8 @@
> >>>>>>>>>>> 
> >>>>>>>>>>>  BASE_DISTRO = "raspios"
> >>>>>>>>>>>  BASE_DISTRO_arm64 = "debian"
> >>>>>>>>>>> 
> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "debian"
> >>>>>>>>>>> +
> >>>>>>>>>>> 
> >>>>>>>>>>>  BASE_DISTRO_CODENAME = "bullseye"
> >>>>>>>>>>>  HOST_DISTRO ?= "debian-bullseye"
> >>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>> 
> >>>>>>>>>>> diff --git a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> b/meta-isar/conf/distro/ubuntu-focal.conf index
> >>>>>>>>>>> 379b5b30..75da79a1
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> +++ b/meta-isar/conf/distro/ubuntu-focal.conf
> >>>>>>>>>>> @@ -8,6 +8,8 @@ require conf/distro/debian-common.conf
> >>>>>>>>>>> 
> >>>>>>>>>>>  BASE_DISTRO = "ubuntu"
> >>>>>>>>>>>  BASE_DISTRO_CODENAME = "focal"
> >>>>>>>>>>> 
> >>>>>>>>>>> +BASE_HOST_DISTRO = "${BASE_DISTRO}"
> >>>>>>>>>>> +
> >>>>>>>>>>> 
> >>>>>>>>>>>  DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}-ports.list"
> >>>>>>>>>>>  HOST_DISTRO_APT_SOURCES_arm64 ?=
> >>>>>>>>>>>  "conf/distro/${HOST_DISTRO}.list
> >>>>>>>>>>>  conf/distro/${HOST_DISTRO}-ports.list">
> >>>>>>>>>>> 
> >>>>>>>>>>> diff --git a/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> b/meta/conf/distro/debian-common.conf index c18eebc2..d7a13f4b
> >>>>>>>>>>> 100644
> >>>>>>>>>>> --- a/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> +++ b/meta/conf/distro/debian-common.conf
> >>>>>>>>>>> @@ -5,6 +5,8 @@
> >>>>>>>>>>> 
> >>>>>>>>>>>  BASE_DISTRO = "debian"
> >>>>>>>>>>> 
> >>>>>>>>>>> +BASE_HOST_DISTRO ?= "${BASE_DISTRO}"
> >>>>>>>>>>> +
> >>>>>>>>>>> 
> >>>>>>>>>>>  HOST_DISTRO_APT_SOURCES ?= "conf/distro/${HOST_DISTRO}.list"
> >>>>>>>>>>>  DISTRO_APT_SOURCES ?=
> >>>>>>>>>>>  "conf/distro/${BASE_DISTRO}-${BASE_DISTRO_CODENAME}.list">
> >>>>>>>>>>> 
> >>>>>>>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>>>>>>>>>> f32d192e..2ee0d4cc 100644
> >>>>>>>>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>>>>>>>>>> @@ -29,6 +29,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg =
> >>>>>>>>>>> ",gnupg"
> >>>>>>>>>>> 
> >>>>>>>>>>>  DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support =
> >>>>>>>>>>>  "${@https_support(d)}" DISTRO_VARS_PREFIX ?= "${@'HOST_' if
> >>>>>>>>>>>  d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
> >>>>>>>>>>>  BOOTSTRAP_DISTRO
> >>>>>>>>>>>  =
> >>>>>>>>>>>  "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') ==
> >>>>>>>>>>>  '1'
> >>>>>>>>>>>  else
> >>>>>>>>>>>  'DISTRO')}">
> >>>>>>>>>>> 
> >>>>>>>>>>> +BOOTSTRAP_BASE_DISTRO = "${@d.getVar('BASE_HOST_DISTRO' if
> >>>>>>>>>>> d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}">
> >>>>>>>>>>> 
> >>>>>>>>>>>  FILESEXTRAPATHS_append = ":${BBPATH}"
> >>>>>>>>>>>  
> >>>>>>>>>>>  inherit deb-dl-dir
> >>>>>>>>>>> 
> >>>>>>>>>>> @@ -111,7 +112,7 @@ def get_apt_source_mirror(d,
> >>>>> 
> >>>>> aptsources_entry_list):
> >>>>>>>>>>>      import re
> >>>>>>>>>>>      
> >>>>>>>>>>>      if
> > 
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>>>>>>>>>> -        premirrors = "\S*
> >>>>>>>>>>> file://${REPO_BASE_DIR}/${BASE_DISTRO}\n"
> >>>>>>>>>>> +        premirrors = "\S*
> >>>>>>>>>>> file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n">
> >>>>>>>>>>> 
> >>>>>>>>>>>      else:
> >>>>>>>>>>>          premirrors = d.getVar('DISTRO_APT_PREMIRRORS', True) or
> >>>>>>>>>>>          ""
> >>>>>>>>>>>      
> >>>>>>>>>>>      mirror_list = [entry.split()
> >>>>>>>>>>> 
> >>>>>>>>>>> @@ -319,11 +320,15 @@ do_bootstrap() {
> >>>>>>>>>>> 
> >>>>>>>>>>>                           "${ROOTFSDIR}/etc/apt/preferences.d/bo
> >>>>>>>>>>>                           ot
> >>>>>>>>>>>                           st
> >>>>>>>>>>>                           ra
> >>>>>>>>>>>                           p"
> >>>>>>>>>>>          
> >>>>>>>>>>>          mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> >>>>>>>>>>>          if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> >>>>>>>>>>> 
> >>>>>>>>>>> -            line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>>>> main" +           
> >>>>>>>>>>> line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME} main">
> >>>>>>>>>>> 
> >>>>>>>>>>>              if [ -z "${BASE_REPO_KEY}" ]; then
> >>>>>>>>>>>              
> >>>>>>>>>>>                  line="[trusted=yes] ${line}"
> >>>>>>>>>>>              
> >>>>>>>>>>>              fi
> >>>>>>>>>>>              echo "deb ${line}" >
> >>>>>>>>>>>              "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> >>>>>>>>>>> 
> >>>>>>>>>>> +            line="file:///base-apt/${BASE_DISTRO}
> >>>>>>>>>>> ${BASE_DISTRO_CODENAME}
> >>>>>>>>>>> main"
> >>>>>>>>>> 
> >>>>>>>>>> Won't this cause duplicate lines (with complaints of apt) in case
> >>>>>>>>>> of
> >>>>>>>>>> BOOTSTRAP_BASE_DISTRO == BASE_DISTRO?
> >>>>>>>>> 
> >>>>>>>>> No, while I don't add new lines (no new "echo"s). I just modify
> >>>>>>>>> "deb-src"
> >>>>>>>>> line while it should point to BASE_DISTRO even for host case.
> >>>>>>>>> Without
> >>>>>>>>> it,
> >>>>>>>>> there was a problem with "hello" (apt_fetch failed to download
> >>>>>>>>> cached
> >>>>>>>>> sources from "host" repo, while sources were cached in "target"
> >>>>>>>>> one.
> >>>>>>>>> 
> >>>>>>>>> That's how it looks now:
> >>>>>>>>> | $ cat
> >>>>>>>>> | isar-bootstrap-target/1.0-r0/rootfs/etc/apt/sources.list.d/base-
> >>>>>>>>> 
> >>>>>>>>> apt.list
> >>>>>>>>> 
> >>>>>>>>> | deb [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>>> | $ cat
> >>>>>>>>> | isar-bootstrap-host/1.0-r0/rootfs/etc/apt/sources.list.d/base-ap
> >>>>>>>>> | t.
> >>>>>>>>> | li
> >>>>>>>>> | st
> >>>>>>>>> | deb [trusted=yes] file:///base-apt/debian bullseye main
> >>>>>>>>> | deb-src [trusted=yes] file:///base-apt/raspios bullseye main
> >>>>>>>> 
> >>>>>>>> Indeed, I misintepreted the code.
> >>>>>>>> 
> >>>>>>>> But I think we still have an issue on the deb-src population side:
> >>>>>>>> 
> >>>>>>>> # ls build/downloads/deb-src/
> >>>>>>>> iot2050-debian
> >>>>>>>> 
> >>>>>>>> And that was with this patch applied.
> >>>>>>>> 
> >>>>>>>> Jan
> >>>>>>> 
> >>>>>>> Actually, I checked the case with the following configuration:
> >>>>>>> 
> >>>>>>> distro: my-debian
> >>>>>>> 
> >>>>>>> local_conf_header:
> >>>>>>>   nodebian: |
> >>>>>>>   
> >>>>>>>     require conf/distro/debian-bullseye.conf
> >>>>>>>     DISTRO_NAME = "My Debian System"
> >>>>>>> 
> >>>>>>> So, I simulated renaming distro that is done in meta-iot2050.
> >>>>>>> 
> >>>>>>> Cached build was OK, "hello" package (that we build from sources)
> >>>>>>> was
> >>>>>>> 
> >>>>>>> downloaded at first build to "my-debian" directory:
> >>>>>>> | $ ls build/downloads/deb-src/my-debian/hello/
> >>>>>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc 
> >>>>>>> | hello_2.10.orig.tar.gz
> >>>>>>> 
> >>>>>>> and second build picked the files with no problem.
> >>>>>>> 
> >>>>>>> The same with "raspios-bullseye" target - "hello" sources are
> >>>>>>> downloaded
> >>>>>>> to
> >>>>>>> 
> >>>>>>> the "target" dir:
> >>>>>>> | $ ls build/downloads/deb-src/raspios-bullseye/hello/
> >>>>>>> | hello_2.10-2.debian.tar.xz  hello_2.10-2.dsc 
> >>>>>>> | hello_2.10.orig.tar.gz
> >>>>>>> 
> >>>>>>> I even tested the case with  BASE_REPO_FEATURES += "cache-deb-src"
> >>>>>>> and
> >>>>>>> it
> >>>>>>> was OK (honestly, I don't know who could require cached build with
> >>>>>>> this
> >>>>>>> option enabled :) ).
> >>>>>>> 
> >>>>>>> Anyway, I'll also try to check my changes with meta-iot2050 layer
> >>>>>>> also
> >>>>>>> (despite some packages in it are currently not compatible with
> >>>>>>> BB_NO_NETWORK option)
> >>>>>> 
> >>>>>> Please then test without the coral packages, they have an own issue
> >>>>>> /wrt
> >>>>>> offline builds.
> >>>>> 
> >>>>> Checked the patchset with `IOT2050_CORAL_SUPPORT = "0"` and it works
> >>>> 
> >>>> ...but it caches to the wrong downloads/deb-src path, no?
> >>>> 
> >>>> Jan
> >>> 
> >>> Why wrong?
> >>> 
> >>> | meta-iot2050/build $ tree -d downloads/deb*
> >>> | downloads/deb
> >>> | ├── debian-bullseye
> >>> | └── iot2050-debian
> >>> | downloads/deb-src
> >>> | └── iot2050-debian
> >>> | 
> >>> |     ├── openssl
> >>> |     └── swig
> >>> 
> >>> `do_apt_fetch` works with `${DISTRO}` and downloads sources to the
> >>> corresponding directory.
> >> 
> >> deb-src fetching still lacks base-distro normalization, like deb also 
has:
> > Saying "normalization", do you mean the following: if ${DISTRO} is just
> > renaming (e.g. "iot2050-debian") everything should be downloaded to and
> > picked up from the directory named "debian-bullseye"?
> 
> Yes, just like you did for binary debs: Reduce the DISTRO to its
> BASE_DISTRO, then use that as key for the cache. The current situation
> is inconsistent and non-optimal, even if working.
> 
> Jan

I've just pushed v4 of patchset that still doesn't include this wish. For now, 
I would apply it to the isar/next to have cached cross build working.

We have some plans to improve base-apt in general (there were some PoC patches 
on list), so improvements you mention could be done a bit later - for now I 
need to switch to some other tasks.

-- 
Uladzimir Bely