Re: Offline build broken with sbuild?

[Uladzimir Bely <ubely@ilbers.de>]

In the email from Wednesday, 23 November 2022 19:17:31 +03 user Jan Kiszka 
wrote:
> On 23.11.22 16:49, Jan Kiszka wrote:
> > On 22.11.22 14:11, Jan Kiszka wrote:
> >> On 22.11.22 13:15, Uladzimir Bely wrote:
> >>> In the email from Monday, 21 November 2022 10:36:43 +03 user Uladzimir
> >>> Bely
> >>> 
> >>> wrote:
> >>>> I'll try to check more things (e.g. Isar revision mentioned in the bug;
> >>>> complete image build, etc), but at the first glance sbuild-chroot-host
> >>>> looks to be cached OK, at least with the latest Isar.
> >>> 
> >>> So, I've got the following result for now:
> >>> 
> >>> - In Isar itself cached cross-build works well (while both HOST_DISTRO
> >>> and
> >>> DISTRO are equal). The only exception: raspberry targets, while DISTRO
> >>> and
> >>> HOST_DISTRO are different by nature.
> >>> 
> >>> - In meta-iot2050, cached cross-build doesn't work for both "sbuild" and
> >>> "pre- sbuild" cases. Evend "isar-bootstrap-host" task doesn't work for
> >>> cached build.
> >>> 
> >>> I've applied the following patch for now and it at least fixed
> >>> sbuild-chroot- host task for cached build. Waiting for full build
> >>> finished and will update you with the result.
> >>> 
> >>> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-
> >>> devtools/base-apt/base-apt.bb
> >>> index 506a28f..316960d 100644
> >>> --- a/meta/recipes-devtools/base-apt/base-apt.bb
> >>> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> >>> @@ -13,7 +13,9 @@ KEYFILES ?= ""
> >>> 
> >>>  BASE_REPO_FEATURES ?= ""
> >>>  
> >>>  populate_base_apt() {
> >>> 
> >>> -    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package;
> >>> do
> >>> +    distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
> >>> +
> >>> +    find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package;
> >>> do
> >>> 
> >>>          # NOTE: due to packages stored by reprepro are not modified, we
> >>>          can
> >>>          # use search by filename to check if package is already in
> >>>          repo. In
> >>>          # addition, md5sums are compared to ensure that the package is
> >>>          the
> >>> 
> >>> @@ -38,7 +40,7 @@ populate_base_apt() {
> >>> 
> >>>              "${package}"
> >>>      
> >>>      done
> >>> 
> >>> -    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read
> >>> package; do +    find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while
> >>> read package; do>>> 
> >>>          repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
> >>>          
> >>>              "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
> >>>              "${BASE_DISTRO_CODENAME}" \
> >> 
> >> Ah, another DISTRO-override issue! Hope we don't have more of that type
> >> in the core. Maybe worth to scan again and check.
> >> 
> >> Thanks for analyzing and (likely) fixing!
> > 
> > sbuild-chroot-host is indeed working now, but there are likely more
> > cases with (custom) DISTRO vs. BASE_DISTRO:
> > 
> > ERROR: isar-bootstrap-target-1.0-r0 do_bootstrap:
> > ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-targe
> > t/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None) ERROR: Logfile of
> > failure stored in:
> > /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/temp/lo
> > g.do_bootstrap.21764> 
> > Log data follows:
> > | DEBUG: Executing python function sstate_task_prefunc
> > | DEBUG: Python function sstate_task_prefunc finished
> > | DEBUG: Executing shell function do_bootstrap
> > | W: qemu-debootstrap is deprecated. Please use regular debootstrap
> > | directly
> > | I: Running command: debootstrap --verbose --variant=minbase
> > | --include=locales --no-check-gpg --arch=arm64
> > | --components=main,contrib,non-free bullseye
> > | /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/rootf
> > | s file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian I:
> > | Retrieving InRelease
> > | I: Retrieving Release
> > | I: Retrieving Packages
> > | I: Validating Packages
> > | I: Resolving dependencies of required packages...
> > | I: Resolving dependencies of base packages...
> > | I: Checking component main on
> > | file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian... E:
> > | Couldn't find these debs: apt
> > | WARNING: exit code 1 from a shell command.
> > | ERROR:
> > | ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-tar
> > | get/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None)> 
> > ERROR: Task
> > (/build/../work/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-targ
> > et.bb:do_bootstrap) failed with exit code '1'
> I think we have two problems here:
> 
>  1. base-apt is set up in sources-list using the BASE_DISTRO, but we are
>     creating it using DISTRO, not only in bast-apt, also dpkg-base
> 
>  2. the whole base-apt caching does not yet consider
>     BASE_DISTRO(DISTRO) != BASE_DISTRO(HOST_DISTRO), and that maybe even
>     before all the sbuild changes
> 
> The latter should break the Raspian case.
> 
> Jan

I've just send patchset that fixes case when DISTRO is changed from default 
value (but still in fact remains "debian").

I set up the following configuration in local.conf:

>    require conf/distro/debian-bullseye.conf
>    DISTRO_NAME = "My Debian System"
>    DISTRO = "my-debian"

And got the same problem with sbuild-chroot-host installation at 2nd (cached) 
build. The patches solved the issue.

The case with Raspberry (target) / Debian (host) is more complex and still 
doesn't work. To make cached build work in this case we at least need to
rework `base-apt` - split it onto 2 repos that are populated only from their 
own original repos. 

What concerns `meta-iot2050`, it still fails to do cached build, but due to 
some local problems. For example, `python-absl` recipe from `meta-coral` layer 
conflicts with BB_NO_NETWORK = "1" option...

-- 
Uladzimir Bely