Re: Offline build broken with sbuild?

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.11.22 07:57, Uladzimir Bely wrote:
> In the email from Wednesday, 23 November 2022 19:17:31 +03 user Jan Kiszka 
> wrote:
>> On 23.11.22 16:49, Jan Kiszka wrote:
>>> On 22.11.22 14:11, Jan Kiszka wrote:
>>>> On 22.11.22 13:15, Uladzimir Bely wrote:
>>>>> In the email from Monday, 21 November 2022 10:36:43 +03 user Uladzimir
>>>>> Bely
>>>>>
>>>>> wrote:
>>>>>> I'll try to check more things (e.g. Isar revision mentioned in the bug;
>>>>>> complete image build, etc), but at the first glance sbuild-chroot-host
>>>>>> looks to be cached OK, at least with the latest Isar.
>>>>>
>>>>> So, I've got the following result for now:
>>>>>
>>>>> - In Isar itself cached cross-build works well (while both HOST_DISTRO
>>>>> and
>>>>> DISTRO are equal). The only exception: raspberry targets, while DISTRO
>>>>> and
>>>>> HOST_DISTRO are different by nature.
>>>>>
>>>>> - In meta-iot2050, cached cross-build doesn't work for both "sbuild" and
>>>>> "pre- sbuild" cases. Evend "isar-bootstrap-host" task doesn't work for
>>>>> cached build.
>>>>>
>>>>> I've applied the following patch for now and it at least fixed
>>>>> sbuild-chroot- host task for cached build. Waiting for full build
>>>>> finished and will update you with the result.
>>>>>
>>>>> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-
>>>>> devtools/base-apt/base-apt.bb
>>>>> index 506a28f..316960d 100644
>>>>> --- a/meta/recipes-devtools/base-apt/base-apt.bb
>>>>> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
>>>>> @@ -13,7 +13,9 @@ KEYFILES ?= ""
>>>>>
>>>>>  BASE_REPO_FEATURES ?= ""
>>>>>  
>>>>>  populate_base_apt() {
>>>>>
>>>>> -    find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package;
>>>>> do
>>>>> +    distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
>>>>> +
>>>>> +    find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package;
>>>>> do
>>>>>
>>>>>          # NOTE: due to packages stored by reprepro are not modified, we
>>>>>          can
>>>>>          # use search by filename to check if package is already in
>>>>>          repo. In
>>>>>          # addition, md5sums are compared to ensure that the package is
>>>>>          the
>>>>>
>>>>> @@ -38,7 +40,7 @@ populate_base_apt() {
>>>>>
>>>>>              "${package}"
>>>>>      
>>>>>      done
>>>>>
>>>>> -    find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read
>>>>> package; do +    find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while
>>>>> read package; do>>> 
>>>>>          repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
>>>>>          
>>>>>              "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
>>>>>              "${BASE_DISTRO_CODENAME}" \
>>>>
>>>> Ah, another DISTRO-override issue! Hope we don't have more of that type
>>>> in the core. Maybe worth to scan again and check.
>>>>
>>>> Thanks for analyzing and (likely) fixing!
>>>
>>> sbuild-chroot-host is indeed working now, but there are likely more
>>> cases with (custom) DISTRO vs. BASE_DISTRO:
>>>
>>> ERROR: isar-bootstrap-target-1.0-r0 do_bootstrap:
>>> ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-targe
>>> t/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None) ERROR: Logfile of
>>> failure stored in:
>>> /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/temp/lo
>>> g.do_bootstrap.21764> 
>>> Log data follows:
>>> | DEBUG: Executing python function sstate_task_prefunc
>>> | DEBUG: Python function sstate_task_prefunc finished
>>> | DEBUG: Executing shell function do_bootstrap
>>> | W: qemu-debootstrap is deprecated. Please use regular debootstrap
>>> | directly
>>> | I: Running command: debootstrap --verbose --variant=minbase
>>> | --include=locales --no-check-gpg --arch=arm64
>>> | --components=main,contrib,non-free bullseye
>>> | /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/rootf
>>> | s file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian I:
>>> | Retrieving InRelease
>>> | I: Retrieving Release
>>> | I: Retrieving Packages
>>> | I: Validating Packages
>>> | I: Resolving dependencies of required packages...
>>> | I: Resolving dependencies of base packages...
>>> | I: Checking component main on
>>> | file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian... E:
>>> | Couldn't find these debs: apt
>>> | WARNING: exit code 1 from a shell command.
>>> | ERROR:
>>> | ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-tar
>>> | get/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None)> 
>>> ERROR: Task
>>> (/build/../work/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-targ
>>> et.bb:do_bootstrap) failed with exit code '1'
>> I think we have two problems here:
>>
>>  1. base-apt is set up in sources-list using the BASE_DISTRO, but we are
>>     creating it using DISTRO, not only in bast-apt, also dpkg-base
>>
>>  2. the whole base-apt caching does not yet consider
>>     BASE_DISTRO(DISTRO) != BASE_DISTRO(HOST_DISTRO), and that maybe even
>>     before all the sbuild changes
>>
>> The latter should break the Raspian case.
>>
>> Jan
> 
> I've just send patchset that fixes case when DISTRO is changed from default 
> value (but still in fact remains "debian").
> 
> I set up the following configuration in local.conf:
> 
>>    require conf/distro/debian-bullseye.conf
>>    DISTRO_NAME = "My Debian System"
>>    DISTRO = "my-debian"
> 
> And got the same problem with sbuild-chroot-host installation at 2nd (cached) 
> build. The patches solved the issue.
> 
> The case with Raspberry (target) / Debian (host) is more complex and still 
> doesn't work. To make cached build work in this case we at least need to
> rework `base-apt` - split it onto 2 repos that are populated only from their 
> own original repos. 

Left a comment

> 
> What concerns `meta-iot2050`, it still fails to do cached build, but due to 
> some local problems. For example, `python-absl` recipe from `meta-coral` layer 
> conflicts with BB_NO_NETWORK = "1" option...
> 

Yes, this is understood already, and a fix for meta-coral is pending.

Thanks,
Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux