RE: [PATCH v2] dpkg-base: Remove newly deployed debs from buildchroots

["Kanagarajan, Vijaikumar" <Vijaikumar_Kanagarajan@mentor.com>]

Hello Jan,

-----Original Message-----
From: isar-users@googlegroups.com [mailto:isar-users@googlegroups.com] On Behalf Of Jan Kiszka
Sent: 22 September 2020 00:48
To: isar-users <isar-users@googlegroups.com>; Baurzhan Ismagulov <ibr@ilbers.de>
Subject: Re: [PATCH v2] dpkg-base: Remove newly deployed debs from buildchroots

On 24.08.20 09:16, [ext] Jan Kiszka wrote:
> On 23.08.20 18:06, Baurzhan Ismagulov wrote:
>> Hello Jan,
>>
>> On Tue, Aug 11, 2020 at 09:05:59AM +0200, Jan Kiszka wrote:
>>> This ensures clean reinstallation after partial rebuilds.
>>>
>>> A typical error pattern so far was that firmware packages pulled by 
>>> the buildchroot-target were not updated there on rebuilds, causing 
>>> the old firmware being deployed into the image.
>>
>> With this applied, we encounter random build failures downstream, e.g.:
>>
>> 1. ERROR: mc:pixi-cdl100-buster:desktop-shortcuts-1.0-r0 do_install_builddeps: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.do_install_builddeps.19582' failed with exit code 1:
>>     mk-build-deps: You must have equivs installed to use this program.
>>     WARNING: exit code 1 from a shell command.
>>
>> 2. Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.dpkg_runbuild.16998' failed with exit code 2:
>>     dpkg-buildpackage: info: source package desktop-shortcuts
>>     dpkg-buildpackage: info: source version 1.0
>>     dpkg-buildpackage: info: source distribution UNRELEASED
>>     dpkg-buildpackage: info: source changed by isar-users <isar-users@googlegroups.com>
>>     dpkg-architecture: warning: specified GNU system type arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try setting a correct CC environment variable
>>      dpkg-source -I --before-build .
>>     dpkg-buildpackage: info: host architecture armhf
>>      fakeroot debian/rules clean
>>     dh clean
>>     make: dh: Command not found
>>
>> Seems we have a race, possibly related to removing packages. What I 
>> don't understand is how it could affect equivs or debhelper -- we 
>> don't do anything with them. Ideas?

I have a  similar condition now. We build openssl with a patch downstream. Looks like with this patch now, in our setup openssl is removed from buildchrootdir, which inturn removes
devscripts(contains mk-build-deps). This causes other recipe builds to fail. Please see the log below.


ERROR: expand-on-first-boot-1.1-r0 do_install_builddeps: Execution of '/work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/run.do_install_builddeps.10653' failed with exit code 1:
/isar/deps.sh: line 39: mk-build-deps: command not found
WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/log.do_install_builddeps.10653
Log data follows:
| DEBUG: Executing shell function do_install_builddeps
| /isar/deps.sh: line 39: mk-build-deps: command not found
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/run.do_install_builddeps.10653' failed with exit code 1:
| /isar/deps.sh: line 39: mk-build-deps: command not found
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/work/isar/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.1.bb:do_install_builddeps) failed with exit code '1'
ERROR: customizations-debug-0.1-ebsy-r0 do_install_builddeps: Execution of '/work/build/tmp/work/ebsy-buster-amd64/customizations-debug/0.1-ebsy-r0/temp/run.do_install_builddeps.10655' failed with exit code 1:
/isar/deps.sh: line 39: mk-build-deps: command not found
WARNING: exit code 1 from a shell command.

Thanks,
Vijai Kumar K

> 
> Which packages are you (re-)building? Debian packages? Or just custom 
> dependencies of that desktop-shortcuts package?
> 
> In theory, do_deploy_deb should be locked against do_install_builddeps.
> Anything that the latter needs should be available after the former.
> Maybe you are missing a build dependency in desktop-shortcuts? Try 
> build that only over a clean buildchroot, that will tell you.
> 

Meanwhile, I understood the issue is it happened on out CI system as well during nightly runs (that shouldn't have happen, but that a local issue). The logs you cited were incomplete:

[...]
NOTE: recipe cowsay-git-r0: task do_dpkg_build: Succeeded
NOTE: Running task 1617 of 1683 (mc:qemumipsel-buster:/builds/ebsy/debian/isar/meta-isar/recipes-app/cowsay/cowsay_git.bb:do_deploy_deb)
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Started [...]
NOTE: Running noexec task 1637 of 1683 (mc:rpi-stretch:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_build)
[...]
ERROR: mc:qemuarm-buster:isar-image-base-1.0-r0 do_rootfs_install: Error executing a python function in exec_python_func() autogenerated:
[...]
E: Failed to fetch file:/isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb  File not found - /isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb (2: No such file or directory)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
WARNING: exit code 100 from a shell command.


ERROR: Logfile of failure stored in: /builds/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/isar-image-base-qemuarm-ext4-img/1.0-r0/temp/log.do_rootfs_install.32804
NOTE: recipe isar-image-base-1.0-r0: task do_rootfs_install: Failed
ERROR: Task (mc:qemuarm-buster:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_rootfs_install) failed with exit code '1'
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Succeeded

The problem is that we have a false sharing between all archs building cowsay for buster. That is because this demo package is arch "all", but the tasks assume it is arch-specific, thus are building and (re-)deploying it happily in parallel. This is similar to the issue I had with [1]. It is not caused by this patch, it is revealed by it, finally.

What can we do?
 - select a different rebuild-example, one that does not generate "all"
   packages (i.e. stick our head into the sand)
 - break up isar-apt on a per-arch basis (as considered for [1])
 - find a magic spell that allows to express the commonality between
   all mc:<arch>-<distro>:<package> recipes that actually produce the
   same output

I suspect the latter will not happen any time soon, so I'm inclined to fix this via the second option.

Jan

[1] https://groups.google.com/d/msgid/isar-users/e4ca6d44-8b99-24ba-4fcd-c36cedac1ff7%40siemens.com

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux

--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/c3998f33-521e-49ed-41c5-95f1f935d5df%40siemens.com.