Re: [PATCH v2] dpkg-base: Remove newly deployed debs from buildchroots

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.08.20 09:16, [ext] Jan Kiszka wrote:
> On 23.08.20 18:06, Baurzhan Ismagulov wrote:
>> Hello Jan,
>>
>> On Tue, Aug 11, 2020 at 09:05:59AM +0200, Jan Kiszka wrote:
>>> This ensures clean reinstallation after partial rebuilds.
>>>
>>> A typical error pattern so far was that firmware packages pulled by the
>>> buildchroot-target were not updated there on rebuilds, causing the old
>>> firmware being deployed into the image.
>>
>> With this applied, we encounter random build failures downstream, e.g.:
>>
>> 1. ERROR: mc:pixi-cdl100-buster:desktop-shortcuts-1.0-r0 do_install_builddeps: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.do_install_builddeps.19582' failed with exit code 1:
>>     mk-build-deps: You must have equivs installed to use this program.
>>     WARNING: exit code 1 from a shell command.
>>
>> 2. Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.dpkg_runbuild.16998' failed with exit code 2:
>>     dpkg-buildpackage: info: source package desktop-shortcuts
>>     dpkg-buildpackage: info: source version 1.0
>>     dpkg-buildpackage: info: source distribution UNRELEASED
>>     dpkg-buildpackage: info: source changed by isar-users <isar-users@googlegroups.com>
>>     dpkg-architecture: warning: specified GNU system type arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try setting a correct CC environment variable
>>      dpkg-source -I --before-build .
>>     dpkg-buildpackage: info: host architecture armhf
>>      fakeroot debian/rules clean
>>     dh clean
>>     make: dh: Command not found
>>
>> Seems we have a race, possibly related to removing packages. What I don't
>> understand is how it could affect equivs or debhelper -- we don't do anything
>> with them. Ideas?
> 
> Which packages are you (re-)building? Debian packages? Or just custom
> dependencies of that desktop-shortcuts package?
> 
> In theory, do_deploy_deb should be locked against do_install_builddeps.
> Anything that the latter needs should be available after the former.
> Maybe you are missing a build dependency in desktop-shortcuts? Try build
> that only over a clean buildchroot, that will tell you.
> 

Meanwhile, I understood the issue is it happened on out CI system as 
well during nightly runs (that shouldn't have happen, but that a local 
issue). The logs you cited were incomplete:

[...]
NOTE: recipe cowsay-git-r0: task do_dpkg_build: Succeeded
NOTE: Running task 1617 of 1683 (mc:qemumipsel-buster:/builds/ebsy/debian/isar/meta-isar/recipes-app/cowsay/cowsay_git.bb:do_deploy_deb)
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Started
[...]
NOTE: Running noexec task 1637 of 1683 (mc:rpi-stretch:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_build)
[...]
ERROR: mc:qemuarm-buster:isar-image-base-1.0-r0 do_rootfs_install: Error executing a python function in exec_python_func() autogenerated:
[...]
E: Failed to fetch file:/isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb  File not found - /isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb (2: No such file or directory)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
WARNING: exit code 100 from a shell command.


ERROR: Logfile of failure stored in: /builds/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/isar-image-base-qemuarm-ext4-img/1.0-r0/temp/log.do_rootfs_install.32804
NOTE: recipe isar-image-base-1.0-r0: task do_rootfs_install: Failed
ERROR: Task (mc:qemuarm-buster:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_rootfs_install) failed with exit code '1'
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Succeeded

The problem is that we have a false sharing between all archs building 
cowsay for buster. That is because this demo package is arch "all", but 
the tasks assume it is arch-specific, thus are building and 
(re-)deploying it happily in parallel. This is similar to the issue I 
had with [1]. It is not caused by this patch, it is revealed by it, 
finally.

What can we do?
 - select a different rebuild-example, one that does not generate "all"
   packages (i.e. stick our head into the sand)
 - break up isar-apt on a per-arch basis (as considered for [1])
 - find a magic spell that allows to express the commonality between
   all mc:<arch>-<distro>:<package> recipes that actually produce the
   same output

I suspect the latter will not happen any time soon, so I'm inclined to 
fix this via the second option.

Jan

[1] https://groups.google.com/d/msgid/isar-users/e4ca6d44-8b99-24ba-4fcd-c36cedac1ff7%40siemens.com

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux