isar-apt locking: install_imager_deps vs rootfs_install (was: Re: dpkg: error: cannot stat pathname '/tmp/apt-dpkg-install...)

[Jan Kiszka <jan.kiszka@siemens.com>]

On 26.08.19 23:28, Jan Kiszka wrote:
> On 26.08.19 22:59, Baurzhan Ismagulov wrote:
>> On Mon, Aug 26, 2019 at 10:22:21PM +0200, Jan Kiszka wrote:
>>> 2019-08-26 19:12:59 - ERROR    - ERROR: 
>>> mc:ultra96-jailhouse-demo:demo-image-1.0-r0 do_rootfs_install: Function 
>>> failed: rootfs_install_pkgs_install (log file is located at 
>>> /builds/ebsy/debian/build/tmp/work/jailhouse-demo-arm64/demo-image-ultra96-wic-img/1.0-r0/temp/log.do_rootfs_install.60341) 
>>>
>>> [...]
>>> 2019-08-26 19:12:59 - INFO     - | DEBUG: Executing shell function 
>>> rootfs_install_pkgs_install
>>> [...]
>>> 2019-08-26 19:12:59 - INFO     - | Get:7 file:/isar-apt isar/main arm64 
>>> u-boot-tools arm64 2019.01-atf1.6 [93.1 kB]
>>> [...]
>>> 2019-08-26 19:12:59 - INFO     - | Setting up python-minimal (2.7.13-2) ...
>>> 2019-08-26 19:12:59 - INFO     - | dpkg: error: cannot stat pathname 
>>> '/tmp/apt-dpkg-install-iVLVJB/57-u-boot-tools_2019.01-atf1.6_arm64.deb': No 
>>> such file or directory
>>> 2019-08-26 19:12:59 - INFO     - | E: Sub-process /usr/bin/dpkg returned an 
>>> error code (2)
>>
>> Wow, I remember having seen this task failing in the past, but not such
>> straight ENOENT. Is this reproducible?
> 
> Only by chance, and only on our CI server so far. That's likely qualifies as "no".
> 

Just had this two times in a row now, so I dug deeper: The failure was 
always around self-built u-boot-tools and two images pulling that in 
more or less in parallel to a third one generating that package once 
again. All are sharing the same u-boot version for custom builds, thus 
u-boot-tools gets deployed multiple times.

Looking at our locking around isar-apt I noticed that we hold the lock 
for install_imager_deps, but we do not for rootfs_install. Why should 
both be different? I played with a shared isar-apt lock around 
rootfs_install, to avoid massive serialization, and that didn't make the 
issue show up again, so far. But does it make sense?

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index c7e0435..5ef66ee 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -135,6 +135,9 @@ do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMA
 do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config"
 do_rootfs_install[deptask] = "do_deploy_deb"
 python do_rootfs_install() {
+    lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
+                             shared=True)
+
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
     install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or "").split()
 
@@ -155,6 +158,8 @@ python do_rootfs_install() {
         progress_reporter.next_stage()
         bb.build.exec_func(cmd, d)
     progress_reporter.finish()
+
+    bb.utils.unlockfile(lock)
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
 

The other issue is that u-boot-tools can get built and deployed multiple 
times if multiple custom u-boots are generated. That may even lead to 
underministic results when the builds use different u-boot sources. I 
will add some control to the recipe to select which one shall deploy the 
tools in case there are multiple options. I suspect we will see similar 
issues when the kernel starts building tools...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux