From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6729573555756859392 X-Received: by 2002:a2e:6344:: with SMTP id x65mr5553411ljb.59.1567411720570; Mon, 02 Sep 2019 01:08:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:1285:: with SMTP id 5ls411613ljs.11.gmail; Mon, 02 Sep 2019 01:08:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqyWO81v+YLWo73K3+Ss7JwAYRHLPf36bdRup5wwJjjJVaxIH08jfoS3xatdWICdyNHAO+PY X-Received: by 2002:a2e:819:: with SMTP id 25mr15547234lji.142.1567411719930; Mon, 02 Sep 2019 01:08:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1567411719; cv=none; d=google.com; s=arc-20160816; b=AFnvAuMvXj3HiqymOgm3ZSJHeLmBSAxdHJOIjtETPyLloPyOZZRoBVroW0tfmdOyYc gztrCV3/lI+2GAy82f7k3qvJ20KqzGflCMmlE5aUZUvNtERIG13V1K1oIxfghsPfPGBh AQxbUapsDJ6t5cYoO1qSFeTxmlRB1hIpADNxeIpVnxUnxc3z7OM2eV5ONJQ1Huidl+er /EGY182br98YBsC1k8ZqlRw8XIvcyLb3MOOZ2n5PTwN+Ad635TLKSqfR9VurNzx+epDe xb/sXRB0FwFRDG6Uj+W+Rd8i71xPaXKOjH4uyvr2roX+1cnXPYYuo2slPDiug1FaL0pm cA0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:cc:references:to:from:subject; bh=0h3qSX8ccpsOTUT66OHS7qHIe7BdqL8OHFBsdzaYuqo=; b=WqnJLu7eObm7090gAaSygJ1rX4ZoC/2pQZ/4k9TOlqdTfo57se3CfKxgiOAxeE8/75 3nyEle2dHKigMYGrRz7tVO16+qcGIKTN5OmC8Z6UofMi/9JVADDRuF3NbgubVYfUHaFe r1+V/pGALkKtKV1d0/YPE8qILJZ5mkkzLmzm9QoACFUNlCi4ToxlNFdE9VGe+yb3brmw FDCogJfPWKO4d5EKbS1BQWzCUhAubhsLV9bT0be84XrUqzk/Kb+Mva+DcPA5dWHFTw6R k9ZwdLsUDKLNhFVS1MNNEezjaWz+BATKeU4OqQoirCMKDrVYlHA10satB9twrRR0hYBX vbXA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id m20si729362lfh.1.2019.09.02.01.08.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Sep 2019 01:08:39 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x8288dKU027057 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 2 Sep 2019 10:08:39 +0200 Received: from [167.87.32.192] ([167.87.32.192]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x8288cgK020342; Mon, 2 Sep 2019 10:08:38 +0200 Subject: isar-apt locking: install_imager_deps vs rootfs_install (was: Re: dpkg: error: cannot stat pathname '/tmp/apt-dpkg-install...) From: Jan Kiszka To: isar-users References: <20190826205952.GI6062@yssyq.m.ilbers.de> <136f3300-8dc3-5f42-13fc-4c04414f7801@siemens.com> Cc: Baurzhan Ismagulov , Cedric Hombourger Message-ID: <0f8729f3-68e0-f6af-b75f-b7729078a039@siemens.com> Date: Mon, 2 Sep 2019 10:08:38 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <136f3300-8dc3-5f42-13fc-4c04414f7801@siemens.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: 5E3g7qQMXIQ+ On 26.08.19 23:28, Jan Kiszka wrote: > On 26.08.19 22:59, Baurzhan Ismagulov wrote: >> On Mon, Aug 26, 2019 at 10:22:21PM +0200, Jan Kiszka wrote: >>> 2019-08-26 19:12:59 - ERROR    - ERROR: >>> mc:ultra96-jailhouse-demo:demo-image-1.0-r0 do_rootfs_install: Function >>> failed: rootfs_install_pkgs_install (log file is located at >>> /builds/ebsy/debian/build/tmp/work/jailhouse-demo-arm64/demo-image-ultra96-wic-img/1.0-r0/temp/log.do_rootfs_install.60341) >>> >>> [...] >>> 2019-08-26 19:12:59 - INFO     - | DEBUG: Executing shell function >>> rootfs_install_pkgs_install >>> [...] >>> 2019-08-26 19:12:59 - INFO     - | Get:7 file:/isar-apt isar/main arm64 >>> u-boot-tools arm64 2019.01-atf1.6 [93.1 kB] >>> [...] >>> 2019-08-26 19:12:59 - INFO     - | Setting up python-minimal (2.7.13-2) ... >>> 2019-08-26 19:12:59 - INFO     - | dpkg: error: cannot stat pathname >>> '/tmp/apt-dpkg-install-iVLVJB/57-u-boot-tools_2019.01-atf1.6_arm64.deb': No >>> such file or directory >>> 2019-08-26 19:12:59 - INFO     - | E: Sub-process /usr/bin/dpkg returned an >>> error code (2) >> >> Wow, I remember having seen this task failing in the past, but not such >> straight ENOENT. Is this reproducible? > > Only by chance, and only on our CI server so far. That's likely qualifies as "no". > Just had this two times in a row now, so I dug deeper: The failure was always around self-built u-boot-tools and two images pulling that in more or less in parallel to a third one generating that package once again. All are sharing the same u-boot version for custom builds, thus u-boot-tools gets deployed multiple times. Looking at our locking around isar-apt I noticed that we hold the lock for install_imager_deps, but we do not for rootfs_install. Why should both be different? I played with a shared isar-apt lock around rootfs_install, to avoid massive serialization, and that didn't make the issue show up again, so far. But does it make sense? diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index c7e0435..5ef66ee 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -135,6 +135,9 @@ do_rootfs_install[vardeps] = "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMA do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config" do_rootfs_install[deptask] = "do_deploy_deb" python do_rootfs_install() { + lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock", + shared=True) + configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split() install_cmds = (d.getVar("ROOTFS_INSTALL_COMMAND", True) or "").split() @@ -155,6 +158,8 @@ python do_rootfs_install() { progress_reporter.next_stage() bb.build.exec_func(cmd, d) progress_reporter.finish() + + bb.utils.unlockfile(lock) } addtask rootfs_install before do_rootfs_postprocess after do_unpack The other issue is that u-boot-tools can get built and deployed multiple times if multiple custom u-boots are generated. That may even lead to underministic results when the builds use different u-boot sources. I will add some control to the recipe to select which one shall deploy the tools in case there are multiple options. I suspect we will see similar issues when the kernel starts building tools... Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux