public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
From: Jan Kiszka <jan.kiszka@siemens.com>
To: isar-users <isar-users@googlegroups.com>
Subject: [PATCH v5 16/27] rootfs: Take isar-apt shared lock during repository operations
Date: Sun, 22 Sep 2019 20:17:00 +0200	[thread overview]
Message-ID: <ad401bee6c6e8aff94b829da510079bce2dc1221.1569176231.git.jan.kiszka@siemens.com> (raw)
In-Reply-To: <cover.1569176231.git.jan.kiszka@siemens.com>
In-Reply-To: <cover.1569176231.git.jan.kiszka@siemens.com>

From: Jan Kiszka <jan.kiszka@siemens.com>

We take this lock already for install_imager_deps, but we also need it
conceptually for the rootfs installation. Also here, pulling from
isar-apt can race with not-yet finished deployments of parallel builds,
even if the target package is not shared. See e9a5d8adf6b5 for further
details.

To avoid holding the lock across the whole lengthy package installation,
introduce command flags that define the beginning and the end of the
locked section so that do_rootfs_install can manage that.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/rootfs.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 5bafe32..9bada0c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -93,6 +93,7 @@ EOSUDO
 
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_update"
 rootfs_install_pkgs_update[weight] = "5"
+rootfs_install_pkgs_update[isar-lock] = "acquire-before"
 rootfs_install_pkgs_update() {
     sudo -E chroot '${ROOTFSDIR}' /usr/bin/apt-get update \
         -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
@@ -108,6 +109,7 @@ rootfs_install_resolvconf() {
 
 ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
 rootfs_install_pkgs_download[weight] = "600"
+rootfs_install_pkgs_download[isar-lock] = "release-after"
 rootfs_install_pkgs_download() {
     sudo -E chroot '${ROOTFSDIR}' \
         /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
@@ -153,7 +155,15 @@ python do_rootfs_install() {
 
     for cmd in cmds:
         progress_reporter.next_stage()
+
+        if (d.getVarFlag(cmd, 'isar-lock') or "") == "acquire-before":
+            lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
+                                     shared=True)
+
         bb.build.exec_func(cmd, d)
+
+        if (d.getVarFlag(cmd, 'isar-lock') or "") == "release-after":
+            bb.utils.unlockfile(lock)
     progress_reporter.finish()
 }
 addtask rootfs_install before do_rootfs_postprocess after do_unpack
-- 
2.16.4


  parent reply	other threads:[~2019-09-22 18:17 UTC|newest]

Thread overview: 78+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-22 18:16 [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 01/27] buildchroot: Properly deploy the build result Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 02/27] sdk: Use clean-package-cache rather than open-coded cleanup Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 03/27] sdk: Deploy earlier Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 04/27] Remove unneeded PF tweakings Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 05/27] image: Stop changing PF, tune WORKDIR and STAMP directly Jan Kiszka
2019-09-25 12:47   ` [PATCH v6 " Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 06/27] image: Make WORKDIR and STAMPs IMAGE_TYPE-specific Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 07/27] Adjust STAMP variables to use PN, rather than PF Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 08/27] Align WORKDIR structure with OE Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 09/27] isar-bootstrap: Unshare host bootstraps across different target distros Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 10/27] isar-bootstrap: Clean up and fix rebuild Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 11/27] gitlab-ci: Keep logs as artifacts on failure Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 12/27] Detect false sharing of recipes Jan Kiszka
2019-09-24 18:02   ` [PATCH v6 " Jan Kiszka
2019-09-29 14:57     ` Baurzhan Ismagulov
2019-09-30  6:00       ` Jan Kiszka
2019-09-30  9:56         ` Baurzhan Ismagulov
2019-09-30 10:14           ` Jan Kiszka
2019-09-30 14:53             ` Baurzhan Ismagulov
2019-09-30 15:26               ` Jan Kiszka
2019-09-30 15:39                 ` Baurzhan Ismagulov
2019-09-30 16:34                   ` Jan Kiszka
2019-09-30 17:20                     ` Baurzhan Ismagulov
2019-09-30 17:26                       ` Jan Kiszka
2019-09-30 20:29                         ` Baurzhan Ismagulov
2019-09-22 18:16 ` [PATCH v5 13/27] dpkg-base: Move do_deploy_deb before do_build Jan Kiszka
2019-10-02  7:09   ` Baurzhan Ismagulov
2019-09-22 18:16 ` [PATCH v5 14/27] base-apt: Move do_cache_config " Jan Kiszka
2019-09-22 18:16 ` [PATCH v5 15/27] wic-img: Use private tmp dir Jan Kiszka
2019-09-22 18:17 ` Jan Kiszka [this message]
2019-09-22 18:17 ` [PATCH v5 17/27] Use deb.debian.org mirrors Jan Kiszka
2019-09-23 10:03   ` Baurzhan Ismagulov
2019-09-23 10:09     ` Jan Kiszka
2019-10-02 11:13   ` Baurzhan Ismagulov
2019-09-22 18:17 ` [PATCH v5 18/27] linux-custom: Control linux-libc-dev deployment manually Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 19/27] dpkg-base: Wait for umount to succeed Jan Kiszka
2019-09-30 11:07   ` Baurzhan Ismagulov
2019-09-30 12:22     ` Jan Kiszka
2019-09-30 13:33       ` Baurzhan Ismagulov
2019-10-07 15:48   ` Baurzhan Ismagulov
2019-09-22 18:17 ` [PATCH v5 20/27] dpkg-base: Permit multiple clean runs Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 21/27] buildchroot: Factor out common packages Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 22/27] buildchroot: Do not build cross when there are no arch-specific outputs Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 23/27] Add git-buildpackage support Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 24/27] dbpg-raw: Only list top-level files in <pn>.install Jan Kiszka
2019-09-23  9:52   ` Baurzhan Ismagulov
2019-09-23 10:29     ` Jan Kiszka
2019-09-23 10:30       ` Baurzhan Ismagulov
2019-09-22 18:17 ` [PATCH v5 25/27] Add OE class and library elements for terminal support Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 26/27] dpkg-base: Add devshell target Jan Kiszka
2019-09-22 18:17 ` [PATCH v5 27/27] linux-custom: Add libncurses-dev to KBUILD_DEPENDS Jan Kiszka
2019-09-23  6:33 ` [PATCH v5 00/27] Pending patches, fix for isar-bootstrap, devshell Baurzhan Ismagulov
2019-09-23  7:33   ` Jan Kiszka
2019-09-23  7:50     ` Baurzhan Ismagulov
2019-09-23  9:50 ` Baurzhan Ismagulov
2019-09-23 10:07   ` Jan Kiszka
2019-09-23 10:23     ` Baurzhan Ismagulov
2019-09-23 10:28       ` Jan Kiszka
2019-09-25  8:41         ` Baurzhan Ismagulov
2019-09-25  8:51           ` Jan Kiszka
2019-09-25  9:22             ` Baurzhan Ismagulov
2019-09-25  9:26               ` Jan Kiszka
2019-09-25  9:35                 ` Baurzhan Ismagulov
2019-09-25  9:35                 ` Jan Kiszka
2019-09-25 10:52                   ` Baurzhan Ismagulov
2019-09-25 11:05                     ` Jan Kiszka
2019-09-25 12:08                       ` Baurzhan Ismagulov
2019-09-25 12:29                         ` Jan Kiszka
2019-09-25 12:44                           ` Baurzhan Ismagulov
2019-09-23 14:24   ` Henning Schild
2019-09-28 15:46     ` Baurzhan Ismagulov
2019-09-25 10:23 ` Henning Schild
2019-09-25 11:41   ` Jan Kiszka
2019-09-30  9:29     ` Baurzhan Ismagulov
2019-09-30 10:02       ` Jan Kiszka
2019-09-30 13:00       ` Baurzhan Ismagulov
2019-10-01 14:47 ` Baurzhan Ismagulov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ad401bee6c6e8aff94b829da510079bce2dc1221.1569176231.git.jan.kiszka@siemens.com \
    --to=jan.kiszka@siemens.com \
    --cc=isar-users@googlegroups.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox