* [PATCH 2/2] meta: Cover bash isar-apt usages by isar.lock
2024-07-12 6:56 [PATCH 0/2] Protect isar-apt usages in shell Anton Mikanovich
2024-07-12 6:56 ` [PATCH 1/2] deb-dl-dir: Avoid multiple find run on export Anton Mikanovich
@ 2024-07-12 6:56 ` Anton Mikanovich
2024-07-23 7:38 ` [PATCH 0/2] Protect isar-apt usages in shell Uladzimir Bely
2 siblings, 0 replies; 4+ messages in thread
From: Anton Mikanovich @ 2024-07-12 6:56 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Cover isar-apt repo usages by ${REPO_ISAR_DIR}/isar.lock to prevent any
conflicts or races.
Introduce lockrun.py helper script to execute bash code covered by
fcntl locks in bitbake compatible way. It means the same lock files as
used in lockfiles task flag can be used with lockrun.py.
Key -r is used for shared locking on read operations.
Usage examples:
${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -c "true"
${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOF
true
EOF
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/deb-dl-dir.bbclass | 3 +-
meta/classes/image-tools-extension.bbclass | 2 +
scripts/lockrun.py | 44 ++++++++++++++++++++++
3 files changed, 48 insertions(+), 1 deletion(-)
create mode 100755 scripts/lockrun.py
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 2d8739dc..55e56c50 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -90,7 +90,8 @@ deb_dl_dir_export() {
export owner=$(id -u):$(id -g)
mkdir -p "${pc}"
- isar_debs="\$(find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print)"
+ isar_debs="$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \
+ "find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")"
flock "${pc}".lock sudo -Es << 'EOSUDO'
set -e
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 46bdf78b..e8ace8f5 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -46,6 +46,7 @@ imager_run() {
E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${schroot_dir} ${distro}
+ ${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOAPT
schroot -r -c ${session_id} -d / -u root -- sh -c " \
apt-get update \
-o Dir::Etc::SourceList='sources.list.d/isar-apt.list' \
@@ -54,6 +55,7 @@ imager_run() {
apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
--allow-unauthenticated --allow-downgrades --download-only install \
${local_install}"
+EOAPT
deb_dl_dir_export ${schroot_dir} ${distro}
schroot -r -c ${session_id} -d / -u root -- sh -c " \
diff --git a/scripts/lockrun.py b/scripts/lockrun.py
new file mode 100755
index 00000000..58a598dd
--- /dev/null
+++ b/scripts/lockrun.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+#
+# Helper script to use bitbake locks in shell
+# Copyright (c) 2024, ilbers GmbH
+
+import argparse
+import os
+import subprocess
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
+
+from bb import utils
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-f', '--file', help='Lock file name.', required=True)
+ parser.add_argument(
+ '-r', '--read', action="store_true", help='Use read (shared) locking.'
+ )
+ group = parser.add_mutually_exclusive_group(required=True)
+ group.add_argument('-c', '--command', help='Command(s) to execute.')
+ group.add_argument(
+ '-s',
+ '--shell',
+ action="store_true",
+ help='Execute commands from stdin.',
+ )
+ return parser.parse_args()
+
+
+if __name__ == "__main__":
+ args = parse_args()
+
+ with utils.fileslocked([args.file], shared=args.read):
+ if args.shell:
+ cmd = sys.stdin.read()
+ else:
+ cmd = args.command
+ try:
+ subprocess.run(cmd, check=True, shell=True)
+ except subprocess.CalledProcessError as e:
+ exit(e.returncode)
--
2.34.1
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240712065646.2156320-3-amikan%40ilbers.de.
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH 0/2] Protect isar-apt usages in shell
2024-07-12 6:56 [PATCH 0/2] Protect isar-apt usages in shell Anton Mikanovich
2024-07-12 6:56 ` [PATCH 1/2] deb-dl-dir: Avoid multiple find run on export Anton Mikanovich
2024-07-12 6:56 ` [PATCH 2/2] meta: Cover bash isar-apt usages by isar.lock Anton Mikanovich
@ 2024-07-23 7:38 ` Uladzimir Bely
2 siblings, 0 replies; 4+ messages in thread
From: Uladzimir Bely @ 2024-07-23 7:38 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On Fri, 2024-07-12 at 09:56 +0300, Anton Mikanovich wrote:
> Read operations on isar-apt should be covered by shared lock.
> This was not implemented for deb_dl_dir_export and imager_run.
> At least following CI error was caused by isar-apt race:
>
> DEBUG: Executing shell function rootfs_export_package_cache
> find: 'isar-apt/debian-bullseye-arm64/apt/debian-
> bullseye/pool/main/e/enable-fsck': No such file or directory
>
> Find call inside deb_dl_dir_export was trying to look into enable-
> fsck
> directory while is was deleted by deb_clean function of do_deploy_deb
> task of enable-fsck package.
>
> The same situation can potentially happen on imager_run call, causing
> races during image preparations.
>
> To protect isar-apt usages inside big shell functions introduce
> lockrun.py helper for using with isar-apt lock.
> More clear implementation will require tasks rewrite in python, which
> is not an option for the big downstream projects.
>
> Anton Mikanovich (2):
> deb-dl-dir: Avoid multiple find run on export
> meta: Cover bash isar-apt usages by isar.lock
>
> meta/classes/deb-dl-dir.bbclass | 9 +++--
> meta/classes/image-tools-extension.bbclass | 2 +
> scripts/lockrun.py | 44
> ++++++++++++++++++++++
> 3 files changed, 52 insertions(+), 3 deletions(-)
> create mode 100755 scripts/lockrun.py
>
> --
> 2.34.1
>
Applied to next.
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/943268259224283c363eefc66452d35cebe81fa5.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 4+ messages in thread