From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7186874795297865728 X-Received: by 2002:a05:622a:1b05:b0:3a6:8be3:301e with SMTP id bb5-20020a05622a1b0500b003a68be3301emr2824270qtb.21.1673324684042; Mon, 09 Jan 2023 20:24:44 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac8:4891:0:b0:3a6:9e8c:1506 with SMTP id i17-20020ac84891000000b003a69e8c1506ls682516qtq.4.-pod-prod-gmail; Mon, 09 Jan 2023 20:24:43 -0800 (PST) X-Google-Smtp-Source: AMrXdXvV/+BXDiM4GZ4taOPwvnpE+AbKpDm3fcC6VegEG28iOZBxpC4Wxldwt1lVEoznS/0qhF3M X-Received: by 2002:a05:622a:410d:b0:3ae:2272:e43c with SMTP id cc13-20020a05622a410d00b003ae2272e43cmr5437909qtb.29.1673324683284; Mon, 09 Jan 2023 20:24:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673324683; cv=none; d=google.com; s=arc-20160816; b=QzoJAAXXqBjJ+YWGVujXRR+Yop9f7RF86NFmSgkMOUo7qMXYNkNsesGkjZ7qWTzIHP Wqk03L8JfdynWMaZ3Jpgr9APle0xJHl4IZZlMzLql4uhodvN2AzwfCbUByxrLI9nJeTL tor4YSZUBJiiEbFSZc5gklgTq6Z66tNUKOFLiNwemVDVmDPzpDzbWkD9w5G6Sb7M2YBs FAmAoPZHHAGievS2U9kibbx2r5F7evI5EY6tV3aTIUbKE5LGr4RP5hPYr/ZFNKB2nTlA MuoanhLh/tlp4vWC7OMcjF1NRyDrQdw+pmeeUhISK5QYk3wlt3eedhCTwxsmUJtV8Qo2 skig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=wZ8ktzUjrIuL/iGInllODbIpfI6MkLc6aeFg90FuTT0=; b=FgOG9vbHvT7S/x3Zw2jQVklzfJ1XKC+nKTHKGO/hqTK7x/MaqQEJWA7EcRVmsn1SDz vPrm25GiNIXtwuTsvPUQ8f/xlIt/wMZ08X+syFbb8x3JPvgSdkEiiDDhQvQc45GSpezo TdsNtsRxAdwnfkIq+hX0tva6DpmOySsh4ojsPw0m0wHHQSWOmbRI7hrJJrSP0OCwoF4t 1Ne+IgpujACwiwKp0NpL2SWoetYBVulwiBDGZBLIC/Mdw5ZNblft8p9N87/c2f+wCuBR K0Jtkgy9HXAa/mA7ltkgXNBOzYEpP7vUdV/39FEXbHl8FjbtTb4pk9PMOMyi8gB8/USf 7AfA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Return-Path: Received: from relay04.th.seeweb.it (relay04.th.seeweb.it. [2001:4b7a:2000:18::165]) by gmr-mx.google.com with ESMTPS id h4-20020ac87764000000b0039a9b745f0asi530242qtu.5.2023.01.09.20.24.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Jan 2023 20:24:42 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) client-ip=2001:4b7a:2000:18::165; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of roberto.foglietta@linuxteam.org designates 2001:4b7a:2000:18::165 as permitted sender) smtp.mailfrom=roberto.foglietta@linuxteam.org Received: from localhost.localdomain (unknown [IPv6:2a02:8071:3187:7b80:2373:f2dd:a867:c892]) by m-r1.th.seeweb.it (Postfix) with ESMTPA id D75A71F8BE; Tue, 10 Jan 2023 05:24:40 +0100 (CET) From: roberto.foglietta@linuxteam.org To: isar-users@googlegroups.com Cc: roberto.foglietta@gmail.com Subject: [PATCH v7] image tools ext.: start_imager_session not break the rebuild, v7 Date: Tue, 10 Jan 2023 05:24:38 +0100 Message-Id: <20230110042438.1226639-1-roberto.foglietta@linuxteam.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: Oc3Pfec/Hmo5 From: "Roberto A. Foglietta" image tools extension, start_imager_session, bugfix and improvment The original Ilbers ISAR code was buggy but the 1st fix do not improve the performances nore solved completely the problem which has been defintely addressed in part.2 - then this p.3 aims to improve the performance brutally deleting the closing schroot session without relies on the slower and sometimes failing schroot -e python script. p.1: bugfix, initial p.2: bugfix completed, code rationalisation p.3: performance improvement bypassing schroot -e and Co. v.4: the squash of the three parts v.5: deep reworking v.6: bugfixing v.7: removed the code left behind but not used Signed-off-by: Roberto A. Foglietta --- meta/classes/image-tools-extension.bbclass | 39 ++++++++-------------- meta/classes/isar-events.bbclass | 31 +++++++++++------ meta/classes/sbuild.bbclass | 3 +- 3 files changed, 37 insertions(+), 36 deletions(-) diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass index 48a93c8..9c46b65 100644 --- a/meta/classes/image-tools-extension.bbclass +++ b/meta/classes/image-tools-extension.bbclass @@ -60,18 +60,20 @@ python do_start_imager_session() { sbuild_chroot = d.getVar("SBUILD_CHROOT", True) session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) try: - if subprocess.run("schroot -l --all-sessions | grep %s" % session_id, shell=True).returncode: - subprocess.run("schroot -b -c %s -n %s" % (sbuild_chroot, session_id), shell=True, check=True) - bb.debug(2, "Open schroot session %s" % session_id) - else: - subprocess.run("schroot --recover-session -c %s" % session_id, shell=True, check=True) - bb.debug(2, "Reuse schroot session %s" % session_id) - d.setVar("SCHROOT_OPEN_SESSION_ID", session_id) + bb.debug(2, "Opening schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -b -c %s -n %s -- printenv -0 SCHROOT_ALIAS_NAME" + % (sbuild_chroot, session_id), shell=True, check=True) except subprocess.CalledProcessError as err: - subprocess.run("schroot -e -c %s" % session_id, shell=True) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) - bb.fatal("Could not create schroot session: %s" % err.output.decode('utf-8') if err.output else "") + try: + bb.debug(2, "Reusing schroot session %s" % sbuild_chroot) + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" + % session_id, shell=True, check=True) + except subprocess.CalledProcessError as err: + bb.debug(2, "Closing schroot session %s (%s)" % (sbuild_chroot, session_id)) +# bb.build.exec_func("stop_imager_session", d) + bb.build.exec_func("stop_schroot_session", d) + if 'id' in locals(): + d.setVar("SBUILD_CHROOT", id) } addtask start_imager_session before do_stop_imager_session after do_rootfs_finalize @@ -79,20 +81,7 @@ do_stop_imager_session[depends] = "${SCHROOT_DEP}" do_stop_imager_session[nostamp] = "1" do_stop_imager_session[network] = "${TASK_USE_SUDO}" python do_stop_imager_session() { - import subprocess - session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) - try: - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) - subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) - except subprocess.CalledProcessError as err: - bb.error("Could not close schroot session %s: %s" % (session_id, err.output.decode('utf-8')) if err.output else "") - finally: - if 'id' in locals(): - d.setVar("SBUILD_CHROOT", id) - bb.build.exec_func("remove_mounts", d) - bb.build.exec_func("schroot_delete_configs", d) + bb.build.exec_func("stop_schroot_session", d) } addtask stop_imager_session before do_deploy after do_image diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass index 5343d4b..7f68ffa 100644 --- a/meta/classes/isar-events.bbclass +++ b/meta/classes/isar-events.bbclass @@ -42,6 +42,25 @@ task_failed[eventmask] = "bb.build.TaskFailed" addhandler build_completed +python stop_schroot_session() { + session_id = d.getVar("IMAGER_SCHROOT_SESSION_ID", True) + import subprocess + try: + id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, + shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') + bb.debug(2, "Close schroot session %s (%s)" % (session_id, id)) + subprocess.run("schroot -e -c %s" % session_id, shell=True, check=True) + d.setVar("SBUILD_CHROOT", id) + bb.build.exec_func("remove_mounts", d) + bb.build.exec_func("schroot_delete_configs", d) + except subprocess.CalledProcessError as err: + bb.error(2, "Could not close schroot session %s: %s" + % (session_id, err.output.decode('utf-8')) if err.output else "") + udir = d.getVar("SCHROOT_OVERLAY_DIR", True) + subprocess.run("sudo rm -rf --one-file-system %s/%s /var/lib/schroot/session/%s" + % (udir, session_id, session_id), shell=True) +} + python build_completed() { import subprocess @@ -56,16 +75,8 @@ python build_completed() { shell=True, stdout=subprocess.PIPE).stdout.decode('utf-8') for line in sessions.splitlines(): session_id = line.split(':', 1)[1] - bb.debug(1, 'Closing imager session %s' % session_id) - id = subprocess.run("schroot -d / -r -c %s -- printenv -0 SCHROOT_ALIAS_NAME" % session_id, - shell=True, check=True, stdout=subprocess.PIPE).stdout.decode('utf-8') - if id: - subprocess.run('schroot --recover-session -c %s' % session_id, shell=True, check=True) - subprocess.run('schroot -e -c %s' % session_id, shell=True, check=True) - if 'id' in locals(): - d.setVar('SBUILD_CHROOT', id) - bb.build.exec_func('remove_mounts', d) - bb.build.exec_func('schroot_delete_configs', d) + bb.debug(2, 'Closing imager session %s' % session_id) + bb.build.exec_func("stop_schroot_session", d) with open('/proc/mounts') as f: for line in f.readlines(): diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass index 06c01d6..25c3e08 100644 --- a/meta/classes/sbuild.bbclass +++ b/meta/classes/sbuild.bbclass @@ -26,6 +26,7 @@ SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${@os.getp SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}" SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}" +SCHROOT_OVERLAY_DIR ?= "${TMPDIR}/schroot-overlay" SBUILD_CONFIG="${WORKDIR}/sbuild.conf" @@ -54,7 +55,7 @@ root-groups=root,sbuild source-root-users=${SCHROOT_USER} source-root-groups=root,sbuild union-type=overlay -union-overlay-directory=${TMPDIR}/schroot-overlay +union-overlay-directory=${SCHROOT_OVERLAY_DIR} preserve-environment=true EOF -- 2.34.1