From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7026321669488640000 X-Received: by 2002:a05:6512:2291:: with SMTP id f17mr47444035lfu.253.1636023928496; Thu, 04 Nov 2021 04:05:28 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:7606:: with SMTP id r6ls908681ljc.6.gmail; Thu, 04 Nov 2021 04:05:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwumF1pB17n07MT+dV7MzyX8NyK64HZiKLosMKnZKJQXrd+q/X+r0SoAg6rKLSA1mF2Mk2F X-Received: by 2002:a2e:b5d2:: with SMTP id g18mr19396998ljn.282.1636023927369; Thu, 04 Nov 2021 04:05:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1636023927; cv=none; d=google.com; s=arc-20160816; b=vaVTFGJHL/9NoH8wDR2NL+O7G8IV1Re/7RLHSoZWfRqs7UWsZLhzqzVpIFg8lN227R hfIYgPmv8YoANKiZt2v2UgGZU7R4wuiEi/9YyEp6GhF5a/taWai+N7vY+A/ct0wao+2g jiZhxeAaw2orAkugtXzF2K0Rvt4NZ6MOnIhDYK0xlcnSRXiaJ9IUkvzqoJmbGCQKtt8h USKygsiKz4rlKtlnRdpvPVF0gtzhtuFbHJYdpiK9Tuq3xZIN0h4/rlVQV0CyHPlTS2fi nxSf/6+QXJ22gmZitR3GS5+EjPFXkS+jcMuiJ3JOeSA2GavCh5ud/qnx2xTuLcqe/w2M 3zmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=QFPtHw563dN5+QztOtiHEaD3YQpiyuxr5e9jKxC/n64=; b=paUDl5w1AsXiMPJaZ1AqxtKrnWsk4eGvgLZk7f1YX7qe8c5NB9w+jrK3hsRBisYNMu CnJO7ZTLMU3dAHyrx+olC6QRqCLH0VglTui6VfUMsbyYHBwfQEizELOtyyOeiTK5Rsbb Gva2eG8aXjmr/7olOjs3QAnvfcOPf9ka8bhhBID+mvA9ls39e7ENkxmdJ/JMeKKNEgxA w/IS8M2TIx7fu1iK0c5Bcu5JL4qN+LyAfV/qwxFofJA3Ql2Gz4RB9AfPTInnd9KxoPnm bFa/WwIBjbtRnNj7ptXfPS6iXPhqnx1i2SHuHMmRG+Ffk7AMMHZzn9BHPhpPt4jiWt2f bUOw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of felix.moessbauer@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=felix.moessbauer@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id e11si329721lfr.10.2021.11.04.04.05.27 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 04 Nov 2021 04:05:27 -0700 (PDT) Received-SPF: pass (google.com: domain of felix.moessbauer@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of felix.moessbauer@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=felix.moessbauer@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 1A4B5QMJ012921 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 4 Nov 2021 12:05:26 +0100 Received: from MD1T0KAC-VM.ad001.siemens.net (md1t0kac.ad001.siemens.net [139.25.68.224]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 1A4B5IRI028841; Thu, 4 Nov 2021 12:05:26 +0100 From: Felix Moessbauer To: isar-users@googlegroups.com Cc: henning.schild@siemens.com, adriaan.schmidt@siemens.com, Felix Moessbauer Subject: [PATCH v4 1/2] always invoke ISAR_RELEASE_CMD to invalidate downstream tasks on change Date: Thu, 4 Nov 2021 12:05:06 +0100 Message-Id: <20211104110507.2358692-2-felix.moessbauer@siemens.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211104110507.2358692-1-felix.moessbauer@siemens.com> References: <20211104110507.2358692-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: 8dIQrdrxqK2P The command in ISAR_RELEASE_CMD is used to add version information of the image under /etc/os-release. As we cannot predict the output of the command, we cannot cache it. Prior to this patch, this situation was just accepted resulting in misleading build-ids on incremental builds. When caching artifacts across builds (e.g. using sstate), this also affects fresh builds. The patch ensures that ISAR_RELEASE_CMD is always invoked and the output is stored in the variable IMAGE_BUILD_ID. This variable is added as a vardep in the rootfs_postprocess task, so changes invalidate all downstream tasks. By that, we ensure that images always contain correct release information. Signed-off-by: Felix Moessbauer --- RECIPE-API-CHANGELOG.md | 13 ++++++++ meta/classes/image-postproc-extension.bbclass | 4 +-- meta/classes/image.bbclass | 32 ++++++++++--------- meta/classes/rootfs.bbclass | 2 +- 4 files changed, 33 insertions(+), 18 deletions(-) diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 7312d4d..48458a5 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -301,3 +301,16 @@ Kernel update with "apt-get" will not work since bootloader configuration will not be updated. It used to "kind of work" for grub and efi, that hack is gone. When using the plugins it is advised to name the partition "/boot" and to exclude boot from the follwing rootfs to not waste space. + +### ISAR_RELEASE_CMD is executed on every build + +The `ISAR_RELEASE_CMD` used to be only executed when changes to the target rootfs where made. +This lead to outdated data in `/etc/os-release` on incremental builds. + +The command is now executed on every build. +Downstream tasks are invalidated in case the output of the command changes. + +When using a custom command it is advised to stick to the following rules: + +- no visible side effects (the command is executed multiple times during a build) +- idempotence (always emit the same output for a single build, i.e. no timestamp) diff --git a/meta/classes/image-postproc-extension.bbclass b/meta/classes/image-postproc-extension.bbclass index 3f00c21..6e6b257 100644 --- a/meta/classes/image-postproc-extension.bbclass +++ b/meta/classes/image-postproc-extension.bbclass @@ -46,11 +46,11 @@ image_postprocess_configure() { } ROOTFS_POSTPROCESS_COMMAND =+ "image_postprocess_mark" +ROOTFS_POSTPROCESS_VARDEPS =+ "IMAGE_BUILD_ID" image_postprocess_mark() { - BUILD_ID=$(get_build_id) update_etc_os_release \ - --build-id "${BUILD_ID}" --variant "${DESCRIPTION}" --version "${PV}" + --build-id "${IMAGE_BUILD_ID}" --variant "${DESCRIPTION}" --version "${PV}" } ROOTFS_POSTPROCESS_COMMAND =+ "image_postprocess_machine_id" diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 5a0f32e..36883a8 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -92,21 +92,23 @@ def get_rootfs_size(d): return base_size + rootfs_extra * 1024 -# here we call a command that should describe your whole build system, -# this could be "git describe" or something similar. -# set ISAR_RELEASE_CMD to customize, or override do_mark_rootfs to do something -# completely different -get_build_id() { - if [ $(echo ${BBLAYERS} | wc -w) -ne 2 ] && - [ "${ISAR_RELEASE_CMD}" = "${ISAR_RELEASE_CMD_DEFAULT}" ]; then - bbwarn "You are using external layers that will not be" \ - "considered in the build_id. Consider changing" \ - "ISAR_RELEASE_CMD." - fi - if ! ( ${ISAR_RELEASE_CMD} ) 2>/dev/null; then - bbwarn "\"${ISAR_RELEASE_CMD}\" failed, returning empty build_id." - echo "" - fi +# the IMAGE_BUILD_ID depends on external conditions injected via +# ISAR_RELEASE_CMD. By that, we have to compute the value +# on each invocation +python() { + bblayers = d.getVar('BBLAYERS', True) + release_cmd = d.getVar('ISAR_RELEASE_CMD', True) + if len(bblayers.split()) != 2: + if release_cmd == d.getVar('ISAR_RELEASE_CMD_DEFAULT', True): + bb.warn('You are using external layers that will not be ' + 'considered in the build_id. Consider changing ' + 'ISAR_RELEASE_CMD.') + try: + out,err = bb.process.run(release_cmd) + d.setVar('IMAGE_BUILD_ID', out.replace('\n', '')) + except(bb.process.ExecutionError): + bb.warn(f'"{release_cmd}" failed, returning empty build_id.') + d.setVar('IMAGE_BUILD_ID', '') } python set_image_size () { diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass index 20ccb00..e8c7fa0 100644 --- a/meta/classes/rootfs.bbclass +++ b/meta/classes/rootfs.bbclass @@ -239,7 +239,7 @@ rootfs_export_dpkg_status() { '${ROOTFS_DPKGSTATUS_DEPLOY_DIR}'/'${PF}'.dpkg_status } -do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}" +do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND} ${ROOTFS_POSTPROCESS_VARDEPS}" python do_rootfs_postprocess() { # Take care that its correctly mounted: bb.build.exec_func('rootfs_do_mounts', d) -- 2.30.2