From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6721684426774806528 X-Received: by 2002:adf:f744:: with SMTP id z4mr19179198wrp.211.1565597042967; Mon, 12 Aug 2019 01:04:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:5709:: with SMTP id a9ls30020209wrv.12.gmail; Mon, 12 Aug 2019 01:04:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqwig5FeX5kOWIhbSUS7UMG1d+93v5hNHKdYRXRN47tQmwMlDPMy0NBi0o2ELXJOZwgrmsBl X-Received: by 2002:a5d:48cf:: with SMTP id p15mr40638860wrs.151.1565597042521; Mon, 12 Aug 2019 01:04:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565597042; cv=none; d=google.com; s=arc-20160816; b=EUrxIeJm8g2GaQ55yfyw75xDQ5oNPAxrIrwVodCrmdiHInLB7jRpbByInky9rm/Lr6 It7Blc6U1cSgY0f54R8UVcuJpgoW7q0OkF0nckMVWu5+scIEm0SBr4TS1aYaB7SthPov 1xrHi1GJgjaD+8FNJZfy7Q/PsCAr0Yq6aYpb+WS/ey6dwDT+XalMM2+4mvoN3md1eUEL gZ05nWVKB4kTaPIfYx2WdCiWhPR/wn2+7z/Tzt2wmqyhpsvv2NcpRB3mlJtYCNEU0cNY wETNXbnlYAdUoX2ot20Yrt2tvssWwIX9PJHkX577c2ZJeGhNB7sZH5BOUzqeNiFyrJd6 VFAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=WRVFZ7rKX8hRnRafS7FKQwtyE75aepBkevr04t5W7co=; b=RYiSY82xUDHgTjqlcMtMnS4WOF2+BVpw2JQhMo66z75TZdemlGEz4ygxeVvLOhleBf jTWK+Nn1ce2nPUHh9qMtlhT9cK6TZrAPbZlUhzP6qUHpdlhrrU1O4tzxtAgqmvSDWnZL tGVEN1QCEocfjdMD6j/TKMJSzaObQV/RbyHpwCUU/Ioj0q+nX04oVTpojJZG58n4hLKi 9cDjN5K1ihKwQ1cFRg2e/qSIdA9Zuxr5cu6ZW4I9tuMHKxiHXEVQTq2QGhkZ56ZzFN/X i/WhZCKV49t/EcybJTxNQEUgcIkuo3S/JNm1HL/hQO76Htr6P178kmsttlF0WZT5Vl3H lT/Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id o4si1952553wrp.4.2019.08.12.01.04.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Aug 2019 01:04:02 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x7C842Jk006740 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 12 Aug 2019 10:04:02 +0200 Received: from [139.25.69.208] (linux-ses-ext02.ppmd.siemens.net [139.25.69.208]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x7C841DZ011500; Mon, 12 Aug 2019 10:04:01 +0200 Subject: Re: [PATCH v3] meta/classes: generate bill of material from image To: "[ext] Q. Gylstorff" , isar-users@googlegroups.com References: <2c58eae5-4d77-776f-3d4e-5fda95dc27d5@siemens.com> <20190809103046.10493-1-Quirin.Gylstorff@siemens.com> From: Claudius Heine Message-ID: <8c27aed7-56b7-89f8-f84d-093334627dae@siemens.com> Date: Mon, 12 Aug 2019 10:04:01 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.1 MIME-Version: 1.0 In-Reply-To: <20190809103046.10493-1-Quirin.Gylstorff@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: csEm7luWcNmi Hi Quirin, On 09/08/2019 12.30, [ext] Q. Gylstorff wrote: > From: Quirin Gylstorff > > To create products it is necessary to have a list > of used packages for clearance and to security monitoring. > To get a simple list of packages use dpkg-query and generate > a list with the following pattern: > > source name| source version | binary package name | binary version > > The list is stored in ${IMAGE_FULLNAME}.rootfs.manifest > > Remove the feature with: > ROOTFS_FEATURES_remove = "generate-manifest" > > Signed-off-by: Quirin Gylstorff > --- > Changes: > v3: > Add list of manifest for buildchroot manifest > This list can be exdent to add additional output generators > v2: > use FEATURE instead of own variable > > .../image-package-list-extension.bbclass | 66 +++++++++++++++++++ > meta/classes/image.bbclass | 3 +- > 2 files changed, 68 insertions(+), 1 deletion(-) > create mode 100644 meta/classes/image-package-list-extension.bbclass > > diff --git a/meta/classes/image-package-list-extension.bbclass b/meta/classes/image-package-list-extension.bbclass > new file mode 100644 > index 0000000..11896f1 > --- /dev/null > +++ b/meta/classes/image-package-list-extension.bbclass > @@ -0,0 +1,66 @@ > +# This software is a part of ISAR. > +# Copyright (C) Siemens AG, 2019 > +# > +# SPDX-License-Identifier: MIT > + > +MANIFESTS ?= "target build" > +MANIFEST_build[rootfs] ?= "/var/lib/dpkg" > +MANIFEST_target[rootfs] ?= "${PP_ROOTFS}/var/lib/dpkg" Have you planned additional flags for this? Currently I think that this mechanism is a bit of an overkill for just two variables. But since you touched this now and are the second users of this, I have further comments ;). > + > +def gen_manifests_array(d, listname, entryname, flags, verb_flags=None): > + from itertools import chain > + > + entries = (d.getVar(listname, True) or "").split() > + return " ".join( > + ":".join( > + chain( > + (entry,), > + ( > + (",".join( > + ( > + d.getVarFlag(entryname + "_" + entry, flag, True) or "" > + ).split() > + ) if flag not in (verb_flags or []) else ( > + d.getVarFlag(entryname + "_" + entry, flag, True) or "" > + )).replace(":","=") > + for flag in flags > + ), > + ) > + ) > + for entry in entries Is this a 1:1 copy from image-account-extension.bbclass? If so, then can you put those implementations together to avoid code duplication? I also have to say, I am not 100% happy with that code. While it is very efficient, it might be not that maintainable compared to simple loops that build the output string using string operators or StringIO. Those are probably easier to understand for people not that fluent in python. > + ) > + > + > +IMAGE_MANIFESTS =+ "${@gen_manifests_array(d, 'MANIFESTS', 'MANIFEST', ['rootfs'])}" > + > +image_generate_manifest[dirs] = "${DEPLOY_DIR_IMAGE}" > +image_generate_manifest() { > + image_do_mounts > + list='${@" ".join(d.getVar('IMAGE_MANIFESTS', True).split())} ' > + while true; do > + list_rest="${list#*:* }" > + entry="${list%%${list_rest}}" > + list="${list_rest}" > + > + if [ -z "${entry}" ]; then > + break > + fi > + # Add colon to the end of the entry and remove trailing space: > + entry="${entry% }:" > + > + # Decode entries: > + name="${entry%%:*}" > + entry="${entry#${name}:}" > + > + rootfs="${entry%%:*}" > + entry="${entry#${rootfs}:}" I also see here some ways to avoid code duplication. Maybe generate this shell code in a python function? Something like: ${@gen_shell_list_processing('IMAGE_MANIFESTS', ['name', 'rootfs'], """ sudo -E chroot ${BUILDCHROOT_DIR} \ /usr/bin/dpkg-query --admindir="$rootfs" \ -f '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W\ ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest """)} Not sure about the shell part. BB seems to not handle line breaks in inline python very well. To move it outside of the inline would also be possible: ${@gen_shell_list_processing_start('IMAGE_MANIFESTS', ['name', 'rootfs'])} sudo -E chroot ${BUILDCHROOT_DIR} \ /usr/bin/dpkg-query --admindir="$rootfs" \ -f '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W\ ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest ${@gen_shell_list_processing_end()} gen_shell_list_processing_end would just return 'done;' but it makes is more flexible in the future... maybe, IDK. But having 'done' directly there would also be ok. > + > + > + sudo -E chroot ${BUILDCHROOT_DIR} \ > + /usr/bin/dpkg-query --admindir="$rootfs" \ > + -f '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W > \ > + ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest Does this also work if you just specify 'dpkg-query' without '/usr/bin/'? Cheers, Claudius > + done > +} > +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'image_generate_manifest', '', d)}" > + > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ec6bd39..60dd9fb 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -58,7 +58,7 @@ image_do_mounts() { > } > > ROOTFSDIR = "${IMAGE_ROOTFS}" > -ROOTFS_FEATURES += "copy-package-cache clean-package-cache finalize-rootfs" > +ROOTFS_FEATURES += "copy-package-cache clean-package-cache finalize-rootfs generate-manifest" > ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" > > inherit rootfs > @@ -68,6 +68,7 @@ inherit image-tools-extension > inherit image-postproc-extension > inherit image-locales-extension > inherit image-account-extension > +inherit image-package-list-extension > > # Extra space for rootfs in MB > ROOTFS_EXTRA ?= "64" > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de