From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6721684426774806528 X-Received: by 2002:a50:8858:: with SMTP id c24mr2771467edc.253.1565601039936; Mon, 12 Aug 2019 02:10:39 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:90d5:: with SMTP id d21ls3039306eda.11.gmail; Mon, 12 Aug 2019 02:10:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqwhnsxE4f3EsEoIsBvBh6iX/Xv1tXJmjt6g1xLofm9zkr642ftOp2ZPsov+zsQrY1FhUs+B X-Received: by 2002:a50:fd82:: with SMTP id o2mr6347256edt.80.1565601039550; Mon, 12 Aug 2019 02:10:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565601039; cv=none; d=google.com; s=arc-20160816; b=lQpDz6g3Y/WcDnWSd3F2L3ul2QqDIlx6gvexv7Zhk18cJFccOQpQRZvfWuGoOZ5xSw R2o4gmBee1eHPYbbXU+u3RBUKiRNh80OGbux8wLD0+evU6G3p19/ygAqKqeqbpE0x0qy rqmeHaeyxRyCwZGu12fTIpsMJQL4fLHtRyFbRaoqGKbWKD9rgvpOP8QCbBV1pfAkgYud GjW6fIbP038WcLd6rgdsi3XwcLwh27CEmNUq/DrWcZD2eesftgj/tyFXGG2IMjTTlwIA 8Gu++3Ejlijad7r9CZmbijrC6na8bDgGAMDBbnGvlWPc4JPhzZDEWj803qvGNxzN6twt +Wpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject; bh=ANv0m8Jb5fzgXnu1IA4Z0ofzIdzRP/BrSRWf99f6XTc=; b=h+kpu1M1LUBfFxCLTcSj8SVFLWOkgSvzxCGPSLZT3J3fEzPxHlhjqaz8TujN+G/cKY x7jbhEvLsbAoi5T1A84BAd/Is3OLB4/MtPIfVmPW8PX4mqMHcF9CjHvRO4vZgL6qn4at yqo8CIc0NrbDPoIYOip/Nh8WPjBbTmtXr7vF2HB3Q5w0DSUtmsXzfogEdrz6EH+rQXO2 2Rpe5et3tAO5g4nkgOGxaTEaexypIqCGnY+JzuXNX8d1JVwvA0gkhZIZbuNqfnzEw9as qz7VqwU/GzcrhgXSycS2xjSWgzXzt6zjV7bKKYdW6qbNYi+notc9UIdHPiX99WQfQCE5 WXJg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id l17si612927ejg.1.2019.08.12.02.10.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Aug 2019 02:10:39 -0700 (PDT) Received-SPF: pass (google.com: domain of quirin.gylstorff@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x7C9AdP1022261 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 12 Aug 2019 11:10:39 +0200 Received: from [139.25.68.238] ([139.25.68.238]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x7C9AdQM007974; Mon, 12 Aug 2019 11:10:39 +0200 Subject: Re: [PATCH v3] meta/classes: generate bill of material from image To: Claudius Heine , isar-users@googlegroups.com References: <2c58eae5-4d77-776f-3d4e-5fda95dc27d5@siemens.com> <20190809103046.10493-1-Quirin.Gylstorff@siemens.com> <8c27aed7-56b7-89f8-f84d-093334627dae@siemens.com> From: Quirin Gylstorff Message-ID: <3221bfdb-641b-7e54-3fb5-1facbf6e5585@siemens.com> Date: Mon, 12 Aug 2019 11:09:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <8c27aed7-56b7-89f8-f84d-093334627dae@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: YyZkBVabFTQ3 On 8/12/19 10:04 AM, Claudius Heine wrote: > Hi Quirin, > > On 09/08/2019 12.30, [ext] Q. Gylstorff wrote: >> From: Quirin Gylstorff >> >> To create products it is necessary to have a list >> of used packages for clearance and to security monitoring. >> To get a simple list of packages use dpkg-query and generate >> a list with the following pattern: >> >> source name| source version | binary package name | binary version >> >> The list is stored in ${IMAGE_FULLNAME}.rootfs.manifest >> >> Remove the feature with: >> ROOTFS_FEATURES_remove = "generate-manifest" >> >> Signed-off-by: Quirin Gylstorff >> --- >> Changes: >> v3: >> Add list of manifest for buildchroot manifest >> This list can be exdent to add additional output generators >> v2: >> use FEATURE instead of own variable >> >> .../image-package-list-extension.bbclass      | 66 +++++++++++++++++++ >>   meta/classes/image.bbclass                    |  3 +- >>   2 files changed, 68 insertions(+), 1 deletion(-) >>   create mode 100644 meta/classes/image-package-list-extension.bbclass >> >> diff --git a/meta/classes/image-package-list-extension.bbclass >> b/meta/classes/image-package-list-extension.bbclass >> new file mode 100644 >> index 0000000..11896f1 >> --- /dev/null >> +++ b/meta/classes/image-package-list-extension.bbclass >> @@ -0,0 +1,66 @@ >> +# This software is a part of ISAR. >> +# Copyright (C) Siemens AG, 2019 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +MANIFESTS ?= "target build" >> +MANIFEST_build[rootfs] ?= "/var/lib/dpkg" >> +MANIFEST_target[rootfs] ?= "${PP_ROOTFS}/var/lib/dpkg" > > Have you planned additional flags for this? > > Currently I think that this mechanism is a bit of an overkill for just > two variables. But since you touched this now and are the second users > of this, I have further comments ;). > One Idea was to add use this to add additional generators. If this is not a use case anymore than it is overkill. >> + >> +def gen_manifests_array(d, listname, entryname, flags, verb_flags=None): >> +    from itertools import chain >> + >> +    entries = (d.getVar(listname, True) or "").split() >> +    return " ".join( >> +        ":".join( >> +            chain( >> +                (entry,), >> +                ( >> +                    (",".join( >> +                        ( >> +                            d.getVarFlag(entryname + "_" + entry, >> flag, True) or "" >> +                        ).split() >> +                    ) if flag not in (verb_flags or []) else ( >> +                        d.getVarFlag(entryname + "_" + entry, flag, >> True) or "" >> +                    )).replace(":","=") >> +                    for flag in flags >> +                ), >> +            ) >> +        ) >> +        for entry in entries > > Is this a 1:1 copy from image-account-extension.bbclass? If so, then can > you put those implementations together to avoid code duplication? > > I also have to say, I am not 100% happy with that code. While it is very > efficient, it might be not that maintainable compared to simple loops > that build the output string using string operators or StringIO. Those > are probably easier to understand for people not that fluent in python. > >> +    ) >> + >> + >> +IMAGE_MANIFESTS =+ "${@gen_manifests_array(d, 'MANIFESTS', >> 'MANIFEST', ['rootfs'])}" >> + >> +image_generate_manifest[dirs] = "${DEPLOY_DIR_IMAGE}" >> +image_generate_manifest() { >> +    image_do_mounts >> +    list='${@" ".join(d.getVar('IMAGE_MANIFESTS', True).split())} ' >> +    while true; do >> +        list_rest="${list#*:* }" >> +        entry="${list%%${list_rest}}" >> +        list="${list_rest}" >> + >> +        if [ -z "${entry}" ]; then >> +            break >> +        fi >> +        # Add colon to the end of the entry and remove trailing space: >> +        entry="${entry% }:" >> + >> +        # Decode entries: >> +        name="${entry%%:*}" >> +        entry="${entry#${name}:}" >> + >> +        rootfs="${entry%%:*}" >> +        entry="${entry#${rootfs}:}" > > I also see here some ways to avoid code duplication. Maybe generate this > shell code in a python function > > Something like: > >     ${@gen_shell_list_processing('IMAGE_MANIFESTS', ['name', 'rootfs'], > """ >         sudo -E chroot ${BUILDCHROOT_DIR} \ >         /usr/bin/dpkg-query --admindir="$rootfs" \ >         -f > '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W\ >         ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest > """)} > > Not sure about the shell part. BB seems to not handle line breaks in > inline python very well. > > To move it outside of the inline would also be possible: > >     ${@gen_shell_list_processing_start('IMAGE_MANIFESTS', ['name', > 'rootfs'])} >         sudo -E chroot ${BUILDCHROOT_DIR} \ >         /usr/bin/dpkg-query --admindir="$rootfs" \ >         -f > '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W\ >         ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest >     ${@gen_shell_list_processing_end()} > > gen_shell_list_processing_end would just return 'done;' but it makes is > more flexible in the future... maybe, IDK. But having 'done' directly > there would also be ok. > > I will test it >> + >> + >> +        sudo -E chroot ${BUILDCHROOT_DIR} \ >> +        /usr/bin/dpkg-query --admindir="$rootfs" \ >> +        -f >> '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' >> -W > \ >> +        ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest > > Does this also work if you just specify 'dpkg-query' without '/usr/bin/'? > > Cheers, > Claudius > >> +    done >> +} >> +ROOTFS_POSTPROCESS_COMMAND += >> "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', >> 'image_generate_manifest', '', d)}" >> + >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index ec6bd39..60dd9fb 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -58,7 +58,7 @@ image_do_mounts() { >>   } >>   ROOTFSDIR = "${IMAGE_ROOTFS}" >> -ROOTFS_FEATURES += "copy-package-cache clean-package-cache >> finalize-rootfs" >> +ROOTFS_FEATURES += "copy-package-cache clean-package-cache >> finalize-rootfs generate-manifest" >>   ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" >>   inherit rootfs >> @@ -68,6 +68,7 @@ inherit image-tools-extension >>   inherit image-postproc-extension >>   inherit image-locales-extension >>   inherit image-account-extension >> +inherit image-package-list-extension >>   # Extra space for rootfs in MB >>   ROOTFS_EXTRA ?= "64" >> > Regards Quirin