From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6721684426774806528 X-Received: by 2002:a19:ed11:: with SMTP id y17mr22052115lfy.141.1565686402558; Tue, 13 Aug 2019 01:53:22 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:4a9a:: with SMTP id l26ls2415466lfp.2.gmail; Tue, 13 Aug 2019 01:53:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqz+pkJb+UNYJIt89NroMj9cUZ18nG5A8so7YdpZJP9/7Bf++lsu/m0ab+CLWoABNYiR3bYT X-Received: by 2002:ac2:5492:: with SMTP id t18mr9695695lfk.41.1565686402048; Tue, 13 Aug 2019 01:53:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565686402; cv=none; d=google.com; s=arc-20160816; b=hpOcn+p60kY1WODjqFaP20C+7kyXGfhd28QrcPh07R2Q2moT1pLh2Dt4u6gjHDjDmq wRzRkNF0IPbN4mxJ8RNoCAxEH8xNqAFPnpf2CJCEbPOKEDxtulPWod7ZjGsayG8dvQ5C Ay4eGQP809eX7pFJzBfIA496wR8fA6MgUPDsFpZClWnyethwzHuaEuHY0BGbhr+9ewKS 5udJu/oSS5VTG3Re77TaDQEKOIrEjUaqFgnkDynJYbQiPNI6ALpepF0Op6ThA2GOoYcy u6h1swAc5dQpuwp6RLqUGWDjIvzqBNnB+GhIt2RSSi+5GV0W3EyJn+eflMOlKqLerQxU PvNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=3r5wKiwjhbxljzSuf7UCJKMcrjX+udE37OZeLmrJLiU=; b=w3+4ziQOYBDFg0C36sR3U9ZT0toxUMuGu2FRkJttmaVSihaLf6x1TCGJ7n4z0r9urP vFH2uWC2pQr7ZAfx6y31eFkoD0bx7Ek8IqnBsXXV5XLo8Fagqyb7oOwKWZIZESllbhe3 WXCMay0HV4IDn0f4lGJ/gSy5hcYBdhfD8oqxzb6ltDu1Bs4Wz81RkhwSgLrte0zGpPyv MOImSKyHlf20lOZXIc05HPL3LxNbCtXHY49zhk17m4Xw8gRdZFszZM0H8sGJnMnszesc 2DQhK9a2eQBZJtH8Tr7PM+g7U2dRzIpctk/8o4XEjD0ubKbVJRvi3vkXvOeC69sR8gRl thrw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id s14si6406678ljg.4.2019.08.13.01.53.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Aug 2019 01:53:22 -0700 (PDT) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id x7D8rKnG016109 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 13 Aug 2019 10:53:21 +0200 Received: from [139.25.69.208] (linux-ses-ext02.ppmd.siemens.net [139.25.69.208]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x7D8rKZh029411; Tue, 13 Aug 2019 10:53:20 +0200 Subject: Re: [PATCH v4] meta/classes: generate bill of material from image To: "[ext] Q. Gylstorff" , isar-users@googlegroups.com Cc: Claudius Heine References: <3221bfdb-641b-7e54-3fb5-1facbf6e5585@siemens.com> <20190813081823.29704-1-Quirin.Gylstorff@siemens.com> From: Claudius Heine Message-ID: <3e792ace-44e8-e1aa-3a44-21a7c2c1f375@siemens.com> Date: Tue, 13 Aug 2019 10:53:20 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.1 MIME-Version: 1.0 In-Reply-To: <20190813081823.29704-1-Quirin.Gylstorff@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: EEoIUlNczfH3 Hi Quirin, On 13/08/2019 10.18, [ext] Q. Gylstorff wrote: > From: Quirin Gylstorff > > To create products it is necessary to have a list > of used packages for clearance and to security monitoring. > To get a simple list of packages use dpkg-query and generate > a list with the following pattern: > > source name| source version | binary package name | binary version > > The list is stored in ${IMAGE_FULLNAME}.rootfs.manifest > > Remove the feature with: > ROOTFS_FEATURES_remove = "generate-manifest" > > Signed-off-by: Quirin Gylstorff > --- > Changes: > v4: > Add sdk rootfs to manifest > Avoid duplicated code and move gen_accounts_array and gen_manifest_array to > shell-list-processing-helper > call dpkg-query from $PATH > > v3: > Add list of manifest for buildchroot manifest > This list can be exdent to add additional output generators > > v2: > use FEATURE instead of own variable > > meta/classes/image-account-extension.bbclass | 28 ++-------- > .../image-package-list-extension.bbclass | 54 +++++++++++++++++++ > meta/classes/image.bbclass | 3 +- > .../shell-list-processing-helper.bbclass | 30 +++++++++++ > 4 files changed, 89 insertions(+), 26 deletions(-) > create mode 100644 meta/classes/image-package-list-extension.bbclass > create mode 100644 meta/classes/shell-list-processing-helper.bbclass > > diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass > index 22754da..df44c49 100644 > --- a/meta/classes/image-account-extension.bbclass > +++ b/meta/classes/image-account-extension.bbclass > @@ -25,36 +25,14 @@ GROUPS ??= "" > #GROUP_root[gid] = "" > #GROUP_root[flags] = "system" > > -def gen_accounts_array(d, listname, entryname, flags, verb_flags=None): > - from itertools import chain > - > - entries = (d.getVar(listname, True) or "").split() > - return " ".join( > - ":".join( > - chain( > - (entry,), > - ( > - (",".join( > - ( > - d.getVarFlag(entryname + "_" + entry, flag, True) or "" > - ).split() > - ) if flag not in (verb_flags or []) else ( > - d.getVarFlag(entryname + "_" + entry, flag, True) or "" > - )).replace(":","=") > - for flag in flags > - ), > - ) > - ) > - for entry in entries > - ) > - > +inherit shell-list-processing-helper > # List of space separated entries, where each entry has the format: > # username:encryptedpassword:expiredate:inactivenumber:userid:groupid:comment:homedir:shell:group1,group2:flag1,flag2 > -IMAGE_ACCOUNTS_USERS =+ "${@gen_accounts_array(d, 'USERS', 'USER', ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', 'shell', 'groups', 'flags'], ['password', 'comment', 'home', 'shell'])}" > +IMAGE_ACCOUNTS_USERS =+ "${@gen_shell_list(d, 'USERS', 'USER', ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', 'shell', 'groups', 'flags'], ['password', 'comment', 'home', 'shell'])}" > > # List of space separated entries, where each entry has the format: > # groupname:groupid:flag1,flag2 > -IMAGE_ACCOUNTS_GROUPS =+ "${@gen_accounts_array(d, 'GROUPS', 'GROUP', ['gid', 'flags'])}" > +IMAGE_ACCOUNTS_GROUPS =+ "${@gen_shell_list(d, 'GROUPS', 'GROUP', ['gid', 'flags'])}" > > ROOTFS_CONFIGURE_COMMAND += "image_configure_accounts" > image_configure_accounts[weight] = "3" > diff --git a/meta/classes/image-package-list-extension.bbclass b/meta/classes/image-package-list-extension.bbclass > new file mode 100644 > index 0000000..0aa3015 > --- /dev/null > +++ b/meta/classes/image-package-list-extension.bbclass > @@ -0,0 +1,54 @@ > +# This software is a part of ISAR. > +# Copyright (C) Siemens AG, 2019 > +# > +# SPDX-License-Identifier: MIT > +MANIFESTS ?= "target build sdk" > +DPKG_DIR ?= "/var/lib/dpkg" > +# rootfs needs to be mounted inside of buildchroot > +MANIFEST_build[rootfs] ?= "${DPKG_DIR}" > +MANIFEST_target[rootfs] ?= "${PP_ROOTFS}${DPKG_DIR}" > +MANIFEST_sdk[rootfs] ?= "/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs${DPKG_DIR}" That is a long and pretty explicit path. Can you change this to use more commonly used variables and if those are missing define them somewhere global where sdkchroot and others can use them? Otherwise if those paths are changed at some point in the future, we have to hunt down every of those magic variables to fix them as well, instead of just at one global point. You should also probably write something about the user facing interface of this class in the documentation. > + > +inherit shell-list-processing-helper > +IMAGE_MANIFESTS =+ "${@gen_shell_list(d, 'MANIFESTS', 'MANIFEST', ['rootfs'])}" > + > +do_image_generate_manifest[dirs] = "${DEPLOY_DIR_IMAGE}" > +image_generate_manifest() { > + image_do_mounts > + # mount working directory to access sdk rootfs > + sudo -s <<'EOSUDO' > + ( flock 9 > + mkdir -p ${BUILDCHROOT_DIR}/work > + if ! mountpoint ${BUILDCHROOT_DIR}/work >/dev/null 2>&1; then > + mount --bind --make-private ${TMPDIR}/work ${BUILDCHROOT_DIR}/work > + fi Does that mean that 'do_image_generate_manifest' now depends on sdkchroot? Is that task dependency missing? Personally I would have the manifest for sdkchroot 'opt-in', since I don't want to create a sdkchroot just to build a image everytime. > + ) 9>${MOUNT_LOCKFILE} > +EOSUDO > + list='${@" ".join(d.getVar('IMAGE_MANIFESTS', True).split())} ' > + while true; do > + list_rest="${list#*:* }" > + entry="${list%%${list_rest}}" > + list="${list_rest}" > + > + if [ -z "${entry}" ]; then > + break > + fi > + # Add colon to the end of the entry and remove trailing space: > + entry="${entry% }:" > + > + # Decode entries: > + name="${entry%%:*}" > + entry="${entry#${name}:}" > + > + rootfs="${entry%%:*}" > + entry="${entry#${rootfs}:}" Maybe put a empty line here. You had 3 here before, now none, just settle with one, maybe two, if you are feeling generous today. :) > + if sudo -E chroot ${BUILDCHROOT_DIR} test -d "$rootfs"; then > + sudo -E chroot ${BUILDCHROOT_DIR} \ > + dpkg-query --admindir="$rootfs" \ > + -f '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' -W > \ > + ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}."$name".manifest > + fi > + done > +} > +ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'image_generate_manifest', '', d)}" > + > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index ec6bd39..60dd9fb 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -58,7 +58,7 @@ image_do_mounts() { > } > > ROOTFSDIR = "${IMAGE_ROOTFS}" > -ROOTFS_FEATURES += "copy-package-cache clean-package-cache finalize-rootfs" > +ROOTFS_FEATURES += "copy-package-cache clean-package-cache finalize-rootfs generate-manifest" > ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}" > > inherit rootfs > @@ -68,6 +68,7 @@ inherit image-tools-extension > inherit image-postproc-extension > inherit image-locales-extension > inherit image-account-extension > +inherit image-package-list-extension > > # Extra space for rootfs in MB > ROOTFS_EXTRA ?= "64" > diff --git a/meta/classes/shell-list-processing-helper.bbclass b/meta/classes/shell-list-processing-helper.bbclass > new file mode 100644 > index 0000000..105066b > --- /dev/null > +++ b/meta/classes/shell-list-processing-helper.bbclass > @@ -0,0 +1,30 @@ > +# This software is a part of ISAR. > +# Copyright (C) Siemens AG, 2019 > +# > +# SPDX-License-Identifier: MIT > +# > +# This class extends the image.bbclass for creating user accounts and groups. To much copy pasta. > + > +def gen_shell_list(d, listname, entryname, flags, verb_flags=None): > + from itertools import chain > + > + entries = (d.getVar(listname, True) or "").split() > + return " ".join( > + ":".join( > + chain( > + (entry,), > + ( > + (",".join( > + ( > + d.getVarFlag(entryname + "_" + entry, flag, True) or "" > + ).split() > + ) if flag not in (verb_flags or []) else ( > + d.getVarFlag(entryname + "_" + entry, flag, True) or "" > + )).replace(":","=") > + for flag in flags > + ), > + ) > + ) > + for entry in entries > + ) > + I a still a bit unconvinced that we need the list feature here. Since that feature is more for the usecase that there needs to be a flexible number of configuration items each with multiple parameters. Like users and groups. That needs to be customized by the end user. Here we now have 3, with each just one setting each. Instead I would think that this could be a 'rootfs' extension, where each rootfs (every image, buildchroot and sdk) can create their own package listing. regards, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de