* [PATCH 0/4] Rewrite the image-account-extension in python
@ 2023-03-30 11:08 T. Schaffner
2023-03-30 11:08 ` [PATCH 1/4] simplify image-account-extension T. Schaffner
` (3 more replies)
0 siblings, 4 replies; 7+ messages in thread
From: T. Schaffner @ 2023-03-30 11:08 UTC (permalink / raw)
To: isar-users; +Cc: quirin.gylstorff, henning.schild, Tobias Schaffner
From: Tobias Schaffner <tobias.schaffner@siemens.com>
This allows us to drop a lot of encoding and parsing code that was used
to transition to shell and therefore makes it easier to read and maintain.
Using python functions for more complex tasks allows us the usage of
unittests. A very basic infrastructure for unittesting using the build
in python unittest and the bb.parse module was added. This was used to
test the re-implementation of the image-account-extension as a first
showcase.
This is a rebased version of the refactoring part of the "allow creation
of users/groups before rootfs creation" patch series to decouple the
merge of the simplifications.
Tobias Schaffner (4):
simplify image-account-extension
create a minimal python unittest infrastructure
add unittests for the image-account-extension
set minimal python version in user_manual to 3.5
doc/user_manual.md | 2 +-
meta/classes/image-account-extension.bbclass | 368 ++++++------------
testsuite/unittests/README.md | 28 ++
testsuite/unittests/bitbake.py | 37 ++
testsuite/unittests/rootfs.py | 45 +++
.../unittests/test_image_account_extension.py | 145 +++++++
6 files changed, 380 insertions(+), 245 deletions(-)
create mode 100644 testsuite/unittests/README.md
create mode 100644 testsuite/unittests/bitbake.py
create mode 100644 testsuite/unittests/rootfs.py
create mode 100644 testsuite/unittests/test_image_account_extension.py
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 1/4] simplify image-account-extension 2023-03-30 11:08 [PATCH 0/4] Rewrite the image-account-extension in python T. Schaffner @ 2023-03-30 11:08 ` T. Schaffner 2023-03-30 19:55 ` Henning Schild 2023-03-30 11:08 ` [PATCH 2/4] create a minimal python unittest infrastructure T. Schaffner ` (2 subsequent siblings) 3 siblings, 1 reply; 7+ messages in thread From: T. Schaffner @ 2023-03-30 11:08 UTC (permalink / raw) To: isar-users; +Cc: quirin.gylstorff, henning.schild, Tobias Schaffner From: Tobias Schaffner <tobias.schaffner@siemens.com> Do the complete user and group creation in python. This allows us to drop the encoding and parsing code that was used to make the user and group lists available in the shell function. Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> --- meta/classes/image-account-extension.bbclass | 368 +++++++------------ 1 file changed, 124 insertions(+), 244 deletions(-) diff --git a/meta/classes/image-account-extension.bbclass b/meta/classes/image-account-extension.bbclass index 1a1f704d..d1133bb4 100644 --- a/meta/classes/image-account-extension.bbclass +++ b/meta/classes/image-account-extension.bbclass @@ -1,5 +1,5 @@ # This software is a part of ISAR. -# Copyright (C) Siemens AG, 2019 +# Copyright (C) Siemens AG, 2023 # # SPDX-License-Identifier: MIT # @@ -25,251 +25,131 @@ GROUPS ??= "" #GROUP_root[gid] = "" #GROUP_root[flags] = "system" -def gen_accounts_array(d, listname, entryname, flags, verb_flags=None): - from itertools import chain - - entries = (d.getVar(listname) or "").split() - return " ".join( - ":".join( - chain( - (entry,), - ( - (",".join( - ( - d.getVarFlag(entryname + "_" + entry, flag, True) or "" - ).split() - ) if flag not in (verb_flags or []) else ( - d.getVarFlag(entryname + "_" + entry, flag, True) or "" - )).replace(":","=") - for flag in flags - ), - ) - ) - for entry in entries - ) - -# List of space separated entries, where each entry has the format: -# username:encryptedpassword:expiredate:inactivenumber:userid:groupid:comment:homedir:shell:group1,group2:flag1,flag2 -IMAGE_ACCOUNTS_USERS =+ "${@gen_accounts_array(d, 'USERS', 'USER', ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', 'shell', 'groups', 'flags'], ['password', 'comment', 'home', 'shell'])}" - -# List of space separated entries, where each entry has the format: -# groupname:groupid:flag1,flag2 -IMAGE_ACCOUNTS_GROUPS =+ "${@gen_accounts_array(d, 'GROUPS', 'GROUP', ['gid', 'flags'])}" - -do_rootfs_install[vardeps] += "${IMAGE_ACCOUNTS_GROUPS} ${IMAGE_ACCOUNTS_USERS}" -ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" -image_postprocess_accounts() { - # Create groups - # Add space to the end of the list: - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_GROUPS').split())} ' - while true; do - # Pop first group entry: - list_rest="${list#*:*:* }" - entry="${list%%${list_rest}}" - list="${list_rest}" - - if [ -z "${entry}" ]; then - break - fi - - # Add colon to the end of the entry and remove trailing space: - entry="${entry% }:" - - # Decode entries: - name="${entry%%:*}" - entry="${entry#${name}:}" - - gid="${entry%%:*}" - entry="${entry#${gid}:}" - - flags="${entry%%:*}" - entry="${entry#${flags}:}" - - flags=",${flags}," # Needed for searching for substrings - - # Check if user already exists: - if grep -q "^${name}:" '${ROOTFSDIR}/etc/group'; then - exists="y" - else - exists="n" - fi - - # Create arguments: - set -- # clear arguments - - if [ -n "$gid" ]; then - set -- "$@" --gid "$gid" - fi - - if [ "n" = "$exists" ]; then - if [ "${flags}" != "${flags%*,system,*}" ]; then - set -- "$@" --system - fi - fi - - # Create or modify groups: - if [ "y" = "$exists" ]; then - if [ -z "$@" ]; then - echo "Do not execute groupmod (no changes)." - else - echo "Execute groupmod with \"$@\" for \"$name\"" - sudo -E chroot '${ROOTFSDIR}' \ - /usr/sbin/groupmod "$@" "$name" - fi - else - echo "Execute groupadd with \"$@\" for \"$name\"" - sudo -E chroot '${ROOTFSDIR}' \ - /usr/sbin/groupadd "$@" "$name" - fi - done - - # Create users - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_USERS').split())} ' - while true; do - # Pop first user entry: - list_rest="${list#*:*:*:*:*:*:*:*:*:*:* }" - entry="${list%%${list_rest}}" - list="${list_rest}" - - if [ -z "${entry}" ]; then - break - fi - - # Add colon to the end of the entry and remove trailing space: - entry="${entry% }:" - - # Decode entries: - name="${entry%%:*}" - entry="${entry#${name}:}" - - password="${entry%%:*}" - entry="${entry#${password}:}" - - expire="${entry%%:*}" - entry="${entry#${expire}:}" - - inactive="${entry%%:*}" - entry="${entry#${inactive}:}" - - uid="${entry%%:*}" - entry="${entry#${uid}:}" - - gid="${entry%%:*}" - entry="${entry#${gid}:}" - - comment="${entry%%:*}" - entry="${entry#${comment}:}" - - home="${entry%%:*}" - entry="${entry#${home}:}" - - shell="${entry%%:*}" - entry="${entry#${shell}:}" - - groups="${entry%%:*}" - entry="${entry#${groups}:}" - - flags="${entry%%:*}" - entry="${entry#${flags}:}" - - flags=",${flags}," # Needed for searching for substrings - - # Check if user already exists: - if grep -q "^${name}:" '${ROOTFSDIR}/etc/passwd'; then - exists="y" - else - exists="n" - fi - - # Create arguments: - set -- # clear arguments - - if [ -n "$expire" ]; then - set -- "$@" --expiredate "$expire" - fi - - if [ -n "$inactive" ]; then - set -- "$@" --inactive "$inactive" - fi - - if [ -n "$uid" ]; then - set -- "$@" --uid "$uid" - fi - - if [ -n "$gid" ]; then - set -- "$@" --gid "$gid" - fi - - if [ -n "$comment" ]; then - set -- "$@" --comment "$comment" - fi - - if [ -n "$home" ]; then - if [ "y" = "$exists" ]; then - set -- "$@" --home "$home" --move-home - else - set -- "$@" --home-dir "$home" - fi - fi - - if [ -n "$shell" ]; then - set -- "$@" --shell "$shell" - fi - - if [ -n "$groups" ]; then - set -- "$@" --groups "$groups" - fi - - if [ "n" = "$exists" ]; then - if [ "${flags}" != "${flags%*,system,*}" ]; then - set -- "$@" --system - fi - if [ "${flags}" != "${flags%*,no-create-home,*}" ]; then - set -- "$@" --no-create-home - else - if [ "${flags}" != "${flags%*,create-home,*}" ]; then - set -- "$@" --create-home - fi - fi - fi - - # Create or modify users: - if [ "y" = "$exists" ]; then - if [ -z "$@" ]; then - echo "Do not execute usermod (no changes)." - else - echo "Execute usermod with \"$@\" for \"$name\"" - sudo -E chroot '${ROOTFSDIR}' \ - /usr/sbin/usermod "$@" "$name" - fi - else - echo "Execute useradd with \"$@\" for \"$name\"" - sudo -E chroot '${ROOTFSDIR}' \ - /usr/sbin/useradd "$@" "$name" - fi - - # Set password: - if [ -n "$password" -o "${flags}" != "${flags%*,allow-empty-password,*}" ]; then - chpasswd_args="-e" - if [ "${flags}" != "${flags%*,clear-text-password,*}" ]; then +def image_create_groups(d: "DataSmart") -> None: + """Creates the groups defined in the ``GROUPS`` bitbake variable. + + Args: + d (DataSmart): The bitbake datastore. + + Returns: + None + """ + entries = (d.getVar("GROUPS") or "").split() + rootfsdir = d.getVar("ROOTFSDIR") + chroot = ["sudo", "-E", "chroot", rootfsdir] + + for entry in entries: + args = [] + group_entry = "GROUP_{}".format(entry) + + with open("{}/etc/group".format(rootfsdir), "r") as group_file: + exists = any(line.startswith("{}:".format(entry)) for line in group_file) + + gid = d.getVarFlag(group_entry, "gid") or "" + if gid: + args.append("--gid") + args.append(gid) + + flags = (d.getVarFlag(group_entry, "flags") or "").split() + if "system" in flags: + args.append("--system") + + if exists: + if args: + bb.process.run([*chroot, "/usr/sbin/groupmod", *args, entry]) + else: + bb.process.run([*chroot, "/usr/sbin/groupadd", *args, entry]) + + +def image_create_users(d: "DataSmart") -> None: + """Creates the users defined in the ``USERS`` bitbake variable. + + Args: + d (DataSmart): The bitbake datastore. + + Returns: + None + """ + import hashlib + import crypt + + entries = (d.getVar("USERS") or "").split() + rootfsdir = d.getVar("ROOTFSDIR") + chroot = ["sudo", "-E", "chroot", rootfsdir] + + for entry in entries: + args = [] + user_entry = "USER_{}".format(entry) + + with open("{}/etc/passwd".format(rootfsdir), "r") as passwd_file: + exists = any(line.startswith("{}:".format(entry)) for line in passwd_file) + + def add_user_option(option_name, flag_name): + flag_value = d.getVarFlag(user_entry, flag_name) or "" + if flag_value: + args.append(option_name) + args.append(flag_value) + + add_user_option("--expire", "expiredate") + add_user_option("--inactive", "inactive") + add_user_option("--uid", "uid") + add_user_option("--gid", "gid") + add_user_option("--comment", "comment") + add_user_option("--shell", "shell") + + groups = d.getVarFlag(user_entry, "groups") or "" + if groups: + args.append("--groups") + args.append(groups.replace(' ', ',')) + + flags = (d.getVarFlag(user_entry, "flags") or "").split() + + if exists: + add_user_option("--home", "home") + if d.getVarFlag(user_entry, "home") or "": + args.append("--move-home") + else: + add_user_option("--home-dir", "home") + + if "system" in flags: + args.append("--system") + if "no-create-home" in flags: + args.append("--no-create-home") + if "create-home" in flags: + args.append("--create-home") + + if exists: + if args: + bb.process.run([*chroot, "/usr/sbin/usermod", *args, entry]) + else: + bb.process.run([*chroot, "/usr/sbin/useradd", *args, entry]) + + command = [*chroot, "/usr/sbin/chpasswd"] + password = d.getVarFlag(user_entry, "password") or "" + if password or "allow-empty-password" in flags: + if "clear-text-password" in flags: + # chpasswd adds a random salt when running against a clear-text password. # For reproducible images, we manually generate the password and use the # SOURCE_DATE_EPOCH to generate the salt in a deterministic way. - if [ -z "${SOURCE_DATE_EPOCH}" ]; then - chpasswd_args="" - else - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum -z | cut -c 1-15)" - password="$(openssl passwd -6 -salt $salt "$password")" - fi - fi - printf '%s:%s' "$name" "$password" | sudo chroot '${ROOTFSDIR}' \ - /usr/sbin/chpasswd $chpasswd_args - fi - if [ "${flags}" != "${flags%*,force-passwd-change,*}" ]; then - echo "Execute passwd to force password change on first boot for \"$name\"" - sudo -E chroot '${ROOTFSDIR}' \ - /usr/bin/passwd --expire "$name" - fi - done + source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") or "" + if source_date_epoch: + command.append("-e") + salt = hashlib.sha256("{}\n".format(source_date_epoch).encode()).hexdigest()[0:15] + password = crypt.crypt(password, "$6${}".format(salt)) + + else: + command.append("-e") + + bb.process.run(command, "{}:{}".format(entry, password).encode()) + + if "force-passwd-change" in flags: + bb.process.run([*chroot, "/usr/sbin/passwd", "--expire", entry]) + + +ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" +python image_postprocess_accounts() { + image_create_groups(d) + image_create_users(d) } -- 2.34.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/4] simplify image-account-extension 2023-03-30 11:08 ` [PATCH 1/4] simplify image-account-extension T. Schaffner @ 2023-03-30 19:55 ` Henning Schild 2023-03-30 21:39 ` Schaffner, Tobias 0 siblings, 1 reply; 7+ messages in thread From: Henning Schild @ 2023-03-30 19:55 UTC (permalink / raw) To: T. Schaffner; +Cc: isar-users, quirin.gylstorff Am Thu, 30 Mar 2023 13:08:01 +0200 schrieb "T. Schaffner" <tobias.schaffner@siemens.com>: > From: Tobias Schaffner <tobias.schaffner@siemens.com> > > Do the complete user and group creation in python. This allows us to > drop the encoding and parsing code that was used to make the user and > group lists available in the shell function. > > Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> > --- > meta/classes/image-account-extension.bbclass | 368 > +++++++------------ 1 file changed, 124 insertions(+), 244 > deletions(-) > > diff --git a/meta/classes/image-account-extension.bbclass > b/meta/classes/image-account-extension.bbclass index > 1a1f704d..d1133bb4 100644 --- > a/meta/classes/image-account-extension.bbclass +++ > b/meta/classes/image-account-extension.bbclass @@ -1,5 +1,5 @@ > # This software is a part of ISAR. > -# Copyright (C) Siemens AG, 2019 > +# Copyright (C) Siemens AG, 2023 2019-2023 did not look at the rest, this feels like it should be tried out ... functional review, which i would try to do please make sure that would pass flake8 ... might be hard since it is "embedded code" a long time ago we had contributions from Harald Seiler, who managed to somehow lint python code inside bb code. Henning > # > # SPDX-License-Identifier: MIT > # > @@ -25,251 +25,131 @@ GROUPS ??= "" > #GROUP_root[gid] = "" > #GROUP_root[flags] = "system" > > -def gen_accounts_array(d, listname, entryname, flags, > verb_flags=None): > - from itertools import chain > - > - entries = (d.getVar(listname) or "").split() > - return " ".join( > - ":".join( > - chain( > - (entry,), > - ( > - (",".join( > - ( > - d.getVarFlag(entryname + "_" + entry, > flag, True) or "" > - ).split() > - ) if flag not in (verb_flags or []) else ( > - d.getVarFlag(entryname + "_" + entry, flag, > True) or "" > - )).replace(":","=") > - for flag in flags > - ), > - ) > - ) > - for entry in entries > - ) > - > -# List of space separated entries, where each entry has the format: > -# > username:encryptedpassword:expiredate:inactivenumber:userid:groupid:comment:homedir:shell:group1,group2:flag1,flag2 > -IMAGE_ACCOUNTS_USERS =+ "${@gen_accounts_array(d, 'USERS', 'USER', > ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', > 'shell', 'groups', 'flags'], ['password', 'comment', 'home', > 'shell'])}" - -# List of space separated entries, where each entry > has the format: -# groupname:groupid:flag1,flag2 > -IMAGE_ACCOUNTS_GROUPS =+ "${@gen_accounts_array(d, 'GROUPS', > 'GROUP', ['gid', 'flags'])}" - -do_rootfs_install[vardeps] += > "${IMAGE_ACCOUNTS_GROUPS} ${IMAGE_ACCOUNTS_USERS}" > -ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" > -image_postprocess_accounts() { > - # Create groups > - # Add space to the end of the list: > - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_GROUPS').split())} ' > - while true; do > - # Pop first group entry: > - list_rest="${list#*:*:* }" > - entry="${list%%${list_rest}}" > - list="${list_rest}" > - > - if [ -z "${entry}" ]; then > - break > - fi > - > - # Add colon to the end of the entry and remove trailing > space: > - entry="${entry% }:" > - > - # Decode entries: > - name="${entry%%:*}" > - entry="${entry#${name}:}" > - > - gid="${entry%%:*}" > - entry="${entry#${gid}:}" > - > - flags="${entry%%:*}" > - entry="${entry#${flags}:}" > - > - flags=",${flags}," # Needed for searching for substrings > - > - # Check if user already exists: > - if grep -q "^${name}:" '${ROOTFSDIR}/etc/group'; then > - exists="y" > - else > - exists="n" > - fi > - > - # Create arguments: > - set -- # clear arguments > - > - if [ -n "$gid" ]; then > - set -- "$@" --gid "$gid" > - fi > - > - if [ "n" = "$exists" ]; then > - if [ "${flags}" != "${flags%*,system,*}" ]; then > - set -- "$@" --system > - fi > - fi > - > - # Create or modify groups: > - if [ "y" = "$exists" ]; then > - if [ -z "$@" ]; then > - echo "Do not execute groupmod (no changes)." > - else > - echo "Execute groupmod with \"$@\" for \"$name\"" > - sudo -E chroot '${ROOTFSDIR}' \ > - /usr/sbin/groupmod "$@" "$name" > - fi > - else > - echo "Execute groupadd with \"$@\" for \"$name\"" > - sudo -E chroot '${ROOTFSDIR}' \ > - /usr/sbin/groupadd "$@" "$name" > - fi > - done > - > - # Create users > - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_USERS').split())} ' > - while true; do > - # Pop first user entry: > - list_rest="${list#*:*:*:*:*:*:*:*:*:*:* }" > - entry="${list%%${list_rest}}" > - list="${list_rest}" > - > - if [ -z "${entry}" ]; then > - break > - fi > - > - # Add colon to the end of the entry and remove trailing > space: > - entry="${entry% }:" > - > - # Decode entries: > - name="${entry%%:*}" > - entry="${entry#${name}:}" > - > - password="${entry%%:*}" > - entry="${entry#${password}:}" > - > - expire="${entry%%:*}" > - entry="${entry#${expire}:}" > - > - inactive="${entry%%:*}" > - entry="${entry#${inactive}:}" > - > - uid="${entry%%:*}" > - entry="${entry#${uid}:}" > - > - gid="${entry%%:*}" > - entry="${entry#${gid}:}" > - > - comment="${entry%%:*}" > - entry="${entry#${comment}:}" > - > - home="${entry%%:*}" > - entry="${entry#${home}:}" > - > - shell="${entry%%:*}" > - entry="${entry#${shell}:}" > - > - groups="${entry%%:*}" > - entry="${entry#${groups}:}" > - > - flags="${entry%%:*}" > - entry="${entry#${flags}:}" > - > - flags=",${flags}," # Needed for searching for substrings > - > - # Check if user already exists: > - if grep -q "^${name}:" '${ROOTFSDIR}/etc/passwd'; then > - exists="y" > - else > - exists="n" > - fi > - > - # Create arguments: > - set -- # clear arguments > - > - if [ -n "$expire" ]; then > - set -- "$@" --expiredate "$expire" > - fi > - > - if [ -n "$inactive" ]; then > - set -- "$@" --inactive "$inactive" > - fi > - > - if [ -n "$uid" ]; then > - set -- "$@" --uid "$uid" > - fi > - > - if [ -n "$gid" ]; then > - set -- "$@" --gid "$gid" > - fi > - > - if [ -n "$comment" ]; then > - set -- "$@" --comment "$comment" > - fi > - > - if [ -n "$home" ]; then > - if [ "y" = "$exists" ]; then > - set -- "$@" --home "$home" --move-home > - else > - set -- "$@" --home-dir "$home" > - fi > - fi > - > - if [ -n "$shell" ]; then > - set -- "$@" --shell "$shell" > - fi > - > - if [ -n "$groups" ]; then > - set -- "$@" --groups "$groups" > - fi > - > - if [ "n" = "$exists" ]; then > - if [ "${flags}" != "${flags%*,system,*}" ]; then > - set -- "$@" --system > - fi > - if [ "${flags}" != "${flags%*,no-create-home,*}" ]; then > - set -- "$@" --no-create-home > - else > - if [ "${flags}" != "${flags%*,create-home,*}" ]; then > - set -- "$@" --create-home > - fi > - fi > - fi > - > - # Create or modify users: > - if [ "y" = "$exists" ]; then > - if [ -z "$@" ]; then > - echo "Do not execute usermod (no changes)." > - else > - echo "Execute usermod with \"$@\" for \"$name\"" > - sudo -E chroot '${ROOTFSDIR}' \ > - /usr/sbin/usermod "$@" "$name" > - fi > - else > - echo "Execute useradd with \"$@\" for \"$name\"" > - sudo -E chroot '${ROOTFSDIR}' \ > - /usr/sbin/useradd "$@" "$name" > - fi > - > - # Set password: > - if [ -n "$password" -o "${flags}" != > "${flags%*,allow-empty-password,*}" ]; then > - chpasswd_args="-e" > - if [ "${flags}" != "${flags%*,clear-text-password,*}" ]; > then +def image_create_groups(d: "DataSmart") -> None: > + """Creates the groups defined in the ``GROUPS`` bitbake variable. > + > + Args: > + d (DataSmart): The bitbake datastore. > + > + Returns: > + None > + """ > + entries = (d.getVar("GROUPS") or "").split() > + rootfsdir = d.getVar("ROOTFSDIR") > + chroot = ["sudo", "-E", "chroot", rootfsdir] > + > + for entry in entries: > + args = [] > + group_entry = "GROUP_{}".format(entry) > + > + with open("{}/etc/group".format(rootfsdir), "r") as > group_file: > + exists = any(line.startswith("{}:".format(entry)) for > line in group_file) + > + gid = d.getVarFlag(group_entry, "gid") or "" > + if gid: > + args.append("--gid") > + args.append(gid) > + > + flags = (d.getVarFlag(group_entry, "flags") or "").split() > + if "system" in flags: > + args.append("--system") > + > + if exists: > + if args: > + bb.process.run([*chroot, "/usr/sbin/groupmod", > *args, entry]) > + else: > + bb.process.run([*chroot, "/usr/sbin/groupadd", *args, > entry]) + > + > +def image_create_users(d: "DataSmart") -> None: > + """Creates the users defined in the ``USERS`` bitbake variable. > + > + Args: > + d (DataSmart): The bitbake datastore. > + > + Returns: > + None > + """ > + import hashlib > + import crypt > + > + entries = (d.getVar("USERS") or "").split() > + rootfsdir = d.getVar("ROOTFSDIR") > + chroot = ["sudo", "-E", "chroot", rootfsdir] > + > + for entry in entries: > + args = [] > + user_entry = "USER_{}".format(entry) > + > + with open("{}/etc/passwd".format(rootfsdir), "r") as > passwd_file: > + exists = any(line.startswith("{}:".format(entry)) for > line in passwd_file) + > + def add_user_option(option_name, flag_name): > + flag_value = d.getVarFlag(user_entry, flag_name) or "" > + if flag_value: > + args.append(option_name) > + args.append(flag_value) > + > + add_user_option("--expire", "expiredate") > + add_user_option("--inactive", "inactive") > + add_user_option("--uid", "uid") > + add_user_option("--gid", "gid") > + add_user_option("--comment", "comment") > + add_user_option("--shell", "shell") > + > + groups = d.getVarFlag(user_entry, "groups") or "" > + if groups: > + args.append("--groups") > + args.append(groups.replace(' ', ',')) > + > + flags = (d.getVarFlag(user_entry, "flags") or "").split() > + > + if exists: > + add_user_option("--home", "home") > + if d.getVarFlag(user_entry, "home") or "": > + args.append("--move-home") > + else: > + add_user_option("--home-dir", "home") > + > + if "system" in flags: > + args.append("--system") > + if "no-create-home" in flags: > + args.append("--no-create-home") > + if "create-home" in flags: > + args.append("--create-home") > + > + if exists: > + if args: > + bb.process.run([*chroot, "/usr/sbin/usermod", *args, > entry]) > + else: > + bb.process.run([*chroot, "/usr/sbin/useradd", *args, > entry]) + > + command = [*chroot, "/usr/sbin/chpasswd"] > + password = d.getVarFlag(user_entry, "password") or "" > + if password or "allow-empty-password" in flags: > + if "clear-text-password" in flags: > + > # chpasswd adds a random salt when running against a > clear-text password. # For reproducible images, we manually generate > the password and use the # SOURCE_DATE_EPOCH to generate the salt in > a deterministic way. > - if [ -z "${SOURCE_DATE_EPOCH}" ]; then > - chpasswd_args="" > - else > - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum > -z | cut -c 1-15)" > - password="$(openssl passwd -6 -salt $salt > "$password")" > - fi > - fi > - printf '%s:%s' "$name" "$password" | sudo chroot > '${ROOTFSDIR}' \ > - /usr/sbin/chpasswd $chpasswd_args > - fi > - if [ "${flags}" != "${flags%*,force-passwd-change,*}" ]; then > - echo "Execute passwd to force password change on first > boot for \"$name\"" > - sudo -E chroot '${ROOTFSDIR}' \ > - /usr/bin/passwd --expire "$name" > - fi > - done > + source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") or > "" > + if source_date_epoch: > + command.append("-e") > + salt = > hashlib.sha256("{}\n".format(source_date_epoch).encode()).hexdigest()[0:15] > + password = crypt.crypt(password, > "$6${}".format(salt)) + > + else: > + command.append("-e") > + > + bb.process.run(command, "{}:{}".format(entry, > password).encode()) + > + if "force-passwd-change" in flags: > + bb.process.run([*chroot, "/usr/sbin/passwd", "--expire", > entry]) + > + > +ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" > +python image_postprocess_accounts() { > + image_create_groups(d) > + image_create_users(d) > } ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/4] simplify image-account-extension 2023-03-30 19:55 ` Henning Schild @ 2023-03-30 21:39 ` Schaffner, Tobias 0 siblings, 0 replies; 7+ messages in thread From: Schaffner, Tobias @ 2023-03-30 21:39 UTC (permalink / raw) To: Schild, Henning; +Cc: isar-users, quirin.gylstorff On 30.03.23 21:55, Schild, Henning (T CED SES-DE) wrote: > Am Thu, 30 Mar 2023 13:08:01 +0200 > schrieb "T. Schaffner" <tobias.schaffner@siemens.com>: > >> From: Tobias Schaffner <tobias.schaffner@siemens.com> >> >> Do the complete user and group creation in python. This allows us to >> drop the encoding and parsing code that was used to make the user and >> group lists available in the shell function. >> >> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> >> --- >> meta/classes/image-account-extension.bbclass | 368 >> +++++++------------ 1 file changed, 124 insertions(+), 244 >> deletions(-) >> >> diff --git a/meta/classes/image-account-extension.bbclass >> b/meta/classes/image-account-extension.bbclass index >> 1a1f704d..d1133bb4 100644 --- >> a/meta/classes/image-account-extension.bbclass +++ >> b/meta/classes/image-account-extension.bbclass @@ -1,5 +1,5 @@ >> # This software is a part of ISAR. >> -# Copyright (C) Siemens AG, 2019 >> +# Copyright (C) Siemens AG, 2023 > > 2019-2023 > > did not look at the rest, this feels like it should be tried out ... > functional review, which i would try to do > > please make sure that would pass flake8 ... might be hard since it is > "embedded code" Hi Henning, the code already pep8 compliant and passes flake8 with two exceptions: * line length may be > 79 chars (E501) as everywhere else in the project * bb and os modules are not visible to flake8 (F821) as they are imported by bitbake Actually it is not that hard. If you want to use flake8 from the command line you can write a wrapper for the handle function in the bb.parse module similar to the bitbake module that I created in testsuite/unittests. If you print the python functions to the stdout you can pipe it to the stdin of flake8. If you use vscode with python extensions its even easier. Just configure your desired linter and set the Language Mode to Python for the open .bbclass. Best, Tobias > a long time ago we had contributions from Harald Seiler, who managed to > somehow lint python code inside bb code. > > Henning >> # >> # SPDX-License-Identifier: MIT >> # >> @@ -25,251 +25,131 @@ GROUPS ??= "" >> #GROUP_root[gid] = "" >> #GROUP_root[flags] = "system" >> >> -def gen_accounts_array(d, listname, entryname, flags, >> verb_flags=None): >> - from itertools import chain >> - >> - entries = (d.getVar(listname) or "").split() >> - return " ".join( >> - ":".join( >> - chain( >> - (entry,), >> - ( >> - (",".join( >> - ( >> - d.getVarFlag(entryname + "_" + entry, >> flag, True) or "" >> - ).split() >> - ) if flag not in (verb_flags or []) else ( >> - d.getVarFlag(entryname + "_" + entry, flag, >> True) or "" >> - )).replace(":","=") >> - for flag in flags >> - ), >> - ) >> - ) >> - for entry in entries >> - ) >> - >> -# List of space separated entries, where each entry has the format: >> -# >> username:encryptedpassword:expiredate:inactivenumber:userid:groupid:comment:homedir:shell:group1,group2:flag1,flag2 >> -IMAGE_ACCOUNTS_USERS =+ "${@gen_accounts_array(d, 'USERS', 'USER', >> ['password', 'expire', 'inactive', 'uid', 'gid', 'comment', 'home', >> 'shell', 'groups', 'flags'], ['password', 'comment', 'home', >> 'shell'])}" - -# List of space separated entries, where each entry >> has the format: -# groupname:groupid:flag1,flag2 >> -IMAGE_ACCOUNTS_GROUPS =+ "${@gen_accounts_array(d, 'GROUPS', >> 'GROUP', ['gid', 'flags'])}" - -do_rootfs_install[vardeps] += >> "${IMAGE_ACCOUNTS_GROUPS} ${IMAGE_ACCOUNTS_USERS}" >> -ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" >> -image_postprocess_accounts() { >> - # Create groups >> - # Add space to the end of the list: >> - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_GROUPS').split())} ' >> - while true; do >> - # Pop first group entry: >> - list_rest="${list#*:*:* }" >> - entry="${list%%${list_rest}}" >> - list="${list_rest}" >> - >> - if [ -z "${entry}" ]; then >> - break >> - fi >> - >> - # Add colon to the end of the entry and remove trailing >> space: >> - entry="${entry% }:" >> - >> - # Decode entries: >> - name="${entry%%:*}" >> - entry="${entry#${name}:}" >> - >> - gid="${entry%%:*}" >> - entry="${entry#${gid}:}" >> - >> - flags="${entry%%:*}" >> - entry="${entry#${flags}:}" >> - >> - flags=",${flags}," # Needed for searching for substrings >> - >> - # Check if user already exists: >> - if grep -q "^${name}:" '${ROOTFSDIR}/etc/group'; then >> - exists="y" >> - else >> - exists="n" >> - fi >> - >> - # Create arguments: >> - set -- # clear arguments >> - >> - if [ -n "$gid" ]; then >> - set -- "$@" --gid "$gid" >> - fi >> - >> - if [ "n" = "$exists" ]; then >> - if [ "${flags}" != "${flags%*,system,*}" ]; then >> - set -- "$@" --system >> - fi >> - fi >> - >> - # Create or modify groups: >> - if [ "y" = "$exists" ]; then >> - if [ -z "$@" ]; then >> - echo "Do not execute groupmod (no changes)." >> - else >> - echo "Execute groupmod with \"$@\" for \"$name\"" >> - sudo -E chroot '${ROOTFSDIR}' \ >> - /usr/sbin/groupmod "$@" "$name" >> - fi >> - else >> - echo "Execute groupadd with \"$@\" for \"$name\"" >> - sudo -E chroot '${ROOTFSDIR}' \ >> - /usr/sbin/groupadd "$@" "$name" >> - fi >> - done >> - >> - # Create users >> - list='${@" ".join(d.getVar('IMAGE_ACCOUNTS_USERS').split())} ' >> - while true; do >> - # Pop first user entry: >> - list_rest="${list#*:*:*:*:*:*:*:*:*:*:* }" >> - entry="${list%%${list_rest}}" >> - list="${list_rest}" >> - >> - if [ -z "${entry}" ]; then >> - break >> - fi >> - >> - # Add colon to the end of the entry and remove trailing >> space: >> - entry="${entry% }:" >> - >> - # Decode entries: >> - name="${entry%%:*}" >> - entry="${entry#${name}:}" >> - >> - password="${entry%%:*}" >> - entry="${entry#${password}:}" >> - >> - expire="${entry%%:*}" >> - entry="${entry#${expire}:}" >> - >> - inactive="${entry%%:*}" >> - entry="${entry#${inactive}:}" >> - >> - uid="${entry%%:*}" >> - entry="${entry#${uid}:}" >> - >> - gid="${entry%%:*}" >> - entry="${entry#${gid}:}" >> - >> - comment="${entry%%:*}" >> - entry="${entry#${comment}:}" >> - >> - home="${entry%%:*}" >> - entry="${entry#${home}:}" >> - >> - shell="${entry%%:*}" >> - entry="${entry#${shell}:}" >> - >> - groups="${entry%%:*}" >> - entry="${entry#${groups}:}" >> - >> - flags="${entry%%:*}" >> - entry="${entry#${flags}:}" >> - >> - flags=",${flags}," # Needed for searching for substrings >> - >> - # Check if user already exists: >> - if grep -q "^${name}:" '${ROOTFSDIR}/etc/passwd'; then >> - exists="y" >> - else >> - exists="n" >> - fi >> - >> - # Create arguments: >> - set -- # clear arguments >> - >> - if [ -n "$expire" ]; then >> - set -- "$@" --expiredate "$expire" >> - fi >> - >> - if [ -n "$inactive" ]; then >> - set -- "$@" --inactive "$inactive" >> - fi >> - >> - if [ -n "$uid" ]; then >> - set -- "$@" --uid "$uid" >> - fi >> - >> - if [ -n "$gid" ]; then >> - set -- "$@" --gid "$gid" >> - fi >> - >> - if [ -n "$comment" ]; then >> - set -- "$@" --comment "$comment" >> - fi >> - >> - if [ -n "$home" ]; then >> - if [ "y" = "$exists" ]; then >> - set -- "$@" --home "$home" --move-home >> - else >> - set -- "$@" --home-dir "$home" >> - fi >> - fi >> - >> - if [ -n "$shell" ]; then >> - set -- "$@" --shell "$shell" >> - fi >> - >> - if [ -n "$groups" ]; then >> - set -- "$@" --groups "$groups" >> - fi >> - >> - if [ "n" = "$exists" ]; then >> - if [ "${flags}" != "${flags%*,system,*}" ]; then >> - set -- "$@" --system >> - fi >> - if [ "${flags}" != "${flags%*,no-create-home,*}" ]; then >> - set -- "$@" --no-create-home >> - else >> - if [ "${flags}" != "${flags%*,create-home,*}" ]; then >> - set -- "$@" --create-home >> - fi >> - fi >> - fi >> - >> - # Create or modify users: >> - if [ "y" = "$exists" ]; then >> - if [ -z "$@" ]; then >> - echo "Do not execute usermod (no changes)." >> - else >> - echo "Execute usermod with \"$@\" for \"$name\"" >> - sudo -E chroot '${ROOTFSDIR}' \ >> - /usr/sbin/usermod "$@" "$name" >> - fi >> - else >> - echo "Execute useradd with \"$@\" for \"$name\"" >> - sudo -E chroot '${ROOTFSDIR}' \ >> - /usr/sbin/useradd "$@" "$name" >> - fi >> - >> - # Set password: >> - if [ -n "$password" -o "${flags}" != >> "${flags%*,allow-empty-password,*}" ]; then >> - chpasswd_args="-e" >> - if [ "${flags}" != "${flags%*,clear-text-password,*}" ]; >> then +def image_create_groups(d: "DataSmart") -> None: >> + """Creates the groups defined in the ``GROUPS`` bitbake variable. >> + >> + Args: >> + d (DataSmart): The bitbake datastore. >> + >> + Returns: >> + None >> + """ >> + entries = (d.getVar("GROUPS") or "").split() >> + rootfsdir = d.getVar("ROOTFSDIR") >> + chroot = ["sudo", "-E", "chroot", rootfsdir] >> + >> + for entry in entries: >> + args = [] >> + group_entry = "GROUP_{}".format(entry) >> + >> + with open("{}/etc/group".format(rootfsdir), "r") as >> group_file: >> + exists = any(line.startswith("{}:".format(entry)) for >> line in group_file) + >> + gid = d.getVarFlag(group_entry, "gid") or "" >> + if gid: >> + args.append("--gid") >> + args.append(gid) >> + >> + flags = (d.getVarFlag(group_entry, "flags") or "").split() >> + if "system" in flags: >> + args.append("--system") >> + >> + if exists: >> + if args: >> + bb.process.run([*chroot, "/usr/sbin/groupmod", >> *args, entry]) >> + else: >> + bb.process.run([*chroot, "/usr/sbin/groupadd", *args, >> entry]) + >> + >> +def image_create_users(d: "DataSmart") -> None: >> + """Creates the users defined in the ``USERS`` bitbake variable. >> + >> + Args: >> + d (DataSmart): The bitbake datastore. >> + >> + Returns: >> + None >> + """ >> + import hashlib >> + import crypt >> + >> + entries = (d.getVar("USERS") or "").split() >> + rootfsdir = d.getVar("ROOTFSDIR") >> + chroot = ["sudo", "-E", "chroot", rootfsdir] >> + >> + for entry in entries: >> + args = [] >> + user_entry = "USER_{}".format(entry) >> + >> + with open("{}/etc/passwd".format(rootfsdir), "r") as >> passwd_file: >> + exists = any(line.startswith("{}:".format(entry)) for >> line in passwd_file) + >> + def add_user_option(option_name, flag_name): >> + flag_value = d.getVarFlag(user_entry, flag_name) or "" >> + if flag_value: >> + args.append(option_name) >> + args.append(flag_value) >> + >> + add_user_option("--expire", "expiredate") >> + add_user_option("--inactive", "inactive") >> + add_user_option("--uid", "uid") >> + add_user_option("--gid", "gid") >> + add_user_option("--comment", "comment") >> + add_user_option("--shell", "shell") >> + >> + groups = d.getVarFlag(user_entry, "groups") or "" >> + if groups: >> + args.append("--groups") >> + args.append(groups.replace(' ', ',')) >> + >> + flags = (d.getVarFlag(user_entry, "flags") or "").split() >> + >> + if exists: >> + add_user_option("--home", "home") >> + if d.getVarFlag(user_entry, "home") or "": >> + args.append("--move-home") >> + else: >> + add_user_option("--home-dir", "home") >> + >> + if "system" in flags: >> + args.append("--system") >> + if "no-create-home" in flags: >> + args.append("--no-create-home") >> + if "create-home" in flags: >> + args.append("--create-home") >> + >> + if exists: >> + if args: >> + bb.process.run([*chroot, "/usr/sbin/usermod", *args, >> entry]) >> + else: >> + bb.process.run([*chroot, "/usr/sbin/useradd", *args, >> entry]) + >> + command = [*chroot, "/usr/sbin/chpasswd"] >> + password = d.getVarFlag(user_entry, "password") or "" >> + if password or "allow-empty-password" in flags: >> + if "clear-text-password" in flags: >> + >> # chpasswd adds a random salt when running against a >> clear-text password. # For reproducible images, we manually generate >> the password and use the # SOURCE_DATE_EPOCH to generate the salt in >> a deterministic way. >> - if [ -z "${SOURCE_DATE_EPOCH}" ]; then >> - chpasswd_args="" >> - else >> - salt="$(echo "${SOURCE_DATE_EPOCH}" | sha256sum >> -z | cut -c 1-15)" >> - password="$(openssl passwd -6 -salt $salt >> "$password")" >> - fi >> - fi >> - printf '%s:%s' "$name" "$password" | sudo chroot >> '${ROOTFSDIR}' \ >> - /usr/sbin/chpasswd $chpasswd_args >> - fi >> - if [ "${flags}" != "${flags%*,force-passwd-change,*}" ]; then >> - echo "Execute passwd to force password change on first >> boot for \"$name\"" >> - sudo -E chroot '${ROOTFSDIR}' \ >> - /usr/bin/passwd --expire "$name" >> - fi >> - done >> + source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") or >> "" >> + if source_date_epoch: >> + command.append("-e") >> + salt = >> hashlib.sha256("{}\n".format(source_date_epoch).encode()).hexdigest()[0:15] >> + password = crypt.crypt(password, >> "$6${}".format(salt)) + >> + else: >> + command.append("-e") >> + >> + bb.process.run(command, "{}:{}".format(entry, >> password).encode()) + >> + if "force-passwd-change" in flags: >> + bb.process.run([*chroot, "/usr/sbin/passwd", "--expire", >> entry]) + >> + >> +ROOTFS_POSTPROCESS_COMMAND += "image_postprocess_accounts" >> +python image_postprocess_accounts() { >> + image_create_groups(d) >> + image_create_users(d) >> } > ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/4] create a minimal python unittest infrastructure 2023-03-30 11:08 [PATCH 0/4] Rewrite the image-account-extension in python T. Schaffner 2023-03-30 11:08 ` [PATCH 1/4] simplify image-account-extension T. Schaffner @ 2023-03-30 11:08 ` T. Schaffner 2023-03-30 11:08 ` [PATCH 3/4] add unittests for the image-account-extension T. Schaffner 2023-03-30 11:08 ` [PATCH 4/4] set minimal python version in user_manual to 3.5 T. Schaffner 3 siblings, 0 replies; 7+ messages in thread From: T. Schaffner @ 2023-03-30 11:08 UTC (permalink / raw) To: isar-users; +Cc: quirin.gylstorff, henning.schild, Tobias Schaffner From: Tobias Schaffner <tobias.schaffner@siemens.com> Add some some infrastructure for python unittesting. The unittest_isar module exposes a function that uses the bb.parse module to import python functions that are defined in bitbake files. Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> --- testsuite/unittests/README.md | 28 +++++++++++++++++++++ testsuite/unittests/bitbake.py | 37 ++++++++++++++++++++++++++++ testsuite/unittests/rootfs.py | 45 ++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+) create mode 100644 testsuite/unittests/README.md create mode 100644 testsuite/unittests/bitbake.py create mode 100644 testsuite/unittests/rootfs.py diff --git a/testsuite/unittests/README.md b/testsuite/unittests/README.md new file mode 100644 index 00000000..75a3bb01 --- /dev/null +++ b/testsuite/unittests/README.md @@ -0,0 +1,28 @@ +# Isar Unittests + +The unittest python module adds some simple infrastructure that allows to +unittest python functions defined in bitbake files. + +## Running the tests + +You can run the tests using avocado with `avocado --show=app,test run testsuite/unittests/` +or by using the buildin module with `python3 -m unittest discover testsuite/unittests/` + +## Creating new tests + +See the [unittest documentation](https://docs.python.org/3/library/unittest.html) +on how to create a test module and name it test_*bitbake_module_name*.py + +Use the function `load_function(file_name: str, function_name: str) -> Callable` +in the bitbake module to load the function. + +Example: +```python +from bitbake import load_function + +my_function = load_function("meta/classes/my_module.bbclass", "my_function") +my_function(arg1, arg2) +``` + +Use the [unittest.mock](https://docs.python.org/3/library/unittest.mock.html) +library to mock the bb modules as needed. diff --git a/testsuite/unittests/bitbake.py b/testsuite/unittests/bitbake.py new file mode 100644 index 00000000..1e2f685a --- /dev/null +++ b/testsuite/unittests/bitbake.py @@ -0,0 +1,37 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2023 +# +# SPDX-License-Identifier: MIT + +import sys +import pathlib +from typing import Callable + +location = pathlib.Path(__file__).parent.resolve() +sys.path.insert(0, "{}/../../bitbake/lib".format(location)) + +from bb.parse import handle +from bb.data import init + +# Modules added for reimport from testfiles +from bb.data_smart import DataSmart + + +def load_function(file_name: str, function_name: str) -> Callable: + """Load a python function defined in a bitbake file. + + Args: + file_name (str): The path to the file e.g. `meta/classes/my_special.bbclass`. + function_name (str): The name of the python function without braces e.g. `my_special_function` + + Returns: + Callable: The loaded function. + """ + d = init() + parse = handle("{}/../../{}".format(location, file_name), d) + if function_name not in parse: + raise KeyError("Function {} does not exist in {}".format( + function_name, file_name)) + namespace = {} + exec(parse[function_name], namespace) + return namespace[function_name] diff --git a/testsuite/unittests/rootfs.py b/testsuite/unittests/rootfs.py new file mode 100644 index 00000000..6c511493 --- /dev/null +++ b/testsuite/unittests/rootfs.py @@ -0,0 +1,45 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2023 +# +# SPDX-License-Identifier: MIT + +import tempfile +import pathlib +import shutil +import atexit + +temp_dirs = [] + + +class TemporaryRootfs: + """ A temporary rootfs folder that will be removed after the testrun. """ + + def __init__(self): + self._rootfs_path = tempfile.mkdtemp() + temp_dirs.append(self._rootfs_path) + + def path(self) -> str: + return self._rootfs_path + + def create_file(self, path: str, content: str) -> None: + """ Create a file with the given content. + + Args: + path (str): The path to the file e.g. `/etc/hostname`. + content (str): The content of the file e.g. `my_special_host` + + Returns: + None + """ + pathlib.Path(self._rootfs_path + + path).parent.mkdir(parents=True, exist_ok=True) + with open(self._rootfs_path + path, 'w') as file: + file.write(content) + + +def cleanup(): + for temp_dir in temp_dirs: + shutil.rmtree(temp_dir) + + +atexit.register(cleanup) -- 2.34.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/4] add unittests for the image-account-extension 2023-03-30 11:08 [PATCH 0/4] Rewrite the image-account-extension in python T. Schaffner 2023-03-30 11:08 ` [PATCH 1/4] simplify image-account-extension T. Schaffner 2023-03-30 11:08 ` [PATCH 2/4] create a minimal python unittest infrastructure T. Schaffner @ 2023-03-30 11:08 ` T. Schaffner 2023-03-30 11:08 ` [PATCH 4/4] set minimal python version in user_manual to 3.5 T. Schaffner 3 siblings, 0 replies; 7+ messages in thread From: T. Schaffner @ 2023-03-30 11:08 UTC (permalink / raw) To: isar-users; +Cc: quirin.gylstorff, henning.schild, Tobias Schaffner From: Tobias Schaffner <tobias.schaffner@siemens.com> This is a first example on how to use the unittest_isar module to test python functions defined in a bitbake file. Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> --- .../unittests/test_image_account_extension.py | 145 ++++++++++++++++++ 1 file changed, 145 insertions(+) create mode 100644 testsuite/unittests/test_image_account_extension.py diff --git a/testsuite/unittests/test_image_account_extension.py b/testsuite/unittests/test_image_account_extension.py new file mode 100644 index 00000000..d35de723 --- /dev/null +++ b/testsuite/unittests/test_image_account_extension.py @@ -0,0 +1,145 @@ +# This software is a part of ISAR. +# Copyright (C) Siemens AG, 2023 +# +# SPDX-License-Identifier: MIT + +from bitbake import load_function, DataSmart +from rootfs import TemporaryRootfs + +import unittest +from unittest.mock import patch +from typing import Tuple + + +file_name = "meta/classes/image-account-extension.bbclass" +image_create_users = load_function(file_name, "image_create_users") +image_create_groups = load_function(file_name, "image_create_groups") + + +class TestImageAccountExtensionCommon(unittest.TestCase): + + def setup(self) -> Tuple[DataSmart, TemporaryRootfs]: + rootfs = TemporaryRootfs() + + d = DataSmart() + d.setVar("ROOTFSDIR", rootfs.path()) + + return (d, rootfs) + + +class TestImageAccountExtensionImageCreateUsers(TestImageAccountExtensionCommon): + + def setup(self, user_name: str) -> Tuple[DataSmart, TemporaryRootfs]: + d, rootfs = super().setup() + rootfs.create_file( + "/etc/passwd", "test:x:1000:1000::/home/test:/bin/sh") + d.setVar("USERS", user_name) + return (d, rootfs) + + def test_new_user(self): + test_user = "new" + d, rootfs = self.setup(test_user) + + with patch.object(bb.process, "run") as run_mock: + image_create_users(d) + + run_mock.assert_called_once_with( + ["sudo", "-E", "chroot", rootfs.path(), "/usr/sbin/useradd", test_user]) + + def test_existing_user_no_change(self): + test_user = "test" + d, _ = self.setup(test_user) + + with patch.object(bb.process, "run") as run_mock: + image_create_users(d) + + run_mock.assert_not_called() + + def test_existing_user_home_change(self): + test_user = "test" + d, _ = self.setup(test_user) + d.setVarFlag("USER_{}".format(test_user), "home", "/home/new_home") + + with patch.object(bb.process, "run") as run_mock: + image_create_users(d) + + assert run_mock.call_count == 1 + assert run_mock.call_args[0][0][-5:] == ["/usr/sbin/usermod", + '--home', '/home/new_home', '--move-home', 'test'] + + def test_deterministic_password(self): + test_user = "new" + cleartext_password = "test" + d, _ = self.setup(test_user) + + d.setVarFlag("USER_{}".format(test_user), + "flags", "clear-text-password") + d.setVarFlag("USER_{}".format(test_user), + "password", cleartext_password) + + source_date_epoch = "1672427776" + d.setVar("SOURCE_DATE_EPOCH", source_date_epoch) + + # openssl passwd -6 -salt $(echo "1672427776" | sha256sum -z | cut -c 1-15) test + encrypted_password = "$6$eb2e2a12cccc88a$IuhgisFe5AKM5.VREKg8wIAcPSkaJDWBM1cMUsEjNZh2Wa6BT2f5OFhqGTGpL4lFzHGN8oiwvAh0jFO1GhO3S." + + with patch.object(bb.process, "run") as run_mock: + image_create_users(d) + + assert run_mock.call_count == 2 + assert run_mock.call_args[0][1] == "{}:{}".format( + test_user, encrypted_password).encode() + + +class TestImageAccountExtensionImageCreateGroups(TestImageAccountExtensionCommon): + + def setup(self, group_name: str) -> Tuple[DataSmart, TemporaryRootfs]: + d, rootfs = super().setup() + rootfs.create_file("/etc/group", "test:x:1000:test") + d.setVar("GROUPS", group_name) + return (d, rootfs) + + def test_new_group(self): + test_group = "new" + d, rootfs = self.setup(test_group) + + with patch.object(bb.process, "run") as run_mock: + image_create_groups(d) + + run_mock.assert_called_once_with( + ["sudo", "-E", "chroot", rootfs.path(), "/usr/sbin/groupadd", test_group]) + + def test_existing_group_no_change(self): + test_group = "test" + d, _ = self.setup(test_group) + + with patch.object(bb.process, "run") as run_mock: + image_create_groups(d) + + run_mock.assert_not_called() + + def test_existing_group_id_change(self): + test_group = "test" + d, rootfs = self.setup(test_group) + d.setVarFlag("GROUP_{}".format(test_group), "gid", "1005") + + with patch.object(bb.process, "run") as run_mock: + image_create_groups(d) + + run_mock.assert_called_once_with( + ["sudo", "-E", "chroot", rootfs.path(), "/usr/sbin/groupmod", "--gid", "1005", test_group]) + + def test_system_flag(self): + test_group = "test" + d, _ = self.setup(test_group) + d.setVarFlag("GROUP_{}".format(test_group), "flags", "system") + + with patch.object(bb.process, "run") as run_mock: + image_create_groups(d) + + assert run_mock.call_count == 1 + assert "--system" in run_mock.call_args[0][0] + + +if __name__ == "__main__": + unittest.main() -- 2.34.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 4/4] set minimal python version in user_manual to 3.5 2023-03-30 11:08 [PATCH 0/4] Rewrite the image-account-extension in python T. Schaffner ` (2 preceding siblings ...) 2023-03-30 11:08 ` [PATCH 3/4] add unittests for the image-account-extension T. Schaffner @ 2023-03-30 11:08 ` T. Schaffner 3 siblings, 0 replies; 7+ messages in thread From: T. Schaffner @ 2023-03-30 11:08 UTC (permalink / raw) To: isar-users; +Cc: quirin.gylstorff, henning.schild, Tobias Schaffner From: Tobias Schaffner <tobias.schaffner@siemens.com> Set the minimal python version to 3.5 in the user_manual as defined in bitbake/lib/bb/__init__.py Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com> --- doc/user_manual.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user_manual.md b/doc/user_manual.md index 2b8542df..5023b37b 100644 --- a/doc/user_manual.md +++ b/doc/user_manual.md @@ -111,7 +111,7 @@ unstable/bullseye included in `/etc/apt/sources.list[.d]`). Notes: -* BitBake requires Python 3.4+. +* BitBake requires Python 3.5+. * The python3 package is required for the correct `alternatives` setting. * If you'd like to run bitbake in a container (chroot, docker, etc.), install the above in the container, and also perform `sudo apt-get install -- 2.34.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-03-30 21:39 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-03-30 11:08 [PATCH 0/4] Rewrite the image-account-extension in python T. Schaffner 2023-03-30 11:08 ` [PATCH 1/4] simplify image-account-extension T. Schaffner 2023-03-30 19:55 ` Henning Schild 2023-03-30 21:39 ` Schaffner, Tobias 2023-03-30 11:08 ` [PATCH 2/4] create a minimal python unittest infrastructure T. Schaffner 2023-03-30 11:08 ` [PATCH 3/4] add unittests for the image-account-extension T. Schaffner 2023-03-30 11:08 ` [PATCH 4/4] set minimal python version in user_manual to 3.5 T. Schaffner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox