* [PATCH v2 0/2] bump bitbake and wic for python 3.10 support
@ 2022-04-21 8:52 Henning Schild
2022-04-21 8:52 ` [PATCH v2 1/2] bitbake: Update to 1.50.5 release Henning Schild
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Henning Schild @ 2022-04-21 8:52 UTC (permalink / raw)
To: isar-users; +Cc: Florian Bezdeka, Felix Moessbauer, Henning Schild
debian bookworm switched to python 3.10 recently, our versions of
bitbake and wic both have some issues with that new interpreter. All
these issues have been fixes in bitbake and OE, so all we need to do is
update the two tools in our tree.
v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", which is p1
here, only later i found that wic also needs a bump to v2 added p2
If this series is applied we do not need "[PATCH] wic-img: use
python3.9 in bookworm", in case this one is applied before it should be
reverted after the version bumps.
Henning Schild (2):
bitbake: Update to 1.50.5 release
wic: Update to the latest revision
bitbake/lib/bb/cache.py | 3 +-
bitbake/lib/bb/cooker.py | 30 +++++++-
bitbake/lib/bb/data_smart.py | 4 +-
bitbake/lib/bb/fetch2/__init__.py | 4 +
bitbake/lib/bb/fetch2/perforce.py | 2 +-
bitbake/lib/bb/fetch2/wget.py | 2 +-
bitbake/lib/bb/persist_data.py | 5 +-
bitbake/lib/bb/process.py | 2 +-
bitbake/lib/bb/runqueue.py | 34 ++++----
bitbake/lib/bb/server/process.py | 2 +-
bitbake/lib/bb/tests/fetch.py | 35 +++++----
bitbake/lib/bb/utils.py | 13 +++-
bitbake/lib/hashserv/server.py | 4 +-
bitbake/lib/toaster/tests/builds/buildtest.py | 2 +-
.../wic/plugins/source/bootimg-efi-isar.py | 77 ++++++++++++++++---
.../wic/plugins/source/bootimg-pcbios-isar.py | 6 +-
scripts/lib/wic/canned-wks/common.wks.inc | 2 +-
scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +-
scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +-
scripts/lib/wic/engine.py | 6 +-
scripts/lib/wic/help.py | 10 ++-
scripts/lib/wic/ksparser.py | 8 +-
scripts/lib/wic/misc.py | 4 +-
scripts/lib/wic/partition.py | 25 ++++--
scripts/lib/wic/pluginbase.py | 8 +-
scripts/lib/wic/plugins/imager/direct.py | 11 ++-
scripts/lib/wic/plugins/source/bootimg-efi.py | 74 +++++++++++++++---
.../lib/wic/plugins/source/bootimg-pcbios.py | 6 +-
scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++-
scripts/lib/wic/plugins/source/rootfs.py | 2 +-
scripts/wic | 9 ++-
31 files changed, 320 insertions(+), 109 deletions(-)
--
2.35.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [PATCH v2 1/2] bitbake: Update to 1.50.5 release 2022-04-21 8:52 [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Henning Schild @ 2022-04-21 8:52 ` Henning Schild 2022-04-21 8:52 ` [PATCH v2 2/2] wic: Update to the latest revision Henning Schild 2022-04-22 7:59 ` [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Anton Mikanovich 2 siblings, 0 replies; 11+ messages in thread From: Henning Schild @ 2022-04-21 8:52 UTC (permalink / raw) To: isar-users; +Cc: Florian Bezdeka, Felix Moessbauer, Henning Schild Update bitbake to the latest release in 1.50 branch. This release is tagged to the commit ID aaa7f7af23d5f89fe4a5ed48c57ea3dfca07c79d in the bitbake upstream. Signed-off-by: Henning Schild <henning.schild@siemens.com> --- bitbake/lib/bb/cache.py | 3 +- bitbake/lib/bb/cooker.py | 30 ++++++++++++++-- bitbake/lib/bb/data_smart.py | 4 +-- bitbake/lib/bb/fetch2/__init__.py | 4 +++ bitbake/lib/bb/fetch2/perforce.py | 2 +- bitbake/lib/bb/fetch2/wget.py | 2 +- bitbake/lib/bb/persist_data.py | 5 +-- bitbake/lib/bb/process.py | 2 +- bitbake/lib/bb/runqueue.py | 34 +++++++++--------- bitbake/lib/bb/server/process.py | 2 +- bitbake/lib/bb/tests/fetch.py | 35 ++++++++++--------- bitbake/lib/bb/utils.py | 13 +++++-- bitbake/lib/hashserv/server.py | 4 +-- bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- 14 files changed, 89 insertions(+), 53 deletions(-) diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py index 27eb271798e8..5f9c0a779d75 100644 --- a/bitbake/lib/bb/cache.py +++ b/bitbake/lib/bb/cache.py @@ -19,7 +19,8 @@ import os import logging import pickle -from collections import defaultdict, Mapping +from collections import defaultdict +from collections.abc import Mapping import bb.utils from bb import PrefixLoggerAdapter import re diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 89f1fad08310..c946800a8c62 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -388,12 +388,22 @@ class BBCooker: # Create a new hash server bound to a unix domain socket if not self.hashserv: dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db" + upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None + if upstream: + import socket + try: + sock = socket.create_connection(upstream.split(":"), 5) + sock.close() + except socket.error as e: + bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s" + % (upstream, repr(e))) + self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR") self.hashserv = hashserv.create_server( self.hashservaddr, dbfile, sync=False, - upstream=self.data.getVar("BB_HASHSERVE_UPSTREAM") or None, + upstream=upstream, ) self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever) self.hashserv.process.start() @@ -805,7 +815,9 @@ class BBCooker: for dep in rq.rqdata.runtaskentries[tid].depends: (depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep) deppn = self.recipecaches[depmc].pkg_fn[deptaskfn] - depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep))) + if depmc: + depmc = "mc:" + depmc + ":" + depend_tree["tdepends"][dotname].append("%s%s.%s" % (depmc, deppn, bb.runqueue.taskname_from_tid(dep))) if taskfn not in seen_fns: seen_fns.append(taskfn) packages = [] @@ -2204,21 +2216,33 @@ class CookerParser(object): yield not cached, mc, infos def parse_generator(self): - while True: + empty = False + while self.processes or not empty: + for process in self.processes.copy(): + if not process.is_alive(): + process.join() + self.processes.remove(process) + if self.parsed >= self.toparse: break try: result = self.result_queue.get(timeout=0.25) except queue.Empty: + empty = True pass else: + empty = False value = result[1] if isinstance(value, BaseException): raise value else: yield result + if not (self.parsed >= self.toparse): + raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None) + + def parse_next(self): result = [] parsed = None diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py index 8291ca65e309..65857a9c7941 100644 --- a/bitbake/lib/bb/data_smart.py +++ b/bitbake/lib/bb/data_smart.py @@ -17,7 +17,7 @@ BitBake build tools. # Based on functions from the base bb module, Copyright 2003 Holger Schurig import copy, re, sys, traceback -from collections import MutableMapping +from collections.abc import MutableMapping import logging import hashlib import bb, bb.codeparser @@ -403,7 +403,7 @@ class DataSmart(MutableMapping): s = __expand_python_regexp__.sub(varparse.python_sub, s) except SyntaxError as e: # Likely unmatched brackets, just don't expand the expression - if e.msg != "EOL while scanning string literal": + if e.msg != "EOL while scanning string literal" and not e.msg.startswith("unterminated string literal"): raise if s == olds: break diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index dbf8b50e68a7..1005ec10c639 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -430,6 +430,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): uri_replace_decoded = list(decodeurl(uri_replace)) logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded)) result_decoded = ['', '', '', '', '', {}] + # 0 - type, 1 - host, 2 - path, 3 - user, 4- pswd, 5 - params for loc, i in enumerate(uri_find_decoded): result_decoded[loc] = uri_decoded[loc] regexp = i @@ -449,6 +450,9 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None): for l in replacements: uri_replace_decoded[loc][k] = uri_replace_decoded[loc][k].replace(l, replacements[l]) result_decoded[loc][k] = uri_replace_decoded[loc][k] + elif (loc == 3 or loc == 4) and uri_replace_decoded[loc]: + # User/password in the replacement is just a straight replacement + result_decoded[loc] = uri_replace_decoded[loc] elif (re.match(regexp, uri_decoded[loc])): if not uri_replace_decoded[loc]: result_decoded[loc] = "" diff --git a/bitbake/lib/bb/fetch2/perforce.py b/bitbake/lib/bb/fetch2/perforce.py index e2a41a4a1287..3b6fa4b1ec9a 100644 --- a/bitbake/lib/bb/fetch2/perforce.py +++ b/bitbake/lib/bb/fetch2/perforce.py @@ -134,7 +134,7 @@ class Perforce(FetchMethod): ud.setup_revisions(d) - ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, cleandedmodule, ud.revision)) + ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, cleanedpath, cleanedmodule, ud.revision)) def _buildp4command(self, ud, d, command, depot_filename=None): """ diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index 784df70c9f62..7fa2a87ffde5 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -322,7 +322,7 @@ class Wget(FetchMethod): except (TypeError, ImportError, IOError, netrc.NetrcParseError): pass - with opener.open(r) as response: + with opener.open(r, timeout=30) as response: pass except urllib.error.URLError as e: if try_again: diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py index c6a209fb3fc1..6f32d81afe80 100644 --- a/bitbake/lib/bb/persist_data.py +++ b/bitbake/lib/bb/persist_data.py @@ -12,6 +12,7 @@ currently, providing a key/value store accessed by 'domain'. # import collections +import collections.abc import contextlib import functools import logging @@ -19,7 +20,7 @@ import os.path import sqlite3 import sys import warnings -from collections import Mapping +from collections.abc import Mapping sqlversion = sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): @@ -29,7 +30,7 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): logger = logging.getLogger("BitBake.PersistData") @functools.total_ordering -class SQLTable(collections.MutableMapping): +class SQLTable(collections.abc.MutableMapping): class _Decorators(object): @staticmethod def retry(*, reconnect=True): diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py index d5a1775fcec0..af5d804a1d59 100644 --- a/bitbake/lib/bb/process.py +++ b/bitbake/lib/bb/process.py @@ -60,7 +60,7 @@ class Popen(subprocess.Popen): "close_fds": True, "preexec_fn": subprocess_setup, "stdout": subprocess.PIPE, - "stderr": subprocess.STDOUT, + "stderr": subprocess.PIPE, "stdin": subprocess.PIPE, "shell": False, } diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 10511a09dc1c..cd10da8b3a6f 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -926,38 +926,36 @@ class RunQueueData: # # Once all active tasks are marked, prune the ones we don't need. - delcount = {} - for tid in list(self.runtaskentries.keys()): - if tid not in runq_build: - delcount[tid] = self.runtaskentries[tid] - del self.runtaskentries[tid] - # Handle --runall if self.cooker.configuration.runall: # re-run the mark_active and then drop unused tasks from new list + reduced_tasklist = set(self.runtaskentries.keys()) + for tid in list(self.runtaskentries.keys()): + if tid not in runq_build: + reduced_tasklist.remove(tid) runq_build = {} for task in self.cooker.configuration.runall: if not task.startswith("do_"): task = "do_{0}".format(task) runall_tids = set() - for tid in list(self.runtaskentries): + for tid in reduced_tasklist: wanttid = "{0}:{1}".format(fn_from_tid(tid), task) - if wanttid in delcount: - self.runtaskentries[wanttid] = delcount[wanttid] if wanttid in self.runtaskentries: runall_tids.add(wanttid) for tid in list(runall_tids): - mark_active(tid,1) + mark_active(tid, 1) if self.cooker.configuration.force: invalidate_task(tid, False) - for tid in list(self.runtaskentries.keys()): - if tid not in runq_build: - delcount[tid] = self.runtaskentries[tid] - del self.runtaskentries[tid] + delcount = set() + for tid in list(self.runtaskentries.keys()): + if tid not in runq_build: + delcount.add(tid) + del self.runtaskentries[tid] + if self.cooker.configuration.runall: if len(self.runtaskentries) == 0: bb.msg.fatal("RunQueue", "Could not find any tasks with the tasknames %s to run within the recipes of the taskgraphs of the targets %s" % (str(self.cooker.configuration.runall), str(self.targets))) @@ -971,16 +969,16 @@ class RunQueueData: for task in self.cooker.configuration.runonly: if not task.startswith("do_"): task = "do_{0}".format(task) - runonly_tids = { k: v for k, v in self.runtaskentries.items() if taskname_from_tid(k) == task } + runonly_tids = [k for k in self.runtaskentries.keys() if taskname_from_tid(k) == task] - for tid in list(runonly_tids): - mark_active(tid,1) + for tid in runonly_tids: + mark_active(tid, 1) if self.cooker.configuration.force: invalidate_task(tid, False) for tid in list(self.runtaskentries.keys()): if tid not in runq_build: - delcount[tid] = self.runtaskentries[tid] + delcount.add(tid) del self.runtaskentries[tid] if len(self.runtaskentries) == 0: diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py index 07bb785a1822..fcdce19717d2 100644 --- a/bitbake/lib/bb/server/process.py +++ b/bitbake/lib/bb/server/process.py @@ -659,7 +659,7 @@ class BBUIEventQueue: self.reader = ConnectionReader(readfd) self.t = threading.Thread() - self.t.setDaemon(True) + self.t.daemon = True self.t.run = self.startCallbackHandler self.t.start() diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index f5d557e8c000..3b64584da0e9 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -431,6 +431,10 @@ class MirrorUriTest(FetcherTest): ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake;branch=master", "git://git.openembedded.org/bitbake;protocol=http") : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + ("git://user1@someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake;branch=master", "git://user2@git.openembedded.org/bitbake;protocol=http") + : "git://user2@git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", + + #Renaming files doesn't work #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz" #("file://sstate-xyz.tgz", "file://.*/.*", "file:///somewhere/1234/sstate-cache") : "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz", @@ -491,7 +495,7 @@ class GitDownloadDirectoryNamingTest(FetcherTest): super(GitDownloadDirectoryNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake" self.recipe_dir = "git.openembedded.org.bitbake" - self.mirror_url = "git://github.com/openembedded/bitbake.git" + self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https" self.mirror_dir = "github.com.openembedded.bitbake.git" self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') @@ -539,7 +543,7 @@ class TarballNamingTest(FetcherTest): super(TarballNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake" self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz" - self.mirror_url = "git://github.com/openembedded/bitbake.git" + self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https" self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz" self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1') @@ -573,7 +577,7 @@ class GitShallowTarballNamingTest(FetcherTest): super(GitShallowTarballNamingTest, self).setUp() self.recipe_url = "git://git.openembedded.org/bitbake" self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz" - self.mirror_url = "git://github.com/openembedded/bitbake.git" + self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https" self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz" self.d.setVar('BB_GIT_SHALLOW', '1') @@ -985,7 +989,7 @@ class FetcherNetworkTest(FetcherTest): def test_git_submodule_dbus_broker(self): # The following external repositories have show failures in fetch and unpack operations # We want to avoid regressions! - url = "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" + url = "gitsm://github.com/bus1/dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1001,7 +1005,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_CLI11(self): - url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main" + url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1016,12 +1020,12 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_update_CLI11(self): """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """ - url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main" + url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # CLI11 that pulls in a newer nlohmann-json - url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main" + url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1035,7 +1039,7 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_aktualizr(self): - url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" + url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1055,7 +1059,7 @@ class FetcherNetworkTest(FetcherTest): """ Prevent regression on deeply nested submodules not being checked out properly, even though they were fetched. """ # This repository also has submodules where the module (name), path and url do not align - url = "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699" + url = "gitsm://github.com/azure/iotedge.git;protocol=https;rev=d76e0316c6f324345d77c48a83ce836d09392699" fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1113,7 +1117,7 @@ class SVNTest(FetcherTest): bb.process.run("svn co %s svnfetch_co" % self.repo_url, cwd=self.tempdir) # Github will emulate SVN. Use this to check if we're downloding... - bb.process.run("svn propset svn:externals 'bitbake svn://vcs.pcre.org/pcre2/code' .", + bb.process.run("svn propset svn:externals 'bitbake https://github.com/PhilipHazel/pcre2.git' .", cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk')) bb.process.run("svn commit --non-interactive -m 'Add external'", cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk')) @@ -1231,7 +1235,7 @@ class FetchLatestVersionTest(FetcherTest): test_git_uris = { # version pattern "X.Y.Z" - ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4", "9b1db6b8060bd00b121a692f942404a24ae2960f", "") + ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "") : "1.99.4", # version pattern "vX.Y" # mirror of git.infradead.org since network issues interfered with testing @@ -1258,9 +1262,9 @@ class FetchLatestVersionTest(FetcherTest): : "0.4.3", ("build-appliance-image", "git://git.yoctoproject.org/poky", "b37dd451a52622d5b570183a81583cc34c2ff555", "(?P<pver>(([0-9][\.|_]?)+[0-9]))") : "11.0.0", - ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot", "cd437ecbd8986c894442f8fce1e0061e20f04dee", "chkconfig\-(?P<pver>((\d+[\.\-_]*)+))") + ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", "chkconfig\-(?P<pver>((\d+[\.\-_]*)+))") : "1.3.59", - ("remake", "git://github.com/rocky/remake.git", "f05508e521987c8494c92d9c2871aec46307d51d", "(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))") + ("remake", "git://github.com/rocky/remake.git;protocol=https", "f05508e521987c8494c92d9c2871aec46307d51d", "(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))") : "3.82+dbg0.9", } @@ -1354,9 +1358,6 @@ class FetchCheckStatusTest(FetcherTest): "http://downloads.yoctoproject.org/releases/opkg/opkg-0.1.7.tar.gz", "http://downloads.yoctoproject.org/releases/opkg/opkg-0.3.0.tar.gz", "ftp://sourceware.org/pub/libffi/libffi-1.20.tar.gz", - "http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz", - "https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz", - "https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz", # GitHub releases are hosted on Amazon S3, which doesn't support HEAD "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz" ] @@ -2047,7 +2048,7 @@ class GitShallowTest(FetcherTest): @skipIfNoNetwork() def test_bitbake(self): - self.git('remote add --mirror=fetch origin git://github.com/openembedded/bitbake', cwd=self.srcdir) + self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir) self.git('config core.bare true', cwd=self.srcdir) self.git('fetch', cwd=self.srcdir) diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index b282d09abfce..2a150fe9c73e 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -16,7 +16,8 @@ import bb.msg import multiprocessing import fcntl import importlib -from importlib import machinery +import importlib.machinery +import importlib.util import itertools import subprocess import glob @@ -451,6 +452,10 @@ def lockfile(name, shared=False, retry=True, block=False): consider the possibility of sending a signal to the process to break out - at which point you want block=True rather than retry=True. """ + if len(name) > 255: + root, ext = os.path.splitext(name) + name = root[:255 - len(ext)] + ext + dirname = os.path.dirname(name) mkdirhier(dirname) @@ -487,7 +492,7 @@ def lockfile(name, shared=False, retry=True, block=False): return lf lf.close() except OSError as e: - if e.errno == errno.EACCES: + if e.errno == errno.EACCES or e.errno == errno.ENAMETOOLONG: logger.error("Unable to acquire lock '%s', %s", e.strerror, name) sys.exit(1) @@ -1616,7 +1621,9 @@ def load_plugins(logger, plugins, pluginpath): logger.debug('Loading plugin %s' % name) spec = importlib.machinery.PathFinder.find_spec(name, path=[pluginpath] ) if spec: - return spec.loader.load_module() + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod logger.debug('Loading plugins from %s...' % pluginpath) diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py index a0dc0c170f2b..df0fa0a07937 100644 --- a/bitbake/lib/hashserv/server.py +++ b/bitbake/lib/hashserv/server.py @@ -521,7 +521,7 @@ class Server(object): def start_tcp_server(self, host, port): self.server = self.loop.run_until_complete( - asyncio.start_server(self.handle_client, host, port, loop=self.loop) + asyncio.start_server(self.handle_client, host, port) ) for s in self.server.sockets: @@ -546,7 +546,7 @@ class Server(object): # Work around path length limits in AF_UNIX os.chdir(os.path.dirname(path)) self.server = self.loop.run_until_complete( - asyncio.start_unix_server(self.handle_client, os.path.basename(path), loop=self.loop) + asyncio.start_unix_server(self.handle_client, os.path.basename(path)) ) finally: os.chdir(cwd) diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py b/bitbake/lib/toaster/tests/builds/buildtest.py index 872bbd377510..13b51fb0d8e4 100644 --- a/bitbake/lib/toaster/tests/builds/buildtest.py +++ b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -119,7 +119,7 @@ class BuildTest(unittest.TestCase): if os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"): ProjectVariable.objects.get_or_create( name="SSTATE_MIRRORS", - value="file://.* http://autobuilder.yoctoproject.org/pub/sstate/PATH;downloadfilename=PATH", + value="file://.* http://sstate.yoctoproject.org/PATH;downloadfilename=PATH", project=project) ProjectTarget.objects.create(project=project, -- 2.35.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v2 2/2] wic: Update to the latest revision 2022-04-21 8:52 [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Henning Schild 2022-04-21 8:52 ` [PATCH v2 1/2] bitbake: Update to 1.50.5 release Henning Schild @ 2022-04-21 8:52 ` Henning Schild 2022-04-21 9:02 ` Henning Schild 2022-04-22 7:59 ` [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Anton Mikanovich 2 siblings, 1 reply; 11+ messages in thread From: Henning Schild @ 2022-04-21 8:52 UTC (permalink / raw) To: isar-users; +Cc: Florian Bezdeka, Felix Moessbauer, Henning Schild Update to the latest wic from OE-core. OE-core Revision: 712552b5cc427d7be1258c45886de9b57f7272c9 Signed-off-by: Henning Schild <henning.schild@siemens.com> --- .../wic/plugins/source/bootimg-efi-isar.py | 77 ++++++++++++++++--- .../wic/plugins/source/bootimg-pcbios-isar.py | 6 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- scripts/lib/wic/engine.py | 6 +- scripts/lib/wic/help.py | 10 ++- scripts/lib/wic/ksparser.py | 8 +- scripts/lib/wic/misc.py | 4 +- scripts/lib/wic/partition.py | 25 ++++-- scripts/lib/wic/pluginbase.py | 8 +- scripts/lib/wic/plugins/imager/direct.py | 11 ++- scripts/lib/wic/plugins/source/bootimg-efi.py | 74 +++++++++++++++--- .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- scripts/lib/wic/plugins/source/rootfs.py | 2 +- scripts/wic | 9 ++- 17 files changed, 231 insertions(+), 56 deletions(-) diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py index 5ba0777ad244..a24e04f309da 100644 --- a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py +++ b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py @@ -12,6 +12,7 @@ import logging import os +import tempfile import shutil import re @@ -129,12 +130,13 @@ class BootimgEFIPlugin(SourcePlugin): bootloader = creator.ks.bootloader loader_conf = "" - loader_conf += "default boot\n" + if source_params.get('create-unified-kernel-image') != "true": + loader_conf += "default boot\n" loader_conf += "timeout %d\n" % bootloader.timeout initrd = source_params.get('initrd') - if initrd: + if initrd and source_params.get('create-unified-kernel-image') != "true": # obviously we need to have a common common deploy var bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") if not bootimg_dir: @@ -195,11 +197,12 @@ class BootimgEFIPlugin(SourcePlugin): for rd in initrds: boot_conf += "initrd /%s\n" % rd - logger.debug("Writing systemd-boot config " - "%s/hdd/boot/loader/entries/boot.conf", cr_workdir) - cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w") - cfg.write(boot_conf) - cfg.close() + if source_params.get('create-unified-kernel-image') != "true": + logger.debug("Writing systemd-boot config " + "%s/hdd/boot/loader/entries/boot.conf", cr_workdir) + cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w") + cfg.write(boot_conf) + cfg.close() @classmethod @@ -300,11 +303,63 @@ class BootimgEFIPlugin(SourcePlugin): kernel = "%s-%s.bin" % \ (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) - install_cmd = "install -m 0644 %s/%s %s/%s" % \ - (staging_kernel_dir, kernel, hdddir, kernel) + if source_params.get('create-unified-kernel-image') == "true": + initrd = source_params.get('initrd') + if not initrd: + raise WicError("initrd= must be specified when create-unified-kernel-image=true, exiting") + + deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") + efi_stub = glob("%s/%s" % (deploy_dir, "linux*.efi.stub")) + if len(efi_stub) == 0: + raise WicError("Unified Kernel Image EFI stub not found, exiting") + efi_stub = efi_stub[0] + + with tempfile.TemporaryDirectory() as tmp_dir: + label = source_params.get('label') + label_conf = "root=%s" % creator.rootdev + if label: + label_conf = "LABEL=%s" % label + + bootloader = creator.ks.bootloader + cmdline = open("%s/cmdline" % tmp_dir, "w") + cmdline.write("%s %s" % (label_conf, bootloader.append)) + cmdline.close() + + initrds = initrd.split(';') + initrd = open("%s/initrd" % tmp_dir, "wb") + for f in initrds: + with open("%s/%s" % (deploy_dir, f), 'rb') as in_file: + shutil.copyfileobj(in_file, initrd) + initrd.close() + + # Searched by systemd-boot: + # https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images + install_cmd = "install -d %s/EFI/Linux" % hdddir + exec_cmd(install_cmd) + + staging_dir_host = get_bitbake_var("STAGING_DIR_HOST") + + # https://www.freedesktop.org/software/systemd/man/systemd-stub.html + objcopy_cmd = "objcopy \ + --add-section .osrel=%s --change-section-vma .osrel=0x20000 \ + --add-section .cmdline=%s --change-section-vma .cmdline=0x30000 \ + --add-section .linux=%s --change-section-vma .linux=0x2000000 \ + --add-section .initrd=%s --change-section-vma .initrd=0x3000000 \ + %s %s" % \ + ("%s/usr/lib/os-release" % staging_dir_host, + cmdline.name, + "%s/%s" % (staging_kernel_dir, kernel), + initrd.name, + efi_stub, + "%s/EFI/Linux/linux.efi" % hdddir) + exec_cmd(objcopy_cmd) + else: + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (staging_kernel_dir, kernel, hdddir, kernel) + + install_cmd = isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir) + exec_cmd(install_cmd) - install_cmd = isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir) - exec_cmd(install_cmd) if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): for src_path, dst_path in cls.install_task: diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py index 9136d4f215ca..7fb0f2a87394 100644 --- a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py +++ b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py @@ -213,8 +213,10 @@ class BootimgPcbiosIsarPlugin(SourcePlugin): # dosfs image, created by mkdosfs bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno) - dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ - (part.fsuuid, bootimg, blocks) + label = part.label if part.label else "boot" + + dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ + (label, part.fsuuid, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) diff --git a/scripts/lib/wic/canned-wks/common.wks.inc b/scripts/lib/wic/canned-wks/common.wks.inc index 4fd29fa8c119..89880b417b6e 100644 --- a/scripts/lib/wic/canned-wks/common.wks.inc +++ b/scripts/lib/wic/canned-wks/common.wks.inc @@ -1,3 +1,3 @@ # This file is included into 3 canned wks files from this directory part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --use-uuid --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 +part / --source rootfs --use-uuid --fstype=ext4 --label platform --align 1024 diff --git a/scripts/lib/wic/canned-wks/directdisk-gpt.wks b/scripts/lib/wic/canned-wks/directdisk-gpt.wks index cf16c0c30bbd..8d7d8de6ea7e 100644 --- a/scripts/lib/wic/canned-wks/directdisk-gpt.wks +++ b/scripts/lib/wic/canned-wks/directdisk-gpt.wks @@ -4,7 +4,7 @@ part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid +part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid bootloader --ptable gpt --timeout=0 --append="rootwait rootfstype=ext4 video=vesafb vga=0x318 console=tty0 console=ttyS0,115200n8" diff --git a/scripts/lib/wic/canned-wks/mkefidisk.wks b/scripts/lib/wic/canned-wks/mkefidisk.wks index d1878e23e5a3..9f534fe18471 100644 --- a/scripts/lib/wic/canned-wks/mkefidisk.wks +++ b/scripts/lib/wic/canned-wks/mkefidisk.wks @@ -4,7 +4,7 @@ part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align 1024 --use-uuid +part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid part swap --ondisk sda --size 44 --label swap1 --fstype=swap diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py index 018815b96688..674ccfc24418 100644 --- a/scripts/lib/wic/engine.py +++ b/scripts/lib/wic/engine.py @@ -19,10 +19,10 @@ import os import tempfile import json import subprocess +import shutil import re from collections import namedtuple, OrderedDict -from distutils.spawn import find_executable from wic import WicError from wic.filemap import sparse_copy @@ -245,7 +245,7 @@ class Disk: for path in pathlist.split(':'): self.paths = "%s%s:%s" % (native_sysroot, path, self.paths) - self.parted = find_executable("parted", self.paths) + self.parted = shutil.which("parted", path=self.paths) if not self.parted: raise WicError("Can't find executable parted") @@ -283,7 +283,7 @@ class Disk: "resize2fs", "mkswap", "mkdosfs", "debugfs","blkid"): aname = "_%s" % name if aname not in self.__dict__: - setattr(self, aname, find_executable(name, self.paths)) + setattr(self, aname, shutil.which(name, path=self.paths)) if aname not in self.__dict__ or self.__dict__[aname] is None: raise WicError("Can't find executable '{}'".format(name)) return self.__dict__[aname] diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py index bd3a2b97dfaf..4ff7470a6a79 100644 --- a/scripts/lib/wic/help.py +++ b/scripts/lib/wic/help.py @@ -637,7 +637,7 @@ DESCRIPTION oe-core: directdisk.bbclass and mkefidisk.sh. The difference between wic and those examples is that with wic the functionality of those scripts is implemented by a general-purpose partitioning - 'language' based on Redhat kickstart syntax). + 'language' based on Red Hat kickstart syntax). The initial motivation and design considerations that lead to the current tool are described exhaustively in Yocto Bug #3847 @@ -840,8 +840,8 @@ DESCRIPTION meanings. The commands are based on the Fedora kickstart documentation but with modifications to reflect wic capabilities. - http://fedoraproject.org/wiki/Anaconda/Kickstart#part_or_partition - http://fedoraproject.org/wiki/Anaconda/Kickstart#bootloader + https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#part-or-partition + https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#bootloader Commands @@ -930,6 +930,7 @@ DESCRIPTION ext4 btrfs squashfs + erofs swap --fsoptions: Specifies a free-form string of options to be @@ -990,6 +991,9 @@ DESCRIPTION multiple partitions and we want to keep the right permissions and usernames in all the partitions. + --no-fstab-update: This option is specific to wic. It does not update the + '/etc/fstab' stock file for the given partition. + --extra-space: This option is specific to wic. It adds extra space after the space filled by the content of the partition. The final size can go diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index 3eb669da39ca..0df9eb0d057d 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -157,7 +157,8 @@ class KickStart(): part.add_argument('--fsoptions', dest='fsopts') part.add_argument('--fstype', default='vfat', choices=('ext2', 'ext3', 'ext4', 'btrfs', - 'squashfs', 'vfat', 'msdos', 'swap')) + 'squashfs', 'vfat', 'msdos', 'erofs', + 'swap')) part.add_argument('--mkfs-extraopts', default='') part.add_argument('--label') part.add_argument('--use-label', action='store_true') @@ -184,6 +185,7 @@ class KickStart(): part.add_argument('--use-uuid', action='store_true') part.add_argument('--uuid') part.add_argument('--fsuuid') + part.add_argument('--no-fstab-update', action='store_true') bootloader = subparsers.add_parser('bootloader') bootloader.add_argument('--append') @@ -229,6 +231,10 @@ class KickStart(): err = "%s:%d: SquashFS does not support LABEL" \ % (confpath, lineno) raise KickStartError(err) + # erofs does not support filesystem labels + if parsed.fstype == 'erofs' and parsed.label: + err = "%s:%d: erofs does not support LABEL" % (confpath, lineno) + raise KickStartError(err) if parsed.fstype == 'msdos' or parsed.fstype == 'vfat': if parsed.fsuuid: if parsed.fsuuid.upper().startswith('0X'): diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py index 57c042c503e6..3e118229960b 100644 --- a/scripts/lib/wic/misc.py +++ b/scripts/lib/wic/misc.py @@ -16,9 +16,9 @@ import logging import os import re import subprocess +import shutil from collections import defaultdict -from distutils import spawn from wic import WicError @@ -122,7 +122,7 @@ def find_executable(cmd, paths): if provided and "%s-native" % recipe in provided: return True - return spawn.find_executable(cmd, paths) + return shutil.which(cmd, path=paths) def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): """ diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index 76d144d12d30..09e491dd494d 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -54,6 +54,7 @@ class Partition(): self.uuid = args.uuid self.fsuuid = args.fsuuid self.type = args.type + self.no_fstab_update = args.no_fstab_update self.updated_fstab_path = None self.has_fstab = False self.update_fstab_in_rootfs = False @@ -104,7 +105,7 @@ class Partition(): extra_blocks = self.extra_space rootfs_size = actual_rootfs_size + extra_blocks - rootfs_size *= self.overhead_factor + rootfs_size = int(rootfs_size * self.overhead_factor) logger.debug("Added %d extra blocks to %s to get to %d total blocks", extra_blocks, self.mountpoint, rootfs_size) @@ -141,9 +142,9 @@ class Partition(): native_sysroot) self.source_file = "%s/fs.%s" % (cr_workdir, self.fstype) else: - if self.fstype == 'squashfs': - raise WicError("It's not possible to create empty squashfs " - "partition '%s'" % (self.mountpoint)) + if self.fstype in ('squashfs', 'erofs'): + raise WicError("It's not possible to create empty %s " + "partition '%s'" % (self.fstype, self.mountpoint)) rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label, self.lineno, self.fstype) @@ -170,7 +171,7 @@ class Partition(): # Split sourceparams string of the form key1=val1[,key2=val2,...] # into a dict. Also accepts valueless keys i.e. without = splitted = self.sourceparams.split(',') - srcparams_dict = dict(par.split('=', 1) for par in splitted if par) + srcparams_dict = dict((par.split('=', 1) + [None])[:2] for par in splitted if par) plugin = PluginMgr.get_plugins('source')[self.source] plugin.do_configure_partition(self, srcparams_dict, creator, @@ -286,7 +287,7 @@ class Partition(): (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - if self.updated_fstab_path and self.has_fstab: + if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") with open(debugfs_script_path, "w") as f: f.write("cd etc\n") @@ -350,7 +351,7 @@ class Partition(): mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) exec_native_cmd(mcopy_cmd, native_sysroot) - if self.updated_fstab_path and self.has_fstab: + if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) exec_native_cmd(mcopy_cmd, native_sysroot) @@ -369,6 +370,16 @@ class Partition(): (rootfs_dir, rootfs, extraopts) exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) + def prepare_rootfs_erofs(self, rootfs, cr_workdir, oe_builddir, rootfs_dir, + native_sysroot, pseudo): + """ + Prepare content for a erofs rootfs partition. + """ + extraopts = self.mkfs_extraopts or '' + erofs_cmd = "mkfs.erofs %s -U %s %s %s" % \ + (extraopts, self.fsuuid, rootfs, rootfs_dir) + exec_native_cmd(erofs_cmd, native_sysroot, pseudo=pseudo) + def prepare_empty_partition_ext(self, rootfs, oe_builddir, native_sysroot): """ diff --git a/scripts/lib/wic/pluginbase.py b/scripts/lib/wic/pluginbase.py index d9b4e57747e7..b64568339b1b 100644 --- a/scripts/lib/wic/pluginbase.py +++ b/scripts/lib/wic/pluginbase.py @@ -9,9 +9,11 @@ __all__ = ['ImagerPlugin', 'SourcePlugin'] import os import logging +import types from collections import defaultdict -from importlib.machinery import SourceFileLoader +import importlib +import importlib.util from wic import WicError from wic.misc import get_bitbake_var @@ -54,7 +56,9 @@ class PluginMgr: mname = fname[:-3] mpath = os.path.join(ppath, fname) logger.debug("loading plugin module %s", mpath) - SourceFileLoader(mname, mpath).load_module() + spec = importlib.util.spec_from_file_location(mname, mpath) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) return PLUGINS.get(ptype) diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py index ea709e8c545d..35fff7c10242 100644 --- a/scripts/lib/wic/plugins/imager/direct.py +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -77,7 +77,8 @@ class DirectPlugin(ImagerPlugin): image_path = self._full_path(self.workdir, self.parts[0].disk, "direct") self._image = PartitionedImage(image_path, self.ptable_format, - self.parts, self.native_sysroot) + self.parts, self.native_sysroot, + options.extra_space) def setup_workdir(self, workdir): if workdir: @@ -116,7 +117,7 @@ class DirectPlugin(ImagerPlugin): updated = False for part in self.parts: if not part.realnum or not part.mountpoint \ - or part.mountpoint == "/": + or part.mountpoint == "/" or not part.mountpoint.startswith('/'): continue if part.use_uuid: @@ -258,6 +259,8 @@ class DirectPlugin(ImagerPlugin): if part.mountpoint == "/": if part.uuid: return "PARTUUID=%s" % part.uuid + elif part.label: + return "PARTLABEL=%s" % part.label else: suffix = 'p' if part.disk.startswith('mmcblk') else '' return "/dev/%s%s%-d" % (part.disk, suffix, part.realnum) @@ -293,7 +296,7 @@ class PartitionedImage(): Partitioned image in a file. """ - def __init__(self, path, ptable_format, partitions, native_sysroot=None): + def __init__(self, path, ptable_format, partitions, native_sysroot=None, extra_space=0): self.path = path # Path to the image file self.numpart = 0 # Number of allocated partitions self.realpart = 0 # Number of partitions in the partition table @@ -314,6 +317,7 @@ class PartitionedImage(): self.sector_size = SECTOR_SIZE self.native_sysroot = native_sysroot num_real_partitions = len([p for p in self.partitions if not p.no_table]) + self.extra_space = extra_space # calculate the real partition number, accounting for partitions not # in the partition table and logical partitions @@ -483,6 +487,7 @@ class PartitionedImage(): self.min_size += GPT_OVERHEAD self.min_size *= self.sector_size + self.min_size += self.extra_space def _create_partition(self, device, parttype, fstype, start, size): """ Create a partition on an image described by the 'device' object. """ diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py index cdc72543c200..0391aebdc840 100644 --- a/scripts/lib/wic/plugins/source/bootimg-efi.py +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -12,6 +12,7 @@ import logging import os +import tempfile import shutil import re @@ -119,12 +120,13 @@ class BootimgEFIPlugin(SourcePlugin): bootloader = creator.ks.bootloader loader_conf = "" - loader_conf += "default boot\n" + if source_params.get('create-unified-kernel-image') != "true": + loader_conf += "default boot\n" loader_conf += "timeout %d\n" % bootloader.timeout initrd = source_params.get('initrd') - if initrd: + if initrd and source_params.get('create-unified-kernel-image') != "true": # obviously we need to have a common common deploy var bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") if not bootimg_dir: @@ -183,11 +185,12 @@ class BootimgEFIPlugin(SourcePlugin): for rd in initrds: boot_conf += "initrd /%s\n" % rd - logger.debug("Writing systemd-boot config " - "%s/hdd/boot/loader/entries/boot.conf", cr_workdir) - cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w") - cfg.write(boot_conf) - cfg.close() + if source_params.get('create-unified-kernel-image') != "true": + logger.debug("Writing systemd-boot config " + "%s/hdd/boot/loader/entries/boot.conf", cr_workdir) + cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w") + cfg.write(boot_conf) + cfg.close() @classmethod @@ -288,9 +291,60 @@ class BootimgEFIPlugin(SourcePlugin): kernel = "%s-%s.bin" % \ (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME")) - install_cmd = "install -m 0644 %s/%s %s/%s" % \ - (staging_kernel_dir, kernel, hdddir, kernel) - exec_cmd(install_cmd) + if source_params.get('create-unified-kernel-image') == "true": + initrd = source_params.get('initrd') + if not initrd: + raise WicError("initrd= must be specified when create-unified-kernel-image=true, exiting") + + deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") + efi_stub = glob("%s/%s" % (deploy_dir, "linux*.efi.stub")) + if len(efi_stub) == 0: + raise WicError("Unified Kernel Image EFI stub not found, exiting") + efi_stub = efi_stub[0] + + with tempfile.TemporaryDirectory() as tmp_dir: + label = source_params.get('label') + label_conf = "root=%s" % creator.rootdev + if label: + label_conf = "LABEL=%s" % label + + bootloader = creator.ks.bootloader + cmdline = open("%s/cmdline" % tmp_dir, "w") + cmdline.write("%s %s" % (label_conf, bootloader.append)) + cmdline.close() + + initrds = initrd.split(';') + initrd = open("%s/initrd" % tmp_dir, "wb") + for f in initrds: + with open("%s/%s" % (deploy_dir, f), 'rb') as in_file: + shutil.copyfileobj(in_file, initrd) + initrd.close() + + # Searched by systemd-boot: + # https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images + install_cmd = "install -d %s/EFI/Linux" % hdddir + exec_cmd(install_cmd) + + staging_dir_host = get_bitbake_var("STAGING_DIR_HOST") + + # https://www.freedesktop.org/software/systemd/man/systemd-stub.html + objcopy_cmd = "objcopy \ + --add-section .osrel=%s --change-section-vma .osrel=0x20000 \ + --add-section .cmdline=%s --change-section-vma .cmdline=0x30000 \ + --add-section .linux=%s --change-section-vma .linux=0x2000000 \ + --add-section .initrd=%s --change-section-vma .initrd=0x3000000 \ + %s %s" % \ + ("%s/usr/lib/os-release" % staging_dir_host, + cmdline.name, + "%s/%s" % (staging_kernel_dir, kernel), + initrd.name, + efi_stub, + "%s/EFI/Linux/linux.efi" % hdddir) + exec_cmd(objcopy_cmd) + else: + install_cmd = "install -m 0644 %s/%s %s/%s" % \ + (staging_kernel_dir, kernel, hdddir, kernel) + exec_cmd(install_cmd) if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): for src_path, dst_path in cls.install_task: diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index f2639e700493..32e47f183146 100644 --- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -186,8 +186,10 @@ class BootimgPcbiosPlugin(SourcePlugin): # dosfs image, created by mkdosfs bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno) - dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ - (part.fsuuid, bootimg, blocks) + label = part.label if part.label else "boot" + + dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ + (label, part.fsuuid, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py index 3c4997d8ba5e..7c90cd3cf82b 100644 --- a/scripts/lib/wic/plugins/source/rawcopy.py +++ b/scripts/lib/wic/plugins/source/rawcopy.py @@ -4,6 +4,8 @@ import logging import os +import signal +import subprocess from wic import WicError from wic.pluginbase import SourcePlugin @@ -29,15 +31,34 @@ class RawCopyPlugin(SourcePlugin): cmd = 'btrfs filesystem label %s %s' % (dst, label) elif fstype == 'swap': cmd = 'mkswap -L %s %s' % (label, dst) - elif fstype == 'squashfs': - raise WicError("It's not possible to update a squashfs " - "filesystem label '%s'" % (label)) + elif fstype in ('squashfs', 'erofs'): + raise WicError("It's not possible to update a %s " + "filesystem label '%s'" % (fstype, label)) else: raise WicError("Cannot update filesystem label: " "Unknown fstype: '%s'" % (fstype)) exec_cmd(cmd) + @staticmethod + def do_image_uncompression(src, dst, workdir): + def subprocess_setup(): + # Python installs a SIGPIPE handler by default. This is usually not what + # non-Python subprocesses expect. + # SIGPIPE errors are known issues with gzip/bash + signal.signal(signal.SIGPIPE, signal.SIG_DFL) + + extension = os.path.splitext(src)[1] + decompressor = { + ".bz2": "bzip2", + ".gz": "gzip", + ".xz": "xz" + }.get(extension) + if not decompressor: + raise WicError("Not supported compressor filename extension: %s" % extension) + cmd = "%s -dc %s > %s" % (decompressor, src, dst) + subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=workdir) + @classmethod def do_prepare_partition(cls, part, source_params, cr, cr_workdir, oe_builddir, bootimg_dir, kernel_dir, @@ -56,7 +77,13 @@ class RawCopyPlugin(SourcePlugin): if 'file' not in source_params: raise WicError("No file specified") - src = os.path.join(kernel_dir, source_params['file']) + if 'unpack' in source_params: + img = os.path.join(kernel_dir, source_params['file']) + src = os.path.join(cr_workdir, os.path.splitext(source_params['file'])[0]) + RawCopyPlugin.do_image_uncompression(img, src, cr_workdir) + else: + src = os.path.join(kernel_dir, source_params['file']) + dst = os.path.join(cr_workdir, "%s.%s" % (os.path.basename(source_params['file']), part.lineno)) if not os.path.exists(os.path.dirname(dst)): diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index 96d940a91d61..2e34e715ca73 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -218,7 +218,7 @@ class RootfsPlugin(SourcePlugin): # Update part.has_fstab here as fstab may have been added or # removed by the above modifications. part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) - if part.update_fstab_in_rootfs and part.has_fstab: + if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update: fstab_path = os.path.join(new_rootfs, "etc/fstab") # Assume that fstab should always be owned by root with fixed permissions install_cmd = "install -m 0644 %s %s" % (part.updated_fstab_path, fstab_path) diff --git a/scripts/wic b/scripts/wic index a741aed364bf..aee63a45aacf 100755 --- a/scripts/wic +++ b/scripts/wic @@ -22,9 +22,9 @@ import sys import argparse import logging import subprocess +import shutil from collections import namedtuple -from distutils import spawn # External modules scripts_path = os.path.dirname(os.path.realpath(__file__)) @@ -47,7 +47,7 @@ if os.environ.get('SDKTARGETSYSROOT'): break sdkroot = os.path.dirname(sdkroot) -bitbake_exe = spawn.find_executable('bitbake') +bitbake_exe = shutil.which('bitbake') if bitbake_exe: bitbake_path = scriptpath.add_bitbake_lib_path() import bb @@ -159,6 +159,9 @@ def wic_create_subcommand(options, usage_str): "(Use -e/--image-name to specify it)") native_sysroot = options.native_sysroot + if options.kernel_dir: + kernel_dir = options.kernel_dir + if not options.vars_dir and (not native_sysroot or not os.path.isdir(native_sysroot)): logger.info("Building wic-tools...\n") subprocess.check_call(["bitbake", "wic-tools"]) @@ -346,6 +349,8 @@ def wic_init_parser_create(subparser): default=False, help="output debug information") subparser.add_argument("-i", "--imager", dest="imager", default="direct", help="the wic imager plugin") + subparser.add_argument("--extra-space", type=int, dest="extra_space", + default=0, help="additional free disk space to add to the image") return -- 2.35.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 2/2] wic: Update to the latest revision 2022-04-21 8:52 ` [PATCH v2 2/2] wic: Update to the latest revision Henning Schild @ 2022-04-21 9:02 ` Henning Schild 0 siblings, 0 replies; 11+ messages in thread From: Henning Schild @ 2022-04-21 9:02 UTC (permalink / raw) To: isar-users; +Cc: Florian Bezdeka, Felix Moessbauer Am Thu, 21 Apr 2022 10:52:32 +0200 schrieb Henning Schild <henning.schild@siemens.com>: > Update to the latest wic from OE-core. > > OE-core Revision: 712552b5cc427d7be1258c45886de9b57f7272c9 > > Signed-off-by: Henning Schild <henning.schild@siemens.com> > --- > .../wic/plugins/source/bootimg-efi-isar.py | 77 > ++++++++++++++++--- .../wic/plugins/source/bootimg-pcbios-isar.py | > 6 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > scripts/lib/wic/engine.py | 6 +- > scripts/lib/wic/help.py | 10 ++- > scripts/lib/wic/ksparser.py | 8 +- > scripts/lib/wic/misc.py | 4 +- > scripts/lib/wic/partition.py | 25 ++++-- > scripts/lib/wic/pluginbase.py | 8 +- > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 +++++++++++++++--- > .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- > scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- > scripts/lib/wic/plugins/source/rootfs.py | 2 +- > scripts/wic | 9 ++- > 17 files changed, 231 insertions(+), 56 deletions(-) > > diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py > b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py index > 5ba0777ad244..a24e04f309da 100644 --- > a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py +++ > b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py @@ -12,6 > +12,7 @@ > import logging > import os > +import tempfile > import shutil > import re > > @@ -129,12 +130,13 @@ class BootimgEFIPlugin(SourcePlugin): > bootloader = creator.ks.bootloader > > loader_conf = "" > - loader_conf += "default boot\n" > + if source_params.get('create-unified-kernel-image') != > "true": > + loader_conf += "default boot\n" > loader_conf += "timeout %d\n" % bootloader.timeout > > initrd = source_params.get('initrd') > > - if initrd: > + if initrd and > source_params.get('create-unified-kernel-image') != "true": # > obviously we need to have a common common deploy var bootimg_dir = > get_bitbake_var("DEPLOY_DIR_IMAGE") if not bootimg_dir: > @@ -195,11 +197,12 @@ class BootimgEFIPlugin(SourcePlugin): > for rd in initrds: > boot_conf += "initrd /%s\n" % rd > > - logger.debug("Writing systemd-boot config " > - "%s/hdd/boot/loader/entries/boot.conf", > cr_workdir) > - cfg = open("%s/hdd/boot/loader/entries/boot.conf" % > cr_workdir, "w") > - cfg.write(boot_conf) > - cfg.close() > + if source_params.get('create-unified-kernel-image') != > "true": > + logger.debug("Writing systemd-boot config " > + "%s/hdd/boot/loader/entries/boot.conf", > cr_workdir) > + cfg = open("%s/hdd/boot/loader/entries/boot.conf" % > cr_workdir, "w") > + cfg.write(boot_conf) > + cfg.close() > > > @classmethod > @@ -300,11 +303,63 @@ class BootimgEFIPlugin(SourcePlugin): > kernel = "%s-%s.bin" % \ > (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) > - install_cmd = "install -m 0644 %s/%s %s/%s" % \ > - (staging_kernel_dir, kernel, hdddir, kernel) > + if source_params.get('create-unified-kernel-image') == > "true": > + initrd = source_params.get('initrd') > + if not initrd: > + raise WicError("initrd= must be specified when > create-unified-kernel-image=true, exiting") + > + deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > + efi_stub = glob("%s/%s" % (deploy_dir, > "linux*.efi.stub")) > + if len(efi_stub) == 0: > + raise WicError("Unified Kernel Image EFI stub not > found, exiting") > + efi_stub = efi_stub[0] > + > + with tempfile.TemporaryDirectory() as tmp_dir: > + label = source_params.get('label') > + label_conf = "root=%s" % creator.rootdev > + if label: > + label_conf = "LABEL=%s" % label > + > + bootloader = creator.ks.bootloader > + cmdline = open("%s/cmdline" % tmp_dir, "w") > + cmdline.write("%s %s" % (label_conf, > bootloader.append)) > + cmdline.close() > + > + initrds = initrd.split(';') > + initrd = open("%s/initrd" % tmp_dir, "wb") > + for f in initrds: > + with open("%s/%s" % (deploy_dir, f), 'rb') as > in_file: > + shutil.copyfileobj(in_file, initrd) > + initrd.close() > + > + # Searched by systemd-boot: > + # > https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images > + install_cmd = "install -d %s/EFI/Linux" % hdddir > + exec_cmd(install_cmd) > + > + staging_dir_host = > get_bitbake_var("STAGING_DIR_HOST") + > + # > https://www.freedesktop.org/software/systemd/man/systemd-stub.html > + objcopy_cmd = "objcopy \ > + --add-section .osrel=%s --change-section-vma > .osrel=0x20000 \ > + --add-section .cmdline=%s --change-section-vma > .cmdline=0x30000 \ > + --add-section .linux=%s --change-section-vma > .linux=0x2000000 \ > + --add-section .initrd=%s --change-section-vma > .initrd=0x3000000 \ > + %s %s" % \ > + ("%s/usr/lib/os-release" % staging_dir_host, > + cmdline.name, > + "%s/%s" % (staging_kernel_dir, kernel), > + initrd.name, > + efi_stub, > + "%s/EFI/Linux/linux.efi" % hdddir) > + exec_cmd(objcopy_cmd) > + else: > + install_cmd = "install -m 0644 %s/%s %s/%s" % \ > + (staging_kernel_dir, kernel, hdddir, kernel) > + > + install_cmd = > isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir) > + exec_cmd(install_cmd) > > - install_cmd = > isar_populate_boot_cmd(rootfs_dir['ROOTFS_DIR'], hdddir) > - exec_cmd(install_cmd) > > if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): > for src_path, dst_path in cls.install_task: > diff --git > a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py > b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py index > 9136d4f215ca..7fb0f2a87394 100644 --- > a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py +++ > b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py @@ > -213,8 +213,10 @@ class BootimgPcbiosIsarPlugin(SourcePlugin): # > dosfs image, created by mkdosfs bootimg = "%s/boot%s.img" % > (cr_workdir, part.lineno) > - dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ > - (part.fsuuid, bootimg, blocks) > + label = part.label if part.label else "boot" > + > + dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ > + (label, part.fsuuid, bootimg, blocks) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) > diff --git a/scripts/lib/wic/canned-wks/common.wks.inc > b/scripts/lib/wic/canned-wks/common.wks.inc index > 4fd29fa8c119..89880b417b6e 100644 --- > a/scripts/lib/wic/canned-wks/common.wks.inc +++ > b/scripts/lib/wic/canned-wks/common.wks.inc @@ -1,3 +1,3 @@ > # This file is included into 3 canned wks files from this directory > part /boot --source bootimg-pcbios --ondisk sda --label boot > --active --align 1024 -part / --source rootfs --use-uuid > --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align > 1024 +part / --source rootfs --use-uuid --fstype=ext4 --label > platform --align 1024 diff --git This is https://github.com/openembedded/openembedded-core/commit/7e8017208bed98b6c90735cb641fc9d7aedf9140 and it was not taken for our forks in meta-isar/scripts/lib/wic/canned-wks/ Because we can not be sure to have a brandnew e2fsprogs always. With scripts/wic_fakeroot we have a way to centrally hook into wic plugins and play with arguments. But i think having our forked wks files is nicer and we might eventually be able to drop wic_fakeroot, at the moment it only deals with one remaining hack for old e2fsck Henning > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks index > cf16c0c30bbd..8d7d8de6ea7e 100644 --- > a/scripts/lib/wic/canned-wks/directdisk-gpt.wks +++ > b/scripts/lib/wic/canned-wks/directdisk-gpt.wks @@ -4,7 +4,7 @@ > part /boot --source bootimg-pcbios --ondisk sda --label boot > --active --align 1024 -part / --source rootfs --ondisk sda > --fstype=ext4 --mkfs-extraopts "-T default" --label platform --align > 1024 --use-uuid +part / --source rootfs --ondisk sda --fstype=ext4 > --label platform --align 1024 --use-uuid bootloader --ptable gpt > --timeout=0 --append="rootwait rootfstype=ext4 video=vesafb > vga=0x318 console=tty0 console=ttyS0,115200n8" diff --git > a/scripts/lib/wic/canned-wks/mkefidisk.wks > b/scripts/lib/wic/canned-wks/mkefidisk.wks index > d1878e23e5a3..9f534fe18471 100644 --- > a/scripts/lib/wic/canned-wks/mkefidisk.wks +++ > b/scripts/lib/wic/canned-wks/mkefidisk.wks @@ -4,7 +4,7 @@ > part /boot --source bootimg-efi --sourceparams="loader=grub-efi" > --ondisk sda --label msdos --active --align 1024 > -part / --source rootfs --ondisk sda --fstype=ext4 --mkfs-extraopts > "-T default" --label platform --align 1024 --use-uuid +part / > --source rootfs --ondisk sda --fstype=ext4 --label platform --align > 1024 --use-uuid part swap --ondisk sda --size 44 --label swap1 > --fstype=swap > diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py > index 018815b96688..674ccfc24418 100644 > --- a/scripts/lib/wic/engine.py > +++ b/scripts/lib/wic/engine.py > @@ -19,10 +19,10 @@ import os > import tempfile > import json > import subprocess > +import shutil > import re > > from collections import namedtuple, OrderedDict > -from distutils.spawn import find_executable > > from wic import WicError > from wic.filemap import sparse_copy > @@ -245,7 +245,7 @@ class Disk: > for path in pathlist.split(':'): > self.paths = "%s%s:%s" % (native_sysroot, path, > self.paths) > - self.parted = find_executable("parted", self.paths) > + self.parted = shutil.which("parted", path=self.paths) > if not self.parted: > raise WicError("Can't find executable parted") > > @@ -283,7 +283,7 @@ class Disk: > "resize2fs", "mkswap", "mkdosfs", > "debugfs","blkid"): aname = "_%s" % name > if aname not in self.__dict__: > - setattr(self, aname, find_executable(name, > self.paths)) > + setattr(self, aname, shutil.which(name, > path=self.paths)) if aname not in self.__dict__ or > self.__dict__[aname] is None: raise WicError("Can't find executable > '{}'".format(name)) return self.__dict__[aname] > diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py > index bd3a2b97dfaf..4ff7470a6a79 100644 > --- a/scripts/lib/wic/help.py > +++ b/scripts/lib/wic/help.py > @@ -637,7 +637,7 @@ DESCRIPTION > oe-core: directdisk.bbclass and mkefidisk.sh. The difference > between wic and those examples is that with wic the functionality > of those scripts is implemented by a general-purpose partitioning > - 'language' based on Redhat kickstart syntax). > + 'language' based on Red Hat kickstart syntax). > > The initial motivation and design considerations that lead to the > current tool are described exhaustively in Yocto Bug #3847 > @@ -840,8 +840,8 @@ DESCRIPTION > meanings. The commands are based on the Fedora kickstart > documentation but with modifications to reflect wic capabilities. > > - > http://fedoraproject.org/wiki/Anaconda/Kickstart#part_or_partition > - http://fedoraproject.org/wiki/Anaconda/Kickstart#bootloader > + > https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#part-or-partition > + > https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#bootloader > Commands > > @@ -930,6 +930,7 @@ DESCRIPTION > ext4 > btrfs > squashfs > + erofs > swap > > --fsoptions: Specifies a free-form string of options to be > @@ -990,6 +991,9 @@ DESCRIPTION > multiple partitions and we want to keep > the right permissions and usernames in all the partitions. > > + --no-fstab-update: This option is specific to wic. It does > not update the > + '/etc/fstab' stock file for the given > partition. + > --extra-space: This option is specific to wic. It adds extra > space after the space filled by the content > of the partition. The final size can go > diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py > index 3eb669da39ca..0df9eb0d057d 100644 > --- a/scripts/lib/wic/ksparser.py > +++ b/scripts/lib/wic/ksparser.py > @@ -157,7 +157,8 @@ class KickStart(): > part.add_argument('--fsoptions', dest='fsopts') > part.add_argument('--fstype', default='vfat', > choices=('ext2', 'ext3', 'ext4', 'btrfs', > - 'squashfs', 'vfat', 'msdos', > 'swap')) > + 'squashfs', 'vfat', 'msdos', > 'erofs', > + 'swap')) > part.add_argument('--mkfs-extraopts', default='') > part.add_argument('--label') > part.add_argument('--use-label', action='store_true') > @@ -184,6 +185,7 @@ class KickStart(): > part.add_argument('--use-uuid', action='store_true') > part.add_argument('--uuid') > part.add_argument('--fsuuid') > + part.add_argument('--no-fstab-update', action='store_true') > > bootloader = subparsers.add_parser('bootloader') > bootloader.add_argument('--append') > @@ -229,6 +231,10 @@ class KickStart(): > err = "%s:%d: SquashFS does not > support LABEL" \ % (confpath, lineno) > raise KickStartError(err) > + # erofs does not support filesystem labels > + if parsed.fstype == 'erofs' and parsed.label: > + err = "%s:%d: erofs does not support > LABEL" % (confpath, lineno) > + raise KickStartError(err) > if parsed.fstype == 'msdos' or parsed.fstype > == 'vfat': if parsed.fsuuid: > if > parsed.fsuuid.upper().startswith('0X'): diff --git > a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py index > 57c042c503e6..3e118229960b 100644 --- a/scripts/lib/wic/misc.py > +++ b/scripts/lib/wic/misc.py > @@ -16,9 +16,9 @@ import logging > import os > import re > import subprocess > +import shutil > > from collections import defaultdict > -from distutils import spawn > > from wic import WicError > > @@ -122,7 +122,7 @@ def find_executable(cmd, paths): > if provided and "%s-native" % recipe in provided: > return True > > - return spawn.find_executable(cmd, paths) > + return shutil.which(cmd, path=paths) > > def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""): > """ > diff --git a/scripts/lib/wic/partition.py > b/scripts/lib/wic/partition.py index 76d144d12d30..09e491dd494d 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -54,6 +54,7 @@ class Partition(): > self.uuid = args.uuid > self.fsuuid = args.fsuuid > self.type = args.type > + self.no_fstab_update = args.no_fstab_update > self.updated_fstab_path = None > self.has_fstab = False > self.update_fstab_in_rootfs = False > @@ -104,7 +105,7 @@ class Partition(): > extra_blocks = self.extra_space > > rootfs_size = actual_rootfs_size + extra_blocks > - rootfs_size *= self.overhead_factor > + rootfs_size = int(rootfs_size * self.overhead_factor) > > logger.debug("Added %d extra blocks to %s to get to %d > total blocks", extra_blocks, self.mountpoint, rootfs_size) > @@ -141,9 +142,9 @@ class Partition(): > native_sysroot) > self.source_file = "%s/fs.%s" % (cr_workdir, > self.fstype) else: > - if self.fstype == 'squashfs': > - raise WicError("It's not possible to create > empty squashfs " > - "partition '%s'" % > (self.mountpoint)) > + if self.fstype in ('squashfs', 'erofs'): > + raise WicError("It's not possible to create > empty %s " > + "partition '%s'" % (self.fstype, > self.mountpoint)) > rootfs = "%s/fs_%s.%s.%s" % (cr_workdir, self.label, > self.lineno, > self.fstype) @@ -170,7 +171,7 @@ class Partition(): > # Split sourceparams string of the form > key1=val1[,key2=val2,...] # into a dict. Also accepts valueless keys > i.e. without = splitted = self.sourceparams.split(',') > - srcparams_dict = dict(par.split('=', 1) for par in > splitted if par) > + srcparams_dict = dict((par.split('=', 1) + [None])[:2] > for par in splitted if par) > plugin = PluginMgr.get_plugins('source')[self.source] > plugin.do_configure_partition(self, srcparams_dict, creator, > @@ -286,7 +287,7 @@ class Partition(): > (self.fstype, extraopts, rootfs, label_str, self.fsuuid, > rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > - if self.updated_fstab_path and self.has_fstab: > + if self.updated_fstab_path and self.has_fstab and not > self.no_fstab_update: debugfs_script_path = os.path.join(cr_workdir, > "debugfs_script") with open(debugfs_script_path, "w") as f: > f.write("cd etc\n") > @@ -350,7 +351,7 @@ class Partition(): > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) > exec_native_cmd(mcopy_cmd, native_sysroot) > > - if self.updated_fstab_path and self.has_fstab: > + if self.updated_fstab_path and self.has_fstab and not > self.no_fstab_update: mcopy_cmd = "mcopy -i %s %s ::/etc/fstab" % > (rootfs, self.updated_fstab_path) exec_native_cmd(mcopy_cmd, > native_sysroot) > @@ -369,6 +370,16 @@ class Partition(): > (rootfs_dir, rootfs, extraopts) > exec_native_cmd(squashfs_cmd, native_sysroot, pseudo=pseudo) > > + def prepare_rootfs_erofs(self, rootfs, cr_workdir, oe_builddir, > rootfs_dir, > + native_sysroot, pseudo): > + """ > + Prepare content for a erofs rootfs partition. > + """ > + extraopts = self.mkfs_extraopts or '' > + erofs_cmd = "mkfs.erofs %s -U %s %s %s" % \ > + (extraopts, self.fsuuid, rootfs, rootfs_dir) > + exec_native_cmd(erofs_cmd, native_sysroot, pseudo=pseudo) > + > def prepare_empty_partition_ext(self, rootfs, oe_builddir, > native_sysroot): > """ > diff --git a/scripts/lib/wic/pluginbase.py > b/scripts/lib/wic/pluginbase.py index d9b4e57747e7..b64568339b1b > 100644 --- a/scripts/lib/wic/pluginbase.py > +++ b/scripts/lib/wic/pluginbase.py > @@ -9,9 +9,11 @@ __all__ = ['ImagerPlugin', 'SourcePlugin'] > > import os > import logging > +import types > > from collections import defaultdict > -from importlib.machinery import SourceFileLoader > +import importlib > +import importlib.util > > from wic import WicError > from wic.misc import get_bitbake_var > @@ -54,7 +56,9 @@ class PluginMgr: > mname = fname[:-3] > mpath = os.path.join(ppath, fname) > logger.debug("loading plugin module %s", > mpath) > - SourceFileLoader(mname, > mpath).load_module() > + spec = > importlib.util.spec_from_file_location(mname, mpath) > + module = > importlib.util.module_from_spec(spec) > + spec.loader.exec_module(module) > > return PLUGINS.get(ptype) > > diff --git a/scripts/lib/wic/plugins/imager/direct.py > b/scripts/lib/wic/plugins/imager/direct.py index > ea709e8c545d..35fff7c10242 100644 --- > a/scripts/lib/wic/plugins/imager/direct.py +++ > b/scripts/lib/wic/plugins/imager/direct.py @@ -77,7 +77,8 @@ class > DirectPlugin(ImagerPlugin): > image_path = self._full_path(self.workdir, > self.parts[0].disk, "direct") self._image = > PartitionedImage(image_path, self.ptable_format, > - self.parts, > self.native_sysroot) > + self.parts, > self.native_sysroot, > + options.extra_space) > > def setup_workdir(self, workdir): > if workdir: > @@ -116,7 +117,7 @@ class DirectPlugin(ImagerPlugin): > updated = False > for part in self.parts: > if not part.realnum or not part.mountpoint \ > - or part.mountpoint == "/": > + or part.mountpoint == "/" or not > part.mountpoint.startswith('/'): continue > > if part.use_uuid: > @@ -258,6 +259,8 @@ class DirectPlugin(ImagerPlugin): > if part.mountpoint == "/": > if part.uuid: > return "PARTUUID=%s" % part.uuid > + elif part.label: > + return "PARTLABEL=%s" % part.label > else: > suffix = 'p' if part.disk.startswith('mmcblk') > else '' return "/dev/%s%s%-d" % (part.disk, suffix, part.realnum) > @@ -293,7 +296,7 @@ class PartitionedImage(): > Partitioned image in a file. > """ > > - def __init__(self, path, ptable_format, partitions, > native_sysroot=None): > + def __init__(self, path, ptable_format, partitions, > native_sysroot=None, extra_space=0): self.path = path # Path to the > image file self.numpart = 0 # Number of allocated partitions > self.realpart = 0 # Number of partitions in the partition > table @@ -314,6 +317,7 @@ class PartitionedImage(): > self.sector_size = SECTOR_SIZE > self.native_sysroot = native_sysroot > num_real_partitions = len([p for p in self.partitions if not > p.no_table]) > + self.extra_space = extra_space > > # calculate the real partition number, accounting for > partitions not # in the partition table and logical partitions > @@ -483,6 +487,7 @@ class PartitionedImage(): > self.min_size += GPT_OVERHEAD > > self.min_size *= self.sector_size > + self.min_size += self.extra_space > > def _create_partition(self, device, parttype, fstype, start, > size): """ Create a partition on an image described by the 'device' > object. """ diff --git > a/scripts/lib/wic/plugins/source/bootimg-efi.py > b/scripts/lib/wic/plugins/source/bootimg-efi.py index > cdc72543c200..0391aebdc840 100644 --- > a/scripts/lib/wic/plugins/source/bootimg-efi.py +++ > b/scripts/lib/wic/plugins/source/bootimg-efi.py @@ -12,6 +12,7 @@ > import logging > import os > +import tempfile > import shutil > import re > > @@ -119,12 +120,13 @@ class BootimgEFIPlugin(SourcePlugin): > bootloader = creator.ks.bootloader > > loader_conf = "" > - loader_conf += "default boot\n" > + if source_params.get('create-unified-kernel-image') != > "true": > + loader_conf += "default boot\n" > loader_conf += "timeout %d\n" % bootloader.timeout > > initrd = source_params.get('initrd') > > - if initrd: > + if initrd and > source_params.get('create-unified-kernel-image') != "true": # > obviously we need to have a common common deploy var bootimg_dir = > get_bitbake_var("DEPLOY_DIR_IMAGE") if not bootimg_dir: > @@ -183,11 +185,12 @@ class BootimgEFIPlugin(SourcePlugin): > for rd in initrds: > boot_conf += "initrd /%s\n" % rd > > - logger.debug("Writing systemd-boot config " > - "%s/hdd/boot/loader/entries/boot.conf", > cr_workdir) > - cfg = open("%s/hdd/boot/loader/entries/boot.conf" % > cr_workdir, "w") > - cfg.write(boot_conf) > - cfg.close() > + if source_params.get('create-unified-kernel-image') != > "true": > + logger.debug("Writing systemd-boot config " > + "%s/hdd/boot/loader/entries/boot.conf", > cr_workdir) > + cfg = open("%s/hdd/boot/loader/entries/boot.conf" % > cr_workdir, "w") > + cfg.write(boot_conf) > + cfg.close() > > > @classmethod > @@ -288,9 +291,60 @@ class BootimgEFIPlugin(SourcePlugin): > kernel = "%s-%s.bin" % \ > (get_bitbake_var("KERNEL_IMAGETYPE"), > get_bitbake_var("INITRAMFS_LINK_NAME")) > - install_cmd = "install -m 0644 %s/%s %s/%s" % \ > - (staging_kernel_dir, kernel, hdddir, kernel) > - exec_cmd(install_cmd) > + if source_params.get('create-unified-kernel-image') == > "true": > + initrd = source_params.get('initrd') > + if not initrd: > + raise WicError("initrd= must be specified when > create-unified-kernel-image=true, exiting") + > + deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE") > + efi_stub = glob("%s/%s" % (deploy_dir, > "linux*.efi.stub")) > + if len(efi_stub) == 0: > + raise WicError("Unified Kernel Image EFI stub not > found, exiting") > + efi_stub = efi_stub[0] > + > + with tempfile.TemporaryDirectory() as tmp_dir: > + label = source_params.get('label') > + label_conf = "root=%s" % creator.rootdev > + if label: > + label_conf = "LABEL=%s" % label > + > + bootloader = creator.ks.bootloader > + cmdline = open("%s/cmdline" % tmp_dir, "w") > + cmdline.write("%s %s" % (label_conf, > bootloader.append)) > + cmdline.close() > + > + initrds = initrd.split(';') > + initrd = open("%s/initrd" % tmp_dir, "wb") > + for f in initrds: > + with open("%s/%s" % (deploy_dir, f), 'rb') as > in_file: > + shutil.copyfileobj(in_file, initrd) > + initrd.close() > + > + # Searched by systemd-boot: > + # > https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images > + install_cmd = "install -d %s/EFI/Linux" % hdddir > + exec_cmd(install_cmd) > + > + staging_dir_host = > get_bitbake_var("STAGING_DIR_HOST") + > + # > https://www.freedesktop.org/software/systemd/man/systemd-stub.html > + objcopy_cmd = "objcopy \ > + --add-section .osrel=%s --change-section-vma > .osrel=0x20000 \ > + --add-section .cmdline=%s --change-section-vma > .cmdline=0x30000 \ > + --add-section .linux=%s --change-section-vma > .linux=0x2000000 \ > + --add-section .initrd=%s --change-section-vma > .initrd=0x3000000 \ > + %s %s" % \ > + ("%s/usr/lib/os-release" % staging_dir_host, > + cmdline.name, > + "%s/%s" % (staging_kernel_dir, kernel), > + initrd.name, > + efi_stub, > + "%s/EFI/Linux/linux.efi" % hdddir) > + exec_cmd(objcopy_cmd) > + else: > + install_cmd = "install -m 0644 %s/%s %s/%s" % \ > + (staging_kernel_dir, kernel, hdddir, kernel) > + exec_cmd(install_cmd) > > if get_bitbake_var("IMAGE_EFI_BOOT_FILES"): > for src_path, dst_path in cls.install_task: > diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py > b/scripts/lib/wic/plugins/source/bootimg-pcbios.py index > f2639e700493..32e47f183146 100644 --- > a/scripts/lib/wic/plugins/source/bootimg-pcbios.py +++ > b/scripts/lib/wic/plugins/source/bootimg-pcbios.py @@ -186,8 +186,10 > @@ class BootimgPcbiosPlugin(SourcePlugin): # dosfs image, created by > mkdosfs bootimg = "%s/boot%s.img" % (cr_workdir, part.lineno) > > - dosfs_cmd = "mkdosfs -n boot -i %s -S 512 -C %s %d" % \ > - (part.fsuuid, bootimg, blocks) > + label = part.label if part.label else "boot" > + > + dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ > + (label, part.fsuuid, bootimg, blocks) > exec_native_cmd(dosfs_cmd, native_sysroot) > > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir) > diff --git a/scripts/lib/wic/plugins/source/rawcopy.py > b/scripts/lib/wic/plugins/source/rawcopy.py index > 3c4997d8ba5e..7c90cd3cf82b 100644 --- > a/scripts/lib/wic/plugins/source/rawcopy.py +++ > b/scripts/lib/wic/plugins/source/rawcopy.py @@ -4,6 +4,8 @@ > > import logging > import os > +import signal > +import subprocess > > from wic import WicError > from wic.pluginbase import SourcePlugin > @@ -29,15 +31,34 @@ class RawCopyPlugin(SourcePlugin): > cmd = 'btrfs filesystem label %s %s' % (dst, label) > elif fstype == 'swap': > cmd = 'mkswap -L %s %s' % (label, dst) > - elif fstype == 'squashfs': > - raise WicError("It's not possible to update a squashfs " > - "filesystem label '%s'" % (label)) > + elif fstype in ('squashfs', 'erofs'): > + raise WicError("It's not possible to update a %s " > + "filesystem label '%s'" % (fstype, label)) > else: > raise WicError("Cannot update filesystem label: " > "Unknown fstype: '%s'" % (fstype)) > > exec_cmd(cmd) > > + @staticmethod > + def do_image_uncompression(src, dst, workdir): > + def subprocess_setup(): > + # Python installs a SIGPIPE handler by default. This is > usually not what > + # non-Python subprocesses expect. > + # SIGPIPE errors are known issues with gzip/bash > + signal.signal(signal.SIGPIPE, signal.SIG_DFL) > + > + extension = os.path.splitext(src)[1] > + decompressor = { > + ".bz2": "bzip2", > + ".gz": "gzip", > + ".xz": "xz" > + }.get(extension) > + if not decompressor: > + raise WicError("Not supported compressor filename > extension: %s" % extension) > + cmd = "%s -dc %s > %s" % (decompressor, src, dst) > + subprocess.call(cmd, preexec_fn=subprocess_setup, > shell=True, cwd=workdir) + > @classmethod > def do_prepare_partition(cls, part, source_params, cr, > cr_workdir, oe_builddir, bootimg_dir, kernel_dir, > @@ -56,7 +77,13 @@ class RawCopyPlugin(SourcePlugin): > if 'file' not in source_params: > raise WicError("No file specified") > > - src = os.path.join(kernel_dir, source_params['file']) > + if 'unpack' in source_params: > + img = os.path.join(kernel_dir, source_params['file']) > + src = os.path.join(cr_workdir, > os.path.splitext(source_params['file'])[0]) > + RawCopyPlugin.do_image_uncompression(img, src, > cr_workdir) > + else: > + src = os.path.join(kernel_dir, source_params['file']) > + > dst = os.path.join(cr_workdir, "%s.%s" % > (os.path.basename(source_params['file']), part.lineno)) > if not os.path.exists(os.path.dirname(dst)): > diff --git a/scripts/lib/wic/plugins/source/rootfs.py > b/scripts/lib/wic/plugins/source/rootfs.py index > 96d940a91d61..2e34e715ca73 100644 --- > a/scripts/lib/wic/plugins/source/rootfs.py +++ > b/scripts/lib/wic/plugins/source/rootfs.py @@ -218,7 +218,7 @@ class > RootfsPlugin(SourcePlugin): # Update part.has_fstab here as fstab may > have been added or # removed by the above modifications. > part.has_fstab = os.path.exists(os.path.join(new_rootfs, > "etc/fstab")) > - if part.update_fstab_in_rootfs and part.has_fstab: > + if part.update_fstab_in_rootfs and part.has_fstab and > not part.no_fstab_update: fstab_path = os.path.join(new_rootfs, > "etc/fstab") # Assume that fstab should always be owned by root with > fixed permissions install_cmd = "install -m 0644 %s %s" % > (part.updated_fstab_path, fstab_path) diff --git a/scripts/wic > b/scripts/wic index a741aed364bf..aee63a45aacf 100755 > --- a/scripts/wic > +++ b/scripts/wic > @@ -22,9 +22,9 @@ import sys > import argparse > import logging > import subprocess > +import shutil > > from collections import namedtuple > -from distutils import spawn > > # External modules > scripts_path = os.path.dirname(os.path.realpath(__file__)) > @@ -47,7 +47,7 @@ if os.environ.get('SDKTARGETSYSROOT'): > break > sdkroot = os.path.dirname(sdkroot) > > -bitbake_exe = spawn.find_executable('bitbake') > +bitbake_exe = shutil.which('bitbake') > if bitbake_exe: > bitbake_path = scriptpath.add_bitbake_lib_path() > import bb > @@ -159,6 +159,9 @@ def wic_create_subcommand(options, usage_str): > "(Use -e/--image-name to specify it)") > native_sysroot = options.native_sysroot > > + if options.kernel_dir: > + kernel_dir = options.kernel_dir > + > if not options.vars_dir and (not native_sysroot or not > os.path.isdir(native_sysroot)): logger.info("Building wic-tools...\n") > subprocess.check_call(["bitbake", "wic-tools"]) > @@ -346,6 +349,8 @@ def wic_init_parser_create(subparser): > default=False, help="output debug information") > subparser.add_argument("-i", "--imager", dest="imager", > default="direct", help="the wic imager plugin") > + subparser.add_argument("--extra-space", type=int, > dest="extra_space", > + default=0, help="additional free disk space to > add to the image") return > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-21 8:52 [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Henning Schild 2022-04-21 8:52 ` [PATCH v2 1/2] bitbake: Update to 1.50.5 release Henning Schild 2022-04-21 8:52 ` [PATCH v2 2/2] wic: Update to the latest revision Henning Schild @ 2022-04-22 7:59 ` Anton Mikanovich 2022-04-22 8:15 ` Bezdeka, Florian 2022-04-22 8:27 ` Henning Schild 2 siblings, 2 replies; 11+ messages in thread From: Anton Mikanovich @ 2022-04-22 7:59 UTC (permalink / raw) To: Henning Schild, isar-users Cc: Florian Bezdeka, Felix Moessbauer, Baurzhan Ismagulov 21.04.2022 11:52, Henning Schild wrote: > debian bookworm switched to python 3.10 recently, our versions of > bitbake and wic both have some issues with that new interpreter. All > these issues have been fixes in bitbake and OE, so all we need to do is > update the two tools in our tree. > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", which is p1 > here, only later i found that wic also needs a bump to v2 added p2 > > If this series is applied we do not need "[PATCH] wic-img: use > python3.9 in bookworm", in case this one is applied before it should be > reverted after the version bumps. > > Henning Schild (2): > bitbake: Update to 1.50.5 release > wic: Update to the latest revision > > bitbake/lib/bb/cache.py | 3 +- > bitbake/lib/bb/cooker.py | 30 +++++++- > bitbake/lib/bb/data_smart.py | 4 +- > bitbake/lib/bb/fetch2/__init__.py | 4 + > bitbake/lib/bb/fetch2/perforce.py | 2 +- > bitbake/lib/bb/fetch2/wget.py | 2 +- > bitbake/lib/bb/persist_data.py | 5 +- > bitbake/lib/bb/process.py | 2 +- > bitbake/lib/bb/runqueue.py | 34 ++++---- > bitbake/lib/bb/server/process.py | 2 +- > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > bitbake/lib/bb/utils.py | 13 +++- > bitbake/lib/hashserv/server.py | 4 +- > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > .../wic/plugins/source/bootimg-efi-isar.py | 77 ++++++++++++++++--- > .../wic/plugins/source/bootimg-pcbios-isar.py | 6 +- > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > scripts/lib/wic/engine.py | 6 +- > scripts/lib/wic/help.py | 10 ++- > scripts/lib/wic/ksparser.py | 8 +- > scripts/lib/wic/misc.py | 4 +- > scripts/lib/wic/partition.py | 25 ++++-- > scripts/lib/wic/pluginbase.py | 8 +- > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 +++++++++++++++--- > .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- > scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- > scripts/lib/wic/plugins/source/rootfs.py | 2 +- > scripts/wic | 9 ++- > 31 files changed, 320 insertions(+), 109 deletions(-) Thanks for update, python issue is really fixed now, but it brakes qemu386 targets booting in QEMU: 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, sector 0 00:46:00 [ 34.428049] floppy: error -5 while reading block 0 00:46:00 [ 34.508133] blk_update_request: I/O error, dev fd0, sector 0 00:46:00 [ 34.512050] floppy: error -5 while reading block 0 00:46:00 Gave up waiting for root file system device. Common problems: 00:46:00 - Boot args (cat /proc/cmdline) 00:46:00 - Check rootdelay= (did the system wait long enough?) 00:46:00 - Missing modules (cat /proc/modules; ls /dev) 00:46:00 ALERT! PARTLABEL=platform does not exist. Dropping to a shell! 00:46:00 (initramfs) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 7:59 ` [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Anton Mikanovich @ 2022-04-22 8:15 ` Bezdeka, Florian 2022-04-22 8:26 ` Henning Schild 2022-04-22 8:27 ` Henning Schild 1 sibling, 1 reply; 11+ messages in thread From: Bezdeka, Florian @ 2022-04-22 8:15 UTC (permalink / raw) To: amikan, isar-users, Schild, Henning Cc: Schmidt, Adriaan, ibr, Moessbauer, Felix On Fri, 2022-04-22 at 10:59 +0300, Anton Mikanovich wrote: > 21.04.2022 11:52, Henning Schild wrote: > > debian bookworm switched to python 3.10 recently, our versions of > > bitbake and wic both have some issues with that new interpreter. All > > these issues have been fixes in bitbake and OE, so all we need to do is > > update the two tools in our tree. > > > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", which is p1 > > here, only later i found that wic also needs a bump to v2 added p2 > > > > If this series is applied we do not need "[PATCH] wic-img: use > > python3.9 in bookworm", in case this one is applied before it should be > > reverted after the version bumps. > > > > Henning Schild (2): > > bitbake: Update to 1.50.5 release > > wic: Update to the latest revision > > > > bitbake/lib/bb/cache.py | 3 +- > > bitbake/lib/bb/cooker.py | 30 +++++++- > > bitbake/lib/bb/data_smart.py | 4 +- > > bitbake/lib/bb/fetch2/__init__.py | 4 + > > bitbake/lib/bb/fetch2/perforce.py | 2 +- > > bitbake/lib/bb/fetch2/wget.py | 2 +- > > bitbake/lib/bb/persist_data.py | 5 +- > > bitbake/lib/bb/process.py | 2 +- > > bitbake/lib/bb/runqueue.py | 34 ++++---- > > bitbake/lib/bb/server/process.py | 2 +- > > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > > bitbake/lib/bb/utils.py | 13 +++- > > bitbake/lib/hashserv/server.py | 4 +- > > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > > .../wic/plugins/source/bootimg-efi-isar.py | 77 ++++++++++++++++--- > > .../wic/plugins/source/bootimg-pcbios-isar.py | 6 +- > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > > scripts/lib/wic/engine.py | 6 +- > > scripts/lib/wic/help.py | 10 ++- > > scripts/lib/wic/ksparser.py | 8 +- > > scripts/lib/wic/misc.py | 4 +- > > scripts/lib/wic/partition.py | 25 ++++-- > > scripts/lib/wic/pluginbase.py | 8 +- > > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 +++++++++++++++--- > > .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- > > scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- > > scripts/lib/wic/plugins/source/rootfs.py | 2 +- > > scripts/wic | 9 ++- > > 31 files changed, 320 insertions(+), 109 deletions(-) > > Thanks for update, python issue is really fixed now, but it brakes > qemu386 targets booting in QEMU: > > 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, sector 0 > 00:46:00 [ 34.428049] floppy: error -5 while reading block 0 > 00:46:00 [ 34.508133] blk_update_request: I/O error, dev fd0, sector 0 > 00:46:00 [ 34.512050] floppy: error -5 while reading block 0 > 00:46:00 Gave up waiting for root file system device. Common problems: > 00:46:00 - Boot args (cat /proc/cmdline) > 00:46:00 - Check rootdelay= (did the system wait long enough?) > 00:46:00 - Missing modules (cat /proc/modules; ls /dev) > 00:46:00 ALERT! PARTLABEL=platform does not exist. Dropping to a shell! > 00:46:00 (initramfs) > Might be upstream commit [1] which Henning pulled in now. Adding Adriaan as author of this commit to CC. I have no idea why this should break exactly this image... [1] https://github.com/openembedded/openembedded-core/commit/2fb247c5ecf057bb96649a3c0234794b4991c050 Regards, Florian ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 8:15 ` Bezdeka, Florian @ 2022-04-22 8:26 ` Henning Schild 2022-04-22 8:52 ` Henning Schild 0 siblings, 1 reply; 11+ messages in thread From: Henning Schild @ 2022-04-22 8:26 UTC (permalink / raw) To: Bezdeka, Florian (T CED SES-DE) Cc: amikan, isar-users, Schmidt, Adriaan (T CED SES-DE), ibr, Moessbauer, Felix (T CED SES-DE) Am Fri, 22 Apr 2022 10:15:22 +0200 schrieb "Bezdeka, Florian (T CED SES-DE)" <florian.bezdeka@siemens.com>: > On Fri, 2022-04-22 at 10:59 +0300, Anton Mikanovich wrote: > > 21.04.2022 11:52, Henning Schild wrote: > > > debian bookworm switched to python 3.10 recently, our versions of > > > bitbake and wic both have some issues with that new interpreter. > > > All these issues have been fixes in bitbake and OE, so all we > > > need to do is update the two tools in our tree. > > > > > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", which > > > is p1 here, only later i found that wic also needs a bump to v2 > > > added p2 > > > > > > If this series is applied we do not need "[PATCH] wic-img: use > > > python3.9 in bookworm", in case this one is applied before it > > > should be reverted after the version bumps. > > > > > > Henning Schild (2): > > > bitbake: Update to 1.50.5 release > > > wic: Update to the latest revision > > > > > > bitbake/lib/bb/cache.py | 3 +- > > > bitbake/lib/bb/cooker.py | 30 +++++++- > > > bitbake/lib/bb/data_smart.py | 4 +- > > > bitbake/lib/bb/fetch2/__init__.py | 4 + > > > bitbake/lib/bb/fetch2/perforce.py | 2 +- > > > bitbake/lib/bb/fetch2/wget.py | 2 +- > > > bitbake/lib/bb/persist_data.py | 5 +- > > > bitbake/lib/bb/process.py | 2 +- > > > bitbake/lib/bb/runqueue.py | 34 ++++---- > > > bitbake/lib/bb/server/process.py | 2 +- > > > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > > > bitbake/lib/bb/utils.py | 13 +++- > > > bitbake/lib/hashserv/server.py | 4 +- > > > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > > > .../wic/plugins/source/bootimg-efi-isar.py | 77 > > > ++++++++++++++++--- .../wic/plugins/source/bootimg-pcbios-isar.py > > > | 6 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > > > scripts/lib/wic/engine.py | 6 +- > > > scripts/lib/wic/help.py | 10 ++- > > > scripts/lib/wic/ksparser.py | 8 +- > > > scripts/lib/wic/misc.py | 4 +- > > > scripts/lib/wic/partition.py | 25 ++++-- > > > scripts/lib/wic/pluginbase.py | 8 +- > > > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > > > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 > > > +++++++++++++++--- .../lib/wic/plugins/source/bootimg-pcbios.py > > > | 6 +- scripts/lib/wic/plugins/source/rawcopy.py | 35 > > > ++++++++- scripts/lib/wic/plugins/source/rootfs.py | 2 +- > > > scripts/wic | 9 ++- > > > 31 files changed, 320 insertions(+), 109 deletions(-) > > > > Thanks for update, python issue is really fixed now, but it brakes > > qemu386 targets booting in QEMU: Thanks, i will look into this. > > 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, > > sector 0 00:46:00 [ 34.428049] floppy: error -5 while reading > > block 0 00:46:00 [ 34.508133] blk_update_request: I/O error, dev > > fd0, sector 0 00:46:00 [ 34.512050] floppy: error -5 while > > reading block 0 00:46:00 Gave up waiting for root file system > > device. Common problems: 00:46:00 - Boot args (cat /proc/cmdline) > > 00:46:00 - Check rootdelay= (did the system wait long enough?) > > 00:46:00 - Missing modules (cat /proc/modules; ls /dev) > > 00:46:00 ALERT! PARTLABEL=platform does not exist. Dropping to a > > shell! 00:46:00 (initramfs) > > > > Might be upstream commit [1] which Henning pulled in now. > > Adding Adriaan as author of this commit to CC. I have no idea why this > should break exactly this image... We boot all our i386 with BIOS directdisk-isar.wks and all amd64 with EFI sdimage-efi.wks, plus a few exceptions from that pattern where we use efi-plus-pcbios.wks or systemd-boot/EFI with sdimage-efi-sd.wks So maybe that directdisk-isar.wks or common-isar.wks.inc need some bits i did not see and good old legacy BIOS is indeed broken in general with these patches. Henning > [1] > https://github.com/openembedded/openembedded-core/commit/2fb247c5ecf057bb96649a3c0234794b4991c050 > > Regards, > Florian > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 8:26 ` Henning Schild @ 2022-04-22 8:52 ` Henning Schild 2022-04-22 9:54 ` Henning Schild 0 siblings, 1 reply; 11+ messages in thread From: Henning Schild @ 2022-04-22 8:52 UTC (permalink / raw) To: Bezdeka, Florian (T CED SES-DE) Cc: amikan, isar-users, Schmidt, Adriaan (T CED SES-DE), ibr, Moessbauer, Felix (T CED SES-DE) Am Fri, 22 Apr 2022 10:26:08 +0200 schrieb Henning Schild <henning.schild@siemens.com>: > Am Fri, 22 Apr 2022 10:15:22 +0200 > schrieb "Bezdeka, Florian (T CED SES-DE)" > <florian.bezdeka@siemens.com>: > > > On Fri, 2022-04-22 at 10:59 +0300, Anton Mikanovich wrote: > > > 21.04.2022 11:52, Henning Schild wrote: > > > > debian bookworm switched to python 3.10 recently, our versions > > > > of bitbake and wic both have some issues with that new > > > > interpreter. All these issues have been fixes in bitbake and > > > > OE, so all we need to do is update the two tools in our tree. > > > > > > > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", > > > > which is p1 here, only later i found that wic also needs a bump > > > > to v2 added p2 > > > > > > > > If this series is applied we do not need "[PATCH] wic-img: use > > > > python3.9 in bookworm", in case this one is applied before it > > > > should be reverted after the version bumps. > > > > > > > > Henning Schild (2): > > > > bitbake: Update to 1.50.5 release > > > > wic: Update to the latest revision > > > > > > > > bitbake/lib/bb/cache.py | 3 +- > > > > bitbake/lib/bb/cooker.py | 30 +++++++- > > > > bitbake/lib/bb/data_smart.py | 4 +- > > > > bitbake/lib/bb/fetch2/__init__.py | 4 + > > > > bitbake/lib/bb/fetch2/perforce.py | 2 +- > > > > bitbake/lib/bb/fetch2/wget.py | 2 +- > > > > bitbake/lib/bb/persist_data.py | 5 +- > > > > bitbake/lib/bb/process.py | 2 +- > > > > bitbake/lib/bb/runqueue.py | 34 ++++---- > > > > bitbake/lib/bb/server/process.py | 2 +- > > > > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > > > > bitbake/lib/bb/utils.py | 13 +++- > > > > bitbake/lib/hashserv/server.py | 4 +- > > > > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > > > > .../wic/plugins/source/bootimg-efi-isar.py | 77 > > > > ++++++++++++++++--- > > > > .../wic/plugins/source/bootimg-pcbios-isar.py | 6 +- > > > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > > > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > > > > scripts/lib/wic/engine.py | 6 +- > > > > scripts/lib/wic/help.py | 10 ++- > > > > scripts/lib/wic/ksparser.py | 8 +- > > > > scripts/lib/wic/misc.py | 4 +- > > > > scripts/lib/wic/partition.py | 25 ++++-- > > > > scripts/lib/wic/pluginbase.py | 8 +- > > > > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > > > > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 > > > > +++++++++++++++--- .../lib/wic/plugins/source/bootimg-pcbios.py > > > > | 6 +- scripts/lib/wic/plugins/source/rawcopy.py | 35 > > > > ++++++++- scripts/lib/wic/plugins/source/rootfs.py | 2 +- > > > > scripts/wic | 9 ++- 31 files > > > > changed, 320 insertions(+), 109 deletions(-) > > > > > > Thanks for update, python issue is really fixed now, but it > > > brakes qemu386 targets booting in QEMU: > > Thanks, i will look into this. > > > > 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, > > > sector 0 00:46:00 [ 34.428049] floppy: error -5 while reading > > > block 0 00:46:00 [ 34.508133] blk_update_request: I/O error, dev > > > fd0, sector 0 00:46:00 [ 34.512050] floppy: error -5 while > > > reading block 0 00:46:00 Gave up waiting for root file system > > > device. Common problems: 00:46:00 - Boot args (cat > > > /proc/cmdline) 00:46:00 - Check rootdelay= (did the system > > > wait long enough?) 00:46:00 - Missing modules (cat > > > /proc/modules; ls /dev) 00:46:00 ALERT! PARTLABEL=platform does > > > not exist. Dropping to a shell! 00:46:00 (initramfs) > > > > > > > Might be upstream commit [1] which Henning pulled in now. > > > > Adding Adriaan as author of this commit to CC. I have no idea why > > this should break exactly this image... > > We boot all our i386 with BIOS directdisk-isar.wks and all amd64 with > EFI sdimage-efi.wks, plus a few exceptions from that pattern where we > use efi-plus-pcbios.wks or systemd-boot/EFI with sdimage-efi-sd.wks > > So maybe that directdisk-isar.wks or common-isar.wks.inc need some > bits i did not see and good old legacy BIOS is indeed broken in > general with these patches. PARTLABEL is a gpt thing, and we use msdos. And our common-isar.wks.inc does not have --use-uuid while upstream common.wks.inc has it. Will send a patch. Could not find if we ever dropped that use-uuid, i think it is more likely it was overlooked while maintaining the fork. Henning > Henning > > > [1] > > https://github.com/openembedded/openembedded-core/commit/2fb247c5ecf057bb96649a3c0234794b4991c050 > > > > Regards, > > Florian > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 8:52 ` Henning Schild @ 2022-04-22 9:54 ` Henning Schild 0 siblings, 0 replies; 11+ messages in thread From: Henning Schild @ 2022-04-22 9:54 UTC (permalink / raw) To: Bezdeka, Florian (T CED SES-DE) Cc: amikan, isar-users, Schmidt, Adriaan (T CED SES-DE), ibr, Moessbauer, Felix (T CED SES-DE) Am Fri, 22 Apr 2022 10:52:11 +0200 schrieb Henning Schild <henning.schild@siemens.com>: > Am Fri, 22 Apr 2022 10:26:08 +0200 > schrieb Henning Schild <henning.schild@siemens.com>: > > > Am Fri, 22 Apr 2022 10:15:22 +0200 > > schrieb "Bezdeka, Florian (T CED SES-DE)" > > <florian.bezdeka@siemens.com>: > > > > > On Fri, 2022-04-22 at 10:59 +0300, Anton Mikanovich wrote: > > > > 21.04.2022 11:52, Henning Schild wrote: > > > > > debian bookworm switched to python 3.10 recently, our versions > > > > > of bitbake and wic both have some issues with that new > > > > > interpreter. All these issues have been fixes in bitbake and > > > > > OE, so all we need to do is update the two tools in our tree. > > > > > > > > > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", > > > > > which is p1 here, only later i found that wic also needs a > > > > > bump to v2 added p2 > > > > > > > > > > If this series is applied we do not need "[PATCH] wic-img: use > > > > > python3.9 in bookworm", in case this one is applied before it > > > > > should be reverted after the version bumps. > > > > > > > > > > Henning Schild (2): > > > > > bitbake: Update to 1.50.5 release > > > > > wic: Update to the latest revision > > > > > > > > > > bitbake/lib/bb/cache.py | 3 +- > > > > > bitbake/lib/bb/cooker.py | 30 +++++++- > > > > > bitbake/lib/bb/data_smart.py | 4 +- > > > > > bitbake/lib/bb/fetch2/__init__.py | 4 + > > > > > bitbake/lib/bb/fetch2/perforce.py | 2 +- > > > > > bitbake/lib/bb/fetch2/wget.py | 2 +- > > > > > bitbake/lib/bb/persist_data.py | 5 +- > > > > > bitbake/lib/bb/process.py | 2 +- > > > > > bitbake/lib/bb/runqueue.py | 34 ++++---- > > > > > bitbake/lib/bb/server/process.py | 2 +- > > > > > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > > > > > bitbake/lib/bb/utils.py | 13 +++- > > > > > bitbake/lib/hashserv/server.py | 4 +- > > > > > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > > > > > .../wic/plugins/source/bootimg-efi-isar.py | 77 > > > > > ++++++++++++++++--- > > > > > .../wic/plugins/source/bootimg-pcbios-isar.py | 6 +- > > > > > scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > > > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > > > > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > > > > > scripts/lib/wic/engine.py | 6 +- > > > > > scripts/lib/wic/help.py | 10 ++- > > > > > scripts/lib/wic/ksparser.py | 8 +- > > > > > scripts/lib/wic/misc.py | 4 +- > > > > > scripts/lib/wic/partition.py | 25 ++++-- > > > > > scripts/lib/wic/pluginbase.py | 8 +- > > > > > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > > > > > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 > > > > > +++++++++++++++--- > > > > > .../lib/wic/plugins/source/bootimg-pcbios.py | 6 +- > > > > > scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- > > > > > scripts/lib/wic/plugins/source/rootfs.py | 2 +- > > > > > scripts/wic | 9 ++- 31 > > > > > files changed, 320 insertions(+), 109 deletions(-) > > > > > > > > Thanks for update, python issue is really fixed now, but it > > > > brakes qemu386 targets booting in QEMU: > > > > Thanks, i will look into this. > > > > > > 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, > > > > sector 0 00:46:00 [ 34.428049] floppy: error -5 while reading > > > > block 0 00:46:00 [ 34.508133] blk_update_request: I/O error, > > > > dev fd0, sector 0 00:46:00 [ 34.512050] floppy: error -5 while > > > > reading block 0 00:46:00 Gave up waiting for root file system > > > > device. Common problems: 00:46:00 - Boot args (cat > > > > /proc/cmdline) 00:46:00 - Check rootdelay= (did the system > > > > wait long enough?) 00:46:00 - Missing modules (cat > > > > /proc/modules; ls /dev) 00:46:00 ALERT! PARTLABEL=platform does > > > > not exist. Dropping to a shell! 00:46:00 (initramfs) > > > > > > > > > > Might be upstream commit [1] which Henning pulled in now. > > > > > > Adding Adriaan as author of this commit to CC. I have no idea why > > > this should break exactly this image... > > > > We boot all our i386 with BIOS directdisk-isar.wks and all amd64 > > with EFI sdimage-efi.wks, plus a few exceptions from that pattern > > where we use efi-plus-pcbios.wks or systemd-boot/EFI with > > sdimage-efi-sd.wks > > > > So maybe that directdisk-isar.wks or common-isar.wks.inc need some > > bits i did not see and good old legacy BIOS is indeed broken in > > general with these patches. > > PARTLABEL is a gpt thing, and we use msdos. And our > common-isar.wks.inc does not have --use-uuid while upstream > common.wks.inc has it. > > Will send a patch. Could not find if we ever dropped that use-uuid, i > think it is more likely it was overlooked while maintaining the fork. v3 solves this with a patch to align our common wks with OE, additionally i started an upstream discussion with a patch to not allow anyone to PARTLABEL on msdos https://lists.openembedded.org/g/openembedded-core/topic/patch_wic_do_not_use/90624635?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,0,90624635,previd%3D1650620877809916310,nextid%3D1650546439549613402&previd=1650620877809916310&nextid=1650546439549613402 Henning > Henning > > > Henning > > > > > [1] > > > https://github.com/openembedded/openembedded-core/commit/2fb247c5ecf057bb96649a3c0234794b4991c050 > > > > > > Regards, > > > Florian > > > > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 7:59 ` [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Anton Mikanovich 2022-04-22 8:15 ` Bezdeka, Florian @ 2022-04-22 8:27 ` Henning Schild 2022-04-22 8:57 ` Anton Mikanovich 1 sibling, 1 reply; 11+ messages in thread From: Henning Schild @ 2022-04-22 8:27 UTC (permalink / raw) To: Anton Mikanovich Cc: isar-users, Florian Bezdeka, Felix Moessbauer, Baurzhan Ismagulov Am Fri, 22 Apr 2022 10:59:15 +0300 schrieb Anton Mikanovich <amikan@ilbers.de>: > 21.04.2022 11:52, Henning Schild wrote: > > debian bookworm switched to python 3.10 recently, our versions of > > bitbake and wic both have some issues with that new interpreter. All > > these issues have been fixes in bitbake and OE, so all we need to > > do is update the two tools in our tree. > > > > v1 was send as "[PATCH] bitbake: Update to 1.50.5 release", which > > is p1 here, only later i found that wic also needs a bump to v2 > > added p2 > > > > If this series is applied we do not need "[PATCH] wic-img: use > > python3.9 in bookworm", in case this one is applied before it > > should be reverted after the version bumps. > > > > Henning Schild (2): > > bitbake: Update to 1.50.5 release > > wic: Update to the latest revision > > > > bitbake/lib/bb/cache.py | 3 +- > > bitbake/lib/bb/cooker.py | 30 +++++++- > > bitbake/lib/bb/data_smart.py | 4 +- > > bitbake/lib/bb/fetch2/__init__.py | 4 + > > bitbake/lib/bb/fetch2/perforce.py | 2 +- > > bitbake/lib/bb/fetch2/wget.py | 2 +- > > bitbake/lib/bb/persist_data.py | 5 +- > > bitbake/lib/bb/process.py | 2 +- > > bitbake/lib/bb/runqueue.py | 34 ++++---- > > bitbake/lib/bb/server/process.py | 2 +- > > bitbake/lib/bb/tests/fetch.py | 35 +++++---- > > bitbake/lib/bb/utils.py | 13 +++- > > bitbake/lib/hashserv/server.py | 4 +- > > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > > .../wic/plugins/source/bootimg-efi-isar.py | 77 > > ++++++++++++++++--- .../wic/plugins/source/bootimg-pcbios-isar.py | > > 6 +- scripts/lib/wic/canned-wks/common.wks.inc | 2 +- > > scripts/lib/wic/canned-wks/directdisk-gpt.wks | 2 +- > > scripts/lib/wic/canned-wks/mkefidisk.wks | 2 +- > > scripts/lib/wic/engine.py | 6 +- > > scripts/lib/wic/help.py | 10 ++- > > scripts/lib/wic/ksparser.py | 8 +- > > scripts/lib/wic/misc.py | 4 +- > > scripts/lib/wic/partition.py | 25 ++++-- > > scripts/lib/wic/pluginbase.py | 8 +- > > scripts/lib/wic/plugins/imager/direct.py | 11 ++- > > scripts/lib/wic/plugins/source/bootimg-efi.py | 74 > > +++++++++++++++--- .../lib/wic/plugins/source/bootimg-pcbios.py | > > 6 +- scripts/lib/wic/plugins/source/rawcopy.py | 35 ++++++++- > > scripts/lib/wic/plugins/source/rootfs.py | 2 +- > > scripts/wic | 9 ++- > > 31 files changed, 320 insertions(+), 109 deletions(-) > > Thanks for update, python issue is really fixed now, but it brakes > qemu386 targets booting in QEMU: All of them or only some suits? I am building qemui386-bullseye now for local repro. Henning > 00:46:00 [ 34.424136] blk_update_request: I/O error, dev fd0, > sector 0 00:46:00 [ 34.428049] floppy: error -5 while reading block > 0 00:46:00 [ 34.508133] blk_update_request: I/O error, dev fd0, > sector 0 00:46:00 [ 34.512050] floppy: error -5 while reading block > 0 00:46:00 Gave up waiting for root file system device. Common > problems: 00:46:00 - Boot args (cat /proc/cmdline) > 00:46:00 - Check rootdelay= (did the system wait long enough?) > 00:46:00 - Missing modules (cat /proc/modules; ls /dev) > 00:46:00 ALERT! PARTLABEL=platform does not exist. Dropping to a > shell! 00:46:00 (initramfs) > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/2] bump bitbake and wic for python 3.10 support 2022-04-22 8:27 ` Henning Schild @ 2022-04-22 8:57 ` Anton Mikanovich 0 siblings, 0 replies; 11+ messages in thread From: Anton Mikanovich @ 2022-04-22 8:57 UTC (permalink / raw) To: Henning Schild Cc: isar-users, Florian Bezdeka, Felix Moessbauer, Baurzhan Ismagulov 22.04.2022 11:27, Henning Schild wrote: > Am Fri, 22 Apr 2022 10:59:15 +0300 > schrieb Anton Mikanovich <amikan@ilbers.de>: > >> Thanks for update, python issue is really fixed now, but it brakes >> qemu386 targets booting in QEMU: > All of them or only some suits? I am building qemui386-bullseye now for > local repro. > > Henning > I've tried buster and stretch (we have them covered by CI start_vm) - both fails. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-04-22 9:54 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-21 8:52 [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Henning Schild 2022-04-21 8:52 ` [PATCH v2 1/2] bitbake: Update to 1.50.5 release Henning Schild 2022-04-21 8:52 ` [PATCH v2 2/2] wic: Update to the latest revision Henning Schild 2022-04-21 9:02 ` Henning Schild 2022-04-22 7:59 ` [PATCH v2 0/2] bump bitbake and wic for python 3.10 support Anton Mikanovich 2022-04-22 8:15 ` Bezdeka, Florian 2022-04-22 8:26 ` Henning Schild 2022-04-22 8:52 ` Henning Schild 2022-04-22 9:54 ` Henning Schild 2022-04-22 8:27 ` Henning Schild 2022-04-22 8:57 ` Anton Mikanovich
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox