* [PATCH 0/8] Update BitBake in Isar to v2.20.0
@ 2026-11-09 9:17 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
This series updates Isar's BitBake from v2.8.0 to v2.20.0, which is the
latest stable release. The update includes Isar adaptions to the
intervening fetcher, signature and metadata parser changes.
Debian Buster remains supported as an image target. Its imaging chroot
provides Python 3.7, while BitBake v2.20.0 requires Python 3.9. The
series moves WIC's Python driver to the build host and keeps image
utilities in the imaging chroot.
The testsuite was run in rootful and rootless modes, using separate
work, download and sstate directories. I tested:
- DevTest.test_dev_apps
- WicTest.test_wic_nodeploy_partitions and
WicTest.test_wic_deploy_partitions
- Buster image builds and EFI/BIOS boot tests
- WIC image builds for qemuamd64-iso/bookworm,
bananapi/bullseye and qemui386/buster
- KernelTests.test_kernel_cross
- NoCrossTest.test_nocross_riscv_trixie in rootless mode
I also executed for a few hours these tests without error, but they
take forever to complete:
- CrossTest.test_cross
- NoCrossTest.test_nocross
Best regards,
Clara Kowalsky
Clara Kowalsky (8):
base: replace BitBake's outdated persist_data
isar-events: preserve cleanup for default multiconfig
isar-events: close /proc/mounts after cleanup
meta: migrate fetcher references to bb.fetch
wic: run the Python driver outside the imaging chroot
bitbake.conf: explicitly exclude export and func from signatures
meta: add missing whitespace around assignment operators
bitbake: update to v2.20.0
.gitignore | 2 +-
RECIPE-API-CHANGELOG.md | 17 +
bitbake/.b4-config | 4 +
bitbake/.gitignore | 2 +
bitbake/AGENTS.md | 55 +
bitbake/README | 2 +-
bitbake/TODO | 65 +-
bitbake/bin/bitbake | 4 +-
bitbake/bin/bitbake-config-build | 1 +
bitbake/bin/bitbake-diffsigs | 16 +-
bitbake/bin/bitbake-getvar | 21 +-
bitbake/bin/bitbake-hashclient | 115 +-
bitbake/bin/bitbake-hashserv | 17 +-
bitbake/bin/bitbake-layers | 46 +-
bitbake/bin/bitbake-prserv | 34 +-
bitbake/bin/bitbake-selftest | 13 +-
bitbake/bin/bitbake-server | 14 +-
bitbake/bin/bitbake-setup | 1481 ++++++
bitbake/bin/bitbake-worker | 42 +-
bitbake/bin/git-make-shallow | 175 -
bitbake/bin/toaster | 23 +-
bitbake/bin/toaster-eventreplay | 7 +
bitbake/conf/bitbake.conf | 2 +-
bitbake/contrib/b4-wrapper-bitbake.py | 40 +
bitbake/contrib/hashserv/Dockerfile | 5 +-
bitbake/contrib/prserv/Dockerfile | 3 +-
bitbake/contrib/vim/LICENSE.txt | 18 -
bitbake/contrib/vim/ftdetect/bitbake.vim | 24 -
bitbake/contrib/vim/ftplugin/bitbake.vim | 13 -
bitbake/contrib/vim/indent/bitbake.vim | 343 --
bitbake/contrib/vim/plugin/newbb.vim | 88 -
bitbake/contrib/vim/plugin/newbbappend.vim | 46 -
bitbake/contrib/vim/syntax/bitbake.vim | 131 -
.../oe-nodistro-blacksail.conf.json | 48 +
.../oe-nodistro-master.conf.json | 48 +
.../oe-nodistro-whinlatter.conf.json | 56 +
.../oe-nodistro-wrynose.conf.json | 48 +
.../configurations/poky-blacksail.conf.json | 74 +
.../configurations/poky-master.conf.json | 74 +
.../configurations/poky-whinlatter.conf.json | 80 +
.../configurations/poky-wrynose.conf.json | 74 +
bitbake/doc/.gitignore | 3 +
bitbake/doc/Makefile | 2 +-
bitbake/doc/README | 2 +-
.../bitbake-user-manual-environment-setup.rst | 1332 +++++
.../bitbake-user-manual-execution.rst | 35 +-
.../bitbake-user-manual-fetching.rst | 305 +-
.../bitbake-user-manual-intro.rst | 267 +-
.../bitbake-user-manual-library-functions.rst | 59 +
.../bitbake-user-manual-metadata.rst | 462 +-
.../bitbake-user-manual-ref-variables.rst | 464 +-
bitbake/doc/conf.py | 41 +-
bitbake/doc/index.rst | 2 +
bitbake/doc/releases.rst | 174 -
bitbake/doc/setversions.py | 374 ++
bitbake/doc/sphinx-static/switchers.js | 233 -
bitbake/doc/sphinx-static/switchers.js.in | 249 +
bitbake/doc/sphinx-static/theme_overrides.css | 17 +-
bitbake/lib/bb/COW.py | 10 +-
bitbake/lib/bb/__init__.py | 158 +-
.../_vendor/beautifulsoup4.LICENSE} | 11 +-
bitbake/lib/bb/_vendor/bs4/__init__.py | 1177 +++++
bitbake/lib/bb/_vendor/bs4/_deprecation.py | 80 +
bitbake/lib/bb/_vendor/bs4/_typing.py | 205 +
bitbake/lib/bb/_vendor/bs4/_warnings.py | 98 +
.../lib/bb/_vendor/bs4/builder/__init__.py | 848 ++++
.../lib/bb/_vendor/bs4/builder/_html5lib.py | 611 +++
.../lib/bb/_vendor/bs4/builder/_htmlparser.py | 517 ++
bitbake/lib/bb/_vendor/bs4/builder/_lxml.py | 501 ++
bitbake/lib/bb/_vendor/bs4/css.py | 330 ++
bitbake/lib/bb/_vendor/bs4/dammit.py | 1525 ++++++
bitbake/lib/bb/_vendor/bs4/diagnose.py | 268 +
bitbake/lib/bb/_vendor/bs4/element.py | 3395 +++++++++++++
bitbake/lib/bb/_vendor/bs4/exceptions.py | 28 +
bitbake/lib/bb/_vendor/bs4/filter.py | 764 +++
bitbake/lib/bb/_vendor/bs4/formatter.py | 276 ++
bitbake/lib/bb/_vendor/bs4/py.typed | 0
bitbake/lib/bb/_vendor/ply.pyi | 1 +
bitbake/lib/{ => bb/_vendor}/ply/__init__.py | 1 +
bitbake/lib/bb/_vendor/ply/cpp.py | 918 ++++
bitbake/lib/bb/_vendor/ply/ctokens.py | 133 +
bitbake/lib/bb/_vendor/ply/lex.py | 901 ++++
bitbake/lib/{ => bb/_vendor}/ply/yacc.py | 2238 +++++----
bitbake/lib/bb/_vendor/ply/ygen.py | 74 +
bitbake/lib/bb/_vendor/progressbar.pyi | 1 +
.../{ => bb/_vendor}/progressbar/LICENSE.txt | 0
.../{ => bb/_vendor}/progressbar/__init__.py | 4 +-
.../{ => bb/_vendor}/progressbar/compat.py | 0
.../_vendor}/progressbar/progressbar.py | 37 +-
.../{ => bb/_vendor}/progressbar/widgets.py | 14 +-
bitbake/lib/bb/_vendor/simplediff.pyi | 1 +
.../lib/{ => bb/_vendor}/simplediff/LICENSE | 0
.../{ => bb/_vendor}/simplediff/__init__.py | 2 +-
bitbake/lib/bb/_vendor/tomli/LICENSE | 21 +
bitbake/lib/bb/_vendor/tomli/__init__.py | 8 +
bitbake/lib/bb/_vendor/tomli/_parser.py | 793 +++
bitbake/lib/bb/_vendor/tomli/_re.py | 119 +
bitbake/lib/bb/_vendor/tomli/_types.py | 10 +
bitbake/lib/bb/_vendor/tomli/py.typed | 1 +
.../lib/bb/_vendor/typing_extensions.LICENSE | 279 ++
bitbake/lib/bb/_vendor/typing_extensions.py | 4317 +++++++++++++++++
bitbake/lib/bb/_vendor/typing_extensions.pyi | 1 +
bitbake/lib/bb/acl.py | 2 -
bitbake/lib/bb/asyncrpc/__init__.py | 3 +-
bitbake/lib/bb/asyncrpc/client.py | 162 +-
bitbake/lib/bb/asyncrpc/serv.py | 58 +-
bitbake/lib/bb/asyncrpc/taskgroup.py | 52 +
bitbake/lib/bb/build.py | 29 +-
bitbake/lib/bb/cache.py | 37 +-
bitbake/lib/bb/checksum.py | 25 +
bitbake/lib/bb/codeparser.py | 102 +-
bitbake/lib/bb/command.py | 50 +-
bitbake/lib/bb/compress/lz4.py | 4 +-
bitbake/lib/bb/cooker.py | 319 +-
bitbake/lib/bb/cookerdata.py | 40 +-
bitbake/lib/bb/daemonize.py | 3 +
bitbake/lib/bb/data.py | 19 +-
bitbake/lib/bb/data_smart.py | 157 +-
bitbake/lib/bb/event.py | 29 +-
bitbake/lib/bb/{fetch2 => fetch}/README | 0
bitbake/lib/bb/{fetch2 => fetch}/__init__.py | 526 +-
bitbake/lib/bb/{fetch2 => fetch}/az.py | 23 +-
bitbake/lib/bb/{fetch2 => fetch}/clearcase.py | 63 +-
bitbake/lib/bb/fetch/crate.py | 198 +
bitbake/lib/bb/{fetch2 => fetch}/gcp.py | 45 +-
bitbake/lib/bb/{fetch2 => fetch}/git.py | 674 +--
bitbake/lib/bb/{fetch2 => fetch}/gitannex.py | 22 +-
bitbake/lib/bb/{fetch2 => fetch}/gitsm.py | 144 +-
bitbake/lib/bb/fetch/gomod.py | 270 ++
bitbake/lib/bb/{fetch2 => fetch}/hg.py | 65 +-
bitbake/lib/bb/{fetch2 => fetch}/local.py | 15 +-
bitbake/lib/bb/{fetch2 => fetch}/npm.py | 169 +-
bitbake/lib/bb/{fetch2 => fetch}/npmsw.py | 120 +-
bitbake/lib/bb/{fetch2 => fetch}/perforce.py | 46 +-
bitbake/lib/bb/{fetch2 => fetch}/repo.py | 26 +-
bitbake/lib/bb/{fetch2 => fetch}/s3.py | 21 +-
bitbake/lib/bb/{fetch2 => fetch}/sftp.py | 23 +-
bitbake/lib/bb/{fetch2 => fetch}/ssh.py | 33 +-
bitbake/lib/bb/{fetch2 => fetch}/svn.py | 88 +-
bitbake/lib/bb/{fetch2 => fetch}/wget.py | 316 +-
bitbake/lib/bb/fetch2.py | 19 +
bitbake/lib/bb/fetch2/bzr.py | 128 -
bitbake/lib/bb/fetch2/crate.py | 141 -
bitbake/lib/bb/fetch2/cvs.py | 157 -
bitbake/lib/bb/fetch2/osc.py | 165 -
bitbake/lib/bb/filter.py | 142 +
bitbake/lib/bb/main.py | 8 +-
bitbake/lib/bb/parse/__init__.py | 49 +-
bitbake/lib/bb/parse/ast.py | 201 +-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 53 +-
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 24 +-
bitbake/lib/bb/persist_data.py | 271 --
bitbake/lib/bb/pysh/pyshlex.py | 2 +-
bitbake/lib/bb/pysh/pyshyacc.py | 2 +-
bitbake/lib/bb/runqueue.py | 487 +-
bitbake/lib/bb/server/process.py | 86 +-
bitbake/lib/bb/server/xmlrpcserver.py | 11 +-
bitbake/lib/bb/siggen.py | 102 +-
bitbake/lib/bb/taskdata.py | 8 +-
bitbake/lib/bb/tests/codeparser.py | 40 +
bitbake/lib/bb/tests/compression.py | 4 +-
bitbake/lib/bb/tests/cooker.py | 342 +-
bitbake/lib/bb/tests/data.py | 207 +-
.../software/miniupnp/download.php | 3528 ++++++++++++++
bitbake/lib/bb/tests/fetch.py | 2583 +++++++---
bitbake/lib/bb/tests/fetch_import.py | 86 +
bitbake/lib/bb/tests/filter.py | 88 +
.../bb/tests/parse-tests/classes/base.bbclass | 5 +
.../classes/recipe-file-class.bbclass | 2 +
.../bb/tests/parse-tests/conf/bitbake.conf | 15 +
bitbake/lib/bb/tests/parse.py | 413 +-
bitbake/lib/bb/tests/persist_data.py | 129 -
.../tests/runqueue-tests/classes/base.bbclass | 4 +-
.../lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 +
.../lib/bb/tests/runqueue-tests/recipes/h1.bb | 0
bitbake/lib/bb/tests/runqueue.py | 20 +-
bitbake/lib/bb/tests/setup.py | 1089 +++++
bitbake/lib/bb/tests/siggen.py | 47 +
bitbake/lib/bb/tests/support/httpserver.py | 4 +-
bitbake/lib/bb/tests/utils.py | 19 +
bitbake/lib/bb/tinfoil.py | 227 +-
bitbake/lib/bb/ui/buildinfohelper.py | 5 +-
bitbake/lib/bb/ui/knotty.py | 203 +-
bitbake/lib/bb/ui/ncurses.py | 2 +-
bitbake/lib/bb/ui/taskexp_ncurses.py | 3 +
bitbake/lib/bb/ui/teamcity.py | 14 +-
bitbake/lib/bb/ui/uihelper.py | 12 +-
bitbake/lib/bb/utils.py | 867 +++-
bitbake/lib/bblayers/action.py | 4 +-
bitbake/lib/bblayers/layerindex.py | 12 +-
bitbake/lib/bblayers/query.py | 37 +-
bitbake/lib/bs4/AUTHORS.txt | 43 -
bitbake/lib/bs4/NEWS.txt | 1066 ----
bitbake/lib/bs4/__init__.py | 468 --
bitbake/lib/bs4/builder/__init__.py | 324 --
bitbake/lib/bs4/builder/_html5lib.py | 338 --
bitbake/lib/bs4/builder/_htmlparser.py | 262 -
bitbake/lib/bs4/builder/_lxml.py | 248 -
bitbake/lib/bs4/dammit.py | 832 ----
bitbake/lib/bs4/diagnose.py | 216 -
bitbake/lib/bs4/element.py | 1724 -------
bitbake/lib/bs4/testing.py | 686 ---
bitbake/lib/bs4/tests/__init__.py | 1 -
.../lib/bs4/tests/test_builder_registry.py | 147 -
bitbake/lib/bs4/tests/test_docs.py | 32 -
bitbake/lib/bs4/tests/test_html5lib.py | 98 -
bitbake/lib/bs4/tests/test_htmlparser.py | 31 -
bitbake/lib/bs4/tests/test_lxml.py | 70 -
bitbake/lib/bs4/tests/test_soup.py | 479 --
bitbake/lib/bs4/tests/test_tree.py | 2004 --------
bitbake/lib/codegen.py | 564 ---
bitbake/lib/hashserv/__init__.py | 15 +-
bitbake/lib/hashserv/client.py | 423 +-
bitbake/lib/hashserv/server.py | 257 +-
bitbake/lib/hashserv/sqlite.py | 17 +
bitbake/lib/hashserv/tests.py | 589 ++-
bitbake/lib/layerindexlib/__init__.py | 6 +-
bitbake/lib/layerindexlib/cooker.py | 16 +-
bitbake/lib/layerindexlib/restapi.py | 2 +-
bitbake/lib/ply/lex.py | 1054 ----
bitbake/lib/prserv/__init__.py | 97 +-
bitbake/lib/prserv/client.py | 15 +-
bitbake/lib/prserv/db.py | 452 +-
bitbake/lib/prserv/serv.py | 140 +-
bitbake/lib/prserv/tests.py | 388 ++
bitbake/lib/pyinotify.py | 2326 ---------
bitbake/lib/toaster/bldcollector/admin.py | 8 +-
bitbake/lib/toaster/bldcontrol/admin.py | 2 +-
.../bldcontrol/localhostbecontroller.py | 175 +-
.../toaster/orm/fixtures/check_fixtures.py | 76 +
.../lib/toaster/orm/fixtures/gen_fixtures.py | 137 +-
bitbake/lib/toaster/orm/fixtures/oe-core.xml | 72 +-
bitbake/lib/toaster/orm/fixtures/poky.xml | 136 +-
.../orm/management/commands/lsupdates.py | 29 +-
bitbake/lib/toaster/orm/models.py | 3 +-
.../tests/browser/selenium_helpers_base.py | 75 +-
.../tests/browser/test_all_builds_page.py | 7 +-
.../tests/browser/test_all_projects_page.py | 12 +-
.../tests/browser/test_builddashboard_page.py | 2 +-
.../tests/browser/test_landing_page.py | 14 +-
.../tests/browser/test_layerdetails_page.py | 65 +-
.../browser/test_new_custom_image_page.py | 2 +-
.../tests/browser/test_new_project_page.py | 12 +-
bitbake/lib/toaster/tests/builds/buildtest.py | 2 +-
.../tests/functional/functional_helpers.py | 88 +
.../functional/test_create_new_project.py | 121 +-
.../tests/functional/test_functional_basic.py | 200 +-
.../tests/functional/test_project_config.py | 99 +-
.../tests/functional/test_project_page.py | 417 +-
.../test_project_page_tab_config.py | 313 +-
bitbake/lib/toaster/tests/functional/utils.py | 21 +-
.../tests/toaster-tests-requirements.txt | 4 +-
bitbake/lib/toaster/tests/views/test_views.py | 2 +-
.../toastergui/templates/layerdetails.html | 8 +-
.../toastergui/templatetags/projecttags.py | 1 -
bitbake/lib/toaster/toastergui/views.py | 18 +-
bitbake/lib/toaster/toastermain/settings.py | 5 +-
bitbake/packaging-pypi/bitbake-setup/BUILD.md | 67 +
bitbake/packaging-pypi/bitbake-setup/LICENSE | 23 +
.../packaging-pypi/bitbake-setup/README.md | 42 +
.../bitbake-setup/package-bitbake-setup.py | 79 +
.../bitbake-setup/pyproject.toml | 92 +
bitbake/pyproject.toml | 16 +
.../setup-schema/bitbake-setup.schema.json | 211 +
bitbake/setup-schema/layers.schema.json | 107 +
bitbake/toaster-requirements.txt | 2 +-
bitbake/vendor.txt | 6 +
bitbake/vendor/licenses/ply/LICENSE | 32 +
...0010-lib-bs4-Avoid-soupsieve-warning.patch | 40 +
.../patches/bs4-remove-double-imports.patch | 23 +
...mplement-basic-task-progress-support.patch | 125 +
...ial-pass-of-SPDX-license-headers-to-.patch | 79 +
...bar-accept-value-over-initial-maxval.patch | 54 +
...self._fd_console-to-use-for-self._ha.patch | 53 +
...ke-bars-show-correctly-with-maxval-0.patch | 31 +
...ty-Allow-mixing-log-messages-and-pro.patch | 31 +
doc/technical_overview.md | 17 +
.../installer-add-rootfs.bbclass | 2 +-
.../recipes-app/hello-isar/hello-isar.bb | 2 +-
meta-isar/recipes-app/libhello/libhello.bb | 2 +-
.../bmap-tools/bmap-tools.bb | 2 +-
meta/classes-global/base.bbclass | 39 +-
meta/classes-global/isar-events.bbclass | 48 +-
meta/classes-global/sstate.bbclass | 14 +-
meta/classes-recipe/bootstrap.bbclass | 4 +-
meta/classes-recipe/dpkg-base.bbclass | 6 +-
meta/classes-recipe/dpkg.bbclass | 4 +-
.../image-tools-extension.bbclass | 17 +-
meta/classes-recipe/imagetypes_wic.bbclass | 43 +-
meta/classes-recipe/rootfs.bbclass | 4 +-
meta/classes-recipe/sbuild.bbclass | 2 +-
meta/conf/bitbake.conf | 8 +-
meta/lib/aptsrc_fetcher.py | 8 +-
meta/lib/container_fetcher.py | 12 +-
meta/lib/oe/patch.py | 10 +-
meta/lib/oe/path.py | 18 +-
.../libubootenv/libubootenv_0.3.5-0.1.bb | 2 +-
meta/recipes-devtools/base-apt/base-apt.bb | 2 +-
.../source/bootimg-biosplusefi-isar.py | 1 +
.../wic/plugins/source/bootimg-pcbios-isar.py | 6 +-
.../plugins/source/isoimage-isohybrid-isar.py | 4 +-
.../lib/wic/plugins/source/rootfs-u-boot.py | 12 +-
scripts/lib/wic/misc.py | 38 +-
scripts/lib/wic/partition.py | 5 +-
scripts/lib/wic/plugins/imager/direct.py | 9 +-
.../lib/wic/plugins/source/bootimg-pcbios.py | 6 +-
.../wic/plugins/source/isoimage-isohybrid.py | 4 +-
scripts/lib/wic/plugins/source/rawcopy.py | 20 +-
scripts/lib/wic/plugins/source/rootfs.py | 17 +-
scripts/wic-chroot | 18 +
testsuite/unittests/test_build_uuid.py | 107 +
311 files changed, 41940 insertions(+), 21357 deletions(-)
create mode 100644 bitbake/.b4-config
create mode 100644 bitbake/AGENTS.md
create mode 120000 bitbake/bin/bitbake-config-build
create mode 100755 bitbake/bin/bitbake-setup
delete mode 100755 bitbake/bin/git-make-shallow
create mode 100755 bitbake/contrib/b4-wrapper-bitbake.py
delete mode 100644 bitbake/contrib/vim/LICENSE.txt
delete mode 100644 bitbake/contrib/vim/ftdetect/bitbake.vim
delete mode 100644 bitbake/contrib/vim/ftplugin/bitbake.vim
delete mode 100644 bitbake/contrib/vim/indent/bitbake.vim
delete mode 100644 bitbake/contrib/vim/plugin/newbb.vim
delete mode 100644 bitbake/contrib/vim/plugin/newbbappend.vim
delete mode 100644 bitbake/contrib/vim/syntax/bitbake.vim
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-master.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-blacksail.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-master.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-whinlatter.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-wrynose.conf.json
create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst
delete mode 100644 bitbake/doc/releases.rst
create mode 100755 bitbake/doc/setversions.py
delete mode 100644 bitbake/doc/sphinx-static/switchers.js
create mode 100644 bitbake/doc/sphinx-static/switchers.js.in
rename bitbake/lib/{bs4/COPYING.txt => bb/_vendor/beautifulsoup4.LICENSE} (80%)
create mode 100644 bitbake/lib/bb/_vendor/bs4/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_deprecation.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_typing.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_warnings.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_lxml.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/css.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/dammit.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/diagnose.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/element.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/exceptions.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/filter.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/formatter.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/py.typed
create mode 100644 bitbake/lib/bb/_vendor/ply.pyi
rename bitbake/lib/{ => bb/_vendor}/ply/__init__.py (80%)
create mode 100644 bitbake/lib/bb/_vendor/ply/cpp.py
create mode 100644 bitbake/lib/bb/_vendor/ply/ctokens.py
create mode 100644 bitbake/lib/bb/_vendor/ply/lex.py
rename bitbake/lib/{ => bb/_vendor}/ply/yacc.py (60%)
create mode 100644 bitbake/lib/bb/_vendor/ply/ygen.py
create mode 100644 bitbake/lib/bb/_vendor/progressbar.pyi
rename bitbake/lib/{ => bb/_vendor}/progressbar/LICENSE.txt (100%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/__init__.py (96%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/compat.py (100%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/progressbar.py (88%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/widgets.py (97%)
create mode 100644 bitbake/lib/bb/_vendor/simplediff.pyi
rename bitbake/lib/{ => bb/_vendor}/simplediff/LICENSE (100%)
rename bitbake/lib/{ => bb/_vendor}/simplediff/__init__.py (99%)
create mode 100644 bitbake/lib/bb/_vendor/tomli/LICENSE
create mode 100644 bitbake/lib/bb/_vendor/tomli/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_parser.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_re.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_types.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/py.typed
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.LICENSE
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.py
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.pyi
create mode 100644 bitbake/lib/bb/asyncrpc/taskgroup.py
rename bitbake/lib/bb/{fetch2 => fetch}/README (100%)
rename bitbake/lib/bb/{fetch2 => fetch}/__init__.py (84%)
rename bitbake/lib/bb/{fetch2 => fetch}/az.py (78%)
rename bitbake/lib/bb/{fetch2 => fetch}/clearcase.py (84%)
create mode 100644 bitbake/lib/bb/fetch/crate.py
rename bitbake/lib/bb/{fetch2 => fetch}/gcp.py (62%)
rename bitbake/lib/bb/{fetch2 => fetch}/git.py (52%)
rename bitbake/lib/bb/{fetch2 => fetch}/gitannex.py (65%)
rename bitbake/lib/bb/{fetch2 => fetch}/gitsm.py (63%)
create mode 100644 bitbake/lib/bb/fetch/gomod.py
rename bitbake/lib/bb/{fetch2 => fetch}/hg.py (72%)
rename bitbake/lib/bb/{fetch2 => fetch}/local.py (83%)
rename bitbake/lib/bb/{fetch2 => fetch}/npm.py (62%)
rename bitbake/lib/bb/{fetch2 => fetch}/npmsw.py (70%)
rename bitbake/lib/bb/{fetch2 => fetch}/perforce.py (87%)
rename bitbake/lib/bb/{fetch2 => fetch}/repo.py (68%)
rename bitbake/lib/bb/{fetch2 => fetch}/s3.py (87%)
rename bitbake/lib/bb/{fetch2 => fetch}/sftp.py (85%)
rename bitbake/lib/bb/{fetch2 => fetch}/ssh.py (85%)
rename bitbake/lib/bb/{fetch2 => fetch}/svn.py (65%)
rename bitbake/lib/bb/{fetch2 => fetch}/wget.py (68%)
create mode 100644 bitbake/lib/bb/fetch2.py
delete mode 100644 bitbake/lib/bb/fetch2/bzr.py
delete mode 100644 bitbake/lib/bb/fetch2/crate.py
delete mode 100644 bitbake/lib/bb/fetch2/cvs.py
delete mode 100644 bitbake/lib/bb/fetch2/osc.py
create mode 100644 bitbake/lib/bb/filter.py
delete mode 100644 bitbake/lib/bb/persist_data.py
create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php
create mode 100644 bitbake/lib/bb/tests/fetch_import.py
create mode 100644 bitbake/lib/bb/tests/filter.py
create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/base.bbclass
create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass
create mode 100644 bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf
delete mode 100644 bitbake/lib/bb/tests/persist_data.py
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
create mode 100644 bitbake/lib/bb/tests/setup.py
delete mode 100644 bitbake/lib/bs4/AUTHORS.txt
delete mode 100644 bitbake/lib/bs4/NEWS.txt
delete mode 100644 bitbake/lib/bs4/__init__.py
delete mode 100644 bitbake/lib/bs4/builder/__init__.py
delete mode 100644 bitbake/lib/bs4/builder/_html5lib.py
delete mode 100644 bitbake/lib/bs4/builder/_htmlparser.py
delete mode 100644 bitbake/lib/bs4/builder/_lxml.py
delete mode 100644 bitbake/lib/bs4/dammit.py
delete mode 100644 bitbake/lib/bs4/diagnose.py
delete mode 100644 bitbake/lib/bs4/element.py
delete mode 100644 bitbake/lib/bs4/testing.py
delete mode 100644 bitbake/lib/bs4/tests/__init__.py
delete mode 100644 bitbake/lib/bs4/tests/test_builder_registry.py
delete mode 100644 bitbake/lib/bs4/tests/test_docs.py
delete mode 100644 bitbake/lib/bs4/tests/test_html5lib.py
delete mode 100644 bitbake/lib/bs4/tests/test_htmlparser.py
delete mode 100644 bitbake/lib/bs4/tests/test_lxml.py
delete mode 100644 bitbake/lib/bs4/tests/test_soup.py
delete mode 100644 bitbake/lib/bs4/tests/test_tree.py
delete mode 100644 bitbake/lib/codegen.py
delete mode 100644 bitbake/lib/ply/lex.py
create mode 100644 bitbake/lib/prserv/tests.py
delete mode 100644 bitbake/lib/pyinotify.py
create mode 100755 bitbake/lib/toaster/orm/fixtures/check_fixtures.py
create mode 100644 bitbake/packaging-pypi/bitbake-setup/BUILD.md
create mode 100644 bitbake/packaging-pypi/bitbake-setup/LICENSE
create mode 100644 bitbake/packaging-pypi/bitbake-setup/README.md
create mode 100755 bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
create mode 100644 bitbake/packaging-pypi/bitbake-setup/pyproject.toml
create mode 100644 bitbake/pyproject.toml
create mode 100644 bitbake/setup-schema/bitbake-setup.schema.json
create mode 100644 bitbake/setup-schema/layers.schema.json
create mode 100644 bitbake/vendor.txt
create mode 100644 bitbake/vendor/licenses/ply/LICENSE
create mode 100644 bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch
create mode 100644 bitbake/vendor/patches/bs4-remove-double-imports.patch
create mode 100644 bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch
create mode 100644 bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch
create mode 100644 bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch
create mode 100644 bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch
create mode 100644 bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch
create mode 100644 bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch
create mode 100755 scripts/wic-chroot
create mode 100644 testsuite/unittests/test_build_uuid.py
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-1-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/8] base: replace BitBake's outdated persist_data
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 2/8] isar-events: preserve cleanup for default multiconfig 'Clara Kowalsky' via isar-users
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
Isar uses a persistent UUID to distinguish build-specific resources,
including sbuild chroot names. The UUID must remain stable throughout a
BitBake run, so parallel builds using separate PID namespaces do not
clash on external mountpoints. It must also be shared across
multiconfigs and cleared when the build completes.
Isar currently stores the UUID through bb.persist_data.persist, which
is removed in BitBake v2.9.2 commit 681a7516 ("persist_data: Remove
it") from October 2024.
Persist the UUID in a file under PERSISTENT_DIR instead. Use BitBake's
file-locking helper when reading, creating and clearing it to ensure
concurrent processes share the same UUID. Clear it on the default
BuildCompleted event, so the next run receives a fresh UUID.
Add unittests for UUID reuse across calls and multiconfigs, cleanup and
concurrent access.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta/classes-global/base.bbclass | 25 +++++--
meta/classes-global/isar-events.bbclass | 2 +-
testsuite/unittests/test_build_uuid.py | 89 +++++++++++++++++++++++++
3 files changed, 109 insertions(+), 7 deletions(-)
create mode 100644 testsuite/unittests/test_build_uuid.py
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 3d5d1c12..c61e59b4 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -341,12 +341,25 @@ def base_set_filespath(path, d):
def calculate_build_uuid(d):
import uuid
- uuid_data = bb.persist_data.persist('BB_ISAR_UUID_DATA', d)
- if "uuid" not in uuid_data or not uuid_data["uuid"]:
- # Generate new UUID
- uuid_data["uuid"] = str(uuid.uuid4())
-
- return uuid_data["uuid"]
+ uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
+ with bb.utils.fileslocked([uuid_file + '.lock']):
+ try:
+ with open(uuid_file) as file:
+ build_uuid = file.read().strip()
+ except FileNotFoundError:
+ build_uuid = ''
+
+ if not build_uuid:
+ build_uuid = str(uuid.uuid4())
+ with open(uuid_file, 'w') as file:
+ file.write(build_uuid + '\n')
+
+ return build_uuid
+
+def clear_build_uuid(d):
+ uuid_file = os.path.join(d.getVar('PERSISTENT_DIR'), 'isar-build-uuid')
+ with bb.utils.fileslocked([uuid_file + '.lock']):
+ bb.utils.remove(uuid_file)
# Unique ID for this build, used to avoid name clashes on external mountpoints
# When running parallel builds in different PID namespaces
diff --git a/meta/classes-global/isar-events.bbclass b/meta/classes-global/isar-events.bbclass
index 15bfdb99..cfe6d7dd 100644
--- a/meta/classes-global/isar-events.bbclass
+++ b/meta/classes-global/isar-events.bbclass
@@ -85,7 +85,7 @@ python build_completed() {
bb.error(str(e))
# Cleanup build UUID, the next bitbake run will generate new one
- bb.persist_data.persist('BB_ISAR_UUID_DATA', d).clear()
+ bb.utils.exec_flat_python_func('clear_build_uuid', d)
}
build_completed[eventmask] = "bb.event.BuildCompleted"
diff --git a/testsuite/unittests/test_build_uuid.py b/testsuite/unittests/test_build_uuid.py
new file mode 100644
index 00000000..374cce64
--- /dev/null
+++ b/testsuite/unittests/test_build_uuid.py
@@ -0,0 +1,89 @@
+# This software is a part of Isar.
+# Copyright (C) Siemens AG, 2026
+#
+# SPDX-License-Identifier: MIT
+
+import multiprocessing
+import os
+import sys
+import tempfile
+import unittest
+import uuid
+from pathlib import Path
+
+root = Path(__file__).resolve().parents[2]
+sys.path.extend([
+ str(root / 'bitbake/lib'),
+ str(root / 'meta/lib'),
+])
+
+import bb
+import bb.data
+import bb.utils
+from bb.parse import handle
+
+
+data = bb.data.init()
+data.setVar('BBPATH', f'{root}/meta:{root}/bitbake')
+data.setVar('__bbclasstype', 'global')
+parsed = handle(str(root / 'meta/classes-global/base.bbclass'), data)
+namespace = {'bb': bb, 'os': os}
+exec(parsed['calculate_build_uuid'], namespace)
+exec(parsed['clear_build_uuid'], namespace)
+calculate_build_uuid = namespace['calculate_build_uuid']
+clear_build_uuid = namespace['clear_build_uuid']
+
+
+def calculate_in_worker(persistent_dir, start, results):
+ data = bb.data.init()
+ data.setVar('PERSISTENT_DIR', persistent_dir)
+ start.wait()
+ results.put(calculate_build_uuid(data))
+
+
+class TestBuildUUID(unittest.TestCase):
+ def setUp(self):
+ directory = tempfile.TemporaryDirectory()
+ self.addCleanup(directory.cleanup)
+ self.persistent_dir = os.path.join(directory.name, 'cache')
+ self.data = bb.data.init()
+ self.data.setVar('PERSISTENT_DIR', self.persistent_dir)
+
+ def test_uuid_is_reused_across_calls_and_multiconfigs(self):
+ build_uuid = calculate_build_uuid(self.data)
+ self.assertEqual(str(uuid.UUID(build_uuid)), build_uuid)
+ self.assertEqual(calculate_build_uuid(self.data), build_uuid)
+ other_config = self.data.createCopy()
+ other_config.setVar('BB_CURRENT_MC', 'other')
+ self.assertEqual(calculate_build_uuid(other_config), build_uuid)
+ clear_build_uuid(self.data)
+ self.assertNotEqual(calculate_build_uuid(self.data), build_uuid)
+
+ def test_concurrent_workers_share_uuid(self):
+ context = multiprocessing.get_context('fork')
+ start = context.Event()
+ results = context.Queue()
+ workers = [
+ context.Process(
+ target=calculate_in_worker,
+ args=(self.persistent_dir, start, results),
+ )
+ for _ in range(8)
+ ]
+ try:
+ for worker in workers:
+ worker.start()
+ start.set()
+ values = [results.get(timeout=15) for worker in workers]
+ for worker in workers:
+ worker.join(timeout=15)
+ self.assertEqual(worker.exitcode, 0)
+ self.assertEqual(len(set(values)), 1)
+ self.assertEqual(values[0], calculate_build_uuid(self.data))
+ finally:
+ for worker in workers:
+ if worker.is_alive():
+ worker.terminate()
+ worker.join(timeout=15)
+ results.close()
+ results.join_thread()
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-2-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/8] isar-events: preserve cleanup for default multiconfig
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 3/8] isar-events: close /proc/mounts after cleanup 'Clara Kowalsky' via isar-users
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
BitBake v2.9.2 includes commit 0fa0d8d764bb ("cooker/cache: Drop mc
'default' string value") from January 2025, which changes BB_CURRENT_MC
for the default multiconfig from "default" to "".
The build completion handler checks for "default" before running
build-wide cleanup. With the new value, it returns early, leaving mounts
behind and the build UUID uncleared.
Fix this by accepting both the empty string and the legacy "default"
value, which can be dropped once BitBake is updated. Add unittests for
both default values and a named multiconfig.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta/classes-global/isar-events.bbclass | 9 +++------
testsuite/unittests/test_build_uuid.py | 18 ++++++++++++++++++
2 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/meta/classes-global/isar-events.bbclass b/meta/classes-global/isar-events.bbclass
index cfe6d7dd..9b66796e 100644
--- a/meta/classes-global/isar-events.bbclass
+++ b/meta/classes-global/isar-events.bbclass
@@ -52,13 +52,10 @@ python build_completed() {
if not tmpdir:
return
- # bitbake calls cleanup for every multiconfig listed in BBMULTICONFIG plus
- # one for the entire build. E.g., if BBMULTICONFIG="mc1 mc2 mc3", we call
- # "bitbake mc1 mc2", the following cleanups would be called:
- # "c1 c2 c3 cdefault".
- # Skip running cleanup for additional multiconfigs
+ # Run cleanup only for the default multiconfig, which is empty on
+ # newer BitBake.
mc = d.getVar('BB_CURRENT_MC')
- if mc != 'default':
+ if mc not in ('', 'default'):
return
fail_on_cleanup = bb.utils.to_boolean(d.getVar('ISAR_FAIL_ON_CLEANUP'))
diff --git a/testsuite/unittests/test_build_uuid.py b/testsuite/unittests/test_build_uuid.py
index 374cce64..5d364ed0 100644
--- a/testsuite/unittests/test_build_uuid.py
+++ b/testsuite/unittests/test_build_uuid.py
@@ -18,6 +18,7 @@ sys.path.extend([
])
import bb
+import bb.build
import bb.data
import bb.utils
from bb.parse import handle
@@ -59,6 +60,23 @@ class TestBuildUUID(unittest.TestCase):
clear_build_uuid(self.data)
self.assertNotEqual(calculate_build_uuid(self.data), build_uuid)
+ def test_completion_resets_only_default_multiconfig(self):
+ data = handle(
+ str(root / 'meta/classes-global/isar-events.bbclass'), self.data
+ )
+ data.setVar('TMPDIR', self.persistent_dir)
+ data.setVar('T', os.path.join(self.persistent_dir, 'temp'))
+ for multiconfig, resets_uuid in [
+ ('', True), ('default', True), ('other', False)
+ ]:
+ with self.subTest(multiconfig=multiconfig):
+ build_uuid = calculate_build_uuid(data)
+ data.setVar('BB_CURRENT_MC', multiconfig)
+ bb.build.exec_func('build_completed', data)
+ self.assertEqual(
+ calculate_build_uuid(data) != build_uuid, resets_uuid
+ )
+
def test_concurrent_workers_share_uuid(self):
context = multiprocessing.get_context('fork')
start = context.Event()
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-3-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/8] isar-events: close /proc/mounts after cleanup
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 2/8] isar-events: preserve cleanup for default multiconfig 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 4/8] meta: migrate fetcher references to bb.fetch 'Clara Kowalsky' via isar-users
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
Running the UUID unittest:
python3 -m unittest discover -s testsuite/unittests \
-p test_build_uuid.py -v
emits the warning:
/isar/meta/classes-global/isar-events.bbclass:65:
ResourceWarning: unclosed file <_io.TextIOWrapper name='/proc/mounts'
mode='r' encoding='UTF-8'>
The test_completion_resets_only_default_multiconfig exercises the
BuildCompleted handler, which reads /proc/mounts using open() without
closing the file. Fix this by reading the mount entries within a context
manager for deterministic file closure.
Fixes: e596cdf97198 ("isar-events: Unhide mounts left behind")
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta/classes-global/isar-events.bbclass | 37 +++++++++++++------------
1 file changed, 19 insertions(+), 18 deletions(-)
diff --git a/meta/classes-global/isar-events.bbclass b/meta/classes-global/isar-events.bbclass
index 9b66796e..48f5f441 100644
--- a/meta/classes-global/isar-events.bbclass
+++ b/meta/classes-global/isar-events.bbclass
@@ -62,24 +62,25 @@ python build_completed() {
basepath = tmpdir + '/work/'
- for line in reversed(list(open('/proc/mounts'))):
- if basepath not in line:
- continue
- msg_line = f"{line.split()[1]} left mounted"
- # If bitbake is started manually, bb.warn and bb.error go to stdout;
- # with bb.error, bitbake additionally fails the build. Under CI,
- # bb.warn and bb.error currently go to a file.
- if fail_on_cleanup:
- bb.error(msg_line)
- else:
- msg_line += ', unmounting...'
- bb.warn(msg_line)
- try:
- subprocess.run(
- f"sudo umount {line.split()[1]}", shell=True, check=True
- )
- except subprocess.CalledProcessError as e:
- bb.error(str(e))
+ with open('/proc/mounts') as mounts:
+ for line in reversed(mounts.readlines()):
+ if basepath not in line:
+ continue
+ msg_line = f"{line.split()[1]} left mounted"
+ # If bitbake is started manually, bb.warn and bb.error go to stdout;
+ # with bb.error, bitbake additionally fails the build. Under CI,
+ # bb.warn and bb.error currently go to a file.
+ if fail_on_cleanup:
+ bb.error(msg_line)
+ else:
+ msg_line += ', unmounting...'
+ bb.warn(msg_line)
+ try:
+ subprocess.run(
+ f"sudo umount {line.split()[1]}", shell=True, check=True
+ )
+ except subprocess.CalledProcessError as e:
+ bb.error(str(e))
# Cleanup build UUID, the next bitbake run will generate new one
bb.utils.exec_flat_python_func('clear_build_uuid', d)
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-4-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 4/8] meta: migrate fetcher references to bb.fetch
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
` (2 preceding siblings ...)
2026-11-09 9:17 ` [PATCH 3/8] isar-events: close /proc/mounts after cleanup 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 5/8] wic: run the Python driver outside the imaging chroot 'Clara Kowalsky' via isar-users
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
BitBake v2.20 includes commit 34ea7cb48090 ("lib: rename fetch2 as
fetch; invert compat shim to make bb.fetch2 = bb.fetch"), which renames
the fetcher API to bb.fetch and keeps bb.fetch2 only for deprecated
compatibility.
In preparation for an update of BitBake in Isar, migrate to bb.fetch to
avoid relying on a deprecated API. The current BitBake v2.8 already
provides bb.fetch as an alias of bb.fetch2, so this change is a pure
rename.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta/classes-global/base.bbclass | 14 +++++++-------
meta/classes-global/sstate.bbclass | 14 +++++++-------
meta/classes-recipe/bootstrap.bbclass | 4 ++--
meta/classes-recipe/dpkg-base.bbclass | 6 +++---
meta/classes-recipe/dpkg.bbclass | 2 +-
meta/conf/bitbake.conf | 6 +++---
meta/lib/aptsrc_fetcher.py | 8 ++++----
meta/lib/container_fetcher.py | 12 ++++++------
meta/lib/oe/patch.py | 10 +++++-----
meta/recipes-devtools/base-apt/base-apt.bb | 2 +-
10 files changed, 39 insertions(+), 39 deletions(-)
diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index c61e59b4..54da1421 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -161,7 +161,7 @@ python() {
needsrcrev = True
if needsrcrev:
- d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
+ d.setVar("SRCPV", "${@bb.fetch.get_srcrev(d)}")
for e in d.keys():
flags = d.getVarFlags(e)
@@ -244,9 +244,9 @@ python do_fetch() {
return
try:
- fetcher = bb.fetch2.Fetch(src_uri, d)
+ fetcher = bb.fetch.Fetch(src_uri, d)
fetcher.download()
- except bb.fetch2.BBFetchException as e:
+ except bb.fetch.BBFetchException as e:
bb.fatal(str(e))
}
@@ -263,9 +263,9 @@ python do_unpack() {
rootdir = d.getVar('WORKDIR')
try:
- fetcher = bb.fetch2.Fetch(src_uri, d)
+ fetcher = bb.fetch.Fetch(src_uri, d)
fetcher.unpack(rootdir)
- except bb.fetch2.BBFetchException as e:
+ except bb.fetch.BBFetchException as e:
bb.fatal(str(e))
}
@@ -314,9 +314,9 @@ python do_cleanall() {
return
try:
- fetcher = bb.fetch2.Fetch(src_uri, d)
+ fetcher = bb.fetch.Fetch(src_uri, d)
fetcher.clean()
- except bb.fetch2.BBFetchException as e:
+ except bb.fetch.BBFetchException as e:
bb.fatal(str(e))
}
diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass
index 76a7b596..4c50f7dd 100644
--- a/meta/classes-global/sstate.bbclass
+++ b/meta/classes-global/sstate.bbclass
@@ -731,7 +731,7 @@ def sstate_package(ss, d):
sstate_package[vardepsexclude] += "SSTATE_SIG_KEY"
def pstaging_fetch(sstatefetch, d):
- import bb.fetch2
+ import bb.fetch
# Only try and fetch if the user has configured a mirror
mirrors = d.getVar('SSTATE_MIRRORS')
@@ -766,11 +766,11 @@ def pstaging_fetch(sstatefetch, d):
localdata.delVar('SRC_URI')
localdata.setVar('SRC_URI', srcuri)
try:
- fetcher = bb.fetch2.Fetch([srcuri], localdata, cache=False)
+ fetcher = bb.fetch.Fetch([srcuri], localdata, cache=False)
fetcher.checkstatus()
fetcher.download()
- except bb.fetch2.BBFetchException:
+ except bb.fetch.BBFetchException:
pass
def sstate_setscene(d):
@@ -1017,7 +1017,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
bb.utils.to_boolean(localdata.getVar('SSTATE_MIRROR_ALLOW_NETWORK')):
localdata.delVar('BB_NO_NETWORK')
- from bb.fetch2 import FetchConnectionCache
+ from bb.fetch import FetchConnectionCache
def checkstatus_init():
while not connection_cache_pool.full():
connection_cache_pool.put(FetchConnectionCache())
@@ -1039,13 +1039,13 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
import traceback
try:
- fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2,
+ fetcher = bb.fetch.Fetch(srcuri.split(), localdata2,
connection_cache=connection_cache)
fetcher.checkstatus()
bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
found.add(tid)
missed.remove(tid)
- except bb.fetch2.FetchError as e:
+ except bb.fetch.FetchError as e:
bb.debug(2, "SState: Unsuccessful fetch test for %s (%s)\n%s" % (srcuri, repr(e), traceback.format_exc()))
except Exception as e:
bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc()))
@@ -1072,7 +1072,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d)
# Have to setup the fetcher environment here rather than in each thread as it would race
- fetcherenv = bb.fetch2.get_fetcher_environment(d)
+ fetcherenv = bb.fetch.get_fetcher_environment(d)
with bb.utils.environment(**fetcherenv):
bb.event.enable_threadlock()
import concurrent.futures
diff --git a/meta/classes-recipe/bootstrap.bbclass b/meta/classes-recipe/bootstrap.bbclass
index c1a59fd2..2671602f 100644
--- a/meta/classes-recipe/bootstrap.bbclass
+++ b/meta/classes-recipe/bootstrap.bbclass
@@ -45,12 +45,12 @@ python () {
distro_bootstrap_keys += own_pub_key.split()
for key in distro_bootstrap_keys:
- fetcher = bb.fetch2.Fetch([key], d)
+ fetcher = bb.fetch.Fetch([key], d)
filename = os.path.relpath(fetcher.localpath(key), topdir)
d.appendVar("DISTRO_BOOTSTRAP_KEYFILES", " ${TOPDIR}/%s" % filename)
for key in third_party_apt_keys:
- fetcher = bb.fetch2.Fetch([key], d)
+ fetcher = bb.fetch.Fetch([key], d)
filename = os.path.relpath(fetcher.localpath(key), topdir)
d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename)
diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index 4ec0ddbe..a237af8f 100644
--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -49,7 +49,7 @@ python do_adjust_git() {
for src_uri in (d.getVar("SRC_URI") or "").split():
try:
- fetcher = bb.fetch2.Fetch([src_uri], d)
+ fetcher = bb.fetch.Fetch([src_uri], d)
ud = fetcher.ud[src_uri]
if ud.type != 'git':
continue
@@ -83,7 +83,7 @@ python do_adjust_git() {
bb.note(' '.join(cmd))
if subprocess.call(cmd) != 0:
bb.fatal("git alternates adjustment failed")
- except bb.fetch2.BBFetchException as e:
+ except bb.fetch.BBFetchException as e:
bb.fatal(str(e))
}
@@ -94,7 +94,7 @@ inherit patch
addtask patch after do_adjust_git
python() {
- from bb.fetch2 import methods
+ from bb.fetch import methods
# apt-src fetcher
import aptsrc_fetcher
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index 35408495..737915b8 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -75,7 +75,7 @@ dpkg_runbuild() {
[ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
# Don't warn environment variables exported to the bitbake fetcher
- case " ${@" ".join(bb.fetch2.FETCH_EXPORT_VARS)} " in
+ case " ${@" ".join(bb.fetch.FETCH_EXPORT_VARS)} " in
*" ${var} "*) continue;;
esac
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index c31719b8..4ffe6d14 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -37,10 +37,10 @@ PR = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
PROVIDES = ""
S = "${WORKDIR}/${P}"
SRCREV ??= "INVALID"
-AUTOREV = "${@bb.fetch2.get_autorev(d)}"
+AUTOREV = "${@bb.fetch.get_autorev(d)}"
AUTOREV[vardepvalue] = "${SRCPV}"
# Set Dynamically in base.bbclass
-# SRCPV = "${@bb.fetch2.get_srcrev(d)}"
+# SRCPV = "${@bb.fetch.get_srcrev(d)}"
SRCPV[vardepvalue] = "${SRCPV}"
SRC_URI = ""
STAMPS_DIR ?= "${TMPDIR}/stamps"
@@ -50,7 +50,7 @@ SVNDIR = "${DL_DIR}/svn"
T = "${WORKDIR}/temp"
TMPDIR = "${TOPDIR}/tmp"
PERSISTENT_DIR = "${TMPDIR}/cache"
-GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
+GITPKGV = "${@bb.fetch.get_srcrev(d, 'gitpkgv_revision')}"
# Isar specific config
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
diff --git a/meta/lib/aptsrc_fetcher.py b/meta/lib/aptsrc_fetcher.py
index fe8f1cd3..8de86fce 100644
--- a/meta/lib/aptsrc_fetcher.py
+++ b/meta/lib/aptsrc_fetcher.py
@@ -3,10 +3,10 @@
#
# SPDX-License-Identifier: MIT
-from bb.fetch2 import FetchError
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import logger
-from bb.fetch2 import runfetchcmd
+from bb.fetch import FetchError
+from bb.fetch import FetchMethod
+from bb.fetch import logger
+from bb.fetch import runfetchcmd
import os
class AptSrc(FetchMethod):
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
index 52a75ac9..7f9bfdbe 100644
--- a/meta/lib/container_fetcher.py
+++ b/meta/lib/container_fetcher.py
@@ -7,12 +7,12 @@ import oe.path
import os
import tempfile
import json
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import logger
-from bb.fetch2 import MissingChecksumEvent
-from bb.fetch2 import NoChecksumError
-from bb.fetch2 import ChecksumError
-from bb.fetch2 import runfetchcmd
+from bb.fetch import FetchMethod
+from bb.fetch import logger
+from bb.fetch import MissingChecksumEvent
+from bb.fetch import NoChecksumError
+from bb.fetch import ChecksumError
+from bb.fetch import runfetchcmd
class Container(FetchMethod):
def supports(self, ud, d):
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 52c0aa4d..a4dda29b 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -92,14 +92,14 @@ class PatchSet(object):
if not patch.get("remote"):
raise PatchError("Patch file must be specified in patch import.")
else:
- patch["file"] = bb.fetch2.localpath(patch["remote"], self.d)
+ patch["file"] = bb.fetch.localpath(patch["remote"], self.d)
for param in PatchSet.defaults:
if not patch.get(param):
patch[param] = PatchSet.defaults[param]
if patch.get("remote"):
- patch["file"] = self.d.expand(bb.fetch2.localpath(patch["remote"], self.d))
+ patch["file"] = self.d.expand(bb.fetch.localpath(patch["remote"], self.d))
patch["filemd5"] = bb.utils.md5_file(patch["file"])
@@ -560,7 +560,7 @@ class GitApplyTree(PatchTree):
shutil.rmtree(tempdir)
def _need_dirty_check(self):
- fetch = bb.fetch2.Fetch([], self.d)
+ fetch = bb.fetch.Fetch([], self.d)
check_dirtyness = False
for url in fetch.urls:
url_data = fetch.ud[url]
@@ -778,7 +778,7 @@ class QuiltTree(PatchSet):
if type == "file":
import shutil
if not patch.get("file") and patch.get("remote"):
- patch["file"] = bb.fetch2.localpath(patch["remote"], self.d)
+ patch["file"] = bb.fetch.localpath(patch["remote"], self.d)
shutil.copyfile(patch["quiltfile"], patch["file"])
else:
@@ -908,7 +908,7 @@ def patch_path(url, fetch, workdir, expand=True):
def src_patches(d, all=False, expand=True):
workdir = d.getVar('WORKDIR')
- fetch = bb.fetch2.Fetch([], d)
+ fetch = bb.fetch.Fetch([], d)
patches = []
sources = []
for url in fetch.urls:
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 97e3ae16..d54786ec 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -81,7 +81,7 @@ python do_cache() {
for key in d.getVar('BASE_REPO_KEY').split():
d.appendVar("SRC_URI", " %s" % key)
- fetcher = bb.fetch2.Fetch([key], d)
+ fetcher = bb.fetch.Fetch([key], d)
filename = fetcher.localpath(key)
d.appendVar("KEYFILES", " %s" % filename)
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-5-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 5/8] wic: run the Python driver outside the imaging chroot
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
` (3 preceding siblings ...)
2026-11-09 9:17 ` [PATCH 4/8] meta: migrate fetcher references to bb.fetch 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures 'Clara Kowalsky' via isar-users
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
WIC is a Python tool that imports BitBake (scripts/wic: "import bb"),
so it needs a Python interpreter that BitBake supports. Isar runs WIC
inside the imaging chroot, which has the userland of the target
distribution, so WIC is executed by the chroot's python3.
This blocks updating BitBake: BitBake 2.8 requires Python 3.8 and
BitBake 2.20 requires 3.9, but Debian buster ships Python 3.7. So far,
Isar worked around this by lowering BitBake's version check (commit
da17d920 "bitbake: Downgrade python requirements"). Carrying it on
would run BitBake 2.20 on an interpreter it does not support.
Fix this by running WIC with the build host's Python, which BitBake
itself already uses, and executing only the image utilities (e.g.,
mkfs, mtools, bmaptool) in the chroot. imager_run_host runs a script
outside the chroot with the same privilege handling and mounts as
imager_run and provides IMAGER_CHROOT. WIC uses WIC_TOOL_RUNNER
(scripts/wic-chroot) to execute utilities in the chroot and
WIC_TOOL_SYSROOT to translate native sysroot paths with
get_native_path(). The rootfs copy helpers in oe.path take an optional
command runner to use the same mechanism.
WIC is invoked as an external script, so its source files are not
included in do_image_wic's task signature automatically. Fix this by
listing the driver, runner and modified plugin files in the
file-checksums, so changes to them rerun the task instead of reusing a
stale image. The WIC work directory is now ${WORKDIR}/wic instead of
/tmp in the chroot.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
RECIPE-API-CHANGELOG.md | 17 ++++++++
doc/technical_overview.md | 17 ++++++++
.../image-tools-extension.bbclass | 17 +++++++-
meta/classes-recipe/imagetypes_wic.bbclass | 43 +++++++++++++++----
meta/lib/oe/path.py | 18 +++++---
.../source/bootimg-biosplusefi-isar.py | 1 +
.../wic/plugins/source/bootimg-efi-isar.py | 8 +++-
.../wic/plugins/source/bootimg-pcbios-isar.py | 6 ++-
.../plugins/source/isoimage-isohybrid-isar.py | 4 +-
.../lib/wic/plugins/source/rootfs-u-boot.py | 15 ++++---
scripts/lib/wic/misc.py | 38 +++++++++++++++-
scripts/lib/wic/partition.py | 5 ++-
scripts/lib/wic/plugins/imager/direct.py | 9 ++--
.../lib/wic/plugins/source/bootimg-pcbios.py | 6 ++-
.../wic/plugins/source/isoimage-isohybrid.py | 4 +-
scripts/lib/wic/plugins/source/rawcopy.py | 20 ++++-----
scripts/lib/wic/plugins/source/rootfs.py | 17 +++++++-
scripts/wic-chroot | 18 ++++++++
18 files changed, 215 insertions(+), 48 deletions(-)
create mode 100755 scripts/wic-chroot
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 3798a9d1..42641d9a 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1225,3 +1225,20 @@ the rootfs. This has been replaced by the `exclude-docs` rootfs feature. The
When enabled, the package changelogs are now removed as well (copyright is still
kept for legal reasons).
+
+### WIC runs on the build host, add imager_run_host
+
+WIC's Python driver now runs on the build host instead of inside the imaging
+chroot. Only the image utilities (e.g. mkfs tools, mtools, bmaptool) still run
+in the chroot. By that, the chroot distribution no longer needs a Python
+interpreter that is compatible with BitBake, which is required to update
+BitBake beyond Debian buster's Python version.
+
+Recipes can use the new `imager_run_host` instead of `imager_run` to run a
+script on the host. It provides the same privilege handling and mounts and
+sets `IMAGER_CHROOT` to the path of the prepared imaging chroot.
+
+Custom WIC plugins that start subprocesses directly must use `exec_cmd()` or
+`exec_native_cmd()` from `wic.misc` to still run their tools in the chroot.
+Native sysroot paths passed to such tools must be converted with
+`get_native_path()`. The WIC work directory is now `${WORKDIR}/wic`.
diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index 5d2e1b8a..d3d4e827 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -243,6 +243,23 @@ This process contains the following steps:
1.1. `IMAGE_CMD:ext4`: target filesystem is packed to extfs image.
1.2. `IMAGE_CMD:wic`: a bootable disk image gets created for the platform
+WIC runs with the build host's Python interpreter. Image utilities and their
+data remain in the imaging chroot, so older target distributions do not need
+to provide a Python interpreter compatible with BitBake.
+
+`imager_run_host` prepares the same tool environment as `imager_run`, but
+executes the supplied shell script outside the chroot. It still uses Isar's
+privilege handling and, in rootless mode, the mapped user and mount namespaces.
+`IMAGER_CHROOT` identifies the prepared chroot for the script.
+
+WIC uses `WIC_TOOL_RUNNER` to execute utility commands in that environment and
+`WIC_TOOL_SYSROOT` to locate native tool data from the host. Native paths are
+converted explicitly with `get_native_path` when constructing utility commands;
+command text and non-path arguments are not rewritten. Rootfs copy helpers use
+an optional command executor to route their utilities through the same runner,
+without changing the default behavior for other callers. Plugins that launch
+subprocesses directly must integrate with this runner to retain tool isolation.
+
## 3.8 Cross-compilation
Isar provides the possibility to cross-compile all or individual packages. If
diff --git a/meta/classes-recipe/image-tools-extension.bbclass b/meta/classes-recipe/image-tools-extension.bbclass
index 8f666444..9532fa47 100644
--- a/meta/classes-recipe/image-tools-extension.bbclass
+++ b/meta/classes-recipe/image-tools-extension.bbclass
@@ -33,6 +33,10 @@ imager_run() {
imager_run_${ISAR_CHROOT_MODE} "$@"
}
+imager_run_host() {
+ IMAGER_HOST_RUN=1 imager_run "$@"
+}
+
imager_run_schroot() {
local_install="${@(d.getVar("INSTALL_%s" % d.getVar("BB_CURRENTTASK")) or '').strip()}"
local_bom="${@(d.getVar("BOM_%s" % d.getVar("BB_CURRENTTASK")) or '').strip()}"
@@ -85,7 +89,11 @@ EOAPT
${local_install}"
fi
- schroot -r -c ${session_id} "$@"
+ if [ "${IMAGER_HOST_RUN:-0}" = "1" ]; then
+ IMAGER_CHROOT="${schroot_dir}" run_privileged_heredoc "$@"
+ else
+ schroot -r -c ${session_id} "$@"
+ fi
if [ -n "${local_bom}" ]; then
schroot -r -c ${session_id} -d / -- \
@@ -198,7 +206,12 @@ EOF
set -e
mkdir -p ${ROOTFS_IMAGETOOLS}/${SCRIPTSDIR}
${@insert_isar_mounts(d, d.getVar('ROOTFS_IMAGETOOLS'), d.getVar('IMAGER_UNSHARE_MOUNTS'))}
- chroot ${ROOTFS_IMAGETOOLS} "$@" <&3
+ if [ "${IMAGER_HOST_RUN:-0}" = "1" ]; then
+ export IMAGER_CHROOT="${ROOTFS_IMAGETOOLS}"
+ "$@" <&3
+ else
+ chroot ${ROOTFS_IMAGETOOLS} "$@" <&3
+ fi
EOF
if [ -n "${local_bom}" ]; then
diff --git a/meta/classes-recipe/imagetypes_wic.bbclass b/meta/classes-recipe/imagetypes_wic.bbclass
index 6cf7bd8e..25ff3095 100644
--- a/meta/classes-recipe/imagetypes_wic.bbclass
+++ b/meta/classes-recipe/imagetypes_wic.bbclass
@@ -144,7 +144,23 @@ check_for_wic_warnings() {
}
do_image_wic[depends] += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-sbom', 'sbom-chroot:do_build', '', d)}"
-do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
+do_image_wic[file-checksums] += " \
+ ${WKS_FILE_CHECKSUM} \
+ ${SCRIPTSDIR}/wic:True \
+ ${SCRIPTSDIR}/wic-chroot:True \
+ ${SCRIPTSDIR}/wic_fakeroot:True \
+ ${SCRIPTSDIR}/lib/wic/misc.py:True \
+ ${SCRIPTSDIR}/lib/wic/partition.py:True \
+ ${SCRIPTSDIR}/lib/wic/plugins/imager/direct.py:True \
+ ${SCRIPTSDIR}/lib/wic/plugins/source/bootimg-pcbios.py:True \
+ ${SCRIPTSDIR}/lib/wic/plugins/source/isoimage-isohybrid.py:True \
+ ${SCRIPTSDIR}/lib/wic/plugins/source/rawcopy.py:True \
+ ${SCRIPTSDIR}/lib/wic/plugins/source/rootfs.py:True \
+ ${LAYERDIR_core}/lib/oe/path.py:True \
+ ${LAYERDIR_core}/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py:True \
+ ${LAYERDIR_core}/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py:True \
+"
+do_image_wic[cleandirs] += "${WORKDIR}/wic"
IMAGE_CMD:wic() {
generate_wic_image
check_for_wic_warnings
@@ -162,9 +178,17 @@ generate_wic_image() {
mkdir -p ${IMAGE_ROOTFS}/../pseudo
touch ${IMAGE_ROOTFS}/../pseudo/files.db
- imager_run -p -d ${PP_WORK} -u root <<'EOIMAGER'
+ imager_run_host <<'EOIMAGER'
set -e
+ export WIC_TOOL_RUNNER="${SCRIPTSDIR}/wic-chroot"
+ export WIC_TOOL_SYSROOT="$IMAGER_CHROOT"
+ export TMPDIR="${WORKDIR}/wic/tmp"
+ mkdir -p "$TMPDIR"
+ cd "${WORKDIR}"
+
+ "$WIC_TOOL_RUNNER" /bin/sh -s <<'EOTOOLS'
+ set -e
# The python path is hard-coded as /usr/bin/python3-native/python3 in wic. Handle that.
mkdir -p /usr/bin/python3-native/
if [ $(head -1 $(which bmaptool) | grep python3) ];then
@@ -172,24 +196,27 @@ generate_wic_image() {
else
ln -fs /usr/bin/python2 /usr/bin/python3-native/python3
fi
+EOTOOLS
export PATH="${BITBAKEDIR}/bin:$PATH"
"${SCRIPTSDIR}"/wic create "${WKS_FULL_PATH}" \
--vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
- -o "/tmp/${IMAGE_FULLNAME}.wic/" \
+ --native-sysroot "$IMAGER_CHROOT" \
+ -o "${WORKDIR}/wic/" \
--bmap \
-e "${IMAGE_BASENAME}" ${WIC_CREATE_EXTRA_ARGS}
- WIC_DIRECT=$(ls -t -1 /tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
- mv -f ${WIC_DIRECT} ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic
- mv -f ${WIC_DIRECT}.bmap ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic.bmap
+ WIC_DIRECT=$(ls -t -1 ${WORKDIR}/wic/*.direct | head -1)
+ mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
+ mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic.bmap
# deploy partition files if requested (ending with .p<x>)
if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
# locate *.direct.p<x> partition files
- find "/tmp/${IMAGE_FULLNAME}.wic/" -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
+ find "${WORKDIR}/wic/" -type f -regextype sed \
+ -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')
- mv -f ${f} ${PP_DEPLOY}/${IMAGE_FULLNAME}.wic${suffix}
+ mv -f ${f} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}
done
fi
EOIMAGER
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 5d21cdcb..76f1b0cc 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -10,6 +10,8 @@ import shutil
import subprocess
import os.path
+import bb
+
def join(*paths):
"""Like os.path.join but doesn't treat absolute RHS specially"""
return os.path.normpath("/".join(paths))
@@ -85,18 +87,22 @@ def format_display(path, metadata):
else:
return rel
-def copytree(src, dst):
+def copytree(src, dst, command_runner=None):
# We could use something like shutil.copytree here but it turns out to
# to be slow. It takes twice as long copying to an empty directory.
# If dst already has contents performance can be 15 time slower
# This way we also preserve hardlinks between files in the tree.
+ if command_runner is None:
+ command_runner = subprocess.check_output
bb.utils.mkdirhier(dst)
cmd = "tar --xattrs --xattrs-include='*' -cf - -S -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
- subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+ command_runner(cmd, shell=True, stderr=subprocess.STDOUT)
-def copyhardlinktree(src, dst):
+def copyhardlinktree(src, dst, command_runner=None):
"""Make a tree of hard links when possible, otherwise copy."""
+ if command_runner is None:
+ command_runner = subprocess.check_output
bb.utils.mkdirhier(dst)
if os.path.isdir(src) and not len(os.listdir(src)):
return
@@ -120,7 +126,7 @@ def copyhardlinktree(src, dst):
# Need to copy directories only with tar first since cp will error if two
# writers try and create a directory at the same time
cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -S -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C %s" % (src, src, dst)
- subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+ command_runner(cmd, shell=True, stderr=subprocess.STDOUT)
source = ''
if os.path.isdir(src):
if len(glob.glob('%s/.??*' % src)) > 0:
@@ -132,9 +138,9 @@ def copyhardlinktree(src, dst):
source = src
s_dir = os.getcwd()
cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst))
- subprocess.check_output(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT)
+ command_runner(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT)
else:
- copytree(src, dst)
+ copytree(src, dst, command_runner=command_runner)
def copyhardlink(src, dst):
"""Make a hard link when possible, otherwise copy."""
diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-biosplusefi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-biosplusefi-isar.py
index fb6b7fcc..df37f97b 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-biosplusefi-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-biosplusefi-isar.py
@@ -18,6 +18,7 @@
# AUTHORS
# William Bourque <wbourque [at) gmail.com>
+import os
import types
from wic.pluginbase import SourcePlugin
diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
index a2592986..7fb5f448 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-efi-isar.py
@@ -521,7 +521,13 @@ class BootimgEFIPlugin(SourcePlugin):
kernel_dir_orig = kernel_dir
kernel_dir = "/usr/lib/systemd/boot/efi/"
- for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
+ efi_dir = kernel_dir
+ if os.environ.get('WIC_TOOL_RUNNER'):
+ tool_sysroot = os.environ.get('WIC_TOOL_SYSROOT')
+ if tool_sysroot:
+ efi_dir = os.path.join(tool_sysroot, kernel_dir.lstrip('/'))
+
+ for mod in [x for x in os.listdir(efi_dir) if x.startswith("systemd-")]:
target = mod[8:]
if target.endswith('.signed'):
target = target[:-7]
diff --git a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
index d5040b72..7569c7b1 100644
--- a/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/bootimg-pcbios-isar.py
@@ -18,7 +18,7 @@ from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
from wic.misc import (exec_cmd, exec_native_cmd,
- get_bitbake_var, BOOTDD_EXTRA_SPACE)
+ get_bitbake_var, get_native_path, BOOTDD_EXTRA_SPACE)
# allow plugins to import from isarpluginbase
if '__file__' in globals():
@@ -83,7 +83,8 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
logger.debug("Installing MBR on disk %s as %s with size %s bytes",
disk_name, full_path, disk.min_size)
- dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
+ dd_cmd = "dd if=%s of=%s conv=notrunc" % (get_native_path(mbrfile),
+ full_path)
exec_cmd(dd_cmd, native_sysroot)
@classmethod
@@ -167,6 +168,7 @@ class BootimgPcbiosIsarPlugin(SourcePlugin):
bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
staging_kernel_dir = kernel_dir
+ bootimg_dir = get_native_path(bootimg_dir)
hdddir = "%s/hdd/boot" % cr_workdir
diff --git a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
index 6aa58870..9a7817e7 100644
--- a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
@@ -16,7 +16,8 @@ import shutil
from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
-from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var
+from wic.misc import (exec_cmd, exec_native_cmd,
+ get_bitbake_var, get_native_path)
# allow plugins to import from isarpluginbase
if '__file__' in globals():
@@ -477,6 +478,7 @@ class IsoImagePlugin(SourcePlugin):
syslinux_dir = get_bitbake_var("STAGING_DATADIR")
if not syslinux_dir:
raise WicError("Couldn't find STAGING_DATADIR, exiting.")
+ syslinux_dir = get_native_path(syslinux_dir)
if os.path.exists("%s/isolinux" % isodir):
shutil.rmtree("%s/isolinux" % isodir)
diff --git a/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py b/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py
index 5611f93a..ab10e391 100644
--- a/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py
+++ b/meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py
@@ -17,6 +17,8 @@ import glob
import logging
import re
import os
+import shlex
+import shutil
from wic import WicError
from wic.plugins.source.rootfs import RootfsPlugin
@@ -98,13 +100,14 @@ class RootfsUBootPlugin(RootfsPlugin):
results = glob.glob(os.path.join("/usr/bin/qemu-*-static"))
qemu_static = results[0] if len(results) > 0 else None
if qemu_static:
- cp_cmd = "cp -L %s %s/usr/bin" % (qemu_static, real_rootfs_dir)
- exec_cmd(cp_cmd)
- update_cmd = "chroot %s sh -c update-u-boot-script" % real_rootfs_dir
- exec_cmd(update_cmd)
+ qemu_target = os.path.join(real_rootfs_dir, "usr/bin",
+ os.path.basename(qemu_static))
+ shutil.copy(qemu_static, qemu_target)
+ update_cmd = "TMPDIR=/tmp chroot %s sh -c update-u-boot-script" % \
+ shlex.quote(real_rootfs_dir)
+ exec_cmd(update_cmd, as_shell=True)
if qemu_static:
- rm_cmd = "rm -f %s/usr/bin/%s" % (real_rootfs_dir, qemu_static)
- exec_cmd(rm_cmd)
+ os.remove(qemu_target)
# For reproducibility set the time stamp of newly updated files
if os.getenv('SOURCE_DATE_EPOCH'):
diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 1a7c140f..dc6114a2 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -15,6 +15,7 @@
import logging
import os
import re
+import shlex
import subprocess
import shutil
@@ -51,6 +52,17 @@ NATIVE_RECIPES = {"bmaptool": "bmaptool",
"tar": "tar"
}
+def get_native_path(path):
+ """Translate a host sysroot path to its matching path in the chroot."""
+ tool_sysroot = os.environ.get('WIC_TOOL_SYSROOT')
+ if tool_sysroot and os.environ.get('WIC_TOOL_RUNNER'):
+ tool_sysroot = os.path.normpath(tool_sysroot)
+ if path == tool_sysroot:
+ return '/'
+ if path.startswith(tool_sysroot + '/'):
+ return path[len(tool_sysroot):]
+ return path
+
def runtool(cmdln_or_args):
""" wrapper for most of the subprocess calls
input:
@@ -62,10 +74,18 @@ def runtool(cmdln_or_args):
cmd = cmdln_or_args[0]
shell = False
else:
- import shlex
cmd = shlex.split(cmdln_or_args)[0]
shell = True
+ tool_runner = os.environ.get('WIC_TOOL_RUNNER')
+ if tool_runner and os.path.basename(cmd) != 'bitbake':
+ runner_args = shlex.split(tool_runner)
+ if shell:
+ cmdln_or_args = runner_args + ['/bin/sh', '-c', cmdln_or_args]
+ else:
+ cmdln_or_args = runner_args + cmdln_or_args
+ shell = False
+
sout = subprocess.PIPE
serr = subprocess.STDOUT
@@ -123,6 +143,14 @@ def find_executable(cmd, paths):
if provided and "%s-native" % recipe in provided:
return True
+ if os.environ.get('WIC_TOOL_RUNNER'):
+ paths = os.pathsep.join(get_native_path(path)
+ for path in paths.split(os.pathsep))
+ ret, _ = runtool(['/bin/sh', '-c',
+ 'PATH=%s command -v %s' %
+ (shlex.quote(paths), shlex.quote(cmd))])
+ return ret == 0
+
return shutil.which(cmd, path=paths)
def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
@@ -133,6 +161,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
Always need to execute native commands as_shell
"""
+ native_sysroot = get_native_path(native_sysroot)
# The reason -1 is used is because there may be "export" commands.
args = cmd_and_args.split(';')[-1].split()
logger.debug(args)
@@ -263,4 +292,9 @@ def get_bitbake_var(var, image=None, cache=True):
Provide old get_bitbake_var API by wrapping
get_var method of BB_VARS singleton.
"""
- return BB_VARS.get_var(var, image, cache)
+ value = BB_VARS.get_var(var, image, cache)
+ tool_sysroot = os.environ.get('WIC_TOOL_SYSROOT')
+ if tool_sysroot and value and var in ('RECIPE_SYSROOT_NATIVE',
+ 'STAGING_DATADIR', 'STAGING_LIBDIR'):
+ return os.path.join(tool_sysroot, value.lstrip('/'))
+ return value
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index bf2c34d5..7c6b83f3 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -15,7 +15,8 @@ import os
import uuid
from wic import WicError
-from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var
+from wic.misc import (exec_cmd, exec_native_cmd,
+ get_bitbake_var, get_native_path)
from wic.pluginbase import PluginMgr
logger = logging.getLogger('wic')
@@ -219,6 +220,7 @@ class Partition():
os.remove(rootfs)
p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
+ p_prefix = get_native_path(p_prefix)
if (pseudo_dir):
# Canonicalize the ignore paths. This corresponds to
# calling oe.path.canonicalize(), which is used in bitbake.conf.
@@ -548,4 +550,3 @@ class Partition():
logger.warn("%s Inodes (of size %d) are too small." %
(get_err_str(self), size))
break
-
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index a65ae36a..60f2dbf7 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -25,7 +25,8 @@ from wic import WicError
from wic.filemap import sparse_copy
from wic.ksparser import KickStart, KickStartError
from wic.pluginbase import PluginMgr, ImagerPlugin
-from wic.misc import get_bitbake_var, exec_cmd, exec_native_cmd
+from wic.misc import (get_bitbake_var, get_native_path,
+ exec_cmd, exec_native_cmd)
logger = logging.getLogger('wic')
@@ -212,8 +213,10 @@ class DirectPlugin(ImagerPlugin):
# Generate .bmap
if self.bmap:
logger.debug("Generating bmap file for %s", disk_name)
- python = os.path.join(self.native_sysroot, 'usr/bin/python3-native/python3')
- bmaptool = os.path.join(self.native_sysroot, 'usr/bin/bmaptool')
+ tool_sysroot = get_native_path(self.native_sysroot)
+ python = os.path.join(tool_sysroot,
+ 'usr/bin/python3-native/python3')
+ bmaptool = os.path.join(tool_sysroot, 'usr/bin/bmaptool')
exec_native_cmd("%s %s create %s -o %s.bmap" % \
(python, bmaptool, full_path, full_path), self.native_sysroot)
# Compress the image
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index a207a835..77e2e52b 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -18,7 +18,7 @@ from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
from wic.misc import (exec_cmd, exec_native_cmd,
- get_bitbake_var, BOOTDD_EXTRA_SPACE)
+ get_bitbake_var, get_native_path, BOOTDD_EXTRA_SPACE)
logger = logging.getLogger('wic')
@@ -73,7 +73,8 @@ class BootimgPcbiosPlugin(SourcePlugin):
logger.debug("Installing MBR on disk %s as %s with size %s bytes",
disk_name, full_path, disk.min_size)
- dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
+ dd_cmd = "dd if=%s of=%s conv=notrunc" % (get_native_path(mbrfile),
+ full_path)
exec_cmd(dd_cmd, native_sysroot)
@classmethod
@@ -146,6 +147,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
staging_kernel_dir = kernel_dir
+ bootimg_dir = get_native_path(bootimg_dir)
hdddir = "%s/hdd/boot" % cr_workdir
diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
index 607356ad..4b8815b6 100644
--- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
+++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
@@ -18,7 +18,8 @@ import shutil
from wic import WicError
from wic.engine import get_custom_config
from wic.pluginbase import SourcePlugin
-from wic.misc import exec_cmd, exec_native_cmd, get_bitbake_var
+from wic.misc import (exec_cmd, exec_native_cmd,
+ get_bitbake_var, get_native_path)
logger = logging.getLogger('wic')
@@ -385,6 +386,7 @@ class IsoImagePlugin(SourcePlugin):
syslinux_dir = get_bitbake_var("STAGING_DATADIR")
if not syslinux_dir:
raise WicError("Couldn't find STAGING_DATADIR, exiting.")
+ syslinux_dir = get_native_path(syslinux_dir)
if os.path.exists("%s/isolinux" % isodir):
shutil.rmtree("%s/isolinux" % isodir)
diff --git a/scripts/lib/wic/plugins/source/rawcopy.py b/scripts/lib/wic/plugins/source/rawcopy.py
index 21903c2f..a5e9b1d6 100644
--- a/scripts/lib/wic/plugins/source/rawcopy.py
+++ b/scripts/lib/wic/plugins/source/rawcopy.py
@@ -6,8 +6,7 @@
import logging
import os
-import signal
-import subprocess
+import shlex
from wic import WicError
from wic.pluginbase import SourcePlugin
@@ -48,12 +47,6 @@ class RawCopyPlugin(SourcePlugin):
@staticmethod
def do_image_uncompression(src, dst, workdir):
- def subprocess_setup():
- # Python installs a SIGPIPE handler by default. This is usually not what
- # non-Python subprocesses expect.
- # SIGPIPE errors are known issues with gzip/bash
- signal.signal(signal.SIGPIPE, signal.SIG_DFL)
-
extension = os.path.splitext(src)[1]
decompressor = {
".bz2": "bzip2",
@@ -63,8 +56,9 @@ class RawCopyPlugin(SourcePlugin):
}.get(extension)
if not decompressor:
raise WicError("Not supported compressor filename extension: %s" % extension)
- cmd = "%s -dc %s > %s" % (decompressor, src, dst)
- subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=workdir)
+ cmd = "cd %s && %s -dc %s > %s" % (shlex.quote(workdir), decompressor,
+ shlex.quote(src), shlex.quote(dst))
+ exec_cmd(cmd, as_shell=True)
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
@@ -90,6 +84,12 @@ class RawCopyPlugin(SourcePlugin):
RawCopyPlugin.do_image_uncompression(img, src, cr_workdir)
else:
src = os.path.join(kernel_dir, source_params['file'])
+ if os.environ.get('WIC_TOOL_RUNNER'):
+ tool_sysroot = os.environ.get('WIC_TOOL_SYSROOT')
+ if tool_sysroot and os.path.isabs(src):
+ host_src = os.path.join(tool_sysroot, src.lstrip('/'))
+ if os.path.lexists(host_src):
+ src = host_src
dst = os.path.join(cr_workdir, "%s.%s" % (os.path.basename(source_params['file']), part.lineno))
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index e29f3a4c..d9629b9c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -13,6 +13,7 @@
import logging
import os
+import shlex
import shutil
import sys
@@ -21,7 +22,8 @@ from pathlib import Path
from wic import WicError
from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var, exec_native_cmd
+from wic.misc import (exec_cmd, exec_native_cmd,
+ get_bitbake_var, get_native_path)
logger = logging.getLogger('wic')
@@ -63,6 +65,7 @@ class RootfsPlugin(SourcePlugin):
@staticmethod
def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+ native_sysroot = get_native_path(native_sysroot)
pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
@@ -70,6 +73,12 @@ class RootfsPlugin(SourcePlugin):
pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
return pseudo
+ @staticmethod
+ def __copy_rootfs_command(command, cwd=None, **kwargs):
+ if cwd:
+ command = "cd %s && %s" % (shlex.quote(cwd), command)
+ return exec_cmd(command, as_shell=True)
+
@classmethod
def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
@@ -122,7 +131,11 @@ class RootfsPlugin(SourcePlugin):
orig_dir = cls.__validate_path("--change-directory", part.rootfs_dir, cd)
else:
orig_dir = part.rootfs_dir
- copyhardlinktree(orig_dir, new_rootfs)
+ command_runner = None
+ if os.environ.get('WIC_TOOL_RUNNER'):
+ command_runner = cls.__copy_rootfs_command
+ copyhardlinktree(orig_dir, new_rootfs,
+ command_runner=command_runner)
# Convert the pseudo directory to its new location
if (pseudo_dir):
diff --git a/scripts/wic-chroot b/scripts/wic-chroot
new file mode 100755
index 00000000..19be0c00
--- /dev/null
+++ b/scripts/wic-chroot
@@ -0,0 +1,18 @@
+#!/bin/sh
+# This software is a part of Isar.
+# Copyright (C) Siemens AG, 2026
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+: "${IMAGER_CHROOT:?No imaging chroot configured}"
+
+# Expand the variables in the chroot shell, not on the host
+# shellcheck disable=SC2016
+exec chroot "$IMAGER_CHROOT" /bin/sh -c '
+ cd "$1" || exit
+ shift
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ exec "$@"
+' sh "$PWD" "$@"
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-6-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
` (4 preceding siblings ...)
2026-11-09 9:17 ` [PATCH 5/8] wic: run the Python driver outside the imaging chroot 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:17 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:18 ` [PATCH 7/8] meta: add missing whitespace around assignment operators 'Clara Kowalsky' via isar-users
2026-11-09 9:18 ` [PATCH 8/8] bitbake: update to v2.20.0 'Clara Kowalsky' via isar-users
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:17 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
BitBake v2.20 includes commit 8ab71d0ce302 ("bitbake: fix issue with
varflag exclusion") which changes BB_SIGNATURE_EXCLUDE_FLAGS to match
complete flag names instead of substrings.
Isar lists export_func, which previously implicitly excluded export and
func. With the updated matching, both flags must be added to the
exclusion list to keep them out of the task signatures.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 4ffe6d14..af536986 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -121,7 +121,7 @@ BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
BB_SETSCENE_ENFORCE BB_CMDLINE BB_SERVER_TIMEOUT"
BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
lockfiles vardepsexclude vardeps vardepvalue vardepvalueexclude \
- file-checksums python task nostamp \
+ file-checksums python task nostamp export func \
sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
progress mcdepends number_threads"
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-7-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 7/8] meta: add missing whitespace around assignment operators
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
` (5 preceding siblings ...)
2026-11-09 9:17 ` [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:18 ` 'Clara Kowalsky' via isar-users
2026-11-09 9:18 ` [PATCH 8/8] bitbake: update to v2.20.0 'Clara Kowalsky' via isar-users
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:18 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
BitBake commit 24772dd2ae6c ("parse/ConfHandler: Add warning for
deprecated whitespace usage"), included in v2.15, emits warnings for
missing whitespace around variable assignment operators.
Fix the warnings by adding the missing spaces. Users can validate their
layers by parsing the metadata on BitBake v2.20 with `bitbake
--parse-only`.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
meta-isar/classes-recipe/installer-add-rootfs.bbclass | 2 +-
meta-isar/recipes-app/hello-isar/hello-isar.bb | 2 +-
meta-isar/recipes-app/libhello/libhello.bb | 2 +-
meta-isar/recipes-installer/bmap-tools/bmap-tools.bb | 2 +-
meta/classes-recipe/dpkg.bbclass | 2 +-
meta/classes-recipe/rootfs.bbclass | 4 ++--
meta/classes-recipe/sbuild.bbclass | 2 +-
meta/recipes-bsp/libubootenv/libubootenv_0.3.5-0.1.bb | 2 +-
8 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/meta-isar/classes-recipe/installer-add-rootfs.bbclass b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
index c4921bc6..2569e81b 100644
--- a/meta-isar/classes-recipe/installer-add-rootfs.bbclass
+++ b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
@@ -78,5 +78,5 @@ python() {
d.setVar("ROOTFS_ADDITIONAL_FILES", " ".join(entries))
}
-INSTALLER_TARGET_TASK ??="do_build"
+INSTALLER_TARGET_TASK ??= "do_build"
do_rootfs_install[mcdepends] += "${@ get_mc_depends(d, d.getVar('INSTALLER_TARGET_TASK'))}"
diff --git a/meta-isar/recipes-app/hello-isar/hello-isar.bb b/meta-isar/recipes-app/hello-isar/hello-isar.bb
index 240b1b6f..db908b21 100644
--- a/meta-isar/recipes-app/hello-isar/hello-isar.bb
+++ b/meta-isar/recipes-app/hello-isar/hello-isar.bb
@@ -23,4 +23,4 @@ SRCREV = "b48d15629667377f544e71c7310b80a71d00d9dd"
inherit dpkg
# Example of using alternative sbuild chroot
-SBUILD_FLAVOR="db2m"
+SBUILD_FLAVOR = "db2m"
diff --git a/meta-isar/recipes-app/libhello/libhello.bb b/meta-isar/recipes-app/libhello/libhello.bb
index 9b4bb9ff..f357ee5b 100644
--- a/meta-isar/recipes-app/libhello/libhello.bb
+++ b/meta-isar/recipes-app/libhello/libhello.bb
@@ -16,4 +16,4 @@ SRCREV = "f4a5669c8c63f7cae8ff268cbf298dd45865b974"
inherit dpkg
# Example of using alternative sbuild chroot
-SBUILD_FLAVOR="db2m"
+SBUILD_FLAVOR = "db2m"
diff --git a/meta-isar/recipes-installer/bmap-tools/bmap-tools.bb b/meta-isar/recipes-installer/bmap-tools/bmap-tools.bb
index e7dd6cbe..15729998 100644
--- a/meta-isar/recipes-installer/bmap-tools/bmap-tools.bb
+++ b/meta-isar/recipes-installer/bmap-tools/bmap-tools.bb
@@ -9,7 +9,7 @@ MAINTAINER = "isar-users <isar-users@googlegroups.com>"
SRC_URI = "apt://${BPN}"
PN = "bmap-tools"
-CHANGELOG_V="<orig-version>+isar"
+CHANGELOG_V = "<orig-version>+isar"
DPKG_ARCH = "all"
SRC_URI += "file://0001-Fix-path-parameter-passing-error-of-set_psplash_pipe.patch;apply=no \
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index 737915b8..525425e7 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -9,7 +9,7 @@ inherit dpkg-source
DPKG_BUILD_TIMEOUT ?= "150"
dpkg_runbuild[vardepsexclude] += "${DPKG_BUILD_TIMEOUT}"
-DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+DPKG_PREBUILD_ENV_FILE = "${WORKDIR}/dpkg_prebuild.env"
# bitbake variables that should be passed into sbuild env
# Note: must not have any logical influence on the generated package
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 11b69d2f..d0c93abf 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -64,9 +64,9 @@ ROOTFS_FEATURES:remove:focal = "generate-sbom"
# Capture all information needed for sbom generation
ROOTFS_APT_STATE = "apt-state.tar.zst"
-ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+ROOTFS_APT_ARGS = "install --yes -o Debug::pkgProblemResolver=yes"
-ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf"
+ROOTFS_CLEAN_FILES = "/etc/hostname /etc/resolv.conf"
ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${DISTRO_ARCH}"
diff --git a/meta/classes-recipe/sbuild.bbclass b/meta/classes-recipe/sbuild.bbclass
index f2916931..a796a737 100644
--- a/meta/classes-recipe/sbuild.bbclass
+++ b/meta/classes-recipe/sbuild.bbclass
@@ -13,7 +13,7 @@ SBUILD_CHROOT:schroot ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${ISAR_BUILD_UUID}-${
SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
-SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+SBUILD_CONFIG = "${WORKDIR}/sbuild.conf"
# Lockfile available for all the users
SCHROOT_LOCKFILE = "/tmp/schroot.lock"
diff --git a/meta/recipes-bsp/libubootenv/libubootenv_0.3.5-0.1.bb b/meta/recipes-bsp/libubootenv/libubootenv_0.3.5-0.1.bb
index 1a6c9013..faaf8e0f 100644
--- a/meta/recipes-bsp/libubootenv/libubootenv_0.3.5-0.1.bb
+++ b/meta/recipes-bsp/libubootenv/libubootenv_0.3.5-0.1.bb
@@ -7,7 +7,7 @@
DESCRIPTION = "Standalone library & tools for accessing the U-Boot environment"
MAINTAINER = "isar-users <isar-users@googlegroups.com>"
-HOMEPAGE= "https://github.com/sbabic/libubootenv"
+HOMEPAGE = "https://github.com/sbabic/libubootenv"
LICENSE = "GPL-2.0"
LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-8-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 8/8] bitbake: update to v2.20.0
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
` (6 preceding siblings ...)
2026-11-09 9:18 ` [PATCH 7/8] meta: add missing whitespace around assignment operators 'Clara Kowalsky' via isar-users
@ 2026-11-09 9:18 ` 'Clara Kowalsky' via isar-users
7 siblings, 0 replies; 9+ messages in thread
From: 'Clara Kowalsky' via isar-users @ 2026-11-09 9:18 UTC (permalink / raw)
To: isar-users; +Cc: Clara Kowalsky
Update BitBake from v2.8.0 to v2.20.0. This is upstream commit
634f8dede6b9d71b5a6691499fd05a8678b46eae, tagged yocto-6.1.
Drop Isar's lowered Python version check and retain upstream's Python
3.9 minimum. WIC now runs with the build host's Python, so imaging
Debian buster no longer requires this workaround.
In gitignore, set /build* so Git does not ignore BitBake's new
bs4/builder directory.
Signed-off-by: Clara Kowalsky <clara.kowalsky@siemens.com>
---
.gitignore | 2 +-
bitbake/.b4-config | 4 +
bitbake/.gitignore | 2 +
bitbake/AGENTS.md | 55 +
bitbake/README | 2 +-
bitbake/TODO | 65 +-
bitbake/bin/bitbake | 4 +-
bitbake/bin/bitbake-config-build | 1 +
bitbake/bin/bitbake-diffsigs | 16 +-
bitbake/bin/bitbake-getvar | 21 +-
bitbake/bin/bitbake-hashclient | 115 +-
bitbake/bin/bitbake-hashserv | 17 +-
bitbake/bin/bitbake-layers | 46 +-
bitbake/bin/bitbake-prserv | 34 +-
bitbake/bin/bitbake-selftest | 13 +-
bitbake/bin/bitbake-server | 14 +-
bitbake/bin/bitbake-setup | 1481 ++++++
bitbake/bin/bitbake-worker | 42 +-
bitbake/bin/git-make-shallow | 175 -
bitbake/bin/toaster | 23 +-
bitbake/bin/toaster-eventreplay | 7 +
bitbake/conf/bitbake.conf | 2 +-
bitbake/contrib/b4-wrapper-bitbake.py | 40 +
bitbake/contrib/hashserv/Dockerfile | 5 +-
bitbake/contrib/prserv/Dockerfile | 3 +-
bitbake/contrib/vim/LICENSE.txt | 18 -
bitbake/contrib/vim/ftdetect/bitbake.vim | 24 -
bitbake/contrib/vim/ftplugin/bitbake.vim | 13 -
bitbake/contrib/vim/indent/bitbake.vim | 343 --
bitbake/contrib/vim/plugin/newbb.vim | 88 -
bitbake/contrib/vim/plugin/newbbappend.vim | 46 -
bitbake/contrib/vim/syntax/bitbake.vim | 131 -
.../oe-nodistro-blacksail.conf.json | 48 +
.../oe-nodistro-master.conf.json | 48 +
.../oe-nodistro-whinlatter.conf.json | 56 +
.../oe-nodistro-wrynose.conf.json | 48 +
.../configurations/poky-blacksail.conf.json | 74 +
.../configurations/poky-master.conf.json | 74 +
.../configurations/poky-whinlatter.conf.json | 80 +
.../configurations/poky-wrynose.conf.json | 74 +
bitbake/doc/.gitignore | 3 +
bitbake/doc/Makefile | 2 +-
bitbake/doc/README | 2 +-
.../bitbake-user-manual-environment-setup.rst | 1332 +++++
.../bitbake-user-manual-execution.rst | 35 +-
.../bitbake-user-manual-fetching.rst | 305 +-
.../bitbake-user-manual-intro.rst | 267 +-
.../bitbake-user-manual-library-functions.rst | 59 +
.../bitbake-user-manual-metadata.rst | 462 +-
.../bitbake-user-manual-ref-variables.rst | 464 +-
bitbake/doc/conf.py | 41 +-
bitbake/doc/index.rst | 2 +
bitbake/doc/releases.rst | 174 -
bitbake/doc/setversions.py | 374 ++
bitbake/doc/sphinx-static/switchers.js | 233 -
bitbake/doc/sphinx-static/switchers.js.in | 249 +
bitbake/doc/sphinx-static/theme_overrides.css | 17 +-
bitbake/lib/bb/COW.py | 10 +-
bitbake/lib/bb/__init__.py | 158 +-
.../_vendor/beautifulsoup4.LICENSE} | 11 +-
bitbake/lib/bb/_vendor/bs4/__init__.py | 1177 +++++
bitbake/lib/bb/_vendor/bs4/_deprecation.py | 80 +
bitbake/lib/bb/_vendor/bs4/_typing.py | 205 +
bitbake/lib/bb/_vendor/bs4/_warnings.py | 98 +
.../lib/bb/_vendor/bs4/builder/__init__.py | 848 ++++
.../lib/bb/_vendor/bs4/builder/_html5lib.py | 611 +++
.../lib/bb/_vendor/bs4/builder/_htmlparser.py | 517 ++
bitbake/lib/bb/_vendor/bs4/builder/_lxml.py | 501 ++
bitbake/lib/bb/_vendor/bs4/css.py | 330 ++
bitbake/lib/bb/_vendor/bs4/dammit.py | 1525 ++++++
bitbake/lib/bb/_vendor/bs4/diagnose.py | 268 +
bitbake/lib/bb/_vendor/bs4/element.py | 3395 +++++++++++++
bitbake/lib/bb/_vendor/bs4/exceptions.py | 28 +
bitbake/lib/bb/_vendor/bs4/filter.py | 764 +++
bitbake/lib/bb/_vendor/bs4/formatter.py | 276 ++
bitbake/lib/bb/_vendor/bs4/py.typed | 0
bitbake/lib/bb/_vendor/ply.pyi | 1 +
bitbake/lib/{ => bb/_vendor}/ply/__init__.py | 1 +
bitbake/lib/bb/_vendor/ply/cpp.py | 918 ++++
bitbake/lib/bb/_vendor/ply/ctokens.py | 133 +
bitbake/lib/bb/_vendor/ply/lex.py | 901 ++++
bitbake/lib/{ => bb/_vendor}/ply/yacc.py | 2238 +++++----
bitbake/lib/bb/_vendor/ply/ygen.py | 74 +
bitbake/lib/bb/_vendor/progressbar.pyi | 1 +
.../{ => bb/_vendor}/progressbar/LICENSE.txt | 0
.../{ => bb/_vendor}/progressbar/__init__.py | 4 +-
.../{ => bb/_vendor}/progressbar/compat.py | 0
.../_vendor}/progressbar/progressbar.py | 37 +-
.../{ => bb/_vendor}/progressbar/widgets.py | 14 +-
bitbake/lib/bb/_vendor/simplediff.pyi | 1 +
.../lib/{ => bb/_vendor}/simplediff/LICENSE | 0
.../{ => bb/_vendor}/simplediff/__init__.py | 2 +-
bitbake/lib/bb/_vendor/tomli/LICENSE | 21 +
bitbake/lib/bb/_vendor/tomli/__init__.py | 8 +
bitbake/lib/bb/_vendor/tomli/_parser.py | 793 +++
bitbake/lib/bb/_vendor/tomli/_re.py | 119 +
bitbake/lib/bb/_vendor/tomli/_types.py | 10 +
bitbake/lib/bb/_vendor/tomli/py.typed | 1 +
.../lib/bb/_vendor/typing_extensions.LICENSE | 279 ++
bitbake/lib/bb/_vendor/typing_extensions.py | 4317 +++++++++++++++++
bitbake/lib/bb/_vendor/typing_extensions.pyi | 1 +
bitbake/lib/bb/acl.py | 2 -
bitbake/lib/bb/asyncrpc/__init__.py | 3 +-
bitbake/lib/bb/asyncrpc/client.py | 162 +-
bitbake/lib/bb/asyncrpc/serv.py | 58 +-
bitbake/lib/bb/asyncrpc/taskgroup.py | 52 +
bitbake/lib/bb/build.py | 29 +-
bitbake/lib/bb/cache.py | 37 +-
bitbake/lib/bb/checksum.py | 25 +
bitbake/lib/bb/codeparser.py | 102 +-
bitbake/lib/bb/command.py | 50 +-
bitbake/lib/bb/compress/lz4.py | 4 +-
bitbake/lib/bb/cooker.py | 319 +-
bitbake/lib/bb/cookerdata.py | 40 +-
bitbake/lib/bb/daemonize.py | 3 +
bitbake/lib/bb/data.py | 19 +-
bitbake/lib/bb/data_smart.py | 157 +-
bitbake/lib/bb/event.py | 29 +-
bitbake/lib/bb/{fetch2 => fetch}/README | 0
bitbake/lib/bb/{fetch2 => fetch}/__init__.py | 526 +-
bitbake/lib/bb/{fetch2 => fetch}/az.py | 23 +-
bitbake/lib/bb/{fetch2 => fetch}/clearcase.py | 63 +-
bitbake/lib/bb/fetch/crate.py | 198 +
bitbake/lib/bb/{fetch2 => fetch}/gcp.py | 45 +-
bitbake/lib/bb/{fetch2 => fetch}/git.py | 674 +--
bitbake/lib/bb/{fetch2 => fetch}/gitannex.py | 22 +-
bitbake/lib/bb/{fetch2 => fetch}/gitsm.py | 144 +-
bitbake/lib/bb/fetch/gomod.py | 270 ++
bitbake/lib/bb/{fetch2 => fetch}/hg.py | 65 +-
bitbake/lib/bb/{fetch2 => fetch}/local.py | 15 +-
bitbake/lib/bb/{fetch2 => fetch}/npm.py | 169 +-
bitbake/lib/bb/{fetch2 => fetch}/npmsw.py | 120 +-
bitbake/lib/bb/{fetch2 => fetch}/perforce.py | 46 +-
bitbake/lib/bb/{fetch2 => fetch}/repo.py | 26 +-
bitbake/lib/bb/{fetch2 => fetch}/s3.py | 21 +-
bitbake/lib/bb/{fetch2 => fetch}/sftp.py | 23 +-
bitbake/lib/bb/{fetch2 => fetch}/ssh.py | 33 +-
bitbake/lib/bb/{fetch2 => fetch}/svn.py | 88 +-
bitbake/lib/bb/{fetch2 => fetch}/wget.py | 316 +-
bitbake/lib/bb/fetch2.py | 19 +
bitbake/lib/bb/fetch2/bzr.py | 128 -
bitbake/lib/bb/fetch2/crate.py | 141 -
bitbake/lib/bb/fetch2/cvs.py | 157 -
bitbake/lib/bb/fetch2/osc.py | 165 -
bitbake/lib/bb/filter.py | 142 +
bitbake/lib/bb/main.py | 8 +-
bitbake/lib/bb/parse/__init__.py | 49 +-
bitbake/lib/bb/parse/ast.py | 201 +-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 53 +-
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 24 +-
bitbake/lib/bb/persist_data.py | 271 --
bitbake/lib/bb/pysh/pyshlex.py | 2 +-
bitbake/lib/bb/pysh/pyshyacc.py | 2 +-
bitbake/lib/bb/runqueue.py | 487 +-
bitbake/lib/bb/server/process.py | 86 +-
bitbake/lib/bb/server/xmlrpcserver.py | 11 +-
bitbake/lib/bb/siggen.py | 102 +-
bitbake/lib/bb/taskdata.py | 8 +-
bitbake/lib/bb/tests/codeparser.py | 40 +
bitbake/lib/bb/tests/compression.py | 4 +-
bitbake/lib/bb/tests/cooker.py | 342 +-
bitbake/lib/bb/tests/data.py | 207 +-
.../software/miniupnp/download.php | 3528 ++++++++++++++
bitbake/lib/bb/tests/fetch.py | 2583 +++++++---
bitbake/lib/bb/tests/fetch_import.py | 86 +
bitbake/lib/bb/tests/filter.py | 88 +
.../bb/tests/parse-tests/classes/base.bbclass | 5 +
.../classes/recipe-file-class.bbclass | 2 +
.../bb/tests/parse-tests/conf/bitbake.conf | 15 +
bitbake/lib/bb/tests/parse.py | 413 +-
bitbake/lib/bb/tests/persist_data.py | 129 -
.../tests/runqueue-tests/classes/base.bbclass | 4 +-
.../lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 +
.../lib/bb/tests/runqueue-tests/recipes/h1.bb | 0
bitbake/lib/bb/tests/runqueue.py | 20 +-
bitbake/lib/bb/tests/setup.py | 1089 +++++
bitbake/lib/bb/tests/siggen.py | 47 +
bitbake/lib/bb/tests/support/httpserver.py | 4 +-
bitbake/lib/bb/tests/utils.py | 19 +
bitbake/lib/bb/tinfoil.py | 227 +-
bitbake/lib/bb/ui/buildinfohelper.py | 5 +-
bitbake/lib/bb/ui/knotty.py | 203 +-
bitbake/lib/bb/ui/ncurses.py | 2 +-
bitbake/lib/bb/ui/taskexp_ncurses.py | 3 +
bitbake/lib/bb/ui/teamcity.py | 14 +-
bitbake/lib/bb/ui/uihelper.py | 12 +-
bitbake/lib/bb/utils.py | 867 +++-
bitbake/lib/bblayers/action.py | 4 +-
bitbake/lib/bblayers/layerindex.py | 12 +-
bitbake/lib/bblayers/query.py | 37 +-
bitbake/lib/bs4/AUTHORS.txt | 43 -
bitbake/lib/bs4/NEWS.txt | 1066 ----
bitbake/lib/bs4/__init__.py | 468 --
bitbake/lib/bs4/builder/__init__.py | 324 --
bitbake/lib/bs4/builder/_html5lib.py | 338 --
bitbake/lib/bs4/builder/_htmlparser.py | 262 -
bitbake/lib/bs4/builder/_lxml.py | 248 -
bitbake/lib/bs4/dammit.py | 832 ----
bitbake/lib/bs4/diagnose.py | 216 -
bitbake/lib/bs4/element.py | 1724 -------
bitbake/lib/bs4/testing.py | 686 ---
bitbake/lib/bs4/tests/__init__.py | 1 -
.../lib/bs4/tests/test_builder_registry.py | 147 -
bitbake/lib/bs4/tests/test_docs.py | 32 -
bitbake/lib/bs4/tests/test_html5lib.py | 98 -
bitbake/lib/bs4/tests/test_htmlparser.py | 31 -
bitbake/lib/bs4/tests/test_lxml.py | 70 -
bitbake/lib/bs4/tests/test_soup.py | 479 --
bitbake/lib/bs4/tests/test_tree.py | 2004 --------
bitbake/lib/codegen.py | 564 ---
bitbake/lib/hashserv/__init__.py | 15 +-
bitbake/lib/hashserv/client.py | 423 +-
bitbake/lib/hashserv/server.py | 257 +-
bitbake/lib/hashserv/sqlite.py | 17 +
bitbake/lib/hashserv/tests.py | 589 ++-
bitbake/lib/layerindexlib/__init__.py | 6 +-
bitbake/lib/layerindexlib/cooker.py | 16 +-
bitbake/lib/layerindexlib/restapi.py | 2 +-
bitbake/lib/ply/lex.py | 1054 ----
bitbake/lib/prserv/__init__.py | 97 +-
bitbake/lib/prserv/client.py | 15 +-
bitbake/lib/prserv/db.py | 452 +-
bitbake/lib/prserv/serv.py | 140 +-
bitbake/lib/prserv/tests.py | 388 ++
bitbake/lib/pyinotify.py | 2326 ---------
bitbake/lib/toaster/bldcollector/admin.py | 8 +-
bitbake/lib/toaster/bldcontrol/admin.py | 2 +-
.../bldcontrol/localhostbecontroller.py | 175 +-
.../toaster/orm/fixtures/check_fixtures.py | 76 +
.../lib/toaster/orm/fixtures/gen_fixtures.py | 137 +-
bitbake/lib/toaster/orm/fixtures/oe-core.xml | 72 +-
bitbake/lib/toaster/orm/fixtures/poky.xml | 136 +-
.../orm/management/commands/lsupdates.py | 29 +-
bitbake/lib/toaster/orm/models.py | 3 +-
.../tests/browser/selenium_helpers_base.py | 75 +-
.../tests/browser/test_all_builds_page.py | 7 +-
.../tests/browser/test_all_projects_page.py | 12 +-
.../tests/browser/test_builddashboard_page.py | 2 +-
.../tests/browser/test_landing_page.py | 14 +-
.../tests/browser/test_layerdetails_page.py | 65 +-
.../browser/test_new_custom_image_page.py | 2 +-
.../tests/browser/test_new_project_page.py | 12 +-
bitbake/lib/toaster/tests/builds/buildtest.py | 2 +-
.../tests/functional/functional_helpers.py | 88 +
.../functional/test_create_new_project.py | 121 +-
.../tests/functional/test_functional_basic.py | 200 +-
.../tests/functional/test_project_config.py | 99 +-
.../tests/functional/test_project_page.py | 417 +-
.../test_project_page_tab_config.py | 313 +-
bitbake/lib/toaster/tests/functional/utils.py | 21 +-
.../tests/toaster-tests-requirements.txt | 4 +-
bitbake/lib/toaster/tests/views/test_views.py | 2 +-
.../toastergui/templates/layerdetails.html | 8 +-
.../toastergui/templatetags/projecttags.py | 1 -
bitbake/lib/toaster/toastergui/views.py | 18 +-
bitbake/lib/toaster/toastermain/settings.py | 5 +-
bitbake/packaging-pypi/bitbake-setup/BUILD.md | 67 +
bitbake/packaging-pypi/bitbake-setup/LICENSE | 23 +
.../packaging-pypi/bitbake-setup/README.md | 42 +
.../bitbake-setup/package-bitbake-setup.py | 79 +
.../bitbake-setup/pyproject.toml | 92 +
bitbake/pyproject.toml | 16 +
.../setup-schema/bitbake-setup.schema.json | 211 +
bitbake/setup-schema/layers.schema.json | 107 +
bitbake/toaster-requirements.txt | 2 +-
bitbake/vendor.txt | 6 +
bitbake/vendor/licenses/ply/LICENSE | 32 +
...0010-lib-bs4-Avoid-soupsieve-warning.patch | 40 +
.../patches/bs4-remove-double-imports.patch | 23 +
...mplement-basic-task-progress-support.patch | 125 +
| 79 +
...bar-accept-value-over-initial-maxval.patch | 54 +
...self._fd_console-to-use-for-self._ha.patch | 53 +
...ke-bars-show-correctly-with-maxval-0.patch | 31 +
...ty-Allow-mixing-log-messages-and-pro.patch | 31 +
275 files changed, 41536 insertions(+), 21231 deletions(-)
create mode 100644 bitbake/.b4-config
create mode 100644 bitbake/AGENTS.md
create mode 120000 bitbake/bin/bitbake-config-build
create mode 100755 bitbake/bin/bitbake-setup
delete mode 100755 bitbake/bin/git-make-shallow
create mode 100755 bitbake/contrib/b4-wrapper-bitbake.py
delete mode 100644 bitbake/contrib/vim/LICENSE.txt
delete mode 100644 bitbake/contrib/vim/ftdetect/bitbake.vim
delete mode 100644 bitbake/contrib/vim/ftplugin/bitbake.vim
delete mode 100644 bitbake/contrib/vim/indent/bitbake.vim
delete mode 100644 bitbake/contrib/vim/plugin/newbb.vim
delete mode 100644 bitbake/contrib/vim/plugin/newbbappend.vim
delete mode 100644 bitbake/contrib/vim/syntax/bitbake.vim
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-master.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json
create mode 100644 bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-blacksail.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-master.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-whinlatter.conf.json
create mode 100644 bitbake/default-registry/configurations/poky-wrynose.conf.json
create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst
delete mode 100644 bitbake/doc/releases.rst
create mode 100755 bitbake/doc/setversions.py
delete mode 100644 bitbake/doc/sphinx-static/switchers.js
create mode 100644 bitbake/doc/sphinx-static/switchers.js.in
rename bitbake/lib/{bs4/COPYING.txt => bb/_vendor/beautifulsoup4.LICENSE} (80%)
create mode 100644 bitbake/lib/bb/_vendor/bs4/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_deprecation.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_typing.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/_warnings.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_lxml.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/css.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/dammit.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/diagnose.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/element.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/exceptions.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/filter.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/formatter.py
create mode 100644 bitbake/lib/bb/_vendor/bs4/py.typed
create mode 100644 bitbake/lib/bb/_vendor/ply.pyi
rename bitbake/lib/{ => bb/_vendor}/ply/__init__.py (80%)
create mode 100644 bitbake/lib/bb/_vendor/ply/cpp.py
create mode 100644 bitbake/lib/bb/_vendor/ply/ctokens.py
create mode 100644 bitbake/lib/bb/_vendor/ply/lex.py
rename bitbake/lib/{ => bb/_vendor}/ply/yacc.py (60%)
create mode 100644 bitbake/lib/bb/_vendor/ply/ygen.py
create mode 100644 bitbake/lib/bb/_vendor/progressbar.pyi
rename bitbake/lib/{ => bb/_vendor}/progressbar/LICENSE.txt (100%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/__init__.py (96%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/compat.py (100%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/progressbar.py (88%)
rename bitbake/lib/{ => bb/_vendor}/progressbar/widgets.py (97%)
create mode 100644 bitbake/lib/bb/_vendor/simplediff.pyi
rename bitbake/lib/{ => bb/_vendor}/simplediff/LICENSE (100%)
rename bitbake/lib/{ => bb/_vendor}/simplediff/__init__.py (99%)
create mode 100644 bitbake/lib/bb/_vendor/tomli/LICENSE
create mode 100644 bitbake/lib/bb/_vendor/tomli/__init__.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_parser.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_re.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/_types.py
create mode 100644 bitbake/lib/bb/_vendor/tomli/py.typed
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.LICENSE
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.py
create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.pyi
create mode 100644 bitbake/lib/bb/asyncrpc/taskgroup.py
rename bitbake/lib/bb/{fetch2 => fetch}/README (100%)
rename bitbake/lib/bb/{fetch2 => fetch}/__init__.py (84%)
rename bitbake/lib/bb/{fetch2 => fetch}/az.py (78%)
rename bitbake/lib/bb/{fetch2 => fetch}/clearcase.py (84%)
create mode 100644 bitbake/lib/bb/fetch/crate.py
rename bitbake/lib/bb/{fetch2 => fetch}/gcp.py (62%)
rename bitbake/lib/bb/{fetch2 => fetch}/git.py (52%)
rename bitbake/lib/bb/{fetch2 => fetch}/gitannex.py (65%)
rename bitbake/lib/bb/{fetch2 => fetch}/gitsm.py (63%)
create mode 100644 bitbake/lib/bb/fetch/gomod.py
rename bitbake/lib/bb/{fetch2 => fetch}/hg.py (72%)
rename bitbake/lib/bb/{fetch2 => fetch}/local.py (83%)
rename bitbake/lib/bb/{fetch2 => fetch}/npm.py (62%)
rename bitbake/lib/bb/{fetch2 => fetch}/npmsw.py (70%)
rename bitbake/lib/bb/{fetch2 => fetch}/perforce.py (87%)
rename bitbake/lib/bb/{fetch2 => fetch}/repo.py (68%)
rename bitbake/lib/bb/{fetch2 => fetch}/s3.py (87%)
rename bitbake/lib/bb/{fetch2 => fetch}/sftp.py (85%)
rename bitbake/lib/bb/{fetch2 => fetch}/ssh.py (85%)
rename bitbake/lib/bb/{fetch2 => fetch}/svn.py (65%)
rename bitbake/lib/bb/{fetch2 => fetch}/wget.py (68%)
create mode 100644 bitbake/lib/bb/fetch2.py
delete mode 100644 bitbake/lib/bb/fetch2/bzr.py
delete mode 100644 bitbake/lib/bb/fetch2/crate.py
delete mode 100644 bitbake/lib/bb/fetch2/cvs.py
delete mode 100644 bitbake/lib/bb/fetch2/osc.py
create mode 100644 bitbake/lib/bb/filter.py
delete mode 100644 bitbake/lib/bb/persist_data.py
create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php
create mode 100644 bitbake/lib/bb/tests/fetch_import.py
create mode 100644 bitbake/lib/bb/tests/filter.py
create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/base.bbclass
create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass
create mode 100644 bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf
delete mode 100644 bitbake/lib/bb/tests/persist_data.py
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
create mode 100644 bitbake/lib/bb/tests/setup.py
delete mode 100644 bitbake/lib/bs4/AUTHORS.txt
delete mode 100644 bitbake/lib/bs4/NEWS.txt
delete mode 100644 bitbake/lib/bs4/__init__.py
delete mode 100644 bitbake/lib/bs4/builder/__init__.py
delete mode 100644 bitbake/lib/bs4/builder/_html5lib.py
delete mode 100644 bitbake/lib/bs4/builder/_htmlparser.py
delete mode 100644 bitbake/lib/bs4/builder/_lxml.py
delete mode 100644 bitbake/lib/bs4/dammit.py
delete mode 100644 bitbake/lib/bs4/diagnose.py
delete mode 100644 bitbake/lib/bs4/element.py
delete mode 100644 bitbake/lib/bs4/testing.py
delete mode 100644 bitbake/lib/bs4/tests/__init__.py
delete mode 100644 bitbake/lib/bs4/tests/test_builder_registry.py
delete mode 100644 bitbake/lib/bs4/tests/test_docs.py
delete mode 100644 bitbake/lib/bs4/tests/test_html5lib.py
delete mode 100644 bitbake/lib/bs4/tests/test_htmlparser.py
delete mode 100644 bitbake/lib/bs4/tests/test_lxml.py
delete mode 100644 bitbake/lib/bs4/tests/test_soup.py
delete mode 100644 bitbake/lib/bs4/tests/test_tree.py
delete mode 100644 bitbake/lib/codegen.py
delete mode 100644 bitbake/lib/ply/lex.py
create mode 100644 bitbake/lib/prserv/tests.py
delete mode 100644 bitbake/lib/pyinotify.py
create mode 100755 bitbake/lib/toaster/orm/fixtures/check_fixtures.py
create mode 100644 bitbake/packaging-pypi/bitbake-setup/BUILD.md
create mode 100644 bitbake/packaging-pypi/bitbake-setup/LICENSE
create mode 100644 bitbake/packaging-pypi/bitbake-setup/README.md
create mode 100755 bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
create mode 100644 bitbake/packaging-pypi/bitbake-setup/pyproject.toml
create mode 100644 bitbake/pyproject.toml
create mode 100644 bitbake/setup-schema/bitbake-setup.schema.json
create mode 100644 bitbake/setup-schema/layers.schema.json
create mode 100644 bitbake/vendor.txt
create mode 100644 bitbake/vendor/licenses/ply/LICENSE
create mode 100644 bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch
create mode 100644 bitbake/vendor/patches/bs4-remove-double-imports.patch
create mode 100644 bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch
create mode 100644 bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch
create mode 100644 bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch
create mode 100644 bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch
create mode 100644 bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch
create mode 100644 bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch
diff --git a/.gitignore b/.gitignore
index b5f6149e..f397ff51 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,4 +4,4 @@ __pycache__
.config.yaml*
.kas_shell_history
.isar_test_shell_history
-build*
+/build*
diff --git a/bitbake/.b4-config b/bitbake/.b4-config
new file mode 100644
index 00000000..047f0b94
--- /dev/null
+++ b/bitbake/.b4-config
@@ -0,0 +1,4 @@
+[b4]
+ send-series-to = bitbake-devel@lists.openembedded.org
+ send-auto-cc-cmd = ./contrib/b4-wrapper-bitbake.py send-auto-cc-cmd
+ prep-pre-flight-checks = disable-needs-checking
diff --git a/bitbake/.gitignore b/bitbake/.gitignore
index fbf9eb20..23cb9688 100644
--- a/bitbake/.gitignore
+++ b/bitbake/.gitignore
@@ -3,6 +3,7 @@
*~
pyshtables.py
/doc/manual/html/
+**/bitbake-builds/
/build/
/bin/bitbakec
*.swp
@@ -16,3 +17,4 @@ lib/toaster/contrib/tts/backlog.txt
lib/toaster/contrib/tts/log/*
lib/toaster/contrib/tts/.cache/*
lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log
+packaging-workspace/
diff --git a/bitbake/AGENTS.md b/bitbake/AGENTS.md
new file mode 100644
index 00000000..833eca7f
--- /dev/null
+++ b/bitbake/AGENTS.md
@@ -0,0 +1,55 @@
+# AGENTS.md — BitBake
+
+Guidance for AI agents working in the `bitbake` repository.
+
+## Making changes
+
+- No web PR workflow. The project uses **patches on mailing lists**
+ (bitbake-devel@lists.openembedded.org), tracked via patchwork. Don't assume
+ GitHub PR conventions apply.
+- One Git commit per logical change (e.g., one commit per recipe added/upgraded).
+- Commit message format:
+ - Summary line prefixed with the recipe name or short file path,
+ e.g. `openssl: upgrade 3.2.0 -> 3.2.1` or `classes/rootfs-postcommands: fix typo`.
+ Check `git log --oneline <path>` for the prefix convention already in use.
+ - Body explains what/why/how, and how it was tested.
+ - Prefer explaining *why* a change was made over restating what's
+ already visible in the diff, unless the diff is long/hard to follow
+ or the reason for the change isn't known — in those cases, summarize
+ what changed instead.
+ - Be succinct: use as few sentences and lines as possible while still
+ covering what/why/how. Cut restatements, filler, and any detail a
+ reviewer wouldn't need (e.g. exact test-run counts, step-by-step
+ narration). One short paragraph is usually enough; only add a second
+ if a distinct point (e.g. testing) needs it.
+ - The most relevant information about the change should be presented first.
+ - Reference bugs as `[YOCTO #1234]` when applicable.
+ - Use `Reported-by`, `Suggested-by`, `Tested-by`, `Reviewed-by`, `Cc` tags
+ where relevant.
+
+### AI-generated code
+
+- **Mandatory**: any AI-generated code/commit must add an `AI-Generated:`
+ tag line in the commit message, placed *before* `Signed-off-by` (if present),
+ naming the tool used, e.g.:
+
+ ```
+ component: Add the ability to ...
+
+ AI-Generated: Uses GitHub Copilot
+ ```
+
+- Also add a code comment or other in-patch indication that the code is
+ AI-generated (recommended, not just the commit tag).
+- **Mandatory** Do not automatically add a `Signed-off-by`. This tag must
+ always be written by the human contributor, never by the AI tool — the
+ DCO/Developer's Statement of Origin still applies in full.
+- Contributor is responsible for confirming the AI tool's terms don't
+ conflict with the project's open source license/IP policy, and that
+ any third-party copyrighted material in the AI output is properly
+ licensed before contributing it. See the Linux Foundation generative AI
+ guidance: https://www.linuxfoundation.org/legal/generative-ai.
+- `Co-authored-by` should be omitted in favor of the `AI-generated` tag.
+
+When in doubt about a convention, grep `git log --oneline <path>` in the
+target repo for precedent before inventing a new style.
diff --git a/bitbake/README b/bitbake/README
index e9f4c858..39e4a978 100644
--- a/bitbake/README
+++ b/bitbake/README
@@ -13,7 +13,7 @@ Bitbake plain documentation can be found under the doc directory or its integrat
html version at the Yocto Project website:
https://docs.yoctoproject.org
-Bitbake requires Python version 3.8 or newer.
+Bitbake requires Python version 3.9 or newer.
Contributing
------------
diff --git a/bitbake/TODO b/bitbake/TODO
index 54d091fb..4e0bd9a7 100644
--- a/bitbake/TODO
+++ b/bitbake/TODO
@@ -1,36 +1,31 @@
-- Reimplement the interactive mode as a proper ui
-- Continue dropping fatal/SystemExit/sys.exit usage in favor of raising
- appropriate exceptions
-- Continue pylint / pyflakes / pychecker / pep8 fixups
+
+- Interfaces
+ - Add an interactive mode to knotty (similar to the interactive mode we dropped)
+ - Fix up or drop the ncurses interface? Currently too limited to be useful.
+ - Concerns about whether teamcity interface works. No testing.
+
+- Improve exception handling:
+ - dropping fatal/SystemExit/sys.exit usage in favor of raising
+ appropriate exceptions
+ - Audit bb.fatal usage - these should all be able to be replaced with
+ exceptions
+
+- Continue pylint / pyflakes / pychecker / pep8 fixups - focus on real issues
+ or performance issues, not cosmetic ones.
+
- Drop os.system usage in favor of direct subprocess usage or a subprocess
wrapper
-- Kill the execution of 'tee' for the task log file in build.py
-- Fix up the exception handling
-
- - Kill exec_task's catch of FuncFailed, instead catch it in the other
- callers of exec_task/exec_func
- - What exactly is the purpose of the "EventException"? I can see using an
- exception like that, *perhaps*, to abstract away exceptions raised by
- event handlers, but it has no place in bb.build.exec_task
- BUG: if you chmod 000 local.conf, it silently doesn't parse it, when it
should really fail, so the user can fix the problem.
-- Audit bb.fatal usage - these should all be able to be replaced with
- exceptions
-- Figure out how to handle the ncurses UI. Should some of our logging
- formatting stuff be made common to all of bitbake, or perhaps all UIs via
- the UIHelper?
Long term, high impact:
- - Change override application to actually *move* it over -- so the original
- override specific version of the variable goes away, rather than sticking
- around as a duplicate.
- - Change the behavior when a variable is referenced and is unset. Today, it
- evaluates to ${FOO} and then shell has a chance to expand it, but this is
- far from ideal. We had considered evaluating it to the empty string, but
- that has other potential problems. Frans Meulenbroeks has proposed just
+ - Consider changing the behavior when a variable is referenced and is unset
+ Today, it evaluates to ${FOO} and then shell has a chance to expand it, but
+ this is far from ideal. We had considered evaluating it to the empty string,
+ but that has other potential problems. Frans Meulenbroeks has proposed just
erroring when this occurs, as we can always define default values for the
variables in bitbake.conf. This seems reasonable. My only concern with
that is the case where you want to reference a shell variable with odd
@@ -38,25 +33,3 @@ Long term, high impact:
expansion rather than normal $. To handle that case, we'd really need a
way to escape / disable bitbake variable expansion, \${} perhaps.
-Uncertain:
-
- - Leverage the python 2.6 multiprocessing module
-
- - Worker processes for bb.cooker
- - Server / UI processes
-
- - Create a bitbake configuration class which is utilized by the library, not
- just bin/bitbake. This class should be responsible for extracting
- configuration parameters from the metadata for bitbake internal use, as well
- as pulling specific items like BBDEBUG, and importing settings from an
- optparse options object.
-
- - Python version bits
-
- - Utilize the new string formatting where appropriate
- - Do we need to take into account the bytes literals changes?
- - Do we have any file-like objects that would benefit from using the "io"
- module?
- - Do we want to leverage the abstract base classes in collections?
- - Aside: Set methods now accept multiple iterables
-
diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index f494eaa1..71bc3791 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -27,11 +27,11 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException
bb.utils.check_system_locale()
-__version__ = "2.8.0"
+__version__ = "2.20.0"
if __name__ == "__main__":
if __version__ != bb.__version__:
- sys.exit("Bitbake core version and program version mismatch!")
+ sys.exit("Bitbake core library version and program version mismatch!")
try:
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
cookerdata.CookerConfiguration()))
diff --git a/bitbake/bin/bitbake-config-build b/bitbake/bin/bitbake-config-build
new file mode 120000
index 00000000..11e6df80
--- /dev/null
+++ b/bitbake/bin/bitbake-config-build
@@ -0,0 +1 @@
+bitbake-layers
\ No newline at end of file
diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs
index 8202c786..99660d5c 100755
--- a/bitbake/bin/bitbake-diffsigs
+++ b/bitbake/bin/bitbake-diffsigs
@@ -22,6 +22,13 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), '
import bb.tinfoil
import bb.siggen
import bb.msg
+from bb import __version__ as libbb_version
+
+__version__ = "2.20.0"
+
+if __version__ != libbb_version:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
myname = os.path.basename(sys.argv[0])
logger = bb.msg.logger_create(myname)
@@ -72,16 +79,17 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None):
elif sig2 not in sigfiles:
logger.error('No sigdata files found matching %s %s with signature %s' % (pn, taskname, sig2))
sys.exit(1)
+
+ latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']]
else:
sigfiles = find_siginfo(bbhandler, pn, taskname)
latestsigs = sorted(sigfiles.keys(), key=lambda h: sigfiles[h]['time'])[-2:]
if not latestsigs:
logger.error('No sigdata files found matching %s %s' % (pn, taskname))
sys.exit(1)
- sig1 = latestsigs[0]
- sig2 = latestsigs[1]
-
- latestfiles = [sigfiles[sig1]['path'], sigfiles[sig2]['path']]
+ latestfiles = [sigfiles[latestsigs[0]]['path']]
+ if len(latestsigs) > 1:
+ latestfiles.append(sigfiles[latestsigs[1]]['path'])
return latestfiles
diff --git a/bitbake/bin/bitbake-getvar b/bitbake/bin/bitbake-getvar
index 8901f99a..7f4d3745 100755
--- a/bitbake/bin/bitbake-getvar
+++ b/bitbake/bin/bitbake-getvar
@@ -10,15 +10,23 @@ import io
import os
import sys
import warnings
+import logging
warnings.simplefilter("default")
bindir = os.path.dirname(__file__)
topdir = os.path.dirname(bindir)
sys.path[0:0] = [os.path.join(topdir, 'lib')]
+import bb.providers
import bb.tinfoil
+from bb import __version__ as libbb_version
+
+__version__ = "2.20.0"
if __name__ == "__main__":
+ if __version__ != libbb_version:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
parser = argparse.ArgumentParser(description="Bitbake Query Variable")
parser.add_argument("variable", help="variable name to query")
parser.add_argument("-r", "--recipe", help="Recipe name to query", default=None, required=False)
@@ -37,13 +45,22 @@ if __name__ == "__main__":
sys.exit("--flag only makes sense with --value")
quiet = args.quiet or args.value
+ if quiet:
+ logger = logging.getLogger("BitBake")
+ logger.setLevel(logging.WARNING)
+
with bb.tinfoil.Tinfoil(tracking=True, setup_logging=not quiet) as tinfoil:
if args.recipe:
tinfoil.prepare(quiet=3 if quiet else 2)
- d = tinfoil.parse_recipe(args.recipe)
+ try:
+ d = tinfoil.parse_recipe(args.recipe)
+ except bb.providers.NoProvider as e:
+ sys.exit(str(e))
else:
tinfoil.prepare(quiet=2, config_only=True)
- d = tinfoil.config_data
+ # Expand keys and run anonymous functions to get identical result to
+ # "bitbake -e"
+ d = tinfoil.finalizeData()
value = None
if args.flag:
diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient
index 610787ed..ce5fd173 100755
--- a/bitbake/bin/bitbake-hashclient
+++ b/bitbake/bin/bitbake-hashclient
@@ -16,8 +16,12 @@ import time
import warnings
import netrc
import json
+import statistics
+import textwrap
warnings.simplefilter("default")
+__version__ = "2.20.0"
+
try:
import tqdm
ProgressBar = tqdm.tqdm
@@ -39,6 +43,12 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib
import hashserv
import bb.asyncrpc
+from bb import __version__ as libbb_version
+
+if __version__ != libbb_version:
+ sys.exit("Bitbake core library version and program version mismatch!")
+if __version__ != hashserv.__version__:
+ sys.exit("Bitbake hashserv library version and program version mismatch!")
DEFAULT_ADDRESS = 'unix://./hashserve.sock'
METHOD = 'stress.test.method'
@@ -81,6 +91,7 @@ def main():
nonlocal found_hashes
nonlocal missed_hashes
nonlocal max_time
+ nonlocal times
with hashserv.create_client(args.address) as client:
for i in range(args.requests):
@@ -98,29 +109,41 @@ def main():
else:
missed_hashes += 1
- max_time = max(elapsed, max_time)
+ times.append(elapsed)
pbar.update()
max_time = 0
found_hashes = 0
missed_hashes = 0
lock = threading.Lock()
- total_requests = args.clients * args.requests
+ times = []
start_time = time.perf_counter()
- with ProgressBar(total=total_requests) as pbar:
+ with ProgressBar(total=args.clients * args.requests) as pbar:
threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)]
for t in threads:
t.start()
for t in threads:
t.join()
+ total_elapsed = time.perf_counter() - start_time
- elapsed = time.perf_counter() - start_time
with lock:
- print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed))
- print("Average request time %.8fs" % (elapsed / total_requests))
- print("Max request time was %.8fs" % max_time)
- print("Found %d hashes, missed %d" % (found_hashes, missed_hashes))
+ mean = statistics.mean(times)
+ median = statistics.median(times)
+ stddev = statistics.pstdev(times)
+
+ print(f"Number of clients: {args.clients}")
+ print(f"Requests per client: {args.requests}")
+ print(f"Number of requests: {len(times)}")
+ print(f"Total elapsed time: {total_elapsed:.3f}s")
+ print(f"Total request rate: {len(times)/total_elapsed:.3f} req/s")
+ print(f"Average request time: {mean:.3f}s")
+ print(f"Median request time: {median:.3f}s")
+ print(f"Request time std dev: {stddev:.3f}s")
+ print(f"Maximum request time: {max(times):.3f}s")
+ print(f"Minimum request time: {min(times):.3f}s")
+ print(f"Hashes found: {found_hashes}")
+ print(f"Hashes missed: {missed_hashes}")
if args.report:
with ProgressBar(total=args.requests) as pbar:
@@ -212,6 +235,27 @@ def main():
print("New hashes marked: %d" % result["count"])
return 0
+ def handle_gc_mark_stream(args, client):
+ stdin = (l.strip() for l in sys.stdin)
+ marked_hashes = 0
+
+ try:
+ result = client.gc_mark_batch(args.mark, stdin)
+ marked_hashes = result["count"]
+ except ConnectionError:
+ logger.warning(
+ "Server doesn't seem to support `gc-mark-stream`. Sending "
+ "hashes sequentially using `gc-mark` API."
+ )
+ for line in stdin:
+ pairs = line.split()
+ condition = dict(zip(pairs[::2], pairs[1::2]))
+ result = client.gc_mark(args.mark, condition)
+ marked_hashes += result["count"]
+
+ print("New hashes marked: %d" % marked_hashes)
+ return 0
+
def handle_gc_sweep(args, client):
result = client.gc_sweep(args.mark)
print("Removed %d rows" % result["count"])
@@ -225,7 +269,45 @@ def main():
print("true" if result else "false")
return 0
- parser = argparse.ArgumentParser(description='Hash Equivalence Client')
+ def handle_ping(args, client):
+ times = []
+ for i in range(1, args.count + 1):
+ if not args.quiet:
+ print(f"Ping {i} of {args.count}... ", end="")
+ start_time = time.perf_counter()
+ client.ping()
+ elapsed = time.perf_counter() - start_time
+ times.append(elapsed)
+ if not args.quiet:
+ print(f"{elapsed:.3f}s")
+
+ mean = statistics.mean(times)
+ median = statistics.median(times)
+ std_dev = statistics.pstdev(times)
+
+ if not args.quiet:
+ print("------------------------")
+ print(f"Number of pings: {len(times)}")
+ print(f"Average round trip time: {mean:.3f}s")
+ print(f"Median round trip time: {median:.3f}s")
+ print(f"Round trip time std dev: {std_dev:.3f}s")
+ print(f"Min time is: {min(times):.3f}s")
+ print(f"Max time is: {max(times):.3f}s")
+ return 0
+
+ parser = argparse.ArgumentParser(
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ description='Hash Equivalence Client',
+ epilog=textwrap.dedent(
+ """
+ Possible ADDRESS options are:
+ unix://PATH Connect to UNIX domain socket at PATH
+ ws://HOST[:PORT] Connect to websocket at HOST:PORT (default port is 80)
+ wss://HOST[:PORT] Connect to secure websocket at HOST:PORT (default port is 443)
+ HOST:PORT Connect to TCP server at HOST:PORT
+ """
+ ),
+ )
parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")')
parser.add_argument('--log', default='WARNING', help='Set logging level')
parser.add_argument('--login', '-l', metavar="USERNAME", help="Authenticate as USERNAME")
@@ -313,6 +395,16 @@ def main():
help="Keep entries in table where KEY == VALUE")
gc_mark_parser.set_defaults(func=handle_gc_mark)
+ gc_mark_parser_stream = subparsers.add_parser(
+ 'gc-mark-stream',
+ help=(
+ "Mark multiple hashes to be retained for garbage collection. Input should be provided via stdin, "
+ "with each line formatted as key-value pairs separated by spaces, for example 'column1 foo column2 bar'."
+ )
+ )
+ gc_mark_parser_stream.add_argument("mark", help="Mark for this garbage collection operation")
+ gc_mark_parser_stream.set_defaults(func=handle_gc_mark_stream)
+
gc_sweep_parser = subparsers.add_parser('gc-sweep', help="Perform garbage collection and delete any entries that are not marked")
gc_sweep_parser.add_argument("mark", help="Mark for this garbage collection operation")
gc_sweep_parser.set_defaults(func=handle_gc_sweep)
@@ -322,6 +414,11 @@ def main():
unihash_exists_parser.add_argument("unihash", help="Unihash to check")
unihash_exists_parser.set_defaults(func=handle_unihash_exists)
+ ping_parser = subparsers.add_parser('ping', help="Ping server")
+ ping_parser.add_argument("-n", "--count", type=int, help="Number of pings. Default is %(default)s", default=10)
+ ping_parser.add_argument("-q", "--quiet", action="store_true", help="Don't print each ping; only print results")
+ ping_parser.set_defaults(func=handle_ping)
+
args = parser.parse_args()
logger = logging.getLogger('hashserv')
diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv
index 4bfb7abf..3e68b6a2 100755
--- a/bitbake/bin/bitbake-hashserv
+++ b/bitbake/bin/bitbake-hashserv
@@ -12,6 +12,8 @@ import argparse
import sqlite3
import warnings
+__version__ = "2.20.0"
+
warnings.simplefilter("default")
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib"))
@@ -19,14 +21,15 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib
import hashserv
from hashserv.server import DEFAULT_ANON_PERMS
-VERSION = "1.0.0"
+if __version__ != hashserv.__version__:
+ sys.exit("Bitbake library hashserv version and program version mismatch!")
DEFAULT_BIND = "unix://./hashserve.sock"
def main():
parser = argparse.ArgumentParser(
- description="Hash Equivalence Reference Server. Version=%s" % VERSION,
+ description="Hash Equivalence Reference Server. Version=%s" % __version__,
formatter_class=argparse.RawTextHelpFormatter,
epilog="""
The bind address may take one of the following formats:
@@ -125,6 +128,11 @@ The following permissions are supported by the server:
default=os.environ.get("HASHSERVER_ADMIN_PASSWORD", None),
help="Create default admin user with password ADMIN_PASSWORD ($HASHSERVER_ADMIN_PASSWORD)",
)
+ parser.add_argument(
+ "--reuseport",
+ action="store_true",
+ help="Enable SO_REUSEPORT, allowing multiple servers to bind to the same port for load balancing",
+ )
args = parser.parse_args()
@@ -132,7 +140,9 @@ The following permissions are supported by the server:
level = getattr(logging, args.log.upper(), None)
if not isinstance(level, int):
- raise ValueError("Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log)
+ raise ValueError(
+ "Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.log
+ )
logger.setLevel(level)
console = logging.StreamHandler()
@@ -155,6 +165,7 @@ The following permissions are supported by the server:
anon_perms=anon_perms,
admin_username=args.admin_user,
admin_password=args.admin_password,
+ reuseport=args.reuseport,
)
server.serve_forever()
return 0
diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
index d4b1d1aa..c1eafb22 100755
--- a/bitbake/bin/bitbake-layers
+++ b/bitbake/bin/bitbake-layers
@@ -17,25 +17,38 @@ import argparse
import warnings
warnings.simplefilter("default")
+__version__ = "2.20.0"
+
bindir = os.path.dirname(__file__)
+toolname = os.path.basename(__file__).split(".")[0]
topdir = os.path.dirname(bindir)
sys.path[0:0] = [os.path.join(topdir, 'lib')]
import bb.tinfoil
+import bb.cooker
import bb.msg
+from bb import __version__ as libbb_version
+
+if __version__ != libbb_version:
+ sys.exit("Bitbake library version and program version mismatch!")
-logger = bb.msg.logger_create('bitbake-layers', sys.stdout)
+logger = bb.msg.logger_create(toolname, sys.stdout)
def main():
parser = argparse.ArgumentParser(
- description="BitBake layers utility",
+ description={"bitbake-layers":"BitBake layers utility", "bitbake-config-build":"BitBake build configuration utility"}[toolname],
epilog="Use %(prog)s <subcommand> --help to get help on a specific command",
add_help=False)
parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
- parser.add_argument('-F', '--force', help='Force add without recipe parse verification', action='store_true')
+ if toolname == "bitbake-layers":
+ parser.add_argument('-F', '--force', help='Forced execution: can be specified multiple times. -F will force add without recipe parse verification and -FF will additionally force the run without layer parsing.', action='count')
parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
+ # Set the default for --force here rather than above so that it is always
+ # defined even if the option is not available.
+ parser.set_defaults(force=0)
+
global_args, unparsed_args = parser.parse_known_args()
# Help is added here rather than via add_help=True, as we don't want it to
@@ -57,18 +70,27 @@ def main():
level=logger.getEffectiveLevel())
plugins = []
- tinfoil = bb.tinfoil.Tinfoil(tracking=True)
- tinfoil.logger.setLevel(logger.getEffectiveLevel())
- try:
- tinfoil.prepare(True)
- for path in ([topdir] +
- tinfoil.config_data.getVar('BBPATH').split(':')):
- pluginpath = os.path.join(path, 'lib', 'bblayers')
+ with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
+ tinfoil.logger.setLevel(logger.getEffectiveLevel())
+
+ if global_args.force > 1:
+ bbpaths = []
+ else:
+ if toolname == "bitbake-config-build":
+ extra_features = [bb.cooker.CookerFeatures.SKIP_FRAGMENTS]
+ else:
+ extra_features = None
+ tinfoil.prepare(True, extra_features=extra_features)
+ bbpaths = tinfoil.config_data.getVar('BBPATH').split(':')
+
+ for path in ([topdir] + bbpaths):
+ pluginbasepath = {"bitbake-layers":'bblayers', 'bitbake-config-build':'bbconfigbuild'}[toolname]
+ pluginpath = os.path.join(path, 'lib', pluginbasepath)
bb.utils.load_plugins(logger, plugins, pluginpath)
registered = False
for plugin in plugins:
- if hasattr(plugin, 'tinfoil_init'):
+ if hasattr(plugin, 'tinfoil_init') and global_args.force <= 1:
plugin.tinfoil_init(tinfoil)
if hasattr(plugin, 'register_commands'):
registered = True
@@ -86,8 +108,6 @@ def main():
tinfoil.config_data.enableTracking()
return args.func(args)
- finally:
- tinfoil.shutdown()
if __name__ == "__main__":
diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
index ad0a0694..8ba246c0 100755
--- a/bitbake/bin/bitbake-prserv
+++ b/bitbake/bin/bitbake-prserv
@@ -11,19 +11,30 @@ import argparse
import warnings
warnings.simplefilter("default")
+__version__ = "2.20.0"
+
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), "lib"))
import prserv
import prserv.serv
-
-VERSION = "1.1.0"
+
+if __name__ == "__main__":
+ if __version__ != prserv.__version__:
+ sys.exit("Bitbake prserv library version and program version mismatch!")
PRHOST_DEFAULT="0.0.0.0"
PRPORT_DEFAULT=8585
+def init_logger(logfile, loglevel):
+ numeric_level = getattr(logging, loglevel.upper(), None)
+ if not isinstance(numeric_level, int):
+ raise ValueError("Invalid log level: %s" % loglevel)
+ FORMAT = "%(asctime)-15s %(message)s"
+ logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
+
def main():
parser = argparse.ArgumentParser(
- description="BitBake PR Server. Version=%s" % VERSION,
+ description="BitBake PR Server. Version=%s" % __version__,
formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument(
@@ -70,12 +81,25 @@ def main():
action="store_true",
help="open database in read-only mode",
)
+ parser.add_argument(
+ "-u",
+ "--upstream",
+ default=os.environ.get("PRSERV_UPSTREAM", None),
+ help="Upstream PR service (host:port)",
+ )
args = parser.parse_args()
- prserv.init_logger(os.path.abspath(args.log), args.loglevel)
+ init_logger(os.path.abspath(args.log), args.loglevel)
if args.start:
- ret=prserv.serv.start_daemon(args.file, args.host, args.port, os.path.abspath(args.log), args.read_only)
+ ret=prserv.serv.start_daemon(
+ args.file,
+ args.host,
+ args.port,
+ os.path.abspath(args.log),
+ args.read_only,
+ args.upstream
+ )
elif args.stop:
ret=prserv.serv.stop_daemon(args.host, args.port)
else:
diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest
index f25f23b1..855abb1f 100755
--- a/bitbake/bin/bitbake-selftest
+++ b/bitbake/bin/bitbake-selftest
@@ -11,14 +11,20 @@ import warnings
warnings.simplefilter("default")
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+__version__ = "2.20.0"
+
import unittest
try:
import bb
import hashserv
+ import prserv
import layerindexlib
except RuntimeError as exc:
sys.exit(str(exc))
+if __version__ != bb.__version__:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
tests = ["bb.tests.codeparser",
"bb.tests.color",
"bb.tests.cooker",
@@ -26,13 +32,16 @@ tests = ["bb.tests.codeparser",
"bb.tests.data",
"bb.tests.event",
"bb.tests.fetch",
+ "bb.tests.fetch_import",
"bb.tests.parse",
- "bb.tests.persist_data",
"bb.tests.runqueue",
+ "bb.tests.setup",
"bb.tests.siggen",
"bb.tests.utils",
"bb.tests.compression",
+ "bb.tests.filter",
"hashserv.tests",
+ "prserv.tests",
"layerindexlib.tests.layerindexobj",
"layerindexlib.tests.restapi",
"layerindexlib.tests.cooker"]
@@ -64,6 +73,8 @@ ENV_HELP = """\
Environment variables:
BB_SKIP_NETTESTS set to 'yes' in order to skip tests using network
connection
+ BB_SKIP_PYPI_TESTS set to 'no' to run PyPI packaging tests
+ (default: yes/skip)
BB_TMPDIR_NOCLEAN set to 'yes' to preserve test tmp directories
"""
diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server
index 454a3919..6157fb33 100755
--- a/bitbake/bin/bitbake-server
+++ b/bitbake/bin/bitbake-server
@@ -12,6 +12,8 @@ warnings.simplefilter("default")
import logging
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+__version__ = "2.20.0"
+
import bb
bb.utils.check_system_locale()
@@ -23,13 +25,17 @@ if len(sys.argv) != 11 or not sys.argv[1].startswith("decafbad"):
import bb.server.process
+if __name__ == "__main__":
+ if __version__ != bb.__version__:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
lockfd = int(sys.argv[2])
readypipeinfd = int(sys.argv[3])
logfile = sys.argv[4]
lockname = sys.argv[5]
sockname = sys.argv[6]
timeout = float(sys.argv[7])
-profile = bool(int(sys.argv[8]))
+profile = sys.argv[8]
xmlrpcinterface = (sys.argv[9], int(sys.argv[10]))
if xmlrpcinterface[0] == "None":
xmlrpcinterface = (None, xmlrpcinterface[1])
@@ -38,9 +44,9 @@ if xmlrpcinterface[0] == "None":
with open('/dev/null', 'r') as si:
os.dup2(si.fileno(), sys.stdin.fileno())
-so = open(logfile, 'a+')
-os.dup2(so.fileno(), sys.stdout.fileno())
-os.dup2(so.fileno(), sys.stderr.fileno())
+with open(logfile, 'a+') as so:
+ os.dup2(so.fileno(), sys.stdout.fileno())
+ os.dup2(so.fileno(), sys.stderr.fileno())
# Have stdout and stderr be the same so log output matches chronologically
# and there aren't two seperate buffers
diff --git a/bitbake/bin/bitbake-setup b/bitbake/bin/bitbake-setup
new file mode 100755
index 00000000..e50117ea
--- /dev/null
+++ b/bitbake/bin/bitbake-setup
@@ -0,0 +1,1481 @@
+#!/usr/bin/env python3
+
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import argparse
+import configparser
+import copy
+import datetime
+import functools
+import glob
+import json
+import logging
+import os
+import shutil
+import signal
+import string
+import subprocess
+import sys
+import textwrap
+import time
+
+__version__ = "2.20.0"
+
+bindir = os.path.abspath(os.path.dirname(__file__))
+sys.path[0:0] = [os.path.join(os.path.dirname(bindir), 'lib')]
+
+import bb.msg # noqa: E402
+import bb.fetch # noqa: E402
+import bb.process # noqa: E402
+from bb import __version__ as libbb_version # noqa: E402
+
+if __name__ == "__main__":
+ if __version__ != libbb_version:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
+logger = bb.msg.logger_create('bitbake-setup', sys.stdout)
+
+# These settings can only be set in the global context
+GLOBAL_ONLY_SETTINGS = (
+ "top-dir-prefix",
+ "top-dir-name",
+)
+
+def color_enabled() -> bool:
+ """
+ Our logger has a BBLogFormatter formatter which holds whether color is
+ enabled or not. Return this value.
+ """
+ return logger.handlers[0].formatter.color_enabled
+
+def get_diff_color_param() -> str:
+ return "--color=always" if color_enabled() else "--color=never"
+
+def print_configs(prompt: str, choices: list[str], descriptions: list[str] = []):
+ """
+ Helper function to print a list of choices and align the output.
+ Each option name is made bold to stand out, unless color is not enabled in
+ our logger.
+ """
+ if not prompt.endswith(':'):
+ prompt += ":"
+ logger.plain(prompt)
+
+ if not descriptions:
+ descriptions = ["" for _ in choices]
+
+ # maximum size of all choices, for alignment
+ cmax = max([len(c) for c in choices]) + 1
+
+ for n, c in enumerate(choices):
+ msg = f"{n + 1}. "
+ if color_enabled():
+ # make it bold
+ msg += "\033[1m"
+ msg += f"{c:<{cmax}}"
+ if color_enabled():
+ msg += "\033[0m"
+ msg += f" {descriptions[n]}"
+ logger.plain(msg)
+
+# If bitbake is from a release tarball or somewhere like pypi where
+# updates may not be straightforward, prefer to use the git repo as the
+# default registry
+def get_default_registry():
+ internal_registry = os.path.normpath(os.path.dirname(__file__) + "/../default-registry")
+ git_registry = "git://git.openembedded.org/bitbake;protocol=https;branch=master;rev=master"
+ if os.path.exists(os.path.dirname(__file__) + "/../.git"):
+ return internal_registry
+ else:
+ return git_registry
+
+def cache_dir(top_dir):
+ return os.path.join(top_dir, '.bitbake-setup-cache')
+
+def init_bb_cache(top_dir, settings, args):
+ dldir = settings["default"]["dl-dir"]
+ bb_cachedir = os.path.join(cache_dir(top_dir), 'bitbake-cache')
+ os.makedirs(bb_cachedir, exist_ok=True)
+
+ d = bb.data.init()
+ d.setVar("DL_DIR", dldir)
+ d.setVar("BB_CACHEDIR", bb_cachedir)
+ d.setVar("__BBSRCREV_SEEN", "1")
+ if args.no_network:
+ d.setVar("BB_SRCREV_POLICY", "cache")
+ bb.fetch.fetcher_init(d)
+ return d
+
+def save_bb_cache():
+ bb.fetch.fetcher_parse_save()
+ bb.fetch.fetcher_parse_done()
+
+def get_config_name(config):
+ suffix = '.conf.json'
+ config_file = os.path.basename(config)
+ if config_file.endswith(suffix):
+ return config_file[:-len(suffix)]
+ else:
+ raise Exception("Config file {} does not end with {}, please rename the file.".format(config, suffix))
+
+def write_upstream_config(config_dir, config_data):
+ with open(os.path.join(config_dir, "config-upstream.json"),'w') as s:
+ json.dump(config_data, s, sort_keys=True, indent=4)
+
+def write_sources_fixed_revisions(config_dir, layer_dir, config_data):
+ json_path = os.path.join(config_dir, "sources-fixed-revisions.json")
+ json_link = os.path.join(layer_dir, "sources-fixed-revisions.json")
+ sources = {}
+ sources['sources'] = config_data
+ with open(os.path.join(config_dir, "sources-fixed-revisions.json"),'w') as s:
+ json.dump(sources, s, sort_keys=True, indent=4)
+ if not os.path.lexists(json_link):
+ os.symlink(os.path.relpath(json_path ,layer_dir), json_link)
+
+def commit_config(config_dir):
+ bb.process.run(['git', '-C', config_dir, 'add', '.'])
+ bb.process.run(['git', '-C', config_dir, 'commit', '--allow-empty', '--no-verify', '-a', '-m', 'Configuration at ' + time.asctime()])
+
+def _write_layer_list(dest, repodirs):
+ layers = []
+ for r in repodirs:
+ for root, _, files in os.walk(os.path.join(dest,r)):
+ if os.path.basename(root) == 'conf' and 'layer.conf' in files:
+ layers.append(os.path.relpath(os.path.dirname(root), dest))
+ layers_f = os.path.join(dest, ".oe-layers.json")
+ with open(layers_f, 'w') as f:
+ json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4)
+
+def add_unique_timestamp_to_path(path):
+ timestamp = time.strftime("%Y%m%d%H%M%S")
+ path_unique = "{}.{}".format(path, timestamp)
+ if os.path.exists(path_unique):
+ import itertools
+ for i in itertools.count(start=1):
+ path_unique = "{}.{}.{}".format(path, timestamp, i)
+ if not os.path.exists(path_unique):
+ break
+ return path_unique
+
+def _get_remotes(r_remote):
+ remotes = []
+
+ if 'remotes' not in r_remote and 'uri' not in r_remote:
+ raise Exception("Expected key(s): 'remotes', 'uri'")
+
+ if 'remotes' in r_remote:
+ for remote in r_remote['remotes']:
+ remotes.append(r_remote['remotes'][remote]['uri'])
+
+ if 'uri' in r_remote:
+ remotes.append(r_remote['uri'])
+
+ return remotes
+
+def checkout_layers(layers, confdir, layerdir, d, rebase_conflicts_strategy='abort'):
+ def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions):
+ rev = r_remote['rev']
+ branch = r_remote.get('branch', None)
+
+ remotes = _get_remotes(r_remote)
+
+ for remote in remotes:
+ prot,host,path,user,pswd,params = bb.fetch.decodeurl(remote)
+ fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params))
+ logger.plain(" {}".format(r_name))
+ if branch:
+ src_uri = f"{fetchuri};protocol={prot};rev={rev};branch={branch};destsuffix={repodir}"
+ else:
+ src_uri = f"{fetchuri};protocol={prot};rev={rev};nobranch=1;destsuffix={repodir}"
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ repodir_path = os.path.join(layerdir, repodir)
+ try:
+ do_fetch(fetcher, layerdir)
+ except (bb.fetch.LocalModificationsError, bb.fetch.RebaseError) as e:
+ if rebase_conflicts_strategy != 'backup':
+ e.msg += ("\nUse 'bitbake-setup update --rebase-conflicts-strategy=backup'"
+ " to automatically back up the directory and re-clone from upstream,"
+ " or use 'bitbake-setup init -L %s /path/to/local/checkout'"
+ " to work with a local checkout instead." % r_name)
+ raise
+ backup_path = add_unique_timestamp_to_path(repodir_path + '-backup')
+ logger.warning(
+ "%s\n"
+ "Renaming %s to %s to preserve your work, then re-cloning from upstream.",
+ e, repodir_path, backup_path)
+ os.rename(repodir_path, backup_path)
+ fetcher.unpack(layerdir)
+ urldata = fetcher.ud[src_uri]
+ revision = urldata.revision
+ layers_fixed_revisions[r_name]['git-remote']['rev'] = revision
+
+ def _symlink_local(src, dst):
+ logger.plain("Making a symbolic link {} pointing to {}".format(dst, src))
+ os.symlink(src, dst)
+
+ layers_fixed_revisions = copy.deepcopy(layers)
+ repodirs = []
+ oesetupbuild = None
+ logger.plain("Fetching layer/tool repositories into {}".format(layerdir))
+ for r_name in layers:
+ r_data = layers[r_name]
+ repodir = r_data.get("path", r_name)
+ repodirs.append(repodir)
+
+ r_remote = r_data.get('git-remote')
+ r_local = r_data.get('local')
+ if r_remote and r_local:
+ raise Exception("Source {} contains both git-remote and local properties.".format(r_name))
+
+ repodir_path = os.path.join(layerdir, repodir)
+ if os.path.lexists(repodir_path):
+ if os.path.islink(repodir_path):
+ os.remove(repodir_path)
+ elif r_local:
+ backup_path = add_unique_timestamp_to_path(repodir_path + '-backup')
+ logger.warning("""Source {} in {} contains local modifications. Renaming to {} to preserve them.
+For local development work it is recommended to clone the needed layers separately and re-initialize using -L option:
+bitbake-setup init -L {} /path/to/repo/checkout""".format(
+ r_name, repodir_path, backup_path, r_name))
+ os.rename(repodir_path, backup_path)
+
+ if r_remote:
+ _checkout_git_remote(r_remote, repodir, layers_fixed_revisions)
+ if r_local:
+ _symlink_local(os.path.expanduser(r_local["path"]), repodir_path)
+
+ if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setup-build')):
+ oesetupbuild = os.path.join(layerdir, repodir, 'scripts/oe-setup-build')
+ oeinitbuildenvdir = os.path.join(layerdir, repodir)
+
+ logger.plain(" ")
+ _write_layer_list(layerdir, repodirs)
+
+ if oesetupbuild:
+ links = {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirname(oesetupbuild), 'oe-init-build-env-dir': oeinitbuildenvdir}
+ for link,item in links.items():
+ symlink = os.path.join(layerdir, link)
+ if os.path.lexists(symlink):
+ os.remove(symlink)
+ os.symlink(os.path.relpath(item,layerdir),symlink)
+
+ return layers_fixed_revisions
+
+def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode=False):
+ def _setup_build_conf(layers, filerelative_layers, build_conf_dir):
+ os.makedirs(build_conf_dir)
+ layers_s = []
+
+ for layer in layers:
+ layer = os.path.join(layerdir, layer)
+ layers_s.append(" {} \\".format(layer))
+
+ for layer in filerelative_layers:
+ if thisdir:
+ layer = os.path.join(thisdir, layer)
+ else:
+ raise Exception("Configuration is using bb-layers-file-relative to specify " \
+ "a layer path relative to itself. This can be done only " \
+ "when the configuration is specified by its path on local " \
+ "disk, not when it's in a registry or is fetched over http.")
+ layers_s.append(" {} \\".format(layer))
+
+ layers_s = "\n".join(layers_s)
+ bblayers_conf = """BBLAYERS ?= " \\
+{}
+ "
+""".format(layers_s)
+ with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f:
+ f.write(bblayers_conf)
+
+ local_conf = """#
+# This file is intended for local configuration tweaks.
+#
+# If you would like to publish and share changes made to this file,
+# it is recommended to put them into a distro config, or to create
+# layer fragments from changes made here.
+#
+"""
+ with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f:
+ f.write(local_conf)
+
+ with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') as f:
+ f.write("")
+
+ with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') as f:
+ f.write(bitbake_config["description"] + "\n")
+
+ with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f:
+ notes = bitbake_config.get("notes")
+ if isinstance(notes, list):
+ notes = "\n".join(notes)
+ f.write(notes + "\n" if notes else "")
+
+ def _make_init_build_env(builddir, oeinitbuildenvdir):
+ builddir = os.path.realpath(builddir)
+ cmd = 'cd {}\nset {}\n. ./oe-init-build-env\nset --\n'.format(oeinitbuildenvdir, builddir)
+ initbuild_in_builddir = os.path.join(builddir, 'init-build-env')
+
+ with open(initbuild_in_builddir, 'w') as f:
+ f.write("# init-build-env wrapper created by bitbake-setup\n")
+ f.write(cmd + '\n')
+
+ def _prepend_passthrough_to_init_build_env(builddir):
+ env = bitbake_config.get("bb-env-passthrough-additions")
+ if not env:
+ return
+
+ initbuild_in_builddir = os.path.join(builddir, 'init-build-env')
+ with open(initbuild_in_builddir) as f:
+ content = f.read()
+
+ joined = " \\\n".join(env)
+ env = "export BB_ENV_PASSTHROUGH_ADDITIONS=\" \\\n"
+ env += "${BB_ENV_PASSTHROUGH_ADDITIONS} \\\n"
+ env += joined
+ env += '"'
+
+ with open(initbuild_in_builddir, 'w') as f:
+ f.write("# environment passthrough added by bitbake-setup\n")
+ f.write(env + '\n')
+ f.write('\n')
+ f.write(content)
+
+ bitbake_builddir = os.path.join(setupdir, "build")
+ logger.plain("Setting up bitbake configuration in\n {}\n".format(bitbake_builddir))
+
+ template = bitbake_config.get("oe-template")
+ layers = bitbake_config.get("bb-layers")
+ if not template and not layers:
+ logger.error("Bitbake configuration does not contain a reference to an OpenEmbedded build template via 'oe-template' or a list of layers via 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechanism manually to complete the setup.")
+ return
+ oesetupbuild = os.path.join(layerdir, 'setup-build')
+ if template and not os.path.exists(oesetupbuild):
+ raise Exception("Cannot complete setting up a bitbake build directory from OpenEmbedded template '{}' as oe-setup-build was not found in any layers; please use oe-init-build-env manually.".format(template))
+
+ bitbake_confdir = os.path.join(bitbake_builddir, 'conf')
+ backup_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-backup'))
+ upstream_bitbake_confdir = add_unique_timestamp_to_path(os.path.join(bitbake_builddir, 'conf-upstream'))
+
+ if os.path.exists(bitbake_confdir):
+ os.rename(bitbake_confdir, backup_bitbake_confdir)
+
+ if layers:
+ filerelative_layers = bitbake_config.get("bb-layers-file-relative") or []
+ _setup_build_conf(layers, filerelative_layers, bitbake_confdir)
+
+ if template:
+ bb.process.run([oesetupbuild, "setup", "-c", template, "-b", bitbake_builddir, "--no-shell"])
+ else:
+ oeinitbuildenvdir = os.path.join(layerdir, 'oe-init-build-env-dir')
+ if not os.path.exists(os.path.join(oeinitbuildenvdir, "oe-init-build-env")):
+ logger.error("Could not find oe-init-build-env in any of the layers; please use another mechanism to initialize the bitbake environment")
+ return
+ _make_init_build_env(bitbake_builddir, os.path.realpath(oeinitbuildenvdir))
+
+ _prepend_passthrough_to_init_build_env(bitbake_builddir)
+
+ siteconf_symlink = os.path.join(bitbake_confdir, "site.conf")
+ siteconf = os.path.normpath(os.path.join(setupdir, '..', "site.conf"))
+ if os.path.lexists(siteconf_symlink):
+ os.remove(siteconf_symlink)
+ os.symlink(os.path.relpath(siteconf, bitbake_confdir), siteconf_symlink)
+
+
+ init_script = os.path.join(bitbake_builddir, "init-build-env")
+ workspace_file = os.path.join(setupdir, "bitbake.code-workspace")
+ shell = "bash"
+ # oe-fragments entries can be strings or dicts with 'name'/'description' keys
+ oe_fragments = [f["name"] if isinstance(f, dict) else f for f in bitbake_config.get("oe-fragments", [])]
+ fragments = oe_fragments + sorted(bitbake_config.get("oe-fragment-choices",{}).values())
+ if fragments:
+ bb.process.run("{} -c '. {} && bitbake-config-build enable-fragment {}'".format(shell, init_script, " ".join(fragments)))
+
+ if os.path.exists(backup_bitbake_confdir):
+ conf_diff = get_diff(backup_bitbake_confdir, bitbake_confdir)
+ if not conf_diff:
+ logger.plain('New bitbake configuration from upstream is the same as the current one, no need to update it.')
+ shutil.rmtree(bitbake_confdir)
+ os.rename(backup_bitbake_confdir, bitbake_confdir)
+ if init_vscode or os.path.exists(workspace_file):
+ configure_vscode(setupdir, layerdir, bitbake_builddir, init_script)
+ return
+
+ logger.plain('Upstream bitbake configuration changes were found:')
+ logger.plain(conf_diff)
+
+ if update_bb_conf == 'prompt':
+ y_or_n = input('Apply these changes to the current configuration? (y/N): ')
+ if y_or_n != 'y':
+ update_bb_conf = 'no'
+
+ if update_bb_conf == 'no':
+ logger.plain('Ignoring upstream bitbake configuration changes')
+ logger.plain(f'Leaving the upstream configuration in {upstream_bitbake_confdir}')
+ os.rename(bitbake_confdir, upstream_bitbake_confdir)
+ os.rename(backup_bitbake_confdir, bitbake_confdir)
+ if init_vscode or os.path.exists(workspace_file):
+ configure_vscode(setupdir, layerdir, bitbake_builddir, init_script)
+ return
+
+ logger.plain('Applying upstream bitbake configuration changes')
+ logger.plain(f'Leaving the previous configuration in {backup_bitbake_confdir}')
+
+ fragment_note = "Run 'bitbake-config-build enable-fragment <fragment-name>' to enable additional fragments or replace built-in ones (e.g. machine/<name> or distro/<name> to change MACHINE or DISTRO)."
+
+ readme_extra = ""
+ if init_vscode:
+ readme_extra = "\nTo edit the code in VSCode, open the workspace: code {}\n".format(workspace_file)
+
+ readme = """{}\n\nAdditional information is in {} and {}\n
+Source the environment using '. {}' to run builds from the command line.\n
+{}\n
+The bitbake configuration files (local.conf, bblayers.conf and more) can be found in {}/conf
+{}""".format(
+ bitbake_config["description"],
+ os.path.join(bitbake_builddir,'conf/conf-summary.txt'),
+ os.path.join(bitbake_builddir,'conf/conf-notes.txt'),
+ init_script,
+ fragment_note,
+ bitbake_builddir,
+ readme_extra
+ )
+ readme_file = os.path.join(bitbake_builddir, "README")
+ with open(readme_file, 'w') as f:
+ f.write(readme)
+
+ logger.plain("This bitbake configuration provides:\n {}\n".format(bitbake_config["description"]))
+ logger.plain("Usage instructions and additional information are in\n {}\n".format(readme_file))
+ logger.plain("To run builds, source the environment using\n . {}\n".format(init_script))
+ logger.plain("{}\n".format(fragment_note))
+ logger.plain("The bitbake configuration files (local.conf, bblayers.conf and more) can be found in\n {}/conf\n".format(bitbake_builddir))
+ if init_vscode:
+ logger.plain("To edit the code in VSCode, open the workspace:\n code {}\n".format(workspace_file))
+
+ if init_vscode or os.path.exists(workspace_file):
+ configure_vscode(setupdir, layerdir, bitbake_builddir, init_script)
+
+def get_registry_config(registry_path, id):
+ for root, _, files in os.walk(registry_path):
+ for file in files:
+ if file.endswith('.conf.json') and id == get_config_name(file):
+ return os.path.join(root, file)
+ raise Exception("Unable to find {} in available configurations; use 'list' sub-command to see what is available".format(id))
+
+def merge_overrides_into_sources(sources, overrides):
+ layers = copy.deepcopy(sources)
+ for k,v in overrides.items():
+ if k in layers:
+ layers[k] = v
+ return layers
+
+def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf="prompt", init_vscode=False, rebase_conflicts_strategy='abort'):
+ layer_config = merge_overrides_into_sources(config["data"]["sources"], config["source-overrides"]["sources"])
+ sources_fixed_revisions = checkout_layers(layer_config, confdir, layerdir, d, rebase_conflicts_strategy=rebase_conflicts_strategy)
+ bitbake_config = config["bitbake-config"]
+ thisdir = os.path.dirname(config["path"]) if config["type"] == 'local' else None
+ setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_conf, init_vscode)
+ write_sources_fixed_revisions(confdir, layerdir, sources_fixed_revisions)
+ commit_config(confdir)
+
+def int_input(allowed_values, prompt=''):
+ n = None
+ while n is None:
+ try:
+ n = int(input(prompt))
+ except ValueError:
+ prompt = 'Not a valid number, please try again: '
+ continue
+ if n not in allowed_values:
+ prompt = 'Number {} not one of {}, please try again: '.format(n, allowed_values)
+ n = None
+ return n
+
+def flatten_bitbake_configs(configs):
+ def merge_configs(c1,c2):
+ c_merged = {}
+ for k,v in c2.items():
+ if k not in c1.keys():
+ c_merged[k] = v
+ for k,v in c1.items():
+ if k not in c2.keys():
+ c_merged[k] = v
+ else:
+ c_merged[k] = c1[k] + c2[k]
+ del c_merged['configurations']
+ return c_merged
+
+ flattened_configs = []
+ for c in configs:
+ if 'configurations' not in c:
+ flattened_configs.append(c)
+ else:
+ for sub_c in flatten_bitbake_configs(c['configurations']):
+ flattened_configs.append(merge_configs(c, sub_c))
+ return flattened_configs
+
+def choose_bitbake_config(configs, parameters, non_interactive):
+ flattened_configs = flatten_bitbake_configs(configs)
+ configs_dict = {i["name"]:i for i in flattened_configs}
+
+ if parameters:
+ config_id = parameters[0]
+ if config_id not in configs_dict:
+ raise Exception("Bitbake configuration {} not found; replace with one of {}".format(config_id, configs_dict))
+ return configs_dict[config_id]
+
+ enumerated_configs = list(enumerate(flattened_configs, 1))
+ if len(enumerated_configs) == 1:
+ only_config = flattened_configs[0]
+ logger.plain("\nSelecting the only available bitbake configuration {}".format(only_config["name"]))
+ return only_config
+
+ if non_interactive:
+ raise Exception("Unable to choose from bitbake configurations in non-interactive mode: {}".format(configs_dict))
+
+ logger.plain("")
+ print_configs("Available bitbake configurations",
+ [c["name"] for c in flattened_configs],
+ [c["description"] for c in flattened_configs])
+ config_n = int_input([i[0] for i in enumerated_configs],
+ "\nPlease select one of the above bitbake configurations by its number: ") - 1
+ return flattened_configs[config_n]
+
+def choose_config(configs, non_interactive):
+ not_expired_configs = [k for k in sorted(configs.keys()) if not has_expired(configs[k].get("expires", None))]
+ if len(not_expired_configs) == 1:
+ only_config = not_expired_configs[0]
+ logger.plain("\nSelecting the only available configuration {}\n".format(only_config))
+ return only_config
+
+ if non_interactive:
+ raise Exception("Unable to choose from configurations in non-interactive mode: {}".format(not_expired_configs))
+
+ descs = []
+ for c in not_expired_configs:
+ d = configs[c]["description"]
+ expiry_date = configs[c].get("expires", None)
+ if expiry_date:
+ d += f" (supported until {expiry_date})"
+ descs.append(d)
+
+ logger.plain("")
+ print_configs("Available Configuration Templates",
+ [c for c in not_expired_configs],
+ descs)
+ config_n = int_input([i[0] for i in list(enumerate(not_expired_configs, 1))],
+ "\nPlease select one of the above configurations by its number: ") - 1
+ return not_expired_configs[config_n]
+
+def choose_fragments(possibilities, parameters, non_interactive, skip_selection):
+ choices = {}
+ for k,v in possibilities.items():
+ if skip_selection and k in skip_selection:
+ logger.info("Skipping a selection of {}, as requested on command line. The resulting bitbake configuration may require further manual adjustments.".format(k))
+ continue
+
+ # options can be a list of strings or a list of dicts
+ options = v["options"]
+ if len(options) > 0 and isinstance(v["options"][0], str):
+ options = [{"name": o, "description": ""} for o in v["options"]]
+
+ choice = [o["name"] for o in options if o["name"] in parameters]
+ if len(choice) > 1:
+ raise Exception("Options specified on command line do not allow a single selection "
+ f"from possibilities {[o['name'] for o in options]}, please "
+ f"remove one or more from {parameters}")
+ if len(choice) == 1:
+ choices[k] = choice[0]
+ continue
+
+ if non_interactive:
+ raise Exception(f"Unable to choose from options in non-interactive mode: {[o['name'] for o in options]}")
+
+ logger.plain("")
+ print_configs(v["description"],
+ [o['name'] for o in options],
+ [o['description'] for o in options])
+ options_enumerated = list(enumerate(options, 1))
+ option_n = int_input([i[0] for i in options_enumerated],
+ "\nPlease select one of the above options by its number: ") - 1
+ choices[k] = options_enumerated[option_n][1]["name"]
+ return choices
+
+def obtain_config(top_dir, registry, args, source_overrides, d):
+ if args.config:
+ config_id = args.config[0]
+ config_parameters = args.config[1:]
+ if os.path.exists(config_id):
+ config_id = os.path.abspath(config_id)
+ logger.info("Reading configuration from local file\n {}".format(config_id))
+ upstream_config = {'type':'local',
+ 'path':config_id,
+ 'name':get_config_name(config_id),
+ 'data':json.load(open(config_id))
+ }
+ elif config_id.startswith("http://") or config_id.startswith("https://"):
+ logger.info("Reading configuration from network URI\n {}".format(config_id))
+ import urllib.request
+ try:
+ with urllib.request.urlopen(config_id) as f:
+ json_data = json.load(f)
+ upstream_config = {'type':'network','uri':config_id,'name':get_config_name(config_id),'data':json_data}
+ except json.JSONDecodeError as e:
+ raise Exception ("Invalid JSON from {}. Are you pointing to an HTML page? {}".format(config_id, e)) from e
+ else:
+ logger.info("Looking up config {} in configuration registry".format(config_id))
+ registry_path = update_registry(registry, cache_dir(top_dir), d)
+ registry_configs = list_registry(registry_path, with_expired=True)
+ if config_id not in registry_configs:
+ raise Exception("Config {} not found in configuration registry, re-run 'init' without parameters to choose from available configurations.".format(config_id))
+ upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))}
+ expiry_date = upstream_config['data'].get("expires", None)
+ if has_expired(expiry_date):
+ logger.warning("This configuration is no longer supported after {}. Please consider changing to a supported configuration.".format(expiry_date))
+ else:
+ registry_path = update_registry(registry, cache_dir(top_dir), d)
+ registry_configs = list_registry(registry_path, with_expired=True)
+ config_id = choose_config(registry_configs, args.non_interactive)
+ config_parameters = []
+ upstream_config = {'type':'registry','registry':registry,'name':config_id,'data':json.load(open(get_registry_config(registry_path,config_id)))}
+
+ upstream_config['bitbake-config'] = choose_bitbake_config(upstream_config['data']['bitbake-setup']['configurations'], config_parameters, args.non_interactive)
+ upstream_config['bitbake-config']['oe-fragment-choices'] = choose_fragments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), config_parameters[1:], args.non_interactive, args.skip_selection)
+ upstream_config['non-interactive-cmdline-options'] = [config_id, upstream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-config']['oe-fragment-choices'].values())
+ upstream_config['source-overrides'] = source_overrides
+ upstream_config['skip-selection'] = args.skip_selection
+ return upstream_config
+
+def obtain_overrides(args):
+ overrides = {'sources':{}}
+ if args.source_overrides:
+ overrides = json.load(open(args.source_overrides))
+ overrides_dir = os.path.dirname(os.path.abspath(args.source_overrides))
+ for s,v in overrides['sources'].items():
+ local = v.get('local')
+ if local:
+ path = os.path.expanduser(local['path'])
+ if not os.path.isabs(path):
+ overrides['sources'][s]['local']['path'] = os.path.join(overrides_dir, path)
+
+ for local_name, local_path in args.use_local_source:
+ overrides['sources'][local_name] = {'local':{'path':os.path.abspath(os.path.expanduser(local_path))}}
+
+ return overrides
+
+def configure_vscode(setupdir, layerdir, builddir, init_script):
+ """
+ Configure the VSCode environment by creating or updating a workspace file.
+
+ Create or update a bitbake.code-workspace file with folders for the layers and build/conf.
+ Managed folders are regenerated; user-added folders are kept. Settings are merged, with
+ managed keys (bitbake.*, python extra paths) always overwritten.
+ """
+ logger.debug("configure_vscode: setupdir={}, layerdir={}, builddir={}, init_script={}".format(
+ setupdir, layerdir, builddir, init_script))
+
+ # Get git repository directories
+ git_repos = []
+ if os.path.exists(layerdir):
+ for entry in os.listdir(layerdir):
+ entry_path = os.path.join(layerdir, entry)
+ if os.path.isdir(entry_path) and not os.path.islink(entry_path):
+ # Check if it's a git repository
+ if os.path.exists(os.path.join(entry_path, '.git')):
+ git_repos.append(entry)
+ logger.debug("configure_vscode: found {} git repos: {}".format(len(git_repos), git_repos))
+
+ conf_path = os.path.relpath(os.path.join(builddir, "conf"), setupdir)
+ repo_paths = [os.path.relpath(os.path.join(layerdir, repo), setupdir) for repo in git_repos]
+ logger.debug("configure_vscode: conf_path={}, repo_paths={}".format(conf_path, repo_paths))
+
+ # Load existing workspace
+ workspace_file = os.path.join(setupdir, "bitbake.code-workspace")
+ workspace = {
+ "extensions": {
+ "recommendations": [
+ "yocto-project.yocto-bitbake"
+ ]
+ }
+ }
+ if os.path.exists(workspace_file):
+ logger.debug("configure_vscode: loading existing workspace file: {}".format(workspace_file))
+ try:
+ with open(workspace_file, 'r') as f:
+ workspace = json.load(f)
+ logger.debug("configure_vscode: loaded workspace with {} folders, {} settings".format(
+ len(workspace.get("folders", [])), len(workspace.get("settings", {}))))
+ except (json.JSONDecodeError, OSError) as e:
+ logger.error(
+ "Unable to read existing workspace file {}: {}. Skipping update.".format(
+ workspace_file, str(e)
+ )
+ )
+ return
+ else:
+ logger.debug("configure_vscode: creating new workspace file: {}".format(workspace_file))
+
+ # Update folders
+ existing_folders = workspace.get("folders", [])
+ new_folders = [{"name": "conf", "path": conf_path}]
+ for rp in repo_paths:
+ repo_name = os.path.basename(rp)
+ new_folders.append({"name": repo_name, "path": rp})
+ # Keep any user-added folders that are not managed
+ managed_paths = {f["path"] for f in new_folders}
+ for f in existing_folders:
+ if f["path"] not in managed_paths:
+ new_folders.append(f)
+ logger.debug("configure_vscode: keeping user-added folder: {}".format(f["path"]))
+ workspace["folders"] = new_folders
+ logger.debug("configure_vscode: updated workspace with {} folders".format(len(new_folders)))
+
+ # Build Python extra paths for each layer - only check top level of each repo
+ extra_paths = []
+ subdirs_to_check = ['lib', 'scripts']
+ for repo in git_repos:
+ repo_path_abs = os.path.join(layerdir, repo)
+ for subdir in subdirs_to_check:
+ sub_path = os.path.join(repo_path_abs, subdir)
+ if os.path.isdir(sub_path):
+ extra_paths.append(sub_path)
+
+ # Update settings
+ existing_settings = workspace.get("settings", {})
+ new_settings = {
+ "bitbake.disableConfigModification": True,
+ "bitbake.pathToBitbakeFolder": os.path.join(layerdir, "bitbake"),
+ "bitbake.pathToBuildFolder": builddir,
+ "bitbake.pathToEnvScript": init_script,
+ "bitbake.workingDirectory": builddir,
+ "files.associations": {
+ "*.conf": "bitbake",
+ "*.inc": "bitbake"
+ },
+ "files.exclude": {
+ "**/.git/**": True
+ },
+ "search.exclude": {
+ "**/.git/**": True,
+ "**/logs/**": True
+ },
+ "files.watcherExclude": {
+ "**/.git/**": True,
+ "**/logs/**": True
+ },
+ "python.analysis.exclude": [
+ "**/.git/**",
+ "**/logs/**"
+ ],
+ "python.autoComplete.extraPaths": extra_paths,
+ "python.analysis.extraPaths": extra_paths
+ }
+
+ # Merge settings: add missing, always update bitbake paths and python extra paths
+ for key, value in new_settings.items():
+ if key not in existing_settings:
+ existing_settings[key] = value
+ elif key.startswith("bitbake.") or key in [
+ "python.autoComplete.extraPaths",
+ "python.analysis.extraPaths",
+ ]:
+ # Always replace - these are managed/machine-generated settings
+ existing_settings[key] = value
+ elif key in [
+ "files.associations",
+ "files.exclude",
+ "search.exclude",
+ "files.watcherExclude",
+ "python.analysis.exclude",
+ ]:
+ # For dicts and lists, merge new values in without removing user additions
+ if isinstance(value, dict):
+ if not isinstance(existing_settings[key], dict):
+ existing_settings[key] = {}
+ for k, v in value.items():
+ if k not in existing_settings[key]:
+ existing_settings[key][k] = v
+ elif isinstance(value, list):
+ if not isinstance(existing_settings[key], list):
+ existing_settings[key] = []
+ for item in value:
+ if item not in existing_settings[key]:
+ existing_settings[key].append(item)
+
+ workspace["settings"] = existing_settings
+ logger.debug("configure_vscode: merged settings, total {} keys".format(len(existing_settings)))
+
+ with open(workspace_file, 'w') as f:
+ json.dump(workspace, f, indent=4)
+ logger.debug("configure_vscode: wrote workspace file: {}".format(workspace_file))
+
+def init_config(top_dir, settings, args):
+ create_siteconf(top_dir, args.non_interactive, settings)
+
+ d = init_bb_cache(top_dir, settings, args)
+
+ def handle_task_progress(event, d):
+ rate = event.rate if event.rate else ''
+ progress = event.progress if event.progress > 0 else 0
+ logger.handlers[0].terminator = '\r'
+ logger.plain("{}% {} ".format(progress, rate))
+ logger.handlers[0].terminator = '\n'
+
+ source_overrides = obtain_overrides(args)
+ upstream_config = obtain_config(top_dir, settings["default"]["registry"], args, source_overrides, d)
+ logger.info("Run 'bitbake-setup init --non-interactive {}' to select this configuration non-interactively.\n".format(" ".join(upstream_config['non-interactive-cmdline-options'])))
+
+ if args.setup_dir_name:
+ setup_dir_name = args.setup_dir_name
+ else:
+ setup_dir_name = "{}-{}".format(upstream_config['name']," ".join(upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").replace("/","_"))
+ if 'setup-dir-name' in upstream_config['bitbake-config']:
+ mapping = {
+ k: v.partition("/")[2].replace(" ", "-").replace("/", "_")
+ for k, v in upstream_config['bitbake-config']['oe-fragment-choices'].items()
+ }
+ config_setup_dir_name = string.Template(upstream_config['bitbake-config']['setup-dir-name']).substitute(mapping)
+ config_setup_dir = os.path.join(top_dir, config_setup_dir_name)
+ if os.path.exists(config_setup_dir):
+ logger.info("Setup directory {} (as suggested by configuration) already exists, using the full name instead.\n".format(config_setup_dir))
+ elif settings['default']['use-full-setup-dir-name'] != 'no':
+ logger.info("Using the full setup directory name instead of {} suggested by configuration, as set in the settings.\n".format(config_setup_dir))
+ else:
+ setup_dir_name = config_setup_dir_name
+
+ if not args.non_interactive:
+ n = input(f"Enter setup directory name [{setup_dir_name}]: ")
+ if n:
+ setup_dir_name = n
+
+ setupdir = os.path.join(os.path.abspath(top_dir), setup_dir_name)
+ if os.path.exists(os.path.join(setupdir, "layers")):
+ logger.info(f"Setup already initialized in:\n {setupdir}\nUse 'bitbake-setup status' to check if it needs to be updated, or 'bitbake-setup update' to perform the update.\nIf you would like to start over and re-initialize in this directory, remove it, and run 'bitbake-setup init' again.")
+ return
+
+ logger.plain("Initializing a setup directory in\n {}".format(setupdir))
+ if not args.non_interactive:
+ y_or_n = input('Continue? (y/N): ')
+ if y_or_n != 'y':
+ exit()
+ logger.plain("")
+
+ os.makedirs(setupdir, exist_ok=True)
+
+ confdir = os.path.join(setupdir, "config")
+ layerdir = os.path.join(setupdir, "layers")
+
+ os.makedirs(confdir)
+ os.makedirs(layerdir)
+
+ bb.process.run(["git", "-C", confdir, "init", "-b", "main"])
+ # Make sure commiting doesn't fail if no default git user is configured on the machine
+ bb.process.run(["git", "-C", confdir, "config", "user.name", "bitbake-setup"])
+ bb.process.run(["git", "-C", confdir, "config", "user.email", "bitbake-setup@not.set"])
+ bb.process.run(["git", "-C", confdir, "commit", "--no-verify", "--allow-empty", "-m", "Initial commit"])
+
+ bb.event.register("bb.build.TaskProgress", handle_task_progress, data=d)
+
+ write_upstream_config(confdir, upstream_config)
+ update_build(upstream_config, confdir, setupdir, layerdir, d, update_bb_conf="yes", init_vscode=args.init_vscode)
+
+ bb.event.remove("bb.build.TaskProgress", None)
+
+def get_diff(file1, file2):
+ try:
+ bb.process.run(["diff", get_diff_color_param(), "-uNr", file1, file2])
+ except bb.process.ExecutionError as e:
+ if e.exitcode == 1:
+ return e.stdout
+ else:
+ raise e
+ return None
+
+def are_layers_changed(layers, layerdir, d):
+ def _is_git_remote_changed(r_remote, repodir):
+ from bb.fetch.git import git_hash_re
+
+ rev = r_remote['rev']
+ branch = r_remote.get('branch', None)
+
+ rev_parse_result = bb.process.run(['git', '-C', os.path.join(layerdir, repodir), "rev-parse", "HEAD"])
+ local_revision = rev_parse_result[0].strip()
+ if git_hash_re.match(rev):
+ if rev != local_revision:
+ logger.info('Layer repository checked out into {} is at revision {} but should be at {}'.format(os.path.join(layerdir, repodir),local_revision, rev))
+ return True
+ return False
+
+ remotes = _get_remotes(r_remote)
+ changed = False
+ for remote in remotes:
+ type,host,path,user,pswd,params = bb.fetch.decodeurl(remote)
+ fetchuri = bb.fetch.encodeurl(('git',host,path,user,pswd,params))
+ if branch:
+ fetcher = bb.fetch.FetchData("{};protocol={};rev={};branch={};destsuffix={}".format(fetchuri,type,rev,branch,repodir), d)
+ else:
+ fetcher = bb.fetch.FetchData("{};protocol={};rev={};nobranch=1;destsuffix={}".format(fetchuri,type,rev,repodir), d)
+ upstream_revision = fetcher.method.latest_revision(fetcher, d, 'default')
+ if upstream_revision != local_revision:
+ changed = True
+ logger.info('Layer repository {} checked out into {} updated revision {} from {} to {}'.format(remote, os.path.join(layerdir, repodir), rev, local_revision, upstream_revision))
+ return changed
+
+ changed = False
+ for r_name in layers:
+ r_data = layers[r_name]
+ repodir = r_data.get("path", r_name)
+
+ git_remote = r_data.get('git-remote')
+ if git_remote:
+ changed = changed | _is_git_remote_changed(git_remote, repodir)
+
+ return changed
+
+def build_status(top_dir, settings, args, d, update=False):
+ setupdir = os.path.abspath(args.setup_dir)
+
+ confdir = os.path.join(setupdir, "config")
+ layerdir = os.path.join(setupdir, "layers")
+
+ current_upstream_config = json.load(open(os.path.join(confdir, "config-upstream.json")))
+
+ args.config = current_upstream_config['non-interactive-cmdline-options']
+ args.non_interactive = True
+ args.skip_selection = current_upstream_config['skip-selection']
+ source_overrides = current_upstream_config["source-overrides"]
+ registry = current_upstream_config.get("registry")
+ new_upstream_config = obtain_config(top_dir, registry, args, source_overrides, d)
+
+ write_upstream_config(confdir, new_upstream_config)
+ config_diff = bb.process.run(["git", "-C" , confdir, "diff", get_diff_color_param()])[0]
+
+ if config_diff:
+ logger.plain('\nConfiguration in {} has changed:\n{}'.format(setupdir, config_diff))
+ if update:
+ update_build(new_upstream_config, confdir, setupdir, layerdir, d,
+ update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy)
+ else:
+ bb.process.run(["git", "-C", confdir, "restore", "config-upstream.json"])
+ return
+
+ layer_config = merge_overrides_into_sources(current_upstream_config["data"]["sources"], current_upstream_config["source-overrides"]["sources"])
+ if are_layers_changed(layer_config, layerdir, d):
+ if update:
+ update_build(current_upstream_config, confdir, setupdir, layerdir,
+ d, update_bb_conf=args.update_bb_conf, rebase_conflicts_strategy=args.rebase_conflicts_strategy)
+ return
+
+ logger.plain("\nConfiguration in {} has not changed.".format(setupdir))
+
+def build_update(top_dir, settings, args, d):
+ build_status(top_dir, settings, args, d, update=True)
+
+def do_fetch(fetcher, dir):
+ # git fetcher simply dumps git output to stdout; in bitbake context that is redirected to temp/log.do_fetch
+ # and we need to set up smth similar here
+ fetchlogdir = os.path.join(dir, 'logs')
+ os.makedirs(fetchlogdir, exist_ok=True)
+ fetchlog = os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.datetime.now().strftime("%Y%m%d%H%M%S")))
+ with open(fetchlog, 'a') as f:
+ oldstdout = sys.stdout
+ sys.stdout = f
+ try:
+ fetcher.download()
+ fetcher.unpack_update(dir)
+ except:
+ if os.path.exists(fetchlog) and os.path.getsize(fetchlog) > 0:
+ logger.error("Check {} for additional information that may help diagnoze the issue.".format(fetchlog))
+ raise
+ finally:
+ sys.stdout = oldstdout
+
+def update_registry(registry, cachedir, d):
+ registrydir = 'configurations'
+ if registry.startswith("."):
+ full_registrydir = os.path.join(os.getcwd(), registry, registrydir)
+ elif registry.startswith("/"):
+ full_registrydir = os.path.join(registry, registrydir)
+ else:
+ full_registrydir = os.path.join(cachedir, registrydir)
+ logger.info("Fetching configuration registry\n {}\ninto\n {}".format(registry, full_registrydir))
+ fetcher = bb.fetch.Fetch(["{};destsuffix={}".format(registry, registrydir)], d)
+ do_fetch(fetcher, cachedir)
+ return full_registrydir
+
+def has_expired(expiry_date):
+ if expiry_date:
+ return datetime.datetime.now() > datetime.datetime.fromisoformat(expiry_date)
+ return False
+
+def list_registry(registry_path, with_expired):
+ json_data = {}
+
+ for root, _, files in os.walk(registry_path):
+ for file in files:
+ if file.endswith('.conf.json'):
+ config_name = get_config_name(file)
+ config_data = json.load(open(os.path.join(root, file)))
+ config_desc = config_data["description"]
+ expiry_date = config_data.get("expires", None)
+ if expiry_date:
+ if with_expired or not has_expired(expiry_date):
+ json_data[config_name] = {"description": config_desc, "expires": expiry_date}
+ else:
+ json_data[config_name] = {"description": config_desc}
+ return json_data
+
+def list_configs(settings, args):
+ import tempfile
+ top_dir = tempfile.mkdtemp(prefix="bitbake-setup-list-")
+ settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads')
+ d = init_bb_cache(top_dir, settings, args)
+ registry_path = update_registry(settings["default"]["registry"], cache_dir(top_dir), d)
+ json_data = list_registry(registry_path, args.with_expired)
+ shutil.rmtree(top_dir)
+
+ logger.plain("Available configurations:")
+ for config_name, config_data in sorted(json_data.items()):
+ expiry_date = config_data.get("expires", None)
+ config_desc = config_data["description"]
+ if expiry_date:
+ if args.with_expired or not has_expired(expiry_date):
+ logger.plain("{}\t{} (supported until {})".format(config_name, config_desc, expiry_date))
+ else:
+ logger.plain("{}\t{}".format(config_name, config_desc))
+ logger.plain("\nRun 'init' with one of the above configuration identifiers to set up a build.")
+
+ if args.write_json:
+ with open(args.write_json, 'w') as f:
+ json.dump(json_data, f, sort_keys=True, indent=4)
+ logger.plain("Available configurations written into {}".format(args.write_json))
+
+def install_buildtools(top_dir, settings, args, d):
+ buildtools_install_dir = os.path.join(args.setup_dir, 'buildtools')
+ if os.path.exists(buildtools_install_dir):
+ if not args.force:
+ logger.plain("Buildtools are already installed in {}.".format(buildtools_install_dir))
+ env_scripts = glob.glob(os.path.join(buildtools_install_dir, 'environment-setup-*'))
+ if env_scripts:
+ logger.plain("If you wish to use them, you need to source the environment setup script e.g.")
+ for s in env_scripts:
+ logger.plain("$ . {}".format(s))
+ logger.plain("You can also re-run bitbake-setup install-buildtools with --force option to force a reinstallation.")
+ return
+ shutil.rmtree(buildtools_install_dir)
+
+ install_buildtools_script = os.path.join(args.setup_dir, 'layers/oe-scripts/install-buildtools')
+
+ buildtools_config = {}
+ upstream_config_path = os.path.join(args.setup_dir, 'config', 'config-upstream.json')
+ try:
+ with open(upstream_config_path) as f:
+ upstream_config = json.load(f)
+ buildtools_config = upstream_config.get('bitbake-config', {}).get('install-buildtools', {})
+ except FileNotFoundError:
+ pass
+ except json.JSONDecodeError as e:
+ logger.error("Failed to parse %s: %s" % (upstream_config_path, e))
+ sys.exit(1)
+
+ url = args.url if args.url is not None else buildtools_config.get('url')
+ sha256 = args.sha256 if args.sha256 is not None else buildtools_config.get('sha256sum')
+
+ buildtools_download_dir = add_unique_timestamp_to_path(
+ os.path.join(args.setup_dir, 'buildtools-downloads/buildtools'))
+
+ if args.default or (not url and not sha256):
+ # No config or --default: fall back to install-buildtools defaults
+ logger.plain("Using install-buildtools script defaults")
+ subprocess.check_call([install_buildtools_script,
+ '-d', buildtools_install_dir,
+ '--downloads-directory', buildtools_download_dir])
+ return
+
+ if not url:
+ logger.error("No buildtools URL configured. "
+ "Set 'url' in the install-buildtools config section or pass --url.")
+ sys.exit(1)
+
+ if not sha256:
+ logger.error("No sha256sum configured for buildtools installer. "
+ "Add 'sha256sum' to the install-buildtools config section or pass --sha256.\n"
+ "You can obtain it with: sha256sum <installer-file>")
+ sys.exit(1)
+
+ # Download using bb.fetch with sha256 validation
+ src_uri = "{};sha256sum={}".format(url, sha256)
+
+ logger.plain("Fetching buildtools installer using bitbake fetcher")
+ logger.plain(" {}".format(url))
+
+ fetcher = bb.fetch.Fetch([src_uri], d)
+ try:
+ fetcher.download()
+ except bb.fetch.ChecksumError as e:
+ logger.error("Checksum mismatch for buildtools installer. "
+ "Verify the URL and downloaded file.\n%s" % str(e))
+ sys.exit(1)
+ except bb.fetch.FetchError as e:
+ logger.error("Failed to download buildtools installer: %s" % str(e))
+ sys.exit(1)
+
+ # Unpack into the standard buildtools download location
+ os.makedirs(buildtools_download_dir, exist_ok=True)
+ fetcher.unpack(buildtools_download_dir)
+
+ filename = os.path.basename(url.rstrip('/'))
+ matches = glob.glob(os.path.join(buildtools_download_dir, '**', filename), recursive=True)
+ if not matches:
+ logger.error("Unpacked file not found: %s" % filename)
+ sys.exit(1)
+ local_path = matches[0]
+
+ logger.plain("Buildtools installer will be installed into {}".format(
+ buildtools_install_dir))
+
+ cmd = [
+ install_buildtools_script,
+ '--local-file', local_path,
+ '-d', buildtools_install_dir,
+ ]
+
+ try:
+ subprocess.check_call(cmd)
+ except subprocess.CalledProcessError as e:
+ logger.error("install-buildtools failed with exit code %d" % e.returncode)
+ sys.exit(1)
+
+def create_siteconf(top_dir, non_interactive, settings):
+ siteconfpath = os.path.join(top_dir, 'site.conf')
+ if os.path.exists(siteconfpath):
+ logger.info('A site.conf file already exists. Please remove it if you would like to replace it with a default one')
+ else:
+ logger.plain(f'{top_dir} looks like a new top directory. If you would like to use a different directory, answer "n" below and either:')
+ logger.plain('\t1) Change the default bitbake-setup settings:')
+ logger.plain('\t\tbitbake-setup settings set default top-dir-prefix <PATH>')
+ logger.plain('\t\tbitbake-setup settings set default top-dir-name <NAME>')
+ logger.plain('\t2) Pass one or more options on the command line to change the top level directory in that invocation only:')
+ logger.plain('\t\tbitbake-setup --setting default top-dir-prefix <PATH> ...')
+ logger.plain('\t\tbitbake-setup --setting default top-dir-name <NAME> ...')
+ logger.plain('')
+
+ logger.plain('A common site.conf file will be created, please check it is correct before running builds\n {}\n'.format(siteconfpath))
+ if not non_interactive:
+ y_or_n = input('Proceed? (y/N): ')
+ if y_or_n != 'y':
+ exit()
+
+ os.makedirs(top_dir, exist_ok=True)
+ with open(siteconfpath, 'w') as siteconffile:
+ sstate_settings = textwrap.dedent(
+ """
+ #
+ # Where to place shared-state files
+ #
+ # BitBake has the capability to accelerate builds based on previously built output.
+ # This is done using "shared state" files which can be thought of as cache objects
+ # and this option determines where those files are placed.
+ #
+ # You can wipe out TMPDIR leaving this directory intact and the build would regenerate
+ # from these files if no changes were made to the configuration. If changes were made
+ # to the configuration, only shared state files where the state was still valid would
+ # be used (done using checksums).
+ SSTATE_DIR ?= "{sstate_dir}"
+ #
+ # Hash Equivalence database location
+ #
+ # Hash equivalence improves reuse of sstate by detecting when a given sstate
+ # artifact can be reused as equivalent, even if the current task hash doesn't
+ # match the one that generated the artifact. This variable controls where the
+ # Hash Equivalence database ("hashserv.db") is stored and can be shared between
+ # concurrent builds.
+ BB_HASHSERVE_DB_DIR ?= "${{SSTATE_DIR}}"
+ """.format(sstate_dir=os.path.join(top_dir, ".sstate-cache"))
+ )
+ siteconffile.write(
+ textwrap.dedent(
+ """\
+ # This file is intended for build host-specific bitbake settings
+
+ # Where to place downloads
+ #
+ # During a first build the system will download many different source code
+ # tarballs from various upstream projects. This can take a while, particularly
+ # if your network connection is slow. These are all stored in DL_DIR. When
+ # wiping and rebuilding you can preserve this directory to speed up this part of
+ # subsequent builds. This directory is safe to share between multiple builds on
+ # the same machine too.
+ DL_DIR ?= "{dl_dir}"
+ """.format(
+ dl_dir=settings["default"]["dl-dir"],
+ )
+ ) + (sstate_settings if settings["default"]["common-sstate"] == 'yes' else "")
+ )
+
+
+def topdir_settings_path(top_dir):
+ return os.path.join(top_dir, 'settings.conf')
+
+def global_settings_path(args):
+ return os.path.abspath(args.global_settings) if args.global_settings else os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'settings.conf')
+
+def load_settings(settings_path):
+ settings = configparser.ConfigParser()
+ if os.path.exists(settings_path):
+ logger.info('Loading settings from {}'.format(settings_path))
+ settings.read_file(open(settings_path))
+
+ for section in settings.sections():
+ for key, value in settings[section].items():
+ settings[section][key] = value.strip('\"\'')
+
+ return settings
+
+def change_setting(top_dir, args):
+ if vars(args)['global']:
+ settings_path = global_settings_path(args)
+ elif args.setting in GLOBAL_ONLY_SETTINGS:
+ logger.info(f"{args.setting} can only be set in the global config; '--global' is implied")
+ settings_path = global_settings_path(args)
+ else:
+ settings_path = topdir_settings_path(top_dir)
+ settings = load_settings(settings_path)
+
+ if args.subcommand == 'set':
+ if args.section not in settings.keys():
+ settings[args.section] = {}
+ settings[args.section][args.setting] = args.value
+ logger.plain(f"From section '{args.section}' the setting '{args.setting}' was changed to '{args.value}'")
+ if args.subcommand == 'unset':
+ if args.section in settings.keys() and args.setting in settings[args.section].keys():
+ del settings[args.section][args.setting]
+ logger.plain(f"From section '{args.section}' the setting '{args.setting}' has been removed")
+
+ os.makedirs(os.path.dirname(settings_path), exist_ok=True)
+ with open(settings_path, 'w') as settingsfile:
+ settings.write(settingsfile)
+ logger.info(f"Settings written to {settings_path}")
+
+def list_settings(all_settings):
+ for section, section_settings in all_settings.items():
+ for key, value in section_settings.items():
+ logger.plain("{} {} {}".format(section, key, value))
+
+def settings_func(top_dir, all_settings, args):
+ if args.subcommand == 'list':
+ list_settings(all_settings)
+ elif args.subcommand == 'set' or args.subcommand == 'unset':
+ change_setting(top_dir, args)
+
+def get_setup_dir_via_bbpath():
+ bbpath = os.environ.get('BBPATH')
+ if bbpath:
+ bitbake_dir = os.path.normpath(bbpath.split(':')[0])
+ if os.path.exists(os.path.join(bitbake_dir,'init-build-env')):
+ setup_dir = os.path.dirname(bitbake_dir)
+ return setup_dir
+ return None
+
+def get_top_dir(args, settings):
+ setup_dir_via_bbpath = get_setup_dir_via_bbpath()
+ if setup_dir_via_bbpath:
+ top_dir = os.path.dirname(setup_dir_via_bbpath)
+ if os.path.exists(cache_dir(top_dir)):
+ return top_dir
+
+ if hasattr(args, 'setup_dir'):
+ top_dir = os.path.dirname(os.path.normpath(args.setup_dir))
+ return top_dir
+
+ top_dir_prefix = settings['default']['top-dir-prefix']
+ top_dir_name = settings['default']['top-dir-name']
+ return os.path.join(top_dir_prefix, top_dir_name)
+
+def merge_settings(builtin_settings, global_settings, topdir_settings, cmdline_settings):
+ all_settings = builtin_settings
+
+ for s in (global_settings, topdir_settings):
+ for section, section_settings in s.items():
+ for setting, value in section_settings.items():
+ if section not in all_settings.keys():
+ all_settings[section] = {}
+ all_settings[section][setting] = value
+
+ for (section, setting, value) in cmdline_settings:
+ if section not in all_settings.keys():
+ all_settings[section] = {}
+ all_settings[section][setting] = value
+
+ return all_settings
+
+def sigint_handler(sig, frame, func, top_dir):
+ logger.plain('\nShutting down...')
+ if isinstance(top_dir, str) and os.path.exists(top_dir):
+ if func in [init_config, build_update]:
+ logger.warning(f'{top_dir} may contain an incomplete setup!')
+ elif func == install_buildtools:
+ logger.warning(f'{top_dir} may contain an incomplete buildtools installation!')
+ exit()
+
+def main():
+ def add_setup_dir_arg(parser):
+ setup_dir = get_setup_dir_via_bbpath()
+ if setup_dir:
+ parser.add_argument('--setup-dir', default=setup_dir, help="Path to the setup, default is %(default)s via BBPATH")
+ else:
+ parser.add_argument('--setup-dir', required=True, help="Path to the setup")
+
+ parser = argparse.ArgumentParser(
+ description="BitBake setup utility. Run with 'init' argument to get started.",
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command"
+ )
+ parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+ parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+ parser.add_argument('--version', action='version', version='%(prog)s ' + __version__)
+ parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
+ parser.add_argument('--no-network', action='store_true', help='Do not check whether configuration repositories and layer repositories have been updated; use only the local cache.')
+ parser.add_argument('--global-settings', action='store', metavar='PATH', help='Path to the global settings file.')
+ parser.add_argument('--setting', default=[], action='append', dest='cmdline_settings',
+ nargs=3, metavar=('SECTION', 'SETTING', 'VALUE'),
+ help='Modify a setting (for this bitbake-setup invocation only), for example "--setting default top-dir-prefix /path/to/top/dir".')
+
+ subparsers = parser.add_subparsers()
+
+ parser_list = subparsers.add_parser('list', help='List available configurations')
+ parser_list.add_argument('--with-expired', action='store_true', help='List also configurations that are no longer supported due to reaching their end-of-life dates.')
+ parser_list.add_argument('--write-json', action='store', help='Write available configurations into a json file so they can be programmatically processed.')
+ parser_list.set_defaults(func=list_configs)
+
+ parser_init = subparsers.add_parser('init', help='Select a configuration and initialize a setup from it')
+ parser_init.add_argument('config', nargs='*', help="path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available choices if command line doesn't completely specify them.")
+ parser_init.add_argument('--non-interactive', action='store_true', help='Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.')
+ parser_init.add_argument('--source-overrides', action='store', help='Override sources information (repositories/revisions) with values from a local json file.')
+ parser_init.add_argument('--setup-dir-name', action='store', help='A custom setup directory name under the top directory.')
+ parser_init.add_argument('--skip-selection', action='append', help='Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.')
+ parser_init.add_argument('-L', '--use-local-source', default=[], action='append', nargs=2, metavar=('SOURCE_NAME', 'PATH'),
+ help='Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).')
+ parser_init.add_argument('--init-vscode', action=argparse.BooleanOptionalAction, default=bool(shutil.which('code')),
+ help='Generate VSCode workspace configuration (default: %(default)s)')
+ parser_init.set_defaults(func=init_config)
+
+ parser_status = subparsers.add_parser('status', help='Check if the setup needs to be synchronized with configuration')
+ add_setup_dir_arg(parser_status)
+ parser_status.set_defaults(func=build_status)
+
+ parser_update = subparsers.add_parser('update', help='Update a setup to be in sync with configuration')
+ add_setup_dir_arg(parser_update)
+ parser_update.add_argument('--update-bb-conf', choices=['prompt', 'yes', 'no'], default='prompt', help='Update bitbake configuration files (bblayers.conf, local.conf) (default: prompt)')
+ parser_update.add_argument('--rebase-conflicts-strategy', choices=['abort', 'backup'], default='abort',
+ help="What to do when a layer repository has local modifications that prevent "
+ "an in-place update: 'abort' (default) aborts with an error message; "
+ "'backup' renames the directory to a timestamped backup and re-clones from upstream.")
+ parser_update.set_defaults(func=build_update)
+
+ parser_install_buildtools = subparsers.add_parser('install-buildtools', help='Install buildtools which can help fulfil missing or incorrect dependencies on the host machine')
+ add_setup_dir_arg(parser_install_buildtools)
+ parser_install_buildtools.add_argument('--force', action='store_true', help='Force a reinstall of buildtools over the previous installation.')
+ parser_install_buildtools.add_argument('--default', action='store_true', help='Use install-buildtools script defaults, ignoring any config.')
+ parser_install_buildtools.add_argument('--url', help='Full URL to the buildtools SDK installer. Overrides config value.')
+ parser_install_buildtools.add_argument('--sha256', help='SHA256 checksum of the buildtools installer. Overrides config value.')
+ parser_install_buildtools.set_defaults(func=install_buildtools)
+
+ parser_settings_arg_global = argparse.ArgumentParser(add_help=False)
+ parser_settings_arg_global.add_argument('--global', action='store_true', help="Modify the setting in a global settings file, rather than one specific to a top directory")
+
+ parser_settings = subparsers.add_parser('settings',
+ help='List current settings, or set or unset a setting in a settings file (e.g. the default prefix and name of the top directory, the location of configuration registry, downloads directory and other settings specific to a top directory)')
+ parser_settings.set_defaults(func=settings_func)
+
+ subparser_settings = parser_settings.add_subparsers(dest="subcommand", required=True, help="The action to perform on the settings file")
+
+ subparser_settings.add_parser('list', help="List all settings with their values")
+
+ parser_settings_set = subparser_settings.add_parser('set', parents=[parser_settings_arg_global],
+ help="In a Section, set a setting to a certain value")
+ parser_settings_set.add_argument("section", metavar="<section>", help="Section in a settings file, typically 'default'")
+ parser_settings_set.add_argument("setting", metavar="<setting>", help="Name of a setting")
+ parser_settings_set.add_argument("value", metavar="<value>", help="The setting value")
+
+ parser_settings_unset = subparser_settings.add_parser('unset', parents=[parser_settings_arg_global],
+ help="Unset a setting, e.g. 'bitbake-setup settings unset default registry' would revert to the registry setting in a global settings file")
+ parser_settings_unset.add_argument("section", metavar="<section>", help="Section in a settings file, typically 'default'")
+ parser_settings_unset.add_argument("setting", metavar="<setting>", help="The setting to remove")
+
+ args = parser.parse_args()
+
+ if args.debug:
+ logger.setLevel(logging.DEBUG)
+ elif args.quiet:
+ logger.setLevel(logging.ERROR)
+
+ # Need to re-run logger_create with color argument
+ # (will be the same logger since it has the same name)
+ bb.msg.logger_create('bitbake-setup', output=sys.stdout,
+ color=args.color,
+ level=logger.getEffectiveLevel())
+
+ if 'func' in args:
+ if hasattr(args, 'setup_dir'):
+ if not os.path.exists(os.path.join(args.setup_dir,'build', 'init-build-env')):
+ logger.error("Not a valid setup directory: build/init-build-env does not exist in {}".format(args.setup_dir))
+ return
+
+ if not hasattr(args, 'non_interactive'):
+ args.non_interactive = True
+
+ builtin_settings = {}
+ builtin_settings['default'] = {
+ 'top-dir-prefix':os.getcwd(),
+ 'top-dir-name':'bitbake-builds',
+ 'registry':get_default_registry(),
+ 'use-full-setup-dir-name':'no',
+ 'common-sstate':'yes',
+ }
+
+ global_settings = load_settings(global_settings_path(args))
+ top_dir = get_top_dir(args, merge_settings(builtin_settings, global_settings, {}, args.cmdline_settings))
+
+ # register handler now to pass top_dir
+ _handler = functools.partial(sigint_handler,
+ func=args.func,
+ top_dir=os.path.abspath(top_dir))
+ signal.signal(signal.SIGINT, _handler)
+
+ # This cannot be set with the rest of the builtin settings as top_dir needs to be determined first
+ builtin_settings['default']['dl-dir'] = os.path.join(top_dir, '.bitbake-setup-downloads')
+
+ topdir_settings = load_settings(topdir_settings_path(top_dir))
+ all_settings = merge_settings(builtin_settings, global_settings, topdir_settings, args.cmdline_settings)
+
+ if args.func == settings_func:
+ settings_func(top_dir, all_settings, args)
+ return
+ if args.func == list_configs:
+ list_configs(all_settings, args)
+ return
+
+ logger.info('Bitbake-setup is using {} as top directory.'.format(top_dir))
+
+ try:
+ if args.func == init_config:
+ init_config(top_dir, all_settings, args)
+ else:
+ d = init_bb_cache(top_dir, all_settings, args)
+ args.func(top_dir, all_settings, args, d)
+
+ save_bb_cache()
+ except (SystemExit, KeyboardInterrupt):
+ raise
+ except Exception as e:
+ if args.debug:
+ raise
+ logger.error(str(e))
+ sys.exit(1)
+ else:
+ parser.print_help()
+
+if __name__ == '__main__':
+ main()
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index e8073f2a..61484a5c 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -9,8 +9,9 @@ import os
import sys
import warnings
warnings.simplefilter("default")
+warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*")
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
-from bb import fetch2
+from bb import fetch
import logging
import bb
import select
@@ -21,9 +22,21 @@ import traceback
import queue
import shlex
import subprocess
-from multiprocessing import Lock
+import fcntl
from threading import Thread
+__version__ = "2.20.0"
+
+if __name__ == "__main__":
+ if __version__ != bb.__version__:
+ sys.exit("Bitbake core library version and program version mismatch!")
+
+Lock = bb.multiprocessing.Lock
+
+# Remove when we have a minimum of python 3.10
+if not hasattr(fcntl, 'F_SETPIPE_SZ'):
+ fcntl.F_SETPIPE_SZ = 1031
+
bb.utils.check_system_locale()
# Users shouldn't be running this code directly
@@ -44,7 +57,6 @@ if sys.argv[1].startswith("decafbadbad"):
# updates to log files for use with tail
try:
if sys.stdout.name == '<stdout>':
- import fcntl
fl = fcntl.fcntl(sys.stdout.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, fl)
@@ -56,6 +68,12 @@ logger = logging.getLogger("BitBake")
worker_pipe = sys.stdout.fileno()
bb.utils.nonblockingfd(worker_pipe)
+# Try to make the pipe buffers larger as it is much more efficient. If we can't
+# e.g. out of buffer space (/proc/sys/fs/pipe-user-pages-soft) then just pass over.
+try:
+ fcntl.fcntl(worker_pipe, fcntl.F_SETPIPE_SZ, 512 * 1024)
+except:
+ pass
# Need to guard against multiprocessing being used in child processes
# and multiple processes trying to write to the parent at the same time
worker_pipe_lock = None
@@ -105,7 +123,7 @@ def worker_flush(worker_queue):
if not worker_queue.empty():
worker_queue_int.extend(worker_queue.get())
written = os.write(worker_pipe, worker_queue_int)
- worker_queue_int = worker_queue_int[written:]
+ del worker_queue_int[0:written]
except (IOError, OSError) as e:
if e.errno != errno.EAGAIN and e.errno != errno.EPIPE:
raise
@@ -171,11 +189,8 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask):
elif workerdata["umask"]:
umask = workerdata["umask"]
if umask:
- # umask might come in as a number or text string..
- try:
- umask = int(umask, 8)
- except TypeError:
- pass
+ # Convert to a python numeric value as it could be a string
+ umask = bb.utils.to_filemode(umask)
dry_run = cfg.dry_run or runtask['dry_run']
@@ -278,9 +293,12 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask):
bb.utils.disable_network(uid, gid)
else:
logger.debug("Skipping disable network for %s since %s is not a local uid." % (taskname, uid))
+ bb.utils.landlock_restrict_network()
- # exported_vars() returns a generator which *cannot* be passed to os.environ.update()
- # successfully. We also need to unset anything from the environment which shouldn't be there
+ # exported_vars() expands each value, so it must be called before
+ # empty_environment() below: an expansion that shells out needs the
+ # environment bitbake was started with. We also need to unset
+ # anything from the environment which shouldn't be there
exports = bb.data.exported_vars(the_data)
bb.utils.empty_environment()
@@ -357,7 +375,7 @@ class runQueueWorkerPipe():
def read(self):
start = len(self.queue)
try:
- self.queue.extend(self.input.read(102400) or b"")
+ self.queue.extend(self.input.read(512*1024) or b"")
except (OSError, IOError) as e:
if e.errno != errno.EAGAIN:
raise
diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow
deleted file mode 100755
index 9de557c1..00000000
--- a/bitbake/bin/git-make-shallow
+++ /dev/null
@@ -1,175 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright BitBake Contributors
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-"""git-make-shallow: make the current git repository shallow
-
-Remove the history of the specified revisions, then optionally filter the
-available refs to those specified.
-"""
-
-import argparse
-import collections
-import errno
-import itertools
-import os
-import subprocess
-import sys
-import warnings
-warnings.simplefilter("default")
-
-version = 1.0
-
-
-git_cmd = ['git', '-c', 'safe.bareRepository=all']
-
-def main():
- if sys.version_info < (3, 4, 0):
- sys.exit('Python 3.4 or greater is required')
-
- git_dir = check_output(git_cmd + ['rev-parse', '--git-dir']).rstrip()
- shallow_file = os.path.join(git_dir, 'shallow')
- if os.path.exists(shallow_file):
- try:
- check_output(git_cmd + ['fetch', '--unshallow'])
- except subprocess.CalledProcessError:
- try:
- os.unlink(shallow_file)
- except OSError as exc:
- if exc.errno != errno.ENOENT:
- raise
-
- args = process_args()
- revs = check_output(git_cmd + ['rev-list'] + args.revisions).splitlines()
-
- make_shallow(shallow_file, args.revisions, args.refs)
-
- ref_revs = check_output(git_cmd + ['rev-list'] + args.refs).splitlines()
- remaining_history = set(revs) & set(ref_revs)
- for rev in remaining_history:
- if check_output(git_cmd + ['rev-parse', '{}^@'.format(rev)]):
- sys.exit('Error: %s was not made shallow' % rev)
-
- filter_refs(args.refs)
-
- if args.shrink:
- shrink_repo(git_dir)
- subprocess.check_call(git_cmd + ['fsck', '--unreachable'])
-
-
-def process_args():
- # TODO: add argument to automatically keep local-only refs, since they
- # can't be easily restored with a git fetch.
- parser = argparse.ArgumentParser(description='Remove the history of the specified revisions, then optionally filter the available refs to those specified.')
- parser.add_argument('--ref', '-r', metavar='REF', action='append', dest='refs', help='remove all but the specified refs (cumulative)')
- parser.add_argument('--shrink', '-s', action='store_true', help='shrink the git repository by repacking and pruning')
- parser.add_argument('revisions', metavar='REVISION', nargs='+', help='a git revision/commit')
- if len(sys.argv) < 2:
- parser.print_help()
- sys.exit(2)
-
- args = parser.parse_args()
-
- if args.refs:
- args.refs = check_output(git_cmd + ['rev-parse', '--symbolic-full-name'] + args.refs).splitlines()
- else:
- args.refs = get_all_refs(lambda r, t, tt: t == 'commit' or tt == 'commit')
-
- args.refs = list(filter(lambda r: not r.endswith('/HEAD'), args.refs))
- args.revisions = check_output(git_cmd + ['rev-parse'] + ['%s^{}' % i for i in args.revisions]).splitlines()
- return args
-
-
-def check_output(cmd, input=None):
- return subprocess.check_output(cmd, universal_newlines=True, input=input)
-
-
-def make_shallow(shallow_file, revisions, refs):
- """Remove the history of the specified revisions."""
- for rev in follow_history_intersections(revisions, refs):
- print("Processing %s" % rev)
- with open(shallow_file, 'a') as f:
- f.write(rev + '\n')
-
-
-def get_all_refs(ref_filter=None):
- """Return all the existing refs in this repository, optionally filtering the refs."""
- ref_output = check_output(git_cmd + ['for-each-ref', '--format=%(refname)\t%(objecttype)\t%(*objecttype)'])
- ref_split = [tuple(iter_extend(l.rsplit('\t'), 3)) for l in ref_output.splitlines()]
- if ref_filter:
- ref_split = (e for e in ref_split if ref_filter(*e))
- refs = [r[0] for r in ref_split]
- return refs
-
-
-def iter_extend(iterable, length, obj=None):
- """Ensure that iterable is the specified length by extending with obj."""
- return itertools.islice(itertools.chain(iterable, itertools.repeat(obj)), length)
-
-
-def filter_refs(refs):
- """Remove all but the specified refs from the git repository."""
- all_refs = get_all_refs()
- to_remove = set(all_refs) - set(refs)
- if to_remove:
- check_output(['xargs', '-0', '-n', '1'] + git_cmd + ['update-ref', '-d', '--no-deref'],
- input=''.join(l + '\0' for l in to_remove))
-
-
-def follow_history_intersections(revisions, refs):
- """Determine all the points where the history of the specified revisions intersects the specified refs."""
- queue = collections.deque(revisions)
- seen = set()
-
- for rev in iter_except(queue.popleft, IndexError):
- if rev in seen:
- continue
-
- parents = check_output(git_cmd + ['rev-parse', '%s^@' % rev]).splitlines()
-
- yield rev
- seen.add(rev)
-
- if not parents:
- continue
-
- check_refs = check_output(git_cmd + ['merge-base', '--independent'] + sorted(refs)).splitlines()
- for parent in parents:
- for ref in check_refs:
- print("Checking %s vs %s" % (parent, ref))
- try:
- merge_base = check_output(git_cmd + ['merge-base', parent, ref]).rstrip()
- except subprocess.CalledProcessError:
- continue
- else:
- queue.append(merge_base)
-
-
-def iter_except(func, exception, start=None):
- """Yield a function repeatedly until it raises an exception."""
- try:
- if start is not None:
- yield start()
- while True:
- yield func()
- except exception:
- pass
-
-
-def shrink_repo(git_dir):
- """Shrink the newly shallow repository, removing the unreachable objects."""
- subprocess.check_call(git_cmd + ['reflog', 'expire', '--expire-unreachable=now', '--all'])
- subprocess.check_call(git_cmd + ['repack', '-ad'])
- try:
- os.unlink(os.path.join(git_dir, 'objects', 'info', 'alternates'))
- except OSError as exc:
- if exc.errno != errno.ENOENT:
- raise
- subprocess.check_call(git_cmd + ['prune', '--expire', 'now'])
-
-
-if __name__ == '__main__':
- main()
diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index f002c8c1..16a64bab 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -164,14 +164,20 @@ fi
# in the local layers that currently make using an arbitrary
# toasterconf.json difficult.
-. $OE_ROOT/.templateconf
-if [ -n "$TEMPLATECONF" ]; then
- if [ ! -d "$TEMPLATECONF" ]; then
- # Allow TEMPLATECONF=meta-xyz/conf as a shortcut
- if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then
- TEMPLATECONF="$OE_ROOT/$TEMPLATECONF"
+# BBSETUP no longer supports .templateconf
+if [ -f $OE_ROOT/.templateconf ] ; then
+ . $OE_ROOT/.templateconf
+ if [ -n "$TEMPLATECONF" ]; then
+ if [ ! -d "$TEMPLATECONF" ]; then
+ # Allow TEMPLATECONF=meta-xyz/conf as a shortcut
+ if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then
+ TEMPLATECONF="$OE_ROOT/$TEMPLATECONF"
+ fi
fi
fi
+else
+ # Force poky
+ TEMPLATECONF="layers/meta-yocto/meta-poky/conf/templates/default"
fi
unset OE_ROOT
@@ -230,7 +236,7 @@ for param in $*; do
manage_cmd="$manage_cmd $param"
else
echo "$HELP"
- exit 1
+ return 1
fi
;;
@@ -255,7 +261,8 @@ fi
# 2) the build dir (in build)
# 3) the sqlite db if that is being used.
# 4) pid's we need to clean up on exit/shutdown
-export TOASTER_DIR=$TOASTERDIR
+# By default we move this all into the Toaster's parent build directory
+export TOASTER_DIR=$BUILDDIR
export BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS TOASTER_DIR"
# Determine the action. If specified by arguments, fine, if not, toggle it
diff --git a/bitbake/bin/toaster-eventreplay b/bitbake/bin/toaster-eventreplay
index 74a31932..742c2b78 100755
--- a/bitbake/bin/toaster-eventreplay
+++ b/bitbake/bin/toaster-eventreplay
@@ -31,6 +31,13 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib'))
import bb.cooker
from bb.ui import toasterui
from bb.ui import eventreplay
+from bb import __version__ as libbb_version
+
+__version__ = "2.20.0"
+
+if __name__ == "__main__":
+ if __version__ != libbb_version:
+ sys.exit("Bitbake core library version and program version mismatch!")
def main(argv):
with open(argv[-1]) as eventfile:
diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf
index f5a5a333..a8cd5c07 100644
--- a/bitbake/conf/bitbake.conf
+++ b/bitbake/conf/bitbake.conf
@@ -43,4 +43,4 @@ T = "${WORKDIR}/temp"
TARGET_ARCH = "${BUILD_ARCH}"
TMPDIR = "${TOPDIR}/tmp"
WORKDIR = "${TMPDIR}/work/${PF}"
-GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
+GITPKGV = "${@bb.fetch.get_srcrev(d, 'gitpkgv_revision')}"
diff --git a/bitbake/contrib/b4-wrapper-bitbake.py b/bitbake/contrib/b4-wrapper-bitbake.py
new file mode 100755
index 00000000..87dff2c3
--- /dev/null
+++ b/bitbake/contrib/b4-wrapper-bitbake.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+# This script is to be called by b4:
+# - through b4.send-auto-cc-cmd with "send-auto-cc-cmd" as first argument,
+#
+# When send-auto-cc-cmd is passed:
+#
+# This returns the list of Cc recipients for a patch.
+#
+# This script takes as stdin a patch.
+
+import subprocess
+import sys
+
+cmd = sys.argv[1]
+if cmd != "send-auto-cc-cmd":
+ sys.exit(-1)
+
+patch = sys.stdin.read()
+
+if subprocess.call(["which", "lsdiff"], stdout=subprocess.DEVNULL) != 0:
+ print("lsdiff missing from host, please install patchutils")
+ sys.exit(-1)
+
+files = subprocess.check_output(["lsdiff", "--strip-match=1", "--strip=1", "--include=doc/*"],
+ input=patch, text=True)
+if len(files):
+ print("docs@lists.yoctoproject.org")
+else:
+# Handle patches made with --no-prefix
+ files = subprocess.check_output(["lsdiff", "--include=doc/*"],
+ input=patch, text=True)
+ if len(files):
+ print("docs@lists.yoctoproject.org")
+
+sys.exit(0)
diff --git a/bitbake/contrib/hashserv/Dockerfile b/bitbake/contrib/hashserv/Dockerfile
index 74b4a3be..5c8b8dd1 100644
--- a/bitbake/contrib/hashserv/Dockerfile
+++ b/bitbake/contrib/hashserv/Dockerfile
@@ -9,14 +9,13 @@
# docker build -f contrib/hashserv/Dockerfile .
#
-FROM alpine:3.13.1
+FROM alpine:3.23
-RUN apk add --no-cache python3
+RUN apk add --no-cache python3 py3-websockets libgcc
COPY bin/bitbake-hashserv /opt/bbhashserv/bin/
COPY lib/hashserv /opt/bbhashserv/lib/hashserv/
COPY lib/bb /opt/bbhashserv/lib/bb/
-COPY lib/codegen.py /opt/bbhashserv/lib/codegen.py
COPY lib/ply /opt/bbhashserv/lib/ply/
COPY lib/bs4 /opt/bbhashserv/lib/bs4/
diff --git a/bitbake/contrib/prserv/Dockerfile b/bitbake/contrib/prserv/Dockerfile
index 9585fe3f..694b9b9f 100644
--- a/bitbake/contrib/prserv/Dockerfile
+++ b/bitbake/contrib/prserv/Dockerfile
@@ -28,14 +28,13 @@
# prserv
#
-FROM alpine:3.14.4
+FROM alpine:3.23
RUN apk add --no-cache python3
COPY bin/bitbake-prserv /opt/bbprserv/bin/
COPY lib/prserv /opt/bbprserv/lib/prserv/
COPY lib/bb /opt/bbprserv/lib/bb/
-COPY lib/codegen.py /opt/bbprserv/lib/codegen.py
COPY lib/ply /opt/bbprserv/lib/ply/
COPY lib/bs4 /opt/bbprserv/lib/bs4/
diff --git a/bitbake/contrib/vim/LICENSE.txt b/bitbake/contrib/vim/LICENSE.txt
deleted file mode 100644
index c7d91502..00000000
--- a/bitbake/contrib/vim/LICENSE.txt
+++ /dev/null
@@ -1,18 +0,0 @@
-The MIT License (MIT)
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
-the Software, and to permit persons to whom the Software is furnished to do so,
-subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/bitbake/contrib/vim/ftdetect/bitbake.vim b/bitbake/contrib/vim/ftdetect/bitbake.vim
deleted file mode 100644
index 09fc4dc7..00000000
--- a/bitbake/contrib/vim/ftdetect/bitbake.vim
+++ /dev/null
@@ -1,24 +0,0 @@
-" Vim filetype detection file
-" Language: BitBake
-" Author: Ricardo Salveti <rsalveti@rsalveti.net>
-" Copyright: Copyright (C) 2008 Ricardo Salveti <rsalveti@rsalveti.net>
-" Licence: You may redistribute this under the same terms as Vim itself
-"
-" This sets up the syntax highlighting for BitBake files, like .bb, .bbclass and .inc
-
-if &compatible || version < 600 || exists("b:loaded_bitbake_plugin")
- finish
-endif
-
-" .bb, .bbappend and .bbclass
-au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=bitbake
-
-" .inc
-au BufNewFile,BufRead *.inc set filetype=bitbake
-
-" .conf
-au BufNewFile,BufRead *.conf
- \ if (match(expand("%:p:h"), "conf") > 0) |
- \ set filetype=bitbake |
- \ endif
-
diff --git a/bitbake/contrib/vim/ftplugin/bitbake.vim b/bitbake/contrib/vim/ftplugin/bitbake.vim
deleted file mode 100644
index 9e8d3e13..00000000
--- a/bitbake/contrib/vim/ftplugin/bitbake.vim
+++ /dev/null
@@ -1,13 +0,0 @@
-" Only do this when not done yet for this buffer
-if exists("b:did_ftplugin")
- finish
-endif
-
-" Don't load another plugin for this buffer
-let b:did_ftplugin = 1
-
-let b:undo_ftplugin = "setl cms< sts< sw< et< sua<"
-
-setlocal commentstring=#\ %s
-setlocal softtabstop=4 shiftwidth=4 expandtab
-setlocal suffixesadd+=.bb,.bbclass
diff --git a/bitbake/contrib/vim/indent/bitbake.vim b/bitbake/contrib/vim/indent/bitbake.vim
deleted file mode 100644
index 7ee9d699..00000000
--- a/bitbake/contrib/vim/indent/bitbake.vim
+++ /dev/null
@@ -1,343 +0,0 @@
-" Vim indent file
-" Language: BitBake
-" Copyright: Copyright (C) 2019 Agilent Technologies, Inc.
-" Maintainer: Chris Laplante <chris.laplante@agilent.com>
-" License: You may redistribute this under the same terms as Vim itself
-
-
-if exists("b:did_indent")
- finish
-endif
-
-if exists("*BitbakeIndent")
- finish
-endif
-
-runtime! indent/sh.vim
-unlet b:did_indent
-
-setlocal indentexpr=BitbakeIndent(v:lnum)
-setlocal autoindent nolisp
-
-function s:is_bb_python_func_def(lnum)
- let stack = synstack(a:lnum, 1)
- if len(stack) == 0
- return 0
- endif
-
- let top = synIDattr(stack[0], "name")
- echo top
-
- return synIDattr(stack[0], "name") == "bbPyFuncDef"
-endfunction
-
-"""" begin modified from indent/python.vim, upstream commit 7a9bd7c1e0ce1baf5a02daf36eeae3638aa315c7
-"""" This copied code is licensed the same as Vim itself.
-setlocal indentkeys+=<:>,=elif,=except
-
-let s:keepcpo= &cpo
-set cpo&vim
-
-let s:maxoff = 50 " maximum number of lines to look backwards for ()
-
-function! GetBBPythonIndent(lnum)
-
- " If this line is explicitly joined: If the previous line was also joined,
- " line it up with that one, otherwise add two 'shiftwidth'
- if getline(a:lnum - 1) =~ '\\$'
- if a:lnum > 1 && getline(a:lnum - 2) =~ '\\$'
- return indent(a:lnum - 1)
- endif
- return indent(a:lnum - 1) + (exists("g:pyindent_continue") ? eval(g:pyindent_continue) : (shiftwidth() * 2))
- endif
-
- " If the start of the line is in a string don't change the indent.
- if has('syntax_items')
- \ && synIDattr(synID(a:lnum, 1, 1), "name") =~ "String$"
- return -1
- endif
-
- " Search backwards for the previous non-empty line.
- let plnum = prevnonblank(v:lnum - 1)
-
- if plnum == 0
- " This is the first non-empty line, use zero indent.
- return 0
- endif
-
- call cursor(plnum, 1)
-
- " Identing inside parentheses can be very slow, regardless of the searchpair()
- " timeout, so let the user disable this feature if he doesn't need it
- let disable_parentheses_indenting = get(g:, "pyindent_disable_parentheses_indenting", 0)
-
- if disable_parentheses_indenting == 1
- let plindent = indent(plnum)
- let plnumstart = plnum
- else
- " searchpair() can be slow sometimes, limit the time to 150 msec or what is
- " put in g:pyindent_searchpair_timeout
- let searchpair_stopline = 0
- let searchpair_timeout = get(g:, 'pyindent_searchpair_timeout', 150)
-
- " If the previous line is inside parenthesis, use the indent of the starting
- " line.
- " Trick: use the non-existing "dummy" variable to break out of the loop when
- " going too far back.
- let parlnum = searchpair('(\|{\|\[', '', ')\|}\|\]', 'nbW',
- \ "line('.') < " . (plnum - s:maxoff) . " ? dummy :"
- \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
- \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
- \ searchpair_stopline, searchpair_timeout)
- if parlnum > 0
- " We may have found the opening brace of a BitBake Python task, e.g. 'python do_task {'
- " If so, ignore it here - it will be handled later.
- if s:is_bb_python_func_def(parlnum)
- let parlnum = 0
- let plindent = indent(plnum)
- let plnumstart = plnum
- else
- let plindent = indent(parlnum)
- let plnumstart = parlnum
- endif
- else
- let plindent = indent(plnum)
- let plnumstart = plnum
- endif
-
- " When inside parenthesis: If at the first line below the parenthesis add
- " two 'shiftwidth', otherwise same as previous line.
- " i = (a
- " + b
- " + c)
- call cursor(a:lnum, 1)
- let p = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW',
- \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :"
- \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
- \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
- \ searchpair_stopline, searchpair_timeout)
- if p > 0
- if s:is_bb_python_func_def(p)
- " Handle first non-empty line inside a BB Python task
- if p == plnum
- return shiftwidth()
- endif
-
- " Handle the user actually trying to close a BitBake Python task
- let line = getline(a:lnum)
- if line =~ '^\s*}'
- return -2
- endif
-
- " Otherwise ignore the brace
- let p = 0
- else
- if p == plnum
- " When the start is inside parenthesis, only indent one 'shiftwidth'.
- let pp = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW',
- \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :"
- \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
- \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
- \ searchpair_stopline, searchpair_timeout)
- if pp > 0
- return indent(plnum) + (exists("g:pyindent_nested_paren") ? eval(g:pyindent_nested_paren) : shiftwidth())
- endif
- return indent(plnum) + (exists("g:pyindent_open_paren") ? eval(g:pyindent_open_paren) : (shiftwidth() * 2))
- endif
- if plnumstart == p
- return indent(plnum)
- endif
- return plindent
- endif
- endif
-
- endif
-
-
- " Get the line and remove a trailing comment.
- " Use syntax highlighting attributes when possible.
- let pline = getline(plnum)
- let pline_len = strlen(pline)
- if has('syntax_items')
- " If the last character in the line is a comment, do a binary search for
- " the start of the comment. synID() is slow, a linear search would take
- " too long on a long line.
- if synIDattr(synID(plnum, pline_len, 1), "name") =~ "\\(Comment\\|Todo\\)$"
- let min = 1
- let max = pline_len
- while min < max
- let col = (min + max) / 2
- if synIDattr(synID(plnum, col, 1), "name") =~ "\\(Comment\\|Todo\\)$"
- let max = col
- else
- let min = col + 1
- endif
- endwhile
- let pline = strpart(pline, 0, min - 1)
- endif
- else
- let col = 0
- while col < pline_len
- if pline[col] == '#'
- let pline = strpart(pline, 0, col)
- break
- endif
- let col = col + 1
- endwhile
- endif
-
- " If the previous line ended with a colon, indent this line
- if pline =~ ':\s*$'
- return plindent + shiftwidth()
- endif
-
- " If the previous line was a stop-execution statement...
- " TODO: utilize this logic to deindent when ending a bbPyDefRegion
- if getline(plnum) =~ '^\s*\(break\|continue\|raise\|return\|pass\|bb\.fatal\)\>'
- " See if the user has already dedented
- if indent(a:lnum) > indent(plnum) - shiftwidth()
- " If not, recommend one dedent
- return indent(plnum) - shiftwidth()
- endif
- " Otherwise, trust the user
- return -1
- endif
-
- " If the current line begins with a keyword that lines up with "try"
- if getline(a:lnum) =~ '^\s*\(except\|finally\)\>'
- let lnum = a:lnum - 1
- while lnum >= 1
- if getline(lnum) =~ '^\s*\(try\|except\)\>'
- let ind = indent(lnum)
- if ind >= indent(a:lnum)
- return -1 " indent is already less than this
- endif
- return ind " line up with previous try or except
- endif
- let lnum = lnum - 1
- endwhile
- return -1 " no matching "try"!
- endif
-
- " If the current line begins with a header keyword, dedent
- if getline(a:lnum) =~ '^\s*\(elif\|else\)\>'
-
- " Unless the previous line was a one-liner
- if getline(plnumstart) =~ '^\s*\(for\|if\|try\)\>'
- return plindent
- endif
-
- " Or the user has already dedented
- if indent(a:lnum) <= plindent - shiftwidth()
- return -1
- endif
-
- return plindent - shiftwidth()
- endif
-
- " When after a () construct we probably want to go back to the start line.
- " a = (b
- " + c)
- " here
- if parlnum > 0
- return plindent
- endif
-
- return -1
-
-endfunction
-
-let &cpo = s:keepcpo
-unlet s:keepcpo
-
-""" end of stuff from indent/python.vim
-
-
-let b:did_indent = 1
-setlocal indentkeys+=0\"
-
-
-function! BitbakeIndent(lnum)
- if !has('syntax_items')
- return -1
- endif
-
- let stack = synstack(a:lnum, 1)
- if len(stack) == 0
- return -1
- endif
-
- let name = synIDattr(stack[0], "name")
-
- " TODO: support different styles of indentation for assignments. For now,
- " we only support like this:
- " VAR = " \
- " value1 \
- " value2 \
- " "
- "
- " i.e. each value indented by shiftwidth(), with the final quote " completely unindented.
- if name == "bbVarValue"
- " Quote handling is tricky. kernel.bbclass has this line for instance:
- " EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" " HOSTCPP="${BUILD_CPP}""
- " Instead of trying to handle crazy cases like that, just assume that a
- " double-quote on a line by itself (following an assignment) means the
- " user is closing the assignment, and de-dent.
- if getline(a:lnum) =~ '^\s*"$'
- return 0
- endif
-
- let prevstack = synstack(a:lnum - 1, 1)
- if len(prevstack) == 0
- return -1
- endif
-
- let prevname = synIDattr(prevstack[0], "name")
-
- " Only indent if there was actually a continuation character on
- " the previous line, to avoid misleading indentation.
- let prevlinelastchar = synIDattr(synID(a:lnum - 1, col([a:lnum - 1, "$"]) - 1, 1), "name")
- let prev_continued = prevlinelastchar == "bbContinue"
-
- " Did the previous line introduce an assignment?
- if index(["bbVarDef", "bbVarFlagDef"], prevname) != -1
- if prev_continued
- return shiftwidth()
- endif
- endif
-
- if !prev_continued
- return 0
- endif
-
- " Autoindent can take it from here
- return -1
- endif
-
- if index(["bbPyDefRegion", "bbPyFuncRegion"], name) != -1
- let ret = GetBBPythonIndent(a:lnum)
- " Should normally always be indented by at least one shiftwidth; but allow
- " return of -1 (defer to autoindent) or -2 (force indent to 0)
- if ret == 0
- return shiftwidth()
- elseif ret == -2
- return 0
- endif
- return ret
- endif
-
- " TODO: GetShIndent doesn't detect tasks prepended with 'fakeroot'
- " Need to submit a patch upstream to Vim to provide an extension point.
- " Unlike the Python indenter, the Sh indenter is way too large to copy and
- " modify here.
- if name == "bbShFuncRegion"
- return GetShIndent()
- endif
-
- " TODO:
- " + heuristics for de-denting out of a bbPyDefRegion? e.g. when the user
- " types an obvious BB keyword like addhandler or addtask, or starts
- " writing a shell task. Maybe too hard to implement...
-
- return -1
-endfunction
diff --git a/bitbake/contrib/vim/plugin/newbb.vim b/bitbake/contrib/vim/plugin/newbb.vim
deleted file mode 100644
index 3a420273..00000000
--- a/bitbake/contrib/vim/plugin/newbb.vim
+++ /dev/null
@@ -1,88 +0,0 @@
-" Vim plugin file
-" Purpose: Create a template for new bb files
-" Author: Ricardo Salveti <rsalveti@gmail.com>
-" Copyright: Copyright (C) 2008 Ricardo Salveti <rsalveti@gmail.com>
-"
-" This file is licensed under the MIT license, see COPYING.MIT in
-" this source distribution for the terms.
-"
-" Based on the gentoo-syntax package
-"
-" Will try to use git to find the user name and email
-
-if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin")
- finish
-endif
-
-fun! <SID>GetUserName()
- let l:user_name = system("git config --get user.name")
- if v:shell_error
- return "Unknown User"
- else
- return substitute(l:user_name, "\n", "", "")
-endfun
-
-fun! <SID>GetUserEmail()
- let l:user_email = system("git config --get user.email")
- if v:shell_error
- return "unknown@user.org"
- else
- return substitute(l:user_email, "\n", "", "")
-endfun
-
-fun! BBHeader()
- let l:current_year = strftime("%Y")
- let l:user_name = <SID>GetUserName()
- let l:user_email = <SID>GetUserEmail()
- 0 put ='# Copyright (C) ' . l:current_year .
- \ ' ' . l:user_name . ' <' . l:user_email . '>'
- put ='# Released under the MIT license (see COPYING.MIT for the terms)'
- $
-endfun
-
-fun! NewBBTemplate()
- if line2byte(line('$') + 1) != -1
- return
- endif
-
- let l:paste = &paste
- set nopaste
-
- " Get the header
- call BBHeader()
-
- " New the bb template
- put ='SUMMARY = \"\"'
- put ='HOMEPAGE = \"\"'
- put ='LICENSE = \"\"'
- put ='SECTION = \"\"'
- put ='DEPENDS = \"\"'
- put =''
- put ='SRC_URI = \"\"'
-
- " Go to the first place to edit
- 0
- /^SUMMARY =/
- exec "normal 2f\""
-
- if paste == 1
- set paste
- endif
-endfun
-
-if !exists("g:bb_create_on_empty")
- let g:bb_create_on_empty = 1
-endif
-
-" disable in case of vimdiff
-if v:progname =~ "vimdiff"
- let g:bb_create_on_empty = 0
-endif
-
-augroup NewBB
- au BufNewFile,BufReadPost *.bb
- \ if g:bb_create_on_empty |
- \ call NewBBTemplate() |
- \ endif
-augroup END
-
diff --git a/bitbake/contrib/vim/plugin/newbbappend.vim b/bitbake/contrib/vim/plugin/newbbappend.vim
deleted file mode 100644
index 3f65f79c..00000000
--- a/bitbake/contrib/vim/plugin/newbbappend.vim
+++ /dev/null
@@ -1,46 +0,0 @@
-" Vim plugin file
-" Purpose: Create a template for new bbappend file
-" Author: Joshua Watt <JPEWhacker@gmail.com>
-" Copyright: Copyright (C) 2017 Joshua Watt <JPEWhacker@gmail.com>
-"
-" This file is licensed under the MIT license, see COPYING.MIT in
-" this source distribution for the terms.
-"
-
-if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin")
- finish
-endif
-
-fun! NewBBAppendTemplate()
- if line2byte(line('$') + 1) != -1
- return
- endif
-
- let l:paste = &paste
- set nopaste
-
- " New bbappend template
- 0 put ='FILESEXTRAPATHS:prepend := \"${THISDIR}/${PN}:\"'
- 2
-
- if paste == 1
- set paste
- endif
-endfun
-
-if !exists("g:bb_create_on_empty")
- let g:bb_create_on_empty = 1
-endif
-
-" disable in case of vimdiff
-if v:progname =~ "vimdiff"
- let g:bb_create_on_empty = 0
-endif
-
-augroup NewBBAppend
- au BufNewFile,BufReadPost *.bbappend
- \ if g:bb_create_on_empty |
- \ call NewBBAppendTemplate() |
- \ endif
-augroup END
-
diff --git a/bitbake/contrib/vim/syntax/bitbake.vim b/bitbake/contrib/vim/syntax/bitbake.vim
deleted file mode 100644
index 8f39b8f9..00000000
--- a/bitbake/contrib/vim/syntax/bitbake.vim
+++ /dev/null
@@ -1,131 +0,0 @@
-" Vim syntax file
-" Language: BitBake bb/bbclasses/inc
-" Author: Chris Larson <kergoth@handhelds.org>
-" Ricardo Salveti <rsalveti@rsalveti.net>
-" Copyright: Copyright (C) 2004 Chris Larson <kergoth@handhelds.org>
-" Copyright (C) 2008 Ricardo Salveti <rsalveti@rsalveti.net>
-"
-" This file is licensed under the MIT license, see COPYING.MIT in
-" this source distribution for the terms.
-"
-" Syntax highlighting for bb, bbclasses and inc files.
-"
-" It's an entirely new type, just has specific syntax in shell and python code
-
-if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin")
- finish
-endif
-if exists("b:current_syntax")
- finish
-endif
-
-syn include @python syntax/python.vim
-if exists("b:current_syntax")
- unlet b:current_syntax
-endif
-
-" BitBake syntax
-
-" Matching case
-syn case match
-
-" Indicates the error when nothing is matched
-syn match bbUnmatched "."
-
-" Comments
-syn cluster bbCommentGroup contains=bbTodo,@Spell
-syn keyword bbTodo COMBAK FIXME TODO XXX contained
-syn match bbComment "#.*$" contains=@bbCommentGroup
-
-" String helpers
-syn match bbQuote +['"]+ contained
-syn match bbDelimiter "[(){}=]" contained
-syn match bbArrayBrackets "[\[\]]" contained
-
-" BitBake strings
-syn match bbContinue "\\$"
-syn region bbString matchgroup=bbQuote start=+"+ skip=+\\$+ end=+"+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
-syn region bbString matchgroup=bbQuote start=+'+ skip=+\\$+ end=+'+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
-
-" Vars definition
-syn match bbExport "^export" nextgroup=bbIdentifier skipwhite
-syn keyword bbExportFlag export contained nextgroup=bbIdentifier skipwhite
-syn match bbIdentifier "[a-zA-Z0-9\-_\.\/\+]\+" display contained
-syn match bbVarDeref "${[a-zA-Z0-9\-_:\.\/\+]\+}" contained
-syn match bbVarEq "\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)" contained nextgroup=bbVarValue
-syn match bbVarDef "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+][${}a-zA-Z0-9\-_:\.\/\+]*\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbOverrideOperator,bbVarDeref nextgroup=bbVarEq
-syn match bbVarValue ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue
-syn region bbVarPyValue start=+${@+ skip=+\\$+ end=+}+ contained contains=@python
-
-" Vars metadata flags
-syn match bbVarFlagDef "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.+]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag
-syn region bbVarFlagFlag matchgroup=bbArrayBrackets start="\[" end="\]\s*\(:=\|=\|.=\|=.|+=\|=+\|?=\)\@=" contained contains=bbIdentifier nextgroup=bbVarEq
-
-" Includes and requires
-syn keyword bbInclude inherit include require contained
-syn match bbIncludeRest ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue
-syn match bbIncludeLine "^\(inherit\|include\|require\)\s\+" contains=bbInclude nextgroup=bbIncludeRest
-
-" Add taks and similar
-syn keyword bbStatement addtask deltask addhandler after before EXPORT_FUNCTIONS contained
-syn match bbStatementRest /[^\\]*$/ skipwhite contained contains=bbStatement,bbVarDeref,bbVarPyValue
-syn region bbStatementRestCont start=/.*\\$/ end=/^[^\\]*$/ contained contains=bbStatement,bbVarDeref,bbVarPyValue,bbContinue keepend
-syn match bbStatementLine "^\(addtask\|deltask\|addhandler\|after\|before\|EXPORT_FUNCTIONS\)\s\+" contains=bbStatement nextgroup=bbStatementRest,bbStatementRestCont
-
-" OE Important Functions
-syn keyword bbOEFunctions do_fetch do_unpack do_patch do_configure do_compile do_stage do_install do_package contained
-
-" Generic Functions
-syn match bbFunction "\h[0-9A-Za-z_\-\.]*" display contained contains=bbOEFunctions
-
-syn keyword bbOverrideOperator append prepend remove contained
-
-" BitBake shell metadata
-syn include @shell syntax/sh.vim
-if exists("b:current_syntax")
- unlet b:current_syntax
-endif
-syn keyword bbShFakeRootFlag fakeroot contained
-syn match bbShFuncDef "^\(fakeroot\s*\)\?\([\.0-9A-Za-z_:${}\-\.]\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbFunction,bbOverrideOperator,bbVarDeref,bbDelimiter nextgroup=bbShFuncRegion skipwhite
-syn region bbShFuncRegion matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@shell
-
-" Python value inside shell functions
-syn region shDeref start=+${@+ skip=+\\$+ excludenl end=+}+ contained contains=@python
-
-" BitBake python metadata
-syn keyword bbPyFlag python contained
-syn match bbPyFuncDef "^\(fakeroot\s*\)\?\(python\)\(\s\+[0-9A-Za-z_:${}\-\.]\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbPyFlag,bbFunction,bbOverrideOperator,bbVarDeref,bbDelimiter nextgroup=bbPyFuncRegion skipwhite
-syn region bbPyFuncRegion matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@python
-
-" BitBake 'def'd python functions
-syn keyword bbPyDef def contained
-syn region bbPyDefRegion start='^\(def\s\+\)\([0-9A-Za-z_-]\+\)\(\s*(.*)\s*\):\s*$' end='^\(\s\|$\)\@!' contains=@python
-
-" Highlighting Definitions
-hi def link bbUnmatched Error
-hi def link bbInclude Include
-hi def link bbTodo Todo
-hi def link bbComment Comment
-hi def link bbQuote String
-hi def link bbString String
-hi def link bbDelimiter Keyword
-hi def link bbArrayBrackets Statement
-hi def link bbContinue Special
-hi def link bbExport Type
-hi def link bbExportFlag Type
-hi def link bbIdentifier Identifier
-hi def link bbVarDeref PreProc
-hi def link bbVarDef Identifier
-hi def link bbVarValue String
-hi def link bbShFakeRootFlag Type
-hi def link bbFunction Function
-hi def link bbPyFlag Type
-hi def link bbPyDef Statement
-hi def link bbStatement Statement
-hi def link bbStatementRest Identifier
-hi def link bbStatementRestCont Identifier
-hi def link bbOEFunctions Special
-hi def link bbVarPyValue PreProc
-hi def link bbOverrideOperator Operator
-
-let b:current_syntax = "bb"
diff --git a/bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json b/bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json
new file mode 100644
index 00000000..c970a96e
--- /dev/null
+++ b/bitbake/default-registry/configurations/oe-nodistro-blacksail.conf.json
@@ -0,0 +1,48 @@
+{
+ "description": "OpenEmbedded - 'nodistro' basic configuration, release 6.0 'blacksail'",
+ "expires": "2027-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "2.20",
+ "rev": "2.20"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "blacksail",
+ "rev": "blacksail"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "blacksail",
+ "rev": "blacksail"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "nodistro",
+ "description": "OpenEmbedded 'nodistro'",
+ "setup-dir-name": "oe-nodistro-blacksail",
+ "bb-layers": ["openembedded-core/meta"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" }
+ ]
+ }
+ }
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/oe-nodistro-master.conf.json b/bitbake/default-registry/configurations/oe-nodistro-master.conf.json
new file mode 100644
index 00000000..c5d96661
--- /dev/null
+++ b/bitbake/default-registry/configurations/oe-nodistro-master.conf.json
@@ -0,0 +1,48 @@
+{
+ "description": "OpenEmbedded - 'nodistro' basic configuration",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "master",
+ "rev": "master"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "master",
+ "rev": "master"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "master",
+ "rev": "master"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "nodistro",
+ "description": "OpenEmbedded 'nodistro'",
+ "setup-dir-name": "oe-nodistro-master",
+ "bb-layers": ["openembedded-core/meta"],
+ "oe-fragments": ["core/yocto/monitor-disk-space"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" }
+ ]
+ }
+ }
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json b/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json
new file mode 100644
index 00000000..7b183ae1
--- /dev/null
+++ b/bitbake/default-registry/configurations/oe-nodistro-whinlatter.conf.json
@@ -0,0 +1,56 @@
+{
+ "description": "OpenEmbedded - 'nodistro' basic configuration, release 5.3 'whinlatter'",
+ "expires": "2026-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ }
+ },
+ "branch": "2.16",
+ "rev": "2.16"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/openembedded-core"
+ }
+ },
+ "branch": "whinlatter",
+ "rev": "whinlatter"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.yoctoproject.org/yocto-docs"
+ }
+ },
+ "branch": "whinlatter",
+ "rev": "whinlatter"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "nodistro",
+ "description": "OpenEmbedded 'nodistro'",
+ "setup-dir-name": "oe-nodistro-whinlatter",
+ "bb-layers": ["openembedded-core/meta"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64"]
+ }
+ }
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json b/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json
new file mode 100644
index 00000000..278ce4dd
--- /dev/null
+++ b/bitbake/default-registry/configurations/oe-nodistro-wrynose.conf.json
@@ -0,0 +1,48 @@
+{
+ "description": "OpenEmbedded - 'nodistro' basic configuration, release 6.0 'wrynose'",
+ "expires": "2030-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "2.18",
+ "rev": "2.18"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "wrynose",
+ "rev": "wrynose"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "wrynose",
+ "rev": "wrynose"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "nodistro",
+ "description": "OpenEmbedded 'nodistro'",
+ "setup-dir-name": "oe-nodistro-wrynose",
+ "bb-layers": ["openembedded-core/meta"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" }
+ ]
+ }
+ }
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/poky-blacksail.conf.json b/bitbake/default-registry/configurations/poky-blacksail.conf.json
new file mode 100644
index 00000000..ff09df99
--- /dev/null
+++ b/bitbake/default-registry/configurations/poky-blacksail.conf.json
@@ -0,0 +1,74 @@
+{
+ "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 6.1 'blacksail'",
+ "expires": "2027-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "2.20",
+ "rev": "2.20"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "blacksail",
+ "rev": "blacksail"
+ }
+ },
+ "meta-yocto": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/meta-yocto",
+ "branch": "blacksail",
+ "rev": "blacksail"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "blacksail",
+ "rev": "blacksail"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "setup-dir-name": "$distro-blacksail",
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" },
+ { "name": "machine/genericarm64", "description": "Arm64 SystemReady IR/ES platforms" },
+ { "name": "machine/genericx86-64", "description": "x86_64 (64-bit) PCs and servers" }
+ ]
+ },
+ "distro": {
+ "description": "Target distributions",
+ "options" : [
+ { "name": "distro/poky", "description": "Yocto Project Reference Distro" },
+ { "name": "distro/poky-altcfg", "description": "Poky alternative with systemd as init manager" },
+ { "name": "distro/poky-tiny", "description": "Poky alternative optimized for size" }
+ ]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/poky-master.conf.json b/bitbake/default-registry/configurations/poky-master.conf.json
new file mode 100644
index 00000000..452d933a
--- /dev/null
+++ b/bitbake/default-registry/configurations/poky-master.conf.json
@@ -0,0 +1,74 @@
+{
+ "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "master",
+ "rev": "master"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "master",
+ "rev": "master"
+ }
+ },
+ "meta-yocto": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/meta-yocto",
+ "branch": "master",
+ "rev": "master"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "master",
+ "rev": "master"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "setup-dir-name": "$distro-master",
+ "oe-fragments": ["core/yocto/monitor-disk-space"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" },
+ { "name": "machine/genericarm64", "description": "Arm64 SystemReady IR/ES platforms" },
+ { "name": "machine/genericx86-64", "description": "x86_64 (64-bit) PCs and servers" }
+ ]
+ },
+ "distro": {
+ "description": "Target distributions",
+ "options" : [
+ { "name": "distro/poky", "description": "Yocto Project Reference Distro" },
+ { "name": "distro/poky-altcfg", "description": "Poky alternative with systemd as init manager" },
+ { "name": "distro/poky-tiny", "description": "Poky alternative optimized for size" }
+ ]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/poky-whinlatter.conf.json b/bitbake/default-registry/configurations/poky-whinlatter.conf.json
new file mode 100644
index 00000000..4827ebe1
--- /dev/null
+++ b/bitbake/default-registry/configurations/poky-whinlatter.conf.json
@@ -0,0 +1,80 @@
+{
+ "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 5.3 'whinlatter'",
+ "expires": "2026-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ }
+ },
+ "branch": "2.16",
+ "rev": "2.16"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/openembedded-core"
+ }
+ },
+ "branch": "whinlatter",
+ "rev": "whinlatter"
+ }
+ },
+ "meta-yocto": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.yoctoproject.org/meta-yocto"
+ }
+ },
+ "branch": "whinlatter",
+ "rev": "whinlatter"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.yoctoproject.org/yocto-docs"
+ }
+ },
+ "branch": "whinlatter",
+ "rev": "whinlatter"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "setup-dir-name": "$distro-whinlatter",
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"]
+ },
+ "distro": {
+ "description": "Target Distributions",
+ "options" : ["distro/poky", "distro/poky-altcfg", "distro/poky-tiny"]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/default-registry/configurations/poky-wrynose.conf.json b/bitbake/default-registry/configurations/poky-wrynose.conf.json
new file mode 100644
index 00000000..d2792843
--- /dev/null
+++ b/bitbake/default-registry/configurations/poky-wrynose.conf.json
@@ -0,0 +1,74 @@
+{
+ "description": "Poky - The Yocto Project testing distribution configurations and hardware test platforms, release 6.0 'wrynose'",
+ "expires": "2030-05-31",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "2.18",
+ "rev": "2.18"
+ }
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "branch": "wrynose",
+ "rev": "wrynose"
+ }
+ },
+ "meta-yocto": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/meta-yocto",
+ "branch": "wrynose",
+ "rev": "wrynose"
+ }
+ },
+ "yocto-docs": {
+ "git-remote": {
+ "uri": "https://git.yoctoproject.org/yocto-docs",
+ "branch": "wrynose",
+ "rev": "wrynose"
+ }
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "setup-dir-name": "$distro-wrynose",
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "RISC-V system on QEMU" },
+ { "name": "machine/genericarm64", "description": "Arm64 SystemReady IR/ES platforms" },
+ { "name": "machine/genericx86-64", "description": "x86_64 (64-bit) PCs and servers" }
+ ]
+ },
+ "distro": {
+ "description": "Target distributions",
+ "options" : [
+ { "name": "distro/poky", "description": "Yocto Project Reference Distro" },
+ { "name": "distro/poky-altcfg", "description": "Poky alternative with systemd as init manager" },
+ { "name": "distro/poky-tiny", "description": "Poky alternative optimized for size" }
+ ]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ },
+ "version": "1.0"
+}
diff --git a/bitbake/doc/.gitignore b/bitbake/doc/.gitignore
index 69fa449d..3a570d8c 100644
--- a/bitbake/doc/.gitignore
+++ b/bitbake/doc/.gitignore
@@ -1 +1,4 @@
_build/
+sphinx-static/switchers.js
+releases.json
+releases.rst
diff --git a/bitbake/doc/Makefile b/bitbake/doc/Makefile
index 996f01b7..2f8e898c 100644
--- a/bitbake/doc/Makefile
+++ b/bitbake/doc/Makefile
@@ -27,7 +27,7 @@ publish: Makefile html singlehtml
sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html
clean:
- @rm -rf $(BUILDDIR)
+ @rm -rf $(BUILDDIR) sphinx-static/switchers.js releases.json releases.rst
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
diff --git a/bitbake/doc/README b/bitbake/doc/README
index d4f56afa..9869d6e3 100644
--- a/bitbake/doc/README
+++ b/bitbake/doc/README
@@ -26,7 +26,7 @@ Additional information related to the Sphinx migration, and guidelines
for developers willing to contribute to the BitBake documentation can
be found in the Yocto Project Documentation README file:
-https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/README
+https://git.yoctoproject.org/yocto-docs/tree/documentation/README
How to build the Yocto Project documentation
============================================
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
new file mode 100644
index 00000000..e15c9ed0
--- /dev/null
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst
@@ -0,0 +1,1332 @@
+.. SPDX-License-Identifier: CC-BY-2.5
+
+=============================================
+Setting Up The Environment With bitbake-setup
+=============================================
+
+|
+
+Setting up layers and configuring builds can be done with the
+``bitbake-setup`` tool. This tool acts as a top-level utility which can perform
+the following tasks:
+
+- Parse a JSON configuration file that describes what layers and which snippets
+ of configuration to use.
+
+- Clone the layers onto the versions specified in the configuration file.
+
+- Create and setup a directory ready for building what is specified in
+ configuration files.
+
+- Behave according to global or per-project settings.
+
+- Synchronize with upstream configuration changes.
+
+Quick Start
+===========
+
+#. First, run the following command to install the ``bitbake-setup``
+ command-line tool in a Python `virtual environment
+ <https://docs.python.org/3/library/venv.html>`__:
+
+ .. code-block:: shell
+
+ $ python3 -m venv --clear ./bitbake-setup-venv
+ $ . ./bitbake-setup-venv/bin/activate
+ $ pip install bitbake-setup
+
+ .. tip::
+
+ To update ``bitbake-setup`` when already installed, you can run:
+
+ .. code-block:: console
+
+ $ pip install --upgrade bitbake-setup
+
+#. To start using ``bitbake-setup``, run:
+
+ .. code-block:: shell
+
+ $ bitbake-setup init
+
+ This command will ask you to choose which configurations to use available as
+ part of the default BitBake :term:`Configuration Templates <Configuration
+ Template>`.
+
+ .. note::
+
+ These default configurations are located in ``./bitbake/default-registry/``.
+ See the :ref:`ref-bbsetup-section-config-reference` section to learn more
+ about ``bitbake-setup`` input configuration files.
+
+#. Here is an example of files and directories structure created by ``bitbake-setup init``::
+
+ ./bitbake-builds/
+ ├── site.conf
+ └── poky-master-poky-distro_poky-machine_qemux86-64/
+ ├── build/
+ ├── config/
+ └── layers/
+
+ With:
+
+ - ``./bitbake-builds``: the :term:`Top Directory`, where ``bitbake-setup``
+ configures everything. This directory can be configured with the
+ :ref:`ref-bbsetup-setting-top-dir-prefix` and
+ :ref:`ref-bbsetup-setting-top-dir-name` settings.
+
+ - ``poky-master-poky-distro_poky-machine_qemux86-64``: a :term:`Setup`
+ directory, which holds a :term:`Setup`: a result of the choices made
+ during the ``bitbake-setup init`` execution.
+
+ The name of the directory will vary depending on the choices.
+
+ - ``config/``: holds the :term:`Configuration Instance`, which embeds the
+ :term:`Configuration Template` (first choice of the :ref:`ref-bbsetup-command-init` command)
+ and the choices made during the initialization.
+
+ - ``build/``: the :term:`BitBake Build` directory, where BitBake stores
+ its own configuration and outputs for the builds.
+
+ - ``layers/``: the directory where :ref:`layers
+ <bitbake-user-manual/bitbake-user-manual-intro:Layers>` and other
+ repositories managed by ``bitbake-setup`` are stored and updated.
+
+ - ``site.conf``: a BitBake configuration file that contains site specific
+ configurations for your build environment. When it is created, it contains
+ some variable definitions that are based on your current :term:`settings`.
+ Comments in this file will help you understand what these variables
+ correspond to.
+
+ These configurations are shared between the :term:`setups <Setup>`.
+
+#. Source the ``init-build-env`` file present in the :term:`BitBake Build`
+ directory:
+
+ .. code-block:: shell
+
+ $ source ./poky-master-poky-distro_poky-machine_qemux86-64/build/init-build-env
+
+ This command will prepare your current shell with the BitBake environment.
+
+#. You can then start running ``bitbake`` in the current shell. For more information
+ on how to use ``bitbake``, see the :doc:`/bitbake-user-manual/bitbake-user-manual-execution`
+ section of this manual.
+
+.. note::
+
+ The ``bitbake-setup`` tool is developed and maintained in the BitBake
+ repository and can also be used directly from there:
+
+ .. code-block:: console
+
+ $ git clone https://git.openembedded.org/bitbake ./bitbake
+ $ ./bitbake/bin/bitbake-setup ...
+
+Terminology
+===========
+
+The ``bitbake-setup`` tool revolves around some common terms we define in this
+section.
+
+``bitbake-setup`` works with a specific hierarchy of files and directories, that
+can be represented as follows::
+
+ Top Directory
+ ├── site.conf
+ ├── Setup 1
+ │ ├── build/
+ │ ├── config/
+ │ └── layers/
+ ├── Setup 2
+ │ ├── build/
+ │ ├── config/
+ │ └── layers/
+ ...
+
+The "Top Directory" and "Setup" directories are defined as follows:
+
+.. glossary::
+ :term:`Top Directory`
+ The top directory is the working directory of ``bitbake-setup``, where its
+ outputs end-up (unless otherwise configured by :term:`settings` such as
+ :ref:`ref-bbsetup-setting-dl-dir`).
+
+ The location of this directory can be changed with the
+ :ref:`ref-bbsetup-setting-top-dir-prefix` and
+ :ref:`ref-bbsetup-setting-top-dir-name` settings.
+
+ The top directory contains one or more :term:`Setup` directories, each of
+ them containing a :term:`Setup`, and a :term:`Site Configuration File`
+ (named ``site.conf``).
+
+ :term:`Setup`
+ A Setup is the result of the :ref:`ref-bbsetup-command-init`
+ command, which creates a :term:`Setup` directory. It is constructed from a
+ :term:`Configuration Template` and choices made during the ``init`` command.
+
+ It contains at least:
+
+ - A :term:`BitBake Build` (``build/`` directory).
+ - A :term:`Configuration Instance` (``config/`` directory).
+ - Sources such as :ref:`layers
+ <bitbake-user-manual/bitbake-user-manual-intro:Layers>` or other
+ repositories managed by ``bitbake-setup`` (``layers/`` directory).
+
+ :term:`Site Configuration File`
+ This file named ``site.conf`` is a unique file located in the :term:`Top
+ Directory`, and holds top-level BitBake configuration statements shared
+ between the :term:`Setups <Setup>`.
+
+ When it is created, it contains some variable definitions that are based
+ on your current :term:`settings`. Comments in this file will help you
+ understand what these variables correspond to.
+
+The following components are involved to create the content of these directories:
+
+.. glossary::
+ :term:`BitBake Build`
+ A BitBake Build is a sub-tree inside a :term:`Setup` that BitBake itself
+ operates on. The files in the ``conf/`` directory of a :term:`BitBake
+ Build` constitute the :ref:`BitBake configuration
+ <bitbake-user-manual/bitbake-user-manual-intro:Configuration Files>`.
+
+ :term:`Configuration Template`
+ A Configuration Template is a file in JSON format containing a template to
+ create a :term:`Setup`. These files are used during the :ref:`ref-bbsetup-command-init`
+ command as a starting point to configure the :term:`Setup`. When the
+ command runs, the user may be prompted with choices to further specify the
+ :term:`Setup` to create.
+
+ It is also possible to specify the choices on the command line for a
+ completely non-interactive initialization.
+
+ :term:`Configuration Template` files are stored in :term:`registries
+ <Registry>`, and can be listed with the :ref:`ref-bbsetup-command-list`
+ command.
+
+ :term:`Configuration Template` files must end with the ``.conf.json``
+ suffix for ``bitbake-setup`` to locate them.
+
+ .. note::
+
+ The default :term:`Configuration Templates <Configuration Template>` are
+ located in the BitBake repository in a local registry. the
+ ``default-registry/`` directory. This can be modified with the
+ :ref:`ref-bbsetup-setting-registry` setting.
+
+ :ref:`ref-bbsetup-command-status` will tell if a :term:`Setup`
+ is in sync with the :term:`Configuration Template` it was constructed from
+ (typically: layer updates).
+
+ :ref:`ref-bbsetup-command-update` will bring a :term:`Setup`
+ in sync with its :term:`Configuration Template`.
+
+ :term:`Configuration Instance`
+ The :term:`Configuration Instance` is stored in the ``config/`` directory
+ in a :term:`Setup`. It embeds the :term:`Configuration Template` and the
+ choices made during the initialization.
+
+ It is also a Git repository, that contains a history of the configuration
+ instance and updates made to it via :ref:`ref-bbsetup-command-update`.
+
+ :term:`Registry`
+ A configuration registry is a place where one or more :term:`Configuration
+ Templates <Configuration Template>` are stored.
+
+ The directory structure of the registry can be any: ``bitbake-setup``
+ recursively find files ending with ``.conf.json`` and consider it a
+ :term:`Configuration Template`.
+
+ The registry location is configured through the
+ :ref:`ref-bbsetup-setting-registry` setting. This location can be the URL to
+ a Git repository, a local directory, or any URI supported by the BitBake
+ fetcher (see the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching`
+ section for more information on fetchers).
+
+ :term:`Settings`
+ Settings are operational parameters that are global to all builds under a
+ :term:`Top Directory`, stored in a ``settings.conf`` file. For example,
+ this could be the location of the configuration registry, or where the
+ BitBake fetcher should store the downloads.
+
+ There are also global settings, common to all top directories that are
+ stored in ``~/.config/bitbake-setup/settings.conf``.
+
+ See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+ section to see the supported settings and where they can be stored.
+
+ :term:`Source Override`
+ A source override is a JSON file that can be used to modify revisions and
+ origins of layers or other sources that need to be checked out into a
+ :term:`Setup` (in the ``layers/`` directory). It can be useful for example
+ when the master branches need to be changed to master-next for the purpose
+ of testing, or to set up a CI pipeline that tests code in a pull request
+ coming from a developer's repository and branch.
+
+ Such a file is specified with a command-line option to
+ :ref:`ref-bbsetup-command-init`.
+
+ See the :ref:`ref-bbsetup-source-overrides` section for more information on
+ the format of these files.
+
+The ``bitbake-setup`` command
+=============================
+
+The ``bitbake-setup`` program has general options and sub-commands. These can be
+obtained using ``bitbake-setup --help``.
+
+The general options, common to all commands, are:
+
+- ``-h`` or ``--help``: Show the help message and exit.
+- ``-d`` or ``--debug``: Enable debug outputs.
+- ``-q`` or ``--quiet``: Print only errors.
+- ``--color``: Colorize output (where COLOR is auto, always, never).
+- ``--no-network``: Do not check whether configuration repositories and layer
+ repositories have been updated; use only the local cache.
+- ``--global-settings``: Path to the global settings file.
+- ``--setting``: Modify a setting (for this bitbake-setup invocation only).
+ For example ``--setting default top-dir-prefix /path/to/top-dir``.
+
+.. _ref-bbsetup-command-init:
+
+``bitbake-setup init``
+----------------------
+
+The ``bitbake-setup init`` sub-command helps initializing a :term:`Setup`.
+
+This command can be run without any arguments to prompt the user with
+configuration options to choose from. These configuration options are taken from
+the input :term:`Configuration Template` files in the :term:`registry`.
+
+.. note::
+
+ The registry location can be set with the :ref:`ref-bbsetup-setting-registry`
+ setting and the :ref:`ref-bbsetup-command-settings` command.
+
+Otherwise, the first argument to :ref:`ref-bbsetup-command-init` can be:
+
+- A Configuration Template ID in the registry.
+- A path to a Configuration Template file on a local disk.
+- An HTTP URI to the Configuration Template file.
+
+The choices made during the bare ``bitbake-setup init`` command can also be
+passed directly on the command-line, for example::
+
+ bitbake-setup init <generic config> poky distro/poky-tiny ...
+
+``bitbake-setup`` will stop and ask to make a choice if the above command does
+not contain all of the required configurations to complete the sequence of
+choices.
+
+In addition, the command can take the following arguments:
+
+- ``--non-interactive``: can be used to create :term:`Setups <Setup>`
+ without interactions from the user. The command will fail if not all the
+ required choices are provided in the command.
+
+- ``--source-overrides``: can be used to pass one or more
+ :ref:`source override <ref-bbsetup-source-overrides>`. See the
+ :ref:`ref-bbsetup-source-overrides` section.
+
+- ``--setup-dir-name``: can be used to configure the name of the :term:`Setup`
+ directory.
+
+- ``--skip-selection``: can be used to skip some of the choices
+ (which may result in an incomplete :term:`Setup`!)
+
+- ``-L`` or ``--use-local-source``: instead of getting a source as prescribed in
+ a configuration, symlink it into a :term:`Setup` from a path on local disk. This
+ is useful for local development where that particular source directory is managed
+ separately, and bitbake-setup will include it in a build but will not otherwise
+ touch or modify it. This option can be specified multiple times to specify multiple
+ local sources.
+
+ The option can be seen as a command line shortcut to providing an override file
+ with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrides` section
+ for more information on source overrides.
+
+- ``--init-vscode`` / ``--no-init-vscode``: generate (or skip generating) a
+ ``bitbake.code-workspace`` file in the :term:`Setup` directory. The workspace
+ file configures the `Yocto Project BitBake
+ <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`_
+ VS Code extension with paths to the build directory and the init script, and
+ lists the layer directories as workspace folders. Any user-added folders or
+ settings in an existing workspace file are preserved across updates.
+ The default is ``true`` when ``code`` (the VS Code binary) is found on
+ ``PATH``, and ``false`` otherwise.
+
+``bitbake-setup init`` Examples
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- This example prompts the user to choose a :term:`Configuration Template` from
+ a custom registry:
+
+ .. code-block:: shell
+
+ $ bitbake-setup \
+ --setting default registry 'git://example.com/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main' \
+ init
+
+- This example takes a :term:`Configuration Template` from a remote location
+ (here, one of the default configuration in BitBake):
+
+ .. code-block:: shell
+
+ $ bitbake-setup init https://git.openembedded.org/bitbake/plain/default-registry/configurations/oe-nodistro-master.conf.json
+
+- This example initializes a :term:`Setup` with:
+
+ - A custom :ref:`ref-bbsetup-setting-top-dir-prefix` and :ref:`ref-bbsetup-setting-top-dir-name`
+ - A :term:`source override`, and additionally overriding openembedded-core with a locally managed directory.
+ - A standalone :term:`Configuration Template` file.
+ - Choices passed on the command-line, applied non-interactively.
+
+ .. code-block:: shell
+
+ $ bitbake-setup \
+ --setting default top-dir-prefix /work/bitbake-setup \
+ --setting default top-dir-name custom-project \
+ init \
+ --non-interactive \
+ --source-overrides develop-branch.json \
+ -L openembedded-core ~/development/openembedded-core-gadget \
+ ./gadget_master.conf.json \
+ gadget distro/gadget machine/gadget
+
+.. _ref-bbsetup-command-list:
+
+``bitbake-setup list``
+----------------------
+
+The ``bitbake-setup list`` sub-command lists the available :term:`Configuration
+Templates <Configuration Template>` in the current :term:`registry`.
+
+In addition, the command can take the following arguments:
+
+- ``--with-expired``: list the expired configuration (e.g. older Yocto releases
+ that have reached their End-Of-Life dates).
+
+- ``--write-json``: write the configurations into a JSON file so they can be
+ programmatically processed.
+
+.. _ref-bbsetup-command-status:
+
+``bitbake-setup status``
+------------------------
+
+The ``bitbake-setup status`` sub-command shows the status of a
+:term:`Setup`. Any differences between the local copy of the :term:`Configuration
+Template` and the upstream one are printed on the console.
+
+If the BitBake environment is sourced and ready to build, the ``bitbake-setup
+status`` command (without any arguments) will show the status of the current
+:term:`Setup`.
+
+In addition, the command can take the following arguments:
+
+- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
+ required if the command is invoked from an initialized BitBake environment
+ that contains :term:`BBPATH`.
+
+.. _ref-bbsetup-command-update:
+
+``bitbake-setup update``
+------------------------
+
+The ``bitbake-setup update`` sub-command updates a :term:`Setup` to sync with
+the latest changes from the :term:`Configuration Template` it was constructed from.
+The :ref:`ref-bbsetup-command-status` command can be used to show the current
+status of the :term:`Setup` before updating it.
+
+This command is intended to be run from a shell where the BitBake environment
+has been sourced (e.g. after ``source build/init-build-env``), so that
+``bitbake-setup`` can automatically identify the current :term:`Setup` without
+requiring the ``--setup-dir`` argument.
+
+When a layer repository already exists in the :term:`Setup` (i.e. it has been
+previously checked out by ``bitbake-setup init`` or a prior ``update``), the
+update is performed *in place* using the fetcher's
+:ref:`unpack_update <bb-the-unpack-update>` method: the new upstream revision
+is fetched into the local download cache and then rebased on top of the
+checkout's current HEAD. This means any local commits in the layer directory
+are preserved and rebased onto the new upstream revision. If the working tree
+contains staged or unstaged changes to tracked files, the update is blocked
+until those changes are committed, stashed or discarded.
+
+.. note::
+
+ ``bitbake-setup`` performs the rebase to fast-forward local commits onto
+ the new upstream revision, but intentionally does not go further by using
+ ``--autostash`` (which would silently stash uncommitted changes before the
+ rebase and pop them afterwards). Any uncommitted modifications are surfaced
+ to the user before the update proceeds, so there are no surprises from an
+ automatic stash/pop cycle.
+
+In addition, the command can take the following arguments:
+
+- ``--update-bb-conf``: whether to update the :term:`BitBake Build`
+ configuration (``local.conf``, ``bblayers.conf``, etc.). This argument can
+ take up to three values:
+
+ - ``prompt`` (default): ask the user whether to update.
+ - ``yes``: update the configuration files.
+ - ``no``: don't update the configuration files.
+
+- ``--rebase-conflicts-strategy``: what to do when a layer repository has
+ local modifications or commits that prevent an in-place update. Accepted
+ values are:
+
+ - ``abort`` (default): stop with an error message describing the problem.
+ The repository is left in its previous state (the failed rebase is
+ automatically aborted). The error message includes a hint to re-run with
+ ``--rebase-conflicts-strategy=backup``.
+ - ``backup``: rename the conflicting layer directory to a timestamped
+ ``<name>-backup-<timestamp>`` path (preserving local work), then
+ re-clone the layer from upstream into a fresh directory.
+
+- ``--setup-dir``: path to the :term:`Setup` to update. Not required if the
+ command is invoked from an initialized BitBake environment that contains
+ :term:`BBPATH`.
+
+``bitbake-setup update`` Examples
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- This example performs a standard update from an initialized BitBake
+ environment. Local commits in any layer directory are rebased on top of
+ the new upstream revision and preserved:
+
+ .. code-block:: shell
+
+ $ bitbake-setup update
+ NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory.
+ NOTE: Looking up config poky-master in configuration registry
+ NOTE: Layer repository https://git.openembedded.org/openembedded-core
+ checked out into /path/to/bitbake-builds/poky-master/layers/openembedded-core
+ updated revision master from d383ea3... to b50d6de...
+ Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers
+ bitbake
+ meta-yocto
+ openembedded-core
+
+- This example shows what happens when a layer directory contains staged or
+ unstaged changes to tracked files. The update is blocked with an error:
+
+ .. code-block:: shell
+
+ $ bitbake-setup update
+ NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory.
+ NOTE: Looking up config poky-master in configuration registry
+ Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers
+ bitbake
+ meta-yocto
+ openembedded-core
+ ERROR: Unpack failure for URL:
+ 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'.
+ Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core has uncommitted changes, unable to update:
+ M meta/recipes-devtools/ccache/ccache_4.13.1.bb
+
+ Commit, stash or discard your changes and re-run the update.
+ Use 'bitbake-setup update --rebase-conflicts-strategy=backup'
+ to automatically back up the directory and re-clone from upstream,
+ or use 'bitbake-setup init -L openembedded-core /path/to/local/checkout'
+ to work with a local checkout instead.
+
+ Stashing the changes and re-running resolves the issue:
+
+ .. code-block:: shell
+
+ $ git -C layers/openembedded-core stash
+ $ bitbake-setup update
+ $ git -C layers/openembedded-core stash pop
+
+- This example shows what happens when a layer directory contains local
+ commits that conflict with the incoming upstream changes. The failed rebase
+ is automatically aborted, and the ``dldir`` remote is left in the repository
+ for manual resolution:
+
+ .. code-block:: shell
+
+ $ bitbake-setup update
+ ERROR: Repository at layers/openembedded-core has local commits that could
+ not be rebased onto the new upstream revision:
+ ...
+ Note: the 'dldir' remote points to the local download cache and may be
+ used to resolve the conflict manually.
+ Once resolved, re-run the update.
+
+ The conflict can be resolved manually using the ``dldir`` remote that
+ ``bitbake-setup`` adds to the repository:
+
+ .. code-block:: shell
+
+ $ git -C layers/openembedded-core rebase dldir/master
+ # fix conflicts in an editor, then stage the resolved files:
+ $ git -C layers/openembedded-core add meta/recipes-core/base-files/base-files.bb
+ $ git -C layers/openembedded-core rebase --continue
+ $ bitbake-setup update
+
+- When manual conflict resolution is not desired, the
+ ``--rebase-conflicts-strategy=backup`` option can be used instead. It
+ preserves the conflicting directory under a timestamped backup path and
+ re-clones the layer cleanly from upstream:
+
+ .. code-block:: shell
+
+ $ bitbake-setup update --rebase-conflicts-strategy=backup
+ NOTE: Bitbake-setup is using /path/to/bitbake-builds as top directory.
+ NOTE: Looking up config poky-master in configuration registry
+ NOTE: Layer repository https://git.openembedded.org/openembedded-core checked
+ out into /path/to/bitbake-builds/poky-master/layers/openembedded-core
+ updated revision master from 2ec283e... to b50d6de...
+ Fetching layer/tool repositories into /path/to/bitbake-builds/poky-master/layers
+ bitbake
+ meta-yocto
+ openembedded-core
+ WARNING: Unpack failure for URL:
+ 'git://git.openembedded.org/openembedded-core;protocol=https;rev=master;branch=master;destsuffix=openembedded-core'.
+ Repository at /path/to/bitbake-builds/poky-master/layers/openembedded-core
+ has local commits that could not be rebased onto the new upstream revision:
+ ...
+ Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually.
+ Once resolved, re-run the update.
+ Renaming /path/to/bitbake-builds/poky-master/layers/openembedded-core to
+ /path/to/bitbake-builds/poky-master/layers/openembedded-core-backup.20260329160426
+ to preserve your work, then re-cloning from upstream.
+
+ The backup directory is a complete git repository. Local commits can be
+ recovered from it after the update by fetching a branch from the backup
+ into the fresh clone (git accepts local paths as remote URLs) and then
+ cherry-picking the desired commits. For example, given a ``my-wip`` branch
+ with two commits existing in the backup repository and not in the fresh clone,
+ the following commands can be used to apply these commits on top of the new
+ upstream revision in the fresh clone:
+
+ .. code-block:: shell
+
+ $ git -C layers/openembedded-core-backup.20260329160426 log --oneline my-wip
+ a1b2c3d u-boot: fix compilation with newer GCC
+ 2ec283e base-files: update version
+ ...
+
+ $ cd layers/openembedded-core
+ $ git checkout -b my-wip
+ Switched to a new branch 'my-wip'
+ $ git fetch ../openembedded-core-backup.20260329160426 my-wip
+
+ $ git cherry-pick 2ec283e
+ # resolve any conflicts, then stage the resolved files:
+ $ git add meta/recipes-core/base-files/base-files.bb
+ $ git cherry-pick --continue
+
+ $ git cherry-pick a1b2c3d
+ # resolve any conflicts, then stage the resolved files:
+ $ git add meta/recipes-devtools/u-boot/u-boot_2026.04.bb
+ $ git cherry-pick --continue
+
+ $ cd ../..
+
+ The sequence above is:
+
+ #. Inspect the backup's branch history to identify the commits to recover.
+ #. Change into the fresh clone and create a matching branch.
+ #. Fetch the backup branch so its objects become available locally. Git
+ accepts filesystem paths as remote URLs, and from inside
+ ``layers/openembedded-core/``, ``../`` points to ``layers/``, where the
+ backup directory sits.
+ #. Cherry-pick the commits in oldest-first order.
+
+ Once all desired commits have been recovered and verified, the backup
+ directory can be removed:
+
+ .. code-block:: shell
+
+ $ rm -rf layers/openembedded-core-backup.20260329160426
+
+ If a VSCode workspace is in use, the backup directory will appear as an
+ additional workspace folder until it is cleaned up. It can be removed from
+ the workspace via the VS Code UI by right-clicking the folder and selecting
+ *Remove Folder from Workspace*.
+
+.. _ref-bbsetup-command-install-buildtools:
+
+``bitbake-setup install-buildtools``
+------------------------------------
+
+The ``bitbake-setup install-buildtools`` sub-command installs and extracts a
+buildtools tarball into the specified :term:`Setup`.
+
+After completion, help is printed to help the user on how to use the extracted
+tarball.
+
+.. note::
+
+ The purpose of the Buildtools tarball is to provide tools needed to run
+ BitBake on build machines where such tools cannot be easily obtained from the
+ host Linux distribution (for example on older distribution versions that do
+ not contain a recent enough GCC compiler or Python interpreter, or machines
+ where the user running BitBake cannot easily install additional software into
+ the system). This command requires that the OpenEmbedded-core layer is
+ present in the BitBake configuration.
+
+ See https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-git-tar-python-make-and-gcc-versions
+ for more information.
+
+In addition, the command can take the following arguments:
+
+- ``--force``: force the re-installation of the tarball.
+
+- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not
+ required if :term:`BBPATH` is already configured.
+
+- ``--url``: full URL to the buildtools SDK installer. Overrides the value
+ from the configuration file.
+
+- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides the
+ value from the configuration file.
+
+When ``url`` and ``sha256sum`` are set in the configuration file,
+the installer is downloaded via ``bb.fetch`` (cached in :term:`DL_DIR`) and its
+checksum is enforced. If no configuration is present, the script falls back
+to its built-in defaults::
+
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "my-config",
+ "install-buildtools": {
+ "url": "https://example.com/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.0.sh",
+ "sha256sum": "af76648b..."
+ },
+ ...
+ }
+ ]
+ }
+
+.. _ref-bbsetup-command-settings:
+
+``bitbake-setup settings``
+--------------------------
+
+The ``bitbake-setup settings`` sub-command helps modifying the settings of
+``bitbake-setup``. This sub-command has sub-commands itself:
+
+- ``bitbake-setup settings list`` lists the current settings and their value.
+- ``bitbake-setup settings set`` sets a setting.
+- ``bitbake-setup settings unset`` removes a setting.
+
+Settings must be set with a section and a value, for example::
+
+ bitbake-setup settings set default top-dir-name bitbake-builds
+
+Will set the value of ``top-dir-name`` in the ``default`` section to
+"bitbake-builds".
+
+In addition, the command can take the following arguments:
+
+- ``--global``: write to the global settings
+ (``~/.config/bitbake-setup/settings.conf``) instead of the :term:`Top
+ Directory` settings.
+
+See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+section to see the supported settings.
+
+.. note::
+
+ The supported setting listed in the
+ :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Settings`
+ section are only affected when set in the ``default`` section.
+
+Settings
+========
+
+The settings allow configuring ``bitbake-setup``. Settings are stored in a file
+named ``settings.conf``, in :wikipedia:`INI <INI_file>` format.
+
+There are multiple locations for storing settings. Settings in different
+locations can override each other, but the final value of a setting is computed
+from reading the files in this order:
+
+#. Global settings file: ``~/.config/bitbake-setup/settings.conf``.
+
+#. Local settings file, taken from a ``settings.conf`` file in the :term:`Top
+ Directory`.
+
+#. Command-line settings, passed with the ``--setting`` argument.
+
+A valid settings file would for example be:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-prefix = /path/to/workspace
+ top-dir-name = bitbake-builds
+ registry = /path/to/bitbake/default-registry
+ dl-dir = /path/to/bitbake-setup-downloads
+ use-full-setup-dir-name = yes
+ common-sstate = yes
+
+Settings and their values can be listed and modified with the ``bitbake-setup
+settings`` command. See the :ref:`ref-bbsetup-command-settings` section for
+more information.
+
+Below are the available settings.
+
+.. _ref-bbsetup-setting-top-dir-prefix:
+
+``top-dir-prefix``
+------------------
+
+The :ref:`ref-bbsetup-setting-top-dir-prefix` setting helps configuring the
+leftmost part of the path to the :term:`Top Directory`.
+
+For example, with:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-prefix = /path/to/workspace
+
+The :term:`top directory` would be ``/path/to/workspace/<top-dir-name>`` with
+the ``<top-dir-name>`` corresponding to the :ref:`ref-bbsetup-setting-top-dir-name`
+setting.
+
+This is most useful to customize on systems where the default location of the
+:term:`Top Directory` (``~/bitbake-builds``) is not suitable, and there is a
+dedicated directory for builds somewhere else.
+
+.. _ref-bbsetup-setting-top-dir-name:
+
+``top-dir-name``
+----------------
+
+The :ref:`ref-bbsetup-setting-top-dir-name` setting helps configuring the
+rightmost part of the path to the :term:`Top Directory`.
+
+For example, with:
+
+.. code-block:: ini
+
+ [default]
+ top-dir-name = builds
+
+The :term:`top directory` would be ``<top-dir-prefix>/builds`` with
+the ``<top-dir-prefix>`` corresponding to the :ref:`ref-bbsetup-setting-top-dir-prefix`
+setting.
+
+.. _ref-bbsetup-setting-registry:
+
+``registry``
+------------
+
+The :ref:`ref-bbsetup-setting-registry` setting sets the URI location of the
+registry. This URI can be any URI supported by the BitBake fetcher.
+
+A local registry would be configured as follows:
+
+.. code-block:: ini
+
+ [default]
+ registry = /path/to/registry
+
+When using another fetcher, it must be specified in the URI scheme. For example:
+
+.. code-block:: ini
+
+ [default]
+ registry = git://example.com/bitbake-setup-configurations;protocol=https;branch=master;rev=master
+
+This would fetch the remote configurations from a remote Git remote repository,
+on the ``master`` branch.
+
+See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section for more
+information on BitBake fetchers.
+
+.. _ref-bbsetup-setting-dl-dir:
+
+``dl-dir``
+----------
+
+The :ref:`ref-bbsetup-setting-dl-dir` setting sets the location of the download
+cache that ``bitbake-setup`` will configure for the purpose of downloading
+configuration repositories, layers and other sources using BitBake fetchers.
+Please see :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` and the
+:term:`DL_DIR` variable for more information.
+
+The location can be set such that it is shared with :term:`DL_DIR` specified by
+BitBake builds, so that there is a single directory containing a copy of
+everything needed to set up and run a BitBake build offline in a reproducible
+manner.
+
+.. _ref-bbsetup-setting-use-full-setup-dir-name:
+
+``use-full-setup-dir-name``
+---------------------------
+
+The :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting, if set to ``yes``
+will override the suggestions for the :term:`Setup` directory name made by
+``setup-dir-name`` entries in :term:`Configuration Template` files. This
+will make the directory names longer, but fully specific: they will contain
+all selections made during initialization.
+
+.. _ref-bbsetup-setting-common-sstate:
+
+``common-sstate``
+-----------------
+
+When this setting is set to ``yes`` (which is also the default), bitbake-setup will
+set up a common sstate directory and common hash equivalency database for all the
+:term:`setups <Setup>` in a :term:`Top Directory`. This is very beneficial for speeding
+up builds as build artefacts will be reused whenever possible between them.
+
+Set this to ``no`` for advanced use cases, such as placing the sstate directory on a NFS
+mount and maintaining a separate hash equivalency server, so that sstate and hash equivalency
+data can be shared between several computers. For such use cases the sstate settings need
+to be added to a build configuration separately.
+
+See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for how to share sstate
+on the network.
+
+.. _ref-bbsetup-section-config-reference:
+
+Configuration Template Files Reference
+======================================
+
+:term:`Configuration Templates <Configuration Template>` are the input files given
+to ``bitbake-setup`` to configure :term:`Setups <Setup>`.
+
+These files are written in the JSON file format and are stored in a
+:term:`Registry`. They can also be standalone files directly passed to the
+:ref:`ref-bbsetup-command-init` command:
+
+.. code-block:: shell
+
+ $ bitbake-setup init /path/to/config.conf.json
+
+They contain the following sections:
+
+- ``version`` (**required**): version of the configuration file.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "version": "1.0"
+ }
+
+- ``description`` (**required**): the description of the configuration.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "description": "OpenEmbedded - 'nodistro' basic configuration"
+ }
+
+- ``sources`` (**required**): sources, such as git repositories that should
+ be provided under ``layers/`` directory of a :term:`Setup`. Although the
+ list of sources can be very flexible, at least BitBake should be cloned
+ to generate a :term:`Setup`.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "branch": "master",
+ "rev": "master"
+ },
+ "path": "bitbake"
+ },
+ "openembedded-core": {
+ "local": {
+ "path": "~/path/to/local/openembedded-core"
+ }
+ },
+ "meta-private": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "ssh://git@git.example.com/meta-private"
+ }
+ },
+ "branch": "master",
+ "rev": "master"
+ },
+ "path": "meta-private"
+ }
+ }
+ }
+
+ Sources can be specified with the following options:
+
+ - ``path`` (*optional*): where the source is extracted, relative to the
+ ``layers/`` directory of a :term:`Setup`. If unspecified, the name of the
+ source is used.
+
+ - ``git-remote`` (*optional*): specifies URI, branch and revision of a git
+ repository to fetch from.
+
+ ``git-remote`` entries are specified with the following options:
+
+ - ``remotes`` (*optional*): a dictionary of git remote names, each containing a ``uri`` property
+ with a URI that follows the git URI syntax. See https://git-scm.com/docs/git-clone#_git_urls
+ and https://git-scm.com/docs/git-remote for more information.
+
+ This structure is useful if you want to provide several remotes. Please
+ note that each URI will be cloned, but only the the last treated one
+ will be used in the unpacked source.
+
+ Example:
+
+ .. code-block:: json
+ :force:
+
+ {
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ },
+ "contrib": {
+ "uri": "https://git.openembedded.org/bitbake-contrib"
+ }
+ },
+ "branch": "master",
+ "rev": "master"
+ },
+ "path": "bitbake"
+ }
+ }
+ }
+ - ``rev`` (**required**): the revision to checkout. Can be the name of the
+ branch to checkout on the latest revision of the specified ``branch``.
+
+ If the value is the branch name, ``bitbake-setup`` will check out the
+ latest revision on that branch, and keep it updated when using the
+ :ref:`ref-bbsetup-command-update` command.
+
+ - ``branch`` (**required**): the Git branch, used to check that the
+ specified ``rev`` is indeed on that branch.
+
+ - ``uri`` (*optional*): a URI that follows the git URI syntax. You can replace
+ the ``remotes`` structure if only one URI is provided. Despite this, ``uri``
+ and ``remotes`` can still be used together.
+
+ See https://git-scm.com/docs/git-clone#_git_urls for more information.
+
+ - ``local`` (*optional*): specifies a path on local disk that should be symlinked
+ to under ``layers/``. This is useful for local development, where some layer
+ or other component used in a build is managed separately, but should still be
+ available for bitbake-setup driven builds.
+
+ ``local`` entries are specified with the following options:
+
+ - ``path`` (**required**): the path on local disk where the externally
+ managed source tree is. ``~`` and ``~user`` are expanded to that user's home
+ directory. Paths in configuration files must be absolute (after possible
+ ~ expansion), paths in override files can be relative to the directory where
+ the override file is.
+
+- ``expires`` (*optional*): Expiration date of the configuration. This date
+ should be in :wikipedia:`ISO 8601 <ISO_8601>` format (``YYYY-MM-DDTHH:MM:SS``).
+
+- ``bitbake-setup`` (**required**): contains a list of configurations.
+
+ Example:
+
+ .. code-block:: json
+
+ {
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yocto-bsp","meta-yocto/meta-poky"],
+ "bb-env-passthrough-additions": ["DL_DIR","SSTATE_DIR"],
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : [
+ { "name": "machine/qemux86-64", "description": "Machine configuration for running an x86-64 system on QEMU" },
+ { "name": "machine/qemuarm64", "description": "Machine configuration for running an ARMv8 system on QEMU" },
+ { "name": "machine/qemuriscv64", "description": "Machine configuration for running an RISC-V system on QEMU" },
+ { "name": "machine/genericarm64", "description": "Machine configuration for Arm64 SystemReady IR/ES platforms" },
+ { "name": "machine/genericx86-64", "description": "Machine configuration for generic x86_64 (64-bit) PCs and servers" }
+ ]
+ },
+ "distro": {
+ "description": "Target Distributions",
+ "options" : [
+ { "name": "distro/poky", "description": "Distro configuration for Poky (Yocto Project Reference Distro)" },
+ { "name": "distro/poky-altcfg", "description": "Distro configuration for Poky (systemd init manager)" },
+ { "name": "distro/poky-tiny", "description": "Distro configuration for Poky (optimized for size)" }
+ ]
+ }
+ },
+ "configurations": [
+ {
+ "name": "poky",
+ "description": "Poky - The Yocto Project testing distribution"
+ },
+ {
+ "name": "poky-with-sstate",
+ "description": "Poky - The Yocto Project testing distribution with internet sstate acceleration. Use with caution as it requires a completely robust local network with sufficient bandwidth.",
+ "oe-fragments": ["core/yocto/sstate-mirror-cdn"]
+ }
+ ]
+ }
+ ]
+ }
+ }
+
+ Configurations can be specified with the following options:
+
+ - ``name`` (**required**): the name of this configuration snippet. This is
+ what is prompted during the :ref:`ref-bbsetup-command-init` command
+ execution.
+
+ - ``description`` (**required**): the description of this configuration
+ snippet. This is what is prompted during the
+ :ref:`ref-bbsetup-command-init` command execution.
+
+ - ``notes`` (*optional*): additional information written to
+ ``build/conf/conf-notes.txt`` when ``bitbake-setup`` generates the build
+ configuration from ``bb-layers``. This can be a string, or a list of
+ strings which will be written one per line. For ``oe-template``
+ configurations, this file is provided by the template.
+
+ - ``configurations``: Configurations can recursively contain as many nested
+ configurations as needed. This will create more choices when running the
+ :ref:`ref-bbsetup-command-init` command.
+
+ The purpose of such nesting is to be able to scale the configurations, for
+ example when there is a need to create multiple configurations that share
+ some parameters (which are specified in their common parent), but differ
+ between themselves in other parameters. ``bitbake-setup`` will assemble
+ configuration choices by putting together information from a leaf
+ configuration and all of its ancestors.
+
+ When the same keyword is present in a nested configuration and in one of
+ its ancestors, the values are merged with Python ``+`` semantics. For
+ example, lists are appended and strings are concatenated directly. String
+ values such as ``description`` should include any needed separators in
+ the configuration data.
+
+ - ``bb-env-passthrough-additions`` (*optional*): List of environment
+ variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`.
+
+ - ``bb-layers`` (*optional*): List of layers to add to the ``bblayers.conf``
+ file. Paths in this list are relative to the ``layers/`` directory of a
+ :term:`Setup`.
+
+ The ``bb-layers`` keyword cannot be used in conjunction with the
+ ``oe-template`` option, as the ``bblayers.conf`` file comes from the
+ template itself.
+
+ - ``bb-layers-file-relative`` (*optional*): List of layers that are not
+ managed by ``bitbake-setup`` but that need to be included as part of the
+ ``bblayers.conf`` file. Paths in this list are relative to the
+ configuration file.
+
+ This is useful when (one or more) configuration files and (one or
+ more) layers are hosted in the same Git repository, which is cloned
+ and managed independently from bitbake-setup workflows. For example::
+
+ ├── meta-myproject/
+ └── myproject.conf.json
+
+ Then ``myproject.conf.json`` can contain the following to add
+ ``meta-myproject`` to ``bblayers.conf``::
+
+ {
+ ...
+ "bb-layers-file-relative": [
+ "meta-myproject"
+ ],
+ ...
+ }
+
+ The ``bb-layers-file-relative`` keyword cannot be used in conjunction with the
+ ``oe-template`` keyword, as the ``bblayers.conf`` file comes from the
+ template itself.
+
+ - ``oe-template`` (*optional*, OpenEmbedded specific): OpenEmbedded template
+ to use. This cannot be used in conjunction with the
+ ``bb-layers`` or ``bb-layers-file-relative`` keywords as it
+ already provides a ready ``bblayers.conf`` file to use.
+
+ See https://docs.yoctoproject.org/dev-manual/custom-template-configuration-directory.html
+ for more information of OpenEmbedded templates.
+
+ - ``oe-fragments-one-of`` (*optional*, OpenEmbedded specific): the OpenEmbedded
+ fragments to select as part of the build.
+
+ This can one of two formats:
+
+ - A list of dictionaries with ``name`` and a ``description``,
+ corresponding to respectively the fragments names and fragment
+ descriptions (like in the example above).
+
+ - A list of fragment names. For example:
+
+ .. code-block:: json
+
+ "options" : ["machine/qemux86-64", "machine/qemuarm64", "machine/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"]
+
+ This will trigger choices to make during the
+ :ref:`ref-bbsetup-command-init` command execution.
+
+ See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for
+ more information of OpenEmbedded configuration fragments.
+
+ - ``oe-fragments`` (*optional*, OpenEmbedded specific): fragments to select
+ as part of the build.
+
+ See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for
+ more information of OpenEmbedded configuration fragments.
+
+ - ``setup-dir-name`` (*optional*): a suggestion for the :term:`Setup`
+ directory name. Bitbake-setup will use it, unless it already exists, or
+ the :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting is set
+ to ``yes`` (in those cases, it will fall back to the built-in full name,
+ containing the full set of choices made during initialization).
+
+ This key can have variable expansions in the same format as python
+ string.Template strings (which matches the shell variable expansion rules):
+ https://docs.python.org/3/library/string.html#template-strings-strings .
+
+ Variables that can be expanded are any fragment configuration prompted
+ by the user (e.g. the keys in "oe-fragments-one-of")::
+
+ {
+ ...
+ "oe-fragments-one-of": {
+ "machine": {
+ "description": "Target machines",
+ "options" : ["machine/gadget", "machine/gizmo"]
+ }
+ },
+ "setup-dir-name": "somebuild-$machine"
+ ...
+ }
+
+ would expand to ``somebuild-machine_gadget``.
+
+Configuration Template Examples
+-------------------------------
+
+OpenEmbedded "nodistro" configuration for master branches:
+
+.. literalinclude:: ../../default-registry/configurations/oe-nodistro-master.conf.json
+ :language: json
+
+Poky distribution configuration for master branches:
+
+.. literalinclude:: ../../default-registry/configurations/poky-master.conf.json
+ :language: json
+
+.. _ref-bbsetup-source-overrides:
+
+Source Overrides
+================
+
+See the definition of :term:`Source Override` in the Terminology section.
+
+These files are written in the JSON file format and are optionally passed to the
+``--source-overrides`` argument of the :ref:`ref-bbsetup-command-init` command.
+The ``--source-overrides`` option can be passed multiple times, in which case the
+overrides are applied in the order specified in the command-line.
+
+Here is an example file that overrides the branch of the BitBake repository to
+"master-next", and provides openembedded-core as a symlink to a path on local disk:
+
+.. code-block:: json
+
+ {
+ "description": "Source override file",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "branch": "master-next",
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ }
+ },
+ "rev": "master-next"
+ }
+ },
+ "openembedded-core": {
+ "local": {
+ "path": "~/path/to/local/openembedded-core"
+ }
+ }
+ },
+ "version": "1.0"
+ }
+
+- The ``version`` parameter contains the version of the used configuration, and
+ should match the one of the :term:`Configuration Template` file in use.
+
+- The ``sources`` section contains the same options as the ``sources`` option
+ of a :term:`Configuration Template` file. See the
+ :ref:`ref-bbsetup-section-config-reference` section for more information.
+
+.. _ref-bbsetup-fixed-revisions:
+
+Fixed source revisions
+======================
+
+:term:`Configuration Template` can set source revisions in ``rev`` to a tag or a branch.
+Bitbake-setup will make sure the actual revision will match the tag or branch when performing
+initializations or updates, and will capture the revisions in a :ref:`ref-bbsetup-source-overrides`
+file.
+
+This file is named ``sources-fixed-revisions.json`` and is available in ``layers/``
+under the :term:`Setup` directory (as well as under ``config/`` for backwards compatibility).
+It can be used to keep record of what was checked out when using configurations that do not
+specify exact revisions.
+
+Also, as it is an override file, it can be combined with the original configurations
+to initialize a setup in a reproducible way that guarantees an exact, never-changing
+set of revisions, by using the ``--source-overrides`` option of the
+:ref:`ref-bbsetup-command-init` command.
+
+For example if the original configuration had specified only a master branch
+for a source::
+
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ }
+ },
+ "branch": "master",
+ "rev": "master"
+ }
+
+
+the override will contain the exact revisions::
+
+ "bitbake": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "https://git.openembedded.org/bitbake"
+ }
+ },
+ "branch": "master",
+ "rev": "720df1a53452983c1c832f624490e255cf389204"
+ }
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
index d58fbb32..1638a8f5 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
@@ -64,11 +64,11 @@ data itself is of various types:
together.
The ``layer.conf`` files are used to construct key variables such as
-:term:`BBPATH` and :term:`BBFILES`.
-:term:`BBPATH` is used to search for configuration and class files under the
-``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used
-to locate both recipe and recipe append files (``.bb`` and
-``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
+:term:`BBPATH` and :term:`BBFILES`. :term:`BBPATH` is used to search for
+configuration files under the ``conf`` directory and class files under the
+``classes-global``, ``classes-recipe`` and ``classes`` directories.
+:term:`BBFILES` is used to locate both recipe and recipe append files (``.bb``
+and ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment.
Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable
@@ -337,21 +337,22 @@ file that BitBake parses, you can change that preference::
.. note::
- It is common for a recipe to provide two versions -- a stable,
- numbered (and preferred) version, and a version that is automatically
- checked out from a source code repository that is considered more
- "bleeding edge" but can be selected only explicitly.
+ It is not uncommon for a given recipe to provide more than one
+ version -- a stable (and preferred) version that should be selected
+ by default, and a newer, development or "bleeding edge" version that
+ will be selected only explicitly by the developer.
- For example, in the OpenEmbedded codebase, there is a standard,
- versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
- is also a Git-based version, ``busybox_git.bb``, which explicitly
- contains the line ::
+ For example, at the moment, in the OpenEmbedded codebase, there are
+ two versions of the ``nginx`` webserver: a stable "1.28.0" version
+ that would be selected by default, and a newer "1.29.1" development
+ version that a developer would need to choose explicitly.
- DEFAULT_PREFERENCE = "-1"
+ This can be seen based on this snippet from the ``nginx`` 1.29.1
+ recipe file::
- to ensure that the
- numbered, stable version is always preferred unless the developer
- selects otherwise.
+ # 1.28.x branch is the current stable branch, the recommended default
+ # 1.29.x is the current mainline branches containing all new features
+ DEFAULT_PREFERENCE = "-1"
.. _bb-bitbake-dependencies:
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index fb4f0a23..9f00545d 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -11,10 +11,11 @@ with the intricacies of downloading source code and files from remote
systems. Fetching source code is one of the cornerstones of building
software. As such, this module forms an important part of BitBake.
-The current fetch module is called "fetch2" and refers to the fact that
-it is the second major version of the API. The original version is
-obsolete and has been removed from the codebase. Thus, in all cases,
-"fetch" refers to "fetch2" in this manual.
+The fetch module was previously called ``fetch2`` to distinguish the second
+major version of the API from the original version, which is obsolete and has
+been removed from the codebase. The module has since been renamed back to
+``fetch``. Existing code using ``bb.fetch2`` remains supported by a
+compatibility alias, but new code should use ``bb.fetch``.
The Download (Fetch)
====================
@@ -30,7 +31,7 @@ The code to execute the first part of this process, a fetch, looks
something like the following::
src_uri = (d.getVar('SRC_URI') or "").split()
- fetcher = bb.fetch2.Fetch(src_uri, d)
+ fetcher = bb.fetch.Fetch(src_uri, d)
fetcher.download()
This code sets up an instance of the fetch class. The instance uses a
@@ -39,10 +40,10 @@ variable and then calls the ``download`` method to download the files.
The instantiation of the fetch class is usually followed by::
- rootdir = l.getVar('WORKDIR')
+ rootdir = l.getVar('UNPACKDIR')
fetcher.unpack(rootdir)
-This code unpacks the downloaded files to the specified by ``WORKDIR``.
+This code unpacks the downloaded files to the specified by ``UNPACKDIR``.
.. note::
@@ -51,7 +52,7 @@ This code unpacks the downloaded files to the specified by ``WORKDIR``.
examine the OpenEmbedded class file ``base.bbclass``
.
-The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the
+The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the
fetcher, since those fetcher methods can be (and are) called with
different variable names. In OpenEmbedded for example, the shared state
(sstate) code uses the fetch module to fetch the sstate files.
@@ -84,11 +85,8 @@ fetcher does know how to use HTTP as a transport.
Here are some examples that show commonly used mirror definitions::
PREMIRRORS ?= "\
- bzr://.*/.\* http://somemirror.org/sources/ \
- cvs://.*/.\* http://somemirror.org/sources/ \
git://.*/.\* http://somemirror.org/sources/ \
hg://.*/.\* http://somemirror.org/sources/ \
- osc://.*/.\* http://somemirror.org/sources/ \
p4://.*/.\* http://somemirror.org/sources/ \
svn://.*/.\* http://somemirror.org/sources/"
@@ -126,7 +124,7 @@ in the previous example, or you can name the URLs. The following syntax
shows how you name the URIs::
SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
- SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
+ SRC_URI[foo.md5sum] = "4a8e0f237e961fd7785d19d07fdb994d"
After a file has been downloaded and
has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`.
@@ -159,19 +157,25 @@ URLs except Git URLs, BitBake uses the common ``unpack`` method.
A number of parameters exist that you can specify within the URL to
govern the behavior of the unpack stage:
-- *unpack:* Controls whether the URL components are unpacked. If set to
+- *"unpack":* Controls whether the URL components are unpacked. If set to
"1", which is the default, the components are unpacked. If set to
"0", the unpack stage leaves the file alone. This parameter is useful
when you want an archive to be copied in and not be unpacked.
-- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether
+- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether
to use DOS line ending conversion on text files.
-- *striplevel:* Strip specified number of leading components (levels)
- from file names on extraction
+- *"striplevel":* Strip specified number of leading components (levels)
+ from file names on extraction.
-- *subdir:* Unpacks the specific URL to the specified subdirectory
- within the root directory.
+- *"subdir":* Unpacks the specific URL to the specified subdirectory
+ within the specified root directory. This path can be further modified
+ by fetcher specific parameters.
+
+- *"name":* Assigns a name to a given component of the :term:`SRC_URI`.
+ This component is later referenced by this name when specifying its
+ :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its
+ revision in the package version string with aid of :term:`SRCREV_FORMAT`.
The unpack call automatically decompresses and extracts files with ".Z",
".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and
@@ -183,6 +187,51 @@ cloning the tree into the final directory. The process is completed
using references so that there is only one central copy of the Git
metadata needed.
+.. _bb-the-unpack-update:
+
+The Non-Destructive Update (``unpack_update``)
+==============================================
+
+.. note::
+
+ This is a specialised method intended for tools that manage persistent
+ layer checkouts on behalf of the user, such as
+ :ref:`bitbake-setup update <ref-bbsetup-command-update>`. It is not part of the
+ normal recipe fetch/unpack flow.
+
+For Git URLs, an alternative ``unpack_update()`` method is available
+that updates an existing checkout *in place* rather than removing and
+re-cloning it. This is useful when the target directory may contain
+local commits that should be preserved across updates.
+
+The code to call the non-destructive update looks like the following::
+
+ rootdir = l.getVar('UNPACKDIR')
+ fetcher.unpack_update(rootdir)
+
+``unpack_update()`` performs the following steps:
+
+1. A ``dldir`` git remote is added (or updated) in the existing
+ checkout, pointing at the local download cache. This remote may
+ also be useful for manually resolving conflicts outside the fetcher.
+2. The new upstream revision is fetched from the ``dldir`` remote into
+ the existing repository.
+3. Any local commits are rebased on top of the new upstream revision,
+ preserving local work.
+
+The method raises an error if:
+
+- The working tree contains staged or unstaged changes to tracked
+ files (``LocalModificationsError``).
+- Local commits cannot be cleanly rebased onto the new upstream
+ revision (``RebaseError``). A failed rebase is automatically aborted
+ before the exception is raised.
+- The download cache does not contain a sufficiently recent clone
+ of the repository, or the checkout is a shallow clone.
+
+Currently only the Git fetcher supports ``unpack_update()``. All other
+fetcher types raise ``RuntimeError`` if it is called.
+
.. _bb-fetchers:
Fetchers
@@ -246,91 +295,19 @@ Some example URLs are as follows::
introduce ambiguity when parsing URLs that also contain semi-colons,
for example::
- SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
-
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
Such URLs should should be modified by replacing semi-colons with '&'
characters::
- SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
-
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
In most cases this should work. Treating semi-colons and '&' in
queries identically is recommended by the World Wide Web Consortium
(W3C). Note that due to the nature of the URL, you may have to
specify the name of the downloaded file as well::
- SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
-
-
-.. _cvs-fetcher:
-
-CVS fetcher (``(cvs://``)
--------------------------
-
-This submodule handles checking out files from the CVS version control
-system. You can configure it using a number of different variables:
-
-- :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running
- the ``cvs`` command. This name is usually "cvs".
-
-- :term:`SRCDATE`: The date to use when fetching the CVS source code. A
- special value of "now" causes the checkout to be updated on every
- build.
-
-- :term:`CVSDIR`: Specifies where a temporary
- checkout is saved. The location is often ``DL_DIR/cvs``.
-
-- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the
- ``cvs`` command.
-
-- CVS_PROXY_PORT: The port number to use as a "proxyport="
- parameter to the ``cvs`` command.
-
-As well as the standard username and password URL syntax, you can also
-configure the fetcher with various URL parameters:
-
-The supported parameters are as follows:
-
-- *"method":* The protocol over which to communicate with the CVS
- server. By default, this protocol is "pserver". If "method" is set to
- "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You
- can use "dir" for local directories.
-
-- *"module":* Specifies the module to check out. You must supply this
- parameter.
-
-- *"tag":* Describes which CVS TAG should be used for the checkout. By
- default, the TAG is empty.
-
-- *"date":* Specifies a date. If no "date" is specified, the
- :term:`SRCDATE` of the configuration is used to
- checkout a specific date. The special value of "now" causes the
- checkout to be updated on every build.
-
-- *"localdir":* Used to rename the module. Effectively, you are
- renaming the output directory to which the module is unpacked. You
- are forcing the module into a special directory relative to
- :term:`CVSDIR`.
-
-- *"rsh":* Used in conjunction with the "method" parameter.
-
-- *"scmdata":* Causes the CVS metadata to be maintained in the tarball
- the fetcher creates when set to "keep". The tarball is expanded into
- the work directory. By default, the CVS metadata is removed.
-
-- *"fullpath":* Controls whether the resulting checkout is at the
- module level, which is the default, or is at deeper paths.
-
-- *"norecurse":* Causes the fetcher to only checkout the specified
- directory with no recurse into any subdirectories.
-
-- *"port":* The port to which the CVS server connects.
-
-Some example URLs are as follows::
-
- SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
- SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
+ SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
.. _svn-fetcher:
@@ -398,16 +375,16 @@ This fetcher supports the following parameters:
.. note::
- When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs
- from the one that is typically passed to ``git clone`` command and provided
- by the Git server to fetch from. For example, the URL returned by GitLab
- server for ``mesa`` when cloning over SSH is
- ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in
- :term:`SRC_URI` is the following::
+ When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs
+ from the one that is typically passed to ``git clone`` command and provided
+ by the Git server to fetch from. For example, the URL returned by GitLab
+ server for ``mesa`` when cloning over SSH is
+ ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in
+ :term:`SRC_URI` is the following::
- SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..."
+ SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..."
- Note the ``:`` character changed for a ``/`` before the path to the project.
+ Note the ``:`` character changed for a ``/`` before the path to the project.
- *"nocheckout":* Tells the fetcher to not checkout source code when
unpacking when set to "1". Set this option for the URL where there is
@@ -436,13 +413,15 @@ This fetcher supports the following parameters:
"nobranch" is set to "1", this is a mandatory parameter. The number of
branch parameters must match the number of name parameters.
-- *"rev":* The revision to use for the checkout. The default is
- "master".
+- *"rev":* The revision to use for the checkout. If :term:`SRCREV` is also set,
+ this parameter must match its value.
-- *"tag":* Specifies a tag to use for the checkout. To correctly
- resolve tags, BitBake must access the network. For that reason, tags
- are often not used. As far as Git is concerned, the "tag" parameter
- behaves effectively the same as the "rev" parameter.
+- *"tag":* Specifies a tag to use when fetching. To correctly resolve
+ tags, BitBake must access the network. If a ``rev`` parameter or
+ :term:`SRCREV` is also specified, network access is not necessary to resolve
+ the tag and instead, it is verified that they both resolve to the same commit
+ SHA at unpack time. The ``tag`` parameter is optional, but strongly
+ recommended if the checked out revision is a tag.
- *"subpath":* Limits the checkout to a specific subpath of the tree.
By default, the whole tree is checked out.
@@ -451,7 +430,7 @@ This fetcher supports the following parameters:
By default, the path is ``git/``.
- *"usehead":* Enables local ``git://`` URLs to use the current branch
- HEAD as the revision for use with ``AUTOREV``. The "usehead"
+ HEAD as the revision for use with :term:`AUTOREV`. The "usehead"
parameter implies no branch and only works when the transfer protocol
is ``file://``.
@@ -461,13 +440,6 @@ Here are some example URLs::
SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main"
SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..."
-.. note::
-
- When using ``git`` as the fetcher of the main source code of your software,
- ``S`` should be set accordingly::
-
- S = "${WORKDIR}/git"
-
.. note::
Specifying passwords directly in ``git://`` urls is not supported.
@@ -476,7 +448,7 @@ Here are some example URLs::
easy to share metadata without removing passwords. SSH keys, ``~/.netrc``
and ``~/.ssh/config`` files can be used as alternatives.
-Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to
+Using tags with the git fetcher may cause surprising behaviour. BitBake needs to
resolve the tag to a specific revision and to do that, it has to connect to and use
the upstream repository. This is because the revision the tags point at can change and
we've seen cases of this happening in well known public repositories. This can mean
@@ -530,10 +502,10 @@ The fetcher uses the ``rcleartool`` or
Following are options for the :term:`SRC_URI` statement:
-- *vob*: The name, which must include the prepending "/" character,
+- *"vob":* The name, which must include the prepending "/" character,
of the ClearCase VOB. This option is required.
-- *module*: The module, which must include the prepending "/"
+- *"module":* The module, which must include the prepending "/"
character, in the selected VOB.
.. note::
@@ -545,7 +517,7 @@ Following are options for the :term:`SRC_URI` statement:
load /example_vob/example_module
-- *proto*: The protocol, which can be either ``http`` or ``https``.
+- *"proto":* The protocol, which can be either ``http`` or ``https``.
By default, the fetcher creates a configuration specification. If you
want this specification written to an area other than the default, use
@@ -554,9 +526,9 @@ the specification is written.
.. note::
- the SRCREV loses its functionality if you specify this variable. However,
- SRCREV is still used to label the archive after a fetch even though it does
- not define what is fetched.
+ the :term:`SRCREV` loses its functionality if you specify this variable.
+ However, :term:`SRCREV` is still used to label the archive after a fetch even
+ though it does not define what is fetched.
Here are a couple of other behaviors worth mentioning:
@@ -598,7 +570,7 @@ and port, username, and password, and fetches the Head Revision::
SRC_URI = "p4://example-depot/main/source/..."
SRCREV = "${AUTOREV}"
PV = "p4-${SRCPV}"
- S = "${WORKDIR}/p4"
+ S = "${UNPACKDIR}/p4"
Here is an example that specifies the server URL and port, username, and
password, and fetches a Revision based on a Label::
@@ -607,25 +579,25 @@ password, and fetches a Revision based on a Label::
SRC_URI = "p4://user:passwd@example-depot/main/source/..."
SRCREV = "release-1.0"
PV = "p4-${SRCPV}"
- S = "${WORKDIR}/p4"
+ S = "${UNPACKDIR}/p4"
.. note::
- You should always set S to "${WORKDIR}/p4" in your recipe.
+ You should always set S to "${UNPACKDIR}/p4" in your recipe.
By default, the fetcher strips the depot location from the local file paths. In
the above example, the content of ``example-depot/main/source/`` will be placed
-in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot
+in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remote depot
paths locally is desirable, the fetcher supports two parameters:
-- *"module":*
- The top-level depot location or directory to fetch. The value of this
- parameter can also point to a single file within the depot, in which case
- the local file path will include the module path.
-- *"remotepath":*
- When used with the value "``keep``", the fetcher will mirror the full depot
- paths locally for the specified location, even in combination with the
- ``module`` parameter.
+- *"module":*
+ The top-level depot location or directory to fetch. The value of this
+ parameter can also point to a single file within the depot, in which case
+ the local file path will include the module path.
+- *"remotepath":*
+ When used with the value "``keep``", the fetcher will mirror the full depot
+ paths locally for the specified location, even in combination with the
+ ``module`` parameter.
Here is an example use of the the ``module`` parameter::
@@ -686,9 +658,9 @@ Such functionality is set by the variable:
delegate access to resources, if this variable is set, the Az Fetcher will
use it when fetching artifacts from the cloud.
-You can specify the AZ_SAS variable as shown below::
+You can specify the AZ_SAS variable prefixed with a ? as shown below::
- AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>"
+ AZ_SAS = "?se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>"
Here is an example URL::
@@ -745,6 +717,11 @@ The format for the :term:`SRC_URI` setting must be::
SRC_URI = "crate://REGISTRY/NAME/VERSION"
+This fetcher supports the following parameters:
+
+- *"protocol":* The protocol used to fetch the crates. The default is "https".
+ You can also use "http".
+
Here is an example URL::
SRC_URI = "crate://crates.io/glob/0.2.11"
@@ -754,6 +731,12 @@ Here is an example URL::
NPM Fetcher (``npm://``)
------------------------
+.. warning::
+
+ The NPM fetcher is currently disabled due to security concerns. See
+ `355cd226e0720a9ed7683bb01c8c0a58eee03664 <https://git.openembedded.org/bitbake/commit/?id=355cd226e0720a9ed7683bb01c8c0a58eee03664>`__
+ for more information.
+
This submodule fetches source code from an
`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__
Javascript package registry.
@@ -793,6 +776,12 @@ to automatically create a recipe from an NPM URL.
NPM shrinkwrap Fetcher (``npmsw://``)
-------------------------------------
+.. warning::
+
+ The NPM fetcher is currently disabled due to security concerns. See
+ `355cd226e0720a9ed7683bb01c8c0a58eee03664 <https://git.openembedded.org/bitbake/commit/?id=355cd226e0720a9ed7683bb01c8c0a58eee03664>`__
+ for more information.
+
This submodule fetches source code from an
`NPM shrinkwrap <https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap>`__
description file, which lists the dependencies
@@ -828,12 +817,8 @@ Other Fetchers
Fetch submodules also exist for the following:
-- Bazaar (``bzr://``)
-
- Mercurial (``hg://``)
-- OSC (``osc://``)
-
- S3 (``s3://``)
- Secure FTP (``sftp://``)
@@ -848,4 +833,42 @@ submodules. However, you might find the code helpful and readable.
Auto Revisions
==============
-We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here.
+For recipes which need to use the latest revision of their source code,
+the way to achieve it is to use :term:`AUTOREV` as the value of the
+source code repository's :term:`SRCREV`::
+
+ SRCREV = "${AUTOREV}"
+
+.. note::
+
+ With :term:`AUTOREV`, BitBake will always need to take the additional step of
+ querying the remote repository to retrieve the latest available revision.
+
+ Also, recipes using it are not part of the parsing-time cache,
+ and hence are parsed every time.
+
+Multiple Source Control Repositories
+====================================
+
+For some recipes it is necessary to make use of more than one
+version controlled source code repository. In such case, the recipe
+must provide BitBake with information about how it should include
+the different SCM revisions in its package version string, instead of its
+usual approach with a single :term:`SRCREV`.
+
+For this purpose, the recipe must set the :term:`SRCREV_FORMAT`
+variable. Consider the following example::
+
+ SRC_URI = " \
+ git://git.some.example.com/source-tree.git;name=machine \
+ git://git.some.example.com/metadata.git;name=meta \
+ "
+ SRCREV_machine = "3f9db490a81eeb0077be3c5a5aa1388a2372232f"
+ SRCREV_meta = "1ac1d0ff730fe1dd1371823d562db8126750a98c"
+ SRCREV_FORMAT ?= "meta_machine"
+
+The value given to :term:`SRCREV_FORMAT` references names, which were
+assigned using the ``name`` parameter in the :term:`SRC_URI` definition,
+and which represent the version controlled source code repositories.
+In the above example, the :term:`SRC_URI` contained two URLs named
+"meta" and "machine".
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst
index 35ffb88b..6801323e 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst
@@ -198,7 +198,7 @@ Class files, which are denoted by the ``.bbclass`` extension, contain
information that is useful to share between metadata files. The BitBake
source tree currently comes with one class metadata file called
``base.bbclass``. You can find this file in the ``classes`` directory.
-The ``base.bbclass`` class files is special since it is always included
+The ``base.bbclass`` class file is special since it is always included
automatically for all recipes and classes. This class contains
definitions for standard basic tasks such as fetching, unpacking,
configuring (empty by default), compiling (runs any Makefile present),
@@ -206,6 +206,35 @@ installing (empty by default) and packaging (empty by default). These
tasks are often overridden or extended by other classes added during the
project development process.
+Class Types
+~~~~~~~~~~~
+
+BitBake supports class files installed in three different directories:
+
+- ``classes-global/``: these classes must be inherited globally through the
+ :term:`INHERIT` variable in a :ref:`configuration file
+ <bitbake-user-manual/bitbake-user-manual-intro:Configuration Files>`. These
+ classes are included for every recipe being built. For example, you would use
+ the global class named ``myclass`` like so::
+
+ INHERIT += "myclass"
+
+- ``classes-recipe/``: these classes must be inherited from a recipe using the
+ :ref:`inherit <ref-bitbake-user-manual-metadata-inherit>` or
+ :ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+ directive. They do not support being inherited globally. For example, you
+ would use the recipe class named ``myclass`` like so::
+
+ inherit myclass
+
+- ``classes/``: this final directory is meant for classes that can be used in
+ the two contexts explained above. In other words, they can be inherited either
+ globally or in a recipe.
+
+For details on how BitBake locates class files, see the
+:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Locating Class Files`
+section of the BitBake User Manual.
+
Layers
------
@@ -242,12 +271,9 @@ same root filename. The filenames can differ only in the file type
suffix used (e.g. ``formfactor_0.0.bb`` and
``formfactor_0.0.bbappend``).
-Information in append files extends or overrides the information in the
-underlying, similarly-named recipe files.
-
When you name an append file, you can use the "``%``" wildcard character
-to allow for matching recipe names. For example, suppose you have an
-append file named as follows::
+to allow for a less strict recipe name matching. For example, suppose you have
+an append file named as follows::
busybox_1.21.%.bbappend
@@ -258,12 +284,8 @@ the append file would match the following recipe names::
busybox_1.21.1.bb
busybox_1.21.2.bb
busybox_1.21.3.bb
-
-.. note::
-
- The use of the " % " character is limited in that it only works directly in
- front of the .bbappend portion of the append file's name. You cannot use the
- wildcard character in any other location of the name.
+ busybox_1.21.10.bb
+ busybox_1.21.11.bb
If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the
append name would not match. However, if you named the append file
@@ -272,6 +294,47 @@ append name would not match. However, if you named the append file
In the most general case, you could name the append file something as
simple as ``busybox_%.bbappend`` to be entirely version independent.
+A recipe named without a version may only be appended by an append file named
+exactly the same. That is, a recipe named::
+
+ mesa.bb
+
+can only be appended by an append file named::
+
+ mesa.bbappend
+
+That is, ``mesa_%.bbappend`` append file will not match a ``mesa.bb`` recipe.
+
+.. note::
+
+ The "``%``" character is a marker for BitBake, and everything after that
+ marker will not be used for matching recipe names. That is, an append file
+ named::
+
+ busybox_1.%.11.bbappend
+
+ will match recipes named::
+
+ busybox_1.0.11.bb
+ busybox_1.2.bb
+ busybox_1.3.48.bb
+
+ Note that the "``%``" character is not specific to the version part of the
+ recipe name. That is, an append file named::
+
+ linux-yocto%.bbappend
+
+ will match recipes named::
+
+ linux-yocto.bb
+ linux-yocto_6.18.bb
+ linux-yocto-tiny_6.12.bb
+ linux-yocto-rt_6.16.bb
+
+ While it can be useful to append to similar recipes with slightly different
+ names, it is highly discouraged to do so as recipes from other layers may be
+ named similarly enough that the append file would unwittingly match them.
+
Obtaining BitBake
=================
@@ -349,40 +412,84 @@ Usage and syntax
Following is the usage and syntax for BitBake::
$ bitbake -h
- Usage: bitbake [options] [recipename/target recipe:do_task ...]
-
- Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
- It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
- will provide the layer, BBFILES and other configuration information.
+ usage: bitbake [-s] [-e] [-g] [-u UI] [--version] [-h] [-f] [-c CMD]
+ [-C INVALIDATE_STAMP] [--runall RUNALL] [--runonly RUNONLY]
+ [--no-setscene] [--skip-setscene] [--setscene-only] [-n] [-p]
+ [-k] [-P] [-S SIGNATURE_HANDLER] [--revisions-changed]
+ [-b BUILDFILE] [-D] [-l DEBUG_DOMAINS] [-v] [-q]
+ [-w WRITEEVENTLOG] [-B BIND] [-T SERVER_TIMEOUT]
+ [--remote-server REMOTE_SERVER] [-m] [--token XMLRPCTOKEN]
+ [--observe-only] [--status-only] [--server-only] [-r PREFILE]
+ [-R POSTFILE] [-I EXTRA_ASSUME_PROVIDED]
+ [recipename/target ...]
+
+ It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH
+ which will provide the layer, BBFILES and other configuration information.
+
+ General options:
+ recipename/target Execute the specified task (default is 'build') for
+ these target recipes (.bb files).
+ -s, --show-versions Show current and preferred versions of all recipes.
+ -e, --environment Show the global or per-recipe environment complete
+ with information about where variables were
+ set/changed.
+ -g, --graphviz Save dependency tree information for the specified
+ targets in the dot syntax.
+ -u UI, --ui UI The user interface to use (knotty, ncurses, taskexp,
+ taskexp_ncurses or teamcity - default knotty).
+ --version Show programs version and exit.
+ -h, --help Show this help message and exit.
- Options:
- --version show program's version number and exit
- -h, --help show this help message and exit
- -b BUILDFILE, --buildfile=BUILDFILE
- Execute tasks from a specific .bb recipe directly.
- WARNING: Does not handle any dependencies from other
- recipes.
- -k, --continue Continue as much as possible after an error. While the
- target that failed and anything depending on it cannot
- be built, as much as possible will be built before
- stopping.
+ Task control options:
-f, --force Force the specified targets/task to run (invalidating
any existing stamp file).
- -c CMD, --cmd=CMD Specify the task to execute. The exact options
+ -c CMD, --cmd CMD Specify the task to execute. The exact options
available depend on the metadata. Some examples might
be 'compile' or 'populate_sysroot' or 'listtasks' may
give a list of the tasks available.
- -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
+ -C INVALIDATE_STAMP, --clear-stamp INVALIDATE_STAMP
Invalidate the stamp for the specified task such as
'compile' and then run the default task for the
specified target(s).
- -r PREFILE, --read=PREFILE
- Read the specified file before bitbake.conf.
- -R POSTFILE, --postread=POSTFILE
- Read the specified file after bitbake.conf.
- -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also
- print bb.note(...) messages to stdout (in addition to
- writing them to ${T}/log.do_<task>).
+ --runall RUNALL Run the specified task for any recipe in the taskgraph
+ of the specified target (even if it wouldn't otherwise
+ have run).
+ --runonly RUNONLY Run only the specified task within the taskgraph of
+ the specified targets (and any task dependencies those
+ tasks may have).
+ --no-setscene Do not run any setscene tasks. sstate will be ignored
+ and everything needed, built.
+ --skip-setscene Skip setscene tasks if they would be executed. Tasks
+ previously restored from sstate will be kept, unlike
+ --no-setscene.
+ --setscene-only Only run setscene tasks, don't run any real tasks.
+
+ Execution control options:
+ -n, --dry-run Don't execute, just go through the motions.
+ -p, --parse-only Quit after parsing the BB recipes.
+ -k, --continue Continue as much as possible after an error. While the
+ target that failed and anything depending on it cannot
+ be built, as much as possible will be built before
+ stopping.
+ -P, --profile Profile the command and save reports.
+ -S SIGNATURE_HANDLER, --dump-signatures SIGNATURE_HANDLER
+ Dump out the signature construction information, with
+ no task execution. The SIGNATURE_HANDLER parameter is
+ passed to the handler. Two common values are none and
+ printdiff but the handler may define more/less. none
+ means only dump the signature, printdiff means
+ recursively compare the dumped signature with the most
+ recent one in a local build or sstate cache (can be
+ used to find out why tasks re-run when that is not
+ expected)
+ --revisions-changed Set the exit code depending on whether upstream
+ floating revisions have changed or not.
+ -b BUILDFILE, --buildfile BUILDFILE
+ Execute tasks from a specific .bb recipe directly.
+ WARNING: Does not handle any dependencies from other
+ recipes.
+
+ Logging/output control options:
-D, --debug Increase the debug level. You can specify this more
than once. -D sets the debug level to 1, where only
bb.debug(1, ...) messages are printed to stdout; -DD
@@ -392,65 +499,47 @@ Following is the usage and syntax for BitBake::
-D only affects output to stdout. All debug messages
are written to ${T}/log.do_taskname, regardless of the
debug level.
+ -l DEBUG_DOMAINS, --log-domains DEBUG_DOMAINS
+ Show debug logging for the specified logging domains.
+ -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also
+ print bb.note(...) messages to stdout (in addition to
+ writing them to ${T}/log.do_<task>).
-q, --quiet Output less log message data to the terminal. You can
specify this more than once.
- -n, --dry-run Don't execute, just go through the motions.
- -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
- Dump out the signature construction information, with
- no task execution. The SIGNATURE_HANDLER parameter is
- passed to the handler. Two common values are none and
- printdiff but the handler may define more/less. none
- means only dump the signature, printdiff means compare
- the dumped signature with the cached one.
- -p, --parse-only Quit after parsing the BB recipes.
- -s, --show-versions Show current and preferred versions of all recipes.
- -e, --environment Show the global or per-recipe environment complete
- with information about where variables were
- set/changed.
- -g, --graphviz Save dependency tree information for the specified
- targets in the dot syntax.
- -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
- Assume these dependencies don't exist and are already
- provided (equivalent to ASSUME_PROVIDED). Useful to
- make dependency graphs more appealing
- -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
- Show debug logging for the specified logging domains
- -P, --profile Profile the command and save reports.
- -u UI, --ui=UI The user interface to use (knotty, ncurses, taskexp or
- teamcity - default knotty).
- --token=XMLRPCTOKEN Specify the connection token to be used when
- connecting to a remote server.
- --revisions-changed Set the exit code depending on whether upstream
- floating revisions have changed or not.
- --server-only Run bitbake without a UI, only starting a server
- (cooker) process.
- -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind
+ -w WRITEEVENTLOG, --write-log WRITEEVENTLOG
+ Writes the event log of the build to a bitbake event
+ json file. Use '' (empty string) to assign the name
+ automatically.
+
+ Server options:
+ -B BIND, --bind BIND The name/address for the bitbake xmlrpc server to bind
to.
- -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT
+ -T SERVER_TIMEOUT, --idle-timeout SERVER_TIMEOUT
Set timeout to unload bitbake server due to
inactivity, set to -1 means no unload, default:
Environment variable BB_SERVER_TIMEOUT.
- --no-setscene Do not run any setscene tasks. sstate will be ignored
- and everything needed, built.
- --skip-setscene Skip setscene tasks if they would be executed. Tasks
- previously restored from sstate will be kept, unlike
- --no-setscene
- --setscene-only Only run setscene tasks, don't run any real tasks.
- --remote-server=REMOTE_SERVER
+ --remote-server REMOTE_SERVER
Connect to the specified server.
-m, --kill-server Terminate any running bitbake server.
+ --token XMLRPCTOKEN Specify the connection token to be used when
+ connecting to a remote server.
--observe-only Connect to a server as an observing-only client.
--status-only Check the status of the remote bitbake server.
- -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
- Writes the event log of the build to a bitbake event
- json file. Use '' (empty string) to assign the name
- automatically.
- --runall=RUNALL Run the specified task for any recipe in the taskgraph
- of the specified target (even if it wouldn't otherwise
- have run).
- --runonly=RUNONLY Run only the specified task within the taskgraph of
- the specified targets (and any task dependencies those
- tasks may have).
+ --server-only Run bitbake without a UI, only starting a server
+ (cooker) process.
+
+ Configuration options:
+ -r PREFILE, --read PREFILE
+ Read the specified file before bitbake.conf.
+ -R POSTFILE, --postread POSTFILE
+ Read the specified file after bitbake.conf.
+ -I EXTRA_ASSUME_PROVIDED, --ignore-deps EXTRA_ASSUME_PROVIDED
+ Assume these dependencies don't exist and are already
+ provided (equivalent to ASSUME_PROVIDED). Useful to
+ make dependency graphs more appealing.
+
+..
+ BitBake help output generated with "stty columns 80; bin/bitbake -h"
.. _bitbake-examples:
@@ -481,7 +570,7 @@ The following command runs the clean task on the ``foo.bb`` recipe file::
.. note::
- The "-b" option explicitly does not handle recipe dependencies. Other
+ The ``-b`` option explicitly does not handle recipe dependencies. Other
than for debugging purposes, it is instead recommended that you use
the syntax presented in the next section.
@@ -496,7 +585,7 @@ dependencies, both for build-time and runtime. There must be a way for
you to express recipe preferences when multiple recipes provide the same
functionality, or when there are multiple versions of a recipe.
-The ``bitbake`` command, when not using "--buildfile" or "-b" only
+The ``bitbake`` command, when not using ``--buildfile`` or ``-b`` only
accepts a "PROVIDES". You cannot provide anything else. By default, a
recipe file generally "PROVIDES" its "packagename" as shown in the
following example::
@@ -504,7 +593,7 @@ following example::
$ bitbake foo
This next example "PROVIDES" the
-package name and also uses the "-c" option to tell BitBake to just
+package name and also uses the ``-c`` option to tell BitBake to just
execute the ``do_clean`` task::
$ bitbake -c clean foo
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst
new file mode 100644
index 00000000..09e35394
--- /dev/null
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions.rst
@@ -0,0 +1,59 @@
+.. SPDX-License-Identifier: CC-BY-2.5
+
+=================
+Library Functions
+=================
+
+|
+
+This chapter lists common library functions available under the ``lib/``
+directory in BitBake.
+
+These functions can be used in recipes or configuration files with
+:ref:`inline-Python <bitbake-user-manual/bitbake-user-manual-metadata:Inline
+Python Variable Expansion>` or :ref:`Python
+<bitbake-user-manual/bitbake-user-manual-metadata:BitBake-Style Python
+Functions>` functions.
+
+Logging utilities
+=================
+
+Different logging utilities can be used from Python code in recipes or
+configuration files.
+
+The strings passed below can be formatted with ``str.format()``, for example::
+
+ bb.warn("Houston, we have a %s", "bit of a problem")
+
+Formatted string can also be used directly::
+
+ bb.error("%s, we have a %s" % ("Houston", "big problem"))
+
+Python f-strings may also be used::
+
+ h = "Houston"
+ bb.fatal(f"{h}, we have a critical problem")
+
+.. automodule:: bb
+ :members:
+ debug,
+ error,
+ erroronce,
+ fatal,
+ note,
+ plain,
+ verbnote,
+ warn,
+ warnonce,
+
+``bb.utils``
+============
+
+.. automodule:: bb.utils
+ :members:
+ :exclude-members:
+ LogCatcher,
+ PrCtlError,
+ VersionStringException,
+ better_compile,
+ better_exec,
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
index 58975f4c..621aab25 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
@@ -172,6 +172,26 @@ assignment, ``BAR`` expands to the literal string "${FOO}" as long as
BAR = "${FOO}"
+Immediate variable expansion (:=)
+---------------------------------
+
+The ":=" operator results in a variable's contents being expanded
+immediately, rather than when the variable is actually used::
+
+ T = "123"
+ A := "test ${T}"
+ T = "456"
+ B := "${T} ${C}"
+ C = "cval"
+ C := "${C}append"
+
+In this example, ``A`` contains "test 123", even though the final value
+of :term:`T` is "456". The variable :term:`B` will end up containing "456
+cvalappend". This is because references to undefined variables are
+preserved as is during (immediate)expansion. This is in contrast to GNU
+Make, where undefined variables expand to nothing. The variable ``C``
+contains "cvalappend" since ``${C}`` immediately expands to "cval".
+
Setting a default value (?=)
----------------------------
@@ -235,26 +255,6 @@ After parsing we will have::
W = "xy"
-Immediate variable expansion (:=)
----------------------------------
-
-The ":=" operator results in a variable's contents being expanded
-immediately, rather than when the variable is actually used::
-
- T = "123"
- A := "test ${T}"
- T = "456"
- B := "${T} ${C}"
- C = "cval"
- C := "${C}append"
-
-In this example, ``A`` contains "test 123", even though the final value
-of :term:`T` is "456". The variable :term:`B` will end up containing "456
-cvalappend". This is because references to undefined variables are
-preserved as is during (immediate)expansion. This is in contrast to GNU
-Make, where undefined variables expand to nothing. The variable ``C``
-contains "cvalappend" since ``${C}`` immediately expands to "cval".
-
.. _appending-and-prepending:
Appending (+=) and prepending (=+) With Spaces
@@ -608,7 +608,7 @@ variable.
DEPENDS = "glibc ncurses"
OVERRIDES = "machine:local"
- DEPENDS:append:machine = "libmad"
+ DEPENDS:append:machine = " libmad"
In this example, :term:`DEPENDS` becomes "glibc ncurses libmad".
@@ -617,8 +617,8 @@ variable.
``KERNEL_FEATURES`` variable based on the architecture::
KERNEL_FEATURES:append = " ${KERNEL_EXTRA_FEATURES}"
- KERNEL_FEATURES:append:qemux86=" cfg/sound.scc cfg/paravirt_kvm.scc"
- KERNEL_FEATURES:append:qemux86-64=" cfg/sound.scc cfg/paravirt_kvm.scc"
+ KERNEL_FEATURES:append:qemux86 = " cfg/sound.scc cfg/paravirt_kvm.scc"
+ KERNEL_FEATURES:append:qemux86-64 = " cfg/sound.scc cfg/paravirt_kvm.scc"
- *Setting a Variable for a Single Task:* BitBake supports setting a
variable just for the duration of a single task. Here is an example::
@@ -753,23 +753,17 @@ share the task.
This section presents the mechanisms BitBake provides to allow you to
share functionality between recipes. Specifically, the mechanisms
-include ``include``, ``inherit``, :term:`INHERIT`, and ``require``
-directives.
-
-Locating Include and Class Files
---------------------------------
-
-BitBake uses the :term:`BBPATH` variable to locate
-needed include and class files. Additionally, BitBake searches the
-current directory for ``include`` and ``require`` directives.
-
-.. note::
-
- The BBPATH variable is analogous to the environment variable PATH .
-
-In order for include and class files to be found by BitBake, they need
-to be located in a "classes" subdirectory that can be found in
-:term:`BBPATH`.
+include :ref:`inherit<ref-bitbake-user-manual-metadata-inherit>`,
+:ref:`inherit_defer<ref-bitbake-user-manual-metadata-inherit-defer>`,
+:ref:`include<ref-include-directive>`,
+:ref:`include_all<ref-include-all-directive>`,
+:ref:`require<require-inclusion>` and
+:ref:`INHERIT<bitbake-user-manual/bitbake-user-manual-metadata:\`\`INHERIT\`\` configuration directive>`
+directives. There is also a higher-level abstraction called
+`configuration fragments` that is enabled with the
+:ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\` directive`.
+
+.. _ref-bitbake-user-manual-metadata-inherit:
``inherit`` Directive
---------------------
@@ -809,65 +803,131 @@ An advantage with the inherit directive as compared to both the
:ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>`
directives is that you can inherit class files conditionally. You can
accomplish this by using a variable expression after the ``inherit``
-statement. Here is an example::
+statement.
- inherit ${VARNAME}
+For inheriting classes conditionally, using the :ref:`inherit_defer
+<ref-bitbake-user-manual-metadata-inherit-defer>` directive is advised as
+:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` is
+evaluated at the end of parsing.
-If ``VARNAME`` is
-going to be set, it needs to be set before the ``inherit`` statement is
-parsed. One way to achieve a conditional inherit in this case is to use
-overrides::
+.. _ref-bitbake-user-manual-metadata-inherit-defer:
- VARIABLE = ""
- VARIABLE:someoverride = "myclass"
+``inherit_defer`` Directive
+---------------------------
-Another method is by using anonymous Python. Here is an example::
+The :ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+directive works like the :ref:`inherit
+<ref-bitbake-user-manual-metadata-inherit>` directive, except that it is only
+evaluated at the end of parsing. Its usage is recommended when a conditional
+expression is used.
- python () {
- if condition == value:
- d.setVar('VARIABLE', 'myclass')
- else:
- d.setVar('VARIABLE', '')
- }
+This allows conditional expressions to be evaluated "late", meaning changes to
+the variable after the line is parsed will take effect. With the :ref:`inherit
+<ref-bitbake-user-manual-metadata-inherit>` directive this is not the case.
+
+Here is an example::
+
+ inherit_defer ${VARNAME}
+
+One way to achieve a conditional inherit in this case is to use
+overrides::
-Alternatively, you could use an in-line Python expression in the
+ VARNAME = ""
+ VARNAME:someoverride = "myclass"
+
+:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+defers the evaluation of ``${VARNAME}`` until the end of
+parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARNAME}``
+expands to ``myclass``, which is then inherited.
+
+Alternatively, you could use an inline Python expression in the
following form::
- inherit ${@'classname' if condition else ''}
- inherit ${@functionname(params)}
+ inherit_defer ${@'classname' if condition else ''}
+
+Or::
+
+ inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)}
In all cases, if the expression evaluates to an
empty string, the statement does not trigger a syntax error because it
becomes a no-op.
+See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes
+``inherit`` calls to ``inherit_defer``.
+
+.. _ref-include-directive:
+
``include`` Directive
---------------------
-BitBake understands the ``include`` directive. This directive causes
-BitBake to parse whatever file you specify, and to insert that file at
-that location. The directive is much like its equivalent in Make except
-that if the path specified on the include line is a relative path,
-BitBake locates the first file it can find within :term:`BBPATH`.
+The ``include`` directive causes BitBake to parse a given file,
+and to include that file's contents at the location of the
+``include`` statement. This directive is similar to its equivalent
+in Make, except that if the path specified on the BitBake ``include``
+line is a relative path, BitBake will search for it on the path designated
+by :term:`BBPATH` and will include *only the first matching file*.
-The include directive is a more generic method of including
+The ``include`` directive is a more generic method of including
functionality as compared to the :ref:`inherit <bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\`\` directive>`
directive, which is restricted to class (i.e. ``.bbclass``) files. The
-include directive is applicable for any other kind of shared or
+``include`` directive is applicable for any other kind of shared or
encapsulated functionality or configuration that does not suit a
``.bbclass`` file.
-As an example, suppose you needed a recipe to include some self-test
-definitions::
+For example, if you needed a recipe to include some self-test definitions,
+you might write::
include test_defs.inc
+The ``include`` directive does not produce an error if the specified file
+cannot be found. If the included file *must* exist, then you should use
+use :ref:`require <require-inclusion>` instead, which will generate an error
+if the file cannot be found.
+
+.. note::
+
+ Note well that the ``include`` directive will include the first matching
+ file and nothing further (which is almost always the behaviour you want).
+ If you need to include all matching files, you need to use the
+ ``include_all`` directive, explained below.
+
+.. _ref-include-all-directive:
+
+``include_all`` Directive
+-------------------------
+
+The ``include_all`` directive works like the :ref:`include
+<bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>`
+directive but will include *all* of the files that match the specified path in
+the enabled layers (layers part of :term:`BBLAYERS`).
+
+Note that only :term:`BBPATH` will be searched, the parent directory of the file
+with the ``include_all`` directive will not be searched (unlike for the
+``include`` directive).
+
.. note::
- The include directive does not produce an error when the file cannot be
- found. Consequently, it is recommended that if the file you are including is
- expected to exist, you should use :ref:`require <require-inclusion>` instead
- of include . Doing so makes sure that an error is produced if the file cannot
- be found.
+ This behaviour is rarely what you want in normal operation, and should
+ be reserved for only those situations when you explicitly want to parse
+ and include all matching files found across all layers, as the following
+ example shows.
+
+As a realistic example of this directive, imagine that all of your active
+layers contain a file ``conf/distro/include/maintainers.inc``, containing
+maintainer information for the recipes in that layer, and you wanted to
+collect all of the content from all of those files across all of those layers.
+You could use the statement::
+
+ include_all conf/distro/include/maintainers.inc
+
+In this case, BitBake will iterate through all of the directories in
+the colon-separated :term:`BBPATH` (from left to right) and collect the
+contents of all matching files, so you end up with the maintainer
+information of all of your active layers, not just the first one.
+
+As the ``include_all`` directive uses the ``include`` directive in the
+background, as with ``include``, no error is produced if no files are matched.
.. _require-inclusion:
@@ -933,6 +993,197 @@ the ``autotools`` and ``pkgconfig`` classes::
INHERIT += "autotools pkgconfig"
+``addfragments`` Directive
+--------------------------
+
+This directive allows fine-tuning local configurations with configuration
+snippets contained in layers in a structured, controlled way. Typically it would
+go into ``bitbake.conf``, for example::
+
+ addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN
+
+``addfragments`` takes four parameters:
+
+- path prefix for fragment files inside the layer file tree that bitbake
+ uses to construct full paths to the fragment files
+
+- name of variable that holds the list of enabled fragments in an
+ active build
+
+- name of variable that contains a list of variable names containing
+ fragment-specific metadata (such as descriptions)
+
+- name of variable that contains definitions for built-in fragments
+
+This allows listing enabled configuration fragments in ``OE_FRAGMENTS``
+variable like this::
+
+ OE_FRAGMENTS = "core/domain/somefragment core/someotherfragment anotherlayer/anotherdomain/anotherfragment"
+
+Fragment names listed in this variable must be prefixed by the layer name
+where a fragment file is located, defined by :term:`BBFILE_COLLECTIONS` in ``layer.conf``.
+
+The implementation then expands this list into
+:ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>`
+directives with full paths to respective layers::
+
+ require /path/to/core-layer/conf/fragments/domain/somefragment.conf
+ require /path/to/core-layer/conf/fragments/someotherfragment.conf
+ require /path/to/another-layer/conf/fragments/anotherdomain/anotherfragment.conf
+
+The variable containing a list of fragment metadata variables could look like this::
+
+ OE_FRAGMENTS_METADATA_VARS = "BB_CONF_FRAGMENT_SUMMARY BB_CONF_FRAGMENT_DESCRIPTION"
+
+The implementation will add a flag containing the fragment name to each of those variables
+when parsing fragments, so that the variables are namespaced by fragment name, and do not override
+each other when several fragments are enabled.
+
+The variable containing a built-in fragment definitions could look like this::
+
+ OE_FRAGMENTS_BUILTIN = "someprefix:SOMEVARIABLE anotherprefix:ANOTHERVARIABLE"
+
+and then if 'someprefix/somevalue' is added to the variable that holds the list
+of enabled fragments::
+
+ OE_FRAGMENTS = "... someprefix/somevalue"
+
+bitbake will treat that as direct value assignment in its configuration::
+
+ SOMEVARIABLE = "somevalue"
+
+Locating Include Files
+----------------------
+
+BitBake first searches the current directory for needed include files for
+:ref:`include <ref-include-directive>` and :ref:`require <require-inclusion>`
+directives.
+BitBake also uses the :term:`BBPATH` variable.
+
+.. note::
+
+ The :term:`BBPATH` variable is analogous to the environment variable ``PATH``.
+
+For these two directives, BitBake includes the first file it finds.
+
+Let's consider the following statement called from a recipe file located in
+``/layers/meta-custom2/recipes-example/example/example_0.1.bb``::
+
+ require myfile.inc
+
+Where ``myfile.inc`` is located in ``/layers/meta-custom2/recipes-example/example``.
+
+And let's assume that the value of :term:`BBPATH` is
+``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to find
+``myfile.inc`` in this order::
+
+ /layers/meta-custom2/recipes-example/example/myfile.inc
+ /layers/meta-custom1/myfile.inc
+ /layers/meta-custom2/myfile.inc
+
+In this case the first path of the list matches and BitBake includes this file
+in ``example_0.1.bb``.
+
+Another common example would be::
+
+ require recipes-other/other/otherfile.inc
+
+Where ``otherfile.inc`` is located in
+``/layers/meta-custom1/recipes-other/other/``.
+
+In this case, the following paths would be searched::
+
+ /layers/meta-custom2/recipes-example/example/recipes-other/other/otherfile.inc
+ /layers/meta-custom1/recipes-other/other/otherfile.inc
+ /layers/meta-custom2/recipes-other/other/otherfile.inc
+
+This time, the second item of this list would be matched.
+
+Note that the first path is based on the location of the file with the
+``require`` (or ``include``) directive. Imagine there's a
+``/layers/meta-custom2/recipes-bbappend/example/example_0.1.bbappend`` with::
+
+ require myappend.inc
+
+In this case, the following paths would be searched::
+
+ /layers/meta-custom2/recipes-bbappend/example/myappend.inc
+ /layers/meta-custom1/myappend.inc
+ /layers/meta-custom2/myappend.inc
+
+.. note::
+
+ In the above examples, the exact same search order applies for the
+ :ref:`include <ref-include-directive>` directive.
+
+It is also possible to include *all* occurences of a file with the same name
+with the :ref:`include_all <ref-include-all-directive>` directive.
+
+Let's consider the following statement called from a recipe file located in
+``/layers/meta-custom2/recipes-example/example/exampleall_0.1.bb``::
+
+ include_all all.inc
+
+Where multiple ``all.inc`` are in located in ``/layers/meta-custom2`` and
+``/layers/meta-custom1``.
+
+And let's assume that the value of :term:`BBPATH` is
+``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to include
+all ``all.inc`` in this order::
+
+ /layers/meta-custom1/all.inc
+ /layers/meta-custom2/all.inc
+
+In this case the first ``/layers/meta-custom1/all.inc`` would be included, and
+then ``/layers/meta-custom2/all.inc`` if both are found.
+
+.. note::
+
+ The same logic as for the :ref:`ref-include-directive` applies, except that
+ the path relative to the file where the directive is specified is not
+ searched when using the ``include_all`` directive.
+
+Locating Class Files
+--------------------
+
+Like include files, class files are located using the :term:`BBPATH` variable.
+The classes can be included in the ``classes-recipe``, ``classes-global`` and
+``classes`` directories, as explained in the
+:ref:`bitbake-user-manual/bitbake-user-manual-intro:Class types` section of the
+BitBake User Manual. Like for the :ref:`include <ref-include-directive>` and
+:ref:`require <require-inclusion>` directives, BitBake stops and inherits the
+first class that it finds.
+
+For classes inherited with the :ref:`inherit
+<ref-bitbake-user-manual-metadata-inherit>` directive, BitBake will try to
+locate the class under each ``classes-recipe`` directory for each path in
+:term:`BBPATH`, and then do the same for each ``classes`` directory for each
+path in :term:`BBPATH`.
+
+For example, if the value :term:`BBPATH` is
+``/layers/meta-custom1:/layers/meta-custom2`` then the ``hello`` class
+would be searched in this order::
+
+ /layers/meta-custom1/classes-recipe/hello.bbclass
+ /layers/meta-custom2/classes-recipe/hello.bbclass
+ /layers/meta-custom1/classes/hello.bbclass
+ /layers/meta-custom2/classes/hello.bbclass
+
+.. note::
+
+ Note that the order of the list above does not depend on where the class is
+ inherited from.
+
+Likewise, for classes inherited with the :term:`INHERIT` variable, the
+``classes-global`` directory is searched first, and the ``classes`` directory is
+searched second. Taking the above example, this would result in the following
+list::
+
+ /layers/meta-custom1/classes-global/hello.bbclass
+ /layers/meta-custom2/classes-global/hello.bbclass
+ /layers/meta-custom1/classes/hello.bbclass
+ /layers/meta-custom2/classes/hello.bbclass
+
Functions
=========
@@ -972,7 +1223,9 @@ When you create these types of functions in
your recipe or class files, you need to follow the shell programming
rules. The scripts are executed by ``/bin/sh``, which may not be a bash
shell but might be something such as ``dash``. You should not use
-Bash-specific script (bashisms).
+Bash-specific script (bashisms), such as ``[[`` (use ``[`` instead). The BitBake
+shell parser also has further
+:ref:`limitations <bitbake-user-manual/bitbake-user-manual-metadata:limitations>`.
Overrides and override-style operators like ``:append`` and ``:prepend``
can also be applied to shell functions. Most commonly, this application
@@ -1013,6 +1266,20 @@ Running ``do_foo`` prints the following::
You can use the ``bitbake -e recipename`` command to view the final
assembled function after all overrides have been applied.
+Limitations
+^^^^^^^^^^^
+
+Shell functions must be parsable by BitBake, which cannot handle all
+syntax supported by POSIX shells. In particular:
+
+#. Arithmetic expansion is not supported. This means you cannot do:
+
+ .. code-block:: shell
+
+ VAR=$(( i + 1 ))
+
+ See https://bugzilla.yoctoproject.org/show_bug.cgi?id=11314.
+
BitBake-Style Python Functions
------------------------------
@@ -1303,8 +1570,8 @@ the task and other tasks. Here is an example that shows how to define a
task and declare some dependencies::
python do_printdate () {
- import time
- print time.strftime('%Y%m%d', time.gmtime())
+ import datetime
+ bb.plain('Date: %s' % (datetime.date.today()))
}
addtask printdate after do_fetch before do_build
@@ -1496,22 +1763,22 @@ functionality of the task:
directory listed is used as the current working directory for the
task.
-- ``[file-checksums]``: Controls the file dependencies for a task. The
- baseline file list is the set of files associated with
- :term:`SRC_URI`. May be used to set additional dependencies on
- files not associated with :term:`SRC_URI`.
+- ``[file-checksums]``: Controls the file dependencies for a task. The
+ baseline file list is the set of files associated with
+ :term:`SRC_URI`. May be used to set additional dependencies on
+ files not associated with :term:`SRC_URI`.
- The value set to the list is a file-boolean pair where the first
- value is the file name and the second is whether or not it
- physically exists on the filesystem. ::
+ The value set to the list is a file-boolean pair where the first
+ value is the file name and the second is whether or not it
+ physically exists on the filesystem. ::
- do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True"
+ do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True"
- It is important to record any paths which the task looked at and
- which didn't exist. This means that if these do exist at a later
- time, the task can be rerun with the new additional files. The
- "exists" True or False value after the path allows this to be
- handled.
+ It is important to record any paths which the task looked at and
+ which didn't exist. This means that if these do exist at a later
+ time, the task can be rerun with the new additional files. The
+ "exists" True or False value after the path allows this to be
+ handled.
- ``[lockfiles]``: Specifies one or more lockfiles to lock while the
task executes. Only one task may hold a lockfile, and any task that
@@ -1524,6 +1791,9 @@ functionality of the task:
shouldn't access the network outside of ``do_fetch`` as it usually
undermines fetcher source mirroring, image and licence manifests, software
auditing and supply chain security.
+ Tasks without the network flag will run with PR_SET_NO_NEW_PRIVS,
+ which prevents executables from gaining privileges through
+ file capabilities.
- ``[noexec]``: When set to "1", marks the task as being empty, with
no execution required. You can use the ``[noexec]`` flag to set up
@@ -1635,8 +1905,9 @@ and the content of the :term:`FILE` variable::
print("The name of the Event is %s" % getName(e))
print("The file we run for is %s" % d.getVar('FILE'))
}
- myclass_eventhandler[eventmask] = "bb.event.BuildStarted
- bb.event.BuildCompleted"
+ myclass_eventhandler[eventmask] = "bb.event.BuildStarted \
+ bb.event.BuildCompleted \
+ "
In the previous example, an eventmask has been
set so that the handler only sees the "BuildStarted" and
@@ -1972,11 +2243,8 @@ access. Here is a list of available operations:
Other Functions
---------------
-You can find many other functions that can be called from Python by
-looking at the source code of the ``bb`` module, which is in
-``bitbake/lib/bb``. For example, ``bitbake/lib/bb/utils.py`` includes
-the commonly used functions ``bb.utils.contains()`` and
-``bb.utils.mkdirhier()``, which come with docstrings.
+Other functions are documented in the
+:doc:`/bitbake-user-manual/bitbake-user-manual-library-functions` document.
Extending Python Library Code
-----------------------------
@@ -2016,12 +2284,12 @@ OpenEmbedded metadata-based example.
These checksums are stored in :term:`STAMP`. You can
examine the checksums using the following BitBake command::
- $ bitbake-dumpsigs
+ $ bitbake-dumpsig
This command returns the signature data in a readable
format that allows you to examine the inputs used when the OpenEmbedded
build system generates signatures. For example, using
-``bitbake-dumpsigs`` allows you to examine the ``do_compile`` task's
+``bitbake-dumpsig`` allows you to examine the ``do_compile`` task's
"sigdata" for a C application (e.g. ``bash``). Running the command also
reveals that the "CC" variable is part of the inputs that are hashed.
Any changes to this variable would invalidate the stamp and cause the
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index 899e584f..fd75275b 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -36,6 +36,14 @@ overview of their function and contents.
when specified allows for the Git binary from the host to be used
rather than building ``git-native``.
+ :term:`AUTOREV`
+ This is a special variable used during fetching. When :term:`SRCREV` is
+ set to the value of this variable, the latest revision from the version
+ controlled source code repository is used.
+ It should be set as follows::
+
+ SRCREV = "${AUTOREV}"
+
:term:`AZ_SAS`
Azure Storage Shared Access Signature, when using the
:ref:`Azure Storage fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`
@@ -108,6 +116,10 @@ overview of their function and contents.
Specifies if SSL certificates should be checked when fetching. The default
value is ``1`` and certificates are not checked if the value is set to ``0``.
+ :term:`BB_CMDLINE`
+ :term:`BB_CMDLINE` is an internal variable and is read-only. It captures
+ the exact command line used for the current invocation of BitBake.
+
:term:`BB_HASH_CODEPARSER_VALS`
Specifies values for variables to use when populating the codeparser cache.
This can be used selectively to set dummy values for variables to avoid
@@ -123,21 +135,24 @@ overview of their function and contents.
Specifies the path to a log file into which BitBake's user interface
writes output during the build.
+ This variable's value remains persistent during the entire execution of
+ the BitBake server. Changes to this variable will only take effect
+ after a server restart, which can be accomplished with the ``bitbake
+ --kill-server`` command.
+
+ .. note::
+
+ For more details on how to make the BitBake server persistent, see the
+ ``BB_SERVER_TIMEOUT`` variable.
+
:term:`BB_CURRENTTASK`
Contains the name of the currently running task. The name does not
include the ``do_`` prefix.
- :term:`BB_DANGLINGAPPENDS_WARNONLY`
- Defines how BitBake handles situations where an append file
- (``.bbappend``) has no corresponding recipe file (``.bb``). This
- condition often occurs when layers get out of sync (e.g. ``oe-core``
- bumps a recipe version and the old recipe no longer exists and the
- other layer has not been updated to the new version of the recipe
- yet).
-
- The default fatal behavior is safest because it is the sane reaction
- given something is out of sync. It is important to realize when your
- changes are no longer being applied.
+ :term:`BB_CURRENT_MC`
+ Contains the name of the current multiconfig a task is being run under.
+ The name is taken from the multiconfig configuration file (a file
+ ``mc1.conf`` would make this variable equal to ``mc1``).
:term:`BB_DEFAULT_TASK`
The default task to use when none is specified (e.g. with the ``-c``
@@ -148,6 +163,25 @@ overview of their function and contents.
The default umask to apply to tasks if specified and no task specific
umask flag is set.
+ :term:`BB_DEFER_BBCLASSES`
+ The classes listed in this variable have their :ref:`inherit
+ <ref-bitbake-user-manual-metadata-inherit>` calls automatically promoted
+ to deferred inherits. See :ref:`inherit_defer
+ <ref-bitbake-user-manual-metadata-inherit-defer>` for more information on
+ deferred inherits.
+
+ This means that if :term:`BB_DEFER_BBCLASSES` is set as follows::
+
+ BB_DEFER_BBCLASSES = "foo"
+
+ The following statement::
+
+ inherit foo
+
+ Will automatically be equal to calling::
+
+ inherit_defer foo
+
:term:`BB_DISKMON_DIRS`
Monitors disk space and available inodes during the build and allows
you to control the build based on these parameters.
@@ -262,8 +296,8 @@ overview of their function and contents.
Specifies the internal list of variables to allow through from
the external environment into BitBake's datastore. If the value of
this variable is not specified (which is the default), the following
- list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`,
- :term:`BB_ENV_PASSTHROUGH`, and :term:`BB_ENV_PASSTHROUGH_ADDITIONS`.
+ list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, and
+ :term:`BB_ENV_PASSTHROUGH_ADDITIONS`.
.. note::
@@ -317,21 +351,42 @@ overview of their function and contents.
For example usage, see :term:`BB_GIT_SHALLOW`.
+ :term:`BB_GIT_DEFAULT_DESTSUFFIX`
+ The default destination directory where the :ref:`Git fetcher
+ <git-fetcher>` unpacks the source code. If this variable is not set, the
+ source code is unpacked in a directory named "git".
+
:term:`BB_GIT_SHALLOW`
Setting this variable to "1" enables the support for fetching, using and
generating mirror tarballs of `shallow git repositories <https://riptutorial.com/git/example/4584/shallow-clone>`_.
- The external `git-make-shallow <https://git.openembedded.org/bitbake/tree/bin/git-make-shallow>`_
- script is used for shallow mirror tarball creation.
+ A git shallow fetch is used for shallow mirror tarball creation (i.e.
+ ``git fetch --depth <depth> ...`` or
+ ``git fetch --shallow-exclude=<revs> ...``.
When :term:`BB_GIT_SHALLOW` is enabled, bitbake will attempt to fetch a shallow
mirror tarball. If the shallow mirror tarball cannot be fetched, it will
try to fetch the full mirror tarball and use that.
- When a mirror tarball is not available, a full git clone will be performed
- regardless of whether this variable is set or not. Support for shallow
- clones is not currently implemented as git does not directly support
- shallow cloning a particular git commit hash (it only supports cloning
- from a tag or branch reference).
+ This setting causes an initial shallow clone instead of an initial full bare clone.
+ The amount of data transferred during the initial clone will be significantly reduced.
+
+ However, every time the source revision (referenced in :term:`SRCREV`)
+ changes, regardless of whether the cache within the download directory
+ (defined by :term:`DL_DIR`) has been cleaned up or not,
+ the data transfer may be significantly higher because entirely
+ new shallow clones are required for each source revision change.
+
+ Over time, numerous shallow clones may cumulatively transfer
+ the same amount of data as an initial full bare clone.
+ This is especially the case with very large repositories.
+
+ Existing initial full bare clones, created without this setting,
+ will still be utilized.
+
+ If the Git error "Server does not allow request for unadvertised object"
+ occurs, an initial full bare clone is fetched automatically.
+ This may happen if the Git server does not allow the request
+ or if the Git client has issues with this functionality.
See also :term:`BB_GIT_SHALLOW_DEPTH` and
:term:`BB_GENERATE_SHALLOW_TARBALLS`.
@@ -359,6 +414,23 @@ overview of their function and contents.
For example usage, see :term:`BB_GIT_SHALLOW`.
+ :term:`BB_GIT_SHALLOW_EXTRA_REFS`
+ With :term:`BB_GIT_SHALLOW` enabled, by default, all unused refs (branches
+ and tags) are removed from the repository, as shallow processing scales
+ with the number of refs it has to process.
+
+ :term:`BB_GIT_SHALLOW_EXTRA_REFS` allows to explicitly specify additional
+ refs to keep. This is particularly useful for recipes with custom checkout
+ processes, or whose Git-based versioning requires a tag be available (i.e.
+ for ``git describe --tags``). The :term:`BB_GIT_SHALLOW_EXTRA_REFS`
+ variable is a space-separated list of refs, fully specified, and supports
+ wildcards.
+
+ Example usage::
+
+ BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0"
+ BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*"
+
:term:`BB_GLOBAL_PYMODULES`
Specifies the list of Python modules to place in the global namespace.
It is intended that only the core layer should set this and it is meant
@@ -414,6 +486,29 @@ overview of their function and contents.
equivalences that correspond to Share State caches that are
only available on specific clients.
+ :term:`BB_HASHSERVE_DB_DIR`
+ When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use
+ a private location inside a particular build directory for the sqlite
+ database file that holds hash equivalency data.
+
+ This variable allows using a different path, which can be shared
+ between multiple build directories and bitbake instances
+ that operate on them. This enables using a common ``${SSTATE_DIR}``
+ together with common hash equivalency data for local builds, without having to
+ separately manage a hash equivalency server.
+
+ .. note::
+
+ This variable cannot be set to a NFS mount and bitbake will error out then.
+ The reason is that NFS implementations can have file locking issues, which
+ can cause data loss and corruption when there are multiple writers operating
+ on a file at the same time as explained in https://sqlite.org/faq.html#q5
+
+ If you'd like to share hash equivalency data between multiple computers, you
+ need to set up a hash equivalency server separately and point :term:`BB_HASHSERVE`
+ to it. See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for
+ additional information.
+
:term:`BB_HASHSERVE_UPSTREAM`
Specifies an upstream Hash Equivalence server.
@@ -424,7 +519,7 @@ overview of their function and contents.
Example usage::
- BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
+ BB_HASHSERVE_UPSTREAM = "hashserv.yoctoproject.org:8686"
:term:`BB_INVALIDCONF`
Used in combination with the ``ConfigParsed`` event to trigger
@@ -525,11 +620,32 @@ overview of their function and contents.
version 4.20 expose under ``/proc/pressure``. The threshold represents
the difference in "total" pressure from the previous second. The
minimum value is 1.0 (extremely slow builds) and the maximum is
- 1000000 (a pressure value unlikely to ever be reached).
+ 1000000 (a pressure value unlikely to ever be reached). See
+ https://docs.kernel.org/accounting/psi.html for more information.
+
+ A default value to limit the CPU pressure to be set in ``conf/local.conf``
+ could be::
+
+ BB_PRESSURE_MAX_CPU = "15000"
+
+ Multiple values should be tested on the build host to determine what suits
+ best, depending on the need for performance versus load average during
+ the build.
+
+ .. note::
+
+ You can track the pressure information while a build is running with:
+
+ .. code-block:: console
- This threshold can be set in ``conf/local.conf`` as::
+ $ tail -F tmp/log/cooker/<machine>/console-latest.log | grep "Pressure status changed"
+ NOTE: Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
+ ...
- BB_PRESSURE_MAX_CPU = "500"
+ If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_CPU`
+ should be increased to a reasonable value for limiting the CPU pressure
+ on the system. Monitor the varying value after ``CPU:`` above to set a
+ sensible value.
:term:`BB_PRESSURE_MAX_IO`
Specifies a maximum I/O pressure threshold, above which BitBake's
@@ -541,14 +657,38 @@ overview of their function and contents.
version 4.20 expose under ``/proc/pressure``. The threshold represents
the difference in "total" pressure from the previous second. The
minimum value is 1.0 (extremely slow builds) and the maximum is
- 1000000 (a pressure value unlikely to ever be reached).
+ 1000000 (a pressure value unlikely to ever be reached). See
+ https://docs.kernel.org/accounting/psi.html for more information.
At this point in time, experiments show that IO pressure tends to
be short-lived and regulating just the CPU with
:term:`BB_PRESSURE_MAX_CPU` can help to reduce it.
- :term:`BB_PRESSURE_MAX_MEMORY`
+ A default value to limit the IO pressure to be set in ``conf/local.conf``
+ could be::
+
+ BB_PRESSURE_MAX_IO = "15000"
+
+ Multiple values should be tested on the build host to determine what suits
+ best, depending on the need for performance versus I/O usage during the
+ build.
+
+ .. note::
+
+ You can track the pressure information while a build is running with:
+
+ .. code-block:: console
+ $ tail -F tmp/log/cooker/<machine>/console-latest.log | grep "Pressure status changed"
+ NOTE: Pressure status changed to CPU: None, IO: True, Mem: False (CPU: 2236.0/None, IO: 153.6/2.0, Mem: 0.0/2.0) - using 19/64 bitbake threads
+ ...
+
+ If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_IO`
+ should be increased to a reasonable value for limiting the I/O pressure
+ on the system. Monitor the varying value after ``IO:`` above to set a
+ sensible value.
+
+ :term:`BB_PRESSURE_MAX_MEMORY`
Specifies a maximum memory pressure threshold, above which BitBake's
scheduler will not start new tasks (providing there is at least
one active task). If no value is set, memory pressure is not
@@ -558,7 +698,8 @@ overview of their function and contents.
version 4.20 expose under ``/proc/pressure``. The threshold represents
the difference in "total" pressure from the previous second. The
minimum value is 1.0 (extremely slow builds) and the maximum is
- 1000000 (a pressure value unlikely to ever be reached).
+ 1000000 (a pressure value unlikely to ever be reached). See
+ https://docs.kernel.org/accounting/psi.html for more information.
Memory pressure is experienced when time is spent swapping,
refaulting pages from the page cache or performing direct reclaim.
@@ -566,6 +707,30 @@ overview of their function and contents.
might be useful as a last resort to prevent OOM errors if they are
occurring during builds.
+ A default value to limit the memory pressure to be set in
+ ``conf/local.conf`` could be::
+
+ BB_PRESSURE_MAX_MEMORY = "15000"
+
+ Multiple values should be tested on the build host to determine what suits
+ best, depending on the need for performance versus memory consumption
+ during the build.
+
+ .. note::
+
+ You can track the pressure information while a build is running with:
+
+ .. code-block:: console
+
+ $ tail -F tmp/log/cooker/<machine>/console-latest.log | grep "Pressure status changed"
+ NOTE: Pressure status changed to CPU: None, IO: False, Mem: True (CPU: 29.5/None, IO: 0.0/2.0, Mem: 2553.3/2.0) - using 17/64 bitbake threads
+ ...
+
+ If these messages are printed a lot, it means that the :term:`BB_PRESSURE_MAX_MEMORY`
+ should be increased to a reasonable value for limiting the memory
+ pressure on the system. Monitor the varying value after ``Mem:`` above
+ to set a sensible value.
+
:term:`BB_RUNFMT`
Specifies the name of the executable script files (i.e. run files)
saved into ``${``\ :term:`T`\ ``}``. By default, the
@@ -680,11 +845,11 @@ overview of their function and contents.
.. note::
In order for your I/O priority settings to take effect, you need the
- Completely Fair Queuing (CFQ) Scheduler selected for the backing block
+ Budget Fair Queuing (BFQ) Scheduler selected for the backing block
device. To select the scheduler, use the following command form where
device is the device (e.g. sda, sdb, and so forth)::
- $ sudo sh -c "echo cfq > /sys/block/device/queu/scheduler"
+ $ sudo sh -c "echo bfq > /sys/block/device/queue/scheduler"
:term:`BB_TASK_NICE_LEVEL`
Allows specific tasks to change their priority (i.e. nice level).
@@ -699,6 +864,12 @@ overview of their function and contents.
Within an executing task, this variable holds the hash of the task as
returned by the currently enabled signature generator.
+ :term:`BB_USE_HOME_NPMRC`
+ Controls whether or not BitBake uses the user's .npmrc file within their
+ home directory within the npm fetcher. This can be used for authentication
+ of private NPM registries, among other uses. This is turned off by default
+ and requires the user to explicitly set it to "1" to enable.
+
:term:`BB_VERBOSE_LOGS`
Controls how verbose BitBake is during builds. If set, shell scripts
echo commands and shell script output appears on standard out
@@ -766,6 +937,10 @@ overview of their function and contents.
:term:`BBFILE_PRIORITY`
Assigns the priority for recipe files in each layer.
+ This variable is used in the ``conf/layer.conf`` file and must be
+ suffixed with a `_` followed by the name of the specific layer (e.g.
+ ``BBFILE_PRIORITY_emenlow``). Colon as separator is not supported.
+
This variable is useful in situations where the same recipe appears
in more than one layer. Setting this variable allows you to
prioritize a layer against other layers that contain the same recipe
@@ -780,7 +955,7 @@ overview of their function and contents.
higher precedence. For example, the value 6 has a higher precedence
than the value 5. If not specified, the :term:`BBFILE_PRIORITY` variable
is set based on layer dependencies (see the :term:`LAYERDEPENDS` variable
- for more information. The default priority, if unspecified for a
+ for more information). The default priority, if unspecified for a
layer with no dependencies, is the lowest defined priority + 1 (or 1
if no priorities are defined).
@@ -884,28 +1059,101 @@ overview of their function and contents.
compiler. Consequently, the syntax follows Python's Regular
Expression (re) syntax. The expressions are compared against the full
paths to the files. For complete syntax information, see Python's
- documentation at http://docs.python.org/3/library/re.html.
+ documentation at https://docs.python.org/3/library/re.html.
The following example uses a complete regular expression to tell
- BitBake to ignore all recipe and recipe append files in the
- ``meta-ti/recipes-misc/`` directory::
+ BitBake to ignore all recipe and recipe append files in
+ ``recipes-bsp`` directory (recursively) of ``meta-ti-bsp``::
- BBMASK = "meta-ti/recipes-misc/"
+ BBMASK = "${BBFILE_PATTERN_meta-ti-bsp}/recipes-bsp/"
If you want to mask out multiple directories or recipes, you can
specify multiple regular expression fragments. This next example
masks out multiple directories and individual recipes::
- BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
- BBMASK += "/meta-oe/recipes-support/"
- BBMASK += "/meta-foo/.*/openldap"
- BBMASK += "opencv.*\.bbappend"
- BBMASK += "lzma"
+ BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/"
+ BBMASK += "${BBFILE_PATTERN_openembedded-layer}/recipes-support/"
+ BBMASK += "${BBFILE_PATTERN_openembedded-layer}/.*/openldap"
+ BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/.*/optee.*\.bbappend"
+
+ This masks:
+
+ - everything under the ``recipes-graphics/libgal/`` directory from
+ ``meta-ti-bsp``,
+ - everything under the ``recipes-support/`` directory in ``meta-oe``,
+ - everything under a directory whose name starts with ``openldap``, and
+ every file with the same naming scheme, in ``meta-oe`` at any directory
+ depth > 1 (e.g. in ``meta-oe``, ``recipes-foo/openldap-stuff/`` or
+ ``recipes-bar/baz/openldap_0.1.bb`` but not ``openldap/``),
+ - every append file whose name starts with ``optee`` in ``meta-ti-bsp`` at
+ any directory depth > 1 (e.g. ``optee/optee-examples_%.bbappend`` and
+ ``recipes-security/optee/optee-client_%.bbappend``).
.. note::
- When specifying a directory name, use the trailing slash character
- to ensure you match just that directory name.
+ Because these are complete regular expressions, if you want to match a
+ directory and not a file, you must end the expression with a trailing
+ slash. That is::
+
+ BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal/"
+
+ Will match anything under ``recipes-graphics/ligbal/`` directory of
+ ``meta-ti-bsp``. And::
+
+ BBMASK += "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libgal"
+
+ Will match in ``meta-ti-bsp`` any file prefixed with ``libgal`` in
+ ``recipes-graphics/`` and any directory (recursively; and its
+ recipes and recipe append files regardless how they are named) prefixed
+ with ``libgal`` in ``recipes-graphics/``. That is, provided your layers
+ are available at ``/bitbake-builds/poky-master/layers/``, it'll match::
+
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal.bb
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal_%.bbappend
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo.bb
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal-foo/foo/bz.bbappend
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/bar.bb
+
+ .. note::
+
+ Because these are complete regular expressions, failing to start the
+ pattern with a ``^`` sign (which is usually the first character in
+ :term:`BBFILE_PATTERN`) means it can match *any* portion of a path.
+ Take the following as an example::
+
+ BBMASK = "recipes-graphics/libgal/"
+
+ This will match subdirectories and files in any path containing
+ ``recipes-graphics/libgal/``, meaning (considering your layers are
+ available at ``/bitbake-builds/poky-master/layers/``)::
+
+ /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes-graphics/libgal/
+ /bitbake-builds/poky-master/layers/my-layer/foo-recipes-graphics/libgal/
+
+ will be both matched. This may be a more relaxed way of matching
+ directories, recipes and recipe append files from any third party layer
+ but is generally discouraged as it may be casting too wide of a net.
+
+ .. note::
+
+ Because these are complete regular expressions, a leading slash does
+ not mean the path is absolute. It simply forces the directory to be
+ named exactly that. Take::
+
+ BBMASK = "recipes-graphics/libgal/"
+
+ If you happen to have a directory ``foo-recipes-graphics/libgal/``, it
+ will be matched.
+
+ Leading with a slash::
+
+ BBMASK = "/recipes-graphics/libgal/"
+
+ makes sure that doesn't happen. However, this doesn't prevent matching
+ a directory ``recipes-graphics/libgal/`` from another layer.
+
+ Again, we highly recommend using :term:`BBFILE_PATTERN` in the
+ expressions to specify absolute paths specific to one layer.
:term:`BBMULTICONFIG`
Enables BitBake to perform multiple configuration builds and lists
@@ -963,18 +1211,10 @@ overview of their function and contents.
A name assigned to the build. The name defaults to a datetime stamp
of when the build was started but can be defined by the metadata.
- :term:`BZRDIR`
- The directory in which files checked out of a Bazaar system are
- stored.
-
:term:`CACHE`
Specifies the directory BitBake uses to store a cache of the metadata
so it does not need to be parsed every time BitBake is started.
- :term:`CVSDIR`
- The directory in which files checked out under the CVS system are
- stored.
-
:term:`DEFAULT_PREFERENCE`
Specifies a weak bias for recipe selection priority.
@@ -1100,9 +1340,17 @@ overview of their function and contents.
:term:`INHERIT`
Causes the named class or classes to be inherited globally. Anonymous
- functions in the class or classes are not executed for the base
- configuration and in each individual recipe. The OpenEmbedded build
- system ignores changes to :term:`INHERIT` in individual recipes.
+ functions in the class or classes are executed in two disjoint situations:
+
+ - When only the
+ :ref:`base configuration <bitbake-user-manual/bitbake-user-manual-execution:parsing the base configuration metadata>`
+ is parsed. For example as a result of the following BitBake invocation::
+
+ $ bitbake -e
+
+ - When recipes are parsed - then for each parsed recipe.
+
+ BitBake ignores changes to :term:`INHERIT` in individual recipes.
For more information on :term:`INHERIT`, see the
":ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\`\` configuration directive`"
@@ -1178,6 +1426,58 @@ overview of their function and contents.
upstream source, and then locations specified by :term:`MIRRORS` in that
order.
+ This variable has a specific syntax. Each line must contain a regular
+ expression matching the original source URL and a replacement URL for it.
+ For example::
+
+ MIRRORS:prepend = " \
+ git://git.openembedded.org/.* git:///mirrors/openembedded/BASENAME \
+ "
+
+ In the above example:
+
+ - The leftmost part means that any Git repository (``git://`` fetcher)
+ whose URL starts with ``git.openembedded.org/`` will be matched.
+
+ - The rightmost part is the replacement URL for the original repository,
+ which is hosted locally in the ``/mirrors/openembedded`` directory. It
+ also uses the ``BASENAME`` keyword which is detailed below.
+
+ The rightmost part of each line can contain keywords which are
+ automatically replaced by BitBake. These keywords can contain either a
+ sub-part of the original URL or meta-information about it.
+
+ For what follows, let's take the following URL as an example for the
+ original source (matched by the leftmost part of the line):
+ ``git://git.openembedded.org/project/repo.git``.
+
+ A replacement for this URL can be created using the following keywords:
+
+ - ``TYPE``: the URL scheme or fetcher type. For the example above: ``git``.
+ - ``HOST``: the URL host. For the example above: ``git.openembedded.org``.
+ - ``PATH``: the URL path. For the example above: ``/project/repo.git``.
+ - ``BASENAME``: the URL basename. For the example above: ``repo.git``.
+ - ``MIRRORNAME``: a translation of the host and path with common
+ characters substituted to dots (``.``). For the example above:
+ ``git.openembedded.org.project.repo.git``.
+
+ The rightmost part of the line can also be a bare URL that uses no special
+ keywords (usually ``https://`` or ``file://``). For example::
+
+ MIRRORS:prepend = " \
+ git://git.openembedded.org/.* file:///mirrors/openembedded/ \
+ "
+
+ In the above example, the fetcher will try to find a mirror tarball in the
+ ``/mirrors/openembedded`` directory. The name of this tarball should follow
+ the name of the tarball created by BitBake (for Git repositories,
+ the :term:`BB_GENERATE_MIRROR_TARBALLS` variable must be set to "1" to
+ generate them).
+
+ If we take ``git://git.openembedded.org/project/repo.git`` as an example,
+ the tarball would have to be named
+ ``git2_git.openembedded.org.project.repo.git.tar.gz``.
+
:term:`OVERRIDES`
A colon-separated list that BitBake uses to control what variables are
overridden after BitBake parses recipes and configuration files.
@@ -1213,8 +1513,13 @@ overview of their function and contents.
:term:`PERSISTENT_DIR`
Specifies the directory BitBake uses to store data that should be
preserved between builds. In particular, the data stored is the data
- that uses BitBake's persistent data API and the data used by the PR
- Server and PR Service.
+ that uses BitBake's persistent data API, the data used by the PR
+ Server and PR Service, and the default location of the Hash Equivalence
+ database (when :term:`BB_HASHSERVE` is set to ``auto``).
+
+ This directory should not be shared between different builds. If you need
+ to share the Hash Equivalence database, you should setup a Hash
+ Equivalence server instead.
:term:`PF`
Specifies the recipe or package name and includes all version and
@@ -1253,6 +1558,20 @@ overview of their function and contents.
PREFERRED_PROVIDER_xxx = "yyy"
PREFERRED_PROVIDER_aaa = "bbb"
+ :term:`PREFERRED_RPROVIDER`
+ Determines which *recipe* should be given preference when
+ multiple packages declare runtime-providing (:term:`RPROVIDES`)
+ the same item. Some examples::
+
+ PREFERRED_RPROVIDER_initd-functions ?= "initscripts"
+ PREFERRED_RPROVIDER_virtual-libegl-icd ?= "mesa"
+
+ The former will select as the package built by the `initscripts`
+ recipe declaring to be runtime-provider for `initd-functions`,
+ which is in this case `initscripts-functions`::
+
+ .../initscripts_1.0.bb:RPROVIDES:${PN}-functions = "initd-functions"
+
:term:`PREFERRED_VERSION`
If there are multiple versions of a recipe available, this variable
determines which version should be given preference. You must always
@@ -1301,6 +1620,10 @@ overview of their function and contents.
use ``file://`` URLs to point to local directories or network shares as
well.
+ The syntax of the :term:`PREMIRRORS` variable is the same as
+ :term:`MIRRORS`. See the definition of the :term:`MIRRORS` for more
+ details.
+
:term:`PROVIDES`
A list of aliases by which a particular recipe can be known. By
default, a recipe's own :term:`PN` is implicitly already in its
@@ -1460,14 +1783,8 @@ overview of their function and contents.
- ``az://``: Fetches files from an Azure Storage account using HTTPS.
- - ``bzr://``: Fetches files from a Bazaar revision control
- repository.
-
- ``ccrc://``: Fetches files from a ClearCase repository.
- - ``cvs://``: Fetches files from a CVS revision control
- repository.
-
- ``file://``: Fetches files, which are usually files shipped
with the Metadata, from the local machine.
The path is relative to the :term:`FILESPATH`
@@ -1502,8 +1819,11 @@ overview of their function and contents.
- ``npm://``: Fetches JavaScript modules from a registry.
- - ``osc://``: Fetches files from an OSC (OpenSUSE Build service)
- revision control repository.
+ .. warning::
+
+ The NPM fetcher is currently disabled due to security concerns. See
+ `355cd226e0720a9ed7683bb01c8c0a58eee03664 <https://git.openembedded.org/bitbake/commit/?id=355cd226e0720a9ed7683bb01c8c0a58eee03664>`__
+ for more information.
- ``p4://``: Fetches files from a Perforce (``p4``) revision
control repository.
@@ -1522,7 +1842,8 @@ overview of their function and contents.
- ``name``: Specifies a name to be used for association with
:term:`SRC_URI` checksums or :term:`SRCREV` when you have more than one
- file or git repository specified in :term:`SRC_URI`. For example::
+ file or source control repository specified in :term:`SRC_URI`.
+ For example::
SRC_URI = "git://example.com/foo.git;branch=main;name=first \
git://example.com/bar.git;branch=main;name=second \
@@ -1535,7 +1856,8 @@ overview of their function and contents.
- ``subdir``: Places the file (or extracts its contents) into the
specified subdirectory. This option is useful for unusual tarballs
or other archives that do not have their files already in a
- subdirectory within the archive.
+ subdirectory within the archive. This path can be further modified
+ by fetcher specific parameters.
- ``subpath``: Limits the checkout to a specific subpath of the
tree when using the Git fetcher is used.
@@ -1550,18 +1872,18 @@ overview of their function and contents.
:term:`SRCREV`
The revision of the source code used to build the package. This
- variable applies only when using Subversion, Git, Mercurial and
- Bazaar. If you want to build a fixed revision and you want to avoid
+ variable applies only when using Subversion, Git and Mercurial.
+ If you want to build a fixed revision and you want to avoid
performing a query on the remote repository every time BitBake parses
your recipe, you should specify a :term:`SRCREV` that is a full revision
identifier and not just a tag.
:term:`SRCREV_FORMAT`
- Helps construct valid :term:`SRCREV` values when
+ Helps construct a valid package version string when
multiple source controlled URLs are used in
:term:`SRC_URI`.
- The system needs help constructing these values under these
+ The system needs help constructing this value under these
circumstances. Each component in the :term:`SRC_URI` is assigned a name
and these are referenced in the :term:`SRCREV_FORMAT` variable. Consider
an example with URLs named "machine" and "meta". In this case,
@@ -1570,6 +1892,12 @@ overview of their function and contents.
``AUTOINC`` placeholder is added and if needed. And, this placeholder
is placed at the start of the returned string.
+ The :term:`SRCREV_FORMAT` can also take the form "_component2".
+ This assumes that there is a component in the :term:`SRC_URI` that does not
+ have a name assigned. While this is not considered good practice, it can be
+ usefull if a ``.bbappend`` file needs to extend the :term:`SRC_URI` with
+ an additional repository.
+
:term:`STAMP`
Specifies the base path used to create recipe stamp files. The path
to an actual stamp file is constructed by evaluating this string and
diff --git a/bitbake/doc/conf.py b/bitbake/doc/conf.py
index fc2ee081..9a53e55c 100644
--- a/bitbake/doc/conf.py
+++ b/bitbake/doc/conf.py
@@ -6,18 +6,27 @@
# -- Path setup --------------------------------------------------------------
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
-
-import sys
import datetime
+import os
+import subprocess
+import sys
+
+from pathlib import Path
-current_version = "dev"
+# Test that we are building from a Git repository
+try:
+ subprocess.run(["git", "rev-parse", "--is-inside-work-tree"],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
+except subprocess.CalledProcessError:
+ sys.exit("Building bitbake's documentation must be done from its Git repository.\n"
+ "Clone the repository with the following command:\n"
+ "git clone https://git.openembedded.org/bitbake ")
+
+sys.path.insert(0, os.path.abspath('.'))
+import setversions
+
+current_version = setversions.get_current_version()
+print(f"Version calculated to be {current_version}")
# String used in sidebar
version = 'Version: ' + current_version
@@ -26,6 +35,12 @@ if current_version == 'dev':
# Version seen in documentation_options.js and hence in js switchers code
release = current_version
+setversions.write_switchers_js("sphinx-static/switchers.js.in",
+ "sphinx-static/switchers.js",
+ current_version)
+
+setversions.write_releases_rst("releases.rst")
+
# -- Project information -----------------------------------------------------
project = 'Bitbake'
@@ -37,6 +52,7 @@ author = 'Richard Purdie, Chris Larson, and Phil Blundell'
extlinks = {
'yocto_docs': ('https://docs.yoctoproject.org%s', None),
'oe_lists': ('https://lists.openembedded.org%s', None),
+ 'wikipedia': ('https://en.wikipedia.org/wiki/%s', None),
}
# -- General configuration ---------------------------------------------------
@@ -47,6 +63,7 @@ extlinks = {
extensions = [
'sphinx.ext.autosectionlabel',
'sphinx.ext.extlinks',
+ 'sphinx.ext.autodoc',
]
autosectionlabel_prefix_document = True
@@ -99,3 +116,7 @@ html_last_updated_fmt = '%b %d, %Y'
# Remove the trailing 'dot' in section numbers
html_secnumber_suffix = " "
+
+# autoconf needs the modules available to auto-generate documentation from the
+# code
+sys.path.insert(0, str(Path('..', 'lib').resolve()))
diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst
index ee1660ac..9f2a9067 100644
--- a/bitbake/doc/index.rst
+++ b/bitbake/doc/index.rst
@@ -12,10 +12,12 @@ BitBake User Manual
bitbake-user-manual/bitbake-user-manual-intro
bitbake-user-manual/bitbake-user-manual-execution
+ bitbake-user-manual/bitbake-user-manual-environment-setup
bitbake-user-manual/bitbake-user-manual-metadata
bitbake-user-manual/bitbake-user-manual-ref-variables-context
bitbake-user-manual/bitbake-user-manual-fetching
bitbake-user-manual/bitbake-user-manual-ref-variables
+ bitbake-user-manual/bitbake-user-manual-library-functions
bitbake-user-manual/bitbake-user-manual-hello
.. toctree::
diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst
deleted file mode 100644
index b38b1c06..00000000
--- a/bitbake/doc/releases.rst
+++ /dev/null
@@ -1,174 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-=================================
-BitBake Supported Release Manuals
-=================================
-
-*******************************
-Release Series 4.2 (mickledore)
-*******************************
-
-- :yocto_docs:`BitBake 2.4 User Manual </bitbake/2.4/>`
-
-******************************
-Release Series 4.0 (kirkstone)
-******************************
-
-- :yocto_docs:`BitBake 2.0 User Manual </bitbake/2.0/>`
-
-****************************
-Release Series 3.1 (dunfell)
-****************************
-
-- :yocto_docs:`BitBake 1.46 User Manual </bitbake/1.46/>`
-
-================================
-BitBake Outdated Release Manuals
-================================
-
-*****************************
-Release Series 4.1 (langdale)
-*****************************
-
-- :yocto_docs:`BitBake 2.2 User Manual </bitbake/2.2/>`
-
-******************************
-Release Series 3.4 (honister)
-******************************
-
-- :yocto_docs:`BitBake 1.52 User Manual </bitbake/1.52/>`
-
-******************************
-Release Series 3.3 (hardknott)
-******************************
-
-- :yocto_docs:`BitBake 1.50 User Manual </bitbake/1.50/>`
-
-*******************************
-Release Series 3.2 (gatesgarth)
-*******************************
-
-- :yocto_docs:`BitBake 1.48 User Manual </bitbake/1.48/>`
-
-*******************************************
-Release Series 3.1 (dunfell first versions)
-*******************************************
-
-- :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.1.3 BitBake User Manual </3.1.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-Release Series 3.0 (zeus)
-*************************
-
-- :yocto_docs:`3.0 BitBake User Manual </3.0/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.1 BitBake User Manual </3.0.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.2 BitBake User Manual </3.0.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.3 BitBake User Manual </3.0.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.4 BitBake User Manual </3.0.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-****************************
-Release Series 2.7 (warrior)
-****************************
-
-- :yocto_docs:`2.7 BitBake User Manual </2.7/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.1 BitBake User Manual </2.7.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.2 BitBake User Manual </2.7.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.3 BitBake User Manual </2.7.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.4 BitBake User Manual </2.7.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-Release Series 2.6 (thud)
-*************************
-
-- :yocto_docs:`2.6 BitBake User Manual </2.6/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.1 BitBake User Manual </2.6.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.2 BitBake User Manual </2.6.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.3 BitBake User Manual </2.6.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.4 BitBake User Manual </2.6.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-Release Series 2.5 (sumo)
-*************************
-
-- :yocto_docs:`2.5 Documentation </2.5>`
-- :yocto_docs:`2.5.1 Documentation </2.5.1>`
-- :yocto_docs:`2.5.2 Documentation </2.5.2>`
-- :yocto_docs:`2.5.3 Documentation </2.5.3>`
-
-**************************
-Release Series 2.4 (rocko)
-**************************
-
-- :yocto_docs:`2.4 BitBake User Manual </2.4/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.1 BitBake User Manual </2.4.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.2 BitBake User Manual </2.4.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.3 BitBake User Manual </2.4.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.4 BitBake User Manual </2.4.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-Release Series 2.3 (pyro)
-*************************
-
-- :yocto_docs:`2.3 BitBake User Manual </2.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.1 BitBake User Manual </2.3.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.2 BitBake User Manual </2.3.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.3 BitBake User Manual </2.3.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.4 BitBake User Manual </2.3.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-Release Series 2.2 (morty)
-**************************
-
-- :yocto_docs:`2.2 BitBake User Manual </2.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.1 BitBake User Manual </2.2.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.2 BitBake User Manual </2.2.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.3 BitBake User Manual </2.2.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-****************************
-Release Series 2.1 (krogoth)
-****************************
-
-- :yocto_docs:`2.1 BitBake User Manual </2.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.1 BitBake User Manual </2.1.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.2 BitBake User Manual </2.1.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.3 BitBake User Manual </2.1.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-***************************
-Release Series 2.0 (jethro)
-***************************
-
-- :yocto_docs:`1.9 BitBake User Manual </1.9/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0 BitBake User Manual </2.0/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.1 BitBake User Manual </2.0.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.2 BitBake User Manual </2.0.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.3 BitBake User Manual </2.0.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-Release Series 1.8 (fido)
-*************************
-
-- :yocto_docs:`1.8 BitBake User Manual </1.8/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.8.1 BitBake User Manual </1.8.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.8.2 BitBake User Manual </1.8.2/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-Release Series 1.7 (dizzy)
-**************************
-
-- :yocto_docs:`1.7 BitBake User Manual </1.7/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.1 BitBake User Manual </1.7.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.2 BitBake User Manual </1.7.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.3 BitBake User Manual </1.7.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-Release Series 1.6 (daisy)
-**************************
-
-- :yocto_docs:`1.6 BitBake User Manual </1.6/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.1 BitBake User Manual </1.6.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.2 BitBake User Manual </1.6.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.3 BitBake User Manual </1.6.3/bitbake-user-manual/bitbake-user-manual.html>`
-
diff --git a/bitbake/doc/setversions.py b/bitbake/doc/setversions.py
new file mode 100755
index 00000000..2df384e9
--- /dev/null
+++ b/bitbake/doc/setversions.py
@@ -0,0 +1,374 @@
+#!/usr/bin/env python3
+#
+# This file defines is used in doc/conf.py to setup the version information for
+# the documentation:
+# - get_current_version() used in doc/conf.py computes the current version by
+# trying to guess the approximate versions we're at using git tags and
+# branches from the repository.
+# - write_switchers_js() write the switchers.js file used for switching between
+# versions of the documentation.
+#
+# Copyright (c) 2026 Antonin Godard <antonin.godard@bootlin.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import itertools
+import json
+import os
+import re
+import subprocess
+import sys
+import textwrap
+
+from urllib.request import urlopen, URLError
+
+# NOTE: the following variables contain default values in case we are not able to fetch
+# the releases.json file from https://dashboard.yoctoproject.org/releases.json
+DEVBRANCH = "2.18"
+LTSSERIES = ["2.8", "2.0"]
+ACTIVERELEASES = ["2.16"] + LTSSERIES
+YOCTO_MAPPING = {
+ "2.18": "wrynose",
+ "2.16": "whinlatter",
+ "2.8": "scarthgap",
+ "2.0": "kirkstone",
+}
+
+RELEASES_FROM_JSON = {}
+
+# Use the local releases.json file if found, fetch it from the dashboard otherwise
+releases_json_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "releases.json")
+try:
+ with open(releases_json_path, "r") as f:
+ RELEASES_FROM_JSON = json.load(f)
+except FileNotFoundError:
+ print("Fetching releases.json from https://dashboard.yoctoproject.org/releases.json...",
+ file=sys.stderr)
+ try:
+ with urlopen("https://dashboard.yoctoproject.org/releases.json") as r, \
+ open(releases_json_path, "w") as f:
+ RELEASES_FROM_JSON = json.load(r)
+ json.dump(RELEASES_FROM_JSON, f)
+ except URLError:
+ print("WARNING: tried to fetch https://dashboard.yoctoproject.org/releases.json "
+ "but failed, using default values for active releases", file=sys.stderr)
+ pass
+
+if RELEASES_FROM_JSON:
+ ACTIVERELEASES = []
+ DEVBRANCH = ""
+ LTSSERIES = []
+ YOCTO_MAPPING = {}
+
+ for release in RELEASES_FROM_JSON:
+ bb_ver = release["bitbake_version"]
+ if release["status"] == "Active Development":
+ DEVBRANCH = bb_ver
+ if "LTS until" in release["status"]:
+ LTSSERIES.append(bb_ver)
+ if release["bitbake_version"]:
+ YOCTO_MAPPING[bb_ver] = release["release_codename"]
+
+ # Find the first non-dev release, which should be displayed as the default
+ # page on the docs website.
+ current_branch = ""
+ for release in RELEASES_FROM_JSON:
+ if release["status"] != "Active Development":
+ current_branch = release["bitbake_version"]
+ break
+
+ if not current_branch:
+ sys.exit("Unable to find a current release! Exiting...")
+
+ # make the list of releases unique, there can be duplication when the
+ # current releases is also an LTS
+ ACTIVERELEASES = list(dict.fromkeys([current_branch] + LTSSERIES))
+
+print(f"ACTIVERELEASES calculated to be {ACTIVERELEASES}", file=sys.stderr)
+print(f"DEVBRANCH calculated to be {DEVBRANCH}", file=sys.stderr)
+print(f"LTSSERIES calculated to be {LTSSERIES}", file=sys.stderr)
+
+BB_RELEASE_TAG_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+$")
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Parse https://dashboard.yoctoproject.org/releases.json to get current releases information"
+ )
+ parser.add_argument("--get-latest-branch",
+ help="Print current latest branch and exit",
+ action="store_true",
+ default=False)
+ args = parser.parse_args()
+
+ if args.get_latest_branch:
+ print(ACTIVERELEASES[0])
+ sys.exit(0)
+
+def get_current_version():
+ # Test tags exist and inform the user to fetch if not
+ try:
+ subprocess.run(["git", "show", f"{LTSSERIES[-1]}.0"],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
+ except subprocess.CalledProcessError:
+ sys.exit("Please run 'git fetch --tags' before building the documentation")
+
+ # Try and figure out what we are
+ tags = subprocess.run(["git", "tag", "--points-at", "HEAD"],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True).stdout
+ for t in tags.split():
+ if re.match(BB_RELEASE_TAG_RE, t):
+ return t
+
+ # We're floating on a branch
+ branch = subprocess.run(["git", "branch", "--show-current"],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True).stdout.strip()
+
+ if branch == "" or branch not in list(YOCTO_MAPPING.keys()) + ["master", "master-next"]:
+ # We're not on a known release branch so we have to guess. Compare the
+ # numbers of commits from each release branch and assume the smallest
+ # number of commits is the one we're based off
+ possible_branch = None
+ branch_count = 0
+ for b in itertools.chain(YOCTO_MAPPING.keys(), ["master"]):
+ result = subprocess.run(["git", "log", "--format=oneline", "HEAD..origin/" + b],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True)
+ if result.returncode == 0:
+ count = result.stdout.count('\n')
+ if not possible_branch or count < branch_count:
+ print("Branch %s has count %s" % (b, count))
+ possible_branch = b
+ branch_count = count
+ if possible_branch:
+ branch = possible_branch
+ else:
+ branch = "master"
+ print("Nearest release branch estimated to be %s" % branch)
+
+ if branch == "master":
+ return "dev"
+
+ if branch == "master-next":
+ return "next"
+
+ ourversion = branch
+ head_commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True).stdout.strip()
+ branch_commit = subprocess.run(["git", "rev-parse", "--short", branch],
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True).stdout.strip()
+ if head_commit != branch_commit:
+ ourversion += f" ({head_commit})"
+
+ return ourversion
+
+def write_switchers_js(js_in, js_out, current_version):
+ with open(js_in, "r") as r, open(js_out, "w") as w:
+ lines = r.readlines()
+ for line in lines:
+ if "VERSIONS_PLACEHOLDER" in line:
+ if current_version != "dev":
+ w.write(" 'dev': 'Unstable (dev)',\n")
+ for series in ACTIVERELEASES:
+ w.write(f" '{series}': '{series} ({YOCTO_MAPPING[series]})',\n")
+ else:
+ w.write(line)
+ print("switchers.js generated from switchers.js.in")
+
+def _release_section(series_version: str, codename: str, bitbake_version: str) -> str:
+ """
+ Helper function to generate a release section, as:
+
+ ********************
+ Release Series xxxxx
+ ********************
+
+ - <link to manual>
+ """
+ section_length = len(series_version) + len(codename) + 18
+ return textwrap.dedent(
+ f"""\
+ {'*' * section_length}
+ Release Series {series_version} ({codename})
+ {'*' * section_length}
+
+ - :yocto_docs:`BitBake {bitbake_version} User Manual </bitbake/{bitbake_version}/>`
+
+ """)
+
+def write_releases_rst(releases_rst_out: str):
+ """
+ Generates the releases.rst file automatically, based on what is found
+ in the releases.json file.
+ """
+ with open(releases_rst_out, "w") as f:
+ f.write(textwrap.dedent("""\
+ .. SPDX-License-Identifier: CC-BY-2.5
+
+ =================================
+ BitBake Supported Release Manuals
+ =================================
+
+ """))
+
+ for release in RELEASES_FROM_JSON:
+ if release["status"] == "Active Development":
+ continue
+
+ if not release["bitbake_version"]:
+ continue
+
+ if release["series"] == "current":
+ f.write(_release_section(
+ release["series_version"],
+ release["release_codename"],
+ release["bitbake_version"]))
+
+ f.write(textwrap.dedent("""\
+ ================================
+ BitBake Outdated Release Manuals
+ ================================
+
+ """))
+
+ for release in RELEASES_FROM_JSON:
+ if not release["series"] == "previous":
+ continue
+
+ if not release["bitbake_version"]:
+ continue
+
+ f.write(_release_section(
+ release["series_version"],
+ release["release_codename"],
+ release["bitbake_version"]))
+
+ # old legacy links, which cannot be auto-generated
+ f.write(textwrap.dedent(
+ """\
+ - :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.1.3 BitBake User Manual </3.1.3/bitbake-user-manual/bitbake-user-manual.html>`
+
+ *************************
+ Release Series 3.0 (Zeus)
+ *************************
+
+ - :yocto_docs:`3.0 BitBake User Manual </3.0/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.0.1 BitBake User Manual </3.0.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.0.2 BitBake User Manual </3.0.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.0.3 BitBake User Manual </3.0.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`3.0.4 BitBake User Manual </3.0.4/bitbake-user-manual/bitbake-user-manual.html>`
+
+ ****************************
+ Release Series 2.7 (Warrior)
+ ****************************
+
+ - :yocto_docs:`2.7 BitBake User Manual </2.7/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.7.1 BitBake User Manual </2.7.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.7.2 BitBake User Manual </2.7.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.7.3 BitBake User Manual </2.7.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.7.4 BitBake User Manual </2.7.4/bitbake-user-manual/bitbake-user-manual.html>`
+
+ *************************
+ Release Series 2.6 (Thud)
+ *************************
+
+ - :yocto_docs:`2.6 BitBake User Manual </2.6/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.6.1 BitBake User Manual </2.6.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.6.2 BitBake User Manual </2.6.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.6.3 BitBake User Manual </2.6.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.6.4 BitBake User Manual </2.6.4/bitbake-user-manual/bitbake-user-manual.html>`
+
+ *************************
+ Release Series 2.5 (Sumo)
+ *************************
+
+ - :yocto_docs:`2.5 Documentation </2.5>`
+ - :yocto_docs:`2.5.1 Documentation </2.5.1>`
+ - :yocto_docs:`2.5.2 Documentation </2.5.2>`
+ - :yocto_docs:`2.5.3 Documentation </2.5.3>`
+
+ **************************
+ Release Series 2.4 (Rocko)
+ **************************
+
+ - :yocto_docs:`2.4 BitBake User Manual </2.4/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.4.1 BitBake User Manual </2.4.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.4.2 BitBake User Manual </2.4.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.4.3 BitBake User Manual </2.4.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.4.4 BitBake User Manual </2.4.4/bitbake-user-manual/bitbake-user-manual.html>`
+
+ *************************
+ Release Series 2.3 (Pyro)
+ *************************
+
+ - :yocto_docs:`2.3 BitBake User Manual </2.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.3.1 BitBake User Manual </2.3.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.3.2 BitBake User Manual </2.3.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.3.3 BitBake User Manual </2.3.3/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.3.4 BitBake User Manual </2.3.4/bitbake-user-manual/bitbake-user-manual.html>`
+
+ **************************
+ Release Series 2.2 (Morty)
+ **************************
+
+ - :yocto_docs:`2.2 BitBake User Manual </2.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.2.1 BitBake User Manual </2.2.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.2.2 BitBake User Manual </2.2.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.2.3 BitBake User Manual </2.2.3/bitbake-user-manual/bitbake-user-manual.html>`
+
+ ****************************
+ Release Series 2.1 (Krogoth)
+ ****************************
+
+ - :yocto_docs:`2.1 BitBake User Manual </2.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.1.1 BitBake User Manual </2.1.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.1.2 BitBake User Manual </2.1.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.1.3 BitBake User Manual </2.1.3/bitbake-user-manual/bitbake-user-manual.html>`
+
+ ***************************
+ Release Series 2.0 (Jethro)
+ ***************************
+
+ - :yocto_docs:`1.9 BitBake User Manual </1.9/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.0 BitBake User Manual </2.0/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.0.1 BitBake User Manual </2.0.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.0.2 BitBake User Manual </2.0.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`2.0.3 BitBake User Manual </2.0.3/bitbake-user-manual/bitbake-user-manual.html>`
+
+ *************************
+ Release Series 1.8 (Fido)
+ *************************
+
+ - :yocto_docs:`1.8 BitBake User Manual </1.8/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.8.1 BitBake User Manual </1.8.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.8.2 BitBake User Manual </1.8.2/bitbake-user-manual/bitbake-user-manual.html>`
+
+ **************************
+ Release Series 1.7 (Dizzy)
+ **************************
+
+ - :yocto_docs:`1.7 BitBake User Manual </1.7/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.7.1 BitBake User Manual </1.7.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.7.2 BitBake User Manual </1.7.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.7.3 BitBake User Manual </1.7.3/bitbake-user-manual/bitbake-user-manual.html>`
+
+ **************************
+ Release Series 1.6 (Daisy)
+ **************************
+
+ - :yocto_docs:`1.6 BitBake User Manual </1.6/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.6.1 BitBake User Manual </1.6.1/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.6.2 BitBake User Manual </1.6.2/bitbake-user-manual/bitbake-user-manual.html>`
+ - :yocto_docs:`1.6.3 BitBake User Manual </1.6.3/bitbake-user-manual/bitbake-user-manual.html>`
+ """))
+
+if __name__ == "__main__":
+ main()
diff --git a/bitbake/doc/sphinx-static/switchers.js b/bitbake/doc/sphinx-static/switchers.js
deleted file mode 100644
index 32113cfa..00000000
--- a/bitbake/doc/sphinx-static/switchers.js
+++ /dev/null
@@ -1,233 +0,0 @@
-(function() {
- 'use strict';
-
- var all_versions = {
- 'dev': 'dev (3.2)',
- '3.1.2': '3.1.2',
- '3.0.3': '3.0.3',
- '2.7.4': '2.7.4',
- };
-
- var all_doctypes = {
- 'single': 'Individual Webpages',
- 'mega': "All-in-one 'Mega' Manual",
- };
-
- // Simple version comparision
- // Return 1 if a > b
- // Return -1 if a < b
- // Return 0 if a == b
- function ver_compare(a, b) {
- if (a == "dev") {
- return 1;
- }
-
- if (a === b) {
- return 0;
- }
-
- var a_components = a.split(".");
- var b_components = b.split(".");
-
- var len = Math.min(a_components.length, b_components.length);
-
- // loop while the components are equal
- for (var i = 0; i < len; i++) {
- // A bigger than B
- if (parseInt(a_components[i]) > parseInt(b_components[i])) {
- return 1;
- }
-
- // B bigger than A
- if (parseInt(a_components[i]) < parseInt(b_components[i])) {
- return -1;
- }
- }
-
- // If one's a prefix of the other, the longer one is greater.
- if (a_components.length > b_components.length) {
- return 1;
- }
-
- if (a_components.length < b_components.length) {
- return -1;
- }
-
- // Otherwise they are the same.
- return 0;
- }
-
- function build_version_select(current_series, current_version) {
- var buf = ['<select>'];
-
- $.each(all_versions, function(version, title) {
- var series = version.substr(0, 3);
- if (series == current_series) {
- if (version == current_version)
- buf.push('<option value="' + version + '" selected="selected">' + title + '</option>');
- else
- buf.push('<option value="' + version + '">' + title + '</option>');
-
- if (version != current_version)
- buf.push('<option value="' + current_version + '" selected="selected">' + current_version + '</option>');
- } else {
- buf.push('<option value="' + version + '">' + title + '</option>');
- }
- });
-
- buf.push('</select>');
- return buf.join('');
- }
-
- function build_doctype_select(current_doctype) {
- var buf = ['<select>'];
-
- $.each(all_doctypes, function(doctype, title) {
- if (doctype == current_doctype)
- buf.push('<option value="' + doctype + '" selected="selected">' +
- all_doctypes[current_doctype] + '</option>');
- else
- buf.push('<option value="' + doctype + '">' + title + '</option>');
- });
- if (!(current_doctype in all_doctypes)) {
- // In case we're browsing a doctype that is not yet in all_doctypes.
- buf.push('<option value="' + current_doctype + '" selected="selected">' +
- current_doctype + '</option>');
- all_doctypes[current_doctype] = current_doctype;
- }
- buf.push('</select>');
- return buf.join('');
- }
-
- function navigate_to_first_existing(urls) {
- // Navigate to the first existing URL in urls.
- var url = urls.shift();
-
- // Web browsers won't redirect file:// urls to file urls using ajax but
- // its useful for local testing
- if (url.startsWith("file://")) {
- window.location.href = url;
- return;
- }
-
- if (urls.length == 0) {
- window.location.href = url;
- return;
- }
- $.ajax({
- url: url,
- success: function() {
- window.location.href = url;
- },
- error: function() {
- navigate_to_first_existing(urls);
- }
- });
- }
-
- function get_docroot_url() {
- var url = window.location.href;
- var root = DOCUMENTATION_OPTIONS.URL_ROOT;
-
- var urlarray = url.split('/');
- // Trim off anything after '/'
- urlarray.pop();
- var depth = (root.match(/\.\.\//g) || []).length;
- for (var i = 0; i < depth; i++) {
- urlarray.pop();
- }
-
- return urlarray.join('/') + '/';
- }
-
- function on_version_switch() {
- var selected_version = $(this).children('option:selected').attr('value');
- var url = window.location.href;
- var current_version = DOCUMENTATION_OPTIONS.VERSION;
- var docroot = get_docroot_url()
-
- var new_versionpath = selected_version + '/';
- if (selected_version == "dev")
- new_versionpath = '';
-
- // dev versions have no version prefix
- if (current_version == "dev") {
- var new_url = docroot + new_versionpath + url.replace(docroot, "");
- var fallback_url = docroot + new_versionpath;
- } else {
- var new_url = url.replace('/' + current_version + '/', '/' + new_versionpath);
- var fallback_url = new_url.replace(url.replace(docroot, ""), "");
- }
-
- console.log(get_docroot_url())
- console.log(url + " to url " + new_url);
- console.log(url + " to fallback " + fallback_url);
-
- if (new_url != url) {
- navigate_to_first_existing([
- new_url,
- fallback_url,
- 'https://www.yoctoproject.org/docs/',
- ]);
- }
- }
-
- function on_doctype_switch() {
- var selected_doctype = $(this).children('option:selected').attr('value');
- var url = window.location.href;
- if (selected_doctype == 'mega') {
- var docroot = get_docroot_url()
- var current_version = DOCUMENTATION_OPTIONS.VERSION;
- // Assume manuals before 3.2 are using old docbook mega-manual
- if (ver_compare(current_version, "3.2") < 0) {
- var new_url = docroot + "mega-manual/mega-manual.html";
- } else {
- var new_url = docroot + "singleindex.html";
- }
- } else {
- var new_url = url.replace("singleindex.html", "index.html")
- }
-
- if (new_url != url) {
- navigate_to_first_existing([
- new_url,
- 'https://www.yoctoproject.org/docs/',
- ]);
- }
- }
-
- // Returns the current doctype based upon the url
- function doctype_segment_from_url(url) {
- if (url.includes("singleindex") || url.includes("mega-manual"))
- return "mega";
- return "single";
- }
-
- $(document).ready(function() {
- var release = DOCUMENTATION_OPTIONS.VERSION;
- var current_doctype = doctype_segment_from_url(window.location.href);
- var current_series = release.substr(0, 3);
- var version_select = build_version_select(current_series, release);
-
- $('.version_switcher_placeholder').html(version_select);
- $('.version_switcher_placeholder select').bind('change', on_version_switch);
-
- var doctype_select = build_doctype_select(current_doctype);
-
- $('.doctype_switcher_placeholder').html(doctype_select);
- $('.doctype_switcher_placeholder select').bind('change', on_doctype_switch);
-
- if (ver_compare(release, "3.1") < 0) {
- $('#outdated-warning').html('Version ' + release + ' of the project is now considered obsolete, please select and use a more recent version');
- $('#outdated-warning').css('padding', '.5em');
- } else if (release != "dev") {
- $.each(all_versions, function(version, title) {
- var series = version.substr(0, 3);
- if (series == current_series && version != release) {
- $('#outdated-warning').html('This document is for outdated version ' + release + ', you should select the latest release version in this series, ' + version + '.');
- $('#outdated-warning').css('padding', '.5em');
- }
- });
- }
- });
-})();
diff --git a/bitbake/doc/sphinx-static/switchers.js.in b/bitbake/doc/sphinx-static/switchers.js.in
new file mode 100644
index 00000000..0b209e95
--- /dev/null
+++ b/bitbake/doc/sphinx-static/switchers.js.in
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: MIT
+(function () {
+ "use strict";
+
+ var all_versions = {
+ VERSIONS_PLACEHOLDER,
+ };
+
+ var all_doctypes = {
+ single: "Individual Webpages",
+ mega: "All-in-one 'Mega' Manual",
+ };
+
+ // Simple version comparision
+ // Return 1 if a > b
+ // Return -1 if a < b
+ // Return 0 if a == b
+ function ver_compare(a, b) {
+ if (a == "dev") {
+ return 1;
+ }
+
+ if (a === b) {
+ return 0;
+ }
+
+ var a_components = a.split(".");
+ var b_components = b.split(".");
+
+ var len = Math.min(a_components.length, b_components.length);
+
+ // loop while the components are equal
+ for (var i = 0; i < len; i++) {
+ // A bigger than B
+ if (parseInt(a_components[i]) > parseInt(b_components[i])) {
+ return 1;
+ }
+
+ // B bigger than A
+ if (parseInt(a_components[i]) < parseInt(b_components[i])) {
+ return -1;
+ }
+ }
+
+ // If one's a prefix of the other, the longer one is greater.
+ if (a_components.length > b_components.length) {
+ return 1;
+ }
+
+ if (a_components.length < b_components.length) {
+ return -1;
+ }
+
+ // Otherwise they are the same.
+ return 0;
+ }
+
+ function build_version_select(current_version) {
+ var buf = ["<select>"];
+
+ $.each(all_versions, function (version, title) {
+ if (current_version != version) {
+ buf.push(
+ '<option value="' +
+ version +
+ '" selected="selected">' +
+ title +
+ "</option>",
+ );
+ }
+ });
+
+ var current_title = current_version;
+ if (current_version in all_versions) {
+ current_title = all_versions[current_version];
+ }
+ buf.push(
+ '<option value="' +
+ current_version +
+ '" selected="selected">' +
+ current_title +
+ "</option>",
+ );
+
+ buf.push("</select>");
+ return buf.join("");
+ }
+
+ function build_doctype_select(current_doctype) {
+ var buf = ["<select>"];
+
+ $.each(all_doctypes, function (doctype, title) {
+ if (doctype == current_doctype)
+ buf.push(
+ '<option value="' +
+ doctype +
+ '" selected="selected">' +
+ all_doctypes[current_doctype] +
+ "</option>",
+ );
+ else buf.push('<option value="' + doctype + '">' + title + "</option>");
+ });
+ if (!(current_doctype in all_doctypes)) {
+ // In case we're browsing a doctype that is not yet in all_doctypes.
+ buf.push(
+ '<option value="' +
+ current_doctype +
+ '" selected="selected">' +
+ current_doctype +
+ "</option>",
+ );
+ all_doctypes[current_doctype] = current_doctype;
+ }
+ buf.push("</select>");
+ return buf.join("");
+ }
+
+ function navigate_to_first_existing(urls) {
+ // Navigate to the first existing URL in urls.
+ var url = urls.shift();
+
+ // Web browsers won't redirect file:// urls to file urls using ajax but
+ // its useful for local testing
+ if (url.startsWith("file://")) {
+ window.location.href = url;
+ return;
+ }
+
+ if (urls.length == 0) {
+ window.location.href = url;
+ return;
+ }
+ $.ajax({
+ url: url,
+ success: function () {
+ window.location.href = url;
+ },
+ error: function () {
+ navigate_to_first_existing(urls);
+ },
+ });
+ }
+
+ function get_docroot_url() {
+ var url = window.location.href;
+ // Try to get the variable from documentation_options.js
+ var root = DOCUMENTATION_OPTIONS.URL_ROOT;
+ if (root == null) {
+ // In recent versions of Sphinx, URL_ROOT was removed from
+ // documentation_options.js, so get it like searchtools.js does.
+ root = document.documentElement.dataset.content_root;
+ }
+
+ var urlarray = url.split("/");
+ // Trim off anything after '/'
+ urlarray.pop();
+ var depth = (root.match(/\.\.\//g) || []).length;
+ for (var i = 0; i < depth; i++) {
+ urlarray.pop();
+ }
+
+ return urlarray.join("/") + "/";
+ }
+
+ function on_version_switch() {
+ var selected_version = $(this).children("option:selected").attr("value");
+ var url = window.location.href;
+ var current_version = DOCUMENTATION_OPTIONS.VERSION;
+ var docroot = get_docroot_url();
+
+ var new_versionpath = selected_version + "/";
+
+ // latest tag is also the default page (without version information)
+ if (docroot.endsWith("dev/")) {
+ var new_url = url.replace("/dev/", "/" + new_versionpath);
+ var fallback_url = new_url.replace(url.replace(docroot, ""), "");
+ } else if (docroot.endsWith(current_version + "/") == false) {
+ var new_url = docroot + new_versionpath + url.replace(docroot, "");
+ var fallback_url = docroot + new_versionpath;
+ } else {
+ var new_url = url.replace(
+ "/" + current_version + "/",
+ "/" + new_versionpath,
+ );
+ var fallback_url = new_url.replace(url.replace(docroot, ""), "");
+ }
+
+ console.log(url + " to url " + new_url);
+ console.log(url + " to fallback " + fallback_url);
+
+ if (new_url != url) {
+ navigate_to_first_existing([
+ new_url,
+ fallback_url,
+ "https://www.yoctoproject.org/bitbake/",
+ ]);
+ }
+ }
+
+ function on_doctype_switch() {
+ var selected_doctype = $(this).children("option:selected").attr("value");
+ var url = window.location.href;
+ if (selected_doctype == "mega") {
+ var docroot = get_docroot_url();
+ var current_version = DOCUMENTATION_OPTIONS.VERSION;
+ var new_url = docroot + "singleindex.html";
+ } else {
+ var new_url = url.replace("singleindex.html", "index.html");
+ }
+
+ if (new_url != url) {
+ navigate_to_first_existing([
+ new_url,
+ "https://www.yoctoproject.org/docs/",
+ ]);
+ }
+ }
+
+ // Returns the current doctype based upon the url
+ function doctype_segment_from_url(url) {
+ if (url.includes("singleindex") || url.includes("mega-manual"))
+ return "mega";
+ return "single";
+ }
+
+ $(document).ready(function () {
+ var release = DOCUMENTATION_OPTIONS.VERSION;
+ var current_doctype = doctype_segment_from_url(window.location.href);
+ var current_series = release.substr(0, 3);
+ var version_select = build_version_select(release);
+
+ $(".version_switcher_placeholder").html(version_select);
+ $(".version_switcher_placeholder select").bind("change", on_version_switch);
+
+ var doctype_select = build_doctype_select(current_doctype);
+
+ $(".doctype_switcher_placeholder").html(doctype_select);
+ $(".doctype_switcher_placeholder select").bind("change", on_doctype_switch);
+
+ // if release = "X.Y (<abbrev hash>)", remove the "(<abbrev hash>)" so that only X.Y is compared
+ release = release.split(" ")[0];
+ if (!(["dev", "next"].includes(release)) && !(release in all_versions)) {
+ $("#outdated-warning").html(
+ "Version " + release + " of the project is now considered obsolete, please select and use a more recent version",
+ );
+ $("#outdated-warning").css("padding", ".5em");
+ }
+ });
+})();
diff --git a/bitbake/doc/sphinx-static/theme_overrides.css b/bitbake/doc/sphinx-static/theme_overrides.css
index e362677a..3a303c8e 100644
--- a/bitbake/doc/sphinx-static/theme_overrides.css
+++ b/bitbake/doc/sphinx-static/theme_overrides.css
@@ -99,14 +99,19 @@ em {
[alt='Permalink'] { color: #eee; }
[alt='Permalink']:hover { color: black; }
+.literal-block {
+ background: #f8f8f8;
+}
+
@media screen {
- /* content column
- *
- * RTD theme's default is 800px as max width for the content, but we have
- * tables with tons of columns, which need the full width of the view-port.
- */
+ .wy-nav-content {
+ max-width: 1000px;
+ background: #fcfcfc;
+ }
- .wy-nav-content{max-width: none; }
+ .wy-nav-content-wrap {
+ background: #efefef;
+ }
/* inline literal: drop the borderbox, padding and red color */
code, .rst-content tt, .rst-content code {
diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py
index 76bc08a3..4af03c54 100644
--- a/bitbake/lib/bb/COW.py
+++ b/bitbake/lib/bb/COW.py
@@ -36,8 +36,9 @@ class COWDictMeta(COWMeta):
__marker__ = tuple()
def __str__(cls):
- # FIXME: I have magic numbers!
- return "<COWDict Level: %i Current Keys: %i>" % (cls.__count__, len(cls.__dict__) - 3)
+ ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"])
+ keys = set(cls.__dict__.keys()) - ignored_keys
+ return "<COWDict Level: %i Current Keys: %i>" % (cls.__count__, len(keys))
__repr__ = __str__
@@ -161,8 +162,9 @@ class COWDictMeta(COWMeta):
class COWSetMeta(COWDictMeta):
def __str__(cls):
- # FIXME: I have magic numbers!
- return "<COWSet Level: %i Current Keys: %i>" % (cls.__count__, len(cls.__dict__) - 3)
+ ignored_keys = set(["__count__", "__doc__", "__module__", "__firstlineno__", "__static_attributes__"])
+ keys = set(cls.__dict__.keys()) - ignored_keys
+ return "<COWSet Level: %i Current Keys: %i>" % (cls.__count__, len(keys))
__repr__ = __str__
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index 7d9bc147..01c73f1b 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -9,12 +9,11 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-__version__ = "2.8.0"
+__version__ = "2.20.0"
import sys
-# It was 3.8.0 originally but set to 3.7.3 for Debian Buster
-if sys.version_info < (3, 7, 3):
- raise RuntimeError("Sorry, python 3.7.3 or later is required for this version of bitbake")
+if sys.version_info < (3, 9, 0):
+ raise RuntimeError("Sorry, python 3.9.0 or later is required for this version of bitbake")
if sys.version_info < (3, 10, 0):
# With python 3.8 and 3.9, we see errors of "libgcc_s.so.1 must be installed for pthread_cancel to work"
@@ -37,6 +36,35 @@ class BBHandledException(Exception):
import os
import logging
+from collections import namedtuple
+import multiprocessing as mp
+
+# Python 3.14 changes the default multiprocessing context from "fork" to
+# "forkserver". However, bitbake heavily relies on "fork" behavior to
+# efficiently pass data to the child processes. Places that need this should do:
+# from bb import multiprocessing
+# in place of
+# import multiprocessing
+
+class MultiprocessingContext(object):
+ """
+ Multiprocessing proxy object that uses the "fork" context for a property if
+ available, otherwise goes to the main multiprocessing module. This allows
+ it to be a drop-in replacement for the multiprocessing module, but use the
+ fork context
+ """
+ def __init__(self):
+ super().__setattr__("_ctx", mp.get_context("fork"))
+
+ def __getattr__(self, name):
+ if hasattr(self._ctx, name):
+ return getattr(self._ctx, name)
+ return getattr(mp, name)
+
+ def __setattr__(self, name, value):
+ raise AttributeError(f"Unable to set attribute {name}")
+
+multiprocessing = MultiprocessingContext()
class NullHandler(logging.Handler):
@@ -62,6 +90,7 @@ class BBLoggerMixin(object):
return self.bbdebug(3, *args, **kwargs)
def bbdebug(self, level, msg, *args, **kwargs):
+ import bb.event
loglevel = logging.DEBUG - level + 1
if not bb.event.worker_pid:
if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
@@ -104,26 +133,6 @@ class BBLoggerAdapter(logging.LoggerAdapter, BBLoggerMixin):
self.setup_bblogger(logger.name)
super().__init__(logger, *args, **kwargs)
- if sys.version_info < (3, 6):
- # These properties were added in Python 3.6. Add them in older versions
- # for compatibility
- @property
- def manager(self):
- return self.logger.manager
-
- @manager.setter
- def manager(self, value):
- self.logger.manager = value
-
- @property
- def name(self):
- return self.logger.name
-
- def __repr__(self):
- logger = self.logger
- level = logger.getLevelName(logger.getEffectiveLevel())
- return '<%s %s (%s)>' % (self.__class__.__name__, logger.name, level)
-
logging.LoggerAdapter = BBLoggerAdapter
logger = logging.getLogger("BitBake")
@@ -144,14 +153,39 @@ class PrefixLoggerAdapter(logging.LoggerAdapter):
# can result in construction of the various loggers.
import bb.msg
-from bb import fetch2 as fetch
-sys.modules['bb.fetch'] = sys.modules['bb.fetch2']
+# There's lots of code (in OE and other layers) that accesses bb.fetch2 without
+# first importing it. It used to work because fetch2 was added to the bb module in
+# bb/__init__.py, as part of installing the fetch = fetch2 shim. But now that
+# the new fetch2 shim is an actual module, this will break. So, provide lazy
+# access to bb.fetch2 via __getattr__ so layers keep working.
+def __getattr__(name):
+ if name == "fetch2":
+ import bb.fetch2 as fetch2
+ return fetch2
+
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# Messaging convenience functions
def plain(*args):
+ """
+ Prints a message at "plain" level (higher level than a ``bb.note()``).
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.plain(''.join(args))
def debug(lvl, *args):
+ """
+ Prints a debug message.
+
+ Arguments:
+
+ - ``lvl``: debug level. Higher value increases the debug level
+ (determined by ``bitbake -D``).
+ - ``args``: one or more strings to print.
+ """
if isinstance(lvl, str):
mainlogger.warning("Passed invalid debug level '%s' to bb.debug", lvl)
args = (lvl,) + args
@@ -159,33 +193,81 @@ def debug(lvl, *args):
mainlogger.bbdebug(lvl, ''.join(args))
def note(*args):
+ """
+ Prints a message at "note" level.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.info(''.join(args))
-#
-# A higher prioity note which will show on the console but isn't a warning
-#
-# Something is happening the user should be aware of but they probably did
-# something to make it happen
-#
def verbnote(*args):
+ """
+ A higher priority note which will show on the console but isn't a warning.
+
+ Use in contexts when something is happening the user should be aware of but
+ they probably did something to make it happen.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.verbnote(''.join(args))
#
# Warnings - things the user likely needs to pay attention to and fix
#
def warn(*args):
+ """
+ Prints a warning message.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.warning(''.join(args))
def warnonce(*args):
+ """
+ Prints a warning message like ``bb.warn()``, but only prints the message
+ once.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.warnonce(''.join(args))
def error(*args, **kwargs):
+ """
+ Prints an error message.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.error(''.join(args), extra=kwargs)
def erroronce(*args):
+ """
+ Prints an error message like ``bb.error()``, but only prints the message
+ once.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.erroronce(''.join(args))
def fatal(*args, **kwargs):
+ """
+ Prints an error message and stops the BitBake execution.
+
+ Arguments:
+
+ - ``args``: one or more strings to print.
+ """
mainlogger.critical(''.join(args), extra=kwargs)
raise BBHandledException()
@@ -214,7 +296,6 @@ def deprecated(func, name=None, advice=""):
# For compatibility
def deprecate_import(current, modulename, fromlist, renames = None):
"""Import objects from one module into another, wrapping them with a DeprecationWarning"""
- import sys
module = __import__(modulename, fromlist = fromlist)
for position, objname in enumerate(fromlist):
@@ -228,3 +309,14 @@ def deprecate_import(current, modulename, fromlist, renames = None):
setattr(sys.modules[current], newname, newobj)
+TaskData = namedtuple("TaskData", [
+ "pn",
+ "taskname",
+ "fn",
+ "deps",
+ "provides",
+ "taskhash",
+ "unihash",
+ "hashfn",
+ "taskhash_deps",
+])
diff --git a/bitbake/lib/bs4/COPYING.txt b/bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE
similarity index 80%
rename from bitbake/lib/bs4/COPYING.txt
rename to bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE
index d668d13f..08e3a9cf 100644
--- a/bitbake/lib/bs4/COPYING.txt
+++ b/bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE
@@ -1,6 +1,6 @@
Beautiful Soup is made available under the MIT license:
- Copyright (c) 2004-2012 Leonard Richardson
+ Copyright (c) Leonard Richardson
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
@@ -20,7 +20,12 @@ Beautiful Soup is made available under the MIT license:
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- SOFTWARE, DAMMIT.
+ SOFTWARE.
Beautiful Soup incorporates code from the html5lib library, which is
-also made available under the MIT license.
+also made available under the MIT license. Copyright (c) James Graham
+and other contributors
+
+Beautiful Soup has an optional dependency on the soupsieve library,
+which is also made available under the MIT license. Copyright (c)
+Isaac Muse
diff --git a/bitbake/lib/bb/_vendor/bs4/__init__.py b/bitbake/lib/bb/_vendor/bs4/__init__.py
new file mode 100644
index 00000000..58079b9e
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/__init__.py
@@ -0,0 +1,1177 @@
+"""Beautiful Soup Elixir and Tonic - "The Screen-Scraper's Friend".
+
+http://www.crummy.com/software/BeautifulSoup/
+
+Beautiful Soup uses a pluggable XML or HTML parser to parse a
+(possibly invalid) document into a tree representation. Beautiful Soup
+provides methods and Pythonic idioms that make it easy to navigate,
+search, and modify the parse tree.
+
+Beautiful Soup works with Python 3.7 and up. It works better if lxml
+and/or html5lib is installed, but they are not required.
+
+For more than you ever wanted to know about Beautiful Soup, see the
+documentation: http://www.crummy.com/software/BeautifulSoup/bs4/doc/
+"""
+
+__author__ = "Leonard Richardson (leonardr@segfault.org)"
+__version__ = "4.15.0"
+__copyright__ = "Copyright (c) 2004-2026 Leonard Richardson"
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+__all__ = [
+ "AttributeResemblesVariableWarning",
+ "BeautifulSoup",
+ "Comment",
+ "Declaration",
+ "ProcessingInstruction",
+ "ResultSet",
+ "CSS",
+ "Script",
+ "Stylesheet",
+ "Tag",
+ "TemplateString",
+ "ElementFilter",
+ "UnicodeDammit",
+ "CData",
+ "Doctype",
+
+ # Exceptions
+ "FeatureNotFound",
+ "ParserRejectedMarkup",
+ "StopParsing",
+
+ # Warnings
+ "AttributeResemblesVariableWarning",
+ "GuessedAtParserWarning",
+ "MarkupResemblesLocatorWarning",
+ "UnusualUsageWarning",
+ "XMLParsedAsHTMLWarning",
+]
+
+from collections import Counter
+import io
+import sys
+import warnings
+
+# The very first thing we do is give a useful error if someone is
+# running this code under Python 2.
+if sys.version_info.major < 3:
+ raise ImportError(
+ "You are trying to use a Python 3-specific version of Beautiful Soup under Python 2. This will not work. The final version of Beautiful Soup to support Python 2 was 4.9.3."
+ )
+
+from .builder import (
+ builder_registry,
+ TreeBuilder,
+)
+from .builder._htmlparser import HTMLParserTreeBuilder
+from .dammit import UnicodeDammit
+from .css import CSS
+from ._deprecation import (
+ _deprecated,
+)
+from .element import (
+ CData,
+ Comment,
+ DEFAULT_OUTPUT_ENCODING,
+ Declaration,
+ Doctype,
+ NavigableString,
+ PageElement,
+ ProcessingInstruction,
+ PYTHON_SPECIFIC_ENCODINGS,
+ ResultSet,
+ Script,
+ Stylesheet,
+ Tag,
+ TemplateString,
+)
+from .formatter import Formatter
+from .filter import (
+ ElementFilter,
+ SoupStrainer,
+)
+from typing import (
+ Any,
+ cast,
+ Counter as CounterType,
+ Dict,
+ Iterator,
+ List,
+ Sequence,
+ Sized,
+ Optional,
+ Type,
+ Union,
+)
+
+from bb._vendor.bs4._typing import (
+ _Encoding,
+ _Encodings,
+ _IncomingMarkup,
+ _InsertableElement,
+ _RawAttributeValue,
+ _RawAttributeValues,
+ _RawMarkup,
+)
+
+# Import all warnings and exceptions into the main package.
+from bb._vendor.bs4.exceptions import (
+ FeatureNotFound,
+ ParserRejectedMarkup,
+ StopParsing,
+)
+from bb._vendor.bs4._warnings import (
+ AttributeResemblesVariableWarning,
+ GuessedAtParserWarning,
+ MarkupResemblesLocatorWarning,
+ UnusualUsageWarning,
+ XMLParsedAsHTMLWarning,
+)
+
+
+class BeautifulSoup(Tag):
+ """A data structure representing a parsed HTML or XML document.
+
+ Most of the methods you'll call on a BeautifulSoup object are inherited from
+ PageElement or Tag.
+
+ Internally, this class defines the basic interface called by the
+ tree builders when converting an HTML/XML document into a data
+ structure. The interface abstracts away the differences between
+ parsers. To write a new tree builder, you'll need to understand
+ these methods as a whole.
+
+ These methods will be called by the BeautifulSoup constructor:
+ * reset()
+ * feed(markup)
+
+ The tree builder may call these methods from its feed() implementation:
+ * handle_starttag(name, attrs) # See note about return value
+ * handle_endtag(name)
+ * handle_data(data) # Appends to the current data node
+ * endData(containerClass) # Ends the current data node
+
+ No matter how complicated the underlying parser is, you should be
+ able to build a tree using 'start tag' events, 'end tag' events,
+ 'data' events, and "done with data" events.
+
+ If you encounter an empty-element tag (aka a self-closing tag,
+ like HTML's <br> tag), call handle_starttag and then
+ handle_endtag.
+ """
+
+ #: Since `BeautifulSoup` subclasses `Tag`, it's possible to treat it as
+ #: a `Tag` with a `Tag.name`. Hoever, this name makes it clear the
+ #: `BeautifulSoup` object isn't a real markup tag.
+ ROOT_TAG_NAME: str = "[document]"
+
+ #: If the end-user gives no indication which tree builder they
+ #: want, look for one with these features.
+ DEFAULT_BUILDER_FEATURES: Sequence[str] = ["html", "fast"]
+
+ #: A string containing all ASCII whitespace characters, used in
+ #: during parsing to detect data chunks that seem 'empty'.
+ ASCII_SPACES: str = "\x20\x0a\x09\x0c\x0d"
+
+ # FUTURE PYTHON:
+ element_classes: Dict[Type[PageElement], Type[PageElement]] #: :meta private:
+ builder: TreeBuilder #: :meta private:
+ is_xml: bool
+ known_xml: Optional[bool]
+ parse_only: Optional[SoupStrainer] #: :meta private:
+
+ # These members are only used while parsing markup.
+ markup: Optional[_RawMarkup] #: :meta private:
+ current_data: List[str] #: :meta private:
+ currentTag: Optional[Tag] #: :meta private:
+ tagStack: List[Tag] #: :meta private:
+ open_tag_counter: CounterType[str] #: :meta private:
+ preserve_whitespace_tag_stack: List[Tag] #: :meta private:
+ string_container_stack: List[Tag] #: :meta private:
+ _most_recent_element: Optional[PageElement] #: :meta private:
+
+ #: Beautiful Soup's best guess as to the character encoding of the
+ #: original document.
+ original_encoding: Optional[_Encoding]
+
+ #: The character encoding, if any, that was explicitly defined
+ #: in the original document. This may or may not match
+ #: `BeautifulSoup.original_encoding`.
+ declared_html_encoding: Optional[_Encoding]
+
+ #: This is True if the markup that was parsed contains
+ #: U+FFFD REPLACEMENT_CHARACTER characters which were not present
+ #: in the original markup. These mark character sequences that
+ #: could not be represented in Unicode.
+ contains_replacement_characters: bool
+
+ def __init__(
+ self,
+ markup: _IncomingMarkup = "",
+ features: Optional[Union[str, Sequence[str]]] = None,
+ builder: Optional[Union[TreeBuilder, Type[TreeBuilder]]] = None,
+ parse_only: Optional[SoupStrainer] = None,
+ from_encoding: Optional[_Encoding] = None,
+ exclude_encodings: Optional[_Encodings] = None,
+ element_classes: Optional[Dict[Type[PageElement], Type[PageElement]]] = None,
+ **kwargs: Any,
+ ):
+ """Constructor.
+
+ :param markup: A string or a file-like object representing
+ markup to be parsed.
+
+ :param features: Desirable features of the parser to be
+ used. This may be the name of a specific parser ("lxml",
+ "lxml-xml", "html.parser", or "html5lib") or it may be the
+ type of markup to be used ("html", "html5", "xml"). It's
+ recommended that you name a specific parser, so that
+ Beautiful Soup gives you the same results across platforms
+ and virtual environments.
+
+ :param builder: A TreeBuilder subclass to instantiate (or
+ instance to use) instead of looking one up based on
+ `features`. You only need to use this if you've implemented a
+ custom TreeBuilder.
+
+ :param parse_only: A SoupStrainer. Only parts of the document
+ matching the SoupStrainer will be considered. This is useful
+ when parsing part of a document that would otherwise be too
+ large to fit into memory.
+
+ :param from_encoding: A string indicating the encoding of the
+ document to be parsed. Pass this in if Beautiful Soup is
+ guessing wrongly about the document's encoding.
+
+ :param exclude_encodings: A list of strings indicating
+ encodings known to be wrong. Pass this in if you don't know
+ the document's encoding but you know Beautiful Soup's guess is
+ wrong.
+
+ :param element_classes: A dictionary mapping BeautifulSoup
+ classes like Tag and NavigableString, to other classes you'd
+ like to be instantiated instead as the parse tree is
+ built. This is useful for subclassing Tag or NavigableString
+ to modify default behavior.
+
+ :param kwargs: For backwards compatibility purposes, the
+ constructor accepts certain keyword arguments used in
+ Beautiful Soup 3. None of these arguments do anything in
+ Beautiful Soup 4; they will result in a warning and then be
+ ignored.
+
+ Apart from this, any keyword arguments passed into the
+ BeautifulSoup constructor are propagated to the TreeBuilder
+ constructor. This makes it possible to configure a
+ TreeBuilder by passing in arguments, not just by saying which
+ one to use.
+ """
+ if "convertEntities" in kwargs:
+ del kwargs["convertEntities"]
+ warnings.warn(
+ "BS4 does not respect the convertEntities argument to the "
+ "BeautifulSoup constructor. Entities are always converted "
+ "to Unicode characters."
+ )
+
+ if "markupMassage" in kwargs:
+ del kwargs["markupMassage"]
+ warnings.warn(
+ "BS4 does not respect the markupMassage argument to the "
+ "BeautifulSoup constructor. The tree builder is responsible "
+ "for any necessary markup massage."
+ )
+
+ if "smartQuotesTo" in kwargs:
+ del kwargs["smartQuotesTo"]
+ warnings.warn(
+ "BS4 does not respect the smartQuotesTo argument to the "
+ "BeautifulSoup constructor. Smart quotes are always converted "
+ "to Unicode characters."
+ )
+
+ if "selfClosingTags" in kwargs:
+ del kwargs["selfClosingTags"]
+ warnings.warn(
+ "Beautiful Soup 4 does not respect the selfClosingTags argument to the "
+ "BeautifulSoup constructor. The tree builder is responsible "
+ "for understanding self-closing tags."
+ )
+
+ if "isHTML" in kwargs:
+ del kwargs["isHTML"]
+ warnings.warn(
+ "Beautiful Soup 4 does not respect the isHTML argument to the "
+ "BeautifulSoup constructor. Suggest you use "
+ "features='lxml' for HTML and features='lxml-xml' for "
+ "XML."
+ )
+
+ def deprecated_argument(old_name: str, new_name: str) -> Optional[Any]:
+ if old_name in kwargs:
+ warnings.warn(
+ 'The "%s" argument to the BeautifulSoup constructor '
+ 'was renamed to "%s" in Beautiful Soup 4.0.0'
+ % (old_name, new_name),
+ DeprecationWarning,
+ stacklevel=3,
+ )
+ return kwargs.pop(old_name)
+ return None
+
+ parse_only = parse_only or deprecated_argument("parseOnlyThese", "parse_only")
+ if parse_only is not None:
+ # Issue a warning if we can tell in advance that
+ # parse_only will exclude the entire tree.
+ if parse_only.excludes_everything:
+ warnings.warn(
+ f"The given value for parse_only will exclude everything: {parse_only}",
+ UserWarning,
+ stacklevel=3,
+ )
+
+ from_encoding = from_encoding or deprecated_argument(
+ "fromEncoding", "from_encoding"
+ )
+
+ if from_encoding and isinstance(markup, str):
+ warnings.warn(
+ "You provided Unicode markup but also provided a value for from_encoding. Your from_encoding will be ignored."
+ )
+ from_encoding = None
+
+ self.element_classes = element_classes or dict()
+
+ # We need this information to track whether or not the builder
+ # was specified well enough that we can omit the 'you need to
+ # specify a parser' warning.
+ original_builder = builder
+ original_features = features
+
+ builder_class: Optional[Type[TreeBuilder]] = None
+ if isinstance(builder, type):
+ # A builder class was passed in; it needs to be instantiated.
+ builder_class = builder
+ builder = None
+ elif builder is None:
+ if isinstance(features, str):
+ features = [features]
+ if features is None or len(features) == 0:
+ features = self.DEFAULT_BUILDER_FEATURES
+ possible_builder_class = builder_registry.lookup(*features)
+ if possible_builder_class is None:
+ raise FeatureNotFound(
+ "Couldn't find a tree builder with the features you "
+ "requested: %s. Do you need to install a parser library?"
+ % ",".join(features)
+ )
+ builder_class = possible_builder_class
+
+ # At this point either we have a TreeBuilder instance in
+ # builder, or we have a builder_class that we can instantiate
+ # with the remaining **kwargs.
+ if builder is None:
+ assert builder_class is not None
+ builder = builder_class(**kwargs)
+ if (
+ not original_builder
+ and not (
+ original_features == builder.NAME
+ or (
+ isinstance(original_features, str)
+ and original_features in builder.ALTERNATE_NAMES
+ )
+ )
+ and markup
+ ):
+ # The user did not tell us which TreeBuilder to use,
+ # and we had to guess. Issue a warning.
+ if builder.is_xml:
+ markup_type = "XML"
+ else:
+ markup_type = "HTML"
+
+ # This code adapted from warnings.py so that we get the same line
+ # of code as our warnings.warn() call gets, even if the answer is wrong
+ # (as it may be in a multithreading situation).
+ caller = None
+ try:
+ caller = sys._getframe(1)
+ except ValueError:
+ pass
+ if caller:
+ globals = caller.f_globals
+ line_number = caller.f_lineno
+ else:
+ globals = sys.__dict__
+ line_number = 1
+ filename = globals.get("__file__")
+ if filename:
+ fnl = filename.lower()
+ if fnl.endswith((".pyc", ".pyo")):
+ filename = filename[:-1]
+ if filename:
+ # If there is no filename at all, the user is most likely in a REPL,
+ # and the warning is not necessary.
+ values = dict(
+ filename=filename,
+ line_number=line_number,
+ parser=builder.NAME,
+ markup_type=markup_type,
+ )
+ warnings.warn(
+ GuessedAtParserWarning.MESSAGE % values,
+ GuessedAtParserWarning,
+ stacklevel=2,
+ )
+ else:
+ if kwargs:
+ warnings.warn(
+ "Keyword arguments to the BeautifulSoup constructor will be ignored. These would normally be passed into the TreeBuilder constructor, but a TreeBuilder instance was passed in as `builder`."
+ )
+
+ self.builder = builder
+ self.is_xml = builder.is_xml
+ self.known_xml = self.is_xml
+ self._namespaces = dict()
+ self.parse_only = parse_only
+
+ if hasattr(markup, "read"): # It's a file-type object.
+ markup = cast(io.IOBase, markup).read()
+ elif not isinstance(markup, (bytes, str)) and not hasattr(markup, "__len__"):
+ raise TypeError(
+ f"Incoming markup is of an invalid type: {markup!r}. Markup must be a string, a bytestring, or an open filehandle."
+ )
+ elif isinstance(markup, Sized) and len(markup) <= 256 and (
+ (isinstance(markup, bytes) and b"<" not in markup and b"\n" not in markup)
+ or (isinstance(markup, str) and "<" not in markup and "\n" not in markup)
+ ):
+ # Issue warnings for a couple beginner problems
+ # involving passing non-markup to Beautiful Soup.
+ # Beautiful Soup will still parse the input as markup,
+ # since that is sometimes the intended behavior.
+ if not self._markup_is_url(markup):
+ self._markup_resembles_filename(markup)
+
+ # At this point we know markup is a string or bytestring. If
+ # it was a file-type object, we've read from it.
+ markup = cast(_RawMarkup, markup)
+
+ rejections = []
+ success = False
+ for (
+ self.markup,
+ self.original_encoding,
+ self.declared_html_encoding,
+ self.contains_replacement_characters,
+ ) in self.builder.prepare_markup(
+ markup, from_encoding, exclude_encodings=exclude_encodings
+ ):
+ self.reset()
+ self.builder.initialize_soup(self)
+ try:
+ self._feed()
+ success = True
+ break
+ except ParserRejectedMarkup as e:
+ rejections.append(e)
+ pass
+
+ if not success:
+ other_exceptions = [str(e) for e in rejections]
+ raise ParserRejectedMarkup(
+ "The markup you provided was rejected by the parser. Trying a different parser or a different encoding may help.\n\nOriginal exception(s) from parser:\n "
+ + "\n ".join(other_exceptions)
+ )
+
+ # Clear out the markup and remove the builder's circular
+ # reference to this object.
+ self.markup = None
+ self.builder.soup = None
+
+ def copy_self(self) -> "BeautifulSoup":
+ """Create a new BeautifulSoup object with the same TreeBuilder,
+ but not associated with any markup.
+
+ This is the first step of the deepcopy process.
+ """
+ clone = type(self)("", None, self.builder)
+
+ # Keep track of the encoding of the original document,
+ # since we won't be parsing it again.
+ clone.original_encoding = self.original_encoding
+ return clone
+
+ def __getstate__(self) -> Dict[str, Any]:
+ # Frequently a tree builder can't be pickled.
+ d = dict(self.__dict__)
+ if "builder" in d and d["builder"] is not None and not self.builder.picklable:
+ d["builder"] = type(self.builder)
+ # Store the contents as a Unicode string.
+ d["contents"] = []
+ d["markup"] = self.decode()
+
+ # If _most_recent_element is present, it's a Tag object left
+ # over from initial parse. It might not be picklable and we
+ # don't need it.
+ if "_most_recent_element" in d:
+ del d["_most_recent_element"]
+ return d
+
+ def __setstate__(self, state: Dict[str, Any]) -> None:
+ # If necessary, restore the TreeBuilder by looking it up.
+ self.__dict__ = state
+ if isinstance(self.builder, type):
+ self.builder = self.builder()
+ elif not self.builder:
+ # We don't know which builder was used to build this
+ # parse tree, so use a default we know is always available.
+ self.builder = HTMLParserTreeBuilder()
+ self.builder.soup = self
+ self.reset()
+ self._feed()
+
+ @property
+ def _is_root(self):
+ """Yes, a BeautifulSoup object is the root of its parse tree. Used by the _root_object internal property."""
+ return True
+
+ @classmethod
+ @_deprecated(
+ replaced_by="nothing (private method, will be removed)", version="4.13.0"
+ )
+ def _decode_markup(cls, markup: _RawMarkup) -> str:
+ """Ensure `markup` is Unicode so it's safe to send into warnings.warn.
+
+ warnings.warn had this problem back in 2010 but fortunately
+ not anymore. This has not been used for a long time; I just
+ noticed that fact while working on 4.13.0.
+ """
+ if isinstance(markup, bytes):
+ decoded = markup.decode("utf-8", "replace")
+ else:
+ decoded = markup
+ return decoded
+
+ @classmethod
+ def _markup_is_url(cls, markup: _RawMarkup) -> bool:
+ """Error-handling method to raise a warning if incoming markup looks
+ like a URL.
+
+ :param markup: A string of markup.
+ :return: Whether or not the markup resembled a URL
+ closely enough to justify issuing a warning.
+ """
+ problem: bool = False
+ if isinstance(markup, bytes):
+ problem = (
+ any(markup.startswith(prefix) for prefix in (b"http:", b"https:"))
+ and b" " not in markup
+ )
+ elif isinstance(markup, str):
+ problem = (
+ any(markup.startswith(prefix) for prefix in ("http:", "https:"))
+ and " " not in markup
+ )
+ else:
+ return False
+
+ if not problem:
+ return False
+ warnings.warn(
+ MarkupResemblesLocatorWarning.URL_MESSAGE % dict(what="URL"),
+ MarkupResemblesLocatorWarning,
+ stacklevel=3,
+ )
+ return True
+
+ @classmethod
+ def _markup_resembles_filename(cls, markup: _RawMarkup) -> bool:
+ """Error-handling method to issue a warning if incoming markup
+ resembles a filename.
+
+ :param markup: A string of markup.
+ :return: Whether or not the markup resembled a filename
+ closely enough to justify issuing a warning.
+ """
+ markup_b: bytes
+
+ # We're only checking ASCII characters, so rather than write
+ # the same tests twice, convert Unicode to a bytestring and
+ # operate on the bytestring.
+ if isinstance(markup, str):
+ markup_b = markup.encode("utf8")
+ else:
+ markup_b = markup
+
+ # Step 1: does it end with a common textual file extension?
+ filelike = False
+ lower = markup_b.lower()
+ extensions = [b".html", b".htm", b".xml", b".xhtml", b".txt"]
+ if any(lower.endswith(ext) for ext in extensions):
+ filelike = True
+ if not filelike:
+ return False
+
+ # Step 2: it _might_ be a file, but there are a few things
+ # we can look for that aren't very common in filenames.
+
+ # Characters that have special meaning to Unix shells. (< was
+ # excluded before this method was called.)
+ #
+ # Many of these are also reserved characters that cannot
+ # appear in Windows filenames.
+ for byte in markup_b:
+ if byte in b"?*#&;>$|":
+ return False
+
+ # Two consecutive forward slashes (as seen in a URL) or two
+ # consecutive spaces (as seen in fixed-width data).
+ #
+ # (Paths to Windows network shares contain consecutive
+ # backslashes, so checking that doesn't seem as helpful.)
+ if b"//" in markup_b:
+ return False
+ if b" " in markup_b:
+ return False
+
+ # A colon in any position other than position 1 (e.g. after a
+ # Windows drive letter).
+ if markup_b.startswith(b":"):
+ return False
+ colon_i = markup_b.rfind(b":")
+ if colon_i not in (-1, 1):
+ return False
+
+ # Step 3: If it survived all of those checks, it's similar
+ # enough to a file to justify issuing a warning.
+ warnings.warn(
+ MarkupResemblesLocatorWarning.FILENAME_MESSAGE % dict(what="filename"),
+ MarkupResemblesLocatorWarning,
+ stacklevel=3,
+ )
+ return True
+
+ def _feed(self) -> None:
+ """Internal method that parses previously set markup, creating a large
+ number of Tag and NavigableString objects.
+ """
+ # Convert the document to Unicode.
+ self.builder.reset()
+
+ if self.markup is not None:
+ self.builder.feed(self.markup)
+ # Close out any unfinished strings and close all the open tags.
+ self.endData()
+ while (
+ self.currentTag is not None and self.currentTag.name != self.ROOT_TAG_NAME
+ ):
+ self.popTag()
+
+ def reset(self) -> None:
+ """Reset this object to a state as though it had never parsed any
+ markup.
+ """
+ Tag.__init__(self, self, self.builder, self.ROOT_TAG_NAME)
+ self.hidden = True
+ self.builder.reset()
+ self.current_data = []
+ self.currentTag = None
+ self.tagStack = []
+ self.open_tag_counter = Counter()
+ self.preserve_whitespace_tag_stack = []
+ self.string_container_stack = []
+ self._most_recent_element = None
+ self.pushTag(self)
+
+ def new_tag(
+ self,
+ name: str,
+ namespace: Optional[str] = None,
+ nsprefix: Optional[str] = None,
+ attrs: Optional[_RawAttributeValues] = None,
+ sourceline: Optional[int] = None,
+ sourcepos: Optional[int] = None,
+ string: Optional[str] = None,
+ **kwattrs: _RawAttributeValue,
+ ) -> Tag:
+ """Create a new Tag associated with this BeautifulSoup object.
+
+ :param name: The name of the new Tag.
+ :param namespace: The URI of the new Tag's XML namespace, if any.
+ :param prefix: The prefix for the new Tag's XML namespace, if any.
+ :param attrs: A dictionary of this Tag's attribute values; can
+ be used instead of ``kwattrs`` for attributes like 'class'
+ that are reserved words in Python.
+ :param sourceline: The line number where this tag was
+ (purportedly) found in its source document.
+ :param sourcepos: The character position within ``sourceline`` where this
+ tag was (purportedly) found.
+ :param string: String content for the new Tag, if any.
+ :param kwattrs: Keyword arguments for the new Tag's attribute values.
+
+ """
+ attr_container = self.builder.attribute_dict_class(**kwattrs)
+ if attrs is not None:
+ attr_container.update(attrs)
+ tag_class = self.element_classes.get(Tag, Tag)
+
+ # Assume that this is either Tag or a subclass of Tag. If not,
+ # the user brought type-unsafety upon themselves.
+ tag_class = cast(Type[Tag], tag_class)
+ tag = tag_class(
+ None,
+ self.builder,
+ name,
+ namespace,
+ nsprefix,
+ attr_container,
+ sourceline=sourceline,
+ sourcepos=sourcepos,
+ )
+
+ if string is not None:
+ tag.string = string
+ return tag
+
+ def string_container(
+ self, base_class: Optional[Type[NavigableString]] = None
+ ) -> Type[NavigableString]:
+ """Find the class that should be instantiated to hold a given kind of
+ string.
+
+ This may be a built-in Beautiful Soup class or a custom class passed
+ in to the BeautifulSoup constructor.
+ """
+ container = base_class or NavigableString
+
+ # The user may want us to use some other class (hopefully a
+ # custom subclass) instead of the one we'd use normally.
+ container = cast(
+ Type[NavigableString], self.element_classes.get(container, container)
+ )
+
+ # On top of that, we may be inside a tag that needs a special
+ # container class.
+ if self.string_container_stack and container is NavigableString:
+ container = self.builder.string_containers.get(
+ self.string_container_stack[-1].name, container
+ )
+ return container
+
+ def new_string(
+ self, s: str, subclass: Optional[Type[NavigableString]] = None
+ ) -> NavigableString:
+ """Create a new `NavigableString` associated with this `BeautifulSoup`
+ object.
+
+ :param s: The string content of the `NavigableString`
+ :param subclass: The subclass of `NavigableString`, if any, to
+ use. If a document is being processed, an appropriate
+ subclass for the current location in the document will
+ be determined automatically.
+ """
+ container = self.string_container(subclass)
+ return container(s)
+
+ def insert_before(self, *args: _InsertableElement) -> List[PageElement]:
+ """This method is part of the PageElement API, but `BeautifulSoup` doesn't implement
+ it because there is nothing before or after it in the parse tree.
+ """
+ raise NotImplementedError(
+ "BeautifulSoup objects don't support insert_before()."
+ )
+
+ def insert_after(self, *args: _InsertableElement) -> List[PageElement]:
+ """This method is part of the PageElement API, but `BeautifulSoup` doesn't implement
+ it because there is nothing before or after it in the parse tree.
+ """
+ raise NotImplementedError("BeautifulSoup objects don't support insert_after().")
+
+ def popTag(self) -> Optional[Tag]:
+ """Internal method called by _popToTag when a tag is closed.
+
+ :meta private:
+ """
+ if not self.tagStack:
+ # Nothing to pop. This shouldn't happen.
+ return None
+ tag = self.tagStack.pop()
+ if tag.name in self.open_tag_counter:
+ self.open_tag_counter[tag.name] -= 1
+ if (
+ self.preserve_whitespace_tag_stack
+ and tag == self.preserve_whitespace_tag_stack[-1]
+ ):
+ self.preserve_whitespace_tag_stack.pop()
+ if self.string_container_stack and tag == self.string_container_stack[-1]:
+ self.string_container_stack.pop()
+ # print("Pop", tag.name)
+ if self.tagStack:
+ self.currentTag = self.tagStack[-1]
+ return self.currentTag
+
+ def pushTag(self, tag: Tag) -> None:
+ """Internal method called by handle_starttag when a tag is opened.
+
+ :meta private:
+ """
+ # print("Push", tag.name)
+ if self.currentTag is not None:
+ self.currentTag.contents.append(tag)
+ self.tagStack.append(tag)
+ self.currentTag = self.tagStack[-1]
+ if tag.name != self.ROOT_TAG_NAME:
+ self.open_tag_counter[tag.name] += 1
+ if tag.name in self.builder.preserve_whitespace_tags:
+ self.preserve_whitespace_tag_stack.append(tag)
+ if tag.name in self.builder.string_containers:
+ self.string_container_stack.append(tag)
+
+ def endData(self, containerClass: Optional[Type[NavigableString]] = None) -> None:
+ """Method called by the TreeBuilder when the end of a data segment
+ occurs.
+
+ :param containerClass: The class to use when incorporating the
+ data segment into the parse tree.
+
+ :meta private:
+ """
+ if self.current_data:
+ current_data = "".join(self.current_data)
+ # If whitespace is not preserved, and this string contains
+ # nothing but ASCII spaces, replace it with a single space
+ # or newline.
+ if not self.preserve_whitespace_tag_stack:
+ strippable = True
+ for i in current_data:
+ if i not in self.ASCII_SPACES:
+ strippable = False
+ break
+ if strippable:
+ if "\n" in current_data:
+ current_data = "\n"
+ else:
+ current_data = " "
+
+ # Reset the data collector.
+ self.current_data = []
+
+ # Should we add this string to the tree at all?
+ if (
+ self.parse_only
+ and len(self.tagStack) <= 1
+ and (not self.parse_only.allow_string_creation(current_data))
+ ):
+ return
+
+ containerClass = self.string_container(containerClass)
+ o = containerClass(current_data)
+ self.object_was_parsed(o)
+
+ def object_was_parsed(
+ self,
+ o: PageElement,
+ parent: Optional[Tag] = None,
+ most_recent_element: Optional[PageElement] = None,
+ ) -> None:
+ """Method called by the TreeBuilder to integrate an object into the
+ parse tree.
+
+ :meta private:
+ """
+ if parent is None:
+ parent = self.currentTag
+ assert parent is not None
+ previous_element: Optional[PageElement]
+ if most_recent_element is not None:
+ previous_element = most_recent_element
+ else:
+ previous_element = self._most_recent_element
+
+ next_element = previous_sibling = next_sibling = None
+ if isinstance(o, Tag):
+ next_element = o.next_element
+ next_sibling = o.next_sibling
+ previous_sibling = o.previous_sibling
+ if previous_element is None:
+ previous_element = o.previous_element
+
+ fix = parent.next_element is not None
+
+ o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)
+
+ self._most_recent_element = o
+ parent.contents.append(o)
+
+ # Check if we are inserting into an already parsed node.
+ if fix:
+ self._linkage_fixer(parent)
+
+ def _linkage_fixer(self, el: Tag) -> None:
+ """Make sure linkage of this fragment is sound."""
+
+ first = el.contents[0]
+ child = el.contents[-1]
+ descendant: PageElement = child
+
+ if child is first and el.parent is not None:
+ # Parent should be linked to first child
+ el.next_element = child
+ # We are no longer linked to whatever this element is
+ prev_el = child.previous_element
+ if prev_el is not None and prev_el is not el:
+ prev_el.next_element = None
+ # First child should be linked to the parent, and no previous siblings.
+ child.previous_element = el
+ child.previous_sibling = None
+
+ # We have no sibling as we've been appended as the last.
+ child.next_sibling = None
+
+ # This index is a tag, dig deeper for a "last descendant"
+ if isinstance(child, Tag) and child.contents:
+ # _last_decendant is typed as returning Optional[PageElement],
+ # but the value can't be None here, because el is a Tag
+ # which we know has contents.
+ descendant = cast(PageElement, child._last_descendant(False))
+
+ # As the final step, link last descendant. It should be linked
+ # to the parent's next sibling (if found), else walk up the chain
+ # and find a parent with a sibling. It should have no next sibling.
+ descendant.next_element = None
+ descendant.next_sibling = None
+
+ target: Optional[Tag] = el
+ while True:
+ if target is None:
+ break
+ elif target.next_sibling is not None:
+ descendant.next_element = target.next_sibling
+ target.next_sibling.previous_element = child
+ break
+ target = target.parent
+
+ def _popToTag(
+ self, name: str, nsprefix: Optional[str] = None, inclusivePop: bool = True
+ ) -> Optional[Tag]:
+ """Pops the tag stack up to and including the most recent
+ instance of the given tag.
+
+ If there are no open tags with the given name, nothing will be
+ popped.
+
+ :param name: Pop up to the most recent tag with this name.
+ :param nsprefix: The namespace prefix that goes with `name`.
+ :param inclusivePop: It this is false, pops the tag stack up
+ to but *not* including the most recent instqance of the
+ given tag.
+
+ :meta private:
+ """
+ # print("Popping to %s" % name)
+ if name == self.ROOT_TAG_NAME:
+ # The BeautifulSoup object itself can never be popped.
+ return None
+
+ most_recently_popped = None
+
+ stack_size = len(self.tagStack)
+ for i in range(stack_size - 1, 0, -1):
+ if not self.open_tag_counter.get(name):
+ break
+ t = self.tagStack[i]
+ if name == t.name and nsprefix == t.prefix:
+ if inclusivePop:
+ most_recently_popped = self.popTag()
+ break
+ most_recently_popped = self.popTag()
+
+ return most_recently_popped
+
+ def handle_starttag(
+ self,
+ name: str,
+ namespace: Optional[str],
+ nsprefix: Optional[str],
+ attrs: _RawAttributeValues,
+ sourceline: Optional[int] = None,
+ sourcepos: Optional[int] = None,
+ namespaces: Optional[Dict[str, str]] = None,
+ ) -> Optional[Tag]:
+ """Called by the tree builder when a new tag is encountered.
+
+ :param name: Name of the tag.
+ :param nsprefix: Namespace prefix for the tag.
+ :param attrs: A dictionary of attribute values. Note that
+ attribute values are expected to be simple strings; processing
+ of multi-valued attributes such as "class" comes later.
+ :param sourceline: The line number where this tag was found in its
+ source document.
+ :param sourcepos: The character position within `sourceline` where this
+ tag was found.
+ :param namespaces: A dictionary of all namespace prefix mappings
+ currently in scope in the document.
+
+ If this method returns None, the tag was rejected by an active
+ `ElementFilter`. You should proceed as if the tag had not occurred
+ in the document. For instance, if this was a self-closing tag,
+ don't call handle_endtag.
+
+ :meta private:
+ """
+ # print("Start tag %s: %s" % (name, attrs))
+ self.endData()
+
+ if (
+ self.parse_only
+ and len(self.tagStack) <= 1
+ and not self.parse_only.allow_tag_creation(nsprefix, name, attrs)
+ ):
+ return None
+
+ tag_class = self.element_classes.get(Tag, Tag)
+ # Assume that this is either Tag or a subclass of Tag. If not,
+ # the user brought type-unsafety upon themselves.
+ tag_class = cast(Type[Tag], tag_class)
+ tag = tag_class(
+ self,
+ self.builder,
+ name,
+ namespace,
+ nsprefix,
+ attrs,
+ self.currentTag,
+ self._most_recent_element,
+ sourceline=sourceline,
+ sourcepos=sourcepos,
+ namespaces=namespaces,
+ )
+ if tag is None:
+ return tag
+ if self._most_recent_element is not None:
+ self._most_recent_element.next_element = tag
+ self._most_recent_element = tag
+ self.pushTag(tag)
+ return tag
+
+ def handle_endtag(self, name: str, nsprefix: Optional[str] = None) -> None:
+ """Called by the tree builder when an ending tag is encountered.
+
+ :param name: Name of the tag.
+ :param nsprefix: Namespace prefix for the tag.
+
+ :meta private:
+ """
+ # print("End tag: " + name)
+ self.endData()
+ self._popToTag(name, nsprefix)
+
+ def handle_data(self, data: str) -> None:
+ """Called by the tree builder when a chunk of textual data is
+ encountered.
+
+ :meta private:
+ """
+ self.current_data.append(data)
+
+ def decode(
+ self,
+ indent_level: Optional[int] = None,
+ eventual_encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ formatter: Union[Formatter, str] = "minimal",
+ iterator: Optional[Iterator[PageElement]] = None,
+ **kwargs: Any,
+ ) -> str:
+ """Returns a string representation of the parse tree
+ as a full HTML or XML document.
+
+ :param indent_level: Each line of the rendering will be
+ indented this many levels. (The ``formatter`` decides what a
+ 'level' means, in terms of spaces or other characters
+ output.) This is used internally in recursive calls while
+ pretty-printing.
+ :param eventual_encoding: The encoding of the final document.
+ If this is None, the document will be a Unicode string.
+ :param formatter: Either a `Formatter` object, or a string naming one of
+ the standard formatters.
+ :param iterator: The iterator to use when navigating over the
+ parse tree. This is only used by `Tag.decode_contents` and
+ you probably won't need to use it.
+ """
+ if self.is_xml:
+ # Print the XML declaration
+ encoding_part = ""
+ declared_encoding: Optional[str] = eventual_encoding
+ if eventual_encoding in PYTHON_SPECIFIC_ENCODINGS:
+ # This is a special Python encoding; it can't actually
+ # go into an XML document because it means nothing
+ # outside of Python.
+ declared_encoding = None
+ if declared_encoding is not None:
+ encoding_part = ' encoding="%s"' % declared_encoding
+ prefix = '<?xml version="1.0"%s?>\n' % encoding_part
+ else:
+ prefix = ""
+
+ # Prior to 4.13.0, the first argument to this method was a
+ # bool called pretty_print, which gave the method a different
+ # signature from its superclass implementation, Tag.decode.
+ #
+ # The signatures of the two methods now match, but just in
+ # case someone is still passing a boolean in as the first
+ # argument to this method (or a keyword argument with the old
+ # name), we can handle it and put out a DeprecationWarning.
+ warning: Optional[str] = None
+ pretty_print: Optional[bool] = None
+ if isinstance(indent_level, bool):
+ if indent_level is True:
+ indent_level = 0
+ elif indent_level is False:
+ indent_level = None
+ warning = f"As of 4.13.0, the first argument to BeautifulSoup.decode has been changed from bool to int, to match Tag.decode. Pass in a value of {indent_level} instead."
+ else:
+ pretty_print = kwargs.pop("pretty_print", None)
+ assert not kwargs
+ if pretty_print is not None:
+ if pretty_print is True:
+ indent_level = 0
+ elif pretty_print is False:
+ indent_level = None
+ warning = f"As of 4.13.0, the pretty_print argument to BeautifulSoup.decode has been removed, to match Tag.decode. Pass in a value of indent_level={indent_level} instead."
+
+ if warning:
+ warnings.warn(warning, DeprecationWarning, stacklevel=2)
+ elif indent_level is False or pretty_print is False:
+ indent_level = None
+ return prefix + super(BeautifulSoup, self).decode(
+ indent_level, eventual_encoding, formatter, iterator
+ )
+
+
+# Aliases to make it easier to get started quickly, e.g. 'from bs4 import _soup'
+_s = BeautifulSoup
+_soup = BeautifulSoup
+
+
+class BeautifulStoneSoup(BeautifulSoup):
+ """Deprecated interface to an XML parser."""
+
+ def __init__(self, *args: Any, **kwargs: Any):
+ kwargs["features"] = "xml"
+ warnings.warn(
+ "The BeautifulStoneSoup class was deprecated in version 4.0.0. Instead of using "
+ 'it, pass features="xml" into the BeautifulSoup constructor.',
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ super(BeautifulStoneSoup, self).__init__(*args, **kwargs)
+
+
+# If this file is run as a script, act as an HTML pretty-printer.
+if __name__ == "__main__":
+ soup = BeautifulSoup(sys.stdin)
+ print((soup.prettify()))
diff --git a/bitbake/lib/bb/_vendor/bs4/_deprecation.py b/bitbake/lib/bb/_vendor/bs4/_deprecation.py
new file mode 100644
index 00000000..a7b5685b
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/_deprecation.py
@@ -0,0 +1,80 @@
+"""Helper functions for deprecation.
+
+This interface is itself unstable and may change without warning. Do
+not use these functions yourself, even as a joke. The underscores are
+there for a reason. No support will be given.
+
+In particular, most of this will go away without warning once
+Beautiful Soup drops support for Python 3.11, since Python 3.12
+defines a `@typing.deprecated()
+decorator. <https://peps.python.org/pep-0702/>`_
+"""
+
+import functools
+import warnings
+
+from typing import (
+ Any,
+ Callable,
+)
+
+
+def _deprecated_alias(old_name: str, new_name: str, version: str):
+ """Alias one attribute name to another for backward compatibility
+
+ :meta private:
+ """
+
+ @property # type:ignore
+ def alias(self) -> Any:
+ ":meta private:"
+ warnings.warn(
+ f"Access to deprecated property {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return getattr(self, new_name)
+
+ @alias.setter
+ def alias(self, value: str) -> None:
+ ":meta private:"
+ warnings.warn(
+ f"Write to deprecated property {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return setattr(self, new_name, value)
+
+ return alias
+
+
+def _deprecated_function_alias(
+ old_name: str, new_name: str, version: str
+) -> Callable[[Any], Any]:
+ def alias(self, *args: Any, **kwargs: Any) -> Any:
+ ":meta private:"
+ warnings.warn(
+ f"Call to deprecated method {old_name}. (Replaced by {new_name}) -- Deprecated since version {version}.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return getattr(self, new_name)(*args, **kwargs)
+
+ return alias
+
+
+def _deprecated(replaced_by: str, version: str) -> Callable:
+ def deprecate(func: Callable) -> Callable:
+ @functools.wraps(func)
+ def with_warning(*args: Any, **kwargs: Any) -> Any:
+ ":meta private:"
+ warnings.warn(
+ f"Call to deprecated method {func.__name__}. (Replaced by {replaced_by}) -- Deprecated since version {version}.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return func(*args, **kwargs)
+
+ return with_warning
+
+ return deprecate
diff --git a/bitbake/lib/bb/_vendor/bs4/_typing.py b/bitbake/lib/bb/_vendor/bs4/_typing.py
new file mode 100644
index 00000000..caf2567a
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/_typing.py
@@ -0,0 +1,205 @@
+# Custom type aliases used throughout Beautiful Soup to improve readability.
+
+# Notes on improvements to the type system in newer versions of Python
+# that can be used once Beautiful Soup drops support for older
+# versions:
+#
+# * ClassVar can be put on class variables now.
+# * In 3.10, x|y is an accepted shorthand for Union[x,y].
+# * In 3.10, TypeAlias gains capabilities that can be used to
+# improve the tree matching types (I don't remember what, exactly).
+# * In 3.9 it's possible to specialize the re.Match type,
+# e.g. re.Match[str]. In 3.8 there's a typing.re namespace for this,
+# but it's removed in 3.12, so to support the widest possible set of
+# versions I'm not using it.
+
+from bb._vendor.typing_extensions import (
+ runtime_checkable,
+ Protocol,
+ TypeAlias,
+)
+from typing import (
+ Any,
+ Callable,
+ Dict,
+ IO,
+ Iterable,
+ Mapping,
+ Optional,
+ Pattern,
+ TYPE_CHECKING,
+ Union,
+)
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4.element import (
+ AttributeValueList,
+ NamespacedAttribute,
+ NavigableString,
+ PageElement,
+ ResultSet,
+ Tag,
+ )
+
+
+@runtime_checkable
+class _RegularExpressionProtocol(Protocol):
+ """A protocol object which can accept either Python's built-in
+ `re.Pattern` objects, or the similar ``Regex`` objects defined by the
+ third-party ``regex`` package.
+ """
+
+ def search(
+ self, string: str, pos: int = ..., endpos: int = ...
+ ) -> Optional[Any]: ...
+
+ @property
+ def pattern(self) -> str: ...
+
+
+# Aliases for markup in various stages of processing.
+#
+#: The rawest form of markup: either a string, bytestring, or an open filehandle.
+_IncomingMarkup: TypeAlias = Union[str, bytes, IO[str], IO[bytes]]
+
+#: Markup that is in memory but has (potentially) yet to be converted
+#: to Unicode.
+_RawMarkup: TypeAlias = Union[str, bytes]
+
+# Aliases for character encodings
+#
+
+#: A data encoding.
+_Encoding: TypeAlias = str
+
+#: One or more data encodings.
+_Encodings: TypeAlias = Iterable[_Encoding]
+
+# Aliases for XML namespaces
+#
+
+#: The prefix for an XML namespace.
+_NamespacePrefix: TypeAlias = str
+
+#: The URL of an XML namespace
+_NamespaceURL: TypeAlias = str
+
+#: A mapping of prefixes to namespace URLs.
+_NamespaceMapping: TypeAlias = Dict[_NamespacePrefix, _NamespaceURL]
+
+#: A mapping of namespace URLs to prefixes
+_InvertedNamespaceMapping: TypeAlias = Dict[_NamespaceURL, _NamespacePrefix]
+
+# Aliases for the attribute values associated with HTML/XML tags.
+#
+
+#: The value associated with an HTML or XML attribute. This is the
+#: relatively unprocessed value Beautiful Soup expects to come from a
+#: `TreeBuilder`.
+_RawAttributeValue: TypeAlias = str
+
+#: A dictionary of names to `_RawAttributeValue` objects. This is how
+#: Beautiful Soup expects a `TreeBuilder` to represent a tag's
+#: attribute values.
+_RawAttributeValues: TypeAlias = (
+ "Mapping[Union[str, NamespacedAttribute], _RawAttributeValue]"
+)
+
+#: An attribute value in its final form, as stored in the
+# `Tag` class, after it has been processed and (in some cases)
+# split into a list of strings.
+_AttributeValue: TypeAlias = Union[str, "AttributeValueList"]
+
+#: A dictionary of names to :py:data:`_AttributeValue` objects. This is what
+#: a tag's attributes look like after processing.
+_AttributeValues: TypeAlias = Dict[str, _AttributeValue]
+
+#: The methods that deal with turning :py:data:`_RawAttributeValue` into
+#: :py:data:`_AttributeValue` may be called several times, even after the values
+#: are already processed (e.g. when cloning a tag), so they need to
+#: be able to acommodate both possibilities.
+_RawOrProcessedAttributeValues: TypeAlias = Union[_RawAttributeValues, _AttributeValues]
+
+#: A number of tree manipulation methods can take either a `PageElement` or a
+#: normal Python string (which will be converted to a `NavigableString`).
+_InsertableElement: TypeAlias = Union["PageElement", str]
+
+# Aliases to represent the many possibilities for matching bits of a
+# parse tree.
+#
+# This is very complicated because we're applying a formal type system
+# to some very DWIM code. The types we end up with will be the types
+# of the arguments to the SoupStrainer constructor and (more
+# familiarly to Beautiful Soup users) the find* methods.
+
+#: A function that takes a PageElement and returns a yes-or-no answer.
+_PageElementMatchFunction: TypeAlias = Callable[["PageElement"], bool]
+
+#: A function that takes the raw parsed ingredients of a markup tag
+#: and returns a yes-or-no answer.
+# Not necessary at the moment.
+# _AllowTagCreationFunction:TypeAlias = Callable[[Optional[str], str, Optional[_RawAttributeValues]], bool]
+
+#: A function that takes the raw parsed ingredients of a markup string node
+#: and returns a yes-or-no answer.
+# Not necessary at the moment.
+# _AllowStringCreationFunction:TypeAlias = Callable[[Optional[str]], bool]
+
+#: A function that takes a `Tag` and returns a yes-or-no answer.
+#: A `TagNameMatchRule` expects this kind of function, if you're
+#: going to pass it a function.
+_TagMatchFunction: TypeAlias = Callable[["Tag"], bool]
+
+#: A function that takes a string (or None) and returns a yes-or-no
+#: answer. An `AttributeValueMatchRule` expects this kind of function, if
+#: you're going to pass it a function.
+_NullableStringMatchFunction: TypeAlias = Callable[[Optional[str]], bool]
+
+#: A function that takes a string and returns a yes-or-no answer. A
+# `StringMatchRule` expects this kind of function, if you're going to
+# pass it a function.
+_StringMatchFunction: TypeAlias = Callable[[str], bool]
+
+#: Either a tag name, an attribute value or a string can be matched
+#: against a string, bytestring, regular expression, or a boolean.
+_BaseStrainable: TypeAlias = Union[str, bytes, Pattern[str], bool]
+
+#: A tag can be matched either with the `_BaseStrainable` options, or
+#: using a function that takes the `Tag` as its sole argument.
+_BaseStrainableElement: TypeAlias = Union[_BaseStrainable, _TagMatchFunction]
+
+#: A tag's attribute value can be matched either with the
+#: `_BaseStrainable` options, or using a function that takes that
+#: value as its sole argument.
+_BaseStrainableAttribute: TypeAlias = Union[_BaseStrainable, _NullableStringMatchFunction]
+
+#: A tag can be matched using either a single criterion or a list of
+#: criteria.
+_StrainableElement: TypeAlias = Union[
+ _BaseStrainableElement, Iterable[_BaseStrainableElement]
+]
+
+#: An attribute value can be matched using either a single criterion
+#: or a list of criteria.
+_StrainableAttribute: TypeAlias = Union[
+ _BaseStrainableAttribute, Iterable[_BaseStrainableAttribute]
+]
+
+#: An string can be matched using the same techniques as
+#: an attribute value.
+_StrainableString: TypeAlias = _StrainableAttribute
+
+#: A dictionary may be used to match against multiple attribute vlaues at once.
+_StrainableAttributes: TypeAlias = Dict[str, _StrainableAttribute]
+
+#: Many Beautiful soup methods return a PageElement or an ResultSet of
+#: PageElements. A PageElement is either a Tag or a NavigableString.
+#: These convenience aliases make it easier for IDE users to see which methods
+#: are available on the objects they're dealing with.
+_OneElement: TypeAlias = Union["PageElement", "Tag", "NavigableString"]
+_AtMostOneElement: TypeAlias = Optional[_OneElement]
+_AtMostOneTag: TypeAlias = Optional["Tag"]
+_AtMostOneNavigableString: TypeAlias = Optional["NavigableString"]
+_QueryResults: TypeAlias = "ResultSet[_OneElement]"
+_SomeTags: TypeAlias = "ResultSet[Tag]"
+_SomeNavigableStrings: TypeAlias = "ResultSet[NavigableString]"
diff --git a/bitbake/lib/bb/_vendor/bs4/_warnings.py b/bitbake/lib/bb/_vendor/bs4/_warnings.py
new file mode 100644
index 00000000..89041705
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/_warnings.py
@@ -0,0 +1,98 @@
+"""Define some custom warnings."""
+
+
+class GuessedAtParserWarning(UserWarning):
+ """The warning issued when BeautifulSoup has to guess what parser to
+ use -- probably because no parser was specified in the constructor.
+ """
+
+ MESSAGE: str = """No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system ("%(parser)s"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
+
+The code that caused this warning is on line %(line_number)s of the file %(filename)s. To get rid of this warning, pass the additional argument 'features="%(parser)s"' to the BeautifulSoup constructor.
+"""
+
+
+class UnusualUsageWarning(UserWarning):
+ """A superclass for warnings issued when Beautiful Soup sees
+ something that is typically the result of a mistake in the calling
+ code, but might be intentional on the part of the user. If it is
+ in fact intentional, you can filter the individual warning class
+ to get rid of the warning. If you don't like Beautiful Soup
+ second-guessing what you are doing, you can filter the
+ UnusualUsageWarningclass itself and get rid of these entirely.
+ """
+
+
+class MarkupResemblesLocatorWarning(UnusualUsageWarning):
+ """The warning issued when BeautifulSoup is given 'markup' that
+ actually looks like a resource locator -- a URL or a path to a file
+ on disk.
+ """
+
+ #: :meta private:
+ GENERIC_MESSAGE: str = """
+
+However, if you want to parse some data that happens to look like a %(what)s, then nothing has gone wrong: you are using Beautiful Soup correctly, and this warning is spurious and can be filtered. To make this warning go away, run this code before calling the BeautifulSoup constructor:
+
+ from bb._vendor.bs4 import MarkupResemblesLocatorWarning
+ import warnings
+
+ warnings.filterwarnings("ignore", category=MarkupResemblesLocatorWarning)
+ """
+
+ URL_MESSAGE: str = (
+ """The input passed in on this line looks more like a URL than HTML or XML.
+
+If you meant to use Beautiful Soup to parse the web page found at a certain URL, then something has gone wrong. You should use an Python package like 'requests' to fetch the content behind the URL. Once you have the content as a string, you can feed that string into Beautiful Soup."""
+ + GENERIC_MESSAGE
+ )
+
+ FILENAME_MESSAGE: str = (
+ """The input passed in on this line looks more like a filename than HTML or XML.
+
+If you meant to use Beautiful Soup to parse the contents of a file on disk, then something has gone wrong. You should open the file first, using code like this:
+
+ filehandle = open(your filename)
+
+You can then feed the open filehandle into Beautiful Soup instead of using the filename."""
+ + GENERIC_MESSAGE
+ )
+
+
+class AttributeResemblesVariableWarning(UnusualUsageWarning, SyntaxWarning):
+ """The warning issued when Beautiful Soup suspects a provided
+ attribute name may actually be the misspelled name of a Beautiful
+ Soup variable. Generally speaking, this is only used in cases like
+ "_class" where it's very unlikely the user would be referencing an
+ XML attribute with that name.
+ """
+
+ MESSAGE: str = """%(original)r is an unusual attribute name and is a common misspelling for %(autocorrect)r.
+
+If you meant %(autocorrect)r, change your code to use it, and this warning will go away.
+
+If you really did mean to check the %(original)r attribute, this warning is spurious and can be filtered. To make it go away, run this code before creating your BeautifulSoup object:
+
+ from bb._vendor.bs4 import AttributeResemblesVariableWarning
+ import warnings
+
+ warnings.filterwarnings("ignore", category=AttributeResemblesVariableWarning)
+"""
+
+
+class XMLParsedAsHTMLWarning(UnusualUsageWarning):
+ """The warning issued when an HTML parser is used to parse
+ XML that is not (as far as we can tell) XHTML.
+ """
+
+ MESSAGE: str = """It looks like you're using an HTML parser to parse an XML document.
+
+Assuming this really is an XML document, what you're doing might work, but you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the Python package 'lxml' installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
+
+If you want or need to use an HTML parser on this document, you can make this warning go away by filtering it. To do that, run this code before calling the BeautifulSoup constructor:
+
+ from bb._vendor.bs4 import XMLParsedAsHTMLWarning
+ import warnings
+
+ warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning)
+"""
diff --git a/bitbake/lib/bb/_vendor/bs4/builder/__init__.py b/bitbake/lib/bb/_vendor/bs4/builder/__init__.py
new file mode 100644
index 00000000..c9d5786d
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/builder/__init__.py
@@ -0,0 +1,848 @@
+from __future__ import annotations
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+from collections import defaultdict
+import re
+from types import ModuleType
+from typing import (
+ Any,
+ cast,
+ Dict,
+ Iterable,
+ List,
+ Optional,
+ Pattern,
+ Set,
+ Tuple,
+ Type,
+ TYPE_CHECKING,
+)
+import warnings
+import sys
+from bb._vendor.bs4.element import (
+ AttributeDict,
+ AttributeValueList,
+ CharsetMetaAttributeValue,
+ ContentMetaAttributeValue,
+ RubyParenthesisString,
+ RubyTextString,
+ Stylesheet,
+ Script,
+ TemplateString,
+ nonwhitespace_re,
+)
+
+# Exceptions were moved to their own module in 4.13. Import here for
+# backwards compatibility.
+from bb._vendor.bs4.exceptions import ParserRejectedMarkup
+
+from bb._vendor.bs4._typing import (
+ _AttributeValues,
+ _RawAttributeValue,
+)
+
+from bb._vendor.bs4._warnings import XMLParsedAsHTMLWarning
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4 import BeautifulSoup
+ from bb._vendor.bs4.element import (
+ NavigableString,
+ Tag,
+ )
+ from bb._vendor.bs4._typing import (
+ _AttributeValue,
+ _Encoding,
+ _Encodings,
+ _RawOrProcessedAttributeValues,
+ _RawMarkup,
+ )
+
+__all__ = [
+ "HTMLTreeBuilder",
+ "SAXTreeBuilder",
+ "TreeBuilder",
+ "TreeBuilderRegistry",
+]
+
+# Some useful features for a TreeBuilder to have.
+FAST = "fast"
+PERMISSIVE = "permissive"
+STRICT = "strict"
+XML = "xml"
+HTML = "html"
+HTML_5 = "html5"
+
+__all__ = [
+ "TreeBuilderRegistry",
+ "TreeBuilder",
+ "HTMLTreeBuilder",
+ "DetectsXMLParsedAsHTML",
+
+ "ParserRejectedMarkup", # backwards compatibility only as of 4.13.0
+]
+
+class TreeBuilderRegistry(object):
+ """A way of looking up TreeBuilder subclasses by their name or by desired
+ features.
+ """
+
+ builders_for_feature: Dict[str, List[Type[TreeBuilder]]]
+ builders: List[Type[TreeBuilder]]
+
+ def __init__(self) -> None:
+ self.builders_for_feature = defaultdict(list)
+ self.builders = []
+
+ def register(self, treebuilder_class: type[TreeBuilder]) -> None:
+ """Register a treebuilder based on its advertised features.
+
+ :param treebuilder_class: A subclass of `TreeBuilder`. its
+ `TreeBuilder.features` attribute should list its features.
+ """
+ for feature in treebuilder_class.features:
+ self.builders_for_feature[feature].insert(0, treebuilder_class)
+ self.builders.insert(0, treebuilder_class)
+
+ def lookup(self, *features: str) -> Optional[Type[TreeBuilder]]:
+ """Look up a TreeBuilder subclass with the desired features.
+
+ :param features: A list of features to look for. If none are
+ provided, the most recently registered TreeBuilder subclass
+ will be used.
+ :return: A TreeBuilder subclass, or None if there's no
+ registered subclass with all the requested features.
+ """
+ if len(self.builders) == 0:
+ # There are no builders at all.
+ return None
+
+ if len(features) == 0:
+ # They didn't ask for any features. Give them the most
+ # recently registered builder.
+ return self.builders[0]
+
+ # Go down the list of features in order, and eliminate any builders
+ # that don't match every feature.
+ feature_list = list(features)
+ feature_list.reverse()
+ candidates = None
+ candidate_set = None
+ while len(feature_list) > 0:
+ feature = feature_list.pop()
+ we_have_the_feature = self.builders_for_feature.get(feature, [])
+ if len(we_have_the_feature) > 0:
+ if candidates is None:
+ candidates = we_have_the_feature
+ candidate_set = set(candidates)
+ elif candidate_set is not None:
+ # Eliminate any candidates that don't have this feature.
+ candidate_set = candidate_set.intersection(set(we_have_the_feature))
+
+ # The only valid candidates are the ones in candidate_set.
+ # Go through the original list of candidates and pick the first one
+ # that's in candidate_set.
+ if candidate_set is None or candidates is None:
+ return None
+ for candidate in candidates:
+ if candidate in candidate_set:
+ return candidate
+ return None
+
+
+#: The `BeautifulSoup` constructor will take a list of features
+#: and use it to look up `TreeBuilder` classes in this registry.
+builder_registry: TreeBuilderRegistry = TreeBuilderRegistry()
+
+
+class TreeBuilder(object):
+ """Turn a textual document into a Beautiful Soup object tree.
+
+ This is an abstract superclass which smooths out the behavior of
+ different parser libraries into a single, unified interface.
+
+ :param multi_valued_attributes: If this is set to None, the
+ TreeBuilder will not turn any values for attributes like
+ 'class' into lists. Setting this to a dictionary will
+ customize this behavior; look at :py:attr:`bs4.builder.HTMLTreeBuilder.DEFAULT_CDATA_LIST_ATTRIBUTES`
+ for an example.
+
+ Internally, these are called "CDATA list attributes", but that
+ probably doesn't make sense to an end-user, so the argument name
+ is ``multi_valued_attributes``.
+
+ :param preserve_whitespace_tags: A set of tags to treat
+ the way <pre> tags are treated in HTML. Tags in this set
+ are immune from pretty-printing; their contents will always be
+ output as-is.
+
+ :param string_containers: A dictionary mapping tag names to
+ the classes that should be instantiated to contain the textual
+ contents of those tags. The default is to use NavigableString
+ for every tag, no matter what the name. You can override the
+ default by changing :py:attr:`DEFAULT_STRING_CONTAINERS`.
+
+ :param store_line_numbers: If the parser keeps track of the line
+ numbers and positions of the original markup, that information
+ will, by default, be stored in each corresponding
+ :py:class:`bs4.element.Tag` object. You can turn this off by
+ passing store_line_numbers=False; then Tag.sourcepos and
+ Tag.sourceline will always be None. If the parser you're using
+ doesn't keep track of this information, then store_line_numbers
+ is irrelevant.
+
+ :param attribute_dict_class: The value of a multi-valued attribute
+ (such as HTML's 'class') willl be stored in an instance of this
+ class. The default is Beautiful Soup's built-in
+ `AttributeValueList`, which is a normal Python list, and you
+ will probably never need to change it.
+ """
+
+ USE_DEFAULT: Any = object() #: :meta private:
+
+ def __init__(
+ self,
+ multi_valued_attributes: Dict[str, Set[str]] = USE_DEFAULT,
+ preserve_whitespace_tags: Set[str] = USE_DEFAULT,
+ store_line_numbers: bool = USE_DEFAULT,
+ string_containers: Dict[str, Type[NavigableString]] = USE_DEFAULT,
+ empty_element_tags: Set[str] = USE_DEFAULT,
+ attribute_dict_class: Type[AttributeDict] = AttributeDict,
+ attribute_value_list_class: Type[AttributeValueList] = AttributeValueList,
+ ):
+ self.soup = None
+ if multi_valued_attributes is self.USE_DEFAULT:
+ multi_valued_attributes = self.DEFAULT_CDATA_LIST_ATTRIBUTES
+ self.cdata_list_attributes = multi_valued_attributes
+ if preserve_whitespace_tags is self.USE_DEFAULT:
+ preserve_whitespace_tags = self.DEFAULT_PRESERVE_WHITESPACE_TAGS
+ self.preserve_whitespace_tags = preserve_whitespace_tags
+ if empty_element_tags is self.USE_DEFAULT:
+ self.empty_element_tags = self.DEFAULT_EMPTY_ELEMENT_TAGS
+ else:
+ self.empty_element_tags = empty_element_tags
+ # TODO: store_line_numbers is probably irrelevant now that
+ # the behavior of sourceline and sourcepos has been made consistent
+ # everywhere.
+ if store_line_numbers == self.USE_DEFAULT:
+ store_line_numbers = self.TRACKS_LINE_NUMBERS
+ self.store_line_numbers = store_line_numbers
+ if string_containers == self.USE_DEFAULT:
+ string_containers = self.DEFAULT_STRING_CONTAINERS
+ self.string_containers = string_containers
+ self.attribute_dict_class = attribute_dict_class
+ self.attribute_value_list_class = attribute_value_list_class
+
+ NAME: str = "[Unknown tree builder]"
+ ALTERNATE_NAMES: Iterable[str] = []
+ features: Iterable[str] = []
+
+ is_xml: bool = False
+ picklable: bool = False
+
+ soup: Optional[BeautifulSoup] #: :meta private:
+
+ #: A tag will be considered an empty-element
+ #: tag when and only when it has no contents.
+ empty_element_tags: Optional[Set[str]] = None #: :meta private:
+ cdata_list_attributes: Dict[str, Set[str]] #: :meta private:
+ preserve_whitespace_tags: Set[str] #: :meta private:
+ string_containers: Dict[str, Type[NavigableString]] #: :meta private:
+ tracks_line_numbers: bool #: :meta private:
+
+ #: A value for these tag/attribute combinations is a space- or
+ #: comma-separated list of CDATA, rather than a single CDATA.
+ DEFAULT_CDATA_LIST_ATTRIBUTES: Dict[str, Set[str]] = defaultdict(set)
+
+ #: Whitespace should be preserved inside these tags.
+ DEFAULT_PRESERVE_WHITESPACE_TAGS: Set[str] = set()
+
+ #: The textual contents of tags with these names should be
+ #: instantiated with some class other than `bs4.element.NavigableString`.
+ DEFAULT_STRING_CONTAINERS: Dict[str, Type[bs4.element.NavigableString]] = {} # type:ignore
+
+ #: By default, tags are treated as empty-element tags if they have
+ #: no contents--that is, using XML rules. HTMLTreeBuilder
+ #: defines a different set of DEFAULT_EMPTY_ELEMENT_TAGS based on the
+ #: HTML 4 and HTML5 standards.
+ DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] = None
+
+ #: Most parsers don't keep track of line numbers.
+ TRACKS_LINE_NUMBERS: bool = False
+
+ def initialize_soup(self, soup: BeautifulSoup) -> None:
+ """The BeautifulSoup object has been initialized and is now
+ being associated with the TreeBuilder.
+
+ :param soup: A BeautifulSoup object.
+ """
+ self.soup = soup
+
+ def reset(self) -> None:
+ """Do any work necessary to reset the underlying parser
+ for a new document.
+
+ By default, this does nothing.
+ """
+ pass
+
+ def can_be_empty_element(self, tag_name: str) -> bool:
+ """Might a tag with this name be an empty-element tag?
+
+ The final markup may or may not actually present this tag as
+ self-closing.
+
+ For instance: an HTMLBuilder does not consider a <p> tag to be
+ an empty-element tag (it's not in
+ HTMLBuilder.empty_element_tags). This means an empty <p> tag
+ will be presented as "<p></p>", not "<p/>" or "<p>".
+
+ The default implementation has no opinion about which tags are
+ empty-element tags, so a tag will be presented as an
+ empty-element tag if and only if it has no children.
+ "<foo></foo>" will become "<foo/>", and "<foo>bar</foo>" will
+ be left alone.
+
+ :param tag_name: The name of a markup tag.
+ """
+ if self.empty_element_tags is None:
+ return True
+ return tag_name in self.empty_element_tags
+
+ def feed(self, markup: _RawMarkup) -> None:
+ """Run incoming markup through some parsing process."""
+ raise NotImplementedError()
+
+ def prepare_markup(
+ self,
+ markup: _RawMarkup,
+ user_specified_encoding: Optional[_Encoding] = None,
+ document_declared_encoding: Optional[_Encoding] = None,
+ exclude_encodings: Optional[_Encodings] = None,
+ ) -> Iterable[Tuple[_RawMarkup, Optional[_Encoding], Optional[_Encoding], bool]]:
+ """Run any preliminary steps necessary to make incoming markup
+ acceptable to the parser.
+
+ :param markup: The markup that's about to be parsed.
+ :param user_specified_encoding: The user asked to try this encoding
+ to convert the markup into a Unicode string.
+ :param document_declared_encoding: The markup itself claims to be
+ in this encoding. NOTE: This argument is not used by the
+ calling code and can probably be removed.
+ :param exclude_encodings: The user asked *not* to try any of
+ these encodings.
+
+ :yield: A series of 4-tuples: (markup, encoding, declared encoding,
+ has undergone character replacement)
+
+ Each 4-tuple represents a strategy that the parser can try
+ to convert the document to Unicode and parse it. Each
+ strategy will be tried in turn.
+
+ By default, the only strategy is to parse the markup
+ as-is. See `LXMLTreeBuilderForXML` and
+ `HTMLParserTreeBuilder` for implementations that take into
+ account the quirks of particular parsers.
+
+ :meta private:
+
+ """
+ yield markup, None, None, False
+
+ def test_fragment_to_document(self, fragment: str) -> str:
+ """Wrap an HTML fragment to make it look like a document.
+
+ Different parsers do this differently. For instance, lxml
+ introduces an empty <head> tag, and html5lib
+ doesn't. Abstracting this away lets us write simple tests
+ which run HTML fragments through the parser and compare the
+ results against other HTML fragments.
+
+ This method should not be used outside of unit tests.
+
+ :param fragment: A fragment of HTML.
+ :return: A full HTML document.
+ :meta private:
+ """
+ return fragment
+
+ def set_up_substitutions(self, tag: Tag) -> bool:
+ """Set up any substitutions that will need to be performed on
+ a `Tag` when it's output as a string.
+
+ By default, this does nothing. See `HTMLTreeBuilder` for a
+ case where this is used.
+
+ :return: Whether or not a substitution was performed.
+ :meta private:
+ """
+ return False
+
+ def _replace_cdata_list_attribute_values(
+ self, tag_name: str, attrs: _RawOrProcessedAttributeValues
+ ) -> _AttributeValues:
+ """When an attribute value is associated with a tag that can
+ have multiple values for that attribute, convert the string
+ value to a list of strings.
+
+ Basically, replaces class="foo bar" with class=["foo", "bar"]
+
+ NOTE: This method modifies its input in place.
+
+ :param tag_name: The name of a tag.
+ :param attrs: A dictionary containing the tag's attributes.
+ Any appropriate attribute values will be modified in place.
+ :return: The modified dictionary that was originally passed in.
+ """
+
+ # First, cast the attrs dict to _AttributeValues. This might
+ # not be accurate yet, but it will be by the time this method
+ # returns.
+ modified_attrs = cast(_AttributeValues, attrs)
+ if not modified_attrs or not self.cdata_list_attributes:
+ # Nothing to do.
+ return modified_attrs
+
+ # There is at least a possibility that we need to modify one of
+ # the attribute values.
+ universal: Set[str] = self.cdata_list_attributes.get("*", set())
+ tag_specific = self.cdata_list_attributes.get(tag_name.lower(), None)
+ for attr in list(modified_attrs.keys()):
+ modified_value: _AttributeValue
+ if attr in universal or (tag_specific and attr in tag_specific):
+ # We have a "class"-type attribute whose string
+ # value is a whitespace-separated list of
+ # values. Split it into a list.
+ original_value: _AttributeValue = modified_attrs[attr]
+ if isinstance(original_value, _RawAttributeValue):
+ # This is a _RawAttributeValue (a string) that
+ # needs to be split and converted to a
+ # AttributeValueList so it can be an
+ # _AttributeValue.
+ modified_value = self.attribute_value_list_class(
+ nonwhitespace_re.findall(original_value)
+ )
+ else:
+ # html5lib calls setAttributes twice for the
+ # same tag when rearranging the parse tree. On
+ # the second call the attribute value here is
+ # already a list. This can also happen when a
+ # Tag object is cloned. If this happens, leave
+ # the value alone rather than trying to split
+ # it again.
+ modified_value = original_value
+ modified_attrs[attr] = modified_value
+ return modified_attrs
+
+
+class SAXTreeBuilder(TreeBuilder):
+ """A Beautiful Soup treebuilder that listens for SAX events.
+
+ This is not currently used for anything, and it will be removed
+ soon. It was a good idea, but it wasn't properly integrated into the
+ rest of Beautiful Soup, so there have been long stretches where it
+ hasn't worked properly.
+ """
+
+ def __init__(self, *args: Any, **kwargs: Any) -> None:
+ warnings.warn(
+ "The SAXTreeBuilder class was deprecated in 4.13.0 and will be removed soon thereafter. It is completely untested and probably doesn't work; do not use it.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ super(SAXTreeBuilder, self).__init__(*args, **kwargs)
+
+ def feed(self, markup: _RawMarkup) -> None:
+ raise NotImplementedError()
+
+ def close(self) -> None:
+ pass
+
+ def startElement(self, name: str, attrs: Dict[str, str]) -> None:
+ attrs = AttributeDict((key[1], value) for key, value in list(attrs.items()))
+ # print("Start %s, %r" % (name, attrs))
+ assert self.soup is not None
+ self.soup.handle_starttag(name, None, None, attrs)
+
+ def endElement(self, name: str) -> None:
+ # print("End %s" % name)
+ assert self.soup is not None
+ self.soup.handle_endtag(name)
+
+ def startElementNS(
+ self, nsTuple: Tuple[str, str], nodeName: str, attrs: Dict[str, str]
+ ) -> None:
+ # Throw away (ns, nodeName) for now.
+ self.startElement(nodeName, attrs)
+
+ def endElementNS(self, nsTuple: Tuple[str, str], nodeName: str) -> None:
+ # Throw away (ns, nodeName) for now.
+ self.endElement(nodeName)
+ # handler.endElementNS((ns, node.nodeName), node.nodeName)
+
+ def startPrefixMapping(self, prefix: str, nodeValue: str) -> None:
+ # Ignore the prefix for now.
+ pass
+
+ def endPrefixMapping(self, prefix: str) -> None:
+ # Ignore the prefix for now.
+ # handler.endPrefixMapping(prefix)
+ pass
+
+ def characters(self, content: str) -> None:
+ assert self.soup is not None
+ self.soup.handle_data(content)
+
+ def startDocument(self) -> None:
+ pass
+
+ def endDocument(self) -> None:
+ pass
+
+
+class HTMLTreeBuilder(TreeBuilder):
+ """This TreeBuilder knows facts about HTML, such as which tags are treated
+ specially by the HTML standard.
+ """
+
+ #: Some HTML tags are defined as having no contents. Beautiful Soup
+ #: treats these specially.
+ DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] = set(
+ [
+ # These are from HTML5.
+ "area",
+ "base",
+ "br",
+ "col",
+ "embed",
+ "hr",
+ "img",
+ "input",
+ "keygen",
+ "link",
+ "menuitem",
+ "meta",
+ "param",
+ "source",
+ "track",
+ "wbr",
+ # These are from earlier versions of HTML and are removed in HTML5.
+ "basefont",
+ "bgsound",
+ "command",
+ "frame",
+ "image",
+ "isindex",
+ "nextid",
+ "spacer",
+ ]
+ )
+
+ #: The HTML standard defines these tags as block-level elements. Beautiful
+ #: Soup does not treat these elements differently from other elements,
+ #: but it may do so eventually, and this information is available if
+ #: you need to use it.
+ DEFAULT_BLOCK_ELEMENTS: Set[str] = set(
+ [
+ "address",
+ "article",
+ "aside",
+ "blockquote",
+ "canvas",
+ "dd",
+ "div",
+ "dl",
+ "dt",
+ "fieldset",
+ "figcaption",
+ "figure",
+ "footer",
+ "form",
+ "h1",
+ "h2",
+ "h3",
+ "h4",
+ "h5",
+ "h6",
+ "header",
+ "hr",
+ "li",
+ "main",
+ "nav",
+ "noscript",
+ "ol",
+ "output",
+ "p",
+ "pre",
+ "section",
+ "table",
+ "tfoot",
+ "ul",
+ "video",
+ ]
+ )
+
+ #: These HTML tags need special treatment so they can be
+ #: represented by a string class other than `bs4.element.NavigableString`.
+ #:
+ #: For some of these tags, it's because the HTML standard defines
+ #: an unusual content model for them. I made this list by going
+ #: through the HTML spec
+ #: (https://html.spec.whatwg.org/#metadata-content) and looking for
+ #: "metadata content" elements that can contain strings.
+ #:
+ #: The Ruby tags (<rt> and <rp>) are here despite being normal
+ #: "phrasing content" tags, because the content they contain is
+ #: qualitatively different from other text in the document, and it
+ #: can be useful to be able to distinguish it.
+ #:
+ #: TODO: Arguably <noscript> could go here but it seems
+ #: qualitatively different from the other tags.
+ DEFAULT_STRING_CONTAINERS: Dict[str, Type[bs4.element.NavigableString]] = { # type:ignore
+ "rt": RubyTextString,
+ "rp": RubyParenthesisString,
+ "style": Stylesheet,
+ "script": Script,
+ "template": TemplateString,
+ }
+
+ #: The HTML standard defines these attributes as containing a
+ #: space-separated list of values, not a single value. That is,
+ #: class="foo bar" means that the 'class' attribute has two values,
+ #: 'foo' and 'bar', not the single value 'foo bar'. When we
+ #: encounter one of these attributes, we will parse its value into
+ #: a list of values if possible. Upon output, the list will be
+ #: converted back into a string.
+ DEFAULT_CDATA_LIST_ATTRIBUTES: Dict[str, Set[str]] = {
+ "*": {"class", "accesskey", "dropzone"},
+ "a": {"rel", "rev"},
+ "link": {"rel", "rev"},
+ "td": {"headers"},
+ "th": {"headers"},
+ "form": {"accept-charset"},
+ "object": {"archive"},
+ # These are HTML5 specific, as are *.accesskey and *.dropzone above.
+ "area": {"rel"},
+ "icon": {"sizes"},
+ "iframe": {"sandbox"},
+ "output": {"for"},
+ }
+
+ #: By default, whitespace inside these HTML tags will be
+ #: preserved rather than being collapsed.
+ DEFAULT_PRESERVE_WHITESPACE_TAGS: set[str] = set(["pre", "textarea"])
+
+ def set_up_substitutions(self, tag: Tag) -> bool:
+ """Replace the declared encoding in a <meta> tag with a placeholder,
+ to be substituted when the tag is output to a string.
+
+ An HTML document may come in to Beautiful Soup as one
+ encoding, but exit in a different encoding, and the <meta> tag
+ needs to be changed to reflect this.
+
+ :return: Whether or not a substitution was performed.
+
+ :meta private:
+ """
+ # We are only interested in <meta> tags
+ if tag.name != "meta":
+ return False
+
+ # TODO: This cast will fail in the (very unlikely) scenario
+ # that the programmer who instantiates the TreeBuilder
+ # specifies meta['content'] or meta['charset'] as
+ # cdata_list_attributes.
+ content: Optional[str] = cast(Optional[str], tag.get("content"))
+ charset: Optional[str] = cast(Optional[str], tag.get("charset"))
+
+ # But we can accommodate meta['http-equiv'] being made a
+ # cdata_list_attribute (again, very unlikely) without much
+ # trouble.
+ http_equiv: List[str] = tag.get_attribute_list("http-equiv")
+
+ # We are interested in <meta> tags that say what encoding the
+ # document was originally in. This means HTML 5-style <meta>
+ # tags that provide the "charset" attribute. It also means
+ # HTML 4-style <meta> tags that provide the "content"
+ # attribute and have "http-equiv" set to "content-type".
+ #
+ # In both cases we will replace the value of the appropriate
+ # attribute with a standin object that can take on any
+ # encoding.
+ substituted = False
+ if charset is not None:
+ # HTML 5 style:
+ # <meta charset="utf8">
+ tag["charset"] = CharsetMetaAttributeValue(charset)
+ substituted = True
+
+ elif content is not None and any(
+ x.lower() == "content-type" for x in http_equiv
+ ):
+ # HTML 4 style:
+ # <meta http-equiv="content-type" content="text/html; charset=utf8">
+ tag["content"] = ContentMetaAttributeValue(content)
+ substituted = True
+
+ return substituted
+
+
+class DetectsXMLParsedAsHTML(object):
+ """A mixin class for any class (a TreeBuilder, or some class used by a
+ TreeBuilder) that's in a position to detect whether an XML
+ document is being incorrectly parsed as HTML, and issue an
+ appropriate warning.
+
+ This requires being able to observe an incoming processing
+ instruction that might be an XML declaration, and also able to
+ observe tags as they're opened. If you can't do that for a given
+ `TreeBuilder`, there's a less reliable implementation based on
+ examining the raw markup.
+ """
+
+ #: Regular expression for seeing if string markup has an <html> tag.
+ LOOKS_LIKE_HTML: Pattern[str] = re.compile("<[^ +]html", re.I)
+
+ #: Regular expression for seeing if byte markup has an <html> tag.
+ LOOKS_LIKE_HTML_B: Pattern[bytes] = re.compile(b"<[^ +]html", re.I)
+
+ #: The start of an XML document string.
+ XML_PREFIX: str = "<?xml"
+
+ #: The start of an XML document bytestring.
+ XML_PREFIX_B: bytes = b"<?xml"
+
+ # This is typed as str, not `ProcessingInstruction`, because this
+ # check may be run before any Beautiful Soup objects are created.
+ _first_processing_instruction: Optional[str] #: :meta private:
+ _root_tag_name: Optional[str] #: :meta private:
+
+ @classmethod
+ def warn_if_markup_looks_like_xml(
+ cls, markup: Optional[_RawMarkup], stacklevel: int = 3
+ ) -> bool:
+ """Perform a check on some markup to see if it looks like XML
+ that's not XHTML. If so, issue a warning.
+
+ This is much less reliable than doing the check while parsing,
+ but some of the tree builders can't do that.
+
+ :param stacklevel: The stacklevel of the code calling this\
+ function.
+
+ :return: True if the markup looks like non-XHTML XML, False
+ otherwise.
+ """
+ if markup is None:
+ return False
+ markup = markup[:500]
+ if isinstance(markup, bytes):
+ markup_b: bytes = markup
+ looks_like_xml = markup_b.startswith(
+ cls.XML_PREFIX_B
+ ) and not cls.LOOKS_LIKE_HTML_B.search(markup)
+ else:
+ markup_s: str = markup
+ looks_like_xml = markup_s.startswith(
+ cls.XML_PREFIX
+ ) and not cls.LOOKS_LIKE_HTML.search(markup)
+
+ if looks_like_xml:
+ cls._warn(stacklevel=stacklevel + 2)
+ return True
+ return False
+
+ @classmethod
+ def _warn(cls, stacklevel: int = 5) -> None:
+ """Issue a warning about XML being parsed as HTML."""
+ warnings.warn(
+ XMLParsedAsHTMLWarning.MESSAGE,
+ XMLParsedAsHTMLWarning,
+ stacklevel=stacklevel,
+ )
+
+ def _initialize_xml_detector(self) -> None:
+ """Call this method before parsing a document."""
+ self._first_processing_instruction = None
+ self._root_tag_name = None
+
+ def _document_might_be_xml(self, processing_instruction: str) -> None:
+ """Call this method when encountering an XML declaration, or a
+ "processing instruction" that might be an XML declaration.
+
+ This helps Beautiful Soup detect potential issues later, if
+ the XML document turns out to be a non-XHTML document that's
+ being parsed as XML.
+ """
+ if (
+ self._first_processing_instruction is not None
+ or self._root_tag_name is not None
+ ):
+ # The document has already started. Don't bother checking
+ # anymore.
+ return
+
+ self._first_processing_instruction = processing_instruction
+
+ # We won't know until we encounter the first tag whether or
+ # not this is actually a problem.
+
+ def _root_tag_encountered(self, name: str) -> None:
+ """Call this when you encounter the document's root tag.
+
+ This is where we actually check whether an XML document is
+ being incorrectly parsed as HTML, and issue the warning.
+ """
+ if self._root_tag_name is not None:
+ # This method was incorrectly called multiple times. Do
+ # nothing.
+ return
+
+ self._root_tag_name = name
+
+ if (
+ name != "html"
+ and self._first_processing_instruction is not None
+ and self._first_processing_instruction.lower().startswith("xml ")
+ ):
+ # We encountered an XML declaration and then a tag other
+ # than 'html'. This is a reliable indicator that a
+ # non-XHTML document is being parsed as XML.
+ self._warn(stacklevel=10)
+
+
+def register_treebuilders_from(module: ModuleType) -> None:
+ """Copy TreeBuilders from the given module into this module."""
+ this_module = sys.modules[__name__]
+ for name in module.__all__:
+ obj = getattr(module, name)
+
+ if issubclass(obj, TreeBuilder):
+ setattr(this_module, name, obj)
+ this_module.__all__.append(name)
+ # Register the builder while we're at it.
+ this_module.builder_registry.register(obj)
+
+
+# Builders are registered in reverse order of priority, so that custom
+# builder registrations will take precedence. In general, we want lxml
+# to take precedence over html5lib, because it's faster. And we only
+# want to use HTMLParser as a last resort.
+from . import _htmlparser # noqa: E402
+
+register_treebuilders_from(_htmlparser)
+try:
+ from . import _html5lib
+
+ register_treebuilders_from(_html5lib)
+except ImportError:
+ # They don't have html5lib installed.
+ pass
+try:
+ from . import _lxml
+
+ register_treebuilders_from(_lxml)
+except ImportError:
+ # They don't have lxml installed.
+ pass
diff --git a/bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py b/bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py
new file mode 100644
index 00000000..2383da93
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py
@@ -0,0 +1,611 @@
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+__all__ = [
+ "HTML5TreeBuilder",
+]
+
+from typing import (
+ Any,
+ cast,
+ Dict,
+ Iterable,
+ Optional,
+ Sequence,
+ TYPE_CHECKING,
+ Tuple,
+ Union,
+)
+from bb._vendor.typing_extensions import TypeAlias
+from bb._vendor.bs4._typing import (
+ _AttributeValue,
+ _AttributeValues,
+ _Encoding,
+ _Encodings,
+ _NamespaceURL,
+ _RawMarkup,
+)
+
+import warnings
+from bb._vendor.bs4.builder import (
+ DetectsXMLParsedAsHTML,
+ PERMISSIVE,
+ HTML,
+ HTML_5,
+ HTMLTreeBuilder,
+)
+from bb._vendor.bs4.element import (
+ NamespacedAttribute,
+ PageElement,
+ nonwhitespace_re,
+)
+import html5lib
+from html5lib.constants import (
+ namespaces,
+)
+from bb._vendor.bs4.element import (
+ Comment,
+ Doctype,
+ NavigableString,
+ Tag,
+)
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4 import BeautifulSoup
+
+from html5lib.treebuilders import base as treebuilder_base
+
+
+class HTML5TreeBuilder(HTMLTreeBuilder):
+ """Use `html5lib <https://github.com/html5lib/html5lib-python>`_ to
+ build a tree.
+
+ Note that `HTML5TreeBuilder` does not support some common HTML
+ `TreeBuilder` features. Some of these features could theoretically
+ be implemented, but at the very least it's quite difficult,
+ because html5lib moves the parse tree around as it's being built.
+
+ Specifically:
+
+ * This `TreeBuilder` doesn't use different subclasses of
+ `NavigableString` (e.g. `Script`) based on the name of the tag
+ in which the string was found.
+ * You can't use a `SoupStrainer` to parse only part of a document.
+ """
+
+ NAME: str = "html5lib"
+
+ features: Iterable[str] = [NAME, PERMISSIVE, HTML_5, HTML]
+
+ #: html5lib can tell us which line number and position in the
+ #: original file is the source of an element.
+ TRACKS_LINE_NUMBERS: bool = True
+
+ underlying_builder: "TreeBuilderForHtml5lib" #: :meta private:
+ user_specified_encoding: Optional[_Encoding]
+
+ def prepare_markup(
+ self,
+ markup: _RawMarkup,
+ user_specified_encoding: Optional[_Encoding] = None,
+ document_declared_encoding: Optional[_Encoding] = None,
+ exclude_encodings: Optional[_Encodings] = None,
+ ) -> Iterable[Tuple[_RawMarkup, Optional[_Encoding], Optional[_Encoding], bool]]:
+ # Store the user-specified encoding for use later on.
+ self.user_specified_encoding = user_specified_encoding
+
+ # document_declared_encoding and exclude_encodings aren't used
+ # ATM because the html5lib TreeBuilder doesn't use
+ # UnicodeDammit.
+ for variable, name in (
+ (document_declared_encoding, "document_declared_encoding"),
+ (exclude_encodings, "exclude_encodings"),
+ ):
+ if variable:
+ warnings.warn(
+ f"You provided a value for {name}, but the html5lib tree builder doesn't support {name}.",
+ stacklevel=3,
+ )
+
+ # html5lib only parses HTML, so if it's given XML that's worth
+ # noting.
+ DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xml(markup, stacklevel=3)
+
+ yield (markup, None, None, False)
+
+ # These methods are defined by Beautiful Soup.
+ def feed(self, markup: _RawMarkup) -> None:
+ """Run some incoming markup through some parsing process,
+ populating the `BeautifulSoup` object in `HTML5TreeBuilder.soup`.
+ """
+ if self.soup is not None and self.soup.parse_only is not None:
+ warnings.warn(
+ "You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.",
+ stacklevel=4,
+ )
+
+ # self.underlying_builder is probably None now, but it'll be set
+ # when html5lib calls self.create_treebuilder().
+ parser = html5lib.HTMLParser(tree=self.create_treebuilder)
+ assert self.underlying_builder is not None
+ self.underlying_builder.parser = parser
+ extra_kwargs = dict()
+ if not isinstance(markup, str):
+ # kwargs, specifically override_encoding, will eventually
+ # be passed in to html5lib's
+ # HTMLBinaryInputStream.__init__.
+ extra_kwargs["override_encoding"] = self.user_specified_encoding
+
+ doc = parser.parse(markup, **extra_kwargs) # type:ignore
+
+ # Set the character encoding detected by the tokenizer.
+ if isinstance(markup, str):
+ # We need to special-case this because html5lib sets
+ # charEncoding to UTF-8 if it gets Unicode input.
+ doc.original_encoding = None
+ else:
+ original_encoding = parser.tokenizer.stream.charEncoding[0] # type:ignore
+ # The encoding is an html5lib Encoding object. We want to
+ # use a string for compatibility with other tree builders.
+ original_encoding = original_encoding.name
+ doc.original_encoding = original_encoding
+ self.underlying_builder.parser = None
+
+ def create_treebuilder(
+ self, namespaceHTMLElements: bool
+ ) -> "TreeBuilderForHtml5lib":
+ """Called by html5lib to instantiate the kind of class it
+ calls a 'TreeBuilder'.
+
+ :param namespaceHTMLElements: Whether or not to namespace HTML elements.
+
+ :meta private:
+ """
+ self.underlying_builder = TreeBuilderForHtml5lib(
+ namespaceHTMLElements, self.soup, store_line_numbers=self.store_line_numbers
+ )
+ return self.underlying_builder
+
+ def test_fragment_to_document(self, fragment: str) -> str:
+ """See `TreeBuilder`."""
+ return "<html><head></head><body>%s</body></html>" % fragment
+
+
+class TreeBuilderForHtml5lib(treebuilder_base.TreeBuilder):
+ soup: "BeautifulSoup" #: :meta private:
+ parser: Optional[html5lib.HTMLParser] #: :meta private:
+
+ def __init__(
+ self,
+ namespaceHTMLElements: bool,
+ soup: Optional["BeautifulSoup"] = None,
+ store_line_numbers: bool = True,
+ **kwargs: Any,
+ ):
+ if soup:
+ self.soup = soup
+ else:
+ warnings.warn(
+ "The optionality of the 'soup' argument to the TreeBuilderForHtml5lib constructor is deprecated as of Beautiful Soup 4.13.0: 'soup' is now required. If you can't pass in a BeautifulSoup object here, or you get this warning and it seems mysterious to you, please contact the Beautiful Soup developer team for possible un-deprecation.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ from bb._vendor.bs4 import BeautifulSoup
+
+ # TODO: Why is the parser 'html.parser' here? Using
+ # html5lib doesn't cause an infinite loop and is more
+ # accurate. Best to get rid of this entire section, I think.
+ self.soup = BeautifulSoup(
+ "", "html.parser", store_line_numbers=store_line_numbers, **kwargs
+ )
+ # TODO: What are **kwargs exactly? Should they be passed in
+ # here in addition to/instead of being passed to the BeautifulSoup
+ # constructor?
+ super(TreeBuilderForHtml5lib, self).__init__(namespaceHTMLElements)
+
+ # This will be set later to a real html5lib HTMLParser object,
+ # which we can use to track the current line number.
+ self.parser = None
+ self.store_line_numbers = store_line_numbers
+
+ def documentClass(self) -> "Element":
+ self.soup.reset()
+ return Element(self.soup, self.soup, None)
+
+ def insertDoctype(self, token: Dict[str, Any]) -> None:
+ name: str = cast(str, token["name"])
+ publicId: Optional[str] = cast(Optional[str], token["publicId"])
+ systemId: Optional[str] = cast(Optional[str], token["systemId"])
+
+ doctype = Doctype.for_name_and_ids(name, publicId, systemId)
+ self.soup.object_was_parsed(doctype)
+
+ def elementClass(self, name: str, namespace: str) -> "Element":
+ sourceline: Optional[int] = None
+ sourcepos: Optional[int] = None
+ if self.parser is not None and self.store_line_numbers:
+ # This represents the point immediately after the end of the
+ # tag. We don't know when the tag started, but we do know
+ # where it ended -- the character just before this one.
+ sourceline, sourcepos = self.parser.tokenizer.stream.position() # type:ignore
+ assert sourcepos is not None
+ sourcepos = sourcepos - 1
+ tag = self.soup.new_tag(
+ name, namespace, sourceline=sourceline, sourcepos=sourcepos
+ )
+
+ return Element(tag, self.soup, namespace)
+
+ def commentClass(self, data: str) -> "TextNode":
+ return TextNode(Comment(data), self.soup)
+
+ def fragmentClass(self) -> "Element":
+ """This is only used by html5lib HTMLParser.parseFragment(),
+ which is never used by Beautiful Soup, only by the html5lib
+ unit tests. Since we don't currently hook into those tests,
+ the implementation is left blank.
+ """
+ raise NotImplementedError()
+
+ def getFragment(self) -> "Element":
+ """This is only used by the html5lib unit tests. Since we
+ don't currently hook into those tests, the implementation is
+ left blank.
+ """
+ raise NotImplementedError()
+
+ def appendChild(self, node: "Element") -> None:
+ # TODO: This code is not covered by the BS4 tests, and
+ # apparently not triggered by the html5lib test suite either.
+ # But it doesn't seem test-specific and there are calls to it
+ # (or a method with the same name) all over html5lib, so I'm
+ # leaving the implementation in place rather than replacing it
+ # with NotImplementedError()
+ self.soup.append(node.element)
+
+ def getDocument(self) -> "BeautifulSoup":
+ return self.soup
+
+ def testSerializer(self, node: "Element") -> None:
+ """This is only used by the html5lib unit tests. Since we
+ don't currently hook into those tests, the implementation is
+ left blank.
+ """
+ raise NotImplementedError()
+
+
+class AttrList(object):
+ """Represents a Tag's attributes in a way compatible with html5lib."""
+
+ element: Tag
+ attrs: _AttributeValues
+
+ def __init__(self, element: Tag):
+ self.element = element
+ self.attrs = dict(self.element.attrs)
+
+ def __iter__(self) -> Iterable[Tuple[str, _AttributeValue]]:
+ return list(self.attrs.items()).__iter__()
+
+ def __setitem__(self, name: str, value: _AttributeValue) -> None:
+ # If this attribute is a multi-valued attribute for this element,
+ # turn its value into a list.
+ list_attr = self.element.cdata_list_attributes or {}
+ if name in list_attr.get("*", []) or (
+ self.element.name in list_attr
+ and name in list_attr.get(self.element.name, [])
+ ):
+ # A node that is being cloned may have already undergone
+ # this procedure. Check for this and skip it.
+ if not isinstance(value, list):
+ assert isinstance(value, str)
+ value = self.element.attribute_value_list_class(
+ nonwhitespace_re.findall(value)
+ )
+ self.element[name] = value
+
+ def items(self) -> Iterable[Tuple[str, _AttributeValue]]:
+ return list(self.attrs.items())
+
+ def keys(self) -> Iterable[str]:
+ return list(self.attrs.keys())
+
+ def __len__(self) -> int:
+ return len(self.attrs)
+
+ def __getitem__(self, name: str) -> _AttributeValue:
+ return self.attrs[name]
+
+ def __contains__(self, name: str) -> bool:
+ return name in list(self.attrs.keys())
+
+
+class BeautifulSoupNode(treebuilder_base.Node):
+ # A node can correspond to _either_ a Tag _or_ a NavigableString.
+ tag: Optional[Tag]
+ string: Optional[NavigableString]
+ soup: "BeautifulSoup"
+ namespace: Optional[_NamespaceURL]
+
+ @property
+ def element(self) -> PageElement:
+ assert self.tag is not None or self.string is not None
+ if self.tag is not None:
+ return self.tag
+ else:
+ assert self.string is not None
+ return self.string
+
+ @property
+ def nodeType(self) -> int:
+ """Return the html5lib constant corresponding to the type of
+ the underlying DOM object.
+
+ NOTE: This property is only accessed by the html5lib test
+ suite, not by Beautiful Soup proper.
+ """
+ raise NotImplementedError()
+
+ # TODO-TYPING: typeshed stubs are incorrect about this;
+ # cloneNode returns a new Node, not None.
+ def cloneNode(self) -> treebuilder_base.Node: # type:ignore
+ raise NotImplementedError()
+
+
+class Element(BeautifulSoupNode):
+ namespace: Optional[_NamespaceURL]
+
+ def __init__(
+ self, element: Tag, soup: "BeautifulSoup", namespace: Optional[_NamespaceURL]
+ ):
+ self.tag = element
+ self.string = None
+ self.soup = soup
+ self.namespace = namespace
+ treebuilder_base.Node.__init__(self, element.name)
+
+ def appendChild(self, node: "BeautifulSoupNode") -> None:
+ string_child: Optional[NavigableString] = None
+ child: PageElement
+ if type(node.string) is NavigableString:
+ # We check for NavigableString *only* because we want to avoid
+ # joining PreformattedStrings, such as Comments, with nearby strings.
+ string_child = child = node.string
+ else:
+ child = node.element
+ node.parent = self
+
+ if (
+ child is not None
+ and child.parent is not None
+ and not isinstance(child, str)
+ ):
+ node.element.extract()
+
+ if (
+ string_child is not None
+ and self.tag is not None and self.tag.contents
+ and type(self.tag.contents[-1]) is NavigableString
+ ):
+ # We are appending a string onto another string.
+ # TODO This has O(n^2) performance, for input like
+ # "a</a>a</a>a</a>..."
+ old_element = self.tag.contents[-1]
+ new_element = self.soup.new_string(old_element + string_child)
+ old_element.replace_with(new_element)
+ self.soup._most_recent_element = new_element
+ else:
+ if isinstance(node, str):
+ # Create a brand new NavigableString from this string.
+ child = self.soup.new_string(node)
+
+ # Tell Beautiful Soup to act as if it parsed this element
+ # immediately after the parent's last descendant. (Or
+ # immediately after the parent, if it has no children.)
+ if self.tag is not None and self.tag.contents:
+ most_recent_element = self.tag._last_descendant(False)
+ elif self.element.next_element is not None:
+ # Something from further ahead in the parse tree is
+ # being inserted into this earlier element. This is
+ # very annoying because it means an expensive search
+ # for the last element in the tree.
+ most_recent_element = self.soup._last_descendant()
+ else:
+ most_recent_element = self.element
+
+ self.soup.object_was_parsed(
+ child, parent=self.tag, most_recent_element=most_recent_element
+ )
+
+ def getAttributes(self) -> AttrList:
+ assert self.tag is not None
+ return AttrList(self.tag)
+
+ # An HTML5lib attribute name may either be a single string,
+ # or a tuple (namespace, name).
+ _Html5libAttributeName: TypeAlias = Union[str, Tuple[str, str]]
+ # Now we can define the type this method accepts as a dictionary
+ # mapping those attribute names to single string values.
+ _Html5libAttributes: TypeAlias = Dict[_Html5libAttributeName, str]
+
+ def setAttributes(self, attributes: Optional[_Html5libAttributes]) -> None:
+ assert self.tag is not None
+ if attributes is not None and len(attributes) > 0:
+ # Replace any namespaced attributes with
+ # NamespacedAttribute objects.
+ for name, value in list(attributes.items()):
+ if isinstance(name, tuple):
+ new_name = NamespacedAttribute(*name)
+ del attributes[name]
+ attributes[new_name] = value
+
+ # We can now cast attributes to the type of Dict
+ # used by Beautiful Soup.
+ normalized_attributes = cast(_AttributeValues, attributes)
+
+ # Values for tags like 'class' came in as single strings;
+ # replace them with lists of strings as appropriate.
+ self.soup.builder._replace_cdata_list_attribute_values(
+ self.name, normalized_attributes
+ )
+
+ # Then set the attributes on the Tag associated with this
+ # BeautifulSoupNode.
+ for name, value_or_values in list(normalized_attributes.items()):
+ self.tag[name] = value_or_values
+
+ # The attributes may contain variables that need substitution.
+ # Call set_up_substitutions manually.
+ #
+ # The Tag constructor called this method when the Tag was created,
+ # but we just set/changed the attributes, so call it again.
+ self.soup.builder.set_up_substitutions(self.tag)
+
+ attributes = property(getAttributes, setAttributes)
+
+ def insertText(
+ self, data: str, insertBefore: Optional["BeautifulSoupNode"] = None
+ ) -> None:
+ text = TextNode(self.soup.new_string(data), self.soup)
+ if insertBefore:
+ self.insertBefore(text, insertBefore)
+ else:
+ self.appendChild(text)
+
+ def insertBefore(
+ self, node: "BeautifulSoupNode", refNode: "BeautifulSoupNode"
+ ) -> None:
+ assert self.tag is not None
+ index = self.tag.index(refNode.element)
+ if (
+ type(node.element) is NavigableString
+ and self.tag.contents
+ and type(self.tag.contents[index - 1]) is NavigableString
+ ):
+ # (See comments in appendChild)
+ old_node = self.tag.contents[index - 1]
+ assert type(old_node) is NavigableString
+ new_str = self.soup.new_string(old_node + node.element)
+ old_node.replace_with(new_str)
+ else:
+ self.tag.insert(index, node.element)
+ node.parent = self
+
+ def removeChild(self, node: "Element") -> None:
+ node.element.extract()
+
+ def reparentChildren(self, newParent: "Element") -> None:
+ """Move all of this tag's children into another tag."""
+ # print("MOVE", self.element.contents)
+ # print("FROM", self.element)
+ # print("TO", new_parent.element)
+
+ element = self.tag
+ assert element is not None
+ new_parent_element = newParent.tag
+ assert new_parent_element is not None
+ # Determine what this tag's next_element will be once all the children
+ # are removed.
+ final_next_element = element.next_sibling
+
+ new_parents_last_descendant = new_parent_element._last_descendant(False, False)
+ if len(new_parent_element.contents) > 0:
+ # The new parent already contains children. We will be
+ # appending this tag's children to the end.
+
+ # We can make this assertion since we know new_parent has
+ # children.
+ assert new_parents_last_descendant is not None
+ new_parents_last_child = new_parent_element.contents[-1]
+ new_parents_last_descendant_next_element = (
+ new_parents_last_descendant.next_element
+ )
+ else:
+ # The new parent contains no children.
+ new_parents_last_child = None
+ new_parents_last_descendant_next_element = new_parent_element.next_element
+
+ to_append = element.contents
+ if len(to_append) > 0:
+ # Set the first child's previous_element and previous_sibling
+ # to elements within the new parent
+ first_child = to_append[0]
+ if new_parents_last_descendant is not None:
+ first_child.previous_element = new_parents_last_descendant
+ else:
+ first_child.previous_element = new_parent_element
+ first_child.previous_sibling = new_parents_last_child
+ if new_parents_last_descendant is not None:
+ new_parents_last_descendant.next_element = first_child
+ else:
+ new_parent_element.next_element = first_child
+ if new_parents_last_child is not None:
+ new_parents_last_child.next_sibling = first_child
+
+ # Find the very last element being moved. It is now the
+ # parent's last descendant. It has no .next_sibling and
+ # its .next_element is whatever the previous last
+ # descendant had.
+ last_childs_last_descendant = to_append[-1]._last_descendant(
+ is_initialized=False, accept_self=True
+ )
+
+ # Since we passed accept_self=True into _last_descendant,
+ # there's no possibility that the result is None.
+ assert last_childs_last_descendant is not None
+ last_childs_last_descendant.next_element = (
+ new_parents_last_descendant_next_element
+ )
+ if new_parents_last_descendant_next_element is not None:
+ # TODO-COVERAGE: This code has no test coverage and
+ # I'm not sure how to get html5lib to go through this
+ # path, but it's just the other side of the previous
+ # line.
+ new_parents_last_descendant_next_element.previous_element = (
+ last_childs_last_descendant
+ )
+ last_childs_last_descendant.next_sibling = None
+
+ for child in to_append:
+ child.parent = new_parent_element
+ new_parent_element.contents.append(child)
+
+ # Now that this element has no children, change its .next_element.
+ element.contents = []
+ element.next_element = final_next_element
+
+ # print("DONE WITH MOVE")
+ # print("FROM", self.element)
+ # print("TO", new_parent_element)
+
+ # TODO-TYPING: typeshed stubs are incorrect about this;
+ # hasContent returns a boolean, not None.
+ def hasContent(self) -> bool: # type:ignore
+ return self.tag is None or len(self.tag.contents) > 0
+
+ # TODO-TYPING: typeshed stubs are incorrect about this;
+ # cloneNode returns a new Node, not None.
+ def cloneNode(self) -> treebuilder_base.Node: # type:ignore
+ assert self.tag is not None
+ tag = self.soup.new_tag(self.tag.name, self.namespace)
+ node = Element(tag, self.soup, self.namespace)
+ for key, value in self.attributes:
+ node.attributes[key] = value
+ return node
+
+ def getNameTuple(self) -> Tuple[Optional[_NamespaceURL], str]:
+ if self.namespace is None:
+ return namespaces["html"], self.name
+ else:
+ return self.namespace, self.name
+
+ nameTuple = property(getNameTuple)
+
+
+class TextNode(BeautifulSoupNode):
+
+ def __init__(self, element: NavigableString, soup: "BeautifulSoup"):
+ treebuilder_base.Node.__init__(self, None)
+ self.tag = None
+ self.string = element
+ self.soup = soup
diff --git a/bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py b/bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py
new file mode 100644
index 00000000..3e916dc7
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py
@@ -0,0 +1,517 @@
+# encoding: utf-8
+"""Use the HTMLParser library to parse HTML files that aren't too bad."""
+from __future__ import annotations
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+__all__ = [
+ "HTMLParserTreeBuilder",
+]
+
+from html.parser import HTMLParser
+import re
+
+from typing import (
+ Any,
+ Callable,
+ cast,
+ Dict,
+ Iterable,
+ List,
+ Optional,
+ TYPE_CHECKING,
+ Tuple,
+ Type,
+ Union,
+)
+
+from bb._vendor.bs4.element import (
+ AttributeDict,
+ CData,
+ Comment,
+ Declaration,
+ Doctype,
+ ProcessingInstruction,
+)
+from bb._vendor.bs4.dammit import EntitySubstitution, UnicodeDammit
+
+from bb._vendor.bs4.builder import (
+ DetectsXMLParsedAsHTML,
+ HTML,
+ HTMLTreeBuilder,
+ STRICT,
+)
+
+from bb._vendor.bs4.exceptions import ParserRejectedMarkup
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4 import BeautifulSoup
+ from bb._vendor.bs4.element import NavigableString
+ from bb._vendor.bs4._typing import (
+ _Encoding,
+ _Encodings,
+ _RawMarkup,
+ )
+
+HTMLPARSER = "html.parser"
+
+_DuplicateAttributeHandler = Callable[[Dict[str, str], str, str], None]
+
+
+class BeautifulSoupHTMLParser(HTMLParser, DetectsXMLParsedAsHTML):
+ #: Constant to handle duplicate attributes by ignoring later values
+ #: and keeping the earlier ones.
+ REPLACE: str = "replace"
+
+ #: Constant to handle duplicate attributes by replacing earlier values
+ #: with later ones.
+ IGNORE: str = "ignore"
+
+ """A subclass of the Python standard library's HTMLParser class, which
+ listens for HTMLParser events and translates them into calls
+ to Beautiful Soup's tree construction API.
+
+ :param on_duplicate_attribute: A strategy for what to do if a
+ tag includes the same attribute more than once. Accepted
+ values are: REPLACE (replace earlier values with later
+ ones, the default), IGNORE (keep the earliest value
+ encountered), or a callable. A callable must take three
+ arguments: the dictionary of attributes already processed,
+ the name of the duplicate attribute, and the most recent value
+ encountered.
+ """
+
+ def __init__(
+ self,
+ soup: BeautifulSoup,
+ *args: Any,
+ on_duplicate_attribute: Union[str, _DuplicateAttributeHandler] = REPLACE,
+ **kwargs: Any,
+ ):
+ self.soup = soup
+ self.on_duplicate_attribute = on_duplicate_attribute
+ self.attribute_dict_class = soup.builder.attribute_dict_class
+ HTMLParser.__init__(self, *args, **kwargs)
+
+ # Keep a list of empty-element tags that were encountered
+ # without an explicit closing tag. If we encounter a closing tag
+ # of this type, we'll associate it with one of those entries.
+ #
+ # This isn't a stack because we don't care about the
+ # order. It's a list of closing tags we've already handled and
+ # will ignore, assuming they ever show up.
+ self.already_closed_empty_element = []
+
+ self._initialize_xml_detector()
+
+ on_duplicate_attribute: Union[str, _DuplicateAttributeHandler]
+ already_closed_empty_element: List[str]
+ soup: BeautifulSoup
+
+ def error(self, message: str) -> None:
+ # NOTE: This method is required so long as Python 3.9 is
+ # supported. The corresponding code is removed from HTMLParser
+ # in 3.5, but not removed from ParserBase until 3.10.
+ # https://github.com/python/cpython/issues/76025
+ #
+ # The original implementation turned the error into a warning,
+ # but in every case I discovered, this made HTMLParser
+ # immediately crash with an error message that was less
+ # helpful than the warning. The new implementation makes it
+ # more clear that html.parser just can't parse this
+ # markup. The 3.10 implementation does the same, though it
+ # raises AssertionError rather than calling a method. (We
+ # catch this error and wrap it in a ParserRejectedMarkup.)
+ raise ParserRejectedMarkup(message)
+
+ def handle_startendtag(
+ self, tag: str, attrs: List[Tuple[str, Optional[str]]]
+ ) -> None:
+ """Handle an incoming empty-element tag.
+
+ html.parser only calls this method when the markup looks like
+ <tag/>.
+ """
+ # `handle_empty_element` tells handle_starttag not to close the tag
+ # just because its name matches a known empty-element tag. We
+ # know that this is an empty-element tag, and we want to call
+ # handle_endtag ourselves.
+ self.handle_starttag(tag, attrs, handle_empty_element=False)
+
+ # Similarly, we set `check_already_closed` when calling
+ # handle_endtag. Since we know the start event is identical to
+ # the end event, we don't want handle_endtag() to cross off
+ # any previous end events for tags of this name.
+ self.handle_endtag(tag, check_already_closed=False)
+
+ def handle_starttag(
+ self,
+ tag: str,
+ attrs: List[Tuple[str, Optional[str]]],
+ handle_empty_element: bool = True,
+ ) -> None:
+ """Handle an opening tag, e.g. '<tag>'
+
+ :param handle_empty_element: True if this tag is known to be
+ an empty-element tag (i.e. there is not expected to be any
+ closing tag).
+ """
+ # TODO: handle namespaces here?
+ attr_dict: AttributeDict = self.attribute_dict_class()
+ for key, value in attrs:
+ # Change None attribute values to the empty string
+ # for consistency with the other tree builders.
+ if value is None:
+ value = ""
+ if key in attr_dict:
+ # A single attribute shows up multiple times in this
+ # tag. How to handle it depends on the
+ # on_duplicate_attribute setting.
+ on_dupe = self.on_duplicate_attribute
+ if on_dupe == self.IGNORE:
+ pass
+ elif on_dupe in (None, self.REPLACE):
+ attr_dict[key] = value
+ else:
+ on_dupe = cast(_DuplicateAttributeHandler, on_dupe)
+ on_dupe(attr_dict, key, value)
+ else:
+ attr_dict[key] = value
+ # print("START", tag)
+ sourceline: Optional[int]
+ sourcepos: Optional[int]
+ if self.soup.builder.store_line_numbers:
+ sourceline, sourcepos = self.getpos()
+ else:
+ sourceline = sourcepos = None
+ tagObj = self.soup.handle_starttag(
+ tag, None, None, attr_dict, sourceline=sourceline, sourcepos=sourcepos
+ )
+ if tagObj is not None and tagObj.is_empty_element and handle_empty_element:
+ # Unlike other parsers, html.parser doesn't send separate end tag
+ # events for empty-element tags. (It's handled in
+ # handle_startendtag, but only if the original markup looked like
+ # <tag/>.)
+ #
+ # So we need to call handle_endtag() ourselves. Since we
+ # know the start event is identical to the end event, we
+ # don't want handle_endtag() to cross off any previous end
+ # events for tags of this name.
+ self.handle_endtag(tag, check_already_closed=False)
+
+ # But we might encounter an explicit closing tag for this tag
+ # later on. If so, we want to ignore it.
+ self.already_closed_empty_element.append(tag)
+
+ if self._root_tag_name is None:
+ self._root_tag_encountered(tag)
+
+ def handle_endtag(self, tag: str, check_already_closed: bool = True) -> None:
+ """Handle a closing tag, e.g. '</tag>'
+
+ :param tag: A tag name.
+ :param check_already_closed: True if this tag is expected to
+ be the closing portion of an empty-element tag,
+ e.g. '<tag></tag>'.
+ """
+ # print("END", tag)
+ if check_already_closed and tag in self.already_closed_empty_element:
+ # This is a redundant end tag for an empty-element tag.
+ # We've already called handle_endtag() for it, so just
+ # check it off the list.
+ # print("ALREADY CLOSED", tag)
+ self.already_closed_empty_element.remove(tag)
+ else:
+ self.soup.handle_endtag(tag)
+
+ def handle_data(self, data: str) -> None:
+ """Handle some textual data that shows up between tags."""
+ self.soup.handle_data(data)
+
+ _DECIMAL_REFERENCE_WITH_FOLLOWING_DATA = re.compile("^([0-9]+)(.*)")
+ _HEX_REFERENCE_WITH_FOLLOWING_DATA = re.compile("^([0-9a-f]+)(.*)")
+
+ @classmethod
+ def _dereference_numeric_character_reference(cls, name:str) -> Tuple[str, bool, str]:
+ """Convert a numeric character reference into an actual character.
+
+ :param name: The number of the character reference, as
+ obtained by html.parser
+
+ :return: A 3-tuple (dereferenced, replacement_added,
+ extra_data). `dereferenced` is the dereferenced character
+ reference, or the empty string if there was no
+ reference. `replacement_added` is True if the reference
+ could only be dereferenced by replacing content with U+FFFD
+ REPLACEMENT CHARACTER. `extra_data` is a portion of data
+ following the character reference, which was deemed to be
+ normal data and not part of the reference at all.
+ """
+ dereferenced:str = ""
+ replacement_added:bool = False
+ extra_data:str = ""
+
+ base:int = 10
+ reg = cls._DECIMAL_REFERENCE_WITH_FOLLOWING_DATA
+ if name.startswith("x") or name.startswith("X"):
+ # Hex reference
+ name = name[1:]
+ base = 16
+ reg = cls._HEX_REFERENCE_WITH_FOLLOWING_DATA
+
+ real_name:Optional[int] = None
+ try:
+ real_name = int(name, base)
+ except ValueError:
+ # This is either bad data that starts with what looks like
+ # a numeric character reference, or a real numeric
+ # reference that wasn't terminated by a semicolon.
+ #
+ # The fix to https://bugs.python.org/issue13633 made it
+ # our responsibility to handle the extra data.
+ #
+ # To preserve the old behavior, we extract the numeric
+ # portion of the incoming "reference" and treat that as a
+ # numeric reference. All subsequent data will be processed
+ # as string data.
+ match = reg.search(name)
+ if match is not None:
+ real_name = int(match.groups()[0], base)
+ extra_data = match.groups()[1]
+
+ if real_name is None:
+ dereferenced = ""
+ extra_data = name
+ else:
+ dereferenced, replacement_added = UnicodeDammit.numeric_character_reference(real_name)
+ return dereferenced, replacement_added, extra_data
+
+ def handle_charref(self, name: str) -> None:
+ """Handle a numeric character reference by converting it to the
+ corresponding Unicode character and treating it as textual
+ data.
+
+ :param name: Character number, possibly in hexadecimal.
+ """
+ dereferenced, replacement_added, extra_data = self._dereference_numeric_character_reference(name)
+ if replacement_added:
+ self.soup.contains_replacement_characters = True
+ if dereferenced is not None:
+ self.handle_data(dereferenced)
+ if extra_data is not None:
+ self.handle_data(extra_data)
+
+ def handle_entityref(self, name: str) -> None:
+ """Handle a named entity reference by converting it to the
+ corresponding Unicode character(s) and treating it as textual
+ data.
+
+ :param name: Name of the entity reference.
+ """
+ character = EntitySubstitution.HTML_ENTITY_TO_CHARACTER.get(name)
+ if character is not None:
+ data = character
+ else:
+ # If this were XML, it would be ambiguous whether "&foo"
+ # was an character entity reference with a missing
+ # semicolon or the literal string "&foo". Since this is
+ # HTML, we have a complete list of all character entity references,
+ # and this one wasn't found, so assume it's the literal string "&foo".
+ data = "&%s" % name
+ self.handle_data(data)
+
+ def handle_comment(self, data: str) -> None:
+ """Handle an HTML comment.
+
+ :param data: The text of the comment.
+ """
+ self.soup.endData()
+ self.soup.handle_data(data)
+ self.soup.endData(Comment)
+
+ def handle_decl(self, decl: str) -> None:
+ """Handle a DOCTYPE declaration.
+
+ :param data: The text of the declaration.
+ """
+ self.soup.endData()
+ decl = decl[len("DOCTYPE ") :]
+ self.soup.handle_data(decl)
+ self.soup.endData(Doctype)
+
+ def unknown_decl(self, data: str) -> None:
+ """Handle a declaration of unknown type -- probably a CDATA block.
+
+ :param data: The text of the declaration.
+ """
+ cls: Type[NavigableString]
+ if data.upper().startswith("CDATA["):
+ cls = CData
+ data = data[len("CDATA[") :]
+ else:
+ cls = Declaration
+ self.soup.endData()
+ self.soup.handle_data(data)
+ self.soup.endData(cls)
+
+ def handle_pi(self, data: str) -> None:
+ """Handle a processing instruction.
+
+ :param data: The text of the instruction.
+ """
+ self.soup.endData()
+ self.soup.handle_data(data)
+ self._document_might_be_xml(data)
+ self.soup.endData(ProcessingInstruction)
+
+
+class HTMLParserTreeBuilder(HTMLTreeBuilder):
+ """A Beautiful soup `bs4.builder.TreeBuilder` that uses the
+ :py:class:`html.parser.HTMLParser` parser, found in the Python
+ standard library.
+
+ """
+
+ is_xml: bool = False
+ picklable: bool = True
+ NAME: str = HTMLPARSER
+ features: Iterable[str] = [NAME, HTML, STRICT]
+ parser_args: Tuple[Iterable[Any], Dict[str, Any]]
+
+ #: The html.parser knows which line number and position in the
+ #: original file is the source of an element.
+ TRACKS_LINE_NUMBERS: bool = True
+
+ def __init__(
+ self,
+ parser_args: Optional[Iterable[Any]] = None,
+ parser_kwargs: Optional[Dict[str, Any]] = None,
+ **kwargs: Any,
+ ):
+ """Constructor.
+
+ :param parser_args: Positional arguments to pass into
+ the BeautifulSoupHTMLParser constructor, once it's
+ invoked.
+ :param parser_kwargs: Keyword arguments to pass into
+ the BeautifulSoupHTMLParser constructor, once it's
+ invoked.
+ :param kwargs: Keyword arguments for the superclass constructor.
+ """
+ # Some keyword arguments will be pulled out of kwargs and placed
+ # into parser_kwargs.
+ extra_parser_kwargs = dict()
+ for arg in ("on_duplicate_attribute",):
+ if arg in kwargs:
+ value = kwargs.pop(arg)
+ extra_parser_kwargs[arg] = value
+ super(HTMLParserTreeBuilder, self).__init__(**kwargs)
+ parser_args = parser_args or []
+ parser_kwargs = parser_kwargs or {}
+ parser_kwargs.update(extra_parser_kwargs)
+ parser_kwargs["convert_charrefs"] = False
+ self.parser_args = (parser_args, parser_kwargs)
+
+ def prepare_markup(
+ self,
+ markup: _RawMarkup,
+ user_specified_encoding: Optional[_Encoding] = None,
+ document_declared_encoding: Optional[_Encoding] = None,
+ exclude_encodings: Optional[_Encodings] = None,
+ ) -> Iterable[Tuple[str, Optional[_Encoding], Optional[_Encoding], bool]]:
+ """Run any preliminary steps necessary to make incoming markup
+ acceptable to the parser.
+
+ :param markup: Some markup -- probably a bytestring.
+ :param user_specified_encoding: The user asked to try this encoding.
+ :param document_declared_encoding: The markup itself claims to be
+ in this encoding.
+ :param exclude_encodings: The user asked _not_ to try any of
+ these encodings.
+
+ :yield: A series of 4-tuples: (markup, encoding, declared encoding,
+ has undergone character replacement)
+
+ Each 4-tuple represents a strategy for parsing the document.
+ This TreeBuilder uses Unicode, Dammit to convert the markup
+ into Unicode, so the ``markup`` element of the tuple will
+ always be a string.
+ """
+ if isinstance(markup, str):
+ # Parse Unicode as-is.
+ yield (markup, None, None, False)
+ return
+
+ # Ask UnicodeDammit to sniff the most likely encoding.
+
+ known_definite_encodings: List[_Encoding] = []
+ if user_specified_encoding:
+ # This was provided by the end-user; treat it as a known
+ # definite encoding per the algorithm laid out in the
+ # HTML5 spec. (See the EncodingDetector class for
+ # details.)
+ known_definite_encodings.append(user_specified_encoding)
+
+ user_encodings: List[_Encoding] = []
+ if document_declared_encoding:
+ # This was found in the document; treat it as a slightly
+ # lower-priority user encoding.
+ user_encodings.append(document_declared_encoding)
+
+ dammit = UnicodeDammit(
+ markup,
+ known_definite_encodings=known_definite_encodings,
+ user_encodings=user_encodings,
+ is_html=True,
+ exclude_encodings=exclude_encodings,
+ )
+
+ if dammit.unicode_markup is None:
+ # In every case I've seen, Unicode, Dammit is able to
+ # convert the markup into Unicode, even if it needs to use
+ # REPLACEMENT CHARACTER. But there is a code path that
+ # could result in unicode_markup being None, and
+ # HTMLParser can only parse Unicode, so here we handle
+ # that code path.
+ raise ParserRejectedMarkup(
+ "Could not convert input to Unicode, and html.parser will not accept bytestrings."
+ )
+ else:
+ yield (
+ dammit.unicode_markup,
+ dammit.original_encoding,
+ dammit.declared_html_encoding,
+ dammit.contains_replacement_characters,
+ )
+
+ def feed(self, markup: _RawMarkup, _parser_class:type[BeautifulSoupHTMLParser] =BeautifulSoupHTMLParser) -> None:
+ """
+ :param markup: The markup to feed into the parser.
+ :param _parser_class: An HTMLParser subclass to use. This is only intended for use in unit tests.
+ """
+ args, kwargs = self.parser_args
+
+ # HTMLParser.feed will only handle str, but
+ # BeautifulSoup.markup is allowed to be _RawMarkup, because
+ # it's set by the yield value of
+ # TreeBuilder.prepare_markup. Fortunately,
+ # HTMLParserTreeBuilder.prepare_markup always yields a str
+ # (UnicodeDammit.unicode_markup).
+ assert isinstance(markup, str)
+
+ # We know BeautifulSoup calls TreeBuilder.initialize_soup
+ # before calling feed(), so we can assume self.soup
+ # is set.
+ assert self.soup is not None
+ parser = _parser_class(self.soup, *args, **kwargs)
+
+ try:
+ parser.feed(markup)
+ parser.close()
+ except AssertionError as e:
+ # html.parser raises AssertionError in rare cases to
+ # indicate a fatal problem with the markup, especially
+ # when there's an error in the doctype declaration.
+ raise ParserRejectedMarkup(e)
+ parser.already_closed_empty_element = []
diff --git a/bitbake/lib/bb/_vendor/bs4/builder/_lxml.py b/bitbake/lib/bb/_vendor/bs4/builder/_lxml.py
new file mode 100644
index 00000000..79ed00e6
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/builder/_lxml.py
@@ -0,0 +1,501 @@
+# encoding: utf-8
+from __future__ import annotations
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+__all__ = [
+ "LXMLTreeBuilderForXML",
+ "LXMLTreeBuilder",
+]
+
+
+from typing import (
+ Any,
+ Dict,
+ Iterable,
+ List,
+ Optional,
+ Set,
+ Tuple,
+ Type,
+ TYPE_CHECKING,
+ Union,
+)
+
+from io import BytesIO
+from io import StringIO
+
+from bb._vendor.typing_extensions import TypeAlias
+
+from lxml import etree # type:ignore
+from bb._vendor.bs4.element import (
+ AttributeDict,
+ XMLAttributeDict,
+ Comment,
+ Doctype,
+ NamespacedAttribute,
+ ProcessingInstruction,
+ XMLProcessingInstruction,
+)
+from bb._vendor.bs4.builder import (
+ DetectsXMLParsedAsHTML,
+ FAST,
+ HTML,
+ HTMLTreeBuilder,
+ PERMISSIVE,
+ TreeBuilder,
+ XML,
+)
+from bb._vendor.bs4.dammit import EncodingDetector
+from bb._vendor.bs4.exceptions import ParserRejectedMarkup
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4._typing import (
+ _Encoding,
+ _Encodings,
+ _NamespacePrefix,
+ _NamespaceURL,
+ _NamespaceMapping,
+ _InvertedNamespaceMapping,
+ _RawMarkup,
+ )
+ from bb._vendor.bs4 import BeautifulSoup
+
+LXML: str = "lxml"
+
+
+def _invert(d: dict[Any, Any]) -> dict[Any, Any]:
+ "Invert a dictionary."
+ return dict((v, k) for k, v in list(d.items()))
+
+
+_LXMLParser: TypeAlias = Union[etree.XMLParser, etree.HTMLParser]
+_ParserOrParserClass: TypeAlias = Union[
+ _LXMLParser, Type[etree.XMLParser], Type[etree.HTMLParser]
+]
+
+
+class LXMLTreeBuilderForXML(TreeBuilder):
+ DEFAULT_PARSER_CLASS: Type[etree.XMLParser] = etree.XMLParser
+
+ is_xml: bool = True
+
+ #: Set this to true (probably by passing huge_tree=True into the :
+ #: BeautifulSoup constructor) to enable the lxml feature "disable security
+ #: restrictions and support very deep trees and very long text
+ #: content".
+ huge_tree: bool
+
+ processing_instruction_class: Type[ProcessingInstruction]
+
+ NAME: str = "lxml-xml"
+ ALTERNATE_NAMES: Iterable[str] = ["xml"]
+
+ # Well, it's permissive by XML parser standards.
+ features: Iterable[str] = [NAME, LXML, XML, FAST, PERMISSIVE]
+
+ CHUNK_SIZE: int = 512
+
+ # This namespace mapping is specified in the XML Namespace
+ # standard.
+ DEFAULT_NSMAPS: _NamespaceMapping = dict(xml="http://www.w3.org/XML/1998/namespace")
+
+ DEFAULT_NSMAPS_INVERTED: _InvertedNamespaceMapping = _invert(DEFAULT_NSMAPS)
+
+ nsmaps: List[Optional[_InvertedNamespaceMapping]]
+ empty_element_tags: Optional[Set[str]]
+ parser: Any
+ _default_parser: Optional[etree.XMLParser]
+
+ # NOTE: If we parsed Element objects and looked at .sourceline,
+ # we'd be able to see the line numbers from the original document.
+ # But instead we build an XMLParser or HTMLParser object to serve
+ # as the target of parse messages, and those messages don't include
+ # line numbers.
+ # See: https://bugs.launchpad.net/lxml/+bug/1846906
+
+ def initialize_soup(self, soup: BeautifulSoup) -> None:
+ """Let the BeautifulSoup object know about the standard namespace
+ mapping.
+
+ :param soup: A `BeautifulSoup`.
+ """
+ # Beyond this point, self.soup is set, so we can assume (and
+ # assert) it's not None whenever necessary.
+ super(LXMLTreeBuilderForXML, self).initialize_soup(soup)
+ self._register_namespaces(self.DEFAULT_NSMAPS)
+
+ def _register_namespaces(self, mapping: Dict[str, str]) -> None:
+ """Let the BeautifulSoup object know about namespaces encountered
+ while parsing the document.
+
+ This might be useful later on when creating CSS selectors.
+
+ This will track (almost) all namespaces, even ones that were
+ only in scope for part of the document. If two namespaces have
+ the same prefix, only the first one encountered will be
+ tracked. Un-prefixed namespaces are not tracked.
+
+ :param mapping: A dictionary mapping namespace prefixes to URIs.
+ """
+ assert self.soup is not None
+ for key, value in list(mapping.items()):
+ # This is 'if key' and not 'if key is not None' because we
+ # don't track un-prefixed namespaces. Soupselect will
+ # treat an un-prefixed namespace as the default, which
+ # causes confusion in some cases.
+ if key and key not in self.soup._namespaces:
+ # Let the BeautifulSoup object know about a new namespace.
+ # If there are multiple namespaces defined with the same
+ # prefix, the first one in the document takes precedence.
+ self.soup._namespaces[key] = value
+
+ def default_parser(self, encoding: Optional[_Encoding]) -> _ParserOrParserClass:
+ """Find the default parser for the given encoding.
+
+ :return: Either a parser object or a class, which
+ will be instantiated with default arguments.
+ """
+ if self._default_parser is not None:
+ return self._default_parser
+ return self.DEFAULT_PARSER_CLASS(target=self, recover=True, huge_tree=self.huge_tree, encoding=encoding)
+
+ def parser_for(self, encoding: Optional[_Encoding]) -> _LXMLParser:
+ """Instantiate an appropriate parser for the given encoding.
+
+ :param encoding: A string.
+ :return: A parser object such as an `etree.XMLParser`.
+ """
+ # Use the default parser.
+ parser = self.default_parser(encoding)
+
+ if callable(parser):
+ # Instantiate the parser with default arguments
+ parser = parser(target=self, recover=True, huge_tree=self.huge_tree, encoding=encoding)
+ return parser
+
+ def __init__(
+ self,
+ parser: Optional[etree.XMLParser] = None,
+ empty_element_tags: Optional[Set[str]] = None,
+ huge_tree: bool = False,
+ **kwargs: Any,
+ ):
+ # TODO: Issue a warning if parser is present but not a
+ # callable, since that means there's no way to create new
+ # parsers for different encodings.
+ self._default_parser = parser
+ self.soup = None
+ self.nsmaps = [self.DEFAULT_NSMAPS_INVERTED]
+ self.active_namespace_prefixes = [dict(self.DEFAULT_NSMAPS)]
+ if self.is_xml:
+ self.processing_instruction_class = XMLProcessingInstruction
+ else:
+ self.processing_instruction_class = ProcessingInstruction
+
+ if "attribute_dict_class" not in kwargs:
+ kwargs["attribute_dict_class"] = XMLAttributeDict
+ self.huge_tree = huge_tree
+
+ super(LXMLTreeBuilderForXML, self).__init__(**kwargs)
+
+ def _getNsTag(self, tag: str) -> Tuple[Optional[str], str]:
+ # Split the namespace URL out of a fully-qualified lxml tag
+ # name. Copied from lxml's src/lxml/sax.py.
+ if tag[0] == "{" and "}" in tag:
+ namespace, name = tag[1:].split("}", 1)
+ return (namespace, name)
+ return (None, tag)
+
+ def prepare_markup(
+ self,
+ markup: _RawMarkup,
+ user_specified_encoding: Optional[_Encoding] = None,
+ document_declared_encoding: Optional[_Encoding] = None,
+ exclude_encodings: Optional[_Encodings] = None,
+ ) -> Iterable[
+ Tuple[Union[str, bytes], Optional[_Encoding], Optional[_Encoding], bool]
+ ]:
+ """Run any preliminary steps necessary to make incoming markup
+ acceptable to the parser.
+
+ lxml really wants to get a bytestring and convert it to
+ Unicode itself. So instead of using UnicodeDammit to convert
+ the bytestring to Unicode using different encodings, this
+ implementation uses EncodingDetector to iterate over the
+ encodings, and tell lxml to try to parse the document as each
+ one in turn.
+
+ :param markup: Some markup -- hopefully a bytestring.
+ :param user_specified_encoding: The user asked to try this encoding.
+ :param document_declared_encoding: The markup itself claims to be
+ in this encoding.
+ :param exclude_encodings: The user asked _not_ to try any of
+ these encodings.
+
+ :yield: A series of 4-tuples: (markup, encoding, declared encoding,
+ has undergone character replacement)
+
+ Each 4-tuple represents a strategy for converting the
+ document to Unicode and parsing it. Each strategy will be tried
+ in turn.
+ """
+ if not self.is_xml:
+ # We're in HTML mode, so if we're given XML, that's worth
+ # noting.
+ DetectsXMLParsedAsHTML.warn_if_markup_looks_like_xml(markup, stacklevel=3)
+
+ if isinstance(markup, str):
+ # We were given Unicode. Maybe lxml can parse Unicode on
+ # this system?
+
+ # TODO: This is a workaround for
+ # https://bugs.launchpad.net/lxml/+bug/1948551.
+ # We can remove it once the upstream issue is fixed.
+ if len(markup) > 0 and markup[0] == "\N{BYTE ORDER MARK}":
+ markup = markup[1:]
+ yield markup, None, document_declared_encoding, False
+
+ if isinstance(markup, str):
+ # No, apparently not. Convert the Unicode to UTF-8 and
+ # tell lxml to parse it as UTF-8.
+ yield (markup.encode("utf8"), "utf8", document_declared_encoding, False)
+
+ # Since the document was Unicode in the first place, there
+ # is no need to try any more strategies; we know this will
+ # work.
+ return
+
+ known_definite_encodings: List[_Encoding] = []
+ if user_specified_encoding:
+ # This was provided by the end-user; treat it as a known
+ # definite encoding per the algorithm laid out in the
+ # HTML5 spec. (See the EncodingDetector class for
+ # details.)
+ known_definite_encodings.append(user_specified_encoding)
+
+ user_encodings: List[_Encoding] = []
+ if document_declared_encoding:
+ # This was found in the document; treat it as a slightly
+ # lower-priority user encoding.
+ user_encodings.append(document_declared_encoding)
+
+ detector = EncodingDetector(
+ markup,
+ known_definite_encodings=known_definite_encodings,
+ user_encodings=user_encodings,
+ is_html=not self.is_xml,
+ exclude_encodings=exclude_encodings,
+ )
+ for encoding in detector.encodings:
+ yield (detector.markup, encoding, document_declared_encoding, False)
+
+ def feed(self, markup: _RawMarkup) -> None:
+ io: Union[BytesIO, StringIO]
+ if isinstance(markup, bytes):
+ io = BytesIO(markup)
+ elif isinstance(markup, str):
+ io = StringIO(markup)
+
+ # initialize_soup is called before feed, so we know this
+ # is not None.
+ assert self.soup is not None
+
+ # Call feed() at least once, even if the markup is empty,
+ # or the parser won't be initialized.
+ data = io.read(self.CHUNK_SIZE)
+ try:
+ self.parser = self.parser_for(self.soup.original_encoding)
+ self.parser.feed(data)
+ while len(data) != 0:
+ # Now call feed() on the rest of the data, chunk by chunk.
+ data = io.read(self.CHUNK_SIZE)
+ if len(data) != 0:
+ self.parser.feed(data)
+ self.parser.close()
+ except (UnicodeDecodeError, LookupError, etree.ParserError) as e:
+ raise ParserRejectedMarkup(e)
+
+ def close(self) -> None:
+ self.nsmaps = [self.DEFAULT_NSMAPS_INVERTED]
+
+ def start(
+ self,
+ tag: str | bytes,
+ attrib: Dict[str | bytes, str | bytes],
+ nsmap: _NamespaceMapping = {},
+ ) -> None:
+ # This is called by lxml code as a result of calling
+ # BeautifulSoup.feed(), and we know self.soup is set by the time feed()
+ # is called.
+ assert self.soup is not None
+ assert isinstance(tag, str)
+
+ # We need to recreate the attribute dict for three
+ # reasons. First, for type checking, so we can assert there
+ # are no bytestrings in the keys or values. Second, because we
+ # need a mutable dict--lxml might send us an immutable
+ # dictproxy. Third, so we can handle namespaced attribute
+ # names by converting the keys to NamespacedAttributes.
+ new_attrib: Dict[Union[str, NamespacedAttribute], str] = (
+ self.attribute_dict_class()
+ )
+ for k, v in attrib.items():
+ assert isinstance(k, str)
+ assert isinstance(v, str)
+ new_attrib[k] = v
+
+ nsprefix: Optional[_NamespacePrefix] = None
+ namespace: Optional[_NamespaceURL] = None
+ # Invert each namespace map as it comes in.
+ if len(nsmap) == 0 and len(self.nsmaps) > 1:
+ # There are no new namespaces for this tag, but
+ # non-default namespaces are in play, so we need a
+ # separate tag stack to know when they end.
+ self.nsmaps.append(None)
+ elif len(nsmap) > 0:
+ # A new namespace mapping has come into play.
+
+ # First, Let the BeautifulSoup object know about it.
+ self._register_namespaces(nsmap)
+
+ # Then, add it to our running list of inverted namespace
+ # mappings.
+ self.nsmaps.append(_invert(nsmap))
+
+ # The currently active namespace prefixes have
+ # changed. Calculate the new mapping so it can be stored
+ # with all Tag objects created while these prefixes are in
+ # scope.
+ current_mapping = dict(self.active_namespace_prefixes[-1])
+ current_mapping.update(nsmap)
+
+ # We should not track un-prefixed namespaces as we can only hold one
+ # and it will be recognized as the default namespace by soupsieve,
+ # which may be confusing in some situations.
+ if "" in current_mapping:
+ del current_mapping[""]
+ self.active_namespace_prefixes.append(current_mapping)
+
+ # Also treat the namespace mapping as a set of attributes on the
+ # tag, so we can recreate it later.
+ for prefix, namespace in list(nsmap.items()):
+ attribute = NamespacedAttribute(
+ "xmlns", prefix, "http://www.w3.org/2000/xmlns/"
+ )
+ new_attrib[attribute] = namespace
+
+ # Namespaces are in play. Find any attributes that came in
+ # from lxml with namespaces attached to their names, and
+ # turn then into NamespacedAttribute objects.
+ final_attrib: AttributeDict = self.attribute_dict_class()
+ for attr, value in list(new_attrib.items()):
+ namespace, attr = self._getNsTag(attr)
+ if namespace is None:
+ final_attrib[attr] = value
+ else:
+ nsprefix = self._prefix_for_namespace(namespace)
+ attr = NamespacedAttribute(nsprefix, attr, namespace)
+ final_attrib[attr] = value
+
+ namespace, tag = self._getNsTag(tag)
+ nsprefix = self._prefix_for_namespace(namespace)
+ self.soup.handle_starttag(
+ tag,
+ namespace,
+ nsprefix,
+ final_attrib,
+ namespaces=self.active_namespace_prefixes[-1],
+ )
+
+ def _prefix_for_namespace(
+ self, namespace: Optional[_NamespaceURL]
+ ) -> Optional[_NamespacePrefix]:
+ """Find the currently active prefix for the given namespace."""
+ if namespace is None:
+ return None
+ for inverted_nsmap in reversed(self.nsmaps):
+ if inverted_nsmap is not None and namespace in inverted_nsmap:
+ return inverted_nsmap[namespace]
+ return None
+
+ def end(self, tag: str | bytes) -> None:
+ assert self.soup is not None
+ assert isinstance(tag, str)
+ self.soup.endData()
+ namespace, tag = self._getNsTag(tag)
+ nsprefix = None
+ if namespace is not None:
+ for inverted_nsmap in reversed(self.nsmaps):
+ if inverted_nsmap is not None and namespace in inverted_nsmap:
+ nsprefix = inverted_nsmap[namespace]
+ break
+ self.soup.handle_endtag(tag, nsprefix)
+ if len(self.nsmaps) > 1:
+ # This tag, or one of its parents, introduced a namespace
+ # mapping, so pop it off the stack.
+ out_of_scope_nsmap = self.nsmaps.pop()
+
+ if out_of_scope_nsmap is not None:
+ # This tag introduced a namespace mapping which is no
+ # longer in scope. Recalculate the currently active
+ # namespace prefixes.
+ self.active_namespace_prefixes.pop()
+
+ def pi(self, target: str, data: str) -> None:
+ assert self.soup is not None
+ self.soup.endData()
+ data = target + " " + data
+ self.soup.handle_data(data)
+ self.soup.endData(self.processing_instruction_class)
+
+ def data(self, data: str | bytes) -> None:
+ assert self.soup is not None
+ assert isinstance(data, str)
+ self.soup.handle_data(data)
+
+ def doctype(self, name: str, pubid: str, system: str) -> None:
+ assert self.soup is not None
+ self.soup.endData()
+ doctype_string = Doctype._string_for_name_and_ids(name, pubid, system)
+ self.soup.handle_data(doctype_string)
+ self.soup.endData(containerClass=Doctype)
+
+ def comment(self, text: str | bytes) -> None:
+ "Handle comments as Comment objects."
+ assert self.soup is not None
+ assert isinstance(text, str)
+ self.soup.endData()
+ self.soup.handle_data(text)
+ self.soup.endData(Comment)
+
+ def test_fragment_to_document(self, fragment: str) -> str:
+ """See `TreeBuilder`."""
+ return '<?xml version="1.0" encoding="utf-8"?>\n%s' % fragment
+
+
+class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
+ NAME: str = LXML
+ ALTERNATE_NAMES: Iterable[str] = ["lxml-html"]
+
+ features: Iterable[str] = list(ALTERNATE_NAMES) + [NAME, HTML, FAST, PERMISSIVE]
+ is_xml: bool = False
+
+ def default_parser(self, encoding: Optional[_Encoding]) -> _ParserOrParserClass:
+ return etree.HTMLParser
+
+ def feed(self, markup: _RawMarkup) -> None:
+ # We know self.soup is set by the time feed() is called.
+ assert self.soup is not None
+ encoding = self.soup.original_encoding
+ try:
+ self.parser = self.parser_for(encoding)
+ self.parser.feed(markup)
+ self.parser.close()
+ except (UnicodeDecodeError, LookupError, etree.ParserError) as e:
+ raise ParserRejectedMarkup(e)
+
+ def test_fragment_to_document(self, fragment: str) -> str:
+ """See `TreeBuilder`."""
+ return "<html><body>%s</body></html>" % fragment
diff --git a/bitbake/lib/bb/_vendor/bs4/css.py b/bitbake/lib/bb/_vendor/bs4/css.py
new file mode 100644
index 00000000..b6e73c2e
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/css.py
@@ -0,0 +1,330 @@
+"""Integration code for CSS selectors using `Soup Sieve <https://facelessuser.github.io/soupsieve/>`_ (pypi: ``soupsieve``).
+
+Acquire a `CSS` object through the `element.Tag.css` attribute of
+the starting point of your CSS selector, or (if you want to run a
+selector against the entire document) of the `BeautifulSoup` object
+itself.
+
+The main advantage of doing this instead of using ``soupsieve``
+functions is that you don't need to keep passing the `element.Tag` to be
+selected against, since the `CSS` object is permanently scoped to that
+`element.Tag`.
+
+"""
+
+from __future__ import annotations
+
+from types import ModuleType
+from typing import (
+ Any,
+ cast,
+ Iterable,
+ Iterator,
+ MutableSequence,
+ Optional,
+ TYPE_CHECKING,
+)
+from bb._vendor.bs4._typing import _NamespaceMapping
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4 import element
+ from bb._vendor.bs4.element import ResultSet, Tag
+
+# We don't use soupsieve
+soupsieve = None
+
+class CSS(object):
+ """A proxy object against the ``soupsieve`` library, to simplify its
+ CSS selector API.
+
+ You don't need to instantiate this class yourself; instead, use
+ `element.Tag.css`.
+
+ :param tag: All CSS selectors run by this object will use this as
+ their starting point.
+
+ :param api: An optional drop-in replacement for the ``soupsieve`` module,
+ intended for use in unit tests.
+ """
+
+ def __init__(self, tag: element.Tag, api: Optional[ModuleType] = None):
+ if api is None:
+ api = soupsieve
+ if api is None:
+ raise NotImplementedError(
+ "Cannot execute CSS selectors because the soupsieve package is not installed."
+ )
+ self.api = api
+ self.tag = tag
+
+ def escape(self, ident: str) -> str:
+ """Escape a CSS identifier.
+
+ This is a simple wrapper around `soupsieve.escape() <https://facelessuser.github.io/soupsieve/api/#soupsieveescape>`_. See the
+ documentation for that function for more information.
+ """
+ if soupsieve is None:
+ raise NotImplementedError(
+ "Cannot escape CSS identifiers because the soupsieve package is not installed."
+ )
+ return cast(str, self.api.escape(ident))
+
+ def _ns(
+ self, ns: Optional[_NamespaceMapping], select: str
+ ) -> Optional[_NamespaceMapping]:
+ """Normalize a dictionary of namespaces."""
+ if not isinstance(select, self.api.SoupSieve) and ns is None:
+ # If the selector is a precompiled pattern, it already has
+ # a namespace context compiled in, which cannot be
+ # replaced.
+ ns = self.tag._namespaces
+ return ns
+
+ def _rs(self, results: MutableSequence[Tag]) -> ResultSet[Tag]:
+ """Normalize a list of results to a py:class:`ResultSet`.
+
+ A py:class:`ResultSet` is more consistent with the rest of
+ Beautiful Soup's API, and :py:meth:`ResultSet.__getattr__` has
+ a helpful error message if you try to treat a list of results
+ as a single result (a common mistake).
+ """
+ # Import here to avoid circular import
+ from bb._vendor.bs4 import ResultSet
+
+ return ResultSet(None, results)
+
+ def compile(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> SoupSieve:
+ """Pre-compile a selector and return the compiled object.
+
+ :param selector: A CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will use the prefixes it encountered while
+ parsing the document.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.compile() <https://facelessuser.github.io/soupsieve/api/#soupsievecompile>`_ method.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ `soupsieve.compile() <https://facelessuser.github.io/soupsieve/api/#soupsievecompile>`_ method.
+
+ :return: A precompiled selector object.
+ :rtype: soupsieve.SoupSieve
+ """
+ return self.api.compile(select, self._ns(namespaces, select), flags, **kwargs)
+
+ def select_one(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> element.Tag | None:
+ """Perform a CSS selection operation on the current Tag and return the
+ first result, if any.
+
+ This uses the Soup Sieve library. For more information, see
+ that library's documentation for the `soupsieve.select_one() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect_one>`_ method.
+
+ :param selector: A CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will use the prefixes it encountered while
+ parsing the document.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.select_one() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect_one>`_ method.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ `soupsieve.select_one() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect_one>`_ method.
+ """
+ return self.api.select_one(
+ select, self.tag, self._ns(namespaces, select), flags, **kwargs
+ )
+
+ def select(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ limit: int = 0,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> ResultSet[element.Tag]:
+ """Perform a CSS selection operation on the current `element.Tag`.
+
+ This uses the Soup Sieve library. For more information, see
+ that library's documentation for the `soupsieve.select() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect>`_ method.
+
+ :param selector: A CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will pass in the prefixes it encountered while
+ parsing the document.
+
+ :param limit: After finding this number of results, stop looking.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.select() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect>`_ method.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ `soupsieve.select() <https://facelessuser.github.io/soupsieve/api/#soupsieveselect>`_ method.
+ """
+ if limit is None:
+ limit = 0
+
+ return self._rs(
+ self.api.select(
+ select, self.tag, self._ns(namespaces, select), limit, flags, **kwargs
+ )
+ )
+
+ def iselect(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ limit: int = 0,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> Iterator[element.Tag]:
+ """Perform a CSS selection operation on the current `element.Tag`.
+
+ This uses the Soup Sieve library. For more information, see
+ that library's documentation for the `soupsieve.iselect()
+ <https://facelessuser.github.io/soupsieve/api/#soupsieveiselect>`_
+ method. It is the same as select(), but it returns a generator
+ instead of a list.
+
+ :param selector: A string containing a CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will pass in the prefixes it encountered while
+ parsing the document.
+
+ :param limit: After finding this number of results, stop looking.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.iselect() <https://facelessuser.github.io/soupsieve/api/#soupsieveiselect>`_ method.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ `soupsieve.iselect() <https://facelessuser.github.io/soupsieve/api/#soupsieveiselect>`_ method.
+ """
+ return self.api.iselect(
+ select, self.tag, self._ns(namespaces, select), limit, flags, **kwargs
+ )
+
+ def closest(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> Optional[element.Tag]:
+ """Find the `element.Tag` closest to this one that matches the given selector.
+
+ This uses the Soup Sieve library. For more information, see
+ that library's documentation for the `soupsieve.closest()
+ <https://facelessuser.github.io/soupsieve/api/#soupsieveclosest>`_
+ method.
+
+ :param selector: A string containing a CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will pass in the prefixes it encountered while
+ parsing the document.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.closest() <https://facelessuser.github.io/soupsieve/api/#soupsieveclosest>`_ method.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ `soupsieve.closest() <https://facelessuser.github.io/soupsieve/api/#soupsieveclosest>`_ method.
+
+ """
+ return self.api.closest(
+ select, self.tag, self._ns(namespaces, select), flags, **kwargs
+ )
+
+ def match(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> bool:
+ """Check whether or not this `element.Tag` matches the given CSS selector.
+
+ This uses the Soup Sieve library. For more information, see
+ that library's documentation for the `soupsieve.match()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievematch>`_
+ method.
+
+ :param: a CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will pass in the prefixes it encountered while
+ parsing the document.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.match()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievematch>`_
+ method.
+
+ :param kwargs: Keyword arguments to be passed into SoupSieve's
+ `soupsieve.match()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievematch>`_
+ method.
+ """
+ return cast(
+ bool,
+ self.api.match(
+ select, self.tag, self._ns(namespaces, select), flags, **kwargs
+ ),
+ )
+
+ def filter(
+ self,
+ select: str,
+ namespaces: Optional[_NamespaceMapping] = None,
+ flags: int = 0,
+ **kwargs: Any,
+ ) -> ResultSet[element.Tag]:
+ """Filter this `element.Tag`'s direct children based on the given CSS selector.
+
+ This uses the Soup Sieve library. It works the same way as
+ passing a `element.Tag` into that library's `soupsieve.filter()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievefilter>`_
+ method. For more information, see the documentation for
+ `soupsieve.filter()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievefilter>`_.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will pass in the prefixes it encountered while
+ parsing the document.
+
+ :param flags: Flags to be passed into Soup Sieve's
+ `soupsieve.filter()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievefilter>`_
+ method.
+
+ :param kwargs: Keyword arguments to be passed into SoupSieve's
+ `soupsieve.filter()
+ <https://facelessuser.github.io/soupsieve/api/#soupsievefilter>`_
+ method.
+ """
+ return self._rs(
+ self.api.filter(
+ select, self.tag, self._ns(namespaces, select), flags, **kwargs
+ )
+ )
diff --git a/bitbake/lib/bb/_vendor/bs4/dammit.py b/bitbake/lib/bb/_vendor/bs4/dammit.py
new file mode 100644
index 00000000..cfd05ab1
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/dammit.py
@@ -0,0 +1,1525 @@
+# -*- coding: utf-8 -*-
+"""Beautiful Soup bonus library: Unicode, Dammit
+
+This library converts a bytestream to Unicode through any means
+necessary. It is heavily based on code from Mark Pilgrim's `Universal
+Feed Parser <https://pypi.org/project/feedparser/>`_, now maintained
+by Kurt McKee. It does not rewrite the body of an XML or HTML document
+to reflect a new encoding; that's the job of `TreeBuilder`.
+
+"""
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+from html.entities import codepoint2name
+from collections import defaultdict
+import codecs
+from html.entities import html5
+import re
+from logging import Logger, getLogger
+from types import ModuleType
+from typing import (
+ Dict,
+ Iterator,
+ List,
+ Optional,
+ Pattern,
+ Set,
+ Tuple,
+ Type,
+ Union,
+ cast,
+)
+from bb._vendor.typing_extensions import Literal
+from bb._vendor.bs4._typing import (
+ _Encoding,
+ _Encodings,
+)
+import warnings
+
+# Import a library to autodetect character encodings. We'll support
+# any of a number of libraries that all support the same API:
+#
+# * cchardet
+# * chardet
+# * charset-normalizer
+chardet_module: Optional[ModuleType] = None
+try:
+ # PyPI package: cchardet
+ import cchardet # type:ignore
+
+ chardet_module = cchardet
+except ImportError:
+ try:
+ # Debian package: python-chardet
+ # PyPI package: chardet
+ import chardet
+
+ chardet_module = chardet
+ except ImportError:
+ try:
+ # PyPI package: charset-normalizer
+ import charset_normalizer # type:ignore
+
+ chardet_module = charset_normalizer
+ except ImportError:
+ # No chardet available.
+ pass
+
+
+def _chardet_dammit(s: bytes) -> Optional[str]:
+ """Try as hard as possible to detect the encoding of a bytestring."""
+ if chardet_module is None or isinstance(s, str):
+ return None
+ module = chardet_module
+ return module.detect(s)["encoding"]
+
+
+# Build bytestring and Unicode versions of regular expressions for finding
+# a declared encoding inside an XML or HTML document.
+xml_encoding: str = "^\\s*<\\?.*encoding=['\"](.*?)['\"].*\\?>" #: :meta private:
+html_meta: str = (
+ "<\\s*meta[^>]+charset\\s*=\\s*[\"']?([^>]*?)[ /;'\">]" #: :meta private:
+)
+
+# TODO-TYPING: The Pattern type here could use more refinement, but it's tricky.
+encoding_res: Dict[Type, Dict[str, Pattern]] = dict()
+encoding_res[bytes] = {
+ "html": re.compile(html_meta.encode("ascii"), re.I),
+ "xml": re.compile(xml_encoding.encode("ascii"), re.I),
+}
+encoding_res[str] = {
+ "html": re.compile(html_meta, re.I),
+ "xml": re.compile(xml_encoding, re.I),
+}
+
+
+class EntitySubstitutionMeta(type):
+ """Provides lazy access to some data structures and regular
+ expressions used by EntitySubstitution which have a measurable
+ startup cost.
+ """
+ # Trigger for
+ _CLASS_VARIABLES_POPULATED: bool = False
+
+ @property
+ def HTML_ENTITY_TO_CHARACTER(self) -> Dict[str, str]:
+ """A mapping of entity names like "angmsdaa" to Unicode
+ strings like "⦨".
+ """
+ if not self._CLASS_VARIABLES_POPULATED:
+ self._populate_class_variables()
+ return self._HTML_ENTITY_TO_CHARACTER
+ _HTML_ENTITY_TO_CHARACTER: Dict[str, str]
+
+ @property
+ def CHARACTER_TO_HTML_ENTITY(self) -> Dict[str, str]:
+ """A mapping of Unicode strings like "⦨" to entity names like
+ "angmsdaa". When a single Unicode string has multiple entity
+ names, we try to choose the most commonly-used name.
+ """
+ if not self._CLASS_VARIABLES_POPULATED:
+ self._populate_class_variables()
+ return self._CHARACTER_TO_HTML_ENTITY
+ _CHARACTER_TO_HTML_ENTITY: Dict[str, str]
+
+ @property
+ def CHARACTER_TO_HTML_ENTITY_RE(self) -> Pattern[str]:
+ """A regular expression matching (almost) any Unicode string
+ that corresponds to an HTML5 named entity.
+ """
+
+ if not self._CLASS_VARIABLES_POPULATED:
+ self._populate_class_variables()
+ return self._CHARACTER_TO_HTML_ENTITY_RE
+ _CHARACTER_TO_HTML_ENTITY_RE: Pattern[str]
+
+ @property
+ def CHARACTER_TO_HTML_ENTITY_WITH_AMPERSAND_RE(self) -> Pattern[str]:
+ """A very similar regular expression to
+ CHARACTER_TO_HTML_ENTITY_RE, but which also matches unescaped
+ ampersands. This is used by the 'html' formatter to provide
+ backwards-compatibility, even though the HTML5 spec allows
+ most ampersands to go unescaped.
+ """
+ if not self._CLASS_VARIABLES_POPULATED:
+ self._populate_class_variables()
+ return self._CHARACTER_TO_HTML_ENTITY_WITH_AMPERSAND_RE
+ _CHARACTER_TO_HTML_ENTITY_WITH_AMPERSAND_RE: Pattern[str]
+
+ def _populate_class_variables(self) -> None:
+ """Initialize variables used by EntitySubstitution to manage the plethora of
+ HTML and HTML5 named entities.
+
+ This method populates the class variables necessary to make
+ the properties defined in the metaclass work.
+ """
+ if self._CLASS_VARIABLES_POPULATED:
+ return
+ unicode_to_name = {}
+ name_to_unicode = {}
+
+ short_entities = set()
+ long_entities_by_first_character = defaultdict(set)
+
+ for name_with_semicolon, character in sorted(html5.items()):
+ # "It is intentional, for legacy compatibility, that many
+ # code points have multiple character reference names. For
+ # example, some appear both with and without the trailing
+ # semicolon, or with different capitalizations."
+ # - https://html.spec.whatwg.org/multipage/named-characters.html#named-character-references
+ #
+ # The parsers are in charge of handling (or not) character
+ # references with no trailing semicolon, so we remove the
+ # semicolon whenever it appears.
+ if name_with_semicolon.endswith(";"):
+ name = name_with_semicolon[:-1]
+ else:
+ name = name_with_semicolon
+
+ # When parsing HTML, we want to recognize any known named
+ # entity and convert it to a sequence of Unicode
+ # characters.
+ if name not in name_to_unicode:
+ name_to_unicode[name] = character
+
+ # When _generating_ HTML, we want to recognize special
+ # character sequences that _could_ be converted to named
+ # entities.
+ unicode_to_name[character] = name
+
+ # We also need to build a regular expression that lets us
+ # _find_ those characters in output strings so we can
+ # replace them.
+ #
+ # This is tricky, for two reasons.
+
+ if len(character) == 1 and ord(character) < 128 and character not in "<>":
+ # First, it would be annoying to turn single ASCII
+ # characters like | into named entities like
+ # |. The exceptions are <>, which we _must_
+ # turn into named entities to produce valid HTML.
+ continue
+
+ if len(character) > 1 and all(ord(x) < 128 for x in character):
+ # We also do not want to turn _combinations_ of ASCII
+ # characters like 'fj' into named entities like 'fj',
+ # though that's more debateable.
+ continue
+
+ # Second, some named entities have a Unicode value that's
+ # a subset of the Unicode value for some _other_ named
+ # entity. As an example, \u2267' is ≧,
+ # but '\u2267\u0338' is ≧̸. Our regular
+ # expression needs to match the first two characters of
+ # "\u2267\u0338foo", but only the first character of
+ # "\u2267foo".
+ #
+ # In this step, we build two sets of characters that
+ # _eventually_ need to go into the regular expression. But
+ # we won't know exactly what the regular expression needs
+ # to look like until we've gone through the entire list of
+ # named entities.
+ if len(character) == 1 and character != "&":
+ short_entities.add(character)
+ else:
+ long_entities_by_first_character[character[0]].add(character)
+
+ # Now that we've been through the entire list of entities, we
+ # can create a regular expression that matches any of them.
+ particles = set()
+ for short in short_entities:
+ long_versions = long_entities_by_first_character[short]
+ if not long_versions:
+ particles.add(short)
+ else:
+ ignore = "".join([x[1] for x in long_versions])
+ # This finds, e.g. \u2267 but only if it is _not_
+ # followed by \u0338.
+ particles.add("%s(?![%s])" % (short, ignore))
+
+ for long_entities in list(long_entities_by_first_character.values()):
+ for long_entity in long_entities:
+ particles.add(long_entity)
+
+ re_definition = "(%s)" % "|".join(particles)
+
+ particles.add("&")
+ re_definition_with_ampersand = "(%s)" % "|".join(particles)
+
+ # If an entity shows up in both html5 and codepoint2name, it's
+ # likely that HTML5 gives it several different names, such as
+ # 'rsquo' and 'rsquor'. When converting Unicode characters to
+ # named entities, the codepoint2name name should take
+ # precedence where possible, since that's the more easily
+ # recognizable one.
+ for codepoint, name in list(codepoint2name.items()):
+ character = chr(codepoint)
+ unicode_to_name[character] = name
+
+ self._CHARACTER_TO_HTML_ENTITY = unicode_to_name
+ self._HTML_ENTITY_TO_CHARACTER = name_to_unicode
+ self._CHARACTER_TO_HTML_ENTITY_RE = re.compile(re_definition)
+ self._CHARACTER_TO_HTML_ENTITY_WITH_AMPERSAND_RE = re.compile(
+ re_definition_with_ampersand
+ )
+ self._CLASS_VARIABLES_POPULATED = True
+
+class EntitySubstitution(metaclass=EntitySubstitutionMeta):
+ """The ability to substitute XML or HTML entities for certain characters."""
+
+ #: A map of Unicode strings to the corresponding named XML entities.
+ #:
+ #: :meta hide-value:
+ CHARACTER_TO_XML_ENTITY: Dict[str, str] = {
+ "'": "apos",
+ '"': "quot",
+ "&": "amp",
+ "<": "lt",
+ ">": "gt",
+ }
+
+ # Matches any named or numeric HTML entity.
+ ANY_ENTITY_RE = re.compile("&(#\\d+|#x[0-9a-fA-F]+|\\w+);", re.I)
+
+ #: A regular expression matching an angle bracket or an ampersand that
+ #: is not part of an XML or HTML entity.
+ #:
+ #: :meta hide-value:
+ BARE_AMPERSAND_OR_BRACKET: Pattern[str] = re.compile(
+ "([<>]|" "&(?!#\\d+;|#x[0-9a-fA-F]+;|\\w+;)" ")"
+ )
+
+ #: A regular expression matching an angle bracket or an ampersand.
+ #:
+ #: :meta hide-value:
+ AMPERSAND_OR_BRACKET: Pattern[str] = re.compile("([<>&])")
+
+ @classmethod
+ def _substitute_html_entity(cls, matchobj: re.Match) -> str:
+ """Used with a regular expression to substitute the
+ appropriate HTML entity for a special character string."""
+ original_entity = matchobj.group(0)
+ entity = cls.CHARACTER_TO_HTML_ENTITY.get(original_entity)
+ if entity is None:
+ return "&%s;" % original_entity
+ return "&%s;" % entity
+
+ @classmethod
+ def _substitute_xml_entity(cls, matchobj: re.Match) -> str:
+ """Used with a regular expression to substitute the
+ appropriate XML entity for a special character string."""
+ entity = cls.CHARACTER_TO_XML_ENTITY[matchobj.group(0)]
+ return "&%s;" % entity
+
+ @classmethod
+ def _escape_entity_name(cls, matchobj: re.Match) -> str:
+ return "&%s;" % matchobj.group(1)
+
+ @classmethod
+ def _escape_unrecognized_entity_name(cls, matchobj: re.Match) -> str:
+ possible_entity = matchobj.group(1)
+ if possible_entity in cls.HTML_ENTITY_TO_CHARACTER:
+ return "&%s;" % possible_entity
+ return "&%s;" % possible_entity
+
+ @classmethod
+ def quoted_attribute_value(cls, value: str) -> str:
+ """Make a value into a quoted XML attribute, possibly escaping it.
+
+ Most strings will be quoted using double quotes.
+
+ Bob's Bar -> "Bob's Bar"
+
+ If a string contains double quotes, it will be quoted using
+ single quotes.
+
+ Welcome to "my bar" -> 'Welcome to "my bar"'
+
+ If a string contains both single and double quotes, the
+ double quotes will be escaped, and the string will be quoted
+ using double quotes.
+
+ Welcome to "Bob's Bar" -> Welcome to "Bob's bar"
+
+ :param value: The XML attribute value to quote
+ :return: The quoted value
+ """
+ quote_with = '"'
+ if '"' in value:
+ if "'" in value:
+ # The string contains both single and double
+ # quotes. Turn the double quotes into
+ # entities. We quote the double quotes rather than
+ # the single quotes because the entity name is
+ # """ whether this is HTML or XML. If we
+ # quoted the single quotes, we'd have to decide
+ # between ' and &squot;.
+ replace_with = """
+ value = value.replace('"', replace_with)
+ else:
+ # There are double quotes but no single quotes.
+ # We can use single quotes to quote the attribute.
+ quote_with = "'"
+ return quote_with + value + quote_with
+
+ @classmethod
+ def substitute_xml(cls, value: str, make_quoted_attribute: bool = False) -> str:
+ """Replace special XML characters with named XML entities.
+
+ The less-than sign will become <, the greater-than sign
+ will become >, and any ampersands will become &. If you
+ want ampersands that seem to be part of an entity definition
+ to be left alone, use `substitute_xml_containing_entities`
+ instead.
+
+ :param value: A string to be substituted.
+
+ :param make_quoted_attribute: If True, then the string will be
+ quoted, as befits an attribute value.
+
+ :return: A version of ``value`` with special characters replaced
+ with named entities.
+ """
+ # Escape angle brackets and ampersands.
+ value = cls.AMPERSAND_OR_BRACKET.sub(cls._substitute_xml_entity, value)
+
+ if make_quoted_attribute:
+ value = cls.quoted_attribute_value(value)
+ return value
+
+ @classmethod
+ def substitute_xml_containing_entities(
+ cls, value: str, make_quoted_attribute: bool = False
+ ) -> str:
+ """Substitute XML entities for special XML characters.
+
+ :param value: A string to be substituted. The less-than sign will
+ become <, the greater-than sign will become >, and any
+ ampersands that are not part of an entity defition will
+ become &.
+
+ :param make_quoted_attribute: If True, then the string will be
+ quoted, as befits an attribute value.
+ """
+ # Escape angle brackets, and ampersands that aren't part of
+ # entities.
+ value = cls.BARE_AMPERSAND_OR_BRACKET.sub(cls._substitute_xml_entity, value)
+
+ if make_quoted_attribute:
+ value = cls.quoted_attribute_value(value)
+ return value
+
+ @classmethod
+ def substitute_html(cls, s: str) -> str:
+ """Replace certain Unicode characters with named HTML entities.
+
+ This differs from ``data.encode(encoding, 'xmlcharrefreplace')``
+ in that the goal is to make the result more readable (to those
+ with ASCII displays) rather than to recover from
+ errors. There's absolutely nothing wrong with a UTF-8 string
+ containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
+ character with "é" will make it more readable to some
+ people.
+
+ :param s: The string to be modified.
+ :return: The string with some Unicode characters replaced with
+ HTML entities.
+ """
+ # Convert any appropriate characters to HTML entities.
+ return cls.CHARACTER_TO_HTML_ENTITY_WITH_AMPERSAND_RE.sub(
+ cls._substitute_html_entity, s
+ )
+
+ @classmethod
+ def substitute_html5(cls, s: str) -> str:
+ """Replace certain Unicode characters with named HTML entities
+ using HTML5 rules.
+
+ Specifically, this method is much less aggressive about
+ escaping ampersands than substitute_html. Only ambiguous
+ ampersands are escaped, per the HTML5 standard:
+
+ "An ambiguous ampersand is a U+0026 AMPERSAND character (&)
+ that is followed by one or more ASCII alphanumerics, followed
+ by a U+003B SEMICOLON character (;), where these characters do
+ not match any of the names given in the named character
+ references section."
+
+ Unlike substitute_html5_raw, this method assumes HTML entities
+ were converted to Unicode characters on the way in, as
+ Beautiful Soup does. By the time Beautiful Soup does its work,
+ the only ambiguous ampersands that need to be escaped are the
+ ones that were escaped in the original markup when mentioning
+ HTML entities.
+
+ :param s: The string to be modified.
+ :return: The string with some Unicode characters replaced with
+ HTML entities.
+ """
+ # First, escape any HTML entities found in the markup.
+ s = cls.ANY_ENTITY_RE.sub(cls._escape_entity_name, s)
+
+ # Next, convert any appropriate characters to unescaped HTML entities.
+ s = cls.CHARACTER_TO_HTML_ENTITY_RE.sub(cls._substitute_html_entity, s)
+
+ return s
+
+ @classmethod
+ def substitute_html5_raw(cls, s: str) -> str:
+ """Replace certain Unicode characters with named HTML entities
+ using HTML5 rules.
+
+ substitute_html5_raw is similar to substitute_html5 but it is
+ designed for standalone use (whereas substitute_html5 is
+ designed for use with Beautiful Soup).
+
+ :param s: The string to be modified.
+ :return: The string with some Unicode characters replaced with
+ HTML entities.
+ """
+ # First, escape the ampersand for anything that looks like an
+ # entity but isn't in the list of recognized entities. All other
+ # ampersands can be left alone.
+ s = cls.ANY_ENTITY_RE.sub(cls._escape_unrecognized_entity_name, s)
+
+ # Then, convert a range of Unicode characters to unescaped
+ # HTML entities.
+ s = cls.CHARACTER_TO_HTML_ENTITY_RE.sub(cls._substitute_html_entity, s)
+
+ return s
+
+
+class EncodingDetector:
+ """This class is capable of guessing a number of possible encodings
+ for a bytestring.
+
+ Order of precedence:
+
+ 1. Encodings you specifically tell EncodingDetector to try first
+ (the ``known_definite_encodings`` argument to the constructor).
+
+ 2. An encoding determined by sniffing the document's byte-order mark.
+
+ 3. Encodings you specifically tell EncodingDetector to try if
+ byte-order mark sniffing fails (the ``user_encodings`` argument to the
+ constructor).
+
+ 4. An encoding declared within the bytestring itself, either in an
+ XML declaration (if the bytestring is to be interpreted as an XML
+ document), or in a <meta> tag (if the bytestring is to be
+ interpreted as an HTML document.)
+
+ 5. An encoding detected through textual analysis by chardet,
+ cchardet, or a similar external library.
+
+ 6. UTF-8.
+
+ 7. Windows-1252.
+
+ :param markup: Some markup in an unknown encoding.
+
+ :param known_definite_encodings: When determining the encoding
+ of ``markup``, these encodings will be tried first, in
+ order. In HTML terms, this corresponds to the "known
+ definite encoding" step defined in `section 13.2.3.1 of the HTML standard <https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding>`_.
+
+ :param user_encodings: These encodings will be tried after the
+ ``known_definite_encodings`` have been tried and failed, and
+ after an attempt to sniff the encoding by looking at a
+ byte order mark has failed. In HTML terms, this
+ corresponds to the step "user has explicitly instructed
+ the user agent to override the document's character
+ encoding", defined in `section 13.2.3.2 of the HTML standard <https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding>`_.
+
+ :param override_encodings: A **deprecated** alias for
+ ``known_definite_encodings``. Any encodings here will be tried
+ immediately after the encodings in
+ ``known_definite_encodings``.
+
+ :param is_html: If True, this markup is considered to be
+ HTML. Otherwise it's assumed to be XML.
+
+ :param exclude_encodings: These encodings will not be tried,
+ even if they otherwise would be.
+
+ """
+
+ def __init__(
+ self,
+ markup: bytes,
+ known_definite_encodings: Optional[_Encodings] = None,
+ is_html: Optional[bool] = False,
+ exclude_encodings: Optional[_Encodings] = None,
+ user_encodings: Optional[_Encodings] = None,
+ override_encodings: Optional[_Encodings] = None,
+ ):
+ self.known_definite_encodings = list(known_definite_encodings or [])
+ if override_encodings:
+ warnings.warn(
+ "The 'override_encodings' argument was deprecated in 4.10.0. Use 'known_definite_encodings' instead.",
+ DeprecationWarning,
+ stacklevel=3,
+ )
+ self.known_definite_encodings += override_encodings
+ self.user_encodings = user_encodings or []
+ exclude_encodings = exclude_encodings or []
+ self.exclude_encodings = set([x.lower() for x in exclude_encodings])
+ self.chardet_encoding = None
+ self.is_html = False if is_html is None else is_html
+ self.declared_encoding: Optional[str] = None
+
+ # First order of business: strip a byte-order mark.
+ self.markup, self.sniffed_encoding = self.strip_byte_order_mark(markup)
+
+ known_definite_encodings: _Encodings
+ user_encodings: _Encodings
+ exclude_encodings: _Encodings
+ chardet_encoding: Optional[_Encoding]
+ is_html: bool
+ declared_encoding: Optional[_Encoding]
+ markup: bytes
+ sniffed_encoding: Optional[_Encoding]
+
+ def _usable(self, encoding: Optional[_Encoding], tried: Set[_Encoding]) -> bool:
+ """Should we even bother to try this encoding?
+
+ :param encoding: Name of an encoding.
+ :param tried: Encodings that have already been tried. This
+ will be modified as a side effect.
+ """
+ if encoding is None:
+ return False
+ encoding = encoding.lower()
+ if encoding in self.exclude_encodings:
+ return False
+ if encoding not in tried:
+ tried.add(encoding)
+ return True
+ return False
+
+ @property
+ def encodings(self) -> Iterator[_Encoding]:
+ """Yield a number of encodings that might work for this markup.
+
+ :yield: A sequence of strings. Each is the name of an encoding
+ that *might* work to convert a bytestring into Unicode.
+ """
+ tried: Set[_Encoding] = set()
+
+ # First, try the known definite encodings
+ for e in self.known_definite_encodings:
+ if self._usable(e, tried):
+ yield e
+
+ # Did the document originally start with a byte-order mark
+ # that indicated its encoding?
+ if self.sniffed_encoding is not None and self._usable(
+ self.sniffed_encoding, tried
+ ):
+ yield self.sniffed_encoding
+
+ # Sniffing the byte-order mark did nothing; try the user
+ # encodings.
+ for e in self.user_encodings:
+ if self._usable(e, tried):
+ yield e
+
+ # Look within the document for an XML or HTML encoding
+ # declaration.
+ if self.declared_encoding is None:
+ self.declared_encoding = self.find_declared_encoding(
+ self.markup, self.is_html
+ )
+ if self.declared_encoding is not None and self._usable(
+ self.declared_encoding, tried
+ ):
+ yield self.declared_encoding
+
+ # Use third-party character set detection to guess at the
+ # encoding.
+ if self.chardet_encoding is None:
+ self.chardet_encoding = _chardet_dammit(self.markup)
+ if self.chardet_encoding is not None and self._usable(
+ self.chardet_encoding, tried
+ ):
+ yield self.chardet_encoding
+
+ # As a last-ditch effort, try utf-8 and windows-1252.
+ for e in ("utf-8", "windows-1252"):
+ if self._usable(e, tried):
+ yield e
+
+ @classmethod
+ def strip_byte_order_mark(cls, data: bytes) -> Tuple[bytes, Optional[_Encoding]]:
+ """If a byte-order mark is present, strip it and return the encoding it implies.
+
+ :param data: A bytestring that may or may not begin with a
+ byte-order mark.
+
+ :return: A 2-tuple (data stripped of byte-order mark, encoding implied by byte-order mark)
+ """
+ encoding = None
+ if isinstance(data, str):
+ # Unicode data cannot have a byte-order mark.
+ return data, encoding
+ if (
+ (len(data) >= 4)
+ and (data[:2] == b"\xfe\xff")
+ and (data[2:4] != b"\x00\x00")
+ ):
+ encoding = "utf-16be"
+ data = data[2:]
+ elif (
+ (len(data) >= 4)
+ and (data[:2] == b"\xff\xfe")
+ and (data[2:4] != b"\x00\x00")
+ ):
+ encoding = "utf-16le"
+ data = data[2:]
+ elif data[:3] == b"\xef\xbb\xbf":
+ encoding = "utf-8"
+ data = data[3:]
+ elif data[:4] == b"\x00\x00\xfe\xff":
+ encoding = "utf-32be"
+ data = data[4:]
+ elif data[:4] == b"\xff\xfe\x00\x00":
+ encoding = "utf-32le"
+ data = data[4:]
+ return data, encoding
+
+ @classmethod
+ def find_declared_encoding(
+ cls,
+ markup: Union[bytes, str],
+ is_html: bool = False,
+ search_entire_document: bool = False,
+ ) -> Optional[_Encoding]:
+ """Given a document, tries to find an encoding declared within the
+ text of the document itself.
+
+ An XML encoding is declared at the beginning of the document.
+
+ An HTML encoding is declared in a <meta> tag, hopefully near the
+ beginning of the document.
+
+ :param markup: Some markup.
+ :param is_html: If True, this markup is considered to be HTML. Otherwise
+ it's assumed to be XML.
+ :param search_entire_document: Since an encoding is supposed
+ to declared near the beginning of the document, most of
+ the time it's only necessary to search a few kilobytes of
+ data. Set this to True to force this method to search the
+ entire document.
+ :return: The declared encoding, if one is found.
+ """
+ if search_entire_document:
+ xml_endpos = html_endpos = len(markup)
+ else:
+ xml_endpos = 1024
+ html_endpos = max(2048, int(len(markup) * 0.05))
+
+ if isinstance(markup, bytes):
+ res = encoding_res[bytes]
+ else:
+ res = encoding_res[str]
+
+ xml_re = res["xml"]
+ html_re = res["html"]
+ declared_encoding: Optional[_Encoding] = None
+ declared_encoding_match = xml_re.search(markup, endpos=xml_endpos)
+ if not declared_encoding_match and is_html:
+ declared_encoding_match = html_re.search(markup, endpos=html_endpos)
+ if declared_encoding_match is not None:
+ declared_encoding = declared_encoding_match.groups()[0]
+ if declared_encoding:
+ if isinstance(declared_encoding, bytes):
+ declared_encoding = declared_encoding.decode("ascii", "replace")
+ return declared_encoding.lower()
+ return None
+
+
+class UnicodeDammit:
+ """A class for detecting the encoding of a bytestring containing an
+ HTML or XML document, and decoding it to Unicode. If the source
+ encoding is windows-1252, `UnicodeDammit` can also replace
+ Microsoft smart quotes with their HTML or XML equivalents.
+
+ :param markup: HTML or XML markup in an unknown encoding.
+
+ :param known_definite_encodings: When determining the encoding
+ of ``markup``, these encodings will be tried first, in
+ order. In HTML terms, this corresponds to the "known
+ definite encoding" step defined in `section 13.2.3.1 of the HTML standard <https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding>`_.
+
+ :param user_encodings: These encodings will be tried after the
+ ``known_definite_encodings`` have been tried and failed, and
+ after an attempt to sniff the encoding by looking at a
+ byte order mark has failed. In HTML terms, this
+ corresponds to the step "user has explicitly instructed
+ the user agent to override the document's character
+ encoding", defined in `section 13.2.3.2 of the HTML standard <https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding>`_.
+
+ :param override_encodings: A **deprecated** alias for
+ ``known_definite_encodings``. Any encodings here will be tried
+ immediately after the encodings in
+ ``known_definite_encodings``.
+
+ :param smart_quotes_to: By default, Microsoft smart quotes will,
+ like all other characters, be converted to Unicode
+ characters. Setting this to ``ascii`` will convert them to ASCII
+ quotes instead. Setting it to ``xml`` will convert them to XML
+ entity references, and setting it to ``html`` will convert them
+ to HTML entity references.
+
+ :param is_html: If True, ``markup`` is treated as an HTML
+ document. Otherwise it's treated as an XML document.
+
+ :param exclude_encodings: These encodings will not be considered,
+ even if the sniffing code thinks they might make sense.
+
+ """
+
+ def __init__(
+ self,
+ markup: bytes,
+ known_definite_encodings: Optional[_Encodings] = [],
+ smart_quotes_to: Optional[Literal["ascii", "xml", "html"]] = None,
+ is_html: bool = False,
+ exclude_encodings: Optional[_Encodings] = [],
+ user_encodings: Optional[_Encodings] = None,
+ override_encodings: Optional[_Encodings] = None,
+ ):
+ self.smart_quotes_to = smart_quotes_to
+ self.tried_encodings = []
+ self.contains_replacement_characters = False
+ self.is_html = is_html
+ self.log = getLogger(__name__)
+ self.detector = EncodingDetector(
+ markup,
+ known_definite_encodings,
+ is_html,
+ exclude_encodings,
+ user_encodings,
+ override_encodings,
+ )
+
+ # Short-circuit if the data is in Unicode to begin with.
+ if isinstance(markup, str):
+ self.markup = markup.encode("utf8")
+ self.unicode_markup = markup
+ self.original_encoding = None
+ return
+
+ # The encoding detector may have stripped a byte-order mark.
+ # Use the stripped markup from this point on.
+ self.markup = self.detector.markup
+
+ u = None
+ for encoding in self.detector.encodings:
+ markup = self.detector.markup
+ u = self._convert_from(encoding)
+ if u is not None:
+ break
+
+ if not u:
+ # None of the encodings worked. As an absolute last resort,
+ # try them again with character replacement.
+
+ for encoding in self.detector.encodings:
+ if encoding != "ascii":
+ u = self._convert_from(encoding, "replace")
+ if u is not None:
+ self.log.warning(
+ "Some characters could not be decoded, and were "
+ "replaced with REPLACEMENT CHARACTER."
+ )
+
+ self.contains_replacement_characters = True
+ break
+
+ # If none of that worked, we could at this point force it to
+ # ASCII, but that would destroy so much data that I think
+ # giving up is better.
+ #
+ # Note that this is extremely unlikely, probably impossible,
+ # because the "replace" strategy is so powerful. Even running
+ # the Python binary through Unicode, Dammit gives you Unicode,
+ # albeit Unicode riddled with REPLACEMENT CHARACTER.
+ if u is None:
+ self.original_encoding = None
+ self.unicode_markup = None
+ else:
+ self.unicode_markup = u
+
+ #: The original markup, before it was converted to Unicode.
+ #: This is not necessarily the same as what was passed in to the
+ #: constructor, since any byte-order mark will be stripped.
+ markup: bytes
+
+ #: The Unicode version of the markup, following conversion. This
+ #: is set to None if there was simply no way to convert the
+ #: bytestring to Unicode (as with binary data).
+ unicode_markup: Optional[str]
+
+ #: This is True if `UnicodeDammit.unicode_markup` contains
+ #: U+FFFD REPLACEMENT_CHARACTER characters which were not present
+ #: in `UnicodeDammit.markup`. These mark character sequences that
+ #: could not be represented in Unicode.
+ contains_replacement_characters: bool
+
+ #: Unicode, Dammit's best guess as to the original character
+ #: encoding of `UnicodeDammit.markup`.
+ original_encoding: Optional[_Encoding]
+
+ #: The strategy used to handle Microsoft smart quotes.
+ smart_quotes_to: Optional[str]
+
+ #: The (encoding, error handling strategy) 2-tuples that were used to
+ #: try and convert the markup to Unicode.
+ tried_encodings: List[Tuple[_Encoding, str]]
+
+ log: Logger #: :meta private:
+
+ def _sub_ms_char(self, match: re.Match) -> bytes:
+ """Changes a MS smart quote character to an XML or HTML
+ entity, or an ASCII character.
+
+ TODO: Since this is only used to convert smart quotes, it
+ could be simplified, and MS_CHARS_TO_ASCII made much less
+ parochial.
+ """
+ orig: bytes = match.group(1)
+ sub: bytes
+ if self.smart_quotes_to == "ascii":
+ if orig in self.MS_CHARS_TO_ASCII:
+ sub = self.MS_CHARS_TO_ASCII[orig].encode()
+ else:
+ # Shouldn't happen; substitute the character
+ # with itself.
+ sub = orig
+ else:
+ if orig in self.MS_CHARS:
+ substitutions = self.MS_CHARS[orig]
+ if type(substitutions) is tuple:
+ if self.smart_quotes_to == "xml":
+ sub = b"&#x" + substitutions[1].encode() + b";"
+ else:
+ sub = b"&" + substitutions[0].encode() + b";"
+ else:
+ substitutions = cast(str, substitutions)
+ sub = substitutions.encode()
+ else:
+ # Shouldn't happen; substitute the character
+ # for itself.
+ sub = orig
+ return sub
+
+ #: This dictionary maps commonly seen values for "charset" in HTML
+ #: meta tags to the corresponding Python codec names. It only covers
+ #: values that aren't in Python's aliases and can't be determined
+ #: by the heuristics in `find_codec`.
+ #:
+ #: :meta hide-value:
+ CHARSET_ALIASES: Dict[str, _Encoding] = {
+ "macintosh": "mac-roman",
+ "x-sjis": "shift-jis",
+ }
+
+ #: A list of encodings that tend to contain Microsoft smart quotes.
+ #:
+ #: :meta hide-value:
+ ENCODINGS_WITH_SMART_QUOTES: _Encodings = [
+ "windows-1252",
+ "iso-8859-1",
+ "iso-8859-2",
+ ]
+
+ def _convert_from(
+ self, proposed: _Encoding, errors: str = "strict"
+ ) -> Optional[str]:
+ """Attempt to convert the markup to the proposed encoding.
+
+ :param proposed: The name of a character encoding.
+ :param errors: An error handling strategy, used when calling `str`.
+ :return: The converted markup, or `None` if the proposed
+ encoding/error handling strategy didn't work.
+ """
+ lookup_result = self.find_codec(proposed)
+ if lookup_result is None or (lookup_result, errors) in self.tried_encodings:
+ return None
+ proposed = lookup_result
+ self.tried_encodings.append((proposed, errors))
+ markup = self.markup
+ # Convert smart quotes to HTML if coming from an encoding
+ # that might have them.
+ if (
+ self.smart_quotes_to is not None
+ and proposed in self.ENCODINGS_WITH_SMART_QUOTES
+ ):
+ smart_quotes_re = b"([\x80-\x9f])"
+ smart_quotes_compiled = re.compile(smart_quotes_re)
+ markup = smart_quotes_compiled.sub(self._sub_ms_char, markup)
+
+ try:
+ # print("Trying to convert document to %s (errors=%s)" % (
+ # proposed, errors))
+ u = self._to_unicode(markup, proposed, errors)
+ self.unicode_markup = u
+ self.original_encoding = proposed
+ except Exception:
+ # print("That didn't work!")
+ # print(e)
+ return None
+ # print("Correct encoding: %s" % proposed)
+ return self.unicode_markup
+
+ def _to_unicode(
+ self, data: bytes, encoding: _Encoding, errors: str = "strict"
+ ) -> str:
+ """Given a bytestring and its encoding, decodes the string into Unicode.
+
+ :param encoding: The name of an encoding.
+ :param errors: An error handling strategy, used when calling `str`.
+ """
+ return str(data, encoding, errors)
+
+ @property
+ def declared_html_encoding(self) -> Optional[_Encoding]:
+ """If the markup is an HTML document, returns the encoding, if any,
+ declared *inside* the document.
+ """
+ if not self.is_html:
+ return None
+ return self.detector.declared_encoding
+
+ def find_codec(self, charset: _Encoding) -> Optional[str]:
+ """Look up the Python codec corresponding to a given character set.
+
+ :param charset: The name of a character set.
+ :return: The name of a Python codec.
+ """
+ value = (
+ self._codec(self.CHARSET_ALIASES.get(charset, charset))
+ or (charset and self._codec(charset.replace("-", "")))
+ or (charset and self._codec(charset.replace("-", "_")))
+ or (charset and charset.lower())
+ or charset
+ )
+ if value:
+ return value.lower()
+ return None
+
+ def _codec(self, charset: _Encoding) -> Optional[str]:
+ if not charset:
+ return charset
+ codec = None
+ try:
+ codecs.lookup(charset)
+ codec = charset
+ except (LookupError, ValueError):
+ pass
+ return codec
+
+ #: A partial mapping of ISO-Latin-1 to HTML entities/XML numeric entities.
+ #:
+ #: :meta hide-value:
+ MS_CHARS: Dict[bytes, Union[str, Tuple[str, str]]] = {
+ b"\x80": ("euro", "20AC"),
+ b"\x81": " ",
+ b"\x82": ("sbquo", "201A"),
+ b"\x83": ("fnof", "192"),
+ b"\x84": ("bdquo", "201E"),
+ b"\x85": ("hellip", "2026"),
+ b"\x86": ("dagger", "2020"),
+ b"\x87": ("Dagger", "2021"),
+ b"\x88": ("circ", "2C6"),
+ b"\x89": ("permil", "2030"),
+ b"\x8a": ("Scaron", "160"),
+ b"\x8b": ("lsaquo", "2039"),
+ b"\x8c": ("OElig", "152"),
+ b"\x8d": "?",
+ b"\x8e": ("#x17D", "17D"),
+ b"\x8f": "?",
+ b"\x90": "?",
+ b"\x91": ("lsquo", "2018"),
+ b"\x92": ("rsquo", "2019"),
+ b"\x93": ("ldquo", "201C"),
+ b"\x94": ("rdquo", "201D"),
+ b"\x95": ("bull", "2022"),
+ b"\x96": ("ndash", "2013"),
+ b"\x97": ("mdash", "2014"),
+ b"\x98": ("tilde", "2DC"),
+ b"\x99": ("trade", "2122"),
+ b"\x9a": ("scaron", "161"),
+ b"\x9b": ("rsaquo", "203A"),
+ b"\x9c": ("oelig", "153"),
+ b"\x9d": "?",
+ b"\x9e": ("#x17E", "17E"),
+ b"\x9f": ("Yuml", ""),
+ }
+
+ #: A parochial partial mapping of ISO-Latin-1 to ASCII. Contains
+ #: horrors like stripping diacritical marks to turn á into a, but also
+ #: contains non-horrors like turning “ into ".
+ #:
+ #: Seriously, don't use this for anything other than removing smart
+ #: quotes.
+ #:
+ #: :meta private:
+ MS_CHARS_TO_ASCII: Dict[bytes, str] = {
+ b"\x80": "EUR",
+ b"\x81": " ",
+ b"\x82": ",",
+ b"\x83": "f",
+ b"\x84": ",,",
+ b"\x85": "...",
+ b"\x86": "+",
+ b"\x87": "++",
+ b"\x88": "^",
+ b"\x89": "%",
+ b"\x8a": "S",
+ b"\x8b": "<",
+ b"\x8c": "OE",
+ b"\x8d": "?",
+ b"\x8e": "Z",
+ b"\x8f": "?",
+ b"\x90": "?",
+ b"\x91": "'",
+ b"\x92": "'",
+ b"\x93": '"',
+ b"\x94": '"',
+ b"\x95": "*",
+ b"\x96": "-",
+ b"\x97": "--",
+ b"\x98": "~",
+ b"\x99": "(TM)",
+ b"\x9a": "s",
+ b"\x9b": ">",
+ b"\x9c": "oe",
+ b"\x9d": "?",
+ b"\x9e": "z",
+ b"\x9f": "Y",
+ b"\xa0": " ",
+ b"\xa1": "!",
+ b"\xa2": "c",
+ b"\xa3": "GBP",
+ b"\xa4": "$", # This approximation is especially parochial--this is the
+ # generic currency symbol.
+ b"\xa5": "YEN",
+ b"\xa6": "|",
+ b"\xa7": "S",
+ b"\xa8": "..",
+ b"\xa9": "",
+ b"\xaa": "(th)",
+ b"\xab": "<<",
+ b"\xac": "!",
+ b"\xad": " ",
+ b"\xae": "(R)",
+ b"\xaf": "-",
+ b"\xb0": "o",
+ b"\xb1": "+-",
+ b"\xb2": "2",
+ b"\xb3": "3",
+ b"\xb4": "'",
+ b"\xb5": "u",
+ b"\xb6": "P",
+ b"\xb7": "*",
+ b"\xb8": ",",
+ b"\xb9": "1",
+ b"\xba": "(th)",
+ b"\xbb": ">>",
+ b"\xbc": "1/4",
+ b"\xbd": "1/2",
+ b"\xbe": "3/4",
+ b"\xbf": "?",
+ b"\xc0": "A",
+ b"\xc1": "A",
+ b"\xc2": "A",
+ b"\xc3": "A",
+ b"\xc4": "A",
+ b"\xc5": "A",
+ b"\xc6": "AE",
+ b"\xc7": "C",
+ b"\xc8": "E",
+ b"\xc9": "E",
+ b"\xca": "E",
+ b"\xcb": "E",
+ b"\xcc": "I",
+ b"\xcd": "I",
+ b"\xce": "I",
+ b"\xcf": "I",
+ b"\xd0": "D",
+ b"\xd1": "N",
+ b"\xd2": "O",
+ b"\xd3": "O",
+ b"\xd4": "O",
+ b"\xd5": "O",
+ b"\xd6": "O",
+ b"\xd7": "*",
+ b"\xd8": "O",
+ b"\xd9": "U",
+ b"\xda": "U",
+ b"\xdb": "U",
+ b"\xdc": "U",
+ b"\xdd": "Y",
+ b"\xde": "b",
+ b"\xdf": "B",
+ b"\xe0": "a",
+ b"\xe1": "a",
+ b"\xe2": "a",
+ b"\xe3": "a",
+ b"\xe4": "a",
+ b"\xe5": "a",
+ b"\xe6": "ae",
+ b"\xe7": "c",
+ b"\xe8": "e",
+ b"\xe9": "e",
+ b"\xea": "e",
+ b"\xeb": "e",
+ b"\xec": "i",
+ b"\xed": "i",
+ b"\xee": "i",
+ b"\xef": "i",
+ b"\xf0": "o",
+ b"\xf1": "n",
+ b"\xf2": "o",
+ b"\xf3": "o",
+ b"\xf4": "o",
+ b"\xf5": "o",
+ b"\xf6": "o",
+ b"\xf7": "/",
+ b"\xf8": "o",
+ b"\xf9": "u",
+ b"\xfa": "u",
+ b"\xfb": "u",
+ b"\xfc": "u",
+ b"\xfd": "y",
+ b"\xfe": "b",
+ b"\xff": "y",
+ }
+
+ #: A map used when removing rogue Windows-1252/ISO-8859-1
+ #: characters in otherwise UTF-8 documents. Also used when a
+ #: numeric character entity has been incorrectly encoded using the
+ #: character's Windows-1252 encoding.
+ #:
+ #: Note that \\x81, \\x8d, \\x8f, \\x90, and \\x9d are undefined in
+ #: Windows-1252.
+ #:
+ #: :meta hide-value:
+ WINDOWS_1252_TO_UTF8: Dict[int, bytes] = {
+ 0x80: b"\xe2\x82\xac", # €
+ 0x82: b"\xe2\x80\x9a", # ‚
+ 0x83: b"\xc6\x92", # ƒ
+ 0x84: b"\xe2\x80\x9e", # „
+ 0x85: b"\xe2\x80\xa6", # …
+ 0x86: b"\xe2\x80\xa0", # †
+ 0x87: b"\xe2\x80\xa1", # ‡
+ 0x88: b"\xcb\x86", # ˆ
+ 0x89: b"\xe2\x80\xb0", # ‰
+ 0x8A: b"\xc5\xa0", # Š
+ 0x8B: b"\xe2\x80\xb9", # ‹
+ 0x8C: b"\xc5\x92", # Œ
+ 0x8E: b"\xc5\xbd", # Ž
+ 0x91: b"\xe2\x80\x98", # ‘
+ 0x92: b"\xe2\x80\x99", # ’
+ 0x93: b"\xe2\x80\x9c", # “
+ 0x94: b"\xe2\x80\x9d", # ”
+ 0x95: b"\xe2\x80\xa2", # •
+ 0x96: b"\xe2\x80\x93", # –
+ 0x97: b"\xe2\x80\x94", # —
+ 0x98: b"\xcb\x9c", # ˜
+ 0x99: b"\xe2\x84\xa2", # ™
+ 0x9A: b"\xc5\xa1", # š
+ 0x9B: b"\xe2\x80\xba", # ›
+ 0x9C: b"\xc5\x93", # œ
+ 0x9E: b"\xc5\xbe", # ž
+ 0x9F: b"\xc5\xb8", # Ÿ
+ 0xA0: b"\xc2\xa0", #
+ 0xA1: b"\xc2\xa1", # ¡
+ 0xA2: b"\xc2\xa2", # ¢
+ 0xA3: b"\xc2\xa3", # £
+ 0xA4: b"\xc2\xa4", # ¤
+ 0xA5: b"\xc2\xa5", # ¥
+ 0xA6: b"\xc2\xa6", # ¦
+ 0xA7: b"\xc2\xa7", # §
+ 0xA8: b"\xc2\xa8", # ¨
+ 0xA9: b"\xc2\xa9", # ©
+ 0xAA: b"\xc2\xaa", # ª
+ 0xAB: b"\xc2\xab", # «
+ 0xAC: b"\xc2\xac", # ¬
+ 0xAD: b"\xc2\xad", #
+ 0xAE: b"\xc2\xae", # ®
+ 0xAF: b"\xc2\xaf", # ¯
+ 0xB0: b"\xc2\xb0", # °
+ 0xB1: b"\xc2\xb1", # ±
+ 0xB2: b"\xc2\xb2", # ²
+ 0xB3: b"\xc2\xb3", # ³
+ 0xB4: b"\xc2\xb4", # ´
+ 0xB5: b"\xc2\xb5", # µ
+ 0xB6: b"\xc2\xb6", # ¶
+ 0xB7: b"\xc2\xb7", # ·
+ 0xB8: b"\xc2\xb8", # ¸
+ 0xB9: b"\xc2\xb9", # ¹
+ 0xBA: b"\xc2\xba", # º
+ 0xBB: b"\xc2\xbb", # »
+ 0xBC: b"\xc2\xbc", # ¼
+ 0xBD: b"\xc2\xbd", # ½
+ 0xBE: b"\xc2\xbe", # ¾
+ 0xBF: b"\xc2\xbf", # ¿
+ 0xC0: b"\xc3\x80", # À
+ 0xC1: b"\xc3\x81", # Á
+ 0xC2: b"\xc3\x82", # Â
+ 0xC3: b"\xc3\x83", # Ã
+ 0xC4: b"\xc3\x84", # Ä
+ 0xC5: b"\xc3\x85", # Å
+ 0xC6: b"\xc3\x86", # Æ
+ 0xC7: b"\xc3\x87", # Ç
+ 0xC8: b"\xc3\x88", # È
+ 0xC9: b"\xc3\x89", # É
+ 0xCA: b"\xc3\x8a", # Ê
+ 0xCB: b"\xc3\x8b", # Ë
+ 0xCC: b"\xc3\x8c", # Ì
+ 0xCD: b"\xc3\x8d", # Í
+ 0xCE: b"\xc3\x8e", # Î
+ 0xCF: b"\xc3\x8f", # Ï
+ 0xD0: b"\xc3\x90", # Ð
+ 0xD1: b"\xc3\x91", # Ñ
+ 0xD2: b"\xc3\x92", # Ò
+ 0xD3: b"\xc3\x93", # Ó
+ 0xD4: b"\xc3\x94", # Ô
+ 0xD5: b"\xc3\x95", # Õ
+ 0xD6: b"\xc3\x96", # Ö
+ 0xD7: b"\xc3\x97", # ×
+ 0xD8: b"\xc3\x98", # Ø
+ 0xD9: b"\xc3\x99", # Ù
+ 0xDA: b"\xc3\x9a", # Ú
+ 0xDB: b"\xc3\x9b", # Û
+ 0xDC: b"\xc3\x9c", # Ü
+ 0xDD: b"\xc3\x9d", # Ý
+ 0xDE: b"\xc3\x9e", # Þ
+ 0xDF: b"\xc3\x9f", # ß
+ 0xE0: b"\xc3\xa0", # à
+ 0xE1: b"\xa1", # á
+ 0xE2: b"\xc3\xa2", # â
+ 0xE3: b"\xc3\xa3", # ã
+ 0xE4: b"\xc3\xa4", # ä
+ 0xE5: b"\xc3\xa5", # å
+ 0xE6: b"\xc3\xa6", # æ
+ 0xE7: b"\xc3\xa7", # ç
+ 0xE8: b"\xc3\xa8", # è
+ 0xE9: b"\xc3\xa9", # é
+ 0xEA: b"\xc3\xaa", # ê
+ 0xEB: b"\xc3\xab", # ë
+ 0xEC: b"\xc3\xac", # ì
+ 0xED: b"\xc3\xad", # í
+ 0xEE: b"\xc3\xae", # î
+ 0xEF: b"\xc3\xaf", # ï
+ 0xF0: b"\xc3\xb0", # ð
+ 0xF1: b"\xc3\xb1", # ñ
+ 0xF2: b"\xc3\xb2", # ò
+ 0xF3: b"\xc3\xb3", # ó
+ 0xF4: b"\xc3\xb4", # ô
+ 0xF5: b"\xc3\xb5", # õ
+ 0xF6: b"\xc3\xb6", # ö
+ 0xF7: b"\xc3\xb7", # ÷
+ 0xF8: b"\xc3\xb8", # ø
+ 0xF9: b"\xc3\xb9", # ù
+ 0xFA: b"\xc3\xba", # ú
+ 0xFB: b"\xc3\xbb", # û
+ 0xFC: b"\xc3\xbc", # ü
+ 0xFD: b"\xc3\xbd", # ý
+ 0xFE: b"\xc3\xbe", # þ
+ 0xFF: b"\xc3\xbf", # ÿ
+ }
+
+ #: :meta private
+ # Note that this isn't all Unicode noncharacters, just the noncontiguous ones that need to be listed.
+ #
+ # "A noncharacter is a code point that is in the range
+ # U+FDD0 to U+FDEF, inclusive, or U+FFFE, U+FFFF, U+1FFFE,
+ # U+1FFFF, U+2FFFE, U+2FFFF, U+3FFFE, U+3FFFF, U+4FFFE,
+ # U+4FFFF, U+5FFFE, U+5FFFF, U+6FFFE, U+6FFFF, U+7FFFE,
+ # U+7FFFF, U+8FFFE, U+8FFFF, U+9FFFE, U+9FFFF, U+AFFFE,
+ # U+AFFFF, U+BFFFE, U+BFFFF, U+CFFFE, U+CFFFF, U+DFFFE,
+ # U+DFFFF, U+EFFFE, U+EFFFF, U+FFFFE, U+FFFFF, U+10FFFE,
+ # or U+10FFFF."
+ ENUMERATED_NONCHARACTERS: Set[int] = set([0xfffe, 0xffff,
+ 0x1fffe, 0x1ffff,
+ 0x2fffe, 0x2ffff,
+ 0x3fffe, 0x3ffff,
+ 0x4fffe, 0x4ffff,
+ 0x5fffe, 0x5ffff,
+ 0x6fffe, 0x6ffff,
+ 0x7fffe, 0x7ffff,
+ 0x8fffe, 0x8ffff,
+ 0x9fffe, 0x9ffff,
+ 0xafffe, 0xaffff,
+ 0xbfffe, 0xbffff,
+ 0xcfffe, 0xcffff,
+ 0xdfffe, 0xdffff,
+ 0xefffe, 0xeffff,
+ 0xffffe, 0xfffff,
+ 0x10fffe, 0x10ffff])
+
+ #: :meta private:
+ MULTIBYTE_MARKERS_AND_SIZES: List[Tuple[int, int, int]] = [
+ (0xC2, 0xDF, 2), # 2-byte characters start with a byte C2-DF
+ (0xE0, 0xEF, 3), # 3-byte characters start with E0-EF
+ (0xF0, 0xF4, 4), # 4-byte characters start with F0-F4
+ ]
+
+ #: :meta private:
+ FIRST_MULTIBYTE_MARKER: int = MULTIBYTE_MARKERS_AND_SIZES[0][0]
+
+ #: :meta private:
+ LAST_MULTIBYTE_MARKER: int = MULTIBYTE_MARKERS_AND_SIZES[-1][1]
+
+ @classmethod
+ def numeric_character_reference(cls, numeric:int) -> Tuple[str, bool]:
+ """This (mostly) implements the algorithm described in "Numeric character
+ reference end state" from the HTML spec:
+ https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state
+
+ The algorithm is designed to convert numeric character references like "☃"
+ to Unicode characters like "☃".
+
+ :return: A 2-tuple (character, replaced). `character` is the Unicode
+ character corresponding to the numeric reference and `replaced` is
+ whether or not an unresolvable character was replaced with REPLACEMENT
+ CHARACTER.
+ """
+ replacement = "\ufffd"
+
+ if numeric == 0x00:
+ # "If the number is 0x00, then this is a
+ # null-character-reference parse error. Set the character
+ # reference code to 0xFFFD."
+ return replacement, True
+
+ if numeric > 0x10ffff:
+ # "If the number is greater than 0x10FFFF, then this is a
+ # character-reference-outside-unicode-range parse
+ # error. Set the character reference code to 0xFFFD."
+ return replacement, True
+
+ if numeric >= 0xd800 and numeric <= 0xdfff:
+ # "If the number is a surrogate, then this is a
+ # surrogate-character-reference parse error. Set the
+ # character reference code to 0xFFFD."
+ return replacement, True
+
+ if (numeric >= 0xfdd0 and numeric <= 0xfdef) or numeric in cls.ENUMERATED_NONCHARACTERS:
+ # "If the number is a noncharacter, then this is a
+ # noncharacter-character-reference parse error."
+ #
+ # "The parser resolves such character references as-is."
+ #
+ # I'm not sure what "as-is" means but I think it means that we act
+ # like there was no error condition.
+ return chr(numeric), False
+
+ # "If the number is 0x0D, or a control that's not ASCII whitespace,
+ # then this is a control-character-reference parse error."
+ #
+ # "A control is a C0 control or a code point in the range
+ # U+007F DELETE to U+009F APPLICATION PROGRAM COMMAND,
+ # inclusive."
+ #
+ # "A C0 control is a code point in the range U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive."
+ #
+ # "The parser resolves such character references as-is except C1 control references that are replaced."
+
+ # First, let's replace the control references that can be replaced.
+ if numeric >= 0x80 and numeric <= 0x9f and numeric in cls.WINDOWS_1252_TO_UTF8:
+ # "If the number is one of the numbers in the first column of the
+ # following table, then find the row with that number in the first
+ # column, and set the character reference code to the number in the
+ # second column of that row."
+ #
+ # This is an attempt to catch characters that were encoded to numeric
+ # entities using their Windows-1252 encodings rather than their UTF-8
+ # encodings.
+ return cls.WINDOWS_1252_TO_UTF8[numeric].decode("utf8"), False
+
+ # Now all that's left are references that should be resolved as-is. This
+ # is also the default path for non-weird character references.
+ try:
+ return chr(numeric), False
+ except (ValueError, OverflowError):
+ # This shouldn't happen, since these cases should have been handled
+ # above, but if it does, return REPLACEMENT CHARACTER
+ return replacement, True
+
+ @classmethod
+ def detwingle(
+ cls,
+ in_bytes: bytes,
+ main_encoding: _Encoding = "utf8",
+ embedded_encoding: _Encoding = "windows-1252",
+ ) -> bytes:
+ """Fix characters from one encoding embedded in some other encoding.
+
+ Currently the only situation supported is Windows-1252 (or its
+ subset ISO-8859-1), embedded in UTF-8.
+
+ :param in_bytes: A bytestring that you suspect contains
+ characters from multiple encodings. Note that this *must*
+ be a bytestring. If you've already converted the document
+ to Unicode, you're too late.
+ :param main_encoding: The primary encoding of ``in_bytes``.
+ :param embedded_encoding: The encoding that was used to embed characters
+ in the main document.
+ :return: A bytestring similar to ``in_bytes``, in which
+ ``embedded_encoding`` characters have been converted to
+ their ``main_encoding`` equivalents.
+ """
+ if embedded_encoding.replace("_", "-").lower() not in (
+ "windows-1252",
+ "windows_1252",
+ ):
+ raise NotImplementedError(
+ "Windows-1252 and ISO-8859-1 are the only currently supported "
+ "embedded encodings."
+ )
+
+ if main_encoding.lower() not in ("utf8", "utf-8"):
+ raise NotImplementedError(
+ "UTF-8 is the only currently supported main encoding."
+ )
+
+ byte_chunks = []
+
+ chunk_start = 0
+ pos = 0
+ while pos < len(in_bytes):
+ byte = in_bytes[pos]
+ if byte >= cls.FIRST_MULTIBYTE_MARKER and byte <= cls.LAST_MULTIBYTE_MARKER:
+ # This is the start of a UTF-8 multibyte character. Skip
+ # to the end.
+ for start, end, size in cls.MULTIBYTE_MARKERS_AND_SIZES:
+ if byte >= start and byte <= end:
+ pos += size
+ break
+ elif byte >= 0x80 and byte in cls.WINDOWS_1252_TO_UTF8:
+ # We found a Windows-1252 character!
+ # Save the string up to this point as a chunk.
+ byte_chunks.append(in_bytes[chunk_start:pos])
+
+ # Now translate the Windows-1252 character into UTF-8
+ # and add it as another, one-byte chunk.
+ byte_chunks.append(cls.WINDOWS_1252_TO_UTF8[byte])
+ pos += 1
+ chunk_start = pos
+ else:
+ # Go on to the next character.
+ pos += 1
+ if chunk_start == 0:
+ # The string is unchanged.
+ return in_bytes
+ else:
+ # Store the final chunk.
+ byte_chunks.append(in_bytes[chunk_start:])
+ return b"".join(byte_chunks)
diff --git a/bitbake/lib/bb/_vendor/bs4/diagnose.py b/bitbake/lib/bb/_vendor/bs4/diagnose.py
new file mode 100644
index 00000000..7c487b3c
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/diagnose.py
@@ -0,0 +1,268 @@
+"""Diagnostic functions, mainly for use when doing tech support."""
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+import cProfile
+from io import BytesIO
+from html.parser import HTMLParser
+from bb._vendor import bs4
+from bb._vendor.bs4 import BeautifulSoup, __version__
+from bb._vendor.bs4.builder import builder_registry
+from typing import (
+ Any,
+ IO,
+ List,
+ Optional,
+ Tuple,
+ TYPE_CHECKING,
+)
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4._typing import _IncomingMarkup
+
+import pstats
+import random
+import tempfile
+import time
+import traceback
+import sys
+
+
+def diagnose(data: "_IncomingMarkup") -> None:
+ """Diagnostic suite for isolating common problems.
+
+ :param data: Some markup that needs to be explained.
+ :return: None; diagnostics are printed to standard output.
+ """
+ print(("Diagnostic running on Beautiful Soup %s" % __version__))
+ print(("Python version %s" % sys.version))
+
+ basic_parsers = ["html.parser", "html5lib", "lxml"]
+ for name in basic_parsers:
+ for builder in builder_registry.builders:
+ if name in builder.features:
+ break
+ else:
+ basic_parsers.remove(name)
+ print(
+ ("I noticed that %s is not installed. Installing it may help." % name)
+ )
+
+ if "lxml" in basic_parsers:
+ basic_parsers.append("lxml-xml")
+ try:
+ from lxml import etree # type:ignore
+
+ print(("Found lxml version %s" % ".".join(map(str, etree.LXML_VERSION))))
+ except ImportError:
+ print("lxml is not installed or couldn't be imported.")
+
+ if "html5lib" in basic_parsers:
+ try:
+ import html5lib
+
+ print(("Found html5lib version %s" % html5lib.__version__))
+ except ImportError:
+ print("html5lib is not installed or couldn't be imported.")
+
+ if hasattr(data, "read"):
+ data = data.read()
+
+ for parser in basic_parsers:
+ print(("Trying to parse your markup with %s" % parser))
+ success = False
+ try:
+ soup = BeautifulSoup(data, features=parser)
+ success = True
+ except Exception:
+ print(("%s could not parse the markup." % parser))
+ traceback.print_exc()
+ if success:
+ print(("Here's what %s did with the markup:" % parser))
+ print((soup.prettify()))
+
+ print(("-" * 80))
+
+
+def lxml_trace(data: "_IncomingMarkup", html: bool = True, **kwargs: Any) -> None:
+ """Print out the lxml events that occur during parsing.
+
+ This lets you see how lxml parses a document when no Beautiful
+ Soup code is running. You can use this to determine whether
+ an lxml-specific problem is in Beautiful Soup's lxml tree builders
+ or in lxml itself.
+
+ :param data: Some markup.
+ :param html: If True, markup will be parsed with lxml's HTML parser.
+ if False, lxml's XML parser will be used.
+ """
+ from lxml import etree
+
+ recover = kwargs.pop("recover", True)
+ if isinstance(data, str):
+ data = data.encode("utf8")
+ if not isinstance(data, IO):
+ reader = BytesIO(data)
+ for event, element in etree.iterparse(reader, html=html, recover=recover, **kwargs):
+ print(("%s, %4s, %s" % (event, element.tag, element.text)))
+
+
+class AnnouncingParser(HTMLParser):
+ """Subclass of HTMLParser that announces parse events, without doing
+ anything else.
+
+ You can use this to get a picture of how html.parser sees a given
+ document. The easiest way to do this is to call `htmlparser_trace`.
+ """
+
+ def _p(self, s: str) -> None:
+ print(s)
+
+ def handle_starttag(
+ self,
+ name: str,
+ attrs: List[Tuple[str, Optional[str]]],
+ handle_empty_element: bool = True,
+ ) -> None:
+ self._p(f"{name} {attrs} START")
+
+ def handle_endtag(self, name: str, check_already_closed: bool = True) -> None:
+ self._p("%s END" % name)
+
+ def handle_data(self, data: str) -> None:
+ self._p("%s DATA" % data)
+
+ def handle_charref(self, name: str) -> None:
+ self._p("%s CHARREF" % name)
+
+ def handle_entityref(self, name: str) -> None:
+ self._p("%s ENTITYREF" % name)
+
+ def handle_comment(self, data: str) -> None:
+ self._p("%s COMMENT" % data)
+
+ def handle_decl(self, data: str) -> None:
+ self._p("%s DECL" % data)
+
+ def unknown_decl(self, data: str) -> None:
+ self._p("%s UNKNOWN-DECL" % data)
+
+ def handle_pi(self, data: str) -> None:
+ self._p("%s PI" % data)
+
+
+def htmlparser_trace(data: str) -> None:
+ """Print out the HTMLParser events that occur during parsing.
+
+ This lets you see how HTMLParser parses a document when no
+ Beautiful Soup code is running.
+
+ :param data: Some markup.
+ """
+ parser = AnnouncingParser()
+ parser.feed(data)
+
+
+_vowels: str = "aeiou"
+_consonants: str = "bcdfghjklmnpqrstvwxyz"
+
+
+def rword(length: int = 5) -> str:
+ """Generate a random word-like string.
+
+ :meta private:
+ """
+ s = ""
+ for i in range(length):
+ if i % 2 == 0:
+ t = _consonants
+ else:
+ t = _vowels
+ s += random.choice(t)
+ return s
+
+
+def rsentence(length: int = 4) -> str:
+ """Generate a random sentence-like string.
+
+ :meta private:
+ """
+ return " ".join(rword(random.randint(4, 9)) for i in range(length))
+
+
+def rdoc(num_elements: int = 1000) -> str:
+ """Randomly generate an invalid HTML document.
+
+ :meta private:
+ """
+ tag_names = ["p", "div", "span", "i", "b", "script", "table"]
+ elements = []
+ for i in range(num_elements):
+ choice = random.randint(0, 3)
+ if choice == 0:
+ # New tag.
+ tag_name = random.choice(tag_names)
+ elements.append("<%s>" % tag_name)
+ elif choice == 1:
+ elements.append(rsentence(random.randint(1, 4)))
+ elif choice == 2:
+ # Close a tag.
+ tag_name = random.choice(tag_names)
+ elements.append("</%s>" % tag_name)
+ return "<html>" + "\n".join(elements) + "</html>"
+
+
+def benchmark_parsers(num_elements: int = 100000) -> None:
+ """Very basic head-to-head performance benchmark."""
+ print(("Comparative parser benchmark on Beautiful Soup %s" % __version__))
+ data = rdoc(num_elements)
+ print(("Generated a large invalid HTML document (%d bytes)." % len(data)))
+
+ for parser_name in ["lxml", ["lxml", "html"], "html5lib", "html.parser"]:
+ success = False
+ try:
+ a = time.time()
+ BeautifulSoup(data, parser_name)
+ b = time.time()
+ success = True
+ except Exception:
+ print(("%s could not parse the markup." % parser_name))
+ traceback.print_exc()
+ if success:
+ print(("BS4+%s parsed the markup in %.2fs." % (parser_name, b - a)))
+
+ from lxml import etree
+
+ a = time.time()
+ etree.HTML(data)
+ b = time.time()
+ print(("Raw lxml parsed the markup in %.2fs." % (b - a)))
+
+ import html5lib
+
+ parser = html5lib.HTMLParser()
+ a = time.time()
+ parser.parse(data)
+ b = time.time()
+ print(("Raw html5lib parsed the markup in %.2fs." % (b - a)))
+
+
+def profile(num_elements: int = 100000, parser: str = "lxml") -> None:
+ """Use Python's profiler on a randomly generated document."""
+ filehandle = tempfile.NamedTemporaryFile()
+ filename = filehandle.name
+
+ data = rdoc(num_elements)
+ vars = dict(bs4=bs4, data=data, parser=parser)
+ cProfile.runctx("bs4.BeautifulSoup(data, parser)", vars, vars, filename)
+
+ stats = pstats.Stats(filename)
+ # stats.strip_dirs()
+ stats.sort_stats("cumulative")
+ stats.print_stats("_html5lib|bs4", 50)
+
+
+# If this file is run as a script, standard input is diagnosed.
+if __name__ == "__main__":
+ diagnose(sys.stdin.read())
diff --git a/bitbake/lib/bb/_vendor/bs4/element.py b/bitbake/lib/bb/_vendor/bs4/element.py
new file mode 100644
index 00000000..e7f97fde
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/element.py
@@ -0,0 +1,3395 @@
+from __future__ import annotations
+
+# Use of this source code is governed by the MIT license.
+__license__ = "MIT"
+
+import inspect
+import re
+import warnings
+
+from bb._vendor.bs4.css import CSS
+from bb._vendor.bs4._deprecation import (
+ _deprecated,
+ _deprecated_alias,
+ _deprecated_function_alias,
+)
+from bb._vendor.bs4.formatter import (
+ Formatter,
+ HTMLFormatter,
+ XMLFormatter,
+)
+from bb._vendor.bs4._warnings import AttributeResemblesVariableWarning
+
+from typing import (
+ Any,
+ Callable,
+ Dict,
+ Generic,
+ Iterable,
+ Iterator,
+ List,
+ Mapping,
+ MutableSequence,
+ Optional,
+ Pattern,
+ Set,
+ TYPE_CHECKING,
+ Tuple,
+ Type,
+ TypeVar,
+ Union,
+ cast,
+ overload,
+)
+from bb._vendor.typing_extensions import (
+ Self,
+ TypeAlias,
+)
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4 import BeautifulSoup
+ from bb._vendor.bs4.builder import TreeBuilder
+ from bb._vendor.bs4.filter import ElementFilter
+ from bb._vendor.bs4.formatter import (
+ _EntitySubstitutionFunction,
+ _FormatterOrName,
+ )
+ from bb._vendor.bs4._typing import (
+ _AtMostOneElement,
+ _AtMostOneNavigableString,
+ _AtMostOneTag,
+ _AttributeValue,
+ _AttributeValues,
+ _Encoding,
+ _InsertableElement,
+ _OneElement,
+ _QueryResults,
+ _RawAttributeValue,
+ _RawAttributeValues,
+ _RawOrProcessedAttributeValues,
+ _SomeNavigableStrings,
+ _SomeTags,
+ _StrainableAttribute,
+ _StrainableAttributes,
+ _StrainableElement,
+ _StrainableString,
+ )
+
+_OneOrMoreStringTypes: TypeAlias = Union[
+ Type["NavigableString"], Iterable[Type["NavigableString"]]
+]
+
+_FindMethodName: TypeAlias = Union["_StrainableElement", "ElementFilter"]
+_OptionalFindMethodName: TypeAlias = Optional[_FindMethodName]
+
+# Deprecated module-level attributes.
+# See https://peps.python.org/pep-0562/
+_deprecated_names = dict(
+ whitespace_re="The {name} attribute was deprecated in version 4.7.0. If you need it, make your own copy."
+)
+#: :meta private:
+_deprecated_whitespace_re: Pattern[str] = re.compile(r"\s+")
+
+
+def __getattr__(name: str) -> Any:
+ if name in _deprecated_names:
+ message = _deprecated_names[name]
+ warnings.warn(message.format(name=name), DeprecationWarning, stacklevel=2)
+
+ return globals()[f"_deprecated_{name}"]
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+#: Documents output by Beautiful Soup will be encoded with
+#: this encoding unless you specify otherwise.
+DEFAULT_OUTPUT_ENCODING: str = "utf-8"
+
+#: A regular expression that can be used to split on whitespace.
+nonwhitespace_re: Pattern[str] = re.compile(r"\S+")
+
+#: These encodings are recognized by Python (so `Tag.encode`
+#: could theoretically support them) but XML and HTML don't recognize
+#: them (so they should not show up in an XML or HTML document as that
+#: document's encoding).
+#:
+#: If an XML document is encoded in one of these encodings, no encoding
+#: will be mentioned in the XML declaration. If an HTML document is
+#: encoded in one of these encodings, and the HTML document has a
+#: <meta> tag that mentions an encoding, the encoding will be given as
+#: the empty string.
+#:
+#: Source:
+#: Python documentation, `Python Specific Encodings <https://docs.python.org/3/library/codecs.html#python-specific-encodings>`_
+PYTHON_SPECIFIC_ENCODINGS: Set[_Encoding] = set(
+ [
+ "idna",
+ "mbcs",
+ "oem",
+ "palmos",
+ "punycode",
+ "raw_unicode_escape",
+ "undefined",
+ "unicode_escape",
+ "raw-unicode-escape",
+ "unicode-escape",
+ "string-escape",
+ "string_escape",
+ ]
+)
+
+
+class NamespacedAttribute(str):
+ """A namespaced attribute (e.g. the 'xml:lang' in 'xml:lang="en"')
+ which remembers the namespace prefix ('xml') and the name ('lang')
+ that were used to create it.
+ """
+
+ prefix: Optional[str]
+ name: Optional[str]
+ namespace: Optional[str]
+
+ def __new__(
+ cls,
+ prefix: Optional[str],
+ name: Optional[str] = None,
+ namespace: Optional[str] = None,
+ ) -> Self:
+ if not name:
+ # This is the default namespace. Its name "has no value"
+ # per https://www.w3.org/TR/xml-names/#defaulting
+ name = None
+
+ if not name:
+ obj = str.__new__(cls, prefix)
+ elif not prefix:
+ # Not really namespaced.
+ obj = str.__new__(cls, name)
+ else:
+ obj = str.__new__(cls, prefix + ":" + name)
+ obj.prefix = prefix
+ obj.name = name
+ obj.namespace = namespace
+ return obj
+
+
+class AttributeValueWithCharsetSubstitution(str):
+ """An abstract class standing in for a character encoding specified
+ inside an HTML ``<meta>`` tag.
+
+ Subclasses exist for each place such a character encoding might be
+ found: either inside the ``charset`` attribute
+ (`CharsetMetaAttributeValue`) or inside the ``content`` attribute
+ (`ContentMetaAttributeValue`)
+
+ This allows Beautiful Soup to replace that part of the HTML file
+ with a different encoding when ouputting a tree as a string.
+ """
+
+ # The original, un-encoded value of the ``content`` attribute.
+ #: :meta private:
+ original_value: str
+
+ def substitute_encoding(self, eventual_encoding: str) -> str:
+ """Do whatever's necessary in this implementation-specific
+ portion an HTML document to substitute in a specific encoding.
+ """
+ raise NotImplementedError()
+
+
+class CharsetMetaAttributeValue(AttributeValueWithCharsetSubstitution):
+ """A generic stand-in for the value of a ``<meta>`` tag's ``charset``
+ attribute.
+
+ When Beautiful Soup parses the markup ``<meta charset="utf8">``, the
+ value of the ``charset`` attribute will become one of these objects.
+
+ If the document is later encoded to an encoding other than UTF-8, its
+ ``<meta>`` tag will mention the new encoding instead of ``utf8``.
+ """
+
+ def __new__(cls, original_value: str) -> Self:
+ # We don't need to use the original value for anything, but
+ # it might be useful for the user to know.
+ obj = str.__new__(cls, original_value)
+ obj.original_value = original_value
+ return obj
+
+ def substitute_encoding(self, eventual_encoding: _Encoding = "utf-8") -> str:
+ """When an HTML document is being encoded to a given encoding, the
+ value of a ``<meta>`` tag's ``charset`` becomes the name of
+ the encoding.
+ """
+ if eventual_encoding in PYTHON_SPECIFIC_ENCODINGS:
+ return ""
+ return eventual_encoding
+
+
+class AttributeValueList(List[str]):
+ """Class for the list used to hold the values of attributes which
+ have multiple values (such as HTML's 'class'). It's just a regular
+ list, but you can subclass it and pass it in to the TreeBuilder
+ constructor as attribute_value_list_class, to have your subclass
+ instantiated instead.
+ """
+
+
+class AttributeDict(Dict[Any,Any]):
+ """Superclass for the dictionary used to hold a tag's
+ attributes. You can use this, but it's just a regular dict with no
+ special logic.
+ """
+
+
+class XMLAttributeDict(AttributeDict):
+ """A dictionary for holding a Tag's attributes, which processes
+ incoming values for consistency with the HTML spec.
+ """
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set an attribute value, possibly modifying it to comply with
+ the XML spec.
+
+ This just means converting common non-string values to
+ strings: XML attributes may have "any literal string as a
+ value."
+ """
+ if value is None:
+ value = ""
+ if isinstance(value, bool):
+ # XML does not define any rules for boolean attributes.
+ # Preserve the old Beautiful Soup behavior (a bool that
+ # gets converted to a string on output) rather than
+ # guessing what the value should be.
+ pass
+ elif isinstance(value, (int, float)):
+ # It's dangerous to convert _every_ attribute value into a
+ # plain string, since an attribute value may be a more
+ # sophisticated string-like object
+ # (e.g. CharsetMetaAttributeValue). But we can definitely
+ # convert numeric values and booleans, which are the most common.
+ value = str(value)
+
+ super().__setitem__(key, value)
+
+
+class HTMLAttributeDict(AttributeDict):
+ """A dictionary for holding a Tag's attributes, which processes
+ incoming values for consistency with the HTML spec, which says
+ 'Attribute values are a mixture of text and character
+ references...'
+
+ Basically, this means converting common non-string values into
+ strings, like XMLAttributeDict, though HTML also has some rules
+ around boolean attributes that XML doesn't have.
+ """
+
+ def __setitem__(self, key: str, value: Any) -> None:
+ """Set an attribute value, possibly modifying it to comply
+ with the HTML spec,
+ """
+ if value in (False, None):
+ # 'The values "true" and "false" are not allowed on
+ # boolean attributes. To represent a false value, the
+ # attribute has to be omitted altogether.'
+ if key in self:
+ del self[key]
+ return
+ if isinstance(value, bool):
+ # 'If the [boolean] attribute is present, its value must
+ # either be the empty string or a value that is an ASCII
+ # case-insensitive match for the attribute's canonical
+ # name, with no leading or trailing whitespace.'
+ #
+ # [fixme] It's not clear to me whether "canonical name"
+ # means fully-qualified name, unqualified name, or
+ # (probably not) name with namespace prefix. For now I'm
+ # going with unqualified name.
+ if isinstance(key, NamespacedAttribute):
+ value = key.name
+ else:
+ value = key
+ elif isinstance(value, (int, float)):
+ # See note in XMLAttributeDict for the reasoning why we
+ # only do this to numbers.
+ value = str(value)
+ super().__setitem__(key, value)
+
+
+class ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution):
+ """A generic stand-in for the value of a ``<meta>`` tag's ``content``
+ attribute.
+
+ When Beautiful Soup parses the markup:
+ ``<meta http-equiv="content-type" content="text/html; charset=utf8">``
+
+ The value of the ``content`` attribute will become one of these objects.
+
+ If the document is later encoded to an encoding other than UTF-8, its
+ ``<meta>`` tag will mention the new encoding instead of ``utf8``.
+ """
+
+ #: Match the 'charset' argument inside the 'content' attribute
+ #: of a <meta> tag.
+ #: :meta private:
+ CHARSET_RE: Pattern[str] = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
+
+ def __new__(cls, original_value: str) -> Self:
+ cls.CHARSET_RE.search(original_value)
+ obj = str.__new__(cls, original_value)
+ obj.original_value = original_value
+ return obj
+
+ def substitute_encoding(self, eventual_encoding: _Encoding = "utf-8") -> str:
+ """When an HTML document is being encoded to a given encoding, the
+ value of the ``charset=`` in a ``<meta>`` tag's ``content`` becomes
+ the name of the encoding.
+ """
+ if eventual_encoding in PYTHON_SPECIFIC_ENCODINGS:
+ return self.CHARSET_RE.sub("", self.original_value)
+
+ def rewrite(match: re.Match[str]) -> str:
+ return match.group(1) + eventual_encoding
+
+ return self.CHARSET_RE.sub(rewrite, self.original_value)
+
+
+class PageElement(object):
+ """An abstract class representing a single element in the parse tree.
+
+ `NavigableString`, `Tag`, etc. are all subclasses of
+ `PageElement`. For this reason you'll see a lot of methods that
+ return `PageElement`, but you'll never see an actual `PageElement`
+ object. For the most part you can think of `PageElement` as
+ meaning "a `Tag` or a `NavigableString`."
+ """
+
+ #: In general, we can't tell just by looking at an element whether
+ #: it's contained in an XML document or an HTML document. But for
+ #: `Tag` objects (q.v.) we can store this information at parse time.
+ #: :meta private:
+ known_xml: Optional[bool] = None
+
+ #: Whether or not this element has been decomposed from the tree
+ #: it was created in.
+ _decomposed: bool
+
+ parent: Optional[Tag]
+ next_element: _AtMostOneElement
+ previous_element: _AtMostOneElement
+ next_sibling: _AtMostOneElement
+ previous_sibling: _AtMostOneElement
+
+ #: Whether or not this element is hidden from generated output.
+ #: Only the `BeautifulSoup` object itself is hidden.
+ hidden: bool = False
+
+ def setup(
+ self,
+ parent: Optional[Tag] = None,
+ previous_element: _AtMostOneElement = None,
+ next_element: _AtMostOneElement = None,
+ previous_sibling: _AtMostOneElement = None,
+ next_sibling: _AtMostOneElement = None,
+ ) -> None:
+ """Sets up the initial relations between this element and
+ other elements.
+
+ :param parent: The parent of this element.
+
+ :param previous_element: The element parsed immediately before
+ this one.
+
+ :param next_element: The element parsed immediately after
+ this one.
+
+ :param previous_sibling: The most recently encountered element
+ on the same level of the parse tree as this one.
+
+ :param previous_sibling: The next element to be encountered
+ on the same level of the parse tree as this one.
+ """
+ self.parent = parent
+
+ self.previous_element = previous_element
+ if self.previous_element is not None:
+ self.previous_element.next_element = self
+
+ self.next_element = next_element
+ if self.next_element is not None:
+ self.next_element.previous_element = self
+
+ self.next_sibling = next_sibling
+ if self.next_sibling is not None:
+ self.next_sibling.previous_sibling = self
+
+ if (
+ previous_sibling is None
+ and self.parent is not None
+ and self.parent.contents
+ ):
+ previous_sibling = self.parent.contents[-1]
+
+ self.previous_sibling = previous_sibling
+ if self.previous_sibling is not None:
+ self.previous_sibling.next_sibling = self
+
+ def format_string(self, s: str, formatter: Optional[_FormatterOrName]) -> str:
+ """Format the given string using the given formatter.
+
+ :param s: A string.
+ :param formatter: A Formatter object, or a string naming one of the standard formatters.
+ """
+ if formatter is None:
+ return s
+ if not isinstance(formatter, Formatter):
+ formatter = self.formatter_for_name(formatter)
+ output = formatter.substitute(s)
+ return output
+
+ def formatter_for_name(
+ self, formatter_name: Union[_FormatterOrName, _EntitySubstitutionFunction]
+ ) -> Formatter:
+ """Look up or create a Formatter for the given identifier,
+ if necessary.
+
+ :param formatter: Can be a `Formatter` object (used as-is), a
+ function (used as the entity substitution hook for an
+ `bs4.formatter.XMLFormatter` or
+ `bs4.formatter.HTMLFormatter`), or a string (used to look
+ up an `bs4.formatter.XMLFormatter` or
+ `bs4.formatter.HTMLFormatter` in the appropriate registry.
+
+ """
+ if isinstance(formatter_name, Formatter):
+ return formatter_name
+ c: type[Formatter]
+ registry: Mapping[Optional[str], Formatter]
+ if self._is_xml:
+ c = XMLFormatter
+ registry = XMLFormatter.REGISTRY
+ else:
+ c = HTMLFormatter
+ registry = HTMLFormatter.REGISTRY
+ if callable(formatter_name):
+ return c(entity_substitution=formatter_name)
+ return registry[formatter_name]
+
+ @property
+ def _is_xml(self) -> bool:
+ """Is this element part of an XML tree or an HTML tree?
+
+ This is used in formatter_for_name, when deciding whether an
+ XMLFormatter or HTMLFormatter is more appropriate. It can be
+ inefficient, but it should be called very rarely.
+ """
+ if self.known_xml is not None:
+ # Most of the time we will have determined this when the
+ # document is parsed.
+ return self.known_xml
+
+ # Otherwise, it's likely that this element was created by
+ # direct invocation of the constructor from within the user's
+ # Python code.
+ if self.parent is None:
+ # This is the top-level object. It should have .known_xml set
+ # from tree creation. If not, take a guess--BS is usually
+ # used on HTML markup.
+ return getattr(self, "is_xml", False)
+ return self.parent._is_xml
+
+ nextSibling = _deprecated_alias("nextSibling", "next_sibling", "4.0.0")
+ previousSibling = _deprecated_alias("previousSibling", "previous_sibling", "4.0.0")
+
+ def __deepcopy__(self, memo: Dict[Any, Any], recursive: bool = False) -> Self:
+ raise NotImplementedError()
+
+ def __copy__(self) -> Self:
+ """A copy of a PageElement can only be a deep copy, because
+ only one PageElement can occupy a given place in a parse tree.
+ """
+ return self.__deepcopy__({})
+
+ default: Iterable[type[NavigableString]] = tuple() #: :meta private:
+
+ def _all_strings(
+ self, strip: bool = False, types: Iterable[type[NavigableString]] = default
+ ) -> Iterator[str]:
+ """Yield all strings of certain classes, possibly stripping them.
+
+ This is implemented differently in `Tag` and `NavigableString`.
+ """
+ raise NotImplementedError()
+
+ @property
+ def stripped_strings(self) -> Iterator[str]:
+ """Yield all interesting strings in this PageElement, stripping them
+ first.
+
+ See `Tag` for information on which strings are considered
+ interesting in a given context.
+ """
+ for string in self._all_strings(True):
+ yield string
+
+ def get_text(
+ self,
+ separator: str = "",
+ strip: bool = False,
+ types: Iterable[Type[NavigableString]] = default,
+ ) -> str:
+ """Get all child strings of this PageElement, concatenated using the
+ given separator.
+
+ :param separator: Strings will be concatenated using this separator.
+
+ :param strip: If True, strings will be stripped before being
+ concatenated.
+
+ :param types: A tuple of NavigableString subclasses. Any
+ strings of a subclass not found in this list will be
+ ignored. Although there are exceptions, the default
+ behavior in most cases is to consider only NavigableString
+ and CData objects. That means no comments, processing
+ instructions, etc.
+
+ :return: A string.
+ """
+ return separator.join([s for s in self._all_strings(strip, types=types)])
+
+ getText = get_text
+
+ @property
+ def text(self) -> str:
+ return self.get_text()
+
+ def replace_with(self, *args: _InsertableElement) -> Self:
+ """Replace this `PageElement` with one or more other elements,
+ objects, keeping the rest of the tree the same.
+
+ :return: This `PageElement`, no longer part of the tree.
+ """
+ if self.parent is None:
+ raise ValueError(
+ "Cannot replace one element with another when the "
+ "element to be replaced is not part of a tree."
+ )
+ if len(args) == 1 and args[0] is self:
+ # Replacing an element with itself is a no-op.
+ return self
+ if any(x is self.parent for x in args):
+ raise ValueError("Cannot replace a Tag with its parent.")
+ old_parent = self.parent
+ my_index = self.parent.index(self)
+ self.extract(_self_index=my_index)
+ for idx, replace_with in enumerate(args, start=my_index):
+ old_parent.insert(idx, replace_with)
+ return self
+
+ replaceWith = _deprecated_function_alias("replaceWith", "replace_with", "4.0.0")
+
+ def wrap(self, wrap_inside: Tag) -> Tag:
+ """Wrap this `PageElement` inside a `Tag`.
+
+ :return: ``wrap_inside``, occupying the position in the tree that used
+ to be occupied by this object, and with this object now inside it.
+ """
+ me = self.replace_with(wrap_inside)
+ wrap_inside.append(me)
+ return wrap_inside
+
+ def extract(self, _self_index: Optional[int] = None) -> Self:
+ """Destructively rips this element out of the tree.
+
+ :param _self_index: The location of this element in its parent's
+ .contents, if known. Passing this in allows for a performance
+ optimization.
+
+ :return: this `PageElement`, no longer part of the tree.
+ """
+ if self.parent is not None:
+ if _self_index is None:
+ _self_index = self.parent.index(self)
+ del self.parent.contents[_self_index]
+
+ # Find the two elements that would be next to each other if
+ # this element (and any children) hadn't been parsed. Connect
+ # the two.
+ last_child = self._last_descendant()
+
+ # last_child can't be None because we passed accept_self=True
+ # into _last_descendant. Worst case, last_child will be
+ # self. Making this cast removes several mypy complaints later
+ # on as we manipulate last_child.
+ last_child = cast(PageElement, last_child)
+ next_element = last_child.next_element
+
+ if self.previous_element is not None:
+ if self.previous_element is not next_element:
+ self.previous_element.next_element = next_element
+ if next_element is not None and next_element is not self.previous_element:
+ next_element.previous_element = self.previous_element
+ self.previous_element = None
+ last_child.next_element = None
+
+ self.parent = None
+ if (
+ self.previous_sibling is not None
+ and self.previous_sibling is not self.next_sibling
+ ):
+ self.previous_sibling.next_sibling = self.next_sibling
+ if (
+ self.next_sibling is not None
+ and self.next_sibling is not self.previous_sibling
+ ):
+ self.next_sibling.previous_sibling = self.previous_sibling
+ self.previous_sibling = self.next_sibling = None
+ return self
+
+ def decompose(self) -> None:
+ """Recursively destroys this `PageElement` and its children.
+
+ The element will be removed from the tree and wiped out; so
+ will everything beneath it.
+
+ The behavior of a decomposed `PageElement` is undefined and you
+ should never use one for anything, but if you need to *check*
+ whether an element has been decomposed, you can use the
+ `PageElement.decomposed` property.
+ """
+ self.extract()
+ e: _AtMostOneElement = self
+ next_up: _AtMostOneElement = None
+ while e is not None:
+ next_up = e.next_element
+ e.__dict__.clear()
+ if isinstance(e, Tag):
+ e.name = ""
+ e.contents = []
+ e._decomposed = True
+ e = next_up
+
+ def _last_descendant(
+ self, is_initialized: bool = True, accept_self: bool = True
+ ) -> _AtMostOneElement:
+ """Finds the last element beneath this object to be parsed.
+
+ Special note to help you figure things out if your type
+ checking is tripped up by the fact that this method returns
+ _AtMostOneElement instead of PageElement: the only time
+ this method returns None is if `accept_self` is False and the
+ `PageElement` has no children--either it's a NavigableString
+ or an empty Tag.
+
+ :param is_initialized: Has `PageElement.setup` been called on
+ this `PageElement` yet?
+
+ :param accept_self: Is ``self`` an acceptable answer to the
+ question?
+ """
+ if is_initialized and self.next_sibling is not None:
+ last_child = self.next_sibling.previous_element
+ else:
+ last_child = self
+ while isinstance(last_child, Tag) and last_child.contents:
+ last_child = last_child.contents[-1]
+ if not accept_self and last_child is self:
+ last_child = None
+ return last_child
+
+ _lastRecursiveChild = _deprecated_alias(
+ "_lastRecursiveChild", "_last_descendant", "4.0.0"
+ )
+
+ def insert_before(self, *args: _InsertableElement) -> List[PageElement]:
+ """Makes the given element(s) the immediate predecessor of this one.
+
+ All the elements will have the same `PageElement.parent` as
+ this one, and the given elements will occur immediately before
+ this one.
+
+ :param args: One or more PageElements.
+
+ :return The list of PageElements that were inserted.
+ """
+ parent = self.parent
+ if parent is None:
+ raise ValueError("Element has no parent, so 'before' has no meaning.")
+ if any(x is self for x in args):
+ raise ValueError("Can't insert an element before itself.")
+ results: List[PageElement] = []
+ for predecessor in args:
+ # Extract first so that the index won't be screwed up if they
+ # are siblings.
+ if isinstance(predecessor, PageElement):
+ predecessor.extract()
+ index = parent.index(self)
+ results.extend(parent.insert(index, predecessor))
+
+ return results
+
+ def insert_after(self, *args: _InsertableElement) -> List[PageElement]:
+ """Makes the given element(s) the immediate successor of this one.
+
+ The elements will have the same `PageElement.parent` as this
+ one, and the given elements will occur immediately after this
+ one.
+
+ :param args: One or more PageElements.
+
+ :return The list of PageElements that were inserted.
+ """
+ # Do all error checking before modifying the tree.
+ parent = self.parent
+ if parent is None:
+ raise ValueError("Element has no parent, so 'after' has no meaning.")
+ if any(x is self for x in args):
+ raise ValueError("Can't insert an element after itself.")
+
+ offset = 0
+ results: List[PageElement] = []
+ for successor in args:
+ # Extract first so that the index won't be screwed up if they
+ # are siblings.
+ if isinstance(successor, PageElement):
+ successor.extract()
+ index = parent.index(self)
+ results.extend(parent.insert(index + 1 + offset, successor))
+ offset += 1
+
+ return results
+
+ def new_tag(
+ self,
+ name: str,
+ namespace: Optional[str] = None,
+ nsprefix: Optional[str] = None,
+ attrs: Optional[_RawAttributeValues] = None,
+ sourceline: Optional[int] = None,
+ sourcepos: Optional[int] = None,
+ string: Optional[str] = None,
+ **kwattrs: _RawAttributeValue,
+ ) -> Tag:
+ """Create a new Tag associated with the same BeautifulSoup object as this PageElement is."""
+ root = self._root_object
+ if root is None:
+ raise ValueError("Cannot call new_tag on a PageElement not contained in a BeautifulSoup object")
+ return root.new_tag(name, namespace, nsprefix, attrs, sourceline, sourcepos, string, **kwattrs)
+
+ def new_string(self, s: str, subclass: Optional[Type[NavigableString]] = None
+ ) -> NavigableString:
+ """Create a new NavigableString associated with the same BeautifulSoup object as this PageElement is."""
+ root = self._root_object
+ if root is None:
+ raise ValueError("Cannot call new_string on a PageElement not contained in a BeautifulSoup object")
+ return root.new_string(s, subclass)
+
+ @property
+ def _root_object(self) -> Optional[BeautifulSoup]:
+ """Find the BeautifulSoup object used to create this PageElement, assuming it's still attached."""
+ parent:Optional[Tag] = self.parent
+ while parent is not None and not parent._is_root:
+ parent = parent.parent
+ if parent is None:
+ return parent
+ return cast('BeautifulSoup', parent)
+
+ @property
+ def _is_root(self) -> bool:
+ """No, this object is not the root of its parse tree; only a BeautifulSoup object can be that."""
+ return False
+
+ # No name or attrs + string -> string
+ @overload
+ def find_next(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneNavigableString:
+ ...
+
+ # No string -> tag
+ @overload
+ def find_next(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None=None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ def find_next(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_AtMostOneTag,_AtMostOneNavigableString,_AtMostOneElement]:
+ """Find the first PageElement that matches the given criteria and
+ appears later in the document than this PageElement.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a NavigableString with specific text.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_one(self.find_all_next, name, attrs, string, **kwargs)
+
+ findNext = _deprecated_function_alias("findNext", "find_next", "4.0.0")
+
+ # No name or attrs + string -> strings
+ @overload
+ def find_all_next(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ # No string -> tags
+ @overload
+ def find_all_next(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ def find_all_next(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings,_QueryResults]:
+ """Find all `PageElement` objects that match the given criteria and
+ appear later in the document than this `PageElement`.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a NavigableString with specific text.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_all(
+ name,
+ attrs,
+ string,
+ limit,
+ self.next_elements,
+ **kwargs,
+ )
+
+ findAllNext = _deprecated_function_alias("findAllNext", "find_all_next", "4.0.0")
+
+ # No name or attrs + string -> strings
+ @overload
+ def find_next_sibling(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneNavigableString:
+ ...
+
+ # No string -> tags
+ @overload
+ def find_next_sibling(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ def find_next_sibling(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_AtMostOneTag,_AtMostOneNavigableString,_AtMostOneElement]:
+ """Find the closest sibling to this PageElement that matches the
+ given criteria and appears later in the document.
+
+ All find_* methods take a common set of arguments. See the
+ online documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a `NavigableString` with specific text.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_one(self.find_next_siblings, name, attrs, string, **kwargs)
+
+ findNextSibling = _deprecated_function_alias(
+ "findNextSibling", "find_next_sibling", "4.0.0"
+ )
+
+ # No name or attrs + string -> strings
+ @overload
+ def find_next_siblings(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ # No string -> tags
+ @overload
+ def find_next_siblings(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ def find_next_siblings(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings,_QueryResults]:
+ """Find all siblings of this `PageElement` that match the given criteria
+ and appear later in the document.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a `NavigableString` with specific text.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_all(
+ name,
+ attrs,
+ string,
+ limit,
+ self.next_siblings,
+ **kwargs,
+ )
+
+ findNextSiblings = _deprecated_function_alias(
+ "findNextSiblings", "find_next_siblings", "4.0.0"
+ )
+ fetchNextSiblings = _deprecated_function_alias(
+ "fetchNextSiblings", "find_next_siblings", "3.0.0"
+ )
+
+ # No name or attrs + string -> string
+ @overload
+ def find_previous(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneNavigableString:
+ ...
+
+ # No string -> tag
+ @overload
+ def find_previous(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None=None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ def find_previous(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_AtMostOneTag,_AtMostOneNavigableString,_AtMostOneElement]:
+ """Look backwards in the document from this `PageElement` and find the
+ first `PageElement` that matches the given criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a `NavigableString` with specific text.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_one(self.find_all_previous, name, attrs, string, **kwargs)
+
+ findPrevious = _deprecated_function_alias("findPrevious", "find_previous", "3.0.0")
+
+ # No name or attrs + string -> strings
+ @overload
+ def find_all_previous(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ # No string -> tags
+ @overload
+ def find_all_previous(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ def find_all_previous(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings,_QueryResults]:
+ """Look backwards in the document from this `PageElement` and find all
+ `PageElement` that match the given criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a `NavigableString` with specific text.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_all(
+ name,
+ attrs,
+ string,
+ limit,
+ self.previous_elements,
+ **kwargs,
+ )
+
+ findAllPrevious = _deprecated_function_alias(
+ "findAllPrevious", "find_all_previous", "4.0.0"
+ )
+ fetchAllPrevious = _deprecated_function_alias(
+ "fetchAllPrevious", "find_all_previous", "3.0.0"
+ )
+
+ # No name or attrs + string -> string
+ @overload
+ def find_previous_sibling(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneNavigableString:
+ ...
+
+ # No string -> tag
+ @overload
+ def find_previous_sibling(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ def find_previous_sibling(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_AtMostOneTag,_AtMostOneNavigableString,_AtMostOneElement]:
+ """Returns the closest sibling to this `PageElement` that matches the
+ given criteria and appears earlier in the document.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a `NavigableString` with specific text.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_one(
+ self.find_previous_siblings, name, attrs, string, **kwargs
+ )
+
+ findPreviousSibling = _deprecated_function_alias(
+ "findPreviousSibling", "find_previous_sibling", "4.0.0"
+ )
+
+ # No name or attrs + string -> strings
+ @overload
+ def find_previous_siblings(
+ self,
+ name: None = None,
+ attrs: None = None,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ # No string -> tags
+ @overload
+ def find_previous_siblings(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ def find_previous_siblings(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings,_QueryResults]:
+ """Returns all siblings to this PageElement that match the
+ given criteria and appear earlier in the document.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param string: A filter for a NavigableString with specific text.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ return self._find_all(
+ name,
+ attrs,
+ string,
+ limit,
+ self.previous_siblings,
+ **kwargs,
+ )
+
+ findPreviousSiblings = _deprecated_function_alias(
+ "findPreviousSiblings", "find_previous_siblings", "4.0.0"
+ )
+ fetchPreviousSiblings = _deprecated_function_alias(
+ "fetchPreviousSiblings", "find_previous_siblings", "3.0.0"
+ )
+
+ def find_parent(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ """Find the closest parent of this PageElement that matches the given
+ criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param self: Whether the PageElement itself should be considered
+ as one of its 'parents'.
+ :kwargs: Additional filters on attribute values.
+ """
+ # NOTE: We can't use _find_one because findParents takes a different
+ # set of arguments.
+ r = None
+ results = self.find_parents(
+ name, attrs, 1, **kwargs
+ )
+ if results:
+ r = results[0]
+ return r
+
+ findParent = _deprecated_function_alias("findParent", "find_parent", "4.0.0")
+
+ def find_parents(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ """Find all parents of this `PageElement` that match the given criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ iterator = self.parents
+ # Only Tags can have children, so this ResultSet will contain
+ # nothing but Tags.
+ return cast(ResultSet[Tag], self._find_all(
+ name, attrs, None, limit, iterator, **kwargs
+ ))
+
+ findParents = _deprecated_function_alias("findParents", "find_parents", "4.0.0")
+ fetchParents = _deprecated_function_alias("fetchParents", "find_parents", "3.0.0")
+
+ @property
+ def next(self) -> _AtMostOneElement:
+ """The `PageElement`, if any, that was parsed just after this one."""
+ return self.next_element
+
+ @property
+ def previous(self) -> _AtMostOneElement:
+ """The `PageElement`, if any, that was parsed just before this one."""
+ return self.previous_element
+
+ # These methods do the real heavy lifting.
+
+ def _find_one(
+ self,
+ # TODO-TYPING: "There is no syntax to indicate optional or
+ # keyword arguments; such function types are rarely used
+ # as callback types." - So, not sure how to get more
+ # specific here.
+ method: Callable,
+ name: _OptionalFindMethodName,
+ attrs: Optional[_StrainableAttributes],
+ string: Optional[_StrainableString],
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneElement:
+ r: _AtMostOneElement = None
+ results: _QueryResults = method(name, attrs, string, 1, **kwargs)
+ if results:
+ r = results[0]
+ return r
+
+ @property
+ def _warning_stack_level(self) -> int:
+ """Find the appropriate stack level to use when issuing a warning relating to one of the find* methods."""
+ # The find* methods call each other, which makes it
+ # difficult to track how deep we are in the stack
+ # vis-a-vis the caller's entry point into the bs4.element
+ # module. However, we know that all of the find* methods
+ # are in bs4.element, and there's no code in this module
+ # that triggers the warnings we need to issue.
+ #
+ # (There is _test_ code that triggers the warnings, but that's
+ # in bs4.tests.)
+ #
+ # Therefore we can go up the stack until we leave the
+ # bs4.element module, and use the distance between here and
+ # there as the stacklevel.
+ stacklevel = 0
+ for frameinfo in inspect.stack(context=0):
+ if (frameinfo.frame is not None
+ and frameinfo.frame.f_globals is not None
+ and frameinfo.frame.f_globals.get('__name__', '') != "bs4.element"):
+ break
+ stacklevel += 1
+ return stacklevel
+
+ def _find_all(
+ self,
+ name: _OptionalFindMethodName,
+ attrs: Optional[_StrainableAttributes],
+ string: Optional[_StrainableString],
+ limit: Optional[int],
+ generator: Iterator[PageElement],
+ **kwargs: _StrainableAttribute,
+ ) -> _QueryResults:
+ """Iterates over a generator looking for things that match."""
+
+ if string is None and "text" in kwargs:
+ string = kwargs.pop("text")
+ warnings.warn(
+ "The 'text' argument to find()-type methods is deprecated. Use 'string' instead.",
+ DeprecationWarning,
+ stacklevel=self._warning_stack_level,
+ )
+
+ if "_class" in kwargs:
+ warnings.warn(
+ AttributeResemblesVariableWarning.MESSAGE
+ % dict(
+ original="_class",
+ autocorrect="class_",
+ ),
+ AttributeResemblesVariableWarning,
+ stacklevel=self._warning_stack_level,
+ )
+
+ from bb._vendor.bs4.filter import ElementFilter
+
+ if isinstance(name, ElementFilter):
+ matcher = name
+ else:
+ matcher = SoupStrainer(name, attrs, string, **kwargs)
+
+ result: MutableSequence[_OneElement]
+ if string is None and not limit and not attrs and not kwargs:
+ if name is True or name is None:
+ # Optimization to find all tags.
+ result = [element for element in generator if isinstance(element, Tag)]
+ return ResultSet(matcher, result)
+ elif isinstance(name, str):
+ # Optimization to find all tags with a given name.
+ if name.count(":") == 1:
+ # This is a name with a prefix. If this is a namespace-aware document,
+ # we need to match the local name against tag.name. If not,
+ # we need to match the fully-qualified name against tag.name.
+ prefix, local_name = name.split(":", 1)
+ else:
+ prefix = None
+ local_name = name
+ result = []
+ for element in generator:
+ if not isinstance(element, Tag):
+ continue
+ if element.name == name or (
+ element.name == local_name
+ and (prefix is None or element.prefix == prefix)
+ ):
+ result.append(element)
+ return ResultSet(matcher, result)
+ return matcher.find_all(generator, limit)
+
+ # These generators can be used to navigate starting from both
+ # NavigableStrings and Tags.
+ @property
+ def next_elements(self) -> Iterator[PageElement]:
+ """All PageElements that were parsed after this one."""
+ i = self.next_element
+ while i is not None:
+ successor = i.next_element
+ yield i
+ i = successor
+
+ @property
+ def self_and_next_elements(self) -> Iterator[PageElement]:
+ """This PageElement, then all PageElements that were parsed after it."""
+ return self._self_and(self.next_elements)
+
+ @property
+ def next_siblings(self) -> Iterator[PageElement]:
+ """All PageElements that are siblings of this one but were parsed
+ later.
+ """
+ i = self.next_sibling
+ while i is not None:
+ successor = i.next_sibling
+ yield i
+ i = successor
+
+ @property
+ def self_and_next_siblings(self) -> Iterator[PageElement]:
+ """This PageElement, then all of its siblings."""
+ return self._self_and(self.next_siblings)
+
+ @property
+ def previous_elements(self) -> Iterator[PageElement]:
+ """All PageElements that were parsed before this one.
+
+ :yield: A sequence of PageElements.
+ """
+ i = self.previous_element
+ while i is not None:
+ successor = i.previous_element
+ yield i
+ i = successor
+
+ @property
+ def self_and_previous_elements(self) -> Iterator[PageElement]:
+ """This PageElement, then all elements that were parsed
+ earlier."""
+ return self._self_and(self.previous_elements)
+
+ @property
+ def previous_siblings(self) -> Iterator[PageElement]:
+ """All PageElements that are siblings of this one but were parsed
+ earlier.
+
+ :yield: A sequence of PageElements.
+ """
+ i = self.previous_sibling
+ while i is not None:
+ successor = i.previous_sibling
+ yield i
+ i = successor
+
+ @property
+ def self_and_previous_siblings(self) -> Iterator[PageElement]:
+ """This PageElement, then all of its siblings that were parsed
+ earlier."""
+ return self._self_and(self.previous_siblings)
+
+ @property
+ def parents(self) -> Iterator[Tag]:
+ """All elements that are parents of this PageElement.
+
+ :yield: A sequence of Tags, ending with a BeautifulSoup object.
+ """
+ i = self.parent
+ while i is not None:
+ successor = i.parent
+ yield i
+ i = successor
+
+ @property
+ def self_and_parents(self) -> Iterator[PageElement]:
+ """This element, then all of its parents.
+
+ :yield: A sequence of PageElements, ending with a BeautifulSoup object.
+ """
+ return self._self_and(self.parents)
+
+ def _self_and(self, other_generator:Iterator[PageElement]) -> Iterator[PageElement]:
+ """Modify a generator by yielding this element, then everything
+ yielded by the other generator.
+ """
+ if not self.hidden:
+ yield self
+ for i in other_generator:
+ yield i
+
+ @property
+ def decomposed(self) -> bool:
+ """Check whether a PageElement has been decomposed."""
+ return getattr(self, "_decomposed", False) or False
+
+ @_deprecated("next_elements", "4.0.0")
+ def nextGenerator(self) -> Iterator[PageElement]:
+ ":meta private:"
+ return self.next_elements
+
+ @_deprecated("next_siblings", "4.0.0")
+ def nextSiblingGenerator(self) -> Iterator[PageElement]:
+ ":meta private:"
+ return self.next_siblings
+
+ @_deprecated("previous_elements", "4.0.0")
+ def previousGenerator(self) -> Iterator[PageElement]:
+ ":meta private:"
+ return self.previous_elements
+
+ @_deprecated("previous_siblings", "4.0.0")
+ def previousSiblingGenerator(self) -> Iterator[PageElement]:
+ ":meta private:"
+ return self.previous_siblings
+
+ @_deprecated("parents", "4.0.0")
+ def parentGenerator(self) -> Iterator[PageElement]:
+ ":meta private:"
+ return self.parents
+
+
+class NavigableString(str, PageElement):
+ """A Python string that is part of a parse tree.
+
+ When Beautiful Soup parses the markup ``<b>penguin</b>``, it will
+ create a `NavigableString` for the string "penguin".
+ """
+
+ #: A string prepended to the body of the 'real' string
+ #: when formatting it as part of a document, such as the '<!--'
+ #: in an HTML comment.
+ PREFIX: str = ""
+
+ #: A string appended to the body of the 'real' string
+ #: when formatting it as part of a document, such as the '-->'
+ #: in an HTML comment.
+ SUFFIX: str = ""
+
+ def __new__(cls, value: Union[str, bytes]) -> Self:
+ """Create a new NavigableString.
+
+ When unpickling a NavigableString, this method is called with
+ the string in DEFAULT_OUTPUT_ENCODING. That encoding needs to be
+ passed in to the superclass's __new__ or the superclass won't know
+ how to handle non-ASCII characters.
+ """
+ if isinstance(value, str):
+ u = str.__new__(cls, value)
+ else:
+ u = str.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
+ u.hidden = False
+ u.setup()
+ return u
+
+ def __deepcopy__(self, memo: Dict[Any, Any], recursive: bool = False) -> Self:
+ """A copy of a NavigableString has the same contents and class
+ as the original, but it is not connected to the parse tree.
+
+ :param recursive: This parameter is ignored; it's only defined
+ so that NavigableString.__deepcopy__ implements the same
+ signature as Tag.__deepcopy__.
+ """
+ return type(self)(self)
+
+ def __getnewargs__(self) -> Tuple[str]:
+ return (str(self),)
+
+ # TODO-TYPING This should be SupportsIndex|slice but SupportsIndex
+ # is introduced in 3.8. This can be changed once 3.7 support is dropped.
+ def __getitem__(self, key: Union[int|slice]) -> str: # type:ignore
+ """Raise an exception """
+ if isinstance(key, str):
+ raise TypeError("string indices must be integers, not '{0}'. Are you treating a NavigableString like a Tag?".format(key.__class__.__name__))
+ return super(NavigableString, self).__getitem__(key)
+
+ @property
+ def string(self) -> str:
+ """Convenience property defined to match `Tag.string`.
+
+ :return: This property always returns the `NavigableString` it was
+ called on.
+
+ :meta private:
+ """
+ return self
+
+ def output_ready(self, formatter: _FormatterOrName = "minimal") -> str:
+ """Run the string through the provided formatter, making it
+ ready for output as part of an HTML or XML document.
+
+ :param formatter: A `Formatter` object, or a string naming one
+ of the standard formatters.
+ """
+ output = self.format_string(self, formatter)
+ return self.PREFIX + output + self.SUFFIX
+
+ @property
+ def name(self) -> None:
+ """Since a NavigableString is not a Tag, it has no .name.
+
+ This property is implemented so that code like this doesn't crash
+ when run on a mixture of Tag and NavigableString objects:
+ [x.name for x in tag.children]
+
+ :meta private:
+ """
+ return None
+
+ @name.setter
+ def name(self, name: str) -> None:
+ """Prevent NavigableString.name from ever being set.
+
+ :meta private:
+ """
+ raise AttributeError("A NavigableString cannot be given a name.")
+
+ def _all_strings(
+ self, strip: bool = False, types: _OneOrMoreStringTypes = PageElement.default
+ ) -> Iterator[str]:
+ """Yield all strings of certain classes, possibly stripping them.
+
+ This makes it easy for NavigableString to implement methods
+ like get_text() as conveniences, creating a consistent
+ text-extraction API across all PageElements.
+
+ :param strip: If True, all strings will be stripped before being
+ yielded.
+
+ :param types: A tuple of NavigableString subclasses. If this
+ NavigableString isn't one of those subclasses, the
+ sequence will be empty. By default, the subclasses
+ considered are NavigableString and CData objects. That
+ means no comments, processing instructions, etc.
+
+ :yield: A sequence that either contains this string, or is empty.
+ """
+ if types is self.default:
+ # This is kept in Tag because it's full of subclasses of
+ # this class, which aren't defined until later in the file.
+ types = Tag.MAIN_CONTENT_STRING_TYPES
+
+ # Do nothing if the caller is looking for specific types of
+ # string, and we're of a different type.
+ #
+ # We check specific types instead of using isinstance(self,
+ # types) because all of these classes subclass
+ # NavigableString. Anyone who's using this feature probably
+ # wants generic NavigableStrings but not other stuff.
+ my_type = type(self)
+ if types is not None:
+ if isinstance(types, type):
+ # Looking for a single type.
+ if my_type is not types:
+ return
+ elif my_type not in types:
+ # Looking for one of a list of types.
+ return
+
+ value = self
+ if strip:
+ final_value = value.strip()
+ else:
+ final_value = self
+ if len(final_value) > 0:
+ yield final_value
+
+ @property
+ def strings(self) -> Iterator[str]:
+ """Yield this string, but only if it is interesting.
+
+ This is defined the way it is for compatibility with
+ `Tag.strings`. See `Tag` for information on which strings are
+ interesting in a given context.
+
+ :yield: A sequence that either contains this string, or is empty.
+ """
+ return self._all_strings()
+
+
+class PreformattedString(NavigableString):
+ """A `NavigableString` not subject to the normal formatting rules.
+
+ This is an abstract class used for special kinds of strings such
+ as comments (`Comment`) and CDATA blocks (`CData`).
+ """
+
+ PREFIX: str = ""
+ SUFFIX: str = ""
+
+ def output_ready(self, formatter: Optional[_FormatterOrName] = None) -> str:
+ """Make this string ready for output by adding any subclass-specific
+ prefix or suffix.
+
+ :param formatter: A `Formatter` object, or a string naming one
+ of the standard formatters. The string will be passed into the
+ `Formatter`, but only to trigger any side effects: the return
+ value is ignored.
+
+ :return: The string, with any subclass-specific prefix and
+ suffix added on.
+ """
+ if formatter is not None:
+ self.format_string(self, formatter)
+ return self.PREFIX + self + self.SUFFIX
+
+
+class CData(PreformattedString):
+ """A `CDATA section <https://dev.w3.org/html5/spec-LC/syntax.html#cdata-sections>`_."""
+
+ PREFIX: str = "<![CDATA["
+ SUFFIX: str = "]]>"
+
+
+class ProcessingInstruction(PreformattedString):
+ """A SGML processing instruction."""
+
+ PREFIX: str = "<?"
+ SUFFIX: str = ">"
+
+
+class XMLProcessingInstruction(ProcessingInstruction):
+ """An `XML processing instruction <https://www.w3.org/TR/REC-xml/#sec-pi>`_."""
+
+ PREFIX: str = "<?"
+ SUFFIX: str = "?>"
+
+
+class Comment(PreformattedString):
+ """An `HTML comment <https://dev.w3.org/html5/spec-LC/syntax.html#comments>`_ or `XML comment <https://www.w3.org/TR/REC-xml/#sec-comments>`_."""
+
+ PREFIX: str = "<!--"
+ SUFFIX: str = "-->"
+
+
+class Declaration(PreformattedString):
+ """An `XML declaration <https://www.w3.org/TR/REC-xml/#sec-prolog-dtd>`_."""
+
+ PREFIX: str = "<?"
+ SUFFIX: str = "?>"
+
+
+class Doctype(PreformattedString):
+ """A `document type declaration <https://www.w3.org/TR/REC-xml/#dt-doctype>`_."""
+
+ @classmethod
+ def for_name_and_ids(
+ cls, name: str, pub_id: Optional[str], system_id: Optional[str]
+ ) -> Doctype:
+ """Generate an appropriate document type declaration for a given
+ public ID and system ID.
+
+ :param name: The name of the document's root element, e.g. 'html'.
+ :param pub_id: The Formal Public Identifier for this document type,
+ e.g. '-//W3C//DTD XHTML 1.1//EN'
+ :param system_id: The system identifier for this document type,
+ e.g. 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'
+ """
+ return Doctype(cls._string_for_name_and_ids(name, pub_id, system_id))
+
+ @classmethod
+ def _string_for_name_and_ids(
+ cls, name: str, pub_id: Optional[str], system_id: Optional[str]
+ ) -> str:
+ """Generate a string to be used as the basis of a Doctype object.
+
+ This is a separate method from for_name_and_ids() because the lxml
+ TreeBuilder needs to call it.
+ """
+ value = name or ""
+ if pub_id is not None:
+ value += ' PUBLIC "%s"' % pub_id
+ if system_id is not None:
+ value += ' "%s"' % system_id
+ elif system_id is not None:
+ value += ' SYSTEM "%s"' % system_id
+ return value
+
+ PREFIX: str = "<!DOCTYPE "
+ SUFFIX: str = ">\n"
+
+
+class Stylesheet(NavigableString):
+ """A `NavigableString` representing the contents of a `<style> HTML
+ tag <https://dev.w3.org/html5/spec-LC/Overview.html#the-style-element>`_
+ (probably CSS).
+
+ Used to distinguish embedded stylesheets from textual content.
+ """
+
+
+class Script(NavigableString):
+ """A `NavigableString` representing the contents of a `<script>
+ HTML tag
+ <https://dev.w3.org/html5/spec-LC/Overview.html#the-script-element>`_
+ (probably Javascript).
+
+ Used to distinguish executable code from textual content.
+ """
+
+
+class TemplateString(NavigableString):
+ """A `NavigableString` representing a string found inside an `HTML
+ <template> tag <https://html.spec.whatwg.org/multipage/scripting.html#the-template-element>`_
+ embedded in a larger document.
+
+ Used to distinguish such strings from the main body of the document.
+ """
+
+
+class RubyTextString(NavigableString):
+ """A NavigableString representing the contents of an `<rt> HTML
+ tag <https://dev.w3.org/html5/spec-LC/text-level-semantics.html#the-rt-element>`_.
+
+ Can be used to distinguish such strings from the strings they're
+ annotating.
+ """
+
+
+class RubyParenthesisString(NavigableString):
+ """A NavigableString representing the contents of an `<rp> HTML
+ tag <https://dev.w3.org/html5/spec-LC/text-level-semantics.html#the-rp-element>`_.
+ """
+
+
+class Tag(PageElement):
+ """An HTML or XML tag that is part of a parse tree, along with its
+ attributes, contents, and relationships to other parts of the tree.
+
+ When Beautiful Soup parses the markup ``<b>penguin</b>``, it will
+ create a `Tag` object representing the ``<b>`` tag. You can
+ instantiate `Tag` objects directly, but it's not necessary unless
+ you're adding entirely new markup to a parsed document. Most of
+ the constructor arguments are intended for use by the `TreeBuilder`
+ that's parsing a document.
+
+ :param parser: A `BeautifulSoup` object representing the parse tree this
+ `Tag` will be part of.
+ :param builder: The `TreeBuilder` being used to build the tree.
+ :param name: The name of the tag.
+ :param namespace: The URI of this tag's XML namespace, if any.
+ :param prefix: The prefix for this tag's XML namespace, if any.
+ :param attrs: A dictionary of attribute values.
+ :param parent: The `Tag` to use as the parent of this `Tag`. May be
+ the `BeautifulSoup` object itself.
+ :param previous: The `PageElement` that was parsed immediately before
+ parsing this tag.
+ :param is_xml: If True, this is an XML tag. Otherwise, this is an
+ HTML tag.
+ :param sourceline: The line number where this tag was found in its
+ source document.
+ :param sourcepos: The character position within ``sourceline`` where this
+ tag was found.
+ :param can_be_empty_element: If True, this tag should be
+ represented as <tag/>. If False, this tag should be represented
+ as <tag></tag>.
+ :param cdata_list_attributes: A dictionary of attributes whose values should
+ be parsed as lists of strings if they ever show up on this tag.
+ :param preserve_whitespace_tags: Names of tags whose contents
+ should have their whitespace preserved if they are encountered inside
+ this tag.
+ :param interesting_string_types: When iterating over this tag's
+ string contents in methods like `Tag.strings` or
+ `PageElement.get_text`, these are the types of strings that are
+ interesting enough to be considered. By default,
+ `NavigableString` (normal strings) and `CData` (CDATA
+ sections) are the only interesting string subtypes.
+ :param namespaces: A dictionary mapping currently active
+ namespace prefixes to URIs, as of the point in the parsing process when
+ this tag was encountered. This can be used later to
+ construct CSS selectors.
+
+ """
+
+ def __init__(
+ self,
+ parser: Optional[BeautifulSoup] = None,
+ builder: Optional[TreeBuilder] = None,
+ name: Optional[str] = None,
+ namespace: Optional[str] = None,
+ prefix: Optional[str] = None,
+ attrs: Optional[_RawOrProcessedAttributeValues] = None,
+ parent: Optional[Union[BeautifulSoup, Tag]] = None,
+ previous: _AtMostOneElement = None,
+ is_xml: Optional[bool] = None,
+ sourceline: Optional[int] = None,
+ sourcepos: Optional[int] = None,
+ can_be_empty_element: Optional[bool] = None,
+ cdata_list_attributes: Optional[Dict[str, Set[str]]] = None,
+ preserve_whitespace_tags: Optional[Set[str]] = None,
+ interesting_string_types: Optional[Set[Type[NavigableString]]] = None,
+ namespaces: Optional[Dict[str, str]] = None,
+ # NOTE: Any new arguments here need to be mirrored in
+ # Tag.copy_self, and potentially BeautifulSoup.new_tag
+ # as well.
+ ):
+ if parser is None:
+ self.parser_class = None
+ else:
+ # We don't actually store the parser object: that lets extracted
+ # chunks be garbage-collected.
+ self.parser_class = parser.__class__
+ if name is None:
+ raise ValueError("No value provided for new tag's name.")
+ self.name = name
+ self.namespace = namespace
+ self._namespaces = namespaces or {}
+ self.prefix = prefix
+ if (not builder or builder.store_line_numbers) and (
+ sourceline is not None or sourcepos is not None
+ ):
+ self.sourceline = sourceline
+ self.sourcepos = sourcepos
+ else:
+ self.sourceline = sourceline
+ self.sourcepos = sourcepos
+
+ attr_dict_class: type[AttributeDict]
+ attribute_value_list_class: type[AttributeValueList]
+ if builder is None:
+ if is_xml:
+ attr_dict_class = XMLAttributeDict
+ else:
+ attr_dict_class = HTMLAttributeDict
+ attribute_value_list_class = AttributeValueList
+ else:
+ attr_dict_class = builder.attribute_dict_class
+ attribute_value_list_class = builder.attribute_value_list_class
+ self.attribute_value_list_class = attribute_value_list_class
+
+ if attrs is None:
+ self.attrs = attr_dict_class()
+ else:
+ if builder is not None and builder.cdata_list_attributes:
+ self.attrs = builder._replace_cdata_list_attribute_values(
+ self.name, attrs
+ )
+ else:
+ self.attrs = attr_dict_class()
+ # Make sure that the values of any multi-valued
+ # attributes (e.g. when a Tag is copied) are stored in
+ # new lists.
+ for k, v in attrs.items():
+ if isinstance(v, list):
+ v = v.__class__(v)
+ self.attrs[k] = v
+
+ # If possible, determine ahead of time whether this tag is an
+ # XML tag.
+ if builder:
+ self.known_xml = builder.is_xml
+ else:
+ self.known_xml = is_xml
+ self.contents: List[PageElement] = []
+ self.setup(parent, previous)
+ self.hidden = False
+
+ if builder is None:
+ # In the absence of a TreeBuilder, use whatever values were
+ # passed in here. They're probably None, unless this is a copy of some
+ # other tag.
+ self.can_be_empty_element = can_be_empty_element
+ self.cdata_list_attributes = cdata_list_attributes
+ self.preserve_whitespace_tags = preserve_whitespace_tags
+ self.interesting_string_types = interesting_string_types
+ else:
+ # Set up any substitutions for this tag, such as the charset in a META tag.
+ self.attribute_value_list_class = builder.attribute_value_list_class
+ builder.set_up_substitutions(self)
+
+ # Ask the TreeBuilder whether this tag might be an empty-element tag.
+ self.can_be_empty_element = builder.can_be_empty_element(name)
+
+ # Keep track of the list of attributes of this tag that
+ # might need to be treated as a list.
+ #
+ # For performance reasons, we store the whole data structure
+ # rather than asking the question of every tag. Asking would
+ # require building a new data structure every time, and
+ # (unlike can_be_empty_element), we almost never need
+ # to check this.
+ self.cdata_list_attributes = builder.cdata_list_attributes
+
+ # Keep track of the names that might cause this tag to be treated as a
+ # whitespace-preserved tag.
+ self.preserve_whitespace_tags = builder.preserve_whitespace_tags
+
+ if self.name in builder.string_containers:
+ # This sort of tag uses a special string container
+ # subclass for most of its strings. We need to be able
+ # to look up the proper container subclass.
+ self.interesting_string_types = {builder.string_containers[self.name]}
+ else:
+ self.interesting_string_types = self.MAIN_CONTENT_STRING_TYPES
+
+ parser_class: Optional[type[BeautifulSoup]]
+ name: str
+ namespace: Optional[str]
+ prefix: Optional[str]
+ attrs: _AttributeValues
+ sourceline: Optional[int]
+ sourcepos: Optional[int]
+ known_xml: Optional[bool]
+ contents: List[PageElement]
+ hidden: bool
+ interesting_string_types: Optional[Set[Type[NavigableString]]]
+
+ can_be_empty_element: Optional[bool]
+ cdata_list_attributes: Optional[Dict[str, Set[str]]]
+ preserve_whitespace_tags: Optional[Set[str]]
+
+ #: :meta private:
+ parserClass = _deprecated_alias("parserClass", "parser_class", "4.0.0")
+
+ def __deepcopy__(self, memo: Dict[Any, Any], recursive: bool = True) -> Self:
+ """A deepcopy of a Tag is a new Tag, unconnected to the parse tree.
+ Its contents are a copy of the old Tag's contents.
+ """
+ clone = self.copy_self()
+
+ if recursive:
+ # Clone this tag's descendants recursively, but without
+ # making any recursive function calls.
+ tag_stack: List[Tag] = [clone]
+ for event, element in self._event_stream(self.descendants):
+ if event is Tag.END_ELEMENT_EVENT:
+ # Stop appending incoming Tags to the Tag that was
+ # just closed.
+ tag_stack.pop()
+ else:
+ descendant_clone = element.__deepcopy__(memo, recursive=False)
+ # Add to its parent's .contents
+ tag_stack[-1].append(descendant_clone)
+
+ if event is Tag.START_ELEMENT_EVENT:
+ # Add the Tag itself to the stack so that its
+ # children will be .appended to it.
+ tag_stack.append(cast(Tag, descendant_clone))
+ return clone
+
+ def copy_self(self) -> Self:
+ """Create a new Tag just like this one, but with no
+ contents and unattached to any parse tree.
+
+ This is the first step in the deepcopy process, but you can
+ call it on its own to create a copy of a Tag without copying its
+ contents.
+ """
+ clone = type(self)(
+ None,
+ None,
+ self.name,
+ self.namespace,
+ self.prefix,
+ self.attrs,
+ is_xml=self._is_xml,
+ sourceline=self.sourceline,
+ sourcepos=self.sourcepos,
+ can_be_empty_element=self.can_be_empty_element,
+ cdata_list_attributes=self.cdata_list_attributes,
+ preserve_whitespace_tags=self.preserve_whitespace_tags,
+ interesting_string_types=self.interesting_string_types,
+ namespaces=self._namespaces,
+ )
+ for attr in ("can_be_empty_element", "hidden"):
+ setattr(clone, attr, getattr(self, attr))
+ return clone
+
+ @property
+ def is_empty_element(self) -> bool:
+ """Is this tag an empty-element tag? (aka a self-closing tag)
+
+ A tag that has contents is never an empty-element tag.
+
+ A tag that has no contents may or may not be an empty-element
+ tag. It depends on the `TreeBuilder` used to create the
+ tag. If the builder has a designated list of empty-element
+ tags, then only a tag whose name shows up in that list is
+ considered an empty-element tag. This is usually the case
+ for HTML documents.
+
+ If the builder has no designated list of empty-element, then
+ any tag with no contents is an empty-element tag. This is usually
+ the case for XML documents.
+ """
+ return len(self.contents) == 0 and self.can_be_empty_element is True
+
+ @_deprecated("is_empty_element", "4.0.0")
+ def isSelfClosing(self) -> bool:
+ ": :meta private:"
+ return self.is_empty_element
+
+ @property
+ def string(self) -> Optional[str]:
+ """Convenience property to get the single string within this
+ `Tag`, assuming there is just one.
+
+ :return: If this `Tag` has a single child that's a
+ `NavigableString`, the return value is that string. If this
+ element has one child `Tag`, the return value is that child's
+ `Tag.string`, recursively. If this `Tag` has no children,
+ or has more than one child, the return value is ``None``.
+
+ If this property is unexpectedly returning ``None`` for you,
+ it's probably because your `Tag` has more than one thing
+ inside it.
+ """
+ if len(self.contents) != 1:
+ return None
+ child = self.contents[0]
+ if isinstance(child, NavigableString):
+ return child
+ elif isinstance(child, Tag):
+ return child.string
+ return None
+
+ @string.setter
+ def string(self, string: str) -> None:
+ """Replace the `Tag.contents` of this `Tag` with a single string."""
+ self.clear()
+ if isinstance(string, NavigableString):
+ new_class = string.__class__
+ else:
+ new_class = NavigableString
+ self.append(new_class(string))
+
+ #: :meta private:
+ MAIN_CONTENT_STRING_TYPES = {NavigableString, CData}
+
+ def _all_strings(
+ self, strip: bool = False, types: _OneOrMoreStringTypes = PageElement.default
+ ) -> Iterator[str]:
+ """Yield all strings of certain classes, possibly stripping them.
+
+ :param strip: If True, all strings will be stripped before being
+ yielded.
+
+ :param types: A tuple of NavigableString subclasses. Any strings of
+ a subclass not found in this list will be ignored. By
+ default, the subclasses considered are the ones found in
+ self.interesting_string_types. If that's not specified,
+ only NavigableString and CData objects will be
+ considered. That means no comments, processing
+ instructions, etc.
+ """
+ if types is self.default:
+ if self.interesting_string_types is None:
+ types = self.MAIN_CONTENT_STRING_TYPES
+ else:
+ types = self.interesting_string_types
+
+ for descendant in self.descendants:
+ if not isinstance(descendant, NavigableString):
+ continue
+ descendant_type = type(descendant)
+ if isinstance(types, type):
+ if descendant_type is not types:
+ # We're not interested in strings of this type.
+ continue
+ elif types is not None and descendant_type not in types:
+ # We're not interested in strings of this type.
+ continue
+ if strip:
+ stripped = descendant.strip()
+ if len(stripped) == 0:
+ continue
+ yield stripped
+ else:
+ yield descendant
+
+ strings = property(_all_strings)
+
+ def insert(self, position: int, *new_children: _InsertableElement) -> List[PageElement]:
+ """Insert one or more new PageElements as a child of this `Tag`.
+
+ This works similarly to :py:meth:`list.insert`, except you can insert
+ multiple elements at once.
+
+ :param position: The numeric position that should be occupied
+ in this Tag's `Tag.children` by the first new `PageElement`.
+
+ :param new_children: The PageElements to insert.
+
+ :return The newly inserted PageElements.
+ """
+ inserted: List[PageElement] = []
+ for new_child in new_children:
+ inserted.extend(self._insert(position, new_child))
+ position += 1
+ return inserted
+
+ def _insert(self, position: int, new_child: _InsertableElement) -> List[PageElement]:
+ if new_child is None:
+ raise ValueError("Cannot insert None into a tag.")
+ if new_child is self:
+ raise ValueError("Cannot insert a tag into itself.")
+ if isinstance(new_child, str) and not isinstance(new_child, NavigableString):
+ new_child = NavigableString(new_child)
+
+ from bb._vendor.bs4 import BeautifulSoup
+ if isinstance(new_child, BeautifulSoup):
+ # We don't want to end up with a situation where one BeautifulSoup
+ # object contains another. Insert the BeautifulSoup's children and
+ # return them.
+ return self.insert(position, *list(new_child.contents))
+ position = min(position, len(self.contents))
+ if hasattr(new_child, "parent") and new_child.parent is not None:
+ # We're 'inserting' an element that's already one
+ # of this object's children.
+ if new_child.parent is self:
+ current_index = self.index(new_child)
+ if current_index < position:
+ # We're moving this element further down the list
+ # of this object's children. That means that when
+ # we extract this element, our target index will
+ # jump down one.
+ position -= 1
+ elif current_index == position:
+ # We're 'inserting' an element into its current location.
+ # This is a no-op.
+ return [new_child]
+ new_child.extract()
+
+ new_child.parent = self
+ previous_child = None
+ if position == 0:
+ new_child.previous_sibling = None
+ new_child.previous_element = self
+ else:
+ previous_child = self.contents[position - 1]
+ new_child.previous_sibling = previous_child
+ new_child.previous_sibling.next_sibling = new_child
+ new_child.previous_element = previous_child._last_descendant(False)
+ if new_child.previous_element is not None:
+ new_child.previous_element.next_element = new_child
+
+ new_childs_last_element = new_child._last_descendant(
+ is_initialized=False, accept_self=True
+ )
+ # new_childs_last_element can't be None because we passed
+ # accept_self=True into _last_descendant. Worst case,
+ # new_childs_last_element will be new_child itself. Making
+ # this cast removes several mypy complaints later on as we
+ # manipulate new_childs_last_element.
+ new_childs_last_element = cast(PageElement, new_childs_last_element)
+
+ if position >= len(self.contents):
+ new_child.next_sibling = None
+
+ parent: Optional[Tag] = self
+ parents_next_sibling = None
+ while parents_next_sibling is None and parent is not None:
+ parents_next_sibling = parent.next_sibling
+ parent = parent.parent
+ if parents_next_sibling is not None:
+ # We found the element that comes next in the document.
+ break
+ if parents_next_sibling is not None:
+ new_childs_last_element.next_element = parents_next_sibling
+ else:
+ # The last element of this tag is the last element in
+ # the document.
+ new_childs_last_element.next_element = None
+ else:
+ next_child = self.contents[position]
+ new_child.next_sibling = next_child
+ if new_child.next_sibling is not None:
+ new_child.next_sibling.previous_sibling = new_child
+ new_childs_last_element.next_element = next_child
+
+ if new_childs_last_element.next_element is not None:
+ new_childs_last_element.next_element.previous_element = (
+ new_childs_last_element
+ )
+ self.contents.insert(position, new_child)
+
+ return [new_child]
+
+ def unwrap(self) -> Self:
+ """Replace this `PageElement` with its contents.
+
+ :return: This object, no longer part of the tree.
+ """
+ my_parent = self.parent
+ if my_parent is None:
+ raise ValueError(
+ "Cannot replace an element with its contents when that "
+ "element is not part of a tree."
+ )
+ my_index = my_parent.index(self)
+ self.extract(_self_index=my_index)
+ for child in reversed(self.contents[:]):
+ my_parent.insert(my_index, child)
+ return self
+
+ replace_with_children = unwrap
+
+ @_deprecated("unwrap", "4.0.0")
+ def replaceWithChildren(self) -> _OneElement:
+ ": :meta private:"
+ return self.unwrap()
+
+ def append(self, tag: _InsertableElement) -> PageElement|List[PageElement]:
+ """Appends the given `PageElement` to the contents of this `Tag`.
+
+ :param tag: A PageElement. If this is another BeautifulSoup
+ object, all of its contents will be inserted into this
+ `Tag`, since one BeautifulSoup object can't contain another
+ one.
+
+ :return: The object that was just appended, or (if `tag` was a BeautifulSoup
+ object) all such objects.
+ """
+ inserted = self.insert(len(self.contents), tag)
+ if isinstance(tag, Tag) and tag.name == "[document]": # TODO: can't reference BeautifulSoup class in this module
+ return inserted
+ else:
+ return inserted[0]
+
+ def extend(self, tags: Union[Iterable[_InsertableElement], Tag]) -> List[PageElement]:
+ """Appends one or more objects to the contents of this
+ `Tag`.
+
+ :param tags: If a list of `PageElement` objects is provided,
+ they will be appended to this tag's contents, one at a time.
+ If a single `Tag` is provided, its `Tag.contents` will be
+ used to extend this object's `Tag.contents`.
+
+ :return The list of PageElements that were appended.
+ """
+ tag_list: Iterable[_InsertableElement]
+
+ if isinstance(tags, Tag):
+ tag_list = list(tags.contents)
+ elif isinstance(tags, (PageElement, str)):
+ # The caller should really be using append() instead,
+ # but we can make it work.
+ warnings.warn(
+ "A single non-Tag item was passed into Tag.extend. Use Tag.append instead.",
+ UserWarning,
+ stacklevel=2,
+ )
+ if isinstance(tags, str) and not isinstance(tags, PageElement):
+ tags = NavigableString(tags)
+ tag_list = [tags]
+ elif isinstance(tags, Iterable):
+ # Moving items around the tree may change their position in
+ # the original list. Make a list that won't change.
+ tag_list = list(tags)
+
+ results: List[PageElement] = []
+ for tag in tag_list:
+ appended = self.append(tag)
+ if isinstance(appended, list):
+ # This can happen if you pass in a mixture of Tag and BeautifulSoup objects.
+ results.extend(appended)
+ else:
+ results.append(appended)
+
+ return results
+
+ def clear(self, decompose: bool = False) -> None:
+ """Destroy all children of this `Tag` by calling
+ `PageElement.extract` on them.
+
+ :param decompose: If this is True, `PageElement.decompose` (a
+ more destructive method) will be called instead of
+ `PageElement.extract`.
+ """
+ for element in self.contents[:]:
+ if decompose:
+ element.decompose()
+ else:
+ element.extract()
+
+ def smooth(self) -> None:
+ """Smooth out the children of this `Tag` by consolidating consecutive
+ strings.
+
+ If you perform a lot of operations that modify the tree,
+ calling this method afterwards can make pretty-printed output
+ look more natural.
+ """
+ # Mark the first position of every pair of children that need
+ # to be consolidated. Do this rather than making a copy of
+ # self.contents, since in most cases very few strings will be
+ # affected.
+ marked = []
+ for i, a in enumerate(self.contents):
+ if isinstance(a, Tag):
+ # Recursively smooth children.
+ a.smooth()
+ if i == len(self.contents) - 1:
+ # This is the last item in .contents, and it's not a
+ # tag. There's no chance it needs any work.
+ continue
+ b = self.contents[i + 1]
+ if (
+ isinstance(a, NavigableString)
+ and isinstance(b, NavigableString)
+ and not isinstance(a, PreformattedString)
+ and not isinstance(b, PreformattedString)
+ ):
+ marked.append(i)
+
+ # Go over the marked positions in reverse order, so that
+ # removing items from .contents won't affect the remaining
+ # positions.
+ for i in reversed(marked):
+ a = cast(NavigableString, self.contents[i])
+ b = cast(NavigableString, self.contents[i + 1])
+ b.extract()
+ n = NavigableString(a + b)
+ a.replace_with(n)
+
+ def index(self, element: PageElement) -> int:
+ """Find the index of a child of this `Tag` (by identity, not value).
+
+ Doing this by identity avoids issues when a `Tag` contains two
+ children that have string equality.
+
+ :param element: Look for this `PageElement` in this object's contents.
+ """
+ for i, child in enumerate(self.contents):
+ if child is element:
+ return i
+ raise ValueError("Tag.index: element not in tag")
+
+ def get(
+ self, key: str, default: Optional[_AttributeValue] = None
+ ) -> Optional[_AttributeValue]:
+ """Returns the value of the 'key' attribute for the tag, or
+ the value given for 'default' if it doesn't have that
+ attribute.
+
+ :param key: The attribute to look for.
+ :param default: Use this value if the attribute is not present
+ on this `Tag`.
+ """
+ return self.attrs.get(key, default)
+
+ def get_attribute_list(
+ self, key: str, default: Optional[AttributeValueList] = None
+ ) -> AttributeValueList:
+ """The same as get(), but always returns a (possibly empty) list.
+
+ :param key: The attribute to look for.
+ :param default: Use this value if the attribute is not present
+ on this `Tag`.
+ :return: A list of strings, usually empty or containing only a single
+ value.
+ """
+ list_value: AttributeValueList
+ value = self.get(key, default)
+ if value is None:
+ list_value = self.attribute_value_list_class()
+ elif isinstance(value, list):
+ list_value = value
+ else:
+ if not isinstance(value, str):
+ value = cast(str, value)
+ list_value = self.attribute_value_list_class([value])
+ return list_value
+
+ def has_attr(self, key: str) -> bool:
+ """Does this `Tag` have an attribute with the given name?"""
+ return key in self.attrs
+
+ def __hash__(self) -> int:
+ return str(self).__hash__()
+
+ def __getitem__(self, key: str) -> _AttributeValue:
+ """tag[key] returns the value of the 'key' attribute for the Tag,
+ and throws an exception if it's not there."""
+ return self.attrs[key]
+
+ def __iter__(self) -> Iterator[PageElement]:
+ "Iterating over a Tag iterates over its contents."
+ return iter(self.contents)
+
+ def __len__(self) -> int:
+ "The length of a Tag is the length of its list of contents."
+ return len(self.contents)
+
+ def __contains__(self, x: Any) -> bool:
+ return x in self.contents
+
+ def __bool__(self) -> bool:
+ "A tag is non-None even if it has no contents."
+ return True
+
+ def __setitem__(self, key: str, value: _AttributeValue) -> None:
+ """Setting tag[key] sets the value of the 'key' attribute for the
+ tag."""
+ self.attrs[key] = value
+
+ def __delitem__(self, key: str) -> None:
+ "Deleting tag[key] deletes all 'key' attributes for the tag."
+ self.attrs.pop(key, None)
+
+ # Since Tag.__call__ is effectively the same as PageElement.find_all, see find_all for notes
+ # on these overloads.
+
+ @overload
+ def __call__(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ @overload
+ def __call__(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ @overload
+ def __call__(
+ self,
+ name: None,
+ attrs: _StrainableAttributes,
+ recursive: bool = True,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ @overload
+ def __call__(
+ self,
+ name: _FindMethodName,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ def __call__(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings,_QueryResults]:
+ """Calling a Tag like a function is the same as calling its
+ find_all() method.
+
+ Eg. tag('a') returns a list of all the A tags found within this tag.
+ """
+ return self._find_all(name, attrs, string, limit, self._generator_for_recursive(recursive), **kwargs)
+
+ def __getattr__(self, subtag: str) -> Optional[Tag]:
+ """Calling tag.subtag is the same as calling tag.find(name="subtag")"""
+ # print("Getattr %s.%s" % (self.__class__, tag))
+ result: _AtMostOneElement
+ if len(subtag) > 3 and subtag.endswith("Tag"):
+ # BS3: soup.aTag -> "soup.find("a")
+ tag_name = subtag[:-3]
+ warnings.warn(
+ '.%(name)sTag is deprecated, use .find("%(name)s") instead. If you really were looking for a tag called %(name)sTag, use .find("%(name)sTag")'
+ % dict(name=tag_name),
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ result = self.find(tag_name)
+ # We special case contents to avoid recursion.
+ elif not subtag.startswith("__") and not subtag == "contents":
+ result = self.find(subtag)
+ else:
+ raise AttributeError(
+ "'%s' object has no attribute '%s'" % (self.__class__, subtag)
+ )
+ return result
+
+ def __eq__(self, other: Any) -> bool:
+ """Returns true iff this Tag has the same name, the same attributes,
+ and the same contents (recursively) as `other`."""
+ if self is other:
+ return True
+ if not isinstance(other, Tag):
+ return False
+ if (
+ not hasattr(other, "name")
+ or not hasattr(other, "attrs")
+ or not hasattr(other, "contents")
+ or self.name != other.name
+ or self.attrs != other.attrs
+ or len(self) != len(other)
+ ):
+ return False
+ for i, my_child in enumerate(self.contents):
+ if my_child != other.contents[i]:
+ return False
+ return True
+
+ def __ne__(self, other: Any) -> bool:
+ """Returns true iff this Tag is not identical to `other`,
+ as defined in __eq__."""
+ return not self == other
+
+ def __repr__(self) -> str:
+ """Renders this `Tag` as a string."""
+ return self.decode()
+
+ __str__ = __unicode__ = __repr__
+
+ def encode(
+ self,
+ encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ indent_level: Optional[int] = None,
+ formatter: _FormatterOrName = "minimal",
+ errors: str = "xmlcharrefreplace",
+ ) -> bytes:
+ """Render this `Tag` and its contents as a bytestring.
+
+ :param encoding: The encoding to use when converting to
+ a bytestring. This may also affect the text of the document,
+ specifically any encoding declarations within the document.
+ :param indent_level: Each line of the rendering will be
+ indented this many levels. (The ``formatter`` decides what a
+ 'level' means, in terms of spaces or other characters
+ output.) This is used internally in recursive calls while
+ pretty-printing.
+ :param formatter: Either a `Formatter` object, or a string naming one of
+ the standard formatters.
+ :param errors: An error handling strategy such as
+ 'xmlcharrefreplace'. This value is passed along into
+ :py:meth:`str.encode` and its value should be one of the `error
+ handling constants defined by Python's codecs module
+ <https://docs.python.org/3/library/codecs.html#error-handlers>`_.
+ """
+ # Turn the data structure into Unicode, then encode the
+ # Unicode.
+ u = self.decode(indent_level, encoding, formatter)
+ return u.encode(encoding, errors)
+
+ def decode(
+ self,
+ indent_level: Optional[int] = None,
+ eventual_encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ formatter: _FormatterOrName = "minimal",
+ iterator: Optional[Iterator[PageElement]] = None,
+ ) -> str:
+ """Render this `Tag` and its contents as a Unicode string.
+
+ :param indent_level: Each line of the rendering will be
+ indented this many levels. (The ``formatter`` decides what a
+ 'level' means, in terms of spaces or other characters
+ output.) This is used internally in recursive calls while
+ pretty-printing.
+ :param encoding: The encoding you intend to use when
+ converting the string to a bytestring. decode() is *not*
+ responsible for performing that encoding. This information
+ is needed so that a real encoding can be substituted in if
+ the document contains an encoding declaration (e.g. in a
+ <meta> tag).
+ :param formatter: Either a `Formatter` object, or a string
+ naming one of the standard formatters.
+ :param iterator: The iterator to use when navigating over the
+ parse tree. This is only used by `Tag.decode_contents` and
+ you probably won't need to use it.
+ """
+ pieces = []
+ # First off, turn a non-Formatter `formatter` into a Formatter
+ # object. This will stop the lookup from happening over and
+ # over again.
+ if not isinstance(formatter, Formatter):
+ formatter = self.formatter_for_name(formatter)
+
+ if indent_level is True:
+ indent_level = 0
+
+ # The currently active tag that put us into string literal
+ # mode. Until this element is closed, children will be treated
+ # as string literals and not pretty-printed. String literal
+ # mode is turned on immediately after this tag begins, and
+ # turned off immediately before it's closed. This means there
+ # will be whitespace before and after the tag itself.
+ string_literal_tag = None
+
+ for event, element in self._event_stream(iterator):
+ if event in (Tag.START_ELEMENT_EVENT, Tag.EMPTY_ELEMENT_EVENT):
+ element = cast(Tag, element)
+ piece = element._format_tag(eventual_encoding, formatter, opening=True)
+ elif event is Tag.END_ELEMENT_EVENT:
+ element = cast(Tag, element)
+ piece = element._format_tag(eventual_encoding, formatter, opening=False)
+ if indent_level is not None:
+ indent_level -= 1
+ else:
+ element = cast(NavigableString, element)
+ piece = element.output_ready(formatter)
+
+ # Now we need to apply the 'prettiness' -- extra
+ # whitespace before and/or after this tag. This can get
+ # complicated because certain tags, like <pre> and
+ # <script>, can't be prettified, since adding whitespace would
+ # change the meaning of the content.
+
+ # The default behavior is to add whitespace before and
+ # after an element when string literal mode is off, and to
+ # leave things as they are when string literal mode is on.
+ if string_literal_tag:
+ indent_before = indent_after = False
+ else:
+ indent_before = indent_after = True
+
+ # The only time the behavior is more complex than that is
+ # when we encounter an opening or closing tag that might
+ # put us into or out of string literal mode.
+ if (
+ event is Tag.START_ELEMENT_EVENT
+ and not string_literal_tag
+ and not cast(Tag, element)._should_pretty_print()
+ ):
+ # We are about to enter string literal mode. Add
+ # whitespace before this tag, but not after. We
+ # will stay in string literal mode until this tag
+ # is closed.
+ indent_before = True
+ indent_after = False
+ string_literal_tag = element
+ elif event is Tag.END_ELEMENT_EVENT and element is string_literal_tag:
+ # We are about to exit string literal mode by closing
+ # the tag that sent us into that mode. Add whitespace
+ # after this tag, but not before.
+ indent_before = False
+ indent_after = True
+ string_literal_tag = None
+
+ # Now we know whether to add whitespace before and/or
+ # after this element.
+ if indent_level is not None:
+ if indent_before or indent_after:
+ if isinstance(element, NavigableString):
+ piece = piece.strip()
+ if piece:
+ piece = self._indent_string(
+ piece, indent_level, formatter, indent_before, indent_after
+ )
+ if event == Tag.START_ELEMENT_EVENT:
+ indent_level += 1
+ pieces.append(piece)
+ return "".join(pieces)
+
+ class _TreeTraversalEvent(object):
+ """An internal class representing an event in the process
+ of traversing a parse tree.
+
+ :meta private:
+ """
+
+ # Stand-ins for the different events yielded by _event_stream
+ START_ELEMENT_EVENT = _TreeTraversalEvent() #: :meta private:
+ END_ELEMENT_EVENT = _TreeTraversalEvent() #: :meta private:
+ EMPTY_ELEMENT_EVENT = _TreeTraversalEvent() #: :meta private:
+ STRING_ELEMENT_EVENT = _TreeTraversalEvent() #: :meta private:
+
+ def _event_stream(
+ self, iterator: Optional[Iterator[PageElement]] = None
+ ) -> Iterator[Tuple[_TreeTraversalEvent, PageElement]]:
+ """Yield a sequence of events that can be used to reconstruct the DOM
+ for this element.
+
+ This lets us recreate the nested structure of this element
+ (e.g. when formatting it as a string) without using recursive
+ method calls.
+
+ This is similar in concept to the SAX API, but it's a simpler
+ interface designed for internal use. The events are different
+ from SAX and the arguments associated with the events are Tags
+ and other Beautiful Soup objects.
+
+ :param iterator: An alternate iterator to use when traversing
+ the tree.
+ """
+ tag_stack: List[Tag] = []
+
+ iterator = iterator or self.self_and_descendants
+
+ for c in iterator:
+ # If the parent of the element we're about to yield is not
+ # the tag currently on the stack, it means that the tag on
+ # the stack closed before this element appeared.
+ while tag_stack and c.parent != tag_stack[-1]:
+ now_closed_tag = tag_stack.pop()
+ yield Tag.END_ELEMENT_EVENT, now_closed_tag
+
+ if isinstance(c, Tag):
+ if c.is_empty_element:
+ yield Tag.EMPTY_ELEMENT_EVENT, c
+ else:
+ yield Tag.START_ELEMENT_EVENT, c
+ tag_stack.append(c)
+ continue
+ else:
+ yield Tag.STRING_ELEMENT_EVENT, c
+
+ while tag_stack:
+ now_closed_tag = tag_stack.pop()
+ yield Tag.END_ELEMENT_EVENT, now_closed_tag
+
+ def _indent_string(
+ self,
+ s: str,
+ indent_level: int,
+ formatter: Formatter,
+ indent_before: bool,
+ indent_after: bool,
+ ) -> str:
+ """Add indentation whitespace before and/or after a string.
+
+ :param s: The string to amend with whitespace.
+ :param indent_level: The indentation level; affects how much
+ whitespace goes before the string.
+ :param indent_before: Whether or not to add whitespace
+ before the string.
+ :param indent_after: Whether or not to add whitespace
+ (a newline) after the string.
+ """
+ space_before = ""
+ if indent_before and indent_level:
+ space_before = formatter.indent * indent_level
+
+ space_after = ""
+ if indent_after:
+ space_after = "\n"
+
+ return space_before + s + space_after
+
+ def _format_tag(
+ self, eventual_encoding: str, formatter: Formatter, opening: bool
+ ) -> str:
+ if self.hidden:
+ # A hidden tag is invisible, although its contents
+ # are visible.
+ return ""
+
+ # A tag starts with the < character (see below).
+
+ # Then the / character, if this is a closing tag.
+ closing_slash = ""
+ if not opening:
+ closing_slash = "/"
+
+ # Then an optional namespace prefix.
+ prefix = ""
+ if self.prefix:
+ prefix = self.prefix + ":"
+
+ # Then a list of attribute values, if this is an opening tag.
+ attribute_string = ""
+ if opening:
+ attributes = formatter.attributes(self)
+ attrs = []
+ for key, val in attributes:
+ if val is None:
+ decoded = key
+ else:
+ if isinstance(val, list) or isinstance(val, tuple):
+ val = " ".join(val)
+ elif not isinstance(val, str):
+ val = str(val)
+ elif (
+ isinstance(val, AttributeValueWithCharsetSubstitution)
+ and eventual_encoding is not None
+ ):
+ val = val.substitute_encoding(eventual_encoding)
+
+ text = formatter.attribute_value(val)
+ decoded = str(key) + "=" + formatter.quoted_attribute_value(text)
+ attrs.append(decoded)
+ if attrs:
+ attribute_string = " " + " ".join(attrs)
+
+ # Then an optional closing slash (for a void element in an
+ # XML document).
+ void_element_closing_slash = ""
+ if self.is_empty_element:
+ void_element_closing_slash = formatter.void_element_close_prefix or ""
+
+ # Put it all together.
+ return (
+ "<"
+ + closing_slash
+ + prefix
+ + self.name
+ + attribute_string
+ + void_element_closing_slash
+ + ">"
+ )
+
+ def _should_pretty_print(self, indent_level: int = 1) -> bool:
+ """Should this tag be pretty-printed?
+
+ Most of them should, but some (such as <pre> in HTML
+ documents) should not.
+ """
+ return indent_level is not None and (
+ not self.preserve_whitespace_tags
+ or self.name not in self.preserve_whitespace_tags
+ )
+
+ @overload
+ def prettify(
+ self,
+ encoding: None = None,
+ formatter: _FormatterOrName = "minimal",
+ ) -> str:
+ ...
+
+ @overload
+ def prettify(
+ self,
+ encoding: _Encoding,
+ formatter: _FormatterOrName = "minimal",
+ ) -> bytes:
+ ...
+
+ def prettify(
+ self,
+ encoding: Optional[_Encoding] = None,
+ formatter: _FormatterOrName = "minimal",
+ ) -> Union[str, bytes]:
+ """Pretty-print this `Tag` as a string or bytestring.
+
+ :param encoding: The encoding of the bytestring, or None if you want Unicode.
+ :param formatter: A Formatter object, or a string naming one of
+ the standard formatters.
+ :return: A string (if no ``encoding`` is provided) or a bytestring
+ (otherwise).
+ """
+ if encoding is None:
+ return self.decode(indent_level=0, formatter=formatter)
+ else:
+ return self.encode(encoding=encoding, indent_level=0, formatter=formatter)
+
+ def decode_contents(
+ self,
+ indent_level: Optional[int] = None,
+ eventual_encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ formatter: _FormatterOrName = "minimal",
+ ) -> str:
+ """Renders the contents of this tag as a Unicode string.
+
+ :param indent_level: Each line of the rendering will be
+ indented this many levels. (The formatter decides what a
+ 'level' means in terms of spaces or other characters
+ output.) Used internally in recursive calls while
+ pretty-printing.
+
+ :param eventual_encoding: The tag is destined to be
+ encoded into this encoding. decode_contents() is *not*
+ responsible for performing that encoding. This information
+ is needed so that a real encoding can be substituted in if
+ the document contains an encoding declaration (e.g. in a
+ <meta> tag).
+
+ :param formatter: A `Formatter` object, or a string naming one of
+ the standard Formatters.
+ """
+ return self.decode(
+ indent_level, eventual_encoding, formatter, iterator=self.descendants
+ )
+
+ def encode_contents(
+ self,
+ indent_level: Optional[int] = None,
+ encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ formatter: _FormatterOrName = "minimal",
+ ) -> bytes:
+ """Renders the contents of this PageElement as a bytestring.
+
+ :param indent_level: Each line of the rendering will be
+ indented this many levels. (The ``formatter`` decides what a
+ 'level' means, in terms of spaces or other characters
+ output.) This is used internally in recursive calls while
+ pretty-printing.
+ :param formatter: Either a `Formatter` object, or a string naming one of
+ the standard formatters.
+ :param encoding: The bytestring will be in this encoding.
+ """
+ contents = self.decode_contents(indent_level, encoding, formatter)
+ return contents.encode(encoding)
+
+ @_deprecated("encode_contents", "4.0.0")
+ def renderContents(
+ self,
+ encoding: _Encoding = DEFAULT_OUTPUT_ENCODING,
+ prettyPrint: bool = False,
+ indentLevel: Optional[int] = 0,
+ ) -> bytes:
+ """Deprecated method for BS3 compatibility.
+
+ :meta private:
+ """
+ if not prettyPrint:
+ indentLevel = None
+ return self.encode_contents(indent_level=indentLevel, encoding=encoding)
+
+ # Soup methods
+ #
+
+ # People who call these methods in a type-safe environment
+ # basically want to know whether the call is going to return
+ # NavigableStrings or Tags. It's always one or the other, never
+ # both, but spelling it out requires a number of overloads for
+ # each method.
+ #
+ # If I had it to do over again I'd design this API differently (it
+ # would look more like ElementFilter), but that's life.
+ #
+ # The overloads all look for a clue in the input which restricts
+ # the method to returning either only strings or only tags. Only
+ # the most common cases are covered.
+
+ # e.g. find(string="foo")
+ # -> string information but no tag information
+ # -> string
+ @overload
+ def find(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ *,
+ string: _StrainableString,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneNavigableString:
+ ...
+
+ # e.g. find() -> default behavior -> tag
+ # find(attr="value") -> only tags have attrs -> tag
+ @overload
+ def find(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ string: None = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ # e.g. find(attrs=dict(attr="value"))
+ # -> only tags have attrs
+ # -> tag
+ @overload
+ def find(
+ self,
+ name: None,
+ attrs: _StrainableAttributes,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ # e.g. find(name="a")) -> only tags have names -> tag
+ #
+ # The confusing and controversial case of find(name="a", string="foo")
+ # also hits this overload.
+ @overload
+ def find(
+ self,
+ name: _FindMethodName,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _AtMostOneTag:
+ ...
+
+ # Some lesser-used cases are not covered by the overrides. Those
+ # cases will hit this method directly and return a very general
+ # type which will need to be cast after the call.
+ def find(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_AtMostOneTag,_AtMostOneNavigableString,_AtMostOneElement]:
+ """Look in the children of this PageElement and find the first
+ PageElement that matches the given criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param recursive: If this is True, find() will perform a
+ recursive search of this Tag's children. Otherwise,
+ only the direct children will be considered.
+ :param string: A filter on the `Tag.string` attribute.
+ :kwargs: Additional filters on attribute values.
+ """
+ tags = self._find_all(name, attrs, string, 1, self._generator_for_recursive(recursive), **kwargs)
+ if tags:
+ return tags[0]
+ return None
+
+ findChild = _deprecated_function_alias("findChild", "find", "3.0.0")
+
+ # e.g. find_all(string="foo")
+ # -> string information but no tag information
+ # -> strings
+ #
+ # Also covers unlikely cases like find_all(name=None, string="foo")
+ #
+ # "To mark parameters as keyword-only, indicating the parameters
+ # must be passed by keyword argument, place an * in the arguments
+ # list just before the first keyword-only parameter."
+ #
+ # --https://peps.python.org/pep-0570/#keyword-only-arguments
+ @overload
+ def find_all(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ *,
+ string: _StrainableString,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeNavigableStrings:
+ ...
+
+ # e.g. find_all() -> default behavior -> tags
+ # find_all(attr="value") -> only tags have attrs -> tags
+ @overload
+ def find_all(
+ self,
+ name: None = None,
+ attrs: None = None,
+ recursive: bool = True,
+ string: None = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ # e.g. find_all(attrs=dict(attr="value"))
+ # -> only tags have attrs
+ # -> tags
+ @overload
+ def find_all(
+ self,
+ name: None,
+ attrs: _StrainableAttributes,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ # e.g. find_all(name="a")) -> only tags have names -> tags
+ #
+ # The confusing and controversial case of find_all(name="a", string="foo")
+ # also hits this overload.
+ @overload
+ def find_all(
+ self,
+ name: _FindMethodName,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> _SomeTags:
+ ...
+
+ # Without the clues above, we don't know whether the method will
+ # return strings or tags. However every common case will trigger one
+ # of the overloads and give us the clue we need.
+ def find_all(
+ self,
+ name: _OptionalFindMethodName = None,
+ attrs: Optional[_StrainableAttributes] = None,
+ recursive: bool = True,
+ string: Optional[_StrainableString] = None,
+ limit: Optional[int] = None,
+ **kwargs: _StrainableAttribute,
+ ) -> Union[_SomeTags,_SomeNavigableStrings]:
+ """Look in the children of this `PageElement` and find all
+ `PageElement` objects that match the given criteria.
+
+ All find_* methods take a common set of arguments. See the online
+ documentation for detailed explanations.
+
+ :param name: A filter on tag name.
+ :param attrs: Additional filters on attribute values.
+ :param recursive: If this is True, find_all() will perform a
+ recursive search of this PageElement's children. Otherwise,
+ only the direct children will be considered.
+ :param limit: Stop looking after finding this many results.
+ :kwargs: Additional filters on attribute values.
+ """
+ generator = self._generator_for_recursive(recursive)
+
+ if string is not None and (name is not None or attrs is not None or kwargs):
+ # TODO: Using the @overload decorator to express the three ways you
+ # could get into this path is way too much code for a rarely(?) used
+ # feature.
+ return cast(ResultSet[Tag],
+ self._find_all(name, attrs, string, limit, generator,
+ **kwargs))
+
+ if string is None:
+ # If string is None, we're searching for tags.
+ return cast(ResultSet[Tag], self._find_all(
+ name, attrs, None, limit, generator, **kwargs
+ ))
+
+ # Otherwise, we're searching for strings.
+ return cast(ResultSet[NavigableString], self._find_all(
+ None, None, string, limit, generator, **kwargs
+ ))
+
+ findAll = _deprecated_function_alias("findAll", "find_all", "4.0.0")
+ findChildren = _deprecated_function_alias("findChildren", "find_all", "3.0.0")
+
+ # Generator methods
+ @property
+ def children(self) -> Iterator[PageElement]:
+ """Iterate over all direct children of this `PageElement`."""
+ return (x for x in self.contents)
+
+ @property
+ def self_and_descendants(self) -> Iterator[PageElement]:
+ """Iterate over this `Tag` and its children in a
+ breadth-first sequence.
+ """
+ return self._self_and(self.descendants)
+
+ @property
+ def descendants(self) -> Iterator[PageElement]:
+ """Iterate over all children of this `Tag` in a
+ breadth-first sequence.
+ """
+ if not len(self.contents):
+ return
+ # _last_descendant() can't return None here because
+ # accept_self is True. Worst case, last_descendant will end up
+ # as self.
+ last_descendant = cast(PageElement, self._last_descendant(accept_self=True))
+ stopNode = last_descendant.next_element
+ current: _AtMostOneElement = self.contents[0]
+ while current is not stopNode and current is not None:
+ successor = current.next_element
+ yield current
+ current = successor
+
+ def _generator_for_recursive(self, recursive:bool) -> Iterator[PageElement]:
+ """Helper method to process the boolean `recursive` argument
+ for find* methods.
+
+ :return: the appropriate generator
+ """
+ if recursive:
+ return self.descendants
+ return self.children
+
+ # CSS selector code
+ def select_one(
+ self, selector: str, namespaces: Optional[Dict[str, str]] = None, **kwargs: Any
+ ) -> Optional[Tag]:
+ """Perform a CSS selection operation on the current element.
+
+ :param selector: A CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will use the prefixes it encountered while
+ parsing the document.
+
+ :param kwargs: Keyword arguments to be passed into Soup Sieve's
+ soupsieve.select() method.
+ """
+ return self.css.select_one(selector, namespaces, **kwargs)
+
+ def select(
+ self,
+ selector: str,
+ namespaces: Optional[Dict[str, str]] = None,
+ limit: int = 0,
+ **kwargs: Any,
+ ) -> ResultSet[Tag]:
+ """Perform a CSS selection operation on the current element.
+
+ This uses the SoupSieve library.
+
+ :param selector: A string containing a CSS selector.
+
+ :param namespaces: A dictionary mapping namespace prefixes
+ used in the CSS selector to namespace URIs. By default,
+ Beautiful Soup will use the prefixes it encountered while
+ parsing the document.
+
+ :param limit: After finding this number of results, stop looking.
+
+ :param kwargs: Keyword arguments to be passed into SoupSieve's
+ soupsieve.select() method.
+ """
+ return self.css.select(selector, namespaces, limit, **kwargs)
+
+ @property
+ def css(self) -> CSS:
+ """Return an interface to the CSS selector API."""
+ return CSS(self)
+
+ # Old names for backwards compatibility
+ @_deprecated("children", "4.0.0")
+ def childGenerator(self) -> Iterator[PageElement]:
+ """Deprecated generator.
+
+ :meta private:
+ """
+ return self.children
+
+ @_deprecated("descendants", "4.0.0")
+ def recursiveChildGenerator(self) -> Iterator[PageElement]:
+ """Deprecated generator.
+
+ :meta private:
+ """
+ return self.descendants
+
+ @_deprecated("has_attr", "4.0.0")
+ def has_key(self, key: str) -> bool:
+ """Deprecated method. This was kind of misleading because has_key()
+ (attributes) was different from __in__ (contents).
+
+ has_key() is gone in Python 3, anyway.
+
+ :meta private:
+ """
+ return self.has_attr(key)
+
+
+_PageElementT = TypeVar("_PageElementT", bound=PageElement)
+
+class ResultSet(List[_PageElementT], Generic[_PageElementT]):
+ """A ResultSet is a list of `PageElement` objects, gathered as the result
+ of matching an :py:class:`ElementFilter` against a parse tree. Basically, a list of
+ search results.
+ """
+
+ source: Optional[ElementFilter]
+
+ def __init__(
+ self, source: Optional[ElementFilter], result: Iterable[_PageElementT] = ()
+ ) -> None:
+ super(ResultSet, self).__init__(result)
+ self.source = source
+
+ def __getattr__(self, key: str) -> None:
+ """Raise a helpful exception to explain a common code fix."""
+ raise AttributeError(
+ f"""ResultSet object has no attribute "{key}". You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?"""
+ )
+
+# Now that all the classes used by SoupStrainer have been defined,
+# import SoupStrainer itself into this module to preserve the
+# backwards compatibility of anyone who imports
+# bs4.element.SoupStrainer.
+from bb._vendor.bs4.filter import SoupStrainer # noqa: E402
diff --git a/bitbake/lib/bb/_vendor/bs4/exceptions.py b/bitbake/lib/bb/_vendor/bs4/exceptions.py
new file mode 100644
index 00000000..1d1a8fb2
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/exceptions.py
@@ -0,0 +1,28 @@
+"""Exceptions defined by Beautiful Soup itself."""
+
+from typing import Union
+
+
+class StopParsing(Exception):
+ """Exception raised by a TreeBuilder if it's unable to continue parsing."""
+
+
+class FeatureNotFound(ValueError):
+ """Exception raised by the BeautifulSoup constructor if no parser with the
+ requested features is found.
+ """
+
+
+class ParserRejectedMarkup(Exception):
+ """An Exception to be raised when the underlying parser simply
+ refuses to parse the given markup.
+ """
+
+ def __init__(self, message_or_exception: Union[str, Exception]):
+ """Explain why the parser rejected the given markup, either
+ with a textual explanation or another exception.
+ """
+ if isinstance(message_or_exception, Exception):
+ e = message_or_exception
+ message_or_exception = "%s: %s" % (e.__class__.__name__, str(e))
+ super(ParserRejectedMarkup, self).__init__(message_or_exception)
diff --git a/bitbake/lib/bb/_vendor/bs4/filter.py b/bitbake/lib/bb/_vendor/bs4/filter.py
new file mode 100644
index 00000000..1d467818
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/filter.py
@@ -0,0 +1,764 @@
+from __future__ import annotations
+from collections import defaultdict
+import re
+from typing import (
+ Any,
+ Callable,
+ cast,
+ Dict,
+ Iterator,
+ Iterable,
+ List,
+ Optional,
+ Sequence,
+ Type,
+ Union,
+)
+import warnings
+
+from bb._vendor.bs4._deprecation import _deprecated
+from bb._vendor.bs4.element import (
+ AttributeDict,
+ NavigableString,
+ PageElement,
+ ResultSet,
+ Tag,
+)
+from bb._vendor.bs4._typing import (
+ _AtMostOneElement,
+ _AttributeValue,
+ _NullableStringMatchFunction,
+ _OneElement,
+ _PageElementMatchFunction,
+ _QueryResults,
+ _RawAttributeValues,
+ _RegularExpressionProtocol,
+ _StrainableAttribute,
+ _StrainableElement,
+ _StrainableString,
+ _StringMatchFunction,
+ _TagMatchFunction,
+)
+
+
+class ElementFilter(object):
+ """`ElementFilter` encapsulates the logic necessary to decide:
+
+ 1. whether a `PageElement` (a `Tag` or a `NavigableString`) matches a
+ user-specified query.
+
+ 2. whether a given sequence of markup found during initial parsing
+ should be turned into a `PageElement` at all, or simply discarded.
+
+ The base class is the simplest `ElementFilter`. By default, it
+ matches everything and allows all markup to become `PageElement`
+ objects. You can make it more selective by passing in a
+ user-defined match function, or defining a subclass.
+
+ Most users of Beautiful Soup will never need to use
+ `ElementFilter`, or its more capable subclass
+ `SoupStrainer`. Instead, they will use methods like
+ :py:meth:`Tag.find`, which will convert their arguments into
+ `SoupStrainer` objects and run them against the tree.
+
+ However, if you find yourself wanting to treat the arguments to
+ Beautiful Soup's find_*() methods as first-class objects, those
+ objects will be `SoupStrainer` objects. You can create them
+ yourself and then make use of functions like
+ `ElementFilter.filter()`.
+ """
+
+ match_function: Optional[_PageElementMatchFunction]
+
+ def __init__(self, match_function: Optional[_PageElementMatchFunction] = None):
+ """Pass in a match function to easily customize the behavior of
+ `ElementFilter.match` without needing to subclass.
+
+ :param match_function: A function that takes a `PageElement`
+ and returns `True` if that `PageElement` matches some criteria.
+ """
+ self.match_function = match_function
+
+ @property
+ def includes_everything(self) -> bool:
+ """Does this `ElementFilter` obviously include everything? If so,
+ the filter process can be made much faster.
+
+ The `ElementFilter` might turn out to include everything even
+ if this returns `False`, but it won't include everything in an
+ obvious way.
+
+ The base `ElementFilter` implementation includes things based on
+ the match function, so includes_everything is only true if
+ there is no match function.
+ """
+ return not self.match_function
+
+ @property
+ def excludes_everything(self) -> bool:
+ """Does this `ElementFilter` obviously exclude everything? If
+ so, Beautiful Soup will issue a warning if you try to use it
+ when parsing a document.
+
+ The `ElementFilter` might turn out to exclude everything even
+ if this returns `False`, but it won't exclude everything in an
+ obvious way.
+
+ The base `ElementFilter` implementation excludes things based
+ on a match function we can't inspect, so excludes_everything
+ is always false.
+ """
+ return False
+
+ def match(self, element: PageElement, _known_rules:bool=False) -> bool:
+ """Does the given PageElement match the rules set down by this
+ ElementFilter?
+
+ The base implementation delegates to the function passed in to
+ the constructor.
+
+ :param _known_rules: Defined for compatibility with
+ SoupStrainer._match(). Used more for consistency than because
+ we need the performance optimization.
+ """
+ if not _known_rules and self.includes_everything:
+ return True
+ if not self.match_function:
+ return True
+ return self.match_function(element)
+
+ def filter(self, generator: Iterator[PageElement]) -> Iterator[_OneElement]:
+ """The most generic search method offered by Beautiful Soup.
+
+ Acts like Python's built-in `filter`, using
+ `ElementFilter.match` as the filtering function.
+ """
+ # If there are no rules at all, don't bother filtering. Let
+ # anything through.
+ if self.includes_everything:
+ yield from generator
+ while True:
+ try:
+ i = next(generator)
+ except StopIteration:
+ break
+ if i:
+ if self.match(i, _known_rules=True):
+ yield i
+
+ def find(self, generator: Iterator[PageElement]) -> _AtMostOneElement:
+ """A lower-level equivalent of :py:meth:`Tag.find`.
+
+ You can pass in your own generator for iterating over
+ `PageElement` objects. The first one that matches this
+ `ElementFilter` will be returned.
+
+ :param generator: A way of iterating over `PageElement`
+ objects.
+ """
+ for match in self.filter(generator):
+ return match
+ return None
+
+ def find_all(
+ self, generator: Iterator[PageElement], limit: Optional[int] = None
+ ) -> _QueryResults:
+ """A lower-level equivalent of :py:meth:`Tag.find_all`.
+
+ You can pass in your own generator for iterating over
+ `PageElement` objects. Only elements that match this
+ `ElementFilter` will be returned in the :py:class:`ResultSet`.
+
+ :param generator: A way of iterating over `PageElement`
+ objects.
+
+ :param limit: Stop looking after finding this many results.
+ """
+ results = []
+ for match in self.filter(generator):
+ results.append(match)
+ if limit is not None and len(results) >= limit:
+ break
+ return ResultSet(self, results)
+
+ def allow_tag_creation(
+ self, nsprefix: Optional[str], name: str, attrs: Optional[_RawAttributeValues]
+ ) -> bool:
+ """Based on the name and attributes of a tag, see whether this
+ `ElementFilter` will allow a `Tag` object to even be created.
+
+ By default, all tags are parsed. To change this, subclass
+ `ElementFilter`.
+
+ :param name: The name of the prospective tag.
+ :param attrs: The attributes of the prospective tag.
+ """
+ return True
+
+ def allow_string_creation(self, string: str) -> bool:
+ """Based on the content of a string, see whether this
+ `ElementFilter` will allow a `NavigableString` object based on
+ this string to be added to the parse tree.
+
+ By default, all strings are processed into `NavigableString`
+ objects. To change this, subclass `ElementFilter`.
+
+ :param str: The string under consideration.
+ """
+ return True
+
+
+class MatchRule(object):
+ """Each MatchRule encapsulates the logic behind a single argument
+ passed in to one of the Beautiful Soup find* methods.
+ """
+
+ string: Optional[str]
+ pattern: Optional[_RegularExpressionProtocol]
+ present: Optional[bool]
+ exclude_everything: Optional[bool]
+ # TODO-TYPING: All MatchRule objects also have an attribute
+ # ``function``, but the type of the function depends on the
+ # subclass.
+
+ def __init__(
+ self,
+ string: Optional[Union[str, bytes]] = None,
+ pattern: Optional[_RegularExpressionProtocol] = None,
+ function: Optional[Callable] = None,
+ present: Optional[bool] = None,
+ exclude_everything: Optional[bool] = None
+ ):
+ if isinstance(string, bytes):
+ string = string.decode("utf8")
+ self.string = string
+ if isinstance(pattern, bytes):
+ self.pattern = re.compile(pattern.decode("utf8"))
+ elif isinstance(pattern, str):
+ self.pattern = re.compile(pattern)
+ else:
+ self.pattern = pattern
+ self.function = function
+ self.present = present
+ self.exclude_everything = exclude_everything
+
+ values = [
+ x
+ for x in (self.string, self.pattern, self.function, self.present, self.exclude_everything)
+ if x is not None
+ ]
+ if len(values) == 0:
+ raise ValueError(
+ "Either string, pattern, function, present, or exclude_everything must be provided."
+ )
+ if len(values) > 1:
+ raise ValueError(
+ "At most one of string, pattern, function, present, and exclude_everything must be provided."
+ )
+
+ def _base_match(self, string: Optional[str]) -> Optional[bool]:
+ """Run the 'cheap' portion of a match, trying to get an answer without
+ calling a potentially expensive custom function.
+
+ :return: True or False if we have a (positive or negative)
+ match; None if we need to keep trying.
+ """
+ # self.exclude_everything matches nothing.
+ if self.exclude_everything:
+ return False
+
+ # self.present==True matches everything except None.
+ if self.present is True:
+ return string is not None
+
+ # self.present==False matches _only_ None.
+ if self.present is False:
+ return string is None
+
+ # self.string does an exact string match.
+ if self.string is not None:
+ # print(f"{self.string} ?= {string}")
+ return self.string == string
+
+ # self.pattern does a regular expression search.
+ if self.pattern is not None:
+ # print(f"{self.pattern} ?~ {string}")
+ if string is None:
+ return False
+ return self.pattern.search(string) is not None
+
+ return None
+
+ def matches_string(self, string: Optional[str]) -> bool:
+ _base_result = self._base_match(string)
+ if _base_result is not None:
+ # No need to invoke the test function.
+ return _base_result
+ if self.function is not None and not self.function(string):
+ # print(f"{self.function}({string}) == False")
+ return False
+ return True
+
+ def __repr__(self) -> str:
+ cls = type(self).__name__
+ return f"<{cls} string={self.string} pattern={self.pattern} function={self.function} present={self.present}>"
+
+ def __eq__(self, other: Any) -> bool:
+ return (
+ isinstance(other, MatchRule)
+ and self.string == other.string
+ and self.pattern == other.pattern
+ and self.function == other.function
+ and self.present == other.present
+ )
+
+
+class TagNameMatchRule(MatchRule):
+ """A MatchRule implementing the rules for matches against tag name."""
+
+ function: Optional[_TagMatchFunction]
+
+ def matches_tag(self, tag: Tag) -> bool:
+ base_value = self._base_match(tag.name)
+ if base_value is not None:
+ return base_value
+
+ # The only remaining possibility is that the match is determined
+ # by a function call. Call the function.
+ function = cast(_TagMatchFunction, self.function)
+ if function(tag):
+ return True
+ return False
+
+
+class AttributeValueMatchRule(MatchRule):
+ """A MatchRule implementing the rules for matches against attribute value."""
+
+ function: Optional[_NullableStringMatchFunction]
+
+
+class StringMatchRule(MatchRule):
+ """A MatchRule implementing the rules for matches against a NavigableString."""
+
+ function: Optional[_StringMatchFunction]
+
+
+class SoupStrainer(ElementFilter):
+ """The `ElementFilter` subclass used internally by Beautiful Soup.
+
+ A `SoupStrainer` encapsulates the logic necessary to perform the
+ kind of matches supported by methods such as
+ :py:meth:`Tag.find`. `SoupStrainer` objects are primarily created
+ internally, but you can create one yourself and pass it in as
+ ``parse_only`` to the `BeautifulSoup` constructor, to parse a
+ subset of a large document.
+
+ Internally, `SoupStrainer` objects work by converting the
+ constructor arguments into `MatchRule` objects. Incoming
+ tags/markup are matched against those rules.
+
+ :param name: One or more restrictions on the tags found in a document.
+
+ :param attrs: A dictionary that maps attribute names to
+ restrictions on tags that use those attributes.
+
+ :param string: One or more restrictions on the strings found in a
+ document.
+
+ :param kwargs: A dictionary that maps attribute names to restrictions
+ on tags that use those attributes. These restrictions are additive to
+ any specified in ``attrs``.
+
+ """
+
+ name_rules: List[TagNameMatchRule]
+ attribute_rules: Dict[str, List[AttributeValueMatchRule]]
+ string_rules: List[StringMatchRule]
+
+ def __init__(
+ self,
+ name: Optional[_StrainableElement] = None,
+ attrs: Optional[Dict[str, _StrainableAttribute]] = None,
+ string: Optional[_StrainableString] = None,
+ **kwargs: _StrainableAttribute,
+ ):
+ if string is None and "text" in kwargs:
+ string = cast(Optional[_StrainableString], kwargs.pop("text"))
+ warnings.warn(
+ "As of version 4.11.0, the 'text' argument to the SoupStrainer constructor is deprecated. Use 'string' instead.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+
+ if name is None and not attrs and not string and not kwargs:
+ # Special case for backwards compatibility. Instantiating
+ # a SoupStrainer with no arguments whatsoever gets you one
+ # that matches all Tags, and only Tags.
+ self.name_rules = [TagNameMatchRule(present=True)]
+ else:
+ self.name_rules = cast(
+ List[TagNameMatchRule], list(self._make_match_rules(name, TagNameMatchRule))
+ )
+ self.attribute_rules = defaultdict(list)
+
+ if attrs is None:
+ attrs = {}
+ if not isinstance(attrs, dict):
+ # Passing something other than a dictionary as attrs is
+ # sugar for matching that thing against the 'class'
+ # attribute.
+ attrs = {"class": attrs}
+
+ for attrdict in attrs, kwargs:
+ for attr, value in attrdict.items():
+ if attr == "class_" and attrdict is kwargs:
+ # If you pass in 'class_' as part of kwargs, it's
+ # because class is a Python reserved word. If you
+ # pass it in as part of the attrs dict, it's
+ # because you really are looking for an attribute
+ # called 'class_'.
+ attr = "class"
+
+ if value is None:
+ value = False
+ for rule_obj in self._make_match_rules(value, AttributeValueMatchRule):
+ self.attribute_rules[attr].append(
+ cast(AttributeValueMatchRule, rule_obj)
+ )
+
+ self.string_rules = cast(
+ List[StringMatchRule], list(self._make_match_rules(string, StringMatchRule))
+ )
+
+ #: DEPRECATED 4.13.0: You shouldn't need to check this under
+ #: any name (.string or .text), and if you do, you're probably
+ #: not taking into account all of the types of values this
+ #: variable might have. Look at the .string_rules list instead.
+ self.__string = string
+
+ @property
+ def includes_everything(self) -> bool:
+ """Check whether the provided rules will obviously include
+ everything. (They might include everything even if this returns `False`,
+ but not in an obvious way.)
+ """
+ return not self.name_rules and not self.string_rules and not self.attribute_rules
+
+ @property
+ def excludes_everything(self) -> bool:
+ """Check whether the provided rules will obviously exclude
+ everything. (They might exclude everything even if this returns `False`,
+ but not in an obvious way.)
+ """
+ if (self.string_rules and (self.name_rules or self.attribute_rules)):
+ # This is self-contradictory, so the rules exclude everything.
+ return True
+
+ # If there's a rule that ended up treated as an "exclude everything"
+ # rule due to creating a logical inconsistency, then the rules
+ # exclude everything.
+ if any(x.exclude_everything for x in self.string_rules):
+ return True
+ if any(x.exclude_everything for x in self.name_rules):
+ return True
+ for ruleset in self.attribute_rules.values():
+ if any(x.exclude_everything for x in ruleset):
+ return True
+ return False
+
+ @property
+ def string(self) -> Optional[_StrainableString]:
+ ":meta private:"
+ warnings.warn(
+ "Access to deprecated property string. (Look at .string_rules instead) -- Deprecated since version 4.13.0.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self.__string
+
+ @property
+ def text(self) -> Optional[_StrainableString]:
+ ":meta private:"
+ warnings.warn(
+ "Access to deprecated property text. (Look at .string_rules instead) -- Deprecated since version 4.13.0.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self.__string
+
+ def __repr__(self) -> str:
+ return f"<{self.__class__.__name__} name={self.name_rules} attrs={self.attribute_rules} string={self.string_rules}>"
+
+ @classmethod
+ def _make_match_rules(
+ cls,
+ obj: Optional[Union[_StrainableElement, _StrainableAttribute]],
+ rule_class: Type[MatchRule],
+ ) -> Iterator[MatchRule]:
+ """Convert a vaguely-specific 'object' into one or more well-defined
+ `MatchRule` objects.
+
+ :param obj: Some kind of object that corresponds to one or more
+ matching rules.
+ :param rule_class: Create instances of this `MatchRule` subclass.
+ """
+ if obj is None:
+ return
+ if isinstance(obj, (str, bytes)):
+ yield rule_class(string=obj)
+ elif isinstance(obj, bool):
+ yield rule_class(present=obj)
+ elif callable(obj):
+ yield rule_class(function=obj)
+ elif isinstance(obj, _RegularExpressionProtocol):
+ yield rule_class(pattern=obj)
+ elif hasattr(obj, "__iter__"):
+ if not obj:
+ # The attribute is being matched against the null set,
+ # which means it should exclude everything.
+ yield rule_class(exclude_everything=True)
+ for o in obj:
+ if not isinstance(o, (bytes, str)) and hasattr(o, "__iter__"):
+ # This is almost certainly the user's
+ # mistake. This list contains another list, which
+ # opens up the possibility of infinite
+ # self-reference. In the interests of avoiding
+ # infinite recursion, we'll treat this as an
+ # impossible match and issue a rule that excludes
+ # everything, rather than looking inside.
+ warnings.warn(
+ f"Ignoring nested list {o} to avoid the possibility of infinite recursion.",
+ stacklevel=5,
+ )
+ yield rule_class(exclude_everything=True)
+ continue
+ for x in cls._make_match_rules(o, rule_class):
+ yield x
+ else:
+ yield rule_class(string=str(obj))
+
+ def matches_tag(self, tag: Tag) -> bool:
+ """Do the rules of this `SoupStrainer` trigger a match against the
+ given `Tag`?
+
+ If the `SoupStrainer` has any `TagNameMatchRule`, at least one
+ must match the `Tag` or its `Tag.name`.
+
+ If there are any `AttributeValueMatchRule` for a given
+ attribute, at least one of them must match the attribute
+ value.
+
+ If there are any `StringMatchRule`, at least one must match,
+ but a `SoupStrainer` that *only* contains `StringMatchRule`
+ cannot match a `Tag`, only a `NavigableString`.
+ """
+ # If there are no rules at all, let anything through.
+ #if self.includes_everything:
+ # return True
+
+ # String rules cannot not match a Tag on their own.
+ if not self.name_rules and not self.attribute_rules:
+ return False
+
+ # Optimization for a very common case where the user is
+ # searching for a tag with one specific name, and we're
+ # looking at a tag with a different name.
+ if (
+ not tag.prefix
+ and len(self.name_rules) == 1
+ and self.name_rules[0].string is not None
+ and tag.name != self.name_rules[0].string
+ ):
+ return False
+
+ # If there are name rules, at least one must match. It can
+ # match either the Tag object itself or the prefixed name of
+ # the tag.
+ prefixed_name = None
+ if tag.prefix:
+ prefixed_name = f"{tag.prefix}:{tag.name}"
+ if self.name_rules:
+ name_matches = False
+ for rule in self.name_rules:
+ # attrs = " ".join(
+ # [f"{k}={v}" for k, v in sorted(tag.attrs.items())]
+ # )
+ # print(f"Testing <{tag.name} {attrs}>{tag.string}</{tag.name}> against {rule}")
+
+ # If the rule contains a function, the function will be called
+ # with `tag`. It will not be called a second time with
+ # `prefixed_name`.
+ if rule.matches_tag(tag) or (
+ not rule.function and prefixed_name is not None and rule.matches_string(prefixed_name)
+ ):
+ name_matches = True
+ break
+
+ if not name_matches:
+ return False
+
+ # If there are attribute rules for a given attribute, at least
+ # one of them must match. If there are rules for multiple
+ # attributes, each attribute must have at least one match.
+ for attr, rules in self.attribute_rules.items():
+ attr_value = tag.get(attr, None)
+ this_attr_match = self._attribute_match(attr_value, rules)
+ if not this_attr_match:
+ return False
+
+ # If there are string rules, at least one must match.
+ if self.string_rules:
+ _str = tag.string
+ if _str is None:
+ return False
+ if not self.matches_any_string_rule(_str):
+ return False
+ return True
+
+ def _attribute_match(
+ self,
+ attr_value: Optional[_AttributeValue],
+ rules: Iterable[AttributeValueMatchRule],
+ ) -> bool:
+ attr_values: Sequence[Optional[str]]
+ if isinstance(attr_value, list):
+ attr_values = attr_value
+ else:
+ attr_values = [cast(str, attr_value)]
+
+ def _match_attribute_value_helper(attr_values: Sequence[Optional[str]]) -> bool:
+ for rule in rules:
+ for attr_value in attr_values:
+ if rule.matches_string(attr_value):
+ return True
+ return False
+
+ this_attr_match = _match_attribute_value_helper(attr_values)
+ if not this_attr_match and len(attr_values) != 1:
+ # Try again but treat the attribute value as a single
+ # string instead of a list. The result can only be
+ # different if the list of values contains more or less
+ # than one item.
+
+ # This cast converts Optional[str] to plain str.
+ #
+ # We know there can't be any None in the list. Beautiful
+ # Soup never uses None as a value of a multi-valued
+ # attribute, and if None is passed in as attr_value, it's
+ # turned into a list with 1 element, which was excluded by
+ # the if statement above.
+ attr_values = cast(Sequence[str], attr_values)
+
+ joined_attr_value = " ".join(attr_values)
+ this_attr_match = _match_attribute_value_helper([joined_attr_value])
+ return this_attr_match
+
+ def allow_tag_creation(
+ self, nsprefix: Optional[str], name: str, attrs: Optional[_RawAttributeValues]
+ ) -> bool:
+ """Based on the name and attributes of a tag, see whether this
+ `SoupStrainer` will allow a `Tag` object to even be created.
+
+ :param name: The name of the prospective tag.
+ :param attrs: The attributes of the prospective tag.
+ """
+ if self.string_rules:
+ # A SoupStrainer that has string rules can't be used to
+ # manage tag creation, because the string rule can't be
+ # evaluated until after the tag and all of its contents
+ # have been parsed.
+ return False
+ prefixed_name = None
+ if nsprefix:
+ prefixed_name = f"{nsprefix}:{name}"
+ if self.name_rules:
+ # At least one name rule must match.
+ name_match = False
+ for rule in self.name_rules:
+ for x in name, prefixed_name:
+ if x is not None:
+ if rule.matches_string(x):
+ name_match = True
+ break
+ if not name_match:
+ return False
+
+ # For each attribute that has rules, at least one rule must
+ # match.
+ if attrs is None:
+ attrs = AttributeDict()
+ for attr, rules in self.attribute_rules.items():
+ attr_value = attrs.get(attr)
+ if not self._attribute_match(attr_value, rules):
+ return False
+
+ return True
+
+ def allow_string_creation(self, string: str) -> bool:
+ """Based on the content of a markup string, see whether this
+ `SoupStrainer` will allow it to be instantiated as a
+ `NavigableString` object, or whether it should be ignored.
+ """
+ if self.name_rules or self.attribute_rules:
+ # A SoupStrainer that has name or attribute rules won't
+ # match any strings; it's designed to match tags with
+ # certain properties.
+ return False
+ if not self.string_rules:
+ # A SoupStrainer with no string rules will match
+ # all strings.
+ return True
+ if not self.matches_any_string_rule(string):
+ return False
+ return True
+
+ def matches_any_string_rule(self, string: str) -> bool:
+ """See whether the content of a string matches any of
+ this `SoupStrainer`'s string rules.
+ """
+ if not self.string_rules:
+ return True
+ for string_rule in self.string_rules:
+ if string_rule.matches_string(string):
+ return True
+ return False
+
+ def match(self, element: PageElement, _known_rules: bool=False) -> bool:
+ """Does the given `PageElement` match the rules set down by this
+ `SoupStrainer`?
+
+ The find_* methods rely heavily on this method to find matches.
+
+ :param element: A `PageElement`.
+ :param _known_rules: Set to true in the common case where
+ we already checked and found at least one rule in this SoupStrainer
+ that might exclude a PageElement. Without this, we need
+ to check .includes_everything every time, just to be safe.
+ :return: `True` if the element matches this `SoupStrainer`'s rules; `False` otherwise.
+ """
+ # If there are no rules at all, let anything through.
+ if not _known_rules and self.includes_everything:
+ return True
+ if isinstance(element, Tag):
+ return self.matches_tag(element)
+ assert isinstance(element, NavigableString)
+ if not (self.name_rules or self.attribute_rules):
+ # A NavigableString can only match a SoupStrainer that
+ # does not define any name or attribute rules.
+ # Then it comes down to the string rules.
+ return self.matches_any_string_rule(element)
+ return False
+
+ @_deprecated("allow_tag_creation", "4.13.0")
+ def search_tag(self, name: str, attrs: Optional[_RawAttributeValues]) -> bool:
+ """A less elegant version of `allow_tag_creation`. Deprecated as of 4.13.0"""
+ ":meta private:"
+ return self.allow_tag_creation(None, name, attrs)
+
+ @_deprecated("match", "4.13.0")
+ def search(self, element: PageElement) -> Optional[PageElement]:
+ """A less elegant version of match(). Deprecated as of 4.13.0.
+
+ :meta private:
+ """
+ return element if self.match(element) else None
diff --git a/bitbake/lib/bb/_vendor/bs4/formatter.py b/bitbake/lib/bb/_vendor/bs4/formatter.py
new file mode 100644
index 00000000..15e6e298
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/bs4/formatter.py
@@ -0,0 +1,276 @@
+from __future__ import annotations
+from typing import Callable, Dict, Iterable, Optional, Set, Tuple, TYPE_CHECKING, Union
+from bb._vendor.typing_extensions import TypeAlias
+from bb._vendor.bs4.dammit import EntitySubstitution
+
+if TYPE_CHECKING:
+ from bb._vendor.bs4._typing import _AttributeValue
+
+
+class Formatter(EntitySubstitution):
+ """Describes a strategy to use when outputting a parse tree to a string.
+
+ Some parts of this strategy come from the distinction between
+ HTML4, HTML5, and XML. Others are configurable by the user.
+
+ Formatters are passed in as the `formatter` argument to methods
+ like `bs4.element.Tag.encode`. Most people won't need to
+ think about formatters, and most people who need to think about
+ them can pass in one of these predefined strings as `formatter`
+ rather than making a new Formatter object:
+
+ For HTML documents:
+ * 'html' - HTML entity substitution for generic HTML documents. (default)
+ * 'html5' - HTML entity substitution for HTML5 documents, as
+ well as some optimizations in the way tags are rendered.
+ * 'html5-4.12.0' - The version of the 'html5' formatter used prior to
+ Beautiful Soup 4.13.0.
+ * 'minimal' - Only make the substitutions necessary to guarantee
+ valid HTML.
+ * None - Do not perform any substitution. This will be faster
+ but may result in invalid markup.
+
+ For XML documents:
+ * 'html' - Entity substitution for XHTML documents.
+ * 'minimal' - Only make the substitutions necessary to guarantee
+ valid XML. (default)
+ * None - Do not perform any substitution. This will be faster
+ but may result in invalid markup.
+
+ """
+
+ #: Constant name denoting HTML markup
+ HTML: str = "html"
+
+ #: Constant name denoting XML markup
+ XML: str = "xml"
+
+ #: Default values for the various constructor options when the
+ #: markup language is HTML.
+ HTML_DEFAULTS: Dict[str, Set[str]] = dict(
+ cdata_containing_tags=set(["script", "style"]),
+ )
+
+ language: Optional[str] #: :meta private:
+ entity_substitution: Optional[_EntitySubstitutionFunction] #: :meta private:
+ void_element_close_prefix: str #: :meta private:
+ cdata_containing_tags: Set[str] #: :meta private:
+ indent: str #: :meta private:
+
+ #: If this is set to true by the constructor, then attributes whose
+ #: values are sent to the empty string will be treated as HTML
+ #: boolean attributes. (Attributes whose value is None are always
+ #: rendered this way.)
+ empty_attributes_are_booleans: bool
+
+ def _default(
+ self, language: str, value: Optional[Set[str]], kwarg: str
+ ) -> Set[str]:
+ if value is not None:
+ return value
+ if language == self.XML:
+ # When XML is the markup language in use, all of the
+ # defaults are the empty list.
+ return set()
+
+ # Otherwise, it depends on what's in HTML_DEFAULTS.
+ return self.HTML_DEFAULTS[kwarg]
+
+ def __init__(
+ self,
+ language: Optional[str] = None,
+ entity_substitution: Optional[_EntitySubstitutionFunction] = None,
+ void_element_close_prefix: str = "/",
+ cdata_containing_tags: Optional[Set[str]] = None,
+ empty_attributes_are_booleans: bool = False,
+ indent: Union[int,str] = 1,
+ ):
+ r"""Constructor.
+
+ :param language: This should be `Formatter.XML` if you are formatting
+ XML markup and `Formatter.HTML` if you are formatting HTML markup.
+
+ :param entity_substitution: A function to call to replace special
+ characters with XML/HTML entities. For examples, see
+ bs4.dammit.EntitySubstitution.substitute_html and substitute_xml.
+ :param void_element_close_prefix: By default, void elements
+ are represented as <tag/> (XML rules) rather than <tag>
+ (HTML rules). To get <tag>, pass in the empty string.
+ :param cdata_containing_tags: The set of tags that are defined
+ as containing CDATA in this dialect. For example, in HTML,
+ <script> and <style> tags are defined as containing CDATA,
+ and their contents should not be formatted.
+ :param empty_attributes_are_booleans: If this is set to true,
+ then attributes whose values are sent to the empty string
+ will be treated as `HTML boolean
+ attributes<https://dev.w3.org/html5/spec-LC/common-microsyntaxes.html#boolean-attributes>`_. (Attributes
+ whose value is None are always rendered this way.)
+ :param indent: If indent is a non-negative integer or string,
+ then the contents of elements will be indented
+ appropriately when pretty-printing. An indent level of 0,
+ negative, or "" will only insert newlines. Using a
+ positive integer indent indents that many spaces per
+ level. If indent is a string (such as "\t"), that string
+ is used to indent each level. The default behavior is to
+ indent one space per level.
+
+ """
+ self.language = language or self.HTML
+ self.entity_substitution = entity_substitution
+ self.void_element_close_prefix = void_element_close_prefix
+ self.cdata_containing_tags = self._default(
+ self.language, cdata_containing_tags, "cdata_containing_tags"
+ )
+ self.empty_attributes_are_booleans = empty_attributes_are_booleans
+ if indent is None:
+ indent = 0
+ indent_str: str
+ if isinstance(indent, int):
+ if indent < 0:
+ indent = 0
+ indent_str = " " * indent
+ elif isinstance(indent, str):
+ indent_str = indent
+ else:
+ indent_str = " "
+ self.indent = indent_str
+
+ def substitute(self, ns: str) -> str:
+ """Process a string that needs to undergo entity substitution.
+ This may be a string encountered in an attribute value or as
+ text.
+
+ :param ns: A string.
+ :return: The same string but with certain characters replaced by named
+ or numeric entities.
+ """
+ if not self.entity_substitution:
+ return ns
+ from .element import NavigableString
+
+ if (
+ isinstance(ns, NavigableString)
+ and ns.parent is not None
+ and ns.parent.name in self.cdata_containing_tags
+ ):
+ # Do nothing.
+ return ns
+ # Substitute.
+ return self.entity_substitution(ns)
+
+ def attribute_value(self, value: str) -> str:
+ """Process the value of an attribute.
+
+ :param ns: A string.
+ :return: A string with certain characters replaced by named
+ or numeric entities.
+ """
+ return self.substitute(value)
+
+ def attributes(
+ self, tag: bs4.element.Tag # type:ignore
+ ) -> Iterable[Tuple[str, Optional[_AttributeValue]]]:
+ """Reorder a tag's attributes however you want.
+
+ By default, attributes are sorted alphabetically. This makes
+ behavior consistent between Python 2 and Python 3, and preserves
+ backwards compatibility with older versions of Beautiful Soup.
+
+ If `empty_attributes_are_booleans` is True, then
+ attributes whose values are set to the empty string will be
+ treated as boolean attributes.
+ """
+ if tag.attrs is None:
+ return []
+
+ items: Iterable[Tuple[str, _AttributeValue]] = list(tag.attrs.items())
+ return sorted(
+ (k, (None if self.empty_attributes_are_booleans and v == "" else v))
+ for k, v in items
+ )
+
+
+class HTMLFormatter(Formatter):
+ """A generic Formatter for HTML."""
+
+ REGISTRY: Dict[Optional[str], HTMLFormatter] = {}
+
+ def __init__(
+ self,
+ entity_substitution: Optional[_EntitySubstitutionFunction] = None,
+ void_element_close_prefix: str = "/",
+ cdata_containing_tags: Optional[Set[str]] = None,
+ empty_attributes_are_booleans: bool = False,
+ indent: Union[int,str] = 1,
+ ):
+ super(HTMLFormatter, self).__init__(
+ self.HTML,
+ entity_substitution,
+ void_element_close_prefix,
+ cdata_containing_tags,
+ empty_attributes_are_booleans,
+ indent=indent
+ )
+
+
+class XMLFormatter(Formatter):
+ """A generic Formatter for XML."""
+
+ REGISTRY: Dict[Optional[str], XMLFormatter] = {}
+
+ def __init__(
+ self,
+ entity_substitution: Optional[_EntitySubstitutionFunction] = None,
+ void_element_close_prefix: str = "/",
+ cdata_containing_tags: Optional[Set[str]] = None,
+ empty_attributes_are_booleans: bool = False,
+ indent: Union[int,str] = 1,
+ ):
+ super(XMLFormatter, self).__init__(
+ self.XML,
+ entity_substitution,
+ void_element_close_prefix,
+ cdata_containing_tags,
+ empty_attributes_are_booleans,
+ indent=indent,
+ )
+
+
+# Set up aliases for the default formatters.
+HTMLFormatter.REGISTRY["html"] = HTMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_html
+)
+
+HTMLFormatter.REGISTRY["html5"] = HTMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_html5,
+ void_element_close_prefix="",
+ empty_attributes_are_booleans=True,
+)
+HTMLFormatter.REGISTRY["html5-4.12"] = HTMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_html,
+ void_element_close_prefix="",
+ empty_attributes_are_booleans=True,
+)
+HTMLFormatter.REGISTRY["minimal"] = HTMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_xml
+)
+HTMLFormatter.REGISTRY[None] = HTMLFormatter(entity_substitution=None)
+XMLFormatter.REGISTRY["html"] = XMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_html
+)
+XMLFormatter.REGISTRY["minimal"] = XMLFormatter(
+ entity_substitution=EntitySubstitution.substitute_xml
+)
+
+XMLFormatter.REGISTRY[None] = XMLFormatter(entity_substitution=None)
+
+# Define type aliases to improve readability.
+#
+
+#: A function to call to replace special characters with XML or HTML
+#: entities.
+_EntitySubstitutionFunction: TypeAlias = Callable[[str], str]
+
+# Many of the output-centered methods take an argument that can either
+# be a Formatter object or the name of a Formatter to be looked up.
+_FormatterOrName = Union[Formatter, str]
diff --git a/bitbake/lib/bb/_vendor/bs4/py.typed b/bitbake/lib/bb/_vendor/bs4/py.typed
new file mode 100644
index 00000000..e69de29b
diff --git a/bitbake/lib/bb/_vendor/ply.pyi b/bitbake/lib/bb/_vendor/ply.pyi
new file mode 100644
index 00000000..5a6b5658
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/ply.pyi
@@ -0,0 +1 @@
+from ply import *
\ No newline at end of file
diff --git a/bitbake/lib/ply/__init__.py b/bitbake/lib/bb/_vendor/ply/__init__.py
similarity index 80%
rename from bitbake/lib/ply/__init__.py
rename to bitbake/lib/bb/_vendor/ply/__init__.py
index 853a9855..6e53cddc 100644
--- a/bitbake/lib/ply/__init__.py
+++ b/bitbake/lib/bb/_vendor/ply/__init__.py
@@ -1,4 +1,5 @@
# PLY package
# Author: David Beazley (dave@dabeaz.com)
+__version__ = '3.9'
__all__ = ['lex','yacc']
diff --git a/bitbake/lib/bb/_vendor/ply/cpp.py b/bitbake/lib/bb/_vendor/ply/cpp.py
new file mode 100644
index 00000000..b049f412
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/ply/cpp.py
@@ -0,0 +1,918 @@
+# -----------------------------------------------------------------------------
+# cpp.py
+#
+# Author: David Beazley (http://www.dabeaz.com)
+# Copyright (C) 2007
+# All rights reserved
+#
+# This module implements an ANSI-C style lexical preprocessor for PLY.
+# -----------------------------------------------------------------------------
+from __future__ import generators
+
+import sys
+
+# Some Python 3 compatibility shims
+if sys.version_info.major < 3:
+ STRING_TYPES = (str, unicode)
+else:
+ STRING_TYPES = str
+ xrange = range
+
+# -----------------------------------------------------------------------------
+# Default preprocessor lexer definitions. These tokens are enough to get
+# a basic preprocessor working. Other modules may import these if they want
+# -----------------------------------------------------------------------------
+
+tokens = (
+ 'CPP_ID','CPP_INTEGER', 'CPP_FLOAT', 'CPP_STRING', 'CPP_CHAR', 'CPP_WS', 'CPP_COMMENT1', 'CPP_COMMENT2', 'CPP_POUND','CPP_DPOUND'
+)
+
+literals = "+-*/%|&~^<>=!?()[]{}.,;:\\\'\""
+
+# Whitespace
+def t_CPP_WS(t):
+ r'\s+'
+ t.lexer.lineno += t.value.count("\n")
+ return t
+
+t_CPP_POUND = r'\#'
+t_CPP_DPOUND = r'\#\#'
+
+# Identifier
+t_CPP_ID = r'[A-Za-z_][\w_]*'
+
+# Integer literal
+def CPP_INTEGER(t):
+ r'(((((0x)|(0X))[0-9a-fA-F]+)|(\d+))([uU][lL]|[lL][uU]|[uU]|[lL])?)'
+ return t
+
+t_CPP_INTEGER = CPP_INTEGER
+
+# Floating literal
+t_CPP_FLOAT = r'((\d+)(\.\d+)(e(\+|-)?(\d+))? | (\d+)e(\+|-)?(\d+))([lL]|[fF])?'
+
+# String literal
+def t_CPP_STRING(t):
+ r'\"([^\\\n]|(\\(.|\n)))*?\"'
+ t.lexer.lineno += t.value.count("\n")
+ return t
+
+# Character constant 'c' or L'c'
+def t_CPP_CHAR(t):
+ r'(L)?\'([^\\\n]|(\\(.|\n)))*?\''
+ t.lexer.lineno += t.value.count("\n")
+ return t
+
+# Comment
+def t_CPP_COMMENT1(t):
+ r'(/\*(.|\n)*?\*/)'
+ ncr = t.value.count("\n")
+ t.lexer.lineno += ncr
+ # replace with one space or a number of '\n'
+ t.type = 'CPP_WS'; t.value = '\n' * ncr if ncr else ' '
+ return t
+
+# Line comment
+def t_CPP_COMMENT2(t):
+ r'(//.*?(\n|$))'
+ # replace with '/n'
+ t.type = 'CPP_WS'; t.value = '\n'
+ return t
+
+def t_error(t):
+ t.type = t.value[0]
+ t.value = t.value[0]
+ t.lexer.skip(1)
+ return t
+
+import re
+import copy
+import time
+import os.path
+
+# -----------------------------------------------------------------------------
+# trigraph()
+#
+# Given an input string, this function replaces all trigraph sequences.
+# The following mapping is used:
+#
+# ??= #
+# ??/ \
+# ??' ^
+# ??( [
+# ??) ]
+# ??! |
+# ??< {
+# ??> }
+# ??- ~
+# -----------------------------------------------------------------------------
+
+_trigraph_pat = re.compile(r'''\?\?[=/\'\(\)\!<>\-]''')
+_trigraph_rep = {
+ '=':'#',
+ '/':'\\',
+ "'":'^',
+ '(':'[',
+ ')':']',
+ '!':'|',
+ '<':'{',
+ '>':'}',
+ '-':'~'
+}
+
+def trigraph(input):
+ return _trigraph_pat.sub(lambda g: _trigraph_rep[g.group()[-1]],input)
+
+# ------------------------------------------------------------------
+# Macro object
+#
+# This object holds information about preprocessor macros
+#
+# .name - Macro name (string)
+# .value - Macro value (a list of tokens)
+# .arglist - List of argument names
+# .variadic - Boolean indicating whether or not variadic macro
+# .vararg - Name of the variadic parameter
+#
+# When a macro is created, the macro replacement token sequence is
+# pre-scanned and used to create patch lists that are later used
+# during macro expansion
+# ------------------------------------------------------------------
+
+class Macro(object):
+ def __init__(self,name,value,arglist=None,variadic=False):
+ self.name = name
+ self.value = value
+ self.arglist = arglist
+ self.variadic = variadic
+ if variadic:
+ self.vararg = arglist[-1]
+ self.source = None
+
+# ------------------------------------------------------------------
+# Preprocessor object
+#
+# Object representing a preprocessor. Contains macro definitions,
+# include directories, and other information
+# ------------------------------------------------------------------
+
+class Preprocessor(object):
+ def __init__(self,lexer=None):
+ if lexer is None:
+ lexer = lex.lexer
+ self.lexer = lexer
+ self.macros = { }
+ self.path = []
+ self.temp_path = []
+
+ # Probe the lexer for selected tokens
+ self.lexprobe()
+
+ tm = time.localtime()
+ self.define("__DATE__ \"%s\"" % time.strftime("%b %d %Y",tm))
+ self.define("__TIME__ \"%s\"" % time.strftime("%H:%M:%S",tm))
+ self.parser = None
+
+ # -----------------------------------------------------------------------------
+ # tokenize()
+ #
+ # Utility function. Given a string of text, tokenize into a list of tokens
+ # -----------------------------------------------------------------------------
+
+ def tokenize(self,text):
+ tokens = []
+ self.lexer.input(text)
+ while True:
+ tok = self.lexer.token()
+ if not tok: break
+ tokens.append(tok)
+ return tokens
+
+ # ---------------------------------------------------------------------
+ # error()
+ #
+ # Report a preprocessor error/warning of some kind
+ # ----------------------------------------------------------------------
+
+ def error(self,file,line,msg):
+ print("%s:%d %s" % (file,line,msg))
+
+ # ----------------------------------------------------------------------
+ # lexprobe()
+ #
+ # This method probes the preprocessor lexer object to discover
+ # the token types of symbols that are important to the preprocessor.
+ # If this works right, the preprocessor will simply "work"
+ # with any suitable lexer regardless of how tokens have been named.
+ # ----------------------------------------------------------------------
+
+ def lexprobe(self):
+
+ # Determine the token type for identifiers
+ self.lexer.input("identifier")
+ tok = self.lexer.token()
+ if not tok or tok.value != "identifier":
+ print("Couldn't determine identifier type")
+ else:
+ self.t_ID = tok.type
+
+ # Determine the token type for integers
+ self.lexer.input("12345")
+ tok = self.lexer.token()
+ if not tok or int(tok.value) != 12345:
+ print("Couldn't determine integer type")
+ else:
+ self.t_INTEGER = tok.type
+ self.t_INTEGER_TYPE = type(tok.value)
+
+ # Determine the token type for strings enclosed in double quotes
+ self.lexer.input("\"filename\"")
+ tok = self.lexer.token()
+ if not tok or tok.value != "\"filename\"":
+ print("Couldn't determine string type")
+ else:
+ self.t_STRING = tok.type
+
+ # Determine the token type for whitespace--if any
+ self.lexer.input(" ")
+ tok = self.lexer.token()
+ if not tok or tok.value != " ":
+ self.t_SPACE = None
+ else:
+ self.t_SPACE = tok.type
+
+ # Determine the token type for newlines
+ self.lexer.input("\n")
+ tok = self.lexer.token()
+ if not tok or tok.value != "\n":
+ self.t_NEWLINE = None
+ print("Couldn't determine token for newlines")
+ else:
+ self.t_NEWLINE = tok.type
+
+ self.t_WS = (self.t_SPACE, self.t_NEWLINE)
+
+ # Check for other characters used by the preprocessor
+ chars = [ '<','>','#','##','\\','(',')',',','.']
+ for c in chars:
+ self.lexer.input(c)
+ tok = self.lexer.token()
+ if not tok or tok.value != c:
+ print("Unable to lex '%s' required for preprocessor" % c)
+
+ # ----------------------------------------------------------------------
+ # add_path()
+ #
+ # Adds a search path to the preprocessor.
+ # ----------------------------------------------------------------------
+
+ def add_path(self,path):
+ self.path.append(path)
+
+ # ----------------------------------------------------------------------
+ # group_lines()
+ #
+ # Given an input string, this function splits it into lines. Trailing whitespace
+ # is removed. Any line ending with \ is grouped with the next line. This
+ # function forms the lowest level of the preprocessor---grouping into text into
+ # a line-by-line format.
+ # ----------------------------------------------------------------------
+
+ def group_lines(self,input):
+ lex = self.lexer.clone()
+ lines = [x.rstrip() for x in input.splitlines()]
+ for i in xrange(len(lines)):
+ j = i+1
+ while lines[i].endswith('\\') and (j < len(lines)):
+ lines[i] = lines[i][:-1]+lines[j]
+ lines[j] = ""
+ j += 1
+
+ input = "\n".join(lines)
+ lex.input(input)
+ lex.lineno = 1
+
+ current_line = []
+ while True:
+ tok = lex.token()
+ if not tok:
+ break
+ current_line.append(tok)
+ if tok.type in self.t_WS and '\n' in tok.value:
+ yield current_line
+ current_line = []
+
+ if current_line:
+ yield current_line
+
+ # ----------------------------------------------------------------------
+ # tokenstrip()
+ #
+ # Remove leading/trailing whitespace tokens from a token list
+ # ----------------------------------------------------------------------
+
+ def tokenstrip(self,tokens):
+ i = 0
+ while i < len(tokens) and tokens[i].type in self.t_WS:
+ i += 1
+ del tokens[:i]
+ i = len(tokens)-1
+ while i >= 0 and tokens[i].type in self.t_WS:
+ i -= 1
+ del tokens[i+1:]
+ return tokens
+
+
+ # ----------------------------------------------------------------------
+ # collect_args()
+ #
+ # Collects comma separated arguments from a list of tokens. The arguments
+ # must be enclosed in parenthesis. Returns a tuple (tokencount,args,positions)
+ # where tokencount is the number of tokens consumed, args is a list of arguments,
+ # and positions is a list of integers containing the starting index of each
+ # argument. Each argument is represented by a list of tokens.
+ #
+ # When collecting arguments, leading and trailing whitespace is removed
+ # from each argument.
+ #
+ # This function properly handles nested parenthesis and commas---these do not
+ # define new arguments.
+ # ----------------------------------------------------------------------
+
+ def collect_args(self,tokenlist):
+ args = []
+ positions = []
+ current_arg = []
+ nesting = 1
+ tokenlen = len(tokenlist)
+
+ # Search for the opening '('.
+ i = 0
+ while (i < tokenlen) and (tokenlist[i].type in self.t_WS):
+ i += 1
+
+ if (i < tokenlen) and (tokenlist[i].value == '('):
+ positions.append(i+1)
+ else:
+ self.error(self.source,tokenlist[0].lineno,"Missing '(' in macro arguments")
+ return 0, [], []
+
+ i += 1
+
+ while i < tokenlen:
+ t = tokenlist[i]
+ if t.value == '(':
+ current_arg.append(t)
+ nesting += 1
+ elif t.value == ')':
+ nesting -= 1
+ if nesting == 0:
+ if current_arg:
+ args.append(self.tokenstrip(current_arg))
+ positions.append(i)
+ return i+1,args,positions
+ current_arg.append(t)
+ elif t.value == ',' and nesting == 1:
+ args.append(self.tokenstrip(current_arg))
+ positions.append(i+1)
+ current_arg = []
+ else:
+ current_arg.append(t)
+ i += 1
+
+ # Missing end argument
+ self.error(self.source,tokenlist[-1].lineno,"Missing ')' in macro arguments")
+ return 0, [],[]
+
+ # ----------------------------------------------------------------------
+ # macro_prescan()
+ #
+ # Examine the macro value (token sequence) and identify patch points
+ # This is used to speed up macro expansion later on---we'll know
+ # right away where to apply patches to the value to form the expansion
+ # ----------------------------------------------------------------------
+
+ def macro_prescan(self,macro):
+ macro.patch = [] # Standard macro arguments
+ macro.str_patch = [] # String conversion expansion
+ macro.var_comma_patch = [] # Variadic macro comma patch
+ i = 0
+ while i < len(macro.value):
+ if macro.value[i].type == self.t_ID and macro.value[i].value in macro.arglist:
+ argnum = macro.arglist.index(macro.value[i].value)
+ # Conversion of argument to a string
+ if i > 0 and macro.value[i-1].value == '#':
+ macro.value[i] = copy.copy(macro.value[i])
+ macro.value[i].type = self.t_STRING
+ del macro.value[i-1]
+ macro.str_patch.append((argnum,i-1))
+ continue
+ # Concatenation
+ elif (i > 0 and macro.value[i-1].value == '##'):
+ macro.patch.append(('c',argnum,i-1))
+ del macro.value[i-1]
+ continue
+ elif ((i+1) < len(macro.value) and macro.value[i+1].value == '##'):
+ macro.patch.append(('c',argnum,i))
+ i += 1
+ continue
+ # Standard expansion
+ else:
+ macro.patch.append(('e',argnum,i))
+ elif macro.value[i].value == '##':
+ if macro.variadic and (i > 0) and (macro.value[i-1].value == ',') and \
+ ((i+1) < len(macro.value)) and (macro.value[i+1].type == self.t_ID) and \
+ (macro.value[i+1].value == macro.vararg):
+ macro.var_comma_patch.append(i-1)
+ i += 1
+ macro.patch.sort(key=lambda x: x[2],reverse=True)
+
+ # ----------------------------------------------------------------------
+ # macro_expand_args()
+ #
+ # Given a Macro and list of arguments (each a token list), this method
+ # returns an expanded version of a macro. The return value is a token sequence
+ # representing the replacement macro tokens
+ # ----------------------------------------------------------------------
+
+ def macro_expand_args(self,macro,args):
+ # Make a copy of the macro token sequence
+ rep = [copy.copy(_x) for _x in macro.value]
+
+ # Make string expansion patches. These do not alter the length of the replacement sequence
+
+ str_expansion = {}
+ for argnum, i in macro.str_patch:
+ if argnum not in str_expansion:
+ str_expansion[argnum] = ('"%s"' % "".join([x.value for x in args[argnum]])).replace("\\","\\\\")
+ rep[i] = copy.copy(rep[i])
+ rep[i].value = str_expansion[argnum]
+
+ # Make the variadic macro comma patch. If the variadic macro argument is empty, we get rid
+ comma_patch = False
+ if macro.variadic and not args[-1]:
+ for i in macro.var_comma_patch:
+ rep[i] = None
+ comma_patch = True
+
+ # Make all other patches. The order of these matters. It is assumed that the patch list
+ # has been sorted in reverse order of patch location since replacements will cause the
+ # size of the replacement sequence to expand from the patch point.
+
+ expanded = { }
+ for ptype, argnum, i in macro.patch:
+ # Concatenation. Argument is left unexpanded
+ if ptype == 'c':
+ rep[i:i+1] = args[argnum]
+ # Normal expansion. Argument is macro expanded first
+ elif ptype == 'e':
+ if argnum not in expanded:
+ expanded[argnum] = self.expand_macros(args[argnum])
+ rep[i:i+1] = expanded[argnum]
+
+ # Get rid of removed comma if necessary
+ if comma_patch:
+ rep = [_i for _i in rep if _i]
+
+ return rep
+
+
+ # ----------------------------------------------------------------------
+ # expand_macros()
+ #
+ # Given a list of tokens, this function performs macro expansion.
+ # The expanded argument is a dictionary that contains macros already
+ # expanded. This is used to prevent infinite recursion.
+ # ----------------------------------------------------------------------
+
+ def expand_macros(self,tokens,expanded=None):
+ if expanded is None:
+ expanded = {}
+ i = 0
+ while i < len(tokens):
+ t = tokens[i]
+ if t.type == self.t_ID:
+ if t.value in self.macros and t.value not in expanded:
+ # Yes, we found a macro match
+ expanded[t.value] = True
+
+ m = self.macros[t.value]
+ if not m.arglist:
+ # A simple macro
+ ex = self.expand_macros([copy.copy(_x) for _x in m.value],expanded)
+ for e in ex:
+ e.lineno = t.lineno
+ tokens[i:i+1] = ex
+ i += len(ex)
+ else:
+ # A macro with arguments
+ j = i + 1
+ while j < len(tokens) and tokens[j].type in self.t_WS:
+ j += 1
+ if tokens[j].value == '(':
+ tokcount,args,positions = self.collect_args(tokens[j:])
+ if not m.variadic and len(args) != len(m.arglist):
+ self.error(self.source,t.lineno,"Macro %s requires %d arguments" % (t.value,len(m.arglist)))
+ i = j + tokcount
+ elif m.variadic and len(args) < len(m.arglist)-1:
+ if len(m.arglist) > 2:
+ self.error(self.source,t.lineno,"Macro %s must have at least %d arguments" % (t.value, len(m.arglist)-1))
+ else:
+ self.error(self.source,t.lineno,"Macro %s must have at least %d argument" % (t.value, len(m.arglist)-1))
+ i = j + tokcount
+ else:
+ if m.variadic:
+ if len(args) == len(m.arglist)-1:
+ args.append([])
+ else:
+ args[len(m.arglist)-1] = tokens[j+positions[len(m.arglist)-1]:j+tokcount-1]
+ del args[len(m.arglist):]
+
+ # Get macro replacement text
+ rep = self.macro_expand_args(m,args)
+ rep = self.expand_macros(rep,expanded)
+ for r in rep:
+ r.lineno = t.lineno
+ tokens[i:j+tokcount] = rep
+ i += len(rep)
+ del expanded[t.value]
+ continue
+ elif t.value == '__LINE__':
+ t.type = self.t_INTEGER
+ t.value = self.t_INTEGER_TYPE(t.lineno)
+
+ i += 1
+ return tokens
+
+ # ----------------------------------------------------------------------
+ # evalexpr()
+ #
+ # Evaluate an expression token sequence for the purposes of evaluating
+ # integral expressions.
+ # ----------------------------------------------------------------------
+
+ def evalexpr(self,tokens):
+ # tokens = tokenize(line)
+ # Search for defined macros
+ i = 0
+ while i < len(tokens):
+ if tokens[i].type == self.t_ID and tokens[i].value == 'defined':
+ j = i + 1
+ needparen = False
+ result = "0L"
+ while j < len(tokens):
+ if tokens[j].type in self.t_WS:
+ j += 1
+ continue
+ elif tokens[j].type == self.t_ID:
+ if tokens[j].value in self.macros:
+ result = "1L"
+ else:
+ result = "0L"
+ if not needparen: break
+ elif tokens[j].value == '(':
+ needparen = True
+ elif tokens[j].value == ')':
+ break
+ else:
+ self.error(self.source,tokens[i].lineno,"Malformed defined()")
+ j += 1
+ tokens[i].type = self.t_INTEGER
+ tokens[i].value = self.t_INTEGER_TYPE(result)
+ del tokens[i+1:j+1]
+ i += 1
+ tokens = self.expand_macros(tokens)
+ for i,t in enumerate(tokens):
+ if t.type == self.t_ID:
+ tokens[i] = copy.copy(t)
+ tokens[i].type = self.t_INTEGER
+ tokens[i].value = self.t_INTEGER_TYPE("0L")
+ elif t.type == self.t_INTEGER:
+ tokens[i] = copy.copy(t)
+ # Strip off any trailing suffixes
+ tokens[i].value = str(tokens[i].value)
+ while tokens[i].value[-1] not in "0123456789abcdefABCDEF":
+ tokens[i].value = tokens[i].value[:-1]
+
+ expr = "".join([str(x.value) for x in tokens])
+ expr = expr.replace("&&"," and ")
+ expr = expr.replace("||"," or ")
+ expr = expr.replace("!"," not ")
+ try:
+ result = eval(expr)
+ except Exception:
+ self.error(self.source,tokens[0].lineno,"Couldn't evaluate expression")
+ result = 0
+ return result
+
+ # ----------------------------------------------------------------------
+ # parsegen()
+ #
+ # Parse an input string/
+ # ----------------------------------------------------------------------
+ def parsegen(self,input,source=None):
+
+ # Replace trigraph sequences
+ t = trigraph(input)
+ lines = self.group_lines(t)
+
+ if not source:
+ source = ""
+
+ self.define("__FILE__ \"%s\"" % source)
+
+ self.source = source
+ chunk = []
+ enable = True
+ iftrigger = False
+ ifstack = []
+
+ for x in lines:
+ for i,tok in enumerate(x):
+ if tok.type not in self.t_WS: break
+ if tok.value == '#':
+ # Preprocessor directive
+
+ # insert necessary whitespace instead of eaten tokens
+ for tok in x:
+ if tok.type in self.t_WS and '\n' in tok.value:
+ chunk.append(tok)
+
+ dirtokens = self.tokenstrip(x[i+1:])
+ if dirtokens:
+ name = dirtokens[0].value
+ args = self.tokenstrip(dirtokens[1:])
+ else:
+ name = ""
+ args = []
+
+ if name == 'define':
+ if enable:
+ for tok in self.expand_macros(chunk):
+ yield tok
+ chunk = []
+ self.define(args)
+ elif name == 'include':
+ if enable:
+ for tok in self.expand_macros(chunk):
+ yield tok
+ chunk = []
+ oldfile = self.macros['__FILE__']
+ for tok in self.include(args):
+ yield tok
+ self.macros['__FILE__'] = oldfile
+ self.source = source
+ elif name == 'undef':
+ if enable:
+ for tok in self.expand_macros(chunk):
+ yield tok
+ chunk = []
+ self.undef(args)
+ elif name == 'ifdef':
+ ifstack.append((enable,iftrigger))
+ if enable:
+ if not args[0].value in self.macros:
+ enable = False
+ iftrigger = False
+ else:
+ iftrigger = True
+ elif name == 'ifndef':
+ ifstack.append((enable,iftrigger))
+ if enable:
+ if args[0].value in self.macros:
+ enable = False
+ iftrigger = False
+ else:
+ iftrigger = True
+ elif name == 'if':
+ ifstack.append((enable,iftrigger))
+ if enable:
+ result = self.evalexpr(args)
+ if not result:
+ enable = False
+ iftrigger = False
+ else:
+ iftrigger = True
+ elif name == 'elif':
+ if ifstack:
+ if ifstack[-1][0]: # We only pay attention if outer "if" allows this
+ if enable: # If already true, we flip enable False
+ enable = False
+ elif not iftrigger: # If False, but not triggered yet, we'll check expression
+ result = self.evalexpr(args)
+ if result:
+ enable = True
+ iftrigger = True
+ else:
+ self.error(self.source,dirtokens[0].lineno,"Misplaced #elif")
+
+ elif name == 'else':
+ if ifstack:
+ if ifstack[-1][0]:
+ if enable:
+ enable = False
+ elif not iftrigger:
+ enable = True
+ iftrigger = True
+ else:
+ self.error(self.source,dirtokens[0].lineno,"Misplaced #else")
+
+ elif name == 'endif':
+ if ifstack:
+ enable,iftrigger = ifstack.pop()
+ else:
+ self.error(self.source,dirtokens[0].lineno,"Misplaced #endif")
+ else:
+ # Unknown preprocessor directive
+ pass
+
+ else:
+ # Normal text
+ if enable:
+ chunk.extend(x)
+
+ for tok in self.expand_macros(chunk):
+ yield tok
+ chunk = []
+
+ # ----------------------------------------------------------------------
+ # include()
+ #
+ # Implementation of file-inclusion
+ # ----------------------------------------------------------------------
+
+ def include(self,tokens):
+ # Try to extract the filename and then process an include file
+ if not tokens:
+ return
+ if tokens:
+ if tokens[0].value != '<' and tokens[0].type != self.t_STRING:
+ tokens = self.expand_macros(tokens)
+
+ if tokens[0].value == '<':
+ # Include <...>
+ i = 1
+ while i < len(tokens):
+ if tokens[i].value == '>':
+ break
+ i += 1
+ else:
+ print("Malformed #include <...>")
+ return
+ filename = "".join([x.value for x in tokens[1:i]])
+ path = self.path + [""] + self.temp_path
+ elif tokens[0].type == self.t_STRING:
+ filename = tokens[0].value[1:-1]
+ path = self.temp_path + [""] + self.path
+ else:
+ print("Malformed #include statement")
+ return
+ for p in path:
+ iname = os.path.join(p,filename)
+ try:
+ data = open(iname,"r").read()
+ dname = os.path.dirname(iname)
+ if dname:
+ self.temp_path.insert(0,dname)
+ for tok in self.parsegen(data,filename):
+ yield tok
+ if dname:
+ del self.temp_path[0]
+ break
+ except IOError:
+ pass
+ else:
+ print("Couldn't find '%s'" % filename)
+
+ # ----------------------------------------------------------------------
+ # define()
+ #
+ # Define a new macro
+ # ----------------------------------------------------------------------
+
+ def define(self,tokens):
+ if isinstance(tokens,STRING_TYPES):
+ tokens = self.tokenize(tokens)
+
+ linetok = tokens
+ try:
+ name = linetok[0]
+ if len(linetok) > 1:
+ mtype = linetok[1]
+ else:
+ mtype = None
+ if not mtype:
+ m = Macro(name.value,[])
+ self.macros[name.value] = m
+ elif mtype.type in self.t_WS:
+ # A normal macro
+ m = Macro(name.value,self.tokenstrip(linetok[2:]))
+ self.macros[name.value] = m
+ elif mtype.value == '(':
+ # A macro with arguments
+ tokcount, args, positions = self.collect_args(linetok[1:])
+ variadic = False
+ for a in args:
+ if variadic:
+ print("No more arguments may follow a variadic argument")
+ break
+ astr = "".join([str(_i.value) for _i in a])
+ if astr == "...":
+ variadic = True
+ a[0].type = self.t_ID
+ a[0].value = '__VA_ARGS__'
+ variadic = True
+ del a[1:]
+ continue
+ elif astr[-3:] == "..." and a[0].type == self.t_ID:
+ variadic = True
+ del a[1:]
+ # If, for some reason, "." is part of the identifier, strip off the name for the purposes
+ # of macro expansion
+ if a[0].value[-3:] == '...':
+ a[0].value = a[0].value[:-3]
+ continue
+ if len(a) > 1 or a[0].type != self.t_ID:
+ print("Invalid macro argument")
+ break
+ else:
+ mvalue = self.tokenstrip(linetok[1+tokcount:])
+ i = 0
+ while i < len(mvalue):
+ if i+1 < len(mvalue):
+ if mvalue[i].type in self.t_WS and mvalue[i+1].value == '##':
+ del mvalue[i]
+ continue
+ elif mvalue[i].value == '##' and mvalue[i+1].type in self.t_WS:
+ del mvalue[i+1]
+ i += 1
+ m = Macro(name.value,mvalue,[x[0].value for x in args],variadic)
+ self.macro_prescan(m)
+ self.macros[name.value] = m
+ else:
+ print("Bad macro definition")
+ except LookupError:
+ print("Bad macro definition")
+
+ # ----------------------------------------------------------------------
+ # undef()
+ #
+ # Undefine a macro
+ # ----------------------------------------------------------------------
+
+ def undef(self,tokens):
+ id = tokens[0].value
+ try:
+ del self.macros[id]
+ except LookupError:
+ pass
+
+ # ----------------------------------------------------------------------
+ # parse()
+ #
+ # Parse input text.
+ # ----------------------------------------------------------------------
+ def parse(self,input,source=None,ignore={}):
+ self.ignore = ignore
+ self.parser = self.parsegen(input,source)
+
+ # ----------------------------------------------------------------------
+ # token()
+ #
+ # Method to return individual tokens
+ # ----------------------------------------------------------------------
+ def token(self):
+ try:
+ while True:
+ tok = next(self.parser)
+ if tok.type not in self.ignore: return tok
+ except StopIteration:
+ self.parser = None
+ return None
+
+if __name__ == '__main__':
+ import bb._vendor.ply.lex as lex
+ lexer = lex.lex()
+
+ # Run a preprocessor
+ import sys
+ f = open(sys.argv[1])
+ input = f.read()
+
+ p = Preprocessor(lexer)
+ p.parse(input,sys.argv[1])
+ while True:
+ tok = p.token()
+ if not tok: break
+ print(p.source, tok)
+
+
+
+
+
+
+
+
+
+
+
diff --git a/bitbake/lib/bb/_vendor/ply/ctokens.py b/bitbake/lib/bb/_vendor/ply/ctokens.py
new file mode 100644
index 00000000..f6f6952d
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/ply/ctokens.py
@@ -0,0 +1,133 @@
+# ----------------------------------------------------------------------
+# ctokens.py
+#
+# Token specifications for symbols in ANSI C and C++. This file is
+# meant to be used as a library in other tokenizers.
+# ----------------------------------------------------------------------
+
+# Reserved words
+
+tokens = [
+ # Literals (identifier, integer constant, float constant, string constant, char const)
+ 'ID', 'TYPEID', 'INTEGER', 'FLOAT', 'STRING', 'CHARACTER',
+
+ # Operators (+,-,*,/,%,|,&,~,^,<<,>>, ||, &&, !, <, <=, >, >=, ==, !=)
+ 'PLUS', 'MINUS', 'TIMES', 'DIVIDE', 'MODULO',
+ 'OR', 'AND', 'NOT', 'XOR', 'LSHIFT', 'RSHIFT',
+ 'LOR', 'LAND', 'LNOT',
+ 'LT', 'LE', 'GT', 'GE', 'EQ', 'NE',
+
+ # Assignment (=, *=, /=, %=, +=, -=, <<=, >>=, &=, ^=, |=)
+ 'EQUALS', 'TIMESEQUAL', 'DIVEQUAL', 'MODEQUAL', 'PLUSEQUAL', 'MINUSEQUAL',
+ 'LSHIFTEQUAL','RSHIFTEQUAL', 'ANDEQUAL', 'XOREQUAL', 'OREQUAL',
+
+ # Increment/decrement (++,--)
+ 'INCREMENT', 'DECREMENT',
+
+ # Structure dereference (->)
+ 'ARROW',
+
+ # Ternary operator (?)
+ 'TERNARY',
+
+ # Delimeters ( ) [ ] { } , . ; :
+ 'LPAREN', 'RPAREN',
+ 'LBRACKET', 'RBRACKET',
+ 'LBRACE', 'RBRACE',
+ 'COMMA', 'PERIOD', 'SEMI', 'COLON',
+
+ # Ellipsis (...)
+ 'ELLIPSIS',
+]
+
+# Operators
+t_PLUS = r'\+'
+t_MINUS = r'-'
+t_TIMES = r'\*'
+t_DIVIDE = r'/'
+t_MODULO = r'%'
+t_OR = r'\|'
+t_AND = r'&'
+t_NOT = r'~'
+t_XOR = r'\^'
+t_LSHIFT = r'<<'
+t_RSHIFT = r'>>'
+t_LOR = r'\|\|'
+t_LAND = r'&&'
+t_LNOT = r'!'
+t_LT = r'<'
+t_GT = r'>'
+t_LE = r'<='
+t_GE = r'>='
+t_EQ = r'=='
+t_NE = r'!='
+
+# Assignment operators
+
+t_EQUALS = r'='
+t_TIMESEQUAL = r'\*='
+t_DIVEQUAL = r'/='
+t_MODEQUAL = r'%='
+t_PLUSEQUAL = r'\+='
+t_MINUSEQUAL = r'-='
+t_LSHIFTEQUAL = r'<<='
+t_RSHIFTEQUAL = r'>>='
+t_ANDEQUAL = r'&='
+t_OREQUAL = r'\|='
+t_XOREQUAL = r'\^='
+
+# Increment/decrement
+t_INCREMENT = r'\+\+'
+t_DECREMENT = r'--'
+
+# ->
+t_ARROW = r'->'
+
+# ?
+t_TERNARY = r'\?'
+
+# Delimeters
+t_LPAREN = r'\('
+t_RPAREN = r'\)'
+t_LBRACKET = r'\['
+t_RBRACKET = r'\]'
+t_LBRACE = r'\{'
+t_RBRACE = r'\}'
+t_COMMA = r','
+t_PERIOD = r'\.'
+t_SEMI = r';'
+t_COLON = r':'
+t_ELLIPSIS = r'\.\.\.'
+
+# Identifiers
+t_ID = r'[A-Za-z_][A-Za-z0-9_]*'
+
+# Integer literal
+t_INTEGER = r'\d+([uU]|[lL]|[uU][lL]|[lL][uU])?'
+
+# Floating literal
+t_FLOAT = r'((\d+)(\.\d+)(e(\+|-)?(\d+))? | (\d+)e(\+|-)?(\d+))([lL]|[fF])?'
+
+# String literal
+t_STRING = r'\"([^\\\n]|(\\.))*?\"'
+
+# Character constant 'c' or L'c'
+t_CHARACTER = r'(L)?\'([^\\\n]|(\\.))*?\''
+
+# Comment (C-Style)
+def t_COMMENT(t):
+ r'/\*(.|\n)*?\*/'
+ t.lexer.lineno += t.value.count('\n')
+ return t
+
+# Comment (C++-Style)
+def t_CPPCOMMENT(t):
+ r'//.*\n'
+ t.lexer.lineno += 1
+ return t
+
+
+
+
+
+
diff --git a/bitbake/lib/bb/_vendor/ply/lex.py b/bitbake/lib/bb/_vendor/ply/lex.py
new file mode 100644
index 00000000..de011fe1
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/ply/lex.py
@@ -0,0 +1,901 @@
+# -----------------------------------------------------------------------------
+# ply: lex.py
+#
+# Copyright (C) 2001-2022
+# David M. Beazley (Dabeaz LLC)
+# All rights reserved.
+#
+# Latest version: https://github.com/dabeaz/ply
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+# * Neither the name of David Beazley or Dabeaz LLC may be used to
+# endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
+
+import re
+import sys
+import types
+import copy
+import os
+import inspect
+
+# This tuple contains acceptable string types
+StringTypes = (str, bytes)
+
+# This regular expression is used to match valid token names
+_is_identifier = re.compile(r'^[a-zA-Z0-9_]+$')
+
+# Exception thrown when invalid token encountered and no default error
+# handler is defined.
+class LexError(Exception):
+ def __init__(self, message, s):
+ self.args = (message,)
+ self.text = s
+
+# Token class. This class is used to represent the tokens produced.
+class LexToken(object):
+ def __repr__(self):
+ return f'LexToken({self.type},{self.value!r},{self.lineno},{self.lexpos})'
+
+# This object is a stand-in for a logging object created by the
+# logging module.
+
+class PlyLogger(object):
+ def __init__(self, f):
+ self.f = f
+
+ def critical(self, msg, *args, **kwargs):
+ self.f.write((msg % args) + '\n')
+
+ def warning(self, msg, *args, **kwargs):
+ self.f.write('WARNING: ' + (msg % args) + '\n')
+
+ def error(self, msg, *args, **kwargs):
+ self.f.write('ERROR: ' + (msg % args) + '\n')
+
+ info = critical
+ debug = critical
+
+# -----------------------------------------------------------------------------
+# === Lexing Engine ===
+#
+# The following Lexer class implements the lexer runtime. There are only
+# a few public methods and attributes:
+#
+# input() - Store a new string in the lexer
+# token() - Get the next token
+# clone() - Clone the lexer
+#
+# lineno - Current line number
+# lexpos - Current position in the input string
+# -----------------------------------------------------------------------------
+
+class Lexer:
+ def __init__(self):
+ self.lexre = None # Master regular expression. This is a list of
+ # tuples (re, findex) where re is a compiled
+ # regular expression and findex is a list
+ # mapping regex group numbers to rules
+ self.lexretext = None # Current regular expression strings
+ self.lexstatere = {} # Dictionary mapping lexer states to master regexs
+ self.lexstateretext = {} # Dictionary mapping lexer states to regex strings
+ self.lexstaterenames = {} # Dictionary mapping lexer states to symbol names
+ self.lexstate = 'INITIAL' # Current lexer state
+ self.lexstatestack = [] # Stack of lexer states
+ self.lexstateinfo = None # State information
+ self.lexstateignore = {} # Dictionary of ignored characters for each state
+ self.lexstateerrorf = {} # Dictionary of error functions for each state
+ self.lexstateeoff = {} # Dictionary of eof functions for each state
+ self.lexreflags = 0 # Optional re compile flags
+ self.lexdata = None # Actual input data (as a string)
+ self.lexpos = 0 # Current position in input text
+ self.lexlen = 0 # Length of the input text
+ self.lexerrorf = None # Error rule (if any)
+ self.lexeoff = None # EOF rule (if any)
+ self.lextokens = None # List of valid tokens
+ self.lexignore = '' # Ignored characters
+ self.lexliterals = '' # Literal characters that can be passed through
+ self.lexmodule = None # Module
+ self.lineno = 1 # Current line number
+
+ def clone(self, object=None):
+ c = copy.copy(self)
+
+ # If the object parameter has been supplied, it means we are attaching the
+ # lexer to a new object. In this case, we have to rebind all methods in
+ # the lexstatere and lexstateerrorf tables.
+
+ if object:
+ newtab = {}
+ for key, ritem in self.lexstatere.items():
+ newre = []
+ for cre, findex in ritem:
+ newfindex = []
+ for f in findex:
+ if not f or not f[0]:
+ newfindex.append(f)
+ continue
+ newfindex.append((getattr(object, f[0].__name__), f[1]))
+ newre.append((cre, newfindex))
+ newtab[key] = newre
+ c.lexstatere = newtab
+ c.lexstateerrorf = {}
+ for key, ef in self.lexstateerrorf.items():
+ c.lexstateerrorf[key] = getattr(object, ef.__name__)
+ c.lexmodule = object
+ return c
+
+ # ------------------------------------------------------------
+ # input() - Push a new string into the lexer
+ # ------------------------------------------------------------
+ def input(self, s):
+ self.lexdata = s
+ self.lexpos = 0
+ self.lexlen = len(s)
+
+ # ------------------------------------------------------------
+ # begin() - Changes the lexing state
+ # ------------------------------------------------------------
+ def begin(self, state):
+ if state not in self.lexstatere:
+ raise ValueError(f'Undefined state {state!r}')
+ self.lexre = self.lexstatere[state]
+ self.lexretext = self.lexstateretext[state]
+ self.lexignore = self.lexstateignore.get(state, '')
+ self.lexerrorf = self.lexstateerrorf.get(state, None)
+ self.lexeoff = self.lexstateeoff.get(state, None)
+ self.lexstate = state
+
+ # ------------------------------------------------------------
+ # push_state() - Changes the lexing state and saves old on stack
+ # ------------------------------------------------------------
+ def push_state(self, state):
+ self.lexstatestack.append(self.lexstate)
+ self.begin(state)
+
+ # ------------------------------------------------------------
+ # pop_state() - Restores the previous state
+ # ------------------------------------------------------------
+ def pop_state(self):
+ self.begin(self.lexstatestack.pop())
+
+ # ------------------------------------------------------------
+ # current_state() - Returns the current lexing state
+ # ------------------------------------------------------------
+ def current_state(self):
+ return self.lexstate
+
+ # ------------------------------------------------------------
+ # skip() - Skip ahead n characters
+ # ------------------------------------------------------------
+ def skip(self, n):
+ self.lexpos += n
+
+ # ------------------------------------------------------------
+ # token() - Return the next token from the Lexer
+ #
+ # Note: This function has been carefully implemented to be as fast
+ # as possible. Don't make changes unless you really know what
+ # you are doing
+ # ------------------------------------------------------------
+ def token(self):
+ # Make local copies of frequently referenced attributes
+ lexpos = self.lexpos
+ lexlen = self.lexlen
+ lexignore = self.lexignore
+ lexdata = self.lexdata
+
+ while lexpos < lexlen:
+ # This code provides some short-circuit code for whitespace, tabs, and other ignored characters
+ if lexdata[lexpos] in lexignore:
+ lexpos += 1
+ continue
+
+ # Look for a regular expression match
+ for lexre, lexindexfunc in self.lexre:
+ m = lexre.match(lexdata, lexpos)
+ if not m:
+ continue
+
+ # Create a token for return
+ tok = LexToken()
+ tok.value = m.group()
+ tok.lineno = self.lineno
+ tok.lexpos = lexpos
+
+ i = m.lastindex
+ func, tok.type = lexindexfunc[i]
+
+ if not func:
+ # If no token type was set, it's an ignored token
+ if tok.type:
+ self.lexpos = m.end()
+ return tok
+ else:
+ lexpos = m.end()
+ break
+
+ lexpos = m.end()
+
+ # If token is processed by a function, call it
+
+ tok.lexer = self # Set additional attributes useful in token rules
+ self.lexmatch = m
+ self.lexpos = lexpos
+ newtok = func(tok)
+ del tok.lexer
+ del self.lexmatch
+
+ # Every function must return a token, if nothing, we just move to next token
+ if not newtok:
+ lexpos = self.lexpos # This is here in case user has updated lexpos.
+ lexignore = self.lexignore # This is here in case there was a state change
+ break
+ return newtok
+ else:
+ # No match, see if in literals
+ if lexdata[lexpos] in self.lexliterals:
+ tok = LexToken()
+ tok.value = lexdata[lexpos]
+ tok.lineno = self.lineno
+ tok.type = tok.value
+ tok.lexpos = lexpos
+ self.lexpos = lexpos + 1
+ return tok
+
+ # No match. Call t_error() if defined.
+ if self.lexerrorf:
+ tok = LexToken()
+ tok.value = self.lexdata[lexpos:]
+ tok.lineno = self.lineno
+ tok.type = 'error'
+ tok.lexer = self
+ tok.lexpos = lexpos
+ self.lexpos = lexpos
+ newtok = self.lexerrorf(tok)
+ if lexpos == self.lexpos:
+ # Error method didn't change text position at all. This is an error.
+ raise LexError(f"Scanning error. Illegal character {lexdata[lexpos]!r}",
+ lexdata[lexpos:])
+ lexpos = self.lexpos
+ if not newtok:
+ continue
+ return newtok
+
+ self.lexpos = lexpos
+ raise LexError(f"Illegal character {lexdata[lexpos]!r} at index {lexpos}",
+ lexdata[lexpos:])
+
+ if self.lexeoff:
+ tok = LexToken()
+ tok.type = 'eof'
+ tok.value = ''
+ tok.lineno = self.lineno
+ tok.lexpos = lexpos
+ tok.lexer = self
+ self.lexpos = lexpos
+ newtok = self.lexeoff(tok)
+ return newtok
+
+ self.lexpos = lexpos + 1
+ if self.lexdata is None:
+ raise RuntimeError('No input string given with input()')
+ return None
+
+ # Iterator interface
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ t = self.token()
+ if t is None:
+ raise StopIteration
+ return t
+
+# -----------------------------------------------------------------------------
+# ==== Lex Builder ===
+#
+# The functions and classes below are used to collect lexing information
+# and build a Lexer object from it.
+# -----------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------------
+# _get_regex(func)
+#
+# Returns the regular expression assigned to a function either as a doc string
+# or as a .regex attribute attached by the @TOKEN decorator.
+# -----------------------------------------------------------------------------
+def _get_regex(func):
+ return getattr(func, 'regex', func.__doc__)
+
+# -----------------------------------------------------------------------------
+# get_caller_module_dict()
+#
+# This function returns a dictionary containing all of the symbols defined within
+# a caller further down the call stack. This is used to get the environment
+# associated with the yacc() call if none was provided.
+# -----------------------------------------------------------------------------
+def get_caller_module_dict(levels):
+ f = sys._getframe(levels)
+ return { **f.f_globals, **f.f_locals }
+
+# -----------------------------------------------------------------------------
+# _form_master_re()
+#
+# This function takes a list of all of the regex components and attempts to
+# form the master regular expression. Given limitations in the Python re
+# module, it may be necessary to break the master regex into separate expressions.
+# -----------------------------------------------------------------------------
+def _form_master_re(relist, reflags, ldict, toknames):
+ if not relist:
+ return [], [], []
+ regex = '|'.join(relist)
+ try:
+ lexre = re.compile(regex, reflags)
+
+ # Build the index to function map for the matching engine
+ lexindexfunc = [None] * (max(lexre.groupindex.values()) + 1)
+ lexindexnames = lexindexfunc[:]
+
+ for f, i in lexre.groupindex.items():
+ handle = ldict.get(f, None)
+ if type(handle) in (types.FunctionType, types.MethodType):
+ lexindexfunc[i] = (handle, toknames[f])
+ lexindexnames[i] = f
+ elif handle is not None:
+ lexindexnames[i] = f
+ if f.find('ignore_') > 0:
+ lexindexfunc[i] = (None, None)
+ else:
+ lexindexfunc[i] = (None, toknames[f])
+
+ return [(lexre, lexindexfunc)], [regex], [lexindexnames]
+ except Exception:
+ m = (len(relist) // 2) + 1
+ llist, lre, lnames = _form_master_re(relist[:m], reflags, ldict, toknames)
+ rlist, rre, rnames = _form_master_re(relist[m:], reflags, ldict, toknames)
+ return (llist+rlist), (lre+rre), (lnames+rnames)
+
+# -----------------------------------------------------------------------------
+# def _statetoken(s,names)
+#
+# Given a declaration name s of the form "t_" and a dictionary whose keys are
+# state names, this function returns a tuple (states,tokenname) where states
+# is a tuple of state names and tokenname is the name of the token. For example,
+# calling this with s = "t_foo_bar_SPAM" might return (('foo','bar'),'SPAM')
+# -----------------------------------------------------------------------------
+def _statetoken(s, names):
+ parts = s.split('_')
+ for i, part in enumerate(parts[1:], 1):
+ if part not in names and part != 'ANY':
+ break
+
+ if i > 1:
+ states = tuple(parts[1:i])
+ else:
+ states = ('INITIAL',)
+
+ if 'ANY' in states:
+ states = tuple(names)
+
+ tokenname = '_'.join(parts[i:])
+ return (states, tokenname)
+
+
+# -----------------------------------------------------------------------------
+# LexerReflect()
+#
+# This class represents information needed to build a lexer as extracted from a
+# user's input file.
+# -----------------------------------------------------------------------------
+class LexerReflect(object):
+ def __init__(self, ldict, log=None, reflags=0):
+ self.ldict = ldict
+ self.error_func = None
+ self.tokens = []
+ self.reflags = reflags
+ self.stateinfo = {'INITIAL': 'inclusive'}
+ self.modules = set()
+ self.error = False
+ self.log = PlyLogger(sys.stderr) if log is None else log
+
+ # Get all of the basic information
+ def get_all(self):
+ self.get_tokens()
+ self.get_literals()
+ self.get_states()
+ self.get_rules()
+
+ # Validate all of the information
+ def validate_all(self):
+ self.validate_tokens()
+ self.validate_literals()
+ self.validate_rules()
+ return self.error
+
+ # Get the tokens map
+ def get_tokens(self):
+ tokens = self.ldict.get('tokens', None)
+ if not tokens:
+ self.log.error('No token list is defined')
+ self.error = True
+ return
+
+ if not isinstance(tokens, (list, tuple)):
+ self.log.error('tokens must be a list or tuple')
+ self.error = True
+ return
+
+ if not tokens:
+ self.log.error('tokens is empty')
+ self.error = True
+ return
+
+ self.tokens = tokens
+
+ # Validate the tokens
+ def validate_tokens(self):
+ terminals = {}
+ for n in self.tokens:
+ if not _is_identifier.match(n):
+ self.log.error(f"Bad token name {n!r}")
+ self.error = True
+ if n in terminals:
+ self.log.warning(f"Token {n!r} multiply defined")
+ terminals[n] = 1
+
+ # Get the literals specifier
+ def get_literals(self):
+ self.literals = self.ldict.get('literals', '')
+ if not self.literals:
+ self.literals = ''
+
+ # Validate literals
+ def validate_literals(self):
+ try:
+ for c in self.literals:
+ if not isinstance(c, StringTypes) or len(c) > 1:
+ self.log.error(f'Invalid literal {c!r}. Must be a single character')
+ self.error = True
+
+ except TypeError:
+ self.log.error('Invalid literals specification. literals must be a sequence of characters')
+ self.error = True
+
+ def get_states(self):
+ self.states = self.ldict.get('states', None)
+ # Build statemap
+ if self.states:
+ if not isinstance(self.states, (tuple, list)):
+ self.log.error('states must be defined as a tuple or list')
+ self.error = True
+ else:
+ for s in self.states:
+ if not isinstance(s, tuple) or len(s) != 2:
+ self.log.error("Invalid state specifier %r. Must be a tuple (statename,'exclusive|inclusive')", s)
+ self.error = True
+ continue
+ name, statetype = s
+ if not isinstance(name, StringTypes):
+ self.log.error('State name %r must be a string', name)
+ self.error = True
+ continue
+ if not (statetype == 'inclusive' or statetype == 'exclusive'):
+ self.log.error("State type for state %r must be 'inclusive' or 'exclusive'", name)
+ self.error = True
+ continue
+ if name in self.stateinfo:
+ self.log.error("State %r already defined", name)
+ self.error = True
+ continue
+ self.stateinfo[name] = statetype
+
+ # Get all of the symbols with a t_ prefix and sort them into various
+ # categories (functions, strings, error functions, and ignore characters)
+
+ def get_rules(self):
+ tsymbols = [f for f in self.ldict if f[:2] == 't_']
+
+ # Now build up a list of functions and a list of strings
+ self.toknames = {} # Mapping of symbols to token names
+ self.funcsym = {} # Symbols defined as functions
+ self.strsym = {} # Symbols defined as strings
+ self.ignore = {} # Ignore strings by state
+ self.errorf = {} # Error functions by state
+ self.eoff = {} # EOF functions by state
+
+ for s in self.stateinfo:
+ self.funcsym[s] = []
+ self.strsym[s] = []
+
+ if len(tsymbols) == 0:
+ self.log.error('No rules of the form t_rulename are defined')
+ self.error = True
+ return
+
+ for f in tsymbols:
+ t = self.ldict[f]
+ states, tokname = _statetoken(f, self.stateinfo)
+ self.toknames[f] = tokname
+
+ if hasattr(t, '__call__'):
+ if tokname == 'error':
+ for s in states:
+ self.errorf[s] = t
+ elif tokname == 'eof':
+ for s in states:
+ self.eoff[s] = t
+ elif tokname == 'ignore':
+ line = t.__code__.co_firstlineno
+ file = t.__code__.co_filename
+ self.log.error("%s:%d: Rule %r must be defined as a string", file, line, t.__name__)
+ self.error = True
+ else:
+ for s in states:
+ self.funcsym[s].append((f, t))
+ elif isinstance(t, StringTypes):
+ if tokname == 'ignore':
+ for s in states:
+ self.ignore[s] = t
+ if '\\' in t:
+ self.log.warning("%s contains a literal backslash '\\'", f)
+
+ elif tokname == 'error':
+ self.log.error("Rule %r must be defined as a function", f)
+ self.error = True
+ else:
+ for s in states:
+ self.strsym[s].append((f, t))
+ else:
+ self.log.error('%s not defined as a function or string', f)
+ self.error = True
+
+ # Sort the functions by line number
+ for f in self.funcsym.values():
+ f.sort(key=lambda x: x[1].__code__.co_firstlineno)
+
+ # Sort the strings by regular expression length
+ for s in self.strsym.values():
+ s.sort(key=lambda x: len(x[1]), reverse=True)
+
+ # Validate all of the t_rules collected
+ def validate_rules(self):
+ for state in self.stateinfo:
+ # Validate all rules defined by functions
+
+ for fname, f in self.funcsym[state]:
+ line = f.__code__.co_firstlineno
+ file = f.__code__.co_filename
+ module = inspect.getmodule(f)
+ self.modules.add(module)
+
+ tokname = self.toknames[fname]
+ if isinstance(f, types.MethodType):
+ reqargs = 2
+ else:
+ reqargs = 1
+ nargs = f.__code__.co_argcount
+ if nargs > reqargs:
+ self.log.error("%s:%d: Rule %r has too many arguments", file, line, f.__name__)
+ self.error = True
+ continue
+
+ if nargs < reqargs:
+ self.log.error("%s:%d: Rule %r requires an argument", file, line, f.__name__)
+ self.error = True
+ continue
+
+ if not _get_regex(f):
+ self.log.error("%s:%d: No regular expression defined for rule %r", file, line, f.__name__)
+ self.error = True
+ continue
+
+ try:
+ c = re.compile('(?P<%s>%s)' % (fname, _get_regex(f)), self.reflags)
+ if c.match(''):
+ self.log.error("%s:%d: Regular expression for rule %r matches empty string", file, line, f.__name__)
+ self.error = True
+ except re.error as e:
+ self.log.error("%s:%d: Invalid regular expression for rule '%s'. %s", file, line, f.__name__, e)
+ if '#' in _get_regex(f):
+ self.log.error("%s:%d. Make sure '#' in rule %r is escaped with '\\#'", file, line, f.__name__)
+ self.error = True
+
+ # Validate all rules defined by strings
+ for name, r in self.strsym[state]:
+ tokname = self.toknames[name]
+ if tokname == 'error':
+ self.log.error("Rule %r must be defined as a function", name)
+ self.error = True
+ continue
+
+ if tokname not in self.tokens and tokname.find('ignore_') < 0:
+ self.log.error("Rule %r defined for an unspecified token %s", name, tokname)
+ self.error = True
+ continue
+
+ try:
+ c = re.compile('(?P<%s>%s)' % (name, r), self.reflags)
+ if (c.match('')):
+ self.log.error("Regular expression for rule %r matches empty string", name)
+ self.error = True
+ except re.error as e:
+ self.log.error("Invalid regular expression for rule %r. %s", name, e)
+ if '#' in r:
+ self.log.error("Make sure '#' in rule %r is escaped with '\\#'", name)
+ self.error = True
+
+ if not self.funcsym[state] and not self.strsym[state]:
+ self.log.error("No rules defined for state %r", state)
+ self.error = True
+
+ # Validate the error function
+ efunc = self.errorf.get(state, None)
+ if efunc:
+ f = efunc
+ line = f.__code__.co_firstlineno
+ file = f.__code__.co_filename
+ module = inspect.getmodule(f)
+ self.modules.add(module)
+
+ if isinstance(f, types.MethodType):
+ reqargs = 2
+ else:
+ reqargs = 1
+ nargs = f.__code__.co_argcount
+ if nargs > reqargs:
+ self.log.error("%s:%d: Rule %r has too many arguments", file, line, f.__name__)
+ self.error = True
+
+ if nargs < reqargs:
+ self.log.error("%s:%d: Rule %r requires an argument", file, line, f.__name__)
+ self.error = True
+
+ for module in self.modules:
+ self.validate_module(module)
+
+ # -----------------------------------------------------------------------------
+ # validate_module()
+ #
+ # This checks to see if there are duplicated t_rulename() functions or strings
+ # in the parser input file. This is done using a simple regular expression
+ # match on each line in the source code of the given module.
+ # -----------------------------------------------------------------------------
+
+ def validate_module(self, module):
+ try:
+ lines, linen = inspect.getsourcelines(module)
+ except IOError:
+ return
+
+ fre = re.compile(r'\s*def\s+(t_[a-zA-Z_0-9]*)\(')
+ sre = re.compile(r'\s*(t_[a-zA-Z_0-9]*)\s*=')
+
+ counthash = {}
+ linen += 1
+ for line in lines:
+ m = fre.match(line)
+ if not m:
+ m = sre.match(line)
+ if m:
+ name = m.group(1)
+ prev = counthash.get(name)
+ if not prev:
+ counthash[name] = linen
+ else:
+ filename = inspect.getsourcefile(module)
+ self.log.error('%s:%d: Rule %s redefined. Previously defined on line %d', filename, linen, name, prev)
+ self.error = True
+ linen += 1
+
+# -----------------------------------------------------------------------------
+# lex(module)
+#
+# Build all of the regular expression rules from definitions in the supplied module
+# -----------------------------------------------------------------------------
+def lex(*, module=None, object=None, debug=False,
+ reflags=int(re.VERBOSE), debuglog=None, errorlog=None):
+
+ global lexer
+
+ ldict = None
+ stateinfo = {'INITIAL': 'inclusive'}
+ lexobj = Lexer()
+ global token, input
+
+ if errorlog is None:
+ errorlog = PlyLogger(sys.stderr)
+
+ if debug:
+ if debuglog is None:
+ debuglog = PlyLogger(sys.stderr)
+
+ # Get the module dictionary used for the lexer
+ if object:
+ module = object
+
+ # Get the module dictionary used for the parser
+ if module:
+ _items = [(k, getattr(module, k)) for k in dir(module)]
+ ldict = dict(_items)
+ # If no __file__ attribute is available, try to obtain it from the __module__ instead
+ if '__file__' not in ldict:
+ ldict['__file__'] = sys.modules[ldict['__module__']].__file__
+ else:
+ ldict = get_caller_module_dict(2)
+
+ # Collect parser information from the dictionary
+ linfo = LexerReflect(ldict, log=errorlog, reflags=reflags)
+ linfo.get_all()
+ if linfo.validate_all():
+ raise SyntaxError("Can't build lexer")
+
+ # Dump some basic debugging information
+ if debug:
+ debuglog.info('lex: tokens = %r', linfo.tokens)
+ debuglog.info('lex: literals = %r', linfo.literals)
+ debuglog.info('lex: states = %r', linfo.stateinfo)
+
+ # Build a dictionary of valid token names
+ lexobj.lextokens = set()
+ for n in linfo.tokens:
+ lexobj.lextokens.add(n)
+
+ # Get literals specification
+ if isinstance(linfo.literals, (list, tuple)):
+ lexobj.lexliterals = type(linfo.literals[0])().join(linfo.literals)
+ else:
+ lexobj.lexliterals = linfo.literals
+
+ lexobj.lextokens_all = lexobj.lextokens | set(lexobj.lexliterals)
+
+ # Get the stateinfo dictionary
+ stateinfo = linfo.stateinfo
+
+ regexs = {}
+ # Build the master regular expressions
+ for state in stateinfo:
+ regex_list = []
+
+ # Add rules defined by functions first
+ for fname, f in linfo.funcsym[state]:
+ regex_list.append('(?P<%s>%s)' % (fname, _get_regex(f)))
+ if debug:
+ debuglog.info("lex: Adding rule %s -> '%s' (state '%s')", fname, _get_regex(f), state)
+
+ # Now add all of the simple rules
+ for name, r in linfo.strsym[state]:
+ regex_list.append('(?P<%s>%s)' % (name, r))
+ if debug:
+ debuglog.info("lex: Adding rule %s -> '%s' (state '%s')", name, r, state)
+
+ regexs[state] = regex_list
+
+ # Build the master regular expressions
+
+ if debug:
+ debuglog.info('lex: ==== MASTER REGEXS FOLLOW ====')
+
+ for state in regexs:
+ lexre, re_text, re_names = _form_master_re(regexs[state], reflags, ldict, linfo.toknames)
+ lexobj.lexstatere[state] = lexre
+ lexobj.lexstateretext[state] = re_text
+ lexobj.lexstaterenames[state] = re_names
+ if debug:
+ for i, text in enumerate(re_text):
+ debuglog.info("lex: state '%s' : regex[%d] = '%s'", state, i, text)
+
+ # For inclusive states, we need to add the regular expressions from the INITIAL state
+ for state, stype in stateinfo.items():
+ if state != 'INITIAL' and stype == 'inclusive':
+ lexobj.lexstatere[state].extend(lexobj.lexstatere['INITIAL'])
+ lexobj.lexstateretext[state].extend(lexobj.lexstateretext['INITIAL'])
+ lexobj.lexstaterenames[state].extend(lexobj.lexstaterenames['INITIAL'])
+
+ lexobj.lexstateinfo = stateinfo
+ lexobj.lexre = lexobj.lexstatere['INITIAL']
+ lexobj.lexretext = lexobj.lexstateretext['INITIAL']
+ lexobj.lexreflags = reflags
+
+ # Set up ignore variables
+ lexobj.lexstateignore = linfo.ignore
+ lexobj.lexignore = lexobj.lexstateignore.get('INITIAL', '')
+
+ # Set up error functions
+ lexobj.lexstateerrorf = linfo.errorf
+ lexobj.lexerrorf = linfo.errorf.get('INITIAL', None)
+ if not lexobj.lexerrorf:
+ errorlog.warning('No t_error rule is defined')
+
+ # Set up eof functions
+ lexobj.lexstateeoff = linfo.eoff
+ lexobj.lexeoff = linfo.eoff.get('INITIAL', None)
+
+ # Check state information for ignore and error rules
+ for s, stype in stateinfo.items():
+ if stype == 'exclusive':
+ if s not in linfo.errorf:
+ errorlog.warning("No error rule is defined for exclusive state %r", s)
+ if s not in linfo.ignore and lexobj.lexignore:
+ errorlog.warning("No ignore rule is defined for exclusive state %r", s)
+ elif stype == 'inclusive':
+ if s not in linfo.errorf:
+ linfo.errorf[s] = linfo.errorf.get('INITIAL', None)
+ if s not in linfo.ignore:
+ linfo.ignore[s] = linfo.ignore.get('INITIAL', '')
+
+ # Create global versions of the token() and input() functions
+ token = lexobj.token
+ input = lexobj.input
+ lexer = lexobj
+
+ return lexobj
+
+# -----------------------------------------------------------------------------
+# runmain()
+#
+# This runs the lexer as a main program
+# -----------------------------------------------------------------------------
+
+def runmain(lexer=None, data=None):
+ if not data:
+ try:
+ filename = sys.argv[1]
+ with open(filename) as f:
+ data = f.read()
+ except IndexError:
+ sys.stdout.write('Reading from standard input (type EOF to end):\n')
+ data = sys.stdin.read()
+
+ if lexer:
+ _input = lexer.input
+ else:
+ _input = input
+ _input(data)
+ if lexer:
+ _token = lexer.token
+ else:
+ _token = token
+
+ while True:
+ tok = _token()
+ if not tok:
+ break
+ sys.stdout.write(f'({tok.type},{tok.value!r},{tok.lineno},{tok.lexpos})\n')
+
+# -----------------------------------------------------------------------------
+# @TOKEN(regex)
+#
+# This decorator function can be used to set the regex expression on a function
+# when its docstring might need to be set in an alternative way
+# -----------------------------------------------------------------------------
+
+def TOKEN(r):
+ def set_regex(f):
+ if hasattr(r, '__call__'):
+ f.regex = _get_regex(r)
+ else:
+ f.regex = r
+ return f
+ return set_regex
diff --git a/bitbake/lib/ply/yacc.py b/bitbake/lib/bb/_vendor/ply/yacc.py
similarity index 60%
rename from bitbake/lib/ply/yacc.py
rename to bitbake/lib/bb/_vendor/ply/yacc.py
index 381b50cf..03bd86ee 100644
--- a/bitbake/lib/ply/yacc.py
+++ b/bitbake/lib/bb/_vendor/ply/yacc.py
@@ -1,22 +1,22 @@
# -----------------------------------------------------------------------------
# ply: yacc.py
#
-# Copyright (C) 2001-2009,
+# Copyright (C) 2001-2017
# David M. Beazley (Dabeaz LLC)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
-#
+#
# * Redistributions of source code must retain the above copyright notice,
-# this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
-# and/or other materials provided with the distribution.
+# and/or other materials provided with the distribution.
# * Neither the name of the David Beazley or Dabeaz LLC may be used to
# endorse or promote products derived from this software without
-# specific prior written permission.
+# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
@@ -59,8 +59,16 @@
# own risk!
# ----------------------------------------------------------------------------
-__version__ = "3.3"
-__tabversion__ = "3.2" # Table version
+import re
+import types
+import sys
+import os.path
+import inspect
+import base64
+import warnings
+
+__version__ = '3.10'
+__tabversion__ = '3.10'
#-----------------------------------------------------------------------------
# === User configurable parameters ===
@@ -68,7 +76,7 @@ __tabversion__ = "3.2" # Table version
# Change these to modify the default behavior of yacc (if you wish)
#-----------------------------------------------------------------------------
-yaccdebug = 0 # Debugging mode. If set, yacc generates a
+yaccdebug = True # Debugging mode. If set, yacc generates a
# a 'parser.out' file in the current directory
debug_file = 'parser.out' # Default name of the debugging file
@@ -77,86 +85,117 @@ default_lr = 'LALR' # Default LR table generation method
error_count = 3 # Number of symbols that must be shifted to leave recovery mode
-yaccdevel = 0 # Set to True if developing yacc. This turns off optimized
+yaccdevel = False # Set to True if developing yacc. This turns off optimized
# implementations of certain functions.
resultlimit = 40 # Size limit of results when running in debug mode.
pickle_protocol = 0 # Protocol to use when writing pickle files
-import re, types, sys, os.path
-
-# Compatibility function for python 2.6/3.0
+# String type-checking compatibility
if sys.version_info[0] < 3:
- def func_code(f):
- return f.func_code
+ string_types = basestring
else:
- def func_code(f):
- return f.__code__
-
-# Compatibility
-try:
- MAXINT = sys.maxint
-except AttributeError:
- MAXINT = sys.maxsize
-
-# Python 2.x/3.0 compatibility.
-def load_ply_lex():
- if sys.version_info[0] < 3:
- import lex
- else:
- import ply.lex as lex
- return lex
+ string_types = str
+
+MAXINT = sys.maxsize
-# This object is a stand-in for a logging object created by the
+# This object is a stand-in for a logging object created by the
# logging module. PLY will use this by default to create things
# such as the parser.out file. If a user wants more detailed
# information, they can create their own logging object and pass
# it into PLY.
class PlyLogger(object):
- def __init__(self,f):
+ def __init__(self, f):
self.f = f
- def debug(self,msg,*args,**kwargs):
- self.f.write((msg % args) + "\n")
- info = debug
- def warning(self,msg,*args,**kwargs):
- self.f.write("WARNING: "+ (msg % args) + "\n")
+ def debug(self, msg, *args, **kwargs):
+ self.f.write((msg % args) + '\n')
+
+ info = debug
- def error(self,msg,*args,**kwargs):
- self.f.write("ERROR: " + (msg % args) + "\n")
+ def warning(self, msg, *args, **kwargs):
+ self.f.write('WARNING: ' + (msg % args) + '\n')
+
+ def error(self, msg, *args, **kwargs):
+ self.f.write('ERROR: ' + (msg % args) + '\n')
critical = debug
# Null logger is used when no output is generated. Does nothing.
class NullLogger(object):
- def __getattribute__(self,name):
+ def __getattribute__(self, name):
return self
- def __call__(self,*args,**kwargs):
+
+ def __call__(self, *args, **kwargs):
return self
-
+
# Exception raised for yacc-related errors
-class YaccError(Exception): pass
+class YaccError(Exception):
+ pass
# Format the result message that the parser produces when running in debug mode.
def format_result(r):
repr_str = repr(r)
- if '\n' in repr_str: repr_str = repr(repr_str)
+ if '\n' in repr_str:
+ repr_str = repr(repr_str)
if len(repr_str) > resultlimit:
- repr_str = repr_str[:resultlimit]+" ..."
- result = "<%s @ 0x%x> (%s)" % (type(r).__name__,id(r),repr_str)
+ repr_str = repr_str[:resultlimit] + ' ...'
+ result = '<%s @ 0x%x> (%s)' % (type(r).__name__, id(r), repr_str)
return result
-
# Format stack entries when the parser is running in debug mode
def format_stack_entry(r):
repr_str = repr(r)
- if '\n' in repr_str: repr_str = repr(repr_str)
+ if '\n' in repr_str:
+ repr_str = repr(repr_str)
if len(repr_str) < 16:
return repr_str
else:
- return "<%s @ 0x%x>" % (type(r).__name__,id(r))
+ return '<%s @ 0x%x>' % (type(r).__name__, id(r))
+
+# Panic mode error recovery support. This feature is being reworked--much of the
+# code here is to offer a deprecation/backwards compatible transition
+
+_errok = None
+_token = None
+_restart = None
+_warnmsg = '''PLY: Don't use global functions errok(), token(), and restart() in p_error().
+Instead, invoke the methods on the associated parser instance:
+
+ def p_error(p):
+ ...
+ # Use parser.errok(), parser.token(), parser.restart()
+ ...
+
+ parser = yacc.yacc()
+'''
+
+def errok():
+ warnings.warn(_warnmsg)
+ return _errok()
+
+def restart():
+ warnings.warn(_warnmsg)
+ return _restart()
+
+def token():
+ warnings.warn(_warnmsg)
+ return _token()
+
+# Utility function to call the p_error() function with some deprecation hacks
+def call_errorfunc(errorfunc, token, parser):
+ global _errok, _token, _restart
+ _errok = parser.errok
+ _token = parser.token
+ _restart = parser.restart
+ r = errorfunc(token)
+ try:
+ del _errok, _token, _restart
+ except NameError:
+ pass
+ return r
#-----------------------------------------------------------------------------
# === LR Parsing Engine ===
@@ -176,8 +215,11 @@ def format_stack_entry(r):
# .endlexpos = Ending lex position (optional, set automatically)
class YaccSymbol:
- def __str__(self): return self.type
- def __repr__(self): return str(self)
+ def __str__(self):
+ return self.type
+
+ def __repr__(self):
+ return str(self)
# This class is a wrapper around the objects actually passed to each
# grammar rule. Index lookup and assignment actually assign the
@@ -189,48 +231,50 @@ class YaccSymbol:
# representing the range of positional information for a symbol.
class YaccProduction:
- def __init__(self,s,stack=None):
+ def __init__(self, s, stack=None):
self.slice = s
self.stack = stack
self.lexer = None
- self.parser= None
- def __getitem__(self,n):
- if isinstance(n,slice):
- return [self[i] for i in range(*(n.indices(len(self.slice))))]
- if n >= 0: return self.slice[n].value
- else: return self.stack[n].value
-
- def __setitem__(self,n,v):
+ self.parser = None
+
+ def __getitem__(self, n):
+ if isinstance(n, slice):
+ return [s.value for s in self.slice[n]]
+ elif n >= 0:
+ return self.slice[n].value
+ else:
+ return self.stack[n].value
+
+ def __setitem__(self, n, v):
self.slice[n].value = v
- def __getslice__(self,i,j):
+ def __getslice__(self, i, j):
return [s.value for s in self.slice[i:j]]
def __len__(self):
return len(self.slice)
- def lineno(self,n):
- return getattr(self.slice[n],"lineno",0)
+ def lineno(self, n):
+ return getattr(self.slice[n], 'lineno', 0)
- def set_lineno(self,n,lineno):
+ def set_lineno(self, n, lineno):
self.slice[n].lineno = lineno
- def linespan(self,n):
- startline = getattr(self.slice[n],"lineno",0)
- endline = getattr(self.slice[n],"endlineno",startline)
- return startline,endline
+ def linespan(self, n):
+ startline = getattr(self.slice[n], 'lineno', 0)
+ endline = getattr(self.slice[n], 'endlineno', startline)
+ return startline, endline
- def lexpos(self,n):
- return getattr(self.slice[n],"lexpos",0)
+ def lexpos(self, n):
+ return getattr(self.slice[n], 'lexpos', 0)
- def lexspan(self,n):
- startpos = getattr(self.slice[n],"lexpos",0)
- endpos = getattr(self.slice[n],"endlexpos",startpos)
- return startpos,endpos
+ def lexspan(self, n):
+ startpos = getattr(self.slice[n], 'lexpos', 0)
+ endpos = getattr(self.slice[n], 'endlexpos', startpos)
+ return startpos, endpos
def error(self):
- raise SyntaxError
-
+ raise SyntaxError
# -----------------------------------------------------------------------------
# == LRParser ==
@@ -239,14 +283,16 @@ class YaccProduction:
# -----------------------------------------------------------------------------
class LRParser:
- def __init__(self,lrtab,errorf):
+ def __init__(self, lrtab, errorf):
self.productions = lrtab.lr_productions
- self.action = lrtab.lr_action
- self.goto = lrtab.lr_goto
- self.errorfunc = errorf
+ self.action = lrtab.lr_action
+ self.goto = lrtab.lr_goto
+ self.errorfunc = errorf
+ self.set_defaulted_states()
+ self.errorok = True
def errok(self):
- self.errorok = 1
+ self.errorok = True
def restart(self):
del self.statestack[:]
@@ -256,24 +302,42 @@ class LRParser:
self.symstack.append(sym)
self.statestack.append(0)
- def parse(self,input=None,lexer=None,debug=0,tracking=0,tokenfunc=None):
+ # Defaulted state support.
+ # This method identifies parser states where there is only one possible reduction action.
+ # For such states, the parser can make a choose to make a rule reduction without consuming
+ # the next look-ahead token. This delayed invocation of the tokenizer can be useful in
+ # certain kinds of advanced parsing situations where the lexer and parser interact with
+ # each other or change states (i.e., manipulation of scope, lexer states, etc.).
+ #
+ # See: http://www.gnu.org/software/bison/manual/html_node/Default-Reductions.html#Default-Reductions
+ def set_defaulted_states(self):
+ self.defaulted_states = {}
+ for state, actions in self.action.items():
+ rules = list(actions.values())
+ if len(rules) == 1 and rules[0] < 0:
+ self.defaulted_states[state] = rules[0]
+
+ def disable_defaulted_states(self):
+ self.defaulted_states = {}
+
+ def parse(self, input=None, lexer=None, debug=False, tracking=False, tokenfunc=None):
if debug or yaccdevel:
- if isinstance(debug,int):
+ if isinstance(debug, int):
debug = PlyLogger(sys.stderr)
- return self.parsedebug(input,lexer,debug,tracking,tokenfunc)
+ return self.parsedebug(input, lexer, debug, tracking, tokenfunc)
elif tracking:
- return self.parseopt(input,lexer,debug,tracking,tokenfunc)
+ return self.parseopt(input, lexer, debug, tracking, tokenfunc)
else:
- return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
-
+ return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
+
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# parsedebug().
#
# This is the debugging enabled version of parse(). All changes made to the
- # parsing engine should be made here. For the non-debugging version,
- # copy this code to a method parseopt() and delete all of the sections
- # enclosed in:
+ # parsing engine should be made here. Optimized versions of this function
+ # are automatically created by the ply/ygen.py script. This script cuts out
+ # sections enclosed in markers such as this:
#
# #--! DEBUG
# statements
@@ -281,22 +345,24 @@ class LRParser:
#
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- def parsedebug(self,input=None,lexer=None,debug=None,tracking=0,tokenfunc=None):
- lookahead = None # Current lookahead symbol
- lookaheadstack = [ ] # Stack of lookahead symbols
- actions = self.action # Local reference to action table (to avoid lookup on self.)
- goto = self.goto # Local reference to goto table (to avoid lookup on self.)
- prod = self.productions # Local reference to production list (to avoid lookup on self.)
- pslice = YaccProduction(None) # Production object passed to grammar rules
- errorcount = 0 # Used during error recovery
-
- # --! DEBUG
- debug.info("PLY: PARSE DEBUG START")
- # --! DEBUG
+ def parsedebug(self, input=None, lexer=None, debug=False, tracking=False, tokenfunc=None):
+ #--! parsedebug-start
+ lookahead = None # Current lookahead symbol
+ lookaheadstack = [] # Stack of lookahead symbols
+ actions = self.action # Local reference to action table (to avoid lookup on self.)
+ goto = self.goto # Local reference to goto table (to avoid lookup on self.)
+ prod = self.productions # Local reference to production list (to avoid lookup on self.)
+ defaulted_states = self.defaulted_states # Local reference to defaulted states
+ pslice = YaccProduction(None) # Production object passed to grammar rules
+ errorcount = 0 # Used during error recovery
+
+ #--! DEBUG
+ debug.info('PLY: PARSE DEBUG START')
+ #--! DEBUG
# If no lexer was given, we will try to use the lex module
if not lexer:
- lex = load_ply_lex()
+ from . import lex
lexer = lex.lexer
# Set up the lexer and parser objects on pslice
@@ -308,16 +374,19 @@ class LRParser:
lexer.input(input)
if tokenfunc is None:
- # Tokenize function
- get_token = lexer.token
+ # Tokenize function
+ get_token = lexer.token
else:
- get_token = tokenfunc
+ get_token = tokenfunc
+
+ # Set the parser() token method (sometimes used in error recovery)
+ self.token = get_token
# Set up the state and symbol stacks
- statestack = [ ] # Stack of parsing states
+ statestack = [] # Stack of parsing states
self.statestack = statestack
- symstack = [ ] # Stack of grammar symbols
+ symstack = [] # Stack of grammar symbols
self.symstack = symstack
pslice.stack = symstack # Put in the production
@@ -327,52 +396,59 @@ class LRParser:
statestack.append(0)
sym = YaccSymbol()
- sym.type = "$end"
+ sym.type = '$end'
symstack.append(sym)
state = 0
- while 1:
+ while True:
# Get the next symbol on the input. If a lookahead symbol
# is already set, we just use that. Otherwise, we'll pull
# the next token off of the lookaheadstack or from the lexer
- # --! DEBUG
+ #--! DEBUG
debug.debug('')
debug.debug('State : %s', state)
- # --! DEBUG
+ #--! DEBUG
- if not lookahead:
- if not lookaheadstack:
- lookahead = get_token() # Get the next token
- else:
- lookahead = lookaheadstack.pop()
+ if state not in defaulted_states:
if not lookahead:
- lookahead = YaccSymbol()
- lookahead.type = "$end"
+ if not lookaheadstack:
+ lookahead = get_token() # Get the next token
+ else:
+ lookahead = lookaheadstack.pop()
+ if not lookahead:
+ lookahead = YaccSymbol()
+ lookahead.type = '$end'
+
+ # Check the action table
+ ltype = lookahead.type
+ t = actions[state].get(ltype)
+ else:
+ t = defaulted_states[state]
+ #--! DEBUG
+ debug.debug('Defaulted state %s: Reduce using %d', state, -t)
+ #--! DEBUG
- # --! DEBUG
+ #--! DEBUG
debug.debug('Stack : %s',
- ("%s . %s" % (" ".join([xx.type for xx in symstack][1:]), str(lookahead))).lstrip())
- # --! DEBUG
-
- # Check the action table
- ltype = lookahead.type
- t = actions[state].get(ltype)
+ ('%s . %s' % (' '.join([xx.type for xx in symstack][1:]), str(lookahead))).lstrip())
+ #--! DEBUG
if t is not None:
if t > 0:
# shift a symbol on the stack
statestack.append(t)
state = t
-
- # --! DEBUG
- debug.debug("Action : Shift and goto state %s", t)
- # --! DEBUG
+
+ #--! DEBUG
+ debug.debug('Action : Shift and goto state %s', t)
+ #--! DEBUG
symstack.append(lookahead)
lookahead = None
# Decrease error count on successful shift
- if errorcount: errorcount -=1
+ if errorcount:
+ errorcount -= 1
continue
if t < 0:
@@ -386,72 +462,77 @@ class LRParser:
sym.type = pname # Production name
sym.value = None
- # --! DEBUG
+ #--! DEBUG
if plen:
- debug.info("Action : Reduce rule [%s] with %s and goto state %d", p.str, "["+",".join([format_stack_entry(_v.value) for _v in symstack[-plen:]])+"]",-t)
+ debug.info('Action : Reduce rule [%s] with %s and goto state %d', p.str,
+ '['+','.join([format_stack_entry(_v.value) for _v in symstack[-plen:]])+']',
+ goto[statestack[-1-plen]][pname])
else:
- debug.info("Action : Reduce rule [%s] with %s and goto state %d", p.str, [],-t)
-
- # --! DEBUG
+ debug.info('Action : Reduce rule [%s] with %s and goto state %d', p.str, [],
+ goto[statestack[-1]][pname])
+
+ #--! DEBUG
if plen:
targ = symstack[-plen-1:]
targ[0] = sym
- # --! TRACKING
+ #--! TRACKING
if tracking:
- t1 = targ[1]
- sym.lineno = t1.lineno
- sym.lexpos = t1.lexpos
- t1 = targ[-1]
- sym.endlineno = getattr(t1,"endlineno",t1.lineno)
- sym.endlexpos = getattr(t1,"endlexpos",t1.lexpos)
-
- # --! TRACKING
+ t1 = targ[1]
+ sym.lineno = t1.lineno
+ sym.lexpos = t1.lexpos
+ t1 = targ[-1]
+ sym.endlineno = getattr(t1, 'endlineno', t1.lineno)
+ sym.endlexpos = getattr(t1, 'endlexpos', t1.lexpos)
+ #--! TRACKING
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# below as a performance optimization. Make sure
# changes get made in both locations.
pslice.slice = targ
-
+
try:
# Call the grammar rule with our special slice object
del symstack[-plen:]
- del statestack[-plen:]
+ self.state = state
p.callable(pslice)
- # --! DEBUG
- debug.info("Result : %s", format_result(pslice[0]))
- # --! DEBUG
+ del statestack[-plen:]
+ #--! DEBUG
+ debug.info('Result : %s', format_result(pslice[0]))
+ #--! DEBUG
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ symstack.extend(targ[1:-1]) # Put the production slice back on the stack
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
+
else:
- # --! TRACKING
+ #--! TRACKING
if tracking:
- sym.lineno = lexer.lineno
- sym.lexpos = lexer.lexpos
- # --! TRACKING
+ sym.lineno = lexer.lineno
+ sym.lexpos = lexer.lexpos
+ #--! TRACKING
- targ = [ sym ]
+ targ = [sym]
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# above as a performance optimization. Make sure
# changes get made in both locations.
@@ -459,41 +540,43 @@ class LRParser:
try:
# Call the grammar rule with our special slice object
+ self.state = state
p.callable(pslice)
- # --! DEBUG
- debug.info("Result : %s", format_result(pslice[0]))
- # --! DEBUG
+ #--! DEBUG
+ debug.info('Result : %s', format_result(pslice[0]))
+ #--! DEBUG
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
if t == 0:
n = symstack[-1]
- result = getattr(n,"value",None)
- # --! DEBUG
- debug.info("Done : Returning %s", format_result(result))
- debug.info("PLY: PARSE DEBUG END")
- # --! DEBUG
+ result = getattr(n, 'value', None)
+ #--! DEBUG
+ debug.info('Done : Returning %s', format_result(result))
+ debug.info('PLY: PARSE DEBUG END')
+ #--! DEBUG
return result
if t is None:
- # --! DEBUG
+ #--! DEBUG
debug.error('Error : %s',
- ("%s . %s" % (" ".join([xx.type for xx in symstack][1:]), str(lookahead))).lstrip())
- # --! DEBUG
+ ('%s . %s' % (' '.join([xx.type for xx in symstack][1:]), str(lookahead))).lstrip())
+ #--! DEBUG
# We have some kind of parsing error here. To handle
# this, we are going to push the current token onto
@@ -507,20 +590,15 @@ class LRParser:
# errorcount == 0.
if errorcount == 0 or self.errorok:
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
errtoken = lookahead
- if errtoken.type == "$end":
+ if errtoken.type == '$end':
errtoken = None # End of file!
if self.errorfunc:
- global errok,token,restart
- errok = self.errok # Set some special functions available in error recovery
- token = get_token
- restart = self.restart
- if errtoken and not hasattr(errtoken,'lexer'):
+ if errtoken and not hasattr(errtoken, 'lexer'):
errtoken.lexer = lexer
- tok = self.errorfunc(errtoken)
- del errok, token, restart # Delete special functions
-
+ self.state = state
+ tok = call_errorfunc(self.errorfunc, errtoken, self)
if self.errorok:
# User must have done some kind of panic
# mode recovery on their own. The
@@ -530,14 +608,16 @@ class LRParser:
continue
else:
if errtoken:
- if hasattr(errtoken,"lineno"): lineno = lookahead.lineno
- else: lineno = 0
+ if hasattr(errtoken, 'lineno'):
+ lineno = lookahead.lineno
+ else:
+ lineno = 0
if lineno:
- sys.stderr.write("yacc: Syntax error at line %d, token=%s\n" % (lineno, errtoken.type))
+ sys.stderr.write('yacc: Syntax error at line %d, token=%s\n' % (lineno, errtoken.type))
else:
- sys.stderr.write("yacc: Syntax error, token=%s" % errtoken.type)
+ sys.stderr.write('yacc: Syntax error, token=%s' % errtoken.type)
else:
- sys.stderr.write("yacc: Parse error in input. EOF\n")
+ sys.stderr.write('yacc: Parse error in input. EOF\n')
return
else:
@@ -547,7 +627,7 @@ class LRParser:
# entire parse has been rolled back and we're completely hosed. The token is
# discarded and we just keep going.
- if len(statestack) <= 1 and lookahead.type != "$end":
+ if len(statestack) <= 1 and lookahead.type != '$end':
lookahead = None
errtoken = None
state = 0
@@ -559,7 +639,7 @@ class LRParser:
# at the end of the file. nuke the top entry and generate an error token
# Start nuking entries on the stack
- if lookahead.type == "$end":
+ if lookahead.type == '$end':
# Whoa. We're really hosed here. Bail out
return
@@ -568,48 +648,67 @@ class LRParser:
if sym.type == 'error':
# Hmmm. Error is on top of stack, we'll just nuke input
# symbol and continue
+ #--! TRACKING
+ if tracking:
+ sym.endlineno = getattr(lookahead, 'lineno', sym.lineno)
+ sym.endlexpos = getattr(lookahead, 'lexpos', sym.lexpos)
+ #--! TRACKING
lookahead = None
continue
+
+ # Create the error symbol for the first time and make it the new lookahead symbol
t = YaccSymbol()
t.type = 'error'
- if hasattr(lookahead,"lineno"):
- t.lineno = lookahead.lineno
+
+ if hasattr(lookahead, 'lineno'):
+ t.lineno = t.endlineno = lookahead.lineno
+ if hasattr(lookahead, 'lexpos'):
+ t.lexpos = t.endlexpos = lookahead.lexpos
t.value = lookahead
lookaheadstack.append(lookahead)
lookahead = t
else:
- symstack.pop()
+ sym = symstack.pop()
+ #--! TRACKING
+ if tracking:
+ lookahead.lineno = sym.lineno
+ lookahead.lexpos = sym.lexpos
+ #--! TRACKING
statestack.pop()
- state = statestack[-1] # Potential bug fix
+ state = statestack[-1]
continue
# Call an error function here
- raise RuntimeError("yacc: internal parser error!!!\n")
+ raise RuntimeError('yacc: internal parser error!!!\n')
+
+ #--! parsedebug-end
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# parseopt().
#
- # Optimized version of parse() method. DO NOT EDIT THIS CODE DIRECTLY.
- # Edit the debug version above, then copy any modifications to the method
- # below while removing #--! DEBUG sections.
+ # Optimized version of parse() method. DO NOT EDIT THIS CODE DIRECTLY!
+ # This code is automatically generated by the ply/ygen.py script. Make
+ # changes to the parsedebug() method instead.
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+ def parseopt(self, input=None, lexer=None, debug=False, tracking=False, tokenfunc=None):
+ #--! parseopt-start
+ lookahead = None # Current lookahead symbol
+ lookaheadstack = [] # Stack of lookahead symbols
+ actions = self.action # Local reference to action table (to avoid lookup on self.)
+ goto = self.goto # Local reference to goto table (to avoid lookup on self.)
+ prod = self.productions # Local reference to production list (to avoid lookup on self.)
+ defaulted_states = self.defaulted_states # Local reference to defaulted states
+ pslice = YaccProduction(None) # Production object passed to grammar rules
+ errorcount = 0 # Used during error recovery
- def parseopt(self,input=None,lexer=None,debug=0,tracking=0,tokenfunc=None):
- lookahead = None # Current lookahead symbol
- lookaheadstack = [ ] # Stack of lookahead symbols
- actions = self.action # Local reference to action table (to avoid lookup on self.)
- goto = self.goto # Local reference to goto table (to avoid lookup on self.)
- prod = self.productions # Local reference to production list (to avoid lookup on self.)
- pslice = YaccProduction(None) # Production object passed to grammar rules
- errorcount = 0 # Used during error recovery
# If no lexer was given, we will try to use the lex module
if not lexer:
- lex = load_ply_lex()
+ from . import lex
lexer = lex.lexer
-
+
# Set up the lexer and parser objects on pslice
pslice.lexer = lexer
pslice.parser = self
@@ -619,16 +718,19 @@ class LRParser:
lexer.input(input)
if tokenfunc is None:
- # Tokenize function
- get_token = lexer.token
+ # Tokenize function
+ get_token = lexer.token
else:
- get_token = tokenfunc
+ get_token = tokenfunc
+
+ # Set the parser() token method (sometimes used in error recovery)
+ self.token = get_token
# Set up the state and symbol stacks
- statestack = [ ] # Stack of parsing states
+ statestack = [] # Stack of parsing states
self.statestack = statestack
- symstack = [ ] # Stack of grammar symbols
+ symstack = [] # Stack of grammar symbols
self.symstack = symstack
pslice.stack = symstack # Put in the production
@@ -641,23 +743,28 @@ class LRParser:
sym.type = '$end'
symstack.append(sym)
state = 0
- while 1:
+ while True:
# Get the next symbol on the input. If a lookahead symbol
# is already set, we just use that. Otherwise, we'll pull
# the next token off of the lookaheadstack or from the lexer
- if not lookahead:
- if not lookaheadstack:
- lookahead = get_token() # Get the next token
- else:
- lookahead = lookaheadstack.pop()
+
+ if state not in defaulted_states:
if not lookahead:
- lookahead = YaccSymbol()
- lookahead.type = '$end'
+ if not lookaheadstack:
+ lookahead = get_token() # Get the next token
+ else:
+ lookahead = lookaheadstack.pop()
+ if not lookahead:
+ lookahead = YaccSymbol()
+ lookahead.type = '$end'
+
+ # Check the action table
+ ltype = lookahead.type
+ t = actions[state].get(ltype)
+ else:
+ t = defaulted_states[state]
- # Check the action table
- ltype = lookahead.type
- t = actions[state].get(ltype)
if t is not None:
if t > 0:
@@ -665,11 +772,13 @@ class LRParser:
statestack.append(t)
state = t
+
symstack.append(lookahead)
lookahead = None
# Decrease error count on successful shift
- if errorcount: errorcount -=1
+ if errorcount:
+ errorcount -= 1
continue
if t < 0:
@@ -683,61 +792,64 @@ class LRParser:
sym.type = pname # Production name
sym.value = None
+
if plen:
targ = symstack[-plen-1:]
targ[0] = sym
- # --! TRACKING
+ #--! TRACKING
if tracking:
- t1 = targ[1]
- sym.lineno = t1.lineno
- sym.lexpos = t1.lexpos
- t1 = targ[-1]
- sym.endlineno = getattr(t1,"endlineno",t1.lineno)
- sym.endlexpos = getattr(t1,"endlexpos",t1.lexpos)
-
- # --! TRACKING
+ t1 = targ[1]
+ sym.lineno = t1.lineno
+ sym.lexpos = t1.lexpos
+ t1 = targ[-1]
+ sym.endlineno = getattr(t1, 'endlineno', t1.lineno)
+ sym.endlexpos = getattr(t1, 'endlexpos', t1.lexpos)
+ #--! TRACKING
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# below as a performance optimization. Make sure
# changes get made in both locations.
pslice.slice = targ
-
+
try:
# Call the grammar rule with our special slice object
del symstack[-plen:]
- del statestack[-plen:]
+ self.state = state
p.callable(pslice)
+ del statestack[-plen:]
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ symstack.extend(targ[1:-1]) # Put the production slice back on the stack
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
+
else:
- # --! TRACKING
+ #--! TRACKING
if tracking:
- sym.lineno = lexer.lineno
- sym.lexpos = lexer.lexpos
- # --! TRACKING
+ sym.lineno = lexer.lineno
+ sym.lexpos = lexer.lexpos
+ #--! TRACKING
- targ = [ sym ]
+ targ = [sym]
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# above as a performance optimization. Make sure
# changes get made in both locations.
@@ -745,29 +857,33 @@ class LRParser:
try:
# Call the grammar rule with our special slice object
+ self.state = state
p.callable(pslice)
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
if t == 0:
n = symstack[-1]
- return getattr(n,"value",None)
+ result = getattr(n, 'value', None)
+ return result
if t is None:
+
# We have some kind of parsing error here. To handle
# this, we are going to push the current token onto
# the tokenstack and replace it with an 'error' token.
@@ -780,20 +896,15 @@ class LRParser:
# errorcount == 0.
if errorcount == 0 or self.errorok:
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
errtoken = lookahead
if errtoken.type == '$end':
errtoken = None # End of file!
if self.errorfunc:
- global errok,token,restart
- errok = self.errok # Set some special functions available in error recovery
- token = get_token
- restart = self.restart
- if errtoken and not hasattr(errtoken,'lexer'):
+ if errtoken and not hasattr(errtoken, 'lexer'):
errtoken.lexer = lexer
- tok = self.errorfunc(errtoken)
- del errok, token, restart # Delete special functions
-
+ self.state = state
+ tok = call_errorfunc(self.errorfunc, errtoken, self)
if self.errorok:
# User must have done some kind of panic
# mode recovery on their own. The
@@ -803,14 +914,16 @@ class LRParser:
continue
else:
if errtoken:
- if hasattr(errtoken,"lineno"): lineno = lookahead.lineno
- else: lineno = 0
+ if hasattr(errtoken, 'lineno'):
+ lineno = lookahead.lineno
+ else:
+ lineno = 0
if lineno:
- sys.stderr.write("yacc: Syntax error at line %d, token=%s\n" % (lineno, errtoken.type))
+ sys.stderr.write('yacc: Syntax error at line %d, token=%s\n' % (lineno, errtoken.type))
else:
- sys.stderr.write("yacc: Syntax error, token=%s" % errtoken.type)
+ sys.stderr.write('yacc: Syntax error, token=%s' % errtoken.type)
else:
- sys.stderr.write("yacc: Parse error in input. EOF\n")
+ sys.stderr.write('yacc: Parse error in input. EOF\n')
return
else:
@@ -841,47 +954,67 @@ class LRParser:
if sym.type == 'error':
# Hmmm. Error is on top of stack, we'll just nuke input
# symbol and continue
+ #--! TRACKING
+ if tracking:
+ sym.endlineno = getattr(lookahead, 'lineno', sym.lineno)
+ sym.endlexpos = getattr(lookahead, 'lexpos', sym.lexpos)
+ #--! TRACKING
lookahead = None
continue
+
+ # Create the error symbol for the first time and make it the new lookahead symbol
t = YaccSymbol()
t.type = 'error'
- if hasattr(lookahead,"lineno"):
- t.lineno = lookahead.lineno
+
+ if hasattr(lookahead, 'lineno'):
+ t.lineno = t.endlineno = lookahead.lineno
+ if hasattr(lookahead, 'lexpos'):
+ t.lexpos = t.endlexpos = lookahead.lexpos
t.value = lookahead
lookaheadstack.append(lookahead)
lookahead = t
else:
- symstack.pop()
+ sym = symstack.pop()
+ #--! TRACKING
+ if tracking:
+ lookahead.lineno = sym.lineno
+ lookahead.lexpos = sym.lexpos
+ #--! TRACKING
statestack.pop()
- state = statestack[-1] # Potential bug fix
+ state = statestack[-1]
continue
# Call an error function here
- raise RuntimeError("yacc: internal parser error!!!\n")
+ raise RuntimeError('yacc: internal parser error!!!\n')
+
+ #--! parseopt-end
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# parseopt_notrack().
#
- # Optimized version of parseopt() with line number tracking removed.
- # DO NOT EDIT THIS CODE DIRECTLY. Copy the optimized version and remove
- # code in the #--! TRACKING sections
+ # Optimized version of parseopt() with line number tracking removed.
+ # DO NOT EDIT THIS CODE DIRECTLY. This code is automatically generated
+ # by the ply/ygen.py script. Make changes to the parsedebug() method instead.
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- def parseopt_notrack(self,input=None,lexer=None,debug=0,tracking=0,tokenfunc=None):
- lookahead = None # Current lookahead symbol
- lookaheadstack = [ ] # Stack of lookahead symbols
- actions = self.action # Local reference to action table (to avoid lookup on self.)
- goto = self.goto # Local reference to goto table (to avoid lookup on self.)
- prod = self.productions # Local reference to production list (to avoid lookup on self.)
- pslice = YaccProduction(None) # Production object passed to grammar rules
- errorcount = 0 # Used during error recovery
+ def parseopt_notrack(self, input=None, lexer=None, debug=False, tracking=False, tokenfunc=None):
+ #--! parseopt-notrack-start
+ lookahead = None # Current lookahead symbol
+ lookaheadstack = [] # Stack of lookahead symbols
+ actions = self.action # Local reference to action table (to avoid lookup on self.)
+ goto = self.goto # Local reference to goto table (to avoid lookup on self.)
+ prod = self.productions # Local reference to production list (to avoid lookup on self.)
+ defaulted_states = self.defaulted_states # Local reference to defaulted states
+ pslice = YaccProduction(None) # Production object passed to grammar rules
+ errorcount = 0 # Used during error recovery
+
# If no lexer was given, we will try to use the lex module
if not lexer:
- lex = load_ply_lex()
+ from . import lex
lexer = lex.lexer
-
+
# Set up the lexer and parser objects on pslice
pslice.lexer = lexer
pslice.parser = self
@@ -891,16 +1024,19 @@ class LRParser:
lexer.input(input)
if tokenfunc is None:
- # Tokenize function
- get_token = lexer.token
+ # Tokenize function
+ get_token = lexer.token
else:
- get_token = tokenfunc
+ get_token = tokenfunc
+
+ # Set the parser() token method (sometimes used in error recovery)
+ self.token = get_token
# Set up the state and symbol stacks
- statestack = [ ] # Stack of parsing states
+ statestack = [] # Stack of parsing states
self.statestack = statestack
- symstack = [ ] # Stack of grammar symbols
+ symstack = [] # Stack of grammar symbols
self.symstack = symstack
pslice.stack = symstack # Put in the production
@@ -913,23 +1049,28 @@ class LRParser:
sym.type = '$end'
symstack.append(sym)
state = 0
- while 1:
+ while True:
# Get the next symbol on the input. If a lookahead symbol
# is already set, we just use that. Otherwise, we'll pull
# the next token off of the lookaheadstack or from the lexer
- if not lookahead:
- if not lookaheadstack:
- lookahead = get_token() # Get the next token
- else:
- lookahead = lookaheadstack.pop()
+
+ if state not in defaulted_states:
if not lookahead:
- lookahead = YaccSymbol()
- lookahead.type = '$end'
+ if not lookaheadstack:
+ lookahead = get_token() # Get the next token
+ else:
+ lookahead = lookaheadstack.pop()
+ if not lookahead:
+ lookahead = YaccSymbol()
+ lookahead.type = '$end'
+
+ # Check the action table
+ ltype = lookahead.type
+ t = actions[state].get(ltype)
+ else:
+ t = defaulted_states[state]
- # Check the action table
- ltype = lookahead.type
- t = actions[state].get(ltype)
if t is not None:
if t > 0:
@@ -937,11 +1078,13 @@ class LRParser:
statestack.append(t)
state = t
+
symstack.append(lookahead)
lookahead = None
# Decrease error count on successful shift
- if errorcount: errorcount -=1
+ if errorcount:
+ errorcount -= 1
continue
if t < 0:
@@ -955,44 +1098,50 @@ class LRParser:
sym.type = pname # Production name
sym.value = None
+
if plen:
targ = symstack[-plen-1:]
targ[0] = sym
+
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# below as a performance optimization. Make sure
# changes get made in both locations.
pslice.slice = targ
-
+
try:
# Call the grammar rule with our special slice object
del symstack[-plen:]
- del statestack[-plen:]
+ self.state = state
p.callable(pslice)
+ del statestack[-plen:]
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ symstack.extend(targ[1:-1]) # Put the production slice back on the stack
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
+
else:
- targ = [ sym ]
+
+ targ = [sym]
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
- # The code enclosed in this section is duplicated
+ # The code enclosed in this section is duplicated
# above as a performance optimization. Make sure
# changes get made in both locations.
@@ -1000,29 +1149,33 @@ class LRParser:
try:
# Call the grammar rule with our special slice object
+ self.state = state
p.callable(pslice)
symstack.append(sym)
state = goto[statestack[-1]][pname]
statestack.append(state)
except SyntaxError:
# If an error was set. Enter error recovery state
- lookaheadstack.append(lookahead)
- symstack.pop()
- statestack.pop()
+ lookaheadstack.append(lookahead) # Save the current lookahead token
+ statestack.pop() # Pop back one state (before the reduce)
state = statestack[-1]
sym.type = 'error'
+ sym.value = 'error'
lookahead = sym
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
+
continue
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
if t == 0:
n = symstack[-1]
- return getattr(n,"value",None)
+ result = getattr(n, 'value', None)
+ return result
if t is None:
+
# We have some kind of parsing error here. To handle
# this, we are going to push the current token onto
# the tokenstack and replace it with an 'error' token.
@@ -1035,20 +1188,15 @@ class LRParser:
# errorcount == 0.
if errorcount == 0 or self.errorok:
errorcount = error_count
- self.errorok = 0
+ self.errorok = False
errtoken = lookahead
if errtoken.type == '$end':
errtoken = None # End of file!
if self.errorfunc:
- global errok,token,restart
- errok = self.errok # Set some special functions available in error recovery
- token = get_token
- restart = self.restart
- if errtoken and not hasattr(errtoken,'lexer'):
+ if errtoken and not hasattr(errtoken, 'lexer'):
errtoken.lexer = lexer
- tok = self.errorfunc(errtoken)
- del errok, token, restart # Delete special functions
-
+ self.state = state
+ tok = call_errorfunc(self.errorfunc, errtoken, self)
if self.errorok:
# User must have done some kind of panic
# mode recovery on their own. The
@@ -1058,14 +1206,16 @@ class LRParser:
continue
else:
if errtoken:
- if hasattr(errtoken,"lineno"): lineno = lookahead.lineno
- else: lineno = 0
+ if hasattr(errtoken, 'lineno'):
+ lineno = lookahead.lineno
+ else:
+ lineno = 0
if lineno:
- sys.stderr.write("yacc: Syntax error at line %d, token=%s\n" % (lineno, errtoken.type))
+ sys.stderr.write('yacc: Syntax error at line %d, token=%s\n' % (lineno, errtoken.type))
else:
- sys.stderr.write("yacc: Syntax error, token=%s" % errtoken.type)
+ sys.stderr.write('yacc: Syntax error, token=%s' % errtoken.type)
else:
- sys.stderr.write("yacc: Parse error in input. EOF\n")
+ sys.stderr.write('yacc: Parse error in input. EOF\n')
return
else:
@@ -1098,32 +1248,37 @@ class LRParser:
# symbol and continue
lookahead = None
continue
+
+ # Create the error symbol for the first time and make it the new lookahead symbol
t = YaccSymbol()
t.type = 'error'
- if hasattr(lookahead,"lineno"):
- t.lineno = lookahead.lineno
+
+ if hasattr(lookahead, 'lineno'):
+ t.lineno = t.endlineno = lookahead.lineno
+ if hasattr(lookahead, 'lexpos'):
+ t.lexpos = t.endlexpos = lookahead.lexpos
t.value = lookahead
lookaheadstack.append(lookahead)
lookahead = t
else:
- symstack.pop()
+ sym = symstack.pop()
statestack.pop()
- state = statestack[-1] # Potential bug fix
+ state = statestack[-1]
continue
# Call an error function here
- raise RuntimeError("yacc: internal parser error!!!\n")
+ raise RuntimeError('yacc: internal parser error!!!\n')
+
+ #--! parseopt-notrack-end
# -----------------------------------------------------------------------------
# === Grammar Representation ===
#
# The following functions, classes, and variables are used to represent and
-# manipulate the rules that make up a grammar.
+# manipulate the rules that make up a grammar.
# -----------------------------------------------------------------------------
-import re
-
# regex matching identifiers
_is_identifier = re.compile(r'^[a-zA-Z0-9_-]+$')
@@ -1133,7 +1288,7 @@ _is_identifier = re.compile(r'^[a-zA-Z0-9_-]+$')
# This class stores the raw information about a single production or grammar rule.
# A grammar rule refers to a specification such as this:
#
-# expr : expr PLUS term
+# expr : expr PLUS term
#
# Here are the basic attributes defined on all productions
#
@@ -1153,7 +1308,7 @@ _is_identifier = re.compile(r'^[a-zA-Z0-9_-]+$')
class Production(object):
reduced = 0
- def __init__(self,number,name,prod,precedence=('right',0),func=None,file='',line=0):
+ def __init__(self, number, name, prod, precedence=('right', 0), func=None, file='', line=0):
self.name = name
self.prod = tuple(prod)
self.number = number
@@ -1164,11 +1319,11 @@ class Production(object):
self.prec = precedence
# Internal settings used during table construction
-
+
self.len = len(self.prod) # Length of the production
# Create a list of unique production symbols used in the production
- self.usyms = [ ]
+ self.usyms = []
for s in self.prod:
if s not in self.usyms:
self.usyms.append(s)
@@ -1179,15 +1334,15 @@ class Production(object):
# Create a string representation
if self.prod:
- self.str = "%s -> %s" % (self.name," ".join(self.prod))
+ self.str = '%s -> %s' % (self.name, ' '.join(self.prod))
else:
- self.str = "%s -> <empty>" % self.name
+ self.str = '%s -> <empty>' % self.name
def __str__(self):
return self.str
def __repr__(self):
- return "Production("+str(self)+")"
+ return 'Production(' + str(self) + ')'
def __len__(self):
return len(self.prod)
@@ -1195,28 +1350,27 @@ class Production(object):
def __nonzero__(self):
return 1
- def __getitem__(self,index):
+ def __getitem__(self, index):
return self.prod[index]
-
- # Return the nth lr_item from the production (or None if at the end)
- def lr_item(self,n):
- if n > len(self.prod): return None
- p = LRItem(self,n)
- # Precompute the list of productions immediately following. Hack. Remove later
+ # Return the nth lr_item from the production (or None if at the end)
+ def lr_item(self, n):
+ if n > len(self.prod):
+ return None
+ p = LRItem(self, n)
+ # Precompute the list of productions immediately following.
try:
- p.lr_after = self.Prodnames[p.prod[n+1]]
- except (IndexError,KeyError):
+ p.lr_after = Prodnames[p.prod[n+1]]
+ except (IndexError, KeyError):
p.lr_after = []
try:
p.lr_before = p.prod[n-1]
except IndexError:
p.lr_before = None
-
return p
-
+
# Bind the production function name to a callable
- def bind(self,pdict):
+ def bind(self, pdict):
if self.func:
self.callable = pdict[self.func]
@@ -1225,7 +1379,7 @@ class Production(object):
# actually used by the LR parsing engine, plus some additional
# debugging information.
class MiniProduction(object):
- def __init__(self,str,name,len,func,file,line):
+ def __init__(self, str, name, len, func, file, line):
self.name = name
self.len = len
self.func = func
@@ -1233,13 +1387,15 @@ class MiniProduction(object):
self.file = file
self.line = line
self.str = str
+
def __str__(self):
return self.str
+
def __repr__(self):
- return "MiniProduction(%s)" % self.str
+ return 'MiniProduction(%s)' % self.str
# Bind the production function name to a callable
- def bind(self,pdict):
+ def bind(self, pdict):
if self.func:
self.callable = pdict[self.func]
@@ -1248,9 +1404,9 @@ class MiniProduction(object):
# class LRItem
#
# This class represents a specific stage of parsing a production rule. For
-# example:
+# example:
#
-# expr : expr . PLUS term
+# expr : expr . PLUS term
#
# In the above, the "." represents the current location of the parse. Here
# basic attributes:
@@ -1269,26 +1425,26 @@ class MiniProduction(object):
# -----------------------------------------------------------------------------
class LRItem(object):
- def __init__(self,p,n):
+ def __init__(self, p, n):
self.name = p.name
self.prod = list(p.prod)
self.number = p.number
self.lr_index = n
- self.lookaheads = { }
- self.prod.insert(n,".")
+ self.lookaheads = {}
+ self.prod.insert(n, '.')
self.prod = tuple(self.prod)
self.len = len(self.prod)
self.usyms = p.usyms
def __str__(self):
if self.prod:
- s = "%s -> %s" % (self.name," ".join(self.prod))
+ s = '%s -> %s' % (self.name, ' '.join(self.prod))
else:
- s = "%s -> <empty>" % self.name
+ s = '%s -> <empty>' % self.name
return s
def __repr__(self):
- return "LRItem("+str(self)+")"
+ return 'LRItem(' + str(self) + ')'
# -----------------------------------------------------------------------------
# rightmost_terminal()
@@ -1311,21 +1467,22 @@ def rightmost_terminal(symbols, terminals):
# This data is used for critical parts of the table generation process later.
# -----------------------------------------------------------------------------
-class GrammarError(YaccError): pass
+class GrammarError(YaccError):
+ pass
class Grammar(object):
- def __init__(self,terminals):
+ def __init__(self, terminals):
self.Productions = [None] # A list of all of the productions. The first
# entry is always reserved for the purpose of
# building an augmented grammar
- self.Prodnames = { } # A dictionary mapping the names of nonterminals to a list of all
+ self.Prodnames = {} # A dictionary mapping the names of nonterminals to a list of all
# productions of that nonterminal.
- self.Prodmap = { } # A dictionary that is only used to detect duplicate
+ self.Prodmap = {} # A dictionary that is only used to detect duplicate
# productions.
- self.Terminals = { } # A dictionary mapping the names of terminal symbols to a
+ self.Terminals = {} # A dictionary mapping the names of terminal symbols to a
# list of the rules where they are used.
for term in terminals:
@@ -1333,17 +1490,17 @@ class Grammar(object):
self.Terminals['error'] = []
- self.Nonterminals = { } # A dictionary mapping names of nonterminals to a list
+ self.Nonterminals = {} # A dictionary mapping names of nonterminals to a list
# of rule numbers where they are used.
- self.First = { } # A dictionary of precomputed FIRST(x) symbols
+ self.First = {} # A dictionary of precomputed FIRST(x) symbols
- self.Follow = { } # A dictionary of precomputed FOLLOW(x) symbols
+ self.Follow = {} # A dictionary of precomputed FOLLOW(x) symbols
- self.Precedence = { } # Precedence rules for each terminal. Contains tuples of the
+ self.Precedence = {} # Precedence rules for each terminal. Contains tuples of the
# form ('right',level) or ('nonassoc', level) or ('left',level)
- self.UsedPrecedence = { } # Precedence rules that were actually used by the grammer.
+ self.UsedPrecedence = set() # Precedence rules that were actually used by the grammer.
# This is only used to provide error checking and to generate
# a warning about unused precedence rules.
@@ -1353,7 +1510,7 @@ class Grammar(object):
def __len__(self):
return len(self.Productions)
- def __getitem__(self,index):
+ def __getitem__(self, index):
return self.Productions[index]
# -----------------------------------------------------------------------------
@@ -1364,14 +1521,14 @@ class Grammar(object):
#
# -----------------------------------------------------------------------------
- def set_precedence(self,term,assoc,level):
- assert self.Productions == [None],"Must call set_precedence() before add_production()"
+ def set_precedence(self, term, assoc, level):
+ assert self.Productions == [None], 'Must call set_precedence() before add_production()'
if term in self.Precedence:
- raise GrammarError("Precedence already specified for terminal '%s'" % term)
- if assoc not in ['left','right','nonassoc']:
+ raise GrammarError('Precedence already specified for terminal %r' % term)
+ if assoc not in ['left', 'right', 'nonassoc']:
raise GrammarError("Associativity must be one of 'left','right', or 'nonassoc'")
- self.Precedence[term] = (assoc,level)
-
+ self.Precedence[term] = (assoc, level)
+
# -----------------------------------------------------------------------------
# add_production()
#
@@ -1389,72 +1546,74 @@ class Grammar(object):
# are valid and that %prec is used correctly.
# -----------------------------------------------------------------------------
- def add_production(self,prodname,syms,func=None,file='',line=0):
+ def add_production(self, prodname, syms, func=None, file='', line=0):
if prodname in self.Terminals:
- raise GrammarError("%s:%d: Illegal rule name '%s'. Already defined as a token" % (file,line,prodname))
+ raise GrammarError('%s:%d: Illegal rule name %r. Already defined as a token' % (file, line, prodname))
if prodname == 'error':
- raise GrammarError("%s:%d: Illegal rule name '%s'. error is a reserved word" % (file,line,prodname))
+ raise GrammarError('%s:%d: Illegal rule name %r. error is a reserved word' % (file, line, prodname))
if not _is_identifier.match(prodname):
- raise GrammarError("%s:%d: Illegal rule name '%s'" % (file,line,prodname))
+ raise GrammarError('%s:%d: Illegal rule name %r' % (file, line, prodname))
- # Look for literal tokens
- for n,s in enumerate(syms):
+ # Look for literal tokens
+ for n, s in enumerate(syms):
if s[0] in "'\"":
- try:
- c = eval(s)
- if (len(c) > 1):
- raise GrammarError("%s:%d: Literal token %s in rule '%s' may only be a single character" % (file,line,s, prodname))
- if not c in self.Terminals:
- self.Terminals[c] = []
- syms[n] = c
- continue
- except SyntaxError:
- pass
+ try:
+ c = eval(s)
+ if (len(c) > 1):
+ raise GrammarError('%s:%d: Literal token %s in rule %r may only be a single character' %
+ (file, line, s, prodname))
+ if c not in self.Terminals:
+ self.Terminals[c] = []
+ syms[n] = c
+ continue
+ except SyntaxError:
+ pass
if not _is_identifier.match(s) and s != '%prec':
- raise GrammarError("%s:%d: Illegal name '%s' in rule '%s'" % (file,line,s, prodname))
-
+ raise GrammarError('%s:%d: Illegal name %r in rule %r' % (file, line, s, prodname))
+
# Determine the precedence level
if '%prec' in syms:
if syms[-1] == '%prec':
- raise GrammarError("%s:%d: Syntax error. Nothing follows %%prec" % (file,line))
+ raise GrammarError('%s:%d: Syntax error. Nothing follows %%prec' % (file, line))
if syms[-2] != '%prec':
- raise GrammarError("%s:%d: Syntax error. %%prec can only appear at the end of a grammar rule" % (file,line))
+ raise GrammarError('%s:%d: Syntax error. %%prec can only appear at the end of a grammar rule' %
+ (file, line))
precname = syms[-1]
- prodprec = self.Precedence.get(precname,None)
+ prodprec = self.Precedence.get(precname)
if not prodprec:
- raise GrammarError("%s:%d: Nothing known about the precedence of '%s'" % (file,line,precname))
+ raise GrammarError('%s:%d: Nothing known about the precedence of %r' % (file, line, precname))
else:
- self.UsedPrecedence[precname] = 1
+ self.UsedPrecedence.add(precname)
del syms[-2:] # Drop %prec from the rule
else:
# If no %prec, precedence is determined by the rightmost terminal symbol
- precname = rightmost_terminal(syms,self.Terminals)
- prodprec = self.Precedence.get(precname,('right',0))
-
+ precname = rightmost_terminal(syms, self.Terminals)
+ prodprec = self.Precedence.get(precname, ('right', 0))
+
# See if the rule is already in the rulemap
- map = "%s -> %s" % (prodname,syms)
+ map = '%s -> %s' % (prodname, syms)
if map in self.Prodmap:
m = self.Prodmap[map]
- raise GrammarError("%s:%d: Duplicate rule %s. " % (file,line, m) +
- "Previous definition at %s:%d" % (m.file, m.line))
+ raise GrammarError('%s:%d: Duplicate rule %s. ' % (file, line, m) +
+ 'Previous definition at %s:%d' % (m.file, m.line))
# From this point on, everything is valid. Create a new Production instance
pnumber = len(self.Productions)
- if not prodname in self.Nonterminals:
- self.Nonterminals[prodname] = [ ]
+ if prodname not in self.Nonterminals:
+ self.Nonterminals[prodname] = []
# Add the production number to Terminals and Nonterminals
for t in syms:
if t in self.Terminals:
self.Terminals[t].append(pnumber)
else:
- if not t in self.Nonterminals:
- self.Nonterminals[t] = [ ]
+ if t not in self.Nonterminals:
+ self.Nonterminals[t] = []
self.Nonterminals[t].append(pnumber)
# Create a production and add it to the list of productions
- p = Production(pnumber,prodname,syms,prodprec,func,file,line)
+ p = Production(pnumber, prodname, syms, prodprec, func, file, line)
self.Productions.append(p)
self.Prodmap[map] = p
@@ -1462,22 +1621,21 @@ class Grammar(object):
try:
self.Prodnames[prodname].append(p)
except KeyError:
- self.Prodnames[prodname] = [ p ]
- return 0
+ self.Prodnames[prodname] = [p]
# -----------------------------------------------------------------------------
# set_start()
#
- # Sets the starting symbol and creates the augmented grammar. Production
+ # Sets the starting symbol and creates the augmented grammar. Production
# rule 0 is S' -> start where start is the start symbol.
# -----------------------------------------------------------------------------
- def set_start(self,start=None):
+ def set_start(self, start=None):
if not start:
start = self.Productions[1].name
if start not in self.Nonterminals:
- raise GrammarError("start symbol %s undefined" % start)
- self.Productions[0] = Production(0,"S'",[start])
+ raise GrammarError('start symbol %s undefined' % start)
+ self.Productions[0] = Production(0, "S'", [start])
self.Nonterminals[start].append(0)
self.Start = start
@@ -1489,26 +1647,20 @@ class Grammar(object):
# -----------------------------------------------------------------------------
def find_unreachable(self):
-
+
# Mark all symbols that are reachable from a symbol s
def mark_reachable_from(s):
- if reachable[s]:
- # We've already reached symbol s.
+ if s in reachable:
return
- reachable[s] = 1
- for p in self.Prodnames.get(s,[]):
+ reachable.add(s)
+ for p in self.Prodnames.get(s, []):
for r in p.prod:
mark_reachable_from(r)
- reachable = { }
- for s in list(self.Terminals) + list(self.Nonterminals):
- reachable[s] = 0
-
- mark_reachable_from( self.Productions[0].prod[0] )
+ reachable = set()
+ mark_reachable_from(self.Productions[0].prod[0])
+ return [s for s in self.Nonterminals if s not in reachable]
- return [s for s in list(self.Nonterminals)
- if not reachable[s]]
-
# -----------------------------------------------------------------------------
# infinite_cycles()
#
@@ -1522,20 +1674,20 @@ class Grammar(object):
# Terminals:
for t in self.Terminals:
- terminates[t] = 1
+ terminates[t] = True
- terminates['$end'] = 1
+ terminates['$end'] = True
# Nonterminals:
# Initialize to false:
for n in self.Nonterminals:
- terminates[n] = 0
+ terminates[n] = False
# Then propagate termination until no change:
- while 1:
- some_change = 0
- for (n,pl) in self.Prodnames.items():
+ while True:
+ some_change = False
+ for (n, pl) in self.Prodnames.items():
# Nonterminal n terminates iff any of its productions terminates.
for p in pl:
# Production p terminates iff all of its rhs symbols terminate.
@@ -1543,19 +1695,19 @@ class Grammar(object):
if not terminates[s]:
# The symbol s does not terminate,
# so production p does not terminate.
- p_terminates = 0
+ p_terminates = False
break
else:
# didn't break from the loop,
# so every symbol s terminates
# so production p terminates.
- p_terminates = 1
+ p_terminates = True
if p_terminates:
# symbol n terminates!
if not terminates[n]:
- terminates[n] = 1
- some_change = 1
+ terminates[n] = True
+ some_change = True
# Don't need to consider any more productions for this n.
break
@@ -1563,9 +1715,9 @@ class Grammar(object):
break
infinite = []
- for (s,term) in terminates.items():
+ for (s, term) in terminates.items():
if not term:
- if not s in self.Prodnames and not s in self.Terminals and s != 'error':
+ if s not in self.Prodnames and s not in self.Terminals and s != 'error':
# s is used-but-not-defined, and we've already warned of that,
# so it would be overkill to say that it's also non-terminating.
pass
@@ -1574,22 +1726,22 @@ class Grammar(object):
return infinite
-
# -----------------------------------------------------------------------------
# undefined_symbols()
#
# Find all symbols that were used the grammar, but not defined as tokens or
# grammar rules. Returns a list of tuples (sym, prod) where sym in the symbol
- # and prod is the production where the symbol was used.
+ # and prod is the production where the symbol was used.
# -----------------------------------------------------------------------------
def undefined_symbols(self):
result = []
for p in self.Productions:
- if not p: continue
+ if not p:
+ continue
for s in p.prod:
- if not s in self.Prodnames and not s in self.Terminals and s != 'error':
- result.append((s,p))
+ if s not in self.Prodnames and s not in self.Terminals and s != 'error':
+ result.append((s, p))
return result
# -----------------------------------------------------------------------------
@@ -1600,7 +1752,7 @@ class Grammar(object):
# -----------------------------------------------------------------------------
def unused_terminals(self):
unused_tok = []
- for s,v in self.Terminals.items():
+ for s, v in self.Terminals.items():
if s != 'error' and not v:
unused_tok.append(s)
@@ -1615,7 +1767,7 @@ class Grammar(object):
def unused_rules(self):
unused_prod = []
- for s,v in self.Nonterminals.items():
+ for s, v in self.Nonterminals.items():
if not v:
p = self.Prodnames[s][0]
unused_prod.append(p)
@@ -1627,15 +1779,15 @@ class Grammar(object):
# Returns a list of tuples (term,precedence) corresponding to precedence
# rules that were never used by the grammar. term is the name of the terminal
# on which precedence was applied and precedence is a string such as 'left' or
- # 'right' corresponding to the type of precedence.
+ # 'right' corresponding to the type of precedence.
# -----------------------------------------------------------------------------
def unused_precedence(self):
unused = []
for termname in self.Precedence:
if not (termname in self.Terminals or termname in self.UsedPrecedence):
- unused.append((termname,self.Precedence[termname][0]))
-
+ unused.append((termname, self.Precedence[termname][0]))
+
return unused
# -------------------------------------------------------------------------
@@ -1646,19 +1798,20 @@ class Grammar(object):
# During execution of compute_first1, the result may be incomplete.
# Afterward (e.g., when called from compute_follow()), it will be complete.
# -------------------------------------------------------------------------
- def _first(self,beta):
+ def _first(self, beta):
# We are computing First(x1,x2,x3,...,xn)
- result = [ ]
+ result = []
for x in beta:
- x_produces_empty = 0
+ x_produces_empty = False
# Add all the non-<empty> symbols of First[x] to the result.
for f in self.First[x]:
if f == '<empty>':
- x_produces_empty = 1
+ x_produces_empty = True
else:
- if f not in result: result.append(f)
+ if f not in result:
+ result.append(f)
if x_produces_empty:
# We have to consider the next x in beta,
@@ -1697,17 +1850,17 @@ class Grammar(object):
self.First[n] = []
# Then propagate symbols until no change:
- while 1:
- some_change = 0
+ while True:
+ some_change = False
for n in self.Nonterminals:
for p in self.Prodnames[n]:
for f in self._first(p.prod):
if f not in self.First[n]:
- self.First[n].append( f )
- some_change = 1
+ self.First[n].append(f)
+ some_change = True
if not some_change:
break
-
+
return self.First
# ---------------------------------------------------------------------
@@ -1717,7 +1870,7 @@ class Grammar(object):
# follow set is the set of all symbols that might follow a given
# non-terminal. See the Dragon book, 2nd Ed. p. 189.
# ---------------------------------------------------------------------
- def compute_follow(self,start=None):
+ def compute_follow(self, start=None):
# If already computed, return the result
if self.Follow:
return self.Follow
@@ -1728,36 +1881,36 @@ class Grammar(object):
# Add '$end' to the follow list of the start symbol
for k in self.Nonterminals:
- self.Follow[k] = [ ]
+ self.Follow[k] = []
if not start:
start = self.Productions[1].name
- self.Follow[start] = [ '$end' ]
+ self.Follow[start] = ['$end']
- while 1:
- didadd = 0
+ while True:
+ didadd = False
for p in self.Productions[1:]:
# Here is the production set
- for i in range(len(p.prod)):
- B = p.prod[i]
+ for i, B in enumerate(p.prod):
if B in self.Nonterminals:
# Okay. We got a non-terminal in a production
fst = self._first(p.prod[i+1:])
- hasempty = 0
+ hasempty = False
for f in fst:
if f != '<empty>' and f not in self.Follow[B]:
self.Follow[B].append(f)
- didadd = 1
+ didadd = True
if f == '<empty>':
- hasempty = 1
+ hasempty = True
if hasempty or i == (len(p.prod)-1):
# Add elements of follow(a) to follow(b)
for f in self.Follow[p.name]:
if f not in self.Follow[B]:
self.Follow[B].append(f)
- didadd = 1
- if not didadd: break
+ didadd = True
+ if not didadd:
+ break
return self.Follow
@@ -1781,15 +1934,15 @@ class Grammar(object):
lastlri = p
i = 0
lr_items = []
- while 1:
+ while True:
if i > len(p):
lri = None
else:
- lri = LRItem(p,i)
+ lri = LRItem(p, i)
# Precompute the list of productions immediately following
try:
lri.lr_after = self.Prodnames[lri.prod[i+1]]
- except (IndexError,KeyError):
+ except (IndexError, KeyError):
lri.lr_after = []
try:
lri.lr_before = lri.prod[i-1]
@@ -1797,7 +1950,8 @@ class Grammar(object):
lri.lr_before = None
lastlri.lr_next = lri
- if not lri: break
+ if not lri:
+ break
lr_items.append(lri)
lastlri = lri
i += 1
@@ -1806,12 +1960,13 @@ class Grammar(object):
# -----------------------------------------------------------------------------
# == Class LRTable ==
#
-# This basic class represents a basic table of LR parsing information.
+# This basic class represents a basic table of LR parsing information.
# Methods for generating the tables are not defined here. They are defined
# in the derived class LRGeneratedTable.
# -----------------------------------------------------------------------------
-class VersionError(YaccError): pass
+class VersionError(YaccError):
+ pass
class LRTable(object):
def __init__(self):
@@ -1820,19 +1975,15 @@ class LRTable(object):
self.lr_productions = None
self.lr_method = None
- def read_table(self,module):
- if isinstance(module,types.ModuleType):
+ def read_table(self, module):
+ if isinstance(module, types.ModuleType):
parsetab = module
else:
- if sys.version_info[0] < 3:
- exec("import %s as parsetab" % module)
- else:
- env = { }
- exec("import %s as parsetab" % module, env, env)
- parsetab = env['parsetab']
+ exec('import %s' % module)
+ parsetab = sys.modules[module]
if parsetab._tabversion != __tabversion__:
- raise VersionError("yacc table file version is out of date")
+ raise VersionError('yacc table file version is out of date')
self.lr_action = parsetab._lr_action
self.lr_goto = parsetab._lr_goto
@@ -1844,17 +1995,20 @@ class LRTable(object):
self.lr_method = parsetab._lr_method
return parsetab._lr_signature
- def read_pickle(self,filename):
+ def read_pickle(self, filename):
try:
import cPickle as pickle
except ImportError:
import pickle
- in_f = open(filename,"rb")
+ if not os.path.exists(filename):
+ raise ImportError
+
+ in_f = open(filename, 'rb')
tabversion = pickle.load(in_f)
if tabversion != __tabversion__:
- raise VersionError("yacc table file version is out of date")
+ raise VersionError('yacc table file version is out of date')
self.lr_method = pickle.load(in_f)
signature = pickle.load(in_f)
self.lr_action = pickle.load(in_f)
@@ -1869,14 +2023,15 @@ class LRTable(object):
return signature
# Bind all production function names to callable objects in pdict
- def bind_callables(self,pdict):
+ def bind_callables(self, pdict):
for p in self.lr_productions:
p.bind(pdict)
-
+
+
# -----------------------------------------------------------------------------
# === LR Generator ===
#
-# The following classes and functions are used to generate LR parsing tables on
+# The following classes and functions are used to generate LR parsing tables on
# a grammar.
# -----------------------------------------------------------------------------
@@ -1897,17 +2052,18 @@ class LRTable(object):
# FP - Set-valued function
# ------------------------------------------------------------------------------
-def digraph(X,R,FP):
- N = { }
+def digraph(X, R, FP):
+ N = {}
for x in X:
- N[x] = 0
+ N[x] = 0
stack = []
- F = { }
+ F = {}
for x in X:
- if N[x] == 0: traverse(x,N,stack,F,X,R,FP)
+ if N[x] == 0:
+ traverse(x, N, stack, F, X, R, FP)
return F
-def traverse(x,N,stack,F,X,R,FP):
+def traverse(x, N, stack, F, X, R, FP):
stack.append(x)
d = len(stack)
N[x] = d
@@ -1916,20 +2072,22 @@ def traverse(x,N,stack,F,X,R,FP):
rel = R(x) # Get y's related to x
for y in rel:
if N[y] == 0:
- traverse(y,N,stack,F,X,R,FP)
- N[x] = min(N[x],N[y])
- for a in F.get(y,[]):
- if a not in F[x]: F[x].append(a)
+ traverse(y, N, stack, F, X, R, FP)
+ N[x] = min(N[x], N[y])
+ for a in F.get(y, []):
+ if a not in F[x]:
+ F[x].append(a)
if N[x] == d:
- N[stack[-1]] = MAXINT
- F[stack[-1]] = F[x]
- element = stack.pop()
- while element != x:
- N[stack[-1]] = MAXINT
- F[stack[-1]] = F[x]
- element = stack.pop()
+ N[stack[-1]] = MAXINT
+ F[stack[-1]] = F[x]
+ element = stack.pop()
+ while element != x:
+ N[stack[-1]] = MAXINT
+ F[stack[-1]] = F[x]
+ element = stack.pop()
-class LALRError(YaccError): pass
+class LALRError(YaccError):
+ pass
# -----------------------------------------------------------------------------
# == LRGeneratedTable ==
@@ -1939,9 +2097,9 @@ class LALRError(YaccError): pass
# -----------------------------------------------------------------------------
class LRGeneratedTable(LRTable):
- def __init__(self,grammar,method='LALR',log=None):
- if method not in ['SLR','LALR']:
- raise LALRError("Unsupported method %s" % method)
+ def __init__(self, grammar, method='LALR', log=None):
+ if method not in ['SLR', 'LALR']:
+ raise LALRError('Unsupported method %s' % method)
self.grammar = grammar
self.lr_method = method
@@ -1976,21 +2134,22 @@ class LRGeneratedTable(LRTable):
# Compute the LR(0) closure operation on I, where I is a set of LR(0) items.
- def lr0_closure(self,I):
+ def lr0_closure(self, I):
self._add_count += 1
# Add everything in I to J
J = I[:]
- didadd = 1
+ didadd = True
while didadd:
- didadd = 0
+ didadd = False
for j in J:
for x in j.lr_after:
- if getattr(x,"lr0_added",0) == self._add_count: continue
+ if getattr(x, 'lr0_added', 0) == self._add_count:
+ continue
# Add B --> .G to J
J.append(x.lr_next)
x.lr0_added = self._add_count
- didadd = 1
+ didadd = True
return J
@@ -2001,43 +2160,43 @@ class LRGeneratedTable(LRTable):
# objects). With uniqueness, we can later do fast set comparisons using
# id(obj) instead of element-wise comparison.
- def lr0_goto(self,I,x):
+ def lr0_goto(self, I, x):
# First we look for a previously cached entry
- g = self.lr_goto_cache.get((id(I),x),None)
- if g: return g
+ g = self.lr_goto_cache.get((id(I), x))
+ if g:
+ return g
# Now we generate the goto set in a way that guarantees uniqueness
# of the result
- s = self.lr_goto_cache.get(x,None)
+ s = self.lr_goto_cache.get(x)
if not s:
- s = { }
+ s = {}
self.lr_goto_cache[x] = s
- gs = [ ]
+ gs = []
for p in I:
n = p.lr_next
if n and n.lr_before == x:
- s1 = s.get(id(n),None)
+ s1 = s.get(id(n))
if not s1:
- s1 = { }
+ s1 = {}
s[id(n)] = s1
gs.append(n)
s = s1
- g = s.get('$end',None)
+ g = s.get('$end')
if not g:
if gs:
g = self.lr0_closure(gs)
s['$end'] = g
else:
s['$end'] = gs
- self.lr_goto_cache[(id(I),x)] = g
+ self.lr_goto_cache[(id(I), x)] = g
return g
# Compute the LR(0) sets of item function
def lr0_items(self):
-
- C = [ self.lr0_closure([self.grammar.Productions[0].lr_next]) ]
+ C = [self.lr0_closure([self.grammar.Productions[0].lr_next])]
i = 0
for I in C:
self.lr0_cidhash[id(I)] = i
@@ -2050,15 +2209,15 @@ class LRGeneratedTable(LRTable):
i += 1
# Collect all of the symbols that could possibly be in the goto(I,X) sets
- asyms = { }
+ asyms = {}
for ii in I:
for s in ii.usyms:
asyms[s] = None
for x in asyms:
- g = self.lr0_goto(I,x)
- if not g: continue
- if id(g) in self.lr0_cidhash: continue
+ g = self.lr0_goto(I, x)
+ if not g or id(g) in self.lr0_cidhash:
+ continue
self.lr0_cidhash[id(g)] = len(C)
C.append(g)
@@ -2093,19 +2252,21 @@ class LRGeneratedTable(LRTable):
# -----------------------------------------------------------------------------
def compute_nullable_nonterminals(self):
- nullable = {}
+ nullable = set()
num_nullable = 0
- while 1:
- for p in self.grammar.Productions[1:]:
- if p.len == 0:
- nullable[p.name] = 1
+ while True:
+ for p in self.grammar.Productions[1:]:
+ if p.len == 0:
+ nullable.add(p.name)
continue
- for t in p.prod:
- if not t in nullable: break
- else:
- nullable[p.name] = 1
- if len(nullable) == num_nullable: break
- num_nullable = len(nullable)
+ for t in p.prod:
+ if t not in nullable:
+ break
+ else:
+ nullable.add(p.name)
+ if len(nullable) == num_nullable:
+ break
+ num_nullable = len(nullable)
return nullable
# -----------------------------------------------------------------------------
@@ -2119,16 +2280,16 @@ class LRGeneratedTable(LRTable):
# The input C is the set of LR(0) items.
# -----------------------------------------------------------------------------
- def find_nonterminal_transitions(self,C):
- trans = []
- for state in range(len(C)):
- for p in C[state]:
- if p.lr_index < p.len - 1:
- t = (state,p.prod[p.lr_index+1])
- if t[1] in self.grammar.Nonterminals:
- if t not in trans: trans.append(t)
- state = state + 1
- return trans
+ def find_nonterminal_transitions(self, C):
+ trans = []
+ for stateno, state in enumerate(C):
+ for p in state:
+ if p.lr_index < p.len - 1:
+ t = (stateno, p.prod[p.lr_index+1])
+ if t[1] in self.grammar.Nonterminals:
+ if t not in trans:
+ trans.append(t)
+ return trans
# -----------------------------------------------------------------------------
# dr_relation()
@@ -2139,21 +2300,22 @@ class LRGeneratedTable(LRTable):
# Returns a list of terminals.
# -----------------------------------------------------------------------------
- def dr_relation(self,C,trans,nullable):
- dr_set = { }
- state,N = trans
+ def dr_relation(self, C, trans, nullable):
+ dr_set = {}
+ state, N = trans
terms = []
- g = self.lr0_goto(C[state],N)
+ g = self.lr0_goto(C[state], N)
for p in g:
- if p.lr_index < p.len - 1:
- a = p.prod[p.lr_index+1]
- if a in self.grammar.Terminals:
- if a not in terms: terms.append(a)
+ if p.lr_index < p.len - 1:
+ a = p.prod[p.lr_index+1]
+ if a in self.grammar.Terminals:
+ if a not in terms:
+ terms.append(a)
# This extra bit is to handle the start state
if state == 0 and N == self.grammar.Productions[0].prod[0]:
- terms.append('$end')
+ terms.append('$end')
return terms
@@ -2163,18 +2325,18 @@ class LRGeneratedTable(LRTable):
# Computes the READS() relation (p,A) READS (t,C).
# -----------------------------------------------------------------------------
- def reads_relation(self,C, trans, empty):
+ def reads_relation(self, C, trans, empty):
# Look for empty transitions
rel = []
state, N = trans
- g = self.lr0_goto(C[state],N)
- j = self.lr0_cidhash.get(id(g),-1)
+ g = self.lr0_goto(C[state], N)
+ j = self.lr0_cidhash.get(id(g), -1)
for p in g:
if p.lr_index < p.len - 1:
- a = p.prod[p.lr_index + 1]
- if a in empty:
- rel.append((j,a))
+ a = p.prod[p.lr_index + 1]
+ if a in empty:
+ rel.append((j, a))
return rel
@@ -2206,8 +2368,7 @@ class LRGeneratedTable(LRTable):
#
# -----------------------------------------------------------------------------
- def compute_lookback_includes(self,C,trans,nullable):
-
+ def compute_lookback_includes(self, C, trans, nullable):
lookdict = {} # Dictionary of lookback relations
includedict = {} # Dictionary of include relations
@@ -2217,11 +2378,12 @@ class LRGeneratedTable(LRTable):
dtrans[t] = 1
# Loop over all transitions and compute lookbacks and includes
- for state,N in trans:
+ for state, N in trans:
lookb = []
includes = []
for p in C[state]:
- if p.name != N: continue
+ if p.name != N:
+ continue
# Okay, we have a name match. We now follow the production all the way
# through the state machine until we get the . on the right hand side
@@ -2229,44 +2391,50 @@ class LRGeneratedTable(LRTable):
lr_index = p.lr_index
j = state
while lr_index < p.len - 1:
- lr_index = lr_index + 1
- t = p.prod[lr_index]
-
- # Check to see if this symbol and state are a non-terminal transition
- if (j,t) in dtrans:
- # Yes. Okay, there is some chance that this is an includes relation
- # the only way to know for certain is whether the rest of the
- # production derives empty
-
- li = lr_index + 1
- while li < p.len:
- if p.prod[li] in self.grammar.Terminals: break # No forget it
- if not p.prod[li] in nullable: break
- li = li + 1
- else:
- # Appears to be a relation between (j,t) and (state,N)
- includes.append((j,t))
-
- g = self.lr0_goto(C[j],t) # Go to next set
- j = self.lr0_cidhash.get(id(g),-1) # Go to next state
+ lr_index = lr_index + 1
+ t = p.prod[lr_index]
+
+ # Check to see if this symbol and state are a non-terminal transition
+ if (j, t) in dtrans:
+ # Yes. Okay, there is some chance that this is an includes relation
+ # the only way to know for certain is whether the rest of the
+ # production derives empty
+
+ li = lr_index + 1
+ while li < p.len:
+ if p.prod[li] in self.grammar.Terminals:
+ break # No forget it
+ if p.prod[li] not in nullable:
+ break
+ li = li + 1
+ else:
+ # Appears to be a relation between (j,t) and (state,N)
+ includes.append((j, t))
+
+ g = self.lr0_goto(C[j], t) # Go to next set
+ j = self.lr0_cidhash.get(id(g), -1) # Go to next state
# When we get here, j is the final state, now we have to locate the production
for r in C[j]:
- if r.name != p.name: continue
- if r.len != p.len: continue
- i = 0
- # This look is comparing a production ". A B C" with "A B C ."
- while i < r.lr_index:
- if r.prod[i] != p.prod[i+1]: break
- i = i + 1
- else:
- lookb.append((j,r))
+ if r.name != p.name:
+ continue
+ if r.len != p.len:
+ continue
+ i = 0
+ # This look is comparing a production ". A B C" with "A B C ."
+ while i < r.lr_index:
+ if r.prod[i] != p.prod[i+1]:
+ break
+ i = i + 1
+ else:
+ lookb.append((j, r))
for i in includes:
- if not i in includedict: includedict[i] = []
- includedict[i].append((state,N))
- lookdict[(state,N)] = lookb
+ if i not in includedict:
+ includedict[i] = []
+ includedict[i].append((state, N))
+ lookdict[(state, N)] = lookb
- return lookdict,includedict
+ return lookdict, includedict
# -----------------------------------------------------------------------------
# compute_read_sets()
@@ -2280,10 +2448,10 @@ class LRGeneratedTable(LRTable):
# Returns a set containing the read sets
# -----------------------------------------------------------------------------
- def compute_read_sets(self,C, ntrans, nullable):
- FP = lambda x: self.dr_relation(C,x,nullable)
- R = lambda x: self.reads_relation(C,x,nullable)
- F = digraph(ntrans,R,FP)
+ def compute_read_sets(self, C, ntrans, nullable):
+ FP = lambda x: self.dr_relation(C, x, nullable)
+ R = lambda x: self.reads_relation(C, x, nullable)
+ F = digraph(ntrans, R, FP)
return F
# -----------------------------------------------------------------------------
@@ -2302,11 +2470,11 @@ class LRGeneratedTable(LRTable):
# Returns a set containing the follow sets
# -----------------------------------------------------------------------------
- def compute_follow_sets(self,ntrans,readsets,inclsets):
- FP = lambda x: readsets[x]
- R = lambda x: inclsets.get(x,[])
- F = digraph(ntrans,R,FP)
- return F
+ def compute_follow_sets(self, ntrans, readsets, inclsets):
+ FP = lambda x: readsets[x]
+ R = lambda x: inclsets.get(x, [])
+ F = digraph(ntrans, R, FP)
+ return F
# -----------------------------------------------------------------------------
# add_lookaheads()
@@ -2320,15 +2488,16 @@ class LRGeneratedTable(LRTable):
# in the lookbacks set
# -----------------------------------------------------------------------------
- def add_lookaheads(self,lookbacks,followset):
- for trans,lb in lookbacks.items():
+ def add_lookaheads(self, lookbacks, followset):
+ for trans, lb in lookbacks.items():
# Loop over productions in lookback
- for state,p in lb:
- if not state in p.lookaheads:
- p.lookaheads[state] = []
- f = followset.get(trans,[])
- for a in f:
- if a not in p.lookaheads[state]: p.lookaheads[state].append(a)
+ for state, p in lb:
+ if state not in p.lookaheads:
+ p.lookaheads[state] = []
+ f = followset.get(trans, [])
+ for a in f:
+ if a not in p.lookaheads[state]:
+ p.lookaheads[state].append(a)
# -----------------------------------------------------------------------------
# add_lalr_lookaheads()
@@ -2337,7 +2506,7 @@ class LRGeneratedTable(LRTable):
# with LALR parsing
# -----------------------------------------------------------------------------
- def add_lalr_lookaheads(self,C):
+ def add_lalr_lookaheads(self, C):
# Determine all of the nullable nonterminals
nullable = self.compute_nullable_nonterminals()
@@ -2345,16 +2514,16 @@ class LRGeneratedTable(LRTable):
trans = self.find_nonterminal_transitions(C)
# Compute read sets
- readsets = self.compute_read_sets(C,trans,nullable)
+ readsets = self.compute_read_sets(C, trans, nullable)
# Compute lookback/includes relations
- lookd, included = self.compute_lookback_includes(C,trans,nullable)
+ lookd, included = self.compute_lookback_includes(C, trans, nullable)
# Compute LALR FOLLOW sets
- followsets = self.compute_follow_sets(trans,readsets,included)
+ followsets = self.compute_follow_sets(trans, readsets, included)
# Add all of the lookaheads
- self.add_lookaheads(lookd,followsets)
+ self.add_lookaheads(lookd, followsets)
# -----------------------------------------------------------------------------
# lr_parse_table()
@@ -2368,9 +2537,9 @@ class LRGeneratedTable(LRTable):
action = self.lr_action # Action array
log = self.log # Logger for output
- actionp = { } # Action production array (temporary)
-
- log.info("Parsing method: %s", self.lr_method)
+ actionp = {} # Action production array (temporary)
+
+ log.info('Parsing method: %s', self.lr_method)
# Step 1: Construct C = { I0, I1, ... IN}, collection of LR(0) items
# This determines the number of states
@@ -2384,23 +2553,23 @@ class LRGeneratedTable(LRTable):
st = 0
for I in C:
# Loop over each production in I
- actlist = [ ] # List of actions
- st_action = { }
- st_actionp = { }
- st_goto = { }
- log.info("")
- log.info("state %d", st)
- log.info("")
+ actlist = [] # List of actions
+ st_action = {}
+ st_actionp = {}
+ st_goto = {}
+ log.info('')
+ log.info('state %d', st)
+ log.info('')
for p in I:
- log.info(" (%d) %s", p.number, str(p))
- log.info("")
+ log.info(' (%d) %s', p.number, p)
+ log.info('')
for p in I:
if p.len == p.lr_index + 1:
if p.name == "S'":
# Start symbol. Accept!
- st_action["$end"] = 0
- st_actionp["$end"] = p
+ st_action['$end'] = 0
+ st_actionp['$end'] = p
else:
# We are at the end of a production. Reduce!
if self.lr_method == 'LALR':
@@ -2408,31 +2577,36 @@ class LRGeneratedTable(LRTable):
else:
laheads = self.grammar.Follow[p.name]
for a in laheads:
- actlist.append((a,p,"reduce using rule %d (%s)" % (p.number,p)))
- r = st_action.get(a,None)
+ actlist.append((a, p, 'reduce using rule %d (%s)' % (p.number, p)))
+ r = st_action.get(a)
if r is not None:
# Whoa. Have a shift/reduce or reduce/reduce conflict
if r > 0:
# Need to decide on shift or reduce here
# By default we favor shifting. Need to add
# some precedence rules here.
- sprec,slevel = Productions[st_actionp[a].number].prec
- rprec,rlevel = Precedence.get(a,('right',0))
+
+ # Shift precedence comes from the token
+ sprec, slevel = Precedence.get(a, ('right', 0))
+
+ # Reduce precedence comes from rule being reduced (p)
+ rprec, rlevel = Productions[p.number].prec
+
if (slevel < rlevel) or ((slevel == rlevel) and (rprec == 'left')):
# We really need to reduce here.
st_action[a] = -p.number
st_actionp[a] = p
if not slevel and not rlevel:
- log.info(" ! shift/reduce conflict for %s resolved as reduce",a)
- self.sr_conflicts.append((st,a,'reduce'))
+ log.info(' ! shift/reduce conflict for %s resolved as reduce', a)
+ self.sr_conflicts.append((st, a, 'reduce'))
Productions[p.number].reduced += 1
elif (slevel == rlevel) and (rprec == 'nonassoc'):
st_action[a] = None
else:
# Hmmm. Guess we'll keep the shift
if not rlevel:
- log.info(" ! shift/reduce conflict for %s resolved as shift",a)
- self.sr_conflicts.append((st,a,'shift'))
+ log.info(' ! shift/reduce conflict for %s resolved as shift', a)
+ self.sr_conflicts.append((st, a, 'shift'))
elif r < 0:
# Reduce/reduce conflict. In this case, we favor the rule
# that was defined first in the grammar file
@@ -2441,15 +2615,16 @@ class LRGeneratedTable(LRTable):
if oldp.line > pp.line:
st_action[a] = -p.number
st_actionp[a] = p
- chosenp,rejectp = pp,oldp
+ chosenp, rejectp = pp, oldp
Productions[p.number].reduced += 1
Productions[oldp.number].reduced -= 1
else:
- chosenp,rejectp = oldp,pp
- self.rr_conflicts.append((st,chosenp,rejectp))
- log.info(" ! reduce/reduce conflict for %s resolved using rule %d (%s)", a,st_actionp[a].number, st_actionp[a])
+ chosenp, rejectp = oldp, pp
+ self.rr_conflicts.append((st, chosenp, rejectp))
+ log.info(' ! reduce/reduce conflict for %s resolved using rule %d (%s)',
+ a, st_actionp[a].number, st_actionp[a])
else:
- raise LALRError("Unknown conflict in state %d" % st)
+ raise LALRError('Unknown conflict in state %d' % st)
else:
st_action[a] = -p.number
st_actionp[a] = p
@@ -2458,99 +2633,106 @@ class LRGeneratedTable(LRTable):
i = p.lr_index
a = p.prod[i+1] # Get symbol right after the "."
if a in self.grammar.Terminals:
- g = self.lr0_goto(I,a)
- j = self.lr0_cidhash.get(id(g),-1)
+ g = self.lr0_goto(I, a)
+ j = self.lr0_cidhash.get(id(g), -1)
if j >= 0:
# We are in a shift state
- actlist.append((a,p,"shift and go to state %d" % j))
- r = st_action.get(a,None)
+ actlist.append((a, p, 'shift and go to state %d' % j))
+ r = st_action.get(a)
if r is not None:
# Whoa have a shift/reduce or shift/shift conflict
if r > 0:
if r != j:
- raise LALRError("Shift/shift conflict in state %d" % st)
+ raise LALRError('Shift/shift conflict in state %d' % st)
elif r < 0:
# Do a precedence check.
# - if precedence of reduce rule is higher, we reduce.
# - if precedence of reduce is same and left assoc, we reduce.
# - otherwise we shift
- rprec,rlevel = Productions[st_actionp[a].number].prec
- sprec,slevel = Precedence.get(a,('right',0))
+
+ # Shift precedence comes from the token
+ sprec, slevel = Precedence.get(a, ('right', 0))
+
+ # Reduce precedence comes from the rule that could have been reduced
+ rprec, rlevel = Productions[st_actionp[a].number].prec
+
if (slevel > rlevel) or ((slevel == rlevel) and (rprec == 'right')):
# We decide to shift here... highest precedence to shift
Productions[st_actionp[a].number].reduced -= 1
st_action[a] = j
st_actionp[a] = p
if not rlevel:
- log.info(" ! shift/reduce conflict for %s resolved as shift",a)
- self.sr_conflicts.append((st,a,'shift'))
+ log.info(' ! shift/reduce conflict for %s resolved as shift', a)
+ self.sr_conflicts.append((st, a, 'shift'))
elif (slevel == rlevel) and (rprec == 'nonassoc'):
st_action[a] = None
else:
# Hmmm. Guess we'll keep the reduce
if not slevel and not rlevel:
- log.info(" ! shift/reduce conflict for %s resolved as reduce",a)
- self.sr_conflicts.append((st,a,'reduce'))
+ log.info(' ! shift/reduce conflict for %s resolved as reduce', a)
+ self.sr_conflicts.append((st, a, 'reduce'))
else:
- raise LALRError("Unknown conflict in state %d" % st)
+ raise LALRError('Unknown conflict in state %d' % st)
else:
st_action[a] = j
st_actionp[a] = p
# Print the actions associated with each terminal
- _actprint = { }
- for a,p,m in actlist:
+ _actprint = {}
+ for a, p, m in actlist:
if a in st_action:
if p is st_actionp[a]:
- log.info(" %-15s %s",a,m)
- _actprint[(a,m)] = 1
- log.info("")
+ log.info(' %-15s %s', a, m)
+ _actprint[(a, m)] = 1
+ log.info('')
# Print the actions that were not used. (debugging)
not_used = 0
- for a,p,m in actlist:
+ for a, p, m in actlist:
if a in st_action:
if p is not st_actionp[a]:
- if not (a,m) in _actprint:
- log.debug(" ! %-15s [ %s ]",a,m)
+ if not (a, m) in _actprint:
+ log.debug(' ! %-15s [ %s ]', a, m)
not_used = 1
- _actprint[(a,m)] = 1
+ _actprint[(a, m)] = 1
if not_used:
- log.debug("")
+ log.debug('')
# Construct the goto table for this state
- nkeys = { }
+ nkeys = {}
for ii in I:
for s in ii.usyms:
if s in self.grammar.Nonterminals:
nkeys[s] = None
for n in nkeys:
- g = self.lr0_goto(I,n)
- j = self.lr0_cidhash.get(id(g),-1)
+ g = self.lr0_goto(I, n)
+ j = self.lr0_cidhash.get(id(g), -1)
if j >= 0:
st_goto[n] = j
- log.info(" %-30s shift and go to state %d",n,j)
+ log.info(' %-30s shift and go to state %d', n, j)
action[st] = st_action
actionp[st] = st_actionp
goto[st] = st_goto
st += 1
-
# -----------------------------------------------------------------------------
# write()
#
# This function writes the LR parsing tables to a file
# -----------------------------------------------------------------------------
- def write_table(self,modulename,outputdir='',signature=""):
- basemodulename = modulename.split(".")[-1]
- filename = os.path.join(outputdir,basemodulename) + ".py"
+ def write_table(self, tabmodule, outputdir='', signature=''):
+ if isinstance(tabmodule, types.ModuleType):
+ raise IOError("Won't overwrite existing tabmodule")
+
+ basemodulename = tabmodule.split('.')[-1]
+ filename = os.path.join(outputdir, basemodulename) + '.py'
try:
- f = open(filename,"w")
+ f = open(filename, 'w')
- f.write("""
+ f.write('''
# %s
# This file is automatically generated. Do not edit.
_tabversion = %r
@@ -2558,105 +2740,103 @@ _tabversion = %r
_lr_method = %r
_lr_signature = %r
- """ % (filename, __tabversion__, self.lr_method, signature))
+ ''' % (os.path.basename(filename), __tabversion__, self.lr_method, signature))
# Change smaller to 0 to go back to original tables
smaller = 1
# Factor out names to try and make smaller
if smaller:
- items = { }
-
- for s,nd in self.lr_action.items():
- for name,v in nd.items():
- i = items.get(name)
- if not i:
- i = ([],[])
- items[name] = i
- i[0].append(s)
- i[1].append(v)
-
- f.write("\n_lr_action_items = {")
- for k,v in items.items():
- f.write("%r:([" % k)
+ items = {}
+
+ for s, nd in self.lr_action.items():
+ for name, v in nd.items():
+ i = items.get(name)
+ if not i:
+ i = ([], [])
+ items[name] = i
+ i[0].append(s)
+ i[1].append(v)
+
+ f.write('\n_lr_action_items = {')
+ for k, v in items.items():
+ f.write('%r:([' % k)
for i in v[0]:
- f.write("%r," % i)
- f.write("],[")
+ f.write('%r,' % i)
+ f.write('],[')
for i in v[1]:
- f.write("%r," % i)
+ f.write('%r,' % i)
- f.write("]),")
- f.write("}\n")
+ f.write(']),')
+ f.write('}\n')
- f.write("""
-_lr_action = { }
+ f.write('''
+_lr_action = {}
for _k, _v in _lr_action_items.items():
for _x,_y in zip(_v[0],_v[1]):
- if not _x in _lr_action: _lr_action[_x] = { }
+ if not _x in _lr_action: _lr_action[_x] = {}
_lr_action[_x][_k] = _y
del _lr_action_items
-""")
+''')
else:
- f.write("\n_lr_action = { ");
- for k,v in self.lr_action.items():
- f.write("(%r,%r):%r," % (k[0],k[1],v))
- f.write("}\n");
+ f.write('\n_lr_action = { ')
+ for k, v in self.lr_action.items():
+ f.write('(%r,%r):%r,' % (k[0], k[1], v))
+ f.write('}\n')
if smaller:
# Factor out names to try and make smaller
- items = { }
-
- for s,nd in self.lr_goto.items():
- for name,v in nd.items():
- i = items.get(name)
- if not i:
- i = ([],[])
- items[name] = i
- i[0].append(s)
- i[1].append(v)
-
- f.write("\n_lr_goto_items = {")
- for k,v in items.items():
- f.write("%r:([" % k)
+ items = {}
+
+ for s, nd in self.lr_goto.items():
+ for name, v in nd.items():
+ i = items.get(name)
+ if not i:
+ i = ([], [])
+ items[name] = i
+ i[0].append(s)
+ i[1].append(v)
+
+ f.write('\n_lr_goto_items = {')
+ for k, v in items.items():
+ f.write('%r:([' % k)
for i in v[0]:
- f.write("%r," % i)
- f.write("],[")
+ f.write('%r,' % i)
+ f.write('],[')
for i in v[1]:
- f.write("%r," % i)
+ f.write('%r,' % i)
- f.write("]),")
- f.write("}\n")
+ f.write(']),')
+ f.write('}\n')
- f.write("""
-_lr_goto = { }
+ f.write('''
+_lr_goto = {}
for _k, _v in _lr_goto_items.items():
- for _x,_y in zip(_v[0],_v[1]):
- if not _x in _lr_goto: _lr_goto[_x] = { }
+ for _x, _y in zip(_v[0], _v[1]):
+ if not _x in _lr_goto: _lr_goto[_x] = {}
_lr_goto[_x][_k] = _y
del _lr_goto_items
-""")
+''')
else:
- f.write("\n_lr_goto = { ");
- for k,v in self.lr_goto.items():
- f.write("(%r,%r):%r," % (k[0],k[1],v))
- f.write("}\n");
+ f.write('\n_lr_goto = { ')
+ for k, v in self.lr_goto.items():
+ f.write('(%r,%r):%r,' % (k[0], k[1], v))
+ f.write('}\n')
# Write production table
- f.write("_lr_productions = [\n")
+ f.write('_lr_productions = [\n')
for p in self.lr_productions:
if p.func:
- f.write(" (%r,%r,%d,%r,%r,%d),\n" % (p.str,p.name, p.len, p.func,p.file,p.line))
+ f.write(' (%r,%r,%d,%r,%r,%d),\n' % (p.str, p.name, p.len,
+ p.func, os.path.basename(p.file), p.line))
else:
- f.write(" (%r,%r,%d,None,None,None),\n" % (str(p),p.name, p.len))
- f.write("]\n")
+ f.write(' (%r,%r,%d,None,None,None),\n' % (str(p), p.name, p.len))
+ f.write(']\n')
f.close()
- except IOError:
- e = sys.exc_info()[1]
- sys.stderr.write("Unable to create '%s'\n" % filename)
- sys.stderr.write(str(e)+"\n")
- return
+ except IOError as e:
+ raise
# -----------------------------------------------------------------------------
@@ -2665,26 +2845,25 @@ del _lr_goto_items
# This function pickles the LR parsing tables to a supplied file object
# -----------------------------------------------------------------------------
- def pickle_table(self,filename,signature=""):
+ def pickle_table(self, filename, signature=''):
try:
import cPickle as pickle
except ImportError:
import pickle
- outf = open(filename,"wb")
- pickle.dump(__tabversion__,outf,pickle_protocol)
- pickle.dump(self.lr_method,outf,pickle_protocol)
- pickle.dump(signature,outf,pickle_protocol)
- pickle.dump(self.lr_action,outf,pickle_protocol)
- pickle.dump(self.lr_goto,outf,pickle_protocol)
-
- outp = []
- for p in self.lr_productions:
- if p.func:
- outp.append((p.str,p.name, p.len, p.func,p.file,p.line))
- else:
- outp.append((str(p),p.name,p.len,None,None,None))
- pickle.dump(outp,outf,pickle_protocol)
- outf.close()
+ with open(filename, 'wb') as outf:
+ pickle.dump(__tabversion__, outf, pickle_protocol)
+ pickle.dump(self.lr_method, outf, pickle_protocol)
+ pickle.dump(signature, outf, pickle_protocol)
+ pickle.dump(self.lr_action, outf, pickle_protocol)
+ pickle.dump(self.lr_goto, outf, pickle_protocol)
+
+ outp = []
+ for p in self.lr_productions:
+ if p.func:
+ outp.append((p.str, p.name, p.len, p.func, os.path.basename(p.file), p.line))
+ else:
+ outp.append((str(p), p.name, p.len, None, None, None))
+ pickle.dump(outp, outf, pickle_protocol)
# -----------------------------------------------------------------------------
# === INTROSPECTION ===
@@ -2702,26 +2881,18 @@ del _lr_goto_items
# -----------------------------------------------------------------------------
def get_caller_module_dict(levels):
- try:
- raise RuntimeError
- except RuntimeError:
- e,b,t = sys.exc_info()
- f = t.tb_frame
- while levels > 0:
- f = f.f_back
- levels -= 1
- ldict = f.f_globals.copy()
- if f.f_globals != f.f_locals:
- ldict.update(f.f_locals)
-
- return ldict
+ f = sys._getframe(levels)
+ ldict = f.f_globals.copy()
+ if f.f_globals != f.f_locals:
+ ldict.update(f.f_locals)
+ return ldict
# -----------------------------------------------------------------------------
# parse_grammar()
#
# This takes a raw grammar rule string and parses it into production data
# -----------------------------------------------------------------------------
-def parse_grammar(doc,file,line):
+def parse_grammar(doc, file, line):
grammar = []
# Split the doc string into lines
pstrings = doc.splitlines()
@@ -2730,12 +2901,13 @@ def parse_grammar(doc,file,line):
for ps in pstrings:
dline += 1
p = ps.split()
- if not p: continue
+ if not p:
+ continue
try:
if p[0] == '|':
# This is a continuation of a previous rule
if not lastp:
- raise SyntaxError("%s:%d: Misplaced '|'" % (file,dline))
+ raise SyntaxError("%s:%d: Misplaced '|'" % (file, dline))
prodname = lastp
syms = p[1:]
else:
@@ -2744,13 +2916,13 @@ def parse_grammar(doc,file,line):
syms = p[2:]
assign = p[1]
if assign != ':' and assign != '::=':
- raise SyntaxError("%s:%d: Syntax error. Expected ':'" % (file,dline))
+ raise SyntaxError("%s:%d: Syntax error. Expected ':'" % (file, dline))
- grammar.append((file,dline,prodname,syms))
+ grammar.append((file, dline, prodname, syms))
except SyntaxError:
raise
except Exception:
- raise SyntaxError("%s:%d: Syntax error in rule '%s'" % (file,dline,ps.strip()))
+ raise SyntaxError('%s:%d: Syntax error in rule %r' % (file, dline, ps.strip()))
return grammar
@@ -2762,14 +2934,14 @@ def parse_grammar(doc,file,line):
# etc.
# -----------------------------------------------------------------------------
class ParserReflect(object):
- def __init__(self,pdict,log=None):
+ def __init__(self, pdict, log=None):
self.pdict = pdict
self.start = None
self.error_func = None
self.tokens = None
- self.files = {}
+ self.modules = set()
self.grammar = []
- self.error = 0
+ self.error = False
if log is None:
self.log = PlyLogger(sys.stderr)
@@ -2783,7 +2955,7 @@ class ParserReflect(object):
self.get_tokens()
self.get_precedence()
self.get_pfunctions()
-
+
# Validate all of the information
def validate_all(self):
self.validate_start()
@@ -2791,36 +2963,28 @@ class ParserReflect(object):
self.validate_tokens()
self.validate_precedence()
self.validate_pfunctions()
- self.validate_files()
+ self.validate_modules()
return self.error
# Compute a signature over the grammar
def signature(self):
- try:
- import hashlib
- except ImportError:
- raise RuntimeError("Unable to import hashlib")
- try:
- sig = hashlib.new('MD5', usedforsecurity=False)
- except TypeError:
- # Some configurations don't appear to support two arguments
- sig = hashlib.new('MD5')
+ parts = []
try:
if self.start:
- sig.update(self.start.encode('latin-1'))
+ parts.append(self.start)
if self.prec:
- sig.update("".join(["".join(p) for p in self.prec]).encode('latin-1'))
+ parts.append(''.join([''.join(p) for p in self.prec]))
if self.tokens:
- sig.update(" ".join(self.tokens).encode('latin-1'))
+ parts.append(' '.join(self.tokens))
for f in self.pfuncs:
if f[3]:
- sig.update(f[3].encode('latin-1'))
- except (TypeError,ValueError):
+ parts.append(f[3])
+ except (TypeError, ValueError):
pass
- return sig.digest()
+ return ''.join(parts)
# -----------------------------------------------------------------------------
- # validate_file()
+ # validate_modules()
#
# This method checks to see if there are duplicated p_rulename() functions
# in the parser module file. Without this function, it is really easy for
@@ -2830,32 +2994,29 @@ class ParserReflect(object):
# to try and detect duplicates.
# -----------------------------------------------------------------------------
- def validate_files(self):
+ def validate_modules(self):
# Match def p_funcname(
fre = re.compile(r'\s*def\s+(p_[a-zA-Z_0-9]*)\(')
- for filename in self.files.keys():
- base,ext = os.path.splitext(filename)
- if ext != '.py': return 1 # No idea. Assume it's okay.
-
+ for module in self.modules:
try:
- f = open(filename)
- lines = f.readlines()
- f.close()
+ lines, linen = inspect.getsourcelines(module)
except IOError:
continue
- counthash = { }
- for linen,l in enumerate(lines):
+ counthash = {}
+ for linen, line in enumerate(lines):
linen += 1
- m = fre.match(l)
+ m = fre.match(line)
if m:
name = m.group(1)
prev = counthash.get(name)
if not prev:
counthash[name] = linen
else:
- self.log.warning("%s:%d: Function %s redefined. Previously defined on line %d", filename,linen,name,prev)
+ filename = inspect.getsourcefile(module)
+ self.log.warning('%s:%d: Function %s redefined. Previously defined on line %d',
+ filename, linen, name, prev)
# Get the start symbol
def get_start(self):
@@ -2864,7 +3025,7 @@ class ParserReflect(object):
# Validate the start symbol
def validate_start(self):
if self.start is not None:
- if not isinstance(self.start,str):
+ if not isinstance(self.start, string_types):
self.log.error("'start' must be a string")
# Look for error handler
@@ -2874,39 +3035,41 @@ class ParserReflect(object):
# Validate the error function
def validate_error_func(self):
if self.error_func:
- if isinstance(self.error_func,types.FunctionType):
+ if isinstance(self.error_func, types.FunctionType):
ismethod = 0
elif isinstance(self.error_func, types.MethodType):
ismethod = 1
else:
self.log.error("'p_error' defined, but is not a function or method")
- self.error = 1
+ self.error = True
return
- eline = func_code(self.error_func).co_firstlineno
- efile = func_code(self.error_func).co_filename
- self.files[efile] = 1
+ eline = self.error_func.__code__.co_firstlineno
+ efile = self.error_func.__code__.co_filename
+ module = inspect.getmodule(self.error_func)
+ self.modules.add(module)
- if (func_code(self.error_func).co_argcount != 1+ismethod):
- self.log.error("%s:%d: p_error() requires 1 argument",efile,eline)
- self.error = 1
+ argcount = self.error_func.__code__.co_argcount - ismethod
+ if argcount != 1:
+ self.log.error('%s:%d: p_error() requires 1 argument', efile, eline)
+ self.error = True
# Get the tokens map
def get_tokens(self):
- tokens = self.pdict.get("tokens",None)
+ tokens = self.pdict.get('tokens')
if not tokens:
- self.log.error("No token list is defined")
- self.error = 1
+ self.log.error('No token list is defined')
+ self.error = True
return
- if not isinstance(tokens,(list, tuple)):
- self.log.error("tokens must be a list or tuple")
- self.error = 1
+ if not isinstance(tokens, (list, tuple)):
+ self.log.error('tokens must be a list or tuple')
+ self.error = True
return
-
+
if not tokens:
- self.log.error("tokens is empty")
- self.error = 1
+ self.log.error('tokens is empty')
+ self.error = True
return
self.tokens = tokens
@@ -2916,120 +3079,129 @@ class ParserReflect(object):
# Validate the tokens.
if 'error' in self.tokens:
self.log.error("Illegal token name 'error'. Is a reserved word")
- self.error = 1
+ self.error = True
return
- terminals = {}
+ terminals = set()
for n in self.tokens:
if n in terminals:
- self.log.warning("Token '%s' multiply defined", n)
- terminals[n] = 1
+ self.log.warning('Token %r multiply defined', n)
+ terminals.add(n)
# Get the precedence map (if any)
def get_precedence(self):
- self.prec = self.pdict.get("precedence",None)
+ self.prec = self.pdict.get('precedence')
# Validate and parse the precedence map
def validate_precedence(self):
preclist = []
if self.prec:
- if not isinstance(self.prec,(list,tuple)):
- self.log.error("precedence must be a list or tuple")
- self.error = 1
+ if not isinstance(self.prec, (list, tuple)):
+ self.log.error('precedence must be a list or tuple')
+ self.error = True
return
- for level,p in enumerate(self.prec):
- if not isinstance(p,(list,tuple)):
- self.log.error("Bad precedence table")
- self.error = 1
+ for level, p in enumerate(self.prec):
+ if not isinstance(p, (list, tuple)):
+ self.log.error('Bad precedence table')
+ self.error = True
return
if len(p) < 2:
- self.log.error("Malformed precedence entry %s. Must be (assoc, term, ..., term)",p)
- self.error = 1
+ self.log.error('Malformed precedence entry %s. Must be (assoc, term, ..., term)', p)
+ self.error = True
return
assoc = p[0]
- if not isinstance(assoc,str):
- self.log.error("precedence associativity must be a string")
- self.error = 1
+ if not isinstance(assoc, string_types):
+ self.log.error('precedence associativity must be a string')
+ self.error = True
return
for term in p[1:]:
- if not isinstance(term,str):
- self.log.error("precedence items must be strings")
- self.error = 1
+ if not isinstance(term, string_types):
+ self.log.error('precedence items must be strings')
+ self.error = True
return
- preclist.append((term,assoc,level+1))
+ preclist.append((term, assoc, level+1))
self.preclist = preclist
# Get all p_functions from the grammar
def get_pfunctions(self):
p_functions = []
for name, item in self.pdict.items():
- if name[:2] != 'p_': continue
- if name == 'p_error': continue
- if isinstance(item,(types.FunctionType,types.MethodType)):
- line = func_code(item).co_firstlineno
- file = func_code(item).co_filename
- p_functions.append((line,file,name,item.__doc__))
-
- # Sort all of the actions by line number
- p_functions.sort()
+ if not name.startswith('p_') or name == 'p_error':
+ continue
+ if isinstance(item, (types.FunctionType, types.MethodType)):
+ line = getattr(item, 'co_firstlineno', item.__code__.co_firstlineno)
+ module = inspect.getmodule(item)
+ p_functions.append((line, module, name, item.__doc__))
+
+ # Sort all of the actions by line number; make sure to stringify
+ # modules to make them sortable, since `line` may not uniquely sort all
+ # p functions
+ p_functions.sort(key=lambda p_function: (
+ p_function[0],
+ str(p_function[1]),
+ p_function[2],
+ p_function[3]))
self.pfuncs = p_functions
-
# Validate all of the p_functions
def validate_pfunctions(self):
grammar = []
# Check for non-empty symbols
if len(self.pfuncs) == 0:
- self.log.error("no rules of the form p_rulename are defined")
- self.error = 1
- return
-
- for line, file, name, doc in self.pfuncs:
+ self.log.error('no rules of the form p_rulename are defined')
+ self.error = True
+ return
+
+ for line, module, name, doc in self.pfuncs:
+ file = inspect.getsourcefile(module)
func = self.pdict[name]
if isinstance(func, types.MethodType):
reqargs = 2
else:
reqargs = 1
- if func_code(func).co_argcount > reqargs:
- self.log.error("%s:%d: Rule '%s' has too many arguments",file,line,func.__name__)
- self.error = 1
- elif func_code(func).co_argcount < reqargs:
- self.log.error("%s:%d: Rule '%s' requires an argument",file,line,func.__name__)
- self.error = 1
+ if func.__code__.co_argcount > reqargs:
+ self.log.error('%s:%d: Rule %r has too many arguments', file, line, func.__name__)
+ self.error = True
+ elif func.__code__.co_argcount < reqargs:
+ self.log.error('%s:%d: Rule %r requires an argument', file, line, func.__name__)
+ self.error = True
elif not func.__doc__:
- self.log.warning("%s:%d: No documentation string specified in function '%s' (ignored)",file,line,func.__name__)
+ self.log.warning('%s:%d: No documentation string specified in function %r (ignored)',
+ file, line, func.__name__)
else:
try:
- parsed_g = parse_grammar(doc,file,line)
+ parsed_g = parse_grammar(doc, file, line)
for g in parsed_g:
grammar.append((name, g))
- except SyntaxError:
- e = sys.exc_info()[1]
+ except SyntaxError as e:
self.log.error(str(e))
- self.error = 1
+ self.error = True
# Looks like a valid grammar rule
# Mark the file in which defined.
- self.files[file] = 1
+ self.modules.add(module)
# Secondary validation step that looks for p_ definitions that are not functions
# or functions that look like they might be grammar rules.
- for n,v in self.pdict.items():
- if n[0:2] == 'p_' and isinstance(v, (types.FunctionType, types.MethodType)): continue
- if n[0:2] == 't_': continue
- if n[0:2] == 'p_' and n != 'p_error':
- self.log.warning("'%s' not defined as a function", n)
- if ((isinstance(v,types.FunctionType) and func_code(v).co_argcount == 1) or
- (isinstance(v,types.MethodType) and func_code(v).co_argcount == 2)):
- try:
- doc = v.__doc__.split(" ")
- if doc[1] == ':':
- self.log.warning("%s:%d: Possible grammar rule '%s' defined without p_ prefix",
- func_code(v).co_filename, func_code(v).co_firstlineno,n)
- except Exception:
- pass
+ for n, v in self.pdict.items():
+ if n.startswith('p_') and isinstance(v, (types.FunctionType, types.MethodType)):
+ continue
+ if n.startswith('t_'):
+ continue
+ if n.startswith('p_') and n != 'p_error':
+ self.log.warning('%r not defined as a function', n)
+ if ((isinstance(v, types.FunctionType) and v.__code__.co_argcount == 1) or
+ (isinstance(v, types.MethodType) and v.__func__.__code__.co_argcount == 2)):
+ if v.__doc__:
+ try:
+ doc = v.__doc__.split(' ')
+ if doc[1] == ':':
+ self.log.warning('%s:%d: Possible grammar rule %r defined without p_ prefix',
+ v.__code__.co_filename, v.__code__.co_firstlineno, n)
+ except IndexError:
+ pass
self.grammar = grammar
@@ -3039,14 +3211,17 @@ class ParserReflect(object):
# Build a parser
# -----------------------------------------------------------------------------
-def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, start=None,
- check_recursion=1, optimize=0, write_tables=1, debugfile=debug_file,outputdir='',
- debuglog=None, errorlog = None, picklefile=None):
+def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, start=None,
+ check_recursion=True, optimize=False, write_tables=True, debugfile=debug_file,
+ outputdir=None, debuglog=None, errorlog=None, picklefile=None):
- global parse # Reference to the parsing method of the last built parser
+ if tabmodule is None:
+ tabmodule = tab_module
- # If pickling is enabled, table files are not created
+ # Reference to the parsing method of the last built parser
+ global parse
+ # If pickling is enabled, table files are not created
if picklefile:
write_tables = 0
@@ -3055,17 +3230,50 @@ def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, star
# Get the module dictionary used for the parser
if module:
- _items = [(k,getattr(module,k)) for k in dir(module)]
+ _items = [(k, getattr(module, k)) for k in dir(module)]
pdict = dict(_items)
+ # If no __file__ attribute is available, try to obtain it from the __module__ instead
+ if '__file__' not in pdict:
+ pdict['__file__'] = sys.modules[pdict['__module__']].__file__
else:
pdict = get_caller_module_dict(2)
+ if outputdir is None:
+ # If no output directory is set, the location of the output files
+ # is determined according to the following rules:
+ # - If tabmodule specifies a package, files go into that package directory
+ # - Otherwise, files go in the same directory as the specifying module
+ if isinstance(tabmodule, types.ModuleType):
+ srcfile = tabmodule.__file__
+ else:
+ if '.' not in tabmodule:
+ srcfile = pdict['__file__']
+ else:
+ parts = tabmodule.split('.')
+ pkgname = '.'.join(parts[:-1])
+ exec('import %s' % pkgname)
+ srcfile = getattr(sys.modules[pkgname], '__file__', '')
+ outputdir = os.path.dirname(srcfile)
+
+ # Determine if the module is package of a package or not.
+ # If so, fix the tabmodule setting so that tables load correctly
+ pkg = pdict.get('__package__')
+ if pkg and isinstance(tabmodule, str):
+ if '.' not in tabmodule:
+ tabmodule = pkg + '.' + tabmodule
+
+
+
+ # Set start symbol if it's specified directly using an argument
+ if start is not None:
+ pdict['start'] = start
+
# Collect parser information from the dictionary
- pinfo = ParserReflect(pdict,log=errorlog)
+ pinfo = ParserReflect(pdict, log=errorlog)
pinfo.get_all()
if pinfo.error:
- raise YaccError("Unable to build parser")
+ raise YaccError('Unable to build parser')
# Check signature against table files (if any)
signature = pinfo.signature()
@@ -3080,35 +3288,36 @@ def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, star
if optimize or (read_signature == signature):
try:
lr.bind_callables(pinfo.pdict)
- parser = LRParser(lr,pinfo.error_func)
+ parser = LRParser(lr, pinfo.error_func)
parse = parser.parse
return parser
- except Exception:
- e = sys.exc_info()[1]
- errorlog.warning("There was a problem loading the table file: %s", repr(e))
- except VersionError:
- e = sys.exc_info()
+ except Exception as e:
+ errorlog.warning('There was a problem loading the table file: %r', e)
+ except VersionError as e:
errorlog.warning(str(e))
- except Exception:
+ except ImportError:
pass
if debuglog is None:
if debug:
- debuglog = PlyLogger(open(debugfile,"w"))
+ try:
+ debuglog = PlyLogger(open(os.path.join(outputdir, debugfile), 'w'))
+ except IOError as e:
+ errorlog.warning("Couldn't open %r. %s" % (debugfile, e))
+ debuglog = NullLogger()
else:
debuglog = NullLogger()
- debuglog.info("Created by PLY version %s (http://www.dabeaz.com/ply)", __version__)
-
+ debuglog.info('Created by PLY version %s (http://www.dabeaz.com/ply)', __version__)
- errors = 0
+ errors = False
# Validate the parser information
if pinfo.validate_all():
- raise YaccError("Unable to build parser")
-
+ raise YaccError('Unable to build parser')
+
if not pinfo.error_func:
- errorlog.warning("no p_error() function is defined")
+ errorlog.warning('no p_error() function is defined')
# Create a grammar object
grammar = Grammar(pinfo.tokens)
@@ -3116,20 +3325,18 @@ def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, star
# Set precedence level for terminals
for term, assoc, level in pinfo.preclist:
try:
- grammar.set_precedence(term,assoc,level)
- except GrammarError:
- e = sys.exc_info()[1]
- errorlog.warning("%s",str(e))
+ grammar.set_precedence(term, assoc, level)
+ except GrammarError as e:
+ errorlog.warning('%s', e)
# Add productions to the grammar
for funcname, gram in pinfo.grammar:
file, line, prodname, syms = gram
try:
- grammar.add_production(prodname,syms,funcname,file,line)
- except GrammarError:
- e = sys.exc_info()[1]
- errorlog.error("%s",str(e))
- errors = 1
+ grammar.add_production(prodname, syms, funcname, file, line)
+ except GrammarError as e:
+ errorlog.error('%s', e)
+ errors = True
# Set the grammar start symbols
try:
@@ -3137,146 +3344,151 @@ def yacc(method='LALR', debug=yaccdebug, module=None, tabmodule=tab_module, star
grammar.set_start(pinfo.start)
else:
grammar.set_start(start)
- except GrammarError:
- e = sys.exc_info()[1]
+ except GrammarError as e:
errorlog.error(str(e))
- errors = 1
+ errors = True
if errors:
- raise YaccError("Unable to build parser")
+ raise YaccError('Unable to build parser')
# Verify the grammar structure
undefined_symbols = grammar.undefined_symbols()
for sym, prod in undefined_symbols:
- errorlog.error("%s:%d: Symbol '%s' used, but not defined as a token or a rule",prod.file,prod.line,sym)
- errors = 1
+ errorlog.error('%s:%d: Symbol %r used, but not defined as a token or a rule', prod.file, prod.line, sym)
+ errors = True
unused_terminals = grammar.unused_terminals()
if unused_terminals:
- debuglog.info("")
- debuglog.info("Unused terminals:")
- debuglog.info("")
+ debuglog.info('')
+ debuglog.info('Unused terminals:')
+ debuglog.info('')
for term in unused_terminals:
- errorlog.warning("Token '%s' defined, but not used", term)
- debuglog.info(" %s", term)
+ errorlog.warning('Token %r defined, but not used', term)
+ debuglog.info(' %s', term)
# Print out all productions to the debug log
if debug:
- debuglog.info("")
- debuglog.info("Grammar")
- debuglog.info("")
- for n,p in enumerate(grammar.Productions):
- debuglog.info("Rule %-5d %s", n, p)
+ debuglog.info('')
+ debuglog.info('Grammar')
+ debuglog.info('')
+ for n, p in enumerate(grammar.Productions):
+ debuglog.info('Rule %-5d %s', n, p)
# Find unused non-terminals
unused_rules = grammar.unused_rules()
for prod in unused_rules:
- errorlog.warning("%s:%d: Rule '%s' defined, but not used", prod.file, prod.line, prod.name)
+ errorlog.warning('%s:%d: Rule %r defined, but not used', prod.file, prod.line, prod.name)
if len(unused_terminals) == 1:
- errorlog.warning("There is 1 unused token")
+ errorlog.warning('There is 1 unused token')
if len(unused_terminals) > 1:
- errorlog.warning("There are %d unused tokens", len(unused_terminals))
+ errorlog.warning('There are %d unused tokens', len(unused_terminals))
if len(unused_rules) == 1:
- errorlog.warning("There is 1 unused rule")
+ errorlog.warning('There is 1 unused rule')
if len(unused_rules) > 1:
- errorlog.warning("There are %d unused rules", len(unused_rules))
+ errorlog.warning('There are %d unused rules', len(unused_rules))
if debug:
- debuglog.info("")
- debuglog.info("Terminals, with rules where they appear")
- debuglog.info("")
+ debuglog.info('')
+ debuglog.info('Terminals, with rules where they appear')
+ debuglog.info('')
terms = list(grammar.Terminals)
terms.sort()
for term in terms:
- debuglog.info("%-20s : %s", term, " ".join([str(s) for s in grammar.Terminals[term]]))
-
- debuglog.info("")
- debuglog.info("Nonterminals, with rules where they appear")
- debuglog.info("")
+ debuglog.info('%-20s : %s', term, ' '.join([str(s) for s in grammar.Terminals[term]]))
+
+ debuglog.info('')
+ debuglog.info('Nonterminals, with rules where they appear')
+ debuglog.info('')
nonterms = list(grammar.Nonterminals)
nonterms.sort()
for nonterm in nonterms:
- debuglog.info("%-20s : %s", nonterm, " ".join([str(s) for s in grammar.Nonterminals[nonterm]]))
- debuglog.info("")
+ debuglog.info('%-20s : %s', nonterm, ' '.join([str(s) for s in grammar.Nonterminals[nonterm]]))
+ debuglog.info('')
if check_recursion:
unreachable = grammar.find_unreachable()
for u in unreachable:
- errorlog.warning("Symbol '%s' is unreachable",u)
+ errorlog.warning('Symbol %r is unreachable', u)
infinite = grammar.infinite_cycles()
for inf in infinite:
- errorlog.error("Infinite recursion detected for symbol '%s'", inf)
- errors = 1
-
+ errorlog.error('Infinite recursion detected for symbol %r', inf)
+ errors = True
+
unused_prec = grammar.unused_precedence()
for term, assoc in unused_prec:
- errorlog.error("Precedence rule '%s' defined for unknown symbol '%s'", assoc, term)
- errors = 1
+ errorlog.error('Precedence rule %r defined for unknown symbol %r', assoc, term)
+ errors = True
if errors:
- raise YaccError("Unable to build parser")
-
+ raise YaccError('Unable to build parser')
+
# Run the LRGeneratedTable on the grammar
if debug:
- errorlog.debug("Generating %s tables", method)
-
- lr = LRGeneratedTable(grammar,method,debuglog)
+ errorlog.debug('Generating %s tables', method)
+
+ lr = LRGeneratedTable(grammar, method, debuglog)
if debug:
num_sr = len(lr.sr_conflicts)
# Report shift/reduce and reduce/reduce conflicts
if num_sr == 1:
- errorlog.warning("1 shift/reduce conflict")
+ errorlog.warning('1 shift/reduce conflict')
elif num_sr > 1:
- errorlog.warning("%d shift/reduce conflicts", num_sr)
+ errorlog.warning('%d shift/reduce conflicts', num_sr)
num_rr = len(lr.rr_conflicts)
if num_rr == 1:
- errorlog.warning("1 reduce/reduce conflict")
+ errorlog.warning('1 reduce/reduce conflict')
elif num_rr > 1:
- errorlog.warning("%d reduce/reduce conflicts", num_rr)
+ errorlog.warning('%d reduce/reduce conflicts', num_rr)
# Write out conflicts to the output file
if debug and (lr.sr_conflicts or lr.rr_conflicts):
- debuglog.warning("")
- debuglog.warning("Conflicts:")
- debuglog.warning("")
+ debuglog.warning('')
+ debuglog.warning('Conflicts:')
+ debuglog.warning('')
for state, tok, resolution in lr.sr_conflicts:
- debuglog.warning("shift/reduce conflict for %s in state %d resolved as %s", tok, state, resolution)
-
- already_reported = {}
+ debuglog.warning('shift/reduce conflict for %s in state %d resolved as %s', tok, state, resolution)
+
+ already_reported = set()
for state, rule, rejected in lr.rr_conflicts:
- if (state,id(rule),id(rejected)) in already_reported:
+ if (state, id(rule), id(rejected)) in already_reported:
continue
- debuglog.warning("reduce/reduce conflict in state %d resolved using rule (%s)", state, rule)
- debuglog.warning("rejected rule (%s) in state %d", rejected,state)
- errorlog.warning("reduce/reduce conflict in state %d resolved using rule (%s)", state, rule)
- errorlog.warning("rejected rule (%s) in state %d", rejected, state)
- already_reported[state,id(rule),id(rejected)] = 1
-
+ debuglog.warning('reduce/reduce conflict in state %d resolved using rule (%s)', state, rule)
+ debuglog.warning('rejected rule (%s) in state %d', rejected, state)
+ errorlog.warning('reduce/reduce conflict in state %d resolved using rule (%s)', state, rule)
+ errorlog.warning('rejected rule (%s) in state %d', rejected, state)
+ already_reported.add((state, id(rule), id(rejected)))
+
warned_never = []
for state, rule, rejected in lr.rr_conflicts:
if not rejected.reduced and (rejected not in warned_never):
- debuglog.warning("Rule (%s) is never reduced", rejected)
- errorlog.warning("Rule (%s) is never reduced", rejected)
+ debuglog.warning('Rule (%s) is never reduced', rejected)
+ errorlog.warning('Rule (%s) is never reduced', rejected)
warned_never.append(rejected)
# Write the table file if requested
if write_tables:
- lr.write_table(tabmodule,outputdir,signature)
+ try:
+ lr.write_table(tabmodule, outputdir, signature)
+ except IOError as e:
+ errorlog.warning("Couldn't create %r. %s" % (tabmodule, e))
# Write a pickled version of the tables
if picklefile:
- lr.pickle_table(picklefile,signature)
+ try:
+ lr.pickle_table(picklefile, signature)
+ except IOError as e:
+ errorlog.warning("Couldn't create %r. %s" % (picklefile, e))
# Build the parser
lr.bind_callables(pinfo.pdict)
- parser = LRParser(lr,pinfo.error_func)
+ parser = LRParser(lr, pinfo.error_func)
parse = parser.parse
return parser
diff --git a/bitbake/lib/bb/_vendor/ply/ygen.py b/bitbake/lib/bb/_vendor/ply/ygen.py
new file mode 100644
index 00000000..acf5ca1a
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/ply/ygen.py
@@ -0,0 +1,74 @@
+# ply: ygen.py
+#
+# This is a support program that auto-generates different versions of the YACC parsing
+# function with different features removed for the purposes of performance.
+#
+# Users should edit the method LParser.parsedebug() in yacc.py. The source code
+# for that method is then used to create the other methods. See the comments in
+# yacc.py for further details.
+
+import os.path
+import shutil
+
+def get_source_range(lines, tag):
+ srclines = enumerate(lines)
+ start_tag = '#--! %s-start' % tag
+ end_tag = '#--! %s-end' % tag
+
+ for start_index, line in srclines:
+ if line.strip().startswith(start_tag):
+ break
+
+ for end_index, line in srclines:
+ if line.strip().endswith(end_tag):
+ break
+
+ return (start_index + 1, end_index)
+
+def filter_section(lines, tag):
+ filtered_lines = []
+ include = True
+ tag_text = '#--! %s' % tag
+ for line in lines:
+ if line.strip().startswith(tag_text):
+ include = not include
+ elif include:
+ filtered_lines.append(line)
+ return filtered_lines
+
+def main():
+ dirname = os.path.dirname(__file__)
+ shutil.copy2(os.path.join(dirname, 'yacc.py'), os.path.join(dirname, 'yacc.py.bak'))
+ with open(os.path.join(dirname, 'yacc.py'), 'r') as f:
+ lines = f.readlines()
+
+ parse_start, parse_end = get_source_range(lines, 'parsedebug')
+ parseopt_start, parseopt_end = get_source_range(lines, 'parseopt')
+ parseopt_notrack_start, parseopt_notrack_end = get_source_range(lines, 'parseopt-notrack')
+
+ # Get the original source
+ orig_lines = lines[parse_start:parse_end]
+
+ # Filter the DEBUG sections out
+ parseopt_lines = filter_section(orig_lines, 'DEBUG')
+
+ # Filter the TRACKING sections out
+ parseopt_notrack_lines = filter_section(parseopt_lines, 'TRACKING')
+
+ # Replace the parser source sections with updated versions
+ lines[parseopt_notrack_start:parseopt_notrack_end] = parseopt_notrack_lines
+ lines[parseopt_start:parseopt_end] = parseopt_lines
+
+ lines = [line.rstrip()+'\n' for line in lines]
+ with open(os.path.join(dirname, 'yacc.py'), 'w') as f:
+ f.writelines(lines)
+
+ print('Updated yacc.py')
+
+if __name__ == '__main__':
+ main()
+
+
+
+
+
diff --git a/bitbake/lib/bb/_vendor/progressbar.pyi b/bitbake/lib/bb/_vendor/progressbar.pyi
new file mode 100644
index 00000000..83480576
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/progressbar.pyi
@@ -0,0 +1 @@
+from progressbar import *
\ No newline at end of file
diff --git a/bitbake/lib/progressbar/LICENSE.txt b/bitbake/lib/bb/_vendor/progressbar/LICENSE.txt
similarity index 100%
rename from bitbake/lib/progressbar/LICENSE.txt
rename to bitbake/lib/bb/_vendor/progressbar/LICENSE.txt
diff --git a/bitbake/lib/progressbar/__init__.py b/bitbake/lib/bb/_vendor/progressbar/__init__.py
similarity index 96%
rename from bitbake/lib/progressbar/__init__.py
rename to bitbake/lib/bb/_vendor/progressbar/__init__.py
index c545a627..623493bc 100644
--- a/bitbake/lib/progressbar/__init__.py
+++ b/bitbake/lib/bb/_vendor/progressbar/__init__.py
@@ -42,9 +42,9 @@ automatically enable features like auto-resizing when the system supports it.
"""
__author__ = 'Nilton Volpato'
-__author_email__ = 'first-name dot last-name @ gmail.com'
+__author_email__ = 'nilton.volpato@gmail.com'
__date__ = '2011-05-14'
-__version__ = '2.3'
+__version__ = '2.5'
from .compat import *
from .widgets import *
diff --git a/bitbake/lib/progressbar/compat.py b/bitbake/lib/bb/_vendor/progressbar/compat.py
similarity index 100%
rename from bitbake/lib/progressbar/compat.py
rename to bitbake/lib/bb/_vendor/progressbar/compat.py
diff --git a/bitbake/lib/progressbar/progressbar.py b/bitbake/lib/bb/_vendor/progressbar/progressbar.py
similarity index 88%
rename from bitbake/lib/progressbar/progressbar.py
rename to bitbake/lib/bb/_vendor/progressbar/progressbar.py
index d4da10ab..3cc49547 100644
--- a/bitbake/lib/progressbar/progressbar.py
+++ b/bitbake/lib/bb/_vendor/progressbar/progressbar.py
@@ -42,9 +42,6 @@ from .compat import * # for: any, next
from . import widgets
-class UnknownLength: pass
-
-
class ProgressBar(object):
"""The ProgressBar class which updates and prints the bar.
@@ -99,29 +96,36 @@ class ProgressBar(object):
_DEFAULT_WIDGETS = [widgets.Percentage(), ' ', widgets.Bar()]
def __init__(self, maxval=None, widgets=None, term_width=None, poll=1,
- left_justify=True, fd=sys.stderr):
+ left_justify=True, fd=None):
"""Initializes a progress bar with sane defaults."""
# Don't share a reference with any other progress bars
if widgets is None:
widgets = list(self._DEFAULT_WIDGETS)
- self.maxval = maxval
+ self.maxval = maxval if maxval != 0 else self._DEFAULT_MAXVAL
self.widgets = widgets
- self.fd = fd
+ self.fd = fd if fd is not None else sys.stderr
self.left_justify = left_justify
+ self._fd_console = None
self.signal_set = False
if term_width is not None:
self.term_width = term_width
else:
try:
- self._handle_resize(None, None)
+ # Check if given file descriptor is resizable for example belong
+ # to a terminal/console as STDOUT or STDERR. If file descriptor
+ # is resizable, let's allow to use for self._handle_resize()
+ # in a dedicated self._fd_console in order to be able to set
+ # temporarily/permanently self.fd to any StringIO or other
+ # file descriptor later.
+ self._fd_console = fd
+ self._handle_resize()
signal.signal(signal.SIGWINCH, self._handle_resize)
self.signal_set = True
except (SystemExit, KeyboardInterrupt): raise
- except Exception as e:
- print("DEBUG 5 %s" % e)
+ except:
self.term_width = self._env_size()
self.__iterable = None
@@ -143,7 +147,7 @@ class ProgressBar(object):
self.maxval = len(iterable)
except:
if self.maxval is None:
- self.maxval = UnknownLength
+ self.maxval = widgets.UnknownLength
self.__iterable = iter(iterable)
return self
@@ -182,12 +186,14 @@ class ProgressBar(object):
def _handle_resize(self, signum=None, frame=None):
"""Tries to catch resize signals sent from the terminal."""
- h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
+ h, w = array('h', ioctl(self._fd_console, termios.TIOCGWINSZ, '\0' * 8))[:2]
self.term_width = w
def percentage(self):
"""Returns the progress as a percentage."""
+ if self.maxval is widgets.UnknownLength:
+ return float("NaN")
if self.currval >= self.maxval:
return 100.0
return (self.currval * 100.0 / self.maxval) if self.maxval else 100.00
@@ -249,8 +255,8 @@ class ProgressBar(object):
def update(self, value=None):
"""Updates the ProgressBar to a new value."""
- if value is not None and value is not UnknownLength:
- if (self.maxval is not UnknownLength
+ if value is not None and value is not widgets.UnknownLength:
+ if (self.maxval is not widgets.UnknownLength
and not 0 <= value <= self.maxval):
self.maxval = value
@@ -271,6 +277,9 @@ class ProgressBar(object):
self.last_update_time = now
return output
+ def clear(self):
+ self.fd.write(" " * self.term_width + '\r')
+ self.fd.flush()
def start(self, update=True):
"""Starts measuring time, and prints the bar at 0%.
@@ -290,7 +299,7 @@ class ProgressBar(object):
self.num_intervals = max(100, self.term_width)
self.next_update = 0
- if self.maxval is not UnknownLength:
+ if self.maxval is not widgets.UnknownLength:
if self.maxval < 0: raise ValueError('Value out of range')
self.update_interval = self.maxval / self.num_intervals
diff --git a/bitbake/lib/progressbar/widgets.py b/bitbake/lib/bb/_vendor/progressbar/widgets.py
similarity index 97%
rename from bitbake/lib/progressbar/widgets.py
rename to bitbake/lib/bb/_vendor/progressbar/widgets.py
index 0772aa53..d844fc36 100644
--- a/bitbake/lib/progressbar/widgets.py
+++ b/bitbake/lib/bb/_vendor/progressbar/widgets.py
@@ -34,6 +34,8 @@ except ImportError:
else:
AbstractWidget = ABCMeta('AbstractWidget', (object,), {})
+class UnknownLength:
+ pass
def format_updatable(updatable, pbar):
if hasattr(updatable, 'update'): return updatable.update(pbar)
@@ -109,7 +111,7 @@ class ETA(Timer):
def update(self, pbar):
"""Updates the widget to show the ETA or total time when finished."""
- if pbar.currval == 0:
+ if pbar.maxval is UnknownLength or pbar.currval == 0:
return 'ETA: --:--:--'
elif pbar.finished:
return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
@@ -147,7 +149,7 @@ class AdaptiveETA(Timer):
def update(self, pbar):
"""Updates the widget to show the ETA or total time when finished."""
- if pbar.currval == 0:
+ if pbar.maxval is UnknownLength or pbar.currval == 0:
return 'ETA: --:--:--'
elif pbar.finished:
return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
@@ -167,7 +169,7 @@ class AdaptiveETA(Timer):
class FileTransferSpeed(Widget):
"""Widget for showing the transfer speed (useful for file transfers)."""
- FORMAT = '%6.2f %s%s/s'
+ FMT = '%6.2f %s%s/s'
PREFIXES = ' kMGTPEZY'
__slots__ = ('unit',)
@@ -184,7 +186,7 @@ class FileTransferSpeed(Widget):
power = int(math.log(speed, 1000))
scaled = speed / 1000.**power
- return self.FORMAT % (scaled, self.PREFIXES[power], self.unit)
+ return self.FMT % (scaled, self.PREFIXES[power], self.unit)
class AnimatedMarker(Widget):
@@ -271,7 +273,7 @@ class SimpleProgress(Widget):
self.sep = sep
def update(self, pbar):
- return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
+ return '%d%s%s' % (pbar.currval, self.sep, pbar.maxval)
class Bar(WidgetHFill):
@@ -304,7 +306,7 @@ class Bar(WidgetHFill):
width -= len(left) + len(right)
# Marked must *always* have length of 1
- if pbar.maxval:
+ if pbar.maxval is not UnknownLength and pbar.maxval:
marked *= int(pbar.currval / pbar.maxval * width)
else:
marked = ''
diff --git a/bitbake/lib/bb/_vendor/simplediff.pyi b/bitbake/lib/bb/_vendor/simplediff.pyi
new file mode 100644
index 00000000..26dabde8
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/simplediff.pyi
@@ -0,0 +1 @@
+from simplediff import *
\ No newline at end of file
diff --git a/bitbake/lib/simplediff/LICENSE b/bitbake/lib/bb/_vendor/simplediff/LICENSE
similarity index 100%
rename from bitbake/lib/simplediff/LICENSE
rename to bitbake/lib/bb/_vendor/simplediff/LICENSE
diff --git a/bitbake/lib/simplediff/__init__.py b/bitbake/lib/bb/_vendor/simplediff/__init__.py
similarity index 99%
rename from bitbake/lib/simplediff/__init__.py
rename to bitbake/lib/bb/_vendor/simplediff/__init__.py
index 57ee3c5c..4e6b59eb 100644
--- a/bitbake/lib/simplediff/__init__.py
+++ b/bitbake/lib/bb/_vendor/simplediff/__init__.py
@@ -11,7 +11,7 @@ May be used and distributed under the zlib/libpng license
'''
__all__ = ['diff', 'string_diff', 'html_diff']
-__version__ = '1.0'
+__version__ = '1.1'
def diff(old, new):
diff --git a/bitbake/lib/bb/_vendor/tomli/LICENSE b/bitbake/lib/bb/_vendor/tomli/LICENSE
new file mode 100644
index 00000000..e859590f
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 Taneli Hukkinen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/bitbake/lib/bb/_vendor/tomli/__init__.py b/bitbake/lib/bb/_vendor/tomli/__init__.py
new file mode 100644
index 00000000..ebe90013
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/__init__.py
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: MIT
+# SPDX-FileCopyrightText: 2021 Taneli Hukkinen
+# Licensed to PSF under a Contributor Agreement.
+
+__all__ = ("loads", "load", "TOMLDecodeError")
+__version__ = "2.4.1" # DO NOT EDIT THIS LINE MANUALLY. LET bump2version UTILITY DO IT
+
+from ._parser import TOMLDecodeError, load, loads
diff --git a/bitbake/lib/bb/_vendor/tomli/_parser.py b/bitbake/lib/bb/_vendor/tomli/_parser.py
new file mode 100644
index 00000000..41b06418
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/_parser.py
@@ -0,0 +1,793 @@
+# SPDX-License-Identifier: MIT
+# SPDX-FileCopyrightText: 2021 Taneli Hukkinen
+# Licensed to PSF under a Contributor Agreement.
+
+from __future__ import annotations
+
+import sys
+from types import MappingProxyType
+
+from ._re import (
+ RE_DATETIME,
+ RE_LOCALTIME,
+ RE_NUMBER,
+ match_to_datetime,
+ match_to_localtime,
+ match_to_number,
+)
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+ from typing import IO, Any, Final
+
+ from ._types import Key, ParseFloat, Pos
+
+# Inline tables/arrays are implemented using recursion. Pathologically
+# nested documents cause pure Python to raise RecursionError (which is OK),
+# but mypyc binary wheels will crash unrecoverably (not OK). According to
+# mypyc docs this will be fixed in the future:
+# https://mypyc.readthedocs.io/en/latest/differences_from_python.html#stack-overflows
+# Before mypyc's fix is in, recursion needs to be limited by this library.
+# Choosing `sys.getrecursionlimit()` as maximum inline table/array nesting
+# level, as it allows more nesting than pure Python, but still seems a far
+# lower number than where mypyc binaries crash.
+MAX_INLINE_NESTING: Final = sys.getrecursionlimit()
+
+# Pathologically excessive number of parts in a key runs into quadratic
+# behavior (e.g. in Flags.is_).
+# Even if keys aren't currently parsed using recursion, they name a
+# recursive structure, so it makes sense to limit it using getrecursionlimit()
+# and RecursionError.
+MAX_KEY_PARTS: Final = sys.getrecursionlimit()
+
+ASCII_CTRL: Final = frozenset(chr(i) for i in range(32)) | frozenset(chr(127))
+
+# Neither of these sets include quotation mark or backslash. They are
+# currently handled as separate cases in the parser functions.
+ILLEGAL_BASIC_STR_CHARS: Final = ASCII_CTRL - frozenset("\t")
+ILLEGAL_MULTILINE_BASIC_STR_CHARS: Final = ASCII_CTRL - frozenset("\t\n")
+
+ILLEGAL_LITERAL_STR_CHARS: Final = ILLEGAL_BASIC_STR_CHARS
+ILLEGAL_MULTILINE_LITERAL_STR_CHARS: Final = ILLEGAL_MULTILINE_BASIC_STR_CHARS
+
+ILLEGAL_COMMENT_CHARS: Final = ILLEGAL_BASIC_STR_CHARS
+
+TOML_WS: Final = frozenset(" \t")
+TOML_WS_AND_NEWLINE: Final = TOML_WS | frozenset("\n")
+BARE_KEY_CHARS: Final = frozenset(
+ "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "0123456789" "-_"
+)
+KEY_INITIAL_CHARS: Final = BARE_KEY_CHARS | frozenset("\"'")
+HEXDIGIT_CHARS: Final = frozenset("abcdef" "ABCDEF" "0123456789")
+
+BASIC_STR_ESCAPE_REPLACEMENTS: Final = MappingProxyType(
+ {
+ "\\b": "\u0008", # backspace
+ "\\t": "\u0009", # tab
+ "\\n": "\u000a", # linefeed
+ "\\f": "\u000c", # form feed
+ "\\r": "\u000d", # carriage return
+ "\\e": "\u001b", # escape
+ '\\"': "\u0022", # quote
+ "\\\\": "\u005c", # backslash
+ }
+)
+
+
+class DEPRECATED_DEFAULT:
+ """Sentinel to be used as default arg during deprecation
+ period of TOMLDecodeError's free-form arguments."""
+
+
+class TOMLDecodeError(ValueError):
+ """An error raised if a document is not valid TOML.
+
+ Adds the following attributes to ValueError:
+ msg: The unformatted error message
+ doc: The TOML document being parsed
+ pos: The index of doc where parsing failed
+ lineno: The line corresponding to pos
+ colno: The column corresponding to pos
+ """
+
+ def __init__(
+ self,
+ msg: str | type[DEPRECATED_DEFAULT] = DEPRECATED_DEFAULT,
+ doc: str | type[DEPRECATED_DEFAULT] = DEPRECATED_DEFAULT,
+ pos: Pos | type[DEPRECATED_DEFAULT] = DEPRECATED_DEFAULT,
+ *args: Any,
+ ):
+ if (
+ args
+ or not isinstance(msg, str)
+ or not isinstance(doc, str)
+ or not isinstance(pos, int)
+ ):
+ import warnings
+
+ warnings.warn(
+ "Free-form arguments for TOMLDecodeError are deprecated. "
+ "Please set 'msg' (str), 'doc' (str) and 'pos' (int) arguments only.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ if pos is not DEPRECATED_DEFAULT:
+ args = pos, *args
+ if doc is not DEPRECATED_DEFAULT:
+ args = doc, *args
+ if msg is not DEPRECATED_DEFAULT:
+ args = msg, *args
+ ValueError.__init__(self, *args)
+ return
+
+ lineno = doc.count("\n", 0, pos) + 1
+ if lineno == 1:
+ colno = pos + 1
+ else:
+ colno = pos - doc.rindex("\n", 0, pos)
+
+ if pos >= len(doc):
+ coord_repr = "end of document"
+ else:
+ coord_repr = f"line {lineno}, column {colno}"
+ errmsg = f"{msg} (at {coord_repr})"
+ ValueError.__init__(self, errmsg)
+
+ self.msg = msg
+ self.doc = doc
+ self.pos = pos
+ self.lineno = lineno
+ self.colno = colno
+
+
+def load(__fp: IO[bytes], *, parse_float: ParseFloat = float) -> dict[str, Any]:
+ """Parse TOML from a binary file object."""
+ b = __fp.read()
+ try:
+ s = b.decode()
+ except AttributeError:
+ raise TypeError(
+ "File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`"
+ ) from None
+ return loads(s, parse_float=parse_float)
+
+
+def loads(__s: str, *, parse_float: ParseFloat = float) -> dict[str, Any]:
+ """Parse TOML from a string."""
+
+ # The spec allows converting "\r\n" to "\n", even in string
+ # literals. Let's do so to simplify parsing.
+ try:
+ src = __s.replace("\r\n", "\n")
+ except (AttributeError, TypeError):
+ raise TypeError(
+ f"Expected str object, not '{type(__s).__qualname__}'"
+ ) from None
+ pos = 0
+ out = Output()
+ header: Key = ()
+ parse_float = make_safe_parse_float(parse_float)
+
+ # Parse one statement at a time
+ # (typically means one line in TOML source)
+ while True:
+ # 1. Skip line leading whitespace
+ pos = skip_chars(src, pos, TOML_WS)
+
+ # 2. Parse rules. Expect one of the following:
+ # - end of file
+ # - end of line
+ # - comment
+ # - key/value pair
+ # - append dict to list (and move to its namespace)
+ # - create dict (and move to its namespace)
+ # Skip trailing whitespace when applicable.
+ try:
+ char = src[pos]
+ except IndexError:
+ break
+ if char == "\n":
+ pos += 1
+ continue
+ if char in KEY_INITIAL_CHARS:
+ pos = key_value_rule(src, pos, out, header, parse_float)
+ pos = skip_chars(src, pos, TOML_WS)
+ elif char == "[":
+ try:
+ second_char: str | None = src[pos + 1]
+ except IndexError:
+ second_char = None
+ out.flags.finalize_pending()
+ if second_char == "[":
+ pos, header = create_list_rule(src, pos, out)
+ else:
+ pos, header = create_dict_rule(src, pos, out)
+ pos = skip_chars(src, pos, TOML_WS)
+ elif char != "#":
+ raise TOMLDecodeError("Invalid statement", src, pos)
+
+ # 3. Skip comment
+ pos = skip_comment(src, pos)
+
+ # 4. Expect end of line or end of file
+ try:
+ char = src[pos]
+ except IndexError:
+ break
+ if char != "\n":
+ raise TOMLDecodeError(
+ "Expected newline or end of document after a statement", src, pos
+ )
+ pos += 1
+
+ return out.data.dict
+
+
+class Flags:
+ """Flags that map to parsed keys/namespaces."""
+
+ # Marks an immutable namespace (inline array or inline table).
+ FROZEN: Final = 0
+ # Marks a nest that has been explicitly created and can no longer
+ # be opened using the "[table]" syntax.
+ EXPLICIT_NEST: Final = 1
+
+ def __init__(self) -> None:
+ self._flags: dict[str, dict[Any, Any]] = {}
+ self._pending_flags: set[tuple[Key, int]] = set()
+
+ def add_pending(self, key: Key, flag: int) -> None:
+ self._pending_flags.add((key, flag))
+
+ def finalize_pending(self) -> None:
+ for key, flag in self._pending_flags:
+ self.set(key, flag, recursive=False)
+ self._pending_flags.clear()
+
+ def unset_all(self, key: Key) -> None:
+ cont = self._flags
+ for k in key[:-1]:
+ if k not in cont:
+ return
+ cont = cont[k]["nested"]
+ cont.pop(key[-1], None)
+
+ def set(self, key: Key, flag: int, *, recursive: bool) -> None: # noqa: A003
+ cont = self._flags
+ key_parent, key_stem = key[:-1], key[-1]
+ for k in key_parent:
+ if k not in cont:
+ cont[k] = {"flags": set(), "recursive_flags": set(), "nested": {}}
+ cont = cont[k]["nested"]
+ if key_stem not in cont:
+ cont[key_stem] = {"flags": set(), "recursive_flags": set(), "nested": {}}
+ cont[key_stem]["recursive_flags" if recursive else "flags"].add(flag)
+
+ def is_(self, key: Key, flag: int) -> bool:
+ if not key:
+ return False # document root has no flags
+ cont = self._flags
+ for k in key[:-1]:
+ if k not in cont:
+ return False
+ inner_cont = cont[k]
+ if flag in inner_cont["recursive_flags"]:
+ return True
+ cont = inner_cont["nested"]
+ key_stem = key[-1]
+ if key_stem in cont:
+ inner_cont = cont[key_stem]
+ return flag in inner_cont["flags"] or flag in inner_cont["recursive_flags"]
+ return False
+
+
+class NestedDict:
+ def __init__(self) -> None:
+ # The parsed content of the TOML document
+ self.dict: dict[str, Any] = {}
+
+ def get_or_create_nest(
+ self,
+ key: Key,
+ *,
+ access_lists: bool = True,
+ ) -> dict[str, Any]:
+ cont: Any = self.dict
+ for k in key:
+ if k not in cont:
+ cont[k] = {}
+ cont = cont[k]
+ if access_lists and isinstance(cont, list):
+ cont = cont[-1]
+ if not isinstance(cont, dict):
+ raise KeyError("There is no nest behind this key")
+ return cont # type: ignore[no-any-return]
+
+ def append_nest_to_list(self, key: Key) -> None:
+ cont = self.get_or_create_nest(key[:-1])
+ last_key = key[-1]
+ if last_key in cont:
+ list_ = cont[last_key]
+ if not isinstance(list_, list):
+ raise KeyError("An object other than list found behind this key")
+ list_.append({})
+ else:
+ cont[last_key] = [{}]
+
+
+class Output:
+ def __init__(self) -> None:
+ self.data = NestedDict()
+ self.flags = Flags()
+
+
+def skip_chars(src: str, pos: Pos, chars: Iterable[str]) -> Pos:
+ try:
+ while src[pos] in chars:
+ pos += 1
+ except IndexError:
+ pass
+ return pos
+
+
+def skip_until(
+ src: str,
+ pos: Pos,
+ expect: str,
+ *,
+ error_on: frozenset[str],
+ error_on_eof: bool,
+) -> Pos:
+ try:
+ new_pos = src.index(expect, pos)
+ except ValueError:
+ new_pos = len(src)
+ if error_on_eof:
+ raise TOMLDecodeError(f"Expected {expect!r}", src, new_pos) from None
+
+ if not error_on.isdisjoint(src[pos:new_pos]):
+ while src[pos] not in error_on:
+ pos += 1
+ raise TOMLDecodeError(f"Found invalid character {src[pos]!r}", src, pos)
+ return new_pos
+
+
+def skip_comment(src: str, pos: Pos) -> Pos:
+ try:
+ char: str | None = src[pos]
+ except IndexError:
+ char = None
+ if char == "#":
+ return skip_until(
+ src, pos + 1, "\n", error_on=ILLEGAL_COMMENT_CHARS, error_on_eof=False
+ )
+ return pos
+
+
+def skip_comments_and_array_ws(src: str, pos: Pos) -> Pos:
+ while True:
+ pos_before_skip = pos
+ pos = skip_chars(src, pos, TOML_WS_AND_NEWLINE)
+ pos = skip_comment(src, pos)
+ if pos == pos_before_skip:
+ return pos
+
+
+def create_dict_rule(src: str, pos: Pos, out: Output) -> tuple[Pos, Key]:
+ pos += 1 # Skip "["
+ pos = skip_chars(src, pos, TOML_WS)
+ pos, key = parse_key(src, pos)
+
+ if out.flags.is_(key, Flags.EXPLICIT_NEST) or out.flags.is_(key, Flags.FROZEN):
+ raise TOMLDecodeError(f"Cannot declare {key} twice", src, pos)
+ out.flags.set(key, Flags.EXPLICIT_NEST, recursive=False)
+ try:
+ out.data.get_or_create_nest(key)
+ except KeyError:
+ raise TOMLDecodeError("Cannot overwrite a value", src, pos) from None
+
+ if not src.startswith("]", pos):
+ raise TOMLDecodeError(
+ "Expected ']' at the end of a table declaration", src, pos
+ )
+ return pos + 1, key
+
+
+def create_list_rule(src: str, pos: Pos, out: Output) -> tuple[Pos, Key]:
+ pos += 2 # Skip "[["
+ pos = skip_chars(src, pos, TOML_WS)
+ pos, key = parse_key(src, pos)
+
+ if out.flags.is_(key, Flags.FROZEN):
+ raise TOMLDecodeError(f"Cannot mutate immutable namespace {key}", src, pos)
+ # Free the namespace now that it points to another empty list item...
+ out.flags.unset_all(key)
+ # ...but this key precisely is still prohibited from table declaration
+ out.flags.set(key, Flags.EXPLICIT_NEST, recursive=False)
+ try:
+ out.data.append_nest_to_list(key)
+ except KeyError:
+ raise TOMLDecodeError("Cannot overwrite a value", src, pos) from None
+
+ if not src.startswith("]]", pos):
+ raise TOMLDecodeError(
+ "Expected ']]' at the end of an array declaration", src, pos
+ )
+ return pos + 2, key
+
+
+def key_value_rule(
+ src: str, pos: Pos, out: Output, header: Key, parse_float: ParseFloat
+) -> Pos:
+ pos, key, value = parse_key_value_pair(src, pos, parse_float, nest_lvl=0)
+ key_parent, key_stem = key[:-1], key[-1]
+ abs_key_parent = header + key_parent
+
+ relative_path_cont_keys = (header + key[:i] for i in range(1, len(key)))
+ for cont_key in relative_path_cont_keys:
+ # Check that dotted key syntax does not redefine an existing table
+ if out.flags.is_(cont_key, Flags.EXPLICIT_NEST):
+ raise TOMLDecodeError(f"Cannot redefine namespace {cont_key}", src, pos)
+ # Containers in the relative path can't be opened with the table syntax or
+ # dotted key/value syntax in following table sections.
+ out.flags.add_pending(cont_key, Flags.EXPLICIT_NEST)
+
+ if out.flags.is_(abs_key_parent, Flags.FROZEN):
+ raise TOMLDecodeError(
+ f"Cannot mutate immutable namespace {abs_key_parent}", src, pos
+ )
+
+ try:
+ nest = out.data.get_or_create_nest(abs_key_parent)
+ except KeyError:
+ raise TOMLDecodeError("Cannot overwrite a value", src, pos) from None
+ if key_stem in nest:
+ raise TOMLDecodeError("Cannot overwrite a value", src, pos)
+ # Mark inline table and array namespaces recursively immutable
+ if isinstance(value, (dict, list)):
+ out.flags.set(header + key, Flags.FROZEN, recursive=True)
+ nest[key_stem] = value
+ return pos
+
+
+def parse_key_value_pair(
+ src: str, pos: Pos, parse_float: ParseFloat, nest_lvl: int
+) -> tuple[Pos, Key, Any]:
+ pos, key = parse_key(src, pos)
+ try:
+ char: str | None = src[pos]
+ except IndexError:
+ char = None
+ if char != "=":
+ raise TOMLDecodeError("Expected '=' after a key in a key/value pair", src, pos)
+ pos += 1
+ pos = skip_chars(src, pos, TOML_WS)
+ pos, value = parse_value(src, pos, parse_float, nest_lvl)
+ return pos, key, value
+
+
+def parse_key(src: str, pos: Pos) -> tuple[Pos, Key]:
+ pos, key_part = parse_key_part(src, pos)
+ key: Key = (key_part,)
+ pos = skip_chars(src, pos, TOML_WS)
+ while True:
+ try:
+ char: str | None = src[pos]
+ except IndexError:
+ char = None
+ if char != ".":
+ return pos, key
+ pos += 1
+ pos = skip_chars(src, pos, TOML_WS)
+ pos, key_part = parse_key_part(src, pos)
+ key += (key_part,)
+ if len(key) > MAX_KEY_PARTS:
+ raise RecursionError(
+ f"TOML key has more than the allowed {MAX_KEY_PARTS} parts"
+ )
+ pos = skip_chars(src, pos, TOML_WS)
+
+
+def parse_key_part(src: str, pos: Pos) -> tuple[Pos, str]:
+ try:
+ char: str | None = src[pos]
+ except IndexError:
+ char = None
+ if char in BARE_KEY_CHARS:
+ start_pos = pos
+ pos = skip_chars(src, pos, BARE_KEY_CHARS)
+ return pos, src[start_pos:pos]
+ if char == "'":
+ return parse_literal_str(src, pos)
+ if char == '"':
+ return parse_one_line_basic_str(src, pos)
+ raise TOMLDecodeError("Invalid initial character for a key part", src, pos)
+
+
+def parse_one_line_basic_str(src: str, pos: Pos) -> tuple[Pos, str]:
+ pos += 1
+ return parse_basic_str(src, pos, multiline=False)
+
+
+def parse_array(
+ src: str, pos: Pos, parse_float: ParseFloat, nest_lvl: int
+) -> tuple[Pos, list[Any]]:
+ pos += 1
+ array: list[Any] = []
+
+ pos = skip_comments_and_array_ws(src, pos)
+ if src.startswith("]", pos):
+ return pos + 1, array
+ while True:
+ pos, val = parse_value(src, pos, parse_float, nest_lvl)
+ array.append(val)
+ pos = skip_comments_and_array_ws(src, pos)
+
+ c = src[pos : pos + 1]
+ if c == "]":
+ return pos + 1, array
+ if c != ",":
+ raise TOMLDecodeError("Unclosed array", src, pos)
+ pos += 1
+
+ pos = skip_comments_and_array_ws(src, pos)
+ if src.startswith("]", pos):
+ return pos + 1, array
+
+
+def parse_inline_table(
+ src: str, pos: Pos, parse_float: ParseFloat, nest_lvl: int
+) -> tuple[Pos, dict[str, Any]]:
+ pos += 1
+ nested_dict = NestedDict()
+ flags = Flags()
+
+ pos = skip_comments_and_array_ws(src, pos)
+ if src.startswith("}", pos):
+ return pos + 1, nested_dict.dict
+ while True:
+ pos, key, value = parse_key_value_pair(src, pos, parse_float, nest_lvl)
+ key_parent, key_stem = key[:-1], key[-1]
+ if flags.is_(key, Flags.FROZEN):
+ raise TOMLDecodeError(f"Cannot mutate immutable namespace {key}", src, pos)
+ try:
+ nest = nested_dict.get_or_create_nest(key_parent, access_lists=False)
+ except KeyError:
+ raise TOMLDecodeError("Cannot overwrite a value", src, pos) from None
+ if key_stem in nest:
+ raise TOMLDecodeError(f"Duplicate inline table key {key_stem!r}", src, pos)
+ nest[key_stem] = value
+ pos = skip_comments_and_array_ws(src, pos)
+ c = src[pos : pos + 1]
+ if c == "}":
+ return pos + 1, nested_dict.dict
+ if c != ",":
+ raise TOMLDecodeError("Unclosed inline table", src, pos)
+ pos += 1
+ pos = skip_comments_and_array_ws(src, pos)
+ if src.startswith("}", pos):
+ return pos + 1, nested_dict.dict
+ if isinstance(value, (dict, list)):
+ flags.set(key, Flags.FROZEN, recursive=True)
+
+
+def parse_basic_str_escape(
+ src: str, pos: Pos, *, multiline: bool = False
+) -> tuple[Pos, str]:
+ escape_id = src[pos : pos + 2]
+ pos += 2
+ if multiline and escape_id in {"\\ ", "\\\t", "\\\n"}:
+ # Skip whitespace until next non-whitespace character or end of
+ # the doc. Error if non-whitespace is found before newline.
+ if escape_id != "\\\n":
+ pos = skip_chars(src, pos, TOML_WS)
+ try:
+ char = src[pos]
+ except IndexError:
+ return pos, ""
+ if char != "\n":
+ raise TOMLDecodeError("Unescaped '\\' in a string", src, pos)
+ pos += 1
+ pos = skip_chars(src, pos, TOML_WS_AND_NEWLINE)
+ return pos, ""
+ if escape_id == "\\x":
+ return parse_hex_char(src, pos, 2)
+ if escape_id == "\\u":
+ return parse_hex_char(src, pos, 4)
+ if escape_id == "\\U":
+ return parse_hex_char(src, pos, 8)
+ try:
+ return pos, BASIC_STR_ESCAPE_REPLACEMENTS[escape_id]
+ except KeyError:
+ raise TOMLDecodeError("Unescaped '\\' in a string", src, pos) from None
+
+
+def parse_basic_str_escape_multiline(src: str, pos: Pos) -> tuple[Pos, str]:
+ return parse_basic_str_escape(src, pos, multiline=True)
+
+
+def parse_hex_char(src: str, pos: Pos, hex_len: int) -> tuple[Pos, str]:
+ hex_str = src[pos : pos + hex_len]
+ if len(hex_str) != hex_len or not HEXDIGIT_CHARS.issuperset(hex_str):
+ raise TOMLDecodeError("Invalid hex value", src, pos)
+ pos += hex_len
+ hex_int = int(hex_str, 16)
+ if not is_unicode_scalar_value(hex_int):
+ raise TOMLDecodeError(
+ "Escaped character is not a Unicode scalar value", src, pos
+ )
+ return pos, chr(hex_int)
+
+
+def parse_literal_str(src: str, pos: Pos) -> tuple[Pos, str]:
+ pos += 1 # Skip starting apostrophe
+ start_pos = pos
+ pos = skip_until(
+ src, pos, "'", error_on=ILLEGAL_LITERAL_STR_CHARS, error_on_eof=True
+ )
+ return pos + 1, src[start_pos:pos] # Skip ending apostrophe
+
+
+def parse_multiline_str(src: str, pos: Pos, *, literal: bool) -> tuple[Pos, str]:
+ pos += 3
+ if src.startswith("\n", pos):
+ pos += 1
+
+ if literal:
+ delim = "'"
+ end_pos = skip_until(
+ src,
+ pos,
+ "'''",
+ error_on=ILLEGAL_MULTILINE_LITERAL_STR_CHARS,
+ error_on_eof=True,
+ )
+ result = src[pos:end_pos]
+ pos = end_pos + 3
+ else:
+ delim = '"'
+ pos, result = parse_basic_str(src, pos, multiline=True)
+
+ # Add at maximum two extra apostrophes/quotes if the end sequence
+ # is 4 or 5 chars long instead of just 3.
+ if not src.startswith(delim, pos):
+ return pos, result
+ pos += 1
+ if not src.startswith(delim, pos):
+ return pos, result + delim
+ pos += 1
+ return pos, result + (delim * 2)
+
+
+def parse_basic_str(src: str, pos: Pos, *, multiline: bool) -> tuple[Pos, str]:
+ if multiline:
+ error_on = ILLEGAL_MULTILINE_BASIC_STR_CHARS
+ parse_escapes = parse_basic_str_escape_multiline
+ else:
+ error_on = ILLEGAL_BASIC_STR_CHARS
+ parse_escapes = parse_basic_str_escape
+ result = ""
+ start_pos = pos
+ while True:
+ try:
+ char = src[pos]
+ except IndexError:
+ raise TOMLDecodeError("Unterminated string", src, pos) from None
+ if char == '"':
+ if not multiline:
+ return pos + 1, result + src[start_pos:pos]
+ if src.startswith('"""', pos):
+ return pos + 3, result + src[start_pos:pos]
+ pos += 1
+ continue
+ if char == "\\":
+ result += src[start_pos:pos]
+ pos, parsed_escape = parse_escapes(src, pos)
+ result += parsed_escape
+ start_pos = pos
+ continue
+ if char in error_on:
+ raise TOMLDecodeError(f"Illegal character {char!r}", src, pos)
+ pos += 1
+
+
+def parse_value(
+ src: str, pos: Pos, parse_float: ParseFloat, nest_lvl: int
+) -> tuple[Pos, Any]:
+ if nest_lvl > MAX_INLINE_NESTING:
+ # Pure Python should have raised RecursionError already.
+ # This ensures mypyc binaries eventually do the same.
+ raise RecursionError( # pragma: no cover
+ "TOML inline arrays/tables are nested more than the allowed"
+ f" {MAX_INLINE_NESTING} levels"
+ )
+
+ try:
+ char: str | None = src[pos]
+ except IndexError:
+ char = None
+
+ # IMPORTANT: order conditions based on speed of checking and likelihood
+
+ # Basic strings
+ if char == '"':
+ if src.startswith('"""', pos):
+ return parse_multiline_str(src, pos, literal=False)
+ return parse_one_line_basic_str(src, pos)
+
+ # Literal strings
+ if char == "'":
+ if src.startswith("'''", pos):
+ return parse_multiline_str(src, pos, literal=True)
+ return parse_literal_str(src, pos)
+
+ # Booleans
+ if char == "t":
+ if src.startswith("true", pos):
+ return pos + 4, True
+ if char == "f":
+ if src.startswith("false", pos):
+ return pos + 5, False
+
+ # Arrays
+ if char == "[":
+ return parse_array(src, pos, parse_float, nest_lvl + 1)
+
+ # Inline tables
+ if char == "{":
+ return parse_inline_table(src, pos, parse_float, nest_lvl + 1)
+
+ # Dates and times
+ datetime_match = RE_DATETIME.match(src, pos)
+ if datetime_match:
+ try:
+ datetime_obj = match_to_datetime(datetime_match)
+ except ValueError as e:
+ raise TOMLDecodeError("Invalid date or datetime", src, pos) from e
+ return datetime_match.end(), datetime_obj
+ localtime_match = RE_LOCALTIME.match(src, pos)
+ if localtime_match:
+ return localtime_match.end(), match_to_localtime(localtime_match)
+
+ # Integers and "normal" floats.
+ # The regex will greedily match any type starting with a decimal
+ # char, so needs to be located after handling of dates and times.
+ number_match = RE_NUMBER.match(src, pos)
+ if number_match:
+ return number_match.end(), match_to_number(number_match, parse_float)
+
+ # Special floats
+ first_three = src[pos : pos + 3]
+ if first_three in {"inf", "nan"}:
+ return pos + 3, parse_float(first_three)
+ first_four = src[pos : pos + 4]
+ if first_four in {"-inf", "+inf", "-nan", "+nan"}:
+ return pos + 4, parse_float(first_four)
+
+ raise TOMLDecodeError("Invalid value", src, pos)
+
+
+def is_unicode_scalar_value(codepoint: int) -> bool:
+ return (0 <= codepoint <= 55295) or (57344 <= codepoint <= 1114111)
+
+
+def make_safe_parse_float(parse_float: ParseFloat) -> ParseFloat:
+ """A decorator to make `parse_float` safe.
+
+ `parse_float` must not return dicts or lists, because these types
+ would be mixed with parsed TOML tables and arrays, thus confusing
+ the parser. The returned decorated callable raises `ValueError`
+ instead of returning illegal types.
+ """
+ # The default `float` callable never returns illegal types. Optimize it.
+ if parse_float is float:
+ return float
+
+ def safe_parse_float(float_str: str) -> Any:
+ float_value = parse_float(float_str)
+ if isinstance(float_value, (dict, list)):
+ raise ValueError("parse_float must not return dicts or lists")
+ return float_value
+
+ return safe_parse_float
diff --git a/bitbake/lib/bb/_vendor/tomli/_re.py b/bitbake/lib/bb/_vendor/tomli/_re.py
new file mode 100644
index 00000000..fc374ed6
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/_re.py
@@ -0,0 +1,119 @@
+# SPDX-License-Identifier: MIT
+# SPDX-FileCopyrightText: 2021 Taneli Hukkinen
+# Licensed to PSF under a Contributor Agreement.
+
+from __future__ import annotations
+
+from datetime import date, datetime, time, timedelta, timezone, tzinfo
+from functools import lru_cache
+import re
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+ from typing import Any, Final
+
+ from ._types import ParseFloat
+
+_TIME_RE_STR: Final = r"""
+([01][0-9]|2[0-3]) # hours
+:([0-5][0-9]) # minutes
+(?:
+ :([0-5][0-9]) # optional seconds
+ (?:\.([0-9]{1,6})[0-9]*)? # optional fractions of a second
+)?
+"""
+
+RE_NUMBER: Final = re.compile(
+ r"""
+0
+(?:
+ x[0-9A-Fa-f](?:_?[0-9A-Fa-f])* # hex
+ |
+ b[01](?:_?[01])* # bin
+ |
+ o[0-7](?:_?[0-7])* # oct
+)
+|
+[+-]?(?:0|[1-9](?:_?[0-9])*) # dec, integer part
+(?P<floatpart>
+ (?:\.[0-9](?:_?[0-9])*)? # optional fractional part
+ (?:[eE][+-]?[0-9](?:_?[0-9])*)? # optional exponent part
+)
+""",
+ flags=re.VERBOSE,
+)
+RE_LOCALTIME: Final = re.compile(_TIME_RE_STR, flags=re.VERBOSE)
+RE_DATETIME: Final = re.compile(
+ rf"""
+([0-9]{{4}})-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01]) # date, e.g. 1988-10-27
+(?:
+ [Tt ]
+ {_TIME_RE_STR}
+ (?:([Zz])|([+-])([01][0-9]|2[0-3]):([0-5][0-9]))? # optional time offset
+)?
+""",
+ flags=re.VERBOSE,
+)
+
+
+def match_to_datetime(match: re.Match[str]) -> datetime | date:
+ """Convert a `RE_DATETIME` match to `datetime.datetime` or `datetime.date`.
+
+ Raises ValueError if the match does not correspond to a valid date
+ or datetime.
+ """
+ (
+ year_str,
+ month_str,
+ day_str,
+ hour_str,
+ minute_str,
+ sec_str,
+ micros_str,
+ zulu_time,
+ offset_sign_str,
+ offset_hour_str,
+ offset_minute_str,
+ ) = match.groups()
+ year, month, day = int(year_str), int(month_str), int(day_str)
+ if hour_str is None:
+ return date(year, month, day)
+ hour, minute = int(hour_str), int(minute_str)
+ sec = int(sec_str) if sec_str else 0
+ micros = int(micros_str.ljust(6, "0")) if micros_str else 0
+ if offset_sign_str:
+ tz: tzinfo | None = cached_tz(
+ offset_hour_str, offset_minute_str, offset_sign_str
+ )
+ elif zulu_time:
+ tz = timezone.utc
+ else: # local date-time
+ tz = None
+ return datetime(year, month, day, hour, minute, sec, micros, tzinfo=tz)
+
+
+# No need to limit cache size. This is only ever called on input
+# that matched RE_DATETIME, so there is an implicit bound of
+# 24 (hours) * 60 (minutes) * 2 (offset direction) = 2880.
+@lru_cache(maxsize=None)
+def cached_tz(hour_str: str, minute_str: str, sign_str: str) -> timezone:
+ sign = 1 if sign_str == "+" else -1
+ return timezone(
+ timedelta(
+ hours=sign * int(hour_str),
+ minutes=sign * int(minute_str),
+ )
+ )
+
+
+def match_to_localtime(match: re.Match[str]) -> time:
+ hour_str, minute_str, sec_str, micros_str = match.groups()
+ sec = int(sec_str) if sec_str else 0
+ micros = int(micros_str.ljust(6, "0")) if micros_str else 0
+ return time(int(hour_str), int(minute_str), sec, micros)
+
+
+def match_to_number(match: re.Match[str], parse_float: ParseFloat) -> Any:
+ if match.group("floatpart"):
+ return parse_float(match.group())
+ return int(match.group(), 0)
diff --git a/bitbake/lib/bb/_vendor/tomli/_types.py b/bitbake/lib/bb/_vendor/tomli/_types.py
new file mode 100644
index 00000000..d949412e
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/_types.py
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: MIT
+# SPDX-FileCopyrightText: 2021 Taneli Hukkinen
+# Licensed to PSF under a Contributor Agreement.
+
+from typing import Any, Callable, Tuple
+
+# Type annotations
+ParseFloat = Callable[[str], Any]
+Key = Tuple[str, ...]
+Pos = int
diff --git a/bitbake/lib/bb/_vendor/tomli/py.typed b/bitbake/lib/bb/_vendor/tomli/py.typed
new file mode 100644
index 00000000..7632ecf7
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/tomli/py.typed
@@ -0,0 +1 @@
+# Marker file for PEP 561
diff --git a/bitbake/lib/bb/_vendor/typing_extensions.LICENSE b/bitbake/lib/bb/_vendor/typing_extensions.LICENSE
new file mode 100644
index 00000000..f26bcf4d
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/typing_extensions.LICENSE
@@ -0,0 +1,279 @@
+A. HISTORY OF THE SOFTWARE
+==========================
+
+Python was created in the early 1990s by Guido van Rossum at Stichting
+Mathematisch Centrum (CWI, see https://www.cwi.nl) in the Netherlands
+as a successor of a language called ABC. Guido remains Python's
+principal author, although it includes many contributions from others.
+
+In 1995, Guido continued his work on Python at the Corporation for
+National Research Initiatives (CNRI, see https://www.cnri.reston.va.us)
+in Reston, Virginia where he released several versions of the
+software.
+
+In May 2000, Guido and the Python core development team moved to
+BeOpen.com to form the BeOpen PythonLabs team. In October of the same
+year, the PythonLabs team moved to Digital Creations, which became
+Zope Corporation. In 2001, the Python Software Foundation (PSF, see
+https://www.python.org/psf/) was formed, a non-profit organization
+created specifically to own Python-related Intellectual Property.
+Zope Corporation was a sponsoring member of the PSF.
+
+All Python releases are Open Source (see https://opensource.org for
+the Open Source Definition). Historically, most, but not all, Python
+releases have also been GPL-compatible; the table below summarizes
+the various releases.
+
+ Release Derived Year Owner GPL-
+ from compatible? (1)
+
+ 0.9.0 thru 1.2 1991-1995 CWI yes
+ 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes
+ 1.6 1.5.2 2000 CNRI no
+ 2.0 1.6 2000 BeOpen.com no
+ 1.6.1 1.6 2001 CNRI yes (2)
+ 2.1 2.0+1.6.1 2001 PSF no
+ 2.0.1 2.0+1.6.1 2001 PSF yes
+ 2.1.1 2.1+2.0.1 2001 PSF yes
+ 2.1.2 2.1.1 2002 PSF yes
+ 2.1.3 2.1.2 2002 PSF yes
+ 2.2 and above 2.1.1 2001-now PSF yes
+
+Footnotes:
+
+(1) GPL-compatible doesn't mean that we're distributing Python under
+ the GPL. All Python licenses, unlike the GPL, let you distribute
+ a modified version without making your changes open source. The
+ GPL-compatible licenses make it possible to combine Python with
+ other software that is released under the GPL; the others don't.
+
+(2) According to Richard Stallman, 1.6.1 is not GPL-compatible,
+ because its license has a choice of law clause. According to
+ CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1
+ is "not incompatible" with the GPL.
+
+Thanks to the many outside volunteers who have worked under Guido's
+direction to make these releases possible.
+
+
+B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON
+===============================================================
+
+Python software and documentation are licensed under the
+Python Software Foundation License Version 2.
+
+Starting with Python 3.8.6, examples, recipes, and other code in
+the documentation are dual licensed under the PSF License Version 2
+and the Zero-Clause BSD license.
+
+Some software incorporated into Python is under different licenses.
+The licenses are listed with code falling under that license.
+
+
+PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
+--------------------------------------------
+
+1. This LICENSE AGREEMENT is between the Python Software Foundation
+("PSF"), and the Individual or Organization ("Licensee") accessing and
+otherwise using this software ("Python") in source or binary form and
+its associated documentation.
+
+2. Subject to the terms and conditions of this License Agreement, PSF hereby
+grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
+analyze, test, perform and/or display publicly, prepare derivative works,
+distribute, and otherwise use Python alone or in any derivative version,
+provided, however, that PSF's License Agreement and PSF's notice of copyright,
+i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
+2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 Python Software Foundation;
+All Rights Reserved" are retained in Python alone or in any derivative version
+prepared by Licensee.
+
+3. In the event Licensee prepares a derivative work that is based on
+or incorporates Python or any part thereof, and wants to make
+the derivative work available to others as provided herein, then
+Licensee hereby agrees to include in any such work a brief summary of
+the changes made to Python.
+
+4. PSF is making Python available to Licensee on an "AS IS"
+basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
+DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
+FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
+INFRINGE ANY THIRD PARTY RIGHTS.
+
+5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
+FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
+A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
+OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
+
+6. This License Agreement will automatically terminate upon a material
+breach of its terms and conditions.
+
+7. Nothing in this License Agreement shall be deemed to create any
+relationship of agency, partnership, or joint venture between PSF and
+Licensee. This License Agreement does not grant permission to use PSF
+trademarks or trade name in a trademark sense to endorse or promote
+products or services of Licensee, or any third party.
+
+8. By copying, installing or otherwise using Python, Licensee
+agrees to be bound by the terms and conditions of this License
+Agreement.
+
+
+BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
+-------------------------------------------
+
+BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1
+
+1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an
+office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
+Individual or Organization ("Licensee") accessing and otherwise using
+this software in source or binary form and its associated
+documentation ("the Software").
+
+2. Subject to the terms and conditions of this BeOpen Python License
+Agreement, BeOpen hereby grants Licensee a non-exclusive,
+royalty-free, world-wide license to reproduce, analyze, test, perform
+and/or display publicly, prepare derivative works, distribute, and
+otherwise use the Software alone or in any derivative version,
+provided, however, that the BeOpen Python License is retained in the
+Software, alone or in any derivative version prepared by Licensee.
+
+3. BeOpen is making the Software available to Licensee on an "AS IS"
+basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND
+DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
+FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT
+INFRINGE ANY THIRD PARTY RIGHTS.
+
+4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
+SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
+AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY
+DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
+
+5. This License Agreement will automatically terminate upon a material
+breach of its terms and conditions.
+
+6. This License Agreement shall be governed by and interpreted in all
+respects by the law of the State of California, excluding conflict of
+law provisions. Nothing in this License Agreement shall be deemed to
+create any relationship of agency, partnership, or joint venture
+between BeOpen and Licensee. This License Agreement does not grant
+permission to use BeOpen trademarks or trade names in a trademark
+sense to endorse or promote products or services of Licensee, or any
+third party. As an exception, the "BeOpen Python" logos available at
+http://www.pythonlabs.com/logos.html may be used according to the
+permissions granted on that web page.
+
+7. By copying, installing or otherwise using the software, Licensee
+agrees to be bound by the terms and conditions of this License
+Agreement.
+
+
+CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1
+---------------------------------------
+
+1. This LICENSE AGREEMENT is between the Corporation for National
+Research Initiatives, having an office at 1895 Preston White Drive,
+Reston, VA 20191 ("CNRI"), and the Individual or Organization
+("Licensee") accessing and otherwise using Python 1.6.1 software in
+source or binary form and its associated documentation.
+
+2. Subject to the terms and conditions of this License Agreement, CNRI
+hereby grants Licensee a nonexclusive, royalty-free, world-wide
+license to reproduce, analyze, test, perform and/or display publicly,
+prepare derivative works, distribute, and otherwise use Python 1.6.1
+alone or in any derivative version, provided, however, that CNRI's
+License Agreement and CNRI's notice of copyright, i.e., "Copyright (c)
+1995-2001 Corporation for National Research Initiatives; All Rights
+Reserved" are retained in Python 1.6.1 alone or in any derivative
+version prepared by Licensee. Alternately, in lieu of CNRI's License
+Agreement, Licensee may substitute the following text (omitting the
+quotes): "Python 1.6.1 is made available subject to the terms and
+conditions in CNRI's License Agreement. This Agreement together with
+Python 1.6.1 may be located on the internet using the following
+unique, persistent identifier (known as a handle): 1895.22/1013. This
+Agreement may also be obtained from a proxy server on the internet
+using the following URL: http://hdl.handle.net/1895.22/1013".
+
+3. In the event Licensee prepares a derivative work that is based on
+or incorporates Python 1.6.1 or any part thereof, and wants to make
+the derivative work available to others as provided herein, then
+Licensee hereby agrees to include in any such work a brief summary of
+the changes made to Python 1.6.1.
+
+4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS"
+basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
+DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
+FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT
+INFRINGE ANY THIRD PARTY RIGHTS.
+
+5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
+1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
+A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1,
+OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
+
+6. This License Agreement will automatically terminate upon a material
+breach of its terms and conditions.
+
+7. This License Agreement shall be governed by the federal
+intellectual property law of the United States, including without
+limitation the federal copyright law, and, to the extent such
+U.S. federal law does not apply, by the law of the Commonwealth of
+Virginia, excluding Virginia's conflict of law provisions.
+Notwithstanding the foregoing, with regard to derivative works based
+on Python 1.6.1 that incorporate non-separable material that was
+previously distributed under the GNU General Public License (GPL), the
+law of the Commonwealth of Virginia shall govern this License
+Agreement only as to issues arising under or with respect to
+Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this
+License Agreement shall be deemed to create any relationship of
+agency, partnership, or joint venture between CNRI and Licensee. This
+License Agreement does not grant permission to use CNRI trademarks or
+trade name in a trademark sense to endorse or promote products or
+services of Licensee, or any third party.
+
+8. By clicking on the "ACCEPT" button where indicated, or by copying,
+installing or otherwise using Python 1.6.1, Licensee agrees to be
+bound by the terms and conditions of this License Agreement.
+
+ ACCEPT
+
+
+CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
+--------------------------------------------------
+
+Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam,
+The Netherlands. All rights reserved.
+
+Permission to use, copy, modify, and distribute this software and its
+documentation for any purpose and without fee is hereby granted,
+provided that the above copyright notice appear in all copies and that
+both that copyright notice and this permission notice appear in
+supporting documentation, and that the name of Stichting Mathematisch
+Centrum or CWI not be used in advertising or publicity pertaining to
+distribution of the software without specific, written prior
+permission.
+
+STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO
+THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
+FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
+OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+ZERO-CLAUSE BSD LICENSE FOR CODE IN THE PYTHON DOCUMENTATION
+----------------------------------------------------------------------
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.
diff --git a/bitbake/lib/bb/_vendor/typing_extensions.py b/bitbake/lib/bb/_vendor/typing_extensions.py
new file mode 100644
index 00000000..51683f22
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/typing_extensions.py
@@ -0,0 +1,4317 @@
+import abc
+import builtins
+import collections
+import collections.abc
+import contextlib
+import enum
+import functools
+import inspect
+import io
+import keyword
+import operator
+import sys
+import types as _types
+import typing
+import warnings
+
+# Breakpoint: https://github.com/python/cpython/pull/119891
+if sys.version_info >= (3, 14):
+ import annotationlib
+
+__all__ = [
+ # Super-special typing primitives.
+ 'Any',
+ 'ClassVar',
+ 'Concatenate',
+ 'Final',
+ 'LiteralString',
+ 'ParamSpec',
+ 'ParamSpecArgs',
+ 'ParamSpecKwargs',
+ 'Self',
+ 'Type',
+ 'TypeVar',
+ 'TypeVarTuple',
+ 'Unpack',
+
+ # ABCs (from collections.abc).
+ 'Awaitable',
+ 'AsyncIterator',
+ 'AsyncIterable',
+ 'Coroutine',
+ 'AsyncGenerator',
+ 'AsyncContextManager',
+ 'Buffer',
+ 'ChainMap',
+
+ # Concrete collection types.
+ 'ContextManager',
+ 'Counter',
+ 'Deque',
+ 'DefaultDict',
+ 'NamedTuple',
+ 'OrderedDict',
+ 'TypedDict',
+
+ # Structural checks, a.k.a. protocols.
+ 'SupportsAbs',
+ 'SupportsBytes',
+ 'SupportsComplex',
+ 'SupportsFloat',
+ 'SupportsIndex',
+ 'SupportsInt',
+ 'SupportsRound',
+ 'Reader',
+ 'Writer',
+
+ # One-off things.
+ 'Annotated',
+ 'assert_never',
+ 'assert_type',
+ 'clear_overloads',
+ 'dataclass_transform',
+ 'deprecated',
+ 'disjoint_base',
+ 'Doc',
+ 'evaluate_forward_ref',
+ 'get_overloads',
+ 'final',
+ 'Format',
+ 'get_annotations',
+ 'get_args',
+ 'get_origin',
+ 'get_original_bases',
+ 'get_protocol_members',
+ 'get_type_hints',
+ 'IntVar',
+ 'is_protocol',
+ 'is_typeddict',
+ 'Literal',
+ 'NewType',
+ 'overload',
+ 'override',
+ 'Protocol',
+ 'Sentinel',
+ 'reveal_type',
+ 'runtime',
+ 'runtime_checkable',
+ 'Text',
+ 'TypeAlias',
+ 'TypeAliasType',
+ 'TypeForm',
+ 'TypeGuard',
+ 'TypeIs',
+ 'TYPE_CHECKING',
+ 'type_repr',
+ 'Never',
+ 'NoReturn',
+ 'ReadOnly',
+ 'Required',
+ 'NotRequired',
+ 'NoDefault',
+ 'NoExtraItems',
+
+ # Pure aliases, have always been in typing
+ 'AbstractSet',
+ 'AnyStr',
+ 'BinaryIO',
+ 'Callable',
+ 'Collection',
+ 'Container',
+ 'Dict',
+ 'ForwardRef',
+ 'FrozenSet',
+ 'Generator',
+ 'Generic',
+ 'Hashable',
+ 'IO',
+ 'ItemsView',
+ 'Iterable',
+ 'Iterator',
+ 'KeysView',
+ 'List',
+ 'Mapping',
+ 'MappingView',
+ 'Match',
+ 'MutableMapping',
+ 'MutableSequence',
+ 'MutableSet',
+ 'Optional',
+ 'Pattern',
+ 'Reversible',
+ 'Sequence',
+ 'Set',
+ 'Sized',
+ 'TextIO',
+ 'Tuple',
+ 'Union',
+ 'ValuesView',
+ 'cast',
+ 'no_type_check',
+ 'no_type_check_decorator',
+]
+
+# for backward compatibility
+PEP_560 = True
+GenericMeta = type
+# Breakpoint: https://github.com/python/cpython/pull/116129
+_PEP_696_IMPLEMENTED = sys.version_info >= (3, 13, 0, "beta")
+
+# Added with bpo-45166 to 3.10.1+ and some 3.9 versions
+_FORWARD_REF_HAS_CLASS = "__forward_is_class__" in typing.ForwardRef.__slots__
+
+# The functions below are modified copies of typing internal helpers.
+# They are needed by _ProtocolMeta and they provide support for PEP 646.
+
+
+class _Sentinel:
+ def __repr__(self):
+ return "<sentinel>"
+
+
+_marker = _Sentinel()
+
+
+# Breakpoint: https://github.com/python/cpython/pull/27342
+if sys.version_info >= (3, 10):
+ def _should_collect_from_parameters(t):
+ return isinstance(
+ t, (typing._GenericAlias, _types.GenericAlias, _types.UnionType)
+ )
+else:
+ def _should_collect_from_parameters(t):
+ return isinstance(t, (typing._GenericAlias, _types.GenericAlias))
+
+
+NoReturn = typing.NoReturn
+
+# Some unconstrained type variables. These are used by the container types.
+# (These are not for export.)
+T = typing.TypeVar('T') # Any type.
+KT = typing.TypeVar('KT') # Key type.
+VT = typing.TypeVar('VT') # Value type.
+T_co = typing.TypeVar('T_co', covariant=True) # Any type covariant containers.
+T_contra = typing.TypeVar('T_contra', contravariant=True) # Ditto contravariant.
+
+
+# Breakpoint: https://github.com/python/cpython/pull/31841
+if sys.version_info >= (3, 11):
+ from typing import Any
+else:
+
+ class _AnyMeta(type):
+ def __instancecheck__(self, obj):
+ if self is Any:
+ raise TypeError("typing_extensions.Any cannot be used with isinstance()")
+ return super().__instancecheck__(obj)
+
+ def __repr__(self):
+ if self is Any:
+ return "typing_extensions.Any"
+ return super().__repr__()
+
+ class Any(metaclass=_AnyMeta):
+ """Special type indicating an unconstrained type.
+ - Any is compatible with every type.
+ - Any assumed to have all methods.
+ - All values assumed to be instances of Any.
+ Note that all the above statements are true from the point of view of
+ static type checkers. At runtime, Any should not be used with instance
+ checks.
+ """
+ def __new__(cls, *args, **kwargs):
+ if cls is Any:
+ raise TypeError("Any cannot be instantiated")
+ return super().__new__(cls, *args, **kwargs)
+
+
+ClassVar = typing.ClassVar
+
+# Vendored from cpython typing._SpecialFrom
+# Having a separate class means that instances will not be rejected by
+# typing._type_check.
+class _SpecialForm(typing._Final, _root=True):
+ __slots__ = ('_name', '__doc__', '_getitem')
+
+ def __init__(self, getitem):
+ self._getitem = getitem
+ self._name = getitem.__name__
+ self.__doc__ = getitem.__doc__
+
+ def __getattr__(self, item):
+ if item in {'__name__', '__qualname__'}:
+ return self._name
+
+ raise AttributeError(item)
+
+ def __mro_entries__(self, bases):
+ raise TypeError(f"Cannot subclass {self!r}")
+
+ def __repr__(self):
+ return f'typing_extensions.{self._name}'
+
+ def __reduce__(self):
+ return self._name
+
+ def __call__(self, *args, **kwds):
+ raise TypeError(f"Cannot instantiate {self!r}")
+
+ def __or__(self, other):
+ return typing.Union[self, other]
+
+ def __ror__(self, other):
+ return typing.Union[other, self]
+
+ def __instancecheck__(self, obj):
+ raise TypeError(f"{self} cannot be used with isinstance()")
+
+ def __subclasscheck__(self, cls):
+ raise TypeError(f"{self} cannot be used with issubclass()")
+
+ @typing._tp_cache
+ def __getitem__(self, parameters):
+ return self._getitem(self, parameters)
+
+
+# Note that inheriting from this class means that the object will be
+# rejected by typing._type_check, so do not use it if the special form
+# is arguably valid as a type by itself.
+class _ExtensionsSpecialForm(typing._SpecialForm, _root=True):
+ def __repr__(self):
+ return 'typing_extensions.' + self._name
+
+
+Final = typing.Final
+
+# Breakpoint: https://github.com/python/cpython/pull/30530
+if sys.version_info >= (3, 11):
+ final = typing.final
+else:
+ # @final exists in 3.8+, but we backport it for all versions
+ # before 3.11 to keep support for the __final__ attribute.
+ # See https://bugs.python.org/issue46342
+ def final(f):
+ """This decorator can be used to indicate to type checkers that
+ the decorated method cannot be overridden, and decorated class
+ cannot be subclassed. For example:
+
+ class Base:
+ @final
+ def done(self) -> None:
+ ...
+ class Sub(Base):
+ def done(self) -> None: # Error reported by type checker
+ ...
+ @final
+ class Leaf:
+ ...
+ class Other(Leaf): # Error reported by type checker
+ ...
+
+ There is no runtime checking of these properties. The decorator
+ sets the ``__final__`` attribute to ``True`` on the decorated object
+ to allow runtime introspection.
+ """
+ try:
+ f.__final__ = True
+ except (AttributeError, TypeError):
+ # Skip the attribute silently if it is not writable.
+ # AttributeError happens if the object has __slots__ or a
+ # read-only property, TypeError if it's a builtin class.
+ pass
+ return f
+
+
+if hasattr(typing, "disjoint_base"): # 3.15
+ disjoint_base = typing.disjoint_base
+else:
+ def disjoint_base(cls):
+ """This decorator marks a class as a disjoint base.
+
+ Child classes of a disjoint base cannot inherit from other disjoint bases that are
+ not parent classes of the disjoint base.
+
+ For example:
+
+ @disjoint_base
+ class Disjoint1: pass
+
+ @disjoint_base
+ class Disjoint2: pass
+
+ class Disjoint3(Disjoint1, Disjoint2): pass # Type checker error
+
+ Type checkers can use knowledge of disjoint bases to detect unreachable code
+ and determine when two types can overlap.
+
+ See PEP 800."""
+ cls.__disjoint_base__ = True
+ return cls
+
+
+def IntVar(name):
+ return typing.TypeVar(name)
+
+
+# A Literal bug was fixed in 3.11.0, 3.10.1 and 3.9.8
+# Breakpoint: https://github.com/python/cpython/pull/29334
+if sys.version_info >= (3, 10, 1):
+ Literal = typing.Literal
+else:
+ def _flatten_literal_params(parameters):
+ """An internal helper for Literal creation: flatten Literals among parameters"""
+ params = []
+ for p in parameters:
+ if isinstance(p, _LiteralGenericAlias):
+ params.extend(p.__args__)
+ else:
+ params.append(p)
+ return tuple(params)
+
+ def _value_and_type_iter(params):
+ for p in params:
+ yield p, type(p)
+
+ class _LiteralGenericAlias(typing._GenericAlias, _root=True):
+ def __eq__(self, other):
+ if not isinstance(other, _LiteralGenericAlias):
+ return NotImplemented
+ these_args_deduped = set(_value_and_type_iter(self.__args__))
+ other_args_deduped = set(_value_and_type_iter(other.__args__))
+ return these_args_deduped == other_args_deduped
+
+ def __hash__(self):
+ return hash(frozenset(_value_and_type_iter(self.__args__)))
+
+ class _LiteralForm(_ExtensionsSpecialForm, _root=True):
+ def __init__(self, doc: str):
+ self._name = 'Literal'
+ self._doc = self.__doc__ = doc
+
+ def __getitem__(self, parameters):
+ if not isinstance(parameters, tuple):
+ parameters = (parameters,)
+
+ parameters = _flatten_literal_params(parameters)
+
+ val_type_pairs = list(_value_and_type_iter(parameters))
+ try:
+ deduped_pairs = set(val_type_pairs)
+ except TypeError:
+ # unhashable parameters
+ pass
+ else:
+ # similar logic to typing._deduplicate on Python 3.9+
+ if len(deduped_pairs) < len(val_type_pairs):
+ new_parameters = []
+ for pair in val_type_pairs:
+ if pair in deduped_pairs:
+ new_parameters.append(pair[0])
+ deduped_pairs.remove(pair)
+ assert not deduped_pairs, deduped_pairs
+ parameters = tuple(new_parameters)
+
+ return _LiteralGenericAlias(self, parameters)
+
+ Literal = _LiteralForm(doc="""\
+ A type that can be used to indicate to type checkers
+ that the corresponding value has a value literally equivalent
+ to the provided parameter. For example:
+
+ var: Literal[4] = 4
+
+ The type checker understands that 'var' is literally equal to
+ the value 4 and no other value.
+
+ Literal[...] cannot be subclassed. There is no runtime
+ checking verifying that the parameter is actually a value
+ instead of a type.""")
+
+
+_overload_dummy = typing._overload_dummy
+
+
+if hasattr(typing, "get_overloads"): # 3.11+
+ overload = typing.overload
+ get_overloads = typing.get_overloads
+ clear_overloads = typing.clear_overloads
+else:
+ # {module: {qualname: {firstlineno: func}}}
+ _overload_registry = collections.defaultdict(
+ functools.partial(collections.defaultdict, dict)
+ )
+
+ def overload(func):
+ """Decorator for overloaded functions/methods.
+
+ In a stub file, place two or more stub definitions for the same
+ function in a row, each decorated with @overload. For example:
+
+ @overload
+ def utf8(value: None) -> None: ...
+ @overload
+ def utf8(value: bytes) -> bytes: ...
+ @overload
+ def utf8(value: str) -> bytes: ...
+
+ In a non-stub file (i.e. a regular .py file), do the same but
+ follow it with an implementation. The implementation should *not*
+ be decorated with @overload. For example:
+
+ @overload
+ def utf8(value: None) -> None: ...
+ @overload
+ def utf8(value: bytes) -> bytes: ...
+ @overload
+ def utf8(value: str) -> bytes: ...
+ def utf8(value):
+ # implementation goes here
+
+ The overloads for a function can be retrieved at runtime using the
+ get_overloads() function.
+ """
+ # classmethod and staticmethod
+ f = getattr(func, "__func__", func)
+ try:
+ _overload_registry[f.__module__][f.__qualname__][
+ f.__code__.co_firstlineno
+ ] = func
+ except AttributeError:
+ # Not a normal function; ignore.
+ pass
+ return _overload_dummy
+
+ def get_overloads(func):
+ """Return all defined overloads for *func* as a sequence."""
+ # classmethod and staticmethod
+ f = getattr(func, "__func__", func)
+ if f.__module__ not in _overload_registry:
+ return []
+ mod_dict = _overload_registry[f.__module__]
+ if f.__qualname__ not in mod_dict:
+ return []
+ return list(mod_dict[f.__qualname__].values())
+
+ def clear_overloads():
+ """Clear all overloads in the registry."""
+ _overload_registry.clear()
+
+
+# This is not a real generic class. Don't use outside annotations.
+Type = typing.Type
+
+# Various ABCs mimicking those in collections.abc.
+# A few are simply re-exported for completeness.
+Awaitable = typing.Awaitable
+Coroutine = typing.Coroutine
+AsyncIterable = typing.AsyncIterable
+AsyncIterator = typing.AsyncIterator
+Deque = typing.Deque
+DefaultDict = typing.DefaultDict
+OrderedDict = typing.OrderedDict
+Counter = typing.Counter
+ChainMap = typing.ChainMap
+Text = typing.Text
+TYPE_CHECKING = typing.TYPE_CHECKING
+
+
+# Breakpoint: https://github.com/python/cpython/pull/118681
+if sys.version_info >= (3, 13, 0, "beta"):
+ from typing import AsyncContextManager, AsyncGenerator, ContextManager, Generator
+else:
+ def _is_dunder(attr):
+ return attr.startswith('__') and attr.endswith('__')
+
+
+ class _SpecialGenericAlias(typing._SpecialGenericAlias, _root=True):
+ def __init__(self, origin, nparams, *, inst=True, name=None, defaults=()):
+ super().__init__(origin, nparams, inst=inst, name=name)
+ self._defaults = defaults
+
+ def __setattr__(self, attr, val):
+ allowed_attrs = {'_name', '_inst', '_nparams', '_defaults'}
+ if _is_dunder(attr) or attr in allowed_attrs:
+ object.__setattr__(self, attr, val)
+ else:
+ setattr(self.__origin__, attr, val)
+
+ @typing._tp_cache
+ def __getitem__(self, params):
+ if not isinstance(params, tuple):
+ params = (params,)
+ msg = "Parameters to generic types must be types."
+ params = tuple(typing._type_check(p, msg) for p in params)
+ if (
+ self._defaults
+ and len(params) < self._nparams
+ and len(params) + len(self._defaults) >= self._nparams
+ ):
+ params = (*params, *self._defaults[len(params) - self._nparams:])
+ actual_len = len(params)
+
+ if actual_len != self._nparams:
+ if self._defaults:
+ expected = f"at least {self._nparams - len(self._defaults)}"
+ else:
+ expected = str(self._nparams)
+ if not self._nparams:
+ raise TypeError(f"{self} is not a generic class")
+ raise TypeError(
+ f"Too {'many' if actual_len > self._nparams else 'few'}"
+ f" arguments for {self};"
+ f" actual {actual_len}, expected {expected}"
+ )
+ return self.copy_with(params)
+
+ _NoneType = type(None)
+ Generator = _SpecialGenericAlias(
+ collections.abc.Generator, 3, defaults=(_NoneType, _NoneType)
+ )
+ AsyncGenerator = _SpecialGenericAlias(
+ collections.abc.AsyncGenerator, 2, defaults=(_NoneType,)
+ )
+ ContextManager = _SpecialGenericAlias(
+ contextlib.AbstractContextManager,
+ 2,
+ name="ContextManager",
+ defaults=(typing.Optional[bool],)
+ )
+ AsyncContextManager = _SpecialGenericAlias(
+ contextlib.AbstractAsyncContextManager,
+ 2,
+ name="AsyncContextManager",
+ defaults=(typing.Optional[bool],)
+ )
+
+
+_PROTO_ALLOWLIST = {
+ 'collections.abc': [
+ 'Callable', 'Awaitable', 'Iterable', 'Iterator', 'AsyncIterable',
+ 'Hashable', 'Sized', 'Container', 'Collection', 'Reversible', 'Buffer',
+ ],
+ 'contextlib': ['AbstractContextManager', 'AbstractAsyncContextManager'],
+ 'typing_extensions': ['Buffer'],
+}
+
+
+_EXCLUDED_ATTRS = frozenset(typing.EXCLUDED_ATTRIBUTES) | {
+ "__match_args__", "__protocol_attrs__", "__non_callable_proto_members__",
+ "__final__",
+}
+
+
+def _get_protocol_attrs(cls):
+ attrs = set()
+ for base in cls.__mro__[:-1]: # without object
+ if base.__name__ in {'Protocol', 'Generic'}:
+ continue
+ annotations = getattr(base, '__annotations__', {})
+ for attr in (*base.__dict__, *annotations):
+ if (not attr.startswith('_abc_') and attr not in _EXCLUDED_ATTRS):
+ attrs.add(attr)
+ return attrs
+
+
+def _caller(depth=1, default='__main__'):
+ try:
+ return sys._getframemodulename(depth + 1) or default
+ except AttributeError: # For platforms without _getframemodulename()
+ pass
+ try:
+ return sys._getframe(depth + 1).f_globals.get('__name__', default)
+ except (AttributeError, ValueError): # For platforms without _getframe()
+ pass
+ return None
+
+
+# `__match_args__` attribute was removed from protocol members in 3.13,
+# we want to backport this change to older Python versions.
+# Breakpoint: https://github.com/python/cpython/pull/110683
+if sys.version_info >= (3, 13):
+ Protocol = typing.Protocol
+else:
+ def _allow_reckless_class_checks(depth=2):
+ """Allow instance and class checks for special stdlib modules.
+ The abc and functools modules indiscriminately call isinstance() and
+ issubclass() on the whole MRO of a user class, which may contain protocols.
+ """
+ return _caller(depth) in {'abc', 'functools', None}
+
+ def _no_init(self, *args, **kwargs):
+ if type(self)._is_protocol:
+ raise TypeError('Protocols cannot be instantiated')
+
+ def _type_check_issubclass_arg_1(arg):
+ """Raise TypeError if `arg` is not an instance of `type`
+ in `issubclass(arg, <protocol>)`.
+
+ In most cases, this is verified by type.__subclasscheck__.
+ Checking it again unnecessarily would slow down issubclass() checks,
+ so, we don't perform this check unless we absolutely have to.
+
+ For various error paths, however,
+ we want to ensure that *this* error message is shown to the user
+ where relevant, rather than a typing.py-specific error message.
+ """
+ if not isinstance(arg, type):
+ # Same error message as for issubclass(1, int).
+ raise TypeError('issubclass() arg 1 must be a class')
+
+ # Inheriting from typing._ProtocolMeta isn't actually desirable,
+ # but is necessary to allow typing.Protocol and typing_extensions.Protocol
+ # to mix without getting TypeErrors about "metaclass conflict"
+ class _ProtocolMeta(type(typing.Protocol)):
+ # This metaclass is somewhat unfortunate,
+ # but is necessary for several reasons...
+ #
+ # NOTE: DO NOT call super() in any methods in this class
+ # That would call the methods on typing._ProtocolMeta on Python <=3.11
+ # and those are slow
+ def __new__(mcls, name, bases, namespace, **kwargs):
+ if name == "Protocol" and len(bases) < 2:
+ pass
+ elif {Protocol, typing.Protocol} & set(bases):
+ for base in bases:
+ if not (
+ base in {object, typing.Generic, Protocol, typing.Protocol}
+ or base.__name__ in _PROTO_ALLOWLIST.get(base.__module__, [])
+ or is_protocol(base)
+ ):
+ raise TypeError(
+ f"Protocols can only inherit from other protocols, "
+ f"got {base!r}"
+ )
+ return abc.ABCMeta.__new__(mcls, name, bases, namespace, **kwargs)
+
+ def __init__(cls, *args, **kwargs):
+ abc.ABCMeta.__init__(cls, *args, **kwargs)
+ if getattr(cls, "_is_protocol", False):
+ cls.__protocol_attrs__ = _get_protocol_attrs(cls)
+
+ def __subclasscheck__(cls, other):
+ if cls is Protocol:
+ return type.__subclasscheck__(cls, other)
+ if (
+ getattr(cls, '_is_protocol', False)
+ and not _allow_reckless_class_checks()
+ ):
+ if not getattr(cls, '_is_runtime_protocol', False):
+ _type_check_issubclass_arg_1(other)
+ raise TypeError(
+ "Instance and class checks can only be used with "
+ "@runtime_checkable protocols"
+ )
+ if (
+ # this attribute is set by @runtime_checkable:
+ cls.__non_callable_proto_members__
+ and cls.__dict__.get("__subclasshook__") is _proto_hook
+ ):
+ _type_check_issubclass_arg_1(other)
+ non_method_attrs = sorted(cls.__non_callable_proto_members__)
+ raise TypeError(
+ "Protocols with non-method members don't support issubclass()."
+ f" Non-method members: {str(non_method_attrs)[1:-1]}."
+ )
+ return abc.ABCMeta.__subclasscheck__(cls, other)
+
+ def __instancecheck__(cls, instance):
+ # We need this method for situations where attributes are
+ # assigned in __init__.
+ if cls is Protocol:
+ return type.__instancecheck__(cls, instance)
+ if not getattr(cls, "_is_protocol", False):
+ # i.e., it's a concrete subclass of a protocol
+ return abc.ABCMeta.__instancecheck__(cls, instance)
+
+ if (
+ not getattr(cls, '_is_runtime_protocol', False) and
+ not _allow_reckless_class_checks()
+ ):
+ raise TypeError("Instance and class checks can only be used with"
+ " @runtime_checkable protocols")
+
+ if abc.ABCMeta.__instancecheck__(cls, instance):
+ return True
+
+ for attr in cls.__protocol_attrs__:
+ try:
+ val = inspect.getattr_static(instance, attr)
+ except AttributeError:
+ break
+ # this attribute is set by @runtime_checkable:
+ if val is None and attr not in cls.__non_callable_proto_members__:
+ break
+ else:
+ return True
+
+ return False
+
+ def __eq__(cls, other):
+ # Hack so that typing.Generic.__class_getitem__
+ # treats typing_extensions.Protocol
+ # as equivalent to typing.Protocol
+ if abc.ABCMeta.__eq__(cls, other) is True:
+ return True
+ return cls is Protocol and other is typing.Protocol
+
+ # This has to be defined, or the abc-module cache
+ # complains about classes with this metaclass being unhashable,
+ # if we define only __eq__!
+ def __hash__(cls) -> int:
+ return type.__hash__(cls)
+
+ @classmethod
+ def _proto_hook(cls, other):
+ if not cls.__dict__.get('_is_protocol', False):
+ return NotImplemented
+
+ for attr in cls.__protocol_attrs__:
+ for base in other.__mro__:
+ # Check if the members appears in the class dictionary...
+ if attr in base.__dict__:
+ if base.__dict__[attr] is None:
+ return NotImplemented
+ break
+
+ # ...or in annotations, if it is a sub-protocol.
+ annotations = getattr(base, '__annotations__', {})
+ if (
+ isinstance(annotations, collections.abc.Mapping)
+ and attr in annotations
+ and is_protocol(other)
+ ):
+ break
+ else:
+ return NotImplemented
+ return True
+
+ class Protocol(typing.Generic, metaclass=_ProtocolMeta):
+ __doc__ = typing.Protocol.__doc__
+ __slots__ = ()
+ _is_protocol = True
+ _is_runtime_protocol = False
+
+ def __init_subclass__(cls, *args, **kwargs):
+ super().__init_subclass__(*args, **kwargs)
+
+ # Determine if this is a protocol or a concrete subclass.
+ if not cls.__dict__.get('_is_protocol', False):
+ cls._is_protocol = any(b is Protocol for b in cls.__bases__)
+
+ # Set (or override) the protocol subclass hook.
+ if '__subclasshook__' not in cls.__dict__:
+ cls.__subclasshook__ = _proto_hook
+
+ # Prohibit instantiation for protocol classes
+ if cls._is_protocol and cls.__init__ is Protocol.__init__:
+ cls.__init__ = _no_init
+
+
+# Breakpoint: https://github.com/python/cpython/pull/113401
+if sys.version_info >= (3, 13):
+ runtime_checkable = typing.runtime_checkable
+else:
+ def runtime_checkable(cls):
+ """Mark a protocol class as a runtime protocol.
+
+ Such protocol can be used with isinstance() and issubclass().
+ Raise TypeError if applied to a non-protocol class.
+ This allows a simple-minded structural check very similar to
+ one trick ponies in collections.abc such as Iterable.
+
+ For example::
+
+ @runtime_checkable
+ class Closable(Protocol):
+ def close(self): ...
+
+ assert isinstance(open('/some/file'), Closable)
+
+ Warning: this will check only the presence of the required methods,
+ not their type signatures!
+ """
+ if not issubclass(cls, typing.Generic) or not getattr(cls, '_is_protocol', False):
+ raise TypeError(f'@runtime_checkable can be only applied to protocol classes,'
+ f' got {cls!r}')
+ cls._is_runtime_protocol = True
+
+ # typing.Protocol classes on <=3.11 break if we execute this block,
+ # because typing.Protocol classes on <=3.11 don't have a
+ # `__protocol_attrs__` attribute, and this block relies on the
+ # `__protocol_attrs__` attribute. Meanwhile, typing.Protocol classes on 3.12.2+
+ # break if we *don't* execute this block, because *they* assume that all
+ # protocol classes have a `__non_callable_proto_members__` attribute
+ # (which this block sets)
+ if isinstance(cls, _ProtocolMeta) or sys.version_info >= (3, 12, 2):
+ # PEP 544 prohibits using issubclass()
+ # with protocols that have non-method members.
+ # See gh-113320 for why we compute this attribute here,
+ # rather than in `_ProtocolMeta.__init__`
+ cls.__non_callable_proto_members__ = set()
+ for attr in cls.__protocol_attrs__:
+ try:
+ is_callable = callable(getattr(cls, attr, None))
+ except Exception as e:
+ raise TypeError(
+ f"Failed to determine whether protocol member {attr!r} "
+ "is a method member"
+ ) from e
+ else:
+ if not is_callable:
+ cls.__non_callable_proto_members__.add(attr)
+
+ return cls
+
+
+# The "runtime" alias exists for backwards compatibility.
+runtime = runtime_checkable
+
+
+# Our version of runtime-checkable protocols is faster on Python <=3.11
+# Breakpoint: https://github.com/python/cpython/pull/112717
+if sys.version_info >= (3, 12):
+ SupportsInt = typing.SupportsInt
+ SupportsFloat = typing.SupportsFloat
+ SupportsComplex = typing.SupportsComplex
+ SupportsBytes = typing.SupportsBytes
+ SupportsIndex = typing.SupportsIndex
+ SupportsAbs = typing.SupportsAbs
+ SupportsRound = typing.SupportsRound
+else:
+ @runtime_checkable
+ class SupportsInt(Protocol):
+ """An ABC with one abstract method __int__."""
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __int__(self) -> int:
+ pass
+
+ @runtime_checkable
+ class SupportsFloat(Protocol):
+ """An ABC with one abstract method __float__."""
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __float__(self) -> float:
+ pass
+
+ @runtime_checkable
+ class SupportsComplex(Protocol):
+ """An ABC with one abstract method __complex__."""
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __complex__(self) -> complex:
+ pass
+
+ @runtime_checkable
+ class SupportsBytes(Protocol):
+ """An ABC with one abstract method __bytes__."""
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __bytes__(self) -> bytes:
+ pass
+
+ @runtime_checkable
+ class SupportsIndex(Protocol):
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __index__(self) -> int:
+ pass
+
+ @runtime_checkable
+ class SupportsAbs(Protocol[T_co]):
+ """
+ An ABC with one abstract method __abs__ that is covariant in its return type.
+ """
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __abs__(self) -> T_co:
+ pass
+
+ @runtime_checkable
+ class SupportsRound(Protocol[T_co]):
+ """
+ An ABC with one abstract method __round__ that is covariant in its return type.
+ """
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def __round__(self, ndigits: int = 0) -> T_co:
+ pass
+
+
+if hasattr(io, "Reader") and hasattr(io, "Writer"):
+ Reader = io.Reader
+ Writer = io.Writer
+else:
+ @runtime_checkable
+ class Reader(Protocol[T_co]):
+ """Protocol for simple I/O reader instances.
+
+ This protocol only supports blocking I/O.
+ """
+
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def read(self, size: int = ..., /) -> T_co:
+ """Read data from the input stream and return it.
+
+ If *size* is specified, at most *size* items (bytes/characters) will be
+ read.
+ """
+
+ @runtime_checkable
+ class Writer(Protocol[T_contra]):
+ """Protocol for simple I/O writer instances.
+
+ This protocol only supports blocking I/O.
+ """
+
+ __slots__ = ()
+
+ @abc.abstractmethod
+ def write(self, data: T_contra, /) -> int:
+ """Write *data* to the output stream and return the number of items written.""" # noqa: E501
+
+
+_NEEDS_SINGLETONMETA = (
+ not hasattr(typing, "NoDefault") or not hasattr(typing, "NoExtraItems")
+)
+
+if _NEEDS_SINGLETONMETA:
+ class SingletonMeta(type):
+ def __setattr__(cls, attr, value):
+ # TypeError is consistent with the behavior of NoneType
+ raise TypeError(
+ f"cannot set {attr!r} attribute of immutable type {cls.__name__!r}"
+ )
+
+
+if hasattr(typing, "NoDefault"):
+ NoDefault = typing.NoDefault
+else:
+ class NoDefaultType(metaclass=SingletonMeta):
+ """The type of the NoDefault singleton."""
+
+ __slots__ = ()
+
+ def __new__(cls):
+ return globals().get("NoDefault") or object.__new__(cls)
+
+ def __repr__(self):
+ return "typing_extensions.NoDefault"
+
+ def __reduce__(self):
+ return "NoDefault"
+
+ NoDefault = NoDefaultType()
+ del NoDefaultType
+
+if hasattr(typing, "NoExtraItems"):
+ NoExtraItems = typing.NoExtraItems
+else:
+ class NoExtraItemsType(metaclass=SingletonMeta):
+ """The type of the NoExtraItems singleton."""
+
+ __slots__ = ()
+
+ def __new__(cls):
+ return globals().get("NoExtraItems") or object.__new__(cls)
+
+ def __repr__(self):
+ return "typing_extensions.NoExtraItems"
+
+ def __reduce__(self):
+ return "NoExtraItems"
+
+ NoExtraItems = NoExtraItemsType()
+ del NoExtraItemsType
+
+if _NEEDS_SINGLETONMETA:
+ del SingletonMeta
+
+
+# Update this to something like >=3.13.0b1 if and when
+# PEP 728 is implemented in CPython
+_PEP_728_IMPLEMENTED = False
+
+if _PEP_728_IMPLEMENTED:
+ # The standard library TypedDict in Python 3.9.0/1 does not honour the "total"
+ # keyword with old-style TypedDict(). See https://bugs.python.org/issue42059
+ # The standard library TypedDict below Python 3.11 does not store runtime
+ # information about optional and required keys when using Required or NotRequired.
+ # Generic TypedDicts are also impossible using typing.TypedDict on Python <3.11.
+ # Aaaand on 3.12 we add __orig_bases__ to TypedDict
+ # to enable better runtime introspection.
+ # On 3.13 we deprecate some odd ways of creating TypedDicts.
+ # Also on 3.13, PEP 705 adds the ReadOnly[] qualifier.
+ # PEP 728 (still pending) makes more changes.
+ TypedDict = typing.TypedDict
+ _TypedDictMeta = typing._TypedDictMeta
+ is_typeddict = typing.is_typeddict
+else:
+ # 3.10.0 and later
+ _TAKES_MODULE = "module" in inspect.signature(typing._type_check).parameters
+
+ def _get_typeddict_qualifiers(annotation_type):
+ while True:
+ annotation_origin = get_origin(annotation_type)
+ if annotation_origin is Annotated:
+ annotation_args = get_args(annotation_type)
+ if annotation_args:
+ annotation_type = annotation_args[0]
+ else:
+ break
+ elif annotation_origin is Required:
+ yield Required
+ annotation_type, = get_args(annotation_type)
+ elif annotation_origin is NotRequired:
+ yield NotRequired
+ annotation_type, = get_args(annotation_type)
+ elif annotation_origin is ReadOnly:
+ yield ReadOnly
+ annotation_type, = get_args(annotation_type)
+ else:
+ break
+
+ class _TypedDictMeta(type):
+
+ def __new__(cls, name, bases, ns, *, total=True, closed=None,
+ extra_items=NoExtraItems):
+ """Create new typed dict class object.
+
+ This method is called when TypedDict is subclassed,
+ or when TypedDict is instantiated. This way
+ TypedDict supports all three syntax forms described in its docstring.
+ Subclasses and instances of TypedDict return actual dictionaries.
+ """
+ for base in bases:
+ if type(base) is not _TypedDictMeta and base is not typing.Generic:
+ raise TypeError('cannot inherit from both a TypedDict type '
+ 'and a non-TypedDict base class')
+ if closed is not None and extra_items is not NoExtraItems:
+ raise TypeError(f"Cannot combine closed={closed!r} and extra_items")
+
+ if any(issubclass(b, typing.Generic) for b in bases):
+ generic_base = (typing.Generic,)
+ else:
+ generic_base = ()
+
+ ns_annotations = ns.pop('__annotations__', None)
+
+ # typing.py generally doesn't let you inherit from plain Generic, unless
+ # the name of the class happens to be "Protocol"
+ tp_dict = type.__new__(_TypedDictMeta, "Protocol", (*generic_base, dict), ns)
+ tp_dict.__name__ = name
+ if tp_dict.__qualname__ == "Protocol":
+ tp_dict.__qualname__ = name
+
+ if not hasattr(tp_dict, '__orig_bases__'):
+ tp_dict.__orig_bases__ = bases
+
+ annotations = {}
+ own_annotate = None
+ if ns_annotations is not None:
+ own_annotations = ns_annotations
+ elif sys.version_info >= (3, 14):
+ if hasattr(annotationlib, "get_annotate_from_class_namespace"):
+ own_annotate = annotationlib.get_annotate_from_class_namespace(ns)
+ else:
+ # 3.14.0a7 and earlier
+ own_annotate = ns.get("__annotate__")
+ if own_annotate is not None:
+ own_annotations = annotationlib.call_annotate_function(
+ own_annotate, Format.FORWARDREF, owner=tp_dict
+ )
+ else:
+ own_annotations = {}
+ else:
+ own_annotations = {}
+ msg = "TypedDict('Name', {f0: t0, f1: t1, ...}); each t must be a type"
+ if _TAKES_MODULE:
+ own_checked_annotations = {
+ n: typing._type_check(tp, msg, module=tp_dict.__module__)
+ for n, tp in own_annotations.items()
+ }
+ else:
+ own_checked_annotations = {
+ n: typing._type_check(tp, msg)
+ for n, tp in own_annotations.items()
+ }
+ required_keys = set()
+ optional_keys = set()
+ readonly_keys = set()
+ mutable_keys = set()
+ extra_items_type = extra_items
+
+ for base in bases:
+ base_dict = base.__dict__
+
+ if sys.version_info <= (3, 14):
+ annotations.update(base_dict.get('__annotations__', {}))
+ required_keys.update(base_dict.get('__required_keys__', ()))
+ optional_keys.update(base_dict.get('__optional_keys__', ()))
+ readonly_keys.update(base_dict.get('__readonly_keys__', ()))
+ mutable_keys.update(base_dict.get('__mutable_keys__', ()))
+
+ # This was specified in an earlier version of PEP 728. Support
+ # is retained for backwards compatibility, but only for Python
+ # 3.13 and lower.
+ if (closed and sys.version_info < (3, 14)
+ and "__extra_items__" in own_checked_annotations):
+ annotation_type = own_checked_annotations.pop("__extra_items__")
+ qualifiers = set(_get_typeddict_qualifiers(annotation_type))
+ if Required in qualifiers:
+ raise TypeError(
+ "Special key __extra_items__ does not support "
+ "Required"
+ )
+ if NotRequired in qualifiers:
+ raise TypeError(
+ "Special key __extra_items__ does not support "
+ "NotRequired"
+ )
+ extra_items_type = annotation_type
+
+ annotations.update(own_checked_annotations)
+ for annotation_key, annotation_type in own_checked_annotations.items():
+ qualifiers = set(_get_typeddict_qualifiers(annotation_type))
+
+ if Required in qualifiers:
+ required_keys.add(annotation_key)
+ elif NotRequired in qualifiers:
+ optional_keys.add(annotation_key)
+ elif total:
+ required_keys.add(annotation_key)
+ else:
+ optional_keys.add(annotation_key)
+ if ReadOnly in qualifiers:
+ mutable_keys.discard(annotation_key)
+ readonly_keys.add(annotation_key)
+ else:
+ mutable_keys.add(annotation_key)
+ readonly_keys.discard(annotation_key)
+
+ # Breakpoint: https://github.com/python/cpython/pull/119891
+ if sys.version_info >= (3, 14):
+ def __annotate__(format):
+ annos = {}
+ for base in bases:
+ if base is Generic:
+ continue
+ base_annotate = base.__annotate__
+ if base_annotate is None:
+ continue
+ base_annos = annotationlib.call_annotate_function(
+ base_annotate, format, owner=base)
+ annos.update(base_annos)
+ if own_annotate is not None:
+ own = annotationlib.call_annotate_function(
+ own_annotate, format, owner=tp_dict)
+ if format != Format.STRING:
+ own = {
+ n: typing._type_check(tp, msg, module=tp_dict.__module__)
+ for n, tp in own.items()
+ }
+ elif format == Format.STRING:
+ own = annotationlib.annotations_to_string(own_annotations)
+ elif format in (Format.FORWARDREF, Format.VALUE):
+ own = own_checked_annotations
+ else:
+ raise NotImplementedError(format)
+ annos.update(own)
+ return annos
+
+ tp_dict.__annotate__ = __annotate__
+ else:
+ tp_dict.__annotations__ = annotations
+ tp_dict.__required_keys__ = frozenset(required_keys)
+ tp_dict.__optional_keys__ = frozenset(optional_keys)
+ tp_dict.__readonly_keys__ = frozenset(readonly_keys)
+ tp_dict.__mutable_keys__ = frozenset(mutable_keys)
+ tp_dict.__total__ = total
+ tp_dict.__closed__ = closed
+ tp_dict.__extra_items__ = extra_items_type
+ return tp_dict
+
+ __call__ = dict # static method
+
+ def __subclasscheck__(cls, other):
+ # Typed dicts are only for static structural subtyping.
+ raise TypeError('TypedDict does not support instance and class checks')
+
+ __instancecheck__ = __subclasscheck__
+
+ _TypedDict = type.__new__(_TypedDictMeta, 'TypedDict', (), {})
+
+ def _create_typeddict(
+ typename,
+ fields,
+ /,
+ *,
+ typing_is_inline,
+ total,
+ closed,
+ extra_items,
+ **kwargs,
+ ):
+ if fields is _marker or fields is None:
+ if fields is _marker:
+ deprecated_thing = (
+ "Failing to pass a value for the 'fields' parameter"
+ )
+ else:
+ deprecated_thing = "Passing `None` as the 'fields' parameter"
+
+ example = f"`{typename} = TypedDict({typename!r}, {{}})`"
+ deprecation_msg = (
+ f"{deprecated_thing} is deprecated and will be disallowed in "
+ "Python 3.15. To create a TypedDict class with 0 fields "
+ "using the functional syntax, pass an empty dictionary, e.g. "
+ ) + example + "."
+ warnings.warn(deprecation_msg, DeprecationWarning, stacklevel=2)
+ # Support a field called "closed"
+ if closed is not False and closed is not True and closed is not None:
+ kwargs["closed"] = closed
+ closed = None
+ # Or "extra_items"
+ if extra_items is not NoExtraItems:
+ kwargs["extra_items"] = extra_items
+ extra_items = NoExtraItems
+ fields = kwargs
+ elif kwargs:
+ raise TypeError("TypedDict takes either a dict or keyword arguments,"
+ " but not both")
+ if kwargs:
+ # Breakpoint: https://github.com/python/cpython/pull/104891
+ if sys.version_info >= (3, 13):
+ raise TypeError("TypedDict takes no keyword arguments")
+ warnings.warn(
+ "The kwargs-based syntax for TypedDict definitions is deprecated "
+ "in Python 3.11, will be removed in Python 3.13, and may not be "
+ "understood by third-party type checkers.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+
+ ns = {'__annotations__': dict(fields)}
+ module = _caller(depth=4 if typing_is_inline else 2)
+ if module is not None:
+ # Setting correct module is necessary to make typed dict classes
+ # pickleable.
+ ns['__module__'] = module
+
+ td = _TypedDictMeta(typename, (), ns, total=total, closed=closed,
+ extra_items=extra_items)
+ td.__orig_bases__ = (TypedDict,)
+ return td
+
+ class _TypedDictSpecialForm(_SpecialForm, _root=True):
+ def __call__(
+ self,
+ typename,
+ fields=_marker,
+ /,
+ *,
+ total=True,
+ closed=None,
+ extra_items=NoExtraItems,
+ **kwargs
+ ):
+ return _create_typeddict(
+ typename,
+ fields,
+ typing_is_inline=False,
+ total=total,
+ closed=closed,
+ extra_items=extra_items,
+ **kwargs,
+ )
+
+ def __mro_entries__(self, bases):
+ return (_TypedDict,)
+
+ @_TypedDictSpecialForm
+ def TypedDict(self, args):
+ """A simple typed namespace. At runtime it is equivalent to a plain dict.
+
+ TypedDict creates a dictionary type such that a type checker will expect all
+ instances to have a certain set of keys, where each key is
+ associated with a value of a consistent type. This expectation
+ is not checked at runtime.
+
+ Usage::
+
+ class Point2D(TypedDict):
+ x: int
+ y: int
+ label: str
+
+ a: Point2D = {'x': 1, 'y': 2, 'label': 'good'} # OK
+ b: Point2D = {'z': 3, 'label': 'bad'} # Fails type check
+
+ assert Point2D(x=1, y=2, label='first') == dict(x=1, y=2, label='first')
+
+ The type info can be accessed via the Point2D.__annotations__ dict, and
+ the Point2D.__required_keys__ and Point2D.__optional_keys__ frozensets.
+ TypedDict supports an additional equivalent form::
+
+ Point2D = TypedDict('Point2D', {'x': int, 'y': int, 'label': str})
+
+ By default, all keys must be present in a TypedDict. It is possible
+ to override this by specifying totality::
+
+ class Point2D(TypedDict, total=False):
+ x: int
+ y: int
+
+ This means that a Point2D TypedDict can have any of the keys omitted. A type
+ checker is only expected to support a literal False or True as the value of
+ the total argument. True is the default, and makes all items defined in the
+ class body be required.
+
+ The Required and NotRequired special forms can also be used to mark
+ individual keys as being required or not required::
+
+ class Point2D(TypedDict):
+ x: int # the "x" key must always be present (Required is the default)
+ y: NotRequired[int] # the "y" key can be omitted
+
+ See PEP 655 for more details on Required and NotRequired.
+ """
+ # This runs when creating inline TypedDicts:
+ if not isinstance(args, dict):
+ raise TypeError(
+ "TypedDict[...] should be used with a single dict argument"
+ )
+
+ return _create_typeddict(
+ "<inline TypedDict>",
+ args,
+ typing_is_inline=True,
+ total=True,
+ closed=True,
+ extra_items=NoExtraItems,
+ )
+
+ _TYPEDDICT_TYPES = (typing._TypedDictMeta, _TypedDictMeta)
+
+ def is_typeddict(tp):
+ """Check if an annotation is a TypedDict class
+
+ For example::
+ class Film(TypedDict):
+ title: str
+ year: int
+
+ is_typeddict(Film) # => True
+ is_typeddict(Union[list, str]) # => False
+ """
+ return isinstance(tp, _TYPEDDICT_TYPES)
+
+
+if hasattr(typing, "assert_type"):
+ assert_type = typing.assert_type
+
+else:
+ def assert_type(val, typ, /):
+ """Assert (to the type checker) that the value is of the given type.
+
+ When the type checker encounters a call to assert_type(), it
+ emits an error if the value is not of the specified type::
+
+ def greet(name: str) -> None:
+ assert_type(name, str) # ok
+ assert_type(name, int) # type checker error
+
+ At runtime this returns the first argument unchanged and otherwise
+ does nothing.
+ """
+ return val
+
+
+if hasattr(typing, "ReadOnly"): # 3.13+
+ get_type_hints = typing.get_type_hints
+else: # <=3.13
+ # replaces _strip_annotations()
+ def _strip_extras(t):
+ """Strips Annotated, Required and NotRequired from a given type."""
+ if isinstance(t, typing._AnnotatedAlias):
+ return _strip_extras(t.__origin__)
+ if hasattr(t, "__origin__") and t.__origin__ in (Required, NotRequired, ReadOnly):
+ return _strip_extras(t.__args__[0])
+ if isinstance(t, typing._GenericAlias):
+ stripped_args = tuple(_strip_extras(a) for a in t.__args__)
+ if stripped_args == t.__args__:
+ return t
+ return t.copy_with(stripped_args)
+ if hasattr(_types, "GenericAlias") and isinstance(t, _types.GenericAlias):
+ stripped_args = tuple(_strip_extras(a) for a in t.__args__)
+ if stripped_args == t.__args__:
+ return t
+ return _types.GenericAlias(t.__origin__, stripped_args)
+ if hasattr(_types, "UnionType") and isinstance(t, _types.UnionType):
+ stripped_args = tuple(_strip_extras(a) for a in t.__args__)
+ if stripped_args == t.__args__:
+ return t
+ return functools.reduce(operator.or_, stripped_args)
+
+ return t
+
+ def get_type_hints(obj, globalns=None, localns=None, include_extras=False):
+ """Return type hints for an object.
+
+ This is often the same as obj.__annotations__, but it handles
+ forward references encoded as string literals, adds Optional[t] if a
+ default value equal to None is set and recursively replaces all
+ 'Annotated[T, ...]', 'Required[T]' or 'NotRequired[T]' with 'T'
+ (unless 'include_extras=True').
+
+ The argument may be a module, class, method, or function. The annotations
+ are returned as a dictionary. For classes, annotations include also
+ inherited members.
+
+ TypeError is raised if the argument is not of a type that can contain
+ annotations, and an empty dictionary is returned if no annotations are
+ present.
+
+ BEWARE -- the behavior of globalns and localns is counterintuitive
+ (unless you are familiar with how eval() and exec() work). The
+ search order is locals first, then globals.
+
+ - If no dict arguments are passed, an attempt is made to use the
+ globals from obj (or the respective module's globals for classes),
+ and these are also used as the locals. If the object does not appear
+ to have globals, an empty dictionary is used.
+
+ - If one dict argument is passed, it is used for both globals and
+ locals.
+
+ - If two dict arguments are passed, they specify globals and
+ locals, respectively.
+ """
+ hint = typing.get_type_hints(
+ obj, globalns=globalns, localns=localns, include_extras=True
+ )
+ # Breakpoint: https://github.com/python/cpython/pull/30304
+ if sys.version_info < (3, 11):
+ _clean_optional(obj, hint, globalns, localns)
+ if include_extras:
+ return hint
+ return {k: _strip_extras(t) for k, t in hint.items()}
+
+ _NoneType = type(None)
+
+ def _could_be_inserted_optional(t):
+ """detects Union[..., None] pattern"""
+ if not isinstance(t, typing._UnionGenericAlias):
+ return False
+ # Assume if last argument is not None they are user defined
+ if t.__args__[-1] is not _NoneType:
+ return False
+ return True
+
+ # < 3.11
+ def _clean_optional(obj, hints, globalns=None, localns=None):
+ # reverts injected Union[..., None] cases from typing.get_type_hints
+ # when a None default value is used.
+ # see https://github.com/python/typing_extensions/issues/310
+ if not hints or isinstance(obj, type):
+ return
+ defaults = typing._get_defaults(obj) # avoid accessing __annotations___
+ if not defaults:
+ return
+ original_hints = obj.__annotations__
+ for name, value in hints.items():
+ # Not a Union[..., None] or replacement conditions not fullfilled
+ if (not _could_be_inserted_optional(value)
+ or name not in defaults
+ or defaults[name] is not None
+ ):
+ continue
+ original_value = original_hints[name]
+ # value=NoneType should have caused a skip above but check for safety
+ if original_value is None:
+ original_value = _NoneType
+ # Forward reference
+ if isinstance(original_value, str):
+ if globalns is None:
+ if isinstance(obj, _types.ModuleType):
+ globalns = obj.__dict__
+ else:
+ nsobj = obj
+ # Find globalns for the unwrapped object.
+ while hasattr(nsobj, '__wrapped__'):
+ nsobj = nsobj.__wrapped__
+ globalns = getattr(nsobj, '__globals__', {})
+ if localns is None:
+ localns = globalns
+ elif localns is None:
+ localns = globalns
+
+ original_value = ForwardRef(
+ original_value,
+ is_argument=not isinstance(obj, _types.ModuleType)
+ )
+ original_evaluated = typing._eval_type(original_value, globalns, localns)
+ # Compare if values differ. Note that even if equal
+ # value might be cached by typing._tp_cache contrary to original_evaluated
+ if original_evaluated != value or (
+ # 3.10: ForwardRefs of UnionType might be turned into _UnionGenericAlias
+ hasattr(_types, "UnionType")
+ and isinstance(original_evaluated, _types.UnionType)
+ and not isinstance(value, _types.UnionType)
+ ):
+ hints[name] = original_evaluated
+
+# Python 3.9 has get_origin() and get_args() but those implementations don't support
+# ParamSpecArgs and ParamSpecKwargs, so only Python 3.10's versions will do.
+# Breakpoint: https://github.com/python/cpython/pull/25298
+if sys.version_info >= (3, 10):
+ get_origin = typing.get_origin
+ get_args = typing.get_args
+# 3.9
+else:
+ def get_origin(tp):
+ """Get the unsubscripted version of a type.
+
+ This supports generic types, Callable, Tuple, Union, Literal, Final, ClassVar
+ and Annotated. Return None for unsupported types. Examples::
+
+ get_origin(Literal[42]) is Literal
+ get_origin(int) is None
+ get_origin(ClassVar[int]) is ClassVar
+ get_origin(Generic) is Generic
+ get_origin(Generic[T]) is Generic
+ get_origin(Union[T, int]) is Union
+ get_origin(List[Tuple[T, T]][int]) == list
+ get_origin(P.args) is P
+ """
+ if isinstance(tp, typing._AnnotatedAlias):
+ return Annotated
+ if isinstance(tp, (typing._BaseGenericAlias, _types.GenericAlias,
+ ParamSpecArgs, ParamSpecKwargs)):
+ return tp.__origin__
+ if tp is typing.Generic:
+ return typing.Generic
+ return None
+
+ def get_args(tp):
+ """Get type arguments with all substitutions performed.
+
+ For unions, basic simplifications used by Union constructor are performed.
+ Examples::
+ get_args(Dict[str, int]) == (str, int)
+ get_args(int) == ()
+ get_args(Union[int, Union[T, int], str][int]) == (int, str)
+ get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int])
+ get_args(Callable[[], T][int]) == ([], int)
+ """
+ if isinstance(tp, typing._AnnotatedAlias):
+ return (tp.__origin__, *tp.__metadata__)
+ if isinstance(tp, (typing._GenericAlias, _types.GenericAlias)):
+ res = tp.__args__
+ if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis:
+ res = (list(res[:-1]), res[-1])
+ return res
+ return ()
+
+
+# 3.10+
+if hasattr(typing, 'TypeAlias'):
+ TypeAlias = typing.TypeAlias
+# 3.9
+else:
+ @_ExtensionsSpecialForm
+ def TypeAlias(self, parameters):
+ """Special marker indicating that an assignment should
+ be recognized as a proper type alias definition by type
+ checkers.
+
+ For example::
+
+ Predicate: TypeAlias = Callable[..., bool]
+
+ It's invalid when used anywhere except as in the example above.
+ """
+ raise TypeError(f"{self} is not subscriptable")
+
+
+def _set_default(type_param, default):
+ type_param.has_default = lambda: default is not NoDefault
+ type_param.__default__ = default
+
+
+def _set_module(typevarlike):
+ # for pickling:
+ def_mod = _caller(depth=2)
+ if def_mod != 'typing_extensions':
+ typevarlike.__module__ = def_mod
+
+
+class _DefaultMixin:
+ """Mixin for TypeVarLike defaults."""
+
+ __slots__ = ()
+ __init__ = _set_default
+
+
+# Classes using this metaclass must provide a _backported_typevarlike ClassVar
+class _TypeVarLikeMeta(type):
+ def __instancecheck__(cls, __instance: Any) -> bool:
+ return isinstance(__instance, cls._backported_typevarlike)
+
+
+if _PEP_696_IMPLEMENTED:
+ from typing import TypeVar
+else:
+ # Add default and infer_variance parameters from PEP 696 and 695
+ class TypeVar(metaclass=_TypeVarLikeMeta):
+ """Type variable."""
+
+ _backported_typevarlike = typing.TypeVar
+
+ def __new__(cls, name, *constraints, bound=None,
+ covariant=False, contravariant=False,
+ default=NoDefault, infer_variance=False):
+ if hasattr(typing, "TypeAliasType"):
+ # PEP 695 implemented (3.12+), can pass infer_variance to typing.TypeVar
+ typevar = typing.TypeVar(name, *constraints, bound=bound,
+ covariant=covariant, contravariant=contravariant,
+ infer_variance=infer_variance)
+ else:
+ typevar = typing.TypeVar(name, *constraints, bound=bound,
+ covariant=covariant, contravariant=contravariant)
+ if infer_variance and (covariant or contravariant):
+ raise ValueError("Variance cannot be specified with infer_variance.")
+ typevar.__infer_variance__ = infer_variance
+
+ _set_default(typevar, default)
+ _set_module(typevar)
+
+ def _tvar_prepare_subst(alias, args):
+ if (
+ typevar.has_default()
+ and alias.__parameters__.index(typevar) == len(args)
+ ):
+ args += (typevar.__default__,)
+ return args
+
+ typevar.__typing_prepare_subst__ = _tvar_prepare_subst
+ return typevar
+
+ def __init_subclass__(cls) -> None:
+ raise TypeError(f"type '{__name__}.TypeVar' is not an acceptable base type")
+
+
+# Python 3.10+ has PEP 612
+if hasattr(typing, 'ParamSpecArgs'):
+ ParamSpecArgs = typing.ParamSpecArgs
+ ParamSpecKwargs = typing.ParamSpecKwargs
+# 3.9
+else:
+ class _Immutable:
+ """Mixin to indicate that object should not be copied."""
+ __slots__ = ()
+
+ def __copy__(self):
+ return self
+
+ def __deepcopy__(self, memo):
+ return self
+
+ class ParamSpecArgs(_Immutable):
+ """The args for a ParamSpec object.
+
+ Given a ParamSpec object P, P.args is an instance of ParamSpecArgs.
+
+ ParamSpecArgs objects have a reference back to their ParamSpec:
+
+ P.args.__origin__ is P
+
+ This type is meant for runtime introspection and has no special meaning to
+ static type checkers.
+ """
+ def __init__(self, origin):
+ self.__origin__ = origin
+
+ def __repr__(self):
+ return f"{self.__origin__.__name__}.args"
+
+ def __eq__(self, other):
+ if not isinstance(other, ParamSpecArgs):
+ return NotImplemented
+ return self.__origin__ == other.__origin__
+
+ class ParamSpecKwargs(_Immutable):
+ """The kwargs for a ParamSpec object.
+
+ Given a ParamSpec object P, P.kwargs is an instance of ParamSpecKwargs.
+
+ ParamSpecKwargs objects have a reference back to their ParamSpec:
+
+ P.kwargs.__origin__ is P
+
+ This type is meant for runtime introspection and has no special meaning to
+ static type checkers.
+ """
+ def __init__(self, origin):
+ self.__origin__ = origin
+
+ def __repr__(self):
+ return f"{self.__origin__.__name__}.kwargs"
+
+ def __eq__(self, other):
+ if not isinstance(other, ParamSpecKwargs):
+ return NotImplemented
+ return self.__origin__ == other.__origin__
+
+
+if _PEP_696_IMPLEMENTED:
+ from typing import ParamSpec
+
+# 3.10+
+elif hasattr(typing, 'ParamSpec'):
+
+ # Add default parameter - PEP 696
+ class ParamSpec(metaclass=_TypeVarLikeMeta):
+ """Parameter specification."""
+
+ _backported_typevarlike = typing.ParamSpec
+
+ def __new__(cls, name, *, bound=None,
+ covariant=False, contravariant=False,
+ infer_variance=False, default=NoDefault):
+ if hasattr(typing, "TypeAliasType"):
+ # PEP 695 implemented, can pass infer_variance to typing.TypeVar
+ paramspec = typing.ParamSpec(name, bound=bound,
+ covariant=covariant,
+ contravariant=contravariant,
+ infer_variance=infer_variance)
+ else:
+ paramspec = typing.ParamSpec(name, bound=bound,
+ covariant=covariant,
+ contravariant=contravariant)
+ paramspec.__infer_variance__ = infer_variance
+
+ _set_default(paramspec, default)
+ _set_module(paramspec)
+
+ def _paramspec_prepare_subst(alias, args):
+ params = alias.__parameters__
+ i = params.index(paramspec)
+ if i == len(args) and paramspec.has_default():
+ args = [*args, paramspec.__default__]
+ if i >= len(args):
+ raise TypeError(f"Too few arguments for {alias}")
+ # Special case where Z[[int, str, bool]] == Z[int, str, bool] in PEP 612.
+ if len(params) == 1 and not typing._is_param_expr(args[0]):
+ assert i == 0
+ args = (args,)
+ # Convert lists to tuples to help other libraries cache the results.
+ elif isinstance(args[i], list):
+ args = (*args[:i], tuple(args[i]), *args[i + 1:])
+ return args
+
+ paramspec.__typing_prepare_subst__ = _paramspec_prepare_subst
+ return paramspec
+
+ def __init_subclass__(cls) -> None:
+ raise TypeError(f"type '{__name__}.ParamSpec' is not an acceptable base type")
+
+# 3.9
+else:
+
+ # Inherits from list as a workaround for Callable checks in Python < 3.9.2.
+ class ParamSpec(list, _DefaultMixin):
+ """Parameter specification variable.
+
+ Usage::
+
+ P = ParamSpec('P')
+
+ Parameter specification variables exist primarily for the benefit of static
+ type checkers. They are used to forward the parameter types of one
+ callable to another callable, a pattern commonly found in higher order
+ functions and decorators. They are only valid when used in ``Concatenate``,
+ or s the first argument to ``Callable``. In Python 3.10 and higher,
+ they are also supported in user-defined Generics at runtime.
+ See class Generic for more information on generic types. An
+ example for annotating a decorator::
+
+ T = TypeVar('T')
+ P = ParamSpec('P')
+
+ def add_logging(f: Callable[P, T]) -> Callable[P, T]:
+ '''A type-safe decorator to add logging to a function.'''
+ def inner(*args: P.args, **kwargs: P.kwargs) -> T:
+ logging.info(f'{f.__name__} was called')
+ return f(*args, **kwargs)
+ return inner
+
+ @add_logging
+ def add_two(x: float, y: float) -> float:
+ '''Add two numbers together.'''
+ return x + y
+
+ Parameter specification variables defined with covariant=True or
+ contravariant=True can be used to declare covariant or contravariant
+ generic types. These keyword arguments are valid, but their actual semantics
+ are yet to be decided. See PEP 612 for details.
+
+ Parameter specification variables can be introspected. e.g.:
+
+ P.__name__ == 'T'
+ P.__bound__ == None
+ P.__covariant__ == False
+ P.__contravariant__ == False
+
+ Note that only parameter specification variables defined in global scope can
+ be pickled.
+ """
+
+ # Trick Generic __parameters__.
+ __class__ = typing.TypeVar
+
+ @property
+ def args(self):
+ return ParamSpecArgs(self)
+
+ @property
+ def kwargs(self):
+ return ParamSpecKwargs(self)
+
+ def __init__(self, name, *, bound=None, covariant=False, contravariant=False,
+ infer_variance=False, default=NoDefault):
+ list.__init__(self, [self])
+ self.__name__ = name
+ self.__covariant__ = bool(covariant)
+ self.__contravariant__ = bool(contravariant)
+ self.__infer_variance__ = bool(infer_variance)
+ if bound:
+ self.__bound__ = typing._type_check(bound, 'Bound must be a type.')
+ else:
+ self.__bound__ = None
+ _DefaultMixin.__init__(self, default)
+
+ # for pickling:
+ def_mod = _caller()
+ if def_mod != 'typing_extensions':
+ self.__module__ = def_mod
+
+ def __repr__(self):
+ if self.__infer_variance__:
+ prefix = ''
+ elif self.__covariant__:
+ prefix = '+'
+ elif self.__contravariant__:
+ prefix = '-'
+ else:
+ prefix = '~'
+ return prefix + self.__name__
+
+ def __hash__(self):
+ return object.__hash__(self)
+
+ def __eq__(self, other):
+ return self is other
+
+ def __reduce__(self):
+ return self.__name__
+
+ # Hack to get typing._type_check to pass.
+ def __call__(self, *args, **kwargs):
+ pass
+
+
+# 3.9
+if not hasattr(typing, 'Concatenate'):
+ # Inherits from list as a workaround for Callable checks in Python < 3.9.2.
+
+ # 3.9.0-1
+ if not hasattr(typing, '_type_convert'):
+ def _type_convert(arg, module=None, *, allow_special_forms=False):
+ """For converting None to type(None), and strings to ForwardRef."""
+ if arg is None:
+ return type(None)
+ if isinstance(arg, str):
+ if sys.version_info <= (3, 9, 6):
+ return ForwardRef(arg)
+ if sys.version_info <= (3, 9, 7):
+ return ForwardRef(arg, module=module)
+ return ForwardRef(arg, module=module, is_class=allow_special_forms)
+ return arg
+ else:
+ _type_convert = typing._type_convert
+
+ class _ConcatenateGenericAlias(list):
+
+ # Trick Generic into looking into this for __parameters__.
+ __class__ = typing._GenericAlias
+
+ def __init__(self, origin, args):
+ super().__init__(args)
+ self.__origin__ = origin
+ self.__args__ = args
+
+ def __repr__(self):
+ _type_repr = typing._type_repr
+ return (f'{_type_repr(self.__origin__)}'
+ f'[{", ".join(_type_repr(arg) for arg in self.__args__)}]')
+
+ def __hash__(self):
+ return hash((self.__origin__, self.__args__))
+
+ # Hack to get typing._type_check to pass in Generic.
+ def __call__(self, *args, **kwargs):
+ pass
+
+ @property
+ def __parameters__(self):
+ return tuple(
+ tp for tp in self.__args__ if isinstance(tp, (typing.TypeVar, ParamSpec))
+ )
+
+ # 3.9 used by __getitem__ below
+ def copy_with(self, params):
+ if isinstance(params[-1], _ConcatenateGenericAlias):
+ params = (*params[:-1], *params[-1].__args__)
+ elif isinstance(params[-1], (list, tuple)):
+ return (*params[:-1], *params[-1])
+ elif (not (params[-1] is ... or isinstance(params[-1], ParamSpec))):
+ raise TypeError("The last parameter to Concatenate should be a "
+ "ParamSpec variable or ellipsis.")
+ return self.__class__(self.__origin__, params)
+
+ # 3.9; accessed during GenericAlias.__getitem__ when substituting
+ def __getitem__(self, args):
+ if self.__origin__ in (Generic, Protocol):
+ # Can't subscript Generic[...] or Protocol[...].
+ raise TypeError(f"Cannot subscript already-subscripted {self}")
+ if not self.__parameters__:
+ raise TypeError(f"{self} is not a generic class")
+
+ if not isinstance(args, tuple):
+ args = (args,)
+ args = _unpack_args(*(_type_convert(p) for p in args))
+ params = self.__parameters__
+ for param in params:
+ prepare = getattr(param, "__typing_prepare_subst__", None)
+ if prepare is not None:
+ args = prepare(self, args)
+ # 3.9 & typing.ParamSpec
+ elif isinstance(param, ParamSpec):
+ i = params.index(param)
+ if (
+ i == len(args)
+ and getattr(param, '__default__', NoDefault) is not NoDefault
+ ):
+ args = [*args, param.__default__]
+ if i >= len(args):
+ raise TypeError(f"Too few arguments for {self}")
+ # Special case for Z[[int, str, bool]] == Z[int, str, bool]
+ if len(params) == 1 and not _is_param_expr(args[0]):
+ assert i == 0
+ args = (args,)
+ elif (
+ isinstance(args[i], list)
+ # 3.9
+ # This class inherits from list do not convert
+ and not isinstance(args[i], _ConcatenateGenericAlias)
+ ):
+ args = (*args[:i], tuple(args[i]), *args[i + 1:])
+
+ alen = len(args)
+ plen = len(params)
+ if alen != plen:
+ raise TypeError(
+ f"Too {'many' if alen > plen else 'few'} arguments for {self};"
+ f" actual {alen}, expected {plen}"
+ )
+
+ subst = dict(zip(self.__parameters__, args))
+ # determine new args
+ new_args = []
+ for arg in self.__args__:
+ if isinstance(arg, type):
+ new_args.append(arg)
+ continue
+ if isinstance(arg, TypeVar):
+ arg = subst[arg]
+ if (
+ (isinstance(arg, typing._GenericAlias) and _is_unpack(arg))
+ or (
+ hasattr(_types, "GenericAlias")
+ and isinstance(arg, _types.GenericAlias)
+ and getattr(arg, "__unpacked__", False)
+ )
+ ):
+ raise TypeError(f"{arg} is not valid as type argument")
+
+ elif isinstance(arg,
+ typing._GenericAlias
+ if not hasattr(_types, "GenericAlias") else
+ (typing._GenericAlias, _types.GenericAlias)
+ ):
+ subparams = arg.__parameters__
+ if subparams:
+ subargs = tuple(subst[x] for x in subparams)
+ arg = arg[subargs]
+ new_args.append(arg)
+ return self.copy_with(tuple(new_args))
+
+# 3.10+
+else:
+ _ConcatenateGenericAlias = typing._ConcatenateGenericAlias
+
+ # 3.10
+ if sys.version_info < (3, 11):
+
+ class _ConcatenateGenericAlias(typing._ConcatenateGenericAlias, _root=True):
+ # needed for checks in collections.abc.Callable to accept this class
+ __module__ = "typing"
+
+ def copy_with(self, params):
+ if isinstance(params[-1], (list, tuple)):
+ return (*params[:-1], *params[-1])
+ if isinstance(params[-1], typing._ConcatenateGenericAlias):
+ params = (*params[:-1], *params[-1].__args__)
+ elif not (params[-1] is ... or isinstance(params[-1], ParamSpec)):
+ raise TypeError("The last parameter to Concatenate should be a "
+ "ParamSpec variable or ellipsis.")
+ return super(typing._ConcatenateGenericAlias, self).copy_with(params)
+
+ def __getitem__(self, args):
+ value = super().__getitem__(args)
+ if isinstance(value, tuple) and any(_is_unpack(t) for t in value):
+ return tuple(_unpack_args(*(n for n in value)))
+ return value
+
+
+# 3.9.2
+class _EllipsisDummy: ...
+
+
+# <=3.10
+def _create_concatenate_alias(origin, parameters):
+ if parameters[-1] is ... and sys.version_info < (3, 9, 2):
+ # Hack: Arguments must be types, replace it with one.
+ parameters = (*parameters[:-1], _EllipsisDummy)
+ if sys.version_info >= (3, 10, 3):
+ concatenate = _ConcatenateGenericAlias(origin, parameters,
+ _typevar_types=(TypeVar, ParamSpec),
+ _paramspec_tvars=True)
+ else:
+ concatenate = _ConcatenateGenericAlias(origin, parameters)
+ if parameters[-1] is not _EllipsisDummy:
+ return concatenate
+ # Remove dummy again
+ concatenate.__args__ = tuple(p if p is not _EllipsisDummy else ...
+ for p in concatenate.__args__)
+ if sys.version_info < (3, 10):
+ # backport needs __args__ adjustment only
+ return concatenate
+ concatenate.__parameters__ = tuple(p for p in concatenate.__parameters__
+ if p is not _EllipsisDummy)
+ return concatenate
+
+
+# <=3.10
+@typing._tp_cache
+def _concatenate_getitem(self, parameters):
+ if parameters == ():
+ raise TypeError("Cannot take a Concatenate of no types.")
+ if not isinstance(parameters, tuple):
+ parameters = (parameters,)
+ if not (parameters[-1] is ... or isinstance(parameters[-1], ParamSpec)):
+ raise TypeError("The last parameter to Concatenate should be a "
+ "ParamSpec variable or ellipsis.")
+ msg = "Concatenate[arg, ...]: each arg must be a type."
+ parameters = (*(typing._type_check(p, msg) for p in parameters[:-1]),
+ parameters[-1])
+ return _create_concatenate_alias(self, parameters)
+
+
+# 3.11+; Concatenate does not accept ellipsis in 3.10
+# Breakpoint: https://github.com/python/cpython/pull/30969
+if sys.version_info >= (3, 11):
+ Concatenate = typing.Concatenate
+# <=3.10
+else:
+ @_ExtensionsSpecialForm
+ def Concatenate(self, parameters):
+ """Used in conjunction with ``ParamSpec`` and ``Callable`` to represent a
+ higher order function which adds, removes or transforms parameters of a
+ callable.
+
+ For example::
+
+ Callable[Concatenate[int, P], int]
+
+ See PEP 612 for detailed information.
+ """
+ return _concatenate_getitem(self, parameters)
+
+
+# 3.10+
+if hasattr(typing, 'TypeGuard'):
+ TypeGuard = typing.TypeGuard
+# 3.9
+else:
+ @_ExtensionsSpecialForm
+ def TypeGuard(self, parameters):
+ """Special typing form used to annotate the return type of a user-defined
+ type guard function. ``TypeGuard`` only accepts a single type argument.
+ At runtime, functions marked this way should return a boolean.
+
+ ``TypeGuard`` aims to benefit *type narrowing* -- a technique used by static
+ type checkers to determine a more precise type of an expression within a
+ program's code flow. Usually type narrowing is done by analyzing
+ conditional code flow and applying the narrowing to a block of code. The
+ conditional expression here is sometimes referred to as a "type guard".
+
+ Sometimes it would be convenient to use a user-defined boolean function
+ as a type guard. Such a function should use ``TypeGuard[...]`` as its
+ return type to alert static type checkers to this intention.
+
+ Using ``-> TypeGuard`` tells the static type checker that for a given
+ function:
+
+ 1. The return value is a boolean.
+ 2. If the return value is ``True``, the type of its argument
+ is the type inside ``TypeGuard``.
+
+ For example::
+
+ def is_str(val: Union[str, float]):
+ # "isinstance" type guard
+ if isinstance(val, str):
+ # Type of ``val`` is narrowed to ``str``
+ ...
+ else:
+ # Else, type of ``val`` is narrowed to ``float``.
+ ...
+
+ Strict type narrowing is not enforced -- ``TypeB`` need not be a narrower
+ form of ``TypeA`` (it can even be a wider form) and this may lead to
+ type-unsafe results. The main reason is to allow for things like
+ narrowing ``List[object]`` to ``List[str]`` even though the latter is not
+ a subtype of the former, since ``List`` is invariant. The responsibility of
+ writing type-safe type guards is left to the user.
+
+ ``TypeGuard`` also works with type variables. For more information, see
+ PEP 647 (User-Defined Type Guards).
+ """
+ item = typing._type_check(parameters, f'{self} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+
+# 3.13+
+if hasattr(typing, 'TypeIs'):
+ TypeIs = typing.TypeIs
+# <=3.12
+else:
+ @_ExtensionsSpecialForm
+ def TypeIs(self, parameters):
+ """Special typing form used to annotate the return type of a user-defined
+ type narrower function. ``TypeIs`` only accepts a single type argument.
+ At runtime, functions marked this way should return a boolean.
+
+ ``TypeIs`` aims to benefit *type narrowing* -- a technique used by static
+ type checkers to determine a more precise type of an expression within a
+ program's code flow. Usually type narrowing is done by analyzing
+ conditional code flow and applying the narrowing to a block of code. The
+ conditional expression here is sometimes referred to as a "type guard".
+
+ Sometimes it would be convenient to use a user-defined boolean function
+ as a type guard. Such a function should use ``TypeIs[...]`` as its
+ return type to alert static type checkers to this intention.
+
+ Using ``-> TypeIs`` tells the static type checker that for a given
+ function:
+
+ 1. The return value is a boolean.
+ 2. If the return value is ``True``, the type of its argument
+ is the intersection of the type inside ``TypeIs`` and the argument's
+ previously known type.
+
+ For example::
+
+ def is_awaitable(val: object) -> TypeIs[Awaitable[Any]]:
+ return hasattr(val, '__await__')
+
+ def f(val: Union[int, Awaitable[int]]) -> int:
+ if is_awaitable(val):
+ assert_type(val, Awaitable[int])
+ else:
+ assert_type(val, int)
+
+ ``TypeIs`` also works with type variables. For more information, see
+ PEP 742 (Narrowing types with TypeIs).
+ """
+ item = typing._type_check(parameters, f'{self} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+
+# 3.14+?
+if hasattr(typing, 'TypeForm'):
+ TypeForm = typing.TypeForm
+# <=3.13
+else:
+ class _TypeFormForm(_ExtensionsSpecialForm, _root=True):
+ # TypeForm(X) is equivalent to X but indicates to the type checker
+ # that the object is a TypeForm.
+ def __call__(self, obj, /):
+ return obj
+
+ @_TypeFormForm
+ def TypeForm(self, parameters):
+ """A special form representing the value that results from the evaluation
+ of a type expression. This value encodes the information supplied in the
+ type expression, and it represents the type described by that type expression.
+
+ When used in a type expression, TypeForm describes a set of type form objects.
+ It accepts a single type argument, which must be a valid type expression.
+ ``TypeForm[T]`` describes the set of all type form objects that represent
+ the type T or types that are assignable to T.
+
+ Usage:
+
+ def cast[T](typ: TypeForm[T], value: Any) -> T: ...
+
+ reveal_type(cast(int, "x")) # int
+
+ See PEP 747 for more information.
+ """
+ item = typing._type_check(parameters, f'{self} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+
+
+
+if hasattr(typing, "LiteralString"): # 3.11+
+ LiteralString = typing.LiteralString
+else:
+ @_SpecialForm
+ def LiteralString(self, params):
+ """Represents an arbitrary literal string.
+
+ Example::
+
+ from bb._vendor.typing_extensions import LiteralString
+
+ def query(sql: LiteralString) -> ...:
+ ...
+
+ query("SELECT * FROM table") # ok
+ query(f"SELECT * FROM {input()}") # not ok
+
+ See PEP 675 for details.
+
+ """
+ raise TypeError(f"{self} is not subscriptable")
+
+
+if hasattr(typing, "Self"): # 3.11+
+ Self = typing.Self
+else:
+ @_SpecialForm
+ def Self(self, params):
+ """Used to spell the type of "self" in classes.
+
+ Example::
+
+ from typing import Self
+
+ class ReturnsSelf:
+ def parse(self, data: bytes) -> Self:
+ ...
+ return self
+
+ """
+
+ raise TypeError(f"{self} is not subscriptable")
+
+
+if hasattr(typing, "Never"): # 3.11+
+ Never = typing.Never
+else:
+ @_SpecialForm
+ def Never(self, params):
+ """The bottom type, a type that has no members.
+
+ This can be used to define a function that should never be
+ called, or a function that never returns::
+
+ from bb._vendor.typing_extensions import Never
+
+ def never_call_me(arg: Never) -> None:
+ pass
+
+ def int_or_str(arg: int | str) -> None:
+ never_call_me(arg) # type checker error
+ match arg:
+ case int():
+ print("It's an int")
+ case str():
+ print("It's a str")
+ case _:
+ never_call_me(arg) # ok, arg is of type Never
+
+ """
+
+ raise TypeError(f"{self} is not subscriptable")
+
+
+if hasattr(typing, 'Required'): # 3.11+
+ Required = typing.Required
+ NotRequired = typing.NotRequired
+else: # <=3.10
+ @_ExtensionsSpecialForm
+ def Required(self, parameters):
+ """A special typing construct to mark a key of a total=False TypedDict
+ as required. For example:
+
+ class Movie(TypedDict, total=False):
+ title: Required[str]
+ year: int
+
+ m = Movie(
+ title='The Matrix', # typechecker error if key is omitted
+ year=1999,
+ )
+
+ There is no runtime checking that a required key is actually provided
+ when instantiating a related TypedDict.
+ """
+ item = typing._type_check(parameters, f'{self._name} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+ @_ExtensionsSpecialForm
+ def NotRequired(self, parameters):
+ """A special typing construct to mark a key of a TypedDict as
+ potentially missing. For example:
+
+ class Movie(TypedDict):
+ title: str
+ year: NotRequired[int]
+
+ m = Movie(
+ title='The Matrix', # typechecker error if key is omitted
+ year=1999,
+ )
+ """
+ item = typing._type_check(parameters, f'{self._name} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+
+if hasattr(typing, 'ReadOnly'):
+ ReadOnly = typing.ReadOnly
+else: # <=3.12
+ @_ExtensionsSpecialForm
+ def ReadOnly(self, parameters):
+ """A special typing construct to mark an item of a TypedDict as read-only.
+
+ For example:
+
+ class Movie(TypedDict):
+ title: ReadOnly[str]
+ year: int
+
+ def mutate_movie(m: Movie) -> None:
+ m["year"] = 1992 # allowed
+ m["title"] = "The Matrix" # typechecker error
+
+ There is no runtime checking for this property.
+ """
+ item = typing._type_check(parameters, f'{self._name} accepts only a single type.')
+ return typing._GenericAlias(self, (item,))
+
+
+_UNPACK_DOC = """\
+Type unpack operator.
+
+The type unpack operator takes the child types from some container type,
+such as `tuple[int, str]` or a `TypeVarTuple`, and 'pulls them out'. For
+example:
+
+ # For some generic class `Foo`:
+ Foo[Unpack[tuple[int, str]]] # Equivalent to Foo[int, str]
+
+ Ts = TypeVarTuple('Ts')
+ # Specifies that `Bar` is generic in an arbitrary number of types.
+ # (Think of `Ts` as a tuple of an arbitrary number of individual
+ # `TypeVar`s, which the `Unpack` is 'pulling out' directly into the
+ # `Generic[]`.)
+ class Bar(Generic[Unpack[Ts]]): ...
+ Bar[int] # Valid
+ Bar[int, str] # Also valid
+
+From Python 3.11, this can also be done using the `*` operator:
+
+ Foo[*tuple[int, str]]
+ class Bar(Generic[*Ts]): ...
+
+The operator can also be used along with a `TypedDict` to annotate
+`**kwargs` in a function signature. For instance:
+
+ class Movie(TypedDict):
+ name: str
+ year: int
+
+ # This function expects two keyword arguments - *name* of type `str` and
+ # *year* of type `int`.
+ def foo(**kwargs: Unpack[Movie]): ...
+
+Note that there is only some runtime checking of this operator. Not
+everything the runtime allows may be accepted by static type checkers.
+
+For more information, see PEP 646 and PEP 692.
+"""
+
+
+# PEP 692 changed the repr of Unpack[]
+# Breakpoint: https://github.com/python/cpython/pull/104048
+if sys.version_info >= (3, 12):
+ Unpack = typing.Unpack
+
+ def _is_unpack(obj):
+ return get_origin(obj) is Unpack
+
+else: # <=3.11
+ class _UnpackSpecialForm(_ExtensionsSpecialForm, _root=True):
+ def __init__(self, getitem):
+ super().__init__(getitem)
+ self.__doc__ = _UNPACK_DOC
+
+ class _UnpackAlias(typing._GenericAlias, _root=True):
+ if sys.version_info < (3, 11):
+ # needed for compatibility with Generic[Unpack[Ts]]
+ __class__ = typing.TypeVar
+
+ @property
+ def __typing_unpacked_tuple_args__(self):
+ assert self.__origin__ is Unpack
+ assert len(self.__args__) == 1
+ arg, = self.__args__
+ if isinstance(arg, (typing._GenericAlias, _types.GenericAlias)):
+ if arg.__origin__ is not tuple:
+ raise TypeError("Unpack[...] must be used with a tuple type")
+ return arg.__args__
+ return None
+
+ @property
+ def __typing_is_unpacked_typevartuple__(self):
+ assert self.__origin__ is Unpack
+ assert len(self.__args__) == 1
+ return isinstance(self.__args__[0], TypeVarTuple)
+
+ def __getitem__(self, args):
+ if self.__typing_is_unpacked_typevartuple__:
+ return args
+ return super().__getitem__(args)
+
+ @_UnpackSpecialForm
+ def Unpack(self, parameters):
+ item = typing._type_check(parameters, f'{self._name} accepts only a single type.')
+ return _UnpackAlias(self, (item,))
+
+ def _is_unpack(obj):
+ return isinstance(obj, _UnpackAlias)
+
+
+def _unpack_args(*args):
+ newargs = []
+ for arg in args:
+ subargs = getattr(arg, '__typing_unpacked_tuple_args__', None)
+ if subargs is not None and (not (subargs and subargs[-1] is ...)):
+ newargs.extend(subargs)
+ else:
+ newargs.append(arg)
+ return newargs
+
+
+if _PEP_696_IMPLEMENTED:
+ from typing import TypeVarTuple
+
+elif hasattr(typing, "TypeVarTuple"): # 3.11+
+
+ # Add default parameter - PEP 696
+ class TypeVarTuple(metaclass=_TypeVarLikeMeta):
+ """Type variable tuple."""
+
+ _backported_typevarlike = typing.TypeVarTuple
+
+ def __new__(cls, name, *, default=NoDefault):
+ tvt = typing.TypeVarTuple(name)
+ _set_default(tvt, default)
+ _set_module(tvt)
+
+ def _typevartuple_prepare_subst(alias, args):
+ params = alias.__parameters__
+ typevartuple_index = params.index(tvt)
+ for param in params[typevartuple_index + 1:]:
+ if isinstance(param, TypeVarTuple):
+ raise TypeError(
+ f"More than one TypeVarTuple parameter in {alias}"
+ )
+
+ alen = len(args)
+ plen = len(params)
+ left = typevartuple_index
+ right = plen - typevartuple_index - 1
+ var_tuple_index = None
+ fillarg = None
+ for k, arg in enumerate(args):
+ if not isinstance(arg, type):
+ subargs = getattr(arg, '__typing_unpacked_tuple_args__', None)
+ if subargs and len(subargs) == 2 and subargs[-1] is ...:
+ if var_tuple_index is not None:
+ raise TypeError(
+ "More than one unpacked "
+ "arbitrary-length tuple argument"
+ )
+ var_tuple_index = k
+ fillarg = subargs[0]
+ if var_tuple_index is not None:
+ left = min(left, var_tuple_index)
+ right = min(right, alen - var_tuple_index - 1)
+ elif left + right > alen:
+ raise TypeError(f"Too few arguments for {alias};"
+ f" actual {alen}, expected at least {plen - 1}")
+ if left == alen - right and tvt.has_default():
+ replacement = _unpack_args(tvt.__default__)
+ else:
+ replacement = args[left: alen - right]
+
+ return (
+ *args[:left],
+ *([fillarg] * (typevartuple_index - left)),
+ replacement,
+ *([fillarg] * (plen - right - left - typevartuple_index - 1)),
+ *args[alen - right:],
+ )
+
+ tvt.__typing_prepare_subst__ = _typevartuple_prepare_subst
+ return tvt
+
+ def __init_subclass__(self, *args, **kwds):
+ raise TypeError("Cannot subclass special typing classes")
+
+else: # <=3.10
+ class TypeVarTuple(_DefaultMixin):
+ """Type variable tuple.
+
+ Usage::
+
+ Ts = TypeVarTuple('Ts')
+
+ In the same way that a normal type variable is a stand-in for a single
+ type such as ``int``, a type variable *tuple* is a stand-in for a *tuple*
+ type such as ``Tuple[int, str]``.
+
+ Type variable tuples can be used in ``Generic`` declarations.
+ Consider the following example::
+
+ class Array(Generic[*Ts]): ...
+
+ The ``Ts`` type variable tuple here behaves like ``tuple[T1, T2]``,
+ where ``T1`` and ``T2`` are type variables. To use these type variables
+ as type parameters of ``Array``, we must *unpack* the type variable tuple using
+ the star operator: ``*Ts``. The signature of ``Array`` then behaves
+ as if we had simply written ``class Array(Generic[T1, T2]): ...``.
+ In contrast to ``Generic[T1, T2]``, however, ``Generic[*Shape]`` allows
+ us to parameterise the class with an *arbitrary* number of type parameters.
+
+ Type variable tuples can be used anywhere a normal ``TypeVar`` can.
+ This includes class definitions, as shown above, as well as function
+ signatures and variable annotations::
+
+ class Array(Generic[*Ts]):
+
+ def __init__(self, shape: Tuple[*Ts]):
+ self._shape: Tuple[*Ts] = shape
+
+ def get_shape(self) -> Tuple[*Ts]:
+ return self._shape
+
+ shape = (Height(480), Width(640))
+ x: Array[Height, Width] = Array(shape)
+ y = abs(x) # Inferred type is Array[Height, Width]
+ z = x + x # ... is Array[Height, Width]
+ x.get_shape() # ... is tuple[Height, Width]
+
+ """
+
+ # Trick Generic __parameters__.
+ __class__ = typing.TypeVar
+
+ def __iter__(self):
+ yield self.__unpacked__
+
+ def __init__(self, name, *, default=NoDefault):
+ self.__name__ = name
+ _DefaultMixin.__init__(self, default)
+
+ # for pickling:
+ def_mod = _caller()
+ if def_mod != 'typing_extensions':
+ self.__module__ = def_mod
+
+ self.__unpacked__ = Unpack[self]
+
+ def __repr__(self):
+ return self.__name__
+
+ def __hash__(self):
+ return object.__hash__(self)
+
+ def __eq__(self, other):
+ return self is other
+
+ def __reduce__(self):
+ return self.__name__
+
+ def __init_subclass__(self, *args, **kwds):
+ if '_root' not in kwds:
+ raise TypeError("Cannot subclass special typing classes")
+
+
+if hasattr(typing, "reveal_type"): # 3.11+
+ reveal_type = typing.reveal_type
+else: # <=3.10
+ def reveal_type(obj: T, /) -> T:
+ """Reveal the inferred type of a variable.
+
+ When a static type checker encounters a call to ``reveal_type()``,
+ it will emit the inferred type of the argument::
+
+ x: int = 1
+ reveal_type(x)
+
+ Running a static type checker (e.g., ``mypy``) on this example
+ will produce output similar to 'Revealed type is "builtins.int"'.
+
+ At runtime, the function prints the runtime type of the
+ argument and returns it unchanged.
+
+ """
+ print(f"Runtime type is {type(obj).__name__!r}", file=sys.stderr)
+ return obj
+
+
+if hasattr(typing, "_ASSERT_NEVER_REPR_MAX_LENGTH"): # 3.11+
+ _ASSERT_NEVER_REPR_MAX_LENGTH = typing._ASSERT_NEVER_REPR_MAX_LENGTH
+else: # <=3.10
+ _ASSERT_NEVER_REPR_MAX_LENGTH = 100
+
+
+if hasattr(typing, "assert_never"): # 3.11+
+ assert_never = typing.assert_never
+else: # <=3.10
+ def assert_never(arg: Never, /) -> Never:
+ """Assert to the type checker that a line of code is unreachable.
+
+ Example::
+
+ def int_or_str(arg: int | str) -> None:
+ match arg:
+ case int():
+ print("It's an int")
+ case str():
+ print("It's a str")
+ case _:
+ assert_never(arg)
+
+ If a type checker finds that a call to assert_never() is
+ reachable, it will emit an error.
+
+ At runtime, this throws an exception when called.
+
+ """
+ value = repr(arg)
+ if len(value) > _ASSERT_NEVER_REPR_MAX_LENGTH:
+ value = value[:_ASSERT_NEVER_REPR_MAX_LENGTH] + '...'
+ raise AssertionError(f"Expected code to be unreachable, but got: {value}")
+
+
+# dataclass_transform exists in 3.11 but lacks the frozen_default parameter
+# Breakpoint: https://github.com/python/cpython/pull/99958
+if sys.version_info >= (3, 12): # 3.12+
+ dataclass_transform = typing.dataclass_transform
+else: # <=3.11
+ def dataclass_transform(
+ *,
+ eq_default: bool = True,
+ order_default: bool = False,
+ kw_only_default: bool = False,
+ frozen_default: bool = False,
+ field_specifiers: typing.Tuple[
+ typing.Union[typing.Type[typing.Any], typing.Callable[..., typing.Any]],
+ ...
+ ] = (),
+ **kwargs: typing.Any,
+ ) -> typing.Callable[[T], T]:
+ """Decorator that marks a function, class, or metaclass as providing
+ dataclass-like behavior.
+
+ Example:
+
+ from bb._vendor.typing_extensions import dataclass_transform
+
+ _T = TypeVar("_T")
+
+ # Used on a decorator function
+ @dataclass_transform()
+ def create_model(cls: type[_T]) -> type[_T]:
+ ...
+ return cls
+
+ @create_model
+ class CustomerModel:
+ id: int
+ name: str
+
+ # Used on a base class
+ @dataclass_transform()
+ class ModelBase: ...
+
+ class CustomerModel(ModelBase):
+ id: int
+ name: str
+
+ # Used on a metaclass
+ @dataclass_transform()
+ class ModelMeta(type): ...
+
+ class ModelBase(metaclass=ModelMeta): ...
+
+ class CustomerModel(ModelBase):
+ id: int
+ name: str
+
+ Each of the ``CustomerModel`` classes defined in this example will now
+ behave similarly to a dataclass created with the ``@dataclasses.dataclass``
+ decorator. For example, the type checker will synthesize an ``__init__``
+ method.
+
+ The arguments to this decorator can be used to customize this behavior:
+ - ``eq_default`` indicates whether the ``eq`` parameter is assumed to be
+ True or False if it is omitted by the caller.
+ - ``order_default`` indicates whether the ``order`` parameter is
+ assumed to be True or False if it is omitted by the caller.
+ - ``kw_only_default`` indicates whether the ``kw_only`` parameter is
+ assumed to be True or False if it is omitted by the caller.
+ - ``frozen_default`` indicates whether the ``frozen`` parameter is
+ assumed to be True or False if it is omitted by the caller.
+ - ``field_specifiers`` specifies a static list of supported classes
+ or functions that describe fields, similar to ``dataclasses.field()``.
+
+ At runtime, this decorator records its arguments in the
+ ``__dataclass_transform__`` attribute on the decorated object.
+
+ See PEP 681 for details.
+
+ """
+ def decorator(cls_or_fn):
+ cls_or_fn.__dataclass_transform__ = {
+ "eq_default": eq_default,
+ "order_default": order_default,
+ "kw_only_default": kw_only_default,
+ "frozen_default": frozen_default,
+ "field_specifiers": field_specifiers,
+ "kwargs": kwargs,
+ }
+ return cls_or_fn
+ return decorator
+
+
+if hasattr(typing, "override"): # 3.12+
+ override = typing.override
+else: # <=3.11
+ _F = typing.TypeVar("_F", bound=typing.Callable[..., typing.Any])
+
+ def override(arg: _F, /) -> _F:
+ """Indicate that a method is intended to override a method in a base class.
+
+ Usage:
+
+ class Base:
+ def method(self) -> None:
+ pass
+
+ class Child(Base):
+ @override
+ def method(self) -> None:
+ super().method()
+
+ When this decorator is applied to a method, the type checker will
+ validate that it overrides a method with the same name on a base class.
+ This helps prevent bugs that may occur when a base class is changed
+ without an equivalent change to a child class.
+
+ There is no runtime checking of these properties. The decorator
+ sets the ``__override__`` attribute to ``True`` on the decorated object
+ to allow runtime introspection.
+
+ See PEP 698 for details.
+
+ """
+ try:
+ arg.__override__ = True
+ except (AttributeError, TypeError):
+ # Skip the attribute silently if it is not writable.
+ # AttributeError happens if the object has __slots__ or a
+ # read-only property, TypeError if it's a builtin class.
+ pass
+ return arg
+
+
+# Python 3.13.3+ contains a fix for the wrapped __new__
+# Breakpoint: https://github.com/python/cpython/pull/132160
+if sys.version_info >= (3, 13, 3):
+ deprecated = warnings.deprecated
+else:
+ _T = typing.TypeVar("_T")
+
+ class deprecated:
+ """Indicate that a class, function or overload is deprecated.
+
+ When this decorator is applied to an object, the type checker
+ will generate a diagnostic on usage of the deprecated object.
+
+ Usage:
+
+ @deprecated("Use B instead")
+ class A:
+ pass
+
+ @deprecated("Use g instead")
+ def f():
+ pass
+
+ @overload
+ @deprecated("int support is deprecated")
+ def g(x: int) -> int: ...
+ @overload
+ def g(x: str) -> int: ...
+
+ The warning specified by *category* will be emitted at runtime
+ on use of deprecated objects. For functions, that happens on calls;
+ for classes, on instantiation and on creation of subclasses.
+ If the *category* is ``None``, no warning is emitted at runtime.
+ The *stacklevel* determines where the
+ warning is emitted. If it is ``1`` (the default), the warning
+ is emitted at the direct caller of the deprecated object; if it
+ is higher, it is emitted further up the stack.
+ Static type checker behavior is not affected by the *category*
+ and *stacklevel* arguments.
+
+ The deprecation message passed to the decorator is saved in the
+ ``__deprecated__`` attribute on the decorated object.
+ If applied to an overload, the decorator
+ must be after the ``@overload`` decorator for the attribute to
+ exist on the overload as returned by ``get_overloads()``.
+
+ See PEP 702 for details.
+
+ """
+ def __init__(
+ self,
+ message: str,
+ /,
+ *,
+ category: typing.Optional[typing.Type[Warning]] = DeprecationWarning,
+ stacklevel: int = 1,
+ ) -> None:
+ if not isinstance(message, str):
+ raise TypeError(
+ "Expected an object of type str for 'message', not "
+ f"{type(message).__name__!r}"
+ )
+ self.message = message
+ self.category = category
+ self.stacklevel = stacklevel
+
+ def __call__(self, arg: _T, /) -> _T:
+ # Make sure the inner functions created below don't
+ # retain a reference to self.
+ msg = self.message
+ category = self.category
+ stacklevel = self.stacklevel
+ if category is None:
+ arg.__deprecated__ = msg
+ return arg
+ elif isinstance(arg, type):
+ import functools
+ from types import MethodType
+
+ original_new = arg.__new__
+
+ @functools.wraps(original_new)
+ def __new__(cls, /, *args, **kwargs):
+ if cls is arg:
+ warnings.warn(msg, category=category, stacklevel=stacklevel + 1)
+ if original_new is not object.__new__:
+ return original_new(cls, *args, **kwargs)
+ # Mirrors a similar check in object.__new__.
+ elif cls.__init__ is object.__init__ and (args or kwargs):
+ raise TypeError(f"{cls.__name__}() takes no arguments")
+ else:
+ return original_new(cls)
+
+ arg.__new__ = staticmethod(__new__)
+
+ original_init_subclass = arg.__init_subclass__
+ # We need slightly different behavior if __init_subclass__
+ # is a bound method (likely if it was implemented in Python)
+ if isinstance(original_init_subclass, MethodType):
+ original_init_subclass = original_init_subclass.__func__
+
+ @functools.wraps(original_init_subclass)
+ def __init_subclass__(*args, **kwargs):
+ warnings.warn(msg, category=category, stacklevel=stacklevel + 1)
+ return original_init_subclass(*args, **kwargs)
+
+ arg.__init_subclass__ = classmethod(__init_subclass__)
+ # Or otherwise, which likely means it's a builtin such as
+ # object's implementation of __init_subclass__.
+ else:
+ @functools.wraps(original_init_subclass)
+ def __init_subclass__(*args, **kwargs):
+ warnings.warn(msg, category=category, stacklevel=stacklevel + 1)
+ return original_init_subclass(*args, **kwargs)
+
+ arg.__init_subclass__ = __init_subclass__
+
+ arg.__deprecated__ = __new__.__deprecated__ = msg
+ __init_subclass__.__deprecated__ = msg
+ return arg
+ elif callable(arg):
+ import asyncio.coroutines
+ import functools
+ import inspect
+
+ @functools.wraps(arg)
+ def wrapper(*args, **kwargs):
+ warnings.warn(msg, category=category, stacklevel=stacklevel + 1)
+ return arg(*args, **kwargs)
+
+ if asyncio.coroutines.iscoroutinefunction(arg):
+ # Breakpoint: https://github.com/python/cpython/pull/99247
+ if sys.version_info >= (3, 12):
+ wrapper = inspect.markcoroutinefunction(wrapper)
+ else:
+ wrapper._is_coroutine = asyncio.coroutines._is_coroutine
+
+ arg.__deprecated__ = wrapper.__deprecated__ = msg
+ return wrapper
+ else:
+ raise TypeError(
+ "@deprecated decorator with non-None category must be applied to "
+ f"a class or callable, not {arg!r}"
+ )
+
+# Breakpoint: https://github.com/python/cpython/pull/23702
+if sys.version_info < (3, 10):
+ def _is_param_expr(arg):
+ return arg is ... or isinstance(
+ arg, (tuple, list, ParamSpec, _ConcatenateGenericAlias)
+ )
+else:
+ def _is_param_expr(arg):
+ return arg is ... or isinstance(
+ arg,
+ (
+ tuple,
+ list,
+ ParamSpec,
+ _ConcatenateGenericAlias,
+ typing._ConcatenateGenericAlias,
+ ),
+ )
+
+
+# We have to do some monkey patching to deal with the dual nature of
+# Unpack/TypeVarTuple:
+# - We want Unpack to be a kind of TypeVar so it gets accepted in
+# Generic[Unpack[Ts]]
+# - We want it to *not* be treated as a TypeVar for the purposes of
+# counting generic parameters, so that when we subscript a generic,
+# the runtime doesn't try to substitute the Unpack with the subscripted type.
+if not hasattr(typing, "TypeVarTuple"):
+ def _check_generic(cls, parameters, elen=_marker):
+ """Check correct count for parameters of a generic cls (internal helper).
+
+ This gives a nice error message in case of count mismatch.
+ """
+ # If substituting a single ParamSpec with multiple arguments
+ # we do not check the count
+ if (inspect.isclass(cls) and issubclass(cls, typing.Generic)
+ and len(cls.__parameters__) == 1
+ and isinstance(cls.__parameters__[0], ParamSpec)
+ and parameters
+ and not _is_param_expr(parameters[0])
+ ):
+ # Generic modifies parameters variable, but here we cannot do this
+ return
+
+ if not elen:
+ raise TypeError(f"{cls} is not a generic class")
+ if elen is _marker:
+ if not hasattr(cls, "__parameters__") or not cls.__parameters__:
+ raise TypeError(f"{cls} is not a generic class")
+ elen = len(cls.__parameters__)
+ alen = len(parameters)
+ if alen != elen:
+ expect_val = elen
+ if hasattr(cls, "__parameters__"):
+ parameters = [p for p in cls.__parameters__ if not _is_unpack(p)]
+ num_tv_tuples = sum(isinstance(p, TypeVarTuple) for p in parameters)
+ if (num_tv_tuples > 0) and (alen >= elen - num_tv_tuples):
+ return
+
+ # deal with TypeVarLike defaults
+ # required TypeVarLikes cannot appear after a defaulted one.
+ if alen < elen:
+ # since we validate TypeVarLike default in _collect_type_vars
+ # or _collect_parameters we can safely check parameters[alen]
+ if (
+ getattr(parameters[alen], '__default__', NoDefault)
+ is not NoDefault
+ ):
+ return
+
+ num_default_tv = sum(getattr(p, '__default__', NoDefault)
+ is not NoDefault for p in parameters)
+
+ elen -= num_default_tv
+
+ expect_val = f"at least {elen}"
+
+ # Breakpoint: https://github.com/python/cpython/pull/27515
+ things = "arguments" if sys.version_info >= (3, 10) else "parameters"
+ raise TypeError(f"Too {'many' if alen > elen else 'few'} {things}"
+ f" for {cls}; actual {alen}, expected {expect_val}")
+else:
+ # Python 3.11+
+
+ def _check_generic(cls, parameters, elen):
+ """Check correct count for parameters of a generic cls (internal helper).
+
+ This gives a nice error message in case of count mismatch.
+ """
+ if not elen:
+ raise TypeError(f"{cls} is not a generic class")
+ alen = len(parameters)
+ if alen != elen:
+ expect_val = elen
+ if hasattr(cls, "__parameters__"):
+ parameters = [p for p in cls.__parameters__ if not _is_unpack(p)]
+
+ # deal with TypeVarLike defaults
+ # required TypeVarLikes cannot appear after a defaulted one.
+ if alen < elen:
+ # since we validate TypeVarLike default in _collect_type_vars
+ # or _collect_parameters we can safely check parameters[alen]
+ if (
+ getattr(parameters[alen], '__default__', NoDefault)
+ is not NoDefault
+ ):
+ return
+
+ num_default_tv = sum(getattr(p, '__default__', NoDefault)
+ is not NoDefault for p in parameters)
+
+ elen -= num_default_tv
+
+ expect_val = f"at least {elen}"
+
+ raise TypeError(f"Too {'many' if alen > elen else 'few'} arguments"
+ f" for {cls}; actual {alen}, expected {expect_val}")
+
+if not _PEP_696_IMPLEMENTED:
+ typing._check_generic = _check_generic
+
+
+def _has_generic_or_protocol_as_origin() -> bool:
+ try:
+ frame = sys._getframe(2)
+ # - Catch AttributeError: not all Python implementations have sys._getframe()
+ # - Catch ValueError: maybe we're called from an unexpected module
+ # and the call stack isn't deep enough
+ except (AttributeError, ValueError):
+ return False # err on the side of leniency
+ else:
+ # If we somehow get invoked from outside typing.py,
+ # also err on the side of leniency
+ if frame.f_globals.get("__name__") != "typing":
+ return False
+ origin = frame.f_locals.get("origin")
+ # Cannot use "in" because origin may be an object with a buggy __eq__ that
+ # throws an error.
+ return origin is typing.Generic or origin is Protocol or origin is typing.Protocol
+
+
+_TYPEVARTUPLE_TYPES = {TypeVarTuple, getattr(typing, "TypeVarTuple", None)}
+
+
+def _is_unpacked_typevartuple(x) -> bool:
+ if get_origin(x) is not Unpack:
+ return False
+ args = get_args(x)
+ return (
+ bool(args)
+ and len(args) == 1
+ and type(args[0]) in _TYPEVARTUPLE_TYPES
+ )
+
+
+# Python 3.11+ _collect_type_vars was renamed to _collect_parameters
+if hasattr(typing, '_collect_type_vars'):
+ def _collect_type_vars(types, typevar_types=None):
+ """Collect all type variable contained in types in order of
+ first appearance (lexicographic order). For example::
+
+ _collect_type_vars((T, List[S, T])) == (T, S)
+ """
+ if typevar_types is None:
+ typevar_types = typing.TypeVar
+ tvars = []
+
+ # A required TypeVarLike cannot appear after a TypeVarLike with a default
+ # if it was a direct call to `Generic[]` or `Protocol[]`
+ enforce_default_ordering = _has_generic_or_protocol_as_origin()
+ default_encountered = False
+
+ # Also, a TypeVarLike with a default cannot appear after a TypeVarTuple
+ type_var_tuple_encountered = False
+
+ for t in types:
+ if _is_unpacked_typevartuple(t):
+ type_var_tuple_encountered = True
+ elif (
+ isinstance(t, typevar_types) and not isinstance(t, _UnpackAlias)
+ and t not in tvars
+ ):
+ if enforce_default_ordering:
+ has_default = getattr(t, '__default__', NoDefault) is not NoDefault
+ if has_default:
+ if type_var_tuple_encountered:
+ raise TypeError('Type parameter with a default'
+ ' follows TypeVarTuple')
+ default_encountered = True
+ elif default_encountered:
+ raise TypeError(f'Type parameter {t!r} without a default'
+ ' follows type parameter with a default')
+
+ tvars.append(t)
+ if _should_collect_from_parameters(t):
+ tvars.extend([t for t in t.__parameters__ if t not in tvars])
+ elif isinstance(t, tuple):
+ # Collect nested type_vars
+ # tuple wrapped by _prepare_paramspec_params(cls, params)
+ for x in t:
+ for collected in _collect_type_vars([x]):
+ if collected not in tvars:
+ tvars.append(collected)
+ return tuple(tvars)
+
+ typing._collect_type_vars = _collect_type_vars
+else:
+ def _collect_parameters(args):
+ """Collect all type variables and parameter specifications in args
+ in order of first appearance (lexicographic order).
+
+ For example::
+
+ assert _collect_parameters((T, Callable[P, T])) == (T, P)
+ """
+ parameters = []
+
+ # A required TypeVarLike cannot appear after a TypeVarLike with default
+ # if it was a direct call to `Generic[]` or `Protocol[]`
+ enforce_default_ordering = _has_generic_or_protocol_as_origin()
+ default_encountered = False
+
+ # Also, a TypeVarLike with a default cannot appear after a TypeVarTuple
+ type_var_tuple_encountered = False
+
+ for t in args:
+ if isinstance(t, type):
+ # We don't want __parameters__ descriptor of a bare Python class.
+ pass
+ elif isinstance(t, tuple):
+ # `t` might be a tuple, when `ParamSpec` is substituted with
+ # `[T, int]`, or `[int, *Ts]`, etc.
+ for x in t:
+ for collected in _collect_parameters([x]):
+ if collected not in parameters:
+ parameters.append(collected)
+ elif hasattr(t, '__typing_subst__'):
+ if t not in parameters:
+ if enforce_default_ordering:
+ has_default = (
+ getattr(t, '__default__', NoDefault) is not NoDefault
+ )
+
+ if type_var_tuple_encountered and has_default:
+ raise TypeError('Type parameter with a default'
+ ' follows TypeVarTuple')
+
+ if has_default:
+ default_encountered = True
+ elif default_encountered:
+ raise TypeError(f'Type parameter {t!r} without a default'
+ ' follows type parameter with a default')
+
+ parameters.append(t)
+ else:
+ if _is_unpacked_typevartuple(t):
+ type_var_tuple_encountered = True
+ for x in getattr(t, '__parameters__', ()):
+ if x not in parameters:
+ parameters.append(x)
+
+ return tuple(parameters)
+
+ if not _PEP_696_IMPLEMENTED:
+ typing._collect_parameters = _collect_parameters
+
+# Backport typing.NamedTuple as it exists in Python 3.13.
+# In 3.11, the ability to define generic `NamedTuple`s was supported.
+# This was explicitly disallowed in 3.9-3.10, and only half-worked in <=3.8.
+# On 3.12, we added __orig_bases__ to call-based NamedTuples
+# On 3.13, we deprecated kwargs-based NamedTuples
+# Breakpoint: https://github.com/python/cpython/pull/105609
+if sys.version_info >= (3, 13):
+ NamedTuple = typing.NamedTuple
+else:
+ def _make_nmtuple(name, types, module, defaults=()):
+ fields = [n for n, t in types]
+ annotations = {n: typing._type_check(t, f"field {n} annotation must be a type")
+ for n, t in types}
+ nm_tpl = collections.namedtuple(name, fields,
+ defaults=defaults, module=module)
+ nm_tpl.__annotations__ = nm_tpl.__new__.__annotations__ = annotations
+ return nm_tpl
+
+ _prohibited_namedtuple_fields = typing._prohibited
+ _special_namedtuple_fields = frozenset({'__module__', '__name__', '__annotations__'})
+
+ class _NamedTupleMeta(type):
+ def __new__(cls, typename, bases, ns):
+ assert _NamedTuple in bases
+ for base in bases:
+ if base is not _NamedTuple and base is not typing.Generic:
+ raise TypeError(
+ 'can only inherit from a NamedTuple type and Generic')
+ bases = tuple(tuple if base is _NamedTuple else base for base in bases)
+ if "__annotations__" in ns:
+ types = ns["__annotations__"]
+ elif "__annotate__" in ns:
+ # TODO: Use inspect.VALUE here, and make the annotations lazily evaluated
+ types = ns["__annotate__"](1)
+ else:
+ types = {}
+ default_names = []
+ for field_name in types:
+ if field_name in ns:
+ default_names.append(field_name)
+ elif default_names:
+ raise TypeError(f"Non-default namedtuple field {field_name} "
+ f"cannot follow default field"
+ f"{'s' if len(default_names) > 1 else ''} "
+ f"{', '.join(default_names)}")
+ nm_tpl = _make_nmtuple(
+ typename, types.items(),
+ defaults=[ns[n] for n in default_names],
+ module=ns['__module__']
+ )
+ nm_tpl.__bases__ = bases
+ if typing.Generic in bases:
+ if hasattr(typing, '_generic_class_getitem'): # 3.12+
+ nm_tpl.__class_getitem__ = classmethod(typing._generic_class_getitem)
+ else:
+ class_getitem = typing.Generic.__class_getitem__.__func__
+ nm_tpl.__class_getitem__ = classmethod(class_getitem)
+ # update from user namespace without overriding special namedtuple attributes
+ for key, val in ns.items():
+ if key in _prohibited_namedtuple_fields:
+ raise AttributeError("Cannot overwrite NamedTuple attribute " + key)
+ elif key not in _special_namedtuple_fields:
+ if key not in nm_tpl._fields:
+ setattr(nm_tpl, key, ns[key])
+ try:
+ set_name = type(val).__set_name__
+ except AttributeError:
+ pass
+ else:
+ try:
+ set_name(val, nm_tpl, key)
+ except BaseException as e:
+ msg = (
+ f"Error calling __set_name__ on {type(val).__name__!r} "
+ f"instance {key!r} in {typename!r}"
+ )
+ # BaseException.add_note() existed on py311,
+ # but the __set_name__ machinery didn't start
+ # using add_note() until py312.
+ # Making sure exceptions are raised in the same way
+ # as in "normal" classes seems most important here.
+ # Breakpoint: https://github.com/python/cpython/pull/95915
+ if sys.version_info >= (3, 12):
+ e.add_note(msg)
+ raise
+ else:
+ raise RuntimeError(msg) from e
+
+ if typing.Generic in bases:
+ nm_tpl.__init_subclass__()
+ return nm_tpl
+
+ _NamedTuple = type.__new__(_NamedTupleMeta, 'NamedTuple', (), {})
+
+ def _namedtuple_mro_entries(bases):
+ assert NamedTuple in bases
+ return (_NamedTuple,)
+
+ def NamedTuple(typename, fields=_marker, /, **kwargs):
+ """Typed version of namedtuple.
+
+ Usage::
+
+ class Employee(NamedTuple):
+ name: str
+ id: int
+
+ This is equivalent to::
+
+ Employee = collections.namedtuple('Employee', ['name', 'id'])
+
+ The resulting class has an extra __annotations__ attribute, giving a
+ dict that maps field names to types. (The field names are also in
+ the _fields attribute, which is part of the namedtuple API.)
+ An alternative equivalent functional syntax is also accepted::
+
+ Employee = NamedTuple('Employee', [('name', str), ('id', int)])
+ """
+ if fields is _marker:
+ if kwargs:
+ deprecated_thing = "Creating NamedTuple classes using keyword arguments"
+ deprecation_msg = (
+ "{name} is deprecated and will be disallowed in Python {remove}. "
+ "Use the class-based or functional syntax instead."
+ )
+ else:
+ deprecated_thing = "Failing to pass a value for the 'fields' parameter"
+ example = f"`{typename} = NamedTuple({typename!r}, [])`"
+ deprecation_msg = (
+ "{name} is deprecated and will be disallowed in Python {remove}. "
+ "To create a NamedTuple class with 0 fields "
+ "using the functional syntax, "
+ "pass an empty list, e.g. "
+ ) + example + "."
+ elif fields is None:
+ if kwargs:
+ raise TypeError(
+ "Cannot pass `None` as the 'fields' parameter "
+ "and also specify fields using keyword arguments"
+ )
+ else:
+ deprecated_thing = "Passing `None` as the 'fields' parameter"
+ example = f"`{typename} = NamedTuple({typename!r}, [])`"
+ deprecation_msg = (
+ "{name} is deprecated and will be disallowed in Python {remove}. "
+ "To create a NamedTuple class with 0 fields "
+ "using the functional syntax, "
+ "pass an empty list, e.g. "
+ ) + example + "."
+ elif kwargs:
+ raise TypeError("Either list of fields or keywords"
+ " can be provided to NamedTuple, not both")
+ if fields is _marker or fields is None:
+ warnings.warn(
+ deprecation_msg.format(name=deprecated_thing, remove="3.15"),
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ fields = kwargs.items()
+ nt = _make_nmtuple(typename, fields, module=_caller())
+ nt.__orig_bases__ = (NamedTuple,)
+ return nt
+
+ NamedTuple.__mro_entries__ = _namedtuple_mro_entries
+
+
+if hasattr(collections.abc, "Buffer"):
+ Buffer = collections.abc.Buffer
+else:
+ class Buffer(abc.ABC): # noqa: B024
+ """Base class for classes that implement the buffer protocol.
+
+ The buffer protocol allows Python objects to expose a low-level
+ memory buffer interface. Before Python 3.12, it is not possible
+ to implement the buffer protocol in pure Python code, or even
+ to check whether a class implements the buffer protocol. In
+ Python 3.12 and higher, the ``__buffer__`` method allows access
+ to the buffer protocol from Python code, and the
+ ``collections.abc.Buffer`` ABC allows checking whether a class
+ implements the buffer protocol.
+
+ To indicate support for the buffer protocol in earlier versions,
+ inherit from this ABC, either in a stub file or at runtime,
+ or use ABC registration. This ABC provides no methods, because
+ there is no Python-accessible methods shared by pre-3.12 buffer
+ classes. It is useful primarily for static checks.
+
+ """
+
+ # As a courtesy, register the most common stdlib buffer classes.
+ Buffer.register(memoryview)
+ Buffer.register(bytearray)
+ Buffer.register(bytes)
+
+
+# Backport of types.get_original_bases, available on 3.12+ in CPython
+if hasattr(_types, "get_original_bases"):
+ get_original_bases = _types.get_original_bases
+else:
+ def get_original_bases(cls, /):
+ """Return the class's "original" bases prior to modification by `__mro_entries__`.
+
+ Examples::
+
+ from typing import TypeVar, Generic
+ from bb._vendor.typing_extensions import NamedTuple, TypedDict
+
+ T = TypeVar("T")
+ class Foo(Generic[T]): ...
+ class Bar(Foo[int], float): ...
+ class Baz(list[str]): ...
+ Eggs = NamedTuple("Eggs", [("a", int), ("b", str)])
+ Spam = TypedDict("Spam", {"a": int, "b": str})
+
+ assert get_original_bases(Bar) == (Foo[int], float)
+ assert get_original_bases(Baz) == (list[str],)
+ assert get_original_bases(Eggs) == (NamedTuple,)
+ assert get_original_bases(Spam) == (TypedDict,)
+ assert get_original_bases(int) == (object,)
+ """
+ try:
+ return cls.__dict__.get("__orig_bases__", cls.__bases__)
+ except AttributeError:
+ raise TypeError(
+ f'Expected an instance of type, not {type(cls).__name__!r}'
+ ) from None
+
+
+# NewType is a class on Python 3.10+, making it pickleable
+# The error message for subclassing instances of NewType was improved on 3.11+
+# Breakpoint: https://github.com/python/cpython/pull/30268
+if sys.version_info >= (3, 11):
+ NewType = typing.NewType
+else:
+ class NewType:
+ """NewType creates simple unique types with almost zero
+ runtime overhead. NewType(name, tp) is considered a subtype of tp
+ by static type checkers. At runtime, NewType(name, tp) returns
+ a dummy callable that simply returns its argument. Usage::
+ UserId = NewType('UserId', int)
+ def name_by_id(user_id: UserId) -> str:
+ ...
+ UserId('user') # Fails type check
+ name_by_id(42) # Fails type check
+ name_by_id(UserId(42)) # OK
+ num = UserId(5) + 1 # type: int
+ """
+
+ def __call__(self, obj, /):
+ return obj
+
+ def __init__(self, name, tp):
+ self.__qualname__ = name
+ if '.' in name:
+ name = name.rpartition('.')[-1]
+ self.__name__ = name
+ self.__supertype__ = tp
+ def_mod = _caller()
+ if def_mod != 'typing_extensions':
+ self.__module__ = def_mod
+
+ def __mro_entries__(self, bases):
+ # We defined __mro_entries__ to get a better error message
+ # if a user attempts to subclass a NewType instance. bpo-46170
+ supercls_name = self.__name__
+
+ class Dummy:
+ def __init_subclass__(cls):
+ subcls_name = cls.__name__
+ raise TypeError(
+ f"Cannot subclass an instance of NewType. "
+ f"Perhaps you were looking for: "
+ f"`{subcls_name} = NewType({subcls_name!r}, {supercls_name})`"
+ )
+
+ return (Dummy,)
+
+ def __repr__(self):
+ return f'{self.__module__}.{self.__qualname__}'
+
+ def __reduce__(self):
+ return self.__qualname__
+
+ # Breakpoint: https://github.com/python/cpython/pull/21515
+ if sys.version_info >= (3, 10):
+ # PEP 604 methods
+ # It doesn't make sense to have these methods on Python <3.10
+
+ def __or__(self, other):
+ return typing.Union[self, other]
+
+ def __ror__(self, other):
+ return typing.Union[other, self]
+
+
+# Breakpoint: https://github.com/python/cpython/pull/124795
+if sys.version_info >= (3, 14):
+ TypeAliasType = typing.TypeAliasType
+# <=3.13
+else:
+ # Breakpoint: https://github.com/python/cpython/pull/103764
+ if sys.version_info >= (3, 12):
+ # 3.12-3.13
+ def _is_unionable(obj):
+ """Corresponds to is_unionable() in unionobject.c in CPython."""
+ return obj is None or isinstance(obj, (
+ type,
+ _types.GenericAlias,
+ _types.UnionType,
+ typing.TypeAliasType,
+ TypeAliasType,
+ ))
+ else:
+ # <=3.11
+ def _is_unionable(obj):
+ """Corresponds to is_unionable() in unionobject.c in CPython."""
+ return obj is None or isinstance(obj, (
+ type,
+ _types.GenericAlias,
+ _types.UnionType,
+ TypeAliasType,
+ ))
+
+ if sys.version_info < (3, 10):
+ # Copied and pasted from https://github.com/python/cpython/blob/986a4e1b6fcae7fe7a1d0a26aea446107dd58dd2/Objects/genericaliasobject.c#L568-L582,
+ # so that we emulate the behaviour of `types.GenericAlias`
+ # on the latest versions of CPython
+ _ATTRIBUTE_DELEGATION_EXCLUSIONS = frozenset({
+ "__class__",
+ "__bases__",
+ "__origin__",
+ "__args__",
+ "__unpacked__",
+ "__parameters__",
+ "__typing_unpacked_tuple_args__",
+ "__mro_entries__",
+ "__reduce_ex__",
+ "__reduce__",
+ "__copy__",
+ "__deepcopy__",
+ })
+
+ class _TypeAliasGenericAlias(typing._GenericAlias, _root=True):
+ def __getattr__(self, attr):
+ if attr in _ATTRIBUTE_DELEGATION_EXCLUSIONS:
+ return object.__getattr__(self, attr)
+ return getattr(self.__origin__, attr)
+
+
+ class TypeAliasType:
+ """Create named, parameterized type aliases.
+
+ This provides a backport of the new `type` statement in Python 3.12:
+
+ type ListOrSet[T] = list[T] | set[T]
+
+ is equivalent to:
+
+ T = TypeVar("T")
+ ListOrSet = TypeAliasType("ListOrSet", list[T] | set[T], type_params=(T,))
+
+ The name ListOrSet can then be used as an alias for the type it refers to.
+
+ The type_params argument should contain all the type parameters used
+ in the value of the type alias. If the alias is not generic, this
+ argument is omitted.
+
+ Static type checkers should only support type aliases declared using
+ TypeAliasType that follow these rules:
+
+ - The first argument (the name) must be a string literal.
+ - The TypeAliasType instance must be immediately assigned to a variable
+ of the same name. (For example, 'X = TypeAliasType("Y", int)' is invalid,
+ as is 'X, Y = TypeAliasType("X", int), TypeAliasType("Y", int)').
+
+ """
+
+ def __init__(self, name: str, value, *, type_params=()):
+ if not isinstance(name, str):
+ raise TypeError("TypeAliasType name must be a string")
+ if not isinstance(type_params, tuple):
+ raise TypeError("type_params must be a tuple")
+ self.__value__ = value
+ self.__type_params__ = type_params
+
+ default_value_encountered = False
+ parameters = []
+ for type_param in type_params:
+ if (
+ not isinstance(type_param, (TypeVar, TypeVarTuple, ParamSpec))
+ # <=3.11
+ # Unpack Backport passes isinstance(type_param, TypeVar)
+ or _is_unpack(type_param)
+ ):
+ raise TypeError(f"Expected a type param, got {type_param!r}")
+ has_default = (
+ getattr(type_param, '__default__', NoDefault) is not NoDefault
+ )
+ if default_value_encountered and not has_default:
+ raise TypeError(f"non-default type parameter '{type_param!r}'"
+ " follows default type parameter")
+ if has_default:
+ default_value_encountered = True
+ if isinstance(type_param, TypeVarTuple):
+ parameters.extend(type_param)
+ else:
+ parameters.append(type_param)
+ self.__parameters__ = tuple(parameters)
+ def_mod = _caller()
+ if def_mod != 'typing_extensions':
+ self.__module__ = def_mod
+ # Setting this attribute closes the TypeAliasType from further modification
+ self.__name__ = name
+
+ def __setattr__(self, name: str, value: object, /) -> None:
+ if hasattr(self, "__name__"):
+ self._raise_attribute_error(name)
+ super().__setattr__(name, value)
+
+ def __delattr__(self, name: str, /) -> Never:
+ self._raise_attribute_error(name)
+
+ def _raise_attribute_error(self, name: str) -> Never:
+ # Match the Python 3.12 error messages exactly
+ if name == "__name__":
+ raise AttributeError("readonly attribute")
+ elif name in {"__value__", "__type_params__", "__parameters__", "__module__"}:
+ raise AttributeError(
+ f"attribute '{name}' of 'typing.TypeAliasType' objects "
+ "is not writable"
+ )
+ else:
+ raise AttributeError(
+ f"'typing.TypeAliasType' object has no attribute '{name}'"
+ )
+
+ def __repr__(self) -> str:
+ return self.__name__
+
+ if sys.version_info < (3, 11):
+ def _check_single_param(self, param, recursion=0):
+ # Allow [], [int], [int, str], [int, ...], [int, T]
+ if param is ...:
+ return ...
+ if param is None:
+ return None
+ # Note in <= 3.9 _ConcatenateGenericAlias inherits from list
+ if isinstance(param, list) and recursion == 0:
+ return [self._check_single_param(arg, recursion+1)
+ for arg in param]
+ return typing._type_check(
+ param, f'Subscripting {self.__name__} requires a type.'
+ )
+
+ def _check_parameters(self, parameters):
+ if sys.version_info < (3, 11):
+ return tuple(
+ self._check_single_param(item)
+ for item in parameters
+ )
+ return tuple(typing._type_check(
+ item, f'Subscripting {self.__name__} requires a type.'
+ )
+ for item in parameters
+ )
+
+ def __getitem__(self, parameters):
+ if not self.__type_params__:
+ raise TypeError("Only generic type aliases are subscriptable")
+ if not isinstance(parameters, tuple):
+ parameters = (parameters,)
+ # Using 3.9 here will create problems with Concatenate
+ if sys.version_info >= (3, 10):
+ return _types.GenericAlias(self, parameters)
+ type_vars = _collect_type_vars(parameters)
+ parameters = self._check_parameters(parameters)
+ alias = _TypeAliasGenericAlias(self, parameters)
+ # alias.__parameters__ is not complete if Concatenate is present
+ # as it is converted to a list from which no parameters are extracted.
+ if alias.__parameters__ != type_vars:
+ alias.__parameters__ = type_vars
+ return alias
+
+ def __reduce__(self):
+ return self.__name__
+
+ def __init_subclass__(cls, *args, **kwargs):
+ raise TypeError(
+ "type 'typing_extensions.TypeAliasType' is not an acceptable base type"
+ )
+
+ # The presence of this method convinces typing._type_check
+ # that TypeAliasTypes are types.
+ def __call__(self):
+ raise TypeError("Type alias is not callable")
+
+ # Breakpoint: https://github.com/python/cpython/pull/21515
+ if sys.version_info >= (3, 10):
+ def __or__(self, right):
+ # For forward compatibility with 3.12, reject Unions
+ # that are not accepted by the built-in Union.
+ if not _is_unionable(right):
+ return NotImplemented
+ return typing.Union[self, right]
+
+ def __ror__(self, left):
+ if not _is_unionable(left):
+ return NotImplemented
+ return typing.Union[left, self]
+
+
+if hasattr(typing, "is_protocol"):
+ is_protocol = typing.is_protocol
+ get_protocol_members = typing.get_protocol_members
+else:
+ def is_protocol(tp: type, /) -> bool:
+ """Return True if the given type is a Protocol.
+
+ Example::
+
+ >>> from typing_extensions import Protocol, is_protocol
+ >>> class P(Protocol):
+ ... def a(self) -> str: ...
+ ... b: int
+ >>> is_protocol(P)
+ True
+ >>> is_protocol(int)
+ False
+ """
+ return (
+ isinstance(tp, type)
+ and getattr(tp, '_is_protocol', False)
+ and tp is not Protocol
+ and tp is not typing.Protocol
+ )
+
+ def get_protocol_members(tp: type, /) -> typing.FrozenSet[str]:
+ """Return the set of members defined in a Protocol.
+
+ Example::
+
+ >>> from typing_extensions import Protocol, get_protocol_members
+ >>> class P(Protocol):
+ ... def a(self) -> str: ...
+ ... b: int
+ >>> get_protocol_members(P)
+ frozenset({'a', 'b'})
+
+ Raise a TypeError for arguments that are not Protocols.
+ """
+ if not is_protocol(tp):
+ raise TypeError(f'{tp!r} is not a Protocol')
+ if hasattr(tp, '__protocol_attrs__'):
+ return frozenset(tp.__protocol_attrs__)
+ return frozenset(_get_protocol_attrs(tp))
+
+
+if hasattr(typing, "Doc"):
+ Doc = typing.Doc
+else:
+ class Doc:
+ """Define the documentation of a type annotation using ``Annotated``, to be
+ used in class attributes, function and method parameters, return values,
+ and variables.
+
+ The value should be a positional-only string literal to allow static tools
+ like editors and documentation generators to use it.
+
+ This complements docstrings.
+
+ The string value passed is available in the attribute ``documentation``.
+
+ Example::
+
+ >>> from typing_extensions import Annotated, Doc
+ >>> def hi(to: Annotated[str, Doc("Who to say hi to")]) -> None: ...
+ """
+ def __init__(self, documentation: str, /) -> None:
+ self.documentation = documentation
+
+ def __repr__(self) -> str:
+ return f"Doc({self.documentation!r})"
+
+ def __hash__(self) -> int:
+ return hash(self.documentation)
+
+ def __eq__(self, other: object) -> bool:
+ if not isinstance(other, Doc):
+ return NotImplemented
+ return self.documentation == other.documentation
+
+
+_CapsuleType = getattr(_types, "CapsuleType", None)
+
+if _CapsuleType is None:
+ try:
+ import _socket
+ except ImportError:
+ pass
+ else:
+ _CAPI = getattr(_socket, "CAPI", None)
+ if _CAPI is not None:
+ _CapsuleType = type(_CAPI)
+
+if _CapsuleType is not None:
+ CapsuleType = _CapsuleType
+ __all__.append("CapsuleType")
+
+
+if sys.version_info >= (3, 14):
+ from annotationlib import Format, get_annotations
+else:
+ # Available since Python 3.14.0a3
+ # PR: https://github.com/python/cpython/pull/124415
+ class Format(enum.IntEnum):
+ VALUE = 1
+ VALUE_WITH_FAKE_GLOBALS = 2
+ FORWARDREF = 3
+ STRING = 4
+
+ # Available since Python 3.14.0a1
+ # PR: https://github.com/python/cpython/pull/119891
+ def get_annotations(obj, *, globals=None, locals=None, eval_str=False,
+ format=Format.VALUE):
+ """Compute the annotations dict for an object.
+
+ obj may be a callable, class, or module.
+ Passing in an object of any other type raises TypeError.
+
+ Returns a dict. get_annotations() returns a new dict every time
+ it's called; calling it twice on the same object will return two
+ different but equivalent dicts.
+
+ This is a backport of `inspect.get_annotations`, which has been
+ in the standard library since Python 3.10. See the standard library
+ documentation for more:
+
+ https://docs.python.org/3/library/inspect.html#inspect.get_annotations
+
+ This backport adds the *format* argument introduced by PEP 649. The
+ three formats supported are:
+ * VALUE: the annotations are returned as-is. This is the default and
+ it is compatible with the behavior on previous Python versions.
+ * FORWARDREF: return annotations as-is if possible, but replace any
+ undefined names with ForwardRef objects. The implementation proposed by
+ PEP 649 relies on language changes that cannot be backported; the
+ typing-extensions implementation simply returns the same result as VALUE.
+ * STRING: return annotations as strings, in a format close to the original
+ source. Again, this behavior cannot be replicated directly in a backport.
+ As an approximation, typing-extensions retrieves the annotations under
+ VALUE semantics and then stringifies them.
+
+ The purpose of this backport is to allow users who would like to use
+ FORWARDREF or STRING semantics once PEP 649 is implemented, but who also
+ want to support earlier Python versions, to simply write:
+
+ typing_extensions.get_annotations(obj, format=Format.FORWARDREF)
+
+ """
+ format = Format(format)
+ if format is Format.VALUE_WITH_FAKE_GLOBALS:
+ raise ValueError(
+ "The VALUE_WITH_FAKE_GLOBALS format is for internal use only"
+ )
+
+ if eval_str and format is not Format.VALUE:
+ raise ValueError("eval_str=True is only supported with format=Format.VALUE")
+
+ if isinstance(obj, type):
+ # class
+ obj_dict = getattr(obj, '__dict__', None)
+ if obj_dict and hasattr(obj_dict, 'get'):
+ ann = obj_dict.get('__annotations__', None)
+ if isinstance(ann, _types.GetSetDescriptorType):
+ ann = None
+ else:
+ ann = None
+
+ obj_globals = None
+ module_name = getattr(obj, '__module__', None)
+ if module_name:
+ module = sys.modules.get(module_name, None)
+ if module:
+ obj_globals = getattr(module, '__dict__', None)
+ obj_locals = dict(vars(obj))
+ unwrap = obj
+ elif isinstance(obj, _types.ModuleType):
+ # module
+ ann = getattr(obj, '__annotations__', None)
+ obj_globals = obj.__dict__
+ obj_locals = None
+ unwrap = None
+ elif callable(obj):
+ # this includes types.Function, types.BuiltinFunctionType,
+ # types.BuiltinMethodType, functools.partial, functools.singledispatch,
+ # "class funclike" from Lib/test/test_inspect... on and on it goes.
+ ann = getattr(obj, '__annotations__', None)
+ obj_globals = getattr(obj, '__globals__', None)
+ obj_locals = None
+ unwrap = obj
+ elif hasattr(obj, '__annotations__'):
+ ann = obj.__annotations__
+ obj_globals = obj_locals = unwrap = None
+ else:
+ raise TypeError(f"{obj!r} is not a module, class, or callable.")
+
+ if ann is None:
+ return {}
+
+ if not isinstance(ann, dict):
+ raise ValueError(f"{obj!r}.__annotations__ is neither a dict nor None")
+
+ if not ann:
+ return {}
+
+ if not eval_str:
+ if format is Format.STRING:
+ return {
+ key: value if isinstance(value, str) else typing._type_repr(value)
+ for key, value in ann.items()
+ }
+ return dict(ann)
+
+ if unwrap is not None:
+ while True:
+ if hasattr(unwrap, '__wrapped__'):
+ unwrap = unwrap.__wrapped__
+ continue
+ if isinstance(unwrap, functools.partial):
+ unwrap = unwrap.func
+ continue
+ break
+ if hasattr(unwrap, "__globals__"):
+ obj_globals = unwrap.__globals__
+
+ if globals is None:
+ globals = obj_globals
+ if locals is None:
+ locals = obj_locals or {}
+
+ # "Inject" type parameters into the local namespace
+ # (unless they are shadowed by assignments *in* the local namespace),
+ # as a way of emulating annotation scopes when calling `eval()`
+ if type_params := getattr(obj, "__type_params__", ()):
+ locals = {param.__name__: param for param in type_params} | locals
+
+ return_value = {key:
+ value if not isinstance(value, str) else eval(value, globals, locals)
+ for key, value in ann.items() }
+ return return_value
+
+
+if hasattr(typing, "evaluate_forward_ref"):
+ evaluate_forward_ref = typing.evaluate_forward_ref
+else:
+ # Implements annotationlib.ForwardRef.evaluate
+ def _eval_with_owner(
+ forward_ref, *, owner=None, globals=None, locals=None, type_params=None
+ ):
+ if forward_ref.__forward_evaluated__:
+ return forward_ref.__forward_value__
+ if getattr(forward_ref, "__cell__", None) is not None:
+ try:
+ value = forward_ref.__cell__.cell_contents
+ except ValueError:
+ pass
+ else:
+ forward_ref.__forward_evaluated__ = True
+ forward_ref.__forward_value__ = value
+ return value
+ if owner is None:
+ owner = getattr(forward_ref, "__owner__", None)
+
+ if (
+ globals is None
+ and getattr(forward_ref, "__forward_module__", None) is not None
+ ):
+ globals = getattr(
+ sys.modules.get(forward_ref.__forward_module__, None), "__dict__", None
+ )
+ if globals is None:
+ globals = getattr(forward_ref, "__globals__", None)
+ if globals is None:
+ if isinstance(owner, type):
+ module_name = getattr(owner, "__module__", None)
+ if module_name:
+ module = sys.modules.get(module_name, None)
+ if module:
+ globals = getattr(module, "__dict__", None)
+ elif isinstance(owner, _types.ModuleType):
+ globals = getattr(owner, "__dict__", None)
+ elif callable(owner):
+ globals = getattr(owner, "__globals__", None)
+
+ # If we pass None to eval() below, the globals of this module are used.
+ if globals is None:
+ globals = {}
+
+ if locals is None:
+ locals = {}
+ if isinstance(owner, type):
+ locals.update(vars(owner))
+
+ if type_params is None and owner is not None:
+ # "Inject" type parameters into the local namespace
+ # (unless they are shadowed by assignments *in* the local namespace),
+ # as a way of emulating annotation scopes when calling `eval()`
+ type_params = getattr(owner, "__type_params__", None)
+
+ # Type parameters exist in their own scope, which is logically
+ # between the locals and the globals. We simulate this by adding
+ # them to the globals.
+ if type_params is not None:
+ globals = dict(globals)
+ for param in type_params:
+ globals[param.__name__] = param
+
+ arg = forward_ref.__forward_arg__
+ if arg.isidentifier() and not keyword.iskeyword(arg):
+ if arg in locals:
+ value = locals[arg]
+ elif arg in globals:
+ value = globals[arg]
+ elif hasattr(builtins, arg):
+ return getattr(builtins, arg)
+ else:
+ raise NameError(arg)
+ else:
+ code = forward_ref.__forward_code__
+ value = eval(code, globals, locals)
+ forward_ref.__forward_evaluated__ = True
+ forward_ref.__forward_value__ = value
+ return value
+
+ def evaluate_forward_ref(
+ forward_ref,
+ *,
+ owner=None,
+ globals=None,
+ locals=None,
+ type_params=None,
+ format=None,
+ _recursive_guard=frozenset(),
+ ):
+ """Evaluate a forward reference as a type hint.
+
+ This is similar to calling the ForwardRef.evaluate() method,
+ but unlike that method, evaluate_forward_ref() also:
+
+ * Recursively evaluates forward references nested within the type hint.
+ * Rejects certain objects that are not valid type hints.
+ * Replaces type hints that evaluate to None with types.NoneType.
+ * Supports the *FORWARDREF* and *STRING* formats.
+
+ *forward_ref* must be an instance of ForwardRef. *owner*, if given,
+ should be the object that holds the annotations that the forward reference
+ derived from, such as a module, class object, or function. It is used to
+ infer the namespaces to use for looking up names. *globals* and *locals*
+ can also be explicitly given to provide the global and local namespaces.
+ *type_params* is a tuple of type parameters that are in scope when
+ evaluating the forward reference. This parameter must be provided (though
+ it may be an empty tuple) if *owner* is not given and the forward reference
+ does not already have an owner set. *format* specifies the format of the
+ annotation and is a member of the annotationlib.Format enum.
+
+ """
+ if format == Format.STRING:
+ return forward_ref.__forward_arg__
+ if forward_ref.__forward_arg__ in _recursive_guard:
+ return forward_ref
+
+ # Evaluate the forward reference
+ try:
+ value = _eval_with_owner(
+ forward_ref,
+ owner=owner,
+ globals=globals,
+ locals=locals,
+ type_params=type_params,
+ )
+ except NameError:
+ if format == Format.FORWARDREF:
+ return forward_ref
+ else:
+ raise
+
+ if isinstance(value, str):
+ value = ForwardRef(value)
+
+ # Recursively evaluate the type
+ if isinstance(value, ForwardRef):
+ if getattr(value, "__forward_module__", True) is not None:
+ globals = None
+ return evaluate_forward_ref(
+ value,
+ globals=globals,
+ locals=locals,
+ type_params=type_params, owner=owner,
+ _recursive_guard=_recursive_guard, format=format
+ )
+ if sys.version_info < (3, 12, 5) and type_params:
+ # Make use of type_params
+ locals = dict(locals) if locals else {}
+ for tvar in type_params:
+ if tvar.__name__ not in locals: # lets not overwrite something present
+ locals[tvar.__name__] = tvar
+ if sys.version_info < (3, 12, 5):
+ return typing._eval_type(
+ value,
+ globals,
+ locals,
+ recursive_guard=_recursive_guard | {forward_ref.__forward_arg__},
+ )
+ else:
+ return typing._eval_type(
+ value,
+ globals,
+ locals,
+ type_params,
+ recursive_guard=_recursive_guard | {forward_ref.__forward_arg__},
+ )
+
+
+class Sentinel:
+ """Create a unique sentinel object.
+
+ *name* should be the name of the variable to which the return value shall be assigned.
+
+ *repr*, if supplied, will be used for the repr of the sentinel object.
+ If not provided, "<name>" will be used.
+ """
+
+ def __init__(
+ self,
+ name: str,
+ repr: typing.Optional[str] = None,
+ ):
+ self._name = name
+ self._repr = repr if repr is not None else f'<{name}>'
+
+ def __repr__(self):
+ return self._repr
+
+ if sys.version_info < (3, 11):
+ # The presence of this method convinces typing._type_check
+ # that Sentinels are types.
+ def __call__(self, *args, **kwargs):
+ raise TypeError(f"{type(self).__name__!r} object is not callable")
+
+ # Breakpoint: https://github.com/python/cpython/pull/21515
+ if sys.version_info >= (3, 10):
+ def __or__(self, other):
+ return typing.Union[self, other]
+
+ def __ror__(self, other):
+ return typing.Union[other, self]
+
+ def __getstate__(self):
+ raise TypeError(f"Cannot pickle {type(self).__name__!r} object")
+
+
+if sys.version_info >= (3, 14, 0, "beta"):
+ type_repr = annotationlib.type_repr
+else:
+ def type_repr(value):
+ """Convert a Python value to a format suitable for use with the STRING format.
+
+ This is intended as a helper for tools that support the STRING format but do
+ not have access to the code that originally produced the annotations. It uses
+ repr() for most objects.
+
+ """
+ if isinstance(value, (type, _types.FunctionType, _types.BuiltinFunctionType)):
+ if value.__module__ == "builtins":
+ return value.__qualname__
+ return f"{value.__module__}.{value.__qualname__}"
+ if value is ...:
+ return "..."
+ return repr(value)
+
+
+# Aliases for items that are in typing in all supported versions.
+# We use hasattr() checks so this library will continue to import on
+# future versions of Python that may remove these names.
+_typing_names = [
+ "AbstractSet",
+ "AnyStr",
+ "BinaryIO",
+ "Callable",
+ "Collection",
+ "Container",
+ "Dict",
+ "FrozenSet",
+ "Hashable",
+ "IO",
+ "ItemsView",
+ "Iterable",
+ "Iterator",
+ "KeysView",
+ "List",
+ "Mapping",
+ "MappingView",
+ "Match",
+ "MutableMapping",
+ "MutableSequence",
+ "MutableSet",
+ "Optional",
+ "Pattern",
+ "Reversible",
+ "Sequence",
+ "Set",
+ "Sized",
+ "TextIO",
+ "Tuple",
+ "Union",
+ "ValuesView",
+ "cast",
+ "no_type_check",
+ "no_type_check_decorator",
+ # This is private, but it was defined by typing_extensions for a long time
+ # and some users rely on it.
+ "_AnnotatedAlias",
+]
+globals().update(
+ {name: getattr(typing, name) for name in _typing_names if hasattr(typing, name)}
+)
+# These are defined unconditionally because they are used in
+# typing-extensions itself.
+Generic = typing.Generic
+ForwardRef = typing.ForwardRef
+Annotated = typing.Annotated
diff --git a/bitbake/lib/bb/_vendor/typing_extensions.pyi b/bitbake/lib/bb/_vendor/typing_extensions.pyi
new file mode 100644
index 00000000..547f7a1b
--- /dev/null
+++ b/bitbake/lib/bb/_vendor/typing_extensions.pyi
@@ -0,0 +1 @@
+from typing_extensions import *
\ No newline at end of file
diff --git a/bitbake/lib/bb/acl.py b/bitbake/lib/bb/acl.py
index 0f41b275..e9dbdb61 100755
--- a/bitbake/lib/bb/acl.py
+++ b/bitbake/lib/bb/acl.py
@@ -195,8 +195,6 @@ class ACL(object):
def main():
import argparse
- import pwd
- import grp
from pathlib import Path
parser = argparse.ArgumentParser()
diff --git a/bitbake/lib/bb/asyncrpc/__init__.py b/bitbake/lib/bb/asyncrpc/__init__.py
index 639e1607..2f0956df 100644
--- a/bitbake/lib/bb/asyncrpc/__init__.py
+++ b/bitbake/lib/bb/asyncrpc/__init__.py
@@ -5,7 +5,7 @@
#
-from .client import AsyncClient, Client, ClientPool
+from .client import AsyncClient, Client
from .serv import AsyncServer, AsyncServerConnection
from .connection import DEFAULT_MAX_CHUNK
from .exceptions import (
@@ -14,3 +14,4 @@ from .exceptions import (
ConnectionClosedError,
InvokeError,
)
+from .taskgroup import TaskGroup
diff --git a/bitbake/lib/bb/asyncrpc/client.py b/bitbake/lib/bb/asyncrpc/client.py
index a350b4fb..dd28fa73 100644
--- a/bitbake/lib/bb/asyncrpc/client.py
+++ b/bitbake/lib/bb/asyncrpc/client.py
@@ -7,15 +7,19 @@
import abc
import asyncio
import json
+import logging
import os
import socket
import sys
import re
import contextlib
+import weakref
from threading import Thread
from .connection import StreamConnection, WebsocketConnection, DEFAULT_MAX_CHUNK
from .exceptions import ConnectionClosedError, InvokeError
+logger = logging.getLogger("bb.asyncrpc.client")
+
UNIX_PREFIX = "unix://"
WS_PREFIX = "ws://"
WSS_PREFIX = "wss://"
@@ -24,6 +28,12 @@ ADDR_TYPE_UNIX = 0
ADDR_TYPE_TCP = 1
ADDR_TYPE_WS = 2
+WEBSOCKETS_MIN_VERSION = (9, 1)
+# Need websockets 10 with python 3.10+
+if sys.version_info >= (3, 10, 0):
+ WEBSOCKETS_MIN_VERSION = (10, 0)
+
+
def parse_address(addr):
if addr.startswith(UNIX_PREFIX):
return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX) :],))
@@ -39,6 +49,7 @@ def parse_address(addr):
return (ADDR_TYPE_TCP, (host, int(port)))
+
class AsyncClient(object):
def __init__(
self,
@@ -86,8 +97,35 @@ class AsyncClient(object):
async def connect_websocket(self, uri):
import websockets
+ try:
+ version = tuple(
+ int(v)
+ for v in websockets.__version__.split(".")[
+ 0 : len(WEBSOCKETS_MIN_VERSION)
+ ]
+ )
+ except ValueError:
+ raise ImportError(
+ f"Unable to parse websockets version '{websockets.__version__}'"
+ )
+
+ if version < WEBSOCKETS_MIN_VERSION:
+ min_ver_str = ".".join(str(v) for v in WEBSOCKETS_MIN_VERSION)
+ raise ImportError(
+ f"Websockets version {websockets.__version__} is less than minimum required version {min_ver_str}"
+ )
+
async def connect_sock():
- websocket = await websockets.connect(uri, ping_interval=None)
+ try:
+ websocket = await websockets.connect(
+ uri,
+ ping_interval=None,
+ open_timeout=self.timeout,
+ )
+ except asyncio.exceptions.TimeoutError:
+ raise ConnectionError("Timeout while connecting to websocket")
+ except (OSError, websockets.InvalidHandshake, websockets.InvalidURI) as exc:
+ raise ConnectionError(f"Could not connect to websocket: {exc}") from exc
return WebsocketConnection(websocket, self.timeout)
self._connect_sock = connect_sock
@@ -190,8 +228,49 @@ class Client(object):
# required (but harmless) with it.
asyncio.set_event_loop(self.loop)
+ # The loop and its transports must be closed even if the caller never
+ # calls close(). Since set_event_loop() above replaces the reference
+ # held by the previous client, an unclosed loop only becomes reachable
+ # for collection once another client is created, at which point
+ # asyncio reports "unclosed transport" and "unclosed event loop"
+ # ResourceWarnings. The finalizer is detached by close() so that the
+ # normal path is unaffected.
+ self._finalizer = weakref.finalize(
+ self, self._close_loop, self.loop, self.client
+ )
+
self._add_methods("connect_tcp", "ping")
+ @staticmethod
+ def _close_loop(loop, client):
+ if loop.is_closed():
+ return
+ try:
+ loop.run_until_complete(client.close())
+ loop.run_until_complete(loop.shutdown_asyncgens())
+ # This can be called from the finalizer, where an exception would be
+ # discarded by the interpreter and reported only as "Exception ignored
+ # in", so report the error here instead. The loop is closed below in
+ # every case.
+ except (OSError, ConnectionClosedError, asyncio.TimeoutError) as exc:
+ # The peer has gone away or is not responding. That is expected
+ # for a client that was never closed, so it is not worth warning
+ # about. asyncio.TimeoutError is only distinct from OSError on
+ # python older than 3.11.
+ logger.debug("Client connection already closed or unreachable: %s" % exc)
+ except RuntimeError as exc:
+ # The loop could not be run, for example because another loop is
+ # already running in this thread. Unlike the above, this means
+ # something is wrong with how the client is being used.
+ logger.warning("Could not shut down client event loop: %s" % exc)
+ except Exception as exc:
+ # Also covers the exceptions raised by websockets, which cannot be
+ # named here as it is imported lazily in connect_websocket(), and
+ # whatever may be raised while the interpreter is shutting down.
+ logger.warning("Error shutting down client connection: %s" % exc)
+ finally:
+ loop.close()
+
@abc.abstractmethod
def _get_async_client(self):
pass
@@ -224,86 +303,9 @@ class Client(object):
def close(self):
if self.loop:
- self.loop.run_until_complete(self.client.close())
- if sys.version_info >= (3, 6):
- self.loop.run_until_complete(self.loop.shutdown_asyncgens())
- self.loop.close()
- self.loop = None
-
- def __enter__(self):
- return self
-
- def __exit__(self, exc_type, exc_value, traceback):
- self.close()
- return False
-
-
-class ClientPool(object):
- def __init__(self, max_clients):
- self.avail_clients = []
- self.num_clients = 0
- self.max_clients = max_clients
+ self._finalizer.detach()
+ self._close_loop(self.loop, self.client)
self.loop = None
- self.client_condition = None
-
- @abc.abstractmethod
- async def _new_client(self):
- raise NotImplementedError("Must be implemented in derived class")
-
- def close(self):
- if self.client_condition:
- self.client_condition = None
-
- if self.loop:
- self.loop.run_until_complete(self.__close_clients())
- self.loop.run_until_complete(self.loop.shutdown_asyncgens())
- self.loop.close()
- self.loop = None
-
- def run_tasks(self, tasks):
- if not self.loop:
- self.loop = asyncio.new_event_loop()
-
- thread = Thread(target=self.__thread_main, args=(tasks,))
- thread.start()
- thread.join()
-
- @contextlib.asynccontextmanager
- async def get_client(self):
- async with self.client_condition:
- if self.avail_clients:
- client = self.avail_clients.pop()
- elif self.num_clients < self.max_clients:
- self.num_clients += 1
- client = await self._new_client()
- else:
- while not self.avail_clients:
- await self.client_condition.wait()
- client = self.avail_clients.pop()
-
- try:
- yield client
- finally:
- async with self.client_condition:
- self.avail_clients.append(client)
- self.client_condition.notify()
-
- def __thread_main(self, tasks):
- async def process_task(task):
- async with self.get_client() as client:
- await task(client)
-
- asyncio.set_event_loop(self.loop)
- if not self.client_condition:
- self.client_condition = asyncio.Condition()
- tasks = [process_task(t) for t in tasks]
- self.loop.run_until_complete(asyncio.gather(*tasks))
-
- async def __close_clients(self):
- for c in self.avail_clients:
- await c.close()
- self.avail_clients = []
- self.num_clients = 0
def __enter__(self):
return self
diff --git a/bitbake/lib/bb/asyncrpc/serv.py b/bitbake/lib/bb/asyncrpc/serv.py
index a66117ac..67fdc44d 100644
--- a/bitbake/lib/bb/asyncrpc/serv.py
+++ b/bitbake/lib/bb/asyncrpc/serv.py
@@ -11,7 +11,8 @@ import os
import signal
import socket
import sys
-import multiprocessing
+import bb
+from bb import multiprocessing
import logging
from .connection import StreamConnection, WebsocketConnection
from .exceptions import ClientError, ServerError, ConnectionClosedError, InvokeError
@@ -56,7 +57,7 @@ class AsyncServerConnection(object):
if not client_protocol:
return
- (client_proto_name, client_proto_version) = client_protocol.split()
+ client_proto_name, client_proto_version = client_protocol.split()
if client_proto_name != self.proto_name:
self.logger.debug("Rejecting invalid protocol %s" % (self.proto_name))
return
@@ -123,6 +124,7 @@ class StreamServer(object):
self.handler = handler
self.logger = logger
self.closed = False
+ self.clients = []
async def handle_stream_client(self, reader, writer):
# writer.transport.set_write_buffer_limits(0)
@@ -131,21 +133,33 @@ class StreamServer(object):
await socket.close()
return
- await self.handler(socket)
+ self.clients.append(socket)
+ try:
+ await self.handler(socket)
+ finally:
+ self.clients.remove(socket)
async def stop(self):
self.closed = True
+ for socket in self.clients:
+ await socket.close()
class TCPStreamServer(StreamServer):
- def __init__(self, host, port, handler, logger):
+ def __init__(self, host, port, handler, logger, *, reuseport=False):
super().__init__(handler, logger)
self.host = host
self.port = port
+ self.reuseport = reuseport
def start(self, loop):
self.server = loop.run_until_complete(
- asyncio.start_server(self.handle_stream_client, self.host, self.port)
+ asyncio.start_server(
+ self.handle_stream_client,
+ self.host,
+ self.port,
+ reuse_port=self.reuseport,
+ )
)
for s in self.server.sockets:
@@ -205,15 +219,19 @@ class UnixStreamServer(StreamServer):
self.server.close()
def cleanup(self):
- os.unlink(self.path)
+ try:
+ os.unlink(self.path)
+ except FileNotFoundError:
+ pass
class WebsocketsServer(object):
- def __init__(self, host, port, handler, logger):
+ def __init__(self, host, port, handler, logger, *, reuseport=False):
self.host = host
self.port = port
self.handler = handler
self.logger = logger
+ self.reuseport = reuseport
def start(self, loop):
import websockets.server
@@ -224,6 +242,7 @@ class WebsocketsServer(object):
self.host,
self.port,
ping_interval=None,
+ reuse_port=self.reuseport,
)
)
@@ -262,14 +281,26 @@ class AsyncServer(object):
self.loop = None
self.run_tasks = []
- def start_tcp_server(self, host, port):
- self.server = TCPStreamServer(host, port, self._client_handler, self.logger)
+ def start_tcp_server(self, host, port, *, reuseport=False):
+ self.server = TCPStreamServer(
+ host,
+ port,
+ self._client_handler,
+ self.logger,
+ reuseport=reuseport,
+ )
def start_unix_server(self, path):
self.server = UnixStreamServer(path, self._client_handler, self.logger)
- def start_websocket_server(self, host, port):
- self.server = WebsocketsServer(host, port, self._client_handler, self.logger)
+ def start_websocket_server(self, host, port, reuseport=False):
+ self.server = WebsocketsServer(
+ host,
+ port,
+ self._client_handler,
+ self.logger,
+ reuseport=reuseport,
+ )
async def _client_handler(self, socket):
address = socket.address
@@ -311,7 +342,7 @@ class AsyncServer(object):
self.loop.add_signal_handler(signal.SIGQUIT, self.signal_handler)
signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGTERM])
- self.loop.run_until_complete(asyncio.gather(*tasks))
+ self.loop.run_until_complete(bb.asyncrpc.TaskGroup.run(*tasks))
self.logger.debug("Server shutting down")
finally:
@@ -368,8 +399,7 @@ class AsyncServer(object):
self._serve_forever(tasks)
- if sys.version_info >= (3, 6):
- self.loop.run_until_complete(self.loop.shutdown_asyncgens())
+ self.loop.run_until_complete(self.loop.shutdown_asyncgens())
self.loop.close()
queue = multiprocessing.Queue()
diff --git a/bitbake/lib/bb/asyncrpc/taskgroup.py b/bitbake/lib/bb/asyncrpc/taskgroup.py
new file mode 100644
index 00000000..b61f4038
--- /dev/null
+++ b/bitbake/lib/bb/asyncrpc/taskgroup.py
@@ -0,0 +1,52 @@
+#
+# Copyright BitBake Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+import asyncio
+import logging
+
+logger = logging.getLogger("asyncio.TaskGroup")
+
+
+class TaskGroup(object):
+ def __init__(self):
+ self._tasks = []
+
+ def create_task(self, coro, **kwargs):
+ self._tasks.append(asyncio.create_task(coro, **kwargs))
+
+ async def __aenter__(self):
+ return self
+
+ async def __aexit__(self, exc_type, exc, tb):
+ try:
+ if exc is None:
+ while self._tasks:
+ done, pending = await asyncio.wait(
+ self._tasks, return_when=asyncio.FIRST_COMPLETED
+ )
+ self._tasks = pending
+
+ for t in done:
+ try:
+ await t
+ except asyncio.CancelledError:
+ pass
+
+ finally:
+ for t in self._tasks:
+ t.cancel()
+ try:
+ await t
+ except:
+ # Ignore exceptions
+ pass
+
+ return False
+
+ @classmethod
+ async def run(cls, *coros):
+ async with cls() as group:
+ for c in coros:
+ group.create_task(c)
diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 44d08f5c..8b49721d 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -24,7 +24,6 @@ import datetime
import bb
import bb.msg
import bb.process
-import bb.progress
from io import StringIO
from bb import data, event, utils
@@ -197,6 +196,8 @@ def exec_func(func, d, dirs = None):
for cdir in d.expand(cleandirs).split():
bb.utils.remove(cdir, True)
bb.utils.mkdirhier(cdir)
+ if cdir == oldcwd:
+ os.chdir(cdir)
if flags and dirs is None:
dirs = flags.get('dirs')
@@ -385,6 +386,7 @@ esac
'''
def create_progress_handler(func, progress, logfile, d):
+ import bb.progress
if progress == 'percent':
# Use default regex
return bb.progress.BasicProgressHandler(d, outfile=logfile)
@@ -395,7 +397,7 @@ def create_progress_handler(func, progress, logfile, d):
# Use specified regex
return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
elif progress.startswith("custom:"):
- # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__
+ # Use a custom progress handler that was injected via other means
import functools
from types import ModuleType
@@ -741,7 +743,7 @@ def _exec_task(fn, task, d, quieterr):
if quieterr:
if not handled:
- logger.warning(repr(exc))
+ logger.warning(str(exc))
event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata)
else:
errprinted = errchk.triggered
@@ -750,7 +752,7 @@ def _exec_task(fn, task, d, quieterr):
if verboseStdoutLogging or handled:
errprinted = True
if not handled:
- logger.error(repr(exc))
+ logger.error(str(exc))
event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata)
return 1
@@ -930,9 +932,13 @@ def add_tasks(tasklist, d):
# don't assume holding a reference
d.setVar('_task_deps', task_deps)
+def ensure_task_prefix(name):
+ if name[:3] != "do_":
+ name = "do_" + name
+ return name
+
def addtask(task, before, after, d):
- if task[:3] != "do_":
- task = "do_" + task
+ task = ensure_task_prefix(task)
d.setVarFlag(task, "task", 1)
bbtasks = d.getVar('__BBTASKS', False) or []
@@ -944,19 +950,20 @@ def addtask(task, before, after, d):
if after is not None:
# set up deps for function
for entry in after.split():
+ entry = ensure_task_prefix(entry)
if entry not in existing:
existing.append(entry)
d.setVarFlag(task, "deps", existing)
if before is not None:
# set up things that depend on this func
for entry in before.split():
+ entry = ensure_task_prefix(entry)
existing = d.getVarFlag(entry, "deps", False) or []
if task not in existing:
d.setVarFlag(entry, "deps", [task] + existing)
def deltask(task, d):
- if task[:3] != "do_":
- task = "do_" + task
+ task = ensure_task_prefix(task)
bbtasks = d.getVar('__BBTASKS', False) or []
if task in bbtasks:
@@ -1021,3 +1028,9 @@ def tasksbetween(task_start, task_end, d):
chain.pop()
follow_chain(task_start, task_end)
return outtasks
+
+def listtasks(d):
+ """
+ Return the list of tasks in the current recipe.
+ """
+ return tuple(d.getVar('__BBTASKS', False) or ())
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 18d5574a..24bde62a 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -28,7 +28,7 @@ import shutil
logger = logging.getLogger("BitBake.Cache")
-__cache_version__ = "155"
+__cache_version__ = "156"
def getCacheFile(path, filename, mc, data_hash):
mcspec = ''
@@ -395,7 +395,7 @@ class Cache(object):
# It will be used later for deciding whether we
# need extra cache file dump/load support
self.mc = mc
- self.logger = PrefixLoggerAdapter("Cache: %s: " % (mc if mc else "default"), logger)
+ self.logger = PrefixLoggerAdapter("Cache: %s: " % (mc if mc else ''), logger)
self.caches_array = caches_array
self.cachedir = self.data.getVar("CACHE")
self.clean = set()
@@ -441,7 +441,7 @@ class Cache(object):
else:
symlink = os.path.join(self.cachedir, "bb_cache.dat")
- if os.path.exists(symlink):
+ if os.path.exists(symlink) or os.path.islink(symlink):
bb.utils.remove(symlink)
try:
os.symlink(os.path.basename(self.cachefile), symlink)
@@ -736,7 +736,7 @@ class MulticonfigCache(Mapping):
# we might have calculated incorrect total size because a file
# might've been written out just after we checked its size
cachesize = current_progress
- current_percent = 100 * current_progress / cachesize
+ current_percent = 100 * current_progress // cachesize
if current_percent > previous_percent:
previous_percent = current_percent
bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
@@ -779,25 +779,6 @@ class MulticonfigCache(Mapping):
for k in self.__caches:
yield k
-def init(cooker):
- """
- The Objective: Cache the minimum amount of data possible yet get to the
- stage of building packages (i.e. tryBuild) without reparsing any .bb files.
-
- To do this, we intercept getVar calls and only cache the variables we see
- being accessed. We rely on the cache getVar calls being made for all
- variables bitbake might need to use to reach this stage. For each cached
- file we need to track:
-
- * Its mtime
- * The mtimes of all its dependencies
- * Whether it caused a parse.SkipRecipe exception
-
- Files causing parsing errors are evicted from the cache.
-
- """
- return Cache(cooker.configuration.data, cooker.configuration.data_hash)
-
class CacheData(object):
"""
@@ -866,6 +847,16 @@ class MultiProcessCache(object):
data = [{}]
return data
+ def clear_cache(self):
+ if not self.cachefile:
+ bb.fatal("Can't clear invalid cachefile")
+
+ self.cachedata = self.create_cachedata()
+ self.cachedata_extras = self.create_cachedata()
+ with bb.utils.fileslocked([self.cachefile + ".lock"]):
+ bb.utils.remove(self.cachefile)
+ bb.utils.remove(self.cachefile + "-*")
+
def save_extras(self):
if not self.cachefile:
return
diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
index 557793d3..3fb39a30 100644
--- a/bitbake/lib/bb/checksum.py
+++ b/bitbake/lib/bb/checksum.py
@@ -142,3 +142,28 @@ class FileChecksumCache(MultiProcessCache):
checksums.sort(key=operator.itemgetter(1))
return checksums
+
+class RevisionsCache(MultiProcessCache):
+ cache_file_name = "local_srcrevisions.dat"
+ CACHE_VERSION = 1
+
+ def __init__(self):
+ MultiProcessCache.__init__(self)
+
+ def get_revs(self):
+ return self.cachedata[0]
+
+ def get_rev(self, k):
+ if k in self.cachedata_extras[0]:
+ return self.cachedata_extras[0][k]
+ if k in self.cachedata[0]:
+ return self.cachedata[0][k]
+ return None
+
+ def set_rev(self, k, v):
+ self.cachedata[0][k] = v
+ self.cachedata_extras[0][k] = v
+
+ def merge_data(self, source, dest):
+ for h in source[0]:
+ dest[0][h] = source[0][h]
diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py
index 2e8b7ced..f906afab 100644
--- a/bitbake/lib/bb/codeparser.py
+++ b/bitbake/lib/bb/codeparser.py
@@ -25,7 +25,6 @@ recipe caches don't trigger "Taskhash mismatch" errors.
import ast
import sys
-import codegen
import logging
import inspect
import bb.pysh as pysh
@@ -69,12 +68,25 @@ def add_module_functions(fn, functions, namespace):
name = "%s.%s" % (namespace, f)
parser = PythonParser(name, logger)
try:
- parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f)
+ parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f, func=functions[f])
#bb.warn("Cached %s" % f)
except KeyError:
- lines, lineno = inspect.getsourcelines(functions[f])
+ try:
+ targetfn = inspect.getsourcefile(functions[f])
+ except TypeError:
+ # Builtin
+ continue
+ if fn != targetfn:
+ # Skip references to other modules outside this file
+ #bb.warn("Skipping %s" % name)
+ continue
+ try:
+ lines, lineno = inspect.getsourcelines(functions[f])
+ except TypeError:
+ # Builtin
+ continue
src = "".join(lines)
- parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
+ parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f, func=functions[f])
#bb.warn("Not cached %s" % f)
execs = parser.execs.copy()
# Expand internal module exec references
@@ -82,14 +94,17 @@ def add_module_functions(fn, functions, namespace):
if e in functions:
execs.remove(e)
execs.add(namespace + "." + e)
- modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy()]
+ visitorcode = None
+ if hasattr(functions[f], 'visitorcode'):
+ visitorcode = getattr(functions[f], "visitorcode")
+ modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy(), parser.extra, visitorcode]
#bb.warn("%s: %s\nRefs:%s Execs: %s %s %s" % (name, fn, parser.references, parser.execs, parser.var_execs, parser.contains))
def update_module_dependencies(d):
for mod in modulecode_deps:
excludes = set((d.getVarFlag(mod, "vardepsexclude") or "").split())
if excludes:
- modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3]]
+ modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3], modulecode_deps[mod][4], modulecode_deps[mod][5]]
# A custom getstate/setstate using tuples is actually worth 15% cachesize by
# avoiding duplication of the attribute names!
@@ -112,21 +127,22 @@ class SetCache(object):
codecache = SetCache()
class pythonCacheLine(object):
- def __init__(self, refs, execs, contains):
+ def __init__(self, refs, execs, contains, extra):
self.refs = codecache.internSet(refs)
self.execs = codecache.internSet(execs)
self.contains = {}
for c in contains:
self.contains[c] = codecache.internSet(contains[c])
+ self.extra = extra
def __getstate__(self):
- return (self.refs, self.execs, self.contains)
+ return (self.refs, self.execs, self.contains, self.extra)
def __setstate__(self, state):
- (refs, execs, contains) = state
- self.__init__(refs, execs, contains)
+ (refs, execs, contains, extra) = state
+ self.__init__(refs, execs, contains, extra)
def __hash__(self):
- l = (hash(self.refs), hash(self.execs))
+ l = (hash(self.refs), hash(self.execs), hash(self.extra))
for c in sorted(self.contains.keys()):
l = l + (c, hash(self.contains[c]))
return hash(l)
@@ -155,7 +171,7 @@ class CodeParserCache(MultiProcessCache):
# so that an existing cache gets invalidated. Additionally you'll need
# to increment __cache_version__ in cache.py in order to ensure that old
# recipe caches don't trigger "Taskhash mismatch" errors.
- CACHE_VERSION = 11
+ CACHE_VERSION = 14
def __init__(self):
MultiProcessCache.__init__(self)
@@ -169,8 +185,8 @@ class CodeParserCache(MultiProcessCache):
self.pythoncachelines = {}
self.shellcachelines = {}
- def newPythonCacheLine(self, refs, execs, contains):
- cacheline = pythonCacheLine(refs, execs, contains)
+ def newPythonCacheLine(self, refs, execs, contains, extra):
+ cacheline = pythonCacheLine(refs, execs, contains, extra)
h = hash(cacheline)
if h in self.pythoncachelines:
return self.pythoncachelines[h]
@@ -246,8 +262,8 @@ class PythonParser():
"""
try:
- funcstr = codegen.to_source(func)
- argstr = codegen.to_source(arg)
+ funcstr = ast.unparse(func)
+ argstr = ast.unparse(arg)
except TypeError:
self.log.debug2('Failed to convert function and argument to source form')
else:
@@ -255,7 +271,15 @@ class PythonParser():
def visit_Call(self, node):
name = self.called_node_name(node.func)
- if name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs):
+ if name and name in modulecode_deps and modulecode_deps[name][5]:
+ visitorcode = modulecode_deps[name][5]
+ contains, execs, warn = visitorcode(name, node.args)
+ for i in contains:
+ self.contains[i] = contains[i]
+ self.execs |= execs
+ if warn:
+ self.warn(node.func, warn)
+ elif name and (name.endswith(self.getvars) or name.endswith(self.getvarflags) or name in self.containsfuncs or name in self.containsanyfuncs):
if isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str):
varname = node.args[0].value
if name in self.containsfuncs and isinstance(node.args[1], ast.Constant):
@@ -323,7 +347,7 @@ class PythonParser():
# For the python module code it is expensive to have the function text so it is
# uses a different fixedhash to cache against. We can take the hit on obtaining the
# text if it isn't in the cache.
- def parse_python(self, node, lineno=0, filename="<string>", fixedhash=None):
+ def parse_python(self, node, lineno=0, filename="<string>", fixedhash=None, func=None):
if not fixedhash and (not node or not node.strip()):
return
@@ -338,6 +362,7 @@ class PythonParser():
self.contains = {}
for i in codeparsercache.pythoncache[h].contains:
self.contains[i] = set(codeparsercache.pythoncache[h].contains[i])
+ self.extra = codeparsercache.pythoncache[h].extra
return
if h in codeparsercache.pythoncacheextras:
@@ -346,6 +371,7 @@ class PythonParser():
self.contains = {}
for i in codeparsercache.pythoncacheextras[h].contains:
self.contains[i] = set(codeparsercache.pythoncacheextras[h].contains[i])
+ self.extra = codeparsercache.pythoncacheextras[h].extra
return
if fixedhash and not node:
@@ -363,9 +389,16 @@ class PythonParser():
if n.__class__.__name__ == "Call":
self.visit_Call(n)
+ if func is not None:
+ self.references |= getattr(func, "bb_vardeps", set())
+ self.references -= getattr(func, "bb_vardepsexclude", set())
+
self.execs.update(self.var_execs)
+ self.extra = None
+ if fixedhash:
+ self.extra = bbhash(str(node))
- codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains)
+ codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains, self.extra)
class ShellParser():
def __init__(self, name, log):
@@ -484,19 +517,34 @@ class ShellParser():
"""
words = list(words)
- for word in list(words):
+ for word in words:
wtree = pyshlex.make_wordtree(word[1])
for part in wtree:
if not isinstance(part, list):
continue
- if part[0] in ('`', '$('):
- command = pyshlex.wordtree_as_string(part[1:-1])
- self._parse_shell(command)
-
- if word[0] in ("cmd_name", "cmd_word"):
- if word in words:
- words.remove(word)
+ candidates = [part]
+
+ # If command is of type:
+ #
+ # var="... $(cmd [...]) ..."
+ #
+ # Then iterate on what's between the quotes and if we find a
+ # list, make that what we check for below.
+ if len(part) >= 3 and part[0] == '"':
+ for p in part[1:-1]:
+ if isinstance(p, list):
+ candidates.append(p)
+
+ for candidate in candidates:
+ if len(candidate) >= 2:
+ if candidate[0] in ('`', '$('):
+ command = pyshlex.wordtree_as_string(candidate[1:-1])
+ self._parse_shell(command)
+
+ if word[0] in ("cmd_name", "cmd_word"):
+ if word in words:
+ words.remove(word)
usetoken = False
for word in words:
diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index 1fcb9bf1..068d95c1 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -22,8 +22,10 @@ from collections import OrderedDict, defaultdict
import io
import bb.event
+import bb.fetch
import bb.cooker
import bb.remotedata
+import bb.parse
class DataStoreConnectionHandle(object):
def __init__(self, dsindex=0):
@@ -108,7 +110,7 @@ class Command:
def runAsyncCommand(self, _, process_server, halt):
try:
- if self.cooker.state in (bb.cooker.state.error, bb.cooker.state.shutdown, bb.cooker.state.forceshutdown):
+ if self.cooker.state in (bb.cooker.State.ERROR, bb.cooker.State.SHUTDOWN, bb.cooker.State.FORCE_SHUTDOWN):
# updateCache will trigger a shutdown of the parser
# and then raise BBHandledException triggering an exit
self.cooker.updateCache()
@@ -118,7 +120,7 @@ class Command:
(command, options) = cmd
commandmethod = getattr(CommandsAsync, command)
needcache = getattr( commandmethod, "needcache" )
- if needcache and self.cooker.state != bb.cooker.state.running:
+ if needcache and self.cooker.state != bb.cooker.State.RUNNING:
self.cooker.updateCache()
return True
else:
@@ -142,14 +144,14 @@ class Command:
return bb.server.process.idleFinish(traceback.format_exc())
def finishAsyncCommand(self, msg=None, code=None):
+ self.cooker.finishcommand()
+ self.process_server.clear_async_cmd()
if msg or msg == "":
bb.event.fire(CommandFailed(msg), self.cooker.data)
elif code:
bb.event.fire(CommandExit(code), self.cooker.data)
else:
bb.event.fire(CommandCompleted(), self.cooker.data)
- self.cooker.finishcommand()
- self.process_server.clear_async_cmd()
def reset(self):
if self.remotedatastores:
@@ -227,7 +229,7 @@ class CommandsSync:
Set the value of variable in configuration
"""
varname = params[0]
- value = str(params[1])
+ value = params[1]
setattr(command.cooker.configuration, varname, value)
def enableDataTracking(self, command, params):
@@ -256,7 +258,7 @@ class CommandsSync:
except IndexError:
mc = ''
return command.cooker.matchFile(fMatch, mc)
- matchFile.needconfig = False
+ matchFile.needconfig = True
def getUIHandlerNum(self, command, params):
return bb.event.get_uihandler()
@@ -310,7 +312,7 @@ class CommandsSync:
def revalidateCaches(self, command, params):
"""Called by UI clients when metadata may have changed"""
command.cooker.revalidateCaches()
- parseConfiguration.needconfig = False
+ revalidateCaches.needconfig = False
def getRecipes(self, command, params):
try:
@@ -420,15 +422,30 @@ class CommandsSync:
return command.cooker.recipecaches[mc].pkg_dp
getDefaultPreference.readonly = True
+
def getSkippedRecipes(self, command, params):
+ """
+ Get the map of skipped recipes for the specified multiconfig/mc name (`params[0]`).
+
+ Invoked by `bb.tinfoil.Tinfoil.get_skipped_recipes`
+
+ :param command: Internally used parameter.
+ :param params: Parameter array. params[0] is multiconfig/mc name. If not given, then default mc '' is assumed.
+ :return: Dict whose keys are virtualfns and values are `bb.cooker.SkippedPackage`
+ """
+ try:
+ mc = params[0]
+ except IndexError:
+ mc = ''
+
# Return list sorted by reverse priority order
import bb.cache
def sortkey(x):
vfn, _ = x
- realfn, _, mc = bb.cache.virtualfn2realfn(vfn)
- return (-command.cooker.collections[mc].calc_bbfile_priority(realfn)[0], vfn)
+ realfn, _, item_mc = bb.cache.virtualfn2realfn(vfn)
+ return -command.cooker.collections[item_mc].calc_bbfile_priority(realfn)[0], vfn
- skipdict = OrderedDict(sorted(command.cooker.skiplist.items(), key=sortkey))
+ skipdict = OrderedDict(sorted(command.cooker.skiplist_by_mc[mc].items(), key=sortkey))
return list(skipdict.items())
getSkippedRecipes.readonly = True
@@ -582,6 +599,13 @@ class CommandsSync:
return DataStoreConnectionHandle(idx)
parseRecipeFile.readonly = True
+ def finalizeData(self, command, params):
+ newdata = command.cooker.data.createCopy()
+ bb.data.expandKeys(newdata)
+ bb.parse.ast.runAnonFuncs(newdata)
+ idx = command.remotedatastores.store(newdata)
+ return DataStoreConnectionHandle(idx)
+
class CommandsAsync:
"""
A class of asynchronous commands
@@ -599,9 +623,13 @@ class CommandsAsync:
internal = params[2]
else:
internal = False
+ if len(params) > 3:
+ taskonly = params[3]
+ else:
+ taskonly = False
if internal:
- command.cooker.buildFileInternal(bfile, task, fireevents=False, quietlog=True)
+ command.cooker.buildFileInternal(bfile, task, fireevents=False, quietlog=True, taskonly=taskonly)
else:
command.cooker.buildFile(bfile, task)
buildFile.needcache = False
diff --git a/bitbake/lib/bb/compress/lz4.py b/bitbake/lib/bb/compress/lz4.py
index 88b09893..2a64681c 100644
--- a/bitbake/lib/bb/compress/lz4.py
+++ b/bitbake/lib/bb/compress/lz4.py
@@ -13,7 +13,7 @@ def open(*args, **kwargs):
class LZ4File(bb.compress._pipecompress.PipeFile):
def get_compress(self):
- return ["lz4c", "-z", "-c"]
+ return ["lz4", "-z", "-c"]
def get_decompress(self):
- return ["lz4c", "-d", "-c"]
+ return ["lz4", "-d", "-c"]
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 701cf51b..8a422e50 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -8,16 +8,17 @@
#
# SPDX-License-Identifier: GPL-2.0-only
#
-
+import enum
import sys, os, glob, os.path, re, time
import itertools
import logging
-import multiprocessing
+from bb import multiprocessing
import threading
from io import StringIO, UnsupportedOperation
from contextlib import closing
from collections import defaultdict, namedtuple
import bb, bb.command
+import bb.fetch
from bb import utils, data, parse, event, cache, providers, taskdata, runqueue, build
import queue
import signal
@@ -26,6 +27,7 @@ import json
import pickle
import codecs
import hashserv
+import ctypes
logger = logging.getLogger("BitBake")
collectlog = logging.getLogger("BitBake.Collection")
@@ -48,16 +50,15 @@ class CollectionError(bb.BBHandledException):
Exception raised when layer configuration is incorrect
"""
-class state:
- initial, parsing, running, shutdown, forceshutdown, stopped, error = list(range(7))
- @classmethod
- def get_name(cls, code):
- for name in dir(cls):
- value = getattr(cls, name)
- if type(value) == type(cls.initial) and value == code:
- return name
- raise ValueError("Invalid status code: %s" % code)
+class State(enum.Enum):
+ INITIAL = 0,
+ PARSING = 1,
+ RUNNING = 2,
+ SHUTDOWN = 3,
+ FORCE_SHUTDOWN = 4,
+ STOPPED = 5,
+ ERROR = 6
class SkippedPackage:
@@ -79,7 +80,7 @@ class SkippedPackage:
class CookerFeatures(object):
- _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO] = list(range(4))
+ _feature_list = [HOB_EXTRA_CACHES, BASEDATASTORE_TRACKING, SEND_SANITYEVENTS, RECIPE_SIGGEN_INFO, SKIP_FRAGMENTS] = list(range(5))
def __init__(self):
self._features=set()
@@ -90,6 +91,11 @@ class CookerFeatures(object):
return
self._features.add(f)
+ def delFeature(self, f):
+ if f not in self._features:
+ return
+ self._features.remove(f)
+
def __contains__(self, f):
return f in self._features
@@ -134,7 +140,8 @@ class BBCooker:
self.baseconfig_valid = False
self.parsecache_valid = False
self.eventlog = None
- self.skiplist = {}
+ # The skiplists, one per multiconfig
+ self.skiplist_by_mc = defaultdict(dict)
self.featureset = CookerFeatures()
if featureSet:
for f in featureSet:
@@ -180,7 +187,7 @@ class BBCooker:
pass
self.command = bb.command.Command(self, self.process_server)
- self.state = state.initial
+ self.state = State.INITIAL
self.parser = None
@@ -226,23 +233,26 @@ class BBCooker:
bb.warn("Cooker received SIGTERM, shutting down...")
elif signum == signal.SIGHUP:
bb.warn("Cooker received SIGHUP, shutting down...")
- self.state = state.forceshutdown
+ self.state = State.FORCE_SHUTDOWN
bb.event._should_exit.set()
def setFeatures(self, features):
# we only accept a new feature set if we're in state initial, so we can reset without problems
- if not self.state in [state.initial, state.shutdown, state.forceshutdown, state.stopped, state.error]:
+ if not self.state in [State.INITIAL, State.SHUTDOWN, State.FORCE_SHUTDOWN, State.STOPPED, State.ERROR]:
raise Exception("Illegal state for feature set change")
original_featureset = list(self.featureset)
for feature in features:
self.featureset.setFeature(feature)
- bb.debug(1, "Features set %s (was %s)" % (original_featureset, list(self.featureset)))
- if (original_featureset != list(self.featureset)) and self.state != state.error and hasattr(self, "data"):
+ for orig_feature in original_featureset:
+ if orig_feature not in features:
+ self.featureset.delFeature(orig_feature)
+ bb.debug(1, "Features set %s (was %s)" % (list(self.featureset), original_featureset))
+ if (original_featureset != list(self.featureset)) and self.state != State.ERROR and hasattr(self, "data"):
+ bb.server.process.serverlog("Reseting due to feature chanages")
self.reset()
def initConfigurationData(self):
-
- self.state = state.initial
+ self.state = State.INITIAL
self.caches_array = []
sys.path = self.orig_syspath.copy()
@@ -250,7 +260,11 @@ class BBCooker:
if mod not in self.orig_sysmodules:
del sys.modules[mod]
+ self._baseconfig_set(False)
+ self._parsecache_set(False)
+
self.configwatched = {}
+ self.revalidateCaches()
# Need to preserve BB_CONSOLELOG over resets
consolelog = None
@@ -278,10 +292,14 @@ class BBCooker:
logger.critical("Unable to import extra RecipeInfo '%s' from '%s': %s" % (cache_name, module_name, exc))
raise bb.BBHandledException()
+ if CookerFeatures.SKIP_FRAGMENTS in self.featureset:
+ self.enableSkipFragments()
+ else:
+ self.disableSkipFragments()
+
self.databuilder = bb.cookerdata.CookerDataBuilder(self.configuration, False)
self.databuilder.parseBaseConfiguration()
self.data = self.databuilder.data
- self.data_hash = self.databuilder.data_hash
self.extraconfigdata = {}
eventlog = self.data.getVar("BB_DEFAULT_EVENTLOG")
@@ -300,7 +318,6 @@ class BBCooker:
self.add_filewatch(mc.getVar("__base_depends", False), configwatcher=True)
self._baseconfig_set(True)
- self._parsecache_set(False)
def handlePRServ(self):
# Setup a PR Server based on the new configuration
@@ -312,16 +329,34 @@ class BBCooker:
if self.data.getVar("BB_HASHSERVE") == "auto":
# Create a new hash server bound to a unix domain socket
if not self.hashserv:
- dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
+ bb_hashserve_db_dir = self.data.getVar("BB_HASHSERVE_DB_DIR")
+ if bb_hashserve_db_dir and utils.is_path_on_nfs(bb_hashserve_db_dir):
+ bb.fatal("""Hash equivalency database location (set via BB_HASHSERVE_DB_DIR to {})
+cannot be on a NFS mount due to potential NFS locking issues between sqlite clients, per https://sqlite.org/faq.html#q5
+
+If you need to share the database between several computers, set up a permanently running hash equivalency server
+according to https://docs.yoctoproject.org/dev-manual/hashequivserver.html""".format(bb_hashserve_db_dir))
+ dbdir = bb_hashserve_db_dir or self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")
+ os.makedirs(dbdir, exist_ok=True)
+ dbfile = dbdir + "/hashserv.db"
upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") or None
if upstream:
- import socket
try:
- sock = socket.create_connection(upstream.split(":"), 5)
- sock.close()
- except socket.error as e:
- bb.warn("BB_HASHSERVE_UPSTREAM is not valid, unable to connect hash equivalence server at '%s': %s"
+ with hashserv.create_client(upstream) as client:
+ client.ping()
+ except ImportError as e:
+ bb.fatal(""""Unable to use hash equivalence server at '%s' due to missing or incorrect python module:
+%s
+Please install the needed module on the build host, or use an environment containing it:
+ - if you are using bitbake-setup, run 'bitbake-setup install-buildtools'
+ - openembedded-core layer contains 'scripts/install-buildtools' that can also be used
+ - or set up pip venv
+You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in significantly longer build times as bitbake will be unable to reuse prebuilt sstate artefacts."""
% (upstream, repr(e)))
+ except ConnectionError as e:
+ bb.warn("Unable to connect to hash equivalence server at '%s', please correct or remove BB_HASHSERVE_UPSTREAM:\n%s"
+ % (upstream, repr(e)))
+ upstream = None
self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
self.hashserv = hashserv.create_server(
@@ -347,17 +382,34 @@ class BBCooker:
if hasattr(self, "data"):
self.data.disableTracking()
+ def enableSkipFragments(self):
+ self.configuration.skip_fragments = True
+
+ def disableSkipFragments(self):
+ self.configuration.skip_fragments = False
+
def revalidateCaches(self):
bb.parse.clear_cache()
+ if not self.baseconfig_valid:
+ bb.parse.BBHandler.cached_statements = {}
+ if hasattr(self, "databuilder"):
+ self.databuilder.calc_datastore_hashes(clean=False)
+ return
+
clean = True
for f in self.configwatched:
if not bb.parse.check_mtime(f, self.configwatched[f]):
bb.server.process.serverlog("Found %s changed, invalid cache" % f)
+ if self.state not in (State.SHUTDOWN, State.FORCE_SHUTDOWN, State.ERROR):
+ self.state = State.INITIAL
self._baseconfig_set(False)
self._parsecache_set(False)
clean = False
- break
+ bb.parse.BBHandler.cached_statements = {}
+ if hasattr(self, "databuilder"):
+ self.databuilder.calc_datastore_hashes(clean=False)
+ return
if clean:
for f in self.parsewatched:
@@ -365,10 +417,13 @@ class BBCooker:
bb.server.process.serverlog("Found %s changed, invalid cache" % f)
self._parsecache_set(False)
clean = False
+ bb.parse.BBHandler.cached_statements = {}
break
- if not clean:
- bb.parse.BBHandler.cached_statements = {}
+ # If writes were made to any of the data stores, we need to recalculate the data
+ # store cache
+ if hasattr(self, "databuilder"):
+ self.databuilder.calc_datastore_hashes(clean=True)
def parseConfiguration(self):
self.updateCacheSync()
@@ -391,6 +446,7 @@ class BBCooker:
self.collections = {}
for mc in self.multiconfigs:
self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)
+ self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
self._parsecache_set(False)
@@ -612,8 +668,8 @@ class BBCooker:
localdata = {}
for mc in self.multiconfigs:
- taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist, allowincomplete=allowincomplete)
- localdata[mc] = data.createCopy(self.databuilder.mcdata[mc])
+ taskdata[mc] = bb.taskdata.TaskData(halt, skiplist=self.skiplist_by_mc[mc], allowincomplete=allowincomplete)
+ localdata[mc] = bb.data.createCopy(self.databuilder.mcdata[mc])
bb.data.expandKeys(localdata[mc])
current = 0
@@ -680,14 +736,14 @@ class BBCooker:
bb.event.fire(bb.event.TreeDataPreparationCompleted(len(fulltargetlist)), self.data)
return taskdata, runlist
- def prepareTreeData(self, pkgs_to_build, task):
+ def prepareTreeData(self, pkgs_to_build, task, halt=False):
"""
Prepare a runqueue and taskdata object for iteration over pkgs_to_build
"""
# We set halt to False here to prevent unbuildable targets raising
# an exception when we're just generating data
- taskdata, runlist = self.buildTaskData(pkgs_to_build, task, False, allowincomplete=True)
+ taskdata, runlist = self.buildTaskData(pkgs_to_build, task, halt, allowincomplete=True)
return runlist, taskdata
@@ -701,7 +757,7 @@ class BBCooker:
if not task.startswith("do_"):
task = "do_%s" % task
- runlist, taskdata = self.prepareTreeData(pkgs_to_build, task)
+ runlist, taskdata = self.prepareTreeData(pkgs_to_build, task, halt=True)
rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
rq.rqdata.prepare()
return self.buildDependTree(rq, taskdata)
@@ -896,10 +952,11 @@ class BBCooker:
depgraph = self.generateTaskDepTreeData(pkgs_to_build, task)
- with open('pn-buildlist', 'w') as f:
- for pn in depgraph["pn"]:
- f.write(pn + "\n")
- logger.info("PN build list saved to 'pn-buildlist'")
+ pns = depgraph["pn"].keys()
+ if pns:
+ with open('pn-buildlist', 'w') as f:
+ f.write("%s\n" % "\n".join(sorted(pns)))
+ logger.info("PN build list saved to 'pn-buildlist'")
# Remove old format output files to ensure no confusion with stale data
try:
@@ -933,7 +990,7 @@ class BBCooker:
for mc in self.multiconfigs:
# First get list of recipes, including skipped
recipefns = list(self.recipecaches[mc].pkg_fn.keys())
- recipefns.extend(self.skiplist.keys())
+ recipefns.extend(self.skiplist_by_mc[mc].keys())
# Work out list of bbappends that have been applied
applied_appends = []
@@ -952,13 +1009,7 @@ class BBCooker:
'\n '.join(appends_without_recipes[mc])))
if msgs:
- msg = "\n".join(msgs)
- warn_only = self.databuilder.mcdata[mc].getVar("BB_DANGLINGAPPENDS_WARNONLY", \
- False) or "no"
- if warn_only.lower() in ("1", "yes", "true"):
- bb.warn(msg)
- else:
- bb.fatal(msg)
+ bb.fatal("\n".join(msgs))
def handlePrefProviders(self):
@@ -1273,8 +1324,8 @@ class BBCooker:
if bf.startswith("/") or bf.startswith("../"):
bf = os.path.abspath(bf)
- collections = {mc: CookerCollectFiles(self.bbfile_config_priorities, mc)}
- filelist, masked, searchdirs = collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
+ filelist = self.collections[mc].bbfiles
+
try:
os.stat(bf)
bf = os.path.abspath(bf)
@@ -1317,7 +1368,7 @@ class BBCooker:
self.buildFileInternal(buildfile, task)
- def buildFileInternal(self, buildfile, task, fireevents=True, quietlog=False):
+ def buildFileInternal(self, buildfile, task, fireevents=True, quietlog=False, taskonly=False):
"""
Build the file matching regexp buildfile
"""
@@ -1338,7 +1389,7 @@ class BBCooker:
self.buildSetVars()
self.reset_mtime_caches()
- bb_caches = bb.cache.MulticonfigCache(self.databuilder, self.data_hash, self.caches_array)
+ bb_caches = bb.cache.MulticonfigCache(self.databuilder, self.databuilder.data_hash, self.caches_array)
layername = self.collections[mc].calc_bbfile_priority(fn)[2]
infos = bb_caches[mc].parse(fn, self.collections[mc].get_file_appends(fn), layername)
@@ -1367,6 +1418,12 @@ class BBCooker:
self.recipecaches[mc].rundeps[fn] = defaultdict(list)
self.recipecaches[mc].runrecs[fn] = defaultdict(list)
+ if taskonly:
+ # Drop the intra-recipe task ordering too ('addtask X after Y'), so
+ # that task is the only entry left in the runqueue.
+ task_deps = self.recipecaches[mc].task_deps[fn]
+ task_deps['parents'] = {t: [] for t in task_deps['tasks']}
+
bb.parse.siggen.setup_datacache(self.recipecaches)
# Invalidate task for target if force mode active
@@ -1399,11 +1456,11 @@ class BBCooker:
msg = None
interrupted = 0
- if halt or self.state == state.forceshutdown:
+ if halt or self.state == State.FORCE_SHUTDOWN:
rq.finish_runqueue(True)
msg = "Forced shutdown"
interrupted = 2
- elif self.state == state.shutdown:
+ elif self.state == State.SHUTDOWN:
rq.finish_runqueue(False)
msg = "Stopped build"
interrupted = 1
@@ -1429,8 +1486,7 @@ class BBCooker:
if quietlog:
bb.runqueue.logger.setLevel(rqloglevel)
return bb.server.process.idleFinish(msg)
- if retval is True:
- return True
+
return retval
self.idleCallBackRegister(buildFileIdle, rq)
@@ -1459,7 +1515,6 @@ class BBCooker:
if t in task or getAllTaskSignatures:
try:
- rq.rqdata.prepare_task_hash(tid)
sig.append([pn, t, rq.rqdata.get_task_unihash(tid)])
except KeyError:
sig.append(self.getTaskSignatures(target, [t])[0])
@@ -1474,12 +1529,12 @@ class BBCooker:
def buildTargetsIdle(server, rq, halt):
msg = None
interrupted = 0
- if halt or self.state == state.forceshutdown:
+ if halt or self.state == State.FORCE_SHUTDOWN:
bb.event._should_exit.set()
rq.finish_runqueue(True)
msg = "Forced shutdown"
interrupted = 2
- elif self.state == state.shutdown:
+ elif self.state == State.SHUTDOWN:
rq.finish_runqueue(False)
msg = "Stopped build"
interrupted = 1
@@ -1500,8 +1555,6 @@ class BBCooker:
bb.event.disable_heartbeat()
return bb.server.process.idleFinish(msg)
- if retval is True:
- return True
return retval
self.reset_mtime_caches()
@@ -1548,8 +1601,8 @@ class BBCooker:
dump = {}
# Horrible but for now we need to avoid any sideeffects of autorev being called
- saved = bb.fetch2.get_autorev
- bb.fetch2.get_autorev = dummy_autorev
+ saved = bb.fetch.get_autorev
+ bb.fetch.get_autorev = dummy_autorev
for k in self.data.keys():
try:
expand = True
@@ -1569,12 +1622,12 @@ class BBCooker:
dump[k][d] = None
except Exception as e:
print(e)
- bb.fetch2.get_autorev = saved
+ bb.fetch.get_autorev = saved
return dump
def updateCacheSync(self):
- if self.state == state.running:
+ if self.state == State.RUNNING:
return
if not self.baseconfig_valid:
@@ -1584,19 +1637,19 @@ class BBCooker:
# This is called for all async commands when self.state != running
def updateCache(self):
- if self.state == state.running:
+ if self.state == State.RUNNING:
return
- if self.state in (state.shutdown, state.forceshutdown, state.error):
+ if self.state in (State.SHUTDOWN, State.FORCE_SHUTDOWN, State.ERROR):
if hasattr(self.parser, 'shutdown'):
self.parser.shutdown(clean=False)
self.parser.final_cleanup()
raise bb.BBHandledException()
- if self.state != state.parsing:
+ if self.state != State.PARSING:
self.updateCacheSync()
- if self.state != state.parsing and not self.parsecache_valid:
+ if self.state != State.PARSING and not self.parsecache_valid:
bb.server.process.serverlog("Parsing started")
self.parsewatched = {}
@@ -1617,11 +1670,9 @@ class BBCooker:
total_masked = 0
searchdirs = set()
for mc in self.multiconfigs:
- (filelist, masked, search) = self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
-
- mcfilelist[mc] = filelist
- total_masked += masked
- searchdirs |= set(search)
+ mcfilelist[mc] = self.collections[mc].bbfiles
+ total_masked += self.collections[mc].maskedcount
+ searchdirs |= set(self.collections[mc].searchdirs)
# Add mtimes for directories searched for bb/bbappend files
for dirent in searchdirs:
@@ -1630,9 +1681,10 @@ class BBCooker:
self.parser = CookerParser(self, mcfilelist, total_masked)
self._parsecache_set(True)
- self.state = state.parsing
+ self.state = State.PARSING
if not self.parser.parse_next():
+ bb.server.process.serverlog("Parsing completed")
collectlog.debug("parsing complete")
if self.parser.error:
raise bb.BBHandledException()
@@ -1640,7 +1692,7 @@ class BBCooker:
self.handlePrefProviders()
for mc in self.multiconfigs:
self.recipecaches[mc].bbfile_priority = self.collections[mc].collection_priorities(self.recipecaches[mc].pkg_fn, self.parser.mcfilelist[mc], self.data)
- self.state = state.running
+ self.state = State.RUNNING
# Send an event listing all stamps reachable after parsing
# which the metadata may use to clean up stale data
@@ -1713,10 +1765,10 @@ class BBCooker:
def shutdown(self, force=False):
if force:
- self.state = state.forceshutdown
+ self.state = State.FORCE_SHUTDOWN
bb.event._should_exit.set()
else:
- self.state = state.shutdown
+ self.state = State.SHUTDOWN
if self.parser:
self.parser.shutdown(clean=False)
@@ -1726,7 +1778,7 @@ class BBCooker:
if hasattr(self.parser, 'shutdown'):
self.parser.shutdown(clean=False)
self.parser.final_cleanup()
- self.state = state.initial
+ self.state = State.INITIAL
bb.event._should_exit.clear()
def reset(self):
@@ -1761,6 +1813,7 @@ class CookerCollectFiles(object):
def __init__(self, priorities, mc=''):
self.mc = mc
self.bbappends = []
+ self.overlayed = None
# Priorities is a list of tuples, with the second element as the pattern.
# We need to sort the list with the longest pattern first, and so on to
# the shortest. This allows nested layers to be properly evaluated.
@@ -1795,7 +1848,10 @@ class CookerCollectFiles(object):
def collect_bbfiles(self, config, eventdata):
"""Collect all available .bb build files"""
+ bbfiles = []
+ bbappend = []
masked = 0
+ searchdirs = []
collectlog.debug("collecting .bb files")
@@ -1813,12 +1869,11 @@ class CookerCollectFiles(object):
bb.event.fire(CookerExit(), eventdata)
# We need to track where we look so that we can know when the cache is invalid. There
- # is no nice way to do this, this is horrid. We intercept the os.listdir()
- # (or os.scandir() for python 3.6+) calls while we run glob().
+ # is no nice way to do this, this is horrid. We intercept the os.listdir() and os.scandir()
+ # calls while we run glob().
origlistdir = os.listdir
if hasattr(os, 'scandir'):
origscandir = os.scandir
- searchdirs = []
def ourlistdir(d):
searchdirs.append(d)
@@ -1880,8 +1935,6 @@ class CookerCollectFiles(object):
collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask)
bbmask = None
- bbfiles = []
- bbappend = []
for f in newfiles:
if bbmask and bbmask_compiled.search(f):
collectlog.debug("skipping masked file %s", f)
@@ -1911,7 +1964,9 @@ class CookerCollectFiles(object):
topfile = bbfile_seen[base]
self.overlayed[topfile].append(f)
- return (bbfiles, masked, searchdirs)
+ self.searchdirs = searchdirs
+ self.maskedcount = masked
+ self.bbfiles = bbfiles
def get_file_appends(self, fn):
"""
@@ -1994,8 +2049,9 @@ class ParsingFailure(Exception):
Exception.__init__(self, realexception, recipe)
class Parser(multiprocessing.Process):
- def __init__(self, jobs, results, quit, profile):
+ def __init__(self, jobs, next_job_id, results, quit, profile):
self.jobs = jobs
+ self.next_job_id = next_job_id
self.results = results
self.quit = quit
multiprocessing.Process.__init__(self)
@@ -2005,6 +2061,7 @@ class Parser(multiprocessing.Process):
self.queue_signals = False
self.signal_received = []
self.signal_threadlock = threading.Lock()
+ self.exit = False
def catch_sig(self, signum, frame):
if self.queue_signals:
@@ -2017,24 +2074,10 @@ class Parser(multiprocessing.Process):
signal.signal(signal.SIGTERM, signal.SIG_DFL)
os.kill(os.getpid(), signal.SIGTERM)
elif signum == signal.SIGINT:
- signal.default_int_handler(signum, frame)
+ self.exit = True
def run(self):
-
- if not self.profile:
- self.realrun()
- return
-
- try:
- import cProfile as profile
- except:
- import profile
- prof = profile.Profile()
- try:
- profile.Profile.runcall(prof, self.realrun)
- finally:
- logfile = "profile-parse-%s.log" % multiprocessing.current_process().name
- prof.dump_stats(logfile)
+ bb.utils.profile_function("parsing" in self.profile, self.realrun, "profile-parse-%s.log" % multiprocessing.current_process().name, process=False)
def realrun(self):
# Signal handling here is hard. We must not terminate any process or thread holding the write
@@ -2055,15 +2098,19 @@ class Parser(multiprocessing.Process):
pending = []
havejobs = True
try:
- while havejobs or pending:
+ while (havejobs or pending) and not self.exit:
if self.quit.is_set():
break
job = None
- try:
- job = self.jobs.pop()
- except IndexError:
- havejobs = False
+ if havejobs:
+ with self.next_job_id.get_lock():
+ if self.next_job_id.value < len(self.jobs):
+ job = self.jobs[self.next_job_id.value]
+ self.next_job_id.value += 1
+ else:
+ havejobs = False
+
if job:
result = self.parse(*job)
# Clear the siggen cache after parsing to control memory usage, its huge
@@ -2112,7 +2159,7 @@ class CookerParser(object):
self.mcfilelist = mcfilelist
self.cooker = cooker
self.cfgdata = cooker.data
- self.cfghash = cooker.data_hash
+ self.cfghash = cooker.databuilder.data_hash
self.cfgbuilder = cooker.databuilder
# Accounting statistics
@@ -2129,13 +2176,13 @@ class CookerParser(object):
self.bb_caches = bb.cache.MulticonfigCache(self.cfgbuilder, self.cfghash, cooker.caches_array)
self.fromcache = set()
- self.willparse = set()
+ self.willparse = []
for mc in self.cooker.multiconfigs:
for filename in self.mcfilelist[mc]:
appends = self.cooker.collections[mc].get_file_appends(filename)
layername = self.cooker.collections[mc].calc_bbfile_priority(filename)[2]
if not self.bb_caches[mc].cacheValid(filename, appends):
- self.willparse.add((mc, self.bb_caches[mc], filename, appends, layername))
+ self.willparse.append((mc, self.bb_caches[mc], filename, appends, layername))
else:
self.fromcache.add((mc, self.bb_caches[mc], filename, appends, layername))
@@ -2154,18 +2201,18 @@ class CookerParser(object):
def start(self):
self.results = self.load_cached()
self.processes = []
+
if self.toparse:
bb.event.fire(bb.event.ParseStarted(self.toparse), self.cfgdata)
+ next_job_id = multiprocessing.Value(ctypes.c_int, 0)
self.parser_quit = multiprocessing.Event()
self.result_queue = multiprocessing.Queue()
- def chunkify(lst,n):
- return [lst[i::n] for i in range(n)]
- self.jobs = chunkify(list(self.willparse), self.num_processes)
-
+ # Have to pass in willparse at fork time so all parsing processes have the unpickleable data
+ # then access it by index from the parse queue.
for i in range(0, self.num_processes):
- parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, self.cooker.configuration.profile)
+ parser = Parser(self.willparse, next_job_id, self.result_queue, self.parser_quit, self.cooker.configuration.profile)
parser.start()
self.process_names.append(parser.name)
self.processes.append(parser)
@@ -2192,11 +2239,17 @@ class CookerParser(object):
# Cleanup the queue before call process.join(), otherwise there might be
# deadlocks.
- while True:
- try:
- self.result_queue.get(timeout=0.25)
- except queue.Empty:
- break
+ def read_results():
+ while True:
+ try:
+ self.result_queue.get(timeout=0.25)
+ except queue.Empty:
+ break
+ except KeyError:
+ # The restore state from SiggenRecipeInfo in cache.py can
+ # fail here if this is an unclean shutdown since the state may have been
+ # reset. Ignore key errors for that reason, we don't care.
+ pass
def sync_caches():
for c in self.bb_caches.values():
@@ -2208,15 +2261,19 @@ class CookerParser(object):
self.parser_quit.set()
+ read_results()
+
for process in self.processes:
- process.join(0.5)
+ process.join(2)
for process in self.processes:
if process.exitcode is None:
os.kill(process.pid, signal.SIGINT)
+ read_results()
+
for process in self.processes:
- process.join(0.5)
+ process.join(2)
for process in self.processes:
if process.exitcode is None:
@@ -2224,9 +2281,8 @@ class CookerParser(object):
for process in self.processes:
process.join()
- # Added in 3.7, cleans up zombies
- if hasattr(process, "close"):
- process.close()
+ # clean up zombies
+ process.close()
bb.codeparser.parser_cache_save()
bb.codeparser.parser_cache_savemerge()
@@ -2236,12 +2292,13 @@ class CookerParser(object):
profiles = []
for i in self.process_names:
logfile = "profile-parse-%s.log" % i
- if os.path.exists(logfile):
+ if os.path.exists(logfile) and os.path.getsize(logfile):
profiles.append(logfile)
- pout = "profile-parse.log.processed"
- bb.utils.process_profilelog(profiles, pout = pout)
- print("Processed parsing statistics saved to %s" % (pout))
+ if profiles:
+ fn_out = "profile-parse.log.report"
+ bb.utils.process_profilelog(profiles, fn_out=fn_out)
+ print("Processed parsing statistics saved to %s" % (fn_out))
def final_cleanup(self):
if self.syncthread:
@@ -2273,7 +2330,7 @@ class CookerParser(object):
yield result
if not (self.parsed >= self.toparse):
- raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None)
+ raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? (%s %s of %s) Exiting." % (len(self.processes), self.parsed, self.toparse), None)
def parse_next(self):
@@ -2359,7 +2416,7 @@ class CookerParser(object):
for virtualfn, info_array in result:
if info_array[0].skipped:
self.skipped += 1
- self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
+ self.cooker.skiplist_by_mc[mc][virtualfn] = SkippedPackage(info_array[0])
self.bb_caches[mc].add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
parsed=parsed, watcher = self.cooker.add_filewatch)
return True
diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 0649e409..d65e3c97 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -1,3 +1,4 @@
+
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -15,9 +16,15 @@ import re
import sys
import hashlib
from functools import wraps
-import bb
-from bb import data
+
+import bb.cache
+import bb.codeparser
+import bb.data
+import bb.data_smart
+import bb.event
+import bb.fetch
import bb.parse
+import bb.utils
logger = logging.getLogger("BitBake")
parselog = logging.getLogger("BitBake.Parsing")
@@ -136,6 +143,7 @@ class CookerConfiguration(object):
self.build_verbose_stdout = False
self.dry_run = False
self.tracking = False
+ self.skip_fragments = False
self.writeeventlog = False
self.limited_deps = False
self.runall = []
@@ -225,12 +233,15 @@ class CookerDataBuilder(object):
self.prefiles = cookercfg.prefile
self.postfiles = cookercfg.postfile
self.tracking = cookercfg.tracking
+ self.skip_fragments = cookercfg.skip_fragments
bb.utils.set_context(bb.utils.clean_context())
bb.event.set_class_handlers(bb.event.clean_class_handlers())
self.basedata = bb.data.init()
if self.tracking:
self.basedata.enableTracking()
+ if self.skip_fragments:
+ self.basedata.setVar("_BB_SKIP_FRAGMENTS", "1")
# Keep a datastore of the initial environment variables and their
# values from when BitBake was launched to enable child processes
@@ -254,14 +265,25 @@ class CookerDataBuilder(object):
self.data = self.basedata
self.mcdata = {}
+ def calc_datastore_hashes(self, clean=True):
+ if not clean:
+ self.data_hash = None
+ return
+
+ data_hash = hashlib.sha256()
+ data_hash.update(self.data.get_hash().encode('utf-8'))
+ multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
+ for config in multiconfig:
+ data_hash.update(self.mcdata[config].get_hash().encode('utf-8'))
+ self.data_hash = data_hash.hexdigest()
+
def parseBaseConfiguration(self, worker=False):
mcdata = {}
- data_hash = hashlib.sha256()
try:
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
- if self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker:
- bb.fetch.fetcher_init(self.data)
+ servercontext = self.data.getVar("BB_WORKERCONTEXT", False) is None and not worker
+ bb.fetch.fetcher_init(self.data, servercontext)
bb.parse.init_parser(self.data)
bb.event.fire(bb.event.ConfigParsed(), self.data)
@@ -279,7 +301,6 @@ class CookerDataBuilder(object):
bb.event.fire(bb.event.ConfigParsed(), self.data)
bb.parse.init_parser(self.data)
- data_hash.update(self.data.get_hash().encode('utf-8'))
mcdata[''] = self.data
multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
@@ -289,11 +310,9 @@ class CookerDataBuilder(object):
parsed_mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
bb.event.fire(bb.event.ConfigParsed(), parsed_mcdata)
mcdata[config] = parsed_mcdata
- data_hash.update(parsed_mcdata.get_hash().encode('utf-8'))
if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(mcdata), self.data)
- self.data_hash = data_hash.hexdigest()
except bb.data_smart.ExpansionError as e:
logger.error(str(e))
raise bb.BBHandledException()
@@ -328,6 +347,7 @@ class CookerDataBuilder(object):
for mc in mcdata:
self.mcdata[mc] = bb.data.createCopy(mcdata[mc])
self.data = self.mcdata['']
+ self.calc_datastore_hashes()
def reset(self):
# We may not have run parseBaseConfiguration() yet
@@ -340,7 +360,7 @@ class CookerDataBuilder(object):
def _findLayerConf(self, data):
return findConfigFile("bblayers.conf", data)
- def parseConfigurationFiles(self, prefiles, postfiles, mc = "default"):
+ def parseConfigurationFiles(self, prefiles, postfiles, mc = ""):
data = bb.data.createCopy(self.basedata)
data.setVar("BB_CURRENT_MC", mc)
@@ -498,6 +518,8 @@ class CookerDataBuilder(object):
bb_data.setVar("__BBMULTICONFIG", mc)
bb_data.setVar("FILE_LAYERNAME", layername)
+ bb_data.setVar("__BB_RECIPE_FILE", bbfile)
+
bbfile_loc = os.path.abspath(os.path.dirname(bbfile))
bb.parse.cached_mtime_noerror(bbfile_loc)
diff --git a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py
index 76894044..70d9b8ee 100644
--- a/bitbake/lib/bb/daemonize.py
+++ b/bitbake/lib/bb/daemonize.py
@@ -15,9 +15,12 @@ import os
import sys
import io
import traceback
+import warnings
import bb
+warnings.filterwarnings("ignore", category=DeprecationWarning, message=".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*")
+
def createDaemon(function, logfile):
"""
Detach a process from the controlling terminal and run it in the
diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index 505f4295..f81f44ea 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -94,7 +94,7 @@ def expandKeys(alterdata, readdata = None):
val = alterdata.getVar(key, False)
if val is not None:
bb.warn("Variable key %s (%s) replaces original key %s (%s)." % (key, val, ekey, newval))
- alterdata.renameVar(key, ekey)
+ alterdata.renameVar(key, ekey, recurse=False)
def inheritFromOS(d, savedenv, permitted):
"""Inherit variables from the initial environment."""
@@ -171,6 +171,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
# if we're going to output this within doublequotes,
# to a shell, we need to escape the quotes in the var
alter = re.sub('"', '\\"', val)
+ alter = re.sub('`', '\\`', alter)
alter = re.sub('\n', ' \\\n', alter)
alter = re.sub('\\$', '\\\\$', alter)
o.write('%s="%s"\n' % (varExpanded, alter))
@@ -192,6 +193,12 @@ def exported_keys(d):
not bb.utils.to_boolean(d.getVarFlag(key, 'unexport')))
def exported_vars(d):
+ """Return the exported variables as a list of (key, value) pairs.
+
+ Every value is expanded before returning, so a caller that changes the
+ environment afterwards still gets what the expansion saw.
+ """
+ exported = []
k = list(exported_keys(d))
for key in k:
try:
@@ -201,7 +208,9 @@ def exported_vars(d):
continue
if value is not None:
- yield key, str(value)
+ exported.append((key, str(value)))
+
+ return exported
def emit_func(func, o=sys.__stdout__, d = init()):
"""Emits all items in the data store in a format such that it can be sourced by a shell."""
@@ -293,7 +302,7 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va
if key in mod_funcs:
exclusions = set()
moddep = bb.codeparser.modulecode_deps[key]
- value = handle_contains("", moddep[3], exclusions, d)
+ value = handle_contains(moddep[4], moddep[3], exclusions, d)
return frozenset((moddep[0] | keys & moddep[1]) - ignored_vars), value
if key[-1] == ']':
@@ -366,7 +375,7 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va
except bb.parse.SkipRecipe:
raise
except Exception as e:
- bb.warn("Exception during build_dependencies for %s" % key)
+ bb.warn("Exception during build_dependencies for %s: %s" % (key, repr(e)))
raise
return frozenset(deps), value
#bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
@@ -377,7 +386,7 @@ def generate_dependencies(d, ignored_vars):
mod_funcs = set(bb.codeparser.modulecode_deps.keys())
keys = set(key for key in d if not key.startswith("__")) | mod_funcs
shelldeps = set(key for key in d.getVar("__exportlist", False) if bb.utils.to_boolean(d.getVarFlag(key, "export")) and not bb.utils.to_boolean(d.getVarFlag(key, "unexport")))
- varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS')
+ varflagsexcl = (d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS') or "").split()
codeparserd = d.createCopy()
for forced in (d.getVar('BB_HASH_CODEPARSER_VALS') or "").split():
diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index 0128a5bb..8aa7c1f9 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -24,6 +24,7 @@ from collections.abc import MutableMapping
import logging
import hashlib
import bb, bb.codeparser
+import bb.filter
from bb import utils
from bb.COW import COWDictBase
@@ -31,7 +32,7 @@ logger = logging.getLogger("BitBake.Data")
__setvar_keyword__ = [":append", ":prepend", ":remove"]
__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$')
-__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}")
+__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+}")
__expand_python_regexp__ = re.compile(r"\${@(?:{.*?}|.)+?}")
__whitespace_split__ = re.compile(r'(\s)')
__override_regexp__ = re.compile(r'[a-z0-9]+')
@@ -106,52 +107,52 @@ class VariableParse:
self.contains = {}
def var_sub(self, match):
- key = match.group()[2:-1]
- if self.varname and key:
- if self.varname == key:
- raise Exception("variable %s references itself!" % self.varname)
- var = self.d.getVarFlag(key, "_content")
- self.references.add(key)
- if var is not None:
- return var
- else:
- return match.group()
+ key = match.group()[2:-1]
+ if self.varname and key:
+ if self.varname == key:
+ raise Exception("variable %s references itself!" % self.varname)
+ var = self.d.getVarFlag(key, "_content")
+ self.references.add(key)
+ if var is not None:
+ return var
+ else:
+ return match.group()
def python_sub(self, match):
- if isinstance(match, str):
- code = match
- else:
- code = match.group()[3:-1]
+ if isinstance(match, str):
+ code = match
+ else:
+ code = match.group()[3:-1]
- # Do not run code that contains one or more unexpanded variables
- # instead return the code with the characters we removed put back
- if __expand_var_regexp__.findall(code):
- return "${@" + code + "}"
+ # Do not run code that contains one or more unexpanded variables
+ # instead return the code with the characters we removed put back
+ if __expand_var_regexp__.search(code):
+ return "${@" + code + "}"
- if self.varname:
- varname = 'Var <%s>' % self.varname
- else:
- varname = '<expansion>'
- codeobj = compile(code.strip(), varname, "eval")
-
- parser = bb.codeparser.PythonParser(self.varname, logger)
- parser.parse_python(code)
- if self.varname:
- vardeps = self.d.getVarFlag(self.varname, "vardeps")
- if vardeps is None:
- parser.log.flush()
- else:
+ if self.varname:
+ varname = 'Var <%s>' % self.varname
+ else:
+ varname = '<expansion>'
+ codeobj = compile(code.strip(), varname, "eval")
+
+ parser = bb.codeparser.PythonParser(self.varname, logger)
+ parser.parse_python(code)
+ if self.varname:
+ vardeps = self.d.getVarFlag(self.varname, "vardeps")
+ if vardeps is None:
parser.log.flush()
- self.references |= parser.references
- self.execs |= parser.execs
+ else:
+ parser.log.flush()
+ self.references |= parser.references
+ self.execs |= parser.execs
- for k in parser.contains:
- if k not in self.contains:
- self.contains[k] = parser.contains[k].copy()
- else:
- self.contains[k].update(parser.contains[k])
- value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
- return str(value)
+ for k in parser.contains:
+ if k not in self.contains:
+ self.contains[k] = parser.contains[k].copy()
+ else:
+ self.contains[k].update(parser.contains[k])
+ value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
+ return str(value)
class DataContext(dict):
excluded = set([i for i in dir(builtins) if not i.startswith('_')] + ['oe'])
@@ -272,12 +273,9 @@ class VariableHistory(object):
return
if 'op' not in loginfo or not loginfo['op']:
loginfo['op'] = 'set'
- if 'detail' in loginfo:
- loginfo['detail'] = str(loginfo['detail'])
if 'variable' not in loginfo or 'file' not in loginfo:
raise ValueError("record() missing variable or file.")
var = loginfo['variable']
-
if var not in self.variables:
self.variables[var] = []
if not isinstance(self.variables[var], list):
@@ -336,7 +334,8 @@ class VariableHistory(object):
flag = '[%s] ' % (event['flag'])
else:
flag = ''
- o.write("# %s %s:%s%s\n# %s\"%s\"\n" % (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n# ', event['detail'])))
+ o.write("# %s %s:%s%s\n# %s\"%s\"\n" % \
+ (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n# ', str(event['detail']))))
if len(history) > 1:
o.write("# pre-expansion value:\n")
o.write('# "%s"\n' % (commentVal))
@@ -390,7 +389,7 @@ class VariableHistory(object):
if isset and event['op'] == 'set?':
continue
isset = True
- items = d.expand(event['detail']).split()
+ items = d.expand(str(event['detail'])).split()
for item in items:
# This is a little crude but is belt-and-braces to avoid us
# having to handle every possible operation type specifically
@@ -429,6 +428,7 @@ class DataSmart(MutableMapping):
self.inchistory = IncludeHistory()
self.varhistory = VariableHistory(self)
+ self.filters = {}
self._tracking = False
self._var_renames = {}
self._var_renames.update(bitbake_renamed_vars)
@@ -582,12 +582,10 @@ class DataSmart(MutableMapping):
else:
loginfo['op'] = keyword
self.varhistory.record(**loginfo)
- # todo make sure keyword is not __doc__ or __module__
# pay the cookie monster
# more cookies for the cookie monster
- if ':' in var:
- self._setvar_update_overrides(base, **loginfo)
+ self._setvar_update_overrides(base, **loginfo)
if base in self.overridevars:
self._setvar_update_overridevars(var, value)
@@ -640,6 +638,7 @@ class DataSmart(MutableMapping):
nextnew.update(vardata.contains.keys())
new = nextnew
self.overrides = None
+ self.expand_cache = {}
def _setvar_update_overrides(self, var, **loginfo):
# aka pay the cookie monster
@@ -662,7 +661,7 @@ class DataSmart(MutableMapping):
def getVar(self, var, expand=True, noweakdefault=False, parsing=False):
return self.getVarFlag(var, "_content", expand, noweakdefault, parsing)
- def renameVar(self, key, newkey, **loginfo):
+ def renameVar(self, key, newkey, recurse=True, **loginfo):
"""
Rename the variable key to newkey
"""
@@ -670,6 +669,8 @@ class DataSmart(MutableMapping):
bb.warn("Calling renameVar with equivalent keys (%s) is invalid" % key)
return
+ found = False
+
val = self.getVar(key, 0, parsing=True)
if val is not None:
self.varhistory.rename_variable_hist(key, newkey)
@@ -678,9 +679,11 @@ class DataSmart(MutableMapping):
loginfo['detail'] = val
self.varhistory.record(**loginfo)
self.setVar(newkey, val, ignore=True, parsing=True)
+ found = True
srcflags = self.getVarFlags(key, False, True) or {}
for i in srcflags:
+
if i not in (__setvar_keyword__):
continue
src = srcflags[i]
@@ -688,12 +691,20 @@ class DataSmart(MutableMapping):
dest = self.getVarFlag(newkey, i, False) or []
dest.extend(src)
self.setVarFlag(newkey, i, dest, ignore=True)
+ found = True
if key in self.overridedata:
+ found = True
self.overridedata[newkey] = []
for (v, o) in self.overridedata[key]:
self.overridedata[newkey].append([v.replace(key, newkey), o])
- self.renameVar(v, v.replace(key, newkey))
+ if recurse:
+ self.renameVar(v, v.replace(key, newkey))
+
+ if not found:
+ # No variable to rename so not worth the work in writing extra
+ # history data for performance
+ return
if ':' in newkey and val is None:
self._setvar_update_overrides(newkey, **loginfo)
@@ -781,10 +792,10 @@ class DataSmart(MutableMapping):
return None
cachename = var + "[" + flag + "]"
- if not expand and retparser and cachename in self.expand_cache:
+ if not expand and retparser and cachename in self.expand_cache and not noweakdefault:
return self.expand_cache[cachename].unexpanded_value, self.expand_cache[cachename]
- if expand and cachename in self.expand_cache:
+ if expand and cachename in self.expand_cache and not noweakdefault:
return self.expand_cache[cachename].value
local_var = self._findVar(var)
@@ -829,6 +840,8 @@ class DataSmart(MutableMapping):
value = copy.copy(local_var[flag])
elif flag == "_content" and "_defaultval" in local_var and not noweakdefault:
value = copy.copy(local_var["_defaultval"])
+ elif "_defaultval_flag_"+flag in local_var and not noweakdefault:
+ value = copy.copy(local_var["_defaultval_flag_"+flag])
if flag == "_content" and local_var is not None and ":append" in local_var and not parsing:
@@ -896,7 +909,13 @@ class DataSmart(MutableMapping):
if expand:
value = parser.value
- if parser:
+ if value and expand and flag == "_content":
+ basevar = var.split(":")[0]
+ if basevar in self.filters:
+ value = bb.filter.apply_filters(value, [self.filters[basevar],])
+ parser.value = value
+
+ if parser and not noweakdefault:
self.expand_cache[cachename] = parser
if retparser:
@@ -920,6 +939,8 @@ class DataSmart(MutableMapping):
self.varhistory.record(**loginfo)
del self.dict[var][flag]
+ if ("_defaultval_flag_" + flag) in self.dict[var]:
+ del self.dict[var]["_defaultval_flag_" + flag]
def appendVarFlag(self, var, flag, value, **loginfo):
loginfo['op'] = 'append'
@@ -954,17 +975,22 @@ class DataSmart(MutableMapping):
flags = {}
if local_var:
- for i in local_var:
- if i.startswith(("_", ":")) and not internalflags:
+ for i, val in local_var.items():
+ if i.startswith("_defaultval_flag_") and not internalflags:
+ i = i[len("_defaultval_flag_"):]
+ if i not in local_var:
+ flags[i] = val
+ elif i.startswith(("_", ":")) and not internalflags:
continue
- flags[i] = local_var[i]
+ else:
+ flags[i] = val
+
if expand and i in expand:
flags[i] = self.expand(flags[i], var + "[" + i + "]")
if len(flags) == 0:
return None
return flags
-
def delVarFlags(self, var, **loginfo):
self.expand_cache = {}
if not var in self.dict:
@@ -994,6 +1020,7 @@ class DataSmart(MutableMapping):
data.varhistory = self.varhistory.copy()
data.varhistory.dataroot = data
data.inchistory = self.inchistory.copy()
+ data.filters = self.filters.copy()
data._tracking = self._tracking
data._var_renames = self._var_renames
@@ -1022,6 +1049,15 @@ class DataSmart(MutableMapping):
if referrervalue and isinstance(referrervalue, str) and ref in referrervalue:
self.setVar(key, referrervalue.replace(ref, value))
+ def setVarFilter(self, var, filter):
+ if filter:
+ self.filters[var] = filter
+ else:
+ try:
+ del self.filters[var]
+ except KeyError:
+ pass
+
def localkeys(self):
for key in self.dict:
if key not in ['_data']:
@@ -1114,5 +1150,10 @@ class DataSmart(MutableMapping):
value = d.getVar(i, False) or ""
data.update({i:value})
+ moddeps = bb.codeparser.modulecode_deps
+ for dep in sorted(moddeps):
+ # Ignore visitor code, sort sets
+ data.update({'moddep[%s]' % dep : [sorted(moddeps[dep][0]), sorted(moddeps[dep][1]), sorted(moddeps[dep][2]), sorted(moddeps[dep][3]), moddeps[dep][4]]})
+
data_str = str([(k, data[k]) for k in sorted(data.keys())])
return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 952c85c0..ddf1006c 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -90,7 +90,7 @@ def disable_heartbeat():
# In long running code, this function should be called periodically
# to check if we should exit due to an interuption (.e.g Ctrl+C from the UI)
#
-def check_for_interrupts(d):
+def check_for_interrupts():
global _should_exit
if _should_exit.is_set():
bb.warn("Exiting due to interrupt.")
@@ -194,7 +194,12 @@ def fire_ui_handlers(event, d):
ui_queue.append(event)
return
- with bb.utils.lock_timeout(_thread_lock):
+ with bb.utils.lock_timeout_nocheck(_thread_lock) as lock:
+ if not lock:
+ # If we can't get the lock, we may be recursively called, queue and return
+ ui_queue.append(event)
+ return
+
errors = []
for h in _ui_handlers:
#print "Sending event %s" % event
@@ -213,6 +218,9 @@ def fire_ui_handlers(event, d):
for h in errors:
del _ui_handlers[h]
+ while ui_queue:
+ fire_ui_handlers(ui_queue.pop(), d)
+
def fire(event, d):
"""Fire off an Event"""
@@ -228,9 +236,12 @@ def fire(event, d):
# If messages have been queued up, clear the queue
global _uiready, ui_queue
if _uiready and ui_queue:
- for queue_event in ui_queue:
+ with bb.utils.lock_timeout_nocheck(_thread_lock):
+ queue = ui_queue
+ ui_queue = []
+ for queue_event in queue:
fire_ui_handlers(queue_event, d)
- ui_queue = []
+
fire_ui_handlers(event, d)
def fire_from_worker(event, d):
@@ -423,6 +434,16 @@ class RecipeEvent(Event):
self.fn = fn
Event.__init__(self)
+class RecipePreDeferredInherits(RecipeEvent):
+ """
+ Called before deferred inherits are processed so code can snoop on class extensions for example
+ Limitations: It won't see inherits of inherited classes and the data is unexpanded
+ """
+ def __init__(self, fn, inherits):
+ self.fn = fn
+ self.inherits = inherits
+ Event.__init__(self)
+
class RecipePreFinalise(RecipeEvent):
""" Recipe Parsing Complete but not yet finalised"""
diff --git a/bitbake/lib/bb/fetch2/README b/bitbake/lib/bb/fetch/README
similarity index 100%
rename from bitbake/lib/bb/fetch2/README
rename to bitbake/lib/bb/fetch/README
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch/__init__.py
similarity index 84%
rename from bitbake/lib/bb/fetch2/__init__.py
rename to bitbake/lib/bb/fetch/__init__.py
index 5bf2c4b8..fb627843 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch/__init__.py
@@ -23,17 +23,19 @@ import collections
import subprocess
import pickle
import errno
-import bb.persist_data, bb.utils
+import shlex
+import bb.utils
import bb.checksum
import bb.process
import bb.event
__version__ = "2"
_checksum_cache = bb.checksum.FileChecksumCache()
+_revisions_cache = bb.checksum.RevisionsCache()
logger = logging.getLogger("BitBake.Fetcher")
-CHECKSUM_LIST = [ "md5", "sha256", "sha1", "sha384", "sha512" ]
+CHECKSUM_LIST = [ "goh1", "md5", "sha256", "sha1", "sha384", "sha512" ]
SHOWN_CHECKSUM_LIST = ["sha256"]
class BBFetchException(Exception):
@@ -49,7 +51,7 @@ class UntrustedUrl(BBFetchException):
"""Exception raised when encountering a host not listed in BB_ALLOWED_NETWORKS"""
def __init__(self, url, message=''):
if message:
- msg = message
+ msg = str(message)
else:
msg = "The URL: '%s' is not trusted and cannot be used" % url
self.url = url
@@ -95,6 +97,23 @@ class UnpackError(BBFetchException):
BBFetchException.__init__(self, msg)
self.args = (message, url)
+class LocalModificationsError(UnpackError):
+ """Exception raised when a checkout cannot be updated due to local modifications"""
+ def __init__(self, repodir, url, git_output):
+ message = ("Repository at %s has uncommitted changes, unable to update:\n%s\n"
+ "Commit, stash or discard your changes and re-run the update." % (repodir, git_output))
+ UnpackError.__init__(self, message, url)
+ self.args = (repodir, url, git_output)
+
+class RebaseError(UnpackError):
+ """Exception raised when a checkout has local commits that could not be rebased onto the new upstream revision"""
+ def __init__(self, repodir, url, git_output):
+ message = ("Repository at %s has local commits that could not be rebased onto the new upstream revision:\n%s\n"
+ "Note: the 'dldir' remote points to the local download cache and may be used to resolve the conflict manually.\n"
+ "Once resolved, re-run the update." % (repodir, git_output))
+ UnpackError.__init__(self, message, url)
+ self.args = (repodir, url, git_output)
+
class NoMethodError(BBFetchException):
"""Exception raised when there is no method to obtain a supplied url or set of urls"""
def __init__(self, url):
@@ -237,7 +256,7 @@ class URI(object):
# to RFC compliant URL format. E.g.:
# file://foo.diff -> file:foo.diff
if urlp.scheme in self._netloc_forbidden:
- uri = re.sub("(?<=:)//(?!/)", "", uri, 1)
+ uri = re.sub(r"(?<=:)//(?!/)", "", uri, count=1)
reparse = 1
if reparse:
@@ -285,7 +304,7 @@ class URI(object):
def _query_str(self):
return (
- ''.join(['?', self._param_str_join(self.query, "&")])
+ ''.join(['?', self._query_str_join(self.query, "&")])
if self.query else '')
def _param_str_split(self, string, elmdelim, kvdelim="="):
@@ -294,9 +313,12 @@ class URI(object):
ret[k] = v
return ret
- def _param_str_join(self, dict_, elmdelim, kvdelim="="):
+ def _query_str_join(self, dict_, elmdelim, kvdelim="="):
return elmdelim.join([kvdelim.join([k, v]) if v else k for k, v in dict_.items()])
+ def _param_str_join(self, dict_, elmdelim, kvdelim="="):
+ return elmdelim.join([kvdelim.join([k, v]) for k, v in dict_.items()])
+
@property
def hostport(self):
if not self.port:
@@ -352,6 +374,14 @@ def decodeurl(url):
user, password, parameters).
"""
+ uri = URI(url)
+ path = uri.path if uri.path else "/"
+ return uri.scheme, uri.hostport, path, uri.username, uri.password, uri.params
+
+def decodemirrorurl(url):
+ """Decodes a mirror URL into the tokens (scheme, network location, path,
+ user, password, parameters).
+ """
m = re.compile('(?P<type>[^:]*)://((?P<user>[^/;]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
if not m:
raise MalformedUrl(url)
@@ -370,6 +400,9 @@ def decodeurl(url):
elif type.lower() == 'file':
host = ""
path = location
+ if user:
+ path = user + '@' + path
+ user = ""
else:
host = location
path = "/"
@@ -402,32 +435,34 @@ def encodeurl(decoded):
if not type:
raise MissingParameterError('type', "encoded from the data %s" % str(decoded))
- url = ['%s://' % type]
+ uri = URI()
+ uri.scheme = type
if user and type != "file":
- url.append("%s" % user)
+ uri.username = user
if pswd:
- url.append(":%s" % pswd)
- url.append("@")
+ uri.password = pswd
if host and type != "file":
- url.append("%s" % host)
+ uri.hostname = host
if path:
# Standardise path to ensure comparisons work
while '//' in path:
path = path.replace("//", "/")
- url.append("%s" % urllib.parse.quote(path))
+ uri.path = path
+ if type == "file":
+ # Use old not IETF compliant style
+ uri.relative = False
if p:
- for parm in p:
- url.append(";%s=%s" % (parm, p[parm]))
+ uri.params = p
- return "".join(url)
+ return str(uri)
def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
if not ud.url or not uri_find or not uri_replace:
logger.error("uri_replace: passed an undefined value, not replacing")
return None
- uri_decoded = list(decodeurl(ud.url))
- uri_find_decoded = list(decodeurl(uri_find))
- uri_replace_decoded = list(decodeurl(uri_replace))
+ uri_decoded = list(decodemirrorurl(ud.url))
+ uri_find_decoded = list(decodemirrorurl(uri_find))
+ uri_replace_decoded = list(decodemirrorurl(uri_replace))
logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded))
result_decoded = ['', '', '', '', '', {}]
# 0 - type, 1 - host, 2 - path, 3 - user, 4- pswd, 5 - params
@@ -460,7 +495,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
for k in replacements:
uri_replace_decoded[loc] = uri_replace_decoded[loc].replace(k, replacements[k])
#bb.note("%s %s %s" % (regexp, uri_replace_decoded[loc], uri_decoded[loc]))
- result_decoded[loc] = re.sub(regexp, uri_replace_decoded[loc], uri_decoded[loc], 1)
+ result_decoded[loc] = re.sub(regexp, uri_replace_decoded[loc], uri_decoded[loc], count=1)
if loc == 2:
# Handle path manipulations
basename = None
@@ -493,18 +528,23 @@ methods = []
urldata_cache = {}
saved_headrevs = {}
-def fetcher_init(d):
+def fetcher_init(d, servercontext=True):
"""
Called to initialize the fetchers once the configuration data is known.
Calls before this must not hit the cache.
"""
- revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
+ _checksum_cache.init_cache(d.getVar("BB_CACHEDIR"))
+ _revisions_cache.init_cache(d.getVar("BB_CACHEDIR"))
+
+ if not servercontext:
+ return
+
try:
# fetcher_init is called multiple times, so make sure we only save the
# revs the first time it is called.
- if not bb.fetch2.saved_headrevs:
- bb.fetch2.saved_headrevs = dict(revs)
+ if not bb.fetch.saved_headrevs:
+ bb.fetch.saved_headrevs = _revisions_cache.get_revs()
except:
pass
@@ -514,11 +554,10 @@ def fetcher_init(d):
logger.debug("Keeping SRCREV cache due to cache policy of: %s", srcrev_policy)
elif srcrev_policy == "clear":
logger.debug("Clearing SRCREV cache due to cache policy of: %s", srcrev_policy)
- revs.clear()
+ _revisions_cache.clear_cache()
else:
raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy)
- _checksum_cache.init_cache(d.getVar("BB_CACHEDIR"))
for m in methods:
if hasattr(m, "init"):
@@ -526,9 +565,11 @@ def fetcher_init(d):
def fetcher_parse_save():
_checksum_cache.save_extras()
+ _revisions_cache.save_extras()
def fetcher_parse_done():
_checksum_cache.save_merge()
+ _revisions_cache.save_merge()
def fetcher_compare_revisions(d):
"""
@@ -536,8 +577,8 @@ def fetcher_compare_revisions(d):
when bitbake was started and return true if they have changed.
"""
- headrevs = dict(bb.persist_data.persist('BB_URI_HEADREVS', d))
- return headrevs != bb.fetch2.saved_headrevs
+ headrevs = _revisions_cache.get_revs()
+ return headrevs != bb.fetch.saved_headrevs
def mirror_from_string(data):
mirrors = (data or "").replace('\\n',' ').split()
@@ -786,8 +827,8 @@ def _get_srcrev(d, method_name='sortable_revision'):
return "", revs
- if len(scms) == 1 and len(urldata[scms[0]].names) == 1:
- autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
+ if len(scms) == 1:
+ autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].name)
revs.append(rev)
if len(rev) > 10:
rev = rev[:10]
@@ -808,13 +849,12 @@ def _get_srcrev(d, method_name='sortable_revision'):
seenautoinc = False
for scm in scms:
ud = urldata[scm]
- for name in ud.names:
- autoinc, rev = getattr(ud.method, method_name)(ud, d, name)
- revs.append(rev)
- seenautoinc = seenautoinc or autoinc
- if len(rev) > 10:
- rev = rev[:10]
- name_to_rev[name] = rev
+ autoinc, rev = getattr(ud.method, method_name)(ud, d, ud.name)
+ revs.append(rev)
+ seenautoinc = seenautoinc or autoinc
+ if len(rev) > 10:
+ rev = rev[:10]
+ name_to_rev[ud.name] = rev
# Replace names by revisions in the SRCREV_FORMAT string. The approach used
# here can handle names being prefixes of other names and names appearing
# as substrings in revisions (in which case the name should not be
@@ -845,7 +885,7 @@ def get_srcrev(d, method_name='sortable_revision'):
return pkgv
def localpath(url, d):
- fetcher = bb.fetch2.Fetch([url], d)
+ fetcher = bb.fetch.Fetch([url], d)
return fetcher.localpath(url)
# Need to export PATH as binary could be in metadata paths
@@ -858,6 +898,7 @@ FETCH_EXPORT_VARS = ['HOME', 'PATH',
'FTPS_PROXY', 'ftps_proxy',
'NO_PROXY', 'no_proxy',
'ALL_PROXY', 'all_proxy',
+ 'GIT_CONFIG_GLOBAL',
'GIT_PROXY_COMMAND',
'GIT_SSH',
'GIT_SSH_COMMAND',
@@ -870,6 +911,7 @@ FETCH_EXPORT_VARS = ['HOME', 'PATH',
'SSL_CERT_FILE',
'NODE_EXTRA_CA_CERTS',
'AWS_PROFILE',
+ 'AWS_SHARED_CREDENTIALS_FILE',
'AWS_ACCESS_KEY_ID',
'AWS_SECRET_ACCESS_KEY',
'AWS_ROLE_ARN',
@@ -878,20 +920,21 @@ FETCH_EXPORT_VARS = ['HOME', 'PATH',
'AWS_SESSION_TOKEN',
'GIT_CACHE_PATH',
'REMOTE_CONTAINERS_IPC',
+ 'GITHUB_TOKEN',
'SSL_CERT_DIR']
def get_fetcher_environment(d):
newenv = {}
origenv = d.getVar("BB_ORIGENV")
- for name in bb.fetch2.FETCH_EXPORT_VARS:
+ for name in bb.fetch.FETCH_EXPORT_VARS:
value = d.getVar(name)
- if not value and origenv:
+ if not value and origenv is not None:
value = origenv.getVar(name)
if value:
newenv[name] = value
return newenv
-def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
+def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None, extraenv=None):
"""
Run cmd returning the command output
Raise an error if interrupted or cmd fails
@@ -916,13 +959,17 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
d.setVar("PR", "fetcheravoidrecurse")
origenv = d.getVar("BB_ORIGENV", False)
+ env = os.environ.copy()
for var in exportvars:
- val = d.getVar(var) or (origenv and origenv.getVar(var))
+ val = d.getVar(var) or (origenv is not None and origenv.getVar(var))
if val:
- cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
+ env[var] = val
+
+ for var in (extraenv or []):
+ env[var] = extraenv[var]
# Disable pseudo as it may affect ssh, potentially causing it to hang.
- cmd = 'export PSEUDO_DISABLED=1; ' + cmd
+ env["PSEUDO_DISABLED"] = "1"
if workdir:
logger.debug("Running '%s' in %s" % (cmd, workdir))
@@ -933,7 +980,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
error_message = ""
try:
- (output, errors) = bb.process.run(cmd, log=log, shell=True, stderr=subprocess.PIPE, cwd=workdir)
+ (output, errors) = bb.process.run(cmd, log=log, stderr=subprocess.PIPE, cwd=workdir, env=env)
success = True
except bb.process.NotFoundError as e:
error_message = "Fetch command %s not found" % (e.command)
@@ -1010,7 +1057,10 @@ def build_mirroruris(origud, mirrors, ld):
newud = FetchData(newuri, ld)
newud.ignore_checksums = True
newud.setup_localpath(ld)
- except bb.fetch2.BBFetchException as e:
+ if hasattr(ud, 'unpack_tracer'):
+ newud.unpack_tracer = ud.unpack_tracer
+ newud.origud = ud
+ except bb.fetch.BBFetchException as e:
logger.debug("Mirror fetch failure for url %s (original url: %s)" % (newuri, origud.url))
logger.debug(str(e))
try:
@@ -1047,10 +1097,11 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
# Return of None or a value means we're finished
# False means try another url
- if ud.lockfile and ud.lockfile != origud.lockfile:
- lf = bb.utils.lockfile(ud.lockfile)
-
try:
+ lf = None
+ if ud.lockfile != origud.lockfile:
+ lf = bb.utils.lockfile(ud.lockfile)
+
if check:
found = ud.method.checkstatus(fetch, ud, ld)
if found:
@@ -1072,6 +1123,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
# If that tarball is a local file:// we need to provide a symlink to it
dldir = ld.getVar("DL_DIR")
+ if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
+ ld = ld.createCopy()
+ ld.setVar("BB_NO_NETWORK", "1")
+
if origud.mirrortarballs and os.path.basename(ud.localpath) in origud.mirrortarballs and os.path.basename(ud.localpath) != os.path.basename(origud.localpath):
# Create donestamp in old format to avoid triggering a re-download
if ud.donestamp:
@@ -1093,20 +1148,17 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
origud.method.build_mirror_data(origud, ld)
return origud.localpath
# Otherwise the result is a local file:// and we symlink to it
- ensure_symlink(ud.localpath, origud.localpath)
+ # This may also be a link to a shallow archive
+ # When using shallow mode, add a symlink to the original fullshallow
+ # path to ensure a valid symlink even in the `PREMIRRORS` case
+ origud.method.update_mirror_links(ud, origud)
update_stamp(origud, ld)
return ud.localpath
- except bb.fetch2.NetworkAccess:
+ except bb.fetch.NetworkAccess:
raise
- except IOError as e:
- if e.errno in [errno.ESTALE]:
- logger.warning("Stale Error Observed %s." % ud.url)
- return False
- raise
-
- except bb.fetch2.BBFetchException as e:
+ except bb.fetch.BBFetchException as e:
if isinstance(e, ChecksumError):
logger.warning("Mirror checksum failure for url %s (original url: %s)\nCleaning and trying again." % (ud.url, origud.url))
logger.warning(str(e))
@@ -1124,37 +1176,21 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
pass
return False
finally:
- if ud.lockfile and ud.lockfile != origud.lockfile:
+ if ud.lockfile != origud.lockfile:
bb.utils.unlockfile(lf)
-
-def ensure_symlink(target, link_name):
- if not os.path.exists(link_name):
- dirname = os.path.dirname(link_name)
- bb.utils.mkdirhier(dirname)
- if os.path.islink(link_name):
- # Broken symbolic link
- os.unlink(link_name)
-
- # In case this is executing without any file locks held (as is
- # the case for file:// URLs), two tasks may end up here at the
- # same time, in which case we do not want the second task to
- # fail when the link has already been created by the first task.
- try:
- os.symlink(target, link_name)
- except FileExistsError:
- pass
-
-
-def try_mirrors(fetch, d, origud, mirrors, check = False):
+def try_mirrors(fetch, d, origud, mirrorvar, check = False):
"""
Try to use a mirrored version of the sources.
This method will be automatically called before the fetchers go.
d Is a bb.data instance
uri is the original uri we're trying to download
- mirrors is the list of mirrors we're going to try
+ mirrorvar is variable name that contains the list of mirrors we're going to try
"""
+ logger.debug("Trying mirrors in %s" % mirrorvar)
+ mirrors = mirror_from_string(d.getVar(mirrorvar))
+
ld = d.createCopy()
uris, uds = build_mirroruris(origud, mirrors, ld)
@@ -1174,7 +1210,7 @@ def trusted_network(d, url):
if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
return True
- pkgname = d.expand(d.getVar('PN', False))
+ pkgname = d.getVar('PN')
trusted_hosts = None
if pkgname:
trusted_hosts = d.getVarFlag('BB_ALLOWED_NETWORKS', pkgname, False)
@@ -1227,20 +1263,17 @@ def srcrev_internal_helper(ud, d, name):
if srcrev and srcrev != "INVALID":
break
- if 'rev' in ud.parm and 'tag' in ud.parm:
- raise FetchError("Please specify a ;rev= parameter or a ;tag= parameter in the url %s but not both." % (ud.url))
-
- if 'rev' in ud.parm or 'tag' in ud.parm:
- if 'rev' in ud.parm:
- parmrev = ud.parm['rev']
- else:
- parmrev = ud.parm['tag']
+ if 'rev' in ud.parm:
+ parmrev = ud.parm['rev']
if srcrev == "INVALID" or not srcrev:
return parmrev
if srcrev != parmrev:
raise FetchError("Conflicting revisions (%s from SRCREV and %s from the url) found, please specify one valid value" % (srcrev, parmrev))
return parmrev
+ if 'tag' in ud.parm and (srcrev == "INVALID" or not srcrev):
+ return ud.parm['tag']
+
if srcrev == "INVALID" or not srcrev:
raise FetchError("Please set a valid SRCREV for url %s (possible key names are %s, or use a ;rev=X URL parameter)" % (str(attempts), ud.url), ud.url)
if srcrev == "AUTOINC":
@@ -1263,7 +1296,7 @@ def get_checksum_file_list(d):
found = False
paths = ud.method.localfile_searchpaths(ud, d)
for f in paths:
- pth = ud.decodedurl
+ pth = ud.path
if os.path.exists(f):
found = True
filelist.append(f + ":" + str(os.path.exists(f)))
@@ -1308,23 +1341,28 @@ class FetchData(object):
self.setup = False
def configure_checksum(checksum_id):
+ checksum_plain_name = "%ssum" % checksum_id
if "name" in self.parm:
checksum_name = "%s.%ssum" % (self.parm["name"], checksum_id)
else:
- checksum_name = "%ssum" % checksum_id
-
- setattr(self, "%s_name" % checksum_id, checksum_name)
+ checksum_name = checksum_plain_name
if checksum_name in self.parm:
checksum_expected = self.parm[checksum_name]
- elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs"]:
+ elif checksum_plain_name in self.parm:
+ checksum_expected = self.parm[checksum_plain_name]
+ checksum_name = checksum_plain_name
+ elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs", "gomod", "npm"]:
checksum_expected = None
else:
checksum_expected = d.getVarFlag("SRC_URI", checksum_name)
+ setattr(self, "%s_name" % checksum_id, checksum_name)
setattr(self, "%s_expected" % checksum_id, checksum_expected)
- self.names = self.parm.get("name",'default').split(',')
+ self.name = self.parm.get("name",'default')
+ if "," in self.name:
+ raise ParameterError("The fetcher no longer supports multiple name parameters in a single url", self.url)
self.method = None
for m in methods:
@@ -1376,13 +1414,7 @@ class FetchData(object):
self.lockfile = basepath + '.lock'
def setup_revisions(self, d):
- self.revisions = {}
- for name in self.names:
- self.revisions[name] = srcrev_internal_helper(self, d, name)
-
- # add compatibility code for non name specified case
- if len(self.names) == 1:
- self.revision = self.revisions[self.names[0]]
+ self.revision = srcrev_internal_helper(self, d, self.name)
def setup_localpath(self, d):
if not self.localpath:
@@ -1407,9 +1439,6 @@ class FetchData(object):
class FetchMethod(object):
"""Base class for 'fetch'ing data"""
- def __init__(self, urls=None):
- self.urls = []
-
def supports(self, urldata, d):
"""
Check to see if this fetch class supports a given url.
@@ -1470,14 +1499,6 @@ class FetchMethod(object):
relpath = relpath[1:]
return relpath
- def setUrls(self, urls):
- self.__urls = urls
-
- def getUrls(self):
- return self.__urls
-
- urls = property(getUrls, setUrls, None, "Urls property")
-
def need_update(self, ud, d):
"""
Force a fetch, even if localpath exists?
@@ -1509,41 +1530,70 @@ class FetchMethod(object):
bb.fatal("Invalid value for 'unpack' parameter for %s: %s" %
(file, urldata.parm.get('unpack')))
+ env = os.environ.copy()
+ path = data.getVar('PATH')
+ if path:
+ env['PATH'] = path
+
+ # If 'subdir' param exists, create a dir and use it as destination for unpack cmd
+ if 'subdir' in urldata.parm:
+ subdir = urldata.parm.get('subdir')
+ if os.path.isabs(subdir):
+ if not os.path.realpath(subdir).startswith(os.path.realpath(rootdir)):
+ raise UnpackError("subdir argument isn't a subdirectory of unpack root %s" % rootdir, urldata.url)
+ unpackdir = subdir
+ else:
+ unpackdir = os.path.join(rootdir, subdir)
+ bb.utils.mkdirhier(unpackdir)
+ else:
+ unpackdir = rootdir
+
+ def _run(cmd):
+ if isinstance(cmd, str):
+ ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir, env=env)
+ else:
+ ret = subprocess.call(cmd, preexec_fn=subprocess_setup, cwd=unpackdir, env=env)
+ if ret != 0:
+ raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url)
+
base, ext = os.path.splitext(file)
- if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz']:
+ if ext in ['.gz', '.bz2', '.Z', '.xz', '.lz', '.zst']:
efile = os.path.join(rootdir, os.path.basename(base))
else:
efile = file
cmd = None
if unpack:
- tar_cmd = 'tar --extract --no-same-owner'
+ tar_cmd = ['tar', '--extract', '--no-same-owner']
if 'striplevel' in urldata.parm:
- tar_cmd += ' --strip-components=%s' % urldata.parm['striplevel']
+ striplevel = urldata.parm['striplevel']
+ if not striplevel.isdigit():
+ raise UnpackError("Invalid striplevel parameter: %s" % striplevel, urldata.url)
+ tar_cmd.append('--strip-components=%s' % striplevel)
if file.endswith('.tar'):
- cmd = '%s -f %s' % (tar_cmd, file)
+ cmd = tar_cmd + ['-f', file]
elif file.endswith('.tgz') or file.endswith('.tar.gz') or file.endswith('.tar.Z'):
- cmd = '%s -z -f %s' % (tar_cmd, file)
+ cmd = tar_cmd + ['-z', '-f', file]
elif file.endswith('.tbz') or file.endswith('.tbz2') or file.endswith('.tar.bz2'):
- cmd = 'bzip2 -dc %s | %s -f -' % (file, tar_cmd)
+ cmd = tar_cmd + ['-j', '-f', file]
elif file.endswith('.gz') or file.endswith('.Z') or file.endswith('.z'):
cmd = 'gzip -dc %s > %s' % (file, efile)
elif file.endswith('.bz2'):
cmd = 'bzip2 -dc %s > %s' % (file, efile)
elif file.endswith('.txz') or file.endswith('.tar.xz'):
- cmd = 'xz -dc %s | %s -f -' % (file, tar_cmd)
+ cmd = tar_cmd + ['-J', '-f', file]
elif file.endswith('.xz'):
cmd = 'xz -dc %s > %s' % (file, efile)
elif file.endswith('.tar.lz'):
- cmd = 'lzip -dc %s | %s -f -' % (file, tar_cmd)
+ cmd = tar_cmd + ['--lzip', '-f', file]
elif file.endswith('.lz'):
cmd = 'lzip -dc %s > %s' % (file, efile)
elif file.endswith('.tar.7z'):
- cmd = '7z x -so %s | %s -f -' % (file, tar_cmd)
+ cmd = '7z x -so %s | %s -f -' % (file, shlex.join(tar_cmd))
elif file.endswith('.7z'):
cmd = '7za x -y %s 1>/dev/null' % file
elif file.endswith('.tzst') or file.endswith('.tar.zst'):
- cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, tar_cmd)
+ cmd = 'zstd --decompress --stdout %s | %s -f -' % (file, shlex.join(tar_cmd))
elif file.endswith('.zst'):
cmd = 'zstd --decompress --stdout %s > %s' % (file, efile)
elif file.endswith('.zip') or file.endswith('.jar'):
@@ -1552,44 +1602,38 @@ class FetchMethod(object):
except ValueError as exc:
bb.fatal("Invalid value for 'dos' parameter for %s: %s" %
(file, urldata.parm.get('dos')))
- cmd = 'unzip -q -o'
+ cmd = ['unzip', '-q', '-o']
if dos:
- cmd = '%s -a' % cmd
- cmd = "%s '%s'" % (cmd, file)
+ cmd.append('-a')
+ cmd.append(file)
elif file.endswith('.rpm') or file.endswith('.srpm'):
if 'extract' in urldata.parm:
unpack_file = urldata.parm.get('extract')
- cmd = 'rpm2cpio.sh %s | cpio -id %s' % (file, unpack_file)
+ cmd = 'rpm2cpio.sh %s | cpio --no-absolute-filenames -id %s' % (file, unpack_file)
iterate = True
iterate_file = unpack_file
else:
- cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
+ cmd = 'rpm2cpio.sh %s | cpio --no-absolute-filenames -id' % (file)
elif file.endswith('.deb') or file.endswith('.ipk'):
- output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup)
+ output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup, env=env)
datafile = None
+ valid_datafiles = ('data.tar', 'data.tar.gz', 'data.tar.xz',
+ 'data.tar.zst', 'data.tar.bz2', 'data.tar.lzma')
if output:
for line in output.decode().splitlines():
- if line.startswith('data.tar.'):
+ if line in valid_datafiles:
datafile = line
break
else:
- raise UnpackError("Unable to unpack deb/ipk package - does not contain data.tar.* file", urldata.url)
+ raise UnpackError("Unable to unpack deb/ipk package - does not contain supported data.tar* file", urldata.url)
else:
raise UnpackError("Unable to unpack deb/ipk package - could not list contents", urldata.url)
- cmd = 'ar x %s %s && %s -p -f %s && rm %s' % (file, datafile, tar_cmd, datafile, datafile)
-
- # If 'subdir' param exists, create a dir and use it as destination for unpack cmd
- if 'subdir' in urldata.parm:
- subdir = urldata.parm.get('subdir')
- if os.path.isabs(subdir):
- if not os.path.realpath(subdir).startswith(os.path.realpath(rootdir)):
- raise UnpackError("subdir argument isn't a subdirectory of unpack root %s" % rootdir, urldata.url)
- unpackdir = subdir
- else:
- unpackdir = os.path.join(rootdir, subdir)
- bb.utils.mkdirhier(unpackdir)
- else:
- unpackdir = rootdir
+ bb.note("Unpacking %s to %s/" % (file, unpackdir))
+ urldata.unpack_tracer.unpack("archive-extract", unpackdir)
+ _run(['ar', 'x', file, datafile])
+ _run(tar_cmd + ['-p' ,'-f', datafile])
+ os.remove(os.path.join(unpackdir, datafile))
+ return
if not unpack or not cmd:
urldata.unpack_tracer.unpack("file-copy", unpackdir)
@@ -1606,21 +1650,15 @@ class FetchMethod(object):
if urlpath.find("/") != -1:
destdir = urlpath.rsplit("/", 1)[0] + '/'
bb.utils.mkdirhier("%s/%s" % (unpackdir, destdir))
- cmd = 'cp -fpPRH "%s" "%s"' % (file, destdir)
+ cmd = ['cp', '--force', '--preserve=timestamps', '--no-dereference', '--recursive', '-H', file, destdir]
else:
urldata.unpack_tracer.unpack("archive-extract", unpackdir)
if not cmd:
return
- path = data.getVar('PATH')
- if path:
- cmd = "PATH=\"%s\" %s" % (path, cmd)
bb.note("Unpacking %s to %s/" % (file, unpackdir))
- ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True, cwd=unpackdir)
-
- if ret != 0:
- raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url)
+ _run(cmd)
if iterate is True:
iterate_urldata = urldata
@@ -1629,23 +1667,48 @@ class FetchMethod(object):
return
+ def unpack_update(self, urldata, rootdir, data):
+ raise RuntimeError("No method available for this url type: %s" % urldata.type)
+
def clean(self, urldata, d):
"""
Clean any existing full or partial download
"""
bb.utils.remove(urldata.localpath)
+ def ensure_symlink(self, target, link_name):
+ if not os.path.exists(link_name):
+ dirname = os.path.dirname(link_name)
+ bb.utils.mkdirhier(dirname)
+ if os.path.islink(link_name):
+ # Broken symbolic link
+ os.unlink(link_name)
+
+ # In case this is executing without any file locks held (as is
+ # the case for file:// URLs), two tasks may end up here at the
+ # same time, in which case we do not want the second task to
+ # fail when the link has already been created by the first task.
+ try:
+ os.symlink(target, link_name)
+ except FileExistsError:
+ pass
+
+ def update_mirror_links(self, ud, origud):
+ # For local file:// results, create a symlink to them
+ # This may also be a link to a shallow archive
+ self.ensure_symlink(ud.localpath, origud.localpath)
+
def try_premirror(self, urldata, d):
"""
Should premirrors be used?
"""
return True
- def try_mirrors(self, fetch, urldata, d, mirrors, check=False):
+ def try_mirrors(self, fetch, urldata, d, mirrorvar, check=False):
"""
Try to use a mirror
"""
- return bool(try_mirrors(fetch, d, urldata, mirrors, check))
+ return bool(try_mirrors(fetch, d, urldata, mirrorvar, check))
def checkstatus(self, fetch, urldata, d):
"""
@@ -1662,13 +1725,13 @@ class FetchMethod(object):
if not hasattr(self, "_latest_revision"):
raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url)
- revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
key = self.generate_revision_key(ud, d, name)
- try:
- return revs[key]
- except KeyError:
- revs[key] = rev = self._latest_revision(ud, d, name)
- return rev
+
+ rev = _revisions_cache.get_rev(key)
+ if rev is None:
+ rev = self._latest_revision(ud, d, name)
+ _revisions_cache.set_rev(key, rev)
+ return rev
def sortable_revision(self, ud, d, name):
latest_rev = self._build_revision(ud, d, name)
@@ -1677,7 +1740,7 @@ class FetchMethod(object):
def generate_revision_key(self, ud, d, name):
return self._revision_key(ud, d, name)
- def latest_versionstring(self, ud, d):
+ def latest_versionstring(self, ud, d, filter_regex=None):
"""
Compute the latest release name like "x.y.x" in "x.y.x+gitHASH"
by searching through the tags output of ls-remote, comparing
@@ -1752,7 +1815,7 @@ class DummyUnpackTracer(object):
class Fetch(object):
def __init__(self, urls, d, cache = True, localonly = False, connection_cache = None):
if localonly and cache:
- raise Exception("bb.fetch2.Fetch.__init__: cannot set cache and localonly at same time")
+ raise Exception("bb.fetch.Fetch.__init__: cannot set cache and localonly at same time")
if not urls:
urls = d.getVar("SRC_URI").split()
@@ -1806,7 +1869,7 @@ class Fetch(object):
self.ud[url] = FetchData(url, self.d)
self.ud[url].setup_localpath(self.d)
- return self.d.expand(self.ud[url].localpath)
+ return self.ud[url].localpath
def localpaths(self):
"""
@@ -1838,48 +1901,41 @@ class Fetch(object):
m = ud.method
done = False
- if ud.lockfile:
- lf = bb.utils.lockfile(ud.lockfile)
-
try:
+ lf = bb.utils.lockfile(ud.lockfile, shared=True)
+
self.d.setVar("BB_NO_NETWORK", network)
if m.verify_donestamp(ud, self.d) and not m.need_update(ud, self.d):
done = True
- elif m.try_premirror(ud, self.d):
- logger.debug("Trying PREMIRRORS")
- mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
- done = m.try_mirrors(self, ud, self.d, mirrors)
- if done:
- try:
- # early checksum verification so that if the checksum of the premirror
- # contents mismatch the fetcher can still try upstream and mirrors
- m.update_donestamp(ud, self.d)
- except ChecksumError as e:
- logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % u)
- logger.debug(str(e))
- done = False
+ if not done:
+ lf = bb.utils.lockfile_to_exclusive(lf)
+ if m.try_premirror(ud, self.d):
+ done = m.try_mirrors(self, ud, self.d, 'PREMIRRORS')
+ d = self.d
if premirroronly:
- self.d.setVar("BB_NO_NETWORK", "1")
+ # Only disable the network in a copy
+ d = bb.data.createCopy(self.d)
+ d.setVar("BB_NO_NETWORK", "1")
firsterr = None
verified_stamp = False
if done:
- verified_stamp = m.verify_donestamp(ud, self.d)
- if not done and (not verified_stamp or m.need_update(ud, self.d)):
+ verified_stamp = m.verify_donestamp(ud, d)
+ if not done and (not verified_stamp or m.need_update(ud, d)):
try:
- if not trusted_network(self.d, ud.url):
+ if not trusted_network(d, ud.url):
raise UntrustedUrl(ud.url)
logger.debug("Trying Upstream")
- m.download(ud, self.d)
+ m.download(ud, d)
if hasattr(m, "build_mirror_data"):
- m.build_mirror_data(ud, self.d)
+ m.build_mirror_data(ud, d)
done = True
# early checksum verify, so that if checksum mismatched,
# fetcher still have chance to fetch from mirror
- m.update_donestamp(ud, self.d)
+ m.update_donestamp(ud, d)
- except bb.fetch2.NetworkAccess:
+ except bb.fetch.NetworkAccess:
raise
except BBFetchException as e:
@@ -1896,22 +1952,15 @@ class Fetch(object):
firsterr = e
# Remove any incomplete fetch
if not verified_stamp and m.cleanup_upon_failure():
- m.clean(ud, self.d)
- logger.debug("Trying MIRRORS")
- mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
- done = m.try_mirrors(self, ud, self.d, mirrors)
+ m.clean(ud, d)
+ done = m.try_mirrors(self, ud, d, 'MIRRORS')
- if not done or not m.done(ud, self.d):
+ if not done or not m.done(ud, d):
if firsterr:
logger.error(str(firsterr))
raise FetchError("Unable to fetch URL from any source.", u)
- m.update_donestamp(ud, self.d)
-
- except IOError as e:
- if e.errno in [errno.ESTALE]:
- logger.error("Stale Error Observed %s." % u)
- raise ChecksumError("Stale Error Detected")
+ m.update_donestamp(ud, d)
except BBFetchException as e:
if isinstance(e, NoChecksumError):
@@ -1923,8 +1972,7 @@ class Fetch(object):
raise
finally:
- if ud.lockfile:
- bb.utils.unlockfile(lf)
+ bb.utils.unlockfile(lf)
if checksum_missing_messages:
logger.error("Missing SRC_URI checksum, please add those to the recipe: \n%s", "\n".join(checksum_missing_messages))
raise BBFetchException("There was some missing checksums in the recipe")
@@ -1948,19 +1996,18 @@ class Fetch(object):
logger.debug("Testing URL %s", u)
# First try checking uri, u, from PREMIRRORS
mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
- ret = m.try_mirrors(self, ud, self.d, mirrors, True)
+ ret = m.try_mirrors(self, ud, self.d, 'PREMIRRORS', True)
if not ret:
# Next try checking from the original uri, u
ret = m.checkstatus(self, ud, self.d)
if not ret:
# Finally, try checking uri, u, from MIRRORS
- mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
- ret = m.try_mirrors(self, ud, self.d, mirrors, True)
+ ret = m.try_mirrors(self, ud, self.d, 'MIRRORS', True)
if not ret:
raise FetchError("URL doesn't work", u)
- def unpack(self, root, urls=None):
+ def unpack(self, root, urls=None, update=False):
"""
Unpack urls to root
"""
@@ -1971,21 +2018,27 @@ class Fetch(object):
unpack_tracer.start(root, self.ud, self.d)
for u in urls:
- ud = self.ud[u]
- ud.setup_localpath(self.d)
+ try:
+ ud = self.ud[u]
+ ud.setup_localpath(self.d)
- if ud.lockfile:
- lf = bb.utils.lockfile(ud.lockfile)
+ lf = bb.utils.lockfile(ud.lockfile, shared=True)
- unpack_tracer.start_url(u)
- ud.method.unpack(ud, root, self.d)
- unpack_tracer.finish_url(u)
+ unpack_tracer.start_url(u)
+ if update:
+ ud.method.unpack_update(ud, root, self.d)
+ else:
+ ud.method.unpack(ud, root, self.d)
+ unpack_tracer.finish_url(u)
- if ud.lockfile:
+ finally:
bb.utils.unlockfile(lf)
unpack_tracer.complete()
+ def unpack_update(self, root, urls=None):
+ self.unpack(root, urls, update=True)
+
def clean(self, urls=None):
"""
Clean files that the fetcher gets or places
@@ -1995,22 +2048,22 @@ class Fetch(object):
urls = self.urls
for url in urls:
- if url not in self.ud:
- self.ud[url] = FetchData(url, self.d)
- ud = self.ud[url]
- ud.setup_localpath(self.d)
+ try:
+ if url not in self.ud:
+ self.ud[url] = FetchData(url, self.d)
+ ud = self.ud[url]
+ ud.setup_localpath(self.d)
- if not ud.localfile and ud.localpath is None:
- continue
+ if not ud.localfile and ud.localpath is None:
+ continue
- if ud.lockfile:
lf = bb.utils.lockfile(ud.lockfile)
- ud.method.clean(ud, self.d)
- if ud.donestamp:
- bb.utils.remove(ud.donestamp)
+ ud.method.clean(ud, self.d)
+ if ud.donestamp:
+ bb.utils.remove(ud.donestamp)
- if ud.lockfile:
+ finally:
bb.utils.unlockfile(lf)
def expanded_urldata(self, urls=None):
@@ -2038,26 +2091,28 @@ class FetchConnectionCache(object):
def __init__(self):
self.cache = {}
- def get_connection_name(self, host, port):
- return host + ':' + str(port)
+ def get_connection_name(self, host, port, connection_id=None):
+ if connection_id is None:
+ return host + ':' + str(port)
+ return (host, port, connection_id)
- def add_connection(self, host, port, connection):
- cn = self.get_connection_name(host, port)
+ def add_connection(self, host, port, connection, connection_id=None):
+ cn = self.get_connection_name(host, port, connection_id)
if cn not in self.cache:
self.cache[cn] = connection
- def get_connection(self, host, port):
+ def get_connection(self, host, port, connection_id=None):
connection = None
- cn = self.get_connection_name(host, port)
+ cn = self.get_connection_name(host, port, connection_id)
if cn in self.cache:
connection = self.cache[cn]
return connection
- def remove_connection(self, host, port):
- cn = self.get_connection_name(host, port)
+ def remove_connection(self, host, port, connection_id=None):
+ cn = self.get_connection_name(host, port, connection_id)
if cn in self.cache:
self.cache[cn].close()
del self.cache[cn]
@@ -2067,7 +2122,6 @@ class FetchConnectionCache(object):
self.cache[cn].close()
del self.cache[cn]
-from . import cvs
from . import git
from . import gitsm
from . import gitannex
@@ -2078,9 +2132,7 @@ from . import ssh
from . import sftp
from . import s3
from . import perforce
-from . import bzr
from . import hg
-from . import osc
from . import repo
from . import clearcase
from . import npm
@@ -2088,6 +2140,7 @@ from . import npmsw
from . import az
from . import crate
from . import gcp
+from . import gomod
methods.append(local.Local())
methods.append(wget.Wget())
@@ -2095,14 +2148,11 @@ methods.append(svn.Svn())
methods.append(git.Git())
methods.append(gitsm.GitSM())
methods.append(gitannex.GitANNEX())
-methods.append(cvs.Cvs())
methods.append(ssh.SSH())
methods.append(sftp.SFTP())
methods.append(s3.S3())
methods.append(perforce.Perforce())
-methods.append(bzr.Bzr())
methods.append(hg.Hg())
-methods.append(osc.Osc())
methods.append(repo.Repo())
methods.append(clearcase.ClearCase())
methods.append(npm.Npm())
@@ -2110,3 +2160,5 @@ methods.append(npmsw.NpmShrinkWrap())
methods.append(az.Az())
methods.append(crate.Crate())
methods.append(gcp.GCP())
+methods.append(gomod.GoMod())
+methods.append(gomod.GoModGit())
diff --git a/bitbake/lib/bb/fetch2/az.py b/bitbake/lib/bb/fetch/az.py
similarity index 78%
rename from bitbake/lib/bb/fetch2/az.py
rename to bitbake/lib/bb/fetch/az.py
index 3ccc594c..76c411ad 100644
--- a/bitbake/lib/bb/fetch2/az.py
+++ b/bitbake/lib/bb/fetch/az.py
@@ -5,7 +5,7 @@ BitBake 'Fetch' Azure Storage implementation
# Copyright (C) 2021 Alejandro Hernandez Samaniego
#
-# Based on bb.fetch2.wget:
+# Based on bb.fetch.wget:
# Copyright (C) 2003, 2004 Chris Larson
#
# SPDX-License-Identifier: GPL-2.0-only
@@ -15,9 +15,9 @@ BitBake 'Fetch' Azure Storage implementation
import shlex
import os
import bb
-from bb.fetch2 import FetchError
-from bb.fetch2 import logger
-from bb.fetch2.wget import Wget
+from bb.fetch import FetchError
+from bb.fetch import logger
+from bb.fetch.wget import Wget
class Az(Wget):
@@ -36,6 +36,8 @@ class Az(Wget):
az_sas = d.getVar('AZ_SAS')
if az_sas and az_sas not in ud.url:
+ if not az_sas.startswith('?'):
+ raise FetchError("When using AZ_SAS, it must start with a '?' character to mark the start of the query-parameters.")
ud.url += az_sas
return Wget.checkstatus(self, fetch, ud, d, try_again)
@@ -46,31 +48,34 @@ class Az(Wget):
# If were reaching the account transaction limit we might be refused a connection,
# retrying allows us to avoid false negatives since the limit changes over time
- fetchcmd = self.basecmd + ' --retry-connrefused --waitretry=5'
+ fetchcmd = self.basecmd + ['--retry-connrefused', '--waitretry=5']
# We need to provide a localpath to avoid wget using the SAS
# ud.localfile either has the downloadfilename or ud.path
localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
bb.utils.mkdirhier(os.path.dirname(localpath))
- fetchcmd += " -O %s" % shlex.quote(localpath)
+ fetchcmd += ["-O", localpath]
if ud.user and ud.pswd:
- fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
+ fetchcmd += ["--user=%s" % ud.user, "--password=%s" % ud.pswd, "--auth-no-challenge"]
# Check if a Shared Access Signature was given and use it
az_sas = d.getVar('AZ_SAS')
if az_sas:
+ if not az_sas.startswith('?'):
+ raise FetchError("When using AZ_SAS, it must start with a '?' character to mark the start of the query-parameters.")
azuri = '%s%s%s%s' % ('https://', ud.host, ud.path, az_sas)
else:
azuri = '%s%s%s' % ('https://', ud.host, ud.path)
+ dldir = d.getVar("DL_DIR")
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again.
- fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % azuri)
+ fetchcmd += ["-c", "-P", dldir, azuri]
else:
- fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % azuri)
+ fetchcmd += ["-P", dldir, azuri]
try:
self._runwget(ud, d, fetchcmd, False)
diff --git a/bitbake/lib/bb/fetch2/clearcase.py b/bitbake/lib/bb/fetch/clearcase.py
similarity index 84%
rename from bitbake/lib/bb/fetch2/clearcase.py
rename to bitbake/lib/bb/fetch/clearcase.py
index 1a9c8637..54f34795 100644
--- a/bitbake/lib/bb/fetch2/clearcase.py
+++ b/bitbake/lib/bb/fetch/clearcase.py
@@ -49,14 +49,15 @@ User credentials:
#
import os
+import shlex
import shutil
import bb
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import ParameterError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import MissingParameterError
+from bb.fetch import ParameterError
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
class ClearCase(FetchMethod):
"""Class to fetch urls via 'clearcase'"""
@@ -94,7 +95,7 @@ class ClearCase(FetchMethod):
else:
ud.module = ""
- ud.basecmd = d.getVar("FETCHCMD_ccrc") or "/usr/bin/env cleartool || rcleartool"
+ ud.fallbackcmd = shlex.split(d.getVar("FETCHCMD_ccrc") or "")
if d.getVar("SRCREV") == "INVALID":
raise FetchError("Set a valid SRCREV for the clearcase fetcher in your recipe, e.g. SRCREV = \"/main/LATEST\" or any other label of your choice.")
@@ -108,7 +109,7 @@ class ClearCase(FetchMethod):
ud.module.replace("/", "."),
ud.label.replace("/", "."))
- ud.viewname = "%s-view%s" % (ud.identifier, d.getVar("DATETIME", d, True))
+ ud.viewname = "%s-view%s" % (ud.identifier, d.getVar("DATETIME"))
ud.csname = "%s-config-spec" % (ud.identifier)
ud.ccasedir = os.path.join(d.getVar("DL_DIR"), ud.type)
ud.viewdir = os.path.join(ud.ccasedir, ud.viewname)
@@ -122,7 +123,6 @@ class ClearCase(FetchMethod):
self.debug("type = %s" % ud.type)
self.debug("vob = %s" % ud.vob)
self.debug("module = %s" % ud.module)
- self.debug("basecmd = %s" % ud.basecmd)
self.debug("label = %s" % ud.label)
self.debug("ccasedir = %s" % ud.ccasedir)
self.debug("viewdir = %s" % ud.viewdir)
@@ -130,41 +130,40 @@ class ClearCase(FetchMethod):
self.debug("configspecfile = %s" % ud.configspecfile)
self.debug("localfile = %s" % ud.localfile)
- ud.localfile = os.path.join(d.getVar("DL_DIR"), ud.localfile)
-
def _build_ccase_command(self, ud, command):
"""
Build up a commandline based on ud
command is: mkview, setcs, rmview
"""
- options = []
-
- if "rcleartool" in ud.basecmd:
- options.append("-server %s" % ud.server)
+ cmd = [shutil.which("cleartool"), command]
+ if cmd[0] is None:
+ cmd = [shutil.which("rcleartool"), command]
+ if cmd[0] is None:
+ cmd = ud.fallbackcmd + [command]
- basecmd = "%s %s" % (ud.basecmd, command)
+ if cmd[0].endswith("rcleartool"):
+ cmd += ['-server', ud.server]
if command == 'mkview':
- if not "rcleartool" in ud.basecmd:
+ if not cmd[0].endswith("rcleartool"):
# Cleartool needs a -snapshot view
- options.append("-snapshot")
- options.append("-tag %s" % ud.viewname)
- options.append(ud.viewdir)
+ cmd.append("-snapshot")
+ cmd += ["-tag", ud.viewname]
+ cmd.append(ud.viewdir)
elif command == 'rmview':
- options.append("-force")
- options.append("%s" % ud.viewdir)
+ cmd.append("-force")
+ cmd.append(ud.viewdir)
elif command == 'setcs':
- options.append("-overwrite")
- options.append(ud.configspecfile)
+ cmd.append("-overwrite")
+ cmd.append(ud.configspecfile)
else:
raise FetchError("Invalid ccase command %s" % command)
- ccasecmd = "%s %s" % (basecmd, " ".join(options))
- self.debug("ccasecmd = %s" % ccasecmd)
- return ccasecmd
+ self.debug("ccasecmd = %s" % cmd)
+ return cmd
def _write_configspec(self, ud, d):
"""
@@ -190,13 +189,13 @@ class ClearCase(FetchMethod):
if os.path.exists(ud.viewdir):
cmd = self._build_ccase_command(ud, 'rmview');
logger.info("cleaning up [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ bb.fetch.check_network_access(d, cmd, ud.url)
output = runfetchcmd(cmd, d, workdir=ud.ccasedir)
logger.info("rmview output: %s", output)
def need_update(self, ud, d):
if ("LATEST" in ud.label) or (ud.customspec and "LATEST" in ud.customspec):
- ud.identifier += "-%s" % d.getVar("DATETIME",d, True)
+ ud.identifier += "-%s" % d.getVar("DATETIME")
return True
if os.path.exists(ud.localpath):
return False
@@ -216,7 +215,7 @@ class ClearCase(FetchMethod):
self._write_configspec(ud, d)
cmd = self._build_ccase_command(ud, 'mkview')
logger.info("creating view [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ bb.fetch.check_network_access(d, cmd, ud.url)
try:
runfetchcmd(cmd, d)
except FetchError as e:
@@ -228,7 +227,7 @@ class ClearCase(FetchMethod):
# Set configspec: Setting the configspec effectively fetches the files as defined in the configspec
cmd = self._build_ccase_command(ud, 'setcs');
logger.info("fetching data [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ bb.fetch.check_network_access(d, cmd, ud.url)
output = runfetchcmd(cmd, d, workdir=ud.viewdir)
logger.info("%s", output)
@@ -237,7 +236,7 @@ class ClearCase(FetchMethod):
# Clean clearcase meta-data before tar
- runfetchcmd('tar -czf "%s" .' % (ud.localpath), d, cleanup = [ud.localpath], workdir = ud.viewdir)
+ runfetchcmd(['tar', '-czf', ud.localpath, '.'], d, cleanup = [ud.localpath], workdir = ud.viewdir)
# Clean up so we can create a new view next time
self.clean(ud, d);
diff --git a/bitbake/lib/bb/fetch/crate.py b/bitbake/lib/bb/fetch/crate.py
new file mode 100644
index 00000000..16ea02d8
--- /dev/null
+++ b/bitbake/lib/bb/fetch/crate.py
@@ -0,0 +1,198 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementation for cargo registries, e.g. crates.io
+
+SRC_URI = "crate://some.host/crate_name/version;OptionA=xxx;OptionB=xxx;..."
+
+Supported SRC_URI options are:
+
+- protocol
+ The method to use to download the crate file. Common options are "http",
+ "https", and "ftp". The default is "https".
+
+"""
+
+# Copyright (C) 2016 Doug Goldstein
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import hashlib
+import json
+import os
+import subprocess
+import re
+from functools import cmp_to_key
+import bb
+from bb.fetch import logger, subprocess_setup, UnpackError, runfetchcmd
+from bb.fetch.wget import Wget
+
+
+class Crate(Wget):
+
+ """Class to fetch crates via wget"""
+
+ def _cargo_bitbake_path(self, rootdir):
+ return os.path.join(rootdir, "cargo_home", "bitbake")
+
+ def supports(self, ud, d):
+ """
+ Check to see if a given url is for this fetcher
+ """
+ return ud.type in ['crate']
+
+ def recommends_checksum(self, urldata):
+ return True
+
+ def urldata_init(self, ud, d):
+ """
+ Sets up to download the respective crate from crates.io
+ """
+
+ if ud.type == 'crate':
+ self._crate_urldata_init(ud, d)
+
+ super(Crate, self).urldata_init(ud, d)
+
+ def _generate_index_path(self, name):
+ # https://doc.rust-lang.org/cargo/reference/registry-index.html#index-files
+ if len(name) == 1:
+ return f"1/{name}"
+ elif len(name) == 2:
+ return f"2/{name}"
+ elif len(name) == 3:
+ return f"3/{name[0]}/{name}"
+ else:
+ return f"{name[0:2]}/{name[2:4]}/{name}"
+
+ def _crate_urldata_init(self, ud, d):
+ """
+ Sets up the download for a crate
+ """
+
+ # URL syntax is: crate://HOST/NAME/VERSION
+ # break the URL apart by /
+ parts = ud.url.split('/')
+ if len(parts) < 5:
+ raise bb.fetch.ParameterError("Invalid URL: Must be crate://HOST/NAME/VERSION", ud.url)
+
+ # version is expected to be the last token
+ # but ignore possible url parameters which will be used
+ # by the top fetcher class
+ version = parts[-1].split(";")[0]
+ # second to last field is name
+ name = parts[-2]
+ # host (this is to allow custom crate registries to be specified
+ host = '/'.join(parts[2:-2])
+
+ if 'protocol' in ud.parm:
+ proto = ud.parm['protocol']
+ else:
+ proto = 'https'
+
+ # If using crates.io use the CDN directly as per https://crates.io/data-access
+ if host == 'crates.io':
+ if proto != 'https':
+ bb.warn("URL: %s does the %s protocol which is not supported by crates.io. Please change to ;protocol=https in the url." % (ud.url, proto))
+ ud.url = "https://static.crates.io/crates/%s/%s/download" % (name, version)
+ ud.versionsurl = 'https://index.crates.io/' + self._generate_index_path(name)
+ else:
+ ud.url = "%s://%s/%s/%s/download" % (proto, host, name, version)
+ ud.versionsurl = "%s://%s/%s/versions" % (proto, host, name)
+
+ ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
+ if 'name' not in ud.parm:
+ ud.parm['name'] = '%s-%s' % (name, version)
+
+ logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
+
+ def unpack(self, ud, rootdir, d):
+ """
+ Uses the crate to build the necessary paths for cargo to utilize it
+ """
+ if ud.type == 'crate':
+ return self._crate_unpack(ud, rootdir, d)
+ else:
+ super(Crate, self).unpack(ud, rootdir, d)
+
+ def _crate_unpack(self, ud, rootdir, d):
+ """
+ Unpacks a crate
+ """
+ thefile = ud.localpath
+
+ # possible metadata we need to write out
+ metadata = {}
+
+ bp = d.getVar('BP')
+ if bp == ud.parm.get('name'):
+ cmd = ['tar', '-xz', '--no-same-owner', '-f', thefile]
+ ud.unpack_tracer.unpack("crate-extract", rootdir)
+ else:
+ cargo_bitbake = self._cargo_bitbake_path(rootdir)
+ ud.unpack_tracer.unpack("cargo-extract", cargo_bitbake)
+
+ cmd = ['tar', '-xz', '--no-same-owner', '-f', thefile, '-C', cargo_bitbake]
+
+ # ensure we've got these paths made
+ bb.utils.mkdirhier(cargo_bitbake)
+
+ # generate metadata necessary
+ with open(thefile, 'rb') as f:
+ # get the SHA256 of the original tarball
+ tarhash = hashlib.sha256(f.read()).hexdigest()
+
+ metadata['files'] = {}
+ metadata['package'] = tarhash
+
+ bb.note("Unpacking %s to %s/" % (thefile, os.getcwd()))
+ runfetchcmd(cmd, d, workdir=rootdir)
+
+ # if we have metadata to write out..
+ if len(metadata) > 0:
+ cratepath = os.path.splitext(os.path.basename(thefile))[0]
+ bbpath = self._cargo_bitbake_path(rootdir)
+ mdfile = '.cargo-checksum.json'
+ mdpath = os.path.join(bbpath, cratepath, mdfile)
+ with open(mdpath, "w") as f:
+ json.dump(metadata, f)
+
+ def latest_versionstring(self, ud, d, filter_regex=None):
+ """
+ Return the latest upstream version, dispatching to the appropriate
+ parser based on the versionsurl format.
+ """
+ if ud.versionsurl.startswith('https://index.crates.io/'):
+ return self._latest_versionstring_from_index(ud, d, filter_regex)
+ return self._latest_versionstring_from_api(ud, d, filter_regex)
+
+ def _latest_versionstring_from_api(self, ud, d, filter_regex=None):
+ """
+ Parse the latest version from a [name]/versions JSON API response.
+ """
+ json_data = json.loads(self._fetch_index(ud.versionsurl, ud, d))
+ versions = [(0, i["num"], "") for i in json_data["versions"]]
+ if filter_regex:
+ versions = [v for v in versions if re.match(filter_regex, v[1])]
+ versions = sorted(versions, key=cmp_to_key(bb.utils.vercmp))
+ return (versions[-1][1], "") if versions else ("", "")
+
+ def _latest_versionstring_from_index(self, ud, d, filter_regex=None):
+ """
+ Parse the latest version from a Cargo sparse index file (NDJSON).
+ https://doc.rust-lang.org/cargo/reference/registry-index.html#index-files
+ """
+ versions = []
+ response = self._fetch_index(ud.versionsurl, ud, d)
+ for line in response.splitlines():
+ data = json.loads(line)
+ if not data.get("yanked", False):
+ versions.append((0, data["vers"], ""))
+
+ if filter_regex:
+ versions = [v for v in versions if re.match(filter_regex, v[1])]
+
+ versions = sorted(versions, key=cmp_to_key(bb.utils.vercmp))
+ return (versions[-1][1], "") if versions else ("", "")
diff --git a/bitbake/lib/bb/fetch2/gcp.py b/bitbake/lib/bb/fetch/gcp.py
similarity index 62%
rename from bitbake/lib/bb/fetch2/gcp.py
rename to bitbake/lib/bb/fetch/gcp.py
index f40ce2ea..dd85b9fd 100644
--- a/bitbake/lib/bb/fetch2/gcp.py
+++ b/bitbake/lib/bb/fetch/gcp.py
@@ -10,7 +10,7 @@ Additionally, gsutil must also be installed.
# Copyright (C) 2023, Snap Inc.
#
-# Based in part on bb.fetch2.s3:
+# Based in part on bb.fetch.s3:
# Copyright (C) 2017 Andre McCurdy
#
# SPDX-License-Identifier: GPL-2.0-only
@@ -20,9 +20,9 @@ Additionally, gsutil must also be installed.
import os
import bb
import urllib.parse, urllib.error
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import logger
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import logger
class GCP(FetchMethod):
"""
@@ -46,8 +46,7 @@ class GCP(FetchMethod):
else:
ud.basename = os.path.basename(ud.path)
- ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
- ud.basecmd = "gsutil stat"
+ ud.localfile = ud.basename
def get_gcp_client(self):
from google.cloud import storage
@@ -58,17 +57,27 @@ class GCP(FetchMethod):
Fetch urls using the GCP API.
Assumes localpath was called first.
"""
+ from google.api_core.exceptions import GatewayTimeout, NotFound
logger.debug2(f"Trying to download gs://{ud.host}{ud.path} to {ud.localpath}")
if self.gcp_client is None:
self.get_gcp_client()
- bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}")
- runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d)
+ bb.fetch.check_network_access(d, "blob.download_to_filename", f"gs://{ud.host}{ud.path}")
# Path sometimes has leading slash, so strip it
path = ud.path.lstrip("/")
blob = self.gcp_client.bucket(ud.host).blob(path)
- blob.download_to_filename(ud.localpath)
+ try:
+ blob.download_to_filename(ud.localpath)
+ except NotFound:
+ raise FetchError("The GCP API threw a NotFound exception")
+ except GatewayTimeout as e:
+ # The GCS client already retries GatewayTimeout internally.
+ # Raise FetchError so mirror fallback can proceed.
+ logger.warning(
+ f"GCP API GatewayTimeout while downloading gs://{ud.host}{ud.path}: {e}"
+ )
+ raise FetchError(f"Transient GCP API GatewayTimeout for gs://{ud.host}{ud.path}")
# Additional sanity checks copied from the wget class (although there
# are no known issues which mean these are required, treat the GCP API
@@ -86,16 +95,28 @@ class GCP(FetchMethod):
"""
Check the status of a URL.
"""
+ from google.api_core.exceptions import GatewayTimeout
+
logger.debug2(f"Checking status of gs://{ud.host}{ud.path}")
if self.gcp_client is None:
self.get_gcp_client()
- bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}")
- runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d)
+ bb.fetch.check_network_access(d, "gcp_client.bucket(ud.host).blob(path).exists()", f"gs://{ud.host}{ud.path}")
# Path sometimes has leading slash, so strip it
path = ud.path.lstrip("/")
- if self.gcp_client.bucket(ud.host).blob(path).exists() == False:
+ try:
+ exists = self.gcp_client.bucket(ud.host).blob(path).exists()
+ except GatewayTimeout as e:
+ # The GCS client already retries GatewayTimeout internally.
+ # Surface a normal checkstatus failure and warn so the timeout
+ # is visible to operators.
+ logger.warning(
+ f"GCP API GatewayTimeout while checking gs://{ud.host}{ud.path}; treating as unavailable: {e}"
+ )
+ raise FetchError(f"Transient GCP API GatewayTimeout for gs://{ud.host}{ud.path}")
+
+ if exists == False:
raise FetchError(f"The GCP API reported that gs://{ud.host}{ud.path} does not exist")
else:
return True
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch/git.py
similarity index 52%
rename from bitbake/lib/bb/fetch2/git.py
rename to bitbake/lib/bb/fetch/git.py
index c7ff769f..dc1b31b5 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch/git.py
@@ -9,15 +9,6 @@ Supported SRC_URI options are:
- branch
The git branch to retrieve from. The default is "master"
- This option also supports multiple branch fetching, with branches
- separated by commas. In multiple branches case, the name option
- must have the same number of names to match the branches, which is
- used to specify the SRC_REV for the branch
- e.g:
- SRC_URI="git://some.host/somepath;branch=branchX,branchY;name=nameX,nameY"
- SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
- SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
-
- tag
The git tag to retrieve. The default is "master"
@@ -81,16 +72,17 @@ import shlex
import shutil
import subprocess
import tempfile
+import urllib
import bb
import bb.progress
from contextlib import contextmanager
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
-from bb.fetch2 import trusted_network
+from bb.fetch import FetchMethod
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
+from bb.fetch import trusted_network
-sha1_re = re.compile(r'^[0-9a-f]{40}$')
+git_hash_re = re.compile(r'^[0-9a-f]{40,64}$')
slash_re = re.compile(r"/+")
class GitProgressHandler(bb.progress.LineFilterProgressHandler):
@@ -135,7 +127,6 @@ class GitProgressHandler(bb.progress.LineFilterProgressHandler):
class Git(FetchMethod):
bitbake_dir = os.path.abspath(os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), '..', '..', '..'))
- make_shallow_path = os.path.join(bitbake_dir, 'bin', 'git-make-shallow')
"""Class to fetch a module or modules from git repositories"""
def init(self, d):
@@ -171,7 +162,7 @@ class Git(FetchMethod):
bb.warn("URL: %s uses git protocol which is no longer supported by github. Please change to ;protocol=https in the url." % ud.url)
if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
- raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
+ raise bb.fetch.ParameterError(f"Invalid protocol type: '{ud.proto}'", ud.url)
ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
@@ -183,74 +174,69 @@ class Git(FetchMethod):
ud.usehead = ud.parm.get("usehead","0") == "1"
if ud.usehead:
if ud.proto != "file":
- raise bb.fetch2.ParameterError("The usehead option is only for use with local ('protocol=file') git repositories", ud.url)
+ raise bb.fetch.ParameterError("The usehead option is only for use with local ('protocol=file') git repositories", ud.url)
ud.nobranch = 1
# bareclone implies nocheckout
ud.bareclone = ud.parm.get("bareclone","0") == "1"
if ud.bareclone:
ud.nocheckout = 1
-
- ud.unresolvedrev = {}
- branches = ud.parm.get("branch", "").split(',')
- if branches == [""] and not ud.nobranch:
- bb.warn("URL: %s does not set any branch parameter. The future default branch used by tools and repositories is uncertain and we will therefore soon require this is set in all git urls." % ud.url)
- branches = ["master"]
- if len(branches) != len(ud.names):
- raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
+
+ ud.unresolvedrev = ""
+ ud.branch = ud.parm.get("branch", "")
+ if not ud.branch and not ud.nobranch:
+ raise bb.fetch.ParameterError("The url does not set any branch parameter or set nobranch=1.", ud.url)
ud.noshared = d.getVar("BB_GIT_NOSHARED") == "1"
- ud.cloneflags = "-n"
+ ud.cloneflags = ["-n"]
if not ud.noshared:
- ud.cloneflags += " -s"
+ ud.cloneflags.append("-s")
if ud.bareclone:
- ud.cloneflags += " --mirror"
+ ud.cloneflags.append("--mirror")
+ ud.shallow_skip_fast = False
ud.shallow = d.getVar("BB_GIT_SHALLOW") == "1"
ud.shallow_extra_refs = (d.getVar("BB_GIT_SHALLOW_EXTRA_REFS") or "").split()
+ if 'tag' in ud.parm:
+ ud.shallow_extra_refs.append("refs/tags/" + ud.parm['tag'])
depth_default = d.getVar("BB_GIT_SHALLOW_DEPTH")
if depth_default is not None:
try:
depth_default = int(depth_default or 0)
except ValueError:
- raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH: %s" % depth_default)
+ raise bb.fetch.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH: %s" % depth_default)
else:
if depth_default < 0:
- raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH: %s" % depth_default)
+ raise bb.fetch.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH: %s" % depth_default)
else:
depth_default = 1
ud.shallow_depths = collections.defaultdict(lambda: depth_default)
revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
ud.shallow_revs = []
- ud.branches = {}
- for pos, name in enumerate(ud.names):
- branch = branches[pos]
- ud.branches[name] = branch
- ud.unresolvedrev[name] = branch
-
- shallow_depth = d.getVar("BB_GIT_SHALLOW_DEPTH_%s" % name)
- if shallow_depth is not None:
- try:
- shallow_depth = int(shallow_depth or 0)
- except ValueError:
- raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (name, shallow_depth))
- else:
- if shallow_depth < 0:
- raise bb.fetch2.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (name, shallow_depth))
- ud.shallow_depths[name] = shallow_depth
-
- revs = d.getVar("BB_GIT_SHALLOW_REVS_%s" % name)
- if revs is not None:
- ud.shallow_revs.extend(revs.split())
- elif revs_default is not None:
- ud.shallow_revs.extend(revs_default.split())
-
- if (ud.shallow and
- not ud.shallow_revs and
- all(ud.shallow_depths[n] == 0 for n in ud.names)):
+
+ ud.unresolvedrev = ud.branch
+
+ shallow_depth = d.getVar("BB_GIT_SHALLOW_DEPTH_%s" % ud.name)
+ if shallow_depth is not None:
+ try:
+ shallow_depth = int(shallow_depth or 0)
+ except ValueError:
+ raise bb.fetch.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (ud.name, shallow_depth))
+ else:
+ if shallow_depth < 0:
+ raise bb.fetch.FetchError("Invalid depth for BB_GIT_SHALLOW_DEPTH_%s: %s" % (ud.name, shallow_depth))
+ ud.shallow_depths[ud.name] = shallow_depth
+
+ revs = d.getVar("BB_GIT_SHALLOW_REVS_%s" % ud.name)
+ if revs is not None:
+ ud.shallow_revs.extend(revs.split())
+ elif revs_default is not None:
+ ud.shallow_revs.extend(revs_default.split())
+
+ if ud.shallow and not ud.shallow_revs and ud.shallow_depths[ud.name] == 0:
# Shallow disabled for this URL
ud.shallow = False
@@ -259,10 +245,11 @@ class Git(FetchMethod):
# rev of this repository. This will get resolved into a revision
# later. If an actual revision happens to have also been provided
# then this setting will be overridden.
- for name in ud.names:
- ud.unresolvedrev[name] = 'HEAD'
+ ud.unresolvedrev = 'HEAD'
+
+ ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin"
- ud.basecmd = d.getVar("FETCHCMD_git") or "git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all"
+ ud.basecmd = shlex.split(ud.basecmd)
write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS") or "0"
ud.write_tarballs = write_tarballs != "0" or ud.rebaseable
@@ -270,12 +257,11 @@ class Git(FetchMethod):
ud.setup_revisions(d)
- for name in ud.names:
- # Ensure any revision that doesn't look like a SHA-1 is translated into one
- if not sha1_re.match(ud.revisions[name] or ''):
- if ud.revisions[name]:
- ud.unresolvedrev[name] = ud.revisions[name]
- ud.revisions[name] = self.latest_revision(ud, d, name)
+ # Ensure any revision that doesn't look like a git hash (SHA-1 or SHA-256) is resolved
+ if not git_hash_re.match(ud.revision or ''):
+ if ud.revision:
+ ud.unresolvedrev = ud.revision
+ ud.revision = self.latest_revision(ud, d, ud.name)
gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
if gitsrcname.startswith('.'):
@@ -286,8 +272,7 @@ class Git(FetchMethod):
# upstream repo in the future, the mirror will remain intact and still
# contain the revision
if ud.rebaseable:
- for name in ud.names:
- gitsrcname = gitsrcname + '_' + ud.revisions[name]
+ gitsrcname = gitsrcname + '_' + ud.revision
dl_dir = d.getVar("DL_DIR")
gitdir = d.getVar("GITDIR") or (dl_dir + "/git2")
@@ -305,15 +290,14 @@ class Git(FetchMethod):
if ud.shallow_revs:
tarballname = "%s_%s" % (tarballname, "_".join(sorted(ud.shallow_revs)))
- for name, revision in sorted(ud.revisions.items()):
- tarballname = "%s_%s" % (tarballname, ud.revisions[name][:7])
- depth = ud.shallow_depths[name]
- if depth:
- tarballname = "%s-%s" % (tarballname, depth)
+ tarballname = "%s_%s" % (tarballname, ud.revision[:7])
+ depth = ud.shallow_depths[ud.name]
+ if depth:
+ tarballname = "%s-%s" % (tarballname, depth)
shallow_refs = []
if not ud.nobranch:
- shallow_refs.extend(ud.branches.values())
+ shallow_refs.append(ud.branch)
if ud.shallow_extra_refs:
shallow_refs.extend(r.replace('refs/heads/', '').replace('*', 'ALL') for r in ud.shallow_extra_refs)
if shallow_refs:
@@ -338,25 +322,28 @@ class Git(FetchMethod):
return True
if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d):
return True
- for name in ud.names:
- if not self._contains_ref(ud, d, name, ud.clonedir):
- return True
+ if not self._contains_ref(ud, d, ud.name, ud.clonedir):
+ return True
+ if 'tag' in ud.parm and not self._contains_ref(ud, d, ud.name, ud.clonedir, tag=True):
+ return True
return False
def lfs_need_update(self, ud, d):
+ if not self._need_lfs(ud):
+ return False
+
if self.clonedir_need_update(ud, d):
return True
- for name in ud.names:
- if not self._lfs_objects_downloaded(ud, d, name, ud.clonedir):
- return True
+ if not self._lfs_objects_downloaded(ud, d, ud.clonedir):
+ return True
return False
def clonedir_need_shallow_revs(self, ud, d):
for rev in ud.shallow_revs:
try:
- runfetchcmd('%s rev-parse -q --verify %s' % (ud.basecmd, rev), d, quiet=True, workdir=ud.clonedir)
- except bb.fetch2.FetchError:
+ runfetchcmd(ud.basecmd + ['rev-parse', '-q', '--verify', rev], d, quiet=True, workdir=ud.clonedir)
+ except bb.fetch.FetchError:
return rev
return None
@@ -366,6 +353,13 @@ class Git(FetchMethod):
def tarball_need_update(self, ud):
return ud.write_tarballs and not os.path.exists(ud.fullmirror)
+ def update_mirror_links(self, ud, origud):
+ super().update_mirror_links(ud, origud)
+ # When using shallow mode, add a symlink to the original fullshallow
+ # path to ensure a valid symlink even in the `PREMIRRORS` case
+ if origud.shallow and not os.path.exists(origud.fullshallow):
+ self.ensure_symlink(ud.localpath, origud.fullshallow)
+
def try_premirror(self, ud, d):
# If we don't do this, updating an existing checkout with only premirrors
# is not possible
@@ -397,16 +391,16 @@ class Git(FetchMethod):
elif os.path.exists(ud.fullmirror) and self.need_update(ud, d):
if not os.path.exists(ud.clonedir):
bb.utils.mkdirhier(ud.clonedir)
- runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
+ runfetchcmd(['tar', '-xzf', ud.fullmirror], d, workdir=ud.clonedir)
else:
- tmpdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR'))
- runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=tmpdir)
- output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
- if 'mirror' in output:
- runfetchcmd("%s remote rm mirror" % ud.basecmd, d, workdir=ud.clonedir)
- runfetchcmd("%s remote add --mirror=fetch mirror %s" % (ud.basecmd, tmpdir), d, workdir=ud.clonedir)
- fetch_cmd = "LANG=C %s fetch -f --update-head-ok --progress mirror " % (ud.basecmd)
- runfetchcmd(fetch_cmd, d, workdir=ud.clonedir)
+ with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
+ runfetchcmd(['tar', '-xzf', ud.fullmirror], d, workdir=tmpdir)
+ output = runfetchcmd(ud.basecmd + ['remote'], d, quiet=True, workdir=ud.clonedir)
+ if 'mirror' in output:
+ runfetchcmd(ud.basecmd + ['remote', 'rm', 'mirror'], d, workdir=ud.clonedir)
+ runfetchcmd(ud.basecmd + ['remote', 'add', '--mirror=fetch', 'mirror', tmpdir], d, workdir=ud.clonedir)
+ fetch_cmd = ud.basecmd + ['fetch', '-f', '--update-head-ok', '--progress', 'mirror']
+ runfetchcmd(fetch_cmd, d, workdir=ud.clonedir, extraenv={'LANG':'C'})
repourl = self._get_repo_url(ud)
needs_clone = False
@@ -415,13 +409,13 @@ class Git(FetchMethod):
# repository in which case it needs to be deleted and re-cloned.
try:
# Since clones can be bare, use --absolute-git-dir instead of --show-toplevel
- output = runfetchcmd("LANG=C %s rev-parse --absolute-git-dir" % ud.basecmd, d, workdir=ud.clonedir)
+ output = runfetchcmd(ud.basecmd + ['rev-parse', '--absolute-git-dir'], d, workdir=ud.clonedir, extraenv={'LANG':'C'})
toplevel = output.rstrip()
if not bb.utils.path_is_descendant(toplevel, ud.clonedir):
logger.warning("Top level directory '%s' is not a descendant of '%s'. Re-cloning", toplevel, ud.clonedir)
needs_clone = True
- except bb.fetch2.FetchError as e:
+ except bb.fetch.FetchError as e:
logger.warning("Unable to get top level for %s (not a git directory?): %s", ud.clonedir, e)
needs_clone = True
except FileNotFoundError as e:
@@ -442,177 +436,234 @@ class Git(FetchMethod):
objects = os.path.join(repourl_path, 'objects')
if os.path.isdir(objects) and not os.path.islink(objects):
repourl = repourl_path
- clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
+ clone_cmd = ud.basecmd + ['clone', '--bare', '--mirror', repourl, ud.clonedir, '--progress']
if ud.proto.lower() != 'file':
- bb.fetch2.check_network_access(d, clone_cmd, ud.url)
+ bb.fetch.check_network_access(d, clone_cmd, ud.url)
progresshandler = GitProgressHandler(d)
- runfetchcmd(clone_cmd, d, log=progresshandler)
+
+ # Try creating a fast initial shallow clone
+ # Enabling ud.shallow_skip_fast will skip this
+ # If the Git error "Server does not allow request for unadvertised object"
+ # occurs, shallow_skip_fast is enabled automatically.
+ # This may happen if the Git server does not allow the request
+ # or if the Git client has issues with this functionality.
+ if ud.shallow and not ud.shallow_skip_fast:
+ try:
+ self.clone_shallow_with_tarball(ud, d)
+ # When the shallow clone has succeeded, use the shallow tarball
+ ud.localpath = ud.fullshallow
+ return
+ except:
+ logger.warning("Creating fast initial shallow clone failed, try initial regular clone now.")
+
+ # When skipping fast initial shallow or the fast inital shallow clone failed:
+ # Try again with an initial regular clone
+ runfetchcmd(clone_cmd, d, log=progresshandler, extraenv={'LANG':'C'})
# Update the checkout if needed
if self.clonedir_need_update(ud, d):
- output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
+ output = runfetchcmd(ud.basecmd + ['remote'], d, quiet=True, workdir=ud.clonedir)
if "origin" in output:
- runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
+ runfetchcmd(ud.basecmd + ['remote', 'rm', 'origin'], d, workdir=ud.clonedir)
- runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
+ runfetchcmd(ud.basecmd + ['remote', 'add', '--mirror=fetch', 'origin', repourl], d, workdir=ud.clonedir)
if ud.nobranch:
- fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+ fetch_cmd = ud.basecmd + ['fetch', '-f', '--progress', repourl, 'refs/*:refs/*']
else:
- fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
+ fetch_cmd = ud.basecmd + ['fetch', '-f', '--progress', repourl, 'refs/heads/*:refs/heads/*', 'refs/tags/*:refs/tags/*']
if ud.proto.lower() != 'file':
- bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
+ bb.fetch.check_network_access(d, fetch_cmd, ud.url)
progresshandler = GitProgressHandler(d)
- runfetchcmd(fetch_cmd, d, log=progresshandler, workdir=ud.clonedir)
- runfetchcmd("%s prune-packed" % ud.basecmd, d, workdir=ud.clonedir)
- runfetchcmd("%s pack-refs --all" % ud.basecmd, d, workdir=ud.clonedir)
- runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d, workdir=ud.clonedir)
+ runfetchcmd(fetch_cmd, d, log=progresshandler, workdir=ud.clonedir, extraenv={'LANG':'C'})
+ runfetchcmd(ud.basecmd + ['repack','-adk'], d, workdir=ud.clonedir)
+ runfetchcmd(ud.basecmd + ['pack-refs','--all'], d, workdir=ud.clonedir)
+ runfetchcmd(ud.basecmd + ['prune-packed'], d, workdir=ud.clonedir)
try:
os.unlink(ud.fullmirror)
except OSError as exc:
if exc.errno != errno.ENOENT:
raise
- for name in ud.names:
- if not self._contains_ref(ud, d, name, ud.clonedir):
- raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name]))
+ if not self._contains_ref(ud, d, ud.name, ud.clonedir):
+ raise bb.fetch.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revision, ud.branch))
if ud.shallow and ud.write_shallow_tarballs:
missing_rev = self.clonedir_need_shallow_revs(ud, d)
if missing_rev:
- raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev)
+ raise bb.fetch.FetchError("Unable to find revision %s even from upstream" % missing_rev)
if self.lfs_need_update(ud, d):
- # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching
- # of all LFS blobs needed at the srcrev.
- #
- # It would be nice to just do this inline here by running 'git-lfs fetch'
- # on the bare clonedir, but that operation requires a working copy on some
- # releases of Git LFS.
- with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
- # Do the checkout. This implicitly involves a Git LFS fetch.
- Git.unpack(self, ud, tmpdir, d)
-
- # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
- # the bare clonedir.
- #
- # As this procedure is invoked repeatedly on incremental fetches as
- # a recipe's SRCREV is bumped throughout its lifetime, this will
- # result in a gradual accumulation of LFS blobs in <ud.clonedir>/lfs
- # corresponding to all the blobs reachable from the different revs
- # fetched across time.
- #
- # Only do this if the unpack resulted in a .git/lfs directory being
- # created; this only happens if at least one blob needed to be
- # downloaded.
- if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")):
- runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir)
+ self.lfs_fetch(ud, d, ud.clonedir, ud.revision)
+
+ def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None):
+ """Helper method for fetching Git LFS data"""
+ if self._need_lfs(ud) and self._contains_lfs(ud, d, clonedir) and len(revision):
+ self._ensure_git_lfs(d, ud)
+
+ # Using worktree with the revision because .lfsconfig may exists
+ worktree_add_cmd = ud.basecmd + ['worktree', 'add', 'wt', revision]
+ # Disable LFS smudging on worktree creation to make it easier
+ # to understand which operation actually failed.
+ runfetchcmd(worktree_add_cmd, d, log=progresshandler, workdir=clonedir, extraenv={'GIT_LFS_SKIP_SMUDGE':'1'})
+ lfs_fetch_cmd = ud.basecmd + ['lfs', 'fetch']
+ if fetchall:
+ lfs_fetch_cmd.append('--all')
+ runfetchcmd(lfs_fetch_cmd, d, log=progresshandler, workdir=(clonedir + "/wt"))
+ worktree_rem_cmd = ud.basecmd + ['worktree', 'remove', '-f', 'wt']
+ runfetchcmd(worktree_rem_cmd, d, log=progresshandler, workdir=clonedir)
+
+
+ @contextmanager
+ def create_atomic(self, filename):
+ """Create as a temp file and move atomically into position to avoid races"""
+ fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename))
+ try:
+ yield tfile
+ umask = os.umask(0o666)
+ os.umask(umask)
+ os.chmod(tfile, (0o666 & ~umask))
+ os.rename(tfile, filename)
+ finally:
+ os.close(fd)
def build_mirror_data(self, ud, d):
-
- # Create as a temp file and move atomically into position to avoid races
- @contextmanager
- def create_atomic(filename):
- fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename))
- try:
- yield tfile
- umask = os.umask(0o666)
- os.umask(umask)
- os.chmod(tfile, (0o666 & ~umask))
- os.rename(tfile, filename)
- finally:
- os.close(fd)
-
if ud.shallow and ud.write_shallow_tarballs:
if not os.path.exists(ud.fullshallow):
if os.path.islink(ud.fullshallow):
os.unlink(ud.fullshallow)
- tempdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR'))
- shallowclone = os.path.join(tempdir, 'git')
- try:
- self.clone_shallow_local(ud, shallowclone, d)
-
- logger.info("Creating tarball of git repository")
- with create_atomic(ud.fullshallow) as tfile:
- runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone)
- runfetchcmd("touch %s.done" % ud.fullshallow, d)
- finally:
- bb.utils.remove(tempdir, recurse=True)
+ self.clone_shallow_with_tarball(ud, d)
elif ud.write_tarballs and not os.path.exists(ud.fullmirror):
if os.path.islink(ud.fullmirror):
os.unlink(ud.fullmirror)
logger.info("Creating tarball of git repository")
- with create_atomic(ud.fullmirror) as tfile:
- mtime = runfetchcmd("{} log --all -1 --format=%cD".format(ud.basecmd), d,
+ with self.create_atomic(ud.fullmirror) as tfile:
+ mtime = runfetchcmd(ud.basecmd + ['log', '--all', '-1', '--format=%cD'], d,
quiet=True, workdir=ud.clonedir)
- runfetchcmd("tar -czf %s --owner oe:0 --group oe:0 --mtime \"%s\" ."
- % (tfile, mtime), d, workdir=ud.clonedir)
- runfetchcmd("touch %s.done" % ud.fullmirror, d)
+ runfetchcmd(['tar', '-czf', tfile, '--owner', 'oe:0', '--group', 'oe:0', '--mtime', mtime, '.'], d, workdir=ud.clonedir)
+ runfetchcmd(['touch', "%s.done" % ud.fullmirror], d)
+
+ def clone_shallow_with_tarball(self, ud, d):
+ for fast in [True, False]:
+ ud.shallow_skip_fast = not fast
+ with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tempdir:
+ shallowclone = os.path.join(tempdir, 'git')
+ try:
+ self.clone_shallow_local(ud, shallowclone, d)
+ except:
+ if not fast:
+ raise
+ logger.warning("Fast shallow clone failed, try to skip fast mode now.")
+ continue
+ logger.info("Creating tarball of git repository")
+ with self.create_atomic(ud.fullshallow) as tfile:
+ runfetchcmd(['tar', '-czf', tfile, '.'], d, workdir=shallowclone)
+ runfetchcmd(['touch', '%s.done' % ud.fullshallow], d)
+ return True
+ return False
def clone_shallow_local(self, ud, dest, d):
- """Clone the repo and make it shallow.
+ """
+ Shallow fetch from ud.clonedir (${DL_DIR}/git2/<gitrepo> by default):
+ - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
+ - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev
+ """
+
+ progresshandler = GitProgressHandler(d)
+ repourl = self._get_repo_url(ud)
+ bb.utils.mkdirhier(dest)
+ init_cmd = ud.basecmd + ['init', '-q']
+ if ud.bareclone:
+ init_cmd.append('--bare')
+ runfetchcmd(init_cmd, d, workdir=dest)
+ # Use repourl when creating a fast initial shallow clone
+ # Prefer already existing full bare clones if available
+ if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir):
+ remote = repourl
+ else:
+ remote = ud.clonedir
+ runfetchcmd(ud.basecmd + ['remote', 'add', 'origin', remote], d, workdir=dest)
- The upstream url of the new clone isn't set at this time, as it'll be
- set correctly when unpacked."""
- runfetchcmd("%s clone %s %s %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, dest), d)
+ # Check the histories which should be excluded
+ shallow_exclude = []
+ for revision in ud.shallow_revs:
+ shallow_exclude.append("--shallow-exclude=%s" % revision)
- to_parse, shallow_branches = [], []
- for name in ud.names:
- revision = ud.revisions[name]
- depth = ud.shallow_depths[name]
- if depth:
- to_parse.append('%s~%d^{}' % (revision, depth - 1))
+ revision = ud.revision
+ depth = ud.shallow_depths[ud.name]
- # For nobranch, we need a ref, otherwise the commits will be
- # removed, and for non-nobranch, we truncate the branch to our
- # srcrev, to avoid keeping unnecessary history beyond that.
- branch = ud.branches[name]
- if ud.nobranch:
- ref = "refs/shallow/%s" % name
- elif ud.bareclone:
- ref = "refs/heads/%s" % branch
- else:
- ref = "refs/remotes/origin/%s" % branch
+ # The --depth and --shallow-exclude can't be used together
+ if depth and shallow_exclude:
+ raise bb.fetch.FetchError("BB_GIT_SHALLOW_REVS is set, but BB_GIT_SHALLOW_DEPTH is not 0.")
+
+ # For nobranch, we need a ref, otherwise the commits will be
+ # removed, and for non-nobranch, we truncate the branch to our
+ # srcrev, to avoid keeping unnecessary history beyond that.
+ branch = ud.branch
+ if ud.nobranch:
+ ref = "refs/shallow/%s" % ud.name
+ elif ud.bareclone:
+ ref = "refs/heads/%s" % branch
+ else:
+ ref = "refs/remotes/origin/%s" % branch
+
+ fetch_cmd = ud.basecmd + ['fetch', 'origin', revision]
+ if depth:
+ fetch_cmd += ['--depth', str(depth)]
- shallow_branches.append(ref)
- runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest)
+ if shallow_exclude:
+ fetch_cmd += shallow_exclude
- # Map srcrev+depths to revisions
- parsed_depths = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join(to_parse)), d, workdir=dest)
+ # Advertise the revision for lower version git such as 2.25.1:
+ # error: Server does not allow request for unadvertised object.
+ # The ud.clonedir is a local temporary dir, will be removed when
+ # fetch is done, so we can do anything on it.
+ adv_cmd = ['git', 'branch', '-f', 'advertise-' + revision, revision]
+ if ud.shallow_skip_fast:
+ runfetchcmd(adv_cmd, d, workdir=ud.clonedir)
- # Resolve specified revisions
- parsed_revs = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join('"%s^{}"' % r for r in ud.shallow_revs)), d, workdir=dest)
- shallow_revisions = parsed_depths.splitlines() + parsed_revs.splitlines()
+ runfetchcmd(fetch_cmd, d, workdir=dest)
+ runfetchcmd(ud.basecmd + ['update-ref', ref, revision], d, workdir=dest)
+ # Fetch Git LFS data
+ self.lfs_fetch(ud, d, dest, ud.revision)
# Apply extra ref wildcards
- all_refs = runfetchcmd('%s for-each-ref "--format=%%(refname)"' % ud.basecmd,
- d, workdir=dest).splitlines()
+ all_refs_remote = runfetchcmd(ud.basecmd + ['ls-remote', 'origin', 'refs/*'], \
+ d, workdir=dest).splitlines()
+ all_refs = []
+ for line in all_refs_remote:
+ all_refs.append(line.split()[-1])
+ extra_refs = []
for r in ud.shallow_extra_refs:
if not ud.bareclone:
r = r.replace('refs/heads/', 'refs/remotes/origin/')
if '*' in r:
matches = filter(lambda a: fnmatch.fnmatchcase(a, r), all_refs)
- shallow_branches.extend(matches)
+ extra_refs.extend(matches)
else:
- shallow_branches.append(r)
+ extra_refs.append(r)
+
+ for ref in extra_refs:
+ ref_fetch = ref.replace('refs/heads/', '').replace('refs/remotes/origin/', '').replace('refs/tags/', '')
+ runfetchcmd(ud.basecmd + ['fetch', 'origin', '--depth', '1', '--', ref_fetch], d, workdir=dest)
+ revision = runfetchcmd(ud.basecmd + ['rev-parse', 'FETCH_HEAD'], d, workdir=dest).strip()
+ runfetchcmd(ud.basecmd + ['update-ref', ref, revision], d, workdir=dest)
+
+ # The url is local ud.clonedir, set it to upstream one
+ runfetchcmd(ud.basecmd + ['remote', 'set-url', 'origin', repourl], d, workdir=dest)
- # Make the repository shallow
- shallow_cmd = [self.make_shallow_path, '-s']
- for b in shallow_branches:
- shallow_cmd.append('-r')
- shallow_cmd.append(b)
- shallow_cmd.extend(shallow_revisions)
- runfetchcmd(subprocess.list2cmdline(shallow_cmd), d, workdir=dest)
+ def unpack_update(self, ud, destdir, d):
+ return self.unpack(ud, destdir, d, update=True)
- def unpack(self, ud, destdir, d):
+ def unpack(self, ud, destdir, d, update=False):
""" unpack the downloaded src to destdir"""
subdir = ud.parm.get("subdir")
subpath = ud.parm.get("subpath")
readpathspec = ""
- def_destsuffix = "git/"
+ def_destsuffix = (d.getVar("BB_GIT_DEFAULT_DESTSUFFIX") or "git") + "/"
if subpath:
readpathspec = ":%s" % subpath
@@ -622,7 +673,7 @@ class Git(FetchMethod):
# If 'subdir' param exists, create a dir and use it as destination for unpack cmd
if os.path.isabs(subdir):
if not os.path.realpath(subdir).startswith(os.path.realpath(destdir)):
- raise bb.fetch2.UnpackError("subdir argument isn't a subdirectory of unpack root %s" % destdir, ud.url)
+ raise bb.fetch.UnpackError("subdir argument isn't a subdirectory of unpack root %s" % destdir, ud.url)
destdir = subdir
else:
destdir = os.path.join(destdir, subdir)
@@ -630,22 +681,27 @@ class Git(FetchMethod):
destsuffix = ud.parm.get("destsuffix", def_destsuffix)
destdir = ud.destdir = os.path.join(destdir, destsuffix)
- if os.path.exists(destdir):
+ if os.path.exists(destdir) and not update:
bb.utils.prunedir(destdir)
if not ud.bareclone:
ud.unpack_tracer.unpack("git", destdir)
need_lfs = self._need_lfs(ud)
+ extraenv = {}
if not need_lfs:
- ud.basecmd = "GIT_LFS_SKIP_SMUDGE=1 " + ud.basecmd
+ extraenv["GIT_LFS_SKIP_SMUDGE"] = "1"
source_found = False
+ update_mode = False
source_error = []
clonedir_is_up_to_date = not self.clonedir_need_update(ud, d)
if clonedir_is_up_to_date:
- runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, destdir), d)
+ if update and os.path.exists(destdir):
+ update_mode = True
+ else:
+ runfetchcmd(ud.basecmd + ['clone'] + ud.cloneflags + [ud.clonedir, destdir], d, extraenv=extraenv)
source_found = True
else:
source_error.append("clone directory not available or not up to date: " + ud.clonedir)
@@ -653,8 +709,11 @@ class Git(FetchMethod):
if not source_found:
if ud.shallow:
if os.path.exists(ud.fullshallow):
- bb.utils.mkdirhier(destdir)
- runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=destdir)
+ if update and os.path.exists(destdir):
+ update_mode = True
+ else:
+ bb.utils.mkdirhier(destdir)
+ runfetchcmd(['tar', '-xzf', ud.fullshallow], d, workdir=destdir)
source_found = True
else:
source_error.append("shallow clone not available: " + ud.fullshallow)
@@ -662,32 +721,80 @@ class Git(FetchMethod):
source_error.append("shallow clone not enabled")
if not source_found:
- raise bb.fetch2.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url)
+ raise bb.fetch.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url)
+
+ if update_mode:
+ if ud.shallow:
+ raise bb.fetch.UnpackError("Can't update shallow clones checkouts without network access, not supported.", ud.url)
+
+ output = runfetchcmd(ud.basecmd + ['status', '--untracked-files=no', '--porcelain'], d, workdir=destdir, extraenv=extraenv)
+ if output:
+ raise bb.fetch.LocalModificationsError(destdir, ud.url, output)
+
+ # Set up remote for the download location if it doesn't exist
+ try:
+ runfetchcmd(ud.basecmd + ['remote', 'get-url', 'dldir'], d, workdir=destdir)
+ except bb.fetch.FetchError:
+ if ud.clonedir:
+ runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', 'file://' + ud.clonedir], d, workdir=destdir)
+ try:
+ runfetchcmd(ud.basecmd + ['fetch', 'dldir'], d, workdir=destdir, extraenv=extraenv)
+ except bb.fetch.FetchError as e:
+ raise bb.fetch.UnpackError("Failed to fetch from dldir remote: %s" % str(e), ud.url)
+ try:
+ runfetchcmd(ud.basecmd + ['rebase', '--no-autosquash', '--no-autostash', ud.revision], d, workdir=destdir, extraenv=extraenv)
+ except bb.fetch.FetchError as e:
+ # If rebase failed, abort it
+ try:
+ runfetchcmd(ud.basecmd + ['rebase', '--abort'], d, workdir=destdir)
+ except Exception:
+ pass
+ raise bb.fetch.RebaseError(destdir, ud.url, str(e))
+ return True
+
+ # If there is a tag parameter in the url and we also have a fixed srcrev, check the tag
+ # matches the revision
+ if 'tag' in ud.parm and git_hash_re.match(ud.revision):
+ output = runfetchcmd(ud.basecmd + ['rev-list', '-n', '1', ud.parm['tag']], d, workdir=destdir, extraenv=extraenv)
+ output = output.strip()
+ if output != ud.revision:
+ # It is possible ud.revision is the revision on an annotated tag which won't match the output of rev-list
+ # If it resolves to the same thing there isn't a problem.
+ output2 = runfetchcmd(ud.basecmd + ['rev-list', '-n', '1', ud.revision], d, workdir=destdir)
+ output2 = output2.strip()
+ if output != output2:
+ raise bb.fetch.FetchError("The revision the git tag '%s' resolved to didn't match the SRCREV in use (%s vs %s)" % (ud.parm['tag'], output, ud.revision), ud.url)
repourl = self._get_repo_url(ud)
- runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir)
+ runfetchcmd(ud.basecmd + ['remote', 'set-url', 'origin', repourl], d, workdir=destdir)
+ if ud.clonedir:
+ try:
+ runfetchcmd(ud.basecmd + ['remote', 'get-url', 'dldir'], d, workdir=destdir)
+ except bb.fetch.FetchError:
+ runfetchcmd(ud.basecmd + ['remote', 'add', 'dldir', "file://" + ud.clonedir], d, workdir=destdir)
if self._contains_lfs(ud, d, destdir):
- if need_lfs and not self._find_git_lfs(d):
- raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
- elif not need_lfs:
+ if not need_lfs:
bb.note("Repository %s has LFS content but it is not being fetched" % (repourl))
else:
- runfetchcmd("%s lfs install --local" % ud.basecmd, d, workdir=destdir)
+ self._ensure_git_lfs(d, ud)
+
+ runfetchcmd(ud.basecmd + ['lfs', 'install', '--local'], d, workdir=destdir)
if not ud.nocheckout:
if subpath:
- runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d,
- workdir=destdir)
- runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d, workdir=destdir)
+ runfetchcmd(ud.basecmd + ['read-tree', ud.revision + readpathspec], d, workdir=destdir, extraenv=extraenv)
+ runfetchcmd(ud.basecmd + ['checkout-index', '-q', '-f', '-a'], d, workdir=destdir, extraenv=extraenv)
+ runfetchcmd(ud.basecmd + ['update-ref', '--no-deref', 'HEAD', ud.revision], d, workdir=destdir, extraenv=extraenv)
+ if not ud.nobranch:
+ branchname = ud.branch
+ runfetchcmd(ud.basecmd + ['update-ref', 'refs/heads/' + branchname, ud.revision], d, workdir=destdir, extraenv=extraenv)
elif not ud.nobranch:
- branchname = ud.branches[ud.names[0]]
- runfetchcmd("%s checkout -B %s %s" % (ud.basecmd, branchname, \
- ud.revisions[ud.names[0]]), d, workdir=destdir)
- runfetchcmd("%s branch %s --set-upstream-to origin/%s" % (ud.basecmd, branchname, \
- branchname), d, workdir=destdir)
+ branchname = ud.branch
+ runfetchcmd(ud.basecmd + ['checkout', '-B', branchname, ud.revision], d, workdir=destdir, extraenv=extraenv)
+ runfetchcmd(ud.basecmd + ['branch', branchname, '--set-upstream-to', 'origin/' + branchname], d, workdir=destdir, extraenv=extraenv)
else:
- runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=destdir)
+ runfetchcmd(ud.basecmd + ['checkout', ud.revision], d, workdir=destdir, extraenv=extraenv)
return True
@@ -701,43 +808,46 @@ class Git(FetchMethod):
clonedir = os.path.realpath(ud.localpath)
to_remove.append(clonedir)
+ # Remove shallow mirror tarball
+ if ud.shallow:
+ to_remove.append(ud.fullshallow)
+ to_remove.append(ud.fullshallow + ".done")
+
for r in to_remove:
- if os.path.exists(r):
+ if os.path.exists(r) or os.path.islink(r):
bb.note('Removing %s' % r)
bb.utils.remove(r, True)
def supports_srcrev(self):
return True
- def _contains_ref(self, ud, d, name, wd):
- cmd = ""
+ def _contains_ref(self, ud, d, name, wd, tag=False):
+ git_ref_name = 'refs/tags/%s' % ud.parm['tag'] if tag else ud.revision
+
if ud.nobranch:
- cmd = "%s log --pretty=oneline -n 1 %s -- 2> /dev/null | wc -l" % (
- ud.basecmd, ud.revisions[name])
+ cmd = ud.basecmd + ['log', '--pretty=oneline', '-n', '1', git_ref_name, "--"]
else:
- cmd = "%s branch --contains %s --list %s 2> /dev/null | wc -l" % (
- ud.basecmd, ud.revisions[name], ud.branches[name])
+ cmd = ud.basecmd + ['branch', '--contains', git_ref_name, '--list', ud.branch]
try:
- output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
- except bb.fetch2.FetchError:
+ output = runfetchcmd(cmd, d, workdir=wd)
+ except (bb.fetch.FetchError):
return False
- if len(output.split()) > 1:
- raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
- return output.split()[0] != "0"
+ return len(output.splitlines()) > 0
- def _lfs_objects_downloaded(self, ud, d, name, wd):
+ def _lfs_objects_downloaded(self, ud, d, wd):
"""
Verifies whether the LFS objects for requested revisions have already been downloaded
"""
# Bail out early if this repository doesn't use LFS
- if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd):
+ if not self._contains_lfs(ud, d, wd):
return True
+ self._ensure_git_lfs(d, ud)
+
# The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file
# existence.
# [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git
- cmd = "%s lfs ls-files -l %s" \
- % (ud.basecmd, ud.revisions[name])
+ cmd = ud.basecmd + ['lfs', 'ls-files', '-l', ud.revision]
output = runfetchcmd(cmd, d, quiet=True, workdir=wd).rstrip()
# Do not do any further matching if no objects are managed by LFS
if not output:
@@ -761,33 +871,23 @@ class Git(FetchMethod):
"""
Check if the repository has 'lfs' (large file) content
"""
-
- if ud.nobranch:
- # If no branch is specified, use the current git commit
- refname = self._build_revision(ud, d, ud.names[0])
- elif wd == ud.clonedir:
- # The bare clonedir doesn't use the remote names; it has the branch immediately.
- refname = ud.branches[ud.names[0]]
- else:
- refname = "origin/%s" % ud.branches[ud.names[0]]
-
- cmd = "%s grep lfs %s:.gitattributes | wc -l" % (
- ud.basecmd, refname)
-
+ cmd = ud.basecmd + ['grep', '^[^#].*lfs', ud.revision + ':.gitattributes']
try:
output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
- if int(output) > 0:
+ if len(output.splitlines()) > 0:
return True
- except (bb.fetch2.FetchError,ValueError):
+ except (bb.fetch.FetchError,ValueError):
pass
return False
- def _find_git_lfs(self, d):
+ def _ensure_git_lfs(self, d, ud):
"""
- Return True if git-lfs can be found, False otherwise.
+ Ensures that git-lfs is available, raising a FetchError if it isn't.
"""
- import shutil
- return shutil.which("git-lfs", path=d.getVar('PATH')) is not None
+ if shutil.which("git-lfs", path=d.getVar('PATH')) is None:
+ raise bb.fetch.FetchError(
+ "Repository %s has LFS content, either install git-lfs on the host to download it, "
+ "or ignore it by adding lfs=0 as a parameter to the URL" % self._get_repo_url(ud))
def _get_repo_url(self, ud):
"""
@@ -795,21 +895,21 @@ class Git(FetchMethod):
"""
# Note that we do not support passwords directly in the git urls. There are several
# reasons. SRC_URI can be written out to things like buildhistory and people don't
- # want to leak passwords like that. Its also all too easy to share metadata without
+ # want to leak passwords like that. Its also all too easy to share metadata without
# removing the password. ssh keys, ~/.netrc and ~/.ssh/config files can be used as
# alternatives so we will not take patches adding password support here.
if ud.user:
username = ud.user + '@'
else:
username = ""
- return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path)
+ return "%s://%s%s%s" % (ud.proto, username, ud.host, urllib.parse.quote(ud.path))
def _revision_key(self, ud, d, name):
"""
Return a unique key for the url
"""
# Collapse adjacent slashes
- return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev[name]
+ return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev
def _lsremote(self, ud, d, search):
"""
@@ -826,13 +926,14 @@ class Git(FetchMethod):
d.setVar('_BB_GIT_IN_LSREMOTE', '1')
try:
repourl = self._get_repo_url(ud)
- cmd = "%s ls-remote %s %s" % \
- (ud.basecmd, shlex.quote(repourl), search)
+ cmd = ud.basecmd + ['ls-remote', repourl]
+ if search:
+ cmd.append(search)
if ud.proto.lower() != 'file':
- bb.fetch2.check_network_access(d, cmd, repourl)
+ bb.fetch.check_network_access(d, cmd, repourl)
output = runfetchcmd(cmd, d, True)
if not output:
- raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+ raise bb.fetch.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
finally:
d.delVar('_BB_GIT_IN_LSREMOTE')
return output
@@ -842,28 +943,28 @@ class Git(FetchMethod):
Compute the HEAD revision for the url
"""
if not d.getVar("__BBSRCREV_SEEN"):
- raise bb.fetch2.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev[name], ud.host+ud.path))
+ raise bb.fetch.FetchError("Recipe uses a floating tag/branch '%s' for repo '%s' without a fixed SRCREV yet doesn't call bb.fetch.get_srcrev() (use SRCPV in PV for OE)." % (ud.unresolvedrev, ud.host+ud.path))
# Ensure we mark as not cached
- bb.fetch2.mark_recipe_nocache(d)
+ bb.fetch.mark_recipe_nocache(d)
output = self._lsremote(ud, d, "")
# Tags of the form ^{} may not work, need to fallback to other form
- if ud.unresolvedrev[name][:5] == "refs/" or ud.usehead:
- head = ud.unresolvedrev[name]
- tag = ud.unresolvedrev[name]
+ if ud.unresolvedrev[:5] == "refs/" or ud.usehead:
+ head = ud.unresolvedrev
+ tag = ud.unresolvedrev
else:
- head = "refs/heads/%s" % ud.unresolvedrev[name]
- tag = "refs/tags/%s" % ud.unresolvedrev[name]
+ head = "refs/heads/%s" % ud.unresolvedrev
+ tag = "refs/tags/%s" % ud.unresolvedrev
for s in [head, tag + "^{}", tag]:
for l in output.strip().split('\n'):
sha1, ref = l.split()
if s == ref:
return sha1
- raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
- (ud.unresolvedrev[name], ud.host+ud.path))
+ raise bb.fetch.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
+ (ud.unresolvedrev, ud.host+ud.path))
- def latest_versionstring(self, ud, d):
+ def latest_versionstring(self, ud, d, filter_regex=None):
"""
Compute the latest release name like "x.y.x" in "x.y.x+gitHASH"
by searching through the tags output of ls-remote, comparing
@@ -873,11 +974,11 @@ class Git(FetchMethod):
try:
output = self._lsremote(ud, d, "refs/tags/*")
- except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
+ except (bb.fetch.FetchError, bb.fetch.NetworkAccess) as e:
bb.note("Could not list remote: %s" % str(e))
return pupver
- rev_tag_re = re.compile(r"([0-9a-f]{40})\s+refs/tags/(.*)")
+ rev_tag_re = re.compile(r"([0-9a-f]{40,64})\s+refs/tags/(.*)")
pver_re = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or r"(?P<pver>([0-9][\.|_]?)+)")
nonrel_re = re.compile(r"(alpha|beta|rc|final)+")
@@ -903,6 +1004,10 @@ class Git(FetchMethod):
pver = m.group('pver').replace("_", ".")
+ if filter_regex:
+ if not re.match(filter_regex, pver):
+ continue
+
if verstring and bb.utils.vercmp(("0", pver, ""), ("0", verstring, "")) < 0:
continue
@@ -912,26 +1017,23 @@ class Git(FetchMethod):
return pupver
def _build_revision(self, ud, d, name):
- return ud.revisions[name]
+ return ud.revision
def gitpkgv_revision(self, ud, d, name):
"""
Return a sortable revision number by counting commits in the history
Based on gitpkgv.bblass in meta-openembedded
"""
- rev = self._build_revision(ud, d, name)
+ rev = ud.revision
localpath = ud.localpath
rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev)
if not os.path.exists(localpath):
commits = None
else:
if not os.path.exists(rev_file) or not os.path.getsize(rev_file):
- from pipes import quote
- commits = bb.fetch2.runfetchcmd(
- "git rev-list %s -- | wc -l" % quote(rev),
- d, quiet=True).strip().lstrip('0')
+ commits = len(bb.fetch.runfetchcmd(['git', 'rev-list', rev, '--'], d).splitlines())
if commits:
- open(rev_file, "w").write("%d\n" % int(commits))
+ open(rev_file, "w").write("%d\n" % commits)
else:
commits = open(rev_file, "r").readline(128).strip()
if commits:
@@ -943,5 +1045,5 @@ class Git(FetchMethod):
try:
self._lsremote(ud, d, "")
return True
- except bb.fetch2.FetchError:
+ except bb.fetch.FetchError:
return False
diff --git a/bitbake/lib/bb/fetch2/gitannex.py b/bitbake/lib/bb/fetch/gitannex.py
similarity index 65%
rename from bitbake/lib/bb/fetch2/gitannex.py
rename to bitbake/lib/bb/fetch/gitannex.py
index 80a808d8..d5917e54 100644
--- a/bitbake/lib/bb/fetch2/gitannex.py
+++ b/bitbake/lib/bb/fetch/gitannex.py
@@ -9,8 +9,8 @@ BitBake 'Fetch' git annex implementation
#
import bb
-from bb.fetch2.git import Git
-from bb.fetch2 import runfetchcmd
+from bb.fetch.git import Git
+from bb.fetch import runfetchcmd
class GitANNEX(Git):
def supports(self, ud, d):
@@ -27,7 +27,7 @@ class GitANNEX(Git):
def uses_annex(self, ud, d, wd):
for name in ud.names:
try:
- runfetchcmd("%s rev-list git-annex" % (ud.basecmd), d, quiet=True, workdir=wd)
+ runfetchcmd(ud.basecmd + ['rev-list', 'git-annex'], d, quiet=True, workdir=wd)
return True
except bb.fetch.FetchError:
pass
@@ -36,10 +36,10 @@ class GitANNEX(Git):
def update_annex(self, ud, d, wd):
try:
- runfetchcmd("%s annex get --all" % (ud.basecmd), d, quiet=True, workdir=wd)
+ runfetchcmd(ud.basecmd + ['annex', 'get', '--all'], d, quiet=True, workdir=wd)
except bb.fetch.FetchError:
return False
- runfetchcmd("chmod u+w -R %s/annex" % (ud.clonedir), d, quiet=True, workdir=wd)
+ runfetchcmd(['chmod', 'u+w', '-R', '%s/annex' % ud.clonedir], d, quiet=True, workdir=wd)
return True
@@ -54,24 +54,24 @@ class GitANNEX(Git):
super(GitANNEX, self).clone_shallow_local(ud, dest, d)
try:
- runfetchcmd("%s annex init" % ud.basecmd, d, workdir=dest)
+ runfetchcmd(ud.basecmd + ['annex', 'init'], d, workdir=dest)
except bb.fetch.FetchError:
pass
if self.uses_annex(ud, d, dest):
- runfetchcmd("%s annex get" % ud.basecmd, d, workdir=dest)
- runfetchcmd("chmod u+w -R %s/.git/annex" % (dest), d, quiet=True, workdir=dest)
+ runfetchcmd(ud.basecmd + ['annex', 'get'], d, workdir=dest)
+ runfetchcmd(['chmod', 'u+w', '-R', '%s/.git/annex' % dest], d, quiet=True, workdir=dest)
def unpack(self, ud, destdir, d):
Git.unpack(self, ud, destdir, d)
try:
- runfetchcmd("%s annex init" % (ud.basecmd), d, workdir=ud.destdir)
+ runfetchcmd(ud.basecmd + ['annex' ,'init'], d, workdir=ud.destdir)
except bb.fetch.FetchError:
pass
annex = self.uses_annex(ud, d, ud.destdir)
if annex:
- runfetchcmd("%s annex get" % (ud.basecmd), d, workdir=ud.destdir)
- runfetchcmd("chmod u+w -R %s/.git/annex" % (ud.destdir), d, quiet=True, workdir=ud.destdir)
+ runfetchcmd(ud.basecmd + ['annex', 'get'], d, workdir=ud.destdir)
+ runfetchcmd(['chmod', 'u+w', '-R', '%s/.git/annex' % ud.destdir], d, quiet=True, workdir=ud.destdir)
diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch/gitsm.py
similarity index 63%
rename from bitbake/lib/bb/fetch2/gitsm.py
rename to bitbake/lib/bb/fetch/gitsm.py
index f7f3af72..d9facdd2 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch/gitsm.py
@@ -22,10 +22,10 @@ import bb
import copy
import shutil
import tempfile
-from bb.fetch2.git import Git
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
-from bb.fetch2 import Fetch
+from bb.fetch.git import Git
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
+from bb.fetch import Fetch
class GitSM(Git):
def supports(self, ud, d):
@@ -62,36 +62,38 @@ class GitSM(Git):
return modules
# Collect the defined submodules, and their attributes
- for name in ud.names:
+ try:
+ gitmodules = runfetchcmd(ud.basecmd + ['show', '%s:.gitmodules' % ud.revision], d, quiet=True, workdir=workdir)
+ except:
+ # No submodules to update
+ gitmodules = ""
+
+ for m, md in parse_gitmodules(gitmodules).items():
try:
- gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
+ module_hash = runfetchcmd(ud.basecmd + ['ls-tree', '-z', '-d', ud.revision, md['path']], d, quiet=True, workdir=workdir)
except:
- # No submodules to update
+ # If the command fails, we don't have a valid file to check. If it doesn't
+ # fail -- it still might be a failure, see next check...
+ module_hash = ""
+
+ if not module_hash:
+ logger.debug("submodule %s is defined, but is not initialized in the repository. Skipping", m)
continue
- for m, md in parse_gitmodules(gitmodules).items():
- try:
- module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=workdir)
- except:
- # If the command fails, we don't have a valid file to check. If it doesn't
- # fail -- it still might be a failure, see next check...
- module_hash = ""
-
- if not module_hash:
- logger.debug("submodule %s is defined, but is not initialized in the repository. Skipping", m)
- continue
-
- submodules.append(m)
- paths[m] = md['path']
- revision[m] = ud.revisions[name]
- uris[m] = md['url']
- subrevision[m] = module_hash.split()[2]
-
- # Convert relative to absolute uri based on parent uri
- if uris[m].startswith('..') or uris[m].startswith('./'):
+ submodules.append(m)
+ paths[m] = md['path']
+ revision[m] = ud.revision
+ uris[m] = md['url']
+ subrevision[m] = module_hash.split()[2]
+
+ # Convert relative to absolute uri based on parent uri
+ if uris[m].startswith('..') or uris[m].startswith('./'):
+ if hasattr(ud, "origud"):
+ newud = copy.copy(ud.origud)
+ else:
newud = copy.copy(ud)
- newud.path = os.path.normpath(os.path.join(newud.path, uris[m]))
- uris[m] = Git._get_repo_url(self, newud)
+ newud.path = os.path.normpath(os.path.join(newud.path, uris[m]))
+ uris[m] = Git._get_repo_url(self, newud)
for module in submodules:
# Translate the module url into a SRC_URI
@@ -116,14 +118,14 @@ class GitSM(Git):
proto = "file"
url = "gitsm://" + uris[module]
if url.endswith("{}{}".format(ud.host, ud.path)):
- raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \
+ raise bb.fetch.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \
"Consider using git fetcher instead.")
url += ';protocol=%s' % proto
url += ";name=%s" % module
url += ";subpath=%s" % module
url += ";nobranch=1"
- url += ";lfs=%s" % self._need_lfs(ud)
+ url += ";lfs=%s" % ("1" if self._need_lfs(ud) else "0")
# Note that adding "user=" here to give credentials to the
# submodule is not supported. Since using SRC_URI to give git://
# URL a password is not supported, one have to use one of the
@@ -147,6 +149,22 @@ class GitSM(Git):
return submodules != []
+ def call_process_submodules(self, ud, d, extra_check, subfunc):
+ # If we're using a shallow mirror tarball it needs to be
+ # unpacked temporarily so that we can examine the .gitmodules file
+ # Unpack even when ud.clonedir is not available,
+ # which may occur during a fast shallow clone
+ unpack = extra_check or not os.path.exists(ud.clonedir)
+ if ud.shallow and os.path.exists(ud.fullshallow) and unpack:
+ tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
+ try:
+ runfetchcmd(['tar', '-xzf', ud.fullshallow], d, workdir=tmpdir)
+ self.process_submodules(ud, tmpdir, subfunc, d)
+ finally:
+ shutil.rmtree(tmpdir)
+ else:
+ self.process_submodules(ud, ud.clonedir, subfunc, d)
+
def need_update(self, ud, d):
if Git.need_update(self, ud, d):
return True
@@ -164,15 +182,7 @@ class GitSM(Git):
logger.error('gitsm: submodule update check failed: %s %s' % (type(e).__name__, str(e)))
need_update_result = True
- # If we're using a shallow mirror tarball it needs to be unpacked
- # temporarily so that we can examine the .gitmodules file
- if ud.shallow and os.path.exists(ud.fullshallow) and not os.path.exists(ud.clonedir):
- tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
- runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir)
- self.process_submodules(ud, tmpdir, need_update_submodule, d)
- shutil.rmtree(tmpdir)
- else:
- self.process_submodules(ud, ud.clonedir, need_update_submodule, d)
+ self.call_process_submodules(ud, d, not os.path.exists(ud.clonedir), need_update_submodule)
if need_update_list:
logger.debug('gitsm: Submodules requiring update: %s' % (' '.join(need_update_list)))
@@ -195,16 +205,7 @@ class GitSM(Git):
raise
Git.download(self, ud, d)
-
- # If we're using a shallow mirror tarball it needs to be unpacked
- # temporarily so that we can examine the .gitmodules file
- if ud.shallow and os.path.exists(ud.fullshallow) and self.need_update(ud, d):
- tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
- runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir)
- self.process_submodules(ud, tmpdir, download_submodule, d)
- shutil.rmtree(tmpdir)
- else:
- self.process_submodules(ud, ud.clonedir, download_submodule, d)
+ self.call_process_submodules(ud, d, self.need_update(ud, d), download_submodule)
def unpack(self, ud, destdir, d):
def unpack_submodules(ud, url, module, modpath, workdir, d):
@@ -230,14 +231,14 @@ class GitSM(Git):
local_path = newfetch.localpath(url)
# Correct the submodule references to the local download version...
- runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir)
+ runfetchcmd(ud.basecmd + ['config', 'submodule.%s.url' % module, local_path], d, workdir=ud.destdir)
if ud.shallow:
- runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir)
+ runfetchcmd(ud.basecmd + ['config', 'submodule.%s.shallow' % module, 'true'], d, workdir=ud.destdir)
# Ensure the submodule repository is NOT set to bare, since we're checking it out...
try:
- runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf, 'modules', module))
+ runfetchcmd(ud.basecmd + ['config', 'core.bare', 'false'], d, quiet=True, workdir=os.path.join(repo_conf, 'modules', module))
except:
logger.error("Unable to set git config core.bare to false for %s" % os.path.join(repo_conf, 'modules', module))
raise
@@ -247,15 +248,28 @@ class GitSM(Git):
ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d)
if not ud.bareclone and ret:
- # All submodules should already be downloaded and configured in the tree. This simply
- # sets up the configuration and checks out the files. The main project config should
- # remain unmodified, and no download from the internet should occur. As such, lfs smudge
- # should also be skipped as these files were already smudged in the fetch stage if lfs
- # was enabled.
- runfetchcmd("GIT_LFS_SKIP_SMUDGE=1 %s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
+ extraenv = {}
+ # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed.
+ if not self._need_lfs(ud):
+ extraenv['GIT_LFS_SKIP_SMUDGE'] = '1'
+ runfetchcmd(ud.basecmd + ['submodule', 'update', '--recursive', '--no-fetch'], d, quiet=True, workdir=ud.destdir, extraenv=extraenv)
+
+ def clean(self, ud, d):
+ def clean_submodule(ud, url, module, modpath, workdir, d):
+ url += ";bareclone=1;nobranch=1"
+ try:
+ newfetch = Fetch([url], d, cache=False)
+ newfetch.clean()
+ except Exception as e:
+ logger.warning('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
+
+ self.call_process_submodules(ud, d, True, clean_submodule)
+
+ # Clean top git dir
+ Git.clean(self, ud, d)
def implicit_urldata(self, ud, d):
- import shutil, subprocess, tempfile
+ import subprocess
urldata = []
def add_submodule(ud, url, module, modpath, workdir, d):
@@ -263,14 +277,6 @@ class GitSM(Git):
newfetch = Fetch([url], d, cache=False)
urldata.extend(newfetch.expanded_urldata())
- # If we're using a shallow mirror tarball it needs to be unpacked
- # temporarily so that we can examine the .gitmodules file
- if ud.shallow and os.path.exists(ud.fullshallow) and ud.method.need_update(ud, d):
- tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
- subprocess.check_call("tar -xzf %s" % ud.fullshallow, cwd=tmpdir, shell=True)
- self.process_submodules(ud, tmpdir, add_submodule, d)
- shutil.rmtree(tmpdir)
- else:
- self.process_submodules(ud, ud.clonedir, add_submodule, d)
+ self.call_process_submodules(ud, d, ud.method.need_update(ud, d), add_submodule)
return urldata
diff --git a/bitbake/lib/bb/fetch/gomod.py b/bitbake/lib/bb/fetch/gomod.py
new file mode 100644
index 00000000..a62d643f
--- /dev/null
+++ b/bitbake/lib/bb/fetch/gomod.py
@@ -0,0 +1,270 @@
+"""
+BitBake 'Fetch' implementation for Go modules
+
+The gomod/gomodgit fetchers are used to download Go modules to the module cache
+from a module proxy or directly from a version control repository.
+
+Example SRC_URI:
+
+SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
+SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;repo=go.googlesource.com/net;srcrev=..."
+
+Required SRC_URI parameters:
+
+- version
+ The version of the module.
+
+Optional SRC_URI parameters:
+
+- mod
+ Fetch and unpack the go.mod file only instead of the complete module.
+ The go command may need to download go.mod files for many different modules
+ when computing the build list, and go.mod files are much smaller than
+ module zip files.
+ The default is "0", set mod=1 for the go.mod file only.
+
+- sha256sum
+ The checksum of the module zip file, or the go.mod file in case of fetching
+ only the go.mod file. Alternatively, set the SRC_URI varible flag for
+ "module@version.sha256sum".
+
+- protocol
+ The method used when fetching directly from a version control repository.
+ The default is "https" for git.
+
+- repo
+ The URL when fetching directly from a version control repository. Required
+ when the URL is different from the module path.
+
+- srcrev
+ The revision identifier used when fetching directly from a version control
+ repository. Alternatively, set the SRCREV varible for "module@version".
+
+- subdir
+ The module subdirectory when fetching directly from a version control
+ repository. Required when the module is not located in the root of the
+ repository.
+
+Related variables:
+
+- GO_MOD_PROXY
+ The module proxy used by the fetcher.
+
+- GO_MOD_CACHE_DIR
+ The directory where the module cache is located.
+ This must match the exported GOMODCACHE variable for the go command to find
+ the downloaded modules.
+
+See the Go modules reference, https://go.dev/ref/mod, for more information
+about the module cache, module proxies and version control systems.
+"""
+
+import hashlib
+import os
+import re
+import shutil
+import subprocess
+import zipfile
+
+import bb
+from bb.fetch import FetchError
+from bb.fetch import MissingParameterError
+from bb.fetch import runfetchcmd
+from bb.fetch import subprocess_setup
+from bb.fetch.git import Git
+from bb.fetch.wget import Wget
+
+
+def escape(path):
+ """Escape capital letters using exclamation points."""
+ return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path)
+
+
+class GoMod(Wget):
+ """Class to fetch Go modules from a Go module proxy via wget"""
+
+ def supports(self, ud, d):
+ """Check to see if a given URL is for this fetcher."""
+ return ud.type == 'gomod'
+
+ def urldata_init(self, ud, d):
+ """Set up to download the module from the module proxy.
+
+ Set up to download the module zip file to the module cache directory
+ and unpack the go.mod file (unless downloading only the go.mod file):
+
+ cache/download/<module>/@v/<version>.zip: The module zip file.
+ cache/download/<module>/@v/<version>.mod: The go.mod file.
+ """
+
+ proxy = d.getVar('GO_MOD_PROXY') or 'proxy.golang.org'
+ moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod'
+
+ if 'version' not in ud.parm:
+ raise MissingParameterError('version', ud.url)
+
+ module = ud.host
+ if ud.path != '/':
+ module += ud.path
+ ud.parm['module'] = module
+ version = ud.parm['version']
+
+ # Set URL and filename for wget download
+ if ud.parm.get('mod', '0') == '1':
+ ext = '.mod'
+ else:
+ ext = '.zip'
+ path = escape(f"{module}/@v/{version}{ext}")
+ ud.url = bb.fetch.encodeurl(
+ ('https', proxy, '/' + path, None, None, None))
+ ud.parm['downloadfilename'] = f"{module.replace('/', '.')}@{version}{ext}"
+
+ # Set name for checksum verification
+ ud.parm['name'] = f"{module}@{version}"
+
+ # Set path for unpack
+ ud.parm['unpackpath'] = os.path.join(moddir, 'cache/download', path)
+
+ super().urldata_init(ud, d)
+
+ def unpack(self, ud, rootdir, d):
+ """Unpack the module in the module cache."""
+
+ # Unpack the module zip file or go.mod file
+ unpackpath = os.path.join(rootdir, ud.parm['unpackpath'])
+ unpackdir = os.path.dirname(unpackpath)
+ bb.utils.mkdirhier(unpackdir)
+ ud.unpack_tracer.unpack("file-copy", unpackdir)
+ cmd = ['cp', ud.localpath, unpackpath]
+ name = os.path.basename(unpackpath)
+ bb.note(f"Unpacking {name} to {unpackdir}/")
+ runfetchcmd(cmd, d)
+
+ if name.endswith('.zip'):
+ # Unpack the go.mod file from the zip file
+ module = ud.parm['module']
+ name = name.rsplit('.', 1)[0] + '.mod'
+ bb.note(f"Unpacking {name} to {unpackdir}/")
+ with zipfile.ZipFile(ud.localpath) as zf:
+ with open(os.path.join(unpackdir, name), mode='wb') as mf:
+ try:
+ f = module + '@' + ud.parm['version'] + '/go.mod'
+ shutil.copyfileobj(zf.open(f), mf)
+ except KeyError:
+ # If the module does not have a go.mod file, synthesize
+ # one containing only a module statement.
+ mf.write(f'module {module}\n'.encode())
+
+
+class GoModGit(Git):
+ """Class to fetch Go modules directly from a git repository"""
+
+ def supports(self, ud, d):
+ """Check to see if a given URL is for this fetcher."""
+ return ud.type == 'gomodgit'
+
+ def urldata_init(self, ud, d):
+ """Set up to download the module from the git repository.
+
+ Set up to download the git repository to the module cache directory and
+ unpack the module zip file and the go.mod file:
+
+ cache/vcs/<hash>: The bare git repository.
+ cache/download/<module>/@v/<version>.zip: The module zip file.
+ cache/download/<module>/@v/<version>.mod: The go.mod file.
+ """
+
+ moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod'
+
+ if 'version' not in ud.parm:
+ raise MissingParameterError('version', ud.url)
+
+ module = ud.host
+ if ud.path != '/':
+ module += ud.path
+ ud.parm['module'] = module
+
+ # Set host, path and srcrev for git download
+ if 'repo' in ud.parm:
+ repo = ud.parm['repo']
+ idx = repo.find('/')
+ if idx != -1:
+ ud.host = repo[:idx]
+ ud.path = repo[idx:]
+ else:
+ ud.host = repo
+ ud.path = ''
+ if 'protocol' not in ud.parm:
+ ud.parm['protocol'] = 'https'
+ ud.name = f"{module}@{ud.parm['version']}"
+ srcrev = d.getVar('SRCREV_' + ud.name)
+ if srcrev:
+ if 'srcrev' not in ud.parm:
+ ud.parm['srcrev'] = srcrev
+ else:
+ if 'srcrev' in ud.parm:
+ d.setVar('SRCREV_' + ud.name, ud.parm['srcrev'])
+ if 'branch' not in ud.parm:
+ ud.parm['nobranch'] = '1'
+
+ # Set subpath, subdir and bareclone for git unpack
+ if 'subdir' in ud.parm:
+ ud.parm['subpath'] = ud.parm['subdir']
+ key = f"git3:{ud.parm['protocol']}://{ud.host}{ud.path}".encode()
+ ud.parm['key'] = key
+ ud.parm['subdir'] = os.path.join(moddir, 'cache/vcs',
+ hashlib.sha256(key).hexdigest())
+ ud.parm['bareclone'] = '1'
+
+ super().urldata_init(ud, d)
+
+ def unpack(self, ud, rootdir, d):
+ """Unpack the module in the module cache."""
+
+ # Unpack the bare git repository
+ super().unpack(ud, rootdir, d)
+
+ moddir = d.getVar('GO_MOD_CACHE_DIR') or 'pkg/mod'
+
+ # Create the info file
+ module = ud.parm['module']
+ repodir = os.path.join(rootdir, ud.parm['subdir'])
+ with open(repodir + '.info', 'wb') as f:
+ f.write(ud.parm['key'])
+
+ # Unpack the go.mod file from the repository
+ unpackdir = os.path.join(rootdir, moddir, 'cache/download',
+ escape(module), '@v')
+ bb.utils.mkdirhier(unpackdir)
+ srcrev = ud.parm['srcrev']
+ version = ud.parm['version']
+ escaped_version = escape(version)
+ cmd = ['git', 'ls-tree', '-r', '--name-only', srcrev]
+ if 'subpath' in ud.parm:
+ cmd.append(ud.parm['subpath'])
+ files = runfetchcmd(cmd, d, workdir=repodir).split()
+ name = escaped_version + '.mod'
+ bb.note(f"Unpacking {name} to {unpackdir}/")
+ with open(os.path.join(unpackdir, name), mode='wb') as mf:
+ f = 'go.mod'
+ if 'subpath' in ud.parm:
+ f = os.path.join(ud.parm['subpath'], f)
+ if f in files:
+ cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f]
+ subprocess.check_call(cmd, stdout=mf, cwd=repodir,
+ preexec_fn=subprocess_setup)
+ else:
+ # If the module does not have a go.mod file, synthesize one
+ # containing only a module statement.
+ mf.write(f'module {module}\n'.encode())
+
+ # Synthesize the module zip file from the repository
+ name = escaped_version + '.zip'
+ bb.note(f"Unpacking {name} to {unpackdir}/")
+ with zipfile.ZipFile(os.path.join(unpackdir, name), mode='w') as zf:
+ prefix = module + '@' + version + '/'
+ for f in files:
+ cmd = ['git', 'cat-file', 'blob', srcrev + ':' + f]
+ data = subprocess.check_output(cmd, cwd=repodir,
+ preexec_fn=subprocess_setup)
+ zf.writestr(prefix + f, data)
diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch/hg.py
similarity index 72%
rename from bitbake/lib/bb/fetch2/hg.py
rename to bitbake/lib/bb/fetch/hg.py
index cbff8c49..9813b0ed 100644
--- a/bitbake/lib/bb/fetch2/hg.py
+++ b/bitbake/lib/bb/fetch/hg.py
@@ -15,11 +15,12 @@ BitBake 'Fetch' implementation for mercurial DRCS (hg).
import os
import bb
import errno
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+import shlex
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import MissingParameterError
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
class Hg(FetchMethod):
"""Class to fetch from mercurial repositories"""
@@ -63,7 +64,7 @@ class Hg(FetchMethod):
ud.pkgdir = os.path.join(hgdir, hgsrcname)
ud.moddir = os.path.join(ud.pkgdir, ud.module)
ud.localfile = ud.moddir
- ud.basecmd = d.getVar("FETCHCMD_hg") or "/usr/bin/env hg"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_hg") or "") or ["hg"]
ud.setup_revisions(d)
@@ -113,7 +114,7 @@ class Hg(FetchMethod):
hgroot = ud.user + "@" + host + ud.path
if command == "info":
- return "%s identify -i %s://%s/%s" % (ud.basecmd, proto, hgroot, ud.module)
+ return ud.basecmd + ['identify', '-i', '%s://%s/%s' % (proto, hgroot, ud.module)]
options = [];
@@ -122,26 +123,21 @@ class Hg(FetchMethod):
# the tag actually exists in the specified revision + 1, so it won't
# be available when used in any successive commands.
if ud.revision and command != "fetch":
- options.append("-r %s" % ud.revision)
+ options += ["-r", ud.revision]
+
+ authconfig = []
+ if ud.user and ud.pswd:
+ authconfig = ['--config', 'auth.default.prefix=*', '--config', 'auth.default.username=' + ud.user, '--config', 'auth.default.password=' + ud.pswd, '--config', 'auth.default.schemes=' + proto]
if command == "fetch":
- if ud.user and ud.pswd:
- cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" clone %s %s://%s/%s %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options), proto, hgroot, ud.module, ud.module)
- else:
- cmd = "%s clone %s %s://%s/%s %s" % (ud.basecmd, " ".join(options), proto, hgroot, ud.module, ud.module)
+ cmd = ud.basecmd + authconfig + ['clone'] + options + ['%s://%s/%s' % (proto, hgroot, ud.module), ud.module]
elif command == "pull":
# do not pass options list; limiting pull to rev causes the local
# repo not to contain it and immediately following "update" command
# will crash
- if ud.user and ud.pswd:
- cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull" % (ud.basecmd, ud.user, ud.pswd, proto)
- else:
- cmd = "%s pull" % (ud.basecmd)
+ cmd = ud.basecmd + authconfig + ['pull']
elif command == "update" or command == "up":
- if ud.user and ud.pswd:
- cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" update -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options))
- else:
- cmd = "%s update -C %s" % (ud.basecmd, " ".join(options))
+ cmd = ud.basecmd + authconfig + ['update', '-C'] + options
else:
raise FetchError("Invalid hg command %s" % command, ud.url)
@@ -155,7 +151,7 @@ class Hg(FetchMethod):
# If the checkout doesn't exist and the mirror tarball does, extract it
if not os.path.exists(ud.pkgdir) and os.path.exists(ud.fullmirror):
bb.utils.mkdirhier(ud.pkgdir)
- runfetchcmd("tar -xzf %s" % (ud.fullmirror), d, workdir=ud.pkgdir)
+ runfetchcmd(['tar', '-xzf', ud.fullmirror], d, workdir=ud.pkgdir)
if os.access(os.path.join(ud.moddir, '.hg'), os.R_OK):
# Found the source, check whether need pull
@@ -163,13 +159,13 @@ class Hg(FetchMethod):
logger.debug("Running %s", updatecmd)
try:
runfetchcmd(updatecmd, d, workdir=ud.moddir)
- except bb.fetch2.FetchError:
+ except bb.fetch.FetchError:
# Runnning pull in the repo
pullcmd = self._buildhgcommand(ud, d, "pull")
logger.info("Pulling " + ud.url)
# update sources there
logger.debug("Running %s", pullcmd)
- bb.fetch2.check_network_access(d, pullcmd, ud.url)
+ bb.fetch.check_network_access(d, pullcmd, ud.url)
runfetchcmd(pullcmd, d, workdir=ud.moddir)
try:
os.unlink(ud.fullmirror)
@@ -184,7 +180,7 @@ class Hg(FetchMethod):
# check out sources there
bb.utils.mkdirhier(ud.pkgdir)
logger.debug("Running %s", fetchcmd)
- bb.fetch2.check_network_access(d, fetchcmd, ud.url)
+ bb.fetch.check_network_access(d, fetchcmd, ud.url)
runfetchcmd(fetchcmd, d, workdir=ud.pkgdir)
# Even when we clone (fetch), we still need to update as hg's clone
@@ -207,7 +203,7 @@ class Hg(FetchMethod):
"""
Compute tip revision for the url
"""
- bb.fetch2.check_network_access(d, self._buildhgcommand(ud, d, "info"), ud.url)
+ bb.fetch.check_network_access(d, self._buildhgcommand(ud, d, "info"), ud.url)
output = runfetchcmd(self._buildhgcommand(ud, d, "info"), d)
return output.strip()
@@ -228,8 +224,8 @@ class Hg(FetchMethod):
os.unlink(ud.fullmirror)
logger.info("Creating tarball of hg repository")
- runfetchcmd("tar -czf %s %s" % (ud.fullmirror, ud.module), d, workdir=ud.pkgdir)
- runfetchcmd("touch %s.done" % (ud.fullmirror), d, workdir=ud.pkgdir)
+ runfetchcmd(['tar', '-czf', ud.fullmirror, ud.module], d, workdir=ud.pkgdir)
+ runfetchcmd(['touch', '%s.done' % ud.fullmirror], d, workdir=ud.pkgdir)
def localpath(self, ud, d):
return ud.pkgdir
@@ -249,16 +245,13 @@ class Hg(FetchMethod):
proto = ud.parm.get('protocol', 'http')
if not os.access(os.path.join(codir, '.hg'), os.R_OK):
logger.debug2("Unpack: creating new hg repository in '" + codir + "'")
- runfetchcmd("%s init %s" % (ud.basecmd, codir), d)
+ runfetchcmd(ud.basecmd + ['init', codir], d)
logger.debug2("Unpack: updating source in '" + codir + "'")
+ authconfig = []
if ud.user and ud.pswd:
- runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull %s" % (ud.basecmd, ud.user, ud.pswd, proto, ud.moddir), d, workdir=codir)
- else:
- runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d, workdir=codir)
- if ud.user and ud.pswd:
- runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" up -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, revflag), d, workdir=codir)
- else:
- runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, workdir=codir)
+ authconfig = ['--config', 'auth.default.prefix=*', '--config', 'auth.default.username=' + ud.user, '--config', 'auth.default.password=' + ud.pswd, '--config', 'auth.default.schemes=' + proto]
+ runfetchcmd(ud.basecmd + authconfig + ['pull', ud.moddir], d, workdir=codir)
+ runfetchcmd(ud.basecmd + authconfig + ['up', '-C', revflag], d, workdir=codir)
else:
logger.debug2("Unpack: extracting source to '" + codir + "'")
- runfetchcmd("%s archive -t files %s %s" % (ud.basecmd, revflag, codir), d, workdir=ud.moddir)
+ runfetchcmd(ud.basecmd + ['archive', '-t', 'files', revflag, codir], d, workdir=ud.moddir)
diff --git a/bitbake/lib/bb/fetch2/local.py b/bitbake/lib/bb/fetch/local.py
similarity index 83%
rename from bitbake/lib/bb/fetch2/local.py
rename to bitbake/lib/bb/fetch/local.py
index 7d766811..dfc5b564 100644
--- a/bitbake/lib/bb/fetch2/local.py
+++ b/bitbake/lib/bb/fetch/local.py
@@ -17,8 +17,8 @@ import os
import urllib.request, urllib.parse, urllib.error
import bb
import bb.utils
-from bb.fetch2 import FetchMethod, FetchError, ParameterError
-from bb.fetch2 import logger
+from bb.fetch import FetchMethod, FetchError, ParameterError
+from bb.fetch import logger
class Local(FetchMethod):
def supports(self, urldata, d):
@@ -29,12 +29,11 @@ class Local(FetchMethod):
def urldata_init(self, ud, d):
# We don't set localfile as for this fetcher the file is already local!
- ud.decodedurl = urllib.parse.unquote(ud.url.split("://")[1].split(";")[0])
- ud.basename = os.path.basename(ud.decodedurl)
- ud.basepath = ud.decodedurl
+ ud.basename = os.path.basename(ud.path)
+ ud.basepath = ud.path
ud.needdonestamp = False
- if "*" in ud.decodedurl:
- raise bb.fetch2.ParameterError("file:// urls using globbing are no longer supported. Please place the files in a directory and reference that instead.", ud.url)
+ if "*" in ud.path:
+ raise bb.fetch.ParameterError("file:// urls using globbing are no longer supported. Please place the files in a directory and reference that instead.", ud.url)
return
def localpath(self, urldata, d):
@@ -48,7 +47,7 @@ class Local(FetchMethod):
Return the local filename of a given url assuming a successful fetch.
"""
searched = []
- path = urldata.decodedurl
+ path = urldata.path
newpath = path
if path[0] == "/":
logger.debug2("Using absolute %s" % (path))
diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch/npm.py
similarity index 62%
rename from bitbake/lib/bb/fetch2/npm.py
rename to bitbake/lib/bb/fetch/npm.py
index 15f3f19b..c8995ba6 100644
--- a/bitbake/lib/bb/fetch2/npm.py
+++ b/bitbake/lib/bb/fetch/npm.py
@@ -16,6 +16,11 @@ Supported SRC_URI options are:
- version
The npm package version. This is a mandatory parameter.
+- sha512sum / sha256sum / sha384sum / sha1sum
+ The expected checksum of the downloaded tarball. Exactly one is
+ mandatory: the registry cannot be trusted to supply its own tamper
+ detection, so the checksum must come from the recipe.
+
- downloadfilename
Specifies the filename used when storing the downloaded file.
@@ -30,23 +35,28 @@ import re
import shlex
import tempfile
import bb
-from bb.fetch2 import Fetch
-from bb.fetch2 import FetchError
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import ParameterError
-from bb.fetch2 import URI
-from bb.fetch2 import check_network_access
-from bb.fetch2 import runfetchcmd
+from bb.fetch import Fetch
+from bb.fetch import FetchError
+from bb.fetch import FetchMethod
+from bb.fetch import MalformedUrl
+from bb.fetch import MissingParameterError
+from bb.fetch import ParameterError
+from bb.fetch import URI
+from bb.fetch import check_network_access
+from bb.fetch import runfetchcmd
from bb.utils import is_semver
+# Preference order matches strength; the first one present in SRC_URI wins.
+CHECKSUM_PARMS = ("sha512sum", "sha256sum", "sha384sum", "sha1sum")
+
def npm_package(package):
"""Convert the npm package name to remove unsupported character"""
- # Scoped package names (with the @) use the same naming convention
- # as the 'npm pack' command.
+ # For scoped package names ('@user/package') the '/' is replaced by a '-'.
+ # This is similar to what 'npm pack' does, but 'npm pack' also strips the
+ # leading '@', which can lead to ambiguous package names.
name = re.sub("/", "-", package)
name = name.lower()
- name = re.sub(r"[^\-a-z0-9]", "", name)
+ name = re.sub(r"[^\-a-z0-9@]", "", name)
name = name.strip("-")
return name
@@ -74,12 +84,9 @@ def npm_integrity(integrity):
def npm_unpack(tarball, destdir, d):
"""Unpack a npm tarball"""
bb.utils.mkdirhier(destdir)
- cmd = "tar --extract --gzip --file=%s" % shlex.quote(tarball)
- cmd += " --no-same-owner"
- cmd += " --delay-directory-restore"
- cmd += " --strip-components=1"
+ cmd = ['tar', '--extract', '--gzip', '--file=%s' % tarball, '--no-same-owner', '--delay-directory-restore', '--strip-components=1']
runfetchcmd(cmd, d, workdir=destdir)
- runfetchcmd("chmod -R +X '%s'" % (destdir), d, quiet=True, workdir=destdir)
+ runfetchcmd(['chmod', '-R', '+X', destdir], d, quiet=True, workdir=destdir)
class NpmEnvironment(object):
"""
@@ -90,6 +97,12 @@ class NpmEnvironment(object):
self.d = d
self.user_config = tempfile.NamedTemporaryFile(mode="w", buffering=1)
+
+ hn = self._home_npmrc(d)
+ if hn is not None:
+ with open(hn, 'r') as hnf:
+ self.user_config.write(hnf.read())
+
for key, value in configs:
self.user_config.write("%s=%s\n" % (key, value))
@@ -102,6 +115,15 @@ class NpmEnvironment(object):
if self.user_config:
self.user_config.close()
+ def _home_npmrc(self, d):
+ """Function to return user's HOME .npmrc file (or None if it doesn't exist)"""
+ home_npmrc_file = os.path.join(os.environ.get("HOME"), ".npmrc")
+ if d.getVar("BB_USE_HOME_NPMRC") == "1" and os.path.exists(home_npmrc_file):
+ bb.warn(f"BB_USE_HOME_NPMRC flag set and valid .npmrc detected - "\
+ f"npm fetcher will use {home_npmrc_file}")
+ return home_npmrc_file
+ return None
+
def run(self, cmd, args=None, configs=None, workdir=None):
"""Run npm command in a controlled environment"""
with tempfile.TemporaryDirectory() as tmpdir:
@@ -113,19 +135,20 @@ class NpmEnvironment(object):
workdir = tmpdir
def _run(cmd):
- cmd = "NPM_CONFIG_USERCONFIG=%s " % (self.user_config.name) + cmd
- cmd = "NPM_CONFIG_GLOBALCONFIG=%s " % (self.global_config_name) + cmd
- return runfetchcmd(cmd, d, workdir=workdir)
+ extraenv = {}
+ extraenv['NPM_CONFIG_USERCONFIG'] = self.user_config.name
+ extraenv['NPM_CONFIG_GLOBALCONFIG'] = self.global_config_name
+ return runfetchcmd(cmd, d, workdir=workdir, extraenv=extraenv)
if configs:
bb.warn("Use of configs argument of NpmEnvironment.run() function"
" is deprecated. Please use args argument instead.")
for key, value in configs:
- cmd += " --%s=%s" % (key, shlex.quote(value))
+ cmd.append('--%s=%s' % (key, value))
if args:
for key, value in args:
- cmd += " --%s=%s" % (key, shlex.quote(value))
+ cmd.append('--%s=%s' % (key, value))
return _run(cmd)
@@ -156,21 +179,42 @@ class Npm(FetchMethod):
if not ud.version:
raise MissingParameterError("Parameter 'version' required", ud.url)
- if not is_semver(ud.version) and not ud.version == "latest":
+ if ud.version == "latest":
+ raise ParameterError(
+ "Version 'latest' is not reproducible; specify an exact semver version",
+ ud.url)
+
+ if not is_semver(ud.version):
raise ParameterError("Invalid 'version' parameter", ud.url)
# Extract the 'registry' part of the url
ud.registry = re.sub(r"^npm://", "https://", ud.url.split(";")[0])
+ if not ud.url.split(";")[0][len("npm://"):]:
+ raise MalformedUrl(ud.url)
+
+ # A checksum is mandatory: the registry cannot be trusted to supply
+ # its own tamper detection, so the recipe must commit to one.
+ for cp in CHECKSUM_PARMS:
+ if cp in ud.parm:
+ ud.checksum_name = cp
+ ud.checksum_expected = ud.parm[cp]
+ break
+ else:
+ raise MissingParameterError(
+ "Missing checksum for npm package '%s@%s': a compromised "
+ "registry could otherwise serve a tampered tarball undetected. "
+ "Add one of %s to SRC_URI." % (ud.package, ud.version, ", ".join(CHECKSUM_PARMS)),
+ ud.url)
# Using the 'downloadfilename' parameter as local filename
# or the npm package name.
if "downloadfilename" in ud.parm:
- ud.localfile = npm_localfile(d.expand(ud.parm["downloadfilename"]))
+ ud.localfile = npm_localfile(ud.parm["downloadfilename"])
else:
ud.localfile = npm_localfile(ud.package, ud.version)
# Get the base 'npm' command
- ud.basecmd = d.getVar("FETCHCMD_npm") or "npm"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_npm") or "") or ["npm"]
# This fetcher resolves a URI from a npm package name and version and
# then forwards it to a proxy fetcher. A resolve file containing the
@@ -182,12 +226,19 @@ class Npm(FetchMethod):
ud.resolvefile = self.localpath(ud, d) + ".resolved"
def _resolve_proxy_url(self, ud, d):
+ """Resolve the tarball URL from the registry and cache it without any checksum.
+
+ Checksums must never be sourced from the registry: a compromised registry
+ controls both the tarball and its advertised hash, so any checksum obtained
+ there provides no tamper detection. Checksums are applied in _setup_proxy
+ from the recipe-provided SRC_URI parameters instead.
+ """
def _npm_view():
args = []
args.append(("json", "true"))
args.append(("registry", ud.registry))
- pkgver = shlex.quote(ud.package + "@" + ud.version)
- cmd = ud.basecmd + " view %s" % pkgver
+ pkgver = ud.package + "@" + ud.version
+ cmd = ud.basecmd + ['view', pkgver]
env = NpmEnvironment(d)
check_network_access(d, cmd, ud.registry)
view_string = env.run(cmd, args=args)
@@ -197,50 +248,27 @@ class Npm(FetchMethod):
try:
view = json.loads(view_string)
-
- error = view.get("error")
- if error is not None:
- raise FetchError(error.get("summary"), ud.url)
-
- if ud.version == "latest":
- bb.warn("The npm package %s is using the latest " \
- "version available. This could lead to " \
- "non-reproducible builds." % pkgver)
- elif ud.version != view.get("version"):
- raise ParameterError("Invalid 'version' parameter", ud.url)
-
- return view
-
- except Exception as e:
+ except json.JSONDecodeError as e:
raise FetchError("Invalid view from npm: %s" % str(e), ud.url)
- def _get_url(view):
- tarball_url = view.get("dist", {}).get("tarball")
-
- if tarball_url is None:
- raise FetchError("Invalid 'dist.tarball' in view", ud.url)
-
- uri = URI(tarball_url)
- uri.params["downloadfilename"] = ud.localfile
+ error = view.get("error")
+ if error is not None:
+ raise FetchError(error.get("summary") or str(error), ud.url)
- integrity = view.get("dist", {}).get("integrity")
- shasum = view.get("dist", {}).get("shasum")
+ if ud.version != view.get("version"):
+ raise ParameterError("Invalid 'version' parameter", ud.url)
- if integrity is not None:
- checksum_name, checksum_expected = npm_integrity(integrity)
- uri.params[checksum_name] = checksum_expected
- elif shasum is not None:
- uri.params["sha1sum"] = shasum
- else:
- raise FetchError("Invalid 'dist.integrity' in view", ud.url)
+ return view
- return str(uri)
+ view = _npm_view()
+ tarball_url = view.get("dist", {}).get("tarball")
- url = _get_url(_npm_view())
+ if tarball_url is None:
+ raise FetchError("Invalid 'dist.tarball' in view", ud.url)
bb.utils.mkdirhier(os.path.dirname(ud.resolvefile))
with open(ud.resolvefile, "w") as f:
- f.write(url)
+ f.write(tarball_url)
def _setup_proxy(self, ud, d):
if ud.proxy is None:
@@ -248,13 +276,26 @@ class Npm(FetchMethod):
self._resolve_proxy_url(ud, d)
with open(ud.resolvefile, "r") as f:
- url = f.read()
+ tarball_url = f.read().strip()
+
+ uri = URI(tarball_url)
+ # Discard any params that may have been embedded in the stored URL
+ # (e.g. from an npmsw-written .resolved file) and rebuild from
+ # scratch so that registry-sourced checksums can never flow through.
+ uri.params = {}
+ uri.params["downloadfilename"] = ud.localfile
+
+ # Inject the recipe-provided checksum into the proxy URL.
+ # Checksums from the remote registry are never used; only the
+ # value validated in urldata_init, sourced from the recipe, is
+ # trusted.
+ uri.params[ud.checksum_name] = ud.checksum_expected
# Avoid conflicts between the environment data and:
# - the proxy url checksum
data = bb.data.createCopy(d)
data.delVarFlags("SRC_URI")
- ud.proxy = Fetch([url], data)
+ ud.proxy = Fetch([str(uri)], data)
def _get_proxy_method(self, ud, d):
self._setup_proxy(ud, d)
@@ -278,8 +319,6 @@ class Npm(FetchMethod):
"""Force a fetch, even if localpath exists ?"""
if not os.path.exists(ud.resolvefile):
return True
- if ud.version == "latest":
- return True
proxy_m, proxy_ud, proxy_d = self._get_proxy_method(ud, d)
return proxy_m.need_update(proxy_ud, proxy_d)
diff --git a/bitbake/lib/bb/fetch2/npmsw.py b/bitbake/lib/bb/fetch/npmsw.py
similarity index 70%
rename from bitbake/lib/bb/fetch2/npmsw.py
rename to bitbake/lib/bb/fetch/npmsw.py
index ff5f8dc7..fde4fe65 100644
--- a/bitbake/lib/bb/fetch2/npmsw.py
+++ b/bitbake/lib/bb/fetch/npmsw.py
@@ -21,14 +21,15 @@ import json
import os
import re
import bb
-from bb.fetch2 import Fetch
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import ParameterError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import URI
-from bb.fetch2.npm import npm_integrity
-from bb.fetch2.npm import npm_localfile
-from bb.fetch2.npm import npm_unpack
+from bb.fetch import FetchError
+from bb.fetch import Fetch
+from bb.fetch import FetchMethod
+from bb.fetch import ParameterError
+from bb.fetch import runfetchcmd
+from bb.fetch import URI
+from bb.fetch.npm import npm_integrity
+from bb.fetch.npm import npm_localfile
+from bb.fetch.npm import npm_unpack
from bb.utils import is_semver
from bb.utils import lockfile
from bb.utils import unlockfile
@@ -37,38 +38,26 @@ def foreach_dependencies(shrinkwrap, callback=None, dev=False):
"""
Run a callback for each dependencies of a shrinkwrap file.
The callback is using the format:
- callback(name, params, deptree)
+ callback(name, data, location)
with:
name = the package name (string)
- params = the package parameters (dictionary)
- destdir = the destination of the package (string)
+ data = the package data (dictionary)
+ location = the location of the package (string)
"""
- # For handling old style dependencies entries in shinkwrap files
- def _walk_deps(deps, deptree):
- for name in deps:
- subtree = [*deptree, name]
- _walk_deps(deps[name].get("dependencies", {}), subtree)
- if callback is not None:
- if deps[name].get("dev", False) and not dev:
- continue
- elif deps[name].get("bundled", False):
- continue
- destsubdirs = [os.path.join("node_modules", dep) for dep in subtree]
- destsuffix = os.path.join(*destsubdirs)
- callback(name, deps[name], destsuffix)
-
- # packages entry means new style shrinkwrap file, else use dependencies
- packages = shrinkwrap.get("packages", None)
- if packages is not None:
- for package in packages:
- if package != "":
- name = package.split('node_modules/')[-1]
- package_infos = packages.get(package, {})
- if dev == False and package_infos.get("dev", False):
- continue
- callback(name, package_infos, package)
- else:
- _walk_deps(shrinkwrap.get("dependencies", {}), [])
+ packages = shrinkwrap.get("packages")
+ if packages is None:
+ raise FetchError("Invalid shrinkwrap file format")
+
+ for location, data in packages.items():
+ # Skip empty main and local link target packages
+ if not location.startswith('node_modules/'):
+ continue
+ elif not dev and data.get("dev", False):
+ continue
+ elif data.get("inBundle", False):
+ continue
+ name = location.split('node_modules/')[-1]
+ callback(name, data, location)
class NpmShrinkWrap(FetchMethod):
"""Class to fetch all package from a shrinkwrap file"""
@@ -95,12 +84,18 @@ class NpmShrinkWrap(FetchMethod):
extrapaths = []
unpack = True
- integrity = params.get("integrity", None)
- resolved = params.get("resolved", None)
- version = params.get("version", None)
+ integrity = params.get("integrity")
+ resolved = params.get("resolved")
+ version = params.get("version")
+ link = params.get("link", False)
+
+ # Handle link sources
+ if link:
+ localpath = resolved
+ unpack = False
# Handle registry sources
- if is_semver(version) and integrity:
+ elif version and is_semver(version) and integrity:
# Handle duplicate dependencies without url
if not resolved:
return
@@ -128,10 +123,13 @@ class NpmShrinkWrap(FetchMethod):
extrapaths.append(resolvefile)
# Handle http tarball sources
- elif version.startswith("http") and integrity:
- localfile = npm_localfile(os.path.basename(version))
+ elif resolved is None:
+ raise ParameterError("Missing 'resolved' field for dependency '%s'" % name, ud.url)
+
+ elif resolved.startswith("http") and integrity:
+ localfile = npm_localfile(os.path.basename(resolved))
- uri = URI(version)
+ uri = URI(resolved)
uri.params["downloadfilename"] = localfile
checksum_name, checksum_expected = npm_integrity(integrity)
@@ -141,28 +139,12 @@ class NpmShrinkWrap(FetchMethod):
localpath = os.path.join(d.getVar("DL_DIR"), localfile)
- # Handle local tarball and link sources
- elif version.startswith("file"):
- localpath = version[5:]
- if not version.endswith(".tgz"):
- unpack = False
+ # Handle local tarball sources
+ elif resolved.startswith("file"):
+ localpath = resolved[5:]
# Handle git sources
- elif version.startswith(("git", "bitbucket","gist")) or (
- not version.endswith((".tgz", ".tar", ".tar.gz"))
- and not version.startswith((".", "@", "/"))
- and "/" in version
- ):
- if version.startswith("github:"):
- version = "git+https://github.com/" + version[len("github:"):]
- elif version.startswith("gist:"):
- version = "git+https://gist.github.com/" + version[len("gist:"):]
- elif version.startswith("bitbucket:"):
- version = "git+https://bitbucket.org/" + version[len("bitbucket:"):]
- elif version.startswith("gitlab:"):
- version = "git+https://gitlab.com/" + version[len("gitlab:"):]
- elif not version.startswith(("git+","git:")):
- version = "git+https://github.com/" + version
+ elif resolved.startswith("git"):
regex = re.compile(r"""
^
git\+
@@ -174,16 +156,16 @@ class NpmShrinkWrap(FetchMethod):
$
""", re.VERBOSE)
- match = regex.match(version)
-
+ match = regex.match(resolved)
if not match:
- raise ParameterError("Invalid git url: %s" % version, ud.url)
+ raise ParameterError("Invalid git url: %s" % resolved, ud.url)
groups = match.groupdict()
uri = URI("git://" + str(groups["url"]))
uri.params["protocol"] = str(groups["protocol"])
uri.params["rev"] = str(groups["rev"])
+ uri.params["nobranch"] = "1"
uri.params["destsuffix"] = destsuffix
url = str(uri)
@@ -268,7 +250,7 @@ class NpmShrinkWrap(FetchMethod):
def unpack(self, ud, rootdir, d):
"""Unpack the downloaded dependencies"""
- destdir = d.getVar("S")
+ destdir = rootdir
destsuffix = ud.parm.get("destsuffix")
if destsuffix:
destdir = os.path.join(rootdir, destsuffix)
@@ -294,7 +276,7 @@ class NpmShrinkWrap(FetchMethod):
npm_unpack(depsrcdir, depdestdir, d)
else:
bb.utils.mkdirhier(depdestdir)
- cmd = 'cp -fpPRH "%s/." .' % (depsrcdir)
+ cmd = ['cp', '-fpPRH', '%s/.' % depsrcdir, "."]
runfetchcmd(cmd, d, workdir=depdestdir)
def clean(self, ud, d):
diff --git a/bitbake/lib/bb/fetch2/perforce.py b/bitbake/lib/bb/fetch/perforce.py
similarity index 87%
rename from bitbake/lib/bb/fetch2/perforce.py
rename to bitbake/lib/bb/fetch/perforce.py
index 3b6fa4b1..aa46e0eb 100644
--- a/bitbake/lib/bb/fetch2/perforce.py
+++ b/bitbake/lib/bb/fetch/perforce.py
@@ -26,10 +26,11 @@ Supported SRC_URI options are:
import os
import bb
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import logger
-from bb.fetch2 import runfetchcmd
+import shlex
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import logger
+from bb.fetch import runfetchcmd
class PerforceProgressHandler (bb.progress.BasicProgressHandler):
"""
@@ -73,7 +74,7 @@ class Perforce(FetchMethod):
provided by the env, use it. If P4PORT is specified by the recipe, use
its values, which may override the settings in P4CONFIG.
"""
- ud.basecmd = d.getVar("FETCHCMD_p4") or "/usr/bin/env p4"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_p4") or "") or ["p4"]
ud.dldir = d.getVar("P4DIR") or (d.getVar("DL_DIR") + "/p4")
@@ -95,10 +96,15 @@ class Perforce(FetchMethod):
else:
logger.debug('Trying to use P4CONFIG to automatically set P4PORT...')
ud.usingp4config = True
- p4cmd = '%s info | grep "Server address"' % ud.basecmd
- bb.fetch2.check_network_access(d, p4cmd, ud.url)
- ud.host = runfetchcmd(p4cmd, d, True)
- ud.host = ud.host.split(': ')[1].strip()
+ p4cmd = ud.basecmd + ['info']
+ bb.fetch.check_network_access(d, p4cmd, ud.url)
+ output = runfetchcmd(p4cmd, d, True)
+ ud.host = None
+ for line in output.splitlines():
+ if "Server address" in line:
+ ud.host = line.split(': ')[1].strip()
+ break
+
logger.debug('Determined P4PORT to be: %s' % ud.host)
if not ud.host:
raise FetchError('Could not determine P4PORT from P4CONFIG')
@@ -142,16 +148,16 @@ class Perforce(FetchMethod):
"files". depot_filename is the full path to the file in the depot
including the trailing '#rev' value.
"""
- p4opt = ""
+ p4opt = []
if ud.user:
- p4opt += ' -u "%s"' % (ud.user)
+ p4opt += ['-u', ud.user]
if ud.pswd:
- p4opt += ' -P "%s"' % (ud.pswd)
+ p4opt += ['-P', ud.pswd]
if ud.host and not ud.usingp4config:
- p4opt += ' -p %s' % (ud.host)
+ p4opt += ['-p', ud.host]
if hasattr(ud, 'revision') and ud.revision:
pathnrev = '%s@%s' % (ud.path, ud.revision)
@@ -176,14 +182,14 @@ class Perforce(FetchMethod):
filename = filename[:filename.find('#')] # Remove trailing '#rev'
if command == 'changes':
- p4cmd = '%s%s changes -m 1 //%s' % (ud.basecmd, p4opt, pathnrev)
+ p4cmd = ud.basecmd + p4opt + ['changes', '-m', '1', '//%s' % pathnrev]
elif command == 'print':
if depot_filename is not None:
- p4cmd = '%s%s print -o "p4/%s" "%s"' % (ud.basecmd, p4opt, filename, depot_filename)
+ p4cmd = ud.basecmd + p4opt + ['print', '-o', 'p4/%s' % filename, depot_filename]
else:
raise FetchError('No depot file name provided to p4 %s' % command, ud.url)
elif command == 'files':
- p4cmd = '%s%s files //%s' % (ud.basecmd, p4opt, pathnrev)
+ p4cmd = ud.basecmd + p4opt + ['files', '//%s' % pathnrev]
else:
raise FetchError('Invalid p4 command %s' % command, ud.url)
@@ -195,7 +201,7 @@ class Perforce(FetchMethod):
'p4 files' command, including trailing '#rev' file revision indicator
"""
p4cmd = self._buildp4command(ud, d, 'files')
- bb.fetch2.check_network_access(d, p4cmd, ud.url)
+ bb.fetch.check_network_access(d, p4cmd, ud.url)
p4fileslist = runfetchcmd(p4cmd, d, True)
p4fileslist = [f.rstrip() for f in p4fileslist.splitlines()]
@@ -228,10 +234,10 @@ class Perforce(FetchMethod):
for afile in filelist:
p4fetchcmd = self._buildp4command(ud, d, 'print', afile)
- bb.fetch2.check_network_access(d, p4fetchcmd, ud.url)
+ bb.fetch.check_network_access(d, p4fetchcmd, ud.url)
runfetchcmd(p4fetchcmd, d, workdir=ud.pkgdir, log=progresshandler)
- runfetchcmd('tar -czf %s p4' % (ud.localpath), d, cleanup=[ud.localpath], workdir=ud.pkgdir)
+ runfetchcmd(['tar', '-czf', ud.localpath, 'p4'], d, cleanup=[ud.localpath], workdir=ud.pkgdir)
def clean(self, ud, d):
""" Cleanup p4 specific files and dirs"""
@@ -248,7 +254,7 @@ class Perforce(FetchMethod):
def _latest_revision(self, ud, d, name):
""" Return the latest upstream scm revision number """
p4cmd = self._buildp4command(ud, d, "changes")
- bb.fetch2.check_network_access(d, p4cmd, ud.url)
+ bb.fetch.check_network_access(d, p4cmd, ud.url)
tip = runfetchcmd(p4cmd, d, True)
if not tip:
diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch/repo.py
similarity index 68%
rename from bitbake/lib/bb/fetch2/repo.py
rename to bitbake/lib/bb/fetch/repo.py
index fa4cb814..c7eaf8b3 100644
--- a/bitbake/lib/bb/fetch2/repo.py
+++ b/bitbake/lib/bb/fetch/repo.py
@@ -13,9 +13,10 @@ BitBake "Fetch" repo (git) implementation
import os
import bb
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+import shlex
+from bb.fetch import FetchMethod
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
class Repo(FetchMethod):
"""Class to fetch a module or modules from repo (git) repositories"""
@@ -33,7 +34,7 @@ class Repo(FetchMethod):
"master".
"""
- ud.basecmd = d.getVar("FETCHCMD_repo") or "/usr/bin/env repo"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_repo") or "") or ["repo"]
ud.proto = ud.parm.get('protocol', 'git')
ud.branch = ud.parm.get('branch', 'master')
@@ -62,20 +63,23 @@ class Repo(FetchMethod):
repodir = os.path.join(codir, "repo")
bb.utils.mkdirhier(repodir)
if not os.path.exists(os.path.join(repodir, ".repo")):
- bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
- runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir)
+ bb.fetch.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
+ runfetchcmd(ud.basecmd + ['init', '-m', ud.manifest, '-b', ud.branch, '-u', '%s://%s%s%s' % (ud.proto, username, ud.host, ud.path)], d, workdir=repodir)
- bb.fetch2.check_network_access(d, "%s sync %s" % (ud.basecmd, ud.url), ud.url)
- runfetchcmd("%s sync" % ud.basecmd, d, workdir=repodir)
+ bb.fetch.check_network_access(d, ud.basecmd + ['sync'], ud.url)
+ runfetchcmd(ud.basecmd + ['sync'], d, workdir=repodir)
scmdata = ud.parm.get("scmdata", "")
if scmdata == "keep":
- tar_flags = ""
+ tar_flags = []
else:
- tar_flags = "--exclude='.repo' --exclude='.git'"
+ tar_flags = ["--exclude=.repo", "--exclude=.git"]
# Create a cache
- runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.join(".", "*") ), d, workdir=codir)
+ tar_paths = [os.path.join(".", name)
+ for name in sorted(os.listdir(codir))
+ if not name.startswith(".")]
+ runfetchcmd(['tar'] + tar_flags + ['-czf', ud.localpath] + tar_paths, d, workdir=codir)
def supports_srcrev(self):
return False
diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch/s3.py
similarity index 87%
rename from bitbake/lib/bb/fetch2/s3.py
rename to bitbake/lib/bb/fetch/s3.py
index 6b8ffd53..ea3dc28f 100644
--- a/bitbake/lib/bb/fetch2/s3.py
+++ b/bitbake/lib/bb/fetch/s3.py
@@ -8,7 +8,7 @@ The aws tool must be correctly installed and configured prior to use.
# Copyright (C) 2017, Andre McCurdy <armccurdy@gmail.com>
#
-# Based in part on bb.fetch2.wget:
+# Based in part on bb.fetch.wget:
# Copyright (C) 2003, 2004 Chris Larson
#
# SPDX-License-Identifier: GPL-2.0-only
@@ -19,9 +19,10 @@ import os
import bb
import urllib.request, urllib.parse, urllib.error
import re
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import runfetchcmd
+import shlex
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import runfetchcmd
def convertToBytes(value, unit):
value = float(value)
@@ -77,9 +78,9 @@ class S3(FetchMethod):
else:
ud.basename = os.path.basename(ud.path)
- ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
+ ud.localfile = ud.basename
- ud.basecmd = d.getVar("FETCHCMD_s3") or "/usr/bin/env aws s3"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_s3") or "") or ["aws", "s3"]
def download(self, ud, d):
"""
@@ -87,8 +88,8 @@ class S3(FetchMethod):
Assumes localpath was called first
"""
- cmd = '%s cp s3://%s%s %s' % (ud.basecmd, ud.host, ud.path, ud.localpath)
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ cmd = ud.basecmd + ['cp', 's3://%s%s' % (ud.host, ud.path), ud.localpath]
+ bb.fetch.check_network_access(d, cmd, ud.url)
progresshandler = S3ProgressHandler(d)
runfetchcmd(cmd, d, False, log=progresshandler)
@@ -111,8 +112,8 @@ class S3(FetchMethod):
Check the status of a URL
"""
- cmd = '%s ls s3://%s%s' % (ud.basecmd, ud.host, ud.path)
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ cmd = ud.basecmd + ['ls', 's3://%s%s' % (ud.host, ud.path)]
+ bb.fetch.check_network_access(d, cmd, ud.url)
output = runfetchcmd(cmd, d)
# "aws s3 ls s3://mybucket/foo" will exit with success even if the file
diff --git a/bitbake/lib/bb/fetch2/sftp.py b/bitbake/lib/bb/fetch/sftp.py
similarity index 85%
rename from bitbake/lib/bb/fetch2/sftp.py
rename to bitbake/lib/bb/fetch/sftp.py
index 7884cce9..4054382d 100644
--- a/bitbake/lib/bb/fetch2/sftp.py
+++ b/bitbake/lib/bb/fetch/sftp.py
@@ -39,7 +39,7 @@ SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
#
-# Based in part on bb.fetch2.wget:
+# Based in part on bb.fetch.wget:
# Copyright (C) 2003, 2004 Chris Larson
#
# SPDX-License-Identifier: GPL-2.0-only
@@ -49,9 +49,9 @@ SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
import os
import bb
import urllib.request, urllib.parse, urllib.error
-from bb.fetch2 import URI
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import runfetchcmd
+from bb.fetch import URI
+from bb.fetch import FetchMethod
+from bb.fetch import runfetchcmd
class SFTP(FetchMethod):
"""Class to fetch urls via 'sftp'"""
@@ -67,7 +67,7 @@ class SFTP(FetchMethod):
def urldata_init(self, ud, d):
if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
- raise bb.fetch2.ParameterError(
+ raise bb.fetch.ParameterError(
"Invalid protocol - if you wish to fetch from a " +
"git repository using ssh, you need to use the " +
"git:// prefix with protocol=ssh", ud.url)
@@ -77,16 +77,15 @@ class SFTP(FetchMethod):
else:
ud.basename = os.path.basename(ud.path)
- ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
+ ud.localfile = ud.basename
def download(self, ud, d):
"""Fetch urls"""
urlo = URI(ud.url)
- basecmd = 'sftp -oBatchMode=yes'
- port = ''
+ basecmd = ['sftp', '-oBatchMode=yes']
if urlo.port:
- port = '-P %d' % urlo.port
+ basecmd += ['-P', str(urlo.port)]
urlo.port = None
dldir = d.getVar('DL_DIR')
@@ -103,10 +102,10 @@ class SFTP(FetchMethod):
if path[:3] == '/~/':
path = path[3:]
- remote = '"%s%s:%s"' % (user, urlo.hostname, path)
+ remote = '%s%s:%s' % (user, urlo.hostname, path)
- cmd = '%s %s %s %s' % (basecmd, port, remote, lpath)
+ cmd = basecmd + [remote, lpath]
- bb.fetch2.check_network_access(d, cmd, ud.url)
+ bb.fetch.check_network_access(d, cmd, ud.url)
runfetchcmd(cmd, d)
return True
diff --git a/bitbake/lib/bb/fetch2/ssh.py b/bitbake/lib/bb/fetch/ssh.py
similarity index 85%
rename from bitbake/lib/bb/fetch2/ssh.py
rename to bitbake/lib/bb/fetch/ssh.py
index 0cbb2a6f..304872ad 100644
--- a/bitbake/lib/bb/fetch2/ssh.py
+++ b/bitbake/lib/bb/fetch/ssh.py
@@ -31,7 +31,7 @@ IETF secsh internet draft:
#
import re, os
-from bb.fetch2 import check_network_access, FetchMethod, ParameterError, runfetchcmd
+from bb.fetch import check_network_access, FetchMethod, ParameterError, runfetchcmd
import urllib
@@ -73,8 +73,7 @@ class SSH(FetchMethod):
path = m.group('path')
path = urllib.parse.unquote(path)
host = m.group('host')
- urldata.localpath = os.path.join(d.getVar('DL_DIR'),
- os.path.basename(os.path.normpath(path)))
+ urldata.localfile = os.path.basename(os.path.normpath(path))
def download(self, urldata, d):
dldir = d.getVar('DL_DIR')
@@ -86,18 +85,16 @@ class SSH(FetchMethod):
user = m.group('user')
password = m.group('pass')
+ portarg = []
if port:
- portarg = '-P %s' % port
- else:
- portarg = ''
+ portarg = ['-P', port]
+ fr = host
if user:
fr = user
if password:
fr += ':%s' % password
fr += '@%s' % host
- else:
- fr = host
if path[0] != '~':
path = '/%s' % path
@@ -105,11 +102,7 @@ class SSH(FetchMethod):
fr += ':%s' % path
- cmd = 'scp -B -r %s %s %s/' % (
- portarg,
- fr,
- dldir
- )
+ cmd = ['scp', '-B', '-r'] + portarg + [fr, dldir + "/"]
check_network_access(d, cmd, urldata.url)
@@ -126,28 +119,22 @@ class SSH(FetchMethod):
user = m.group('user')
password = m.group('pass')
+ portarg = []
if port:
- portarg = '-P %s' % port
- else:
- portarg = ''
+ portarg = ['-P', port]
+ fr = host
if user:
fr = user
if password:
fr += ':%s' % password
fr += '@%s' % host
- else:
- fr = host
if path[0] != '~':
path = '/%s' % path
path = urllib.parse.unquote(path)
- cmd = 'ssh -o BatchMode=true %s %s [ -f %s ]' % (
- portarg,
- fr,
- path
- )
+ cmd = ['ssh', '-o', 'BatchMode=true'] + portarg + [fr, '[', '-f', path, ']']
check_network_access(d, cmd, urldata.url)
runfetchcmd(cmd, d)
diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch/svn.py
similarity index 65%
rename from bitbake/lib/bb/fetch2/svn.py
rename to bitbake/lib/bb/fetch/svn.py
index d40e4d29..5d577934 100644
--- a/bitbake/lib/bb/fetch2/svn.py
+++ b/bitbake/lib/bb/fetch/svn.py
@@ -13,11 +13,12 @@ BitBake 'Fetch' implementation for svn.
import os
import bb
import re
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
+import shlex
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import MissingParameterError
+from bb.fetch import runfetchcmd
+from bb.fetch import logger
class Svn(FetchMethod):
"""Class to fetch a module or modules from svn repositories"""
@@ -34,7 +35,7 @@ class Svn(FetchMethod):
if not "module" in ud.parm:
raise MissingParameterError('module', ud.url)
- ud.basecmd = d.getVar("FETCHCMD_svn") or "/usr/bin/env svn --non-interactive --trust-server-cert"
+ ud.basecmd = shlex.split(d.getVar("FETCHCMD_svn") or "") or ['svn', '--non-interactive']
ud.module = ud.parm["module"]
@@ -83,15 +84,17 @@ class Svn(FetchMethod):
options.append("--no-auth-cache")
if ud.user:
- options.append("--username %s" % ud.user)
+ options.append("--username")
+ options.append(ud.user)
if ud.pswd:
- options.append("--password %s" % ud.pswd)
+ options.append("--password")
+ options.append(ud.pswd)
if command == "info":
- svncmd = "%s info %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
+ svncmd = ud.basecmd + ['info'] + options + ['%s://%s/%s/' % (proto, svnroot, ud.module)]
elif command == "log1":
- svncmd = "%s log --limit 1 --quiet %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
+ svncmd = ud.basecmd + ['log', '--limit', '1', '--quiet'] + options + ['%s://%s/%s/' % (proto, svnroot, ud.module)]
else:
suffix = ""
@@ -102,22 +105,24 @@ class Svn(FetchMethod):
options.append("--ignore-externals")
if ud.revision:
- options.append("-r %s" % ud.revision)
+ options.append("-r")
+ options.append(ud.revision)
if ud.pegrevision:
suffix = "@%s" % (ud.revision)
if command == "fetch":
transportuser = ud.parm.get("transportuser", "")
- svncmd = "%s co %s %s://%s%s/%s%s %s" % (ud.basecmd, " ".join(options), proto, transportuser, svnroot, ud.module, suffix, ud.path_spec)
+ svncmd = ud.basecmd + ['co'] + options + ['%s://%s%s/%s%s' % (proto, transportuser, svnroot, ud.module, suffix), ud.path_spec]
elif command == "update":
- svncmd = "%s update %s" % (ud.basecmd, " ".join(options))
+ svncmd = ud.basecmd + ['update'] + options
else:
raise FetchError("Invalid svn command %s" % command, ud.url)
+ extraenv = {}
if svn_ssh:
- svncmd = "SVN_SSH=\"%s\" %s" % (svn_ssh, svncmd)
+ extraenv['SVN_SSH'] = svn_ssh
- return svncmd
+ return svncmd, extraenv
def download(self, ud, d):
"""Fetch url"""
@@ -128,45 +133,47 @@ class Svn(FetchMethod):
try:
if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
- svncmd = self._buildsvncommand(ud, d, "update")
+ svncmd, extraenv = self._buildsvncommand(ud, d, "update")
logger.info("Update " + ud.url)
# We need to attempt to run svn upgrade first in case its an older working format
try:
- runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir)
+ runfetchcmd(ud.basecmd + ["upgrade"], d, workdir=ud.moddir)
except FetchError:
pass
logger.debug("Running %s", svncmd)
- bb.fetch2.check_network_access(d, svncmd, ud.url)
- runfetchcmd(svncmd, d, workdir=ud.moddir)
+ bb.fetch.check_network_access(d, svncmd, ud.url)
+ runfetchcmd(svncmd, d, workdir=ud.moddir, extraenv=extraenv)
else:
- svncmd = self._buildsvncommand(ud, d, "fetch")
+ svncmd, extraenv = self._buildsvncommand(ud, d, "fetch")
logger.info("Fetch " + ud.url)
# check out sources there
bb.utils.mkdirhier(ud.pkgdir)
logger.debug("Running %s", svncmd)
- bb.fetch2.check_network_access(d, svncmd, ud.url)
- runfetchcmd(svncmd, d, workdir=ud.pkgdir)
+ bb.fetch.check_network_access(d, svncmd, ud.url)
+ runfetchcmd(svncmd, d, workdir=ud.pkgdir, extraenv=extraenv)
if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
# Warn the user if this had externals (won't catch them all)
- output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
- if output:
- if "--ignore-externals" in svncmd.split():
- bb.warn("%s contains svn:externals." % ud.url)
- bb.warn("These should be added to the recipe SRC_URI as necessary.")
- bb.warn("svn fetch has ignored externals:\n%s" % output)
- bb.warn("To disable this warning add ';externals=nowarn' to the url.")
- else:
- bb.debug(1, "svn repository has externals:\n%s" % output)
-
+ try:
+ output = runfetchcmd(['svn', 'propget', 'svn:externals'], d, workdir=ud.moddir)
+ if output:
+ if "--ignore-externals" in svncmd:
+ bb.warn("%s contains svn:externals." % ud.url)
+ bb.warn("These should be added to the recipe SRC_URI as necessary.")
+ bb.warn("svn fetch has ignored externals:\n%s" % output)
+ bb.warn("To disable this warning add ';externals=nowarn' to the url.")
+ else:
+ bb.debug(1, "svn repository has externals:\n%s" % output)
+ except bb.fetch.FetchError:
+ pass
scmdata = ud.parm.get("scmdata", "")
if scmdata == "keep":
- tar_flags = ""
+ tar_flags = []
else:
- tar_flags = "--exclude='.svn'"
+ tar_flags = ["--exclude=.svn"]
# tar them up to a defined filename
- runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d,
+ runfetchcmd(['tar'] + tar_flags + ['-czf', ud.localpath, ud.path_spec], d,
cleanup=[ud.localpath], workdir=ud.pkgdir)
finally:
bb.utils.unlockfile(lf)
@@ -191,9 +198,11 @@ class Svn(FetchMethod):
"""
Return the latest upstream revision number
"""
- bb.fetch2.check_network_access(d, self._buildsvncommand(ud, d, "log1"), ud.url)
-
- output = runfetchcmd("LANG=C LC_ALL=C " + self._buildsvncommand(ud, d, "log1"), d, True)
+ cmd, extraenv = self._buildsvncommand(ud, d, "log1")
+ bb.fetch.check_network_access(d, cmd, ud.url)
+ extraenv['LANG'] = 'C'
+ extraenv['LC_ALL'] = 'C'
+ output = runfetchcmd(cmd, d, True, extraenv=extraenv)
# skip the first line, as per output of svn log
# then we expect the revision on the 2nd line
@@ -210,3 +219,6 @@ class Svn(FetchMethod):
def _build_revision(self, ud, d):
return ud.revision
+
+ def supports_checksum(self, urldata):
+ return False
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch/wget.py
similarity index 68%
rename from bitbake/lib/bb/fetch2/wget.py
rename to bitbake/lib/bb/fetch/wget.py
index fbfa6938..972c8404 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch/wget.py
@@ -22,12 +22,13 @@ import bb.progress
import socket
import http.client
import urllib.request, urllib.parse, urllib.error
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import logger
-from bb.fetch2 import runfetchcmd
-from bs4 import BeautifulSoup
-from bs4 import SoupStrainer
+import subprocess
+from bb.fetch import FetchMethod
+from bb.fetch import FetchError
+from bb.fetch import logger
+from bb.fetch import runfetchcmd
+from bb._vendor.bs4 import BeautifulSoup
+from bb._vendor.bs4 import SoupStrainer
class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
"""
@@ -53,10 +54,10 @@ class WgetProgressHandler(bb.progress.LineFilterProgressHandler):
class Wget(FetchMethod):
"""Class to fetch urls via 'wget'"""
- # CDNs like CloudFlare may do a 'browser integrity test' which can fail
- # with the standard wget/urllib User-Agent, so pretend to be a modern
- # browser.
- user_agent = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0"
+ def __init__(self):
+ super().__init__()
+ self._unverified_ssl_context = None
+ self._verified_ssl_context = None
def check_certs(self, d):
"""
@@ -64,6 +65,23 @@ class Wget(FetchMethod):
"""
return (d.getVar("BB_CHECK_SSL_CERTS") or "1") != "0"
+ def ssl_context(self, d):
+ """
+ Get an SSL context, caching it to avoid creating every time.
+ """
+ if self.check_certs(d):
+ # Cache verified SSL context
+ if self._verified_ssl_context is None:
+ import ssl
+ self._verified_ssl_context = ssl.create_default_context()
+ return self._verified_ssl_context
+
+ # Cache unverified SSL context
+ if self._unverified_ssl_context is None:
+ import ssl
+ self._unverified_ssl_context = ssl._create_unverified_context()
+ return self._unverified_ssl_context
+
def supports(self, ud, d):
"""
Check to see if a given url can be fetched with wget.
@@ -76,44 +94,49 @@ class Wget(FetchMethod):
def urldata_init(self, ud, d):
if 'protocol' in ud.parm:
if ud.parm['protocol'] == 'git':
- raise bb.fetch2.ParameterError("Invalid protocol - if you wish to fetch from a git repository using http, you need to instead use the git:// prefix with protocol=http", ud.url)
+ raise bb.fetch.ParameterError("Invalid protocol - if you wish to fetch from a git repository using http, you need to instead use the git:// prefix with protocol=http", ud.url)
if 'downloadfilename' in ud.parm:
ud.basename = ud.parm['downloadfilename']
else:
ud.basename = os.path.basename(ud.path)
- ud.localfile = d.expand(urllib.parse.unquote(ud.basename))
+ ud.localfile = ud.basename
if not ud.localfile:
- ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."))
+ ud.localfile = ud.host + ud.path.replace("/", ".")
- self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30"
+ self.basecmd = shlex.split(d.getVar("FETCHCMD_wget") or "") or ['wget', '--tries=2', '--timeout=100']
if ud.type == 'ftp' or ud.type == 'ftps':
- self.basecmd += " --passive-ftp"
+ self.basecmd.append("--passive-ftp")
if not self.check_certs(d):
- self.basecmd += " --no-check-certificate"
+ self.basecmd.append("--no-check-certificate")
def _runwget(self, ud, d, command, quiet, workdir=None):
progresshandler = WgetProgressHandler(d)
logger.debug2("Fetching %s using command '%s'" % (ud.url, command))
- bb.fetch2.check_network_access(d, command, ud.url)
- runfetchcmd(command + ' --progress=dot -v', d, quiet, log=progresshandler, workdir=workdir)
+ bb.fetch.check_network_access(d, command, ud.url)
+
+ runfetchcmd(command + ['--progress=dot', '--verbose'], d, quiet, log=progresshandler, workdir=workdir)
def download(self, ud, d):
"""Fetch urls"""
- fetchcmd = self.basecmd
+ fetchcmd = self.basecmd.copy()
- localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp"
+ dldir = os.path.realpath(d.getVar("DL_DIR"))
+ # Where we ultimately want the file
+ finalpath = os.path.join(dldir, ud.localfile)
+ # A temp location while processing, keeping in mind max path lengths
+ localpath = finalpath[:250] + ".tmp"
bb.utils.mkdirhier(os.path.dirname(localpath))
- fetchcmd += " -O %s" % shlex.quote(localpath)
+ fetchcmd.append("--output-document=%s" % localpath)
if ud.user and ud.pswd:
- fetchcmd += " --auth-no-challenge"
+ fetchcmd.append("--auth-no-challenge")
if ud.parm.get("redirectauth", "1") == "1":
# An undocumented feature of wget is that if the
# username/password are specified on the URI, wget will only
@@ -123,59 +146,84 @@ class Wget(FetchMethod):
# AWS will reject any request that has authentication both in
# the query parameters (from the redirect) and in the
# Authorization header.
- fetchcmd += " --user=%s --password=%s" % (ud.user, ud.pswd)
+ fetchcmd.append("--user=" + ud.user)
+ fetchcmd.append("--password=" + ud.pswd)
uri = ud.url.split(";")[0]
- if os.path.exists(ud.localpath):
- # file exists, but we didnt complete it.. trying again..
- fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
- else:
- fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri)
-
+ fetchcmd.append("--continue")
+ fetchcmd.append("--directory-prefix=" + dldir)
+ fetchcmd.append(uri)
self._runwget(ud, d, fetchcmd, False)
+ # Sanity check since wget can pretend it succeed when it didn't
+ # Also, this used to happen if sourceforge sent us to the mirror page
+ if not os.path.exists(localpath):
+ raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, localpath), uri)
+
+ if os.path.getsize(localpath) == 0:
+ os.remove(localpath)
+ raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri)
+
# Try and verify any checksum now, meaning if it isn't correct, we don't remove the
# original file, which might be a race (imagine two recipes referencing the same
# source, one with an incorrect checksum)
- bb.fetch2.verify_checksum(ud, d, localpath=localpath, fatal_nochecksum=False)
+ bb.fetch.verify_checksum(ud, d, localpath=localpath, fatal_nochecksum=False)
# Remove the ".tmp" and move the file into position atomically
# Our lock prevents multiple writers but mirroring code may grab incomplete files
- os.rename(localpath, localpath[:-4])
-
- # Sanity check since wget can pretend it succeed when it didn't
- # Also, this used to happen if sourceforge sent us to the mirror page
- if not os.path.exists(ud.localpath):
- raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri)
-
- if os.path.getsize(ud.localpath) == 0:
- os.remove(ud.localpath)
- raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri)
+ os.rename(localpath, finalpath)
return True
def checkstatus(self, fetch, ud, d, try_again=True):
+ check_certs = self.check_certs(d)
+ newenv = bb.fetch.get_fetcher_environment(d)
+
class HTTPConnectionCache(http.client.HTTPConnection):
+ def cache_id(self):
+ return None
+
if fetch.connection_cache:
def connect(self):
"""Connect to the host and port specified in __init__."""
- sock = fetch.connection_cache.get_connection(self.host, self.port)
+ sock = fetch.connection_cache.get_connection(
+ self.host, self.port, self.cache_id())
if sock:
self.sock = sock
else:
self.sock = socket.create_connection((self.host, self.port),
self.timeout, self.source_address)
- fetch.connection_cache.add_connection(self.host, self.port, self.sock)
+ fetch.connection_cache.add_connection(
+ self.host, self.port, self.sock, self.cache_id())
if self._tunnel_host:
self._tunnel()
+ class HTTPSConnectionCache(http.client.HTTPSConnection):
+ def cache_id(self):
+ return ("https", check_certs,
+ newenv.get("SSL_CERT_FILE"),
+ self._tunnel_host, self._tunnel_port)
+
+ if fetch.connection_cache:
+ def connect(self):
+ """Reuse an established TLS connection when available."""
+
+ sock = fetch.connection_cache.get_connection(
+ self.host, self.port, self.cache_id())
+ if sock:
+ self.sock = sock
+ else:
+ super().connect()
+ fetch.connection_cache.add_connection(
+ self.host, self.port, self.sock, self.cache_id())
+
class CacheHTTPHandler(urllib.request.HTTPHandler):
def http_open(self, req):
return self.do_open(HTTPConnectionCache, req)
- def do_open(self, http_class, req):
+ def do_open(self, http_class, req, **http_conn_args):
"""Return an addinfourl object for the request, using http_class.
http_class must implement the HTTPConnection API from httplib.
@@ -189,7 +237,7 @@ class Wget(FetchMethod):
if not host:
raise urllib.error.URLError('no host given')
- h = http_class(host, timeout=req.timeout) # will parse host:port
+ h = http_class(host, timeout=req.timeout, **http_conn_args) # will parse host:port
h.set_debuglevel(self._debuglevel)
headers = dict(req.unredirected_hdrs)
@@ -224,26 +272,21 @@ class Wget(FetchMethod):
try:
h.request(req.get_method(), req.selector, req.data, headers)
- except socket.error as err: # XXX what error?
- # Don't close connection when cache is enabled.
- # Instead, try to detect connections that are no longer
- # usable (for example, closed unexpectedly) and remove
- # them from the cache.
- if fetch.connection_cache is None:
- h.close()
- elif isinstance(err, OSError) and err.errno == errno.EBADF:
- # This happens when the server closes the connection despite the Keep-Alive.
- # Apparently urllib then uses the file descriptor, expecting it to be
- # connected, when in reality the connection is already gone.
- # We let the request fail and expect it to be
- # tried once more ("try_again" in check_status()),
- # with the dead connection removed from the cache.
- # If it still fails, we give up, which can happen for bad
- # HTTP proxy settings.
- fetch.connection_cache.remove_connection(h.host, h.port)
- raise urllib.error.URLError(err)
- else:
r = h.getresponse()
+ except:
+ # This can happen when the server closes the connection despite the Keep-Alive.
+ # Apparently urllib then uses the file descriptor, expecting it to be
+ # connected, when in reality the connection is already gone.
+ # We let the request fail and expect it to be
+ # tried once more ("try_again" in check_status()),
+ # with the dead connection removed from the cache.
+ # If it still fails, we give up, which can happen for bad
+ # HTTP proxy settings.
+ if fetch.connection_cache:
+ fetch.connection_cache.remove_connection(
+ h.host, h.port, h.cache_id())
+ h.close()
+ raise
# Pick apart the HTTPResponse object to get the addinfourl
# object initialized properly.
@@ -275,10 +318,20 @@ class Wget(FetchMethod):
# Close connection when server request it.
if fetch.connection_cache is not None:
if 'Connection' in r.msg and r.msg['Connection'] == 'close':
- fetch.connection_cache.remove_connection(h.host, h.port)
+ fetch.connection_cache.remove_connection(
+ h.host, h.port, h.cache_id())
return resp
+ class CacheHTTPSHandler(CacheHTTPHandler, urllib.request.HTTPSHandler):
+ def __init__(self, debuglevel=0, context=None, check_hostname=None):
+ urllib.request.HTTPSHandler.__init__(self, debuglevel, context,
+ check_hostname)
+
+ def https_open(self, req):
+ return self.do_open(HTTPSConnectionCache, req,
+ context=self._context)
+
class HTTPMethodFallback(urllib.request.BaseHandler):
"""
Fallback to GET if HEAD is not allowed (405 HTTP error)
@@ -302,15 +355,70 @@ class Wget(FetchMethod):
http_error_403 = http_error_405
+ def _url_origin(url):
+ parsed = urllib.parse.urlsplit(url)
+ scheme = parsed.scheme.lower()
+ host = parsed.hostname.lower() if parsed.hostname else ""
+ port = parsed.port
+ if port is None:
+ port = {"http": 80, "https": 443}.get(scheme)
+ return (scheme, host, port)
+
+ def _same_origin(url_a, url_b):
+ return _url_origin(url_a) == _url_origin(url_b)
+
class FixedHTTPRedirectHandler(urllib.request.HTTPRedirectHandler):
"""
- urllib2.HTTPRedirectHandler resets the method to GET on redirect,
- when we want to follow redirects using the original method.
+ urllib2.HTTPRedirectHandler before 3.13 has two flaws:
+
+ It resets the method to GET on redirect when we want to follow
+ redirects using the original method (typically HEAD). This was fixed
+ in 759e8e7.
+
+ It also doesn't handle 308 (Permanent Redirect). This was fixed in
+ c379bc5.
+
+ Until we depend on Python 3.13 onwards, copy the redirect_request
+ method to fix these issues.
+
+ Additionally, strip sensitive headers (Authorization, Cookie) when
+ redirecting to a different origin to avoid credential leaks.
"""
def redirect_request(self, req, fp, code, msg, headers, newurl):
- newreq = urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
- newreq.get_method = req.get_method
- return newreq
+ m = req.get_method()
+ if (not (code in (301, 302, 303, 307, 308) and m in ("GET", "HEAD")
+ or code in (301, 302, 303) and m == "POST")):
+ raise urllib.HTTPError(req.full_url, code, msg, headers, fp)
+
+ # Strictly (according to RFC 2616), 301 or 302 in response to
+ # a POST MUST NOT cause a redirection without confirmation
+ # from the user (of urllib.request, in this case). In practice,
+ # essentially all clients do redirect in this case, so we do
+ # the same.
+
+ # Be conciliant with URIs containing a space. This is mainly
+ # redundant with the more complete encoding done in http_error_302(),
+ # but it is kept for compatibility with other callers.
+ newurl = newurl.replace(' ', '%20')
+
+ CONTENT_HEADERS = ("content-length", "content-type")
+ SENSITIVE_REDIRECT_HEADERS = ("authorization", "cookie")
+ same_origin = _same_origin(req.get_full_url(), newurl)
+ newheaders = {}
+ for k, v in req.headers.items():
+ header = k.lower()
+ if header in CONTENT_HEADERS:
+ continue
+ if not same_origin and header in SENSITIVE_REDIRECT_HEADERS:
+ continue
+ newheaders[k] = v
+ return urllib.request.Request(newurl,
+ method="HEAD" if m == "HEAD" else "GET",
+ headers=newheaders,
+ origin_req_host=req.origin_req_host,
+ unverifiable=True)
+
+ http_error_308 = urllib.request.HTTPRedirectHandler.http_error_302
# We need to update the environment here as both the proxy and HTTPS
# handlers need variables set. The proxy needs http_proxy and friends to
@@ -325,32 +433,27 @@ class Wget(FetchMethod):
# Avoid tramping the environment too much by using bb.utils.environment
# to scope the changes to the build_opener request, which is when the
# environment lookups happen.
- newenv = bb.fetch2.get_fetcher_environment(d)
-
with bb.utils.environment(**newenv):
- import ssl
-
- if self.check_certs(d):
- context = ssl.create_default_context()
- else:
- context = ssl._create_unverified_context()
-
+ context = self.ssl_context(d)
handlers = [FixedHTTPRedirectHandler,
HTTPMethodFallback,
urllib.request.ProxyHandler(),
CacheHTTPHandler(),
- urllib.request.HTTPSHandler(context=context)]
+ CacheHTTPSHandler(context=context)]
opener = urllib.request.build_opener(*handlers)
try:
- uri_base = ud.url.split(";")[0]
- uri = "{}://{}{}".format(urllib.parse.urlparse(uri_base).scheme, ud.host, ud.path)
+ parts = urllib.parse.urlparse(ud.url.split(";")[0])
+ if parts.query:
+ uri = "{}://{}{}?{}".format(parts.scheme, parts.netloc, parts.path, parts.query)
+ else:
+ uri = "{}://{}{}".format(parts.scheme, parts.netloc, parts.path)
r = urllib.request.Request(uri)
r.get_method = lambda: "HEAD"
# Some servers (FusionForge, as used on Alioth) require that the
# optional Accept header is set.
r.add_header("Accept", "*/*")
- r.add_header("User-Agent", self.user_agent)
+ r.add_header("User-Agent", "bitbake/{}".format(bb.__version__))
def add_basic_auth(login_str, request):
'''Adds Basic auth to http request, pass in login:password as string'''
import base64
@@ -370,15 +473,20 @@ class Wget(FetchMethod):
except (FileNotFoundError, netrc.NetrcParseError):
pass
- with opener.open(r, timeout=30) as response:
+ with opener.open(r, timeout=100) as response:
pass
- except (urllib.error.URLError, ConnectionResetError, TimeoutError) as e:
- if try_again:
- logger.debug2("checkstatus: trying again")
+ except (urllib.error.URLError, OSError, http.client.RemoteDisconnected) as e:
+ if try_again and getattr(e, 'code', None) != 404:
+ logger.debug2("checkstatus: trying again after exception %s" % str(e))
return self.checkstatus(fetch, ud, d, False)
else:
# debug for now to avoid spamming the logs in e.g. remote sstate searches
- logger.debug2("checkstatus() urlopen failed for %s: %s" % (uri,e))
+ logger.debug2("checkstatus() urlopen failed for %s: %s" % (uri, e))
+ if "Bad file descriptor" in str(e):
+ files = os.listdir("/proc/self/fd")
+ logger.warn("listing of /proc/self/fd (%d):" % len(files) + str(files))
+ logger.warn("lsof output: " + subprocess.check_output(["lsof", "-p", str(os.getpid())]))
+ logger.warn("checkstatus() urlopen failed for %s: %s" % (uri, e))
return False
return True
@@ -454,19 +562,17 @@ class Wget(FetchMethod):
"""
Run fetch checkstatus to get directory information
"""
- f = tempfile.NamedTemporaryFile()
with tempfile.TemporaryDirectory(prefix="wget-index-") as workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="wget-listing-") as f:
- fetchcmd = self.basecmd
- fetchcmd += " -O " + f.name + " --user-agent='" + self.user_agent + "' '" + uri + "'"
+ fetchcmd = self.basecmd + ["--output-document=%s" % f.name, uri]
try:
self._runwget(ud, d, fetchcmd, True, workdir=workdir)
fetchresult = f.read()
- except bb.fetch2.BBFetchException:
+ except bb.fetch.BBFetchException:
fetchresult = ""
return fetchresult
- def _check_latest_version(self, url, package, package_regex, current_version, ud, d):
+ def _check_latest_version(self, url, package, package_regex, current_version, ud, d, filter_regex=None):
"""
Return the latest version of a package inside a given directory path
If error or no version, return ""
@@ -489,6 +595,11 @@ class Wget(FetchMethod):
newver = self._parse_path(package_regex, str(line))
if newver:
+ if filter_regex:
+ # filter_regex is derived from PV, and uses . as separator
+ # while upstream may be using _ as separator
+ if not re.match(filter_regex, re.sub('_', '.', newver[1])):
+ continue
bb.debug(3, "Upstream version found: %s" % newver[1])
if valid == 0:
version = newver
@@ -506,14 +617,14 @@ class Wget(FetchMethod):
return ""
- def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d):
+ def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d, filter_regex=None):
"""
Scan every directory in order to get upstream version.
"""
version_dir = ['', '', '']
version = ['', '', '']
- dirver_regex = re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])*(\d+))")
+ dirver_regex = re.compile(r"^(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])*(\d+))$")
s = dirver_regex.search(dirver)
if s:
version_dir[1] = s.group('ver')
@@ -550,11 +661,10 @@ class Wget(FetchMethod):
ud.user, ud.pswd, {}])
pupver = self._check_latest_version(uri,
- package, package_regex, current_version, ud, d)
+ package, package_regex, current_version, ud, d, filter_regex)
if pupver:
version[1] = pupver
-
- version_dir = version_dir_new
+ version_dir = version_dir_new
return version[1]
@@ -611,19 +721,23 @@ class Wget(FetchMethod):
return package_custom_regex_comp
- def latest_versionstring(self, ud, d):
+ def latest_versionstring(self, ud, d, filter_regex=None):
"""
Manipulate the URL and try to obtain the latest package version
sanity check to ensure same name and type.
"""
- package = ud.path.split("/")[-1]
+ if 'downloadfilename' in ud.parm:
+ package = ud.parm['downloadfilename']
+ else:
+ package = ud.path.split("/")[-1]
current_version = ['', d.getVar('PV'), '']
"""possible to have no version in pkg name, such as spectrum-fw"""
if not re.search(r"\d+", package):
current_version[1] = re.sub('_', '.', current_version[1])
current_version[1] = re.sub('-', '.', current_version[1])
+ bb.debug(3, "latest_versionstring: no version found in %s" % package)
return (current_version[1], '')
package_regex = self._init_regexes(package, ud, d)
@@ -648,11 +762,11 @@ class Wget(FetchMethod):
dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn)))
if not dirver_pn_regex.search(dirver):
return (self._check_latest_version_by_dir(dirver,
- package, package_regex, current_version, ud, d), '')
+ package, package_regex, current_version, ud, d, filter_regex), '')
uri = bb.fetch.encodeurl([ud.type, ud.host, path, ud.user, ud.pswd, {}])
else:
uri = regex_uri
return (self._check_latest_version(uri, package, package_regex,
- current_version, ud, d), '')
+ current_version, ud, d, filter_regex), '')
diff --git a/bitbake/lib/bb/fetch2.py b/bitbake/lib/bb/fetch2.py
new file mode 100644
index 00000000..e6401701
--- /dev/null
+++ b/bitbake/lib/bb/fetch2.py
@@ -0,0 +1,19 @@
+#
+# Copyright (C) 2026 Agilent Technologies, Inc.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import sys
+import warnings
+
+import bb.fetch
+
+warnings.warn(
+ "bb.fetch2 is deprecated; use bb.fetch instead",
+ DeprecationWarning,
+ stacklevel=2,
+)
+
+# Compatibility shim for code using bb.fetch2
+sys.modules[__name__] = bb.fetch
diff --git a/bitbake/lib/bb/fetch2/bzr.py b/bitbake/lib/bb/fetch2/bzr.py
deleted file mode 100644
index fc558f50..00000000
--- a/bitbake/lib/bb/fetch2/bzr.py
+++ /dev/null
@@ -1,128 +0,0 @@
-"""
-BitBake 'Fetch' implementation for bzr.
-
-"""
-
-# Copyright (C) 2007 Ross Burton
-# Copyright (C) 2007 Richard Purdie
-#
-# Classes for obtaining upstream sources for the
-# BitBake build tools.
-# Copyright (C) 2003, 2004 Chris Larson
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import os
-import bb
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
-
-class Bzr(FetchMethod):
- def supports(self, ud, d):
- return ud.type in ['bzr']
-
- def urldata_init(self, ud, d):
- """
- init bzr specific variable within url data
- """
- # Create paths to bzr checkouts
- bzrdir = d.getVar("BZRDIR") or (d.getVar("DL_DIR") + "/bzr")
- relpath = self._strip_leading_slashes(ud.path)
- ud.pkgdir = os.path.join(bzrdir, ud.host, relpath)
-
- ud.setup_revisions(d)
-
- if not ud.revision:
- ud.revision = self.latest_revision(ud, d)
-
- ud.localfile = d.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision))
-
- def _buildbzrcommand(self, ud, d, command):
- """
- Build up an bzr commandline based on ud
- command is "fetch", "update", "revno"
- """
-
- basecmd = d.getVar("FETCHCMD_bzr") or "/usr/bin/env bzr"
-
- proto = ud.parm.get('protocol', 'http')
-
- bzrroot = ud.host + ud.path
-
- options = []
-
- if command == "revno":
- bzrcmd = "%s revno %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
- else:
- if ud.revision:
- options.append("-r %s" % ud.revision)
-
- if command == "fetch":
- bzrcmd = "%s branch %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
- elif command == "update":
- bzrcmd = "%s pull %s --overwrite" % (basecmd, " ".join(options))
- else:
- raise FetchError("Invalid bzr command %s" % command, ud.url)
-
- return bzrcmd
-
- def download(self, ud, d):
- """Fetch url"""
-
- if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK):
- bzrcmd = self._buildbzrcommand(ud, d, "update")
- logger.debug("BZR Update %s", ud.url)
- bb.fetch2.check_network_access(d, bzrcmd, ud.url)
- runfetchcmd(bzrcmd, d, workdir=os.path.join(ud.pkgdir, os.path.basename(ud.path)))
- else:
- bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
- bzrcmd = self._buildbzrcommand(ud, d, "fetch")
- bb.fetch2.check_network_access(d, bzrcmd, ud.url)
- logger.debug("BZR Checkout %s", ud.url)
- bb.utils.mkdirhier(ud.pkgdir)
- logger.debug("Running %s", bzrcmd)
- runfetchcmd(bzrcmd, d, workdir=ud.pkgdir)
-
- scmdata = ud.parm.get("scmdata", "")
- if scmdata == "keep":
- tar_flags = ""
- else:
- tar_flags = "--exclude='.bzr' --exclude='.bzrtags'"
-
- # tar them up to a defined filename
- runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(ud.pkgdir)),
- d, cleanup=[ud.localpath], workdir=ud.pkgdir)
-
- def supports_srcrev(self):
- return True
-
- def _revision_key(self, ud, d, name):
- """
- Return a unique key for the url
- """
- return "bzr:" + ud.pkgdir
-
- def _latest_revision(self, ud, d, name):
- """
- Return the latest upstream revision number
- """
- logger.debug2("BZR fetcher hitting network for %s", ud.url)
-
- bb.fetch2.check_network_access(d, self._buildbzrcommand(ud, d, "revno"), ud.url)
-
- output = runfetchcmd(self._buildbzrcommand(ud, d, "revno"), d, True)
-
- return output.strip()
-
- def sortable_revision(self, ud, d, name):
- """
- Return a sortable revision number which in our case is the revision number
- """
-
- return False, self._build_revision(ud, d)
-
- def _build_revision(self, ud, d):
- return ud.revision
diff --git a/bitbake/lib/bb/fetch2/crate.py b/bitbake/lib/bb/fetch2/crate.py
deleted file mode 100644
index 01d49435..00000000
--- a/bitbake/lib/bb/fetch2/crate.py
+++ /dev/null
@@ -1,141 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementation for crates.io
-"""
-
-# Copyright (C) 2016 Doug Goldstein
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import hashlib
-import json
-import os
-import subprocess
-import bb
-from bb.fetch2 import logger, subprocess_setup, UnpackError
-from bb.fetch2.wget import Wget
-
-
-class Crate(Wget):
-
- """Class to fetch crates via wget"""
-
- def _cargo_bitbake_path(self, rootdir):
- return os.path.join(rootdir, "cargo_home", "bitbake")
-
- def supports(self, ud, d):
- """
- Check to see if a given url is for this fetcher
- """
- return ud.type in ['crate']
-
- def recommends_checksum(self, urldata):
- return True
-
- def urldata_init(self, ud, d):
- """
- Sets up to download the respective crate from crates.io
- """
-
- if ud.type == 'crate':
- self._crate_urldata_init(ud, d)
-
- super(Crate, self).urldata_init(ud, d)
-
- def _crate_urldata_init(self, ud, d):
- """
- Sets up the download for a crate
- """
-
- # URL syntax is: crate://NAME/VERSION
- # break the URL apart by /
- parts = ud.url.split('/')
- if len(parts) < 5:
- raise bb.fetch2.ParameterError("Invalid URL: Must be crate://HOST/NAME/VERSION", ud.url)
-
- # version is expected to be the last token
- # but ignore possible url parameters which will be used
- # by the top fetcher class
- version = parts[-1].split(";")[0]
- # second to last field is name
- name = parts[-2]
- # host (this is to allow custom crate registries to be specified
- host = '/'.join(parts[2:-2])
-
- # if using upstream just fix it up nicely
- if host == 'crates.io':
- host = 'crates.io/api/v1/crates'
-
- ud.url = "https://%s/%s/%s/download" % (host, name, version)
- ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
- if 'name' not in ud.parm:
- ud.parm['name'] = '%s-%s' % (name, version)
-
- logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
-
- def unpack(self, ud, rootdir, d):
- """
- Uses the crate to build the necessary paths for cargo to utilize it
- """
- if ud.type == 'crate':
- return self._crate_unpack(ud, rootdir, d)
- else:
- super(Crate, self).unpack(ud, rootdir, d)
-
- def _crate_unpack(self, ud, rootdir, d):
- """
- Unpacks a crate
- """
- thefile = ud.localpath
-
- # possible metadata we need to write out
- metadata = {}
-
- # change to the rootdir to unpack but save the old working dir
- save_cwd = os.getcwd()
- os.chdir(rootdir)
-
- bp = d.getVar('BP')
- if bp == ud.parm.get('name'):
- cmd = "tar -xz --no-same-owner -f %s" % thefile
- ud.unpack_tracer.unpack("crate-extract", rootdir)
- else:
- cargo_bitbake = self._cargo_bitbake_path(rootdir)
- ud.unpack_tracer.unpack("cargo-extract", cargo_bitbake)
-
- cmd = "tar -xz --no-same-owner -f %s -C %s" % (thefile, cargo_bitbake)
-
- # ensure we've got these paths made
- bb.utils.mkdirhier(cargo_bitbake)
-
- # generate metadata necessary
- with open(thefile, 'rb') as f:
- # get the SHA256 of the original tarball
- tarhash = hashlib.sha256(f.read()).hexdigest()
-
- metadata['files'] = {}
- metadata['package'] = tarhash
-
- path = d.getVar('PATH')
- if path:
- cmd = "PATH=\"%s\" %s" % (path, cmd)
- bb.note("Unpacking %s to %s/" % (thefile, os.getcwd()))
-
- ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True)
-
- os.chdir(save_cwd)
-
- if ret != 0:
- raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), ud.url)
-
- # if we have metadata to write out..
- if len(metadata) > 0:
- cratepath = os.path.splitext(os.path.basename(thefile))[0]
- bbpath = self._cargo_bitbake_path(rootdir)
- mdfile = '.cargo-checksum.json'
- mdpath = os.path.join(bbpath, cratepath, mdfile)
- with open(mdpath, "w") as f:
- json.dump(metadata, f)
diff --git a/bitbake/lib/bb/fetch2/cvs.py b/bitbake/lib/bb/fetch2/cvs.py
deleted file mode 100644
index 01de5ff4..00000000
--- a/bitbake/lib/bb/fetch2/cvs.py
+++ /dev/null
@@ -1,157 +0,0 @@
-"""
-BitBake 'Fetch' implementations
-
-Classes for obtaining upstream sources for the
-BitBake build tools.
-
-"""
-
-# Copyright (C) 2003, 2004 Chris Larson
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-#
-
-import os
-import bb
-from bb.fetch2 import FetchMethod, FetchError, MissingParameterError, logger
-from bb.fetch2 import runfetchcmd
-
-class Cvs(FetchMethod):
- """
- Class to fetch a module or modules from cvs repositories
- """
- def supports(self, ud, d):
- """
- Check to see if a given url can be fetched with cvs.
- """
- return ud.type in ['cvs']
-
- def urldata_init(self, ud, d):
- if not "module" in ud.parm:
- raise MissingParameterError("module", ud.url)
- ud.module = ud.parm["module"]
-
- ud.tag = ud.parm.get('tag', "")
-
- # Override the default date in certain cases
- if 'date' in ud.parm:
- ud.date = ud.parm['date']
- elif ud.tag:
- ud.date = ""
-
- norecurse = ''
- if 'norecurse' in ud.parm:
- norecurse = '_norecurse'
-
- fullpath = ''
- if 'fullpath' in ud.parm:
- fullpath = '_fullpath'
-
- ud.localfile = d.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath))
-
- pkg = d.getVar('PN')
- cvsdir = d.getVar("CVSDIR") or (d.getVar("DL_DIR") + "/cvs")
- ud.pkgdir = os.path.join(cvsdir, pkg)
-
- def need_update(self, ud, d):
- if (ud.date == "now"):
- return True
- if not os.path.exists(ud.localpath):
- return True
- return False
-
- def download(self, ud, d):
-
- method = ud.parm.get('method', 'pserver')
- localdir = ud.parm.get('localdir', ud.module)
- cvs_port = ud.parm.get('port', '')
-
- cvs_rsh = None
- if method == "ext":
- if "rsh" in ud.parm:
- cvs_rsh = ud.parm["rsh"]
-
- if method == "dir":
- cvsroot = ud.path
- else:
- cvsroot = ":" + method
- cvsproxyhost = d.getVar('CVS_PROXY_HOST')
- if cvsproxyhost:
- cvsroot += ";proxy=" + cvsproxyhost
- cvsproxyport = d.getVar('CVS_PROXY_PORT')
- if cvsproxyport:
- cvsroot += ";proxyport=" + cvsproxyport
- cvsroot += ":" + ud.user
- if ud.pswd:
- cvsroot += ":" + ud.pswd
- cvsroot += "@" + ud.host + ":" + cvs_port + ud.path
-
- options = []
- if 'norecurse' in ud.parm:
- options.append("-l")
- if ud.date:
- # treat YYYYMMDDHHMM specially for CVS
- if len(ud.date) == 12:
- options.append("-D \"%s %s:%s UTC\"" % (ud.date[0:8], ud.date[8:10], ud.date[10:12]))
- else:
- options.append("-D \"%s UTC\"" % ud.date)
- if ud.tag:
- options.append("-r %s" % ud.tag)
-
- cvsbasecmd = d.getVar("FETCHCMD_cvs") or "/usr/bin/env cvs"
- cvscmd = cvsbasecmd + " '-d" + cvsroot + "' co " + " ".join(options) + " " + ud.module
- cvsupdatecmd = cvsbasecmd + " '-d" + cvsroot + "' update -d -P " + " ".join(options)
-
- if cvs_rsh:
- cvscmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvscmd)
- cvsupdatecmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvsupdatecmd)
-
- # create module directory
- logger.debug2("Fetch: checking for module directory")
- moddir = os.path.join(ud.pkgdir, localdir)
- workdir = None
- if os.access(os.path.join(moddir, 'CVS'), os.R_OK):
- logger.info("Update " + ud.url)
- bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url)
- # update sources there
- workdir = moddir
- cmd = cvsupdatecmd
- else:
- logger.info("Fetch " + ud.url)
- # check out sources there
- bb.utils.mkdirhier(ud.pkgdir)
- workdir = ud.pkgdir
- logger.debug("Running %s", cvscmd)
- bb.fetch2.check_network_access(d, cvscmd, ud.url)
- cmd = cvscmd
-
- runfetchcmd(cmd, d, cleanup=[moddir], workdir=workdir)
-
- if not os.access(moddir, os.R_OK):
- raise FetchError("Directory %s was not readable despite sucessful fetch?!" % moddir, ud.url)
-
- scmdata = ud.parm.get("scmdata", "")
- if scmdata == "keep":
- tar_flags = ""
- else:
- tar_flags = "--exclude='CVS'"
-
- # tar them up to a defined filename
- workdir = None
- if 'fullpath' in ud.parm:
- workdir = ud.pkgdir
- cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir)
- else:
- workdir = os.path.dirname(os.path.realpath(moddir))
- cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir))
-
- runfetchcmd(cmd, d, cleanup=[ud.localpath], workdir=workdir)
-
- def clean(self, ud, d):
- """ Clean CVS Files and tarballs """
-
- bb.utils.remove(ud.pkgdir, True)
- bb.utils.remove(ud.localpath)
-
diff --git a/bitbake/lib/bb/fetch2/osc.py b/bitbake/lib/bb/fetch2/osc.py
deleted file mode 100644
index 495ac8a3..00000000
--- a/bitbake/lib/bb/fetch2/osc.py
+++ /dev/null
@@ -1,165 +0,0 @@
-#
-# Copyright BitBake Contributors
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-"""
-Bitbake "Fetch" implementation for osc (Opensuse build service client).
-Based on the svn "Fetch" implementation.
-
-"""
-
-import logging
-import os
-import re
-import bb
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import runfetchcmd
-
-logger = logging.getLogger(__name__)
-
-class Osc(FetchMethod):
- """Class to fetch a module or modules from Opensuse build server
- repositories."""
-
- def supports(self, ud, d):
- """
- Check to see if a given url can be fetched with osc.
- """
- return ud.type in ['osc']
-
- def urldata_init(self, ud, d):
- if not "module" in ud.parm:
- raise MissingParameterError('module', ud.url)
-
- ud.module = ud.parm["module"]
-
- # Create paths to osc checkouts
- oscdir = d.getVar("OSCDIR") or (d.getVar("DL_DIR") + "/osc")
- relpath = self._strip_leading_slashes(ud.path)
- ud.oscdir = oscdir
- ud.pkgdir = os.path.join(oscdir, ud.host)
- ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module)
-
- if 'rev' in ud.parm:
- ud.revision = ud.parm['rev']
- else:
- pv = d.getVar("PV", False)
- rev = bb.fetch2.srcrev_internal_helper(ud, d, '')
- if rev:
- ud.revision = rev
- else:
- ud.revision = ""
-
- ud.localfile = d.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), relpath.replace('/', '.'), ud.revision))
-
- def _buildosccommand(self, ud, d, command):
- """
- Build up an ocs commandline based on ud
- command is "fetch", "update", "info"
- """
-
- basecmd = d.getVar("FETCHCMD_osc") or "/usr/bin/env osc"
-
- proto = ud.parm.get('protocol', 'https')
-
- options = []
-
- config = "-c %s" % self.generate_config(ud, d)
-
- if getattr(ud, 'revision', ''):
- options.append("-r %s" % ud.revision)
-
- coroot = self._strip_leading_slashes(ud.path)
-
- if command == "fetch":
- osccmd = "%s %s -A %s://%s co %s/%s %s" % (basecmd, config, proto, ud.host, coroot, ud.module, " ".join(options))
- elif command == "update":
- osccmd = "%s %s -A %s://%s up %s" % (basecmd, config, proto, ud.host, " ".join(options))
- elif command == "api_source":
- osccmd = "%s %s -A %s://%s api source/%s/%s" % (basecmd, config, proto, ud.host, coroot, ud.module)
- else:
- raise FetchError("Invalid osc command %s" % command, ud.url)
-
- return osccmd
-
- def _latest_revision(self, ud, d, name):
- """
- Fetch latest revision for the given package
- """
- api_source_cmd = self._buildosccommand(ud, d, "api_source")
-
- output = runfetchcmd(api_source_cmd, d)
- match = re.match(r'<directory ?.* rev="(\d+)".*>', output)
- if match is None:
- raise FetchError("Unable to parse osc response", ud.url)
- return match.groups()[0]
-
- def _revision_key(self, ud, d, name):
- """
- Return a unique key for the url
- """
- # Collapse adjacent slashes
- slash_re = re.compile(r"/+")
- rev = getattr(ud, 'revision', "latest")
- return "osc:%s%s.%s.%s" % (ud.host, slash_re.sub(".", ud.path), name, rev)
-
- def download(self, ud, d):
- """
- Fetch url
- """
-
- logger.debug2("Fetch: checking for module directory '" + ud.moddir + "'")
-
- if os.access(ud.moddir, os.R_OK):
- oscupdatecmd = self._buildosccommand(ud, d, "update")
- logger.info("Update "+ ud.url)
- # update sources there
- logger.debug("Running %s", oscupdatecmd)
- bb.fetch2.check_network_access(d, oscupdatecmd, ud.url)
- runfetchcmd(oscupdatecmd, d, workdir=ud.moddir)
- else:
- oscfetchcmd = self._buildosccommand(ud, d, "fetch")
- logger.info("Fetch " + ud.url)
- # check out sources there
- bb.utils.mkdirhier(ud.pkgdir)
- logger.debug("Running %s", oscfetchcmd)
- bb.fetch2.check_network_access(d, oscfetchcmd, ud.url)
- runfetchcmd(oscfetchcmd, d, workdir=ud.pkgdir)
-
- # tar them up to a defined filename
- runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d,
- cleanup=[ud.localpath], workdir=os.path.join(ud.pkgdir + ud.path))
-
- def supports_srcrev(self):
- return False
-
- def generate_config(self, ud, d):
- """
- Generate a .oscrc to be used for this run.
- """
-
- config_path = os.path.join(ud.oscdir, "oscrc")
- if not os.path.exists(ud.oscdir):
- bb.utils.mkdirhier(ud.oscdir)
-
- if (os.path.exists(config_path)):
- os.remove(config_path)
-
- f = open(config_path, 'w')
- proto = ud.parm.get('protocol', 'https')
- f.write("[general]\n")
- f.write("apiurl = %s://%s\n" % (proto, ud.host))
- f.write("su-wrapper = su -c\n")
- f.write("build-root = %s\n" % d.getVar('WORKDIR'))
- f.write("urllist = %s\n" % d.getVar("OSCURLLIST"))
- f.write("extra-pkgs = gzip\n")
- f.write("\n")
- f.write("[%s://%s]\n" % (proto, ud.host))
- f.write("user = %s\n" % ud.parm["user"])
- f.write("pass = %s\n" % ud.parm["pswd"])
- f.close()
-
- return config_path
diff --git a/bitbake/lib/bb/filter.py b/bitbake/lib/bb/filter.py
new file mode 100644
index 00000000..0b5b5d92
--- /dev/null
+++ b/bitbake/lib/bb/filter.py
@@ -0,0 +1,142 @@
+#
+# Copyright (C) 2025 Garmin Ltd. or its subsidiaries
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import builtins
+
+# Purposely blank out __builtins__ which prevents users from
+# calling any normal builtin python functions
+FILTERS = {
+ "__builtins__": {},
+}
+
+CACHE = {}
+
+
+def apply_filters(val, expressions):
+ g = FILTERS.copy()
+
+ for e in expressions:
+ e = e.strip()
+ if not e:
+ continue
+
+ k = (val, e)
+ if k not in CACHE:
+ # Set val as a local so it can be cleared out while keeping the
+ # globals
+ l = {"val": val}
+
+ CACHE[k] = eval(e, g, l)
+
+ val = CACHE[k]
+
+ return val
+
+
+class Namespace(object):
+ """
+ Helper class to simulate a python namespace. The object properties can be
+ set as if it were a dictionary. Properties cannot be changed or deleted
+ through the object interface
+ """
+
+ def __getitem__(self, name):
+ return self.__dict__[name]
+
+ def __setitem__(self, name, value):
+ self.__dict__[name] = value
+
+ def __contains__(self, name):
+ return name in self.__dict__
+
+ def __setattr__(self, name, value):
+ raise AttributeError(f"Attribute {name!r} cannot be changed")
+
+ def __delattr__(self, name):
+ raise AttributeError(f"Attribute {name!r} cannot be deleted")
+
+
+def filter_proc(*, name=None):
+ """
+ Decorator to mark a function that can be called in `apply_filters`, either
+ directly in a filter expression, or indirectly. The `name` argument can be
+ used to specify an alternate name for the function if the actual name is
+ not desired. The `name` can be a fully qualified namespace if desired.
+
+ All functions must be "pure" in that they do not depend on global state and
+ have no global side effects (e.g. the output only depends on the input
+ arguments); the results of filter expressions are cached to optimize
+ repeated calls.
+ """
+
+ def inner(func):
+ global FILTERS
+ nonlocal name
+
+ if name is None:
+ name = func.__name__
+
+ ns = name.split(".")
+ o = FILTERS
+ for n in ns[:-1]:
+ if not n in o:
+ o[n] = Namespace()
+ o = o[n]
+
+ o[ns[-1]] = func
+
+ return func
+
+ return inner
+
+
+# A select set of builtins that are supported in filter expressions
+filter_proc()(all)
+filter_proc()(all)
+filter_proc()(any)
+filter_proc()(bin)
+filter_proc()(bool)
+filter_proc()(chr)
+filter_proc()(enumerate)
+filter_proc()(float)
+filter_proc()(format)
+filter_proc()(hex)
+filter_proc()(int)
+filter_proc()(len)
+filter_proc()(map)
+filter_proc()(max)
+filter_proc()(min)
+filter_proc()(oct)
+filter_proc()(ord)
+filter_proc()(pow)
+filter_proc()(str)
+filter_proc()(sum)
+
+
+@filter_proc()
+def suffix(val, suffix):
+ return " ".join(v + suffix for v in val.split())
+
+
+@filter_proc()
+def prefix(val, prefix):
+ return " ".join(prefix + v for v in val.split())
+
+
+@filter_proc()
+def sort(val):
+ return " ".join(sorted(val.split()))
+
+
+@filter_proc()
+def remove(val, remove, sep=None):
+ if isinstance(remove, str):
+ remove = remove.split(sep)
+ new = [i for i in val.split(sep) if not i in remove]
+
+ if not sep:
+ return " ".join(new)
+ return sep.join(new)
diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index bca8ebfa..73a66468 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -208,8 +208,10 @@ def create_bitbake_parser():
"failed and anything depending on it cannot be built, as much as "
"possible will be built before stopping.")
- exec_group.add_argument("-P", "--profile", action="store_true",
- help="Profile the command and save reports.")
+ exec_group.add_argument("-P", "--profile", action="append",
+ default=[],
+ help="Profile the command and save reports. Specify 'main', 'idle' or 'parsing' "
+ "to indicate which bitbake code to profile.")
exec_group.add_argument("-S", "--dump-signatures", action="append",
default=[], metavar="SIGNATURE_HANDLER",
@@ -314,7 +316,7 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
options = parser.parse_intermixed_args(argv[1:])
if options.version:
- print("BitBake Build Tool Core version %s" % bb.__version__)
+ print("BitBake Build Tool version %s" % bb.__version__)
sys.exit(0)
if options.quiet and options.verbose:
diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index a4358f13..d428d8a4 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -49,20 +49,23 @@ class SkipPackage(SkipRecipe):
__mtime_cache = {}
def cached_mtime(f):
if f not in __mtime_cache:
- __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+ res = os.stat(f)
+ __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino)
return __mtime_cache[f]
def cached_mtime_noerror(f):
if f not in __mtime_cache:
try:
- __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+ res = os.stat(f)
+ __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino)
except OSError:
return 0
return __mtime_cache[f]
def check_mtime(f, mtime):
try:
- current_mtime = os.stat(f)[stat.ST_MTIME]
+ res = os.stat(f)
+ current_mtime = (res.st_mtime_ns, res.st_size, res.st_ino)
__mtime_cache[f] = current_mtime
except OSError:
current_mtime = 0
@@ -70,7 +73,8 @@ def check_mtime(f, mtime):
def update_mtime(f):
try:
- __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+ res = os.stat(f)
+ __mtime_cache[f] = (res.st_mtime_ns, res.st_size, res.st_ino)
except OSError:
if f in __mtime_cache:
del __mtime_cache[f]
@@ -172,4 +176,41 @@ def get_file_depends(d):
dep_files.append(os.path.abspath(fn))
return " ".join(dep_files)
+def vardeps(*varnames):
+ """
+ Function decorator that can be used to instruct the bitbake dependency
+ parsing to add a dependency on the specified variables names
+
+ Example:
+
+ @bb.parse.vardeps("FOO", "BAR")
+ def my_function():
+ ...
+
+ """
+ def inner(f):
+ if not hasattr(f, "bb_vardeps"):
+ f.bb_vardeps = set()
+ f.bb_vardeps |= set(varnames)
+ return f
+ return inner
+
+def vardepsexclude(*varnames):
+ """
+ Function decorator that can be used to instruct the bitbake dependency
+ parsing to ignore dependencies on the specified variable names in the code
+
+ Example:
+
+ @bb.parse.vardepsexclude("FOO", "BAR")
+ def my_function():
+ ...
+ """
+ def inner(f):
+ if not hasattr(f, "bb_vardepsexclude"):
+ f.bb_vardepsexclude = set()
+ f.bb_vardepsexclude |= set(varnames)
+ return f
+ return inner
+
from bb.parse.parse_py import __version__, ConfHandler, BBHandler
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 7581d003..5f2aaadf 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -43,6 +43,21 @@ class IncludeNode(AstNode):
else:
bb.parse.ConfHandler.include(self.filename, s, self.lineno, data, False)
+class IncludeAllNode(AstNode):
+ def __init__(self, filename, lineno, what_file):
+ AstNode.__init__(self, filename, lineno)
+ self.what_file = what_file
+
+ def eval(self, data):
+ """
+ Include the file and evaluate the statements
+ """
+ s = data.expand(self.what_file)
+ logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s)
+
+ for path in data.getVar("BBPATH").split(":"):
+ bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False)
+
class ExportNode(AstNode):
def __init__(self, filename, lineno, var):
AstNode.__init__(self, filename, lineno)
@@ -137,7 +152,10 @@ class DataNode(AstNode):
flag = None
if 'flag' in groupd and groupd['flag'] is not None:
- flag = groupd['flag']
+ if groupd["lazyques"]:
+ flag = "_defaultval_flag_"+groupd['flag']
+ else:
+ flag = groupd['flag']
elif groupd["lazyques"]:
flag = "_defaultval"
@@ -240,14 +258,16 @@ class ExportFuncsNode(AstNode):
data.setVar(func, sentinel + " " + calledfunc + "\n", parsing=True)
class AddTaskNode(AstNode):
- def __init__(self, filename, lineno, func, before, after):
+ def __init__(self, filename, lineno, tasks, before, after):
AstNode.__init__(self, filename, lineno)
- self.func = func
+ self.tasks = tasks
self.before = before
self.after = after
def eval(self, data):
- bb.build.addtask(self.func, self.before, self.after, data)
+ tasks = self.tasks.split()
+ for task in tasks:
+ bb.build.addtask(task, self.before, self.after, data)
class DelTaskNode(AstNode):
def __init__(self, filename, lineno, tasks):
@@ -290,8 +310,9 @@ class PyLibNode(AstNode):
bb.utils._context[self.namespace] = __import__(self.namespace)
toimport = getattr(bb.utils._context[self.namespace], "BBIMPORTS", [])
for i in toimport:
- bb.utils._context[self.namespace] = __import__(self.namespace + "." + i)
- mod = getattr(bb.utils._context[self.namespace], i)
+ mod = bb.utils._context[self.namespace] = __import__(self.namespace + "." + i)
+ for submod in i.split("."):
+ mod = getattr(mod, submod)
fn = getattr(mod, "__file__")
funcs = {}
for f in dir(mod):
@@ -320,13 +341,101 @@ class InheritDeferredNode(AstNode):
self.inherit = (classes, filename, lineno)
def eval(self, data):
- inherits = data.getVar('__BBDEFINHERITS', False) or []
- inherits.append(self.inherit)
- data.setVar('__BBDEFINHERITS', inherits)
+ bb.parse.BBHandler.inherit_defer(*self.inherit, data)
+
+class AddFragmentsNode(AstNode):
+ def __init__(self, filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable):
+ AstNode.__init__(self, filename, lineno)
+ self.fragments_path_prefix = fragments_path_prefix
+ self.fragments_variable = fragments_variable
+ self.flagged_variables_list_variable = flagged_variables_list_variable
+ self.builtin_fragments_variable = builtin_fragments_variable
+
+ def eval(self, data):
+ if data.getVar('_BB_SKIP_FRAGMENTS') == '1':
+ return
+
+ # No need to use mark_dependency since we would only match a fragment
+ # from a specific layer and there can only be a single layer with a
+ # given namespace.
+ def find_fragment(layers, layerid, full_fragment_name):
+ for layerpath in layers.split():
+ candidate_fragment_path = os.path.join(layerpath, full_fragment_name)
+ if os.path.exists(candidate_fragment_path) and bb.utils.get_file_layer(candidate_fragment_path, data) == layerid:
+ return candidate_fragment_path
+ return None
+
+ def check_and_set_builtin_fragment(fragment, data, builtin_fragments):
+ prefix, value = fragment.split('/', 1)
+ if prefix in builtin_fragments.keys():
+ if data.getVar(builtin_fragments[prefix], noweakdefault=True) != None:
+ bb.fatal(
+ ("A builtin fragment '%s' is used while %s has already got an assignment.\n"
+ "Please either disable the fragment or remove the value assignment.\n"
+ "To disable the fragment, use 'bitbake-config-build disable-fragment %s'."
+ ) % (fragment, builtin_fragments[prefix], fragment))
+ fragment_history = data.varhistory.variable(self.fragments_variable)
+ loginfo={}
+ for fh in fragment_history[::-1]:
+ if fh['op'] in ("set", "append") and fragment in fh["detail"]:
+ loginfo["file"] = fh["file"]
+ loginfo["line"] = fh["line"]
+ loginfo["detail"] = f"{value} ({self.fragments_variable} contains \"{fragment}\")"
+ break
+ # parsing=True since we want to emulate X=Y and allow X:override=Z to continue to exist
+ data.setVar(builtin_fragments[prefix], value, parsing=True, **loginfo)
+ return True
+ return False
+
+ fragments = data.getVar(self.fragments_variable)
+ layers = data.getVar('BBLAYERS')
+ flagged_variables = data.getVar(self.flagged_variables_list_variable).split()
+ builtin_fragments = {f[0]:f[1] for f in [f.split(':') for f in data.getVar(self.builtin_fragments_variable).split()] }
+
+ if not fragments:
+ return
+
+ # Check for multiple builtin fragments setting the same variable
+ for builtin_fragment_key in builtin_fragments.keys():
+ builtin_fragments_list = list(
+ filter(
+ lambda f: f.startswith(builtin_fragment_key + "/"),
+ fragments.split(),
+ )
+ )
+ if len(builtin_fragments_list) > 1:
+ bb.warn(
+ ("Multiple builtin fragments are enabled for %s via variable %s: %s. "
+ "This likely points to a mis-configuration in the metadata, as only "
+ "one of them should be set. The build will use the last value.")
+ % (
+ builtin_fragment_key,
+ self.fragments_variable,
+ " ".join(builtin_fragments_list),
+ )
+ )
+
+ for f in fragments.split():
+ if check_and_set_builtin_fragment(f, data, builtin_fragments):
+ continue
+ layerid, fragment_name = f.split('/', 1)
+ full_fragment_name = data.expand("{}/{}.conf".format(self.fragments_path_prefix, fragment_name))
+ fragment_path = find_fragment(layers, layerid, full_fragment_name)
+ if fragment_path:
+ bb.parse.ConfHandler.include(self.filename, fragment_path, self.lineno, data, "include fragment")
+ for flagged_var in flagged_variables:
+ val = data.getVar(flagged_var)
+ data.setVarFlag(flagged_var, f, val)
+ data.setVar(flagged_var, None)
+ else:
+ bb.fatal("Could not find fragment {} in enabled layers:\n\t{}".format(f, "\n\t".join(layers.split())))
def handleInclude(statements, filename, lineno, m, force):
statements.append(IncludeNode(filename, lineno, m.group(1), force))
+def handleIncludeAll(statements, filename, lineno, m):
+ statements.append(IncludeAllNode(filename, lineno, m.group(1)))
+
def handleExport(statements, filename, lineno, m):
statements.append(ExportNode(filename, lineno, m.group(1)))
@@ -348,21 +457,11 @@ def handlePythonMethod(statements, filename, lineno, funcname, modulename, body)
def handleExportFuncs(statements, filename, lineno, m, classname):
statements.append(ExportFuncsNode(filename, lineno, m.group(1), classname))
-def handleAddTask(statements, filename, lineno, m):
- func = m.group("func")
- before = m.group("before")
- after = m.group("after")
- if func is None:
- return
-
- statements.append(AddTaskNode(filename, lineno, func, before, after))
+def handleAddTask(statements, filename, lineno, tasks, before, after):
+ statements.append(AddTaskNode(filename, lineno, tasks, before, after))
-def handleDelTask(statements, filename, lineno, m):
- func = m.group(1)
- if func is None:
- return
-
- statements.append(DelTaskNode(filename, lineno, func))
+def handleDelTask(statements, filename, lineno, tasks):
+ statements.append(DelTaskNode(filename, lineno, tasks))
def handleBBHandlers(statements, filename, lineno, m):
statements.append(BBHandlerNode(filename, lineno, m.group(1)))
@@ -378,12 +477,46 @@ def handleInheritDeferred(statements, filename, lineno, m):
classes = m.group(1)
statements.append(InheritDeferredNode(filename, lineno, classes))
+def handleAddFragments(statements, filename, lineno, m):
+ fragments_path_prefix = m.group(1)
+ fragments_variable = m.group(2)
+ flagged_variables_list_variable = m.group(3)
+ builtin_fragments_variable = m.group(4)
+ statements.append(AddFragmentsNode(filename, lineno, fragments_path_prefix, fragments_variable, flagged_variables_list_variable, builtin_fragments_variable))
+
def runAnonFuncs(d):
code = []
for funcname in d.getVar("__BBANONFUNCS", False) or []:
code.append("%s(d)" % funcname)
bb.utils.better_exec("\n".join(code), {"d": d})
+# Handle recipe level PREFERRED_PROVIDERs
+def handleVirtRecipeProviders(tasklist, d):
+ depends = (d.getVar("DEPENDS") or "").split()
+ virtprovs = (d.getVar("BB_RECIPE_VIRTUAL_PROVIDERS") or "").split()
+ newdeps = []
+ for dep in depends:
+ if dep in virtprovs:
+ newdep = d.getVar("PREFERRED_PROVIDER_" + dep)
+ if not newdep:
+ bb.fatal("Error, recipe virtual provider PREFERRED_PROVIDER_%s not set" % dep)
+ newdeps.append(newdep)
+ else:
+ newdeps.append(dep)
+ d.setVar("DEPENDS", " ".join(newdeps))
+ for task in tasklist:
+ taskdeps = (d.getVarFlag(task, "depends") or "").split()
+ remapped = []
+ for entry in taskdeps:
+ try:
+ r, t = entry.split(":")
+ except ValueError:
+ bb.fatal("Error, incorrect format for task dependency (task '%s'): %s" % (task, entry))
+ if r in virtprovs:
+ r = d.getVar("PREFERRED_PROVIDER_" + r)
+ remapped.append("%s:%s" % (r, t))
+ d.setVarFlag(task, "depends", " ".join(remapped))
+
def finalize(fn, d, variant = None):
saved_handlers = bb.event.get_handlers().copy()
try:
@@ -391,6 +524,17 @@ def finalize(fn, d, variant = None):
if d.getVar("_FAILPARSINGERRORHANDLED", False) == True:
raise bb.BBHandledException()
+ inherits = [x[0] for x in (d.getVar('__BBDEFINHERITS', False) or [('',)])]
+ bb.event.fire(bb.event.RecipePreDeferredInherits(fn, inherits), d)
+
+ while True:
+ inherits = d.getVar('__BBDEFINHERITS', False) or []
+ if not inherits:
+ break
+ inherit, filename, lineno = inherits.pop(0)
+ d.setVar('__BBDEFINHERITS', inherits)
+ bb.parse.BBHandler.inherit(inherit, filename, lineno, d, deferred=True)
+
for var in d.getVar('__BBHANDLERS', False) or []:
# try to add the handler
handlerfn = d.getVarFlag(var, "filename", False)
@@ -409,6 +553,7 @@ def finalize(fn, d, variant = None):
tasklist = d.getVar('__BBTASKS', False) or []
bb.event.fire(bb.event.RecipeTaskPreProcess(fn, list(tasklist)), d)
+ handleVirtRecipeProviders(tasklist, d)
bb.build.add_tasks(tasklist, d)
bb.parse.siggen.finalise(fn, d, variant)
@@ -444,14 +589,6 @@ def multi_finalize(fn, d):
logger.debug("Appending .bbappend file %s to %s", append, fn)
bb.parse.BBHandler.handle(append, d, True)
- while True:
- inherits = d.getVar('__BBDEFINHERITS', False) or []
- if not inherits:
- break
- inherit, filename, lineno = inherits.pop(0)
- d.setVar('__BBDEFINHERITS', inherits)
- bb.parse.BBHandler.inherit(inherit, filename, lineno, d, deferred=True)
-
onlyfinalise = d.getVar("__ONLYFINALISE", False)
safe_d = d
@@ -487,7 +624,7 @@ def multi_finalize(fn, d):
d.setVar("BBEXTENDVARIANT", variantmap[name])
else:
d.setVar("PN", "%s-%s" % (pn, name))
- bb.parse.BBHandler.inherit(extendedmap[name], fn, 0, d)
+ bb.parse.BBHandler.inherit_defer(extendedmap[name], fn, 0, d)
safe_d.setVar("BBCLASSEXTEND", extended)
_create_variants(datastores, extendedmap.keys(), extendfunc, onlyfinalise)
diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index c13e4b97..008fec23 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -23,8 +23,8 @@ __func_start_regexp__ = re.compile(r"(((?P<py>python(?=(\s|\()))|(?P<fr>faker
__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
__inherit_def_regexp__ = re.compile(r"inherit_defer\s+(.+)" )
__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
-__addtask_regexp__ = re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
-__deltask_regexp__ = re.compile(r"deltask\s+(.+)")
+__addtask_regexp__ = re.compile(r"addtask\s+([^#\n]+)(?P<comment>#.*|.*?)")
+__deltask_regexp__ = re.compile(r"deltask\s+([^#\n]+)(?P<comment>#.*|.*?)")
__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
@@ -42,12 +42,22 @@ def supports(fn, d):
"""Return True if fn has a supported extension"""
return os.path.splitext(fn)[-1] in [".bb", ".bbclass", ".inc"]
+def inherit_defer(expression, fn, lineno, d):
+ inherit = (expression, fn, lineno)
+ inherits = d.getVar('__BBDEFINHERITS', False) or []
+ inherits.append(inherit)
+ d.setVar('__BBDEFINHERITS', inherits)
+
def inherit(files, fn, lineno, d, deferred=False):
__inherit_cache = d.getVar('__inherit_cache', False) or []
#if "${" in files and not deferred:
# bb.warn("%s:%s has non deferred conditional inherit" % (fn, lineno))
files = d.expand(files).split()
for file in files:
+ defer = (d.getVar("BB_DEFER_BBCLASSES") or "").split()
+ if not deferred and file in defer:
+ inherit_defer(file, fn, lineno, d)
+ continue
classtype = d.getVar("__bbclasstype", False)
origfile = file
for t in ["classes-" + classtype, "classes"]:
@@ -239,29 +249,38 @@ def feeder(lineno, s, fn, root, statements, eof=False):
m = __addtask_regexp__.match(s)
if m:
- if len(m.group().split()) == 2:
- # Check and warn for "addtask task1 task2"
- m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", s)
- if m2 and m2.group('ignores'):
- logger.warning('addtask ignored: "%s"' % m2.group('ignores'))
-
- # Check and warn for "addtask task1 before task2 before task3", the
- # similar to "after"
- taskexpression = s.split()
- for word in ('before', 'after'):
- if taskexpression.count(word) > 1:
- logger.warning("addtask contained multiple '%s' keywords, only one is supported" % word)
+ after = ""
+ before = ""
+
+ # This code splits on 'before' and 'after' instead of on whitespace so we can defer
+ # evaluation to as late as possible.
+ tasks = m.group(1).split(" before ")[0].split(" after ")[0]
+
+ for exp in m.group(1).split(" before "):
+ exp2 = exp.split(" after ")
+ if len(exp2) > 1:
+ after = after + " ".join(exp2[1:])
- # Check and warn for having task with exprssion as part of task name
+ for exp in m.group(1).split(" after "):
+ exp2 = exp.split(" before ")
+ if len(exp2) > 1:
+ before = before + " ".join(exp2[1:])
+
+ # Check and warn for having task with a keyword as part of task name
+ taskexpression = s.split()
for te in taskexpression:
if any( ( "%s_" % keyword ) in te for keyword in bb.data_smart.__setvar_keyword__ ):
raise ParseError("Task name '%s' contains a keyword which is not recommended/supported.\nPlease rename the task not to include the keyword.\n%s" % (te, ("\n".join(map(str, bb.data_smart.__setvar_keyword__)))), fn)
- ast.handleAddTask(statements, fn, lineno, m)
+
+ if tasks is not None:
+ ast.handleAddTask(statements, fn, lineno, tasks, before, after)
return
m = __deltask_regexp__.match(s)
if m:
- ast.handleDelTask(statements, fn, lineno, m)
+ task = m.group(1)
+ if task is not None:
+ ast.handleDelTask(statements, fn, lineno, task)
return
m = __addhandler_regexp__.match(s)
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index 7826dee7..9ddbae12 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -20,10 +20,10 @@ from bb.parse import ParseError, resolve_file, ast, logger, handle
__config_regexp__ = re.compile( r"""
^
(?P<exp>export\s+)?
- (?P<var>[a-zA-Z0-9\-_+.${}/~:]+?)
- (\[(?P<flag>[a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]*)\])?
+ (?P<var>[a-zA-Z0-9\-_+.${}/~:]*?)
+ (\[(?P<flag>[a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@/]*)\])?
- \s* (
+ (?P<whitespace>\s*) (
(?P<colon>:=) |
(?P<lazyques>\?\?=) |
(?P<ques>\?=) |
@@ -32,7 +32,7 @@ __config_regexp__ = re.compile( r"""
(?P<predot>=\.) |
(?P<postdot>\.=) |
=
- ) \s*
+ ) (?P<whitespace2>\s*)
(?!'[^']*'[^']*'$)
(?!\"[^\"]*\"[^\"]*\"$)
@@ -43,10 +43,12 @@ __config_regexp__ = re.compile( r"""
""", re.X)
__include_regexp__ = re.compile( r"include\s+(.+)" )
__require_regexp__ = re.compile( r"require\s+(.+)" )
+__includeall_regexp__ = re.compile( r"include_all\s+(.+)" )
__export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/~]+)$" )
__unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)$" )
__unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/~]+)\[([a-zA-Z0-9\-_+.][a-zA-Z0-9\-_+.@]+)\]$" )
__addpylib_regexp__ = re.compile(r"addpylib\s+(.+)\s+(.+)" )
+__addfragments_regexp__ = re.compile(r"addfragments\s+(.+)\s+(.+)\s+(.+)\s+(.+)" )
def init(data):
return
@@ -164,6 +166,10 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True):
m = __config_regexp__.match(s)
if m:
groupd = m.groupdict()
+ if groupd['var'] == "":
+ raise ParseError("Empty variable name in assignment: '%s'" % s, fn, lineno);
+ if not groupd['whitespace'] or not groupd['whitespace2']:
+ logger.warning("%s:%s has a lack of whitespace around the assignment: '%s'" % (fn, lineno, s))
ast.handleData(statements, fn, lineno, groupd)
return
@@ -177,6 +183,11 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True):
ast.handleInclude(statements, fn, lineno, m, True)
return
+ m = __includeall_regexp__.match(s)
+ if m:
+ ast.handleIncludeAll(statements, fn, lineno, m)
+ return
+
m = __export_regexp__.match(s)
if m:
ast.handleExport(statements, fn, lineno, m)
@@ -197,6 +208,11 @@ def feeder(lineno, s, fn, statements, baseconfig=False, conffile=True):
ast.handlePyLib(statements, fn, lineno, m)
return
+ m = __addfragments_regexp__.match(s)
+ if m:
+ ast.handleAddFragments(statements, fn, lineno, m)
+ return
+
raise ParseError("unparsed line: '%s'" % s, fn, lineno);
# Add us to the handlers list
diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
deleted file mode 100644
index bcca791e..00000000
--- a/bitbake/lib/bb/persist_data.py
+++ /dev/null
@@ -1,271 +0,0 @@
-"""BitBake Persistent Data Store
-
-Used to store data in a central location such that other threads/tasks can
-access them at some future date. Acts as a convenience wrapper around sqlite,
-currently, providing a key/value store accessed by 'domain'.
-"""
-
-# Copyright (C) 2007 Richard Purdie
-# Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import collections
-import collections.abc
-import contextlib
-import functools
-import logging
-import os.path
-import sqlite3
-import sys
-from collections.abc import Mapping
-
-sqlversion = sqlite3.sqlite_version_info
-if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
- raise Exception("sqlite3 version 3.3.0 or later is required.")
-
-
-logger = logging.getLogger("BitBake.PersistData")
-
-@functools.total_ordering
-class SQLTable(collections.abc.MutableMapping):
- class _Decorators(object):
- @staticmethod
- def retry(*, reconnect=True):
- """
- Decorator that restarts a function if a database locked sqlite
- exception occurs. If reconnect is True, the database connection
- will be closed and reopened each time a failure occurs
- """
- def retry_wrapper(f):
- def wrap_func(self, *args, **kwargs):
- # Reconnect if necessary
- if self.connection is None and reconnect:
- self.reconnect()
-
- count = 0
- while True:
- try:
- return f(self, *args, **kwargs)
- except sqlite3.OperationalError as exc:
- if count < 500 and ('is locked' in str(exc) or 'locking protocol' in str(exc)):
- count = count + 1
- if reconnect:
- self.reconnect()
- continue
- raise
- return wrap_func
- return retry_wrapper
-
- @staticmethod
- def transaction(f):
- """
- Decorator that starts a database transaction and creates a database
- cursor for performing queries. If no exception is thrown, the
- database results are committed. If an exception occurs, the database
- is rolled back. In all cases, the cursor is closed after the
- function ends.
-
- Note that the cursor is passed as an extra argument to the function
- after `self` and before any of the normal arguments
- """
- def wrap_func(self, *args, **kwargs):
- # Context manager will COMMIT the database on success,
- # or ROLLBACK on an exception
- with self.connection:
- # Automatically close the cursor when done
- with contextlib.closing(self.connection.cursor()) as cursor:
- return f(self, cursor, *args, **kwargs)
- return wrap_func
-
- """Object representing a table/domain in the database"""
- def __init__(self, cachefile, table):
- self.cachefile = cachefile
- self.table = table
-
- self.connection = None
- self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table)
-
- @_Decorators.retry(reconnect=False)
- @_Decorators.transaction
- def _setup_database(self, cursor):
- cursor.execute("pragma synchronous = off;")
- # Enable WAL and keep the autocheckpoint length small (the default is
- # usually 1000). Persistent caches are usually read-mostly, so keeping
- # this short will keep readers running quickly
- cursor.execute("pragma journal_mode = WAL;")
- cursor.execute("pragma wal_autocheckpoint = 100;")
-
- def reconnect(self):
- if self.connection is not None:
- self.connection.close()
- self.connection = sqlite3.connect(self.cachefile, timeout=5)
- self.connection.text_factory = str
- self._setup_database()
-
- @_Decorators.retry()
- @_Decorators.transaction
- def _execute_single(self, cursor, *query):
- """
- Executes a single query and discards the results. This correctly closes
- the database cursor when finished
- """
- cursor.execute(*query)
-
- @_Decorators.retry()
- def _row_iter(self, f, *query):
- """
- Helper function that returns a row iterator. Each time __next__ is
- called on the iterator, the provided function is evaluated to determine
- the return value
- """
- class CursorIter(object):
- def __init__(self, cursor):
- self.cursor = cursor
-
- def __iter__(self):
- return self
-
- def __next__(self):
- row = self.cursor.fetchone()
- if row is None:
- self.cursor.close()
- raise StopIteration
- return f(row)
-
- def __enter__(self):
- return self
-
- def __exit__(self, typ, value, traceback):
- self.cursor.close()
- return False
-
- cursor = self.connection.cursor()
- try:
- cursor.execute(*query)
- return CursorIter(cursor)
- except:
- cursor.close()
-
- def __enter__(self):
- self.connection.__enter__()
- return self
-
- def __exit__(self, *excinfo):
- self.connection.__exit__(*excinfo)
-
- @_Decorators.retry()
- @_Decorators.transaction
- def __getitem__(self, cursor, key):
- cursor.execute("SELECT * from %s where key=?;" % self.table, [key])
- row = cursor.fetchone()
- if row is not None:
- return row[1]
- raise KeyError(key)
-
- @_Decorators.retry()
- @_Decorators.transaction
- def __delitem__(self, cursor, key):
- if key not in self:
- raise KeyError(key)
- cursor.execute("DELETE from %s where key=?;" % self.table, [key])
-
- @_Decorators.retry()
- @_Decorators.transaction
- def __setitem__(self, cursor, key, value):
- if not isinstance(key, str):
- raise TypeError('Only string keys are supported')
- elif not isinstance(value, str):
- raise TypeError('Only string values are supported')
-
- # Ensure the entire transaction (including SELECT) executes under write lock
- cursor.execute("BEGIN EXCLUSIVE")
-
- cursor.execute("SELECT * from %s where key=?;" % self.table, [key])
- row = cursor.fetchone()
- if row is not None:
- cursor.execute("UPDATE %s SET value=? WHERE key=?;" % self.table, [value, key])
- else:
- cursor.execute("INSERT into %s(key, value) values (?, ?);" % self.table, [key, value])
-
- @_Decorators.retry()
- @_Decorators.transaction
- def __contains__(self, cursor, key):
- cursor.execute('SELECT * from %s where key=?;' % self.table, [key])
- return cursor.fetchone() is not None
-
- @_Decorators.retry()
- @_Decorators.transaction
- def __len__(self, cursor):
- cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
- row = cursor.fetchone()
- if row is not None:
- return row[0]
-
- def __iter__(self):
- return self._row_iter(lambda row: row[0], "SELECT key from %s;" % self.table)
-
- def __lt__(self, other):
- if not isinstance(other, Mapping):
- raise NotImplementedError()
-
- return len(self) < len(other)
-
- def get_by_pattern(self, pattern):
- return self._row_iter(lambda row: row[1], "SELECT * FROM %s WHERE key LIKE ?;" %
- self.table, [pattern])
-
- def values(self):
- return list(self.itervalues())
-
- def itervalues(self):
- return self._row_iter(lambda row: row[0], "SELECT value FROM %s;" %
- self.table)
-
- def items(self):
- return list(self.iteritems())
-
- def iteritems(self):
- return self._row_iter(lambda row: (row[0], row[1]), "SELECT * FROM %s;" %
- self.table)
-
- @_Decorators.retry()
- @_Decorators.transaction
- def clear(self, cursor):
- cursor.execute("DELETE FROM %s;" % self.table)
-
- def has_key(self, key):
- return key in self
-
-def persist(domain, d):
- """Convenience factory for SQLTable objects based upon metadata"""
- import bb.utils
- cachedir = (d.getVar("PERSISTENT_DIR") or
- d.getVar("CACHE"))
- if not cachedir:
- logger.critical("Please set the 'PERSISTENT_DIR' or 'CACHE' variable")
- sys.exit(1)
-
- bb.utils.mkdirhier(cachedir)
- cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
-
- try:
- return SQLTable(cachefile, domain)
- except sqlite3.OperationalError:
- # Sqlite fails to open database when its path is too long.
- # After testing, 504 is the biggest path length that can be opened by
- # sqlite.
- # Note: This code is called before sanity.bbclass and its path length
- # check
- max_len = 504
- if len(cachefile) > max_len:
- logger.critical("The path of the cache file is too long "
- "({0} chars > {1}) to be opened by sqlite! "
- "Your cache file is \"{2}\"".format(
- len(cachefile),
- max_len,
- cachefile))
- sys.exit(1)
- else:
- raise
diff --git a/bitbake/lib/bb/pysh/pyshlex.py b/bitbake/lib/bb/pysh/pyshlex.py
index a42c2944..3f99cd1b 100644
--- a/bitbake/lib/bb/pysh/pyshlex.py
+++ b/bitbake/lib/bb/pysh/pyshlex.py
@@ -14,7 +14,7 @@
# that hard to enable pull mode.
import re
-from ply import lex
+from bb._vendor.ply import lex
from bb.pysh.sherrors import *
class NeedMore(Exception):
diff --git a/bitbake/lib/bb/pysh/pyshyacc.py b/bitbake/lib/bb/pysh/pyshyacc.py
index 924860a6..67e22c96 100644
--- a/bitbake/lib/bb/pysh/pyshyacc.py
+++ b/bitbake/lib/bb/pysh/pyshyacc.py
@@ -13,7 +13,7 @@ import sys
import bb.pysh.pyshlex as pyshlex
tokens = pyshlex.tokens
-from ply import yacc
+from bb._vendor.ply import yacc
import bb.pysh.sherrors as sherrors
class IORedirect:
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index bc7e1817..ef95eb46 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -10,25 +10,36 @@ Handles preparation and execution of a queue of tasks
#
import copy
+import enum
import os
import sys
import stat
import errno
+import itertools
import logging
import re
-import bb
-from bb import msg, event
-from bb import monitordisk
import subprocess
import pickle
-from multiprocessing import Process
import shlex
import pprint
import time
+import bb.build
+import bb.cache
+import bb.cooker
+import bb.event
+import bb.monitordisk
+import bb.parse
+import bb.progress
+import bb.utils
+from bb import multiprocessing
+
+Process = multiprocessing.Process
+
bblogger = logging.getLogger("BitBake")
logger = logging.getLogger("BitBake.RunQueue")
hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv")
+psi_logger = logging.getLogger("BitBake.RunQueue.PSI")
__find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
@@ -80,7 +91,7 @@ def pending_hash_index(tid, rqdata):
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
h = rqdata.runtaskentries[tid].unihash
- return pn + ":" + "taskname" + h
+ return pn + ":" + taskname + h
class RunQueueStats:
"""
@@ -124,14 +135,16 @@ class RunQueueStats:
def updateActiveSetscene(self, active):
self.setscene_active = active
-# These values indicate the next step due to be run in the
-# runQueue state machine
-runQueuePrepare = 2
-runQueueSceneInit = 3
-runQueueRunning = 6
-runQueueFailed = 7
-runQueueCleanUp = 8
-runQueueComplete = 9
+
+# Indicates the next step due to run in the runQueue state machine
+class RunQueueState(enum.Enum):
+ PREPARE = 0
+ SCENE_INIT = 1
+ DUMP_SIGS = 2
+ RUNNING = 3
+ FAILED = 4
+ CLEAN_UP = 5
+ COMPLETE = 6
class RunQueueScheduler(object):
"""
@@ -148,7 +161,7 @@ class RunQueueScheduler(object):
self.rqdata = rqdata
self.numTasks = len(self.rqdata.runtaskentries)
- self.prio_map = [self.rqdata.runtaskentries.keys()]
+ self.prio_map = list(self.rqdata.runtaskentries.keys())
self.buildable = set()
self.skip_maxthread = {}
@@ -160,6 +173,13 @@ class RunQueueScheduler(object):
self.buildable.add(tid)
self.rev_prio_map = None
+ self.prev_cpu_pressure = 0.0
+ self.prev_io_pressure = 0.0
+ self.prev_memory_pressure = 0.0
+ self.prev_pressure_time = 0.0
+ self.check_pressure = False
+ self.pressure_state = None
+ self.loadfactor_limit = None
self.is_pressure_usable()
def is_pressure_usable(self):
@@ -174,9 +194,9 @@ class RunQueueScheduler(object):
open("/proc/pressure/io") as io_pressure_fds, \
open("/proc/pressure/memory") as memory_pressure_fds:
- self.prev_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1]
- self.prev_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1]
- self.prev_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1]
+ self.prev_cpu_pressure = float(cpu_pressure_fds.readline().split()[4].split("=")[1])
+ self.prev_io_pressure = float(io_pressure_fds.readline().split()[4].split("=")[1])
+ self.prev_memory_pressure = float(memory_pressure_fds.readline().split()[4].split("=")[1])
self.prev_pressure_time = time.time()
self.check_pressure = True
except:
@@ -195,15 +215,15 @@ class RunQueueScheduler(object):
open("/proc/pressure/io") as io_pressure_fds, \
open("/proc/pressure/memory") as memory_pressure_fds:
# extract "total" from /proc/pressure/{cpu|io}
- curr_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1]
- curr_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1]
- curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1]
+ curr_cpu_pressure = float(cpu_pressure_fds.readline().split()[4].split("=")[1])
+ curr_io_pressure = float(io_pressure_fds.readline().split()[4].split("=")[1])
+ curr_memory_pressure = float(memory_pressure_fds.readline().split()[4].split("=")[1])
now = time.time()
tdiff = now - self.prev_pressure_time
psi_accumulation_interval = 1.0
- cpu_pressure = (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) / tdiff
- io_pressure = (float(curr_io_pressure) - float(self.prev_io_pressure)) / tdiff
- memory_pressure = (float(curr_memory_pressure) - float(self.prev_memory_pressure)) / tdiff
+ cpu_pressure = (curr_cpu_pressure - self.prev_cpu_pressure) / tdiff
+ io_pressure = (curr_io_pressure - self.prev_io_pressure) / tdiff
+ memory_pressure = (curr_memory_pressure - self.prev_memory_pressure) / tdiff
exceeds_cpu_pressure = self.rq.max_cpu_pressure and cpu_pressure > self.rq.max_cpu_pressure
exceeds_io_pressure = self.rq.max_io_pressure and io_pressure > self.rq.max_io_pressure
exceeds_memory_pressure = self.rq.max_memory_pressure and memory_pressure > self.rq.max_memory_pressure
@@ -216,8 +236,9 @@ class RunQueueScheduler(object):
pressure_state = (exceeds_cpu_pressure, exceeds_io_pressure, exceeds_memory_pressure)
pressure_values = (round(cpu_pressure,1), self.rq.max_cpu_pressure, round(io_pressure,1), self.rq.max_io_pressure, round(memory_pressure,1), self.rq.max_memory_pressure)
- if hasattr(self, "pressure_state") and pressure_state != self.pressure_state:
- bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s) - using %s/%s bitbake threads" % (pressure_state + pressure_values + (len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks)))
+ if self.pressure_state is not None and pressure_state != self.pressure_state:
+ psi_logger.verbose("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s) - using %s/%s bitbake threads" % (pressure_state + pressure_values + (len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks)))
+ bb.event.fire(PSIEvent(pressure_state, pressure_values), self.rq.cfgData)
self.pressure_state = pressure_state
return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure)
elif self.rq.max_loadfactor:
@@ -226,7 +247,7 @@ class RunQueueScheduler(object):
# bb.warn("Comparing %s to %s" % (loadfactor, self.rq.max_loadfactor))
if loadfactor > self.rq.max_loadfactor:
limit = True
- if hasattr(self, "loadfactor_limit") and limit != self.loadfactor_limit:
+ if self.loadfactor_limit is not None and limit != self.loadfactor_limit:
bb.note("Load average limiting set to %s as load average: %s - using %s/%s bitbake threads" % (limit, loadfactor, len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks))
self.loadfactor_limit = limit
return limit
@@ -273,9 +294,10 @@ class RunQueueScheduler(object):
return tid
if not self.rev_prio_map:
- self.rev_prio_map = {}
- for tid in self.rqdata.runtaskentries:
- self.rev_prio_map[tid] = self.prio_map.index(tid)
+ self.rev_prio_map = {
+ tid: priority
+ for priority, tid in enumerate(self.prio_map)
+ }
best = None
bestprio = None
@@ -448,7 +470,6 @@ class RunTaskEntry(object):
self.revdeps = set()
self.hash = None
self.unihash = None
- self.task = None
self.weight = 1
class RunQueueData:
@@ -465,9 +486,10 @@ class RunQueueData:
self.multi_provider_allowed = (cfgData.getVar("BB_MULTI_PROVIDER_ALLOWED") or "").split()
self.setscene_ignore_tasks = get_setscene_enforce_ignore_tasks(cfgData, targets)
- self.setscene_ignore_tasks_checked = False
self.setscene_enforce = (cfgData.getVar('BB_SETSCENE_ENFORCE') == "1")
self.init_progress_reporter = bb.progress.DummyMultiStageProcessProgressReporter()
+ self.target_tids = []
+ self.runq_setscene_tids = set()
self.reset()
@@ -475,7 +497,6 @@ class RunQueueData:
self.runtaskentries = {}
def runq_depends_names(self, ids):
- import re
ret = []
for id in ids:
nam = os.path.basename(id)
@@ -489,22 +510,11 @@ class RunQueueData:
def get_task_unihash(self, tid):
return self.runtaskentries[tid].unihash
- def get_user_idstring(self, tid, task_name_suffix = ""):
- return tid + task_name_suffix
-
- def get_short_user_idstring(self, task, task_name_suffix = ""):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- pn = self.dataCaches[mc].pkg_fn[taskfn]
- taskname = taskname_from_tid(task) + task_name_suffix
- return "%s:%s" % (pn, taskname)
-
def circular_depchains_handler(self, tasks):
"""
Some tasks aren't buildable, likely due to circular dependency issues.
Identify the circular dependencies and print them in a user readable format.
"""
- from copy import deepcopy
-
valid_chains = []
explored_deps = {}
msgs = []
@@ -600,7 +610,6 @@ class RunQueueData:
possible to execute due to circular dependencies.
"""
- numTasks = len(self.runtaskentries)
weight = {}
deps_left = {}
task_done = {}
@@ -649,8 +658,6 @@ class RunQueueData:
message = message + msg
bb.msg.fatal("RunQueue", message)
- return weight
-
def prepare(self):
"""
Turn a set of taskData into a RunQueue and compute data needed
@@ -677,7 +684,7 @@ class RunQueueData:
self.init_progress_reporter.start()
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Step A - Work out a list of tasks to run
#
@@ -688,7 +695,7 @@ class RunQueueData:
# To create the actual list of tasks to execute we fix the list of
# providers and then resolve the dependencies into task IDs. This
# process is repeated for each type of dependency (tdepends, deptask,
- # rdeptast, recrdeptask, idepends).
+ # rdeptask, recrdeptask, idepends).
def add_build_dependencies(depids, tasknames, depends, mc):
for depname in depids:
@@ -728,6 +735,8 @@ class RunQueueData:
if mc == frommc:
fn = taskData[mcdep].build_targets[pn][0]
newdep = '%s:%s' % (fn,deptask)
+ if newdep not in taskData[mcdep].taskentries:
+ bb.fatal("Task mcdepends on non-existent task %s" % (newdep))
taskData[mc].taskentries[tid].tdepends.append(newdep)
for mc in taskData:
@@ -826,7 +835,7 @@ class RunQueueData:
#self.dump_data()
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Resolve recursive 'recrdeptask' dependencies (Part B)
#
@@ -834,7 +843,7 @@ class RunQueueData:
# (makes sure sometask runs after someothertask of all DEPENDS, RDEPENDS and intertask dependencies, recursively)
# We need to do this separately since we need all of runtaskentries[*].depends to be complete before this is processed
- # Generating/interating recursive lists of dependencies is painful and potentially slow
+ # Generating/iterating recursive lists of dependencies is painful and potentially slow
# Precompute recursive task dependencies here by:
# a) create a temp list of reverse dependencies (revdeps)
# b) walk up the ends of the chains (when a given task no longer has dependencies i.e. len(deps) == 0)
@@ -865,8 +874,7 @@ class RunQueueData:
for dep in revdeps[tid]:
cumulativedeps[dep].add(fn_from_tid(tid))
cumulativedeps[dep].update(cumulativedeps[tid])
- if tid in deps[dep]:
- deps[dep].remove(tid)
+ deps[dep].discard(tid)
if not deps[dep]:
next.add(dep)
endpoints = next
@@ -923,7 +931,7 @@ class RunQueueData:
self.runtaskentries[tid].depends.difference_update(recursivetasksselfref)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
#self.dump_data()
@@ -1005,7 +1013,7 @@ class RunQueueData:
mark_active(tid, 1)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Step C - Prune all inactive tasks
#
@@ -1052,7 +1060,7 @@ class RunQueueData:
bb.msg.fatal("RunQueue", "Could not find any tasks with the tasknames %s to run within the recipes of the taskgraphs of the targets %s" % (str(self.cooker.configuration.runall), str(self.targets)))
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Handle runonly
if self.cooker.configuration.runonly:
@@ -1093,7 +1101,7 @@ class RunQueueData:
logger.verbose("Assign Weightings")
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Generate a list of reverse dependencies to ease future calculations
for tid in self.runtaskentries:
@@ -1101,7 +1109,7 @@ class RunQueueData:
self.runtaskentries[dep].revdeps.add(tid)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Identify tasks at the end of dependency chains
# Error on circular dependency loops (length two)
@@ -1118,14 +1126,14 @@ class RunQueueData:
logger.verbose("Compute totals (have %s endpoint(s))", len(endpoints))
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Calculate task weights
# Check of higher length circular dependencies
- self.runq_weight = self.calculate_task_weights(endpoints)
+ self.calculate_task_weights(endpoints)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Sanity Check - Check for multiple tasks building the same provider
for mc in self.dataCaches:
@@ -1226,7 +1234,7 @@ class RunQueueData:
self.init_progress_reporter.next_stage()
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Iterate over the task list looking for tasks with a 'setscene' function
self.runq_setscene_tids = set()
@@ -1239,7 +1247,7 @@ class RunQueueData:
self.runq_setscene_tids.add(tid)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Invalidate task if force mode active
if self.cooker.configuration.force:
@@ -1256,7 +1264,7 @@ class RunQueueData:
invalidate_task(fn + ":" + st, True)
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
# Create and print to the logs a virtual/xxxx -> PN (fn) table
for mc in taskData:
@@ -1269,31 +1277,45 @@ class RunQueueData:
bb.parse.siggen.tasks_resolved(virtmap, virtpnmap, self.dataCaches[mc])
self.init_progress_reporter.next_stage()
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
+ starttime = time.time()
+ lasttime = starttime
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(self.runtaskentries)
while todeal:
+ ready = set()
for tid in todeal.copy():
if not (self.runtaskentries[tid].depends - dealtwith):
- dealtwith.add(tid)
- todeal.remove(tid)
- self.prepare_task_hash(tid)
- bb.event.check_for_interrupts(self.cooker.data)
+ self.runtaskentries[tid].taskhash_deps = bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches)
+ # get_taskhash for a given tid *must* be called before get_unihash* below
+ self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches)
+ ready.add(tid)
+ unihashes = bb.parse.siggen.get_unihashes(ready)
+ for tid in ready:
+ dealtwith.add(tid)
+ todeal.remove(tid)
+ self.runtaskentries[tid].unihash = unihashes[tid]
+
+ bb.event.check_for_interrupts()
+
+ if time.time() > (lasttime + 30):
+ lasttime = time.time()
+ hashequiv_logger.verbose("Initial setup loop progress: %s of %s in %s" % (len(todeal), len(self.runtaskentries), lasttime - starttime))
+
+ endtime = time.time()
+ if (endtime-starttime > 60):
+ hashequiv_logger.verbose("Initial setup loop took: %s" % (endtime-starttime))
bb.parse.siggen.writeout_file_checksum_cache()
#self.dump_data()
return len(self.runtaskentries)
- def prepare_task_hash(self, tid):
- bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches)
- self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches)
- self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
-
def dump_data(self):
"""
Dump some debug information on the internal data structures
@@ -1320,20 +1342,23 @@ class RunQueue:
self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION") or None
self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
- self.state = runQueuePrepare
+ self.state = RunQueueState.PREPARE
# For disk space monitor
# Invoked at regular time intervals via the bitbake heartbeat event
# while the build is running. We generate a unique name for the handler
# here, just in case that there ever is more than one RunQueue instance,
- # start the handler when reaching runQueueSceneInit, and stop it when
+ # start the handler when reaching RunQueueState.SCENE_INIT, and stop it when
# done with the build.
- self.dm = monitordisk.diskMonitor(cfgData)
+ self.dm = bb.monitordisk.diskMonitor(cfgData)
self.dm_event_handler_name = '_bb_diskmonitor_' + str(id(self))
self.dm_event_handler_registered = False
self.rqexe = None
self.worker = {}
self.fakeworker = {}
+ self.teardown = False
+ self.invalidtasks_dump = set()
+ self.dumpsigs_launched = None
@staticmethod
def send_pickled_data(worker, data, name):
@@ -1538,9 +1563,9 @@ class RunQueue:
"""
retval = True
- bb.event.check_for_interrupts(self.cooker.data)
+ bb.event.check_for_interrupts()
- if self.state is runQueuePrepare:
+ if self.state == RunQueueState.PREPARE:
# NOTE: if you add, remove or significantly refactor the stages of this
# process then you should recalculate the weightings here. This is quite
# easy to do - just change the next line temporarily to pass debug=True as
@@ -1551,12 +1576,12 @@ class RunQueue:
"Initialising tasks",
[43, 967, 4, 3, 1, 5, 3, 7, 13, 1, 2, 1, 1, 246, 35, 1, 38, 1, 35, 2, 338, 204, 142, 3, 3, 37, 244])
if self.rqdata.prepare() == 0:
- self.state = runQueueComplete
+ self.state = RunQueueState.COMPLETE
else:
- self.state = runQueueSceneInit
+ self.state = RunQueueState.SCENE_INIT
bb.parse.siggen.save_unitaskhashes()
- if self.state is runQueueSceneInit:
+ if self.state == RunQueueState.SCENE_INIT:
self.rqdata.init_progress_reporter.next_stage()
# we are ready to run, emit dependency info to any UI or class which
@@ -1566,25 +1591,30 @@ class RunQueue:
bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
if not self.dm_event_handler_registered:
- res = bb.event.register(self.dm_event_handler_name,
- lambda x, y: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False,
- ('bb.event.HeartbeatEvent',), data=self.cfgData)
- self.dm_event_handler_registered = True
+ bb.event.register(self.dm_event_handler_name,
+ lambda x, y: self.dm.check(self) if self.state in [RunQueueState.RUNNING, RunQueueState.CLEAN_UP] else False,
+ ('bb.event.HeartbeatEvent',), data=self.cfgData)
+ self.dm_event_handler_registered = True
self.rqdata.init_progress_reporter.next_stage()
self.rqexe = RunQueueExecute(self)
- dump = self.cooker.configuration.dump_signatures
- if dump:
+ dumpsigs = self.cooker.configuration.dump_signatures
+ if dumpsigs:
self.rqdata.init_progress_reporter.finish()
- if 'printdiff' in dump:
- invalidtasks = self.print_diffscenetasks()
- self.dump_signatures(dump)
- if 'printdiff' in dump:
- self.write_diffscenetasks(invalidtasks)
- self.state = runQueueComplete
-
- if self.state is runQueueSceneInit:
+ if 'printdiff' in dumpsigs:
+ self.invalidtasks_dump = self.print_diffscenetasks()
+ self.state = RunQueueState.DUMP_SIGS
+
+ if self.state == RunQueueState.DUMP_SIGS:
+ dumpsigs = self.cooker.configuration.dump_signatures
+ retval = self.dump_signatures(dumpsigs)
+ if retval is False:
+ if 'printdiff' in dumpsigs:
+ self.write_diffscenetasks(self.invalidtasks_dump)
+ self.state = RunQueueState.COMPLETE
+
+ if self.state == RunQueueState.SCENE_INIT:
self.start_worker(self.rqexe)
self.rqdata.init_progress_reporter.finish()
@@ -1597,15 +1627,15 @@ class RunQueue:
self.rqexe.tasks_notcovered.add(tid)
self.rqexe.sqdone = True
logger.info('Executing Tasks')
- self.state = runQueueRunning
+ self.state = RunQueueState.RUNNING
- if self.state is runQueueRunning:
+ if self.state == RunQueueState.RUNNING:
retval = self.rqexe.execute()
- if self.state is runQueueCleanUp:
+ if self.state == RunQueueState.CLEAN_UP:
retval = self.rqexe.finish()
- build_done = self.state is runQueueComplete or self.state is runQueueFailed
+ build_done = self.state in [RunQueueState.COMPLETE, RunQueueState.FAILED]
if build_done and self.dm_event_handler_registered:
bb.event.remove(self.dm_event_handler_name, None, data=self.cfgData)
@@ -1621,10 +1651,10 @@ class RunQueue:
# Let's avoid the word "failed" if nothing actually did
logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped)
- if self.state is runQueueFailed:
- raise bb.runqueue.TaskFailure(self.rqexe.failed_tids)
+ if self.state == RunQueueState.FAILED:
+ raise TaskFailure(self.rqexe.failed_tids)
- if self.state is runQueueComplete:
+ if self.state == RunQueueState.COMPLETE:
# All done
return False
@@ -1635,7 +1665,7 @@ class RunQueue:
# Catch unexpected exceptions and ensure we exit when an error occurs, not loop.
try:
return self._execute_runqueue()
- except bb.runqueue.TaskFailure:
+ except TaskFailure:
raise
except SystemExit:
raise
@@ -1644,7 +1674,7 @@ class RunQueue:
self.teardown_workers()
except:
pass
- self.state = runQueueComplete
+ self.state = RunQueueState.COMPLETE
raise
except Exception as err:
logger.exception("An uncaught exception occurred in runqueue")
@@ -1652,12 +1682,12 @@ class RunQueue:
self.teardown_workers()
except:
pass
- self.state = runQueueComplete
+ self.state = RunQueueState.COMPLETE
raise
def finish_runqueue(self, now = False):
if not self.rqexe:
- self.state = runQueueComplete
+ self.state = RunQueueState.COMPLETE
return
if now:
@@ -1672,33 +1702,42 @@ class RunQueue:
bb.parse.siggen.dump_sigtask(taskfn, taskname, dataCaches[mc].stamp[taskfn], True)
def dump_signatures(self, options):
- if bb.cooker.CookerFeatures.RECIPE_SIGGEN_INFO not in self.cooker.featureset:
- bb.fatal("The dump signatures functionality needs the RECIPE_SIGGEN_INFO feature enabled")
-
- bb.note("Writing task signature files")
-
- max_process = int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() or 1)
- def chunkify(l, n):
- return [l[i::n] for i in range(n)]
- tids = chunkify(list(self.rqdata.runtaskentries), max_process)
- # We cannot use the real multiprocessing.Pool easily due to some local data
- # that can't be pickled. This is a cheap multi-process solution.
- launched = []
- while tids:
- if len(launched) < max_process:
- p = Process(target=self._rq_dump_sigtid, args=(tids.pop(), ))
+ if self.dumpsigs_launched is None:
+ if bb.cooker.CookerFeatures.RECIPE_SIGGEN_INFO not in self.cooker.featureset:
+ bb.fatal("The dump signatures functionality needs the RECIPE_SIGGEN_INFO feature enabled")
+
+ bb.note("Writing task signature files")
+
+ max_process = int(self.cfgData.getVar("BB_NUMBER_PARSE_THREADS") or os.cpu_count() or 1)
+ def chunkify(l, n):
+ return [l[i::n] for i in range(n)]
+ dumpsigs_tids = chunkify(list(self.rqdata.runtaskentries), max_process)
+
+ # We cannot use the real multiprocessing.Pool easily due to some local data
+ # that can't be pickled. This is a cheap multi-process solution.
+ self.dumpsigs_launched = []
+
+ for tids in dumpsigs_tids:
+ p = Process(target=self._rq_dump_sigtid, args=(tids, ))
p.start()
- launched.append(p)
- for q in launched:
- # The finished processes are joined when calling is_alive()
- if not q.is_alive():
- launched.remove(q)
- for p in launched:
- p.join()
+ self.dumpsigs_launched.append(p)
+
+ return 1.0
+
+ for q in self.dumpsigs_launched.copy():
+ # The finished processes are joined when calling is_alive()
+ if not q.is_alive():
+ self.dumpsigs_launched.remove(q)
+
+ if self.dumpsigs_launched:
+ return 1.0
+
+ for p in self.dumpsigs_launched:
+ p.join()
bb.parse.siggen.dump_sigs(self.rqdata.dataCaches, options)
- return
+ return False
def print_diffscenetasks(self):
def get_root_invalid_tasks(task, taskdepends, valid, noexec, visited_invalid):
@@ -1847,7 +1886,6 @@ class RunQueueExecute:
self.runq_tasksrun = set()
self.build_stamps = {}
- self.build_stamps2 = []
self.failed_tids = []
self.sq_deferred = {}
self.sq_needed_harddeps = set()
@@ -1858,6 +1896,7 @@ class RunQueueExecute:
self.holdoff_tasks = set()
self.holdoff_need_update = True
self.sqdone = False
+ self.taskdepdata_cache = {}
self.stats = RunQueueStats(len(self.rqdata.runtaskentries), len(self.rqdata.runq_setscene_tids))
@@ -1885,13 +1924,13 @@ class RunQueueExecute:
if self.max_memory_pressure < lower_limit:
bb.fatal("Invalid BB_PRESSURE_MAX_MEMORY %s, minimum value is %s." % (self.max_memory_pressure, lower_limit))
if self.max_memory_pressure > upper_limit:
- bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_MEMORY is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_io_pressure))
+ bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_MEMORY is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_memory_pressure))
if self.max_loadfactor:
self.max_loadfactor = float(self.max_loadfactor)
if self.max_loadfactor <= 0:
bb.fatal("Invalid BB_LOADFACTOR_MAX %s, needs to be greater than zero." % (self.max_loadfactor))
-
+
# List of setscene tasks which we've covered
self.scenequeue_covered = set()
# List of tasks which are covered (including setscene ones)
@@ -1901,6 +1940,8 @@ class RunQueueExecute:
self.tasks_notcovered = set()
self.scenequeue_notneeded = set()
+ self.setscene_tids_generator = None
+
schedulers = self.get_schedulers()
for scheduler in schedulers:
if self.scheduler == scheduler.name:
@@ -1937,7 +1978,6 @@ class RunQueueExecute:
# self.build_stamps[pid] may not exist when use shared work directory.
if task in self.build_stamps:
- self.build_stamps2.remove(self.build_stamps[task])
del self.build_stamps[task]
if task in self.sq_live:
@@ -1971,14 +2011,14 @@ class RunQueueExecute:
pass
if self.failed_tids:
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
return
- self.rq.state = runQueueComplete
+ self.rq.state = RunQueueState.COMPLETE
return
def finish(self):
- self.rq.state = runQueueCleanUp
+ self.rq.state = RunQueueState.CLEAN_UP
active = self.stats.active + len(self.sq_live)
if active > 0:
@@ -1987,10 +2027,10 @@ class RunQueueExecute:
return self.rq.active_fds()
if self.failed_tids:
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
return True
- self.rq.state = runQueueComplete
+ self.rq.state = RunQueueState.COMPLETE
return True
# Used by setscene only
@@ -2109,7 +2149,7 @@ class RunQueueExecute:
bb.event.fire(runQueueTaskFailed(task, self.stats, exitcode, self.rq, fakeroot_log=("".join(fakeroot_log) or None)), self.cfgData)
if self.rqdata.taskData[''].halt:
- self.rq.state = runQueueCleanUp
+ self.rq.state = RunQueueState.CLEAN_UP
def task_skip(self, task, reason):
self.runq_running.add(task)
@@ -2172,15 +2212,21 @@ class RunQueueExecute:
if self.updated_taskhash_queue or self.pending_migrations:
self.process_possible_migrations()
- if not hasattr(self, "sorted_setscene_tids"):
- # Don't want to sort this set every execution
- self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids)
+ if self.setscene_tids_generator is None:
+ # Resume looping where we left off when we returned to feed the mainloop
+ self.setscene_tids_generator = itertools.cycle(self.rqdata.runq_setscene_tids)
task = None
if not self.sqdone and self.can_start_task():
- # Find the next setscene to run
- for nexttask in self.sorted_setscene_tids:
+ loopcount = 0
+ # Find the next setscene to run, exit the loop when we've processed all tids or found something to execute
+ while loopcount < len(self.rqdata.runq_setscene_tids):
+ loopcount += 1
+ nexttask = next(self.setscene_tids_generator)
if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred:
+ if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete:
+ # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds
+ continue
if nexttask not in self.sqdata.unskippable and self.sqdata.sq_revdeps[nexttask] and \
nexttask not in self.sq_needed_harddeps and \
self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and \
@@ -2206,12 +2252,10 @@ class RunQueueExecute:
return True
continue
# If covered tasks are running, need to wait for them to complete
- for t in self.sqdata.sq_covered_tasks[nexttask]:
- if t in self.runq_running and t not in self.runq_complete:
- continue
+ if any(t in self.runq_running and t not in self.runq_complete for t in self.sqdata.sq_covered_tasks[nexttask]):
+ continue
if nexttask in self.sq_deferred:
- if self.sq_deferred[nexttask] not in self.runq_complete:
- continue
+ # Deferred tasks that were still deferred were skipped above so we now need to process
logger.debug("Task %s no longer deferred" % nexttask)
del self.sq_deferred[nexttask]
valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False, summary=False)
@@ -2282,7 +2326,6 @@ class RunQueueExecute:
self.rq.worker[mc].process.stdin.flush()
self.build_stamps[task] = bb.parse.siggen.stampfile_mcfn(taskname, taskfn, extrainfo=False)
- self.build_stamps2.append(self.build_stamps[task])
self.sq_running.add(task)
self.sq_live.add(task)
self.stats.updateActiveSetscene(len(self.sq_live))
@@ -2296,17 +2339,17 @@ class RunQueueExecute:
err = self.summarise_scenequeue_errors()
if err:
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
return True
if self.cooker.configuration.setsceneonly:
- self.rq.state = runQueueComplete
+ self.rq.state = RunQueueState.COMPLETE
return True
self.sqdone = True
if self.stats.total == 0:
# nothing to do
- self.rq.state = runQueueComplete
+ self.rq.state = RunQueueState.COMPLETE
return True
if self.cooker.configuration.setsceneonly:
@@ -2373,7 +2416,7 @@ class RunQueueExecute:
self.rq.start_fakeworker(self, mc)
except OSError as exc:
logger.critical("Failed to spawn fakeroot worker to run %s: %s" % (task, str(exc)))
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
self.stats.taskFailed()
return True
RunQueue.send_pickled_data(self.rq.fakeworker[mc].process, runtask, "runtask")
@@ -2383,7 +2426,6 @@ class RunQueueExecute:
self.rq.worker[mc].process.stdin.flush()
self.build_stamps[task] = bb.parse.siggen.stampfile_mcfn(taskname, taskfn, extrainfo=False)
- self.build_stamps2.append(self.build_stamps[task])
self.runq_running.add(task)
self.stats.taskActive()
if self.can_start_task():
@@ -2401,7 +2443,7 @@ class RunQueueExecute:
return True
if self.failed_tids:
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
return True
# Sanity Checks
@@ -2418,9 +2460,9 @@ class RunQueueExecute:
err = True
if err:
- self.rq.state = runQueueFailed
+ self.rq.state = RunQueueState.FAILED
else:
- self.rq.state = runQueueComplete
+ self.rq.state = RunQueueState.COMPLETE
return True
@@ -2438,14 +2480,17 @@ class RunQueueExecute:
taskdepdata_cache = {}
for task in self.rqdata.runtaskentries:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
- deps = self.rqdata.runtaskentries[task].depends
- provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
- taskhash = self.rqdata.runtaskentries[task].hash
- unihash = self.rqdata.runtaskentries[task].unihash
- deps = self.filtermcdeps(task, mc, deps)
- hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn]
- taskdepdata_cache[task] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn]
+ taskdepdata_cache[task] = bb.TaskData(
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn],
+ taskname = taskname,
+ fn = fn,
+ deps = self.filtermcdeps(task, mc, self.rqdata.runtaskentries[task].depends),
+ provides = self.rqdata.dataCaches[mc].fn_provides[taskfn],
+ taskhash = self.rqdata.runtaskentries[task].hash,
+ unihash = self.rqdata.runtaskentries[task].unihash,
+ hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn],
+ taskhash_deps = self.rqdata.runtaskentries[task].taskhash_deps,
+ )
self.taskdepdata_cache = taskdepdata_cache
@@ -2460,9 +2505,11 @@ class RunQueueExecute:
while next:
additional = []
for revdep in next:
- self.taskdepdata_cache[revdep][6] = self.rqdata.runtaskentries[revdep].unihash
+ self.taskdepdata_cache[revdep] = self.taskdepdata_cache[revdep]._replace(
+ unihash=self.rqdata.runtaskentries[revdep].unihash
+ )
taskdepdata[revdep] = self.taskdepdata_cache[revdep]
- for revdep2 in self.taskdepdata_cache[revdep][3]:
+ for revdep2 in self.taskdepdata_cache[revdep].deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
next = additional
@@ -2531,9 +2578,6 @@ class RunQueueExecute:
self.rqdata.runtaskentries[hashtid].unihash = unihash
bb.parse.siggen.set_unihash(hashtid, unihash)
toprocess.add(hashtid)
- if torehash:
- # Need to save after set_unihash above
- bb.parse.siggen.save_unitaskhashes()
# Work out all tasks which depend upon these
total = set()
@@ -2556,17 +2600,28 @@ class RunQueueExecute:
elif self.rqdata.runtaskentries[p].depends.isdisjoint(total):
next.add(p)
+ starttime = time.time()
+ lasttime = starttime
+
# When an item doesn't have dependencies in total, we can process it. Drop items from total when handled
while next:
current = next.copy()
next = set()
+ ready = {}
for tid in current:
- if self.rqdata.runtaskentries[p].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
+ if self.rqdata.runtaskentries[tid].depends and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
continue
+ # get_taskhash for a given tid *must* be called before get_unihash* below
+ ready[tid] = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches)
+
+ unihashes = bb.parse.siggen.get_unihashes(ready.keys())
+
+ for tid in ready:
orighash = self.rqdata.runtaskentries[tid].hash
- newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches)
+ newhash = ready[tid]
origuni = self.rqdata.runtaskentries[tid].unihash
- newuni = bb.parse.siggen.get_unihash(tid)
+ newuni = unihashes[tid]
+
# FIXME, need to check it can come from sstate at all for determinism?
remapped = False
if newuni == origuni:
@@ -2587,6 +2642,15 @@ class RunQueueExecute:
next |= self.rqdata.runtaskentries[tid].revdeps
total.remove(tid)
next.intersection_update(total)
+ bb.event.check_for_interrupts()
+
+ if time.time() > (lasttime + 30):
+ lasttime = time.time()
+ hashequiv_logger.verbose("Rehash loop slow progress: %s in %s" % (len(total), lasttime - starttime))
+
+ endtime = time.time()
+ if (endtime-starttime > 60):
+ hashequiv_logger.verbose("Rehash loop took more than 60s: %s" % (endtime-starttime))
if changed:
for mc in self.rq.worker:
@@ -2622,31 +2686,24 @@ class RunQueueExecute:
self.pending_migrations.remove(tid)
changed = True
- if tid in self.tasks_scenequeue_done:
- self.tasks_scenequeue_done.remove(tid)
+ self.tasks_scenequeue_done.discard(tid)
for dep in self.sqdata.sq_covered_tasks[tid]:
if dep in self.runq_complete and dep not in self.runq_tasksrun:
bb.error("Task %s marked as completed but now needing to rerun? Halting build." % dep)
self.failed_tids.append(tid)
- self.rq.state = runQueueCleanUp
+ self.rq.state = RunQueueState.CLEAN_UP
return
if dep not in self.runq_complete:
if dep in self.tasks_scenequeue_done and dep not in self.sqdata.unskippable:
self.tasks_scenequeue_done.remove(dep)
- if tid in self.sq_buildable:
- self.sq_buildable.remove(tid)
- if tid in self.sq_running:
- self.sq_running.remove(tid)
- if tid in self.sqdata.outrightfail:
- self.sqdata.outrightfail.remove(tid)
- if tid in self.scenequeue_notcovered:
- self.scenequeue_notcovered.remove(tid)
- if tid in self.scenequeue_covered:
- self.scenequeue_covered.remove(tid)
- if tid in self.scenequeue_notneeded:
- self.scenequeue_notneeded.remove(tid)
+ self.sq_buildable.discard(tid)
+ self.sq_running.discard(tid)
+ self.sqdata.outrightfail.discard(tid)
+ self.scenequeue_notcovered.discard(tid)
+ self.scenequeue_covered.discard(tid)
+ self.scenequeue_notneeded.discard(tid)
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
self.sqdata.stamps[tid] = bb.parse.siggen.stampfile_mcfn(taskname, taskfn, extrainfo=False)
@@ -2712,8 +2769,12 @@ class RunQueueExecute:
logger.debug2("%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
self.sq_task_failoutright(dep)
continue
+
+ # For performance, only compute allcovered once if needed
+ if self.sqdata.sq_deps[task]:
+ allcovered = self.scenequeue_covered | self.scenequeue_notcovered
for dep in sorted(self.sqdata.sq_deps[task]):
- if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered | self.scenequeue_notcovered):
+ if self.sqdata.sq_revdeps[dep].issubset(allcovered):
if dep not in self.sq_buildable:
self.sq_buildable.add(dep)
@@ -2759,7 +2820,7 @@ class RunQueueExecute:
pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
if not check_setscene_enforce_ignore_tasks(pn, taskname, self.rqdata.setscene_ignore_tasks):
logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
- self.rq.state = runQueueCleanUp
+ self.rq.state = RunQueueState.CLEAN_UP
def sq_task_complete(self, task):
bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
@@ -2806,13 +2867,19 @@ class RunQueueExecute:
additional = []
for revdep in next:
(mc, fn, taskname, taskfn) = split_tid_mcfn(revdep)
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
deps = getsetscenedeps(revdep)
- provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
- taskhash = self.rqdata.runtaskentries[revdep].hash
- unihash = self.rqdata.runtaskentries[revdep].unihash
- hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn]
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash, hashfn]
+
+ taskdepdata[revdep] = bb.TaskData(
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn],
+ taskname = taskname,
+ fn = fn,
+ deps = deps,
+ provides = self.rqdata.dataCaches[mc].fn_provides[taskfn],
+ taskhash = self.rqdata.runtaskentries[revdep].hash,
+ unihash = self.rqdata.runtaskentries[revdep].unihash,
+ hashfn = self.rqdata.dataCaches[mc].hashfn[taskfn],
+ taskhash_deps = self.rqdata.runtaskentries[revdep].taskhash_deps,
+ )
for revdep2 in deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
@@ -2900,8 +2967,7 @@ def build_scenequeue_data(sqdata, rqdata, sqrq):
sq_collated_deps[tid] = set()
#bb.warn("Added endpoint 2 %s" % (tid))
for dep in rqdata.runtaskentries[tid].depends:
- if tid in sq_revdeps[dep]:
- sq_revdeps[dep].remove(tid)
+ sq_revdeps[dep].discard(tid)
if dep not in endpoints:
endpoints[dep] = set()
#bb.warn(" Added endpoint 3 %s" % (dep))
@@ -2925,8 +2991,7 @@ def build_scenequeue_data(sqdata, rqdata, sqrq):
sq_revdeps_squash[point] = tasks
continue
for dep in rqdata.runtaskentries[point].depends:
- if point in sq_revdeps[dep]:
- sq_revdeps[dep].remove(point)
+ sq_revdeps[dep].discard(point)
if tasks:
sq_revdeps_squash[dep] |= tasks
if not sq_revdeps[dep] and dep not in rqdata.runq_setscene_tids:
@@ -2964,14 +3029,13 @@ def build_scenequeue_data(sqdata, rqdata, sqrq):
rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
# Sanity check all dependencies could be changed to setscene task references
- for taskcounter, tid in enumerate(rqdata.runtaskentries):
+ for tid in rqdata.runtaskentries:
if tid in rqdata.runq_setscene_tids:
pass
elif sq_revdeps_squash[tid]:
bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, halting. Please report this problem.")
else:
del sq_revdeps_squash[tid]
- rqdata.init_progress_reporter.update(taskcounter)
rqdata.init_progress_reporter.next_stage()
@@ -3076,12 +3140,9 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s
tocheck = set()
for tid in sorted(tids):
- if tid in sqdata.stamppresent:
- sqdata.stamppresent.remove(tid)
- if tid in sqdata.valid:
- sqdata.valid.remove(tid)
- if tid in sqdata.outrightfail:
- sqdata.outrightfail.remove(tid)
+ sqdata.stamppresent.discard(tid)
+ sqdata.valid.discard(tid)
+ sqdata.outrightfail.discard(tid)
noexec, stamppresent = check_setscene_stamps(tid, rqdata, rq, stampcache, noexecstamp=True)
@@ -3236,6 +3297,12 @@ class taskUniHashUpdate(bb.event.Event):
self.unihash = unihash
bb.event.Event.__init__(self)
+class PSIEvent(bb.event.Event):
+ def __init__(self, pressure_state, pressure_values):
+ super().__init__()
+ self.pressure_state = pressure_state
+ self.pressure_values = pressure_values
+
class runQueuePipe():
"""
Abstraction for a pipe between a worker thread and the server
@@ -3261,7 +3328,7 @@ class runQueuePipe():
start = len(self.queue)
try:
- self.queue.extend(self.input.read(102400) or b"")
+ self.queue.extend(self.input.read(512 * 1024) or b"")
except (OSError, IOError) as e:
if e.errno != errno.EAGAIN:
raise
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 76b18929..d0f73590 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -13,7 +13,7 @@
import bb
import bb.event
import logging
-import multiprocessing
+from bb import multiprocessing
import threading
import array
import os
@@ -80,9 +80,6 @@ class idleFinish():
self.msg = msg
class ProcessServer():
- profile_filename = "profile.log"
- profile_processed_filename = "profile.log.processed"
-
def __init__(self, lock, lockname, sock, sockname, server_timeout, xmlrpcinterface):
self.command_channel = False
self.command_channel_reply = False
@@ -140,23 +137,7 @@ class ProcessServer():
serverlog("Error writing to lock file: %s" % str(e))
pass
- if self.cooker.configuration.profile:
- try:
- import cProfile as profile
- except:
- import profile
- prof = profile.Profile()
-
- ret = profile.Profile.runcall(prof, self.main)
-
- prof.dump_stats("profile.log")
- bb.utils.process_profilelog("profile.log")
- serverlog("Raw profiling information saved to profile.log and processed statistics to profile.log.processed")
-
- else:
- ret = self.main()
-
- return ret
+ return bb.utils.profile_function("main" in self.cooker.configuration.profile, self.main, "profile-mainloop.log")
def _idle_check(self):
return len(self._idlefuns) == 0 and self.cooker.command.currentAsyncCommand is None
@@ -321,7 +302,22 @@ class ProcessServer():
bb.warn('Ignoring invalid BB_SERVER_TIMEOUT=%s, must be a float specifying seconds.' % self.timeout)
seendata = True
- ready = self.idle_commands(.1, fds)
+ if not self.idle:
+ self.idle = threading.Thread(target=self.idle_thread)
+ self.idle.start()
+ elif self.idle and not self.idle.is_alive():
+ serverlog("Idle thread terminated, main thread exiting too")
+ bb.error("Idle thread terminated, main thread exiting too")
+ self.quit = True
+
+ nextsleep = 1.0
+ if self.xmlrpc:
+ nextsleep = self.xmlrpc.get_timeout(nextsleep)
+ try:
+ ready = select.select(fds,[],[],nextsleep)[0]
+ except InterruptedError:
+ # Ignore EINTR
+ ready = []
if self.idle:
self.idle.join()
@@ -402,20 +398,7 @@ class ProcessServer():
serverlog("".join(msg))
def idle_thread(self):
- if self.cooker.configuration.profile:
- try:
- import cProfile as profile
- except:
- import profile
- prof = profile.Profile()
-
- ret = profile.Profile.runcall(prof, self.idle_thread_internal)
-
- prof.dump_stats("profile-mainloop.log")
- bb.utils.process_profilelog("profile-mainloop.log")
- serverlog("Raw profiling information saved to profile-mainloop.log and processed statistics to profile-mainloop.log.processed")
- else:
- self.idle_thread_internal()
+ bb.utils.profile_function("idle" in self.cooker.configuration.profile, self.idle_thread_internal, "profile-idleloop.log")
def idle_thread_internal(self):
def remove_idle_func(function):
@@ -424,7 +407,7 @@ class ProcessServer():
self.idle_cond.notify_all()
while not self.quit:
- nextsleep = 0.1
+ nextsleep = 1.0
fds = []
with bb.utils.lock_timeout(self._idlefuncsLock):
@@ -462,7 +445,7 @@ class ProcessServer():
# Create new heartbeat event?
now = time.time()
- if bb.event._heartbeat_enabled and now >= self.next_heartbeat:
+ if items and bb.event._heartbeat_enabled and now >= self.next_heartbeat:
# We might have missed heartbeats. Just trigger once in
# that case and continue after the usual delay.
self.next_heartbeat += self.heartbeat_seconds
@@ -485,31 +468,6 @@ class ProcessServer():
if nextsleep is not None:
select.select(fds,[],[],nextsleep)[0]
- def idle_commands(self, delay, fds=None):
- nextsleep = delay
- if not fds:
- fds = []
-
- if not self.idle:
- self.idle = threading.Thread(target=self.idle_thread)
- self.idle.start()
- elif self.idle and not self.idle.is_alive():
- serverlog("Idle thread terminated, main thread exiting too")
- bb.error("Idle thread terminated, main thread exiting too")
- self.quit = True
-
- if nextsleep is not None:
- if self.xmlrpc:
- nextsleep = self.xmlrpc.get_timeout(nextsleep)
- try:
- return select.select(fds,[],[],nextsleep)[0]
- except InterruptedError:
- # Ignore EINTR
- return []
- else:
- return select.select(fds,[],[],0)[0]
-
-
class ServerCommunicator():
def __init__(self, connection, recv):
self.connection = connection
@@ -642,7 +600,7 @@ class BitBakeServer(object):
os.set_inheritable(self.bitbake_lock.fileno(), True)
os.set_inheritable(self.readypipein, True)
serverscript = os.path.realpath(os.path.dirname(__file__) + "/../../../bin/bitbake-server")
- os.execl(sys.executable, sys.executable, serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname, str(self.server_timeout or 0), str(int(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1]))
+ os.execl(sys.executable, sys.executable, serverscript, "decafbad", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, self.bitbake_lock.name, self.sockname, str(self.server_timeout or 0), str(list(self.profile)), str(self.xmlrpcinterface[0]), str(self.xmlrpcinterface[1]))
def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, xmlrpcinterface, profile):
diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py
index 04b0b17d..ebc271ac 100644
--- a/bitbake/lib/bb/server/xmlrpcserver.py
+++ b/bitbake/lib/bb/server/xmlrpcserver.py
@@ -14,6 +14,8 @@ from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
import bb.server.xmlrpcclient
import bb
+import bb.cooker
+import bb.event
# This request handler checks if the request has a "Bitbake-token" header
# field (this comes from the client side) and compares it with its internal
@@ -54,7 +56,7 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
def __init__(self, interface, cooker, parent):
# Use auto port configuration
- if (interface[1] == -1):
+ if interface[1] == -1:
interface = (interface[0], 0)
SimpleXMLRPCServer.__init__(self, interface,
requestHandler=BitBakeXMLRPCRequestHandler,
@@ -87,11 +89,12 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
def handle_requests(self):
self._handle_request_noblock()
-class BitBakeXMLRPCServerCommands():
+class BitBakeXMLRPCServerCommands:
def __init__(self, server):
self.server = server
self.has_client = False
+ self.event_handle = None
def registerEventHandler(self, host, port):
"""
@@ -100,8 +103,8 @@ class BitBakeXMLRPCServerCommands():
s, t = bb.server.xmlrpcclient._create_server(host, port)
# we don't allow connections if the cooker is running
- if (self.server.cooker.state in [bb.cooker.state.parsing, bb.cooker.state.running]):
- return None, "Cooker is busy: %s" % bb.cooker.state.get_name(self.server.cooker.state)
+ if self.server.cooker.state in [bb.cooker.State.PARSING, bb.cooker.State.RUNNING]:
+ return None, f"Cooker is busy: {self.server.cooker.state.name}"
self.event_handle = bb.event.register_UIHhandler(s, True)
return self.event_handle, 'OK'
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 8ab08ec9..76c82c68 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -11,8 +11,9 @@ import re
import tempfile
import pickle
import bb.data
+import bb.fetch
import difflib
-import simplediff
+from bb._vendor import simplediff
import json
import types
from contextlib import contextmanager
@@ -43,6 +44,10 @@ def check_siggen_version(siggen):
if siggen.find_siginfo_version < siggen.find_siginfo_minversion:
bb.fatal("Siggen from metadata (OE-Core?) is too old, please update it (%s vs %s)" % (siggen.find_siginfo_version, siggen.find_siginfo_minversion))
+def check_hashserv_unihash(unihash):
+ if not hashserv.is_valid_unihash(unihash):
+ bb.fatal("Hash Equivalence Server returned invalid unihash")
+
class SetEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, set) or isinstance(obj, frozenset):
@@ -201,9 +206,6 @@ class SignatureGenerator(object):
def save_unitaskhashes(self):
return
- def copy_unitaskhashes(self, targetdir):
- return
-
def set_setscene_tasks(self, setscene_tasks):
return
@@ -361,7 +363,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
if self.checksum_cache:
checksums = self.checksum_cache.get_checksums(dataCaches[mc].file_checksums[mcfn][task], recipename, self.localdirsexclude)
else:
- checksums = bb.fetch2.get_file_checksums(dataCaches[mc].file_checksums[mcfn][task], recipename, self.localdirsexclude)
+ checksums = bb.fetch.get_file_checksums(dataCaches[mc].file_checksums[mcfn][task], recipename, self.localdirsexclude)
for (f,cs) in checksums:
self.file_checksum_values[tid].append((f,cs))
@@ -381,7 +383,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.taints[tid] = taint
logger.warning("%s is tainted from a forced run" % tid)
- return
+ return set(dep for _, dep in self.runtaskdeps[tid])
def get_taskhash(self, tid, deps, dataCaches):
@@ -412,15 +414,12 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.checksum_cache.save_extras()
self.checksum_cache.save_merge()
else:
- bb.fetch2.fetcher_parse_save()
- bb.fetch2.fetcher_parse_done()
+ bb.fetch.fetcher_parse_save()
+ bb.fetch.fetcher_parse_done()
def save_unitaskhashes(self):
self.unihash_cache.save(self.unitaskhashes)
- def copy_unitaskhashes(self, targetdir):
- self.unihash_cache.copyfile(targetdir)
-
def dump_sigtask(self, mcfn, task, stampbase, runtime):
tid = mcfn + ":" + task
mc = bb.runqueue.mc_from_tid(mcfn)
@@ -540,7 +539,7 @@ class SignatureGeneratorUniHashMixIn(object):
def __init__(self, data):
self.extramethod = {}
# NOTE: The cache only tracks hashes that exist. Hashes that don't
- # exist are always queries from the server since it is possible for
+ # exist are always queried from the server since it is possible for
# hashes to appear over time, but much less likely for them to
# disappear
self.unihash_exists_cache = set()
@@ -558,11 +557,11 @@ class SignatureGeneratorUniHashMixIn(object):
super().__init__(data)
def get_taskdata(self):
- return (self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env) + super().get_taskdata()
+ return (self.server, self.method, self.extramethod, self.username, self.password, self.env) + super().get_taskdata()
def set_taskdata(self, data):
- self.server, self.method, self.extramethod, self.max_parallel, self.username, self.password, self.env = data[:7]
- super().set_taskdata(data[7:])
+ self.server, self.method, self.extramethod, self.username, self.password, self.env = data[:6]
+ super().set_taskdata(data[6:])
def get_hashserv_creds(self):
if self.username and self.password:
@@ -595,13 +594,6 @@ class SignatureGeneratorUniHashMixIn(object):
self._client = hashserv.create_client(self.server, **self.get_hashserv_creds())
yield self._client
- @contextmanager
- def client_pool(self):
- with self._client_env():
- if getattr(self, '_client_pool', None) is None:
- self._client_pool = hashserv.client.ClientPool(self.server, self.max_parallel, **self.get_hashserv_creds())
- yield self._client_pool
-
def reset(self, data):
self.__close_clients()
return super().reset(data)
@@ -678,25 +670,20 @@ class SignatureGeneratorUniHashMixIn(object):
if len(query) == 0:
return {}
- uncached_query = {}
+ query_keys = []
result = {}
for key, unihash in query.items():
if unihash in self.unihash_exists_cache:
result[key] = True
else:
- uncached_query[key] = unihash
+ query_keys.append(key)
- if self.max_parallel <= 1 or len(uncached_query) <= 1:
- # No parallelism required. Make the query serially with the single client
+ if query_keys:
with self.client() as client:
- uncached_result = {
- key: client.unihash_exists(value) for key, value in uncached_query.items()
- }
- else:
- with self.client_pool() as client_pool:
- uncached_result = client_pool.unihashes_exist(uncached_query)
+ query_result = client.unihash_exists_batch(query[k] for k in query_keys)
- for key, exists in uncached_result.items():
+ for idx, key in enumerate(query_keys):
+ exists = query_result[idx]
if exists:
self.unihash_exists_cache.add(query[key])
result[key] = exists
@@ -712,29 +699,24 @@ class SignatureGeneratorUniHashMixIn(object):
unihash
"""
result = {}
- queries = {}
- query_result = {}
+ query_tids = []
for tid in tids:
unihash = self.get_cached_unihash(tid)
if unihash:
result[tid] = unihash
else:
- queries[tid] = (self._get_method(tid), self.taskhash[tid])
-
- if len(queries) == 0:
- return result
+ query_tids.append(tid)
- if self.max_parallel <= 1 or len(queries) <= 1:
- # No parallelism required. Make the query serially with the single client
- with self.client() as client:
- for tid, args in queries.items():
- query_result[tid] = client.get_unihash(*args)
- else:
- with self.client_pool() as client_pool:
- query_result = client_pool.get_unihashes(queries)
+ if query_tids:
+ unihashes = []
+ try:
+ with self.client() as client:
+ unihashes = client.get_unihash_batch((self._get_method(tid), self.taskhash[tid]) for tid in query_tids)
+ except (ConnectionError, FileNotFoundError, EOFError) as e:
+ bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
- for tid, unihash in query_result.items():
+ for idx, tid in enumerate(query_tids):
# In the absence of being able to discover a unique hash from the
# server, make it be equivalent to the taskhash. The unique "hash" only
# really needs to be a unique string (not even necessarily a hash), but
@@ -749,7 +731,10 @@ class SignatureGeneratorUniHashMixIn(object):
# to the server, there is a better chance that they will agree on
# the unique hash.
taskhash = self.taskhash[tid]
- if unihash:
+
+ if unihashes and unihashes[idx]:
+ unihash = unihashes[idx]
+ check_hashserv_unihash(unihash)
# A unique hash equal to the taskhash is not very interesting,
# so it is reported it at debug level 2. If they differ, that
# is much more interesting, so it is reported at debug level 1
@@ -758,7 +743,6 @@ class SignatureGeneratorUniHashMixIn(object):
hashequiv_logger.debug2('No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
unihash = taskhash
-
self.set_unihash(tid, unihash)
self.unihash[tid] = unihash
result[tid] = unihash
@@ -769,7 +753,7 @@ class SignatureGeneratorUniHashMixIn(object):
import importlib
taskhash = d.getVar('BB_TASKHASH')
- unihash = d.getVar('BB_UNIHASH')
+ unihash = d.getVar('BB_UNIHASH', expand=False)
report_taskdata = d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
tempdir = d.getVar('T')
mcfn = d.getVar('BB_FILENAME')
@@ -831,6 +815,7 @@ class SignatureGeneratorUniHashMixIn(object):
data = client.report_unihash(taskhash, method, outhash, unihash, extra_data)
new_unihash = data['unihash']
+ check_hashserv_unihash(new_unihash)
if new_unihash != unihash:
hashequiv_logger.debug('Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
@@ -839,7 +824,7 @@ class SignatureGeneratorUniHashMixIn(object):
d.setVar('BB_UNIHASH', new_unihash)
else:
hashequiv_logger.debug('Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
- except ConnectionError as e:
+ except (ConnectionError, FileNotFoundError, EOFError) as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
finally:
if sigfile:
@@ -870,6 +855,7 @@ class SignatureGeneratorUniHashMixIn(object):
return False
finalunihash = data['unihash']
+ check_hashserv_unihash(finalunihash)
if finalunihash == current_unihash:
hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash))
@@ -881,7 +867,7 @@ class SignatureGeneratorUniHashMixIn(object):
# TODO: What to do here?
hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
- except ConnectionError as e:
+ except (ConnectionError, FileNotFoundError, EOFError) as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
return False
@@ -895,13 +881,12 @@ class SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn, SignatureG
super().init_rundepcheck(data)
self.server = data.getVar('BB_HASHSERVE')
self.method = "sstate_output_hash"
- self.max_parallel = 1
def clean_checksum_file_path(file_checksum_tuple):
f, cs = file_checksum_tuple
if "/./" in f:
return "./" + f.split("/./")[1]
- return f
+ return os.path.basename(f)
def dump_this_task(outfile, d):
import bb.parse
@@ -1286,7 +1271,10 @@ def dump_sigfile(a):
else:
output.append("Unable to compute base hash")
- computed_taskhash = calc_taskhash(a_data)
- output.append("Computed task hash is %s" % computed_taskhash)
+ if 'runtaskdeps' in a_data:
+ computed_taskhash = calc_taskhash(a_data)
+ output.append("Computed task hash is %s" % computed_taskhash)
+ else:
+ output.append("Unable to compute task hash")
return output
diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
index 66545a65..b40afc02 100644
--- a/bitbake/lib/bb/taskdata.py
+++ b/bitbake/lib/bb/taskdata.py
@@ -138,15 +138,15 @@ class TaskData:
rdependids = set()
rdepends = dataCache.rundeps[fn]
rrecs = dataCache.runrecs[fn]
- rdependlist = []
- rreclist = []
+ rdependlist = set()
+ rreclist = set()
for package in rdepends:
for rdepend in rdepends[package]:
- rdependlist.append(rdepend)
+ rdependlist.add(rdepend)
rdependids.add(rdepend)
for package in rrecs:
for rdepend in rrecs[package]:
- rreclist.append(rdepend)
+ rreclist.add(rdepend)
rdependids.add(rdepend)
if rdependlist:
logger.debug2("Added runtime dependencies %s for %s", str(rdependlist), fn)
diff --git a/bitbake/lib/bb/tests/codeparser.py b/bitbake/lib/bb/tests/codeparser.py
index f6585fb3..c0d1362a 100644
--- a/bitbake/lib/bb/tests/codeparser.py
+++ b/bitbake/lib/bb/tests/codeparser.py
@@ -106,6 +106,46 @@ ${D}${libdir}/pkgconfig/*.pc
self.parseExpression("foo=$(echo bar)")
self.assertExecs(set(["echo"]))
+ def test_assign_subshell_expansion_quotes(self):
+ self.parseExpression('foo="$(echo bar)"')
+ self.assertExecs(set(["echo"]))
+
+ def test_assign_subshell_expansion_nested(self):
+ self.parseExpression('foo="$(func1 "$(func2 bar$(func3))")"')
+ self.assertExecs(set(["func1", "func2", "func3"]))
+
+ def test_assign_subshell_expansion_multiple(self):
+ self.parseExpression('foo="$(func1 "$(func2)") $(func3)"')
+ self.assertExecs(set(["func1", "func2", "func3"]))
+
+ def test_assign_subshell_expansion_escaped_quotes(self):
+ self.parseExpression('foo="\\"fo\\"o$(func1)"')
+ self.assertExecs(set(["func1"]))
+
+ def test_assign_subshell_expansion_empty(self):
+ self.parseExpression('foo="bar$()foo"')
+ self.assertExecs(set())
+
+ def test_assign_subshell_backticks(self):
+ self.parseExpression("foo=`echo bar`")
+ self.assertExecs(set(["echo"]))
+
+ def test_assign_subshell_backticks_quotes(self):
+ self.parseExpression('foo="`echo bar`"')
+ self.assertExecs(set(["echo"]))
+
+ def test_assign_subshell_backticks_multiple(self):
+ self.parseExpression('foo="`func1 bar` `func2`"')
+ self.assertExecs(set(["func1", "func2"]))
+
+ def test_assign_subshell_backticks_escaped_quotes(self):
+ self.parseExpression('foo="\\"fo\\"o`func1`"')
+ self.assertExecs(set(["func1"]))
+
+ def test_assign_subshell_backticks_empty(self):
+ self.parseExpression('foo="bar``foo"')
+ self.assertExecs(set())
+
def test_shell_unexpanded(self):
self.setEmptyVars(["QT_BASE_NAME"])
self.parseExpression('echo "${QT_BASE_NAME}"')
diff --git a/bitbake/lib/bb/tests/compression.py b/bitbake/lib/bb/tests/compression.py
index 95af3f96..16c297b3 100644
--- a/bitbake/lib/bb/tests/compression.py
+++ b/bitbake/lib/bb/tests/compression.py
@@ -66,8 +66,8 @@ class CompressionTests(object):
class LZ4Tests(CompressionTests, unittest.TestCase):
def setUp(self):
- if shutil.which("lz4c") is None:
- self.skipTest("'lz4c' not found")
+ if shutil.which("lz4") is None:
+ self.skipTest("'lz4' not found")
super().setUp()
@contextlib.contextmanager
diff --git a/bitbake/lib/bb/tests/cooker.py b/bitbake/lib/bb/tests/cooker.py
index 9e524ae3..db4104d4 100644
--- a/bitbake/lib/bb/tests/cooker.py
+++ b/bitbake/lib/bb/tests/cooker.py
@@ -7,12 +7,306 @@
#
import unittest
+import contextlib
import os
+import subprocess
+import sys
+import tempfile
+import time
import bb, bb.cooker
import re
import logging
-# Cooker tests
+
+class _BitbakeSubprocessTestCase(unittest.TestCase):
+ """Common helpers for tests that run bitbake/tinfoil in a subprocess.
+
+ Shared because every such subprocess can start a memory-resident bitbake
+ server (and, if BB_HASHSERVE=auto, a hashserv) rooted at TOPDIR, and both
+ must release that directory before the caller's TemporaryDirectory can be
+ safely removed.
+ """
+
+ def _run_subprocess(self, cmd, env, cwd):
+ proc = subprocess.run(
+ cmd,
+ env=env,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ universal_newlines=True,
+ cwd=cwd,
+ )
+ if proc.returncode:
+ self.fail('%s failed: %s' % (cmd, proc.stdout))
+ return proc.stdout
+
+ def _shutdown(self, builddir):
+ """Wait for the bitbake server and hashserv to release builddir.
+
+ Must run before the caller's TemporaryDirectory is removed, so it
+ cannot be a tearDown().
+ """
+ deadline = time.monotonic() + 30
+ while time.monotonic() < deadline:
+ if not any(os.path.exists(os.path.join(builddir, p))
+ for p in ('hashserve.sock', 'bitbake.lock')):
+ return
+ time.sleep(0.5)
+
+ @contextlib.contextmanager
+ def _build_dir(self, prefix='tinfoiltest'):
+ """TemporaryDirectory that also waits out _shutdown() before removal."""
+ with tempfile.TemporaryDirectory(prefix=prefix) as builddir:
+ try:
+ yield builddir
+ finally:
+ self._shutdown(builddir)
+
+
+class TinfoilTests(_BitbakeSubprocessTestCase):
+ """Tests for the Tinfoil API that require a running bitbake server."""
+
+ # Library directory containing bb.tinfoil
+ _bblib = os.path.realpath(os.path.join(os.path.dirname(__file__), '..', '..'))
+ # runqueue-tests BBPATH (provides the simple a1/b1/... test recipes)
+ _runqueuetests = os.path.realpath(os.path.join(os.path.dirname(__file__), 'runqueue-tests'))
+
+ failing_recipe = """\
+python do_install() {
+ bb.fatal("deliberate failure")
+}
+addtask install
+"""
+
+ # do_install genuinely depends on do_compile's output, unlike the
+ # a1 fixture recipe's dummy stamptask() tasks.
+ dependent_recipe = """\
+python do_compile() {
+ with open(d.expand("${TOPDIR}/compiled"), "w") as f:
+ f.write("compiled")
+}
+addtask compile
+
+python do_install() {
+ if not os.path.exists(d.expand("${TOPDIR}/compiled")):
+ bb.fatal("do_install ran without do_compile's output being present")
+}
+addtask install after do_compile
+"""
+
+ # Same contract as dependent_recipe, but with shell tasks, since a real
+ # recipe's do_install is usually shell (e.g. install/cp under pseudo)
+ # rather than a python function.
+ shell_dependent_recipe = """\
+do_compile() {
+ echo compiled > "${TOPDIR}/compiled"
+}
+addtask compile
+
+do_install() {
+ if [ ! -e "${TOPDIR}/compiled" ]; then
+ echo "do_install ran without do_compile's output being present" >&2
+ exit 1
+ fi
+}
+addtask install after do_compile
+"""
+
+ def _make_env(self, builddir, extra=None):
+ env = os.environ.copy()
+ env['PYTHONPATH'] = self._bblib + (':' + env['PYTHONPATH'] if 'PYTHONPATH' in env else '')
+ env['BBPATH'] = self._runqueuetests
+ env['BB_ENV_PASSTHROUGH_ADDITIONS'] = 'SSTATEVALID SLOWTASKS TOPDIR BB_HASHSERVE BB_SIGNATURE_HANDLER EXTRA_BBFILES'
+ env['SSTATEVALID'] = ''
+ env['SLOWTASKS'] = ''
+ env['TOPDIR'] = builddir
+ # TestEquivHash creates taint files so that force=True actually
+ # invalidates the task hash; the default noop siggen cannot do this.
+ env['BB_HASHSERVE'] = 'auto'
+ env['BB_SIGNATURE_HANDLER'] = 'TestEquivHash'
+ if extra:
+ env.update(extra)
+ return env
+
+ def _run_script(self, builddir, script, extra=None):
+ """Run script in a subprocess to isolate tinfoil's server lifecycle."""
+ return self._run_subprocess([sys.executable, '-c', script],
+ self._make_env(builddir, extra), builddir)
+
+ def _read_tasklog(self, builddir, cleanup=True):
+ tasklog = os.path.join(builddir, 'task.log')
+ tasks = []
+ if os.path.exists(tasklog):
+ with open(tasklog) as f:
+ tasks = [line.rstrip() for line in f]
+ if cleanup:
+ os.remove(tasklog)
+ return tasks
+
+ def test_run_prepared_task(self):
+ """tinfoil.run_prepared_task() reruns one task without resolving deps.
+
+ Uses do_install since that's the real devtool ide-sdk scenario: it
+ needs pseudo and so must run via bitbake, unlike the compile step which
+ the IDE invokes directly (e.g. via cmake/meson).
+
+ Builds a1 completely so all stamps/hashes are valid, then calls
+ run_prepared_task('a1', 'install') through the Python API and verifies
+ that only do_install re-runs while its intra-recipe predecessors
+ (fetch, unpack, patch, prepare_recipe_sysroot, configure, compile) are
+ skipped.
+ """
+ # The script runs inside a subprocess so that tinfoil's server
+ # lifecycle and environment modifications are isolated.
+ script = """
+import os, sys
+import bb.tinfoil
+
+builddir = os.environ['TOPDIR']
+tasklog = os.path.join(builddir, 'task.log')
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ # Full build so all stamps and hashes are valid.
+ tinfoil.build_targets(['a1'])
+ # Clear the log so only the run_prepared_task() entries are counted.
+ if os.path.exists(tasklog):
+ os.remove(tasklog)
+ # run_prepared_task() sets force=True (taint) and calls build_file_sync
+ # with the recipe file resolved via get_recipe_file(), bypassing the
+ # normal runqueue dependency resolver.
+ tinfoil.run_prepared_task('a1', 'install')
+"""
+ with self._build_dir() as builddir:
+ self._run_script(builddir, script)
+
+ tasks = self._read_tasklog(builddir)
+ self.assertEqual(tasks, ['a1:install'],
+ 'run_prepared_task should rerun only install, got: %s' % tasks)
+
+ def test_run_prepared_task_unbuilt(self):
+ """run_prepared_task() does not pull any dependency task into the runqueue.
+
+ buildFileInternal()'s taskonly=True clears task_deps['parents'] for
+ every task of the recipe, so do_install's intra-recipe predecessors
+ (fetch, unpack, patch, ...) are never added to the runqueue at all.
+ The recipe was never built here (no stamps exist for any of them), so
+ this is the case that would actually catch a regression: without
+ taskonly, those missing-stamp tasks would have to run to satisfy the
+ runqueue, and the tasklog assertion below would show more than just
+ 'a1:install'.
+
+ Whether do_install itself then succeeds or fails is a separate matter
+ that does depend on the recipe: this fixture's do_install has no real
+ prerequisites, so it succeeds here; test_run_prepared_task_unbuilt_dependent_fails()
+ uses a recipe whose do_install does have one, and fails instead.
+ """
+ script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ assert tinfoil.run_prepared_task('a1', 'install') is True
+"""
+ with self._build_dir() as builddir:
+ self._run_script(builddir, script)
+
+ tasks = self._read_tasklog(builddir)
+ self.assertEqual(tasks, ['a1:install'],
+ 'run_prepared_task should run no dependency task, got: %s' % tasks)
+
+ def test_run_prepared_task_unbuilt_dependent_fails(self):
+ """A task with a genuine dependency on a predecessor's output fails
+ when that predecessor never ran.
+
+ Unlike test_run_prepared_task_unbuilt()'s fixture recipe, whose tasks
+ are dummy stamptask() calls with no real prerequisites,
+ dependent_recipe's do_install actually needs do_compile's output.
+ run_prepared_task() only skips bitbake's own dependency resolution;
+ it does not make the prerequisites appear, exactly as documented:
+ "everything the task consumes must already be in place".
+ """
+ script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ assert tinfoil.run_prepared_task('dependent', 'install') is False
+"""
+ with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+ self._build_dir() as builddir:
+ with open(os.path.join(recipes, 'dependent.bb'), 'w') as f:
+ f.write(self.dependent_recipe)
+ self._run_script(builddir, script,
+ {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+ def test_run_prepared_task_unbuilt_dependent_fails_shell(self):
+ """Same as test_run_prepared_task_unbuilt_dependent_fails(), but with
+ shell do_compile/do_install tasks instead of python ones, matching how
+ a real recipe's do_install is usually written.
+ """
+ script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ assert tinfoil.run_prepared_task('shelldependent', 'install') is False
+"""
+ with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+ self._build_dir() as builddir:
+ with open(os.path.join(recipes, 'shelldependent.bb'), 'w') as f:
+ f.write(self.shell_dependent_recipe)
+ self._run_script(builddir, script,
+ {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+ def test_run_prepared_task_failure(self):
+ """A failing task makes run_prepared_task() return False, not raise."""
+ script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ assert tinfoil.run_prepared_task('failer', 'install') is False
+"""
+ with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+ self._build_dir() as builddir:
+ with open(os.path.join(recipes, 'failer.bb'), 'w') as f:
+ f.write(self.failing_recipe)
+ self._run_script(builddir, script,
+ {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+ def test_run_prepared_task_recipecache_self_heals(self):
+ """A normal full parse after run_prepared_task() sees every recipe.
+
+ run_prepared_task() -> buildFileInternal() -> parseConfiguration()
+ wipes and repopulates self.recipecaches[mc]/self.collections[mc]
+ for just the one recipe it builds - that has always been true of
+ "bitbake -b" too. Prove this is only a transient footprint: a
+ subsequent normal, full parse (as any non -b build would trigger)
+ must see the complete recipe set again, not just the one recipe
+ run_prepared_task() touched.
+ """
+ script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(quiet=2)
+ tinfoil.parse_recipes()
+ recipes_before = tinfoil.run_command('getRecipes')
+ assert len(recipes_before) > 1, 'test fixture should have more than one recipe'
+
+ tinfoil.run_prepared_task('a1', 'install')
+
+ tinfoil.parse_recipes()
+ recipes_after = tinfoil.run_command('getRecipes')
+ assert len(recipes_after) == len(recipes_before), (
+ 'recipe cache not fully restored after run_prepared_task(): '
+ 'before=%d after=%d' % (len(recipes_before), len(recipes_after)))
+"""
+ with self._build_dir() as builddir:
+ self._run_script(builddir, script)
+
+
class CookerTest(unittest.TestCase):
def setUp(self):
# At least one variable needs to be set
@@ -69,3 +363,49 @@ class CookerTest(unittest.TestCase):
expected = []
self.assertEqual(log_handler.logdata, expected)
+
+
+class BuildFileTest(_BitbakeSubprocessTestCase):
+ """Tests for the buildfile ("bitbake -b") mode."""
+
+ # parse-tests BBPATH: minimal bitbake.conf whose BBFILES honours
+ # EXTRA_BBFILES and already includes *.bbappend
+ _parsetests = os.path.realpath(os.path.join(os.path.dirname(__file__), "parse-tests"))
+
+ recipe = """\
+MARKER ??= "no-bbappend"
+python do_marker() {
+ with open(d.expand("${TOPDIR}/marker.log"), "w") as f:
+ f.write(d.getVar("MARKER"))
+}
+addtask marker
+"""
+
+ bbappend = 'MARKER = "bbappend-applied"\n'
+
+ def test_buildfile_applies_bbappends(self):
+ """bitbake -b must build the recipe with its bbappends applied.
+
+ buildFileInternal() looks the appends up in self.collections[mc], which
+ on the -b path is only ever populated by matchFiles().
+ """
+ with tempfile.TemporaryDirectory(prefix="buildfilerecipes") as recipes, \
+ self._build_dir(prefix="buildfiletest") as builddir:
+ recipe = os.path.join(recipes, "appendtest.bb")
+ with open(recipe, "w") as f:
+ f.write(self.recipe)
+ with open(os.path.join(recipes, "appendtest.bbappend"), "w") as f:
+ f.write(self.bbappend)
+
+ env = os.environ.copy()
+ env["BBPATH"] = self._parsetests
+ env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "TOPDIR EXTRA_BBFILES"
+ env["TOPDIR"] = builddir
+ env["EXTRA_BBFILES"] = "%s/*.bb %s/*.bbappend" % (recipes, recipes)
+
+ cmd = ["bitbake", "-b", recipe, "-c", "marker"]
+ self._run_subprocess(cmd, env, builddir)
+
+ with open(os.path.join(builddir, "marker.log")) as f:
+ self.assertEqual(f.read(), "bbappend-applied",
+ "bitbake -b did not apply the recipe's bbappend")
diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py
index cbc7c1ec..49177474 100644
--- a/bitbake/lib/bb/tests/data.py
+++ b/bitbake/lib/bb/tests/data.py
@@ -11,6 +11,7 @@ import unittest
import bb
import bb.data
import bb.parse
+import io
import logging
import os
@@ -405,6 +406,31 @@ class TestOverrides(unittest.TestCase):
bb.data.expandKeys(self.d)
self.assertEqual(self.d.getVar("VERSION"), "2")
+ # Test an :append whose override name is only resolved by key expansion
+ def test_append_in_expanded_override(self):
+ self.d.setVar("MACHINE", "qemux86")
+ self.d.setVar("PN", "gizmo")
+ self.d.setVar("OVERRIDES", "gizmo:pn-gizmo-qemux86")
+ self.d.setVar("TEST:${PN}", "base")
+ self.d.setVar("TEST:${PN}:append:pn-gizmo-${MACHINE}", " appended")
+ bb.data.expandKeys(self.d)
+ self.assertEqual(self.d.getVar("TEST"), "base appended")
+
+ # Test renaming to a key which is itself not expanded yet, as native.bbclass
+ # does. The dependent override keys must be left for expandKeys() to rename.
+ def test_rename_to_unexpanded_key_with_override(self):
+ self.d.setVar("BPN", "gizmo")
+ self.d.setVar("PN", "gizmo-native")
+ self.d.setVar("OVERRIDES", "class-target")
+ self.d.setVar("TEST:${PN}-lib", "base")
+ self.d.setVar("TEST:${PN}-lib:class-target", "target")
+ with LogRecord() as logs:
+ self.d.renameVar("TEST:${PN}-lib", "TEST:${BPN}-lib-native")
+ bb.data.expandKeys(self.d)
+ self.assertFalse(logContains("renameVar with equivalent keys", logs))
+ self.assertFalse(logContains("replaces original key", logs))
+ self.assertEqual(self.d.getVar("TEST:gizmo-lib-native"), "target")
+
def test_remove_with_override(self):
self.d.setVar("TEST:bar", "testvalue2")
self.d.setVar("TEST:some_val", "testvalue3 testvalue5")
@@ -450,17 +476,64 @@ class TestFlags(unittest.TestCase):
self.d = bb.data.init()
self.d.setVar("foo", "value of foo")
self.d.setVarFlag("foo", "flag1", "value of flag1")
+ self.d.setVarFlag("foo", "_defaultval_flag_flag1", "default of flag1")
self.d.setVarFlag("foo", "flag2", "value of flag2")
+ self.d.setVarFlag("foo", "_defaultval_flag_flag2", "default of flag2")
+ self.d.setVarFlag("foo", "flag3", "value of flag3")
+ self.d.setVarFlag("foo", "_defaultval_flag_flagnovalue", "default of flagnovalue")
def test_setflag(self):
self.assertEqual(self.d.getVarFlag("foo", "flag1", False), "value of flag1")
self.assertEqual(self.d.getVarFlag("foo", "flag2", False), "value of flag2")
+ self.assertDictEqual(
+ self.d.getVarFlags("foo"),
+ {
+ "flag1": "value of flag1",
+ "flag2": "value of flag2",
+ "flag3": "value of flag3",
+ "flagnovalue": "default of flagnovalue",
+ }
+ )
+ self.assertDictEqual(
+ self.d.getVarFlags("foo", internalflags=True),
+ {
+ "_content": "value of foo",
+ "flag1": "value of flag1",
+ "flag2": "value of flag2",
+ "flag3": "value of flag3",
+ "_defaultval_flag_flag1": "default of flag1",
+ "_defaultval_flag_flag2": "default of flag2",
+ "_defaultval_flag_flagnovalue": "default of flagnovalue",
+ }
+ )
def test_delflag(self):
self.d.delVarFlag("foo", "flag2")
+ self.d.delVarFlag("foo", "flag3")
self.assertEqual(self.d.getVarFlag("foo", "flag1", False), "value of flag1")
self.assertEqual(self.d.getVarFlag("foo", "flag2", False), None)
-
+ self.assertDictEqual(
+ self.d.getVarFlags("foo"),
+ {
+ "flag1": "value of flag1",
+ "flagnovalue": "default of flagnovalue",
+ }
+ )
+ self.assertDictEqual(
+ self.d.getVarFlags("foo", internalflags=True),
+ {
+ "_content": "value of foo",
+ "flag1": "value of flag1",
+ "_defaultval_flag_flag1": "default of flag1",
+ "_defaultval_flag_flagnovalue": "default of flagnovalue",
+ }
+ )
+
+ def test_delvar(self):
+ self.d.delVar("foo")
+ self.assertEqual(self.d.getVarFlag("foo", "flag1", False), None)
+ self.assertEqual(self.d.getVarFlag("foo", "flag2", False), None)
+ self.assertEqual(self.d.getVarFlags("foo", internalflags=True), None)
class Contains(unittest.TestCase):
def setUp(self):
@@ -570,4 +643,136 @@ class Serialize(unittest.TestCase):
self.assertEqual(newd.getVar('HELLO'), 'world')
self.assertEqual(newd.getVarFlag('HELLO', 'other'), 'planet')
+class EmitVar(unittest.TestCase):
+ def setUp(self):
+ self.d = bb.data.init()
+
+ self.d.setVar("foo", "value of foo")
+
+ self.d.setVar("foo_exported", "value of foo_exported")
+ self.d.setVarFlag("foo_exported", "export", "1")
+
+ self.d.setVar("foo_unexported", "value of foo_unexported")
+ self.d.setVarFlag("foo_unexported", "unexport", "1")
+
+ self.d.setVar("test_func", "echo test_func_l1\necho test_func_l2")
+ self.d.setVarFlag("test_func", "func", "1")
+
+ self.d.setVar("bad_chars", 'a"b\nc`d\ne$f')
+
+ @staticmethod
+ def get_output(out):
+ lines = [line for line in out.getvalue().splitlines()
+ if len(line) == 0 or line[0] != '#'
+ ]
+ out.truncate(0)
+ out.seek(0)
+ return lines
+
+ def test_simple_variable(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('foo', out, self.d, all=True)
+ self.assertEqual(self.get_output(out), ['foo="value of foo"'])
+
+ def test_exported(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('foo_exported', out, self.d)
+ self.assertEqual(self.get_output(out),
+ ['export foo_exported="value of foo_exported"'])
+
+ def test_unexported(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('foo_unexported', out, self.d)
+ self.assertEqual(self.get_output(out), ['unset foo_unexported'])
+
+ def test_function(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('test_func', out, self.d)
+ self.assertEqual(self.get_output(out), ['test_func() {',
+ 'echo test_func_l1',
+ 'echo test_func_l2',
+ '}'])
+
+ def test_all(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('foo', out, self.d, all=True)
+ self.assertEqual(self.get_output(out), ['foo="value of foo"'])
+
+ bb.data.emit_var('foo_exported', out, self.d, all=True)
+ self.assertEqual(self.get_output(out),
+ ['export foo_exported="value of foo_exported"'])
+ bb.data.emit_var('foo_unexported', out, self.d, all=True)
+
+ self.assertEqual(self.get_output(out), ['unset foo_unexported'])
+
+ bb.data.emit_var('test_func', out, self.d, all=True)
+ self.assertEqual(self.get_output(out), ['test_func() {',
+ 'echo test_func_l1',
+ 'echo test_func_l2',
+ '}'])
+
+ def test_not_all(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('foo', out, self.d)
+ self.assertEqual(self.get_output(out), [])
+
+ bb.data.emit_var('foo_exported', out, self.d)
+ self.assertEqual(self.get_output(out),
+ ['export foo_exported="value of foo_exported"'])
+
+ bb.data.emit_var('foo_unexported', out, self.d)
+ self.assertEqual(self.get_output(out), ['unset foo_unexported'])
+
+ bb.data.emit_var('test_func', out, self.d)
+ self.assertEqual(self.get_output(out), ['test_func() {',
+ 'echo test_func_l1',
+ 'echo test_func_l2',
+ '}'])
+
+ def test_not_set(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('random_name', out, self.d)
+ self.assertEqual(self.get_output(out), [])
+
+ def test_bad_chars(self):
+ out = io.StringIO()
+
+ bb.data.emit_var('bad_chars', out, self.d, all=True)
+ self.assertEqual(self.get_output(out), ['bad_chars="a\\"b \\',
+ 'c\\`d \\',
+ 'e\\$f"'])
+
+class ExportedVars(unittest.TestCase):
+ def test_expanded_before_returning(self):
+ # Called while the variable is set, read once it is gone: the value
+ # has to be the one from the call, not from the read.
+ d = bb.data.init()
+ d.setVar("TESTVAR", "${@os.environ.get('BB_TEST_EXPORT', 'gone')}")
+ d.setVarFlag("TESTVAR", "export", "1")
+
+ with bb.utils.environment(BB_TEST_EXPORT="present"):
+ exported = bb.data.exported_vars(d)
+
+ self.assertEqual(dict(exported), {"TESTVAR": "present"})
+ self.assertIsInstance(exported, list)
+
+ def test_unexpandable_value_warns_and_is_skipped(self):
+ # The warning belongs to the call, not to a later iteration.
+ d = bb.data.init()
+ d.setVar("TESTVAR", "value")
+ d.setVarFlag("TESTVAR", "export", "1")
+ d.setVar("TESTBROKEN", "${@int('not a number')}")
+ d.setVarFlag("TESTBROKEN", "export", "1")
+
+ with LogRecord() as logs:
+ exported = bb.data.exported_vars(d)
+ self.assertEqual(dict(exported), {"TESTVAR": "value"})
+ self.assertTrue(logContains("Unable to export ${TESTBROKEN}", logs))
diff --git a/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php b/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php
new file mode 100644
index 00000000..e27ee134
--- /dev/null
+++ b/bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/download.php
@@ -0,0 +1,3528 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+<title>MiniUPnP download zone</title>
+<link href="../css/miniupnp.css" rel="stylesheet" type="text/css"/>
+<meta name="description" content="files download of the miniupnp project"/>
+<meta name="keywords" content="upnp,download,openbsd,freebsd,linux,windows"/>
+<meta name="viewport" content="width=device-width" />
+<link href="rss.php" title="MiniUPnPd, MiniUPnPc and MiniSSDPd Files" type="application/rss+xml" rel="alternate" />
+<link rel="canonical" href="http://miniupnp.free.fr/files/" />
+<link rel="alternate" hreflang="fr" href="/files/index_fr.php" />
+<script async="async" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js" type="text/javascript"></script>
+<script type="text/javascript">
+ (adsbygoogle = window.adsbygoogle || []).push({
+ google_ad_client: "ca-pub-6883148866513192",
+ enable_page_level_ads: true
+ });
+</script>
+</head>
+<body>
+<h2>MiniUPnP Project</h2>
+
+<p align="center">
+<a href="../">Home</a> |
+<b>Downloads</b> |
+<a href="../devicelist.php">Compatibility list</a> |
+<a href="../libnatpmp.html">libnatpmp</a> |
+<a href="../minissdpd.html">MiniSSDPd</a> |
+<a href="../xchat-upnp.html">xchat upnp patch</a> |
+<a href="../search.html">Search</a> |
+<a href="https://miniupnp.tuxfamily.org/forum/">Forum</a>
+</p>
+<p align="center">
+<b>English</b> | <a href="/files/index_fr.php">Français</a>
+</p>
+
+<div align="center">
+<script type="text/javascript"><!--
+google_ad_client = "pub-6883148866513192";
+/* 728x90, created 7/10/08 */
+google_ad_slot = "0774293141";
+google_ad_width = 728;
+google_ad_height = 90;
+//-->
+</script>
+<script type="text/javascript"
+src="https://pagead2.googlesyndication.com/pagead/show_ads.js">
+</script>
+</div>
+
+<h2>MiniUPnP download zone</h2>
+<p>
+Find on this page the source of miniupnp and
+some related files. You will also find precompiled binaries
+of the UPnP client sample program for windows compiled using
+<a href="https://mingw.osdn.io/">MinGW</a>. There are also Windows
+binaries (including python module) automatically built using
+<a href="https://ci.appveyor.com/project/miniupnp/miniupnp/build/artifacts">AppVeyor</a>.
+</p>
+<p>If you just need one of the software installed on your machine,
+you probably don't need to download and compile the source files.
+It is very likely that a package/port already exists for
+your system/distribution. Refer to your system documentation
+to find how to search and install a package/port.
+Mac OS X does have port systems too : see
+<a href="http://www.macports.org/">MacPorts</a> or
+<a href="http://mxcl.github.com/homebrew/">Homebrew</a> or
+<a href="http://www.finkproject.org/">Fink</a>.
+</p>
+<p>
+The miniupnpc (client) sources have been successfully compiled
+under Windows XP/vista/7/10/etc. (using
+<a href="https://mingw.osdn.io/">MinGW</a>,
+<a href="https://www.mingw-w64.org/">Mingw-w64</a>
+or <a href="http://www.cygwin.com/">Cygwin</a>),
+Linux, OpenBSD, FreeBSD, NetBSD, DragonFlyBSD,
+Solaris, MacOS X and AmigaOS. <br/>
+The Makefile of the client is made for GNU make :
+check which version your system have
+with the command "make --version". On some systems, such as OpenBSD,
+you have to use "gmake". Under Windows with MinGW, GNU make is
+called "mingw32-make" and a sligthly modified version of the Makefile
+should be used : Makefile.mingw. Run "mingw32make.bat" to compile. <br/>
+If you have any compatibility problem, please post on the
+<a href="https://miniupnp.tuxfamily.org/forum/">forum</a>
+or contact me by email.
+</p>
+<!--
+<p>A devoted user compiled miniupnp<strong>c</strong> for
+Openwrt (currently Kamikaze 7.09)
+and his work is available here :
+<a href="http://replay.waybackmachine.org/20081120030628/http://www.myantihero.net/pub/openwrt/packages/">http://myantihero.net/pub/openwrt/packages/</a>.</p>
+-->
+<p>Get miniupnpc under AmigaOS 4 on
+<a href="http://os4depot.net/index.php?function=showfile&file=network/misc/miniupnpc.lha">OS4Depot</a>.
+</p>
+<p>
+Dario Meloni has made a Ruby Gem embedding miniupnpc :
+<a href="https://rubygems.org/gems/mupnp">https://rubygems.org/gems/mupnp</a>.
+</p>
+<p>
+The python module is available on pypi.org :
+<a href="https://pypi.org/project/miniupnpc/">pip install miniupnpc</a>.
+</p>
+<p>
+The daemon (starting in November 2006) compiles with BSD make under BSD
+and Solaris.<br/>
+To compile the daemon under linux, use "make -f Makefile.linux"<br/>
+To compile for <a href="http://openwrt.org/">OpenWRT</a>
+please read the README.openwrt file, or use the packages
+<a href="https://openwrt.org/packages/pkgdata/miniupnpd">miniupnpc</a> and
+<a href="https://openwrt.org/packages/pkgdata/miniupnpd">miniupnpd</a>.
+<!-- The
+<a href="http://www.x-wrt.org/">X-Wrt</a> project is providing
+precompiled ipkg packages for OpenWrt for both OpenWrt
+<a href="ftp://ftp.berlios.de/pub/xwrt/packages/">White Russian</a>
+and OpenWrt
+<a href="ftp://ftp.berlios.de/pub/xwrt/kamikaze/packages">kamikaze</a>.
+Check
+<a href="ftp://ftp.berlios.de/pub/xwrt/">ftp://ftp.berlios.de/pub/xwrt/</a>.
+For White Russian, take a look at
+<a href="http://jackassofalltrades.com/openwrt/">this</a>. -->
+<br/>
+<a href="http://pfsense.com">pfSense</a> users are advised to use the
+miniupnpd port available for their system. Recent versions of
+pfSense include MiniUPnPd in the base system.
+<br/>
+For <a href="http://en.wikipedia.org/wiki/WRT54G">Linksys WRT54G</a>
+and WRT54GL owners,
+<a href="http://sourceforge.net/projects/tarifa/">Tarifa firmware</a>
+is another alternative to get miniUPnPd running on the router.
+</p>
+<p>
+Please read README and
+LICENCE files included with the distribution for further informations.
+</p>
+<p>
+The MiniUPnP daemon (miniupnpd) is working under
+<a href="http://www.openbsd.org/">OpenBSD</a>,
+<a href="http://www.netbsd.org/">NetBSD</a>,
+<a href="http://www.freebsd.org/">FreeBSD</a>,
+<a href="http://www.dragonflybsd.org/">DragonFlyBSD</a>,
+<a href="http://www.apple.com/macosx/">Mac OS X</a> and
+(<a href="https://en.wikipedia.org/wiki/OpenSolaris">Open</a>)<a href="http://www.oracle.com/us/products/servers-storage/solaris/solaris11/overview/index.html">Solaris</a>
+with <a href="http://www.openbsd.org/faq/pf/">pf</a>,
+with <a href="https://en.wikipedia.org/wiki/IPFilter">IP Filter</a> or
+with <a href="http://en.wikipedia.org/wiki/Ipfirewall">ipfw</a>.
+The linux version uses either libiptc which permits to access
+<a href="http://netfilter.org/">netfilter</a>
+rules inside the kernel the same way as
+<a href="https://www.netfilter.org/projects/iptables/index.html">iptables</a>, or
+<a href="https://www.netfilter.org/projects/libnftnl/index.html">libnftnl</a>
+which is the equivalent for
+<a href="https://www.netfilter.org/projects/nftables/index.html">nftables</a>.
+</p>
+
+<p>Releases are now GPG signed with the key <a href="../A31ACAAF.asc">A31ACAAF</a>.
+Previous signing key was <a href="../A5C0863C.asc">A5C0863C</a>.
+Get it from your favorite
+<a href="https://pgp.mit.edu/pks/lookup?search=0xA31ACAAF&op=index&fingerprint=on">key server</a>.</p>
+
+<h4>REST API</h4>
+<p>You can use the REST API to get the latest releases available:</p>
+<ul>
+<li><a href="rest.php/tags/miniupnpd?count=1">rest.php/tags/miniupnpd?count=1</a>: latest miniupnpd.</li>
+<li><a href="rest.php/tags?count=1">rest.php/tags?count=1</a>: miniupnpc, miniupnpd and minissdpd.</li>
+</ul>
+
+<h4>You can help !</h4>
+<p>If you make a package/port for your favorite OS distribution,
+inform me so I can upload the package here or add a link to your
+repository.
+</p>
+
+<h4>Latest files</h4>
+<table>
+<tr><th>name</th>
+<th>size</th>
+<th>date</th>
+<th>comment</th>
+<th><!-- Changelog --></th>
+<th><!-- Signature --></th>
+</tr>
+<tr>
+ <td class="filename"><a href='miniupnpc-2.3.2.tar.gz'>miniupnpc-2.3.2.tar.gz</a></td>
+ <td class="filesize">140137</td>
+ <td class="filedate">05/03/2025 10:31</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="changelog.php?file=miniupnpc-2.3.2.tar.gz">changelog</a></td>
+ <td><a href="miniupnpc-2.3.2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='miniupnpd-2.3.7.tar.gz'>miniupnpd-2.3.7.tar.gz</a></td>
+ <td class="filesize">265329</td>
+ <td class="filedate">22/06/2024 22:31</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="changelog.php?file=miniupnpd-2.3.7.tar.gz">changelog</a></td>
+ <td><a href="miniupnpd-2.3.7.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='libnatpmp-20230423.tar.gz'>libnatpmp-20230423.tar.gz</a></td>
+ <td class="filesize">26506</td>
+ <td class="filedate">23/04/2023 11:02</td>
+ <td class="comment">latest libnatpmp source code</td>
+ <td><a href="changelog.php?file=libnatpmp-20230423.tar.gz">changelog</a></td>
+ <td><a href="libnatpmp-20230423.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='minissdpd-1.6.0.tar.gz'>minissdpd-1.6.0.tar.gz</a></td>
+ <td class="filesize">39077</td>
+ <td class="filedate">22/10/2022 18:41</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td><a href="changelog.php?file=minissdpd-1.6.0.tar.gz">changelog</a></td>
+ <td><a href="minissdpd-1.6.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='upnpc-exe-win32-20220515.zip'>upnpc-exe-win32-20220515.zip</a></td>
+ <td class="filesize">69503</td>
+ <td class="filedate">15/05/2022 14:31</td>
+ <td class="comment">Windows executable</td>
+ <td><a href="changelog.php?file=upnpc-exe-win32-20220515.zip">changelog</a></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='minissdpd-1.5.20211105.tar.gz'>minissdpd-1.5.20211105.tar.gz</a></td>
+ <td class="filesize">38870</td>
+ <td class="filedate">04/11/2021 23:34</td>
+ <td class="comment">latest MiniSSDPd source code</td>
+ <td><a href="changelog.php?file=minissdpd-1.5.20211105.tar.gz">changelog</a></td>
+ <td><a href="minissdpd-1.5.20211105.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='miniupnpc-2.1.20201016.tar.gz'>miniupnpc-2.1.20201016.tar.gz</a></td>
+ <td class="filesize">97682</td>
+ <td class="filedate">15/10/2020 22:31</td>
+ <td class="comment">latest MiniUPnP client source code</td>
+ <td><a href="changelog.php?file=miniupnpc-2.1.20201016.tar.gz">changelog</a></td>
+ <td><a href="miniupnpc-2.1.20201016.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='miniupnpd-2.1.20200510.tar.gz'>miniupnpd-2.1.20200510.tar.gz</a></td>
+ <td class="filesize">245426</td>
+ <td class="filedate">10/05/2020 18:23</td>
+ <td class="comment">latest MiniUPnP daemon source code</td>
+ <td><a href="changelog.php?file=miniupnpd-2.1.20200510.tar.gz">changelog</a></td>
+ <td><a href="miniupnpd-2.1.20200510.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='xchat-upnp20110811.patch'>xchat-upnp20110811.patch</a></td>
+ <td class="filesize">10329</td>
+ <td class="filedate">11/08/2011 15:18</td>
+ <td class="comment">Patch to add UPnP capabilities to xchat</td>
+ <td><a href="changelog.php?file=xchat-upnp20110811.patch">changelog</a></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='minidlna_1.0.21.minissdp1.patch'>minidlna_1.0.21.minissdp1.patch</a></td>
+ <td class="filesize">7598</td>
+ <td class="filedate">25/07/2011 14:57</td>
+ <td class="comment">Patch for MiniDLNA to use miniSSDPD</td>
+ <td><a href="changelog.php?file=minidlna_1.0.21.minissdp1.patch">changelog</a></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='miniupnpc-new20060630.tar.gz'>miniupnpc-new20060630.tar.gz</a></td>
+ <td class="filesize">14840</td>
+ <td class="filedate">04/11/2006 18:16</td>
+ <td class="comment">João Paulo Barraca version of the upnp client</td>
+ <td><a href="changelog.php?file=miniupnpc-new20060630.tar.gz">changelog</a></td>
+ <td></td>
+</tr>
+</table>
+
+<h4>All files</h4>
+<table>
+<tr><th>name</th>
+<th>size</th>
+<th>date</th>
+<th>comment</th>
+<th><!-- signature --></th>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.3.2.tar.gz'>miniupnpc-2.3.2.tar.gz</a></td>
+ <td class="filesize">140137</td>
+ <td class="filedate">05/03/2025 10:31:36 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.3.2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.3.1.tar.gz'>miniupnpc-2.3.1.tar.gz</a></td>
+ <td class="filesize">139499</td>
+ <td class="filedate">23/02/2025 16:44:16 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.3.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.3.0.tar.gz'>miniupnpc-2.3.0.tar.gz</a></td>
+ <td class="filesize">105071</td>
+ <td class="filedate">10/01/2025 23:16:45 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.3.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.7.tar.gz'>miniupnpd-2.3.7.tar.gz</a></td>
+ <td class="filesize">265329</td>
+ <td class="filedate">22/06/2024 22:31:38 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.7.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.8.tar.gz'>miniupnpc-2.2.8.tar.gz</a></td>
+ <td class="filesize">104603</td>
+ <td class="filedate">08/06/2024 22:13:39 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.8.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.6.tar.gz'>miniupnpd-2.3.6.tar.gz</a></td>
+ <td class="filesize">263018</td>
+ <td class="filedate">19/03/2024 23:39:51 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.6.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.7.tar.gz'>miniupnpc-2.2.7.tar.gz</a></td>
+ <td class="filesize">104258</td>
+ <td class="filedate">19/03/2024 23:25:18 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.7.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.5.tar.gz'>miniupnpd-2.3.5.tar.gz</a></td>
+ <td class="filesize">261952</td>
+ <td class="filedate">02/03/2024 11:04:07 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.5.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.4.tar.gz'>miniupnpd-2.3.4.tar.gz</a></td>
+ <td class="filesize">260810</td>
+ <td class="filedate">04/01/2024 00:53:17 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.4.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.6.tar.gz'>miniupnpc-2.2.6.tar.gz</a></td>
+ <td class="filesize">103949</td>
+ <td class="filedate">04/01/2024 00:27:14 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.6.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.5.tar.gz'>miniupnpc-2.2.5.tar.gz</a></td>
+ <td class="filesize">103654</td>
+ <td class="filedate">11/06/2023 23:14:56 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.5.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20230423.tar.gz'>libnatpmp-20230423.tar.gz</a></td>
+ <td class="filesize">26506</td>
+ <td class="filedate">23/04/2023 11:02:09 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td><a href="libnatpmp-20230423.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.3.tar.gz'>miniupnpd-2.3.3.tar.gz</a></td>
+ <td class="filesize">260079</td>
+ <td class="filedate">17/02/2023 03:07:46 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.3.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.2.tar.gz'>miniupnpd-2.3.2.tar.gz</a></td>
+ <td class="filesize">259686</td>
+ <td class="filedate">19/01/2023 23:18:08 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.6.0.tar.gz'>minissdpd-1.6.0.tar.gz</a></td>
+ <td class="filesize">39077</td>
+ <td class="filedate">22/10/2022 18:41:54 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td><a href="minissdpd-1.6.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.4.tar.gz'>miniupnpc-2.2.4.tar.gz</a></td>
+ <td class="filesize">102932</td>
+ <td class="filedate">21/10/2022 21:01:01 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.4.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.1.tar.gz'>miniupnpd-2.3.1.tar.gz</a></td>
+ <td class="filesize">258050</td>
+ <td class="filedate">16/10/2022 05:58:44 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20220515.zip'>upnpc-exe-win32-20220515.zip</a></td>
+ <td class="filesize">69503</td>
+ <td class="filedate">15/05/2022 14:31:25 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=hexchat-2.16.patch'>hexchat-2.16.patch</a></td>
+ <td class="filesize">8147</td>
+ <td class="filedate">19/03/2022 16:52:05 +0000</td>
+ <td class="comment"></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.3.0.tar.gz'>miniupnpd-2.3.0.tar.gz</a></td>
+ <td class="filesize">256069</td>
+ <td class="filedate">23/01/2022 00:23:32 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.3.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20211105.tar.gz'>minissdpd-1.5.20211105.tar.gz</a></td>
+ <td class="filesize">38870</td>
+ <td class="filedate">04/11/2021 23:34:49 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td><a href="minissdpd-1.5.20211105.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.3.tar.gz'>miniupnpc-2.2.3.tar.gz</a></td>
+ <td class="filesize">101360</td>
+ <td class="filedate">28/09/2021 21:43:32 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.3.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.3.tar.gz'>miniupnpd-2.2.3.tar.gz</a></td>
+ <td class="filesize">254752</td>
+ <td class="filedate">21/08/2021 08:35:13 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.3.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.2.tar.gz'>miniupnpd-2.2.2.tar.gz</a></td>
+ <td class="filesize">250649</td>
+ <td class="filedate">13/05/2021 11:30:11 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.2.tar.gz'>miniupnpc-2.2.2.tar.gz</a></td>
+ <td class="filesize">100008</td>
+ <td class="filedate">02/03/2021 23:44:52 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.1.tar.gz'>miniupnpd-2.2.1.tar.gz</a></td>
+ <td class="filesize">250023</td>
+ <td class="filedate">20/12/2020 18:08:08 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.1.tar.gz'>miniupnpc-2.2.1.tar.gz</a></td>
+ <td class="filesize">99595</td>
+ <td class="filedate">20/12/2020 18:08:02 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.2.0.tar.gz'>miniupnpc-2.2.0.tar.gz</a></td>
+ <td class="filesize">98348</td>
+ <td class="filedate">09/11/2020 19:51:24 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.2.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.0.tar.gz'>miniupnpd-2.2.0.tar.gz</a></td>
+ <td class="filesize">249858</td>
+ <td class="filedate">31/10/2020 09:20:59 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.0-RC3.tar.gz'>miniupnpd-2.2.0-RC3.tar.gz</a></td>
+ <td class="filesize">249879</td>
+ <td class="filedate">30/10/2020 21:49:49 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.0-RC3.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20201016.tar.gz'>miniupnpc-2.1.20201016.tar.gz</a></td>
+ <td class="filesize">97682</td>
+ <td class="filedate">15/10/2020 22:31:09 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20201016.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.0-RC2.tar.gz'>miniupnpd-2.2.0-RC2.tar.gz</a></td>
+ <td class="filesize">248756</td>
+ <td class="filedate">28/09/2020 21:57:22 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.0-RC2.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20200928.tar.gz'>miniupnpc-2.1.20200928.tar.gz</a></td>
+ <td class="filesize">96508</td>
+ <td class="filedate">28/09/2020 21:56:09 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20200928.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20200928.tar.gz'>minissdpd-1.5.20200928.tar.gz</a></td>
+ <td class="filesize">37860</td>
+ <td class="filedate">28/09/2020 21:55:40 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td><a href="minissdpd-1.5.20200928.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.0-RC1.tar.gz'>miniupnpd-2.2.0-RC1.tar.gz</a></td>
+ <td class="filesize">247772</td>
+ <td class="filedate">06/06/2020 18:34:50 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.0-RC1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.2.0-RC0.tar.gz'>miniupnpd-2.2.0-RC0.tar.gz</a></td>
+ <td class="filesize">245507</td>
+ <td class="filedate">16/05/2020 18:03:17 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.2.0-RC0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20200510.tar.gz'>miniupnpd-2.1.20200510.tar.gz</a></td>
+ <td class="filesize">245426</td>
+ <td class="filedate">10/05/2020 18:23:13 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20200510.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20200329.tar.gz'>miniupnpd-2.1.20200329.tar.gz</a></td>
+ <td class="filesize">243725</td>
+ <td class="filedate">29/03/2020 09:11:02 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20200329.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20191224.tar.gz'>miniupnpc-2.1.20191224.tar.gz</a></td>
+ <td class="filesize">94740</td>
+ <td class="filedate">23/12/2019 23:37:32 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20191224.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20191006.tar.gz'>miniupnpd-2.1.20191006.tar.gz</a></td>
+ <td class="filesize">243255</td>
+ <td class="filedate">06/10/2019 21:02:31 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20191006.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20191005.tar.gz'>miniupnpd-2.1.20191005.tar.gz</a></td>
+ <td class="filesize">244100</td>
+ <td class="filedate">05/10/2019 21:33:08 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20191005.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20191003.tar.gz'>miniupnpd-2.1.20191003.tar.gz</a></td>
+ <td class="filesize">243287</td>
+ <td class="filedate">02/10/2019 22:23:51 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20191003.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190924.tar.gz'>miniupnpd-2.1.20190924.tar.gz</a></td>
+ <td class="filesize">241008</td>
+ <td class="filedate">24/09/2019 11:58:15 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190924.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190902.tar.gz'>miniupnpd-2.1.20190902.tar.gz</a></td>
+ <td class="filesize">240742</td>
+ <td class="filedate">01/09/2019 23:03:03 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190902.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190824.tar.gz'>miniupnpd-2.1.20190824.tar.gz</a></td>
+ <td class="filesize">240490</td>
+ <td class="filedate">24/08/2019 09:21:52 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190824.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20190824.tar.gz'>minissdpd-1.5.20190824.tar.gz</a></td>
+ <td class="filesize">37300</td>
+ <td class="filedate">24/08/2019 09:17:32 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td><a href="minissdpd-1.5.20190824.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20190824.tar.gz'>miniupnpc-2.1.20190824.tar.gz</a></td>
+ <td class="filesize">94564</td>
+ <td class="filedate">24/08/2019 09:12:50 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20190824.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190630.tar.gz'>miniupnpd-2.1.20190630.tar.gz</a></td>
+ <td class="filesize">240466</td>
+ <td class="filedate">30/06/2019 20:27:38 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190630.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190625.tar.gz'>miniupnpd-2.1.20190625.tar.gz</a></td>
+ <td class="filesize">240120</td>
+ <td class="filedate">25/06/2019 21:33:49 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190625.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20190625.tar.gz'>miniupnpc-2.1.20190625.tar.gz</a></td>
+ <td class="filesize">94461</td>
+ <td class="filedate">25/06/2019 21:33:26 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20190625.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190502.tar.gz'>miniupnpd-2.1.20190502.tar.gz</a></td>
+ <td class="filesize">236052</td>
+ <td class="filedate">02/05/2019 17:22:23 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190502.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20190408.tar.gz'>miniupnpc-2.1.20190408.tar.gz</a></td>
+ <td class="filesize">94216</td>
+ <td class="filedate">08/04/2019 12:50:21 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20190408.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190408.tar.gz'>miniupnpd-2.1.20190408.tar.gz</a></td>
+ <td class="filesize">235989</td>
+ <td class="filedate">08/04/2019 12:50:01 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190408.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20190403.tar.gz'>miniupnpc-2.1.20190403.tar.gz</a></td>
+ <td class="filesize">94204</td>
+ <td class="filedate">03/04/2019 15:41:36 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20190403.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190403.tar.gz'>miniupnpd-2.1.20190403.tar.gz</a></td>
+ <td class="filesize">235909</td>
+ <td class="filedate">03/04/2019 15:41:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190403.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20190210.tar.gz'>minissdpd-1.5.20190210.tar.gz</a></td>
+ <td class="filesize">37227</td>
+ <td class="filedate">10/02/2019 15:21:49 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td><a href="minissdpd-1.5.20190210.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.20190210.tar.gz'>miniupnpc-2.1.20190210.tar.gz</a></td>
+ <td class="filesize">94125</td>
+ <td class="filedate">10/02/2019 12:46:09 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td><a href="miniupnpc-2.1.20190210.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20190210.tar.gz'>miniupnpd-2.1.20190210.tar.gz</a></td>
+ <td class="filesize">235093</td>
+ <td class="filedate">10/02/2019 11:20:11 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20190210.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.20180706.tar.gz'>miniupnpd-2.1.20180706.tar.gz</a></td>
+ <td class="filesize">233675</td>
+ <td class="filedate">06/07/2018 12:44:24 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td><a href="miniupnpd-2.1.20180706.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.1.tar.gz'>miniupnpd-2.1.tar.gz</a></td>
+ <td class="filesize">225458</td>
+ <td class="filedate">08/05/2018 21:50:32 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.1.tar.gz'>miniupnpc-2.1.tar.gz</a></td>
+ <td class="filesize">91914</td>
+ <td class="filedate">07/05/2018 11:10:59 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td><a href="miniupnpc-2.1.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180503.tar.gz'>miniupnpd-2.0.20180503.tar.gz</a></td>
+ <td class="filesize">225454</td>
+ <td class="filedate">03/05/2018 08:33:10 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20180503.tar.gz'>miniupnpc-2.0.20180503.tar.gz</a></td>
+ <td class="filesize">88207</td>
+ <td class="filedate">03/05/2018 08:31:22 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180422.tar.gz'>miniupnpd-2.0.20180422.tar.gz</a></td>
+ <td class="filesize">224942</td>
+ <td class="filedate">22/04/2018 19:48:54 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180412.tar.gz'>miniupnpd-2.0.20180412.tar.gz</a></td>
+ <td class="filesize">224831</td>
+ <td class="filedate">12/04/2018 08:16:25 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180410.tar.gz'>miniupnpd-2.0.20180410.tar.gz</a></td>
+ <td class="filesize">224736</td>
+ <td class="filedate">10/04/2018 07:58:28 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20180410.tar.gz'>miniupnpc-2.0.20180410.tar.gz</a></td>
+ <td class="filesize">87363</td>
+ <td class="filedate">10/04/2018 07:52:55 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20180406.tar.gz'>miniupnpc-2.0.20180406.tar.gz</a></td>
+ <td class="filesize">87374</td>
+ <td class="filedate">06/04/2018 10:55:21 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20180223.tar.gz'>minissdpd-1.5.20180223.tar.gz</a></td>
+ <td class="filesize">36179</td>
+ <td class="filedate">23/02/2018 14:24:07 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20180222.tar.gz'>miniupnpc-2.0.20180222.tar.gz</a></td>
+ <td class="filesize">87018</td>
+ <td class="filedate">22/02/2018 15:09:24 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180222.tar.gz'>miniupnpd-2.0.20180222.tar.gz</a></td>
+ <td class="filesize">223697</td>
+ <td class="filedate">22/02/2018 15:09:14 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20180203.tar.gz'>miniupnpd-2.0.20180203.tar.gz</a></td>
+ <td class="filesize">223084</td>
+ <td class="filedate">03/02/2018 22:34:46 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20180203.tar.gz'>miniupnpc-2.0.20180203.tar.gz</a></td>
+ <td class="filesize">86772</td>
+ <td class="filedate">03/02/2018 22:34:32 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20180203.tar.gz'>minissdpd-1.5.20180203.tar.gz</a></td>
+ <td class="filesize">35848</td>
+ <td class="filedate">03/02/2018 22:33:08 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20171212.tar.gz'>miniupnpc-2.0.20171212.tar.gz</a></td>
+ <td class="filesize">86607</td>
+ <td class="filedate">12/12/2017 12:03:38 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20171212.tar.gz'>miniupnpd-2.0.20171212.tar.gz</a></td>
+ <td class="filesize">222617</td>
+ <td class="filedate">12/12/2017 12:03:32 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20171102.tar.gz'>miniupnpc-2.0.20171102.tar.gz</a></td>
+ <td class="filesize">86363</td>
+ <td class="filedate">02/11/2017 17:58:34 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20170509.tar.gz'>miniupnpc-2.0.20170509.tar.gz</a></td>
+ <td class="filesize">86055</td>
+ <td class="filedate">09/05/2017 10:14:56 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20170421.tar.gz'>miniupnpc-2.0.20170421.tar.gz</a></td>
+ <td class="filesize">85984</td>
+ <td class="filedate">21/04/2017 12:02:26 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20170421.tar.gz'>miniupnpd-2.0.20170421.tar.gz</a></td>
+ <td class="filesize">219191</td>
+ <td class="filedate">21/04/2017 12:02:06 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.20161216.tar.gz'>miniupnpd-2.0.20161216.tar.gz</a></td>
+ <td class="filesize">218119</td>
+ <td class="filedate">16/12/2016 09:34:08 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.20161216.tar.gz'>miniupnpc-2.0.20161216.tar.gz</a></td>
+ <td class="filesize">85780</td>
+ <td class="filedate">16/12/2016 09:34:03 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20161216.tar.gz'>minissdpd-1.5.20161216.tar.gz</a></td>
+ <td class="filesize">35078</td>
+ <td class="filedate">16/12/2016 09:33:59 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-2.0.tar.gz'>miniupnpd-2.0.tar.gz</a></td>
+ <td class="filesize">217802</td>
+ <td class="filedate">19/04/2016 21:12:01 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td><a href="miniupnpd-2.0.tar.gz.sig">Signature</a></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-2.0.tar.gz'>miniupnpc-2.0.tar.gz</a></td>
+ <td class="filesize">85287</td>
+ <td class="filedate">19/04/2016 21:07:52 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20160301.tar.gz'>minissdpd-1.5.20160301.tar.gz</a></td>
+ <td class="filesize">34827</td>
+ <td class="filedate">01/03/2016 18:08:23 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20160222.tar.gz'>miniupnpd-1.9.20160222.tar.gz</a></td>
+ <td class="filesize">217541</td>
+ <td class="filedate">22/02/2016 10:21:40 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20160216.tar.gz'>miniupnpd-1.9.20160216.tar.gz</a></td>
+ <td class="filesize">217007</td>
+ <td class="filedate">16/02/2016 12:41:44 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20160212.tar.gz'>miniupnpd-1.9.20160212.tar.gz</a></td>
+ <td class="filesize">215866</td>
+ <td class="filedate">12/02/2016 15:22:04 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20160209.tar.gz'>miniupnpd-1.9.20160209.tar.gz</a></td>
+ <td class="filesize">213416</td>
+ <td class="filedate">09/02/2016 09:47:03 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20160209.tar.gz'>miniupnpc-1.9.20160209.tar.gz</a></td>
+ <td class="filesize">85268</td>
+ <td class="filedate">09/02/2016 09:44:50 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.20160119.tar.gz'>minissdpd-1.5.20160119.tar.gz</a></td>
+ <td class="filesize">34711</td>
+ <td class="filedate">19/01/2016 13:39:51 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20160113.tar.gz'>miniupnpd-1.9.20160113.tar.gz</a></td>
+ <td class="filesize">211437</td>
+ <td class="filedate">13/01/2016 16:03:14 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.5.tar.gz'>minissdpd-1.5.tar.gz</a></td>
+ <td class="filesize">34404</td>
+ <td class="filedate">13/01/2016 15:26:53 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20151212.tar.gz'>miniupnpd-1.9.20151212.tar.gz</a></td>
+ <td class="filesize">210912</td>
+ <td class="filedate">12/12/2015 10:06:07 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20151118.tar.gz'>miniupnpd-1.9.20151118.tar.gz</a></td>
+ <td class="filesize">210322</td>
+ <td class="filedate">18/11/2015 08:59:46 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20151026.tar.gz'>miniupnpc-1.9.20151026.tar.gz</a></td>
+ <td class="filesize">84208</td>
+ <td class="filedate">26/10/2015 17:07:34 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20151008.tar.gz'>miniupnpc-1.9.20151008.tar.gz</a></td>
+ <td class="filesize">83538</td>
+ <td class="filedate">08/10/2015 16:22:28 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150922.tar.gz'>miniupnpd-1.9.20150922.tar.gz</a></td>
+ <td class="filesize">208700</td>
+ <td class="filedate">22/09/2015 10:21:50 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20150918.zip'>upnpc-exe-win32-20150918.zip</a></td>
+ <td class="filesize">100004</td>
+ <td class="filedate">18/09/2015 12:50:51 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150917.tar.gz'>miniupnpc-1.9.20150917.tar.gz</a></td>
+ <td class="filesize">82609</td>
+ <td class="filedate">17/09/2015 14:09:14 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20150824.zip'>upnpc-exe-win32-20150824.zip</a></td>
+ <td class="filesize">99520</td>
+ <td class="filedate">24/08/2015 15:25:18 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.4.tar.gz'>minissdpd-1.4.tar.gz</a></td>
+ <td class="filesize">32017</td>
+ <td class="filedate">06/08/2015 13:38:37 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150730.tar.gz'>miniupnpc-1.9.20150730.tar.gz</a></td>
+ <td class="filesize">81431</td>
+ <td class="filedate">29/07/2015 22:10:00 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150721.tar.gz'>miniupnpd-1.9.20150721.tar.gz</a></td>
+ <td class="filesize">207562</td>
+ <td class="filedate">21/07/2015 13:35:51 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150721.tar.gz'>miniupnpc-1.9.20150721.tar.gz</a></td>
+ <td class="filesize">80521</td>
+ <td class="filedate">21/07/2015 13:27:00 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20150609.tar.gz'>libnatpmp-20150609.tar.gz</a></td>
+ <td class="filesize">24392</td>
+ <td class="filedate">09/06/2015 15:40:28 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150609.tar.gz'>miniupnpc-1.9.20150609.tar.gz</a></td>
+ <td class="filesize">79311</td>
+ <td class="filedate">09/06/2015 15:39:48 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150609.tar.gz'>miniupnpd-1.9.20150609.tar.gz</a></td>
+ <td class="filesize">207088</td>
+ <td class="filedate">09/06/2015 15:39:36 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.3.20150527.tar.gz'>minissdpd-1.3.20150527.tar.gz</a></td>
+ <td class="filesize">31025</td>
+ <td class="filedate">27/05/2015 09:17:15 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150522.tar.gz'>miniupnpc-1.9.20150522.tar.gz</a></td>
+ <td class="filesize">79080</td>
+ <td class="filedate">22/05/2015 11:02:27 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.3.20150522.tar.gz'>minissdpd-1.3.20150522.tar.gz</a></td>
+ <td class="filesize">30334</td>
+ <td class="filedate">22/05/2015 11:02:04 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150430.tar.gz'>miniupnpd-1.9.20150430.tar.gz</a></td>
+ <td class="filesize">205930</td>
+ <td class="filedate">30/04/2015 09:09:27 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150430.tar.gz'>miniupnpc-1.9.20150430.tar.gz</a></td>
+ <td class="filesize">78459</td>
+ <td class="filedate">30/04/2015 08:39:31 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150427.tar.gz'>miniupnpc-1.9.20150427.tar.gz</a></td>
+ <td class="filesize">78424</td>
+ <td class="filedate">27/04/2015 16:08:42 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150427.tar.gz'>miniupnpd-1.9.20150427.tar.gz</a></td>
+ <td class="filesize">191157</td>
+ <td class="filedate">27/04/2015 16:08:27 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20150307.tar.gz'>miniupnpd-1.9.20150307.tar.gz</a></td>
+ <td class="filesize">190913</td>
+ <td class="filedate">07/03/2015 16:11:51 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20150206.tar.gz'>miniupnpc-1.9.20150206.tar.gz</a></td>
+ <td class="filesize">76864</td>
+ <td class="filedate">06/02/2015 14:38:00 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20141209.tar.gz'>miniupnpd-1.9.20141209.tar.gz</a></td>
+ <td class="filesize">193183</td>
+ <td class="filedate">09/12/2014 09:58:34 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.3.tar.gz'>minissdpd-1.3.tar.gz</a></td>
+ <td class="filesize">30326</td>
+ <td class="filedate">09/12/2014 09:57:30 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20141204.tar.gz'>minissdpd-1.2.20141204.tar.gz</a></td>
+ <td class="filesize">26978</td>
+ <td class="filedate">04/12/2014 10:55:26 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20141204.tar.gz'>miniupnpd-1.9.20141204.tar.gz</a></td>
+ <td class="filesize">192597</td>
+ <td class="filedate">04/12/2014 10:55:03 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20141128.tar.gz'>minissdpd-1.2.20141128.tar.gz</a></td>
+ <td class="filesize">26795</td>
+ <td class="filedate">28/11/2014 16:33:10 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20141128.tar.gz'>miniupnpd-1.9.20141128.tar.gz</a></td>
+ <td class="filesize">192558</td>
+ <td class="filedate">28/11/2014 13:31:36 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20141128.tar.gz'>miniupnpc-1.9.20141128.tar.gz</a></td>
+ <td class="filesize">76541</td>
+ <td class="filedate">28/11/2014 13:31:15 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20141117.tar.gz'>miniupnpc-1.9.20141117.tar.gz</a></td>
+ <td class="filesize">73865</td>
+ <td class="filedate">17/11/2014 09:51:36 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20141113.tar.gz'>miniupnpc-1.9.20141113.tar.gz</a></td>
+ <td class="filesize">72857</td>
+ <td class="filedate">13/11/2014 10:36:44 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20141108.tar.gz'>minissdpd-1.2.20141108.tar.gz</a></td>
+ <td class="filesize">22001</td>
+ <td class="filedate">08/11/2014 13:55:41 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20141108.tar.gz'>miniupnpc-1.9.20141108.tar.gz</a></td>
+ <td class="filesize">72781</td>
+ <td class="filedate">08/11/2014 13:53:48 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.20141108.tar.gz'>miniupnpd-1.9.20141108.tar.gz</a></td>
+ <td class="filesize">192413</td>
+ <td class="filedate">08/11/2014 13:53:38 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.9.tar.gz'>miniupnpd-1.9.tar.gz</a></td>
+ <td class="filesize">192183</td>
+ <td class="filedate">27/10/2014 16:45:34 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20141027.tar.gz'>miniupnpc-1.9.20141027.tar.gz</a></td>
+ <td class="filesize">76763</td>
+ <td class="filedate">27/10/2014 16:45:25 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20141022.tar.gz'>miniupnpd-1.8.20141022.tar.gz</a></td>
+ <td class="filesize">191630</td>
+ <td class="filedate">22/10/2014 09:17:41 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20141021.tar.gz'>miniupnpd-1.8.20141021.tar.gz</a></td>
+ <td class="filesize">191270</td>
+ <td class="filedate">21/10/2014 14:18:58 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20140911.tar.gz'>miniupnpc-1.9.20140911.tar.gz</a></td>
+ <td class="filesize">76855</td>
+ <td class="filedate">11/09/2014 14:15:23 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20140906.tar.gz'>minissdpd-1.2.20140906.tar.gz</a></td>
+ <td class="filesize">21956</td>
+ <td class="filedate">06/09/2014 08:34:10 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140906.tar.gz'>miniupnpd-1.8.20140906.tar.gz</a></td>
+ <td class="filesize">191183</td>
+ <td class="filedate">06/09/2014 08:34:02 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20140906.tar.gz'>miniupnpc-1.9.20140906.tar.gz</a></td>
+ <td class="filesize">76791</td>
+ <td class="filedate">06/09/2014 08:33:45 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20140701.tar.gz'>miniupnpc-1.9.20140701.tar.gz</a></td>
+ <td class="filesize">76735</td>
+ <td class="filedate">01/07/2014 13:06:51 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20140610.tar.gz'>miniupnpc-1.9.20140610.tar.gz</a></td>
+ <td class="filesize">76674</td>
+ <td class="filedate">10/06/2014 10:28:27 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20140610.tar.gz'>minissdpd-1.2.20140610.tar.gz</a></td>
+ <td class="filesize">21909</td>
+ <td class="filedate">10/06/2014 10:03:29 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140523.tar.gz'>miniupnpd-1.8.20140523.tar.gz</a></td>
+ <td class="filesize">190936</td>
+ <td class="filedate">23/05/2014 15:48:03 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20140422.zip'>upnpc-exe-win32-20140422.zip</a></td>
+ <td class="filesize">97505</td>
+ <td class="filedate">22/04/2014 10:10:07 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140422.tar.gz'>miniupnpd-1.8.20140422.tar.gz</a></td>
+ <td class="filesize">187225</td>
+ <td class="filedate">22/04/2014 08:58:56 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140401.tar.gz'>miniupnpd-1.8.20140401.tar.gz</a></td>
+ <td class="filesize">183131</td>
+ <td class="filedate">01/04/2014 10:07:20 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.20140401.tar.gz'>miniupnpc-1.9.20140401.tar.gz</a></td>
+ <td class="filesize">74703</td>
+ <td class="filedate">01/04/2014 09:49:46 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20140401.tar.gz'>libnatpmp-20140401.tar.gz</a></td>
+ <td class="filesize">23302</td>
+ <td class="filedate">01/04/2014 09:49:44 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140313.tar.gz'>miniupnpd-1.8.20140313.tar.gz</a></td>
+ <td class="filesize">177120</td>
+ <td class="filedate">13/03/2014 10:39:11 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140310.tar.gz'>miniupnpd-1.8.20140310.tar.gz</a></td>
+ <td class="filesize">176585</td>
+ <td class="filedate">09/03/2014 23:16:49 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140225.tar.gz'>miniupnpd-1.8.20140225.tar.gz</a></td>
+ <td class="filesize">175183</td>
+ <td class="filedate">25/02/2014 11:01:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140203.tar.gz'>miniupnpd-1.8.20140203.tar.gz</a></td>
+ <td class="filesize">170112</td>
+ <td class="filedate">03/02/2014 09:56:05 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.9.tar.gz'>miniupnpc-1.9.tar.gz</a></td>
+ <td class="filesize">74230</td>
+ <td class="filedate">31/01/2014 13:57:40 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20140127.tar.gz'>miniupnpd-1.8.20140127.tar.gz</a></td>
+ <td class="filesize">170467</td>
+ <td class="filedate">27/01/2014 11:25:34 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20140117.zip'>upnpc-exe-win32-20140117.zip</a></td>
+ <td class="filesize">97270</td>
+ <td class="filedate">17/01/2014 11:37:53 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20131216.tar.gz'>miniupnpd-1.8.20131216.tar.gz</a></td>
+ <td class="filesize">170277</td>
+ <td class="filedate">16/12/2013 16:15:40 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20131213.tar.gz'>miniupnpd-1.8.20131213.tar.gz</a></td>
+ <td class="filesize">169753</td>
+ <td class="filedate">13/12/2013 16:18:10 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.20131209.tar.gz'>miniupnpc-1.8.20131209.tar.gz</a></td>
+ <td class="filesize">73900</td>
+ <td class="filedate">09/12/2013 20:52:54 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20131126.tar.gz'>libnatpmp-20131126.tar.gz</a></td>
+ <td class="filesize">22972</td>
+ <td class="filedate">26/11/2013 08:51:36 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.20131007.tar.gz'>miniupnpc-1.8.20131007.tar.gz</a></td>
+ <td class="filesize">73750</td>
+ <td class="filedate">07/10/2013 10:10:25 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20130911.tar.gz'>libnatpmp-20130911.tar.gz</a></td>
+ <td class="filesize">18744</td>
+ <td class="filedate">11/09/2013 07:35:51 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20130910.tar.gz'>libnatpmp-20130910.tar.gz</a></td>
+ <td class="filesize">18734</td>
+ <td class="filedate">10/09/2013 20:15:34 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20130907.tar.gz'>minissdpd-1.2.20130907.tar.gz</a></td>
+ <td class="filesize">20237</td>
+ <td class="filedate">07/09/2013 06:46:31 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.20130819.tar.gz'>minissdpd-1.2.20130819.tar.gz</a></td>
+ <td class="filesize">20772</td>
+ <td class="filedate">19/08/2013 16:50:29 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.20130801.tar.gz'>miniupnpc-1.8.20130801.tar.gz</a></td>
+ <td class="filesize">73426</td>
+ <td class="filedate">01/08/2013 21:38:05 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130730.tar.gz'>miniupnpd-1.8.20130730.tar.gz</a></td>
+ <td class="filesize">149904</td>
+ <td class="filedate">30/07/2013 11:37:48 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130607.tar.gz'>miniupnpd-1.8.20130607.tar.gz</a></td>
+ <td class="filesize">149521</td>
+ <td class="filedate">07/06/2013 08:46:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130521.tar.gz'>miniupnpd-1.8.20130521.tar.gz</a></td>
+ <td class="filesize">149276</td>
+ <td class="filedate">21/05/2013 09:01:33 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130503.tar.gz'>miniupnpd-1.8.20130503.tar.gz</a></td>
+ <td class="filesize">148420</td>
+ <td class="filedate">03/05/2013 19:27:16 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.20130503.tar.gz'>miniupnpc-1.8.20130503.tar.gz</a></td>
+ <td class="filesize">71858</td>
+ <td class="filedate">03/05/2013 19:27:07 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130426.tar.gz'>miniupnpd-1.8.20130426.tar.gz</a></td>
+ <td class="filesize">147890</td>
+ <td class="filedate">26/04/2013 16:57:20 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.20130211.tar.gz'>miniupnpc-1.8.20130211.tar.gz</a></td>
+ <td class="filesize">70723</td>
+ <td class="filedate">11/02/2013 10:32:44 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.20130207.tar.gz'>miniupnpd-1.8.20130207.tar.gz</a></td>
+ <td class="filesize">147325</td>
+ <td class="filedate">07/02/2013 12:29:32 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.8.tar.gz'>miniupnpc-1.8.tar.gz</a></td>
+ <td class="filesize">70624</td>
+ <td class="filedate">06/02/2013 14:31:06 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.8.tar.gz'>miniupnpd-1.8.tar.gz</a></td>
+ <td class="filesize">146679</td>
+ <td class="filedate">06/02/2013 14:30:59 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20121009.zip'>upnpc-exe-win32-20121009.zip</a></td>
+ <td class="filesize">96513</td>
+ <td class="filedate">09/10/2012 17:54:12 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.7.20121005.tar.gz'>miniupnpd-1.7.20121005.tar.gz</a></td>
+ <td class="filesize">144393</td>
+ <td class="filedate">04/10/2012 22:39:05 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.7.20120830.tar.gz'>miniupnpc-1.7.20120830.tar.gz</a></td>
+ <td class="filesize">70074</td>
+ <td class="filedate">30/08/2012 08:41:51 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.7.20120824.tar.gz'>miniupnpd-1.7.20120824.tar.gz</a></td>
+ <td class="filesize">141960</td>
+ <td class="filedate">24/08/2012 18:15:01 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20120821.tar.gz'>libnatpmp-20120821.tar.gz</a></td>
+ <td class="filesize">17832</td>
+ <td class="filedate">21/08/2012 17:24:46 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.7.20120714.tar.gz'>miniupnpc-1.7.20120714.tar.gz</a></td>
+ <td class="filesize">69570</td>
+ <td class="filedate">14/07/2012 14:40:47 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.7.20120711.tar.gz'>miniupnpc-1.7.20120711.tar.gz</a></td>
+ <td class="filesize">69580</td>
+ <td class="filedate">10/07/2012 22:27:05 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.7.20120711.tar.gz'>miniupnpd-1.7.20120711.tar.gz</a></td>
+ <td class="filesize">141380</td>
+ <td class="filedate">10/07/2012 22:26:58 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.7.tar.gz'>miniupnpd-1.7.tar.gz</a></td>
+ <td class="filesize">138047</td>
+ <td class="filedate">27/05/2012 23:13:30 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.7.tar.gz'>miniupnpc-1.7.tar.gz</a></td>
+ <td class="filesize">68327</td>
+ <td class="filedate">24/05/2012 18:17:48 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.2.tar.gz'>minissdpd-1.2.tar.gz</a></td>
+ <td class="filesize">19874</td>
+ <td class="filedate">24/05/2012 18:06:24 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120509.tar.gz'>miniupnpd-1.6.20120509.tar.gz</a></td>
+ <td class="filesize">137147</td>
+ <td class="filedate">09/05/2012 10:45:44 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120509.tar.gz'>miniupnpc-1.6.20120509.tar.gz</a></td>
+ <td class="filesize">68205</td>
+ <td class="filedate">09/05/2012 10:45:41 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.1.20120509.tar.gz'>minissdpd-1.1.20120509.tar.gz</a></td>
+ <td class="filesize">18123</td>
+ <td class="filedate">09/05/2012 10:45:39 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120502.tar.gz'>miniupnpd-1.6.20120502.tar.gz</a></td>
+ <td class="filesize">136688</td>
+ <td class="filedate">01/05/2012 22:51:18 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120502.tar.gz'>miniupnpc-1.6.20120502.tar.gz</a></td>
+ <td class="filesize">68170</td>
+ <td class="filedate">01/05/2012 22:51:11 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120426.tar.gz'>miniupnpd-1.6.20120426.tar.gz</a></td>
+ <td class="filesize">134764</td>
+ <td class="filedate">26/04/2012 16:24:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120424.tar.gz'>miniupnpd-1.6.20120424.tar.gz</a></td>
+ <td class="filesize">132522</td>
+ <td class="filedate">23/04/2012 22:43:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120424.tar.gz'>miniupnpc-1.6.20120424.tar.gz</a></td>
+ <td class="filesize">68067</td>
+ <td class="filedate">23/04/2012 22:43:10 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120420.tar.gz'>miniupnpd-1.6.20120420.tar.gz</a></td>
+ <td class="filesize">131972</td>
+ <td class="filedate">20/04/2012 14:58:57 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120420.tar.gz'>miniupnpc-1.6.20120420.tar.gz</a></td>
+ <td class="filesize">68068</td>
+ <td class="filedate">20/04/2012 14:58:39 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120419.tar.gz'>miniupnpd-1.6.20120419.tar.gz</a></td>
+ <td class="filesize">131088</td>
+ <td class="filedate">18/04/2012 23:41:36 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120418.tar.gz'>miniupnpd-1.6.20120418.tar.gz</a></td>
+ <td class="filesize">130879</td>
+ <td class="filedate">18/04/2012 21:01:10 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.1.20120410.tar.gz'>minissdpd-1.1.20120410.tar.gz</a></td>
+ <td class="filesize">18059</td>
+ <td class="filedate">09/04/2012 22:45:38 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120410.tar.gz'>miniupnpc-1.6.20120410.tar.gz</a></td>
+ <td class="filesize">67934</td>
+ <td class="filedate">09/04/2012 22:45:10 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120406.tar.gz'>miniupnpd-1.6.20120406.tar.gz</a></td>
+ <td class="filesize">128992</td>
+ <td class="filedate">06/04/2012 17:52:57 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120320.tar.gz'>miniupnpc-1.6.20120320.tar.gz</a></td>
+ <td class="filesize">67374</td>
+ <td class="filedate">20/03/2012 16:55:48 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120320.tar.gz'>miniupnpd-1.6.20120320.tar.gz</a></td>
+ <td class="filesize">127968</td>
+ <td class="filedate">20/03/2012 16:46:07 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120305.tar.gz'>miniupnpd-1.6.20120305.tar.gz</a></td>
+ <td class="filesize">126985</td>
+ <td class="filedate">05/03/2012 20:42:01 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120207.tar.gz'>miniupnpd-1.6.20120207.tar.gz</a></td>
+ <td class="filesize">127425</td>
+ <td class="filedate">07/02/2012 10:21:16 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120203.tar.gz'>miniupnpd-1.6.20120203.tar.gz</a></td>
+ <td class="filesize">126599</td>
+ <td class="filedate">03/02/2012 15:14:13 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120125.tar.gz'>miniupnpc-1.6.20120125.tar.gz</a></td>
+ <td class="filesize">67354</td>
+ <td class="filedate">25/01/2012 21:12:28 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.20120121.tar.gz'>miniupnpc-1.6.20120121.tar.gz</a></td>
+ <td class="filesize">67347</td>
+ <td class="filedate">21/01/2012 14:07:41 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20120121.tar.gz'>miniupnpd-1.6.20120121.tar.gz</a></td>
+ <td class="filesize">126021</td>
+ <td class="filedate">21/01/2012 14:07:33 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.1.20120121.tar.gz'>minissdpd-1.1.20120121.tar.gz</a></td>
+ <td class="filesize">17762</td>
+ <td class="filedate">21/01/2012 14:07:16 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20120121.zip'>upnpc-exe-win32-20120121.zip</a></td>
+ <td class="filesize">94575</td>
+ <td class="filedate">21/01/2012 13:59:11 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20111212.zip'>upnpc-exe-win32-20111212.zip</a></td>
+ <td class="filesize">94507</td>
+ <td class="filedate">12/12/2011 12:33:48 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20111118.tar.gz'>miniupnpd-1.6.20111118.tar.gz</a></td>
+ <td class="filesize">125683</td>
+ <td class="filedate">18/11/2011 11:26:12 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.1.20111007.tar.gz'>minissdpd-1.1.20111007.tar.gz</a></td>
+ <td class="filesize">17611</td>
+ <td class="filedate">07/10/2011 09:47:51 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=xchat-upnp20110811.patch'>xchat-upnp20110811.patch</a></td>
+ <td class="filesize">10329</td>
+ <td class="filedate">11/08/2011 15:18:25 +0000</td>
+ <td class="comment">Patch to add UPnP capabilities to xchat</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=xchat-upnp20110811-2.8.8.patch'>xchat-upnp20110811-2.8.8.patch</a></td>
+ <td class="filesize">11529</td>
+ <td class="filedate">11/08/2011 15:18:23 +0000</td>
+ <td class="comment">Patch to add UPnP capabilities to xchat</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110808.tar.gz'>libnatpmp-20110808.tar.gz</a></td>
+ <td class="filesize">17762</td>
+ <td class="filedate">08/08/2011 21:21:34 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110730.tar.gz'>libnatpmp-20110730.tar.gz</a></td>
+ <td class="filesize">17687</td>
+ <td class="filedate">30/07/2011 13:19:31 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.1.tar.gz'>minissdpd-1.1.tar.gz</a></td>
+ <td class="filesize">17481</td>
+ <td class="filedate">30/07/2011 13:17:30 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.20110730.tar.gz'>miniupnpd-1.6.20110730.tar.gz</a></td>
+ <td class="filesize">125583</td>
+ <td class="filedate">30/07/2011 13:17:09 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0.20110729.tar.gz'>minissdpd-1.0.20110729.tar.gz</a></td>
+ <td class="filesize">15898</td>
+ <td class="filedate">29/07/2011 08:47:26 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.6.tar.gz'>miniupnpc-1.6.tar.gz</a></td>
+ <td class="filesize">66454</td>
+ <td class="filedate">25/07/2011 18:03:09 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.6.tar.gz'>miniupnpd-1.6.tar.gz</a></td>
+ <td class="filesize">124917</td>
+ <td class="filedate">25/07/2011 16:37:57 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minidlna_1.0.21.minissdp1.patch'>minidlna_1.0.21.minissdp1.patch</a></td>
+ <td class="filesize">7598</td>
+ <td class="filedate">25/07/2011 14:57:50 +0000</td>
+ <td class="comment">Patch for MiniDLNA to use miniSSDPD</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110715.tar.gz'>libnatpmp-20110715.tar.gz</a></td>
+ <td class="filesize">17943</td>
+ <td class="filedate">15/07/2011 08:31:40 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110715.tar.gz'>miniupnpd-1.5.20110715.tar.gz</a></td>
+ <td class="filesize">124519</td>
+ <td class="filedate">15/07/2011 07:55:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20110714.zip'>upnpc-exe-win32-20110714.zip</a></td>
+ <td class="filesize">94236</td>
+ <td class="filedate">13/07/2011 23:16:01 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110623.tar.gz'>miniupnpd-1.5.20110623.tar.gz</a></td>
+ <td class="filesize">123529</td>
+ <td class="filedate">22/06/2011 22:29:15 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110620.tar.gz'>miniupnpd-1.5.20110620.tar.gz</a></td>
+ <td class="filesize">123221</td>
+ <td class="filedate">20/06/2011 14:11:11 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110618.tar.gz'>miniupnpd-1.5.20110618.tar.gz</a></td>
+ <td class="filesize">123176</td>
+ <td class="filedate">17/06/2011 23:29:18 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110618.tar.gz'>miniupnpc-1.5.20110618.tar.gz</a></td>
+ <td class="filesize">66401</td>
+ <td class="filedate">17/06/2011 23:29:17 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110618.tar.gz'>libnatpmp-20110618.tar.gz</a></td>
+ <td class="filesize">17901</td>
+ <td class="filedate">17/06/2011 23:29:16 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0.20110618.tar.gz'>minissdpd-1.0.20110618.tar.gz</a></td>
+ <td class="filesize">15193</td>
+ <td class="filedate">17/06/2011 23:29:16 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename" colspan="2"><a href='download.php?file=minidlna_cvs20110529_minissdp1.patch'>minidlna_cvs20110529_minissdp1.patch</a></td>
+ <td class="filedate">29/05/2011 21:19:09 +0000</td>
+ <td class="comment">Patch for MiniDLNA to use miniSSDPD</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110528.tar.gz'>miniupnpd-1.5.20110528.tar.gz</a></td>
+ <td class="filesize">121985</td>
+ <td class="filedate">28/05/2011 09:39:04 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minidlna_1.0.19_minissdp1.patch'>minidlna_1.0.19_minissdp1.patch</a></td>
+ <td class="filesize">9080</td>
+ <td class="filedate">27/05/2011 09:55:04 +0000</td>
+ <td class="comment">Patch for MiniDLNA to use miniSSDPD</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110527.tar.gz'>miniupnpd-1.5.20110527.tar.gz</a></td>
+ <td class="filesize">120896</td>
+ <td class="filedate">27/05/2011 08:28:35 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110527.tar.gz'>miniupnpc-1.5.20110527.tar.gz</a></td>
+ <td class="filesize">66279</td>
+ <td class="filedate">27/05/2011 08:28:34 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110527.tar.gz'>libnatpmp-20110527.tar.gz</a></td>
+ <td class="filesize">17627</td>
+ <td class="filedate">27/05/2011 08:28:33 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0.20110523.tar.gz'>minissdpd-1.0.20110523.tar.gz</a></td>
+ <td class="filesize">15024</td>
+ <td class="filedate">23/05/2011 12:55:31 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110520.tar.gz'>miniupnpd-1.5.20110520.tar.gz</a></td>
+ <td class="filesize">119227</td>
+ <td class="filedate">20/05/2011 18:00:41 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110519.tar.gz'>miniupnpd-1.5.20110519.tar.gz</a></td>
+ <td class="filesize">114735</td>
+ <td class="filedate">18/05/2011 22:29:06 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110516.tar.gz'>miniupnpd-1.5.20110516.tar.gz</a></td>
+ <td class="filesize">113348</td>
+ <td class="filedate">16/05/2011 09:32:51 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110515.tar.gz'>miniupnpd-1.5.20110515.tar.gz</a></td>
+ <td class="filesize">113135</td>
+ <td class="filedate">15/05/2011 21:51:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110515.tar.gz'>miniupnpc-1.5.20110515.tar.gz</a></td>
+ <td class="filesize">66112</td>
+ <td class="filedate">15/05/2011 21:51:28 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110513.tar.gz'>miniupnpd-1.5.20110513.tar.gz</a></td>
+ <td class="filesize">111029</td>
+ <td class="filedate">13/05/2011 14:03:12 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110506.tar.gz'>miniupnpc-1.5.20110506.tar.gz</a></td>
+ <td class="filesize">65536</td>
+ <td class="filedate">06/05/2011 16:35:38 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4-v6.20100505.zip'>miniupnpc-1.4-v6.20100505.zip</a></td>
+ <td class="filesize">91833</td>
+ <td class="filedate">18/04/2011 20:14:11 +0000</td>
+ <td class="comment"></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4-v6.20100823.zip'>miniupnpd-1.4-v6.20100823.zip</a></td>
+ <td class="filesize">222235</td>
+ <td class="filedate">18/04/2011 20:14:07 +0000</td>
+ <td class="comment"></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110418.tar.gz'>miniupnpc-1.5.20110418.tar.gz</a></td>
+ <td class="filesize">61820</td>
+ <td class="filedate">18/04/2011 20:09:22 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20110418.zip'>upnpc-exe-win32-20110418.zip</a></td>
+ <td class="filesize">94183</td>
+ <td class="filedate">18/04/2011 17:53:26 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110314.tar.gz'>miniupnpc-1.5.20110314.tar.gz</a></td>
+ <td class="filesize">57210</td>
+ <td class="filedate">14/03/2011 14:27:29 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110309.tar.gz'>miniupnpd-1.5.20110309.tar.gz</a></td>
+ <td class="filesize">100073</td>
+ <td class="filedate">09/03/2011 15:36:12 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110302.tar.gz'>miniupnpd-1.5.20110302.tar.gz</a></td>
+ <td class="filesize">100756</td>
+ <td class="filedate">02/03/2011 16:17:44 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110221.tar.gz'>miniupnpd-1.5.20110221.tar.gz</a></td>
+ <td class="filesize">100092</td>
+ <td class="filedate">20/02/2011 23:48:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20110215.zip'>upnpc-exe-win32-20110215.zip</a></td>
+ <td class="filesize">55409</td>
+ <td class="filedate">15/02/2011 23:05:00 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.20110215.tar.gz'>miniupnpc-1.5.20110215.tar.gz</a></td>
+ <td class="filesize">54880</td>
+ <td class="filedate">15/02/2011 11:16:04 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110214.tar.gz'>miniupnpd-1.5.20110214.tar.gz</a></td>
+ <td class="filesize">99629</td>
+ <td class="filedate">14/02/2011 18:00:43 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minidlna_1.0.18_minissdp1.patch'>minidlna_1.0.18_minissdp1.patch</a></td>
+ <td class="filesize">9747</td>
+ <td class="filedate">02/02/2011 15:12:19 +0000</td>
+ <td class="comment">Patch for MiniDLNA to use miniSSDPD</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.20110127.tar.gz'>miniupnpd-1.5.20110127.tar.gz</a></td>
+ <td class="filesize">97421</td>
+ <td class="filedate">27/01/2011 17:51:25 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.5.tar.gz'>miniupnpd-1.5.tar.gz</a></td>
+ <td class="filesize">98993</td>
+ <td class="filedate">04/01/2011 09:45:10 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.5.tar.gz'>miniupnpc-1.5.tar.gz</a></td>
+ <td class="filesize">53309</td>
+ <td class="filedate">04/01/2011 09:45:06 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20110103.tar.gz'>libnatpmp-20110103.tar.gz</a></td>
+ <td class="filesize">17529</td>
+ <td class="filedate">03/01/2011 17:33:16 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20101221.tar.gz'>miniupnpc-1.4.20101221.tar.gz</a></td>
+ <td class="filesize">52342</td>
+ <td class="filedate">21/12/2010 16:15:38 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20101213.zip'>upnpc-exe-win32-20101213.zip</a></td>
+ <td class="filesize">52359</td>
+ <td class="filedate">12/12/2010 23:44:01 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20101211.tar.gz'>libnatpmp-20101211.tar.gz</a></td>
+ <td class="filesize">17324</td>
+ <td class="filedate">11/12/2010 17:20:36 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20101209.tar.gz'>miniupnpc-1.4.20101209.tar.gz</a></td>
+ <td class="filesize">51900</td>
+ <td class="filedate">09/12/2010 16:17:30 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.20100921.tar.gz'>miniupnpd-1.4.20100921.tar.gz</a></td>
+ <td class="filesize">95483</td>
+ <td class="filedate">21/09/2010 15:50:00 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20100825.zip'>upnpc-exe-win32-20100825.zip</a></td>
+ <td class="filesize">50636</td>
+ <td class="filedate">25/08/2010 08:42:59 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100609.tar.gz'>miniupnpc-1.4.20100609.tar.gz</a></td>
+ <td class="filesize">50390</td>
+ <td class="filedate">09/06/2010 11:03:11 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20100513.zip'>upnpc-exe-win32-20100513.zip</a></td>
+ <td class="filesize">50950</td>
+ <td class="filedate">13/05/2010 16:54:33 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.20100511.tar.gz'>miniupnpd-1.4.20100511.tar.gz</a></td>
+ <td class="filesize">93281</td>
+ <td class="filedate">11/05/2010 16:22:33 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20100418.zip'>upnpc-exe-win32-20100418.zip</a></td>
+ <td class="filesize">40758</td>
+ <td class="filedate">17/04/2010 23:00:37 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100418.tar.gz'>miniupnpc-1.4.20100418.tar.gz</a></td>
+ <td class="filesize">50245</td>
+ <td class="filedate">17/04/2010 22:18:31 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100412.tar.gz'>miniupnpc-1.4.20100412.tar.gz</a></td>
+ <td class="filesize">50145</td>
+ <td class="filedate">12/04/2010 20:42:53 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100407.tar.gz'>miniupnpc-1.4.20100407.tar.gz</a></td>
+ <td class="filesize">49756</td>
+ <td class="filedate">07/04/2010 10:05:08 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100405.tar.gz'>miniupnpc-1.4.20100405.tar.gz</a></td>
+ <td class="filesize">49549</td>
+ <td class="filedate">05/04/2010 14:34:38 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.20100308.tar.gz'>miniupnpd-1.4.20100308.tar.gz</a></td>
+ <td class="filesize">92889</td>
+ <td class="filedate">08/03/2010 17:18:00 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20100202.tar.gz'>libnatpmp-20100202.tar.gz</a></td>
+ <td class="filesize">17231</td>
+ <td class="filedate">02/02/2010 18:41:13 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100202.tar.gz'>miniupnpc-1.4.20100202.tar.gz</a></td>
+ <td class="filesize">46710</td>
+ <td class="filedate">02/02/2010 18:41:13 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20100106.tar.gz'>miniupnpc-1.4.20100106.tar.gz</a></td>
+ <td class="filesize">46659</td>
+ <td class="filedate">06/01/2010 10:08:21 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.20091222.tar.gz'>miniupnpd-1.4.20091222.tar.gz</a></td>
+ <td class="filesize">90993</td>
+ <td class="filedate">22/12/2009 17:23:48 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20091219.tar.gz'>libnatpmp-20091219.tar.gz</a></td>
+ <td class="filesize">16839</td>
+ <td class="filedate">19/12/2009 14:35:22 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20091213.tar.gz'>miniupnpc-1.4.20091213.tar.gz</a></td>
+ <td class="filesize">46510</td>
+ <td class="filedate">12/12/2009 23:05:40 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20091211.tar.gz'>miniupnpc-1.4.20091211.tar.gz</a></td>
+ <td class="filesize">45852</td>
+ <td class="filedate">11/12/2009 16:43:01 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20091210.zip'>upnpc-exe-win32-20091210.zip</a></td>
+ <td class="filesize">38666</td>
+ <td class="filedate">10/12/2009 18:50:27 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20091208.tar.gz'>miniupnpc-1.4.20091208.tar.gz</a></td>
+ <td class="filesize">43392</td>
+ <td class="filedate">08/12/2009 10:58:26 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.20091203.tar.gz'>miniupnpc-1.4.20091203.tar.gz</a></td>
+ <td class="filesize">42040</td>
+ <td class="filedate">03/12/2009 13:56:28 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.20091106.tar.gz'>miniupnpd-1.4.20091106.tar.gz</a></td>
+ <td class="filesize">90787</td>
+ <td class="filedate">06/11/2009 21:18:50 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.4.tar.gz'>miniupnpd-1.4.tar.gz</a></td>
+ <td class="filesize">90071</td>
+ <td class="filedate">30/10/2009 09:20:05 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.4.tar.gz'>miniupnpc-1.4.tar.gz</a></td>
+ <td class="filesize">41790</td>
+ <td class="filedate">30/10/2009 09:20:04 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20091016.tar.gz'>miniupnpc-20091016.tar.gz</a></td>
+ <td class="filesize">41792</td>
+ <td class="filedate">16/10/2009 09:04:35 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20091010.tar.gz'>miniupnpd-20091010.tar.gz</a></td>
+ <td class="filesize">90043</td>
+ <td class="filedate">10/10/2009 19:21:30 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20091010.tar.gz'>miniupnpc-20091010.tar.gz</a></td>
+ <td class="filesize">41671</td>
+ <td class="filedate">10/10/2009 19:21:28 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090921.tar.gz'>miniupnpd-20090921.tar.gz</a></td>
+ <td class="filesize">89476</td>
+ <td class="filedate">21/09/2009 13:00:04 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090921.tar.gz'>miniupnpc-20090921.tar.gz</a></td>
+ <td class="filesize">41630</td>
+ <td class="filedate">21/09/2009 13:00:03 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090904.tar.gz'>miniupnpd-20090904.tar.gz</a></td>
+ <td class="filesize">89344</td>
+ <td class="filedate">04/09/2009 16:24:26 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090820.tar.gz'>miniupnpd-20090820.tar.gz</a></td>
+ <td class="filesize">89149</td>
+ <td class="filedate">20/08/2009 09:35:58 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090807.tar.gz'>miniupnpc-20090807.tar.gz</a></td>
+ <td class="filesize">41288</td>
+ <td class="filedate">07/08/2009 14:46:11 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090729.tar.gz'>miniupnpc-20090729.tar.gz</a></td>
+ <td class="filesize">40588</td>
+ <td class="filedate">29/07/2009 08:47:43 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=xchat-upnp20061022.patch'>xchat-upnp20061022.patch</a></td>
+ <td class="filesize">10258</td>
+ <td class="filedate">17/07/2009 15:49:46 +0000</td>
+ <td class="comment">Patch to add UPnP capabilities to xchat</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090713.tar.gz'>miniupnpc-20090713.tar.gz</a></td>
+ <td class="filesize">40206</td>
+ <td class="filedate">13/07/2009 08:53:49 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20090713.tar.gz'>libnatpmp-20090713.tar.gz</a></td>
+ <td class="filesize">14262</td>
+ <td class="filedate">13/07/2009 08:53:49 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090605.tar.gz'>miniupnpd-20090605.tar.gz</a></td>
+ <td class="filesize">83774</td>
+ <td class="filedate">04/06/2009 23:32:20 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090605.tar.gz'>miniupnpc-20090605.tar.gz</a></td>
+ <td class="filesize">40077</td>
+ <td class="filedate">04/06/2009 23:32:16 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20090605.tar.gz'>libnatpmp-20090605.tar.gz</a></td>
+ <td class="filesize">13817</td>
+ <td class="filedate">04/06/2009 23:32:15 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090516.tar.gz'>miniupnpd-20090516.tar.gz</a></td>
+ <td class="filesize">83689</td>
+ <td class="filedate">16/05/2009 08:47:31 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.3.tar.gz'>miniupnpc-1.3.tar.gz</a></td>
+ <td class="filesize">40058</td>
+ <td class="filedate">17/04/2009 21:27:55 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.3.tar.gz'>miniupnpd-1.3.tar.gz</a></td>
+ <td class="filesize">83464</td>
+ <td class="filedate">17/04/2009 20:11:21 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20090310.tar.gz'>libnatpmp-20090310.tar.gz</a></td>
+ <td class="filesize">11847</td>
+ <td class="filedate">10/03/2009 10:19:45 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090214.tar.gz'>miniupnpd-20090214.tar.gz</a></td>
+ <td class="filesize">82921</td>
+ <td class="filedate">14/02/2009 11:27:03 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090213.tar.gz'>miniupnpd-20090213.tar.gz</a></td>
+ <td class="filesize">82594</td>
+ <td class="filedate">13/02/2009 19:48:01 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20090129.tar.gz'>libnatpmp-20090129.tar.gz</a></td>
+ <td class="filesize">11748</td>
+ <td class="filedate">29/01/2009 21:50:31 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20090129.tar.gz'>miniupnpc-20090129.tar.gz</a></td>
+ <td class="filesize">39976</td>
+ <td class="filedate">29/01/2009 21:50:30 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20090129.tar.gz'>miniupnpd-20090129.tar.gz</a></td>
+ <td class="filesize">82487</td>
+ <td class="filedate">29/01/2009 21:50:27 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20081009.tar.gz'>miniupnpd-20081009.tar.gz</a></td>
+ <td class="filesize">81732</td>
+ <td class="filedate">09/10/2008 12:53:02 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0.tar.gz'>minissdpd-1.0.tar.gz</a></td>
+ <td class="filesize">12996</td>
+ <td class="filedate">07/10/2008 14:03:49 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.2.tar.gz'>miniupnpc-1.2.tar.gz</a></td>
+ <td class="filesize">38787</td>
+ <td class="filedate">07/10/2008 14:03:47 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.2.tar.gz'>miniupnpd-1.2.tar.gz</a></td>
+ <td class="filesize">81025</td>
+ <td class="filedate">07/10/2008 14:03:45 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20081006.tar.gz'>miniupnpd-20081006.tar.gz</a></td>
+ <td class="filesize">80510</td>
+ <td class="filedate">06/10/2008 15:50:34 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-20081006.tar.gz'>minissdpd-20081006.tar.gz</a></td>
+ <td class="filesize">12230</td>
+ <td class="filedate">06/10/2008 15:50:33 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20081006.tar.gz'>libnatpmp-20081006.tar.gz</a></td>
+ <td class="filesize">11710</td>
+ <td class="filedate">06/10/2008 15:50:31 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename" colspan="2"><a href='download.php?file=mediatomb_minissdp-20081006.patch'>mediatomb_minissdp-20081006.patch</a></td>
+ <td class="filedate">06/10/2008 15:48:18 +0000</td>
+ <td class="comment"></td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20081002.tar.gz'>miniupnpc-20081002.tar.gz</a></td>
+ <td class="filesize">38291</td>
+ <td class="filedate">02/10/2008 09:20:18 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20081001.tar.gz'>miniupnpd-20081001.tar.gz</a></td>
+ <td class="filesize">79696</td>
+ <td class="filedate">01/10/2008 13:11:20 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20080925.zip'>upnpc-exe-win32-20080925.zip</a></td>
+ <td class="filesize">36602</td>
+ <td class="filedate">25/09/2008 06:59:33 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080710.tar.gz'>miniupnpd-20080710.tar.gz</a></td>
+ <td class="filesize">78898</td>
+ <td class="filedate">10/07/2008 09:38:41 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080707.tar.gz'>libnatpmp-20080707.tar.gz</a></td>
+ <td class="filesize">11679</td>
+ <td class="filedate">06/07/2008 22:05:23 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.1.tar.gz'>miniupnpc-1.1.tar.gz</a></td>
+ <td class="filesize">38235</td>
+ <td class="filedate">04/07/2008 16:45:24 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20080703.tar.gz'>miniupnpc-20080703.tar.gz</a></td>
+ <td class="filesize">38204</td>
+ <td class="filedate">03/07/2008 15:47:37 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080703.tar.gz'>libnatpmp-20080703.tar.gz</a></td>
+ <td class="filesize">11570</td>
+ <td class="filedate">03/07/2008 15:47:25 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20080703.zip'>upnpc-exe-win32-20080703.zip</a></td>
+ <td class="filesize">36137</td>
+ <td class="filedate">02/07/2008 23:35:14 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080702.tar.gz'>libnatpmp-20080702.tar.gz</a></td>
+ <td class="filesize">8873</td>
+ <td class="filedate">02/07/2008 17:32:35 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080630.tar.gz'>libnatpmp-20080630.tar.gz</a></td>
+ <td class="filesize">8864</td>
+ <td class="filedate">30/06/2008 14:20:16 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080529.tar.gz'>libnatpmp-20080529.tar.gz</a></td>
+ <td class="filesize">7397</td>
+ <td class="filedate">29/05/2008 09:06:25 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20080514.zip'>upnpc-exe-win32-20080514.zip</a></td>
+ <td class="filesize">14227</td>
+ <td class="filedate">14/05/2008 20:23:19 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20080428.tar.gz'>libnatpmp-20080428.tar.gz</a></td>
+ <td class="filesize">7295</td>
+ <td class="filedate">28/04/2008 03:09:14 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080427.tar.gz'>miniupnpd-20080427.tar.gz</a></td>
+ <td class="filesize">78765</td>
+ <td class="filedate">27/04/2008 18:16:36 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20080427.tar.gz'>miniupnpc-20080427.tar.gz</a></td>
+ <td class="filesize">37610</td>
+ <td class="filedate">27/04/2008 18:16:35 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.1.tar.gz'>miniupnpd-1.1.tar.gz</a></td>
+ <td class="filesize">78594</td>
+ <td class="filedate">25/04/2008 17:38:05 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20080423.tar.gz'>miniupnpc-20080423.tar.gz</a></td>
+ <td class="filesize">36818</td>
+ <td class="filedate">23/04/2008 11:57:36 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080308.tar.gz'>miniupnpd-20080308.tar.gz</a></td>
+ <td class="filesize">75679</td>
+ <td class="filedate">08/03/2008 11:13:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080303.tar.gz'>miniupnpd-20080303.tar.gz</a></td>
+ <td class="filesize">74202</td>
+ <td class="filedate">03/03/2008 01:43:16 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080224.tar.gz'>miniupnpd-20080224.tar.gz</a></td>
+ <td class="filesize">72773</td>
+ <td class="filedate">24/02/2008 11:23:17 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0.tar.gz'>miniupnpc-1.0.tar.gz</a></td>
+ <td class="filesize">36223</td>
+ <td class="filedate">21/02/2008 13:26:46 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080221.tar.gz'>miniupnpd-20080221.tar.gz</a></td>
+ <td class="filesize">70823</td>
+ <td class="filedate">21/02/2008 10:23:46 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20080217.tar.gz'>miniupnpc-20080217.tar.gz</a></td>
+ <td class="filesize">35243</td>
+ <td class="filedate">16/02/2008 23:47:59 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20080207.tar.gz'>miniupnpd-20080207.tar.gz</a></td>
+ <td class="filesize">70647</td>
+ <td class="filedate">07/02/2008 21:21:00 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20080203.tar.gz'>miniupnpc-20080203.tar.gz</a></td>
+ <td class="filesize">34921</td>
+ <td class="filedate">03/02/2008 22:28:11 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0.tar.gz'>miniupnpd-1.0.tar.gz</a></td>
+ <td class="filesize">69427</td>
+ <td class="filedate">27/01/2008 22:41:25 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20080118.zip'>upnpc-exe-win32-20080118.zip</a></td>
+ <td class="filesize">13582</td>
+ <td class="filedate">18/01/2008 11:42:16 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC13.tar.gz'>miniupnpd-1.0-RC13.tar.gz</a></td>
+ <td class="filesize">67892</td>
+ <td class="filedate">03/01/2008 16:50:21 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC13.tar.gz'>miniupnpc-1.0-RC13.tar.gz</a></td>
+ <td class="filesize">34820</td>
+ <td class="filedate">03/01/2008 16:50:20 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20071220.tar.gz'>miniupnpd-20071220.tar.gz</a></td>
+ <td class="filesize">67211</td>
+ <td class="filedate">20/12/2007 12:08:34 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20071219.tar.gz'>miniupnpc-20071219.tar.gz</a></td>
+ <td class="filesize">34290</td>
+ <td class="filedate">19/12/2007 18:31:47 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0-RC12.tar.gz'>minissdpd-1.0-RC12.tar.gz</a></td>
+ <td class="filesize">9956</td>
+ <td class="filedate">19/12/2007 18:30:12 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC12.tar.gz'>miniupnpd-1.0-RC12.tar.gz</a></td>
+ <td class="filesize">66911</td>
+ <td class="filedate">14/12/2007 17:39:20 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC12.tar.gz'>miniupnpc-1.0-RC12.tar.gz</a></td>
+ <td class="filesize">32543</td>
+ <td class="filedate">14/12/2007 17:39:19 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20071213.tar.gz'>miniupnpc-20071213.tar.gz</a></td>
+ <td class="filesize">32541</td>
+ <td class="filedate">13/12/2007 17:09:51 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20071213.tar.gz'>miniupnpd-20071213.tar.gz</a></td>
+ <td class="filesize">66826</td>
+ <td class="filedate">13/12/2007 16:42:50 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20071213.tar.gz'>libnatpmp-20071213.tar.gz</a></td>
+ <td class="filesize">5997</td>
+ <td class="filedate">13/12/2007 14:56:30 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=libnatpmp-20071202.tar.gz'>libnatpmp-20071202.tar.gz</a></td>
+ <td class="filesize">5664</td>
+ <td class="filedate">02/12/2007 00:15:28 +0000</td>
+ <td class="comment">libnatpmp source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20071103.tar.gz'>miniupnpd-20071103.tar.gz</a></td>
+ <td class="filesize">65740</td>
+ <td class="filedate">02/11/2007 23:58:38 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20071102.tar.gz'>miniupnpd-20071102.tar.gz</a></td>
+ <td class="filesize">65733</td>
+ <td class="filedate">02/11/2007 23:05:44 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20071103.tar.gz'>miniupnpc-20071103.tar.gz</a></td>
+ <td class="filesize">32239</td>
+ <td class="filedate">02/11/2007 23:05:34 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC11.tar.gz'>miniupnpd-1.0-RC11.tar.gz</a></td>
+ <td class="filesize">64828</td>
+ <td class="filedate">25/10/2007 13:27:18 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC11.tar.gz'>miniupnpc-1.0-RC11.tar.gz</a></td>
+ <td class="filesize">32161</td>
+ <td class="filedate">25/10/2007 13:27:17 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20071025.zip'>upnpc-exe-win32-20071025.zip</a></td>
+ <td class="filesize">12809</td>
+ <td class="filedate">24/10/2007 23:15:55 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC10.tar.gz'>miniupnpd-1.0-RC10.tar.gz</a></td>
+ <td class="filesize">62674</td>
+ <td class="filedate">12/10/2007 08:38:33 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC10.tar.gz'>miniupnpc-1.0-RC10.tar.gz</a></td>
+ <td class="filesize">31962</td>
+ <td class="filedate">12/10/2007 08:38:31 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0-RC10.tar.gz'>minissdpd-1.0-RC10.tar.gz</a></td>
+ <td class="filesize">9517</td>
+ <td class="filedate">12/10/2007 08:38:30 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20071003.tar.gz'>miniupnpc-20071003.tar.gz</a></td>
+ <td class="filesize">31199</td>
+ <td class="filedate">03/10/2007 15:30:13 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20071001.zip'>upnpc-exe-win32-20071001.zip</a></td>
+ <td class="filesize">12604</td>
+ <td class="filedate">01/10/2007 17:09:22 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC9.tar.gz'>miniupnpd-1.0-RC9.tar.gz</a></td>
+ <td class="filesize">54778</td>
+ <td class="filedate">27/09/2007 19:38:36 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-1.0-RC9.tar.gz'>minissdpd-1.0-RC9.tar.gz</a></td>
+ <td class="filesize">9163</td>
+ <td class="filedate">27/09/2007 17:00:03 +0000</td>
+ <td class="comment">MiniSSDPd release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC9.tar.gz'>miniupnpc-1.0-RC9.tar.gz</a></td>
+ <td class="filesize">30538</td>
+ <td class="filedate">27/09/2007 17:00:03 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070924.tar.gz'>miniupnpd-20070924.tar.gz</a></td>
+ <td class="filesize">52338</td>
+ <td class="filedate">24/09/2007 20:26:05 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070923.tar.gz'>miniupnpd-20070923.tar.gz</a></td>
+ <td class="filesize">51060</td>
+ <td class="filedate">23/09/2007 21:13:34 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20070923.tar.gz'>miniupnpc-20070923.tar.gz</a></td>
+ <td class="filesize">30246</td>
+ <td class="filedate">23/09/2007 21:13:33 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-20070923.tar.gz'>minissdpd-20070923.tar.gz</a></td>
+ <td class="filesize">8978</td>
+ <td class="filedate">23/09/2007 21:13:32 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20070902.tar.gz'>miniupnpc-20070902.tar.gz</a></td>
+ <td class="filesize">30205</td>
+ <td class="filedate">01/09/2007 23:47:23 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=minissdpd-20070902.tar.gz'>minissdpd-20070902.tar.gz</a></td>
+ <td class="filesize">6539</td>
+ <td class="filedate">01/09/2007 23:47:20 +0000</td>
+ <td class="comment">MiniSSDPd source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC8.tar.gz'>miniupnpd-1.0-RC8.tar.gz</a></td>
+ <td class="filesize">50952</td>
+ <td class="filedate">29/08/2007 10:56:09 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC8.tar.gz'>miniupnpc-1.0-RC8.tar.gz</a></td>
+ <td class="filesize">29312</td>
+ <td class="filedate">29/08/2007 10:56:08 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC7.tar.gz'>miniupnpd-1.0-RC7.tar.gz</a></td>
+ <td class="filesize">50613</td>
+ <td class="filedate">20/07/2007 00:15:45 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC6.tar.gz'>miniupnpd-1.0-RC6.tar.gz</a></td>
+ <td class="filesize">49986</td>
+ <td class="filedate">12/06/2007 17:12:07 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC6.tar.gz'>miniupnpc-1.0-RC6.tar.gz</a></td>
+ <td class="filesize">29032</td>
+ <td class="filedate">12/06/2007 17:12:06 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070607.tar.gz'>miniupnpd-20070607.tar.gz</a></td>
+ <td class="filesize">49768</td>
+ <td class="filedate">06/06/2007 23:12:00 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070605.tar.gz'>miniupnpd-20070605.tar.gz</a></td>
+ <td class="filesize">49710</td>
+ <td class="filedate">05/06/2007 21:01:53 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070521.tar.gz'>miniupnpd-20070521.tar.gz</a></td>
+ <td class="filesize">48374</td>
+ <td class="filedate">21/05/2007 13:07:43 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20070519.zip'>upnpc-exe-win32-20070519.zip</a></td>
+ <td class="filesize">10836</td>
+ <td class="filedate">19/05/2007 13:14:15 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20070515.tar.gz'>miniupnpc-20070515.tar.gz</a></td>
+ <td class="filesize">25802</td>
+ <td class="filedate">15/05/2007 18:15:25 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC5.tar.gz'>miniupnpd-1.0-RC5.tar.gz</a></td>
+ <td class="filesize">48064</td>
+ <td class="filedate">10/05/2007 20:22:48 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC5.tar.gz'>miniupnpc-1.0-RC5.tar.gz</a></td>
+ <td class="filesize">25242</td>
+ <td class="filedate">10/05/2007 20:22:46 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070412.tar.gz'>miniupnpd-20070412.tar.gz</a></td>
+ <td class="filesize">47807</td>
+ <td class="filedate">12/04/2007 20:21:48 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC4.tar.gz'>miniupnpd-1.0-RC4.tar.gz</a></td>
+ <td class="filesize">47687</td>
+ <td class="filedate">17/03/2007 11:43:13 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC4.tar.gz'>miniupnpc-1.0-RC4.tar.gz</a></td>
+ <td class="filesize">25085</td>
+ <td class="filedate">17/03/2007 11:43:10 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070311.tar.gz'>miniupnpd-20070311.tar.gz</a></td>
+ <td class="filesize">47599</td>
+ <td class="filedate">11/03/2007 00:25:26 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070208.tar.gz'>miniupnpd-20070208.tar.gz</a></td>
+ <td class="filesize">45084</td>
+ <td class="filedate">07/02/2007 23:04:06 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC3.tar.gz'>miniupnpd-1.0-RC3.tar.gz</a></td>
+ <td class="filesize">44683</td>
+ <td class="filedate">30/01/2007 23:00:44 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC3.tar.gz'>miniupnpc-1.0-RC3.tar.gz</a></td>
+ <td class="filesize">25055</td>
+ <td class="filedate">30/01/2007 23:00:42 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070130.tar.gz'>miniupnpd-20070130.tar.gz</a></td>
+ <td class="filesize">43735</td>
+ <td class="filedate">29/01/2007 23:26:16 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20070130.tar.gz'>miniupnpc-20070130.tar.gz</a></td>
+ <td class="filesize">24466</td>
+ <td class="filedate">29/01/2007 23:26:13 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070127.tar.gz'>miniupnpd-20070127.tar.gz</a></td>
+ <td class="filesize">42643</td>
+ <td class="filedate">27/01/2007 16:02:35 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20070127.tar.gz'>miniupnpc-20070127.tar.gz</a></td>
+ <td class="filesize">24241</td>
+ <td class="filedate">27/01/2007 16:02:33 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC2.tar.gz'>miniupnpd-1.0-RC2.tar.gz</a></td>
+ <td class="filesize">40424</td>
+ <td class="filedate">17/01/2007 16:13:05 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070112.tar.gz'>miniupnpd-20070112.tar.gz</a></td>
+ <td class="filesize">40708</td>
+ <td class="filedate">12/01/2007 13:40:50 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070111.tar.gz'>miniupnpd-20070111.tar.gz</a></td>
+ <td class="filesize">40651</td>
+ <td class="filedate">11/01/2007 18:50:21 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070108.tar.gz'>miniupnpd-20070108.tar.gz</a></td>
+ <td class="filesize">40025</td>
+ <td class="filedate">08/01/2007 10:02:14 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20070103.tar.gz'>miniupnpd-20070103.tar.gz</a></td>
+ <td class="filesize">40065</td>
+ <td class="filedate">03/01/2007 14:39:11 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-20061214.tar.gz'>miniupnpc-20061214.tar.gz</a></td>
+ <td class="filesize">24106</td>
+ <td class="filedate">14/12/2006 15:43:54 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-20061214.tar.gz'>miniupnpd-20061214.tar.gz</a></td>
+ <td class="filesize">39750</td>
+ <td class="filedate">14/12/2006 13:44:51 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd-1.0-RC1.tar.gz'>miniupnpd-1.0-RC1.tar.gz</a></td>
+ <td class="filesize">39572</td>
+ <td class="filedate">07/12/2006 10:55:31 +0000</td>
+ <td class="comment">MiniUPnP daemon release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-1.0-RC1.tar.gz'>miniupnpc-1.0-RC1.tar.gz</a></td>
+ <td class="filesize">23582</td>
+ <td class="filedate">07/12/2006 10:55:30 +0000</td>
+ <td class="comment">MiniUPnP client release source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20061201.zip'>upnpc-exe-win32-20061201.zip</a></td>
+ <td class="filesize">10378</td>
+ <td class="filedate">01/12/2006 00:33:08 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061130.tar.gz'>miniupnpd20061130.tar.gz</a></td>
+ <td class="filesize">37184</td>
+ <td class="filedate">30/11/2006 12:25:25 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061129.tar.gz'>miniupnpd20061129.tar.gz</a></td>
+ <td class="filesize">36045</td>
+ <td class="filedate">29/11/2006 00:10:49 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061127.tar.gz'>miniupnpd20061127.tar.gz</a></td>
+ <td class="filesize">34155</td>
+ <td class="filedate">26/11/2006 23:15:28 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061123.tar.gz'>miniupnpc20061123.tar.gz</a></td>
+ <td class="filesize">21004</td>
+ <td class="filedate">23/11/2006 22:41:46 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename" colspan="2"><a href='download.php?file=miniupnpd-bin-openwrt20061123.tar.gz'>miniupnpd-bin-openwrt20061123.tar.gz</a></td>
+ <td class="filedate">23/11/2006 22:41:44 +0000</td>
+ <td class="comment">Precompiled binaries for openwrt</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061123.tar.gz'>miniupnpd20061123.tar.gz</a></td>
+ <td class="filesize">33809</td>
+ <td class="filedate">23/11/2006 22:28:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061119.tar.gz'>miniupnpc20061119.tar.gz</a></td>
+ <td class="filesize">20897</td>
+ <td class="filedate">19/11/2006 22:50:37 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061119.tar.gz'>miniupnpd20061119.tar.gz</a></td>
+ <td class="filesize">32580</td>
+ <td class="filedate">19/11/2006 22:50:36 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061117.tar.gz'>miniupnpd20061117.tar.gz</a></td>
+ <td class="filesize">32646</td>
+ <td class="filedate">17/11/2006 13:29:33 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20061112.zip'>upnpc-exe-win32-20061112.zip</a></td>
+ <td class="filesize">10262</td>
+ <td class="filedate">12/11/2006 22:41:25 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061112.tar.gz'>miniupnpd20061112.tar.gz</a></td>
+ <td class="filesize">32023</td>
+ <td class="filedate">12/11/2006 21:30:32 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061112.tar.gz'>miniupnpc20061112.tar.gz</a></td>
+ <td class="filesize">21047</td>
+ <td class="filedate">12/11/2006 21:30:31 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061110.tar.gz'>miniupnpd20061110.tar.gz</a></td>
+ <td class="filesize">27926</td>
+ <td class="filedate">09/11/2006 23:35:02 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061110.tar.gz'>miniupnpc20061110.tar.gz</a></td>
+ <td class="filesize">21009</td>
+ <td class="filedate">09/11/2006 23:32:19 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20061101.zip'>upnpc-exe-win32-20061101.zip</a></td>
+ <td class="filesize">10089</td>
+ <td class="filedate">08/11/2006 20:35:09 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20061020.zip'>upnpc-exe-win32-20061020.zip</a></td>
+ <td class="filesize">9183</td>
+ <td class="filedate">08/11/2006 20:35:08 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20060909.zip'>upnpc-exe-win32-20060909.zip</a></td>
+ <td class="filesize">9994</td>
+ <td class="filedate">08/11/2006 20:35:07 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20060801.zip'>upnpc-exe-win32-20060801.zip</a></td>
+ <td class="filesize">10002</td>
+ <td class="filedate">08/11/2006 20:35:06 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20060711.zip'>upnpc-exe-win32-20060711.zip</a></td>
+ <td class="filesize">13733</td>
+ <td class="filedate">08/11/2006 20:35:05 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20060709.zip'>upnpc-exe-win32-20060709.zip</a></td>
+ <td class="filesize">13713</td>
+ <td class="filedate">08/11/2006 20:35:04 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=upnpc-exe-win32-20060704.zip'>upnpc-exe-win32-20060704.zip</a></td>
+ <td class="filesize">13297</td>
+ <td class="filedate">08/11/2006 20:35:03 +0000</td>
+ <td class="comment">Windows executable</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061107.tar.gz'>miniupnpc20061107.tar.gz</a></td>
+ <td class="filesize">20708</td>
+ <td class="filedate">06/11/2006 23:36:57 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061107.tar.gz'>miniupnpd20061107.tar.gz</a></td>
+ <td class="filesize">26992</td>
+ <td class="filedate">06/11/2006 23:35:06 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061106.tar.gz'>miniupnpc20061106.tar.gz</a></td>
+ <td class="filesize">20575</td>
+ <td class="filedate">06/11/2006 17:02:15 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061106.tar.gz'>miniupnpd20061106.tar.gz</a></td>
+ <td class="filesize">26597</td>
+ <td class="filedate">06/11/2006 15:39:10 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061101.tar.gz'>miniupnpc20061101.tar.gz</a></td>
+ <td class="filesize">20395</td>
+ <td class="filedate">04/11/2006 18:16:15 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061031.tar.gz'>miniupnpc20061031.tar.gz</a></td>
+ <td class="filesize">20396</td>
+ <td class="filedate">04/11/2006 18:16:13 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061023.tar.gz'>miniupnpc20061023.tar.gz</a></td>
+ <td class="filesize">20109</td>
+ <td class="filedate">04/11/2006 18:16:12 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20061020.tar.gz'>miniupnpc20061020.tar.gz</a></td>
+ <td class="filesize">19739</td>
+ <td class="filedate">04/11/2006 18:16:10 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20060909.tar.gz'>miniupnpc20060909.tar.gz</a></td>
+ <td class="filesize">19285</td>
+ <td class="filedate">04/11/2006 18:16:09 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20060731.tar.gz'>miniupnpc20060731.tar.gz</a></td>
+ <td class="filesize">19032</td>
+ <td class="filedate">04/11/2006 18:16:07 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20060711.tar.gz'>miniupnpc20060711.tar.gz</a></td>
+ <td class="filesize">19151</td>
+ <td class="filedate">04/11/2006 18:16:06 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20060709.tar.gz'>miniupnpc20060709.tar.gz</a></td>
+ <td class="filesize">19080</td>
+ <td class="filedate">04/11/2006 18:16:04 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc20060703.tar.gz'>miniupnpc20060703.tar.gz</a></td>
+ <td class="filesize">17906</td>
+ <td class="filedate">04/11/2006 18:16:03 +0000</td>
+ <td class="comment">MiniUPnP client source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpc-new20060630.tar.gz'>miniupnpc-new20060630.tar.gz</a></td>
+ <td class="filesize">14840</td>
+ <td class="filedate">04/11/2006 18:16:01 +0000</td>
+ <td class="comment">João Paulo Barraca version of the upnp client</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061029.tar.gz'>miniupnpd20061029.tar.gz</a></td>
+ <td class="filesize">24197</td>
+ <td class="filedate">03/11/2006 13:40:30 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061027.tar.gz'>miniupnpd20061027.tar.gz</a></td>
+ <td class="filesize">23904</td>
+ <td class="filedate">03/11/2006 13:40:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061028.tar.gz'>miniupnpd20061028.tar.gz</a></td>
+ <td class="filesize">24383</td>
+ <td class="filedate">03/11/2006 13:40:29 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061018.tar.gz'>miniupnpd20061018.tar.gz</a></td>
+ <td class="filesize">23051</td>
+ <td class="filedate">03/11/2006 13:40:28 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20061023.tar.gz'>miniupnpd20061023.tar.gz</a></td>
+ <td class="filesize">23478</td>
+ <td class="filedate">03/11/2006 13:40:28 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20060930.tar.gz'>miniupnpd20060930.tar.gz</a></td>
+ <td class="filesize">22832</td>
+ <td class="filedate">03/11/2006 13:40:28 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20060924.tar.gz'>miniupnpd20060924.tar.gz</a></td>
+ <td class="filesize">22038</td>
+ <td class="filedate">03/11/2006 13:40:27 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20060919.tar.gz'>miniupnpd20060919.tar.gz</a></td>
+ <td class="filesize">21566</td>
+ <td class="filedate">03/11/2006 13:40:27 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20060729.tar.gz'>miniupnpd20060729.tar.gz</a></td>
+ <td class="filesize">19202</td>
+ <td class="filedate">03/11/2006 13:40:26 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+<tr>
+ <td class="filename"><a href='download.php?file=miniupnpd20060909.tar.gz'>miniupnpd20060909.tar.gz</a></td>
+ <td class="filesize">19952</td>
+ <td class="filedate">03/11/2006 13:40:26 +0000</td>
+ <td class="comment">MiniUPnP daemon source code</td>
+ <td></td>
+</tr>
+</table>
+
+<p><a href="..">Home</a></p>
+<p>Contact: miniupnp _AT_ free _DOT_ fr</p>
+<p align="center">
+<a href="https://validator.w3.org/check?uri=referer"><img src="https://www.w3.org/Icons/valid-xhtml10" alt="Valid XHTML 1.0 Transitional" height="31" width="88" /></a>
+<a href="https://jigsaw.w3.org/css-validator/check/referer"><img style="border:0;width:88px;height:31px" src="https://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!" /></a>
+<!--
+ <a href="https://freshmeat.net/projects/miniupnp"><img src="https://s3.amazonaws.com/entp-tender-production/assets/bc5be96f147ec8db3c10fc017f1f53889904ef5b/fm_logo_white_150_normal.png" border="0" alt="freshmeat.net" /></a>
+-->
+<!-- https://futuresimple.github.com/images/github_logo.png -->
+<!-- <a href="https://github.com/miniupnp/miniupnp"><img src="https://assets-cdn.github.com/images/modules/logos_page/GitHub-Logo.png" alt="github.com" height="31" /></a> -->
+<a href="https://github.com/miniupnp/miniupnp"><img style="position: absolute; top: 0; left: 0; border: 0;" src="https://github.blog/wp-content/uploads/2008/12/forkme_left_green_007200.png" alt="Fork me on GitHub" /></a>
+</p>
+
+<script type="text/javascript">
+var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
+document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
+</script>
+<script type="text/javascript">
+try {
+ var ua = 'UA-10295521';
+ if(window.location.hostname == 'miniupnp.free.fr')
+ ua += '-1';
+ else if(window.location.hostname == 'miniupnp.tuxfamily.org')
+ ua += '-2';
+ else ua = '';
+ if(ua != '') {
+ var pageTracker = _gat._getTracker(ua);
+ pageTracker._trackPageview();
+ }
+} catch(err) {}</script>
+</body>
+</html>
+
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 85c1f79f..71b0a63f 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -1,5 +1,5 @@
#
-# BitBake Tests for the Fetcher (fetch2/)
+# BitBake Tests for the Fetcher (fetch/)
#
# Copyright (C) 2012 Richard Purdie
#
@@ -7,16 +7,23 @@
#
import contextlib
+import http.server
+import shutil
import unittest
+import unittest.mock
+import urllib.parse
import hashlib
import tempfile
import collections
import os
import signal
+import subprocess
+import json
import tarfile
-from bb.fetch2 import URI
-from bb.fetch2 import FetchMethod
+import threading
+from bb.fetch import URI
import bb
+import bb.utils
from bb.tests.support.httpserver import HTTPService
def skipIfNoNetwork():
@@ -24,6 +31,18 @@ def skipIfNoNetwork():
return unittest.skip("network test")
return lambda f: f
+
+@contextlib.contextmanager
+def hide_directory(directory):
+ """Hide the given directory and restore it after the context is left"""
+ temp_name = directory + ".bak"
+ os.rename(directory, temp_name)
+ try:
+ yield
+ finally:
+ os.rename(temp_name, directory)
+
+
class TestTimeout(Exception):
# Indicate to pytest that this is not a test suite
__test__ = False
@@ -142,39 +161,6 @@ class URITest(unittest.TestCase):
'query': {},
'relative': False
},
- "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : {
- 'uri': 'cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg',
- 'scheme': 'cvs',
- 'hostname': 'cvs.handhelds.org',
- 'port': None,
- 'hostport': 'cvs.handhelds.org',
- 'path': '/cvs',
- 'userinfo': 'anoncvs',
- 'username': 'anoncvs',
- 'password': '',
- 'params': {
- 'module': 'familiar/dist/ipkg'
- },
- 'query': {},
- 'relative': False
- },
- "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg": {
- 'uri': 'cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg',
- 'scheme': 'cvs',
- 'hostname': 'cvs.handhelds.org',
- 'port': None,
- 'hostport': 'cvs.handhelds.org',
- 'path': '/cvs',
- 'userinfo': 'anoncvs:anonymous',
- 'username': 'anoncvs',
- 'password': 'anonymous',
- 'params': collections.OrderedDict([
- ('tag', 'V0-99-81'),
- ('module', 'familiar/dist/ipkg')
- ]),
- 'query': {},
- 'relative': False
- },
"file://example.diff": { # NOTE: Not RFC compliant!
'uri': 'file:example.diff',
'scheme': 'file',
@@ -309,8 +295,8 @@ class URITest(unittest.TestCase):
'query': {},
'relative': True
},
- "https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip": {
- 'uri': 'https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip',
+ "https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip;someparam=": {
+ 'uri': 'https://www.innodisk.com/Download_file?9BE0BF6657;downloadfilename=EGPL-T101.zip;someparam=',
'scheme': 'https',
'hostname': 'www.innodisk.com',
'port': None,
@@ -320,9 +306,24 @@ class URITest(unittest.TestCase):
'userinfo': '',
'username': '',
'password': '',
- 'params': {"downloadfilename" : "EGPL-T101.zip"},
+ 'params': {"downloadfilename" : "EGPL-T101.zip", "someparam" : ""},
'query': {"9BE0BF6657": None},
'relative': False
+ },
+ "file://example@.service": {
+ 'uri': 'file:example%40.service',
+ 'scheme': 'file',
+ 'hostname': '',
+ 'port': None,
+ 'hostport': '',
+ 'path': 'example@.service',
+ 'userinfo': '',
+ 'userinfo': '',
+ 'username': '',
+ 'password': '',
+ 'params': {},
+ 'query': {},
+ 'relative': True
}
}
@@ -427,20 +428,17 @@ class FetcherTest(unittest.TestCase):
if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
print("Not cleaning up %s. Please remove manually." % self.tempdir)
else:
- bb.process.run('chmod u+rw -R %s' % self.tempdir)
+ bb.process.run(['chmod', 'u+rw', '-R', self.tempdir])
bb.utils.prunedir(self.tempdir)
def git(self, cmd, cwd=None):
- if isinstance(cmd, str):
- cmd = 'git -c safe.bareRepository=all ' + cmd
- else:
- cmd = ['git', '-c', 'safe.bareRepository=all'] + cmd
+ cmd = ['git', '-c', 'safe.bareRepository=all'] + cmd
if cwd is None:
cwd = self.gitdir
return bb.process.run(cmd, cwd=cwd)[0]
def git_init(self, cwd=None):
- self.git('init', cwd=cwd)
+ self.git(['init'], cwd=cwd)
# Explicitly set initial branch to master as
# a common setup is to use other default
# branch than master.
@@ -459,16 +457,16 @@ class FetcherTest(unittest.TestCase):
class MirrorUriTest(FetcherTest):
replaceuris = {
- ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "http://somewhere.org/somedir/")
+ ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "http://somewhere.org/somedir/")
: "http://somewhere.org/somedir/git2_git.invalid.infradead.org.mtd-utils.git.tar.gz",
- ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http")
- : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
- ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http")
- : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
- ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/\\2;protocol=http")
- : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
- ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake")
- : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890",
+ ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http")
+ : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+ ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/somedir/\\2;protocol=http")
+ : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+ ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/([^/]+/)*([^/]*)", "git://somewhere.org/\\2;protocol=http")
+ : "git://somewhere.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+ ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", "git://someserver.org/bitbake", "git://git.openembedded.org/bitbake")
+ : "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master",
("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache")
: "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz",
("file://sstate-xyz.tgz", "file://.*", "file:///somewhere/1234/sstate-cache/")
@@ -481,12 +479,12 @@ class MirrorUriTest(FetcherTest):
: "http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2",
("http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2", "http://.*/.*", "file:///somepath/downloads/")
: "file:///somepath/downloads/subversion-1.7.1.tar.bz2",
- ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
- : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
- ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
- : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
- ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
- : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;protocol=http",
+ ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
+ : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+ ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/BASENAME;protocol=http")
+ : "git://somewhere.org/somedir/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
+ ("git://git.invalid.infradead.org/foo/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", "git://.*/.*", "git://somewhere.org/somedir/MIRRORNAME;protocol=http")
+ : "git://somewhere.org/somedir/git.invalid.infradead.org.foo.mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master;protocol=http",
("http://somewhere.org/somedir1/somedir2/somefile_1.2.3.tar.gz", "http://.*/.*", "http://somewhere2.org")
: "http://somewhere2.org/somefile_1.2.3.tar.gz",
("http://somewhere.org/somedir1/somedir2/somefile_1.2.3.tar.gz", "http://.*/.*", "http://somewhere2.org/")
@@ -502,6 +500,10 @@ class MirrorUriTest(FetcherTest):
: "file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz",
("https://somewhere.org/example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz", "https://.*/.*", "file:///mirror/some-example-1.0.0.tgz")
: "file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz",
+ ("git://git.invalid.infradead.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", r"git://(?!internal\.git\.server).*/.*", "http://somewhere.org/somedir/")
+ : "http://somewhere.org/somedir/git2_git.invalid.infradead.org.mtd-utils.git.tar.gz",
+ ("git://internal.git.server.org/mtd-utils.git;tag=1234567890123456789012345678901234567890;branch=master", r"git://(?!internal\.git\.server).*/.*", "http://somewhere.org/somedir/")
+ : None,
#Renaming files doesn't work
#("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz"
@@ -510,53 +512,60 @@ class MirrorUriTest(FetcherTest):
mirrorvar = "http://.*/.* file:///somepath/downloads/ " \
"git://someserver.org/bitbake git://git.openembedded.org/bitbake " \
- "https://.*/.* file:///someotherpath/downloads/ " \
- "http://.*/.* file:///someotherpath/downloads/"
+ "https?://.*/.* file:///someotherpath/downloads/ " \
+ "svn://svn.server1.com/ svn://svn.server2.com/"
def test_urireplace(self):
self.d.setVar("FILESPATH", ".")
for k, v in self.replaceuris.items():
ud = bb.fetch.FetchData(k[0], self.d)
ud.setup_localpath(self.d)
- mirrors = bb.fetch2.mirror_from_string("%s %s" % (k[1], k[2]))
- newuris, uds = bb.fetch2.build_mirroruris(ud, mirrors, self.d)
- self.assertEqual([v], newuris)
+ mirrors = bb.fetch.mirror_from_string("%s %s" % (k[1], k[2]))
+ newuris, uds = bb.fetch.build_mirroruris(ud, mirrors, self.d)
+ self.assertEqual([v] if v else [], newuris)
def test_urilist1(self):
fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
- mirrors = bb.fetch2.mirror_from_string(self.mirrorvar)
- uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
+ mirrors = bb.fetch.mirror_from_string(self.mirrorvar)
+ uris, uds = bb.fetch.build_mirroruris(fetcher, mirrors, self.d)
self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz', 'file:///someotherpath/downloads/bitbake-1.0.tar.gz'])
def test_urilist2(self):
# Catch https:// -> files:// bug
fetcher = bb.fetch.FetchData("https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
- mirrors = bb.fetch2.mirror_from_string(self.mirrorvar)
- uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
+ mirrors = bb.fetch.mirror_from_string(self.mirrorvar)
+ uris, uds = bb.fetch.build_mirroruris(fetcher, mirrors, self.d)
self.assertEqual(uris, ['file:///someotherpath/downloads/bitbake-1.0.tar.gz'])
+ def test_urilistsvn(self):
+ # Catch svn:// -> svn:// bug
+ fetcher = bb.fetch.FetchData("svn://svn.server1.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2", self.d)
+ mirrors = bb.fetch.mirror_from_string(self.mirrorvar)
+ uris, uds = bb.fetch.build_mirroruris(fetcher, mirrors, self.d)
+ self.assertEqual(uris, ['svn://svn.server2.com/isource/svnroot/reponame/tags/tagname;module=path_in_tagnamefolder;protocol=https;rev=2'])
+
def test_mirror_of_mirror(self):
# Test if mirror of a mirror works
mirrorvar = self.mirrorvar + " http://.*/.* http://otherdownloads.yoctoproject.org/downloads/"
mirrorvar = mirrorvar + " http://otherdownloads.yoctoproject.org/.* http://downloads2.yoctoproject.org/downloads/"
fetcher = bb.fetch.FetchData("http://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
- mirrors = bb.fetch2.mirror_from_string(mirrorvar)
- uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
- self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz',
- 'file:///someotherpath/downloads/bitbake-1.0.tar.gz',
+ mirrors = bb.fetch.mirror_from_string(mirrorvar)
+ uris, uds = bb.fetch.build_mirroruris(fetcher, mirrors, self.d)
+ self.assertEqual(uris, ['file:///somepath/downloads/bitbake-1.0.tar.gz',
+ 'file:///someotherpath/downloads/bitbake-1.0.tar.gz',
'http://otherdownloads.yoctoproject.org/downloads/bitbake-1.0.tar.gz',
'http://downloads2.yoctoproject.org/downloads/bitbake-1.0.tar.gz'])
- recmirrorvar = "https://.*/[^/]* http://AAAA/A/A/A/ " \
- "https://.*/[^/]* https://BBBB/B/B/B/"
+ recmirrorvar = "https://.*/[^/]* http://aaaa/A/A/A/ " \
+ "https://.*/[^/]* https://bbbb/B/B/B/"
def test_recursive(self):
fetcher = bb.fetch.FetchData("https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz", self.d)
- mirrors = bb.fetch2.mirror_from_string(self.recmirrorvar)
- uris, uds = bb.fetch2.build_mirroruris(fetcher, mirrors, self.d)
- self.assertEqual(uris, ['http://AAAA/A/A/A/bitbake/bitbake-1.0.tar.gz',
- 'https://BBBB/B/B/B/bitbake/bitbake-1.0.tar.gz',
- 'http://AAAA/A/A/A/B/B/bitbake/bitbake-1.0.tar.gz'])
+ mirrors = bb.fetch.mirror_from_string(self.recmirrorvar)
+ uris, uds = bb.fetch.build_mirroruris(fetcher, mirrors, self.d)
+ self.assertEqual(uris, ['http://aaaa/A/A/A/bitbake/bitbake-1.0.tar.gz',
+ 'https://bbbb/B/B/B/bitbake/bitbake-1.0.tar.gz',
+ 'http://aaaa/A/A/A/B/B/bitbake/bitbake-1.0.tar.gz'])
class GitDownloadDirectoryNamingTest(FetcherTest):
@@ -564,8 +573,8 @@ class GitDownloadDirectoryNamingTest(FetcherTest):
super(GitDownloadDirectoryNamingTest, self).setUp()
self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
self.recipe_dir = "git.openembedded.org.bitbake"
- self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
- self.mirror_dir = "github.com.openembedded.bitbake.git"
+ self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+ self.mirror_dir = "git.openembedded.org.bitbake-contrib.git"
self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
@@ -612,8 +621,8 @@ class TarballNamingTest(FetcherTest):
super(TarballNamingTest, self).setUp()
self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
- self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
- self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz"
+ self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+ self.mirror_tarball = "git2_git.openembedded.org.bitbake-contrib.git.tar.gz"
self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
@@ -646,8 +655,8 @@ class GitShallowTarballNamingTest(FetcherTest):
super(GitShallowTarballNamingTest, self).setUp()
self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz"
- self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
- self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz"
+ self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+ self.mirror_tarball = "gitshallow_git.openembedded.org.bitbake-contrib.git_82ea737-1_master.tar.gz"
self.d.setVar('BB_GIT_SHALLOW', '1')
self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
@@ -679,7 +688,7 @@ class GitShallowTarballNamingTest(FetcherTest):
class CleanTarballTest(FetcherTest):
def setUp(self):
super(CleanTarballTest, self).setUp()
- self.recipe_url = "git://git.openembedded.org/bitbake;protocol=https"
+ self.recipe_url = "git://git.openembedded.org/bitbake;protocol=https;branch=master"
self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
@@ -692,7 +701,7 @@ class CleanTarballTest(FetcherTest):
fetcher.download()
fetcher.unpack(self.unpackdir)
- mtime = bb.process.run('git log --all -1 --format=%ct',
+ mtime = bb.process.run(['git', 'log', '--all', '-1', '--format=%ct'],
cwd=os.path.join(self.unpackdir, 'git'))
self.assertEqual(len(mtime), 2)
mtime = int(mtime[0])
@@ -720,17 +729,46 @@ class FetcherLocalTest(FetcherTest):
os.makedirs(self.localsrcdir)
touch(os.path.join(self.localsrcdir, 'a'))
touch(os.path.join(self.localsrcdir, 'b'))
+ touch(os.path.join(self.localsrcdir, 'c@d'))
os.makedirs(os.path.join(self.localsrcdir, 'dir'))
touch(os.path.join(self.localsrcdir, 'dir', 'c'))
touch(os.path.join(self.localsrcdir, 'dir', 'd'))
os.makedirs(os.path.join(self.localsrcdir, 'dir', 'subdir'))
touch(os.path.join(self.localsrcdir, 'dir', 'subdir', 'e'))
touch(os.path.join(self.localsrcdir, r'backslash\x2dsystemd-unit.device'))
- bb.process.run('tar cf archive.tar -C dir .', cwd=self.localsrcdir)
- bb.process.run('tar czf archive.tar.gz -C dir .', cwd=self.localsrcdir)
- bb.process.run('tar cjf archive.tar.bz2 -C dir .', cwd=self.localsrcdir)
+ bb.process.run(['tar', 'cf', 'archive.tar', '-C', 'dir', '.'], cwd=self.localsrcdir)
+ bb.process.run(['tar', 'czf', 'archive.tar.gz', '-C', 'dir', '.'], cwd=self.localsrcdir)
+ bb.process.run(['tar', 'cjf', 'archive.tar.bz2', '-C', 'dir', '.'], cwd=self.localsrcdir)
self.d.setVar("FILESPATH", self.localsrcdir)
+ def make_ar_package(self, package_name, data_member="data.tar"):
+ if not shutil.which("ar"):
+ self.skipTest("ar not installed")
+
+ workdir = tempfile.mkdtemp(dir=self.tempdir)
+ payload = os.path.join(workdir, "payload")
+ with open(payload, "w") as f:
+ f.write("payload\n")
+
+ data_path = os.path.join(workdir, data_member)
+ mode = "w:gz" if data_member.endswith(".gz") else "w"
+ with tarfile.open(data_path, mode) as archive:
+ archive.add(payload, arcname="payload")
+
+ with open(os.path.join(workdir, "debian-binary"), "w") as f:
+ f.write("2.0\n")
+
+ control = os.path.join(workdir, "control")
+ with open(control, "w") as f:
+ f.write("Package: fetch-test\nVersion: 1\nArchitecture: all\n")
+ with tarfile.open(os.path.join(workdir, "control.tar"), "w") as archive:
+ archive.add(control, arcname="control")
+
+ package_path = os.path.join(self.localsrcdir, package_name)
+ subprocess.check_call(["ar", "r", package_path, "debian-binary", "control.tar", data_member],
+ cwd=workdir, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ return package_name
+
def fetchUnpack(self, uris):
fetcher = bb.fetch.Fetch(uris, self.d)
fetcher.download()
@@ -751,12 +789,16 @@ class FetcherLocalTest(FetcherTest):
tree = self.fetchUnpack(['file://a', 'file://dir/c'])
self.assertEqual(tree, ['a', 'dir/c'])
+ def test_local_at(self):
+ tree = self.fetchUnpack(['file://c@d'])
+ self.assertEqual(tree, ['c@d'])
+
def test_local_backslash(self):
tree = self.fetchUnpack([r'file://backslash\x2dsystemd-unit.device'])
self.assertEqual(tree, [r'backslash\x2dsystemd-unit.device'])
def test_local_wildcard(self):
- with self.assertRaises(bb.fetch2.ParameterError):
+ with self.assertRaises(bb.fetch.ParameterError):
tree = self.fetchUnpack(['file://a', 'file://dir/*'])
def test_local_dir(self):
@@ -785,7 +827,7 @@ class FetcherLocalTest(FetcherTest):
tree = self.fetchUnpack(['file://a;subdir=%s' % os.path.join(self.unpackdir, 'bar')])
# Unpacking to an absolute path outside of the root should fail
- with self.assertRaises(bb.fetch2.UnpackError):
+ with self.assertRaises(bb.fetch.UnpackError):
self.fetchUnpack(['file://a;subdir=/bin/sh'])
def test_local_striplevel(self):
@@ -800,6 +842,40 @@ class FetcherLocalTest(FetcherTest):
tree = self.fetchUnpack(['file://archive.tar.bz2;subdir=bar;striplevel=1'])
self.assertEqual(tree, ['bar/c', 'bar/d', 'bar/subdir/e'])
+ def test_local_deb_quoted_filename(self):
+ package = self.make_ar_package("archive$(id).deb")
+ tree = self.fetchUnpack(['file://%s' % package])
+ self.assertEqual(tree, ['payload'])
+
+ def test_local_ipk_gz_data_member(self):
+ package = self.make_ar_package("archive.ipk", data_member="data.tar.gz")
+ tree = self.fetchUnpack(['file://%s' % package])
+ self.assertEqual(tree, ['payload'])
+
+ def test_local_deb_rejects_unknown_data_member_suffix(self):
+ package = self.make_ar_package("archive.deb", data_member="data.tar.foo")
+ with self.assertRaises(bb.fetch.UnpackError) as context:
+ self.fetchUnpack(['file://%s' % package])
+
+ self.assertIn("does not contain supported data.tar* file", str(context.exception))
+
+ def test_local_deb_rejects_unsafe_data_member(self):
+ package = self.make_ar_package("archive.deb", data_member="data.tar.xz;id")
+ with self.assertRaises(bb.fetch.UnpackError) as context:
+ self.fetchUnpack(['file://%s' % package])
+
+ self.assertIn("does not contain supported data.tar* file", str(context.exception))
+
+ def assertInvalidStriplevel(self, value):
+ with self.assertRaises(bb.fetch.UnpackError) as context:
+ self.fetchUnpack(['file://archive.tar;subdir=bar;striplevel=%s' % value])
+ self.assertIn("Invalid striplevel parameter", str(context.exception))
+
+ def test_local_striplevel_rejects_invalid_values(self):
+ for value in ("abc", "", "-1", "1\n", "1 2"):
+ with self.subTest(striplevel=repr(value)):
+ self.assertInvalidStriplevel(value)
+
def dummyGitTest(self, suffix):
# Create dummy local Git repo
src_dir = tempfile.mkdtemp(dir=self.tempdir,
@@ -855,7 +931,7 @@ class FetcherNoNetworkTest(FetcherTest):
self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz")))
self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz.done")))
fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/test-file.tar.gz"], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
def test_valid_missing_donestamp(self):
@@ -885,7 +961,7 @@ class FetcherNoNetworkTest(FetcherTest):
self.assertTrue(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz")))
self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz.done")))
fetcher = bb.fetch.Fetch(["http://invalid.yoctoproject.org/test-file.tar.gz"], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
# the existing file should not exist or should have be moved to "bad-checksum"
self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz")))
@@ -896,7 +972,7 @@ class FetcherNoNetworkTest(FetcherTest):
# ssh fetch does not support checksums
fetcher = bb.fetch.Fetch(["ssh://invalid@invalid.yoctoproject.org/test-file.tar.gz"], self.d)
# attempts to download with missing donestamp
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
def test_nochecksums_missing_donestamp(self):
@@ -909,7 +985,7 @@ class FetcherNoNetworkTest(FetcherTest):
# ssh fetch does not support checksums
fetcher = bb.fetch.Fetch(["ssh://invalid@invalid.yoctoproject.org/test-file.tar.gz"], self.d)
# attempts to download with missing donestamp
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
def test_nochecksums_has_donestamp(self):
@@ -938,7 +1014,7 @@ class FetcherNoNetworkTest(FetcherTest):
self.assertTrue(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz.done")))
# ssh fetch does not support checksums
fetcher = bb.fetch.Fetch(["ssh://invalid@invalid.yoctoproject.org/test-file.tar.gz"], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
# both files should still exist
self.assertFalse(os.path.exists(os.path.join(self.dldir, "test-file.tar.gz")))
@@ -1055,12 +1131,6 @@ class FetcherNetworkTest(FetcherTest):
url1 = url2 = "git://git.openembedded.org/bitbake;rev=dead05b0b4ba0959fe0624d2a4885d7b70426da5;branch=master;protocol=https"
self.assertRaises(bb.fetch.FetchError, self.gitfetcher, url1, url2)
- @skipIfNoNetwork()
- def test_gitfetch_tagandrev(self):
- # SRCREV is set but does not match rev= parameter
- url1 = url2 = "git://git.openembedded.org/bitbake;rev=270a05b0b4ba0959fe0624d2a4885d7b70426da5;tag=270a05b0b4ba0959fe0624d2a4885d7b70426da5;protocol=https"
- self.assertRaises(bb.fetch.FetchError, self.gitfetcher, url1, url2)
-
@skipIfNoNetwork()
def test_gitfetch_usehead(self):
# Since self.gitfetcher() sets SRCREV we expect this to override
@@ -1095,7 +1165,7 @@ class FetcherNetworkTest(FetcherTest):
@skipIfNoNetwork()
def test_gitfetch_finds_local_repository_when_premirror_rewrites_the_recipe_url(self):
realurl = "https://git.openembedded.org/bitbake"
- recipeurl = "git://someserver.org/bitbake;protocol=https"
+ recipeurl = "git://someserver.org/bitbake;protocol=https;branch=master"
self.sourcedir = self.unpackdir.replace("unpacked", "sourcemirror.git")
os.chdir(self.tempdir)
self.git(['clone', realurl, self.sourcedir], cwd=self.tempdir)
@@ -1107,7 +1177,7 @@ class FetcherNetworkTest(FetcherTest):
# URL with ssh submodules
url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7;branch=master;protocol=https"
# Original URL (comment this if you have ssh access to git.yoctoproject.org)
- url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee;branch=master;protocol=https"
+ url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
@@ -1127,7 +1197,7 @@ class FetcherNetworkTest(FetcherTest):
def test_git_submodule_restricted_network_premirrors(self):
# this test is to ensure that premirrors will be tried in restricted network
# that is, BB_ALLOWED_NETWORKS does not contain the domain the url uses
- url = "gitsm://github.com/grpc/grpc.git;protocol=https;name=grpc;branch=v1.60.x;rev=0ef13a7555dbaadd4633399242524129eef5e231"
+ url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https"
# create a download directory to be used as premirror later
tempdir = tempfile.mkdtemp(prefix="bitbake-fetch-")
dl_premirror = os.path.join(tempdir, "download-premirror")
@@ -1146,7 +1216,7 @@ class FetcherNetworkTest(FetcherTest):
def test_git_submodule_dbus_broker(self):
# The following external repositories have show failures in fetch and unpack operations
# We want to avoid regressions!
- url = "gitsm://github.com/bus1/dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main"
+ url = "gitsm://git.yoctoproject.org/bbfetchtests-bus1-dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
@@ -1162,7 +1232,7 @@ class FetcherNetworkTest(FetcherTest):
@skipIfNoNetwork()
def test_git_submodule_CLI11(self):
- url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main"
+ url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=474872a222f61e57b16c2feb774610a4595a729e;branch=main"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
@@ -1172,17 +1242,16 @@ class FetcherNetworkTest(FetcherTest):
repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"')
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
- self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
@skipIfNoNetwork()
def test_git_submodule_update_CLI11(self):
""" Prevent regression on update detection not finding missing submodule, or modules without needed commits """
- url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main"
+ url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=fe0e149a0624db734de37dc827af3328855f51cf;branch=main"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# CLI11 that pulls in a newer nlohmann-json
- url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main"
+ url = "gitsm://git.yoctoproject.org/bbfetchtests-cliutils-cli11;protocol=https;rev=141f9dca45bd5ef42e7b1a06637860b7581370c8;branch=main"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
@@ -1192,11 +1261,10 @@ class FetcherNetworkTest(FetcherTest):
repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"')
self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
- self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
@skipIfNoNetwork()
def test_git_submodule_aktualizr(self):
- url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
+ url = "gitsm://git.yoctoproject.org/bbfetchtests-advancedtelematic-aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
@@ -1241,16 +1309,15 @@ class FetcherNetworkTest(FetcherTest):
@skipIfNoNetwork()
def test_git_submodule_reference_to_parent(self):
- self.recipe_url = "gitsm://github.com/gflags/gflags.git;protocol=https;branch=master"
+ self.recipe_url = "gitsm://git.yoctoproject.org/bbfetchtests-gflags;protocol=https;branch=master"
self.d.setVar("SRCREV", "14e1138441bbbb584160cb1c0a0426ec1bac35f1")
with Timeout(60):
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
fetcher.download()
class SVNTest(FetcherTest):
def skipIfNoSvn():
- import shutil
if not shutil.which("svn"):
return unittest.skip("svn not installed, tests being skipped")
@@ -1275,18 +1342,18 @@ class SVNTest(FetcherTest):
repo_dir = tempfile.mkdtemp(dir=self.tempdir,
prefix='svnfetch_localrepo_')
repo_dir = os.path.abspath(repo_dir)
- bb.process.run("svnadmin create project", cwd=repo_dir)
+ bb.process.run(['svnadmin', 'create', 'project'], cwd=repo_dir)
self.repo_url = "file://%s/project" % repo_dir
- bb.process.run("svn import --non-interactive -m 'Initial import' %s %s/trunk" % (src_dir, self.repo_url),
+ bb.process.run(['svn', 'import', '--non-interactive', '-m', 'Initial import', src_dir, "%s/trunk" % self.repo_url],
cwd=repo_dir)
- bb.process.run("svn co %s svnfetch_co" % self.repo_url, cwd=self.tempdir)
+ bb.process.run(['svn', 'co', self.repo_url, 'svnfetch_co'], cwd=self.tempdir)
# Github won't emulate SVN anymore (see https://github.blog/2023-01-20-sunsetting-subversion-support/)
# Use still accessible svn repo (only trunk to avoid longer downloads)
- bb.process.run("svn propset svn:externals 'bitbake https://svn.apache.org/repos/asf/serf/trunk' .",
+ bb.process.run(['svn', 'propset', 'svn:externals', "'bitbake https://svn.apache.org/repos/asf/serf/trunk'", "."],
cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
- bb.process.run("svn commit --non-interactive -m 'Add external'",
+ bb.process.run(['svn', 'commit', '--non-interactive', '-m', 'Add external'],
cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
self.src_dir = src_dir
@@ -1373,17 +1440,17 @@ class TrustedNetworksTest(FetcherTest):
self.assertFalse(bb.fetch.trusted_network(self.d, url))
class URLHandle(unittest.TestCase):
-
+ # Quote password as per RFC3986
+ password = urllib.parse.quote(r"!#$%^&*()-_={}[]\|:?,.<>~`", r"!$&'/()*+,;=")
datatable = {
"http://www.google.com/index.html" : ('http', 'www.google.com', '/index.html', '', '', {}),
- "cvs://anoncvs@cvs.handhelds.org/cvs;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', '', {'module': 'familiar/dist/ipkg'}),
- "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous', collections.OrderedDict([('tag', 'V0-99-81'), ('module', 'familiar/dist/ipkg')])),
"git://git.openembedded.org/bitbake;branch=@foo;protocol=https" : ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo', 'protocol' : 'https'}),
"file://somelocation;someparam=1": ('file', '', 'somelocation', '', '', {'someparam': '1'}),
+ "file://example@.service": ('file', '', 'example@.service', '', '', {}),
"https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" : ('https', 'somesite.com', '/somerepo.git', '', '', {'user': 'anyUser:idtoken=1234'}),
- r'git://s.o-me_ONE:!#$%^&*()-_={}[]\|:?,.<>~`@git.openembedded.org/bitbake;branch=main;protocol=https': ('git', 'git.openembedded.org', '/bitbake', 's.o-me_ONE', r'!#$%^&*()-_={}[]\|:?,.<>~`', {'branch': 'main', 'protocol' : 'https'}),
+ 'git://s.o-me_ONE:%s@git.openembedded.org/bitbake;branch=main;protocol=https' % password: ('git', 'git.openembedded.org', '/bitbake', 's.o-me_ONE', password, {'branch': 'main', 'protocol' : 'https'}),
}
- # we require a pathname to encodeurl but users can still pass such urls to
+ # we require a pathname to encodeurl but users can still pass such urls to
# decodeurl and we need to handle them
decodedata = datatable.copy()
decodedata.update({
@@ -1399,13 +1466,15 @@ class URLHandle(unittest.TestCase):
def test_encodeurl(self):
for k, v in self.datatable.items():
result = bb.fetch.encodeurl(v)
+ if result.startswith("file:"):
+ result = urllib.parse.unquote(result)
self.assertEqual(result, k)
class FetchLatestVersionTest(FetcherTest):
test_git_uris = {
# version pattern "X.Y.Z"
- ("mx-1.0", "git://github.com/clutter-project/mx.git;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "", "")
+ ("mx-1.0", "git://git.yoctoproject.org/bbfetchtests-clutter-mx;branch=mx-1.4;protocol=https", "9b1db6b8060bd00b121a692f942404a24ae2960f", "", "")
: "1.99.4",
# version pattern "vX.Y"
# mirror of git.infradead.org since network issues interfered with testing
@@ -1419,12 +1488,12 @@ class FetchLatestVersionTest(FetcherTest):
("dtc", "git://git.yoctoproject.org/bbfetchtests-dtc.git;branch=master;protocol=https", "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf", "", "")
: "1.4.0",
# combination version pattern
- ("sysprof", "git://gitlab.gnome.org/GNOME/sysprof.git;protocol=https;branch=master", "cd44ee6644c3641507fb53b8a2a69137f2971219", "", "")
+ ("sysprof", "git://git.yoctoproject.org/sysprof.git;protocol=https;branch=master", "cd44ee6644c3641507fb53b8a2a69137f2971219", "", "")
: "1.2.0",
- ("u-boot-mkimage", "git://git.denx.de/u-boot.git;branch=master;protocol=git", "62c175fbb8a0f9a926c88294ea9f7e88eb898f6c", "", "")
+ ("u-boot-mkimage", "git://git.yoctoproject.org/bbfetchtests-u-boot.git;branch=master;protocol=https", "62c175fbb8a0f9a926c88294ea9f7e88eb898f6c", "", "")
: "2014.01",
# version pattern "yyyymmdd"
- ("mobile-broadband-provider-info", "git://gitlab.gnome.org/GNOME/mobile-broadband-provider-info.git;protocol=https;branch=master", "4ed19e11c2975105b71b956440acdb25d46a347d", "", "")
+ ("mobile-broadband-provider-info", "git://git.yoctoproject.org/mobile-broadband-provider-info.git;protocol=https;branch=master", "4ed19e11c2975105b71b956440acdb25d46a347d", "", "")
: "20120614",
# packages with a valid UPSTREAM_CHECK_GITTAGREGEX
# mirror of git://anongit.freedesktop.org/xorg/driver/xf86-video-omap since network issues interfered with testing
@@ -1432,106 +1501,210 @@ class FetchLatestVersionTest(FetcherTest):
: "0.4.3",
("build-appliance-image", "git://git.yoctoproject.org/poky;branch=master;protocol=https", "b37dd451a52622d5b570183a81583cc34c2ff555", r"(?P<pver>(([0-9][\.|_]?)+[0-9]))", "")
: "11.0.0",
- ("chkconfig-alternatives-native", "git://github.com/kergoth/chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", r"chkconfig\-(?P<pver>((\d+[\.\-_]*)+))", "")
+ ("chkconfig-alternatives-native", "git://git.yoctoproject.org/bbfetchtests-kergoth-chkconfig;branch=sysroot;protocol=https", "cd437ecbd8986c894442f8fce1e0061e20f04dee", r"chkconfig\-(?P<pver>((\d+[\.\-_]*)+))", "")
: "1.3.59",
- ("remake", "git://github.com/rocky/remake.git;protocol=https;branch=master", "f05508e521987c8494c92d9c2871aec46307d51d", r"(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))", "")
+ ("remake", "git://git.yoctoproject.org/bbfetchtests-rocky-remake;protocol=https;branch=master", "f05508e521987c8494c92d9c2871aec46307d51d", r"(?P<pver>(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))", "")
: "3.82+dbg0.9",
- ("sysdig", "git://github.com/draios/sysdig.git;branch=dev;protocol=https", "4fb6288275f567f63515df0ff0a6518043ecfa9b", r"^(?P<pver>\d+(\.\d+)+)", "10.0.0")
+ ("sysdig", "git://git.yoctoproject.org/bbfetchtests-draios-sysdig;branch=dev;protocol=https", "4fb6288275f567f63515df0ff0a6518043ecfa9b", r"^(?P<pver>\d+(\.\d+)+)", "10.0.0")
: "0.28.0",
}
+ WgetTestData = collections.namedtuple("WgetTestData", ["pn", "path", "pv", "check_uri", "check_regex"], defaults=[None, None, None])
test_wget_uris = {
#
# packages with versions inside directory name
#
# http://kernel.org/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2
- ("util-linux", "/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2", "", "")
+ WgetTestData("util-linux", "/pub/linux/utils/util-linux/v2.23/util-linux-2.24.2.tar.bz2")
: "2.24.2",
# http://www.abisource.com/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz
- ("enchant", "/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz", "", "")
+ WgetTestData("enchant", "/downloads/enchant/1.6.0/enchant-1.6.0.tar.gz")
: "1.6.0",
# http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz
- ("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz", "", "")
+ WgetTestData("cmake", "/files/v2.8/cmake-2.8.12.1.tar.gz")
: "2.8.12.1",
# https://download.gnome.org/sources/libxml2/2.9/libxml2-2.9.14.tar.xz
- ("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz", "", "")
+ WgetTestData("libxml2", "/software/libxml2/2.9/libxml2-2.9.14.tar.xz")
: "2.10.3",
#
# packages with versions only in current directory
#
# https://downloads.yoctoproject.org/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2
- ("eglic", "/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2", "", "")
+ WgetTestData("eglic", "/releases/eglibc/eglibc-2.18-svnr23787.tar.bz2")
: "2.19",
# https://downloads.yoctoproject.org/releases/gnu-config/gnu-config-20120814.tar.bz2
- ("gnu-config", "/releases/gnu-config/gnu-config-20120814.tar.bz2", "", "")
+ WgetTestData("gnu-config", "/releases/gnu-config/gnu-config-20120814.tar.bz2")
: "20120814",
#
# packages with "99" in the name of possible version
#
# http://freedesktop.org/software/pulseaudio/releases/pulseaudio-4.0.tar.xz
- ("pulseaudio", "/software/pulseaudio/releases/pulseaudio-4.0.tar.xz", "", "")
+ WgetTestData("pulseaudio", "/software/pulseaudio/releases/pulseaudio-4.0.tar.xz")
: "5.0",
# http://xorg.freedesktop.org/releases/individual/xserver/xorg-server-1.15.1.tar.bz2
- ("xserver-xorg", "/releases/individual/xserver/xorg-server-1.15.1.tar.bz2", "", "")
+ WgetTestData("xserver-xorg", "/releases/individual/xserver/xorg-server-1.15.1.tar.bz2")
: "1.15.1",
#
# packages with valid UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX
#
# http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2
# https://github.com/apple/cups/releases
- ("cups", "/software/1.7.2/cups-1.7.2-source.tar.bz2", "/apple/cups/releases", r"(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
+ WgetTestData("cups", "/software/1.7.2/cups-1.7.2-source.tar.bz2", check_uri="/apple/cups/releases", check_regex=r"(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
: "2.0.0",
# http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz
# http://ftp.debian.org/debian/pool/main/d/db5.3/
- ("db", "/berkeley-db/db-5.3.21.tar.gz", "/debian/pool/main/d/db5.3/", r"(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
+ WgetTestData("db", "/berkeley-db/db-5.3.21.tar.gz", check_uri="/debian/pool/main/d/db5.3/", check_regex=r"(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
: "5.3.10",
#
# packages where the tarball compression changed in the new version
#
# http://ftp.debian.org/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz
- ("minicom", "/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz", "", "")
+ WgetTestData("minicom", "/debian/pool/main/m/minicom/minicom_2.7.1.orig.tar.gz")
: "2.8",
+
+ #
+ # packages where the path doesn't actually contain the filename, so downloadfilename should be respected
+ #
+ WgetTestData("miniupnpd", "/software/miniupnp/download.php?file=miniupnpd_2.1.20191006.tar.gz;downloadfilename=miniupnpd_2.1.20191006.tar.gz", pv="2.1.20191006", check_uri="/software/miniupnp/download.php", check_regex=r"miniupnpd-(?P<pver>\d+(\.\d+)+)\.tar")
+ : "2.3.7",
+ }
+
+ test_crate_uris = {
+ # basic example; version pattern "A.B.C+cargo-D.E.F"
+ ("cargo-c", "crate://crates.io/cargo-c/0.9.18+cargo-0.69")
+ : "0.9.29"
+ }
+
+ test_git_stable_uris = {
+ ("dtc", "git://git.yoctoproject.org/bbfetchtests-dtc.git;branch=master;protocol=https", "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf", "", r"^1\.4\.\d+$")
+ : ("1.4.0", "1.5.0"),
+ ("systemd", "git://git.yoctoproject.org/bbfetchtests-systemd;protocol=https;branch=stable/v259-stable", "b3d8fc43e9cb531d958c17ef2cd93b374bc14e8a", "", r"^259\.\d+$")
+ : ("259.5", "260")
+ }
+
+ test_wget_stable_uris = {
+ ("openssh", "https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-10.2p1.tar.gz", "10.2p1", "", "", r"^10\.2p\d+$")
+ : ("10.2p1", "10.3")
+ }
+
+ test_crate_stable_uris = {
+ ("cargo-c", "crate://crates.io/cargo-c/0.9.18+cargo-0.69", r"^0\.9\.\d+")
+ : ("0.9.29", "0.10.0")
}
@skipIfNoNetwork()
def test_git_latest_versionstring(self):
for k, v in self.test_git_uris.items():
- self.d.setVar("PN", k[0])
- self.d.setVar("SRCREV", k[2])
- self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3])
- ud = bb.fetch2.FetchData(k[1], self.d)
- pupver= ud.method.latest_versionstring(ud, self.d)
- verstring = pupver[0]
- self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
- r = bb.utils.vercmp_string(v, verstring)
- self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring))
- if k[4]:
- r = bb.utils.vercmp_string(verstring, k[4])
- self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], verstring, k[4]))
+ with self.subTest(pn=k[0]):
+ self.d.setVar("PN", k[0])
+ self.d.setVar("SRCREV", k[2])
+ self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3])
+ ud = bb.fetch.FetchData(k[1], self.d)
+ pupver= ud.method.latest_versionstring(ud, self.d)
+ verstring = pupver[0]
+ self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
+ r = bb.utils.vercmp_string(v, verstring)
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring))
+ if k[4]:
+ r = bb.utils.vercmp_string(verstring, k[4])
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], verstring, k[4]))
def test_wget_latest_versionstring(self):
testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata"
- server = HTTPService(testdata)
+ server = HTTPService(testdata, host="127.0.0.1")
server.start()
port = server.port
try:
- for k, v in self.test_wget_uris.items():
+ for data, v in self.test_wget_uris.items():
+ with self.subTest(pn=data.pn):
+ self.d.setVar("PN", data.pn)
+ self.d.setVar("PV", data.pv)
+ if data.check_uri:
+ checkuri = "http://127.0.0.1:%s/%s" % (port, data.check_uri)
+ self.d.setVar("UPSTREAM_CHECK_URI", checkuri)
+ if data.check_regex:
+ self.d.setVar("UPSTREAM_CHECK_REGEX", data.check_regex)
+
+ url = "http://127.0.0.1:%s/%s" % (port, data.path)
+ ud = bb.fetch.FetchData(url, self.d)
+ pupver = ud.method.latest_versionstring(ud, self.d)
+ verstring = pupver[0]
+ self.assertTrue(verstring, msg="Could not find upstream version for %s" % data.pn)
+ r = bb.utils.vercmp_string(v, verstring)
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (data.pn, v, verstring))
+ finally:
+ server.stop()
+
+ @skipIfNoNetwork()
+ def test_crate_latest_versionstring(self):
+ for k, v in self.test_crate_uris.items():
+ with self.subTest(pn=k[0]):
self.d.setVar("PN", k[0])
- checkuri = ""
- if k[2]:
- checkuri = "http://localhost:%s/" % port + k[2]
- self.d.setVar("UPSTREAM_CHECK_URI", checkuri)
- self.d.setVar("UPSTREAM_CHECK_REGEX", k[3])
- url = "http://localhost:%s/" % port + k[1]
- ud = bb.fetch2.FetchData(url, self.d)
+ ud = bb.fetch.FetchData(k[1], self.d)
pupver = ud.method.latest_versionstring(ud, self.d)
verstring = pupver[0]
self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
r = bb.utils.vercmp_string(v, verstring)
self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s <= %s" % (k[0], v, verstring))
- finally:
- server.stop()
+ @skipIfNoNetwork()
+ def test_git_latest_versionstring_stable(self):
+ for k, v in self.test_git_stable_uris.items():
+ with self.subTest(pn=k[0]):
+ self.d.setVar("PN", k[0])
+ self.d.setVar("SRCREV", k[2])
+ self.d.setVar("UPSTREAM_CHECK_GITTAGREGEX", k[3])
+ filter_regex = k[4]
+ ud = bb.fetch.FetchData(k[1], self.d)
+ pupver= ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex)
+ verstring = pupver[0]
+ self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
+ v_less_or_equal = v[0]
+ v_larger = v[1]
+ r = bb.utils.vercmp_string(v_less_or_equal, verstring)
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring))
+ r = bb.utils.vercmp_string(verstring, v_larger)
+ self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring))
+
+ @skipIfNoNetwork()
+ def test_wget_latest_versionstring_stable(self):
+ for k, v in self.test_wget_stable_uris.items():
+ with self.subTest(pn=k[0]):
+ self.d.setVar("PN", k[0])
+ url = k[1]
+ self.d.setVar("PV", k[2])
+ if k[3]:
+ self.d.setVar("UPSTREAM_CHECK_URI", k[3])
+ if k[4]:
+ self.d.setVar("UPSTREAM_CHECK_REGEX", k[4])
+ filter_regex = k[5]
+ ud = bb.fetch.FetchData(url, self.d)
+ pupver= ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex)
+ verstring = pupver[0]
+ self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
+ v_less_or_equal = v[0]
+ v_larger = v[1]
+ r = bb.utils.vercmp_string(v_less_or_equal, verstring)
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring))
+ r = bb.utils.vercmp_string(verstring, v_larger)
+ self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring))
+
+ @skipIfNoNetwork()
+ def test_crate_latest_versionstring_stable(self):
+ for k, v in self.test_crate_stable_uris.items():
+ with self.subTest(pn=k[0]):
+ self.d.setVar("PN", k[0])
+ ud = bb.fetch.FetchData(k[1], self.d)
+ filter_regex = k[2]
+ pupver = ud.method.latest_versionstring(ud, self.d, filter_regex=filter_regex)
+ verstring = pupver[0]
+ self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
+ v_less_or_equal = v[0]
+ v_larger = v[1]
+ r = bb.utils.vercmp_string(v_less_or_equal, verstring)
+ self.assertTrue(r == -1 or r == 0, msg="Package %s, version: %s < %s" % (k[0], v_less_or_equal, verstring))
+ r = bb.utils.vercmp_string(verstring, v_larger)
+ self.assertTrue(r == -1, msg="Package %s, version: %s <= %s" % (k[0], v_larger, verstring))
class FetchCheckStatusTest(FetcherTest):
test_wget_uris = ["https://downloads.yoctoproject.org/releases/sato/sato-engine-0.1.tar.gz",
@@ -1541,14 +1714,46 @@ class FetchCheckStatusTest(FetcherTest):
"https://docs.yoctoproject.org",
"https://downloads.yoctoproject.org/releases/opkg/opkg-0.1.7.tar.gz",
"https://downloads.yoctoproject.org/releases/opkg/opkg-0.3.0.tar.gz",
- "ftp://sourceware.org/pub/libffi/libffi-1.20.tar.gz",
- # GitHub releases are hosted on Amazon S3, which doesn't support HEAD
- "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz"
]
+ def _start_checkstatus_server(self):
+ class CheckStatusHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
+ def do_HEAD(self):
+ self.server.requests.append((self.path, dict(self.headers)))
+ if self.path == "/a" and self.server.redirect_url:
+ self.send_response(302)
+ self.send_header("Location", self.server.redirect_url)
+ self.end_headers()
+ return
+ self.send_response(200)
+ self.end_headers()
+
+ def log_message(self, format_str, *args):
+ pass
+
+ server = http.server.HTTPServer(("127.0.0.1", 0), CheckStatusHTTPRequestHandler)
+ server.redirect_url = None
+ server.requests = []
+ thread = threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.05})
+ thread.daemon = True
+ thread.start()
+
+ def stop_server():
+ server.shutdown()
+ thread.join()
+ server.server_close()
+
+ self.addCleanup(stop_server)
+ return server
+
+ def _checkstatus(self, url):
+ fetch = bb.fetch.Fetch([url], self.d)
+ ud = fetch.ud[url]
+ return ud.method.checkstatus(fetch, ud, self.d)
+
@skipIfNoNetwork()
def test_wget_checkstatus(self):
- fetch = bb.fetch2.Fetch(self.test_wget_uris, self.d)
+ fetch = bb.fetch.Fetch(self.test_wget_uris, self.d)
for u in self.test_wget_uris:
with self.subTest(url=u):
ud = fetch.ud[u]
@@ -1558,10 +1763,10 @@ class FetchCheckStatusTest(FetcherTest):
@skipIfNoNetwork()
def test_wget_checkstatus_connection_cache(self):
- from bb.fetch2 import FetchConnectionCache
+ from bb.fetch import FetchConnectionCache
connection_cache = FetchConnectionCache()
- fetch = bb.fetch2.Fetch(self.test_wget_uris, self.d,
+ fetch = bb.fetch.Fetch(self.test_wget_uris, self.d,
connection_cache = connection_cache)
for u in self.test_wget_uris:
@@ -1573,6 +1778,99 @@ class FetchCheckStatusTest(FetcherTest):
connection_cache.close_connections()
+ @unittest.skipUnless(shutil.which("openssl"), "openssl not installed")
+ def test_wget_checkstatus_https_connection_cache(self):
+ import ssl
+ from socketserver import ThreadingMixIn
+ from bb.fetch import FetchConnectionCache
+
+ class HTTPSRequestHandler(http.server.BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def do_HEAD(self):
+ self.send_response(200)
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+
+ def log_message(self, format_str, *args):
+ pass
+
+ class HTTPSServer(ThreadingMixIn, http.server.HTTPServer):
+ daemon_threads = True
+
+ def __init__(self, *args, **kwargs):
+ self.connection_count = 0
+ super().__init__(*args, **kwargs)
+
+ def get_request(self):
+ request, client_address = super().get_request()
+ self.connection_count += 1
+ return request, client_address
+
+ certificate = os.path.join(self.tempdir, "certificate.pem")
+ private_key = os.path.join(self.tempdir, "private-key.pem")
+ result = subprocess.run(
+ ["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
+ "-keyout", private_key, "-out", certificate, "-days", "1",
+ "-subj", "/CN=127.0.0.1"],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+ self.assertEqual(result.returncode, 0,
+ "openssl certificate generation failed:\n%s" % result.stdout)
+
+ server = HTTPSServer(("127.0.0.1", 0), HTTPSRequestHandler)
+ context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
+ context.load_cert_chain(certificate, private_key)
+ server.socket = context.wrap_socket(server.socket, server_side=True)
+ server_thread = threading.Thread(target=server.serve_forever)
+ server_thread.daemon = True
+ server_thread.start()
+
+ connection_cache = FetchConnectionCache()
+ try:
+ url = "https://127.0.0.1:%s/test" % server.server_port
+ self.d.setVar("BB_CHECK_SSL_CERTS", "0")
+ fetch = bb.fetch.Fetch([url], self.d,
+ connection_cache=connection_cache)
+ ud = fetch.ud[url]
+ self.assertTrue(ud.method.checkstatus(fetch, ud, self.d))
+ self.assertTrue(ud.method.checkstatus(fetch, ud, self.d))
+ self.assertEqual(server.connection_count, 1)
+
+ # A connection established without certificate checks must not be
+ # reused after certificate checking is enabled.
+ self.d.setVar("BB_CHECK_SSL_CERTS", "1")
+ self.assertFalse(ud.method.checkstatus(fetch, ud, self.d))
+ finally:
+ connection_cache.close_connections()
+ server.shutdown()
+ server_thread.join()
+ server.server_close()
+
+ def test_wget_checkstatus_same_origin_redirect_keeps_auth(self):
+ server = self._start_checkstatus_server()
+ server.redirect_url = "http://127.0.0.1:%s/b" % server.server_port
+
+ url = "http://127.0.0.1:%s/a;user=user;pswd=pass" % server.server_port
+ self.assertTrue(self._checkstatus(url))
+
+ self.assertEqual(len(server.requests), 2)
+ redirected_headers = {k.lower(): v for k, v in server.requests[1][1].items()}
+ self.assertIn("authorization", redirected_headers)
+
+ def test_wget_checkstatus_different_origin_redirect_drops_auth(self):
+ origin = self._start_checkstatus_server()
+ target = self._start_checkstatus_server()
+ # Same host but different port is a different origin.
+ origin.redirect_url = "http://127.0.0.1:%s/b" % target.server_port
+
+ url = "http://127.0.0.1:%s/a;user=user;pswd=pass" % origin.server_port
+ self.assertTrue(self._checkstatus(url))
+
+ self.assertEqual(len(origin.requests), 1)
+ self.assertEqual(len(target.requests), 1)
+ redirected_headers = {k.lower(): v for k, v in target.requests[0][1].items()}
+ self.assertNotIn("authorization", redirected_headers)
+
class GitMakeShallowTest(FetcherTest):
def setUp(self):
@@ -1593,11 +1891,6 @@ class GitMakeShallowTest(FetcherTest):
actual_count = len(revs.splitlines())
self.assertEqual(expected_count, actual_count, msg='Object count `%d` is not the expected `%d`' % (actual_count, expected_count))
- def make_shallow(self, args=None):
- if args is None:
- args = ['HEAD']
- return bb.process.run([bb.fetch2.git.Git.make_shallow_path] + args, cwd=self.gitdir)
-
def add_empty_file(self, path, msg=None):
if msg is None:
msg = path
@@ -1605,88 +1898,6 @@ class GitMakeShallowTest(FetcherTest):
self.git(['add', path])
self.git(['commit', '-m', msg, path])
- def test_make_shallow_single_branch_no_merge(self):
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.assertRevCount(2)
- self.make_shallow()
- self.assertRevCount(1)
-
- def test_make_shallow_single_branch_one_merge(self):
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.git('checkout -b a_branch')
- self.add_empty_file('c')
- self.git('checkout master')
- self.add_empty_file('d')
- self.git('merge --no-ff --no-edit a_branch')
- self.git('branch -d a_branch')
- self.add_empty_file('e')
- self.assertRevCount(6)
- self.make_shallow(['HEAD~2'])
- self.assertRevCount(5)
-
- def test_make_shallow_at_merge(self):
- self.add_empty_file('a')
- self.git('checkout -b a_branch')
- self.add_empty_file('b')
- self.git('checkout master')
- self.git('merge --no-ff --no-edit a_branch')
- self.git('branch -d a_branch')
- self.assertRevCount(3)
- self.make_shallow()
- self.assertRevCount(1)
-
- def test_make_shallow_annotated_tag(self):
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.git('tag -a -m a_tag a_tag')
- self.assertRevCount(2)
- self.make_shallow(['a_tag'])
- self.assertRevCount(1)
-
- def test_make_shallow_multi_ref(self):
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.git('checkout -b a_branch')
- self.add_empty_file('c')
- self.git('checkout master')
- self.add_empty_file('d')
- self.git('checkout -b a_branch_2')
- self.add_empty_file('a_tag')
- self.git('tag a_tag')
- self.git('checkout master')
- self.git('branch -D a_branch_2')
- self.add_empty_file('e')
- self.assertRevCount(6, ['--all'])
- self.make_shallow()
- self.assertRevCount(5, ['--all'])
-
- def test_make_shallow_multi_ref_trim(self):
- self.add_empty_file('a')
- self.git('checkout -b a_branch')
- self.add_empty_file('c')
- self.git('checkout master')
- self.assertRevCount(1)
- self.assertRevCount(2, ['--all'])
- self.assertRefs(['master', 'a_branch'])
- self.make_shallow(['-r', 'master', 'HEAD'])
- self.assertRevCount(1, ['--all'])
- self.assertRefs(['master'])
-
- def test_make_shallow_noop(self):
- self.add_empty_file('a')
- self.assertRevCount(1)
- self.make_shallow()
- self.assertRevCount(1)
-
- @skipIfNoNetwork()
- def test_make_shallow_bitbake(self):
- self.git('remote add origin https://github.com/openembedded/bitbake')
- self.git('fetch --tags origin')
- orig_revs = len(self.git('rev-list --all').splitlines())
- self.make_shallow(['refs/tags/1.10.0'])
- self.assertRevCount(orig_revs - 1746, ['--all'])
class GitShallowTest(FetcherTest):
def setUp(self):
@@ -1704,7 +1915,7 @@ class GitShallowTest(FetcherTest):
uri = 'git://%s;protocol=file;subdir=${S};branch=master' % self.srcdir
self.d.setVar('SRC_URI', uri)
self.d.setVar('SRCREV', '${AUTOREV}')
- self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
+ self.d.setVar('AUTOREV', '${@bb.fetch.get_autorev(d)}')
self.d.setVar('BB_GIT_SHALLOW', '1')
self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '0')
@@ -1715,6 +1926,8 @@ class GitShallowTest(FetcherTest):
if cwd is None:
cwd = self.gitdir
actual_refs = self.git(['for-each-ref', '--format=%(refname)'], cwd=cwd).splitlines()
+ # Resolve references into the same format as the comparision (needed by git 2.48 onwards)
+ actual_refs = self.git(['rev-parse', '--symbolic-full-name'] + actual_refs, cwd=cwd).splitlines()
full_expected = self.git(['rev-parse', '--symbolic-full-name'] + expected_refs, cwd=cwd).splitlines()
self.assertEqual(sorted(set(full_expected)), sorted(set(actual_refs)))
@@ -1747,7 +1960,7 @@ class GitShallowTest(FetcherTest):
uri = d.expand(uri)
uris = [uri]
- fetcher = bb.fetch2.Fetch(uris, d)
+ fetcher = bb.fetch.Fetch(uris, d)
fetcher.download()
ud = fetcher.ud[uri]
return fetcher, ud
@@ -1761,7 +1974,6 @@ class GitShallowTest(FetcherTest):
def fetch_shallow(self, uri=None, disabled=False, keepclone=False):
"""Fetch a uri, generating a shallow tarball, then unpack using it"""
fetcher, ud = self.fetch_and_unpack(uri)
- assert os.path.exists(ud.clonedir), 'Git clone in DLDIR (%s) does not exist for uri %s' % (ud.clonedir, uri)
# Confirm that the unpacked repo is unshallow
if not disabled:
@@ -1769,9 +1981,10 @@ class GitShallowTest(FetcherTest):
# fetch and unpack, from the shallow tarball
bb.utils.remove(self.gitdir, recurse=True)
- bb.process.run('chmod u+w -R "%s"' % ud.clonedir)
- bb.utils.remove(ud.clonedir, recurse=True)
- bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True)
+ if os.path.exists(ud.clonedir):
+ bb.process.run(['chmod', 'u+w', '-R', ud.clonedir])
+ bb.utils.remove(ud.clonedir, recurse=True)
+ bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True)
# confirm that the unpacked repo is used when no git clone or git
# mirror tarball is available
@@ -1796,7 +2009,7 @@ class GitShallowTest(FetcherTest):
self.add_empty_file('b')
self.assertRevCount(2, cwd=self.srcdir)
- srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
+ srcrev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
self.d.setVar('SRCREV', srcrev)
uri = self.d.getVar('SRC_URI').split()[0]
uri = '%s;nobranch=1;bare=1' % uri
@@ -1854,11 +2067,16 @@ class GitShallowTest(FetcherTest):
self.add_empty_file('c')
self.assertRevCount(3, cwd=self.srcdir)
+ # Clone without tarball
+ self.d.setVar('BB_GIT_SHALLOW', '0')
+ fetcher, ud = self.fetch()
+
# Clone and generate mirror tarball
+ self.d.setVar('BB_GIT_SHALLOW', '1')
fetcher, ud = self.fetch()
# Ensure we have a current mirror tarball, but an out of date clone
- self.git('update-ref refs/heads/master refs/heads/master~1', cwd=ud.clonedir)
+ self.git(['update-ref', 'refs/heads/master', 'refs/heads/master~1'], cwd=ud.clonedir)
self.assertRevCount(2, cwd=ud.clonedir)
# Fetch and unpack, from the current tarball, not the out of date clone
@@ -1866,6 +2084,7 @@ class GitShallowTest(FetcherTest):
fetcher, ud = self.fetch()
fetcher.unpack(self.d.getVar('WORKDIR'))
self.assertRevCount(1)
+ assert os.path.exists(os.path.join(self.d.getVar('WORKDIR'), 'git', 'c'))
def test_shallow_single_branch_no_merge(self):
self.add_empty_file('a')
@@ -1884,12 +2103,12 @@ class GitShallowTest(FetcherTest):
self.fetch_shallow()
self.assertRevCount(1)
- assert not self.git('fsck --dangling')
+ assert not self.git(['fsck', '--dangling'])
def test_shallow_srcrev_branch_truncation(self):
self.add_empty_file('a')
self.add_empty_file('b')
- b_commit = self.git('rev-parse HEAD', cwd=self.srcdir).rstrip()
+ b_commit = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).rstrip()
self.add_empty_file('c')
self.assertRevCount(3, cwd=self.srcdir)
@@ -1898,7 +2117,7 @@ class GitShallowTest(FetcherTest):
# The 'c' commit was removed entirely, and 'a' was removed from history
self.assertRevCount(1, ['--all'])
- self.assertEqual(self.git('rev-parse HEAD').strip(), b_commit)
+ self.assertEqual(self.git(['rev-parse', 'HEAD']).strip(), b_commit)
assert os.path.exists(os.path.join(self.gitdir, 'a'))
assert os.path.exists(os.path.join(self.gitdir, 'b'))
assert not os.path.exists(os.path.join(self.gitdir, 'c'))
@@ -1906,7 +2125,7 @@ class GitShallowTest(FetcherTest):
def test_shallow_ref_pruning(self):
self.add_empty_file('a')
self.add_empty_file('b')
- self.git('branch a_branch', cwd=self.srcdir)
+ self.git(['branch', 'a_branch'], cwd=self.srcdir)
self.assertRefs(['master', 'a_branch'], cwd=self.srcdir)
self.assertRevCount(2, cwd=self.srcdir)
@@ -1923,15 +2142,15 @@ class GitShallowTest(FetcherTest):
bb.utils.mkdirhier(smdir)
self.git_init(cwd=smdir)
# Make this look like it was cloned from a remote...
- self.git('config --add remote.origin.url "%s"' % smdir, cwd=smdir)
- self.git('config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
+ self.git(['config', '--add', 'remote.origin.url', '"%s"' % smdir], cwd=smdir)
+ self.git(['config', '--add', 'remote.origin.fetch', '"+refs/heads/*:refs/remotes/origin/*"'], cwd=smdir)
self.add_empty_file('asub', cwd=smdir)
self.add_empty_file('bsub', cwd=smdir)
- self.git('submodule init', cwd=self.srcdir)
- self.git('-c protocol.file.allow=always submodule add file://%s' % smdir, cwd=self.srcdir)
- self.git('submodule update', cwd=self.srcdir)
- self.git('commit -m submodule -a', cwd=self.srcdir)
+ self.git(['submodule', 'init'], cwd=self.srcdir)
+ self.git(['-c', 'protocol.file.allow=always', 'submodule', 'add', 'file://%s' % smdir], cwd=self.srcdir)
+ self.git(['submodule', 'update'], cwd=self.srcdir)
+ self.git(['commit', '-m', 'submodule', '-a'], cwd=self.srcdir)
uri = 'gitsm://%s;protocol=file;subdir=${S};branch=master' % self.srcdir
fetcher, ud = self.fetch_shallow(uri)
@@ -1953,17 +2172,17 @@ class GitShallowTest(FetcherTest):
bb.utils.mkdirhier(smdir)
self.git_init(cwd=smdir)
# Make this look like it was cloned from a remote...
- self.git('config --add remote.origin.url "%s"' % smdir, cwd=smdir)
- self.git('config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
+ self.git(['config', '--add', 'remote.origin.url', '"%s"' % smdir], cwd=smdir)
+ self.git(['config', '--add', 'remote.origin.fetch', '"+refs/heads/*:refs/remotes/origin/*"'], cwd=smdir)
self.add_empty_file('asub', cwd=smdir)
self.add_empty_file('bsub', cwd=smdir)
- self.git('submodule init', cwd=self.srcdir)
- self.git('-c protocol.file.allow=always submodule add file://%s' % smdir, cwd=self.srcdir)
- self.git('submodule update', cwd=self.srcdir)
- self.git('commit -m submodule -a', cwd=self.srcdir)
+ self.git(['submodule', 'init'], cwd=self.srcdir)
+ self.git(['-c', 'protocol.file.allow=always', 'submodule', 'add', 'file://%s' % smdir], cwd=self.srcdir)
+ self.git(['submodule', 'update'], cwd=self.srcdir)
+ self.git(['commit', '-m', 'submodule', '-a'], cwd=self.srcdir)
- uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
+ uri = 'gitsm://%s;protocol=file;subdir=${S};branch=master' % self.srcdir
# Fetch once to generate the shallow tarball
fetcher, ud = self.fetch(uri)
@@ -1991,83 +2210,30 @@ class GitShallowTest(FetcherTest):
def test_shallow_annex(self):
self.add_empty_file('a')
self.add_empty_file('b')
- self.git('annex init', cwd=self.srcdir)
+ self.git(['annex', 'init'], cwd=self.srcdir)
open(os.path.join(self.srcdir, 'c'), 'w').close()
- self.git('annex add c', cwd=self.srcdir)
- self.git('commit --author "Foo Bar <foo@bar>" -m annex-c -a', cwd=self.srcdir)
- bb.process.run('chmod u+w -R %s' % self.srcdir)
+ self.git(['annex', 'add', 'c'], cwd=self.srcdir)
+ self.git(['commit', '--author', '"Foo Bar <foo@bar>"', '-m', 'annex-c', '-a'], cwd=self.srcdir)
+ bb.process.run(['chmod', 'u+w', '-R', self.srcdir])
uri = 'gitannex://%s;protocol=file;subdir=${S};branch=master' % self.srcdir
fetcher, ud = self.fetch_shallow(uri)
self.assertRevCount(1)
- assert './.git/annex/' in bb.process.run('tar -tzf %s' % os.path.join(self.dldir, ud.mirrortarballs[0]))[0]
+ assert './.git/annex/' in bb.process.run(['tar', '-tzf', os.path.join(self.dldir, ud.mirrortarballs[0])])[0]
assert os.path.exists(os.path.join(self.gitdir, 'c'))
- def test_shallow_multi_one_uri(self):
- # Create initial git repo
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.git('checkout -b a_branch', cwd=self.srcdir)
- self.add_empty_file('c')
- self.add_empty_file('d')
- self.git('checkout master', cwd=self.srcdir)
- self.git('tag v0.0 a_branch', cwd=self.srcdir)
- self.add_empty_file('e')
- self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir)
- self.add_empty_file('f')
- self.assertRevCount(7, cwd=self.srcdir)
-
- uri = self.d.getVar('SRC_URI').split()[0]
- uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
-
- self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
- self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0')
- self.d.setVar('SRCREV_master', '${AUTOREV}')
- self.d.setVar('SRCREV_a_branch', '${AUTOREV}')
-
- self.fetch_shallow(uri)
-
- self.assertRevCount(5)
- self.assertRefs(['master', 'origin/master', 'origin/a_branch'])
-
- def test_shallow_multi_one_uri_depths(self):
- # Create initial git repo
- self.add_empty_file('a')
- self.add_empty_file('b')
- self.git('checkout -b a_branch', cwd=self.srcdir)
- self.add_empty_file('c')
- self.add_empty_file('d')
- self.git('checkout master', cwd=self.srcdir)
- self.add_empty_file('e')
- self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir)
- self.add_empty_file('f')
- self.assertRevCount(7, cwd=self.srcdir)
-
- uri = self.d.getVar('SRC_URI').split()[0]
- uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
-
- self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
- self.d.setVar('BB_GIT_SHALLOW_DEPTH_master', '3')
- self.d.setVar('BB_GIT_SHALLOW_DEPTH_a_branch', '1')
- self.d.setVar('SRCREV_master', '${AUTOREV}')
- self.d.setVar('SRCREV_a_branch', '${AUTOREV}')
-
- self.fetch_shallow(uri)
-
- self.assertRevCount(4, ['--all'])
- self.assertRefs(['master', 'origin/master', 'origin/a_branch'])
-
def test_shallow_clone_preferred_over_shallow(self):
self.add_empty_file('a')
self.add_empty_file('b')
# Fetch once to generate the shallow tarball
+ self.d.setVar('BB_GIT_SHALLOW', '0')
fetcher, ud = self.fetch()
- assert os.path.exists(os.path.join(self.dldir, ud.mirrortarballs[0]))
# Fetch and unpack with both the clonedir and shallow tarball available
bb.utils.remove(self.gitdir, recurse=True)
+ self.d.setVar('BB_GIT_SHALLOW', '1')
fetcher, ud = self.fetch_and_unpack()
# The unpacked tree should *not* be shallow
@@ -2097,12 +2263,35 @@ class GitShallowTest(FetcherTest):
self.fetch_and_unpack()
self.assertRevCount(1)
+ @skipIfNoNetwork()
+ def test_shallow_mirrors_with_multiple_same_urls(self):
+ url = "git://git.openembedded.org/bitbake;branch=master;protocol=https;rev=82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40"
+
+ d = self.d.createCopy()
+ d.delVar('AUTOREV')
+ d.delVar('SRCREV')
+
+ # prepare premirror
+ premirror = os.path.join(self.tempdir, "premirror")
+ os.mkdir(premirror)
+ d.setVar("DL_DIR", premirror)
+ fetcher = bb.fetch.Fetch([url], d)
+ fetcher.download()
+
+ # set PREMIRRORS
+ d.setVar('PREMIRRORS', 'git://.*/.* file://%s/' % premirror)
+
+ # set DL_DIR back and use the same url multiple times to fetch
+ d.setVar("DL_DIR", self.dldir)
+ fetcher2 = bb.fetch.Fetch([url, url], d)
+ fetcher2.download()
+
def test_shallow_invalid_depth(self):
self.add_empty_file('a')
self.add_empty_file('b')
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '-12')
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
self.fetch()
def test_shallow_invalid_depth_default(self):
@@ -2110,13 +2299,13 @@ class GitShallowTest(FetcherTest):
self.add_empty_file('b')
self.d.setVar('BB_GIT_SHALLOW_DEPTH_default', '-12')
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
self.fetch()
def test_shallow_extra_refs(self):
self.add_empty_file('a')
self.add_empty_file('b')
- self.git('branch a_branch', cwd=self.srcdir)
+ self.git(['branch', 'a_branch'], cwd=self.srcdir)
self.assertRefs(['master', 'a_branch'], cwd=self.srcdir)
self.assertRevCount(2, cwd=self.srcdir)
@@ -2129,8 +2318,8 @@ class GitShallowTest(FetcherTest):
def test_shallow_extra_refs_wildcard(self):
self.add_empty_file('a')
self.add_empty_file('b')
- self.git('branch a_branch', cwd=self.srcdir)
- self.git('tag v1.0', cwd=self.srcdir)
+ self.git(['branch', 'a_branch'], cwd=self.srcdir)
+ self.git(['tag', 'v1.0'], cwd=self.srcdir)
self.assertRefs(['master', 'a_branch', 'v1.0'], cwd=self.srcdir)
self.assertRevCount(2, cwd=self.srcdir)
@@ -2140,12 +2329,42 @@ class GitShallowTest(FetcherTest):
self.assertRefs(['master', 'origin/master', 'v1.0'])
self.assertRevCount(1)
+ def test_shallow_extra_refs_wildcard_shell_quoted(self):
+ self.add_empty_file('a')
+ marker = os.path.join(self.tempdir, 'ref-command-marker')
+ ref = 'refs/tags/poc;touch${IFS}%s' % marker
+ self.git(['update-ref', ref, 'HEAD'], cwd=self.srcdir)
+
+ self.d.setVar('BB_GIT_SHALLOW_EXTRA_REFS', 'refs/tags/*')
+ self.fetch_shallow()
+
+ self.assertFalse(os.path.exists(marker))
+ self.assertRefs(['master', 'origin/master', ref])
+
+ def test_shallow_extra_refs_wildcard_fetch_options(self):
+ self.add_empty_file('a')
+ marker = os.path.join(self.tempdir, 'ref-option-marker')
+ helper = os.path.join(self.tempdir, 'upload-pack-helper')
+ with open(helper, 'w') as f:
+ f.write('#!/bin/sh\n')
+ f.write('touch "%s"\n' % marker)
+ f.write('exec git-upload-pack "$@"\n')
+ os.chmod(helper, 0o755)
+ ref = 'refs/tags/--upload-pack=%s' % helper
+ self.git(['update-ref', ref, 'HEAD'], cwd=self.srcdir)
+
+ self.d.setVar('BB_GIT_SHALLOW_EXTRA_REFS', 'refs/tags/*')
+ self.fetch_shallow()
+
+ self.assertFalse(os.path.exists(marker))
+ self.assertRefs(['master', 'origin/master', ref])
+
def test_shallow_missing_extra_refs(self):
self.add_empty_file('a')
self.add_empty_file('b')
self.d.setVar('BB_GIT_SHALLOW_EXTRA_REFS', 'refs/heads/foo')
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
self.fetch()
def test_shallow_missing_extra_refs_wildcard(self):
@@ -2159,14 +2378,14 @@ class GitShallowTest(FetcherTest):
# Create initial git repo
self.add_empty_file('a')
self.add_empty_file('b')
- self.git('checkout -b a_branch', cwd=self.srcdir)
+ self.git(['checkout', '-b', 'a_branch'], cwd=self.srcdir)
self.add_empty_file('c')
self.add_empty_file('d')
- self.git('checkout master', cwd=self.srcdir)
- self.git('tag v0.0 a_branch', cwd=self.srcdir)
+ self.git(['checkout', 'master'], cwd=self.srcdir)
+ self.git(['tag', 'v0.0', 'a_branch'], cwd=self.srcdir)
self.add_empty_file('e')
- self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir)
- self.git('branch -d a_branch', cwd=self.srcdir)
+ self.git(['merge', '--no-ff', '--no-edit', 'a_branch'], cwd=self.srcdir)
+ self.git(['branch', '-d', 'a_branch'], cwd=self.srcdir)
self.add_empty_file('f')
self.assertRevCount(7, cwd=self.srcdir)
@@ -2175,7 +2394,7 @@ class GitShallowTest(FetcherTest):
self.fetch_shallow()
- self.assertRevCount(5)
+ self.assertRevCount(2)
def test_shallow_invalid_revs(self):
self.add_empty_file('a')
@@ -2184,17 +2403,20 @@ class GitShallowTest(FetcherTest):
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0')
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
self.fetch()
def test_shallow_fetch_missing_revs(self):
self.add_empty_file('a')
self.add_empty_file('b')
fetcher, ud = self.fetch(self.d.getVar('SRC_URI'))
- self.git('tag v0.0 master', cwd=self.srcdir)
+ self.git(['tag', 'v0.0', 'master'], cwd=self.srcdir)
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0')
- self.fetch_shallow()
+
+ with self.assertRaises(bb.fetch.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm:
+ self.fetch_shallow()
+ self.assertIn("fatal: no commits selected for shallow requests", cm.output[0])
def test_shallow_fetch_missing_revs_fails(self):
self.add_empty_file('a')
@@ -2203,15 +2425,42 @@ class GitShallowTest(FetcherTest):
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0')
- with self.assertRaises(bb.fetch2.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm:
+ with self.assertRaises(bb.fetch.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm:
self.fetch_shallow()
self.assertIn("Unable to find revision v0.0 even from upstream", cm.output[0])
+ @skipIfNoNetwork()
+ def test_git_shallow_fetch_premirrors(self):
+ url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
+
+ # Create a separate premirror directory within tempdir
+ premirror = os.path.join(self.tempdir, "premirror")
+ os.mkdir(premirror)
+
+ # Fetch a non-shallow clone into the premirror subdir
+ self.d.setVar('BB_GIT_SHALLOW', '0')
+ self.d.setVar("DL_DIR", premirror)
+ fetcher, ud = self.fetch(url)
+
+ # Fetch a shallow clone from the premirror subdir with unpacking
+ # using the original recipe URL and the premirror mapping
+ self.d.setVar('BB_GIT_SHALLOW', '1')
+ self.d.setVar("DL_DIR", self.dldir)
+ self.d.setVar('BB_FETCH_PREMIRRORONLY', '1')
+ self.d.setVar('BB_NO_NETWORK', '1')
+ self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '0')
+ self.d.setVar("PREMIRRORS", "git://.*/.* git://{0};protocol=file".format(premirror + "/git2/" + ud.host + ud.path.replace("/", ".")))
+ fetcher = self.fetch_and_unpack(url)
+
+ # Verify that the unpacked sources are shallow clones
+ self.assertRevCount(1)
+ assert os.path.exists(os.path.join(self.gitdir, '.git', 'shallow'))
+
@skipIfNoNetwork()
def test_bitbake(self):
- self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir)
- self.git('config core.bare true', cwd=self.srcdir)
- self.git('fetch', cwd=self.srcdir)
+ self.git(['remote', 'add', '--mirror=fetch', 'origin', 'https://git.openembedded.org/bitbake-contrib'], cwd=self.srcdir)
+ self.git(['config', 'core.bare', 'true'], cwd=self.srcdir)
+ self.git(['fetch'], cwd=self.srcdir)
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
# Note that the 1.10.0 tag is annotated, so this also tests
@@ -2221,11 +2470,11 @@ class GitShallowTest(FetcherTest):
self.fetch_shallow()
# Confirm that the history of 1.10.0 was removed
- orig_revs = len(self.git('rev-list master', cwd=self.srcdir).splitlines())
- revs = len(self.git('rev-list master').splitlines())
+ orig_revs = len(self.git(['rev-list', 'master'], cwd=self.srcdir).splitlines())
+ revs = len(self.git(['rev-list', 'master']).splitlines())
self.assertNotEqual(orig_revs, revs)
self.assertRefs(['master', 'origin/master'])
- self.assertRevCount(orig_revs - 1758)
+ self.assertRevCount(orig_revs - 1760)
def test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist(self):
self.add_empty_file('a')
@@ -2233,29 +2482,39 @@ class GitShallowTest(FetcherTest):
bb.utils.remove(self.gitdir, recurse=True)
bb.utils.remove(self.dldir, recurse=True)
- with self.assertRaises(bb.fetch2.UnpackError) as context:
+ with self.assertRaises(bb.fetch.UnpackError) as context:
fetcher.unpack(self.d.getVar('WORKDIR'))
self.assertIn("No up to date source found", context.exception.msg)
self.assertIn("clone directory not available or not up to date", context.exception.msg)
- @skipIfNoNetwork()
- def test_that_unpack_does_work_when_using_git_shallow_tarball_but_tarball_is_not_available(self):
- self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0')
- self.d.setVar('BB_GIT_SHALLOW', '1')
- self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
- fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests;branch=master;protocol=https"], self.d)
- fetcher.download()
+ def test_shallow_check_is_shallow(self):
+ self.add_empty_file('a')
+ self.add_empty_file('b')
- bb.utils.remove(self.dldir + "/*.tar.gz")
- fetcher.unpack(self.unpackdir)
+ # Fetch and unpack without the clonedir and *only* shallow tarball available
+ bb.utils.remove(self.gitdir, recurse=True)
+ fetcher, ud = self.fetch_and_unpack()
+
+ # The unpacked tree *should* be shallow
+ self.assertRevCount(1)
+ assert os.path.exists(os.path.join(self.gitdir, '.git', 'shallow'))
- dir = os.listdir(self.unpackdir + "/git/")
- self.assertIn("fstests.doap", dir)
+ def test_shallow_succeeds_with_tag_containing_slash(self):
+ self.add_empty_file('a')
+ self.add_empty_file('b')
+ self.git(['tag', 't1/t2/t3'], cwd=self.srcdir)
+ self.assertRevCount(2, cwd=self.srcdir)
+
+ srcrev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+ self.d.setVar('SRCREV', srcrev)
+ uri = self.d.getVar('SRC_URI').split()[0]
+ uri = '%s;tag=t1/t2/t3' % uri
+ self.fetch_shallow(uri)
+ self.assertRevCount(1)
class GitLfsTest(FetcherTest):
def skipIfNoGitLFS():
- import shutil
if not shutil.which('git-lfs'):
return unittest.skip('git-lfs not installed')
return lambda f: f
@@ -2272,26 +2531,32 @@ class GitLfsTest(FetcherTest):
self.d.delVar('MIRRORS')
self.d.setVar('SRCREV', '${AUTOREV}')
- self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
+ self.d.setVar('AUTOREV', '${@bb.fetch.get_autorev(d)}')
self.d.setVar("__BBSRCREV_SEEN", "1")
bb.utils.mkdirhier(self.srcdir)
self.git_init(cwd=self.srcdir)
self.commit_file('.gitattributes', '*.mp3 filter=lfs -text')
- def commit_file(self, filename, content):
- with open(os.path.join(self.srcdir, filename), "w") as f:
+ def commit(self, *, cwd=None):
+ cwd = cwd or self.srcdir
+ self.git(["commit", "-m", "Change"], cwd=cwd)
+ return self.git(["rev-parse", "HEAD"], cwd=cwd).strip()
+
+ def commit_file(self, filename, content, *, cwd=None):
+ cwd = cwd or self.srcdir
+
+ with open(os.path.join(cwd, filename), "w") as f:
f.write(content)
- self.git(["add", filename], cwd=self.srcdir)
- self.git(["commit", "-m", "Change"], cwd=self.srcdir)
- return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip()
+ self.git(["add", filename], cwd=cwd)
+ return self.commit(cwd=cwd)
def fetch(self, uri=None, download=True):
uris = self.d.getVar('SRC_URI').split()
uri = uris[0]
d = self.d
- fetcher = bb.fetch2.Fetch(uris, d)
+ fetcher = bb.fetch.Fetch(uris, d)
if download:
fetcher.download()
ud = fetcher.ud[uri]
@@ -2305,52 +2570,139 @@ class GitLfsTest(FetcherTest):
return unpacked_lfs_file
@skipIfNoGitLFS()
- def test_fetch_lfs_on_srcrev_change(self):
- """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested"""
+ def test_gitsm_lfs(self):
+ """Test that the gitsm fetcher caches objects stored via LFS"""
self.git(["lfs", "install", "--local"], cwd=self.srcdir)
- @contextlib.contextmanager
- def hide_upstream_repository():
- """Hide the upstream repository to make sure that git lfs cannot pull from it"""
- temp_name = self.srcdir + ".bak"
- os.rename(self.srcdir, temp_name)
- try:
- yield
- finally:
- os.rename(temp_name, self.srcdir)
-
def fetch_and_verify(revision, filename, content):
self.d.setVar('SRCREV', revision)
fetcher, ud = self.fetch()
- with hide_upstream_repository():
+ with hide_directory(submoduledir), hide_directory(self.srcdir):
workdir = self.d.getVar('WORKDIR')
fetcher.unpack(workdir)
with open(os.path.join(workdir, "git", filename)) as f:
self.assertEqual(f.read(), content)
- commit_1 = self.commit_file("a.mp3", "version 1")
- commit_2 = self.commit_file("a.mp3", "version 2")
-
- self.d.setVar('SRC_URI', "git://%s;protocol=file;lfs=1;branch=master" % self.srcdir)
-
- # Seed the local download folder by fetching the latest commit and verifying that the LFS contents are
- # available even when the upstream repository disappears.
- fetch_and_verify(commit_2, "a.mp3", "version 2")
- # Verify that even when an older revision is fetched, the needed LFS objects are fetched into the download
- # folder.
- fetch_and_verify(commit_1, "a.mp3", "version 1")
+ # Create the git repository that will later be used as a submodule
+ submoduledir = self.tempdir + "/submodule"
+ bb.utils.mkdirhier(submoduledir)
+ self.git_init(submoduledir)
+ self.git(["lfs", "install", "--local"], cwd=submoduledir)
+ self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir)
+
+ submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir)
+ _ = self.commit_file("a.mp3", "submodule version 2", cwd=submoduledir)
+
+ # Add the submodule to the repository at its current HEAD revision
+ self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"],
+ cwd=self.srcdir)
+ base_commit_1 = self.commit()
+
+ # Let the submodule point at a different revision
+ self.git(["checkout", submodule_commit_1], self.srcdir + "/submodule")
+ self.git(["add", "submodule"], cwd=self.srcdir)
+ base_commit_2 = self.commit()
+
+ # Add a LFS file to the repository
+ base_commit_3 = self.commit_file("a.mp3", "version 1")
+ # Update the added LFS file
+ base_commit_4 = self.commit_file("a.mp3", "version 2")
+
+ self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master" % self.srcdir)
+
+ # Verify that LFS objects referenced from submodules are fetched and checked out
+ fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 2")
+ # Verify that the repository inside the download cache of a submodile is extended with any
+ # additional LFS objects needed when checking out a different revision.
+ fetch_and_verify(base_commit_2, "submodule/a.mp3", "submodule version 1")
+ # Verify that LFS objects referenced from the base repository are fetched and checked out
+ fetch_and_verify(base_commit_3, "a.mp3", "version 1")
+ # Verify that the cached repository is extended with any additional LFS objects required
+ # when checking out a different revision.
+ fetch_and_verify(base_commit_4, "a.mp3", "version 2")
@skipIfNoGitLFS()
- @skipIfNoNetwork()
- def test_real_git_lfs_repo_succeeds_without_lfs_param(self):
- self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master")
- f = self.get_real_git_lfs_file()
- self.assertTrue(os.path.exists(f))
- self.assertEqual("c0baab607a97839c9a328b4310713307", bb.utils.md5_file(f))
+ def test_gitsm_lfs_disabled(self):
+ """Test that the gitsm fetcher does not use LFS when explicitly disabled"""
+ self.git(["lfs", "install", "--local"], cwd=self.srcdir)
- @skipIfNoGitLFS()
+ def fetch_and_verify(revision, filename, content):
+ self.d.setVar('SRCREV', revision)
+ fetcher, ud = self.fetch()
+
+ with hide_directory(submoduledir), hide_directory(self.srcdir):
+ workdir = self.d.getVar('WORKDIR')
+ fetcher.unpack(workdir)
+
+ with open(os.path.join(workdir, "git", filename)) as f:
+ # Assume that LFS did not perform smudging when the expected content is
+ # missing.
+ self.assertNotEqual(f.read(), content)
+
+ # Create the git repository that will later be used as a submodule
+ submoduledir = self.tempdir + "/submodule"
+ bb.utils.mkdirhier(submoduledir)
+ self.git_init(submoduledir)
+ self.git(["lfs", "install", "--local"], cwd=submoduledir)
+ self.commit_file('.gitattributes', '*.mp3 filter=lfs -text', cwd=submoduledir)
+
+ submodule_commit_1 = self.commit_file("a.mp3", "submodule version 1", cwd=submoduledir)
+
+ # Add the submodule to the repository at its current HEAD revision
+ self.git(["-c", "protocol.file.allow=always", "submodule", "add", submoduledir, "submodule"],
+ cwd=self.srcdir)
+ base_commit_1 = self.commit()
+
+ # Add a LFS file to the repository
+ base_commit_2 = self.commit_file("a.mp3", "version 1")
+
+ self.d.setVar('SRC_URI', "gitsm://%s;protocol=file;lfs=1;branch=master;lfs=0" % self.srcdir)
+
+ # Verify that LFS objects referenced from submodules are not fetched nor checked out
+ fetch_and_verify(base_commit_1, "submodule/a.mp3", "submodule version 1")
+ # Verify that the LFS objects referenced from the base repository are not fetched nor
+ # checked out
+ fetch_and_verify(base_commit_2, "a.mp3", "version 1")
+
+ @skipIfNoGitLFS()
+ def test_fetch_lfs_on_srcrev_change(self):
+ """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested"""
+ self.git(["lfs", "install", "--local"], cwd=self.srcdir)
+
+ def fetch_and_verify(revision, filename, content):
+ self.d.setVar('SRCREV', revision)
+ fetcher, ud = self.fetch()
+
+ with hide_directory(self.srcdir):
+ workdir = self.d.getVar('WORKDIR')
+ fetcher.unpack(workdir)
+
+ with open(os.path.join(workdir, "git", filename)) as f:
+ self.assertEqual(f.read(), content)
+
+ commit_1 = self.commit_file("a.mp3", "version 1")
+ commit_2 = self.commit_file("a.mp3", "version 2")
+
+ self.d.setVar('SRC_URI', "git://%s;protocol=file;lfs=1;branch=master" % self.srcdir)
+
+ # Seed the local download folder by fetching the latest commit and verifying that the LFS contents are
+ # available even when the upstream repository disappears.
+ fetch_and_verify(commit_2, "a.mp3", "version 2")
+ # Verify that even when an older revision is fetched, the needed LFS objects are fetched into the download
+ # folder.
+ fetch_and_verify(commit_1, "a.mp3", "version 1")
+
+ @skipIfNoGitLFS()
+ @skipIfNoNetwork()
+ def test_real_git_lfs_repo_succeeds_without_lfs_param(self):
+ self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master")
+ f = self.get_real_git_lfs_file()
+ self.assertTrue(os.path.exists(f))
+ self.assertEqual("c0baab607a97839c9a328b4310713307", bb.utils.md5_file(f))
+
+ @skipIfNoGitLFS()
@skipIfNoNetwork()
def test_real_git_lfs_repo_succeeds(self):
self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master;lfs=1")
@@ -2375,8 +2727,6 @@ class GitLfsTest(FetcherTest):
@skipIfNoGitLFS()
def test_lfs_enabled(self):
- import shutil
-
uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
self.d.setVar('SRC_URI', uri)
@@ -2387,8 +2737,6 @@ class GitLfsTest(FetcherTest):
@skipIfNoGitLFS()
def test_lfs_disabled(self):
- import shutil
-
uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir
self.d.setVar('SRC_URI', uri)
@@ -2397,58 +2745,91 @@ class GitLfsTest(FetcherTest):
fetcher, ud = self.fetch()
fetcher.unpack(self.d.getVar('WORKDIR'))
- def test_lfs_enabled_not_installed(self):
- import shutil
-
+ @skipIfNoGitLFS()
+ def test_lfs_enabled_not_installed_during_unpack(self):
uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
self.d.setVar('SRC_URI', uri)
# Careful: suppress initial attempt at downloading
fetcher, ud = self.fetch(uri=None, download=False)
- # Artificially assert that git-lfs is not installed, so
- # we can verify a failure to unpack in it's absence.
- old_find_git_lfs = ud.method._find_git_lfs
- try:
- # If git-lfs cannot be found, the unpack should throw an error
- with self.assertRaises(bb.fetch2.FetchError):
- fetcher.download()
- ud.method._find_git_lfs = lambda d: False
+ fetcher.download()
+ # If git-lfs cannot be found, the unpack should throw an error
+ with self.assertRaises(bb.fetch.FetchError):
+ with unittest.mock.patch("shutil.which", return_value=None):
shutil.rmtree(self.gitdir, ignore_errors=True)
fetcher.unpack(self.d.getVar('WORKDIR'))
- finally:
- ud.method._find_git_lfs = old_find_git_lfs
- def test_lfs_disabled_not_installed(self):
- import shutil
+ def test_lfs_enabled_not_installed(self):
+ uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+ # Careful: suppress initial attempt at downloading
+ fetcher, ud = self.fetch(uri=None, download=False)
+
+ # If git-lfs cannot be found, the download should throw an error
+ with unittest.mock.patch("shutil.which", return_value=None):
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher.download()
+
+ def test_lfs_disabled_not_installed(self):
uri = 'git://%s;protocol=file;lfs=0;branch=master' % self.srcdir
self.d.setVar('SRC_URI', uri)
# Careful: suppress initial attempt at downloading
fetcher, ud = self.fetch(uri=None, download=False)
- # Artificially assert that git-lfs is not installed, so
- # we can verify a failure to unpack in it's absence.
- old_find_git_lfs = ud.method._find_git_lfs
- try:
- # Even if git-lfs cannot be found, the unpack should be successful
+ # Even if git-lfs cannot be found, the download / unpack should be successful
+ with unittest.mock.patch("shutil.which", return_value=None):
fetcher.download()
- ud.method._find_git_lfs = lambda d: False
shutil.rmtree(self.gitdir, ignore_errors=True)
fetcher.unpack(self.d.getVar('WORKDIR'))
- finally:
- ud.method._find_git_lfs = old_find_git_lfs
+
+ def test_lfs_enabled_not_installed_but_not_needed(self):
+ srcdir = os.path.join(self.tempdir, "emptygit")
+ bb.utils.mkdirhier(srcdir)
+ self.git_init(srcdir)
+ self.commit_file("test", "test content", cwd=srcdir)
+
+ uri = 'git://%s;protocol=file;lfs=1;branch=master' % srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ # Careful: suppress initial attempt at downloading
+ fetcher, ud = self.fetch(uri=None, download=False)
+
+ # It shouldnt't matter that git-lfs cannot be found as the repository configuration does not
+ # specify any LFS filters.
+ with unittest.mock.patch("shutil.which", return_value=None):
+ fetcher.download()
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ @skipIfNoGitLFS()
+ def test_lfs_fetch_failure_raises(self):
+ self.commit_file("a.mp3", "version 1")
+
+ uri = 'git://%s;protocol=file;lfs=1;branch=master' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ # Simulate a fetch failure by removing the .git/lfs/objects directory
+ # from the source repository
+ shutil.rmtree(os.path.join(self.srcdir, ".git", "lfs", "objects"))
+
+ # Test than exception is raised when LFS objects could not be fetched
+ with self.assertRaises(bb.fetch.FetchError):
+ self.fetch()
class GitURLWithSpacesTest(FetcherTest):
test_git_urls = {
"git://tfs-example.org:22/tfs/example%20path/example.git;branch=master" : {
'url': 'git://tfs-example.org:22/tfs/example%20path/example.git;branch=master',
+ 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example.git',
'gitsrcname': 'tfs-example.org.22.tfs.example_path.example.git',
'path': '/tfs/example path/example.git'
},
"git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master" : {
'url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git;branch=master',
+ 'repo_url': 'git://tfs-example.org:22/tfs/example%20path/example%20repo.git',
'gitsrcname': 'tfs-example.org.22.tfs.example_path.example_repo.git',
'path': '/tfs/example path/example repo.git'
}
@@ -2471,6 +2852,51 @@ class GitURLWithSpacesTest(FetcherTest):
self.assertEqual(ud.lockfile, os.path.join(self.dldir, "git2", ref['gitsrcname'] + '.lock'))
self.assertEqual(ud.clonedir, os.path.join(self.dldir, "git2", ref['gitsrcname']))
self.assertEqual(ud.fullmirror, os.path.join(self.dldir, "git2_" + ref['gitsrcname'] + '.tar.gz'))
+ self.assertEqual(ud.method._get_repo_url(ud), ref['repo_url'])
+
+
+class FetchLocallyMissingTagFromRemote(FetcherTest):
+ def setUp(self):
+ FetcherTest.setUp(self)
+ self.gitdir = os.path.join(self.tempdir, 'git')
+ self.srcdir = os.path.join(self.tempdir, 'gitsource')
+
+ bb.utils.mkdirhier(self.srcdir)
+ self.git_init(cwd=self.srcdir)
+ self.d.setVar('WORKDIR', self.tempdir)
+ self.d.setVar('S', self.gitdir)
+
+ uri = 'git://%s;protocol=file;subdir=${S};branch=master' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ open(os.path.join(self.srcdir, 'dummyfile'), 'w').close()
+ self.git(['add', 'dummyfile'], self.srcdir)
+ self.git(['commit', '-m', 'dummymsg', 'dummyfile'], self.srcdir)
+
+ def _fetch_and_unpack(self, uri_to_fetch):
+ fetcher = bb.fetch.Fetch([uri_to_fetch], self.d)
+ fetcher.download()
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ def test_tag_present_in_remote_but_not_local(self):
+ # fetch a repo that has no tag in it
+ # then add a tag to this repo, and fetch it again, without
+ # changing SRC_REV, but by adding ';tag=tag1` to SRC_URI
+ # the new tag should be fetched and unpacked
+ srcrev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+ self.d.setVar('SRCREV', srcrev)
+ src_uri = self.d.getVar('SRC_URI')
+ self._fetch_and_unpack(src_uri)
+
+ self.git(['tag', '-m', '-a', 'tag1'], cwd=self.srcdir)
+
+ src_uri = '%s;tag=tag1' % self.d.getVar('SRC_URI').split()[0]
+ self.d.setVar('SRC_URI', src_uri)
+ self._fetch_and_unpack(src_uri)
+
+ output = self.git(['log', '--pretty=oneline', '-n', '1', 'refs/tags/tag1'], cwd=self.gitdir)
+ assert "fatal: ambiguous argument" not in output
+
class CrateTest(FetcherTest):
@skipIfNoNetwork()
@@ -2482,7 +2908,7 @@ class CrateTest(FetcherTest):
uris = self.d.getVar('SRC_URI').split()
d = self.d
- fetcher = bb.fetch2.Fetch(uris, self.d)
+ fetcher = bb.fetch.Fetch(uris, self.d)
ud = fetcher.ud[fetcher.urls[0]]
self.assertIn("name", ud.parm)
@@ -2508,7 +2934,7 @@ class CrateTest(FetcherTest):
uris = self.d.getVar('SRC_URI').split()
d = self.d
- fetcher = bb.fetch2.Fetch(uris, self.d)
+ fetcher = bb.fetch.Fetch(uris, self.d)
ud = fetcher.ud[fetcher.urls[0]]
self.assertIn("name", ud.parm)
@@ -2531,7 +2957,7 @@ class CrateTest(FetcherTest):
uris = self.d.getVar('SRC_URI').split()
d = self.d
- fetcher = bb.fetch2.Fetch(uris, self.d)
+ fetcher = bb.fetch.Fetch(uris, self.d)
ud = fetcher.ud[fetcher.urls[0]]
self.assertIn("name", ud.parm)
@@ -2555,7 +2981,7 @@ class CrateTest(FetcherTest):
uris = self.d.getVar('SRC_URI').split()
d = self.d
- fetcher = bb.fetch2.Fetch(uris, self.d)
+ fetcher = bb.fetch.Fetch(uris, self.d)
ud = fetcher.ud[fetcher.urls[0]]
self.assertIn("name", ud.parm)
@@ -2586,13 +3012,12 @@ class CrateTest(FetcherTest):
uris = self.d.getVar('SRC_URI').split()
- fetcher = bb.fetch2.Fetch(uris, self.d)
- with self.assertRaisesRegex(bb.fetch2.FetchError, "Fetcher failure for URL"):
+ fetcher = bb.fetch.Fetch(uris, self.d)
+ with self.assertRaisesRegex(bb.fetch.FetchError, "Fetcher failure for URL"):
fetcher.download()
class NPMTest(FetcherTest):
def skipIfNoNpm():
- import shutil
if not shutil.which('npm'):
return unittest.skip('npm not installed')
return lambda f: f
@@ -2600,8 +3025,10 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2614,9 +3041,11 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_bad_checksum(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2633,9 +3062,11 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_premirrors(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2655,7 +3086,7 @@ class NPMTest(FetcherTest):
# while the fetcher object exists, which it does when we rename the
# download directory to "mirror" above. Thus we need a new fetcher to go
# with the now empty download directory.
- fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2663,9 +3094,11 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_premirrors_with_specified_filename(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2685,8 +3118,10 @@ class NPMTest(FetcherTest):
@skipIfNoNetwork()
def test_npm_mirrors(self):
# Fetch once to get a tarball
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
+ fetcher = bb.fetch.Fetch(urls, self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -2710,27 +3145,35 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_destsuffix_downloadfilename(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0;destsuffix=foo/bar;downloadfilename=foo-bar.tgz'
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223'
+ ';destsuffix=foo/bar;downloadfilename=foo-bar.tgz']
+ fetcher = bb.fetch.Fetch(urls, self.d)
fetcher.download()
self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'foo-bar.tgz')))
fetcher.unpack(self.unpackdir)
unpackdir = os.path.join(self.unpackdir, 'foo', 'bar')
self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json')))
+ @skipIfNoNpm()
def test_npm_no_network_no_tarball(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
self.d.setVar('BB_NO_NETWORK', '1')
- fetcher = bb.fetch.Fetch([url], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_no_network_with_tarball(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher = bb.fetch.Fetch(urls, self.d)
fetcher.download()
# Disable network access
self.d.setVar('BB_NO_NETWORK', '1')
@@ -2743,66 +3186,122 @@ class NPMTest(FetcherTest):
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_registry_alternate(self):
- url = 'npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0'
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
+ fetcher = bb.fetch.Fetch(urls, self.d)
fetcher.download()
fetcher.unpack(self.unpackdir)
unpackdir = os.path.join(self.unpackdir, 'npm')
self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json')))
@skipIfNoNpm()
- @skipIfNoNetwork()
- def test_npm_version_latest(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest'
- fetcher = bb.fetch.Fetch([url], self.d)
- fetcher.download()
- fetcher.unpack(self.unpackdir)
- unpackdir = os.path.join(self.unpackdir, 'npm')
- self.assertTrue(os.path.exists(os.path.join(unpackdir, 'package.json')))
+ def test_npm_version_latest_rejected(self):
+ url = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest']
+ with self.assertRaises(bb.fetch.ParameterError):
+ bb.fetch.Fetch(url, self.d)
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_registry_invalid(self):
- url = 'npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
- fetcher = bb.fetch.Fetch([url], self.d)
- with self.assertRaises(bb.fetch2.FetchError):
+ urls = ['npm://registry.invalid.org;package=@savoirfairelinux/node-server-example;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ with self.assertRaises(bb.fetch.FetchError):
fetcher.download()
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_package_invalid(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0'
- fetcher = bb.fetch.Fetch([url], self.d)
- with self.assertRaises(bb.fetch2.FetchError):
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/invalid;version=1.0.0'
+ ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+ '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223']
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ with self.assertRaises(bb.fetch.FetchError):
fetcher.download()
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_version_invalid(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid'
- with self.assertRaises(bb.fetch2.ParameterError):
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=invalid']
+ with self.assertRaises(bb.fetch.ParameterError):
+ fetcher = bb.fetch.Fetch(urls, self.d)
+
+ @skipIfNoNpm()
+ @skipIfNoNetwork()
+ def test_npm_recipe_checksum(self):
+ """A sha512sum param in SRC_URI is forwarded to the proxy and verified."""
+ import subprocess
+ from bb.fetch.npm import npm_integrity
+ result = subprocess.run(
+ ['npm', 'view', '--json', '@savoirfairelinux/node-server-example@1.0.0'],
+ capture_output=True, text=True)
+ if result.returncode != 0:
+ self.skipTest('npm view failed: %s' % result.stderr.strip())
+ try:
+ view = json.loads(result.stdout)
+ except json.JSONDecodeError:
+ self.skipTest('npm view returned invalid JSON')
+ integrity = view.get('dist', {}).get('integrity')
+ if not integrity:
+ self.skipTest('npm view response missing dist.integrity')
+ checksum_name, hexsum = npm_integrity(integrity)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'
+ ';version=1.0.0;%s=%s' % (checksum_name, hexsum)]
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[fetcher.urls[0]]
+ fetcher.download()
+ self.assertTrue(os.path.exists(ud.localpath))
+
+ @skipIfNoNpm()
+ @skipIfNoNetwork()
+ def test_npm_bad_recipe_checksum_rejected(self):
+ """A wrong sha512sum param in SRC_URI causes the fetch to fail."""
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'
+ ';version=1.0.0;sha512sum=deadbeef00']
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher.download()
+
+ @skipIfNoNpm()
+ def test_npm_no_checksum_rejected(self):
+ """A missing checksum in SRC_URI is rejected regardless of BB_STRICT_CHECKSUM.
+
+ Unlike wget and other fetchers, npm must not fall back to an
+ unverified download: the registry cannot be trusted to supply its
+ own tamper detection, so the checksum requirement is not gated by
+ the usual opt-in strict-checksum setting.
+ """
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=1.0.0']
+ for strict in (None, '0', '1', 'ignore'):
+ if strict is None:
+ self.d.delVar('BB_STRICT_CHECKSUM')
+ else:
+ self.d.setVar('BB_STRICT_CHECKSUM', strict)
+ with self.assertRaises(bb.fetch.MissingParameterError):
+ bb.fetch.Fetch(urls, self.d)
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_registry_none(self):
- url = 'npm://;package=@savoirfairelinux/node-server-example;version=1.0.0'
- with self.assertRaises(bb.fetch2.MalformedUrl):
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://;package=@savoirfairelinux/node-server-example;version=1.0.0']
+ with self.assertRaises(bb.fetch.MalformedUrl):
+ fetcher = bb.fetch.Fetch(urls, self.d)
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_package_none(self):
- url = 'npm://registry.npmjs.org;version=1.0.0'
- with self.assertRaises(bb.fetch2.MissingParameterError):
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;version=1.0.0']
+ with self.assertRaises(bb.fetch.MissingParameterError):
+ fetcher = bb.fetch.Fetch(urls, self.d)
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npm_version_none(self):
- url = 'npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example'
- with self.assertRaises(bb.fetch2.MissingParameterError):
- fetcher = bb.fetch.Fetch([url], self.d)
+ urls = ['npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example']
+ with self.assertRaises(bb.fetch.MissingParameterError):
+ fetcher = bb.fetch.Fetch(urls, self.d)
def create_shrinkwrap_file(self, data):
import json
@@ -2811,32 +3310,30 @@ class NPMTest(FetcherTest):
bb.utils.mkdirhier(datadir)
with open(swfile, 'w') as f:
json.dump(data, f)
- # Also configure the S directory
- self.sdir = os.path.join(self.unpackdir, 'S')
- self.d.setVar('S', self.sdir)
return swfile
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw(self):
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=',
'dependencies': {
- 'content-type': {
- 'version': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz',
- 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==',
- 'dependencies': {
- 'cookie': {
- 'version': 'git+https://github.com/jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09',
- 'from': 'git+https://github.com/jshttp/cookie.git'
- }
- }
- }
+ 'content-type': "1.0.4"
+ }
+ },
+ 'node_modules/array-flatten/node_modules/content-type': {
+ 'version': '1.0.4',
+ 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz',
+ 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==',
+ 'dependencies': {
+ 'cookie': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09'
}
+ },
+ 'node_modules/array-flatten/node_modules/content-type/node_modules/cookie': {
+ 'resolved': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09'
}
}
})
@@ -2844,63 +3341,36 @@ class NPMTest(FetcherTest):
fetcher.download()
self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz')))
self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz')))
- self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git')))
+ self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'git.yoctoproject.org.bbfetchtests-jshttp-cookie')))
fetcher.unpack(self.unpackdir)
- self.assertTrue(os.path.exists(os.path.join(self.sdir, 'npm-shrinkwrap.json')))
- self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'package.json')))
- self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'package.json')))
- self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'node_modules', 'cookie', 'package.json')))
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm-shrinkwrap.json')))
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json')))
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'package.json')))
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'node_modules', 'content-type', 'node_modules', 'cookie', 'package.json')))
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_git(self):
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'cookie': {
- 'version': 'github:jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09',
- 'from': 'github:jshttp/cookie.git'
- }
- }
- })
- fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
- fetcher.download()
- self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git')))
-
- swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'cookie': {
- 'version': 'jshttp/cookie.git#aec1177c7da67e3b3273df96cf476824dbc9ae09',
- 'from': 'jshttp/cookie.git'
- }
- }
- })
- fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
- fetcher.download()
- self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'github.com.jshttp.cookie.git')))
-
- swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'nodejs': {
- 'version': 'gitlab:gitlab-examples/nodejs.git#892a1f16725e56cc3a2cb0d677be42935c8fc262',
- 'from': 'gitlab:gitlab-examples/nodejs'
+ 'packages': {
+ 'node_modules/cookie': {
+ 'resolved': 'git+https://git.yoctoproject.org/bbfetchtests-jshttp-cookie#aec1177c7da67e3b3273df96cf476824dbc9ae09'
}
}
})
fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
fetcher.download()
- self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'gitlab.com.gitlab-examples.nodejs.git')))
+ self.assertTrue(os.path.exists(os.path.join(self.dldir, 'git2', 'git.yoctoproject.org.bbfetchtests-jshttp-cookie')))
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_dev(self):
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
},
- 'content-type': {
+ 'node_modules/content-type': {
'version': '1.0.4',
'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz',
'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==',
@@ -2919,12 +3389,11 @@ class NPMTest(FetcherTest):
self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz')))
self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz')))
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_destsuffix(self):
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -2938,8 +3407,8 @@ class NPMTest(FetcherTest):
def test_npmsw_no_network_no_tarball(self):
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -2948,21 +3417,21 @@ class NPMTest(FetcherTest):
})
self.d.setVar('BB_NO_NETWORK', '1')
fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
@skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_no_network_with_tarball(self):
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d)
+ fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1;sha1sum=9a5f699051b1e7073328f2a008968b64ea2955d2'], self.d)
fetcher.download()
# Disable network access
self.d.setVar('BB_NO_NETWORK', '1')
# Fetch again
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -2972,15 +3441,14 @@ class NPMTest(FetcherTest):
fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
fetcher.download()
fetcher.unpack(self.unpackdir)
- self.assertTrue(os.path.exists(os.path.join(self.sdir, 'node_modules', 'array-flatten', 'package.json')))
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'node_modules', 'array-flatten', 'package.json')))
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_npm_reusability(self):
# Fetch once with npmsw
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -2992,18 +3460,17 @@ class NPMTest(FetcherTest):
# Disable network access
self.d.setVar('BB_NO_NETWORK', '1')
# Fetch again with npm
- fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d)
+ fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1;sha1sum=9a5f699051b1e7073328f2a008968b64ea2955d2'], self.d)
fetcher.download()
fetcher.unpack(self.unpackdir)
self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'npm', 'package.json')))
- @skipIfNoNpm()
@skipIfNoNetwork()
def test_npmsw_bad_checksum(self):
# Try to fetch with bad checksum
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-gfNEp2hqgLTFKT6P3AsBYMgsBqg='
@@ -3011,12 +3478,12 @@ class NPMTest(FetcherTest):
}
})
fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
- with self.assertRaises(bb.fetch2.FetchError):
+ with self.assertRaises(bb.fetch.FetchError):
fetcher.download()
# Fetch correctly to get a tarball
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -3041,7 +3508,7 @@ class NPMTest(FetcherTest):
@skipIfNoNetwork()
def test_npmsw_premirrors(self):
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d)
+ fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1;sha1sum=9a5f699051b1e7073328f2a008968b64ea2955d2'], self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -3054,8 +3521,8 @@ class NPMTest(FetcherTest):
# Fetch again
self.assertFalse(os.path.exists(ud.localpath))
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -3070,7 +3537,7 @@ class NPMTest(FetcherTest):
@skipIfNoNetwork()
def test_npmsw_mirrors(self):
# Fetch once to get a tarball
- fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1'], self.d)
+ fetcher = bb.fetch.Fetch(['npm://registry.npmjs.org;package=array-flatten;version=1.1.1;sha1sum=9a5f699051b1e7073328f2a008968b64ea2955d2'], self.d)
ud = fetcher.ud[fetcher.urls[0]]
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
@@ -3082,8 +3549,8 @@ class NPMTest(FetcherTest):
# Fetch again with invalid url
self.assertFalse(os.path.exists(ud.localpath))
swfile = self.create_shrinkwrap_file({
- 'dependencies': {
- 'array-flatten': {
+ 'packages': {
+ 'node_modules/array-flatten': {
'version': '1.1.1',
'resolved': 'https://invalid',
'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
@@ -3094,6 +3561,28 @@ class NPMTest(FetcherTest):
fetcher.download()
self.assertTrue(os.path.exists(ud.localpath))
+ @skipIfNoNetwork()
+ def test_npmsw_bundled(self):
+ swfile = self.create_shrinkwrap_file({
+ 'packages': {
+ 'node_modules/array-flatten': {
+ 'version': '1.1.1',
+ 'resolved': 'https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz',
+ 'integrity': 'sha1-ml9pkFGx5wczKPKgCJaLZOopVdI='
+ },
+ 'node_modules/content-type': {
+ 'version': '1.0.4',
+ 'resolved': 'https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz',
+ 'integrity': 'sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==',
+ 'inBundle': True
+ }
+ }
+ })
+ fetcher = bb.fetch.Fetch(['npmsw://' + swfile], self.d)
+ fetcher.download()
+ self.assertTrue(os.path.exists(os.path.join(self.dldir, 'npm2', 'array-flatten-1.1.1.tgz')))
+ self.assertFalse(os.path.exists(os.path.join(self.dldir, 'npm2', 'content-type-1.0.4.tgz')))
+
class GitSharedTest(FetcherTest):
def setUp(self):
super(GitSharedTest, self).setUp()
@@ -3121,6 +3610,72 @@ class GitSharedTest(FetcherTest):
alt = os.path.join(self.unpackdir, 'git/.git/objects/info/alternates')
self.assertFalse(os.path.exists(alt))
+class GitTagVerificationTests(FetcherTest):
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match(self):
+ # Test a url with rev= and tag= set works
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ def test_annotated_tag_rev_match(self):
+ # Test a url with rev= and tag= set works
+ # rev is the annotated tag revision in this case
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=fa30183549bd09f33fd4eebf56771ca5393526a6;tag=2.12.0"], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match2(self):
+ # Test a url with SRCREV and tag= set works
+ self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757')
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;tag=2.12.0"], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match3(self):
+ # Test a url with SRCREV, rev= and tag= set works
+ self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757')
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match4(self):
+ # Test a url with SRCREV and rev= mismatching errors
+ self.d.setVar('SRCREV', 'bade540fc31a1c26839efd2c7785a751ce24ebfb')
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match5(self):
+ # Test a url with SRCREV, rev= and tag= set works when using shallow clones
+ self.d.setVar('BB_GIT_SHALLOW', '1')
+ self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757')
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.12.0"], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match6(self):
+ # Test a url with SRCREV, rev= and a mismatched tag= when using shallow clones
+ self.d.setVar('BB_GIT_SHALLOW', '1')
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.8.0"], self.d)
+ fetcher.download()
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher.unpack(self.unpackdir)
+
+ @skipIfNoNetwork()
+ def test_tag_rev_match7(self):
+ # Test a url with SRCREV, rev= and a mismatched tag=
+ self.d.setVar('SRCREV', '5b4e20377eea8d428edf1aeb2187c18f82ca6757')
+ fetcher = bb.fetch.Fetch(["git://git.openembedded.org/bitbake;branch=2.12;protocol=https;rev=5b4e20377eea8d428edf1aeb2187c18f82ca6757;tag=2.8.0"], self.d)
+ fetcher.download()
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher.unpack(self.unpackdir)
+
class FetchPremirroronlyLocalTest(FetcherTest):
@@ -3144,25 +3699,25 @@ class FetchPremirroronlyLocalTest(FetcherTest):
self.git_init(cwd=self.gitdir)
for i in range(0):
self.git_new_commit()
- bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd = self.gitdir)
+ bb.process.run(['tar', '-czvf', os.path.join(self.mirrordir, self.mirrorname), '.'], cwd=self.gitdir)
def git_new_commit(self):
import random
os.unlink(os.path.join(self.mirrordir, self.mirrorname))
- branch = self.git("branch --show-current", self.gitdir).split()
+ branch = self.git(["branch", "--show-current"], self.gitdir).split()
with open(os.path.join(self.gitdir, self.testfilename), "w") as testfile:
testfile.write("File {} from branch {}; Useless random data {}".format(self.testfilename, branch, random.random()))
- self.git("add {}".format(self.testfilename), self.gitdir)
- self.git("commit -a -m \"This random commit {} in branch {}. I'm useless.\"".format(random.random(), branch), self.gitdir)
- bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd = self.gitdir)
- return self.git("rev-parse HEAD", self.gitdir).strip()
+ self.git(['add', self.testfilename], self.gitdir)
+ self.git(['commit', '-a', '-m', "\"This random commit {} in branch {}. I'm useless.\"".format(random.random(), branch)], self.gitdir)
+ bb.process.run(['tar', '-czvf', os.path.join(self.mirrordir, self.mirrorname), '.'], cwd=self.gitdir)
+ return self.git(["rev-parse", "HEAD"], self.gitdir).strip()
def git_new_branch(self, name):
self.git_new_commit()
- head = self.git("rev-parse HEAD", self.gitdir).strip()
- self.git("checkout -b {}".format(name), self.gitdir)
+ head = self.git(["rev-parse", "HEAD"], self.gitdir).strip()
+ self.git(["checkout", "-b", name], self.gitdir)
newrev = self.git_new_commit()
- self.git("checkout {}".format(head), self.gitdir)
+ self.git(["checkout", head], self.gitdir)
return newrev
def test_mirror_multiple_fetches(self):
@@ -3187,7 +3742,7 @@ class FetchPremirroronlyLocalTest(FetcherTest):
self.make_git_repo()
self.d.setVar("SRCREV", "0"*40)
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
def test_mirror_commit_exists(self):
@@ -3200,59 +3755,7 @@ class FetchPremirroronlyLocalTest(FetcherTest):
def test_mirror_tarball_nonexistent(self):
self.d.setVar("SRCREV", "0"*40)
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
- fetcher.download()
-
- def test_mirror_tarball_multiple_branches(self):
- """
- test if PREMIRRORS can handle multiple name/branches correctly
- both branches have required revisions
- """
- self.make_git_repo()
- branch1rev = self.git_new_branch("testbranch1")
- branch2rev = self.git_new_branch("testbranch2")
- self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1,testbranch2;protocol=https;name=branch1,branch2"
- self.d.setVar("SRCREV_branch1", branch1rev)
- self.d.setVar("SRCREV_branch2", branch2rev)
- fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- self.assertTrue(os.path.exists(self.mirrorfile), "Mirror file doesn't exist")
- fetcher.download()
- fetcher.unpack(os.path.join(self.tempdir, "unpacked"))
- unpacked = os.path.join(self.tempdir, "unpacked", "git", self.testfilename)
- self.assertTrue(os.path.exists(unpacked), "Repo has not been unpackaged properly!")
- with open(unpacked, 'r') as f:
- content = f.read()
- ## We expect to see testbranch1 in the file, not master, not testbranch2
- self.assertTrue(content.find("testbranch1") != -1, "Wrong branch has been checked out!")
-
- def test_mirror_tarball_multiple_branches_nobranch(self):
- """
- test if PREMIRRORS can handle multiple name/branches correctly
- Unbalanced name/branches raises ParameterError
- """
- self.make_git_repo()
- branch1rev = self.git_new_branch("testbranch1")
- branch2rev = self.git_new_branch("testbranch2")
- self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1;protocol=https;name=branch1,branch2"
- self.d.setVar("SRCREV_branch1", branch1rev)
- self.d.setVar("SRCREV_branch2", branch2rev)
- with self.assertRaises(bb.fetch2.ParameterError):
- fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
-
- def test_mirror_tarball_multiple_branches_norev(self):
- """
- test if PREMIRRORS can handle multiple name/branches correctly
- one of the branches specifies non existing SRCREV
- """
- self.make_git_repo()
- branch1rev = self.git_new_branch("testbranch1")
- branch2rev = self.git_new_branch("testbranch2")
- self.recipe_url = "git://git.fake.repo/bitbake;branch=testbranch1,testbranch2;protocol=https;name=branch1,branch2"
- self.d.setVar("SRCREV_branch1", branch1rev)
- self.d.setVar("SRCREV_branch2", "0"*40)
- fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- self.assertTrue(os.path.exists(self.mirrorfile), "Mirror file doesn't exist")
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
@@ -3265,24 +3768,25 @@ class FetchPremirroronlyNetworkTest(FetcherTest):
self.reponame = "fstests"
self.clonedir = os.path.join(self.tempdir, "git")
self.gitdir = os.path.join(self.tempdir, "git", "{}.git".format(self.reponame))
- self.recipe_url = "git://git.yoctoproject.org/fstests;protocol=https"
+ self.giturl = "https://git.yoctoproject.org/fstests"
+ self.recipe_url = "git://git.yoctoproject.org/fstests;protocol=https;branch=master"
self.d.setVar("BB_FETCH_PREMIRRORONLY", "1")
self.d.setVar("BB_NO_NETWORK", "0")
self.d.setVar("PREMIRRORS", self.recipe_url + " " + "file://{}".format(self.mirrordir) + " \n")
def make_git_repo(self):
- import shutil
self.mirrorname = "git2_git.yoctoproject.org.fstests.tar.gz"
os.makedirs(self.clonedir)
- self.git("clone --bare --shallow-since=\"01.01.2013\" {}".format(self.recipe_url), self.clonedir)
- bb.process.run('tar -czvf {} .'.format(os.path.join(self.mirrordir, self.mirrorname)), cwd = self.gitdir)
+ self.git(["clone", "--bare", self.giturl], self.clonedir)
+ self.git(["update-ref", "HEAD", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f"], self.gitdir)
+ bb.process.run(['tar', '-czvf', os.path.join(self.mirrordir, self.mirrorname), '.'], cwd=self.gitdir)
shutil.rmtree(self.clonedir)
@skipIfNoNetwork()
def test_mirror_tarball_updated(self):
self.make_git_repo()
## Upstream commit is in the mirror
- self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec")
+ self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f")
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
fetcher.download()
@@ -3290,9 +3794,9 @@ class FetchPremirroronlyNetworkTest(FetcherTest):
def test_mirror_tarball_outdated(self):
self.make_git_repo()
## Upstream commit not in the mirror
- self.d.setVar("SRCREV", "15413486df1f5a5b5af699b6f3ba5f0984e52a9f")
+ self.d.setVar("SRCREV", "49d65d53c2bf558ae6e9185af0f3af7b79d255ec")
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- with self.assertRaises(bb.fetch2.NetworkAccess):
+ with self.assertRaises(bb.fetch.NetworkAccess):
fetcher.download()
class FetchPremirroronlyMercurialTest(FetcherTest):
@@ -3300,7 +3804,6 @@ class FetchPremirroronlyMercurialTest(FetcherTest):
the test covers also basic hg:// clone (see fetch_and_create_tarball
"""
def skipIfNoHg():
- import shutil
if not shutil.which('hg'):
return unittest.skip('Mercurial not installed')
return lambda f: f
@@ -3347,7 +3850,7 @@ class FetchPremirroronlyBrokenTarball(FetcherTest):
os.mkdir(self.mirrordir)
self.reponame = "bitbake"
self.gitdir = os.path.join(self.tempdir, "git", self.reponame)
- self.recipe_url = "git://git.fake.repo/bitbake;protocol=https"
+ self.recipe_url = "git://git.fake.repo/bitbake;protocol=https;branch=master"
self.d.setVar("BB_FETCH_PREMIRRORONLY", "1")
self.d.setVar("BB_NO_NETWORK", "1")
self.d.setVar("PREMIRRORS", self.recipe_url + " " + "file://{}".format(self.mirrordir) + " \n")
@@ -3356,10 +3859,782 @@ class FetchPremirroronlyBrokenTarball(FetcherTest):
targz.write("This is not tar.gz file!")
def test_mirror_broken_download(self):
- import sys
self.d.setVar("SRCREV", "0"*40)
fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
- with self.assertRaises(bb.fetch2.FetchError), self.assertLogs() as logs:
+ with self.assertRaises(bb.fetch.FetchError), self.assertLogs() as logs:
fetcher.download()
output = "".join(logs.output)
self.assertFalse(" not a git repository (or any parent up to mount point /)" in output)
+
+class GoModTest(FetcherTest):
+
+ @skipIfNoNetwork()
+ def test_gomod_url(self):
+ urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;'
+ 'sha256sum=9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.zip')
+ self.assertEqual(ud.parm['downloadfilename'], 'github.com.Azure.azure-sdk-for-go.sdk.storage.azblob@v1.0.0.zip')
+ self.assertEqual(ud.parm['name'], 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')),
+ '7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873')
+
+ @skipIfNoNetwork()
+ def test_gomod_url_go_mod_only(self):
+ urls = ['gomod://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;mod=1;'
+ 'sha256sum=7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.url, 'https://proxy.golang.org/github.com/%21azure/azure-sdk-for-go/sdk/storage/azblob/%40v/v1.0.0.mod')
+ self.assertEqual(ud.parm['downloadfilename'], 'github.com.Azure.azure-sdk-for-go.sdk.storage.azblob@v1.0.0.mod')
+ self.assertEqual(ud.parm['name'], 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')))
+
+ @skipIfNoNetwork()
+ def test_gomod_url_sha256sum_varflag(self):
+ urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0']
+ self.d.setVarFlag('SRC_URI', 'gopkg.in/ini.v1@v1.67.0.sha256sum', 'bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6')
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip')
+ self.assertEqual(ud.parm['downloadfilename'], 'gopkg.in.ini.v1@v1.67.0.zip')
+ self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')),
+ '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1')
+
+ @skipIfNoNetwork()
+ def test_gomod_url_no_go_mod_in_module(self):
+ urls = ['gomod://gopkg.in/ini.v1;version=v1.67.0;'
+ 'sha256sum=bd845dfc762a87a56e5a32a07770dc83e86976db7705d7f89c5dbafdc60b06c6']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.url, 'https://proxy.golang.org/gopkg.in/ini.v1/%40v/v1.67.0.zip')
+ self.assertEqual(ud.parm['downloadfilename'], 'gopkg.in.ini.v1@v1.67.0.zip')
+ self.assertEqual(ud.parm['name'], 'gopkg.in/ini.v1@v1.67.0')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')),
+ '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1')
+
+ @skipIfNoNetwork()
+ def test_gomod_url_host_only(self):
+ urls = ['gomod://go.opencensus.io;version=v0.24.0;'
+ 'sha256sum=203a767d7f8e7c1ebe5588220ad168d1e15b14ae70a636de7ca9a4a88a7e0d0c']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.url, 'https://proxy.golang.org/go.opencensus.io/%40v/v0.24.0.zip')
+ self.assertEqual(ud.parm['downloadfilename'], 'go.opencensus.io@v0.24.0.zip')
+ self.assertEqual(ud.parm['name'], 'go.opencensus.io@v0.24.0')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')),
+ '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96')
+
+class GoModGitTest(FetcherTest):
+
+ @skipIfNoNetwork()
+ def test_gomodgit_url_repo(self):
+ urls = ['gomodgit://golang.org/x/net;version=v0.9.0;'
+ 'repo=go.googlesource.com/net;'
+ 'srcrev=694cff8668bac64e0864b552bffc280cd27f21b1']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.host, 'go.googlesource.com')
+ self.assertEqual(ud.path, '/net')
+ self.assertEqual(ud.name, 'golang.org/x/net@v0.9.0')
+ self.assertEqual(self.d.getVar('SRCREV_golang.org/x/net@v0.9.0'), '694cff8668bac64e0864b552bffc280cd27f21b1')
+
+ fetcher.download()
+ self.assertTrue(os.path.exists(ud.localpath))
+
+ fetcher.unpack(self.unpackdir)
+ vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs')
+ self.assertTrue(os.path.exists(os.path.join(vcsdir, 'ed42bd05533fd84ae290a5d33ebd3695a0a2b06131beebd5450825bee8603aca')))
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'golang.org/x/net/@v/v0.9.0.mod')),
+ 'c5d6851ede50ec1c001afb763040194b68961bf06997e2605e8bf06dcd2aeb2e')
+
+ @skipIfNoNetwork()
+ def test_gomodgit_url_subdir(self):
+ urls = ['gomodgit://github.com/Azure/azure-sdk-for-go/sdk/storage/azblob;version=v1.0.0;'
+ 'repo=github.com/Azure/azure-sdk-for-go;subdir=sdk/storage/azblob;'
+ 'srcrev=ec928e0ed34db682b3f783d3739d1c538142e0c3']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.host, 'github.com')
+ self.assertEqual(ud.path, '/Azure/azure-sdk-for-go')
+ self.assertEqual(ud.parm['subpath'], 'sdk/storage/azblob')
+ self.assertEqual(ud.name, 'github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0')
+ self.assertEqual(self.d.getVar('SRCREV_github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@v1.0.0'), 'ec928e0ed34db682b3f783d3739d1c538142e0c3')
+
+ fetcher.download()
+ self.assertTrue(os.path.exists(ud.localpath))
+
+ fetcher.unpack(self.unpackdir)
+ vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs')
+ self.assertTrue(os.path.exists(os.path.join(vcsdir, 'd31d6145676ed3066ce573a8198f326dea5be45a43b3d8f41ce7787fd71d66b3')))
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'github.com/!azure/azure-sdk-for-go/sdk/storage/azblob/@v/v1.0.0.mod')),
+ '7873b8544842329b4f385a3aa6cf82cc2bc8defb41a04fa5291c35fd5900e873')
+
+ @skipIfNoNetwork()
+ def test_gomodgit_url_srcrev_var(self):
+ urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0']
+ self.d.setVar('SRCREV_gopkg.in/ini.v1@v1.67.0', 'b2f570e5b5b844226bbefe6fb521d891f529a951')
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.host, 'gopkg.in')
+ self.assertEqual(ud.path, '/ini.v1')
+ self.assertEqual(ud.name, 'gopkg.in/ini.v1@v1.67.0')
+ self.assertEqual(ud.parm['srcrev'], 'b2f570e5b5b844226bbefe6fb521d891f529a951')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs')
+ self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3')))
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')),
+ '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1')
+
+ @skipIfNoNetwork()
+ def test_gomodgit_url_no_go_mod_in_module(self):
+ urls = ['gomodgit://gopkg.in/ini.v1;version=v1.67.0;'
+ 'srcrev=b2f570e5b5b844226bbefe6fb521d891f529a951']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.host, 'gopkg.in')
+ self.assertEqual(ud.path, '/ini.v1')
+ self.assertEqual(ud.name, 'gopkg.in/ini.v1@v1.67.0')
+ self.assertEqual(self.d.getVar('SRCREV_gopkg.in/ini.v1@v1.67.0'), 'b2f570e5b5b844226bbefe6fb521d891f529a951')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs')
+ self.assertTrue(os.path.exists(os.path.join(vcsdir, 'b7879a4be9ba8598851b8278b14c4f71a8316be64913298d1639cce6bde59bc3')))
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'gopkg.in/ini.v1/@v/v1.67.0.mod')),
+ '13aedd85db8e555104108e0e613bb7e4d1242af7f27c15423dd9ab63b60b72a1')
+
+ @skipIfNoNetwork()
+ def test_gomodgit_url_host_only(self):
+ urls = ['gomodgit://go.opencensus.io;version=v0.24.0;'
+ 'repo=git.yoctoproject.org/bbfetchtests-census-instrumentation-opencensus-go;'
+ 'srcrev=b1a01ee95db0e690d91d7193d037447816fae4c5']
+
+ fetcher = bb.fetch.Fetch(urls, self.d)
+ ud = fetcher.ud[urls[0]]
+ self.assertEqual(ud.host, 'git.yoctoproject.org')
+ self.assertEqual(ud.path, '/bbfetchtests-census-instrumentation-opencensus-go')
+ self.assertEqual(ud.name, 'go.opencensus.io@v0.24.0')
+ self.assertEqual(self.d.getVar('SRCREV_go.opencensus.io@v0.24.0'), 'b1a01ee95db0e690d91d7193d037447816fae4c5')
+
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+ vcsdir = os.path.join(self.unpackdir, 'pkg/mod/cache/vcs')
+ self.assertTrue(os.path.exists(os.path.join(vcsdir, '24e4a8be150f9e7343b72e40000115a175fb2f19d015542f54abb6ccdbd52f14')))
+ downloaddir = os.path.join(self.unpackdir, 'pkg/mod/cache/download')
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.zip')))
+ self.assertTrue(os.path.exists(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')))
+ self.assertEqual(bb.utils.sha256_file(os.path.join(downloaddir, 'go.opencensus.io/@v/v0.24.0.mod')),
+ '0dc9ccc660ad21cebaffd548f2cc6efa27891c68b4fbc1f8a3893b00f1acec96')
+
+
+class GitUnpackUpdateTest(FetcherTest):
+ """Test the unpack_update functionality for git fetcher.
+
+ Intended workflow
+ 1. First-time setup:
+ 1. download() - clones the upstream repo into DL_DIR/git2/... (clonedir).
+ 2. unpack() - clones from clonedir into the workspace (S/workdir) and
+ registers a 'dldir' git remote pointing at
+ file://DL_DIR/git2/... for later offline use.
+
+ 2. Subsequent updates (what unpack_update is designed for):
+ 1. The user works in the unpacked source tree.
+ 2. Upstream advances - SRCREV changes in the recipe.
+ 3. download() - fetches the new revision into the local clonedir.
+ 4. unpack_update() - instead of wiping the workspace and re-cloning:
+ * fetches the new revision from the local 'dldir' remote
+ * rebases the user's local commits on top of the new SRCREV
+ * raises LocalModificationsError if uncommitted changes block the
+ update, RebaseError if local commits cannot be rebased, or a
+ plain UnpackError for other failures (shallow clone, stale dldir);
+ in all cases the caller (e.g. bitbake-setup) can fall back to
+ backup + re-clone.
+
+ Key design constraints:
+ * unpack_update() never deletes existing data (unlike unpack()).
+ * Only staged/modified tracked files block the update; untracked files and
+ committed local work are handled gracefully.
+ * The 'dldir' remote is intentionally visible to users outside the
+ fetcher (e.g. for manual 'git log dldir/master').
+ * Currently only git is supported.
+ """
+
+ def setUp(self):
+ """Set up a local bare git source repository with two commits on 'master'.
+
+ self.initial_rev - the first commit (testfile.txt: 'initial content')
+ self.updated_rev - the second commit (testfile.txt: 'updated content')
+
+ SRCREV is initialised to self.initial_rev so individual tests can
+ advance it to self.updated_rev (or create further commits) as needed.
+ """
+ FetcherTest.setUp(self)
+
+ self.gitdir = os.path.join(self.tempdir, 'gitrepo')
+ self.srcdir = os.path.join(self.tempdir, 'gitsource')
+
+ self.d.setVar('WORKDIR', self.tempdir)
+ self.d.setVar('S', self.gitdir)
+ self.d.delVar('PREMIRRORS')
+ self.d.delVar('MIRRORS')
+
+ # Create a source git repository
+ bb.utils.mkdirhier(self.srcdir)
+ self.git_init(cwd=self.srcdir)
+
+ # Create initial commit
+ with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f:
+ f.write('initial content\n')
+ self.git(['add', 'testfile.txt'], cwd=self.srcdir)
+ self.git(['commit', '-m', 'Initial commit'], cwd=self.srcdir)
+ self.initial_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+
+ # Create a second commit
+ with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f:
+ f.write('updated content\n')
+ self.git(['add', 'testfile.txt'], cwd=self.srcdir)
+ self.git(['commit', '-m', 'Update commit'], cwd=self.srcdir)
+ self.updated_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+
+ self.d.setVar('SRCREV', self.initial_rev)
+ self.d.setVar('SRC_URI', 'git://%s;branch=master;protocol=file' % self.srcdir)
+
+ def test_unpack_update_full_clone(self):
+ """Test that unpack_update updates an existing checkout in place for a full clone.
+
+ Steps:
+ 1. Fetch and unpack at self.initial_rev - verify 'initial content'.
+ 2. Advance SRCREV to self.updated_rev and re-download.
+ 3. Call unpack_update() instead of unpack() - the existing checkout
+ must be updated via 'git fetch dldir' + 'git rebase' without
+ re-cloning the directory.
+ 4. Verify testfile.txt now contains 'updated content'.
+ """
+ # First fetch at initial revision
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ # Verify initial state
+ unpack_path = os.path.join(self.unpackdir, 'git')
+ self.assertTrue(os.path.exists(os.path.join(unpack_path, 'testfile.txt')))
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f:
+ self.assertEqual(f.read(), 'initial content\n')
+
+ # Update to new revision
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ # Use unpack_update
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ # Verify updated state
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f:
+ self.assertEqual(f.read(), 'updated content\n')
+
+ def test_unpack_update_dldir_remote_setup(self):
+ """Test that unpack() adds a 'dldir' git remote pointing at ud.clonedir.
+
+ The 'dldir' remote is used by subsequent unpack_update() calls to fetch
+ new commits from the local download cache (${DL_DIR}/git2/…) without
+ requiring network access. After a normal unpack the remote must exist
+ and its URL must be 'file://<ud.clonedir>'.
+ """
+ # First fetch
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Check that dldir remote exists
+ remotes = self.git(['remote'], cwd=unpack_path).strip().split('\n')
+ self.assertIn('dldir', remotes)
+
+ # Verify it points to the clonedir
+ dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip()
+ self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir))
+
+ def test_unpack_update_ff_with_local_changes(self):
+ """Test that unpack_update rebases local commits fast forward.
+
+ Full workflow:
+ 1. Fetch + unpack at initial_rev - verify 'dldir' remote is created
+ pointing at ud.clonedir.
+ 2. Add a local commit touching localfile.txt.
+ 3. Advance SRCREV to updated_rev and call download() - verify that
+ ud.clonedir (the dldir bare clone) now contains updated_rev.
+ 4. Call unpack_update() - it fetches updated_rev from dldir into the
+ working tree and rebases the local commit on top.
+ 5. Verify the final commit graph: HEAD's parent is updated_rev, and
+ both testfile.txt ('updated content') and localfile.txt ('local
+ change') are present.
+
+ Note: git rebase operates the same way regardless of whether HEAD is
+ detached or on a named branch (e.g. 'master' or a local feature branch),
+ so this test covers those scenarios implicitly.
+ """
+ # Step 1 - fetch + unpack at initial_rev
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # The normal unpack must have set up the 'dldir' remote pointing at
+ # ud.clonedir so that subsequent unpack_update() calls work offline.
+ dldir_url = self.git(['remote', 'get-url', 'dldir'], cwd=unpack_path).strip()
+ self.assertEqual(dldir_url, 'file://{}'.format(ud.clonedir))
+
+ # Step 2 - add a local commit that touches a new file
+ with open(os.path.join(unpack_path, 'localfile.txt'), 'w') as f:
+ f.write('local change\n')
+ self.git(['add', 'localfile.txt'], cwd=unpack_path)
+ self.git(['commit', '-m', 'Local commit'], cwd=unpack_path)
+ local_commit = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip()
+
+ # Step 3 - advance SRCREV and download; clonedir must now contain
+ # updated_rev so that unpack_update can fetch it without network access.
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ ud = fetcher.ud[uri]
+ clonedir_refs = self.git(['rev-parse', self.updated_rev], cwd=ud.clonedir).strip()
+ self.assertEqual(clonedir_refs, self.updated_rev,
+ "clonedir must contain updated_rev after download()")
+
+ # Step 4 - unpack_update fetches from dldir and rebases
+ git_fetcher = ud.method
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ # Step 5 - verify the commit graph and working tree content
+ # HEAD is the rebased local commit; its parent must be updated_rev
+ head_rev = self.git(['rev-parse', 'HEAD'], cwd=unpack_path).strip()
+ parent_rev = self.git(['rev-parse', 'HEAD^'], cwd=unpack_path).strip()
+ self.assertNotEqual(head_rev, local_commit,
+ "local commit should have a new SHA after rebase")
+ self.assertEqual(parent_rev, self.updated_rev,
+ "HEAD's parent must be updated_rev after fast-forward rebase")
+
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f:
+ self.assertEqual(f.read(), 'updated content\n')
+ with open(os.path.join(unpack_path, 'localfile.txt'), 'r') as f:
+ self.assertEqual(f.read(), 'local change\n')
+
+ def test_unpack_update_already_at_target_revision(self):
+ """Test that unpack_update is a no-op when the checkout is already at SRCREV.
+
+ Calling unpack_update() without advancing SRCREV must succeed and leave
+ the working tree unchanged. No rebase should be attempted because the
+ checkout already points at ud.revision.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+ with open(os.path.join(unpack_path, 'testfile.txt')) as f:
+ self.assertEqual(f.read(), 'initial content\n')
+
+ # Call unpack_update with SRCREV still at initial_rev - no upstream change
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+ result = git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+ self.assertTrue(result)
+
+ # Content must be unchanged
+ with open(os.path.join(unpack_path, 'testfile.txt')) as f:
+ self.assertEqual(f.read(), 'initial content\n')
+
+ def test_unpack_update_with_untracked_file(self):
+ """Test that unpack_update succeeds when the checkout has an untracked file.
+
+ The status check uses '--untracked-files=no', so untracked files are not
+ detected and do not trigger the fallback path. git rebase also leaves
+ untracked files untouched, so both the upstream update and the untracked
+ file must be present after the call.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Create an untracked file (not staged, not committed)
+ untracked = os.path.join(unpack_path, 'untracked.txt')
+ with open(untracked, 'w') as f:
+ f.write('untracked content\n')
+
+ # Update to new upstream revision
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ # --untracked-files=no means the status check passes; rebase preserves the file
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'r') as f:
+ self.assertEqual(f.read(), 'updated content\n')
+
+ # Untracked file must survive the rebase
+ self.assertTrue(os.path.exists(untracked))
+ with open(untracked, 'r') as f:
+ self.assertEqual(f.read(), 'untracked content\n')
+
+ def test_unpack_update_with_staged_changes(self):
+ """Test that unpack_update fails when the checkout has staged (but not committed) changes.
+
+ The rebase is run with --no-autostash so git refuses to rebase over a
+ dirty index. The caller (bitbake-setup) is expected to catch the
+ resulting LocalModificationsError and fall back to backup + re-fetch.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Stage a new file without committing it
+ staged = os.path.join(unpack_path, 'staged.txt')
+ with open(staged, 'w') as f:
+ f.write('staged content\n')
+ self.git(['add', 'staged.txt'], cwd=unpack_path)
+
+ # Update to new upstream revision
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ # Should fail - git rebase refuses to run with a dirty index
+ with self.assertRaises(bb.fetch.LocalModificationsError):
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ def test_unpack_update_with_modified_tracked_file(self):
+ """Test that unpack_update fails when a tracked file has unstaged modifications.
+
+ 'git status --untracked-files=no --porcelain' reports unstaged modifications
+ to tracked files (output line ' M filename'), which must block the update so
+ the caller can fall back to backup + re-fetch rather than silently discarding
+ work in progress.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Modify a tracked file without staging or committing
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f:
+ f.write('locally modified content\n')
+
+ # Update to new upstream revision
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ # Should fail - unstaged modification to tracked file is detected by
+ # 'git status --untracked-files=no --porcelain'
+ with self.assertRaises(bb.fetch.LocalModificationsError):
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ def test_unpack_update_conflict_raises_rebase_error(self):
+ """Test that unpack_update raises RebaseError on a rebase conflict.
+
+ When a local commit modifies the same lines as an incoming upstream commit,
+ git rebase cannot resolve the conflict automatically. unpack_update must
+ abort the failed rebase and raise RebaseError so the caller can fall back
+ to a backup + re-fetch.
+ """
+ # Fetch and unpack at the initial revision
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Make a local commit that edits the same lines as the upcoming upstream commit
+ with open(os.path.join(unpack_path, 'testfile.txt'), 'w') as f:
+ f.write('conflicting local content\n')
+ self.git(['add', 'testfile.txt'], cwd=unpack_path)
+ self.git(['commit', '-m', 'Local conflicting commit'], cwd=unpack_path)
+
+ # Add a third upstream commit that also edits testfile.txt differently
+ with open(os.path.join(self.srcdir, 'testfile.txt'), 'w') as f:
+ f.write('conflicting upstream content\n')
+ self.git(['add', 'testfile.txt'], cwd=self.srcdir)
+ self.git(['commit', '-m', 'Upstream conflicting commit'], cwd=self.srcdir)
+ conflict_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+
+ # Update SRCREV to the new upstream commit
+ self.d.setVar('SRCREV', conflict_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ # unpack_update must fail and clean up (rebase --abort) rather than
+ # leaving the repo in a mid-rebase state
+ with self.assertRaises(bb.fetch.RebaseError):
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ # Verify the repo is not left in a conflicted / mid-rebase state
+ rebase_merge = os.path.join(unpack_path, '.git', 'rebase-merge')
+ rebase_apply = os.path.join(unpack_path, '.git', 'rebase-apply')
+ self.assertFalse(os.path.exists(rebase_merge),
+ "rebase-merge dir should not exist after failed unpack_update")
+ self.assertFalse(os.path.exists(rebase_apply),
+ "rebase-apply dir should not exist after failed unpack_update")
+
+ def test_unpack_update_untracked_file_overwritten_by_upstream(self):
+ """Test that unpack_update raises RebaseError when an untracked file would be
+ overwritten by an incoming upstream commit.
+
+ We skip untracked files in the pre-check (git rebase doesn't touch harmless
+ untracked files), but git itself refuses to rebase when an untracked file would
+ be overwritten by the incoming changes. The resulting FetchError must be caught
+ and re-raised as RebaseError without leaving the repo in a mid-rebase state.
+
+ Two sub-cases are covered:
+ - top-level untracked file clashing with an incoming upstream file
+ - untracked file inside a subdirectory (xxx/somefile) clashing with an
+ upstream commit that adds the same path
+ """
+ def _run_case(upstream_path, local_rel_path, commit_msg):
+ """
+ Add upstream_path to self.srcdir, create local_rel_path as an
+ untracked file in the checkout, then assert that unpack_update
+ raises RebaseError and leaves no mid-rebase state, and that the
+ local file is untouched.
+ """
+ # Fresh fetch + unpack at the current SRCREV
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Upstream adds the file (potentially inside a subdirectory)
+ full_upstream = os.path.join(self.srcdir, upstream_path)
+ os.makedirs(os.path.dirname(full_upstream), exist_ok=True)
+ with open(full_upstream, 'w') as f:
+ f.write('upstream content\n')
+ self.git(['add', upstream_path], cwd=self.srcdir)
+ self.git(['commit', '-m', commit_msg], cwd=self.srcdir)
+ new_rev = self.git(['rev-parse', 'HEAD'], cwd=self.srcdir).strip()
+
+ # Create the clashing untracked file in the checkout
+ full_local = os.path.join(unpack_path, local_rel_path)
+ os.makedirs(os.path.dirname(full_local), exist_ok=True)
+ with open(full_local, 'w') as f:
+ f.write('local untracked content\n')
+
+ self.d.setVar('SRCREV', new_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ # git rebase refuses because the untracked file would be overwritten
+ with self.assertRaises(bb.fetch.RebaseError):
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ # Repo must not be left in a mid-rebase state
+ self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-merge')))
+ self.assertFalse(os.path.exists(os.path.join(unpack_path, '.git', 'rebase-apply')))
+
+ # The local untracked file must be untouched
+ self.assertTrue(os.path.exists(full_local))
+ with open(full_local) as f:
+ self.assertEqual(f.read(), 'local untracked content\n')
+
+ # Reset unpackdir for the next sub-case
+ import shutil as _shutil
+ _shutil.rmtree(self.unpackdir)
+ os.makedirs(self.unpackdir)
+
+ # Sub-case 1: top-level file clash
+ _run_case('newfile.txt', 'newfile.txt',
+ 'Upstream adds newfile.txt')
+
+ # Sub-case 2: file inside a subdirectory (xxx/somefile)
+ _run_case('xxx/somefile.txt', 'xxx/somefile.txt',
+ 'Upstream adds xxx/somefile.txt')
+
+ def test_unpack_update_shallow_clone_fails(self):
+ """Test that unpack_update raises UnpackError for shallow-tarball checkouts.
+
+ Shallow clones lack full history, which makes an in-place rebase impossible
+ without network access. After fetching with BB_GIT_SHALLOW=1 the clonedir
+ is deleted so that unpack() is forced to use the shallow tarball.
+ A subsequent call to unpack_update() must raise UnpackError and the message
+ must mention 'shallow clone' so callers can distinguish this case.
+ """
+ self.d.setVar('BB_GIT_SHALLOW', '1')
+ self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
+
+ # First fetch at initial revision
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ # Remove clonedir to force use of shallow tarball
+ clonedir = os.path.join(self.dldir, 'git2')
+ if os.path.exists(clonedir):
+ shutil.rmtree(clonedir)
+
+ fetcher.unpack(self.unpackdir)
+
+ # Update to new revision
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ # unpack_update should fail for shallow clones
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+ git_fetcher = ud.method
+
+ with self.assertRaises(bb.fetch.UnpackError) as context:
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ self.assertIn("shallow clone", str(context.exception).lower())
+
+ def test_unpack_update_stale_dldir_remote(self):
+ """Test that unpack_update raises UnpackError when the dldir remote URL is stale.
+
+ If the clonedir has been removed after the initial unpack (e.g. DL_DIR was
+ cleaned) the 'dldir' remote URL no longer resolves. The fetch inside
+ update_mode will fail with a FetchError which must be re-raised as
+ UnpackError so the caller can fall back to a full re-fetch.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+
+ # Advance SRCREV to trigger update_mode
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ uri = self.d.getVar('SRC_URI')
+ ud = fetcher.ud[uri]
+
+ # Delete the clonedir and corrupt the dldir remote URL so that
+ # 'git fetch dldir' fails, simulating a missing or relocated DL_DIR.
+ shutil.rmtree(ud.clonedir)
+ self.git(['remote', 'set-url', 'dldir', 'file://' + ud.clonedir],
+ cwd=unpack_path)
+
+ git_fetcher = ud.method
+ with self.assertRaises(bb.fetch.UnpackError):
+ git_fetcher.unpack_update(ud, self.unpackdir, self.d)
+
+ def test_fetch_unpack_update_toplevel_api(self):
+ """Test that the top-level Fetch.unpack_update() dispatches to Git.unpack_update().
+
+ Callers such as bitbake-setup use fetcher.unpack_update(root) rather than
+ calling the method on the Git fetcher directly. Verify that the public API
+ works end-to-end: fetch at initial_rev, unpack, advance to updated_rev,
+ fetch again, then call fetcher.unpack_update(root) and confirm the content
+ is updated.
+ """
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+ fetcher.unpack(self.unpackdir)
+
+ unpack_path = os.path.join(self.unpackdir, 'git')
+ with open(os.path.join(unpack_path, 'testfile.txt')) as f:
+ self.assertEqual(f.read(), 'initial content\n')
+
+ self.d.setVar('SRCREV', self.updated_rev)
+ fetcher = bb.fetch.Fetch([self.d.getVar('SRC_URI')], self.d)
+ fetcher.download()
+
+ # Use the public Fetch.unpack_update() rather than the method directly
+ fetcher.unpack_update(self.unpackdir)
+
+ with open(os.path.join(unpack_path, 'testfile.txt')) as f:
+ self.assertEqual(f.read(), 'updated content\n')
diff --git a/bitbake/lib/bb/tests/fetch_import.py b/bitbake/lib/bb/tests/fetch_import.py
new file mode 100644
index 00000000..17217842
--- /dev/null
+++ b/bitbake/lib/bb/tests/fetch_import.py
@@ -0,0 +1,86 @@
+#
+# Copyright (C) 2026 Agilent Technologies, Inc.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import subprocess
+import sys
+import unittest
+
+
+class FetchImportTests(unittest.TestCase):
+ def run_python(self, code):
+ env = os.environ.copy()
+ libdir = os.path.realpath(os.path.join(os.path.dirname(__file__), "..", ".."))
+ env["PYTHONPATH"] = os.pathsep.join(filter(None, (libdir, env.get("PYTHONPATH"))))
+ result = subprocess.run(
+ [sys.executable, "-c", code],
+ env=env,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ )
+ self.assertEqual(result.returncode, 0, result.stderr)
+
+ def test_fetch_is_lazy(self):
+ self.run_python("""
+import sys
+import bb
+
+assert "bb.fetch" not in sys.modules
+assert "bb.fetch2" not in sys.modules
+""")
+
+ def test_fetch_import(self):
+ self.run_python("""
+import sys
+import bb.fetch
+from bb.fetch import Fetch
+
+assert sys.modules["bb.fetch"] is bb.fetch
+assert Fetch is bb.fetch.Fetch
+""")
+
+ def test_fetch2_import_compatibility(self):
+ self.run_python("""
+import sys
+import warnings
+
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ import bb.fetch2
+ from bb.fetch2 import Fetch
+
+assert len(caught) == 1
+assert caught[0].category is DeprecationWarning
+assert str(caught[0].message) == "bb.fetch2 is deprecated; use bb.fetch instead"
+assert caught[0].filename == "<string>"
+
+assert bb.fetch2 is bb.fetch
+assert sys.modules["bb.fetch2"] is bb.fetch
+assert Fetch is bb.fetch.Fetch
+""")
+
+ def test_fetch2_attribute_access(self):
+ self.run_python("""
+import sys
+import warnings
+import bb
+
+assert "bb.fetch" not in sys.modules
+assert "bb.fetch2" not in sys.modules
+
+with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ fetch2 = bb.fetch2
+
+assert len(caught) == 1
+assert caught[0].category is DeprecationWarning
+assert str(caught[0].message) == "bb.fetch2 is deprecated; use bb.fetch instead"
+
+assert fetch2 is bb.fetch
+assert bb.fetch2 is bb.fetch
+assert sys.modules["bb.fetch2"] is bb.fetch
+""")
\ No newline at end of file
diff --git a/bitbake/lib/bb/tests/filter.py b/bitbake/lib/bb/tests/filter.py
new file mode 100644
index 00000000..245df7b2
--- /dev/null
+++ b/bitbake/lib/bb/tests/filter.py
@@ -0,0 +1,88 @@
+#
+# Copyright (C) 2025 Garmin Ltd. or its subsidiaries
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import unittest
+import bb.filter
+
+
+class BuiltinFilterTest(unittest.TestCase):
+ def test_disallowed_builtins(self):
+ with self.assertRaises(NameError):
+ val = bb.filter.apply_filters("1", ["open('foo.txt', 'rb')"])
+
+ def test_prefix(self):
+ val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')"])
+ self.assertEqual(val, "a1 a2 a3")
+
+ val = bb.filter.apply_filters("", ["prefix(val, 'a')"])
+ self.assertEqual(val, "")
+
+ def test_suffix(self):
+ val = bb.filter.apply_filters("1 2 3", ["suffix(val, 'b')"])
+ self.assertEqual(val, "1b 2b 3b")
+
+ val = bb.filter.apply_filters("", ["suffix(val, 'b')"])
+ self.assertEqual(val, "")
+
+ def test_sort(self):
+ val = bb.filter.apply_filters("z y x", ["sort(val)"])
+ self.assertEqual(val, "x y z")
+
+ val = bb.filter.apply_filters("", ["sort(val)"])
+ self.assertEqual(val, "")
+
+ def test_identity(self):
+ val = bb.filter.apply_filters("1 2 3", ["val"])
+ self.assertEqual(val, "1 2 3")
+
+ val = bb.filter.apply_filters("123", ["val"])
+ self.assertEqual(val, "123")
+
+ def test_empty(self):
+ val = bb.filter.apply_filters("1 2 3", ["", "prefix(val, 'a')", ""])
+ self.assertEqual(val, "a1 a2 a3")
+
+ def test_nested(self):
+ val = bb.filter.apply_filters("1 2 3", ["prefix(prefix(val, 'a'), 'b')"])
+ self.assertEqual(val, "ba1 ba2 ba3")
+
+ val = bb.filter.apply_filters("1 2 3", ["prefix(prefix(val, 'b'), 'a')"])
+ self.assertEqual(val, "ab1 ab2 ab3")
+
+ def test_filter_order(self):
+ val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')", "prefix(val, 'b')"])
+ self.assertEqual(val, "ba1 ba2 ba3")
+
+ val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'b')", "prefix(val, 'a')"])
+ self.assertEqual(val, "ab1 ab2 ab3")
+
+ val = bb.filter.apply_filters("1 2 3", ["prefix(val, 'a')", "suffix(val, 'b')"])
+ self.assertEqual(val, "a1b a2b a3b")
+
+ val = bb.filter.apply_filters("1 2 3", ["suffix(val, 'b')", "prefix(val, 'a')"])
+ self.assertEqual(val, "a1b a2b a3b")
+
+ def test_remove(self):
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, ['2'])"])
+ self.assertEqual(val, "1 3")
+
+ val = bb.filter.apply_filters("1,2,3", ["remove(val, ['2'], ',')"])
+ self.assertEqual(val, "1,3")
+
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, ['4'])"])
+ self.assertEqual(val, "1 2 3")
+
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, ['1', '2'])"])
+ self.assertEqual(val, "3")
+
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, '2')"])
+ self.assertEqual(val, "1 3")
+
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, '4')"])
+ self.assertEqual(val, "1 2 3")
+
+ val = bb.filter.apply_filters("1 2 3", ["remove(val, '1 2')"])
+ self.assertEqual(val, "3")
diff --git a/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass b/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass
new file mode 100644
index 00000000..db8898e1
--- /dev/null
+++ b/bitbake/lib/bb/tests/parse-tests/classes/base.bbclass
@@ -0,0 +1,5 @@
+# At least one task is required for bitbake to parse
+do_fetch() {
+ :
+}
+addtask do_fetch
diff --git a/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass b/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass
new file mode 100644
index 00000000..4682dc6c
--- /dev/null
+++ b/bitbake/lib/bb/tests/parse-tests/classes/recipe-file-class.bbclass
@@ -0,0 +1,2 @@
+BBCLASS_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}"
+BBCLASS_FILE := "${@os.path.basename(d.getVar('FILE'))}"
diff --git a/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf b/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf
new file mode 100644
index 00000000..e0362506
--- /dev/null
+++ b/bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf
@@ -0,0 +1,15 @@
+CACHE = "${TOPDIR}/cache"
+THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
+COREBASE := "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
+EXTRA_BBFILES ?= ""
+BBFILES = "${COREBASE}/recipes/*.bb ${COREBASE}/recipes/*.bbappend ${EXTRA_BBFILES}"
+PROVIDES = "${PN}"
+PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
+PF = "${BB_CURRENT_MC}:${PN}"
+export PATH
+TMPDIR ??= "${TOPDIR}"
+STAMP = "${TMPDIR}/stamps/${PN}"
+T = "${TMPDIR}/workdir/${PN}/temp"
+BB_NUMBER_THREADS = "4"
+
+BB_BASEHASH_IGNORE_VARS = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR SLOWTASKS SSTATEVALID FILE BB_CURRENTTASK"
diff --git a/bitbake/lib/bb/tests/parse.py b/bitbake/lib/bb/tests/parse.py
index 72d1962e..6ac2137e 100644
--- a/bitbake/lib/bb/tests/parse.py
+++ b/bitbake/lib/bb/tests/parse.py
@@ -11,6 +11,8 @@ import tempfile
import logging
import bb
import os
+import subprocess
+import textwrap
logger = logging.getLogger('BitBake.TestParse')
@@ -39,7 +41,6 @@ C = "3"
os.chdir(self.origdir)
def parsehelper(self, content, suffix = ".bb"):
-
f = tempfile.NamedTemporaryFile(suffix = suffix)
f.write(bytes(content, "utf-8"))
f.flush()
@@ -47,17 +48,17 @@ C = "3"
return f
def test_parse_simple(self):
- f = self.parsehelper(self.testfile)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.testfile) as f:
+ d = bb.parse.handle(f.name, self.d)['']
self.assertEqual(d.getVar("A"), "1")
self.assertEqual(d.getVar("B"), "2")
self.assertEqual(d.getVar("C"), "3")
def test_parse_incomplete_function(self):
testfileB = self.testfile.replace("}", "")
- f = self.parsehelper(testfileB)
- with self.assertRaises(bb.parse.ParseError):
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(testfileB) as f:
+ with self.assertRaises(bb.parse.ParseError):
+ d = bb.parse.handle(f.name, self.d)['']
unsettest = """
A = "1"
@@ -69,12 +70,65 @@ unset B[flag]
"""
def test_parse_unset(self):
- f = self.parsehelper(self.unsettest)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.unsettest) as f:
+ d = bb.parse.handle(f.name, self.d)['']
self.assertEqual(d.getVar("A"), None)
self.assertEqual(d.getVarFlag("A","flag"), None)
self.assertEqual(d.getVar("B"), "2")
+ defaulttest = """
+A = "set value"
+A ??= "default value"
+
+A[flag_set_vs_question] = "set flag"
+A[flag_set_vs_question] ?= "question flag"
+
+A[flag_set_vs_default] = "set flag"
+A[flag_set_vs_default] ??= "default flag"
+
+A[flag_question] ?= "question flag"
+
+A[flag_default] ??= "default flag"
+
+A[flag_question_vs_default] ?= "question flag"
+A[flag_question_vs_default] ??= "default flag"
+
+A[flag_default_vs_question] ??= "default flag"
+A[flag_default_vs_question] ?= "question flag"
+
+A[flag_set_question_default] = "set flag"
+A[flag_set_question_default] ?= "question flag"
+A[flag_set_question_default] ??= "default flag"
+
+A[flag_set_default_question] = "set flag"
+A[flag_set_default_question] ??= "default flag"
+A[flag_set_default_question] ?= "question flag"
+
+A[flag_set_twice] = "set flag first"
+A[flag_set_twice] = "set flag second"
+
+A[flag_question_twice] ?= "question flag first"
+A[flag_question_twice] ?= "question flag second"
+
+A[flag_default_twice] ??= "default flag first"
+A[flag_default_twice] ??= "default flag second"
+"""
+ def test_parse_defaulttest(self):
+ with self.parsehelper(self.defaulttest) as f:
+ d = bb.parse.handle(f.name, self.d)['']
+ self.assertEqual(d.getVar("A"), "set value")
+ self.assertEqual(d.getVarFlag("A","flag_set_vs_question"), "set flag")
+ self.assertEqual(d.getVarFlag("A","flag_set_vs_default"), "set flag")
+ self.assertEqual(d.getVarFlag("A","flag_question"), "question flag")
+ self.assertEqual(d.getVarFlag("A","flag_default"), "default flag")
+ self.assertEqual(d.getVarFlag("A","flag_question_vs_default"), "question flag")
+ self.assertEqual(d.getVarFlag("A","flag_default_vs_question"), "question flag")
+ self.assertEqual(d.getVarFlag("A","flag_set_question_default"), "set flag")
+ self.assertEqual(d.getVarFlag("A","flag_set_default_question"), "set flag")
+ self.assertEqual(d.getVarFlag("A","flag_set_twice"), "set flag second")
+ self.assertEqual(d.getVarFlag("A","flag_question_twice"), "question flag first")
+ self.assertEqual(d.getVarFlag("A","flag_default_twice"), "default flag second")
+
exporttest = """
A = "a"
export B = "b"
@@ -83,8 +137,8 @@ exportD = "d"
"""
def test_parse_exports(self):
- f = self.parsehelper(self.exporttest)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.exporttest) as f:
+ d = bb.parse.handle(f.name, self.d)['']
self.assertEqual(d.getVar("A"), "a")
self.assertIsNone(d.getVarFlag("A", "export"))
self.assertEqual(d.getVar("B"), "b")
@@ -96,7 +150,6 @@ exportD = "d"
self.assertEqual(d.getVar("exportD"), "d")
self.assertIsNone(d.getVarFlag("exportD", "export"))
-
overridetest = """
RRECOMMENDS:${PN} = "a"
RRECOMMENDS:${PN}:libc = "b"
@@ -105,8 +158,8 @@ PN = "gtk+"
"""
def test_parse_overrides(self):
- f = self.parsehelper(self.overridetest)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.overridetest) as f:
+ d = bb.parse.handle(f.name, self.d)['']
self.assertEqual(d.getVar("RRECOMMENDS"), "b")
bb.data.expandKeys(d)
self.assertEqual(d.getVar("RRECOMMENDS"), "b")
@@ -120,8 +173,8 @@ EXTRA_OECONF:append = " c"
"""
def test_parse_overrides2(self):
- f = self.parsehelper(self.overridetest2)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.overridetest2) as f:
+ d = bb.parse.handle(f.name, self.d)['']
d.appendVar("EXTRA_OECONF", " d")
d.setVar("OVERRIDES", "class-target")
self.assertEqual(d.getVar("EXTRA_OECONF"), "b c d")
@@ -133,8 +186,8 @@ PN = "bc"
"""
def test_parse_combinations(self):
- f = self.parsehelper(self.overridetest3)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.overridetest3) as f:
+ d = bb.parse.handle(f.name, self.d)['']
bb.data.expandKeys(d)
self.assertEqual(d.getVar("DESCRIPTION:bc-dev"), "A B")
d.setVar("DESCRIPTION", "E")
@@ -142,7 +195,6 @@ PN = "bc"
d.setVar("OVERRIDES", "bc-dev")
self.assertEqual(d.getVar("DESCRIPTION"), "C D")
-
classextend = """
VAR_var:override1 = "B"
EXTRA = ":override1"
@@ -160,24 +212,36 @@ python () {
#
# Test based upon a real world data corruption issue. One
# data store changing a variable poked through into a different data
- # store. This test case replicates that issue where the value 'B' would
+ # store. This test case replicates that issue where the value 'B' would
# become unset/disappear.
#
def test_parse_classextend_contamination(self):
self.d.setVar("__bbclasstype", "recipe")
- cls = self.parsehelper(self.classextend_bbclass, suffix=".bbclass")
- #clsname = os.path.basename(cls.name).replace(".bbclass", "")
- self.classextend = self.classextend.replace("###CLASS###", cls.name)
- f = self.parsehelper(self.classextend)
- alldata = bb.parse.handle(f.name, self.d)
- d1 = alldata['']
- d2 = alldata[cls.name]
- self.assertEqual(d1.getVar("VAR_var"), "B")
- self.assertEqual(d2.getVar("VAR_var"), None)
+ with self.parsehelper(self.classextend_bbclass, suffix=".bbclass") as cls:
+ #clsname = os.path.basename(cls.name).replace(".bbclass", "")
+ self.classextend = self.classextend.replace("###CLASS###", cls.name)
+ with self.parsehelper(self.classextend) as f:
+ alldata = bb.parse.handle(f.name, self.d)
+ d1 = alldata['']
+ d2 = alldata[cls.name]
+ self.assertEqual(d1.getVar("VAR_var"), "B")
+ self.assertEqual(d2.getVar("VAR_var"), None)
addtask_deltask = """
addtask do_patch after do_foo after do_unpack before do_configure before do_compile
-addtask do_fetch do_patch
+addtask do_fetch2 do_patch2
+
+addtask do_myplaintask
+addtask do_myplaintask2
+deltask do_myplaintask2
+addtask do_mytask# comment
+addtask do_mytask2 # comment2
+addtask do_mytask3
+deltask do_mytask3# comment
+deltask do_mytask4 # comment2
+
+# Ensure a missing task prefix on after works
+addtask do_mytask5 after mytask
MYVAR = "do_patch"
EMPTYVAR = ""
@@ -185,17 +249,10 @@ deltask do_fetch ${MYVAR} ${EMPTYVAR}
deltask ${EMPTYVAR}
"""
def test_parse_addtask_deltask(self):
- import sys
-
- with self.assertLogs() as logs:
- f = self.parsehelper(self.addtask_deltask)
+ with self.parsehelper(self.addtask_deltask) as f:
d = bb.parse.handle(f.name, self.d)['']
-
- output = "".join(logs.output)
- self.assertTrue("addtask contained multiple 'before' keywords" in output)
- self.assertTrue("addtask contained multiple 'after' keywords" in output)
- self.assertTrue('addtask ignored: " do_patch"' in output)
- #self.assertTrue('dependent task do_foo for do_patch does not exist' in output)
+ self.assertSequenceEqual(['do_fetch2', 'do_patch2', 'do_myplaintask', 'do_mytask', 'do_mytask2', 'do_mytask5'], bb.build.listtasks(d))
+ self.assertEqual(['do_mytask'], d.getVarFlag("do_mytask5", "deps"))
broken_multiline_comment = """
# First line of comment \\
@@ -203,10 +260,9 @@ deltask ${EMPTYVAR}
"""
def test_parse_broken_multiline_comment(self):
- f = self.parsehelper(self.broken_multiline_comment)
- with self.assertRaises(bb.BBHandledException):
- d = bb.parse.handle(f.name, self.d)['']
-
+ with self.parsehelper(self.broken_multiline_comment) as f:
+ with self.assertRaises(bb.BBHandledException):
+ d = bb.parse.handle(f.name, self.d)['']
comment_in_var = """
VAR = " \\
@@ -216,10 +272,9 @@ VAR = " \\
"
"""
def test_parse_comment_in_var(self):
- f = self.parsehelper(self.comment_in_var)
- with self.assertRaises(bb.BBHandledException):
- d = bb.parse.handle(f.name, self.d)['']
-
+ with self.parsehelper(self.comment_in_var) as f:
+ with self.assertRaises(bb.BBHandledException):
+ d = bb.parse.handle(f.name, self.d)['']
at_sign_in_var_flag = """
A[flag@.service] = "nonet"
@@ -229,8 +284,8 @@ C[f] = "flag"
unset A[flag@.service]
"""
def test_parse_at_sign_in_var_flag(self):
- f = self.parsehelper(self.at_sign_in_var_flag)
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(self.at_sign_in_var_flag) as f:
+ d = bb.parse.handle(f.name, self.d)['']
self.assertEqual(d.getVar("A"), None)
self.assertEqual(d.getVar("B"), None)
self.assertEqual(d.getVarFlag("A","flag@.service"), None)
@@ -239,9 +294,9 @@ unset A[flag@.service]
def test_parse_invalid_at_sign_in_var_flag(self):
invalid_at_sign = self.at_sign_in_var_flag.replace("B[f", "B[@f")
- f = self.parsehelper(invalid_at_sign)
- with self.assertRaises(bb.parse.ParseError):
- d = bb.parse.handle(f.name, self.d)['']
+ with self.parsehelper(invalid_at_sign) as f:
+ with self.assertRaises(bb.parse.ParseError):
+ d = bb.parse.handle(f.name, self.d)['']
export_function_recipe = """
inherit someclass
@@ -296,14 +351,11 @@ EXPORT_FUNCTIONS do_compile do_compilepython
os.makedirs(tempdir + "/classes")
with open(tempdir + "/classes/someclass.bbclass", "w") as f:
f.write(self.export_function_class)
- f.flush()
with open(tempdir + "/classes/secondclass.bbclass", "w") as f:
f.write(self.export_function_class2)
- f.flush()
with open(recipename, "w") as f:
f.write(self.export_function_recipe)
- f.flush()
os.chdir(tempdir)
d = bb.parse.handle(recipename, bb.data.createCopy(self.d))['']
self.assertIn("someclass_do_compile", d.getVar("do_compile"))
@@ -313,7 +365,6 @@ EXPORT_FUNCTIONS do_compile do_compilepython
recipename2 = tempdir + "/recipe2.bb"
with open(recipename2, "w") as f:
f.write(self.export_function_recipe2)
- f.flush()
d = bb.parse.handle(recipename2, bb.data.createCopy(self.d))['']
self.assertNotIn("someclass_do_compile", d.getVar("do_compile"))
@@ -324,10 +375,8 @@ EXPORT_FUNCTIONS do_compile do_compilepython
with open(recipename, "a+") as f:
f.write("\ninherit secondclass\n")
- f.flush()
with open(recipename2, "a+") as f:
f.write("\ninherit secondclass\n")
- f.flush()
d = bb.parse.handle(recipename, bb.data.createCopy(self.d))['']
self.assertIn("secondclass_do_compile", d.getVar("do_compile"))
@@ -341,3 +390,251 @@ EXPORT_FUNCTIONS do_compile do_compilepython
self.assertIn("else", d.getVar("do_compilepython"))
check_function_flags(d)
+ export_function_unclosed_tab = """
+do_compile () {
+ bb.note("Something")
+\t}
+"""
+ export_function_unclosed_space = """
+do_compile () {
+ bb.note("Something")
+ }
+"""
+ export_function_residue = """
+do_compile () {
+ bb.note("Something")
+}
+
+include \\
+"""
+
+ def test_unclosed_functions(self):
+ def test_helper(content, expected_error):
+ with tempfile.TemporaryDirectory() as tempdir:
+ recipename = tempdir + "/recipe_unclosed.bb"
+ with open(recipename, "w") as f:
+ f.write(content)
+ os.chdir(tempdir)
+ with self.assertRaises(bb.parse.ParseError) as error:
+ bb.parse.handle(recipename, bb.data.createCopy(self.d))
+ self.assertIn(expected_error, str(error.exception))
+
+ with tempfile.TemporaryDirectory() as tempdir:
+ test_helper(self.export_function_unclosed_tab, "Unparsed lines from unclosed function")
+ test_helper(self.export_function_unclosed_space, "Unparsed lines from unclosed function")
+ test_helper(self.export_function_residue, "Unparsed lines")
+
+ recipename_closed = tempdir + "/recipe_closed.bb"
+ with open(recipename_closed, "w") as in_file:
+ lines = self.export_function_unclosed_tab.split("\n")
+ lines[3] = "}"
+ in_file.write("\n".join(lines))
+ bb.parse.handle(recipename_closed, bb.data.createCopy(self.d))
+
+ special_character_assignment = """
+A+="a"
+A+ = "b"
++ = "c"
+"""
+ ambigous_assignment = """
++= "d"
+"""
+ def test_parse_special_character_assignment(self):
+ with self.parsehelper(self.special_character_assignment) as f:
+ d = bb.parse.handle(f.name, self.d)['']
+ self.assertEqual(d.getVar("A"), " a")
+ self.assertEqual(d.getVar("A+"), "b")
+ self.assertEqual(d.getVar("+"), "c")
+
+ with self.parsehelper(self.ambigous_assignment) as f:
+ with self.assertRaises(bb.parse.ParseError) as error:
+ bb.parse.handle(f.name, self.d)
+ self.assertIn("Empty variable name in assignment", str(error.exception))
+
+ someconf1 = """
+EXTRA_OECONF:append = " foo"
+"""
+
+ someconf2 = """
+EXTRA_OECONF:append = " bar"
+"""
+
+ someconf3 = """
+EXTRA_OECONF:append = " foobar"
+"""
+
+ def test_include_and_require(self):
+ def test_helper(content, result):
+ with self.parsehelper(content) as f:
+ if isinstance(result, type) and issubclass(result, Exception):
+ with self.assertRaises(result):
+ d = bb.parse.handle(f.name, bb.data.createCopy(self.d))['']
+ else:
+ d = bb.parse.handle(f.name, bb.data.createCopy(self.d))['']
+ self.assertEqual(d.getVar("EXTRA_OECONF"), result)
+
+ with tempfile.TemporaryDirectory() as tempdir:
+ os.makedirs(tempdir + "/conf1")
+ os.makedirs(tempdir + "/conf2")
+
+ with open(tempdir + "/conf1/some.conf", "w") as f:
+ f.write(self.someconf1)
+ with open(tempdir + "/conf2/some.conf", "w") as f:
+ f.write(self.someconf2)
+ with open(tempdir + "/conf2/some3.conf", "w") as f:
+ f.write(self.someconf3)
+
+ self.d.setVar("BBPATH", tempdir + "/conf1" + ":" + tempdir + "/conf2")
+
+ test_helper("include some.conf", " foo")
+ test_helper("include someother.conf", None)
+ test_helper("include some3.conf", " foobar")
+ test_helper("include ${@''}", None)
+ test_helper("include " + tempdir + "/conf2/some.conf", " bar")
+
+ test_helper("require some.conf", " foo")
+ test_helper("require someother.conf", bb.parse.ParseError)
+ test_helper("require some3.conf", " foobar")
+ test_helper("require ${@''}", None)
+ test_helper("require " + tempdir + "/conf2/some.conf", " bar")
+
+ test_helper("include_all some.conf", " foo bar")
+ test_helper("include_all someother.conf", None)
+ test_helper("include_all some3.conf", " foobar")
+
+ self.d.setVar("BBPATH", tempdir + "/conf2" + ":" + tempdir + "/conf1")
+
+ test_helper("include some.conf", " bar")
+ test_helper("include some3.conf", " foobar")
+ test_helper("require some.conf", " bar")
+ test_helper("require some3.conf", " foobar")
+ test_helper("include_all some.conf", " bar foo")
+ test_helper("include_all some3.conf", " foobar")
+
+ def test_file_variables(self):
+ # Tests the values of FILE and __BB_RECIPE_FILE in different
+ # combinations of bbappends, includes, and inherits
+
+ def write_file(path, data):
+ with open(path, "w") as f:
+ f.write(textwrap.dedent(data))
+
+ def run_bitbake(cmd, builddir, extraenv={}):
+ env = os.environ.copy()
+ env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "parse-tests"))
+ env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "TOPDIR"
+ env["TOPDIR"] = builddir
+ for k, v in extraenv.items():
+ env[k] = v
+ env["BB_ENV_PASSTHROUGH_ADDITIONS"] = env["BB_ENV_PASSTHROUGH_ADDITIONS"] + " " + k
+ try:
+ return subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT, universal_newlines=True, cwd=builddir)
+ except subprocess.CalledProcessError as e:
+ self.fail("Command %s failed with %s" % (cmd, e.output))
+
+ with tempfile.TemporaryDirectory(prefix="parserecipes") as recipes, tempfile.TemporaryDirectory(prefix="parsetest") as builddir:
+ extraenv = {
+ "EXTRA_BBFILES": f"{recipes}/*.bb {recipes}/*.bbappend",
+ }
+
+ inc_path = f"{recipes}/recipe-file.inc"
+ bbappend_path = f"{recipes}/recipe-%.bbappend"
+ recipe_path = f"{recipes}/recipe-file1.bb"
+
+ # __BB_RECIPE_FILE should always be the name of .bb file, even
+ # when set in a bbappend. FILE is the name of the bbappend
+ write_file(recipe_path, "")
+ write_file(bbappend_path,
+ """\
+ BBAPPEND_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}"
+ BBAPPEND_FILE := "${@os.path.basename(d.getVar('FILE'))}"
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('BBAPPEND_FILE="recipe-%.bbappend"', output)
+ self.assertIn(f'BBAPPEND_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # __BB_RECIPE_FILE should always be the name of .bb file, even when
+ # set in an include file. FILE is the name of the include
+ write_file(recipe_path,
+ """\
+ require recipe-file.inc
+ """
+ )
+ write_file(inc_path,
+ """\
+ INC_RECIPE_FILE := "${@os.path.basename(d.getVar('__BB_RECIPE_FILE'))}"
+ INC_FILE := "${@os.path.basename(d.getVar('FILE'))}"
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('INC_FILE="recipe-file.inc"', output)
+ self.assertIn(f'INC_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # Test when the include file is included from a bbappend
+ write_file(recipe_path, "")
+ write_file(bbappend_path,
+ """\
+ require recipe-file.inc
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('INC_FILE="recipe-file.inc"', output)
+ self.assertIn(f'INC_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # Test the variables in a bbclass when inherited directly in the
+ # recipe. Note that FILE still refers to the recipe in a bbclass
+ write_file(recipe_path,
+ """\
+ inherit recipe-file-class
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('BBCLASS_FILE="recipe-file1.bb"', output)
+ self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # Test the variables when the inherit is in a bbappend. In this
+ # case, FILE is the bbappend
+ write_file(recipe_path, "")
+ write_file(bbappend_path,
+ """\
+ inherit recipe-file-class
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('BBCLASS_FILE="recipe-%.bbappend"', output)
+ self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # Test the variables when the inherit is in a include. In this
+ # case, FILE is the include file
+ write_file(recipe_path,
+ """\
+ require recipe-file.inc
+ """
+ )
+ write_file(bbappend_path, "")
+ write_file(inc_path,
+ """\
+ inherit recipe-file-class
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('BBCLASS_FILE="recipe-file.inc"', output)
+ self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output)
+
+ # Test the variables when the inherit is in a include included from
+ # a bbappend. In this case, FILE is the include file
+ write_file(recipe_path, "")
+ write_file(bbappend_path,
+ """\
+ require recipe-file.inc
+ """
+ )
+ write_file(inc_path,
+ """\
+ inherit recipe-file-class
+ """
+ )
+ output = run_bitbake(["bitbake", "-e", "recipe-file1"], builddir, extraenv).splitlines()
+ self.assertIn('BBCLASS_FILE="recipe-file.inc"', output)
+ self.assertIn(f'BBCLASS_RECIPE_FILE="recipe-file1.bb"', output)
diff --git a/bitbake/lib/bb/tests/persist_data.py b/bitbake/lib/bb/tests/persist_data.py
deleted file mode 100644
index f641b5ac..00000000
--- a/bitbake/lib/bb/tests/persist_data.py
+++ /dev/null
@@ -1,129 +0,0 @@
-#
-# BitBake Test for lib/bb/persist_data/
-#
-# Copyright (C) 2018 Garmin Ltd.
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import unittest
-import bb.data
-import bb.persist_data
-import tempfile
-import threading
-
-class PersistDataTest(unittest.TestCase):
- def _create_data(self):
- return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
-
- def setUp(self):
- self.d = bb.data.init()
- self.tempdir = tempfile.TemporaryDirectory()
- self.d['PERSISTENT_DIR'] = self.tempdir.name
- self.data = self._create_data()
- self.items = {
- 'A1': '1',
- 'B1': '2',
- 'C2': '3'
- }
- self.stress_count = 10000
- self.thread_count = 5
-
- for k,v in self.items.items():
- self.data[k] = v
-
- def tearDown(self):
- self.tempdir.cleanup()
-
- def _iter_helper(self, seen, iterator):
- with iter(iterator):
- for v in iterator:
- self.assertTrue(v in seen)
- seen.remove(v)
- self.assertEqual(len(seen), 0, '%s not seen' % seen)
-
- def test_get(self):
- for k, v in self.items.items():
- self.assertEqual(self.data[k], v)
-
- self.assertIsNone(self.data.get('D'))
- with self.assertRaises(KeyError):
- self.data['D']
-
- def test_set(self):
- for k, v in self.items.items():
- self.data[k] += '-foo'
-
- for k, v in self.items.items():
- self.assertEqual(self.data[k], v + '-foo')
-
- def test_delete(self):
- self.data['D'] = '4'
- self.assertEqual(self.data['D'], '4')
- del self.data['D']
- self.assertIsNone(self.data.get('D'))
- with self.assertRaises(KeyError):
- self.data['D']
-
- def test_contains(self):
- for k in self.items:
- self.assertTrue(k in self.data)
- self.assertTrue(self.data.has_key(k))
- self.assertFalse('NotFound' in self.data)
- self.assertFalse(self.data.has_key('NotFound'))
-
- def test_len(self):
- self.assertEqual(len(self.data), len(self.items))
-
- def test_iter(self):
- self._iter_helper(set(self.items.keys()), self.data)
-
- def test_itervalues(self):
- self._iter_helper(set(self.items.values()), self.data.itervalues())
-
- def test_iteritems(self):
- self._iter_helper(set(self.items.items()), self.data.iteritems())
-
- def test_get_by_pattern(self):
- self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
-
- def _stress_read(self, data):
- for i in range(self.stress_count):
- for k in self.items:
- data[k]
-
- def _stress_write(self, data):
- for i in range(self.stress_count):
- for k, v in self.items.items():
- data[k] = v + str(i)
-
- def _validate_stress(self):
- for k, v in self.items.items():
- self.assertEqual(self.data[k], v + str(self.stress_count - 1))
-
- def test_stress(self):
- self._stress_read(self.data)
- self._stress_write(self.data)
- self._validate_stress()
-
- def test_stress_threads(self):
- def read_thread():
- data = self._create_data()
- self._stress_read(data)
-
- def write_thread():
- data = self._create_data()
- self._stress_write(data)
-
- threads = []
- for i in range(self.thread_count):
- threads.append(threading.Thread(target=read_thread))
- threads.append(threading.Thread(target=write_thread))
-
- for t in threads:
- t.start()
- self._stress_read(self.data)
- for t in threads:
- t.join()
- self._validate_stress()
-
diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
index b57650d5..3a3db55d 100644
--- a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
+++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
@@ -9,7 +9,7 @@ def stamptask(d):
with open(stampname, "a+") as f:
f.write(d.getVar("BB_UNIHASH") + "\n")
- if d.getVar("BB_CURRENT_MC") != "default":
+ if d.getVar("BB_CURRENT_MC") != "":
thistask = d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
if thistask in d.getVar("SLOWTASKS").split():
bb.note("Slowing task %s" % thistask)
@@ -98,7 +98,7 @@ do_package_qa[rdeptask] = "do_packagedata"
do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
DEBIANRDEP = "do_packagedata"
-oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
+do_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
addtask fetch
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
new file mode 100644
index 00000000..3c7dca02
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_invalid"
+
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
new file mode 100644
index 00000000..e69de29b
diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py
index cc87e8d6..42c02cd1 100644
--- a/bitbake/lib/bb/tests/runqueue.py
+++ b/bitbake/lib/bb/tests/runqueue.py
@@ -26,7 +26,7 @@ class RunQueueTests(unittest.TestCase):
a1_sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic a1:do_populate_sysroot"
b1_sstatevalid = "b1:do_package b1:do_package_qa b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm b1:do_populate_lic b1:do_populate_sysroot"
- def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None, cleanup=False):
+ def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None, cleanup=False, allowfailure=False):
env = os.environ.copy()
env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "runqueue-tests"))
env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "SSTATEVALID SLOWTASKS TOPDIR"
@@ -41,6 +41,8 @@ class RunQueueTests(unittest.TestCase):
output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
print(output)
except subprocess.CalledProcessError as e:
+ if allowfailure:
+ return e.output
self.fail("Command %s failed with %s" % (cmd, e.output))
tasks = []
tasklog = builddir + "/task.log"
@@ -61,6 +63,15 @@ class RunQueueTests(unittest.TestCase):
self.shutdown(tempdir)
+ def test_basic_scheduler(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, extraenv={"BB_SCHEDULER": "basic"})
+ expected = ['a1:' + task for task in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+
+ self.shutdown(tempdir)
+
def test_single_setscenevalid(self):
with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
cmd = ["bitbake", "a1"]
@@ -314,6 +325,13 @@ class RunQueueTests(unittest.TestCase):
["mc_2:a1:%s" % t for t in rerun_tasks]
self.assertEqual(set(tasks), set(expected))
+ # Check that a multiconfig that doesn't exist rasies a correct error message
+ error_output = self.run_bitbakecmd(["bitbake", "g1"], tempdir, "", extraenv=extraenv, cleanup=True, allowfailure=True)
+ self.assertIn("non-existent task", error_output)
+ # If the word 'Traceback' or 'KeyError' is in the output we've regressed
+ self.assertNotIn("Traceback", error_output)
+ self.assertNotIn("KeyError", error_output)
+
self.shutdown(tempdir)
def test_hashserv_single(self):
diff --git a/bitbake/lib/bb/tests/setup.py b/bitbake/lib/bb/tests/setup.py
new file mode 100644
index 00000000..53a07ee3
--- /dev/null
+++ b/bitbake/lib/bb/tests/setup.py
@@ -0,0 +1,1089 @@
+#
+# Copyright BitBake Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from bb.tests.fetch import FetcherTest
+import bb
+import bb.process
+import glob
+import hashlib
+import json
+import os
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import unittest
+import venv
+import zipfile
+from bb.tests.support.httpserver import HTTPService
+
+class BitbakeSetupTest(FetcherTest):
+ def setUp(self):
+ super(BitbakeSetupTest, self).setUp()
+
+ self.registrypath = os.path.join(self.tempdir, "bitbake-setup-configurations")
+
+ os.makedirs(self.registrypath)
+ self.git_init(cwd=self.registrypath)
+ self.git(['commit', '--allow-empty', '-m', "Initial commit"], cwd=self.registrypath)
+
+ self.testrepopath = os.path.join(self.tempdir, "test-repo")
+ os.makedirs(self.testrepopath)
+ self.git_init(cwd=self.testrepopath)
+ self.git(['commit', '--allow-empty', '-m', "Initial commit"], cwd=self.testrepopath)
+
+ oeinitbuildenv = """BBPATH=$1
+export BBPATH
+PATH={}:$PATH
+""".format(os.path.join(self.testrepopath, 'scripts'))
+ self.add_file_to_testrepo('oe-init-build-env',oeinitbuildenv, script=True)
+
+ oesetupbuild = """#!/usr/bin/env python3
+import getopt
+import sys
+import os
+import shutil
+opts, args = getopt.getopt(sys.argv[2:], "c:b:", ["no-shell"])
+for option, value in opts:
+ if option == '-c':
+ template = value
+ if option == '-b':
+ builddir = value
+confdir = os.path.join(builddir, 'conf')
+os.makedirs(confdir, exist_ok=True)
+with open(os.path.join(confdir, 'conf-summary.txt'), 'w') as f:
+ f.write(template)
+shutil.copy(os.path.join(os.path.dirname(__file__), 'test-repo/test-file'), confdir)
+with open(os.path.join(builddir, 'init-build-env'), 'w') as f:
+ f.write("BBPATH={}\\nexport BBPATH\\nPATH={}:$PATH".format(builddir, os.path.join(os.path.dirname(__file__), 'test-repo/scripts')))
+"""
+ self.add_file_to_testrepo('scripts/oe-setup-build', oesetupbuild, script=True)
+
+ installbuildtools = """#!/usr/bin/env python3
+import getopt
+import sys
+import os
+
+opts, args = getopt.getopt(sys.argv[1:], "d:", ["downloads-directory=", "local-file="])
+installdir = None
+local_file = None
+for option, value in opts:
+ if option == '-d':
+ installdir = value
+ elif option == '--local-file':
+ local_file = value
+ print("install-buildtools local-file={}".format(value))
+
+print("Buildtools installed into {}".format(installdir))
+os.makedirs(installdir, exist_ok=True)
+"""
+ self.add_file_to_testrepo('scripts/install-buildtools', installbuildtools, script=True)
+
+ # Dummy buildtools installer for bb.fetch testing
+ self.buildtools_dir = os.path.join(self.tempdir, "buildtools-dl")
+ os.makedirs(self.buildtools_dir)
+ self.buildtools_filename = "x86_64-buildtools-nativesdk-standalone-test.sh"
+ buildtools_filepath = os.path.join(self.buildtools_dir, self.buildtools_filename)
+ with open(buildtools_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho dummy\n")
+ with open(buildtools_filepath, 'rb') as f:
+ self.buildtools_sha256 = hashlib.sha256(f.read()).hexdigest()
+
+ bitbakeconfigbuild = """#!/usr/bin/env python3
+import os
+import sys
+confdir = os.path.join(os.environ['BBPATH'], 'conf')
+fragment = sys.argv[2]
+with open(os.path.join(confdir, fragment), 'w') as f:
+ f.write('')
+"""
+ self.add_file_to_testrepo('scripts/bitbake-config-build', bitbakeconfigbuild, script=True)
+
+ sometargetexecutable_template = """#!/usr/bin/env python3
+import os
+print("This is {}")
+print("BBPATH is {{}}".format(os.environ["BBPATH"]))
+"""
+ for e_name in ("some-target-executable-1", "some-target-executable-2"):
+ sometargetexecutable = sometargetexecutable_template.format(e_name)
+ self.add_file_to_testrepo('scripts/{}'.format(e_name), sometargetexecutable, script=True)
+
+ def runbbsetup(self, cmd):
+ bbsetup = os.path.abspath(os.path.dirname(__file__) + "/../../../bin/bitbake-setup")
+ # Set PYTHONPATH so subprocess can find bb module instead of relying on the current directory
+ env = os.environ.copy()
+ libdir = os.path.abspath(os.path.dirname(__file__) + "/../..")
+ env["PYTHONPATH"] = libdir + ":" + env.get("PYTHONPATH", "")
+ return bb.process.run([bbsetup, '--global-settings', os.path.join(self.tempdir, 'global-config')] + cmd, env=env, cwd=self.tempdir)
+
+
+ def _add_json_config_to_registry_helper(self, name, sources):
+ config = """
+{
+ "sources": {
+%s
+ },
+ "description": "Test configuration",
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name": "gadget",
+ "description": "Gadget configuration",
+ "oe-template": "test-configuration-gadget",
+ "oe-fragments": ["test-fragment-1"],
+ "install-buildtools": {
+ "url": "file://%s/%s",
+ "sha256sum": "%s"
+ }
+ },
+ {
+ "name": "gizmo",
+ "description": "Gizmo configuration",
+ "oe-template": "test-configuration-gizmo",
+ "oe-fragments": ["test-fragment-2"],
+ "setup-dir-name": "this-is-a-custom-gizmo-build"
+ },
+ {
+ "name": "gizmo-env-passthrough",
+ "description": "Gizmo configuration with environment-passthrough",
+ "bb-layers": ["layerC","layerD/meta-layer"],
+ "oe-fragments": ["test-fragment-1"],
+ "bb-env-passthrough-additions": [
+ "BUILD_ID",
+ "BUILD_DATE",
+ "BUILD_SERVER"
+ ]
+ },
+ {
+ "name": "gizmo-no-fragment",
+ "description": "Gizmo no-fragment template-only configuration",
+ "oe-template": "test-configuration-gizmo"
+ },
+ {
+ "name": "gadget-notemplate",
+ "description": "Gadget notemplate configuration",
+ "notes": [
+ "Gadget notemplate notes",
+ "Second line"
+ ],
+ "bb-layers": ["layerA","layerB/meta-layer"],
+ "oe-fragments": ["test-fragment-1"]
+ },
+ {
+ "name": "gizmo-notemplate-fragments-one-of",
+ "description": "Gizmo notemplate configuration",
+ "bb-layers": ["layerC","layerD/meta-layer"],
+ "oe-fragments": ["test-fragment-2"],
+ "oe-fragments-one-of": {
+ "fragment-desc": {
+ "description": "Test fragments (with description)",
+ "options" : [
+ { "name": "fragment-desc-1", "description": "fragment 1 desc" },
+ { "name": "fragment-desc-2", "description": "fragment 2 desc" }
+ ]
+ },
+ "fragment-nodesc": {
+ "description": "Test fragments (no description)",
+ "options" : [
+ "fragment-nodesc-1",
+ "fragment-nodesc-2"
+ ]
+ }
+ }
+ },
+ {
+ "name": "gizmo-notemplate-with-filerelative-layers",
+ "description": "Gizmo notemplate configuration using filerelative layers",
+ "bb-layers": ["layerC","layerD/meta-layer"],
+ "bb-layers-file-relative": ["layerE/meta-layer"],
+ "oe-fragments": ["test-fragment-2"]
+ }
+ ]
+ },
+ "version": "1.0"
+}
+""" % (sources, self.buildtools_dir, self.buildtools_filename, self.buildtools_sha256)
+ os.makedirs(os.path.join(self.registrypath, os.path.dirname(name)), exist_ok=True)
+ with open(os.path.join(self.registrypath, name), 'w') as f:
+ f.write(config)
+ self.git(['add', name], cwd=self.registrypath)
+ self.git(['commit', '-m', "Adding " + name], cwd=self.registrypath)
+ return json.loads(config)
+
+ def add_json_config_to_registry(self, name, rev, branch):
+ sources = """
+ "test-repo": {
+ "git-remote": {
+ "remotes": {
+ "origin": {
+ "uri": "file://%s"
+ }
+ },
+ "branch": "%s",
+ "rev": "%s"
+ }
+ }
+""" % (self.testrepopath, branch, rev)
+ return self._add_json_config_to_registry_helper(name, sources)
+
+ def add_local_json_config_to_registry(self, name, path):
+ sources = """
+ "test-repo": {
+ "local": {
+ "path": "%s"
+ }
+ }
+""" % (path)
+ return self._add_json_config_to_registry_helper(name, sources)
+
+ def add_file_to_testrepo(self, name, content, script=False):
+ fullname = os.path.join(self.testrepopath, name)
+ os.makedirs(os.path.join(self.testrepopath, os.path.dirname(name)), exist_ok=True)
+ with open(fullname, 'w') as f:
+ f.write(content)
+ if script:
+ st = os.stat(fullname)
+ os.chmod(fullname, st.st_mode | stat.S_IEXEC)
+ self.git(['add', name], cwd=self.testrepopath)
+ self.git(['commit', '-m', "Adding " + name], cwd=self.testrepopath)
+
+ def config_is_unchanged(self, setuppath):
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["status"])
+ self.assertIn("Configuration in {} has not changed".format(setuppath), out[0])
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ self.assertIn("Configuration in {} has not changed".format(setuppath), out[0])
+ del os.environ['BBPATH']
+
+ def check_setupdir_files(self, setuppath, test_file_content):
+ with open(os.path.join(setuppath, 'config', "config-upstream.json")) as f:
+ config_upstream = json.load(f)
+ with open(os.path.join(setuppath, 'layers', 'test-repo', 'test-file')) as f:
+ self.assertEqual(f.read(), test_file_content)
+ bitbake_config = config_upstream["bitbake-config"]
+ bb_build_path = os.path.join(setuppath, 'build')
+ bb_conf_path = os.path.join(bb_build_path, 'conf')
+ self.assertTrue(os.path.exists(os.path.join(bb_build_path, 'init-build-env')))
+
+ with open(os.path.join(setuppath, 'config', "sources-fixed-revisions.json")) as f:
+ sources_fixed_revisions = json.load(f)
+ self.assertTrue('test-repo' in sources_fixed_revisions['sources'].keys())
+ revision = self.git(['rev-parse', 'HEAD'], cwd=self.testrepopath).strip()
+ self.assertEqual(revision, sources_fixed_revisions['sources']['test-repo']['git-remote']['rev'])
+
+ if "oe-template" in bitbake_config:
+ with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f:
+ self.assertEqual(f.read(), bitbake_config["oe-template"])
+ with open(os.path.join(bb_conf_path, 'test-file')) as f:
+ self.assertEqual(f.read(), test_file_content)
+ else:
+ with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f:
+ self.assertIn(bitbake_config["description"], f.read())
+ with open(os.path.join(bb_conf_path, 'conf-notes.txt')) as f:
+ expected_notes = bitbake_config.get("notes")
+ if isinstance(expected_notes, list):
+ expected_notes = "\n".join(expected_notes)
+ self.assertEqual(f.read(), expected_notes + "\n" if expected_notes else "")
+ with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f:
+ bblayers = f.read()
+ for l in bitbake_config["bb-layers"]:
+ self.assertIn(os.path.join(setuppath, "layers", l), bblayers)
+ for l in bitbake_config.get("bb-layers-file-relative") or []:
+ filerelative_layer = os.path.join(
+ os.path.dirname(config_upstream["path"]),
+ l,
+ )
+ self.assertIn(filerelative_layer, bblayers)
+
+ if 'oe-fragments' in bitbake_config:
+ for f in bitbake_config["oe-fragments"]:
+ self.assertTrue(os.path.exists(os.path.join(bb_conf_path, f)))
+
+ if 'bb-env-passthrough-additions' in bitbake_config:
+ with open(os.path.join(bb_build_path, 'init-build-env'), 'r') as f:
+ init_build_env = f.read()
+ self.assertTrue('BB_ENV_PASSTHROUGH_ADDITIONS' in init_build_env)
+ self.assertTrue('BUILD_ID' in init_build_env)
+ self.assertTrue('BUILD_DATE' in init_build_env)
+ self.assertTrue('BUILD_SERVER' in init_build_env)
+ # a more throrough test could be to initialize a bitbake build-env, export FOO to the shell environment, set the env-passthrough on it and finally check against 'bitbake-getvar FOO'
+
+ self.config_is_unchanged(setuppath)
+
+ def get_setup_path(self, cf, c):
+ if c == 'gizmo':
+ return os.path.join(self.tempdir, 'bitbake-builds', 'this-is-a-custom-gizmo-build')
+ return os.path.join(self.tempdir, 'bitbake-builds', '{}-{}'.format(cf, "-".join(c.split())))
+
+ def test_setup(self):
+ # unset BBPATH to ensure tests run in isolation from the existing bitbake environment
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+
+ # check that no arguments works
+ self.runbbsetup([])
+
+ # check that --help works
+ self.runbbsetup(["--help"])
+
+ # change to self.tempdir to work with cwd-based default settings
+ os.chdir(self.tempdir)
+
+ # check that the default top-dir-prefix is cwd (now self.tempdir) with no global settings
+ out = self.runbbsetup(["settings", "list"])
+ self.assertIn("default top-dir-prefix {}".format(os.getcwd()), out[0])
+
+ # set up global location for dl-dir
+ settings_path = "{}/global-config".format(self.tempdir)
+ out = self.runbbsetup(["settings", "set", "--global", "default", "dl-dir", os.path.join(self.tempdir, 'downloads')])
+ self.assertIn("From section 'default' the setting 'dl-dir' was changed to", out[0])
+ self.assertIn("Settings written to".format(settings_path), out[0])
+
+ # check that writing settings works and then adjust them to point to
+ # test registry repo
+ out = self.runbbsetup(["settings", "set", "default", "registry", "'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)])
+ settings_path = "{}/bitbake-builds/settings.conf".format(self.tempdir)
+ self.assertIn(settings_path, out[0])
+ self.assertIn("From section 'default' the setting 'registry' was changed to", out[0])
+ self.assertIn("Settings written to".format(settings_path), out[0])
+
+ # check that listing settings works
+ out = self.runbbsetup(["settings", "list"])
+ self.assertIn("default top-dir-prefix {}".format(self.tempdir), out[0])
+ self.assertIn("default dl-dir {}".format(os.path.join(self.tempdir, 'downloads')), out[0])
+ self.assertIn("default registry {}".format('git://{};protocol=file;branch=master;rev=master'.format(self.registrypath)), out[0])
+
+ # check that 'list' produces correct output with no configs, one config and two configs
+ out = self.runbbsetup(["list"])
+ self.assertNotIn("test-config-1", out[0])
+ self.assertNotIn("test-config-2", out[0])
+
+ json_1 = self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master')
+ out = self.runbbsetup(["list"])
+ self.assertIn("test-config-1", out[0])
+ self.assertNotIn("test-config-2", out[0])
+
+ json_2 = self.add_json_config_to_registry('config-2/test-config-2.conf.json', 'master', 'master')
+ out = self.runbbsetup(["list", "--write-json={}".format(os.path.join(self.tempdir, "test-configs.json"))])
+ self.assertIn("test-config-1", out[0])
+ self.assertIn("test-config-2", out[0])
+ with open(os.path.join(self.tempdir, "test-configs.json")) as f:
+ json_configs = json.load(f)
+ self.assertIn("test-config-1", json_configs)
+ self.assertIn("test-config-2", json_configs)
+
+ # check that init/status/update work
+ # (the latter two should do nothing and say that config hasn't changed)
+ test_file_content = 'initial\n'
+ self.add_file_to_testrepo('test-file', test_file_content)
+
+ # test-config-1 is tested as a registry config and over http, test-config-2 as a local file
+ server = HTTPService(self.registrypath, host="127.0.0.1")
+ server.start()
+ variants = (
+ 'gadget',
+ 'gizmo',
+ 'gizmo-env-passthrough',
+ 'gizmo-no-fragment',
+ 'gadget-notemplate',
+ 'gizmo-notemplate-fragments-one-of fragment-desc-1 fragment-nodesc-1',
+ )
+ variants_local = variants + ('gizmo-notemplate-with-filerelative-layers',)
+ test_configurations = ({'name':'test-config-1','cmdline': 'test-config-1',
+ 'buildconfigs': variants},
+ {'name':'test-config-2','cmdline': os.path.join(self.registrypath,'config-2/test-config-2.conf.json'),
+ 'buildconfigs': variants_local},
+ {'name':'test-config-1','cmdline':'http://127.0.0.1:{}/test-config-1.conf.json'.format(server.port),
+ 'buildconfigs': variants}
+ )
+ try:
+ for v in test_configurations:
+ for c in v['buildconfigs']:
+ out = self.runbbsetup(["init", "--non-interactive", v['cmdline']] + c.split())
+ setuppath = self.get_setup_path(v['name'], c)
+ self.check_setupdir_files(setuppath, test_file_content)
+ finally:
+ server.stop()
+
+ # install buildtools
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # change a file in the test layer repo, make a new commit and
+ # test that status/update correctly report the change and update the config
+ prev_test_file_content = test_file_content
+ test_file_content = 'modified\n'
+ self.add_file_to_testrepo('test-file', test_file_content)
+ for c in variants:
+ setuppath = self.get_setup_path('test-config-1', c)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["status"])
+ self.assertIn("Layer repository file://{} checked out into {}/layers/test-repo updated revision master from".format(self.testrepopath, setuppath), out[0])
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ if c in ('gadget', 'gizmo'):
+ self.assertIn("Leaving the previous configuration in {}/build/conf-backup.".format(setuppath), out[0])
+ self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0])
+ self.check_setupdir_files(setuppath, test_file_content)
+
+ # make a new branch in the test layer repo, change a file on that branch,
+ # make a new commit, update the top level json config to refer to that branch,
+ # and test that status/update correctly report the change and update the config
+ prev_test_file_content = test_file_content
+ test_file_content = 'modified-in-branch\n'
+ branch = "another-branch"
+ self.git(['checkout', '-b', branch], cwd=self.testrepopath)
+ self.add_file_to_testrepo('test-file', test_file_content)
+ json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch)
+ for c in variants:
+ setuppath = self.get_setup_path('test-config-1', c)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["status"])
+ self.assertIn("Configuration in {} has changed:".format(setuppath), out[0])
+ self.assertIn('- "rev": "master"\n+ "rev": "another-branch"', out[0])
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ if c in ('gadget', 'gizmo'):
+ self.assertIn("Leaving the previous configuration in {}/build/conf-backup.".format(setuppath), out[0])
+ self.assertIn('-{}+{}'.format(prev_test_file_content, test_file_content), out[0])
+ self.check_setupdir_files(setuppath, test_file_content)
+
+ # do the same as the previous test, but now without updating the bitbake configuration (--update-bb-conf=no)
+ # and check that files have not been modified
+ def _conf_chksum(confdir: str) -> list:
+ sums = []
+ for f in glob.glob(f'{confdir}/*'):
+ if os.path.isfile(f) and not os.path.islink(f):
+ with open(f, 'rb') as fd:
+ sha = hashlib.sha256()
+ sha.update(fd.read())
+ sums.append(os.path.basename(f) + '_' + sha.hexdigest())
+ return sums
+
+ prev_test_file_content = test_file_content
+ test_file_content = 'modified-in-branch-no-bb-conf-update\n'
+ branch = "another-branch-no-bb-conf-update"
+ self.git(['checkout', '-b', branch], cwd=self.testrepopath)
+ self.add_file_to_testrepo('test-file', test_file_content)
+ json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch)
+ for c in variants:
+ setuppath = self.get_setup_path('test-config-1', c)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ # write something in local.conf and bblayers.conf
+ for f in ["local.conf", "bblayers.conf"]:
+ with open(f"{setuppath}/build/conf/{f}", "w") as fd:
+ fd.write("deadbeef")
+ sums_before = _conf_chksum(f"{setuppath}/build/conf")
+ out = self.runbbsetup(["update", "--update-bb-conf=no"])
+ sums_after = _conf_chksum(f"{setuppath}/build/conf")
+ self.assertEqual(sums_before, sums_after)
+
+ def _check_local_sources(custom_setup_dir):
+ custom_setup_path = os.path.join(self.tempdir, 'bitbake-builds', custom_setup_dir)
+ custom_layer_path = os.path.join(custom_setup_path, 'layers', 'test-repo')
+ self.assertTrue(os.path.islink(custom_layer_path))
+ self.assertEqual(self.testrepopath, os.path.realpath(custom_layer_path))
+ self.config_is_unchanged(custom_setup_path)
+
+ def _check_layer_backups(layer_path, expected_backups):
+ files = os.listdir(layer_path)
+ backups = len([f for f in files if 'backup' in f])
+ self.assertEqual(backups, expected_backups, msg = "Expected {} layer backups, got {}, directory listing: {}".format(expected_backups, backups, files))
+
+ # Change the configuration to refer to a local source, then to another local source, then back to a git remote
+ # Run status/update after each change and verify that nothing breaks
+ # Also check that layer backups happen when expected
+ c = 'gadget'
+ setuppath = self.get_setup_path('test-config-1', c)
+ self.config_is_unchanged(setuppath)
+
+ layers_path = os.path.join(setuppath, 'layers')
+ layer_path = os.path.join(layers_path, 'test-repo')
+ _check_layer_backups(layers_path, 0)
+
+ json_1 = self.add_local_json_config_to_registry('test-config-1.conf.json', self.testrepopath)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ _check_local_sources(setuppath)
+ _check_layer_backups(layers_path, 1)
+
+ prev_path = self.testrepopath
+ self.testrepopath = prev_path + "-2"
+ self.git(['clone', prev_path, self.testrepopath], cwd=self.tempdir)
+ json_1 = self.add_local_json_config_to_registry('test-config-1.conf.json', self.testrepopath)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ _check_local_sources(setuppath)
+ _check_layer_backups(layers_path, 1)
+
+ self.testrepopath = prev_path
+ json_1 = self.add_json_config_to_registry('test-config-1.conf.json', branch, branch)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ self.check_setupdir_files(setuppath, test_file_content)
+ _check_layer_backups(layers_path, 1)
+
+ ## edit a file without making a commit
+ with open(os.path.join(layer_path, 'local-modification'), 'w') as f:
+ f.write('locally-modified\n')
+ test_file_content = "modified-again\n"
+ self.add_file_to_testrepo('test-file', test_file_content)
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ _check_layer_backups(layers_path, 1)
+
+ ## edit a file and make a commit such that no rebase conflicts occur
+ with open(os.path.join(layer_path, 'local-modification'), 'w') as f:
+ f.write('locally-modified-again\n')
+ self.git(['add', '.'], cwd=layer_path)
+ self.git(['commit', '-m', 'Adding a local modification'], cwd=layer_path)
+ test_file_content = "modified-again-and-again\n"
+ self.add_file_to_testrepo('test-file', test_file_content)
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ _check_layer_backups(layers_path, 1)
+
+ ## edit a file and make a commit in a way that causes a rebase conflict
+ with open(os.path.join(layer_path, 'test-file'), 'w') as f:
+ f.write('locally-modified\n')
+ self.git(['add', '.'], cwd=layer_path)
+ self.git(['commit', '-m', 'Adding a local modification'], cwd=layer_path)
+ test_file_content = "remotely-modified\n"
+ self.add_file_to_testrepo('test-file', test_file_content)
+ with self.assertRaisesRegex(bb.process.ExecutionError, "Merge conflict in test-file"):
+ out = self.runbbsetup(["update", "--update-bb-conf=yes"])
+ _check_layer_backups(layers_path, 1)
+
+ # check source overrides, local sources provided with symlinks, and custom setup dir name
+ source_override_content = """
+{
+ "sources": {
+ "test-repo": {
+ "local": {
+ "path": "."
+ }
+ }
+ }
+}"""
+ override_filename = 'source-overrides.json'
+ custom_setup_dir = 'special-setup-dir'
+ self.add_file_to_testrepo(override_filename, source_override_content)
+ out = self.runbbsetup(["init", "--non-interactive", "--source-overrides", os.path.join(self.testrepopath, override_filename), "--setup-dir-name", custom_setup_dir, "test-config-1", "gadget"])
+ _check_local_sources(custom_setup_dir)
+
+ # same but use command line options to specify local overrides
+ custom_setup_dir = 'special-setup-dir-with-cmdline-overrides'
+ out = self.runbbsetup(["init", "--non-interactive", "-L", "test-repo", self.testrepopath, "--setup-dir-name", custom_setup_dir, "test-config-1", "gadget"])
+ _check_local_sources(custom_setup_dir)
+
+ def test_vscode(self):
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ self.runbbsetup(["settings", "set", "default", "registry", "'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master')
+
+ # --init-vscode should create bitbake.code-workspace
+ self.runbbsetup(["init", "--non-interactive", "--init-vscode", "test-config-1", "gadget"])
+ setuppath = self.get_setup_path('test-config-1', 'gadget')
+ workspace_file = os.path.join(setuppath, 'bitbake.code-workspace')
+ self.assertTrue(os.path.exists(workspace_file),
+ "bitbake.code-workspace should be created with --init-vscode")
+
+ with open(workspace_file) as f:
+ workspace = json.load(f)
+
+ # top-level structure
+ self.assertIn('folders', workspace)
+ self.assertIn('settings', workspace)
+ self.assertIn('extensions', workspace)
+ self.assertIn('yocto-project.yocto-bitbake',
+ workspace['extensions']['recommendations'])
+
+ # folders: conf dir + test-repo (symlinks like oe-init-build-env-dir are skipped)
+ folder_names = {f['name'] for f in workspace['folders']}
+ self.assertIn('conf', folder_names)
+ self.assertIn('test-repo', folder_names)
+
+ # folder paths must be relative so the workspace is portable
+ for f in workspace['folders']:
+ self.assertFalse(os.path.isabs(f['path']),
+ "Folder path should be relative, got: {}".format(f['path']))
+
+ # bitbake extension settings
+ settings = workspace['settings']
+ self.assertTrue(settings.get('bitbake.disableConfigModification'))
+ self.assertEqual(settings['bitbake.pathToBuildFolder'],
+ os.path.join(setuppath, 'build'))
+ self.assertEqual(settings['bitbake.pathToEnvScript'],
+ os.path.join(setuppath, 'build', 'init-build-env'))
+
+ # file associations
+ self.assertIn('*.conf', settings.get('files.associations', {}))
+ self.assertIn('*.inc', settings.get('files.associations', {}))
+
+ # python extra paths: test-repo/scripts/ exists and should be listed
+ extra_paths = settings.get('python.analysis.extraPaths', [])
+ self.assertTrue(any('scripts' in p for p in extra_paths),
+ "python.analysis.extraPaths should include the scripts dir")
+ self.assertEqual(settings.get('python.analysis.extraPaths'),
+ settings.get('python.autoComplete.extraPaths'))
+
+ # --no-init-vscode should NOT create a workspace file
+ self.runbbsetup(["init", "--non-interactive", "--no-init-vscode", "test-config-1", "gadget-notemplate"])
+ notemplate_path = self.get_setup_path('test-config-1', 'gadget-notemplate')
+ self.assertFalse(
+ os.path.exists(os.path.join(notemplate_path, 'bitbake.code-workspace')),
+ "bitbake.code-workspace should not be created with --no-init-vscode")
+
+ # update with --init-vscode after a layer change should preserve
+ # user-added folders and settings while still rewriting managed ones
+ workspace['folders'].append({"name": "user-folder", "path": "user/custom"})
+ workspace['settings']['my.user.setting'] = 'preserved'
+ with open(workspace_file, 'w') as f:
+ json.dump(workspace, f, indent=4)
+
+ self.add_file_to_testrepo('test-file', 'updated\n')
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ self.runbbsetup(["update", "--update-bb-conf=no"])
+ del os.environ['BBPATH']
+
+ with open(workspace_file) as f:
+ updated = json.load(f)
+ self.assertIn('user/custom', {f['path'] for f in updated['folders']},
+ "User-added folder was removed during update")
+ self.assertIn('my.user.setting', updated['settings'],
+ "User-added setting was removed during update")
+
+ # update with a corrupt workspace file should log an error and leave it unchanged
+ self.add_file_to_testrepo('test-file', 'updated-again\n')
+ with open(workspace_file, 'w') as f:
+ f.write('{invalid json')
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ self.runbbsetup(["update", "--update-bb-conf=no"])
+ del os.environ['BBPATH']
+ with open(workspace_file) as f:
+ content = f.read()
+ self.assertEqual(content, '{invalid json',
+ "Corrupt workspace file should not be modified")
+
+ def _count_layer_backups(self, layers_path):
+ return len([f for f in os.listdir(layers_path) if 'backup' in f])
+
+ def test_install_buildtools_fetch(self):
+ """Test that install-buildtools uses bb.fetch with sha256 and passes --local-file"""
+ import shutil
+
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ registry_uri = "git://{};protocol=file;branch=master;rev=master".format(
+ self.registrypath)
+ self.runbbsetup(["settings", "set", "default", "registry", registry_uri])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-bt.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-bt", "gadget"])
+ setuppath = self.get_setup_path('test-config-bt', 'gadget')
+
+ # test config-driven install (url + sha256sum from config)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertTrue(os.path.exists(os.path.join(setuppath, 'buildtools')))
+
+ # test CLI overrides (use a different file to prove precedence)
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'))
+ alt_filename = "alt-buildtools-test.sh"
+ alt_filepath = os.path.join(self.buildtools_dir, alt_filename)
+ with open(alt_filepath, 'w') as f:
+ f.write("#!/bin/sh\necho alt\n")
+ with open(alt_filepath, 'rb') as f:
+ alt_sha256 = hashlib.sha256(f.read()).hexdigest()
+ alt_url = "file://{}/{}".format(self.buildtools_dir, alt_filename)
+ out = self.runbbsetup(["install-buildtools", "--setup-dir", setuppath,
+ "--url", alt_url,
+ "--sha256", alt_sha256])
+ self.assertIn("Buildtools installed into", out[0])
+ self.assertIn("install-buildtools local-file=", out[0])
+ self.assertIn(alt_filename, out[0])
+
+ # test missing sha256sum is a hard error
+ shutil.rmtree(os.path.join(setuppath, 'buildtools'), ignore_errors=True)
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json')) as f:
+ config_upstream = json.load(f)
+ config_upstream['bitbake-config']['install-buildtools'] = {
+ 'url': 'file://{}/{}'.format(self.buildtools_dir, self.buildtools_filename)
+ }
+ with open(os.path.join(setuppath, 'config', 'config-upstream.json'), 'w') as f:
+ json.dump(config_upstream, f)
+ with self.assertRaises(bb.process.ExecutionError):
+ self.runbbsetup(["install-buildtools", "--setup-dir", setuppath, "--force"])
+
+ def test_update_rebase_conflicts_strategy(self):
+ """Test the --rebase-conflicts-strategy option for the update command.
+
+ Covers three scenarios not exercised by test_setup:
+ 1. Uncommitted tracked-file change (LocalModificationsError) + default 'abort'
+ strategy → clean error message containing 'has uncommitted changes' and a
+ hint at --rebase-conflicts-strategy=backup; no backup directory is created.
+ 2. Same uncommitted change + 'backup' strategy → directory is renamed to a
+ timestamped backup and the layer is re-cloned cleanly.
+ 3. Committed local change that conflicts with an incoming upstream commit
+ (RebaseError):
+ a. Default 'abort' strategy → error containing 'Merge conflict' and the
+ --rebase-conflicts-strategy=backup hint; no backup directory is created.
+ b. 'backup' strategy → backup + re-clone instead of a hard failure.
+ """
+ if 'BBPATH' in os.environ:
+ del os.environ['BBPATH']
+ os.chdir(self.tempdir)
+
+ self.runbbsetup(["settings", "set", "default", "registry", "'git://{};protocol=file;branch=master;rev=master'".format(self.registrypath)])
+ self.add_file_to_testrepo('test-file', 'initial\n')
+ self.add_json_config_to_registry('test-config-1.conf.json', 'master', 'master')
+ self.runbbsetup(["init", "--non-interactive", "test-config-1", "gadget"])
+
+ setuppath = self.get_setup_path('test-config-1', 'gadget')
+ layer_path = os.path.join(setuppath, 'layers', 'test-repo')
+ layers_path = os.path.join(setuppath, 'layers')
+
+ # Scenario 1: uncommitted tracked change, default 'abort' strategy
+ # Advance upstream so an update is required.
+ self.add_file_to_testrepo('test-file', 'upstream-v2\n')
+ # Modify the same tracked file in the layer without committing.
+ with open(os.path.join(layer_path, 'test-file'), 'w') as f:
+ f.write('locally-modified\n')
+
+ os.environ['BBPATH'] = os.path.join(setuppath, 'build')
+ with self.assertRaises(bb.process.ExecutionError) as ctx:
+ self.runbbsetup(["update", "--update-bb-conf=no"])
+ self.assertIn('has uncommitted changes', str(ctx.exception))
+ self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception))
+ # No backup directory must have been created.
+ self.assertEqual(self._count_layer_backups(layers_path), 0,
+ "abort strategy must not create any backup")
+
+ # Scenario 2: same uncommitted change, 'backup' strategy
+ out = self.runbbsetup(["update", "--update-bb-conf=no", "--rebase-conflicts-strategy=backup"])
+ # One backup directory must now exist.
+ self.assertEqual(self._count_layer_backups(layers_path), 1,
+ "backup strategy must create exactly one backup")
+ # The re-cloned layer must be clean and at the upstream revision.
+ with open(os.path.join(layer_path, 'test-file')) as f:
+ self.assertEqual(f.read(), 'upstream-v2\n',
+ "re-cloned layer must contain the upstream content")
+ status = self.git(['status', '--porcelain'], cwd=layer_path).strip()
+ self.assertEqual(status, '',
+ "re-cloned layer must have no local modifications")
+ del os.environ['BBPATH']
+
+ # Scenario 3: committed conflicting change, 'backup' strategy
+ # Re-initialise a fresh setup so we start from a clean state.
+ self.runbbsetup(["init", "--non-interactive", "--setup-dir-name", "rebase-conflict-setup", "test-config-1", "gadget"])
+ conflict_setup = os.path.join(self.tempdir, 'bitbake-builds', 'rebase-conflict-setup')
+ conflict_layer = os.path.join(conflict_setup, 'layers', 'test-repo')
+ conflict_layers = os.path.join(conflict_setup, 'layers')
+
+ # Commit a local change that touches the same file as the next upstream commit.
+ with open(os.path.join(conflict_layer, 'test-file'), 'w') as f:
+ f.write('conflicting-local\n')
+ self.git(['add', 'test-file'], cwd=conflict_layer)
+ self.git(['commit', '-m', 'Local conflicting change'], cwd=conflict_layer)
+
+ # Advance upstream with a conflicting edit.
+ self.add_file_to_testrepo('test-file', 'conflicting-upstream\n')
+
+ os.environ['BBPATH'] = os.path.join(conflict_setup, 'build')
+ # Default stop strategy must still fail with a conflict error and include
+ # the --rebase-conflicts-strategy=backup hint (same handler as LocalModificationsError).
+ with self.assertRaises(bb.process.ExecutionError) as ctx:
+ self.runbbsetup(["update", "--update-bb-conf=no"])
+ self.assertIn('Merge conflict in test-file', str(ctx.exception))
+ self.assertIn('--rebase-conflicts-strategy=backup', str(ctx.exception))
+ self.assertEqual(self._count_layer_backups(conflict_layers), 0)
+
+ # Backup strategy must succeed: backup the conflicted dir and re-clone.
+ self.runbbsetup(["update", "--update-bb-conf=no", "--rebase-conflicts-strategy=backup"])
+ self.assertEqual(self._count_layer_backups(conflict_layers), 1,
+ "backup strategy must create exactly one backup after a conflict")
+ with open(os.path.join(conflict_layer, 'test-file')) as f:
+ self.assertEqual(f.read(), 'conflicting-upstream\n',
+ "re-cloned layer must contain the upstream content after conflict backup")
+ del os.environ['BBPATH']
+
+@unittest.skipIf(os.environ.get("BB_SKIP_PYPI_TESTS", "yes") != "no",
+ "PyPI packaging test (set BB_SKIP_PYPI_TESTS=no to run)")
+class PyPIPackagingTest(unittest.TestCase):
+ """Tests for PyPI packaging of bitbake-setup.
+
+ These tests build a wheel from source, install it in an isolated venv,
+ and verify the package works correctly. Skipped by default unless
+ BB_SKIP_PYPI_TESTS=no is set.
+ """
+
+ wheel_path = None
+ build_tempdir = None
+ workspace_dir = None
+
+ @classmethod
+ def setUpClass(cls):
+ """Build wheel once for all tests in this class."""
+ # Locate packaging-pypi/bitbake-setup directory
+ cls.pypi_dir = os.path.abspath(
+ os.path.join(os.path.dirname(__file__), '..', '..', '..', 'packaging-pypi', 'bitbake-setup')
+ )
+
+ # Check for required build tools
+ cls._check_build_tools()
+
+ # Create temporary directory for packaging workspace
+ cls.build_tempdir = tempfile.mkdtemp(prefix="bitbake-pypi-build-")
+
+ # Run package-bitbake-setup.py to create packaging workspace
+ cls._create_packaging_workspace()
+
+ # Build the wheel
+ cls._build_wheel()
+
+ @classmethod
+ def _check_build_tools(cls):
+ """Verify build tools are available, skip if not."""
+ try:
+ result = subprocess.run(
+ [sys.executable, "-m", "build", "--version"],
+ capture_output=True, check=True
+ )
+ except (subprocess.CalledProcessError, FileNotFoundError):
+ raise unittest.SkipTest("'build' package not installed (pip install build)")
+
+ @classmethod
+ def _create_packaging_workspace(cls):
+ """Create packaging workspace using package-bitbake-setup.py."""
+ script = os.path.join(cls.pypi_dir, 'package-bitbake-setup.py')
+ cls.workspace_dir = os.path.join(cls.build_tempdir, 'workspace')
+ result = subprocess.run(
+ [sys.executable, script, '-d', cls.workspace_dir],
+ capture_output=True, text=True
+ )
+ if result.returncode != 0:
+ raise unittest.SkipTest(f"Packaging workspace creation failed: {result.stderr}")
+
+ @classmethod
+ def _build_wheel(cls):
+ """Build the wheel in the packaging workspace."""
+ result = subprocess.run(
+ [sys.executable, "-m", "build", "--wheel"],
+ cwd=cls.workspace_dir,
+ capture_output=True, text=True
+ )
+ if result.returncode != 0:
+ raise unittest.SkipTest(f"Wheel build failed: {result.stderr}")
+
+ # Find the built wheel
+ dist_dir = os.path.join(cls.workspace_dir, 'dist')
+ wheels = glob.glob(os.path.join(dist_dir, '*.whl'))
+ if not wheels:
+ raise unittest.SkipTest("No wheel file found after build")
+ cls.wheel_path = wheels[0]
+
+ @classmethod
+ def tearDownClass(cls):
+ """Clean up the shared wheel build artifacts."""
+ if cls.build_tempdir and os.environ.get("BB_TMPDIR_NOCLEAN") != "yes":
+ shutil.rmtree(cls.build_tempdir, ignore_errors=True)
+ elif cls.build_tempdir:
+ print(f"Not cleaning up {cls.build_tempdir}. Please remove manually.")
+
+ def setUp(self):
+ """Create isolated venv for testing the installed package."""
+ self.venv_dir = tempfile.mkdtemp(prefix="bitbake-pypi-venv-")
+
+ # Create venv without pip (faster, no network needed)
+ venv.create(self.venv_dir, with_pip=False, symlinks=True)
+
+ # Get paths to venv python and bin directory
+ self.venv_python = os.path.join(self.venv_dir, 'bin', 'python')
+ self.venv_bin = os.path.join(self.venv_dir, 'bin')
+
+ # Install wheel using pip from the outer environment (offline, no-deps)
+ site_packages = self._get_site_packages()
+ result = subprocess.run(
+ [sys.executable, "-m", "pip", "install",
+ "--target", site_packages,
+ "--no-deps", "--no-index",
+ self.wheel_path],
+ capture_output=True, text=True
+ )
+ if result.returncode != 0:
+ self.fail(f"Failed to install wheel: {result.stderr}")
+
+ # Install console script entry point manually
+ self._install_console_script()
+
+ def _get_site_packages(self):
+ """Get the site-packages directory in the venv."""
+ lib_dir = os.path.join(self.venv_dir, 'lib')
+ # Find python version directory
+ for d in os.listdir(lib_dir):
+ if d.startswith('python'):
+ return os.path.join(lib_dir, d, 'site-packages')
+ raise RuntimeError("Could not find site-packages in venv")
+
+ def _install_console_script(self):
+ """Create console script wrapper in venv bin directory."""
+ site_packages = self._get_site_packages()
+ script_path = os.path.join(self.venv_bin, 'bitbake-setup')
+ script_content = f'''#!{self.venv_python}
+import sys
+sys.path.insert(0, "{site_packages}")
+from bitbake_setup.__main__ import main
+sys.exit(main())
+'''
+ with open(script_path, 'w') as f:
+ f.write(script_content)
+ os.chmod(script_path, 0o755)
+
+ def tearDown(self):
+ """Remove venv after test."""
+ if os.environ.get("BB_TMPDIR_NOCLEAN") != "yes":
+ shutil.rmtree(self.venv_dir, ignore_errors=True)
+ else:
+ print(f"Not cleaning up {self.venv_dir}. Please remove manually.")
+
+ def test_imports(self):
+ """Verify all expected modules can be imported from installed package."""
+
+ site_packages = self._get_site_packages()
+ imports = ['bb', 'bitbake_setup']
+ for module in imports:
+ result = subprocess.run(
+ [self.venv_python, '-c', f'import sys; sys.path.insert(0, "{site_packages}"); import {module}'],
+ capture_output=True, text=True
+ )
+ self.assertEqual(result.returncode, 0,
+ f"Failed to import {module}: {result.stderr}")
+
+ def test_console_script_help(self):
+ """Verify bitbake-setup --help runs successfully."""
+ script = os.path.join(self.venv_bin, 'bitbake-setup')
+ result = subprocess.run(
+ [script, '--help'],
+ capture_output=True, text=True
+ )
+ self.assertEqual(result.returncode, 0,
+ f"bitbake-setup --help failed: {result.stderr}")
+ self.assertIn('usage:', result.stdout.lower())
+
+ def test_console_script_list(self):
+ """Verify bitbake-setup list runs successfully."""
+ script = os.path.join(self.venv_bin, 'bitbake-setup')
+ result = subprocess.run(
+ [script, 'list'],
+ capture_output=True, text=True
+ )
+ # List may return 0 even with no configurations
+ self.assertEqual(result.returncode, 0,
+ f"bitbake-setup list failed: {result.stderr}")
+
+ def test_package_metadata(self):
+ """Verify package metadata is correctly set."""
+ site_packages = self._get_site_packages()
+ code = '''
+import json
+import sys
+sys.path.insert(0, "{}")
+from importlib.metadata import metadata
+m = metadata("bitbake-setup")
+print(json.dumps({{
+ "name": m["Name"],
+ "version": m["Version"],
+ "requires_python": m.get("Requires-Python", ""),
+ "license": m.get("License", ""),
+}}))
+'''.format(site_packages)
+ result = subprocess.run(
+ [self.venv_python, '-c', code],
+ capture_output=True, text=True
+ )
+ self.assertEqual(result.returncode, 0,
+ f"Failed to get metadata: {result.stderr}")
+
+ meta = json.loads(result.stdout)
+ self.assertEqual(meta['name'], 'bitbake-setup')
+ self.assertIn('>=3.9', meta['requires_python'])
+
+ def test_vendored_dependencies(self):
+ """Verify vendored dependencies (bs4, ply, progressbar, simplediff) are not bundled in package."""
+ # Check that vendored packages do not exist in root of wheel
+ with zipfile.ZipFile(self.wheel_path, 'r') as whl:
+ names = whl.namelist()
+
+ # Check for expected package directories
+ expected = ['bs4/', 'ply/', 'progressbar/', 'simplediff/']
+ for pkg in expected:
+ found = any(n.startswith(pkg) for n in names)
+ self.assertFalse(found,
+ f"Unexpected vendored package '{pkg}' found in wheel")
+
+ def test_version_from_wheel(self):
+ """Verify version is set correctly (not fallback 0.0.0 unless expected)."""
+ import re
+ # Extract version from wheel filename
+ wheel_name = os.path.basename(self.wheel_path)
+ # Wheel format: {name}-{version}(-{build})?-{python}-{abi}-{platform}.whl
+ parts = wheel_name.split('-')
+ version = parts[1]
+
+ # Check version format (should be semver-like or contain git info)
+ version_pattern = r'^\d+\.\d+\.\d+.*$'
+ self.assertTrue(re.match(version_pattern, version),
+ f"Version '{version}' doesn't match expected pattern")
+
+ print(f"Extracted version from wheel: {version}")
+
+ self.assertNotEqual(version, '0.0.0',
+ "Version is fallback 0.0.0 - no git tags found")
+
+ def test_wheel_metadata_file(self):
+ """Verify wheel METADATA file contains required fields."""
+ with zipfile.ZipFile(self.wheel_path, 'r') as whl:
+ # Find METADATA file in dist-info
+ metadata_path = None
+ for name in whl.namelist():
+ if name.endswith('.dist-info/METADATA'):
+ metadata_path = name
+ break
+
+ self.assertIsNotNone(metadata_path, "METADATA file not found in wheel")
+
+ # Parse metadata
+ metadata_content = whl.read(metadata_path).decode('utf-8')
+
+ # Check required fields
+ self.assertIn('Metadata-Version:', metadata_content)
+ self.assertIn('Name: bitbake-setup', metadata_content)
+ self.assertIn('Version:', metadata_content)
+ self.assertIn('Requires-Python:', metadata_content)
+
+ def test_entry_points(self):
+ """Verify console script entry points are correctly defined."""
+ with zipfile.ZipFile(self.wheel_path, 'r') as whl:
+ # Find entry_points.txt in dist-info
+ entry_points_path = None
+ for name in whl.namelist():
+ if name.endswith('.dist-info/entry_points.txt'):
+ entry_points_path = name
+ break
+
+ self.assertIsNotNone(entry_points_path,
+ "entry_points.txt not found in wheel")
+
+ content = whl.read(entry_points_path).decode('utf-8')
+ self.assertIn('[console_scripts]', content)
+ self.assertIn('bitbake-setup', content)
+ self.assertIn('bitbake_setup.__main__:main', content)
diff --git a/bitbake/lib/bb/tests/siggen.py b/bitbake/lib/bb/tests/siggen.py
index 0dc67e6c..eb07cc92 100644
--- a/bitbake/lib/bb/tests/siggen.py
+++ b/bitbake/lib/bb/tests/siggen.py
@@ -9,7 +9,9 @@
import unittest
import logging
import bb
+import bb.data
import time
+from contextlib import contextmanager
logger = logging.getLogger('BitBake.TestSiggen')
@@ -26,3 +28,48 @@ class SiggenTest(unittest.TestCase):
for t in tests:
self.assertEqual(bb.siggen.build_pnid(*t), tests[t])
+ def test_get_unihashes_rejects_invalid_hashserv_unihash(self):
+ class TestClient:
+ def get_unihash_batch(self, query):
+ list(query)
+ return ["${@os.system('true')}"]
+
+ class TestSiggen(bb.siggen.SignatureGeneratorUniHashMixIn):
+ def __init__(self):
+ self.server = "test-server"
+ self.method = "test-method"
+ self.extramethod = {}
+ self.taskhash = {"test.bb:do_compile": "a" * 64}
+ self.unihash = {}
+ self.unitaskhashes = {}
+ self.tidtopn = {}
+ self.setscenetasks = set()
+
+ @contextmanager
+ def client(self):
+ yield TestClient()
+
+ siggen = TestSiggen()
+
+ with self.assertRaises(bb.BBHandledException):
+ siggen.get_unihashes(["test.bb:do_compile"])
+
+ self.assertEqual(siggen.unihash, {})
+ self.assertEqual(siggen.unitaskhashes, {})
+
+ def test_report_unihash_reads_bb_unihash_without_expansion(self):
+ class TestSiggen(bb.siggen.SignatureGeneratorUniHashMixIn):
+ def __init__(self):
+ self.setscenetasks = set()
+ self.taskhash = {"test.bb:do_compile": "b" * 64}
+
+ d = bb.data.init()
+ d.setVar("BB_TASKHASH", "a" * 64)
+ d.setVar("BB_UNIHASH", "${@d.setVar('EXPANDED_UNIHASH', '1') or 'bad'}")
+ d.setVar("SSTATE_HASHEQUIV_REPORT_TASKDATA", "0")
+ d.setVar("T", "/tmp")
+ d.setVar("BB_FILENAME", "test.bb")
+
+ TestSiggen().report_unihash(".", "compile", d)
+
+ self.assertIsNone(d.getVar("EXPANDED_UNIHASH"))
diff --git a/bitbake/lib/bb/tests/support/httpserver.py b/bitbake/lib/bb/tests/support/httpserver.py
index 78f76600..03327e92 100644
--- a/bitbake/lib/bb/tests/support/httpserver.py
+++ b/bitbake/lib/bb/tests/support/httpserver.py
@@ -3,7 +3,7 @@
#
import http.server
-import multiprocessing
+from bb import multiprocessing
import os
import traceback
import signal
@@ -43,7 +43,7 @@ class HTTPService(object):
self.process = multiprocessing.Process(target=self.server.server_start, args=[self.root_dir, self.logger])
# The signal handler from testimage.bbclass can cause deadlocks here
- # if the HTTPServer is terminated before it can restore the standard
+ # if the HTTPServer is terminated before it can restore the standard
#signal behaviour
orig = signal.getsignal(signal.SIGTERM)
signal.signal(signal.SIGTERM, signal.SIG_DFL)
diff --git a/bitbake/lib/bb/tests/utils.py b/bitbake/lib/bb/tests/utils.py
index c363f62d..52b7bf85 100644
--- a/bitbake/lib/bb/tests/utils.py
+++ b/bitbake/lib/bb/tests/utils.py
@@ -130,6 +130,14 @@ class Checksum(unittest.TestCase):
checksum = bb.utils.sha256_file(f.name)
self.assertEqual(checksum, "fcfbae8bf6b721dbb9d2dc6a9334a58f2031a9a9b302999243f99da4d7f12d0f")
+ def test_goh1(self):
+ import hashlib
+ with tempfile.NamedTemporaryFile() as f:
+ f.write(self.filler)
+ f.flush()
+ checksum = bb.utils.goh1_file(f.name)
+ self.assertEqual(checksum, "81191f04d4abf413e5badd234814e4202d9efa73e6f9437e9ddd6b8165b569ef")
+
class EditMetadataFile(unittest.TestCase):
_origfile = """
# A comment
@@ -684,3 +692,14 @@ class EnvironmentTests(unittest.TestCase):
self.assertIn("A", os.environ)
self.assertEqual(os.environ["A"], "this is A")
self.assertNotIn("B", os.environ)
+
+class FilemodeTests(unittest.TestCase):
+ def test_filemode_convert(self):
+ self.assertEqual(0o775, bb.utils.to_filemode("0o775"))
+ self.assertEqual(0o775, bb.utils.to_filemode(0o775))
+ self.assertEqual(0o775, bb.utils.to_filemode("775"))
+ with self.assertRaises(ValueError):
+ bb.utils.to_filemode("xyz")
+ with self.assertRaises(ValueError):
+ bb.utils.to_filemode("999")
+
diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
index dcd3910c..9b9925e5 100644
--- a/bitbake/lib/bb/tinfoil.py
+++ b/bitbake/lib/bb/tinfoil.py
@@ -14,7 +14,8 @@ import time
import atexit
import re
from collections import OrderedDict, defaultdict
-from functools import partial
+from functools import partial, wraps
+from contextlib import contextmanager
import bb.cache
import bb.cooker
@@ -24,7 +25,136 @@ import bb.utils
import bb.command
import bb.remotedata
from bb.main import setup_bitbake, BitBakeConfigParameters
-import bb.fetch2
+import bb.fetch
+
+def wait_for(f):
+ """
+ Wrap a function that makes an asynchronous tinfoil call using
+ self.run_command() and wait for events to say that the call has been
+ successful, or an error has occurred.
+ """
+ @wraps(f)
+ def wrapper(self, *args, handle_events=True, extra_events=None, event_callback=None, **kwargs):
+ if handle_events:
+ # A reasonable set of default events matching up with those we handle below
+ eventmask = [
+ 'bb.event.BuildStarted',
+ 'bb.event.BuildCompleted',
+ 'logging.LogRecord',
+ 'bb.event.NoProvider',
+ 'bb.command.CommandCompleted',
+ 'bb.command.CommandFailed',
+ 'bb.build.TaskStarted',
+ 'bb.build.TaskFailed',
+ 'bb.build.TaskSucceeded',
+ 'bb.build.TaskFailedSilent',
+ 'bb.build.TaskProgress',
+ 'bb.runqueue.runQueueTaskStarted',
+ 'bb.runqueue.sceneQueueTaskStarted',
+ 'bb.event.ProcessStarted',
+ 'bb.event.ProcessProgress',
+ 'bb.event.ProcessFinished',
+ ]
+ if extra_events:
+ eventmask.extend(extra_events)
+ ret = self.set_event_mask(eventmask)
+
+ includelogs = self.config_data.getVar('BBINCLUDELOGS')
+ loglines = self.config_data.getVar('BBINCLUDELOGS_LINES')
+
+ # Call actual function
+ ret = f(self, *args, **kwargs)
+
+ if handle_events:
+ lastevent = time.time()
+ result = False
+ # Borrowed from knotty, instead somewhat hackily we use the helper
+ # as the object to store "shutdown" on
+ helper = bb.ui.uihelper.BBUIHelper()
+ helper.shutdown = 0
+ parseprogress = None
+ termfilter = bb.ui.knotty.TerminalFilter(helper, helper, self.logger.handlers, quiet=self.quiet)
+ try:
+ while True:
+ try:
+ event = self.wait_event(0.25)
+ if event:
+ lastevent = time.time()
+ if event_callback and event_callback(event):
+ continue
+ if helper.eventHandler(event):
+ if isinstance(event, bb.build.TaskFailedSilent):
+ self.logger.warning("Logfile for failed setscene task is %s" % event.logfile)
+ elif isinstance(event, bb.build.TaskFailed):
+ bb.ui.knotty.print_event_log(event, includelogs, loglines, termfilter)
+ continue
+ if isinstance(event, bb.event.ProcessStarted):
+ if self.quiet > 1:
+ continue
+ if parseprogress:
+ parseprogress.finish()
+ parseprogress = bb.ui.knotty.new_progress(event.processname, event.total)
+ parseprogress.start(False)
+ continue
+ if isinstance(event, bb.event.ProcessProgress):
+ if self.quiet > 1:
+ continue
+ if parseprogress and parseprogress.id == event.processname:
+ parseprogress.update(event.progress)
+ continue
+ if isinstance(event, bb.event.ProcessFinished):
+ if self.quiet > 1:
+ continue
+ if parseprogress and parseprogress.id == event.processname:
+ parseprogress.finish()
+ parseprogress = None
+ continue
+ if isinstance(event, bb.command.CommandCompleted):
+ result = True
+ break
+ if isinstance(event, (bb.command.CommandFailed, bb.command.CommandExit)):
+ self.logger.error(str(event))
+ result = False
+ break
+ if isinstance(event, logging.LogRecord):
+ if event.taskpid == 0 or event.levelno > logging.INFO:
+ self.logger.handle(event)
+ continue
+ if isinstance(event, bb.event.NoProvider):
+ self.logger.error(str(event))
+ result = False
+ break
+ elif helper.shutdown > 1:
+ break
+ termfilter.updateFooter()
+ if time.time() > (lastevent + (3*60)):
+ if not self.run_command('ping', handle_events=False):
+ print("\nUnable to ping server and no events, closing down...\n")
+ return False
+ except KeyboardInterrupt:
+ termfilter.clearFooter()
+ if helper.shutdown == 1:
+ print("\nSecond Keyboard Interrupt, stopping...\n")
+ ret = self.run_command("stateForceShutdown")
+ if ret and ret[2]:
+ self.logger.error("Unable to cleanly stop: %s" % ret[2])
+ elif helper.shutdown == 0:
+ print("\nKeyboard Interrupt, closing down...\n")
+ interrupted = True
+ ret = self.run_command("stateShutdown")
+ if ret and ret[2]:
+ self.logger.error("Unable to cleanly shutdown: %s" % ret[2])
+ helper.shutdown = helper.shutdown + 1
+ termfilter.clearFooter()
+ finally:
+ termfilter.finish()
+ if helper.failed_tasks:
+ result = False
+ return result
+ else:
+ return ret
+
+ return wrapper
# We need this in order to shut down the connection to the bitbake server,
@@ -188,11 +318,19 @@ class TinfoilCookerAdapter:
self._cache[name] = attrvalue
return attrvalue
+ class TinfoilSkiplistByMcAdapter:
+ def __init__(self, tinfoil):
+ self.tinfoil = tinfoil
+
+ def __getitem__(self, mc):
+ return self.tinfoil.get_skipped_recipes(mc)
+
def __init__(self, tinfoil):
self.tinfoil = tinfoil
self.multiconfigs = [''] + (tinfoil.config_data.getVar('BBMULTICONFIG') or '').split()
self.collections = {}
self.recipecaches = {}
+ self.skiplist_by_mc = self.TinfoilSkiplistByMcAdapter(tinfoil)
for mc in self.multiconfigs:
self.collections[mc] = self.TinfoilCookerCollectionAdapter(tinfoil, mc)
self.recipecaches[mc] = self.TinfoilRecipeCacheAdapter(tinfoil, mc)
@@ -201,8 +339,6 @@ class TinfoilCookerAdapter:
# Grab these only when they are requested since they aren't always used
if name in self._cache:
return self._cache[name]
- elif name == 'skiplist':
- attrvalue = self.tinfoil.get_skipped_recipes()
elif name == 'bbfile_config_priorities':
ret = self.tinfoil.run_command('getLayerPriorities')
bbfile_config_priorities = []
@@ -514,12 +650,12 @@ class Tinfoil:
"""
return defaultdict(list, self.run_command('getOverlayedRecipes', mc))
- def get_skipped_recipes(self):
+ def get_skipped_recipes(self, mc=''):
"""
Find recipes which were skipped (i.e. SkipRecipe was raised
during parsing).
"""
- return OrderedDict(self.run_command('getSkippedRecipes'))
+ return OrderedDict(self.run_command('getSkippedRecipes', mc))
def get_all_providers(self, mc=''):
return defaultdict(list, self.run_command('allProviders', mc))
@@ -533,6 +669,7 @@ class Tinfoil:
def get_runtime_providers(self, rdep):
return self.run_command('getRuntimeProviders', rdep)
+ # TODO: teach this method about mc
def get_recipe_file(self, pn):
"""
Get the file name for the specified recipe/target. Raises
@@ -541,13 +678,21 @@ class Tinfoil:
"""
best = self.find_best_provider(pn)
if not best or (len(best) > 3 and not best[3]):
+ # TODO: pass down mc
skiplist = self.get_skipped_recipes()
taskdata = bb.taskdata.TaskData(None, skiplist=skiplist)
skipreasons = taskdata.get_reasons(pn)
if skipreasons:
raise bb.providers.NoProvider('%s is unavailable:\n %s' % (pn, ' \n'.join(skipreasons)))
else:
- raise bb.providers.NoProvider('Unable to find any recipe file matching "%s"' % pn)
+ msg = f'Unable to find any recipe file matching "{pn}"'
+ import difflib
+ providers = self.get_all_providers()
+ close_matches = difflib.get_close_matches(pn, providers, cutoff=0.7)
+ if close_matches:
+ close_matches = "\n ".join(close_matches)
+ msg += f'. Close matches:\n {close_matches}'
+ raise bb.providers.NoProvider(msg)
return best[3]
def get_file_appends(self, fn, mc=''):
@@ -633,6 +778,29 @@ class Tinfoil:
fn = self.get_recipe_file(pn)
return self.parse_recipe_file(fn)
+ @contextmanager
+ def _data_tracked_if_enabled(self):
+ """
+ A context manager to enable data tracking for a code segment if data
+ tracking was enabled for this tinfoil instance.
+ """
+ if self.tracking:
+ # Enable history tracking just for the operation
+ self.run_command('enableDataTracking')
+
+ # Here goes the operation with the optional data tracking
+ yield
+
+ if self.tracking:
+ self.run_command('disableDataTracking')
+
+ def finalizeData(self):
+ """
+ Run anonymous functions and expand keys
+ """
+ with self._data_tracked_if_enabled():
+ return self._reconvert_type(self.run_command('finalizeData'), 'DataStoreConnectionHandle')
+
def parse_recipe_file(self, fn, appends=True, appendlist=None, config_data=None):
"""
Parse the specified recipe file (with or without bbappends)
@@ -645,10 +813,7 @@ class Tinfoil:
appendlist: optional list of bbappend files to apply, if you
want to filter them
"""
- if self.tracking:
- # Enable history tracking just for the parse operation
- self.run_command('enableDataTracking')
- try:
+ with self._data_tracked_if_enabled():
if appends and appendlist == []:
appends = False
if config_data:
@@ -660,19 +825,39 @@ class Tinfoil:
return self._reconvert_type(dscon, 'DataStoreConnectionHandle')
else:
return None
- finally:
- if self.tracking:
- self.run_command('disableDataTracking')
- def build_file(self, buildfile, task, internal=True):
+ def build_file(self, buildfile, task, internal=True, taskonly=False):
"""
Runs the specified task for just a single recipe (i.e. no dependencies).
This is equivalent to bitbake -b, except with the default internal=True
no warning about dependencies will be produced, normal info messages
from the runqueue will be silenced and BuildInit, BuildStarted and
BuildCompleted events will not be fired.
+ With taskonly=True the recipe's own task ordering is dropped as well, so
+ only the requested task runs.
+ """
+ return self.run_command('buildFile', buildfile, task, internal, taskonly)
+
+ @wait_for
+ def build_file_sync(self, *args):
+ self.build_file(*args)
+
+ def run_prepared_task(self, recipe, task):
+ """Run *task* for one parsed recipe, and nothing else.
+
+ No dependencies are resolved and no other task of the recipe is run,
+ so everything the task consumes must already be in place. The task
+ runs through the normal BitBake worker path, including fakeroot setup
+ and dispatch of shell or Python task bodies.
+
+ The task is forced to execute rather than being skipped as up to date.
+ Returns False if the task failed.
"""
- return self.run_command('buildFile', buildfile, task, internal)
+ self.run_command('setConfig', 'force', True)
+ try:
+ return self.build_file_sync(self.get_recipe_file(recipe), task, True, True)
+ finally:
+ self.run_command('setConfig', 'force', False)
def build_targets(self, targets, task=None, handle_events=True, extra_events=None, event_callback=None):
"""
@@ -761,23 +946,23 @@ class Tinfoil:
if isinstance(event, bb.event.ProcessStarted):
if self.quiet > 1:
continue
+ if parseprogress:
+ parseprogress.finish()
parseprogress = bb.ui.knotty.new_progress(event.processname, event.total)
parseprogress.start(False)
continue
if isinstance(event, bb.event.ProcessProgress):
if self.quiet > 1:
continue
- if parseprogress:
+ if parseprogress and parseprogress.id == event.processname:
parseprogress.update(event.progress)
- else:
- bb.warn("Got ProcessProgress event for something that never started?")
continue
if isinstance(event, bb.event.ProcessFinished):
if self.quiet > 1:
continue
- if parseprogress:
+ if parseprogress and parseprogress.id == event.processname:
parseprogress.finish()
- parseprogress = None
+ parseprogress = None
continue
if isinstance(event, bb.command.CommandCompleted):
result = True
diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 8b212b78..4ee45d67 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -559,7 +559,10 @@ class ORMWrapper(object):
# we might have an invalid link; no way to detect this. just set it to None
filetarget_obj = None
- parent_obj = Target_File.objects.get(target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
+ try:
+ parent_obj = Target_File.objects.get(target = target_obj, path = parent_path, inodetype = Target_File.ITYPE_DIRECTORY)
+ except Target_File.DoesNotExist:
+ parent_obj = None
Target_File.objects.create(
target = target_obj,
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index f86999bb..1a3198b5 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -10,10 +10,11 @@
from __future__ import division
+import io
import os
import sys
import logging
-import progressbar
+from bb._vendor import progressbar
import signal
import bb.msg
import time
@@ -24,6 +25,12 @@ import atexit
from itertools import groupby
from bb.ui import uihelper
+import bb.build
+import bb.command
+import bb.cooker
+import bb.event
+import bb.runqueue
+import bb.utils
featureSet = [bb.cooker.CookerFeatures.SEND_SANITYEVENTS, bb.cooker.CookerFeatures.BASEDATASTORE_TRACKING]
@@ -32,6 +39,7 @@ interactive = sys.stdout.isatty()
class BBProgress(progressbar.ProgressBar):
def __init__(self, msg, maxval, widgets=None, extrapos=-1, resize_handler=None):
+ self.id = msg
self.msg = msg
self.extrapos = extrapos
if not widgets:
@@ -77,7 +85,9 @@ class NonInteractiveProgress(object):
fobj = sys.stdout
def __init__(self, msg, maxval):
+ self.id = msg
self.msg = msg
+ self.currval = 0
self.maxval = maxval
self.finished = False
@@ -87,6 +97,10 @@ class NonInteractiveProgress(object):
return self
def update(self, value):
+ self.currval = value
+ pass
+
+ def clear(self):
pass
def finish(self):
@@ -103,23 +117,42 @@ def new_progress(msg, maxval):
return NonInteractiveProgress(msg, maxval)
def pluralise(singular, plural, qty):
- if(qty == 1):
+ if qty == 1:
return singular % qty
else:
return plural % qty
+def get_pressure_message(pressure_state, pressure_values):
+ cpu_pressure, io_pressure, mem_pressure = pressure_state
+ pressure_strs = []
+ if cpu_pressure:
+ pressure_strs.append("CPU pressure (>%sus)" % pressure_values[1])
+ if io_pressure:
+ pressure_strs.append("I/O pressure (>%sus)" % pressure_values[3])
+ if mem_pressure:
+ pressure_strs.append("MEM pressure (>%sus)" % pressure_values[5])
+
+ if not pressure_strs:
+ pressure_strs.append("No pressure")
+
+ return "%s is limiting task startup" % ", ".join(pressure_strs)
class InteractConsoleLogFilter(logging.Filter):
def __init__(self, tf):
self.tf = tf
+ super().__init__()
def filter(self, record):
- if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ") or "limiting task startup" in record.msg):
return False
self.tf.clearFooter()
return True
class TerminalFilter(object):
+
+ # 40 Hz (FPS) -> 0.025 secs
+ _DEFAULT_PRINT_INTERVAL = 0.025
+
rows = 25
columns = 80
@@ -131,7 +164,7 @@ class TerminalFilter(object):
def getTerminalColumns(self):
def ioctl_GWINSZ(fd):
try:
- cr = struct.unpack('hh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, '1234'))
+ cr = struct.unpack('hhhh', fcntl.ioctl(fd, self.termios.TIOCGWINSZ, b'12345678'))[0:2]
except:
return None
return cr
@@ -145,7 +178,7 @@ class TerminalFilter(object):
pass
if not cr:
try:
- cr = (os.environ['LINES'], os.environ['COLUMNS'])
+ cr = (int(os.environ['LINES']), int(os.environ['COLUMNS']))
except:
cr = (25, 80)
return cr
@@ -157,10 +190,12 @@ class TerminalFilter(object):
self.stdinbackup = None
self.interactive = sys.stdout.isatty()
self.footer_present = False
- self.lastpids = []
- self.lasttime = None
+ self.lasttime = time.time()
self.quiet = quiet
+ self._footer_buf = io.StringIO()
+ self._footer_lines = None
+
if not self.interactive:
return
@@ -208,11 +243,11 @@ class TerminalFilter(object):
def clearFooter(self):
if self.footer_present:
- lines = self.footer_present
- sys.stdout.buffer.write(self.curses.tparm(self.cuu, lines))
+ sys.stdout.buffer.write(self.curses.tparm(self.cuu, self._footer_lines))
sys.stdout.buffer.write(self.curses.tparm(self.ed))
sys.stdout.flush()
self.footer_present = False
+ self._footer_lines = None
def elapsed(self, sec):
hrs = int(sec / 3600.0)
@@ -228,9 +263,10 @@ class TerminalFilter(object):
def keepAlive(self, t):
if not self.cuu:
- print("Bitbake still alive (no events for %ds). Active tasks:" % t)
+ msgbuf = ["Bitbake still alive (no events for %ds). Active tasks:" % t]
for t in self.helper.running_tasks:
- print(t)
+ msgbuf.append(str(t))
+ print("\n".join(msgbuf))
sys.stdout.flush()
def updateFooter(self):
@@ -238,29 +274,44 @@ class TerminalFilter(object):
return
activetasks = self.helper.running_tasks
failedtasks = self.helper.failed_tasks
- runningpids = self.helper.running_pids
currenttime = time.time()
- if not self.lasttime or (currenttime - self.lasttime > 5):
+ deltatime = currenttime - self.lasttime
+
+ if (deltatime > 5.0):
self.helper.needUpdate = True
- self.lasttime = currenttime
- if self.footer_present and not self.helper.needUpdate:
+ need_update = self.helper.needUpdate
+ else:
+ # Do not update faster than _DEFAULT_PRINT_INTERVAL
+ # to avoid heavy print() flooding.
+ need_update = self.helper.needUpdate and (deltatime > self._DEFAULT_PRINT_INTERVAL)
+
+ if self.footer_present and not need_update:
+ # Footer update is not needed.
return
+
+ # Remember the time when the footer was last updated.
+ self.lasttime = currenttime
+
self.helper.needUpdate = False
- if self.footer_present:
- self.clearFooter()
if (not self.helper.tasknumber_total or self.helper.tasknumber_current == self.helper.tasknumber_total) and not len(activetasks):
+ self.clearFooter()
return
+
+ # Clear footer buffer.
+ self._footer_buf.truncate(0)
+ self._footer_buf.seek(0)
+
tasks = []
- for t in runningpids:
- start_time = activetasks[t].get("starttime", None)
+ for task in activetasks.values():
+ start_time = task.get("starttime", None)
if start_time:
- msg = "%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), activetasks[t]["pid"])
+ msg = "%s - %s (pid %s)" % (task["title"], self.elapsed(currenttime - start_time), task["pid"])
else:
- msg = "%s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"])
- progress = activetasks[t].get("progress", None)
+ msg = "%s (pid %s)" % (task["title"], task["pid"])
+ progress = task.get("progress", None)
if progress is not None:
- pbar = activetasks[t].get("progressbar", None)
- rate = activetasks[t].get("rate", None)
+ pbar = task.get("progressbar", None)
+ rate = task.get("rate", None)
if not pbar or pbar.bouncing != (progress < 0):
if progress < 0:
pbar = BBProgress("0: %s" % msg, 100, widgets=[' ', progressbar.BouncingSlider(), ''], extrapos=3, resize_handler=self.sigwinch_handle)
@@ -268,7 +319,8 @@ class TerminalFilter(object):
else:
pbar = BBProgress("0: %s" % msg, 100, widgets=[' ', progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=5, resize_handler=self.sigwinch_handle)
pbar.bouncing = False
- activetasks[t]["progressbar"] = pbar
+ pbar.fd = self._footer_buf
+ task["progressbar"] = pbar
tasks.append((pbar, msg, progress, rate, start_time))
else:
tasks.append(msg)
@@ -278,7 +330,7 @@ class TerminalFilter(object):
"Waiting for %s running tasks to finish", len(activetasks))
if not self.quiet:
content += ':'
- print(content)
+ print(content, file=self._footer_buf)
else:
scene_tasks = "%s of %s" % (self.helper.setscene_current, self.helper.setscene_total)
cur_tasks = "%s of %s" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
@@ -287,7 +339,12 @@ class TerminalFilter(object):
if not self.quiet:
msg = "Setscene tasks: %s" % scene_tasks
content += msg + "\n"
- print(msg)
+ print(msg, file=self._footer_buf)
+
+ if any(self.helper.pressure_state):
+ msg = get_pressure_message(self.helper.pressure_state, self.helper.pressure_values)
+ content += msg + "\n"
+ print(msg, file=self._footer_buf)
if self.quiet:
msg = "Running tasks (%s, %s)" % (scene_tasks, cur_tasks)
@@ -297,13 +354,14 @@ class TerminalFilter(object):
msg = "Currently %2s running tasks (%s)" % (len(activetasks), cur_tasks)
maxtask = self.helper.tasknumber_total
if not self.main_progress or self.main_progress.maxval != maxtask:
- widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()]
+ widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar(), ' ', progressbar.Timer()]
self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets, resize_handler=self.sigwinch_handle)
+ self.main_progress.fd = self._footer_buf
self.main_progress.start(False)
self.main_progress.setmessage(msg)
progress = max(0, self.helper.tasknumber_current - 1)
content += self.main_progress.update(progress)
- print('')
+ print('', file=self._footer_buf)
lines = self.getlines(content)
if not self.quiet:
for tasknum, task in enumerate(tasks[:(self.rows - 1 - lines)]):
@@ -319,15 +377,19 @@ class TerminalFilter(object):
content = pbar.update(progress)
else:
content = pbar.update(1)
- print('')
+ print('', file=self._footer_buf)
else:
content = "%s: %s" % (tasknum, task)
- print(content)
+ print(content, file=self._footer_buf)
lines = lines + self.getlines(content)
- self.footer_present = lines
- self.lastpids = runningpids[:]
self.lastcount = self.helper.tasknumber_current
+ # Clear footer and Print buffer.
+ self.clearFooter()
+ print(self._footer_buf.getvalue(), end='')
+ self._footer_lines = lines
+ self.footer_present = True
+
def getlines(self, content):
lines = 0
for line in content.split("\n"):
@@ -335,18 +397,18 @@ class TerminalFilter(object):
return lines
def finish(self):
+ self._footer_buf.close()
if self.stdinbackup:
fd = sys.stdin.fileno()
self.termios.tcsetattr(fd, self.termios.TCSADRAIN, self.stdinbackup)
def print_event_log(event, includelogs, loglines, termfilter):
- # FIXME refactor this out further
logfile = event.logfile
if logfile and os.path.exists(logfile):
termfilter.clearFooter()
bb.error("Logfile of failure stored in: %s" % logfile)
if includelogs and not event.errprinted:
- print("Log data follows:")
+ msgbuf = ["Log data follows:"]
f = open(logfile, "r")
lines = []
while True:
@@ -359,11 +421,11 @@ def print_event_log(event, includelogs, loglines, termfilter):
if len(lines) > int(loglines):
lines.pop(0)
else:
- print('| %s' % l)
+ msgbuf.append('| %s' % l)
f.close()
if lines:
- for line in lines:
- print(line)
+ msgbuf.extend(lines)
+ print("\n".join(msgbuf))
def _log_settings_from_server(server, observe_only):
# Get values of variables which control our output
@@ -397,7 +459,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
"bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
"bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
"bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
- "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished"]
+ "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.event.ProcessProgress", "bb.event.ProcessFinished",
+ "bb.runqueue.PSIEvent"]
def drain_events_errorhandling(eventHandler):
# We don't have logging setup, we do need to show any events we see before exiting
@@ -551,17 +614,31 @@ def main(server, eventHandler, params, tf = TerminalFilter):
"BitBake.RunQueue.HashEquiv": {
"level": "VERBOSE",
"handlers": ["BitBake.verbconsolelog"],
+ },
+ "BitBake.RunQueue.PSI": {
+ "level": "VERBOSE",
+ "handlers": ["BitBake.verbconsolelog"],
}
}
})
- bb.utils.mkdirhier(os.path.dirname(consolelogfile))
- loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+ consolelogdirname = os.path.dirname(consolelogfile)
+ # `bb.utils.mkdirhier` has this check, but it reports failure using bb.fatal, which logs
+ # to the very logger we are trying to set up.
+ if '${' in str(consolelogdirname):
+ print(
+ "FATAL: Directory name {} contains unexpanded bitbake variable. This may cause build failures and WORKDIR pollution.".format(
+ consolelogdirname))
+ if '${MACHINE}' in consolelogdirname:
+ print("HINT: It looks like you forgot to set MACHINE in local.conf.")
+
+ bb.utils.mkdirhier(consolelogdirname)
+ loglink = os.path.join(consolelogdirname, 'console-latest.log')
bb.utils.remove(loglink)
try:
- os.symlink(os.path.basename(consolelogfile), loglink)
+ os.symlink(os.path.basename(consolelogfile), loglink)
except OSError:
- pass
+ pass
# Add the logging domains specified by the user on the command line
for (domainarg, iterator) in groupby(params.debug_domains):
@@ -577,6 +654,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
else:
log_exec_tty = False
+ should_print_hyperlinks = sys.stdout.isatty() and os.environ.get('NO_COLOR', '') == ''
+
helper = uihelper.BBUIHelper()
# Look for the specially designated handlers which need to be passed to the
@@ -640,7 +719,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
return_value = 0
errors = 0
warnings = 0
- taskfailures = []
+ taskfailures = {}
printintervaldelta = 10 * 60 # 10 minutes
printinterval = printintervaldelta
@@ -717,7 +796,12 @@ def main(server, eventHandler, params, tf = TerminalFilter):
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn') and event.levelno not in [bb.msg.BBLogFormatter.WARNONCE, bb.msg.BBLogFormatter.ERRORONCE]:
event.msg = event.fn + ': ' + event.msg
+ # Need to remove any progress bar, then add it back after we print this message
+ if parseprogress:
+ parseprogress.clear()
logging.getLogger(event.name).handle(event)
+ if parseprogress:
+ parseprogress.update(parseprogress.currval)
continue
if isinstance(event, bb.build.TaskFailedSilent):
@@ -726,6 +810,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.build.TaskFailed):
return_value = 1
print_event_log(event, includelogs, loglines, termfilter)
+ k = "{}:{}".format(event._fn, event._task)
+ taskfailures[k] = event.logfile
if isinstance(event, bb.build.TaskBase):
logger.info(event._message)
continue
@@ -821,7 +907,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.runqueue.runQueueTaskFailed):
return_value = 1
- taskfailures.append(event.taskstring)
+ taskfailures.setdefault(event.taskstring)
logger.error(str(event))
continue
@@ -836,23 +922,28 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if params.options.quiet > 1:
continue
termfilter.clearFooter()
+ if parseprogress:
+ parseprogress.finish()
parseprogress = new_progress(event.processname, event.total)
parseprogress.start(False)
continue
if isinstance(event, bb.event.ProcessProgress):
if params.options.quiet > 1:
continue
- if parseprogress:
+ if parseprogress and parseprogress.id == event.processname:
parseprogress.update(event.progress)
- else:
- bb.warn("Got ProcessProgress event for someting that never started?")
continue
if isinstance(event, bb.event.ProcessFinished):
if params.options.quiet > 1:
continue
- if parseprogress:
+ if parseprogress and parseprogress.id == event.processname:
parseprogress.finish()
- parseprogress = None
+ parseprogress = None
+ continue
+ if isinstance(event, bb.runqueue.PSIEvent):
+ if params.options.quiet > 1:
+ continue
+ logger.info(get_pressure_message(event.pressure_state, event.pressure_values))
continue
# ignore
@@ -942,11 +1033,21 @@ def main(server, eventHandler, params, tf = TerminalFilter):
try:
termfilter.clearFooter()
summary = ""
+ def format_hyperlink(url, link_text):
+ if should_print_hyperlinks:
+ start = f'\033]8;;{url}\033\\'
+ end = '\033]8;;\033\\'
+ return f'{start}{link_text}{end}'
+ return link_text
+
if taskfailures:
summary += pluralise("\nSummary: %s task failed:",
"\nSummary: %s tasks failed:", len(taskfailures))
- for failure in taskfailures:
+ for (failure, log_file) in taskfailures.items():
summary += "\n %s" % failure
+ if log_file:
+ hyperlink = format_hyperlink(f"file://{log_file}", log_file)
+ summary += "\n log: {}".format(hyperlink)
if warnings:
summary += pluralise("\nSummary: There was %s WARNING message.",
"\nSummary: There were %s WARNING messages.", warnings)
diff --git a/bitbake/lib/bb/ui/ncurses.py b/bitbake/lib/bb/ui/ncurses.py
index 18a70654..228fa9a6 100644
--- a/bitbake/lib/bb/ui/ncurses.py
+++ b/bitbake/lib/bb/ui/ncurses.py
@@ -291,7 +291,7 @@ class NCursesUI:
# bb.error("log data follows (%s)" % logfile)
# number_of_lines = data.getVar("BBINCLUDELOGS_LINES", d)
# if number_of_lines:
-# subprocess.check_call('tail -n%s %s' % (number_of_lines, logfile), shell=True)
+# subprocess.check_call(['tail', '-n' + number_of_lines, logfile])
# else:
# f = open(logfile, "r")
# while True:
diff --git a/bitbake/lib/bb/ui/taskexp_ncurses.py b/bitbake/lib/bb/ui/taskexp_ncurses.py
index ea94a498..551ad142 100755
--- a/bitbake/lib/bb/ui/taskexp_ncurses.py
+++ b/bitbake/lib/bb/ui/taskexp_ncurses.py
@@ -146,6 +146,9 @@ def debug_frame(nbox_ojb):
unit_test = os.environ.get('TASK_EXP_UNIT_TEST')
unit_test_cmnds=[
'# Default selected task in primary box',
+ 'tst_selected=<TASK>.do_create_recipe_spdx',
+ '# Move to next entry, more predictable',
+ 'tst_entry=<DOWN>', # optional injected error
'tst_selected=<TASK>.do_recipe_qa',
'# Default selected task in deps',
'tst_entry=<TAB>',
diff --git a/bitbake/lib/bb/ui/teamcity.py b/bitbake/lib/bb/ui/teamcity.py
index 7eeaab8d..3177f4aa 100644
--- a/bitbake/lib/bb/ui/teamcity.py
+++ b/bitbake/lib/bb/ui/teamcity.py
@@ -200,6 +200,7 @@ def main(server, eventHandler, params):
logger.error("XMLRPC Fault getting commandline: {0}".format(x))
return 1
+ active_process = None
active_process_total = None
is_tasks_running = False
@@ -300,16 +301,23 @@ def main(server, eventHandler, params):
if isinstance(event, bb.event.ProcessStarted):
if event.processname in ["Initialising tasks", "Checking sstate mirror object availability"]:
+ if active_process:
+ ui.progress(active_process, 100)
+ ui.block_end()
+ active_process = event.processname
active_process_total = event.total
ui.block_start(event.processname)
if isinstance(event, bb.event.ProcessFinished):
if event.processname in ["Initialising tasks", "Checking sstate mirror object availability"]:
- ui.progress(event.processname, 100)
- ui.block_end()
+ if active_process and active_process == event.processname:
+ ui.progress(event.processname, 100)
+ ui.block_end()
+ active_process = None
if isinstance(event, bb.event.ProcessProgress):
if event.processname in ["Initialising tasks",
"Checking sstate mirror object availability"] and active_process_total != 0:
- ui.progress(event.processname, event.progress * 100 / active_process_total)
+ if active_process and active_process == event.processname:
+ ui.progress(event.processname, event.progress * 100 / active_process_total)
if isinstance(event, bb.event.CacheLoadStarted):
ui.block_start("Loading cache")
if isinstance(event, bb.event.CacheLoadProgress):
diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py
index 82913e0d..3e077f04 100644
--- a/bitbake/lib/bb/ui/uihelper.py
+++ b/bitbake/lib/bb/ui/uihelper.py
@@ -6,6 +6,7 @@
#
import bb.build
+import bb.runqueue
import time
class BBUIHelper:
@@ -13,17 +14,17 @@ class BBUIHelper:
self.needUpdate = False
self.running_tasks = {}
# Running PIDs preserves the order tasks were executed in
- self.running_pids = []
self.failed_tasks = []
self.pidmap = {}
self.tasknumber_current = 0
self.tasknumber_total = 0
+ self.pressure_state = (False, False, False)
+ self.pressure_values = None
def eventHandler(self, event):
# PIDs are a bad idea as they can be reused before we process all UI events.
# We maintain a 'fuzzy' match for TaskProgress since there is no other way to match
def removetid(pid, tid):
- self.running_pids.remove(tid)
del self.running_tasks[tid]
if self.pidmap[pid] == tid:
del self.pidmap[pid]
@@ -31,11 +32,10 @@ class BBUIHelper:
if isinstance(event, bb.build.TaskStarted):
tid = event._fn + ":" + event._task
- if event._mc != "default":
+ if event._mc != "":
self.running_tasks[tid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid }
else:
self.running_tasks[tid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid }
- self.running_pids.append(tid)
self.pidmap[event.pid] = tid
self.needUpdate = True
elif isinstance(event, bb.build.TaskSucceeded):
@@ -60,6 +60,10 @@ class BBUIHelper:
self.running_tasks[self.pidmap[event.pid]]['progress'] = event.progress
self.running_tasks[self.pidmap[event.pid]]['rate'] = event.rate
self.needUpdate = True
+ elif isinstance(event, bb.runqueue.PSIEvent):
+ self.pressure_state = event.pressure_state
+ self.pressure_values = event.pressure_values
+ self.needUpdate = True
else:
return False
return True
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index ebee65d3..ec6eb218 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -11,11 +11,8 @@ import re, fcntl, os, string, stat, shutil, time
import sys
import errno
import logging
-import bb
-import bb.msg
import locale
-import multiprocessing
-import fcntl
+from bb import multiprocessing
import importlib
import importlib.machinery
import importlib.util
@@ -24,7 +21,6 @@ import subprocess
import glob
import fnmatch
import traceback
-import errno
import signal
import collections
import copy
@@ -36,6 +32,9 @@ import tempfile
from subprocess import getstatusoutput
from contextlib import contextmanager
from ctypes import cdll
+import bb
+import bb.msg
+import bb.filter
logger = logging.getLogger("BitBake.Util")
python_extensions = importlib.machinery.all_suffixes()
@@ -84,7 +83,16 @@ def explode_version(s):
return r
def split_version(s):
- """Split a version string into its constituent parts (PE, PV, PR)"""
+ """Split a version string into its constituent parts (PE, PV, PR).
+
+ Arguments:
+
+ - ``s``: version string. The format of the input string should be::
+
+ ${PE}:${PV}-${PR}
+
+ Returns a tuple ``(pe, pv, pr)``.
+ """
s = s.strip(" <>=")
e = 0
if s.count(':'):
@@ -136,16 +144,30 @@ def vercmp(ta, tb):
return r
def vercmp_string(a, b):
- """ Split version strings and compare them """
+ """ Split version strings using ``bb.utils.split_version()`` and compare
+ them with ``bb.utils.vercmp().``
+
+ Arguments:
+
+ - ``a``: left version string operand.
+ - ``b``: right version string operand.
+
+ Returns what ``bb.utils.vercmp()`` returns."""
ta = split_version(a)
tb = split_version(b)
return vercmp(ta, tb)
def vercmp_string_op(a, b, op):
"""
- Compare two versions and check if the specified comparison operator matches the result of the comparison.
- This function is fairly liberal about what operators it will accept since there are a variety of styles
- depending on the context.
+ Takes the return value ``bb.utils.vercmp()`` and returns the operation
+ defined by ``op`` between the return value and 0.
+
+ Arguments:
+
+ - ``a``: left version string operand.
+ - ``b``: right version string operand.
+ - ``op``: operator string. Can be one of ``=``, ``==``, ``<=``, ``>=``,
+ ``>``, ``>>``, ``<``, ``<<`` or ``!=``.
"""
res = vercmp_string(a, b)
if op in ('=', '=='):
@@ -163,11 +185,19 @@ def vercmp_string_op(a, b, op):
else:
raise VersionStringException('Unsupported comparison operator "%s"' % op)
+@bb.filter.filter_proc(name="bb.utils.explode_deps")
def explode_deps(s):
"""
- Take an RDEPENDS style string of format:
- "DEPEND1 (optional version) DEPEND2 (optional version) ..."
- and return a list of dependencies.
+ Takes an RDEPENDS style string of format::
+
+ DEPEND1 (optional version) DEPEND2 (optional version) ...
+
+ Arguments:
+
+ - ``s``: input RDEPENDS style string
+
+ Returns a list of dependencies.
+
Version information is ignored.
"""
r = []
@@ -189,9 +219,17 @@ def explode_deps(s):
def explode_dep_versions2(s, *, sort=True):
"""
- Take an RDEPENDS style string of format:
- "DEPEND1 (optional version) DEPEND2 (optional version) ..."
- and return a dictionary of dependencies and versions.
+ Takes an RDEPENDS style string of format::
+
+ DEPEND1 (optional version) DEPEND2 (optional version) ...
+
+ Arguments:
+
+ - ``s``: input RDEPENDS style string
+ - ``*``: *Unused*.
+ - ``sort``: whether to sort the output or not.
+
+ Returns a dictionary of dependencies and versions.
"""
r = collections.OrderedDict()
l = s.replace(",", "").split()
@@ -256,10 +294,17 @@ def explode_dep_versions2(s, *, sort=True):
def explode_dep_versions(s):
"""
- Take an RDEPENDS style string of format:
- "DEPEND1 (optional version) DEPEND2 (optional version) ..."
- skip null value and items appeared in dependency string multiple times
- and return a dictionary of dependencies and versions.
+ Take an RDEPENDS style string of format::
+
+ DEPEND1 (optional version) DEPEND2 (optional version) ...
+
+ Skips null values and items appeared in dependency string multiple times.
+
+ Arguments:
+
+ - ``s``: input RDEPENDS style string
+
+ Returns a dictionary of dependencies and versions.
"""
r = explode_dep_versions2(s)
for d in r:
@@ -273,7 +318,17 @@ def explode_dep_versions(s):
def join_deps(deps, commasep=True):
"""
- Take the result from explode_dep_versions and generate a dependency string
+ Take a result from ``bb.utils.explode_dep_versions()`` and generate a
+ dependency string.
+
+ Arguments:
+
+ - ``deps``: dictionary of dependencies and versions.
+ - ``commasep``: makes the return value separated by commas if ``True``,
+ separated by spaces otherwise.
+
+ Returns a comma-separated (space-separated if ``comma-sep`` is ``False``)
+ string of dependencies and versions.
"""
result = []
for dep in deps:
@@ -435,7 +490,11 @@ def better_eval(source, locals, extraglobals = None):
@contextmanager
def fileslocked(files, *args, **kwargs):
- """Context manager for locking and unlocking file locks."""
+ """Context manager for locking and unlocking file locks. Uses
+ ``bb.utils.lockfile()`` and ``bb.utils.unlockfile()`` to lock and unlock
+ files.
+
+ No return value."""
locks = []
if files:
for lockfile in files:
@@ -446,20 +505,33 @@ def fileslocked(files, *args, **kwargs):
try:
yield
finally:
+ locks.reverse()
for lock in locks:
bb.utils.unlockfile(lock)
def lockfile(name, shared=False, retry=True, block=False):
"""
- Use the specified file as a lock file, return when the lock has
- been acquired. Returns a variable to pass to unlockfile().
- Parameters:
- retry: True to re-try locking if it fails, False otherwise
- block: True to block until the lock succeeds, False otherwise
+ Use the specified file (with filename ``name``) as a lock file, return when
+ the lock has been acquired. Returns a variable to pass to unlockfile().
+
+ Arguments:
+
+ - ``shared``: sets the lock as a shared lock instead of an
+ exclusive lock.
+ - ``retry``: ``True`` to re-try locking if it fails, ``False``
+ otherwise.
+ - ``block``: ``True`` to block until the lock succeeds,
+ ``False`` otherwise.
+
The retry and block parameters are kind of equivalent unless you
consider the possibility of sending a signal to the process to break
out - at which point you want block=True rather than retry=True.
+
+ Returns the locked file descriptor in case of success, ``None`` otherwise.
"""
+ if not name:
+ return None
+
basename = os.path.basename(name)
if len(basename) > 255:
root, ext = os.path.splitext(basename)
@@ -515,10 +587,27 @@ def lockfile(name, shared=False, retry=True, block=False):
if not retry:
return None
+# We have to drop the existing lock to avoid deadlocks
+def lockfile_to_exclusive(lf):
+ if not lf:
+ return
+ name = lf.name
+ unlockfile(lf)
+ return lockfile(name)
+
def unlockfile(lf):
"""
- Unlock a file locked using lockfile()
+ Unlock a file locked using ``bb.utils.lockfile()``.
+
+ Arguments:
+
+ - ``lf``: the locked file descriptor.
+
+ No return value.
"""
+ if not lf:
+ return
+
try:
# If we had a shared lock, we need to promote to exclusive before
# removing the lockfile. Attempt this, ignore failures.
@@ -545,7 +634,11 @@ def _hasher(method, filename):
def md5_file(filename):
"""
- Return the hex string representation of the MD5 checksum of filename.
+ Arguments:
+
+ - ``filename``: path to the input file.
+
+ Returns the hexadecimal string representation of the MD5 checksum of filename.
"""
import hashlib
try:
@@ -557,36 +650,86 @@ def md5_file(filename):
def sha256_file(filename):
"""
- Return the hex string representation of the 256-bit SHA checksum of
+ Returns the hexadecimal representation of the 256-bit SHA checksum of
filename.
+
+ Arguments:
+
+ - ``filename``: path to the file.
"""
import hashlib
return _hasher(hashlib.sha256(), filename)
def sha1_file(filename):
"""
- Return the hex string representation of the SHA1 checksum of the filename
+ Returns the hexadecimal representation of the SHA1 checksum of the filename
+
+ Arguments:
+
+ - ``filename``: path to the file.
"""
import hashlib
return _hasher(hashlib.sha1(), filename)
def sha384_file(filename):
"""
- Return the hex string representation of the SHA384 checksum of the filename
+ Returns the hexadecimal representation of the SHA384 checksum of the filename
+
+ Arguments:
+
+ - ``filename``: path to the file.
"""
import hashlib
return _hasher(hashlib.sha384(), filename)
def sha512_file(filename):
"""
- Return the hex string representation of the SHA512 checksum of the filename
+ Returns the hexadecimal representation of the SHA512 checksum of the filename
+
+ Arguments:
+
+ - ``filename``: path to the file.
"""
import hashlib
return _hasher(hashlib.sha512(), filename)
+def goh1_file(filename):
+ """
+ Returns the hexadecimal string representation of the Go mod h1 checksum of the
+ filename. The Go mod h1 checksum uses the Go dirhash package. The package
+ defines hashes over directory trees and is used by go mod for mod files and
+ zip archives.
+
+ Arguments:
+
+ - ``filename``: path to the file.
+ """
+ import hashlib
+ import zipfile
+
+ lines = []
+ is_zipfile = False
+ if zipfile.is_zipfile(filename):
+ try:
+ with zipfile.ZipFile(filename) as archive:
+ for fn in sorted(archive.namelist()):
+ method = hashlib.sha256()
+ method.update(archive.read(fn))
+ hash = method.hexdigest()
+ lines.append("%s %s\n" % (hash, fn))
+ is_zipfile = True
+ except zipfile.BadZipFile:
+ is_zipfile = False
+ if not is_zipfile:
+ hash = _hasher(hashlib.sha256(), filename)
+ lines.append("%s go.mod\n" % hash)
+ method = hashlib.sha256()
+ method.update("".join(lines).encode('utf-8'))
+ return method.hexdigest()
+
def preserved_envvars_exported():
- """Variables which are taken from the environment and placed in and exported
- from the metadata"""
+ """Returns the list of variables which are taken from the environment and
+ placed in and exported from the metadata."""
return [
'BB_TASKHASH',
'HOME',
@@ -600,7 +743,8 @@ def preserved_envvars_exported():
]
def preserved_envvars():
- """Variables which are taken from the environment and placed in the metadata"""
+ """Returns the list of variables which are taken from the environment and
+ placed in the metadata."""
v = [
'BBPATH',
'BB_PRESERVE_ENV',
@@ -609,7 +753,9 @@ def preserved_envvars():
return v + preserved_envvars_exported()
def check_system_locale():
- """Make sure the required system locale are available and configured"""
+ """Make sure the required system locale are available and configured.
+
+ No return value."""
default_locale = locale.getlocale(locale.LC_CTYPE)
try:
@@ -627,6 +773,12 @@ def filter_environment(good_vars):
"""
Create a pristine environment for bitbake. This will remove variables that
are not known and may influence the build in a negative way.
+
+ Arguments:
+
+ - ``good_vars``: list of variable to exclude from the filtering.
+
+ No return value.
"""
removed_vars = {}
@@ -671,6 +823,8 @@ def clean_environment():
"""
Clean up any spurious environment variables. This will remove any
variables the user hasn't chosen to preserve.
+
+ No return value.
"""
if 'BB_PRESERVE_ENV' not in os.environ:
good_vars = approved_variables()
@@ -681,6 +835,8 @@ def clean_environment():
def empty_environment():
"""
Remove all variables from the environment.
+
+ No return value.
"""
for s in list(os.environ.keys()):
os.unsetenv(s)
@@ -689,6 +845,12 @@ def empty_environment():
def build_environment(d):
"""
Build an environment from all exported variables.
+
+ Arguments:
+
+ - ``d``: the data store.
+
+ No return value.
"""
import bb.data
for var in bb.data.keys(d):
@@ -713,7 +875,17 @@ def _check_unsafe_delete_path(path):
return False
def remove(path, recurse=False, ionice=False):
- """Equivalent to rm -f or rm -rf"""
+ """Equivalent to rm -f or rm -rf.
+
+ Arguments:
+
+ - ``path``: path to file/directory to remove.
+ - ``recurse``: deletes recursively if ``True``.
+ - ``ionice``: prepends ``ionice -c 3`` to the ``rm`` command. See ``man
+ ionice``.
+
+ No return value.
+ """
if not path:
return
if recurse:
@@ -734,7 +906,17 @@ def remove(path, recurse=False, ionice=False):
raise
def prunedir(topdir, ionice=False):
- """ Delete everything reachable from the directory named in 'topdir'. """
+ """
+ Delete everything reachable from the directory named in ``topdir``.
+
+ Arguments:
+
+ - ``topdir``: directory path.
+ - ``ionice``: prepends ``ionice -c 3`` to the ``rm`` command. See ``man
+ ionice``.
+
+ No return value.
+ """
# CAUTION: This is dangerous!
if _check_unsafe_delete_path(topdir):
raise Exception('bb.utils.prunedir: called with dangerous path "%s", refusing to delete!' % topdir)
@@ -746,8 +928,15 @@ def prunedir(topdir, ionice=False):
#
def prune_suffix(var, suffixes, d):
"""
- See if var ends with any of the suffixes listed and
- remove it if found
+ Check if ``var`` ends with any of the suffixes listed in ``suffixes`` and
+ remove it if found.
+
+ Arguments:
+
+ - ``var``: string to check for suffixes.
+ - ``suffixes``: list of strings representing suffixes to check for.
+
+ Returns the string ``var`` without the suffix.
"""
for suffix in suffixes:
if suffix and var.endswith(suffix):
@@ -756,21 +945,53 @@ def prune_suffix(var, suffixes, d):
def mkdirhier(directory):
"""Create a directory like 'mkdir -p', but does not complain if
- directory already exists like os.makedirs
+ directory already exists like ``os.makedirs()``.
+
+ Arguments:
+
+ - ``directory``: path to the directory.
+
+ No return value.
"""
if '${' in str(directory):
bb.fatal("Directory name {} contains unexpanded bitbake variable. This may cause build failures and WORKDIR polution.".format(directory))
try:
os.makedirs(directory)
except OSError as e:
- if e.errno != errno.EEXIST or not os.path.isdir(directory):
+ if e.errno != errno.EEXIST:
raise e
+ if os.path.isdir(directory):
+ return
+ # We can end up here if there is a race between two mkdirs on an NFS mount,
+ # which happens more often with sstate that you'd think. The server returns
+ # EEXIST but the local attribute cache is out of date. It can be refreshed with
+ # an opendir call, so try that (via listdir) and check the directory again
+ # before we really fail.
+ os.listdir(os.path.dirname(directory))
+ if os.path.isdir(directory):
+ return
+ bb.warn("mkdir: %s is not a directory?")
+ raise e
def movefile(src, dest, newmtime = None, sstat = None):
- """Moves a file from src to dest, preserving all permissions and
+ """Moves a file from ``src`` to ``dest``, preserving all permissions and
attributes; mtime will be preserved even when moving across
- filesystems. Returns true on success and false on failure. Move is
+ filesystems. Returns ``True`` on success and ``False`` on failure. Move is
atomic.
+
+ Arguments:
+
+ - ``src`` -- Source file.
+ - ``dest`` -- Destination file.
+ - ``newmtime`` -- new mtime to be passed as float seconds since the epoch.
+ - ``sstat`` -- os.stat_result to use for the destination file.
+
+ Returns an ``os.stat_result`` of the destination file if the
+ source file is a symbolic link or the ``sstat`` argument represents a
+ symbolic link - in which case the destination file will also be created as
+ a symbolic link.
+
+ Otherwise, returns ``newmtime`` on success and ``False`` on failure.
"""
#print "movefile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")"
@@ -861,9 +1082,24 @@ def movefile(src, dest, newmtime = None, sstat = None):
def copyfile(src, dest, newmtime = None, sstat = None):
"""
- Copies a file from src to dest, preserving all permissions and
+ Copies a file from ``src`` to ``dest``, preserving all permissions and
attributes; mtime will be preserved even when moving across
- filesystems. Returns true on success and false on failure.
+ filesystems.
+
+ Arguments:
+
+ - ``src``: Source file.
+ - ``dest``: Destination file.
+ - ``newmtime``: new mtime to be passed as float seconds since the epoch.
+ - ``sstat``: os.stat_result to use for the destination file.
+
+ Returns an ``os.stat_result`` of the destination file if the
+ source file is a symbolic link or the ``sstat`` argument represents a
+ symbolic link - in which case the destination file will also be created as
+ a symbolic link.
+
+ Otherwise, returns ``newmtime`` on success and ``False`` on failure.
+
"""
#print "copyfile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")"
try:
@@ -941,10 +1177,16 @@ def copyfile(src, dest, newmtime = None, sstat = None):
def break_hardlinks(src, sstat = None):
"""
- Ensures src is the only hardlink to this file. Other hardlinks,
+ Ensures ``src`` is the only hardlink to this file. Other hardlinks,
if any, are not affected (other than in their st_nlink value, of
- course). Returns true on success and false on failure.
+ course).
+
+ Arguments:
+ - ``src``: source file path.
+ - ``sstat``: os.stat_result to use when checking if the file is a link.
+
+ Returns ``True`` on success and ``False`` on failure.
"""
try:
if not sstat:
@@ -958,11 +1200,24 @@ def break_hardlinks(src, sstat = None):
def which(path, item, direction = 0, history = False, executable=False):
"""
- Locate `item` in the list of paths `path` (colon separated string like $PATH).
- If `direction` is non-zero then the list is reversed.
- If `history` is True then the list of candidates also returned as result,history.
- If `executable` is True then the candidate has to be an executable file,
- otherwise the candidate simply has to exist.
+ Locate ``item`` in the list of paths ``path`` (colon separated string like
+ ``$PATH``).
+
+ Arguments:
+
+ - ``path``: list of colon-separated paths.
+ - ``item``: string to search for.
+ - ``direction``: if non-zero then the list is reversed.
+ - ``history``: if ``True`` then the list of candidates also returned as
+ ``result,history`` where ``history`` is the list of previous path
+ checked.
+ - ``executable``: if ``True`` then the candidate defined by ``path`` has
+ to be an executable file, otherwise if ``False`` the candidate simply
+ has to exist.
+
+ Returns the item if found in the list of path, otherwise an empty string.
+ If ``history`` is ``True``, return the list of previous path checked in a
+ tuple with the found (or not found) item as ``(item, history)``.
"""
if executable:
@@ -989,10 +1244,29 @@ def which(path, item, direction = 0, history = False, executable=False):
return "", hist
return ""
+def to_filemode(input):
+ """
+ Take a bitbake variable contents defining a file mode and return
+ the proper python representation of the number
+
+ Arguments:
+
+ - ``input``: a string or number to convert, e.g. a bitbake variable
+ string, assumed to be an octal representation
+
+ Returns the python file mode as a number
+ """
+ # umask might come in as a number or text string..
+ if type(input) is int:
+ return input
+ return int(input, 8)
+
@contextmanager
def umask(new_mask):
"""
Context manager to set the umask to a specific mask, and restore it afterwards.
+
+ No return value.
"""
current_mask = os.umask(new_mask)
try:
@@ -1003,7 +1277,17 @@ def umask(new_mask):
def to_boolean(string, default=None):
"""
Check input string and return boolean value True/False/None
- depending upon the checks
+ depending upon the checks.
+
+ Arguments:
+
+ - ``string``: input string.
+ - ``default``: default return value if the input ``string`` is ``None``,
+ ``0``, ``False`` or an empty string.
+
+ Returns ``True`` if the string is one of "y", "yes", "1", "true", ``False``
+ if the string is one of "n", "no", "0", or "false". Return ``default`` if
+ the input ``string`` is ``None``, ``0``, ``False`` or an empty string.
"""
if not string:
return default
@@ -1024,18 +1308,17 @@ def contains(variable, checkvalues, truevalue, falsevalue, d):
Arguments:
- variable -- the variable name. This will be fetched and expanded (using
- d.getVar(variable)) and then split into a set().
-
- checkvalues -- if this is a string it is split on whitespace into a set(),
- otherwise coerced directly into a set().
-
- truevalue -- the value to return if checkvalues is a subset of variable.
+ - ``variable``: the variable name. This will be fetched and expanded (using
+ d.getVar(variable)) and then split into a set().
+ - ``checkvalues``: if this is a string it is split on whitespace into a set(),
+ otherwise coerced directly into a set().
+ - ``truevalue``: the value to return if checkvalues is a subset of variable.
+ - ``falsevalue``: the value to return if variable is empty or if checkvalues is
+ not a subset of variable.
+ - ``d``: the data store.
- falsevalue -- the value to return if variable is empty or if checkvalues is
- not a subset of variable.
-
- d -- the data store.
+ Returns ``truevalue`` if the variable contains the values specified,
+ ``falsevalue`` otherwise.
"""
val = d.getVar(variable)
@@ -1055,18 +1338,17 @@ def contains_any(variable, checkvalues, truevalue, falsevalue, d):
Arguments:
- variable -- the variable name. This will be fetched and expanded (using
- d.getVar(variable)) and then split into a set().
-
- checkvalues -- if this is a string it is split on whitespace into a set(),
- otherwise coerced directly into a set().
-
- truevalue -- the value to return if checkvalues is a subset of variable.
-
- falsevalue -- the value to return if variable is empty or if checkvalues is
- not a subset of variable.
+ - ``variable``: the variable name. This will be fetched and expanded (using
+ d.getVar(variable)) and then split into a set().
+ - ``checkvalues``: if this is a string it is split on whitespace into a set(),
+ otherwise coerced directly into a set().
+ - ``truevalue``: the value to return if checkvalues is a subset of variable.
+ - ``falsevalue``: the value to return if variable is empty or if checkvalues is
+ not a subset of variable.
+ - ``d``: the data store.
- d -- the data store.
+ Returns ``truevalue`` if the variable contains any of the values specified,
+ ``falsevalue`` otherwise.
"""
val = d.getVar(variable)
if not val:
@@ -1080,21 +1362,18 @@ def contains_any(variable, checkvalues, truevalue, falsevalue, d):
return truevalue
return falsevalue
-def filter(variable, checkvalues, d):
- """Return all words in the variable that are present in the checkvalues.
+def filter_string(val, checkvalues):
+ """Return all words in the string that are present in the ``checkvalues``.
Arguments:
- variable -- the variable name. This will be fetched and expanded (using
- d.getVar(variable)) and then split into a set().
-
- checkvalues -- if this is a string it is split on whitespace into a set(),
- otherwise coerced directly into a set().
+ - ``val``: the string data to filter after being split into a set().
+ - ``checkvalues``: if this is a string it is split on whitespace into a set(),
+ otherwise coerced directly into a set().
+ - ``d``: the data store.
- d -- the data store.
+ Returns a list of string.
"""
-
- val = d.getVar(variable)
if not val:
return ''
val = set(val.split())
@@ -1104,11 +1383,46 @@ def filter(variable, checkvalues, d):
checkvalues = set(checkvalues)
return ' '.join(sorted(checkvalues & val))
+def filter(variable, checkvalues, d):
+ """Return all words in the variable that are present in the ``checkvalues``.
+
+ Arguments:
+
+ - ``variable``: the variable name. This will be fetched and expanded (using
+ d.getVar(variable)) and then split into a set().
+ - ``checkvalues``: if this is a string it is split on whitespace into a set(),
+ otherwise coerced directly into a set().
+ - ``d``: the data store.
+
+ Returns a list of string.
+ """
+
+ val = d.getVar(variable)
+ return filter_string(val, checkvalues)
def get_referenced_vars(start_expr, d):
"""
- :return: names of vars referenced in start_expr (recursively), in quasi-BFS order (variables within the same level
- are ordered arbitrarily)
+ Get the names of the variables referenced in a given expression.
+
+ Arguments:
+
+ - ``start_expr``: the expression where to look for variables references.
+
+ For example::
+
+ ${VAR_A} string ${VAR_B}
+
+ Or::
+
+ ${@d.getVar('VAR')}
+
+ If a variables makes references to other variables, the latter are also
+ returned recursively.
+
+ - ``d``: the data store.
+
+ Returns the names of vars referenced in ``start_expr`` (recursively), in
+ quasi-BFS order (variables within the same level are ordered arbitrarily).
"""
seen = set()
@@ -1150,45 +1464,71 @@ def cpu_count():
def nonblockingfd(fd):
fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
-def process_profilelog(fn, pout = None):
+def profile_function(profile, function, output_fn, process=True):
+ """Common function to profile a code block and optionally process the
+ output using or processing function.
+
+ Arguments:
+
+ - ``profile``: a boolean saying whether to enable profiling or not
+ - ``function``: the function call to profile/run
+ - ``outputfn``: where to write the profiling data
+ - ``process``: whether to process the profiling data and write a report
+
+ Returns the wrapped function return value
+ """
+ if profile:
+ try:
+ import cProfile as profile
+ except:
+ import profile
+ prof = profile.Profile()
+ ret = profile.Profile.runcall(prof, function)
+ prof.dump_stats(output_fn)
+ if process:
+ process_profilelog(output_fn)
+ serverlog("Raw profiling information saved to %s and processed statistics to %s.report*" % (output_fn, output_fn))
+ return ret
+ else:
+ return function()
+
+def process_profilelog(fn, fn_out = None):
# Either call with a list of filenames and set pout or a filename and optionally pout.
- if not pout:
- pout = fn + '.processed'
+ import pstats
- with open(pout, 'w') as pout:
- import pstats
+ if not fn_out:
+ fn_out = fn + '.report'
+
+ def pstatopen():
if isinstance(fn, list):
- p = pstats.Stats(*fn, stream=pout)
- else:
- p = pstats.Stats(fn, stream=pout)
+ return pstats.Stats(*fn, stream=pout)
+ return pstats.Stats(fn, stream=pout)
+
+ with open(fn_out + '.time', 'w') as pout:
+ p = pstatopen()
p.sort_stats('time')
p.print_stats()
+
+ with open(fn_out + '.time-callers', 'w') as pout:
+ p = pstatopen()
+ p.sort_stats('time')
p.print_callers()
+
+ with open(fn_out + '.cumulative', 'w') as pout:
+ p = pstatopen()
p.sort_stats('cumulative')
p.print_stats()
- pout.flush()
-
-#
-# Was present to work around multiprocessing pool bugs in python < 2.7.3
-#
-def multiprocessingpool(*args, **kwargs):
-
- import multiprocessing.pool
- #import multiprocessing.util
- #multiprocessing.util.log_to_stderr(10)
- # Deal with a multiprocessing bug where signals to the processes would be delayed until the work
- # completes. Putting in a timeout means the signals (like SIGINT/SIGTERM) get processed.
- def wrapper(func):
- def wrap(self, timeout=None):
- return func(self, timeout=timeout if timeout is not None else 1e100)
- return wrap
- multiprocessing.pool.IMapIterator.next = wrapper(multiprocessing.pool.IMapIterator.next)
+ with open(fn_out + '.cumulative-callers', 'w') as pout:
+ p = pstatopen()
+ p.sort_stats('cumulative')
+ p.print_callers()
- return multiprocessing.Pool(*args, **kwargs)
def exec_flat_python_func(func, *args, **kwargs):
- """Execute a flat python function (defined with def funcname(args):...)"""
+ """Execute a flat python function (defined with ``def funcname(args): ...``)
+
+ Returns the return value of the function."""
# Prepare a small piece of python code which calls the requested function
# To do this we need to prepare two things - a set of variables we can use to pass
# the values of arguments into the calling function, and the list of arguments for
@@ -1214,48 +1554,57 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False):
"""Edit lines from a recipe or config file and modify one or more
specified variable values set in the file using a specified callback
function. Lines are expected to have trailing newlines.
- Parameters:
- meta_lines: lines from the file; can be a list or an iterable
- (e.g. file pointer)
- variables: a list of variable names to look for. Functions
- may also be specified, but must be specified with '()' at
- the end of the name. Note that the function doesn't have
- any intrinsic understanding of :append, :prepend, :remove,
- or overrides, so these are considered as part of the name.
- These values go into a regular expression, so regular
- expression syntax is allowed.
- varfunc: callback function called for every variable matching
- one of the entries in the variables parameter. The function
- should take four arguments:
- varname: name of variable matched
- origvalue: current value in file
- op: the operator (e.g. '+=')
- newlines: list of lines up to this point. You can use
- this to prepend lines before this variable setting
- if you wish.
- and should return a four-element tuple:
- newvalue: new value to substitute in, or None to drop
- the variable setting entirely. (If the removal
- results in two consecutive blank lines, one of the
- blank lines will also be dropped).
- newop: the operator to use - if you specify None here,
- the original operation will be used.
- indent: number of spaces to indent multi-line entries,
- or -1 to indent up to the level of the assignment
- and opening quote, or a string to use as the indent.
- minbreak: True to allow the first element of a
- multi-line value to continue on the same line as
- the assignment, False to indent before the first
- element.
- To clarify, if you wish not to change the value, then you
- would return like this: return origvalue, None, 0, True
- match_overrides: True to match items with _overrides on the end,
- False otherwise
+
+ Arguments:
+
+ - ``meta_lines``: lines from the file; can be a list or an iterable
+ (e.g. file pointer)
+ - ``variables``: a list of variable names to look for. Functions
+ may also be specified, but must be specified with ``()`` at
+ the end of the name. Note that the function doesn't have
+ any intrinsic understanding of ``:append``, ``:prepend``, ``:remove``,
+ or overrides, so these are considered as part of the name.
+ These values go into a regular expression, so regular
+ expression syntax is allowed.
+ - ``varfunc``: callback function called for every variable matching
+ one of the entries in the variables parameter.
+
+ The function should take four arguments:
+
+ - ``varname``: name of variable matched
+ - ``origvalue``: current value in file
+ - ``op``: the operator (e.g. ``+=``)
+ - ``newlines``: list of lines up to this point. You can use
+ this to prepend lines before this variable setting
+ if you wish.
+
+ And should return a four-element tuple:
+
+ - ``newvalue``: new value to substitute in, or ``None`` to drop
+ the variable setting entirely. (If the removal
+ results in two consecutive blank lines, one of the
+ blank lines will also be dropped).
+ - ``newop``: the operator to use - if you specify ``None`` here,
+ the original operation will be used.
+ - ``indent``: number of spaces to indent multi-line entries,
+ or ``-1`` to indent up to the level of the assignment
+ and opening quote, or a string to use as the indent.
+ - ``minbreak``: ``True`` to allow the first element of a
+ multi-line value to continue on the same line as
+ the assignment, ``False`` to indent before the first
+ element.
+
+ To clarify, if you wish not to change the value, then you
+ would return like this::
+
+ return origvalue, None, 0, True
+ - ``match_overrides``: True to match items with _overrides on the end,
+ False otherwise
+
Returns a tuple:
- updated:
- True if changes were made, False otherwise.
- newlines:
- Lines after processing
+
+ - ``updated``: ``True`` if changes were made, ``False`` otherwise.
+ - ``newlines``: Lines after processing.
"""
var_res = {}
@@ -1399,12 +1748,13 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False):
def edit_metadata_file(meta_file, variables, varfunc):
- """Edit a recipe or config file and modify one or more specified
+ """Edit a recipe or configuration file and modify one or more specified
variable values set in the file using a specified callback function.
The file is only written to if the value(s) actually change.
- This is basically the file version of edit_metadata(), see that
+ This is basically the file version of ``bb.utils.edit_metadata()``, see that
function's description for parameter/usage information.
- Returns True if the file was written to, False otherwise.
+
+ Returns ``True`` if the file was written to, ``False`` otherwise.
"""
with open(meta_file, 'r') as f:
(updated, newlines) = edit_metadata(f, variables, varfunc)
@@ -1415,23 +1765,25 @@ def edit_metadata_file(meta_file, variables, varfunc):
def edit_bblayers_conf(bblayers_conf, add, remove, edit_cb=None):
- """Edit bblayers.conf, adding and/or removing layers
- Parameters:
- bblayers_conf: path to bblayers.conf file to edit
- add: layer path (or list of layer paths) to add; None or empty
- list to add nothing
- remove: layer path (or list of layer paths) to remove; None or
- empty list to remove nothing
- edit_cb: optional callback function that will be called after
- processing adds/removes once per existing entry.
+ """Edit ``bblayers.conf``, adding and/or removing layers.
+
+ Arguments:
+
+ - ``bblayers_conf``: path to ``bblayers.conf`` file to edit
+ - ``add``: layer path (or list of layer paths) to add; ``None`` or empty
+ list to add nothing
+ - ``remove``: layer path (or list of layer paths) to remove; ``None`` or
+ empty list to remove nothing
+ - ``edit_cb``: optional callback function that will be called
+ after processing adds/removes once per existing entry.
+
Returns a tuple:
- notadded: list of layers specified to be added but weren't
- (because they were already in the list)
- notremoved: list of layers that were specified to be removed
- but weren't (because they weren't in the list)
- """
- import fnmatch
+ - ``notadded``: list of layers specified to be added but weren't
+ (because they were already in the list)
+ - ``notremoved``: list of layers that were specified to be removed
+ but weren't (because they weren't in the list)
+ """
def remove_trailing_sep(pth):
if pth and pth[-1] == os.sep:
@@ -1550,7 +1902,22 @@ def get_collection_res(d):
def get_file_layer(filename, d, collection_res={}):
- """Determine the collection (as defined by a layer's layer.conf file) containing the specified file"""
+ """Determine the collection (or layer name, as defined by a layer's
+ ``layer.conf`` file) containing the specified file.
+
+ Arguments:
+
+ - ``filename``: the filename to look for.
+ - ``d``: the data store.
+ - ``collection_res``: dictionary with the layer names as keys and file
+ patterns to match as defined with the BBFILE_COLLECTIONS and
+ BBFILE_PATTERN variables respectively. The return value of
+ ``bb.utils.get_collection_res()`` is the default if this variable is
+ not specified.
+
+ Returns the layer name containing the file. If multiple layers contain the
+ file, the last matching layer name from collection_res is returned.
+ """
if not collection_res:
collection_res = get_collection_res(d)
@@ -1588,7 +1955,13 @@ class PrCtlError(Exception):
def signal_on_parent_exit(signame):
"""
- Trigger signame to be sent when the parent process dies
+ Trigger ``signame`` to be sent when the parent process dies.
+
+ Arguments:
+
+ - ``signame``: name of the signal. See ``man signal``.
+
+ No return value.
"""
signum = getattr(signal, signame)
# http://linux.die.net/man/2/prctl
@@ -1623,7 +1996,7 @@ def ioprio_set(who, cls, value):
bb.warn("Unable to set IO Prio for arch %s" % _unamearch)
def set_process_name(name):
- from ctypes import cdll, byref, create_string_buffer
+ from ctypes import byref, create_string_buffer
# This is nice to have for debugging, not essential
try:
libc = cdll.LoadLibrary('libc.so.6')
@@ -1675,6 +2048,13 @@ def disable_network(uid=None, gid=None):
Disable networking in the current process if the kernel supports it, else
just return after logging to debug. To do this we need to create a new user
namespace, then map back to the original uid/gid.
+
+ Arguments:
+
+ - ``uid``: original user id.
+ - ``gid``: original user group id.
+
+ No return value.
"""
libc = ctypes.CDLL('libc.so.6')
@@ -1700,8 +2080,46 @@ def disable_network(uid=None, gid=None):
with open("/proc/self/gid_map", "w") as f:
f.write("%s %s 1" % (gid, gid))
+def landlock_restrict_network():
+ """Block TCP bind/connect using Landlock LSM (ABI v4+, kernel 6.7+).
+ Gracefully skipped on older kernels. Stacks with disable_network()."""
+
+ NR_landlock_create_ruleset = 444
+ NR_landlock_restrict_self = 446
+
+ LANDLOCK_ACCESS_NET_BIND_TCP = 0x1
+ LANDLOCK_ACCESS_NET_CONNECT_TCP = 0x2
+
+ LANDLOCK_CREATE_RULESET_VERSION = 1
+
+ PR_SET_NO_NEW_PRIVS = 38
+
+ libc = ctypes.CDLL('libc.so.6')
+
+ libc.prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)
+ abi = libc.syscall(NR_landlock_create_ruleset,
+ 0, 0,
+ LANDLOCK_CREATE_RULESET_VERSION)
+ if abi < 4:
+ logger.debug("System doesn't support disabling network via landlock")
+ return
+
+ net_access = LANDLOCK_ACCESS_NET_BIND_TCP | LANDLOCK_ACCESS_NET_CONNECT_TCP
+ attr = struct.pack("QQ", 0, net_access)
+ buf = ctypes.create_string_buffer(attr)
+ fd = libc.syscall(NR_landlock_create_ruleset, buf, len(attr), 0)
+ if fd < 0:
+ logger.debug("Failed to create landlock ruleset for network restriction")
+ return
+
+ r = libc.syscall(NR_landlock_restrict_self, fd, 0)
+ if r < 0:
+ logger.debug("Failed to landlock restrict_self")
+
+ os.close(fd)
+
def export_proxies(d):
- from bb.fetch2 import get_fetcher_environment
+ from bb.fetch import get_fetcher_environment
""" export common proxies variables from datastore to environment """
newenv = get_fetcher_environment(d)
for v in newenv:
@@ -1744,9 +2162,14 @@ class LogCatcher(logging.Handler):
def is_semver(version):
"""
- Is the version string following the semver semantic?
+ Arguments:
+
+ - ``version``: the version string.
+
+ Returns ``True`` if the version string follow semantic versioning, ``False``
+ otherwise.
- https://semver.org/spec/v2.0.0.html
+ See https://semver.org/spec/v2.0.0.html.
"""
regex = re.compile(
r"""
@@ -1784,6 +2207,8 @@ def rename(src, dst):
def environment(**envvars):
"""
Context manager to selectively update the environment with the specified mapping.
+
+ No return value.
"""
backup = dict(os.environ)
try:
@@ -1800,6 +2225,13 @@ def is_local_uid(uid=''):
"""
Check whether uid is a local one or not.
Can't use pwd module since it gets all UIDs, not local ones only.
+
+ Arguments:
+
+ - ``uid``: user id. If not specified the user id is determined from
+ ``os.getuid()``.
+
+ Returns ``True`` is the user id is local, ``False`` otherwise.
"""
if not uid:
uid = os.getuid()
@@ -1814,7 +2246,7 @@ def is_local_uid(uid=''):
def mkstemp(suffix=None, prefix=None, dir=None, text=False):
"""
- Generates a unique filename, independent of time.
+ Generates a unique temporary file, independent of time.
mkstemp() in glibc (at least) generates unique file names based on the
current system time. When combined with highly parallel builds, and
@@ -1823,6 +2255,18 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False):
This function adds additional entropy to the file name so that a collision
is independent of time and thus extremely unlikely.
+
+ Arguments:
+
+ - ``suffix``: filename suffix.
+ - ``prefix``: filename prefix.
+ - ``dir``: directory where the file will be created.
+ - ``text``: if ``True``, the file is opened in text mode.
+
+ Returns a tuple containing:
+
+ - the file descriptor for the created file
+ - the name of the file.
"""
entropy = "".join(random.choices("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890", k=20))
if prefix:
@@ -1833,12 +2277,20 @@ def mkstemp(suffix=None, prefix=None, dir=None, text=False):
def path_is_descendant(descendant, ancestor):
"""
- Returns True if the path `descendant` is a descendant of `ancestor`
- (including being equivalent to `ancestor` itself). Otherwise returns False.
+ Returns ``True`` if the path ``descendant`` is a descendant of ``ancestor``
+ (including being equivalent to ``ancestor`` itself). Otherwise returns
+ ``False``.
+
Correctly accounts for symlinks, bind mounts, etc. by using
- os.path.samestat() to compare paths
+ ``os.path.samestat()`` to compare paths.
+
+ May raise any exception that ``os.stat()`` raises.
- May raise any exception that os.stat() raises
+ Arguments:
+
+ - ``descendant``: path to check for being an ancestor.
+ - ``ancestor``: path to the ancestor ``descendant`` will be checked
+ against.
"""
ancestor_stat = os.stat(ancestor)
@@ -1854,15 +2306,56 @@ def path_is_descendant(descendant, ancestor):
return False
+# Recomputing the sets in signal.py is expensive (bitbake -pP idle)
+# so try and use _signal directly to avoid it
+valid_signals = signal.valid_signals()
+try:
+ import _signal
+ sigmask = _signal.pthread_sigmask
+except ImportError:
+ sigmask = signal.pthread_sigmask
+
# If we don't have a timeout of some kind and a process/thread exits badly (for example
# OOM killed) and held a lock, we'd just hang in the lock futex forever. It is better
# we exit at some point than hang. 5 minutes with no progress means we're probably deadlocked.
+# This function can still deadlock python since it can't signal the other threads to exit
+# (signals are handled in the main thread) and even os._exit() will wait on non-daemon threads
+# to exit.
@contextmanager
def lock_timeout(lock):
- held = lock.acquire(timeout=5*60)
try:
+ s = sigmask(signal.SIG_BLOCK, valid_signals)
+ held = lock.acquire(timeout=5*60)
if not held:
+ bb.server.process.serverlog("Couldn't get the lock for 5 mins, timed out, exiting.\n%s" % traceback.format_stack())
os._exit(1)
yield held
finally:
lock.release()
+ sigmask(signal.SIG_SETMASK, s)
+
+# A version of lock_timeout without the check that the lock was locked and a shorter timeout
+@contextmanager
+def lock_timeout_nocheck(lock):
+ l = False
+ try:
+ s = sigmask(signal.SIG_BLOCK, valid_signals)
+ l = lock.acquire(timeout=10)
+ yield l
+ finally:
+ if l:
+ lock.release()
+ sigmask(signal.SIG_SETMASK, s)
+
+def is_path_on_nfs(path):
+ """
+ Returns True if ``path`` argument is on a NFS mount.
+ """
+ # strip not existing path
+ if os.path.isabs(path):
+ while not os.path.exists(path):
+ path = os.path.dirname(path)
+
+ import bb.process
+ fstype = bb.process.run(['stat', '-f', '-c', '%T', path])[0].strip()
+ return fstype == "nfs"
diff --git a/bitbake/lib/bblayers/action.py b/bitbake/lib/bblayers/action.py
index a8f26993..a14f1994 100644
--- a/bitbake/lib/bblayers/action.py
+++ b/bitbake/lib/bblayers/action.py
@@ -50,8 +50,8 @@ class ActionPlugin(LayerPlugin):
try:
notadded, _ = bb.utils.edit_bblayers_conf(bblayers_conf, layerdirs, None)
- self.tinfoil.modified_files()
if not (args.force or notadded):
+ self.tinfoil.modified_files()
try:
self.tinfoil.run_command('parseConfiguration')
except (bb.tinfoil.TinfoilUIException, bb.BBHandledException):
@@ -83,6 +83,8 @@ class ActionPlugin(LayerPlugin):
layerdir = os.path.abspath(item)
layerdirs.append(layerdir)
(_, notremoved) = bb.utils.edit_bblayers_conf(bblayers_conf, None, layerdirs)
+ if args.force > 1:
+ return 0
self.tinfoil.modified_files()
if notremoved:
for item in notremoved:
diff --git a/bitbake/lib/bblayers/layerindex.py b/bitbake/lib/bblayers/layerindex.py
index ba91fac6..2ba1103a 100644
--- a/bitbake/lib/bblayers/layerindex.py
+++ b/bitbake/lib/bblayers/layerindex.py
@@ -55,13 +55,15 @@ class LayerIndexPlugin(ActionPlugin):
switching branches if necessary and possible.
"""
base_cmd = ['git', '--git-dir=%s/.git' % repodir, '--work-tree=%s' % repodir]
- cmd = base_cmd + ['branch']
+ # Get current branch name
+ cmd = base_cmd + ['name-rev', '--name-only', 'HEAD']
completed_proc = subprocess.run(cmd, text=True, capture_output=True)
if completed_proc.returncode:
- logger.error("Unable to validate repo %s (%s)" % (repodir, stderr))
+ logger.error("Unable to validate repo %s (%s)" % (repodir, completed_proc.stderr))
return None, None, None
else:
- if branch != completed_proc.stdout[2:-1]:
+ current_branch = completed_proc.stdout.strip()
+ if branch != current_branch:
cmd = base_cmd + ['status', '--short']
completed_proc = subprocess.run(cmd, text=True, capture_output=True)
if completed_proc.stdout.count('\n') != 0:
@@ -114,7 +116,7 @@ class LayerIndexPlugin(ActionPlugin):
# Load the cooker DB
cookerIndex = layerindexlib.LayerIndex(self.tinfoil.config_data)
- cookerIndex.load_layerindex('cooker://', load='layerDependencies')
+ cookerIndex.load_layerindex('cooker://', load=['layerDependencies'])
# Fast path, check if we already have what has been requested!
(dependencies, invalidnames) = cookerIndex.find_dependencies(names=args.layername, ignores=ignore_layers)
@@ -135,7 +137,7 @@ class LayerIndexPlugin(ActionPlugin):
for remoteurl in _construct_url(apiurl, branches):
logger.plain("Loading %s..." % remoteurl)
- remoteIndex.load_layerindex(remoteurl)
+ remoteIndex.load_layerindex(remoteurl, load=['layerDependencies'])
if remoteIndex.is_empty():
logger.error("Remote layer index %s is empty for branches %s" % (apiurl, branches))
diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py
index bfc18a75..7b8d1afd 100644
--- a/bitbake/lib/bblayers/query.py
+++ b/bitbake/lib/bblayers/query.py
@@ -52,7 +52,7 @@ layer, with the preferred version first. Note that skipped recipes that
are overlayed will also be listed, with a " (skipped)" suffix.
"""
- items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None, args.mc)
+ items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None, args.mc, False)
# Check for overlayed .bbclass files
classes = collections.defaultdict(list)
@@ -119,9 +119,9 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
title = 'Matching recipes:'
else:
title = 'Available recipes:'
- self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist, args.mc)
+ self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist, args.mc, args.show_variants)
- def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits, mc):
+ def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits, mc, show_variants=False):
if inherits:
bbpath = str(self.tinfoil.config_data.getVar('BBPATH'))
for classname in inherits:
@@ -142,10 +142,10 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
# Ensure we list skipped recipes
# We are largely guessing about PN, PV and the preferred version here,
# but we have no choice since skipped recipes are not fully parsed
- skiplist = list(self.tinfoil.cooker.skiplist.keys())
- mcspec = 'mc:%s:' % mc
+ skiplist = list(self.tinfoil.cooker.skiplist_by_mc[mc].keys())
+
if mc:
- skiplist = [s[len(mcspec):] for s in skiplist if s.startswith(mcspec)]
+ skiplist = [s.removeprefix(f'mc:{mc}:') for s in skiplist]
for fn in skiplist:
recipe_parts = os.path.splitext(os.path.basename(fn))[0].split('_')
@@ -162,7 +162,7 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
def print_item(f, pn, ver, layer, ispref):
if not selected_layer or layer == selected_layer:
if not bare and f in skiplist:
- skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist[f].skipreason
+ skipped = ' (skipped: %s)' % self.tinfoil.cooker.skiplist_by_mc[mc][f].skipreason
else:
skipped = ''
if show_filenames:
@@ -203,8 +203,10 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
# We only display once per recipe, we should prefer non extended versions of the
# recipe if present (so e.g. in OpenEmbedded, openssl rather than nativesdk-openssl
# which would otherwise sort first).
+ # However, if show_variants is True, we show all BBCLASSEXTEND variants.
if realfn[1] and realfn[0] in self.tinfoil.cooker.recipecaches[mc].pkg_fn:
- continue
+ if not show_variants:
+ continue
if inherits:
matchcount = 0
@@ -229,7 +231,11 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
for prov in allproviders[p]:
provfile = bb.cache.virtualfn2realfn(prov[1])[0]
provlayer = self.get_file_layer(provfile)
- provs.append((provfile, provlayer, prov[0]))
+ # When show_variants is True, use the virtual filename to distinguish BBCLASSEXTEND variants
+ if show_variants:
+ provs.append((prov[1], provlayer, prov[0]))
+ else:
+ provs.append((provfile, provlayer, prov[0]))
if provlayer != preflayer:
multilayer = True
if prov[0] != pref[0]:
@@ -238,10 +244,12 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
if not items_listed:
logger.plain('=== %s ===' % title)
items_listed = True
+ # When show_variants is True, use the virtual filename for preference check
+ pref_cmp_file = pref[1] if show_variants else preffile
print_item(preffile, p, self.version_str(pref[0][0], pref[0][1]), preflayer, True)
for (provfile, provlayer, provver) in provs:
- if provfile != preffile:
- print_item(provfile, p, self.version_str(provver[0], provver[1]), provlayer, False)
+ if provfile != pref_cmp_file:
+ print_item(bb.cache.virtualfn2realfn(provfile)[0] if show_variants else provfile, p, self.version_str(provver[0], provver[1]), provlayer, False)
# Ensure we don't show two entries for BBCLASSEXTENDed recipes
preffiles.append(preffile)
@@ -301,7 +309,7 @@ Lists recipes with the bbappends that apply to them as subitems.
if self.show_appends_for_pn(pn, cooker_data, args.mc):
appends = True
- if not args.pnspec and self.show_appends_for_skipped():
+ if not args.pnspec and self.show_appends_for_skipped(args.mc):
appends = True
if not appends:
@@ -317,9 +325,9 @@ Lists recipes with the bbappends that apply to them as subitems.
return self.show_appends_output(filenames, best_filename)
- def show_appends_for_skipped(self):
+ def show_appends_for_skipped(self, mc):
filenames = [os.path.basename(f)
- for f in self.tinfoil.cooker.skiplist.keys()]
+ for f in self.tinfoil.cooker.skiplist_by_mc[mc].keys()]
return self.show_appends_output(filenames, None, " (skipped)")
def show_appends_output(self, filenames, best_filename, name_suffix = ''):
@@ -530,6 +538,7 @@ NOTE: .bbappend files can impact the dependencies.
parser_show_recipes.add_argument('-i', '--inherits', help='only list recipes that inherit the named class(es) - separate multiple classes using , (without spaces)', metavar='CLASS', default='')
parser_show_recipes.add_argument('-l', '--layer', help='only list recipes from the selected layer', default='')
parser_show_recipes.add_argument('-b', '--bare', help='output just names without the "(skipped)" marker', action='store_true')
+ parser_show_recipes.add_argument('--show-variants', help='show all BBCLASSEXTEND recipe variants (e.g., devupstream, nativesdk)', action='store_true')
parser_show_recipes.add_argument('--mc', help='use specified multiconfig', default='')
parser_show_recipes.add_argument('pnspec', nargs='*', help='optional recipe name specification (wildcards allowed, enclose in quotes to avoid shell expansion)')
diff --git a/bitbake/lib/bs4/AUTHORS.txt b/bitbake/lib/bs4/AUTHORS.txt
deleted file mode 100644
index 2ac8fcc8..00000000
--- a/bitbake/lib/bs4/AUTHORS.txt
+++ /dev/null
@@ -1,43 +0,0 @@
-Behold, mortal, the origins of Beautiful Soup...
-================================================
-
-Leonard Richardson is the primary programmer.
-
-Aaron DeVore is awesome.
-
-Mark Pilgrim provided the encoding detection code that forms the base
-of UnicodeDammit.
-
-Thomas Kluyver and Ezio Melotti finished the work of getting Beautiful
-Soup 4 working under Python 3.
-
-Simon Willison wrote soupselect, which was used to make Beautiful Soup
-support CSS selectors.
-
-Sam Ruby helped with a lot of edge cases.
-
-Jonathan Ellis was awarded the prestigous Beau Potage D'Or for his
-work in solving the nestable tags conundrum.
-
-An incomplete list of people have contributed patches to Beautiful
-Soup:
-
- Istvan Albert, Andrew Lin, Anthony Baxter, Andrew Boyko, Tony Chang,
- Zephyr Fang, Fuzzy, Roman Gaufman, Yoni Gilad, Richie Hindle, Peteris
- Krumins, Kent Johnson, Ben Last, Robert Leftwich, Staffan Malmgren,
- Ksenia Marasanova, JP Moins, Adam Monsen, John Nagle, "Jon", Ed
- Oskiewicz, Greg Phillips, Giles Radford, Arthur Rudolph, Marko
- Samastur, Jouni Sepp�nen, Alexander Schmolck, Andy Theyers, Glyn
- Webster, Paul Wright, Danny Yoo
-
-An incomplete list of people who made suggestions or found bugs or
-found ways to break Beautiful Soup:
-
- Hanno B�ck, Matteo Bertini, Chris Curvey, Simon Cusack, Bruce Eckel,
- Matt Ernst, Michael Foord, Tom Harris, Bill de hOra, Donald Howes,
- Matt Patterson, Scott Roberts, Steve Strassmann, Mike Williams,
- warchild at redho dot com, Sami Kuisma, Carlos Rocha, Bob Hutchison,
- Joren Mc, Michal Migurski, John Kleven, Tim Heaney, Tripp Lilley, Ed
- Summers, Dennis Sutch, Chris Smith, Aaron Sweep^W Swartz, Stuart
- Turner, Greg Edwards, Kevin J Kalupson, Nikos Kouremenos, Artur de
- Sousa Rocha, Yichun Wei, Per Vognsen
diff --git a/bitbake/lib/bs4/NEWS.txt b/bitbake/lib/bs4/NEWS.txt
deleted file mode 100644
index 88a60a24..00000000
--- a/bitbake/lib/bs4/NEWS.txt
+++ /dev/null
@@ -1,1066 +0,0 @@
-= 4.3.2 (20131002) =
-
-* Fixed a bug in which short Unicode input was improperly encoded to
- ASCII when checking whether or not it was the name of a file on
- disk. [bug=1227016]
-
-* Fixed a crash when a short input contains data not valid in
- filenames. [bug=1232604]
-
-* Fixed a bug that caused Unicode data put into UnicodeDammit to
- return None instead of the original data. [bug=1214983]
-
-* Combined two tests to stop a spurious test failure when tests are
- run by nosetests. [bug=1212445]
-
-= 4.3.1 (20130815) =
-
-* Fixed yet another problem with the html5lib tree builder, caused by
- html5lib's tendency to rearrange the tree during
- parsing. [bug=1189267]
-
-* Fixed a bug that caused the optimized version of find_all() to
- return nothing. [bug=1212655]
-
-= 4.3.0 (20130812) =
-
-* Instead of converting incoming data to Unicode and feeding it to the
- lxml tree builder in chunks, Beautiful Soup now makes successive
- guesses at the encoding of the incoming data, and tells lxml to
- parse the data as that encoding. Giving lxml more control over the
- parsing process improves performance and avoids a number of bugs and
- issues with the lxml parser which had previously required elaborate
- workarounds:
-
- - An issue in which lxml refuses to parse Unicode strings on some
- systems. [bug=1180527]
-
- - A returning bug that truncated documents longer than a (very
- small) size. [bug=963880]
-
- - A returning bug in which extra spaces were added to a document if
- the document defined a charset other than UTF-8. [bug=972466]
-
- This required a major overhaul of the tree builder architecture. If
- you wrote your own tree builder and didn't tell me, you'll need to
- modify your prepare_markup() method.
-
-* The UnicodeDammit code that makes guesses at encodings has been
- split into its own class, EncodingDetector. A lot of apparently
- redundant code has been removed from Unicode, Dammit, and some
- undocumented features have also been removed.
-
-* Beautiful Soup will issue a warning if instead of markup you pass it
- a URL or the name of a file on disk (a common beginner's mistake).
-
-* A number of optimizations improve the performance of the lxml tree
- builder by about 33%, the html.parser tree builder by about 20%, and
- the html5lib tree builder by about 15%.
-
-* All find_all calls should now return a ResultSet object. Patch by
- Aaron DeVore. [bug=1194034]
-
-= 4.2.1 (20130531) =
-
-* The default XML formatter will now replace ampersands even if they
- appear to be part of entities. That is, "<" will become
- "&lt;". The old code was left over from Beautiful Soup 3, which
- didn't always turn entities into Unicode characters.
-
- If you really want the old behavior (maybe because you add new
- strings to the tree, those strings include entities, and you want
- the formatter to leave them alone on output), it can be found in
- EntitySubstitution.substitute_xml_containing_entities(). [bug=1182183]
-
-* Gave new_string() the ability to create subclasses of
- NavigableString. [bug=1181986]
-
-* Fixed another bug by which the html5lib tree builder could create a
- disconnected tree. [bug=1182089]
-
-* The .previous_element of a BeautifulSoup object is now always None,
- not the last element to be parsed. [bug=1182089]
-
-* Fixed test failures when lxml is not installed. [bug=1181589]
-
-* html5lib now supports Python 3. Fixed some Python 2-specific
- code in the html5lib test suite. [bug=1181624]
-
-* The html.parser treebuilder can now handle numeric attributes in
- text when the hexidecimal name of the attribute starts with a
- capital X. Patch by Tim Shirley. [bug=1186242]
-
-= 4.2.0 (20130514) =
-
-* The Tag.select() method now supports a much wider variety of CSS
- selectors.
-
- - Added support for the adjacent sibling combinator (+) and the
- general sibling combinator (~). Tests by "liquider". [bug=1082144]
-
- - The combinators (>, +, and ~) can now combine with any supported
- selector, not just one that selects based on tag name.
-
- - Added limited support for the "nth-of-type" pseudo-class. Code
- by Sven Slootweg. [bug=1109952]
-
-* The BeautifulSoup class is now aliased to "_s" and "_soup", making
- it quicker to type the import statement in an interactive session:
-
- from bs4 import _s
- or
- from bs4 import _soup
-
- The alias may change in the future, so don't use this in code you're
- going to run more than once.
-
-* Added the 'diagnose' submodule, which includes several useful
- functions for reporting problems and doing tech support.
-
- - diagnose(data) tries the given markup on every installed parser,
- reporting exceptions and displaying successes. If a parser is not
- installed, diagnose() mentions this fact.
-
- - lxml_trace(data, html=True) runs the given markup through lxml's
- XML parser or HTML parser, and prints out the parser events as
- they happen. This helps you quickly determine whether a given
- problem occurs in lxml code or Beautiful Soup code.
-
- - htmlparser_trace(data) is the same thing, but for Python's
- built-in HTMLParser class.
-
-* In an HTML document, the contents of a <script> or <style> tag will
- no longer undergo entity substitution by default. XML documents work
- the same way they did before. [bug=1085953]
-
-* Methods like get_text() and properties like .strings now only give
- you strings that are visible in the document--no comments or
- processing commands. [bug=1050164]
-
-* The prettify() method now leaves the contents of <pre> tags
- alone. [bug=1095654]
-
-* Fix a bug in the html5lib treebuilder which sometimes created
- disconnected trees. [bug=1039527]
-
-* Fix a bug in the lxml treebuilder which crashed when a tag included
- an attribute from the predefined "xml:" namespace. [bug=1065617]
-
-* Fix a bug by which keyword arguments to find_parent() were not
- being passed on. [bug=1126734]
-
-* Stop a crash when unwisely messing with a tag that's been
- decomposed. [bug=1097699]
-
-* Now that lxml's segfault on invalid doctype has been fixed, fixed a
- corresponding problem on the Beautiful Soup end that was previously
- invisible. [bug=984936]
-
-* Fixed an exception when an overspecified CSS selector didn't match
- anything. Code by Stefaan Lippens. [bug=1168167]
-
-= 4.1.3 (20120820) =
-
-* Skipped a test under Python 2.6 and Python 3.1 to avoid a spurious
- test failure caused by the lousy HTMLParser in those
- versions. [bug=1038503]
-
-* Raise a more specific error (FeatureNotFound) when a requested
- parser or parser feature is not installed. Raise NotImplementedError
- instead of ValueError when the user calls insert_before() or
- insert_after() on the BeautifulSoup object itself. Patch by Aaron
- Devore. [bug=1038301]
-
-= 4.1.2 (20120817) =
-
-* As per PEP-8, allow searching by CSS class using the 'class_'
- keyword argument. [bug=1037624]
-
-* Display namespace prefixes for namespaced attribute names, instead of
- the fully-qualified names given by the lxml parser. [bug=1037597]
-
-* Fixed a crash on encoding when an attribute name contained
- non-ASCII characters.
-
-* When sniffing encodings, if the cchardet library is installed,
- Beautiful Soup uses it instead of chardet. cchardet is much
- faster. [bug=1020748]
-
-* Use logging.warning() instead of warning.warn() to notify the user
- that characters were replaced with REPLACEMENT
- CHARACTER. [bug=1013862]
-
-= 4.1.1 (20120703) =
-
-* Fixed an html5lib tree builder crash which happened when html5lib
- moved a tag with a multivalued attribute from one part of the tree
- to another. [bug=1019603]
-
-* Correctly display closing tags with an XML namespace declared. Patch
- by Andreas Kostyrka. [bug=1019635]
-
-* Fixed a typo that made parsing significantly slower than it should
- have been, and also waited too long to close tags with XML
- namespaces. [bug=1020268]
-
-* get_text() now returns an empty Unicode string if there is no text,
- rather than an empty bytestring. [bug=1020387]
-
-= 4.1.0 (20120529) =
-
-* Added experimental support for fixing Windows-1252 characters
- embedded in UTF-8 documents. (UnicodeDammit.detwingle())
-
-* Fixed the handling of " with the built-in parser. [bug=993871]
-
-* Comments, processing instructions, document type declarations, and
- markup declarations are now treated as preformatted strings, the way
- CData blocks are. [bug=1001025]
-
-* Fixed a bug with the lxml treebuilder that prevented the user from
- adding attributes to a tag that didn't originally have
- attributes. [bug=1002378] Thanks to Oliver Beattie for the patch.
-
-* Fixed some edge-case bugs having to do with inserting an element
- into a tag it's already inside, and replacing one of a tag's
- children with another. [bug=997529]
-
-* Added the ability to search for attribute values specified in UTF-8. [bug=1003974]
-
- This caused a major refactoring of the search code. All the tests
- pass, but it's possible that some searches will behave differently.
-
-= 4.0.5 (20120427) =
-
-* Added a new method, wrap(), which wraps an element in a tag.
-
-* Renamed replace_with_children() to unwrap(), which is easier to
- understand and also the jQuery name of the function.
-
-* Made encoding substitution in <meta> tags completely transparent (no
- more %SOUP-ENCODING%).
-
-* Fixed a bug in decoding data that contained a byte-order mark, such
- as data encoded in UTF-16LE. [bug=988980]
-
-* Fixed a bug that made the HTMLParser treebuilder generate XML
- definitions ending with two question marks instead of
- one. [bug=984258]
-
-* Upon document generation, CData objects are no longer run through
- the formatter. [bug=988905]
-
-* The test suite now passes when lxml is not installed, whether or not
- html5lib is installed. [bug=987004]
-
-* Print a warning on HTMLParseErrors to let people know they should
- install a better parser library.
-
-= 4.0.4 (20120416) =
-
-* Fixed a bug that sometimes created disconnected trees.
-
-* Fixed a bug with the string setter that moved a string around the
- tree instead of copying it. [bug=983050]
-
-* Attribute values are now run through the provided output formatter.
- Previously they were always run through the 'minimal' formatter. In
- the future I may make it possible to specify different formatters
- for attribute values and strings, but for now, consistent behavior
- is better than inconsistent behavior. [bug=980237]
-
-* Added the missing renderContents method from Beautiful Soup 3. Also
- added an encode_contents() method to go along with decode_contents().
-
-* Give a more useful error when the user tries to run the Python 2
- version of BS under Python 3.
-
-* UnicodeDammit can now convert Microsoft smart quotes to ASCII with
- UnicodeDammit(markup, smart_quotes_to="ascii").
-
-= 4.0.3 (20120403) =
-
-* Fixed a typo that caused some versions of Python 3 to convert the
- Beautiful Soup codebase incorrectly.
-
-* Got rid of the 4.0.2 workaround for HTML documents--it was
- unnecessary and the workaround was triggering a (possibly different,
- but related) bug in lxml. [bug=972466]
-
-= 4.0.2 (20120326) =
-
-* Worked around a possible bug in lxml that prevents non-tiny XML
- documents from being parsed. [bug=963880, bug=963936]
-
-* Fixed a bug where specifying `text` while also searching for a tag
- only worked if `text` wanted an exact string match. [bug=955942]
-
-= 4.0.1 (20120314) =
-
-* This is the first official release of Beautiful Soup 4. There is no
- 4.0.0 release, to eliminate any possibility that packaging software
- might treat "4.0.0" as being an earlier version than "4.0.0b10".
-
-* Brought BS up to date with the latest release of soupselect, adding
- CSS selector support for direct descendant matches and multiple CSS
- class matches.
-
-= 4.0.0b10 (20120302) =
-
-* Added support for simple CSS selectors, taken from the soupselect project.
-
-* Fixed a crash when using html5lib. [bug=943246]
-
-* In HTML5-style <meta charset="foo"> tags, the value of the "charset"
- attribute is now replaced with the appropriate encoding on
- output. [bug=942714]
-
-* Fixed a bug that caused calling a tag to sometimes call find_all()
- with the wrong arguments. [bug=944426]
-
-* For backwards compatibility, brought back the BeautifulStoneSoup
- class as a deprecated wrapper around BeautifulSoup.
-
-= 4.0.0b9 (20120228) =
-
-* Fixed the string representation of DOCTYPEs that have both a public
- ID and a system ID.
-
-* Fixed the generated XML declaration.
-
-* Renamed Tag.nsprefix to Tag.prefix, for consistency with
- NamespacedAttribute.
-
-* Fixed a test failure that occured on Python 3.x when chardet was
- installed.
-
-* Made prettify() return Unicode by default, so it will look nice on
- Python 3 when passed into print().
-
-= 4.0.0b8 (20120224) =
-
-* All tree builders now preserve namespace information in the
- documents they parse. If you use the html5lib parser or lxml's XML
- parser, you can access the namespace URL for a tag as tag.namespace.
-
- However, there is no special support for namespace-oriented
- searching or tree manipulation. When you search the tree, you need
- to use namespace prefixes exactly as they're used in the original
- document.
-
-* The string representation of a DOCTYPE always ends in a newline.
-
-* Issue a warning if the user tries to use a SoupStrainer in
- conjunction with the html5lib tree builder, which doesn't support
- them.
-
-= 4.0.0b7 (20120223) =
-
-* Upon decoding to string, any characters that can't be represented in
- your chosen encoding will be converted into numeric XML entity
- references.
-
-* Issue a warning if characters were replaced with REPLACEMENT
- CHARACTER during Unicode conversion.
-
-* Restored compatibility with Python 2.6.
-
-* The install process no longer installs docs or auxillary text files.
-
-* It's now possible to deepcopy a BeautifulSoup object created with
- Python's built-in HTML parser.
-
-* About 100 unit tests that "test" the behavior of various parsers on
- invalid markup have been removed. Legitimate changes to those
- parsers caused these tests to fail, indicating that perhaps
- Beautiful Soup should not test the behavior of foreign
- libraries.
-
- The problematic unit tests have been reformulated as informational
- comparisons generated by the script
- scripts/demonstrate_parser_differences.py.
-
- This makes Beautiful Soup compatible with html5lib version 0.95 and
- future versions of HTMLParser.
-
-= 4.0.0b6 (20120216) =
-
-* Multi-valued attributes like "class" always have a list of values,
- even if there's only one value in the list.
-
-* Added a number of multi-valued attributes defined in HTML5.
-
-* Stopped generating a space before the slash that closes an
- empty-element tag. This may come back if I add a special XHTML mode
- (http://www.w3.org/TR/xhtml1/#C_2), but right now it's pretty
- useless.
-
-* Passing text along with tag-specific arguments to a find* method:
-
- find("a", text="Click here")
-
- will find tags that contain the given text as their
- .string. Previously, the tag-specific arguments were ignored and
- only strings were searched.
-
-* Fixed a bug that caused the html5lib tree builder to build a
- partially disconnected tree. Generally cleaned up the html5lib tree
- builder.
-
-* If you restrict a multi-valued attribute like "class" to a string
- that contains spaces, Beautiful Soup will only consider it a match
- if the values correspond to that specific string.
-
-= 4.0.0b5 (20120209) =
-
-* Rationalized Beautiful Soup's treatment of CSS class. A tag
- belonging to multiple CSS classes is treated as having a list of
- values for the 'class' attribute. Searching for a CSS class will
- match *any* of the CSS classes.
-
- This actually affects all attributes that the HTML standard defines
- as taking multiple values (class, rel, rev, archive, accept-charset,
- and headers), but 'class' is by far the most common. [bug=41034]
-
-* If you pass anything other than a dictionary as the second argument
- to one of the find* methods, it'll assume you want to use that
- object to search against a tag's CSS classes. Previously this only
- worked if you passed in a string.
-
-* Fixed a bug that caused a crash when you passed a dictionary as an
- attribute value (possibly because you mistyped "attrs"). [bug=842419]
-
-* Unicode, Dammit now detects the encoding in HTML 5-style <meta> tags
- like <meta charset="utf-8" />. [bug=837268]
-
-* If Unicode, Dammit can't figure out a consistent encoding for a
- page, it will try each of its guesses again, with errors="replace"
- instead of errors="strict". This may mean that some data gets
- replaced with REPLACEMENT CHARACTER, but at least most of it will
- get turned into Unicode. [bug=754903]
-
-* Patched over a bug in html5lib (?) that was crashing Beautiful Soup
- on certain kinds of markup. [bug=838800]
-
-* Fixed a bug that wrecked the tree if you replaced an element with an
- empty string. [bug=728697]
-
-* Improved Unicode, Dammit's behavior when you give it Unicode to
- begin with.
-
-= 4.0.0b4 (20120208) =
-
-* Added BeautifulSoup.new_string() to go along with BeautifulSoup.new_tag()
-
-* BeautifulSoup.new_tag() will follow the rules of whatever
- tree-builder was used to create the original BeautifulSoup object. A
- new <p> tag will look like "<p />" if the soup object was created to
- parse XML, but it will look like "<p></p>" if the soup object was
- created to parse HTML.
-
-* We pass in strict=False to html.parser on Python 3, greatly
- improving html.parser's ability to handle bad HTML.
-
-* We also monkeypatch a serious bug in html.parser that made
- strict=False disastrous on Python 3.2.2.
-
-* Replaced the "substitute_html_entities" argument with the
- more general "formatter" argument.
-
-* Bare ampersands and angle brackets are always converted to XML
- entities unless the user prevents it.
-
-* Added PageElement.insert_before() and PageElement.insert_after(),
- which let you put an element into the parse tree with respect to
- some other element.
-
-* Raise an exception when the user tries to do something nonsensical
- like insert a tag into itself.
-
-
-= 4.0.0b3 (20120203) =
-
-Beautiful Soup 4 is a nearly-complete rewrite that removes Beautiful
-Soup's custom HTML parser in favor of a system that lets you write a
-little glue code and plug in any HTML or XML parser you want.
-
-Beautiful Soup 4.0 comes with glue code for four parsers:
-
- * Python's standard HTMLParser (html.parser in Python 3)
- * lxml's HTML and XML parsers
- * html5lib's HTML parser
-
-HTMLParser is the default, but I recommend you install lxml if you
-can.
-
-For complete documentation, see the Sphinx documentation in
-bs4/doc/source/. What follows is a summary of the changes from
-Beautiful Soup 3.
-
-=== The module name has changed ===
-
-Previously you imported the BeautifulSoup class from a module also
-called BeautifulSoup. To save keystrokes and make it clear which
-version of the API is in use, the module is now called 'bs4':
-
- >>> from bs4 import BeautifulSoup
-
-=== It works with Python 3 ===
-
-Beautiful Soup 3.1.0 worked with Python 3, but the parser it used was
-so bad that it barely worked at all. Beautiful Soup 4 works with
-Python 3, and since its parser is pluggable, you don't sacrifice
-quality.
-
-Special thanks to Thomas Kluyver and Ezio Melotti for getting Python 3
-support to the finish line. Ezio Melotti is also to thank for greatly
-improving the HTML parser that comes with Python 3.2.
-
-=== CDATA sections are normal text, if they're understood at all. ===
-
-Currently, the lxml and html5lib HTML parsers ignore CDATA sections in
-markup:
-
- <p><![CDATA[foo]]></p> => <p></p>
-
-A future version of html5lib will turn CDATA sections into text nodes,
-but only within tags like <svg> and <math>:
-
- <svg><![CDATA[foo]]></svg> => <p>foo</p>
-
-The default XML parser (which uses lxml behind the scenes) turns CDATA
-sections into ordinary text elements:
-
- <p><![CDATA[foo]]></p> => <p>foo</p>
-
-In theory it's possible to preserve the CDATA sections when using the
-XML parser, but I don't see how to get it to work in practice.
-
-=== Miscellaneous other stuff ===
-
-If the BeautifulSoup instance has .is_xml set to True, an appropriate
-XML declaration will be emitted when the tree is transformed into a
-string:
-
- <?xml version="1.0" encoding="utf-8">
- <markup>
- ...
- </markup>
-
-The ['lxml', 'xml'] tree builder sets .is_xml to True; the other tree
-builders set it to False. If you want to parse XHTML with an HTML
-parser, you can set it manually.
-
-
-= 3.2.0 =
-
-The 3.1 series wasn't very useful, so I renamed the 3.0 series to 3.2
-to make it obvious which one you should use.
-
-= 3.1.0 =
-
-A hybrid version that supports 2.4 and can be automatically converted
-to run under Python 3.0. There are three backwards-incompatible
-changes you should be aware of, but no new features or deliberate
-behavior changes.
-
-1. str() may no longer do what you want. This is because the meaning
-of str() inverts between Python 2 and 3; in Python 2 it gives you a
-byte string, in Python 3 it gives you a Unicode string.
-
-The effect of this is that you can't pass an encoding to .__str__
-anymore. Use encode() to get a string and decode() to get Unicode, and
-you'll be ready (well, readier) for Python 3.
-
-2. Beautiful Soup is now based on HTMLParser rather than SGMLParser,
-which is gone in Python 3. There's some bad HTML that SGMLParser
-handled but HTMLParser doesn't, usually to do with attribute values
-that aren't closed or have brackets inside them:
-
- <a href="foo</a>, </a><a href="bar">baz</a>
- <a b="<a>">', '<a b="<a>"></a><a>"></a>
-
-A later version of Beautiful Soup will allow you to plug in different
-parsers to make tradeoffs between speed and the ability to handle bad
-HTML.
-
-3. In Python 3 (but not Python 2), HTMLParser converts entities within
-attributes to the corresponding Unicode characters. In Python 2 it's
-possible to parse this string and leave the é intact.
-
- <a href="http://crummy.com?sacré&bleu">
-
-In Python 3, the é is always converted to \xe9 during
-parsing.
-
-
-= 3.0.7a =
-
-Added an import that makes BS work in Python 2.3.
-
-
-= 3.0.7 =
-
-Fixed a UnicodeDecodeError when unpickling documents that contain
-non-ASCII characters.
-
-Fixed a TypeError that occured in some circumstances when a tag
-contained no text.
-
-Jump through hoops to avoid the use of chardet, which can be extremely
-slow in some circumstances. UTF-8 documents should never trigger the
-use of chardet.
-
-Whitespace is preserved inside <pre> and <textarea> tags that contain
-nothing but whitespace.
-
-Beautiful Soup can now parse a doctype that's scoped to an XML namespace.
-
-
-= 3.0.6 =
-
-Got rid of a very old debug line that prevented chardet from working.
-
-Added a Tag.decompose() method that completely disconnects a tree or a
-subset of a tree, breaking it up into bite-sized pieces that are
-easy for the garbage collecter to collect.
-
-Tag.extract() now returns the tag that was extracted.
-
-Tag.findNext() now does something with the keyword arguments you pass
-it instead of dropping them on the floor.
-
-Fixed a Unicode conversion bug.
-
-Fixed a bug that garbled some <meta> tags when rewriting them.
-
-
-= 3.0.5 =
-
-Soup objects can now be pickled, and copied with copy.deepcopy.
-
-Tag.append now works properly on existing BS objects. (It wasn't
-originally intended for outside use, but it can be now.) (Giles
-Radford)
-
-Passing in a nonexistent encoding will no longer crash the parser on
-Python 2.4 (John Nagle).
-
-Fixed an underlying bug in SGMLParser that thinks ASCII has 255
-characters instead of 127 (John Nagle).
-
-Entities are converted more consistently to Unicode characters.
-
-Entity references in attribute values are now converted to Unicode
-characters when appropriate. Numeric entities are always converted,
-because SGMLParser always converts them outside of attribute values.
-
-ALL_ENTITIES happens to just be the XHTML entities, so I renamed it to
-XHTML_ENTITIES.
-
-The regular expression for bare ampersands was too loose. In some
-cases ampersands were not being escaped. (Sam Ruby?)
-
-Non-breaking spaces and other special Unicode space characters are no
-longer folded to ASCII spaces. (Robert Leftwich)
-
-Information inside a TEXTAREA tag is now parsed literally, not as HTML
-tags. TEXTAREA now works exactly the same way as SCRIPT. (Zephyr Fang)
-
-= 3.0.4 =
-
-Fixed a bug that crashed Unicode conversion in some cases.
-
-Fixed a bug that prevented UnicodeDammit from being used as a
-general-purpose data scrubber.
-
-Fixed some unit test failures when running against Python 2.5.
-
-When considering whether to convert smart quotes, UnicodeDammit now
-looks at the original encoding in a case-insensitive way.
-
-= 3.0.3 (20060606) =
-
-Beautiful Soup is now usable as a way to clean up invalid XML/HTML (be
-sure to pass in an appropriate value for convertEntities, or XML/HTML
-entities might stick around that aren't valid in HTML/XML). The result
-may not validate, but it should be good enough to not choke a
-real-world XML parser. Specifically, the output of a properly
-constructed soup object should always be valid as part of an XML
-document, but parts may be missing if they were missing in the
-original. As always, if the input is valid XML, the output will also
-be valid.
-
-= 3.0.2 (20060602) =
-
-Previously, Beautiful Soup correctly handled attribute values that
-contained embedded quotes (sometimes by escaping), but not other kinds
-of XML character. Now, it correctly handles or escapes all special XML
-characters in attribute values.
-
-I aliased methods to the 2.x names (fetch, find, findText, etc.) for
-backwards compatibility purposes. Those names are deprecated and if I
-ever do a 4.0 I will remove them. I will, I tell you!
-
-Fixed a bug where the findAll method wasn't passing along any keyword
-arguments.
-
-When run from the command line, Beautiful Soup now acts as an HTML
-pretty-printer, not an XML pretty-printer.
-
-= 3.0.1 (20060530) =
-
-Reintroduced the "fetch by CSS class" shortcut. I thought keyword
-arguments would replace it, but they don't. You can't call soup('a',
-class='foo') because class is a Python keyword.
-
-If Beautiful Soup encounters a meta tag that declares the encoding,
-but a SoupStrainer tells it not to parse that tag, Beautiful Soup will
-no longer try to rewrite the meta tag to mention the new
-encoding. Basically, this makes SoupStrainers work in real-world
-applications instead of crashing the parser.
-
-= 3.0.0 "Who would not give all else for two p" (20060528) =
-
-This release is not backward-compatible with previous releases. If
-you've got code written with a previous version of the library, go
-ahead and keep using it, unless one of the features mentioned here
-really makes your life easier. Since the library is self-contained,
-you can include an old copy of the library in your old applications,
-and use the new version for everything else.
-
-The documentation has been rewritten and greatly expanded with many
-more examples.
-
-Beautiful Soup autodetects the encoding of a document (or uses the one
-you specify), and converts it from its native encoding to
-Unicode. Internally, it only deals with Unicode strings. When you
-print out the document, it converts to UTF-8 (or another encoding you
-specify). [Doc reference]
-
-It's now easy to make large-scale changes to the parse tree without
-screwing up the navigation members. The methods are extract,
-replaceWith, and insert. [Doc reference. See also Improving Memory
-Usage with extract]
-
-Passing True in as an attribute value gives you tags that have any
-value for that attribute. You don't have to create a regular
-expression. Passing None for an attribute value gives you tags that
-don't have that attribute at all.
-
-Tag objects now know whether or not they're self-closing. This avoids
-the problem where Beautiful Soup thought that tags like <BR /> were
-self-closing even in XML documents. You can customize the self-closing
-tags for a parser object by passing them in as a list of
-selfClosingTags: you don't have to subclass anymore.
-
-There's a new built-in parser, MinimalSoup, which has most of
-BeautifulSoup's HTML-specific rules, but no tag nesting rules. [Doc
-reference]
-
-You can use a SoupStrainer to tell Beautiful Soup to parse only part
-of a document. This saves time and memory, often making Beautiful Soup
-about as fast as a custom-built SGMLParser subclass. [Doc reference,
-SoupStrainer reference]
-
-You can (usually) use keyword arguments instead of passing a
-dictionary of attributes to a search method. That is, you can replace
-soup(args={"id" : "5"}) with soup(id="5"). You can still use args if
-(for instance) you need to find an attribute whose name clashes with
-the name of an argument to findAll. [Doc reference: **kwargs attrs]
-
-The method names have changed to the better method names used in
-Rubyful Soup. Instead of find methods and fetch methods, there are
-only find methods. Instead of a scheme where you can't remember which
-method finds one element and which one finds them all, we have find
-and findAll. In general, if the method name mentions All or a plural
-noun (eg. findNextSiblings), then it finds many elements
-method. Otherwise, it only finds one element. [Doc reference]
-
-Some of the argument names have been renamed for clarity. For instance
-avoidParserProblems is now parserMassage.
-
-Beautiful Soup no longer implements a feed method. You need to pass a
-string or a filehandle into the soup constructor, not with feed after
-the soup has been created. There is still a feed method, but it's the
-feed method implemented by SGMLParser and calling it will bypass
-Beautiful Soup and cause problems.
-
-The NavigableText class has been renamed to NavigableString. There is
-no NavigableUnicodeString anymore, because every string inside a
-Beautiful Soup parse tree is a Unicode string.
-
-findText and fetchText are gone. Just pass a text argument into find
-or findAll.
-
-Null was more trouble than it was worth, so I got rid of it. Anything
-that used to return Null now returns None.
-
-Special XML constructs like comments and CDATA now have their own
-NavigableString subclasses, instead of being treated as oddly-formed
-data. If you parse a document that contains CDATA and write it back
-out, the CDATA will still be there.
-
-When you're parsing a document, you can get Beautiful Soup to convert
-XML or HTML entities into the corresponding Unicode characters. [Doc
-reference]
-
-= 2.1.1 (20050918) =
-
-Fixed a serious performance bug in BeautifulStoneSoup which was
-causing parsing to be incredibly slow.
-
-Corrected several entities that were previously being incorrectly
-translated from Microsoft smart-quote-like characters.
-
-Fixed a bug that was breaking text fetch.
-
-Fixed a bug that crashed the parser when text chunks that look like
-HTML tag names showed up within a SCRIPT tag.
-
-THEAD, TBODY, and TFOOT tags are now nestable within TABLE
-tags. Nested tables should parse more sensibly now.
-
-BASE is now considered a self-closing tag.
-
-= 2.1.0 "Game, or any other dish?" (20050504) =
-
-Added a wide variety of new search methods which, given a starting
-point inside the tree, follow a particular navigation member (like
-nextSibling) over and over again, looking for Tag and NavigableText
-objects that match certain criteria. The new methods are findNext,
-fetchNext, findPrevious, fetchPrevious, findNextSibling,
-fetchNextSiblings, findPreviousSibling, fetchPreviousSiblings,
-findParent, and fetchParents. All of these use the same basic code
-used by first and fetch, so you can pass your weird ways of matching
-things into these methods.
-
-The fetch method and its derivatives now accept a limit argument.
-
-You can now pass keyword arguments when calling a Tag object as though
-it were a method.
-
-Fixed a bug that caused all hand-created tags to share a single set of
-attributes.
-
-= 2.0.3 (20050501) =
-
-Fixed Python 2.2 support for iterators.
-
-Fixed a bug that gave the wrong representation to tags within quote
-tags like <script>.
-
-Took some code from Mark Pilgrim that treats CDATA declarations as
-data instead of ignoring them.
-
-Beautiful Soup's setup.py will now do an install even if the unit
-tests fail. It won't build a source distribution if the unit tests
-fail, so I can't release a new version unless they pass.
-
-= 2.0.2 (20050416) =
-
-Added the unit tests in a separate module, and packaged it with
-distutils.
-
-Fixed a bug that sometimes caused renderContents() to return a Unicode
-string even if there was no Unicode in the original string.
-
-Added the done() method, which closes all of the parser's open
-tags. It gets called automatically when you pass in some text to the
-constructor of a parser class; otherwise you must call it yourself.
-
-Reinstated some backwards compatibility with 1.x versions: referencing
-the string member of a NavigableText object returns the NavigableText
-object instead of throwing an error.
-
-= 2.0.1 (20050412) =
-
-Fixed a bug that caused bad results when you tried to reference a tag
-name shorter than 3 characters as a member of a Tag, eg. tag.table.td.
-
-Made sure all Tags have the 'hidden' attribute so that an attempt to
-access tag.hidden doesn't spawn an attempt to find a tag named
-'hidden'.
-
-Fixed a bug in the comparison operator.
-
-= 2.0.0 "Who cares for fish?" (20050410)
-
-Beautiful Soup version 1 was very useful but also pretty stupid. I
-originally wrote it without noticing any of the problems inherent in
-trying to build a parse tree out of ambiguous HTML tags. This version
-solves all of those problems to my satisfaction. It also adds many new
-clever things to make up for the removal of the stupid things.
-
-== Parsing ==
-
-The parser logic has been greatly improved, and the BeautifulSoup
-class should much more reliably yield a parse tree that looks like
-what the page author intended. For a particular class of odd edge
-cases that now causes problems, there is a new class,
-ICantBelieveItsBeautifulSoup.
-
-By default, Beautiful Soup now performs some cleanup operations on
-text before parsing it. This is to avoid common problems with bad
-definitions and self-closing tags that crash SGMLParser. You can
-provide your own set of cleanup operations, or turn it off
-altogether. The cleanup operations include fixing self-closing tags
-that don't close, and replacing Microsoft smart quotes and similar
-characters with their HTML entity equivalents.
-
-You can now get a pretty-print version of parsed HTML to get a visual
-picture of how Beautiful Soup parses it, with the Tag.prettify()
-method.
-
-== Strings and Unicode ==
-
-There are separate NavigableText subclasses for ASCII and Unicode
-strings. These classes directly subclass the corresponding base data
-types. This means you can treat NavigableText objects as strings
-instead of having to call methods on them to get the strings.
-
-str() on a Tag always returns a string, and unicode() always returns
-Unicode. Previously it was inconsistent.
-
-== Tree traversal ==
-
-In a first() or fetch() call, the tag name or the desired value of an
-attribute can now be any of the following:
-
- * A string (matches that specific tag or that specific attribute value)
- * A list of strings (matches any tag or attribute value in the list)
- * A compiled regular expression object (matches any tag or attribute
- value that matches the regular expression)
- * A callable object that takes the Tag object or attribute value as a
- string. It returns None/false/empty string if the given string
- doesn't match, and any other value if it does.
-
-This is much easier to use than SQL-style wildcards (see, regular
-expressions are good for something). Because of this, I took out
-SQL-style wildcards. I'll put them back if someone complains, but
-their removal simplifies the code a lot.
-
-You can use fetch() and first() to search for text in the parse tree,
-not just tags. There are new alias methods fetchText() and firstText()
-designed for this purpose. As with searching for tags, you can pass in
-a string, a regular expression object, or a method to match your text.
-
-If you pass in something besides a map to the attrs argument of
-fetch() or first(), Beautiful Soup will assume you want to match that
-thing against the "class" attribute. When you're scraping
-well-structured HTML, this makes your code a lot cleaner.
-
-1.x and 2.x both let you call a Tag object as a shorthand for
-fetch(). For instance, foo("bar") is a shorthand for
-foo.fetch("bar"). In 2.x, you can also access a specially-named member
-of a Tag object as a shorthand for first(). For instance, foo.barTag
-is a shorthand for foo.first("bar"). By chaining these shortcuts you
-traverse a tree in very little code: for header in
-soup.bodyTag.pTag.tableTag('th'):
-
-If an element relationship (like parent or next) doesn't apply to a
-tag, it'll now show up Null instead of None. first() will also return
-Null if you ask it for a nonexistent tag. Null is an object that's
-just like None, except you can do whatever you want to it and it'll
-give you Null instead of throwing an error.
-
-This lets you do tree traversals like soup.htmlTag.headTag.titleTag
-without having to worry if the intermediate stages are actually
-there. Previously, if there was no 'head' tag in the document, headTag
-in that instance would have been None, and accessing its 'titleTag'
-member would have thrown an AttributeError. Now, you can get what you
-want when it exists, and get Null when it doesn't, without having to
-do a lot of conditionals checking to see if every stage is None.
-
-There are two new relations between page elements: previousSibling and
-nextSibling. They reference the previous and next element at the same
-level of the parse tree. For instance, if you have HTML like this:
-
- <p><ul><li>Foo<br /><li>Bar</ul>
-
-The first 'li' tag has a previousSibling of Null and its nextSibling
-is the second 'li' tag. The second 'li' tag has a nextSibling of Null
-and its previousSibling is the first 'li' tag. The previousSibling of
-the 'ul' tag is the first 'p' tag. The nextSibling of 'Foo' is the
-'br' tag.
-
-I took out the ability to use fetch() to find tags that have a
-specific list of contents. See, I can't even explain it well. It was
-really difficult to use, I never used it, and I don't think anyone
-else ever used it. To the extent anyone did, they can probably use
-fetchText() instead. If it turns out someone needs it I'll think of
-another solution.
-
-== Tree manipulation ==
-
-You can add new attributes to a tag, and delete attributes from a
-tag. In 1.x you could only change a tag's existing attributes.
-
-== Porting Considerations ==
-
-There are three changes in 2.0 that break old code:
-
-In the post-1.2 release you could pass in a function into fetch(). The
-function took a string, the tag name. In 2.0, the function takes the
-actual Tag object.
-
-It's no longer to pass in SQL-style wildcards to fetch(). Use a
-regular expression instead.
-
-The different parsing algorithm means the parse tree may not be shaped
-like you expect. This will only actually affect you if your code uses
-one of the affected parts. I haven't run into this problem yet while
-porting my code.
-
-= Between 1.2 and 2.0 =
-
-This is the release to get if you want Python 1.5 compatibility.
-
-The desired value of an attribute can now be any of the following:
-
- * A string
- * A string with SQL-style wildcards
- * A compiled RE object
- * A callable that returns None/false/empty string if the given value
- doesn't match, and any other value otherwise.
-
-This is much easier to use than SQL-style wildcards (see, regular
-expressions are good for something). Because of this, I no longer
-recommend you use SQL-style wildcards. They may go away in a future
-release to clean up the code.
-
-Made Beautiful Soup handle processing instructions as text instead of
-ignoring them.
-
-Applied patch from Richie Hindle (richie at entrian dot com) that
-makes tag.string a shorthand for tag.contents[0].string when the tag
-has only one string-owning child.
-
-Added still more nestable tags. The nestable tags thing won't work in
-a lot of cases and needs to be rethought.
-
-Fixed an edge case where searching for "%foo" would match any string
-shorter than "foo".
-
-= 1.2 "Who for such dainties would not stoop?" (20040708) =
-
-Applied patch from Ben Last (ben at benlast dot com) that made
-Tag.renderContents() correctly handle Unicode.
-
-Made BeautifulStoneSoup even dumber by making it not implicitly close
-a tag when another tag of the same type is encountered; only when an
-actual closing tag is encountered. This change courtesy of Fuzzy (mike
-at pcblokes dot com). BeautifulSoup still works as before.
-
-= 1.1 "Swimming in a hot tureen" =
-
-Added more 'nestable' tags. Changed popping semantics so that when a
-nestable tag is encountered, tags are popped up to the previously
-encountered nestable tag (of whatever kind). I will revert this if
-enough people complain, but it should make more people's lives easier
-than harder. This enhancement was suggested by Anthony Baxter (anthony
-at interlink dot com dot au).
-
-= 1.0 "So rich and green" (20040420) =
-
-Initial release.
diff --git a/bitbake/lib/bs4/__init__.py b/bitbake/lib/bs4/__init__.py
deleted file mode 100644
index e35725b8..00000000
--- a/bitbake/lib/bs4/__init__.py
+++ /dev/null
@@ -1,468 +0,0 @@
-"""Beautiful Soup
-Elixir and Tonic
-"The Screen-Scraper's Friend"
-http://www.crummy.com/software/BeautifulSoup/
-
-Beautiful Soup uses a pluggable XML or HTML parser to parse a
-(possibly invalid) document into a tree representation. Beautiful Soup
-provides provides methods and Pythonic idioms that make it easy to
-navigate, search, and modify the parse tree.
-
-Beautiful Soup works with Python 2.6 and up. It works better if lxml
-and/or html5lib is installed.
-
-For more than you ever wanted to know about Beautiful Soup, see the
-documentation:
-http://www.crummy.com/software/BeautifulSoup/bs4/doc/
-"""
-
-__author__ = "Leonard Richardson (leonardr@segfault.org)"
-__version__ = "4.4.1"
-__copyright__ = "Copyright (c) 2004-2015 Leonard Richardson"
-__license__ = "MIT"
-
-__all__ = ['BeautifulSoup']
-
-import os
-import re
-import warnings
-
-from .builder import builder_registry, ParserRejectedMarkup
-from .dammit import UnicodeDammit
-from .element import (
- CData,
- Comment,
- DEFAULT_OUTPUT_ENCODING,
- Declaration,
- Doctype,
- NavigableString,
- PageElement,
- ProcessingInstruction,
- ResultSet,
- SoupStrainer,
- Tag,
- )
-
-# The very first thing we do is give a useful error if someone is
-# running this code under Python 3 without converting it.
-'You are trying to run the Python 2 version of Beautiful Soup under Python 3. This will not work.'!='You need to convert the code, either by installing it (`python setup.py install`) or by running 2to3 (`2to3 -w bs4`).'
-
-class BeautifulSoup(Tag):
- """
- This class defines the basic interface called by the tree builders.
-
- These methods will be called by the parser:
- reset()
- feed(markup)
-
- The tree builder may call these methods from its feed() implementation:
- handle_starttag(name, attrs) # See note about return value
- handle_endtag(name)
- handle_data(data) # Appends to the current data node
- endData(containerClass=NavigableString) # Ends the current data node
-
- No matter how complicated the underlying parser is, you should be
- able to build a tree using 'start tag' events, 'end tag' events,
- 'data' events, and "done with data" events.
-
- If you encounter an empty-element tag (aka a self-closing tag,
- like HTML's <br> tag), call handle_starttag and then
- handle_endtag.
- """
- ROOT_TAG_NAME = '[document]'
-
- # If the end-user gives no indication which tree builder they
- # want, look for one with these features.
- DEFAULT_BUILDER_FEATURES = ['html', 'fast']
-
- ASCII_SPACES = '\x20\x0a\x09\x0c\x0d'
-
- NO_PARSER_SPECIFIED_WARNING = "No parser was explicitly specified, so I'm using the best available %(markup_type)s parser for this system (\"%(parser)s\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n\nTo get rid of this warning, change this:\n\n BeautifulSoup([your markup])\n\nto this:\n\n BeautifulSoup([your markup], \"%(parser)s\")\n"
-
- def __init__(self, markup="", features=None, builder=None,
- parse_only=None, from_encoding=None, exclude_encodings=None,
- **kwargs):
- """The Soup object is initialized as the 'root tag', and the
- provided markup (which can be a string or a file-like object)
- is fed into the underlying parser."""
-
- if 'convertEntities' in kwargs:
- warnings.warn(
- "BS4 does not respect the convertEntities argument to the "
- "BeautifulSoup constructor. Entities are always converted "
- "to Unicode characters.")
-
- if 'markupMassage' in kwargs:
- del kwargs['markupMassage']
- warnings.warn(
- "BS4 does not respect the markupMassage argument to the "
- "BeautifulSoup constructor. The tree builder is responsible "
- "for any necessary markup massage.")
-
- if 'smartQuotesTo' in kwargs:
- del kwargs['smartQuotesTo']
- warnings.warn(
- "BS4 does not respect the smartQuotesTo argument to the "
- "BeautifulSoup constructor. Smart quotes are always converted "
- "to Unicode characters.")
-
- if 'selfClosingTags' in kwargs:
- del kwargs['selfClosingTags']
- warnings.warn(
- "BS4 does not respect the selfClosingTags argument to the "
- "BeautifulSoup constructor. The tree builder is responsible "
- "for understanding self-closing tags.")
-
- if 'isHTML' in kwargs:
- del kwargs['isHTML']
- warnings.warn(
- "BS4 does not respect the isHTML argument to the "
- "BeautifulSoup constructor. Suggest you use "
- "features='lxml' for HTML and features='lxml-xml' for "
- "XML.")
-
- def deprecated_argument(old_name, new_name):
- if old_name in kwargs:
- warnings.warn(
- 'The "%s" argument to the BeautifulSoup constructor '
- 'has been renamed to "%s."' % (old_name, new_name))
- value = kwargs[old_name]
- del kwargs[old_name]
- return value
- return None
-
- parse_only = parse_only or deprecated_argument(
- "parseOnlyThese", "parse_only")
-
- from_encoding = from_encoding or deprecated_argument(
- "fromEncoding", "from_encoding")
-
- if len(kwargs) > 0:
- arg = list(kwargs.keys()).pop()
- raise TypeError(
- "__init__() got an unexpected keyword argument '%s'" % arg)
-
- if builder is None:
- original_features = features
- if isinstance(features, str):
- features = [features]
- if features is None or len(features) == 0:
- features = self.DEFAULT_BUILDER_FEATURES
- builder_class = builder_registry.lookup(*features)
- if builder_class is None:
- raise FeatureNotFound(
- "Couldn't find a tree builder with the features you "
- "requested: %s. Do you need to install a parser library?"
- % ",".join(features))
- builder = builder_class()
- if not (original_features == builder.NAME or
- original_features in builder.ALTERNATE_NAMES):
- if builder.is_xml:
- markup_type = "XML"
- else:
- markup_type = "HTML"
- warnings.warn(self.NO_PARSER_SPECIFIED_WARNING % dict(
- parser=builder.NAME,
- markup_type=markup_type))
-
- self.builder = builder
- self.is_xml = builder.is_xml
- self.builder.soup = self
-
- self.parse_only = parse_only
-
- if hasattr(markup, 'read'): # It's a file-type object.
- markup = markup.read()
- elif len(markup) <= 256:
- # Print out warnings for a couple beginner problems
- # involving passing non-markup to Beautiful Soup.
- # Beautiful Soup will still parse the input as markup,
- # just in case that's what the user really wants.
- if (isinstance(markup, str)
- and not os.path.supports_unicode_filenames):
- possible_filename = markup.encode("utf8")
- else:
- possible_filename = markup
- is_file = False
- try:
- is_file = os.path.exists(possible_filename)
- except Exception as e:
- # This is almost certainly a problem involving
- # characters not valid in filenames on this
- # system. Just let it go.
- pass
- if is_file:
- if isinstance(markup, str):
- markup = markup.encode("utf8")
- warnings.warn(
- '"%s" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)
- if markup[:5] == "http:" or markup[:6] == "https:":
- # TODO: This is ugly but I couldn't get it to work in
- # Python 3 otherwise.
- if ((isinstance(markup, bytes) and not b' ' in markup)
- or (isinstance(markup, str) and not ' ' in markup)):
- if isinstance(markup, str):
- markup = markup.encode("utf8")
- warnings.warn(
- '"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
-
- for (self.markup, self.original_encoding, self.declared_html_encoding,
- self.contains_replacement_characters) in (
- self.builder.prepare_markup(
- markup, from_encoding, exclude_encodings=exclude_encodings)):
- self.reset()
- try:
- self._feed()
- break
- except ParserRejectedMarkup:
- pass
-
- # Clear out the markup and remove the builder's circular
- # reference to this object.
- self.markup = None
- self.builder.soup = None
-
- def __copy__(self):
- return type(self)(self.encode(), builder=self.builder)
-
- def __getstate__(self):
- # Frequently a tree builder can't be pickled.
- d = dict(self.__dict__)
- if 'builder' in d and not self.builder.picklable:
- del d['builder']
- return d
-
- def _feed(self):
- # Convert the document to Unicode.
- self.builder.reset()
-
- self.builder.feed(self.markup)
- # Close out any unfinished strings and close all the open tags.
- self.endData()
- while self.currentTag.name != self.ROOT_TAG_NAME:
- self.popTag()
-
- def reset(self):
- Tag.__init__(self, self, self.builder, self.ROOT_TAG_NAME)
- self.hidden = 1
- self.builder.reset()
- self.current_data = []
- self.currentTag = None
- self.tagStack = []
- self.preserve_whitespace_tag_stack = []
- self.pushTag(self)
-
- def new_tag(self, name, namespace=None, nsprefix=None, **attrs):
- """Create a new tag associated with this soup."""
- return Tag(None, self.builder, name, namespace, nsprefix, attrs)
-
- def new_string(self, s, subclass=NavigableString):
- """Create a new NavigableString associated with this soup."""
- return subclass(s)
-
- def insert_before(self, successor):
- raise NotImplementedError("BeautifulSoup objects don't support insert_before().")
-
- def insert_after(self, successor):
- raise NotImplementedError("BeautifulSoup objects don't support insert_after().")
-
- def popTag(self):
- tag = self.tagStack.pop()
- if self.preserve_whitespace_tag_stack and tag == self.preserve_whitespace_tag_stack[-1]:
- self.preserve_whitespace_tag_stack.pop()
- #print "Pop", tag.name
- if self.tagStack:
- self.currentTag = self.tagStack[-1]
- return self.currentTag
-
- def pushTag(self, tag):
- #print "Push", tag.name
- if self.currentTag:
- self.currentTag.contents.append(tag)
- self.tagStack.append(tag)
- self.currentTag = self.tagStack[-1]
- if tag.name in self.builder.preserve_whitespace_tags:
- self.preserve_whitespace_tag_stack.append(tag)
-
- def endData(self, containerClass=NavigableString):
- if self.current_data:
- current_data = ''.join(self.current_data)
- # If whitespace is not preserved, and this string contains
- # nothing but ASCII spaces, replace it with a single space
- # or newline.
- if not self.preserve_whitespace_tag_stack:
- strippable = True
- for i in current_data:
- if i not in self.ASCII_SPACES:
- strippable = False
- break
- if strippable:
- if '\n' in current_data:
- current_data = '\n'
- else:
- current_data = ' '
-
- # Reset the data collector.
- self.current_data = []
-
- # Should we add this string to the tree at all?
- if self.parse_only and len(self.tagStack) <= 1 and \
- (not self.parse_only.text or \
- not self.parse_only.search(current_data)):
- return
-
- o = containerClass(current_data)
- self.object_was_parsed(o)
-
- def object_was_parsed(self, o, parent=None, most_recent_element=None):
- """Add an object to the parse tree."""
- parent = parent or self.currentTag
- previous_element = most_recent_element or self._most_recent_element
-
- next_element = previous_sibling = next_sibling = None
- if isinstance(o, Tag):
- next_element = o.next_element
- next_sibling = o.next_sibling
- previous_sibling = o.previous_sibling
- if not previous_element:
- previous_element = o.previous_element
-
- o.setup(parent, previous_element, next_element, previous_sibling, next_sibling)
-
- self._most_recent_element = o
- parent.contents.append(o)
-
- if parent.next_sibling:
- # This node is being inserted into an element that has
- # already been parsed. Deal with any dangling references.
- index = parent.contents.index(o)
- if index == 0:
- previous_element = parent
- previous_sibling = None
- else:
- previous_element = previous_sibling = parent.contents[index-1]
- if index == len(parent.contents)-1:
- next_element = parent.next_sibling
- next_sibling = None
- else:
- next_element = next_sibling = parent.contents[index+1]
-
- o.previous_element = previous_element
- if previous_element:
- previous_element.next_element = o
- o.next_element = next_element
- if next_element:
- next_element.previous_element = o
- o.next_sibling = next_sibling
- if next_sibling:
- next_sibling.previous_sibling = o
- o.previous_sibling = previous_sibling
- if previous_sibling:
- previous_sibling.next_sibling = o
-
- def _popToTag(self, name, nsprefix=None, inclusivePop=True):
- """Pops the tag stack up to and including the most recent
- instance of the given tag. If inclusivePop is false, pops the tag
- stack up to but *not* including the most recent instqance of
- the given tag."""
- #print "Popping to %s" % name
- if name == self.ROOT_TAG_NAME:
- # The BeautifulSoup object itself can never be popped.
- return
-
- most_recently_popped = None
-
- stack_size = len(self.tagStack)
- for i in range(stack_size - 1, 0, -1):
- t = self.tagStack[i]
- if (name == t.name and nsprefix == t.prefix):
- if inclusivePop:
- most_recently_popped = self.popTag()
- break
- most_recently_popped = self.popTag()
-
- return most_recently_popped
-
- def handle_starttag(self, name, namespace, nsprefix, attrs):
- """Push a start tag on to the stack.
-
- If this method returns None, the tag was rejected by the
- SoupStrainer. You should proceed as if the tag had not occured
- in the document. For instance, if this was a self-closing tag,
- don't call handle_endtag.
- """
-
- # print "Start tag %s: %s" % (name, attrs)
- self.endData()
-
- if (self.parse_only and len(self.tagStack) <= 1
- and (self.parse_only.text
- or not self.parse_only.search_tag(name, attrs))):
- return None
-
- tag = Tag(self, self.builder, name, namespace, nsprefix, attrs,
- self.currentTag, self._most_recent_element)
- if tag is None:
- return tag
- if self._most_recent_element:
- self._most_recent_element.next_element = tag
- self._most_recent_element = tag
- self.pushTag(tag)
- return tag
-
- def handle_endtag(self, name, nsprefix=None):
- #print "End tag: " + name
- self.endData()
- self._popToTag(name, nsprefix)
-
- def handle_data(self, data):
- self.current_data.append(data)
-
- def decode(self, pretty_print=False,
- eventual_encoding=DEFAULT_OUTPUT_ENCODING,
- formatter="minimal"):
- """Returns a string or Unicode representation of this document.
- To get Unicode, pass None for encoding."""
-
- if self.is_xml:
- # Print the XML declaration
- encoding_part = ''
- if eventual_encoding is not None:
- encoding_part = ' encoding="%s"' % eventual_encoding
- prefix = '<?xml version="1.0"%s?>\n' % encoding_part
- else:
- prefix = ''
- if not pretty_print:
- indent_level = None
- else:
- indent_level = 0
- return prefix + super(BeautifulSoup, self).decode(
- indent_level, eventual_encoding, formatter)
-
-# Alias to make it easier to type import: 'from bs4 import _soup'
-_s = BeautifulSoup
-_soup = BeautifulSoup
-
-class BeautifulStoneSoup(BeautifulSoup):
- """Deprecated interface to an XML parser."""
-
- def __init__(self, *args, **kwargs):
- kwargs['features'] = 'xml'
- warnings.warn(
- 'The BeautifulStoneSoup class is deprecated. Instead of using '
- 'it, pass features="xml" into the BeautifulSoup constructor.')
- super(BeautifulStoneSoup, self).__init__(*args, **kwargs)
-
-
-class StopParsing(Exception):
- pass
-
-class FeatureNotFound(ValueError):
- pass
-
-
-#By default, act as an HTML pretty-printer.
-if __name__ == '__main__':
- import sys
- soup = BeautifulSoup(sys.stdin)
- print(soup.prettify())
diff --git a/bitbake/lib/bs4/builder/__init__.py b/bitbake/lib/bs4/builder/__init__.py
deleted file mode 100644
index 6ccd4d23..00000000
--- a/bitbake/lib/bs4/builder/__init__.py
+++ /dev/null
@@ -1,324 +0,0 @@
-from collections import defaultdict
-import itertools
-import sys
-from bs4.element import (
- CharsetMetaAttributeValue,
- ContentMetaAttributeValue,
- whitespace_re
- )
-
-__all__ = [
- 'HTMLTreeBuilder',
- 'SAXTreeBuilder',
- 'TreeBuilder',
- 'TreeBuilderRegistry',
- ]
-
-# Some useful features for a TreeBuilder to have.
-FAST = 'fast'
-PERMISSIVE = 'permissive'
-STRICT = 'strict'
-XML = 'xml'
-HTML = 'html'
-HTML_5 = 'html5'
-
-
-class TreeBuilderRegistry(object):
-
- def __init__(self):
- self.builders_for_feature = defaultdict(list)
- self.builders = []
-
- def register(self, treebuilder_class):
- """Register a treebuilder based on its advertised features."""
- for feature in treebuilder_class.features:
- self.builders_for_feature[feature].insert(0, treebuilder_class)
- self.builders.insert(0, treebuilder_class)
-
- def lookup(self, *features):
- if len(self.builders) == 0:
- # There are no builders at all.
- return None
-
- if len(features) == 0:
- # They didn't ask for any features. Give them the most
- # recently registered builder.
- return self.builders[0]
-
- # Go down the list of features in order, and eliminate any builders
- # that don't match every feature.
- features = list(features)
- features.reverse()
- candidates = None
- candidate_set = None
- while len(features) > 0:
- feature = features.pop()
- we_have_the_feature = self.builders_for_feature.get(feature, [])
- if len(we_have_the_feature) > 0:
- if candidates is None:
- candidates = we_have_the_feature
- candidate_set = set(candidates)
- else:
- # Eliminate any candidates that don't have this feature.
- candidate_set = candidate_set.intersection(
- set(we_have_the_feature))
-
- # The only valid candidates are the ones in candidate_set.
- # Go through the original list of candidates and pick the first one
- # that's in candidate_set.
- if candidate_set is None:
- return None
- for candidate in candidates:
- if candidate in candidate_set:
- return candidate
- return None
-
-# The BeautifulSoup class will take feature lists from developers and use them
-# to look up builders in this registry.
-builder_registry = TreeBuilderRegistry()
-
-class TreeBuilder(object):
- """Turn a document into a Beautiful Soup object tree."""
-
- NAME = "[Unknown tree builder]"
- ALTERNATE_NAMES = []
- features = []
-
- is_xml = False
- picklable = False
- preserve_whitespace_tags = set()
- empty_element_tags = None # A tag will be considered an empty-element
- # tag when and only when it has no contents.
-
- # A value for these tag/attribute combinations is a space- or
- # comma-separated list of CDATA, rather than a single CDATA.
- cdata_list_attributes = {}
-
-
- def __init__(self):
- self.soup = None
-
- def reset(self):
- pass
-
- def can_be_empty_element(self, tag_name):
- """Might a tag with this name be an empty-element tag?
-
- The final markup may or may not actually present this tag as
- self-closing.
-
- For instance: an HTMLBuilder does not consider a <p> tag to be
- an empty-element tag (it's not in
- HTMLBuilder.empty_element_tags). This means an empty <p> tag
- will be presented as "<p></p>", not "<p />".
-
- The default implementation has no opinion about which tags are
- empty-element tags, so a tag will be presented as an
- empty-element tag if and only if it has no contents.
- "<foo></foo>" will become "<foo />", and "<foo>bar</foo>" will
- be left alone.
- """
- if self.empty_element_tags is None:
- return True
- return tag_name in self.empty_element_tags
-
- def feed(self, markup):
- raise NotImplementedError()
-
- def prepare_markup(self, markup, user_specified_encoding=None,
- document_declared_encoding=None):
- return markup, None, None, False
-
- def test_fragment_to_document(self, fragment):
- """Wrap an HTML fragment to make it look like a document.
-
- Different parsers do this differently. For instance, lxml
- introduces an empty <head> tag, and html5lib
- doesn't. Abstracting this away lets us write simple tests
- which run HTML fragments through the parser and compare the
- results against other HTML fragments.
-
- This method should not be used outside of tests.
- """
- return fragment
-
- def set_up_substitutions(self, tag):
- return False
-
- def _replace_cdata_list_attribute_values(self, tag_name, attrs):
- """Replaces class="foo bar" with class=["foo", "bar"]
-
- Modifies its input in place.
- """
- if not attrs:
- return attrs
- if self.cdata_list_attributes:
- universal = self.cdata_list_attributes.get('*', [])
- tag_specific = self.cdata_list_attributes.get(
- tag_name.lower(), None)
- for attr in list(attrs.keys()):
- if attr in universal or (tag_specific and attr in tag_specific):
- # We have a "class"-type attribute whose string
- # value is a whitespace-separated list of
- # values. Split it into a list.
- value = attrs[attr]
- if isinstance(value, str):
- values = whitespace_re.split(value)
- else:
- # html5lib sometimes calls setAttributes twice
- # for the same tag when rearranging the parse
- # tree. On the second call the attribute value
- # here is already a list. If this happens,
- # leave the value alone rather than trying to
- # split it again.
- values = value
- attrs[attr] = values
- return attrs
-
-class SAXTreeBuilder(TreeBuilder):
- """A Beautiful Soup treebuilder that listens for SAX events."""
-
- def feed(self, markup):
- raise NotImplementedError()
-
- def close(self):
- pass
-
- def startElement(self, name, attrs):
- attrs = dict((key[1], value) for key, value in list(attrs.items()))
- #print "Start %s, %r" % (name, attrs)
- self.soup.handle_starttag(name, attrs)
-
- def endElement(self, name):
- #print "End %s" % name
- self.soup.handle_endtag(name)
-
- def startElementNS(self, nsTuple, nodeName, attrs):
- # Throw away (ns, nodeName) for now.
- self.startElement(nodeName, attrs)
-
- def endElementNS(self, nsTuple, nodeName):
- # Throw away (ns, nodeName) for now.
- self.endElement(nodeName)
- #handler.endElementNS((ns, node.nodeName), node.nodeName)
-
- def startPrefixMapping(self, prefix, nodeValue):
- # Ignore the prefix for now.
- pass
-
- def endPrefixMapping(self, prefix):
- # Ignore the prefix for now.
- # handler.endPrefixMapping(prefix)
- pass
-
- def characters(self, content):
- self.soup.handle_data(content)
-
- def startDocument(self):
- pass
-
- def endDocument(self):
- pass
-
-
-class HTMLTreeBuilder(TreeBuilder):
- """This TreeBuilder knows facts about HTML.
-
- Such as which tags are empty-element tags.
- """
-
- preserve_whitespace_tags = set(['pre', 'textarea'])
- empty_element_tags = set(['br' , 'hr', 'input', 'img', 'meta',
- 'spacer', 'link', 'frame', 'base'])
-
- # The HTML standard defines these attributes as containing a
- # space-separated list of values, not a single value. That is,
- # class="foo bar" means that the 'class' attribute has two values,
- # 'foo' and 'bar', not the single value 'foo bar'. When we
- # encounter one of these attributes, we will parse its value into
- # a list of values if possible. Upon output, the list will be
- # converted back into a string.
- cdata_list_attributes = {
- "*" : ['class', 'accesskey', 'dropzone'],
- "a" : ['rel', 'rev'],
- "link" : ['rel', 'rev'],
- "td" : ["headers"],
- "th" : ["headers"],
- "td" : ["headers"],
- "form" : ["accept-charset"],
- "object" : ["archive"],
-
- # These are HTML5 specific, as are *.accesskey and *.dropzone above.
- "area" : ["rel"],
- "icon" : ["sizes"],
- "iframe" : ["sandbox"],
- "output" : ["for"],
- }
-
- def set_up_substitutions(self, tag):
- # We are only interested in <meta> tags
- if tag.name != 'meta':
- return False
-
- http_equiv = tag.get('http-equiv')
- content = tag.get('content')
- charset = tag.get('charset')
-
- # We are interested in <meta> tags that say what encoding the
- # document was originally in. This means HTML 5-style <meta>
- # tags that provide the "charset" attribute. It also means
- # HTML 4-style <meta> tags that provide the "content"
- # attribute and have "http-equiv" set to "content-type".
- #
- # In both cases we will replace the value of the appropriate
- # attribute with a standin object that can take on any
- # encoding.
- meta_encoding = None
- if charset is not None:
- # HTML 5 style:
- # <meta charset="utf8">
- meta_encoding = charset
- tag['charset'] = CharsetMetaAttributeValue(charset)
-
- elif (content is not None and http_equiv is not None
- and http_equiv.lower() == 'content-type'):
- # HTML 4 style:
- # <meta http-equiv="content-type" content="text/html; charset=utf8">
- tag['content'] = ContentMetaAttributeValue(content)
-
- return (meta_encoding is not None)
-
-def register_treebuilders_from(module):
- """Copy TreeBuilders from the given module into this module."""
- # I'm fairly sure this is not the best way to do this.
- this_module = sys.modules['bs4.builder']
- for name in module.__all__:
- obj = getattr(module, name)
-
- if issubclass(obj, TreeBuilder):
- setattr(this_module, name, obj)
- this_module.__all__.append(name)
- # Register the builder while we're at it.
- this_module.builder_registry.register(obj)
-
-class ParserRejectedMarkup(Exception):
- pass
-
-# Builders are registered in reverse order of priority, so that custom
-# builder registrations will take precedence. In general, we want lxml
-# to take precedence over html5lib, because it's faster. And we only
-# want to use HTMLParser as a last result.
-from . import _htmlparser
-register_treebuilders_from(_htmlparser)
-try:
- from . import _html5lib
- register_treebuilders_from(_html5lib)
-except ImportError:
- # They don't have html5lib installed.
- pass
-try:
- from . import _lxml
- register_treebuilders_from(_lxml)
-except ImportError:
- # They don't have lxml installed.
- pass
diff --git a/bitbake/lib/bs4/builder/_html5lib.py b/bitbake/lib/bs4/builder/_html5lib.py
deleted file mode 100644
index 9e9216ef..00000000
--- a/bitbake/lib/bs4/builder/_html5lib.py
+++ /dev/null
@@ -1,338 +0,0 @@
-__all__ = [
- 'HTML5TreeBuilder',
- ]
-
-import warnings
-from bs4.builder import (
- PERMISSIVE,
- HTML,
- HTML_5,
- HTMLTreeBuilder,
- )
-from bs4.element import (
- NamespacedAttribute,
- whitespace_re,
-)
-import html5lib
-try:
- # html5lib >= 0.99999999/1.0b9
- from html5lib.treebuilders import base as treebuildersbase
-except ImportError:
- # html5lib <= 0.9999999/1.0b8
- from html5lib.treebuilders import _base as treebuildersbase
-from html5lib.constants import namespaces
-
-from bs4.element import (
- Comment,
- Doctype,
- NavigableString,
- Tag,
- )
-
-class HTML5TreeBuilder(HTMLTreeBuilder):
- """Use html5lib to build a tree."""
-
- NAME = "html5lib"
-
- features = [NAME, PERMISSIVE, HTML_5, HTML]
-
- def prepare_markup(self, markup, user_specified_encoding,
- document_declared_encoding=None, exclude_encodings=None):
- # Store the user-specified encoding for use later on.
- self.user_specified_encoding = user_specified_encoding
-
- # document_declared_encoding and exclude_encodings aren't used
- # ATM because the html5lib TreeBuilder doesn't use
- # UnicodeDammit.
- if exclude_encodings:
- warnings.warn("You provided a value for exclude_encoding, but the html5lib tree builder doesn't support exclude_encoding.")
- yield (markup, None, None, False)
-
- # These methods are defined by Beautiful Soup.
- def feed(self, markup):
- if self.soup.parse_only is not None:
- warnings.warn("You provided a value for parse_only, but the html5lib tree builder doesn't support parse_only. The entire document will be parsed.")
- parser = html5lib.HTMLParser(tree=self.create_treebuilder)
- doc = parser.parse(markup, encoding=self.user_specified_encoding)
-
- # Set the character encoding detected by the tokenizer.
- if isinstance(markup, str):
- # We need to special-case this because html5lib sets
- # charEncoding to UTF-8 if it gets Unicode input.
- doc.original_encoding = None
- else:
- doc.original_encoding = parser.tokenizer.stream.charEncoding[0]
-
- def create_treebuilder(self, namespaceHTMLElements):
- self.underlying_builder = TreeBuilderForHtml5lib(
- self.soup, namespaceHTMLElements)
- return self.underlying_builder
-
- def test_fragment_to_document(self, fragment):
- """See `TreeBuilder`."""
- return '<html><head></head><body>%s</body></html>' % fragment
-
-
-class TreeBuilderForHtml5lib(treebuildersbase.TreeBuilder):
-
- def __init__(self, soup, namespaceHTMLElements):
- self.soup = soup
- super(TreeBuilderForHtml5lib, self).__init__(namespaceHTMLElements)
-
- def documentClass(self):
- self.soup.reset()
- return Element(self.soup, self.soup, None)
-
- def insertDoctype(self, token):
- name = token["name"]
- publicId = token["publicId"]
- systemId = token["systemId"]
-
- doctype = Doctype.for_name_and_ids(name, publicId, systemId)
- self.soup.object_was_parsed(doctype)
-
- def elementClass(self, name, namespace):
- tag = self.soup.new_tag(name, namespace)
- return Element(tag, self.soup, namespace)
-
- def commentClass(self, data):
- return TextNode(Comment(data), self.soup)
-
- def fragmentClass(self):
- self.soup = BeautifulSoup("")
- self.soup.name = "[document_fragment]"
- return Element(self.soup, self.soup, None)
-
- def appendChild(self, node):
- # XXX This code is not covered by the BS4 tests.
- self.soup.append(node.element)
-
- def getDocument(self):
- return self.soup
-
- def getFragment(self):
- return treebuildersbase.TreeBuilder.getFragment(self).element
-
-class AttrList(object):
- def __init__(self, element):
- self.element = element
- self.attrs = dict(self.element.attrs)
- def __iter__(self):
- return list(self.attrs.items()).__iter__()
- def __setitem__(self, name, value):
- # If this attribute is a multi-valued attribute for this element,
- # turn its value into a list.
- list_attr = HTML5TreeBuilder.cdata_list_attributes
- if (name in list_attr['*']
- or (self.element.name in list_attr
- and name in list_attr[self.element.name])):
- # A node that is being cloned may have already undergone
- # this procedure.
- if not isinstance(value, list):
- value = whitespace_re.split(value)
- self.element[name] = value
- def items(self):
- return list(self.attrs.items())
- def keys(self):
- return list(self.attrs.keys())
- def __len__(self):
- return len(self.attrs)
- def __getitem__(self, name):
- return self.attrs[name]
- def __contains__(self, name):
- return name in list(self.attrs.keys())
-
-
-class Element(treebuildersbase.Node):
- def __init__(self, element, soup, namespace):
- treebuildersbase.Node.__init__(self, element.name)
- self.element = element
- self.soup = soup
- self.namespace = namespace
-
- def appendChild(self, node):
- string_child = child = None
- if isinstance(node, str):
- # Some other piece of code decided to pass in a string
- # instead of creating a TextElement object to contain the
- # string.
- string_child = child = node
- elif isinstance(node, Tag):
- # Some other piece of code decided to pass in a Tag
- # instead of creating an Element object to contain the
- # Tag.
- child = node
- elif node.element.__class__ == NavigableString:
- string_child = child = node.element
- else:
- child = node.element
-
- if not isinstance(child, str) and child.parent is not None:
- node.element.extract()
-
- if (string_child and self.element.contents
- and self.element.contents[-1].__class__ == NavigableString):
- # We are appending a string onto another string.
- # TODO This has O(n^2) performance, for input like
- # "a</a>a</a>a</a>..."
- old_element = self.element.contents[-1]
- new_element = self.soup.new_string(old_element + string_child)
- old_element.replace_with(new_element)
- self.soup._most_recent_element = new_element
- else:
- if isinstance(node, str):
- # Create a brand new NavigableString from this string.
- child = self.soup.new_string(node)
-
- # Tell Beautiful Soup to act as if it parsed this element
- # immediately after the parent's last descendant. (Or
- # immediately after the parent, if it has no children.)
- if self.element.contents:
- most_recent_element = self.element._last_descendant(False)
- elif self.element.next_element is not None:
- # Something from further ahead in the parse tree is
- # being inserted into this earlier element. This is
- # very annoying because it means an expensive search
- # for the last element in the tree.
- most_recent_element = self.soup._last_descendant()
- else:
- most_recent_element = self.element
-
- self.soup.object_was_parsed(
- child, parent=self.element,
- most_recent_element=most_recent_element)
-
- def getAttributes(self):
- return AttrList(self.element)
-
- def setAttributes(self, attributes):
-
- if attributes is not None and len(attributes) > 0:
-
- converted_attributes = []
- for name, value in list(attributes.items()):
- if isinstance(name, tuple):
- new_name = NamespacedAttribute(*name)
- del attributes[name]
- attributes[new_name] = value
-
- self.soup.builder._replace_cdata_list_attribute_values(
- self.name, attributes)
- for name, value in list(attributes.items()):
- self.element[name] = value
-
- # The attributes may contain variables that need substitution.
- # Call set_up_substitutions manually.
- #
- # The Tag constructor called this method when the Tag was created,
- # but we just set/changed the attributes, so call it again.
- self.soup.builder.set_up_substitutions(self.element)
- attributes = property(getAttributes, setAttributes)
-
- def insertText(self, data, insertBefore=None):
- if insertBefore:
- text = TextNode(self.soup.new_string(data), self.soup)
- self.insertBefore(data, insertBefore)
- else:
- self.appendChild(data)
-
- def insertBefore(self, node, refNode):
- index = self.element.index(refNode.element)
- if (node.element.__class__ == NavigableString and self.element.contents
- and self.element.contents[index-1].__class__ == NavigableString):
- # (See comments in appendChild)
- old_node = self.element.contents[index-1]
- new_str = self.soup.new_string(old_node + node.element)
- old_node.replace_with(new_str)
- else:
- self.element.insert(index, node.element)
- node.parent = self
-
- def removeChild(self, node):
- node.element.extract()
-
- def reparentChildren(self, new_parent):
- """Move all of this tag's children into another tag."""
- # print "MOVE", self.element.contents
- # print "FROM", self.element
- # print "TO", new_parent.element
- element = self.element
- new_parent_element = new_parent.element
- # Determine what this tag's next_element will be once all the children
- # are removed.
- final_next_element = element.next_sibling
-
- new_parents_last_descendant = new_parent_element._last_descendant(False, False)
- if len(new_parent_element.contents) > 0:
- # The new parent already contains children. We will be
- # appending this tag's children to the end.
- new_parents_last_child = new_parent_element.contents[-1]
- new_parents_last_descendant_next_element = new_parents_last_descendant.next_element
- else:
- # The new parent contains no children.
- new_parents_last_child = None
- new_parents_last_descendant_next_element = new_parent_element.next_element
-
- to_append = element.contents
- append_after = new_parent_element.contents
- if len(to_append) > 0:
- # Set the first child's previous_element and previous_sibling
- # to elements within the new parent
- first_child = to_append[0]
- if new_parents_last_descendant:
- first_child.previous_element = new_parents_last_descendant
- else:
- first_child.previous_element = new_parent_element
- first_child.previous_sibling = new_parents_last_child
- if new_parents_last_descendant:
- new_parents_last_descendant.next_element = first_child
- else:
- new_parent_element.next_element = first_child
- if new_parents_last_child:
- new_parents_last_child.next_sibling = first_child
-
- # Fix the last child's next_element and next_sibling
- last_child = to_append[-1]
- last_child.next_element = new_parents_last_descendant_next_element
- if new_parents_last_descendant_next_element:
- new_parents_last_descendant_next_element.previous_element = last_child
- last_child.next_sibling = None
-
- for child in to_append:
- child.parent = new_parent_element
- new_parent_element.contents.append(child)
-
- # Now that this element has no children, change its .next_element.
- element.contents = []
- element.next_element = final_next_element
-
- # print "DONE WITH MOVE"
- # print "FROM", self.element
- # print "TO", new_parent_element
-
- def cloneNode(self):
- tag = self.soup.new_tag(self.element.name, self.namespace)
- node = Element(tag, self.soup, self.namespace)
- for key,value in self.attributes:
- node.attributes[key] = value
- return node
-
- def hasContent(self):
- return self.element.contents
-
- def getNameTuple(self):
- if self.namespace is None:
- return namespaces["html"], self.name
- else:
- return self.namespace, self.name
-
- nameTuple = property(getNameTuple)
-
-class TextNode(Element):
- def __init__(self, element, soup):
- treebuildersbase.Node.__init__(self, None)
- self.element = element
- self.soup = soup
-
- def cloneNode(self):
- raise NotImplementedError
diff --git a/bitbake/lib/bs4/builder/_htmlparser.py b/bitbake/lib/bs4/builder/_htmlparser.py
deleted file mode 100644
index bb0a63f2..00000000
--- a/bitbake/lib/bs4/builder/_htmlparser.py
+++ /dev/null
@@ -1,262 +0,0 @@
-"""Use the HTMLParser library to parse HTML files that aren't too bad."""
-
-__all__ = [
- 'HTMLParserTreeBuilder',
- ]
-
-from html.parser import HTMLParser
-
-try:
- from html.parser import HTMLParseError
-except ImportError as e:
- # HTMLParseError is removed in Python 3.5. Since it can never be
- # thrown in 3.5, we can just define our own class as a placeholder.
- class HTMLParseError(Exception):
- pass
-
-import sys
-import warnings
-
-# Starting in Python 3.2, the HTMLParser constructor takes a 'strict'
-# argument, which we'd like to set to False. Unfortunately,
-# http://bugs.python.org/issue13273 makes strict=True a better bet
-# before Python 3.2.3.
-#
-# At the end of this file, we monkeypatch HTMLParser so that
-# strict=True works well on Python 3.2.2.
-major, minor, release = sys.version_info[:3]
-CONSTRUCTOR_TAKES_STRICT = major == 3 and minor == 2 and release >= 3
-CONSTRUCTOR_STRICT_IS_DEPRECATED = major == 3 and minor == 3
-CONSTRUCTOR_TAKES_CONVERT_CHARREFS = major == 3 and minor >= 4
-
-
-from bs4.element import (
- CData,
- Comment,
- Declaration,
- Doctype,
- ProcessingInstruction,
- )
-from bs4.dammit import EntitySubstitution, UnicodeDammit
-
-from bs4.builder import (
- HTML,
- HTMLTreeBuilder,
- STRICT,
- )
-
-
-HTMLPARSER = 'html.parser'
-
-class BeautifulSoupHTMLParser(HTMLParser):
- def handle_starttag(self, name, attrs):
- # XXX namespace
- attr_dict = {}
- for key, value in attrs:
- # Change None attribute values to the empty string
- # for consistency with the other tree builders.
- if value is None:
- value = ''
- attr_dict[key] = value
- attrvalue = '""'
- self.soup.handle_starttag(name, None, None, attr_dict)
-
- def handle_endtag(self, name):
- self.soup.handle_endtag(name)
-
- def handle_data(self, data):
- self.soup.handle_data(data)
-
- def handle_charref(self, name):
- # XXX workaround for a bug in HTMLParser. Remove this once
- # it's fixed in all supported versions.
- # http://bugs.python.org/issue13633
- if name.startswith('x'):
- real_name = int(name.lstrip('x'), 16)
- elif name.startswith('X'):
- real_name = int(name.lstrip('X'), 16)
- else:
- real_name = int(name)
-
- try:
- data = chr(real_name)
- except (ValueError, OverflowError) as e:
- data = "\N{REPLACEMENT CHARACTER}"
-
- self.handle_data(data)
-
- def handle_entityref(self, name):
- character = EntitySubstitution.HTML_ENTITY_TO_CHARACTER.get(name)
- if character is not None:
- data = character
- else:
- data = "&%s;" % name
- self.handle_data(data)
-
- def handle_comment(self, data):
- self.soup.endData()
- self.soup.handle_data(data)
- self.soup.endData(Comment)
-
- def handle_decl(self, data):
- self.soup.endData()
- if data.startswith("DOCTYPE "):
- data = data[len("DOCTYPE "):]
- elif data == 'DOCTYPE':
- # i.e. "<!DOCTYPE>"
- data = ''
- self.soup.handle_data(data)
- self.soup.endData(Doctype)
-
- def unknown_decl(self, data):
- if data.upper().startswith('CDATA['):
- cls = CData
- data = data[len('CDATA['):]
- else:
- cls = Declaration
- self.soup.endData()
- self.soup.handle_data(data)
- self.soup.endData(cls)
-
- def handle_pi(self, data):
- self.soup.endData()
- self.soup.handle_data(data)
- self.soup.endData(ProcessingInstruction)
-
-
-class HTMLParserTreeBuilder(HTMLTreeBuilder):
-
- is_xml = False
- picklable = True
- NAME = HTMLPARSER
- features = [NAME, HTML, STRICT]
-
- def __init__(self, *args, **kwargs):
- if CONSTRUCTOR_TAKES_STRICT and not CONSTRUCTOR_STRICT_IS_DEPRECATED:
- kwargs['strict'] = False
- if CONSTRUCTOR_TAKES_CONVERT_CHARREFS:
- kwargs['convert_charrefs'] = False
- self.parser_args = (args, kwargs)
-
- def prepare_markup(self, markup, user_specified_encoding=None,
- document_declared_encoding=None, exclude_encodings=None):
- """
- :return: A 4-tuple (markup, original encoding, encoding
- declared within markup, whether any characters had to be
- replaced with REPLACEMENT CHARACTER).
- """
- if isinstance(markup, str):
- yield (markup, None, None, False)
- return
-
- try_encodings = [user_specified_encoding, document_declared_encoding]
- dammit = UnicodeDammit(markup, try_encodings, is_html=True,
- exclude_encodings=exclude_encodings)
- yield (dammit.markup, dammit.original_encoding,
- dammit.declared_html_encoding,
- dammit.contains_replacement_characters)
-
- def feed(self, markup):
- args, kwargs = self.parser_args
- parser = BeautifulSoupHTMLParser(*args, **kwargs)
- parser.soup = self.soup
- try:
- parser.feed(markup)
- except HTMLParseError as e:
- warnings.warn(RuntimeWarning(
- "Python's built-in HTMLParser cannot parse the given document. This is not a bug in Beautiful Soup. The best solution is to install an external parser (lxml or html5lib), and use Beautiful Soup with that parser. See http://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for help."))
- raise e
-
-# Patch 3.2 versions of HTMLParser earlier than 3.2.3 to use some
-# 3.2.3 code. This ensures they don't treat markup like <p></p> as a
-# string.
-#
-# XXX This code can be removed once most Python 3 users are on 3.2.3.
-if major == 3 and minor == 2 and not CONSTRUCTOR_TAKES_STRICT:
- import re
- attrfind_tolerant = re.compile(
- r'\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*'
- r'(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?')
- HTMLParserTreeBuilder.attrfind_tolerant = attrfind_tolerant
-
- locatestarttagend = re.compile(r"""
- <[a-zA-Z][-.a-zA-Z0-9:_]* # tag name
- (?:\s+ # whitespace before attribute name
- (?:[a-zA-Z_][-.:a-zA-Z0-9_]* # attribute name
- (?:\s*=\s* # value indicator
- (?:'[^']*' # LITA-enclosed value
- |\"[^\"]*\" # LIT-enclosed value
- |[^'\">\s]+ # bare value
- )
- )?
- )
- )*
- \s* # trailing whitespace
-""", re.VERBOSE)
- BeautifulSoupHTMLParser.locatestarttagend = locatestarttagend
-
- from html.parser import tagfind, attrfind
-
- def parse_starttag(self, i):
- self.__starttag_text = None
- endpos = self.check_for_whole_start_tag(i)
- if endpos < 0:
- return endpos
- rawdata = self.rawdata
- self.__starttag_text = rawdata[i:endpos]
-
- # Now parse the data between i+1 and j into a tag and attrs
- attrs = []
- match = tagfind.match(rawdata, i+1)
- assert match, 'unexpected call to parse_starttag()'
- k = match.end()
- self.lasttag = tag = rawdata[i+1:k].lower()
- while k < endpos:
- if self.strict:
- m = attrfind.match(rawdata, k)
- else:
- m = attrfind_tolerant.match(rawdata, k)
- if not m:
- break
- attrname, rest, attrvalue = m.group(1, 2, 3)
- if not rest:
- attrvalue = None
- elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
- attrvalue[:1] == '"' == attrvalue[-1:]:
- attrvalue = attrvalue[1:-1]
- if attrvalue:
- attrvalue = self.unescape(attrvalue)
- attrs.append((attrname.lower(), attrvalue))
- k = m.end()
-
- end = rawdata[k:endpos].strip()
- if end not in (">", "/>"):
- lineno, offset = self.getpos()
- if "\n" in self.__starttag_text:
- lineno = lineno + self.__starttag_text.count("\n")
- offset = len(self.__starttag_text) \
- - self.__starttag_text.rfind("\n")
- else:
- offset = offset + len(self.__starttag_text)
- if self.strict:
- self.error("junk characters in start tag: %r"
- % (rawdata[k:endpos][:20],))
- self.handle_data(rawdata[i:endpos])
- return endpos
- if end.endswith('/>'):
- # XHTML-style empty tag: <span attr="value" />
- self.handle_startendtag(tag, attrs)
- else:
- self.handle_starttag(tag, attrs)
- if tag in self.CDATA_CONTENT_ELEMENTS:
- self.set_cdata_mode(tag)
- return endpos
-
- def set_cdata_mode(self, elem):
- self.cdata_elem = elem.lower()
- self.interesting = re.compile(r'</\s*%s\s*>' % self.cdata_elem, re.I)
-
- BeautifulSoupHTMLParser.parse_starttag = parse_starttag
- BeautifulSoupHTMLParser.set_cdata_mode = set_cdata_mode
-
- CONSTRUCTOR_TAKES_STRICT = True
diff --git a/bitbake/lib/bs4/builder/_lxml.py b/bitbake/lib/bs4/builder/_lxml.py
deleted file mode 100644
index 9c6c14ee..00000000
--- a/bitbake/lib/bs4/builder/_lxml.py
+++ /dev/null
@@ -1,248 +0,0 @@
-__all__ = [
- 'LXMLTreeBuilderForXML',
- 'LXMLTreeBuilder',
- ]
-
-from io import BytesIO
-from io import StringIO
-import collections
-from lxml import etree
-from bs4.element import (
- Comment,
- Doctype,
- NamespacedAttribute,
- ProcessingInstruction,
-)
-from bs4.builder import (
- FAST,
- HTML,
- HTMLTreeBuilder,
- PERMISSIVE,
- ParserRejectedMarkup,
- TreeBuilder,
- XML)
-from bs4.dammit import EncodingDetector
-
-LXML = 'lxml'
-
-class LXMLTreeBuilderForXML(TreeBuilder):
- DEFAULT_PARSER_CLASS = etree.XMLParser
-
- is_xml = True
-
- NAME = "lxml-xml"
- ALTERNATE_NAMES = ["xml"]
-
- # Well, it's permissive by XML parser standards.
- features = [NAME, LXML, XML, FAST, PERMISSIVE]
-
- CHUNK_SIZE = 512
-
- # This namespace mapping is specified in the XML Namespace
- # standard.
- DEFAULT_NSMAPS = {'http://www.w3.org/XML/1998/namespace' : "xml"}
-
- def default_parser(self, encoding):
- # This can either return a parser object or a class, which
- # will be instantiated with default arguments.
- if self._default_parser is not None:
- return self._default_parser
- return etree.XMLParser(
- target=self, strip_cdata=False, recover=True, encoding=encoding)
-
- def parser_for(self, encoding):
- # Use the default parser.
- parser = self.default_parser(encoding)
-
- if isinstance(parser, collections.Callable):
- # Instantiate the parser with default arguments
- parser = parser(target=self, strip_cdata=False, encoding=encoding)
- return parser
-
- def __init__(self, parser=None, empty_element_tags=None):
- # TODO: Issue a warning if parser is present but not a
- # callable, since that means there's no way to create new
- # parsers for different encodings.
- self._default_parser = parser
- if empty_element_tags is not None:
- self.empty_element_tags = set(empty_element_tags)
- self.soup = None
- self.nsmaps = [self.DEFAULT_NSMAPS]
-
- def _getNsTag(self, tag):
- # Split the namespace URL out of a fully-qualified lxml tag
- # name. Copied from lxml's src/lxml/sax.py.
- if tag[0] == '{':
- return tuple(tag[1:].split('}', 1))
- else:
- return (None, tag)
-
- def prepare_markup(self, markup, user_specified_encoding=None,
- exclude_encodings=None,
- document_declared_encoding=None):
- """
- :yield: A series of 4-tuples.
- (markup, encoding, declared encoding,
- has undergone character replacement)
-
- Each 4-tuple represents a strategy for parsing the document.
- """
- if isinstance(markup, str):
- # We were given Unicode. Maybe lxml can parse Unicode on
- # this system?
- yield markup, None, document_declared_encoding, False
-
- if isinstance(markup, str):
- # No, apparently not. Convert the Unicode to UTF-8 and
- # tell lxml to parse it as UTF-8.
- yield (markup.encode("utf8"), "utf8",
- document_declared_encoding, False)
-
- # Instead of using UnicodeDammit to convert the bytestring to
- # Unicode using different encodings, use EncodingDetector to
- # iterate over the encodings, and tell lxml to try to parse
- # the document as each one in turn.
- is_html = not self.is_xml
- try_encodings = [user_specified_encoding, document_declared_encoding]
- detector = EncodingDetector(
- markup, try_encodings, is_html, exclude_encodings)
- for encoding in detector.encodings:
- yield (detector.markup, encoding, document_declared_encoding, False)
-
- def feed(self, markup):
- if isinstance(markup, bytes):
- markup = BytesIO(markup)
- elif isinstance(markup, str):
- markup = StringIO(markup)
-
- # Call feed() at least once, even if the markup is empty,
- # or the parser won't be initialized.
- data = markup.read(self.CHUNK_SIZE)
- try:
- self.parser = self.parser_for(self.soup.original_encoding)
- self.parser.feed(data)
- while len(data) != 0:
- # Now call feed() on the rest of the data, chunk by chunk.
- data = markup.read(self.CHUNK_SIZE)
- if len(data) != 0:
- self.parser.feed(data)
- self.parser.close()
- except (UnicodeDecodeError, LookupError, etree.ParserError) as e:
- raise ParserRejectedMarkup(str(e))
-
- def close(self):
- self.nsmaps = [self.DEFAULT_NSMAPS]
-
- def start(self, name, attrs, nsmap={}):
- # Make sure attrs is a mutable dict--lxml may send an immutable dictproxy.
- attrs = dict(attrs)
- nsprefix = None
- # Invert each namespace map as it comes in.
- if len(self.nsmaps) > 1:
- # There are no new namespaces for this tag, but
- # non-default namespaces are in play, so we need a
- # separate tag stack to know when they end.
- self.nsmaps.append(None)
- elif len(nsmap) > 0:
- # A new namespace mapping has come into play.
- inverted_nsmap = dict((value, key) for key, value in list(nsmap.items()))
- self.nsmaps.append(inverted_nsmap)
- # Also treat the namespace mapping as a set of attributes on the
- # tag, so we can recreate it later.
- attrs = attrs.copy()
- for prefix, namespace in list(nsmap.items()):
- attribute = NamespacedAttribute(
- "xmlns", prefix, "http://www.w3.org/2000/xmlns/")
- attrs[attribute] = namespace
-
- # Namespaces are in play. Find any attributes that came in
- # from lxml with namespaces attached to their names, and
- # turn then into NamespacedAttribute objects.
- new_attrs = {}
- for attr, value in list(attrs.items()):
- namespace, attr = self._getNsTag(attr)
- if namespace is None:
- new_attrs[attr] = value
- else:
- nsprefix = self._prefix_for_namespace(namespace)
- attr = NamespacedAttribute(nsprefix, attr, namespace)
- new_attrs[attr] = value
- attrs = new_attrs
-
- namespace, name = self._getNsTag(name)
- nsprefix = self._prefix_for_namespace(namespace)
- self.soup.handle_starttag(name, namespace, nsprefix, attrs)
-
- def _prefix_for_namespace(self, namespace):
- """Find the currently active prefix for the given namespace."""
- if namespace is None:
- return None
- for inverted_nsmap in reversed(self.nsmaps):
- if inverted_nsmap is not None and namespace in inverted_nsmap:
- return inverted_nsmap[namespace]
- return None
-
- def end(self, name):
- self.soup.endData()
- completed_tag = self.soup.tagStack[-1]
- namespace, name = self._getNsTag(name)
- nsprefix = None
- if namespace is not None:
- for inverted_nsmap in reversed(self.nsmaps):
- if inverted_nsmap is not None and namespace in inverted_nsmap:
- nsprefix = inverted_nsmap[namespace]
- break
- self.soup.handle_endtag(name, nsprefix)
- if len(self.nsmaps) > 1:
- # This tag, or one of its parents, introduced a namespace
- # mapping, so pop it off the stack.
- self.nsmaps.pop()
-
- def pi(self, target, data):
- self.soup.endData()
- self.soup.handle_data(target + ' ' + data)
- self.soup.endData(ProcessingInstruction)
-
- def data(self, content):
- self.soup.handle_data(content)
-
- def doctype(self, name, pubid, system):
- self.soup.endData()
- doctype = Doctype.for_name_and_ids(name, pubid, system)
- self.soup.object_was_parsed(doctype)
-
- def comment(self, content):
- "Handle comments as Comment objects."
- self.soup.endData()
- self.soup.handle_data(content)
- self.soup.endData(Comment)
-
- def test_fragment_to_document(self, fragment):
- """See `TreeBuilder`."""
- return '<?xml version="1.0" encoding="utf-8"?>\n%s' % fragment
-
-
-class LXMLTreeBuilder(HTMLTreeBuilder, LXMLTreeBuilderForXML):
-
- NAME = LXML
- ALTERNATE_NAMES = ["lxml-html"]
-
- features = ALTERNATE_NAMES + [NAME, HTML, FAST, PERMISSIVE]
- is_xml = False
-
- def default_parser(self, encoding):
- return etree.HTMLParser
-
- def feed(self, markup):
- encoding = self.soup.original_encoding
- try:
- self.parser = self.parser_for(encoding)
- self.parser.feed(markup)
- self.parser.close()
- except (UnicodeDecodeError, LookupError, etree.ParserError) as e:
- raise ParserRejectedMarkup(str(e))
-
-
- def test_fragment_to_document(self, fragment):
- """See `TreeBuilder`."""
- return '<html><body>%s</body></html>' % fragment
diff --git a/bitbake/lib/bs4/dammit.py b/bitbake/lib/bs4/dammit.py
deleted file mode 100644
index 7ad9e0dd..00000000
--- a/bitbake/lib/bs4/dammit.py
+++ /dev/null
@@ -1,832 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Beautiful Soup bonus library: Unicode, Dammit
-
-This library converts a bytestream to Unicode through any means
-necessary. It is heavily based on code from Mark Pilgrim's Universal
-Feed Parser. It works best on XML and HTML, but it does not rewrite the
-XML or HTML to reflect a new encoding; that's the tree builder's job.
-"""
-__license__ = "MIT"
-
-import codecs
-from html.entities import codepoint2name
-import re
-import logging
-
-# Import a library to autodetect character encodings.
-chardet_type = None
-try:
- # First try the fast C implementation.
- # PyPI package: cchardet
- import cchardet
- def chardet_dammit(s):
- return cchardet.detect(s)['encoding']
-except ImportError:
- try:
- # Fall back to the pure Python implementation
- # Debian package: python-chardet
- # PyPI package: chardet
- import chardet
- def chardet_dammit(s):
- return chardet.detect(s)['encoding']
- #import chardet.constants
- #chardet.constants._debug = 1
- except ImportError:
- # No chardet available.
- def chardet_dammit(s):
- return None
-
-xml_encoding_re = re.compile(
- r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
-html_meta_re = re.compile(
- r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(), re.I)
-
-class EntitySubstitution(object):
-
- """Substitute XML or HTML entities for the corresponding characters."""
-
- def _populate_class_variables():
- lookup = {}
- reverse_lookup = {}
- characters_for_re = []
- for codepoint, name in list(codepoint2name.items()):
- character = chr(codepoint)
- if codepoint != 34:
- # There's no point in turning the quotation mark into
- # ", unless it happens within an attribute value, which
- # is handled elsewhere.
- characters_for_re.append(character)
- lookup[character] = name
- # But we do want to turn " into the quotation mark.
- reverse_lookup[name] = character
- re_definition = "[%s]" % "".join(characters_for_re)
- return lookup, reverse_lookup, re.compile(re_definition)
- (CHARACTER_TO_HTML_ENTITY, HTML_ENTITY_TO_CHARACTER,
- CHARACTER_TO_HTML_ENTITY_RE) = _populate_class_variables()
-
- CHARACTER_TO_XML_ENTITY = {
- "'": "apos",
- '"': "quot",
- "&": "amp",
- "<": "lt",
- ">": "gt",
- }
-
- BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
- r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
- r")")
-
- AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
-
- @classmethod
- def _substitute_html_entity(cls, matchobj):
- entity = cls.CHARACTER_TO_HTML_ENTITY.get(matchobj.group(0))
- return "&%s;" % entity
-
- @classmethod
- def _substitute_xml_entity(cls, matchobj):
- """Used with a regular expression to substitute the
- appropriate XML entity for an XML special character."""
- entity = cls.CHARACTER_TO_XML_ENTITY[matchobj.group(0)]
- return "&%s;" % entity
-
- @classmethod
- def quoted_attribute_value(self, value):
- """Make a value into a quoted XML attribute, possibly escaping it.
-
- Most strings will be quoted using double quotes.
-
- Bob's Bar -> "Bob's Bar"
-
- If a string contains double quotes, it will be quoted using
- single quotes.
-
- Welcome to "my bar" -> 'Welcome to "my bar"'
-
- If a string contains both single and double quotes, the
- double quotes will be escaped, and the string will be quoted
- using double quotes.
-
- Welcome to "Bob's Bar" -> "Welcome to "Bob's bar"
- """
- quote_with = '"'
- if '"' in value:
- if "'" in value:
- # The string contains both single and double
- # quotes. Turn the double quotes into
- # entities. We quote the double quotes rather than
- # the single quotes because the entity name is
- # """ whether this is HTML or XML. If we
- # quoted the single quotes, we'd have to decide
- # between ' and &squot;.
- replace_with = """
- value = value.replace('"', replace_with)
- else:
- # There are double quotes but no single quotes.
- # We can use single quotes to quote the attribute.
- quote_with = "'"
- return quote_with + value + quote_with
-
- @classmethod
- def substitute_xml(cls, value, make_quoted_attribute=False):
- """Substitute XML entities for special XML characters.
-
- :param value: A string to be substituted. The less-than sign
- will become <, the greater-than sign will become >,
- and any ampersands will become &. If you want ampersands
- that appear to be part of an entity definition to be left
- alone, use substitute_xml_containing_entities() instead.
-
- :param make_quoted_attribute: If True, then the string will be
- quoted, as befits an attribute value.
- """
- # Escape angle brackets and ampersands.
- value = cls.AMPERSAND_OR_BRACKET.sub(
- cls._substitute_xml_entity, value)
-
- if make_quoted_attribute:
- value = cls.quoted_attribute_value(value)
- return value
-
- @classmethod
- def substitute_xml_containing_entities(
- cls, value, make_quoted_attribute=False):
- """Substitute XML entities for special XML characters.
-
- :param value: A string to be substituted. The less-than sign will
- become <, the greater-than sign will become >, and any
- ampersands that are not part of an entity defition will
- become &.
-
- :param make_quoted_attribute: If True, then the string will be
- quoted, as befits an attribute value.
- """
- # Escape angle brackets, and ampersands that aren't part of
- # entities.
- value = cls.BARE_AMPERSAND_OR_BRACKET.sub(
- cls._substitute_xml_entity, value)
-
- if make_quoted_attribute:
- value = cls.quoted_attribute_value(value)
- return value
-
- @classmethod
- def substitute_html(cls, s):
- """Replace certain Unicode characters with named HTML entities.
-
- This differs from data.encode(encoding, 'xmlcharrefreplace')
- in that the goal is to make the result more readable (to those
- with ASCII displays) rather than to recover from
- errors. There's absolutely nothing wrong with a UTF-8 string
- containg a LATIN SMALL LETTER E WITH ACUTE, but replacing that
- character with "é" will make it more readable to some
- people.
- """
- return cls.CHARACTER_TO_HTML_ENTITY_RE.sub(
- cls._substitute_html_entity, s)
-
-
-class EncodingDetector:
- """Suggests a number of possible encodings for a bytestring.
-
- Order of precedence:
-
- 1. Encodings you specifically tell EncodingDetector to try first
- (the override_encodings argument to the constructor).
-
- 2. An encoding declared within the bytestring itself, either in an
- XML declaration (if the bytestring is to be interpreted as an XML
- document), or in a <meta> tag (if the bytestring is to be
- interpreted as an HTML document.)
-
- 3. An encoding detected through textual analysis by chardet,
- cchardet, or a similar external library.
-
- 4. UTF-8.
-
- 5. Windows-1252.
- """
- def __init__(self, markup, override_encodings=None, is_html=False,
- exclude_encodings=None):
- self.override_encodings = override_encodings or []
- exclude_encodings = exclude_encodings or []
- self.exclude_encodings = set([x.lower() for x in exclude_encodings])
- self.chardet_encoding = None
- self.is_html = is_html
- self.declared_encoding = None
-
- # First order of business: strip a byte-order mark.
- self.markup, self.sniffed_encoding = self.strip_byte_order_mark(markup)
-
- def _usable(self, encoding, tried):
- if encoding is not None:
- encoding = encoding.lower()
- if encoding in self.exclude_encodings:
- return False
- if encoding not in tried:
- tried.add(encoding)
- return True
- return False
-
- @property
- def encodings(self):
- """Yield a number of encodings that might work for this markup."""
- tried = set()
- for e in self.override_encodings:
- if self._usable(e, tried):
- yield e
-
- # Did the document originally start with a byte-order mark
- # that indicated its encoding?
- if self._usable(self.sniffed_encoding, tried):
- yield self.sniffed_encoding
-
- # Look within the document for an XML or HTML encoding
- # declaration.
- if self.declared_encoding is None:
- self.declared_encoding = self.find_declared_encoding(
- self.markup, self.is_html)
- if self._usable(self.declared_encoding, tried):
- yield self.declared_encoding
-
- # Use third-party character set detection to guess at the
- # encoding.
- if self.chardet_encoding is None:
- self.chardet_encoding = chardet_dammit(self.markup)
- if self._usable(self.chardet_encoding, tried):
- yield self.chardet_encoding
-
- # As a last-ditch effort, try utf-8 and windows-1252.
- for e in ('utf-8', 'windows-1252'):
- if self._usable(e, tried):
- yield e
-
- @classmethod
- def strip_byte_order_mark(cls, data):
- """If a byte-order mark is present, strip it and return the encoding it implies."""
- encoding = None
- if isinstance(data, str):
- # Unicode data cannot have a byte-order mark.
- return data, encoding
- if (len(data) >= 4) and (data[:2] == b'\xfe\xff') \
- and (data[2:4] != '\x00\x00'):
- encoding = 'utf-16be'
- data = data[2:]
- elif (len(data) >= 4) and (data[:2] == b'\xff\xfe') \
- and (data[2:4] != '\x00\x00'):
- encoding = 'utf-16le'
- data = data[2:]
- elif data[:3] == b'\xef\xbb\xbf':
- encoding = 'utf-8'
- data = data[3:]
- elif data[:4] == b'\x00\x00\xfe\xff':
- encoding = 'utf-32be'
- data = data[4:]
- elif data[:4] == b'\xff\xfe\x00\x00':
- encoding = 'utf-32le'
- data = data[4:]
- return data, encoding
-
- @classmethod
- def find_declared_encoding(cls, markup, is_html=False, search_entire_document=False):
- """Given a document, tries to find its declared encoding.
-
- An XML encoding is declared at the beginning of the document.
-
- An HTML encoding is declared in a <meta> tag, hopefully near the
- beginning of the document.
- """
- if search_entire_document:
- xml_endpos = html_endpos = len(markup)
- else:
- xml_endpos = 1024
- html_endpos = max(2048, int(len(markup) * 0.05))
-
- declared_encoding = None
- declared_encoding_match = xml_encoding_re.search(markup, endpos=xml_endpos)
- if not declared_encoding_match and is_html:
- declared_encoding_match = html_meta_re.search(markup, endpos=html_endpos)
- if declared_encoding_match is not None:
- declared_encoding = declared_encoding_match.groups()[0].decode(
- 'ascii', 'replace')
- if declared_encoding:
- return declared_encoding.lower()
- return None
-
-class UnicodeDammit:
- """A class for detecting the encoding of a *ML document and
- converting it to a Unicode string. If the source encoding is
- windows-1252, can replace MS smart quotes with their HTML or XML
- equivalents."""
-
- # This dictionary maps commonly seen values for "charset" in HTML
- # meta tags to the corresponding Python codec names. It only covers
- # values that aren't in Python's aliases and can't be determined
- # by the heuristics in find_codec.
- CHARSET_ALIASES = {"macintosh": "mac-roman",
- "x-sjis": "shift-jis"}
-
- ENCODINGS_WITH_SMART_QUOTES = [
- "windows-1252",
- "iso-8859-1",
- "iso-8859-2",
- ]
-
- def __init__(self, markup, override_encodings=[],
- smart_quotes_to=None, is_html=False, exclude_encodings=[]):
- self.smart_quotes_to = smart_quotes_to
- self.tried_encodings = []
- self.contains_replacement_characters = False
- self.is_html = is_html
-
- self.detector = EncodingDetector(
- markup, override_encodings, is_html, exclude_encodings)
-
- # Short-circuit if the data is in Unicode to begin with.
- if isinstance(markup, str) or markup == '':
- self.markup = markup
- self.unicode_markup = str(markup)
- self.original_encoding = None
- return
-
- # The encoding detector may have stripped a byte-order mark.
- # Use the stripped markup from this point on.
- self.markup = self.detector.markup
-
- u = None
- for encoding in self.detector.encodings:
- markup = self.detector.markup
- u = self._convert_from(encoding)
- if u is not None:
- break
-
- if not u:
- # None of the encodings worked. As an absolute last resort,
- # try them again with character replacement.
-
- for encoding in self.detector.encodings:
- if encoding != "ascii":
- u = self._convert_from(encoding, "replace")
- if u is not None:
- logging.warning(
- "Some characters could not be decoded, and were "
- "replaced with REPLACEMENT CHARACTER.")
- self.contains_replacement_characters = True
- break
-
- # If none of that worked, we could at this point force it to
- # ASCII, but that would destroy so much data that I think
- # giving up is better.
- self.unicode_markup = u
- if not u:
- self.original_encoding = None
-
- def _sub_ms_char(self, match):
- """Changes a MS smart quote character to an XML or HTML
- entity, or an ASCII character."""
- orig = match.group(1)
- if self.smart_quotes_to == 'ascii':
- sub = self.MS_CHARS_TO_ASCII.get(orig).encode()
- else:
- sub = self.MS_CHARS.get(orig)
- if type(sub) == tuple:
- if self.smart_quotes_to == 'xml':
- sub = '&#x'.encode() + sub[1].encode() + ';'.encode()
- else:
- sub = '&'.encode() + sub[0].encode() + ';'.encode()
- else:
- sub = sub.encode()
- return sub
-
- def _convert_from(self, proposed, errors="strict"):
- proposed = self.find_codec(proposed)
- if not proposed or (proposed, errors) in self.tried_encodings:
- return None
- self.tried_encodings.append((proposed, errors))
- markup = self.markup
- # Convert smart quotes to HTML if coming from an encoding
- # that might have them.
- if (self.smart_quotes_to is not None
- and proposed in self.ENCODINGS_WITH_SMART_QUOTES):
- smart_quotes_re = b"([\x80-\x9f])"
- smart_quotes_compiled = re.compile(smart_quotes_re)
- markup = smart_quotes_compiled.sub(self._sub_ms_char, markup)
-
- try:
- #print "Trying to convert document to %s (errors=%s)" % (
- # proposed, errors)
- u = self._to_unicode(markup, proposed, errors)
- self.markup = u
- self.original_encoding = proposed
- except Exception as e:
- #print "That didn't work!"
- #print e
- return None
- #print "Correct encoding: %s" % proposed
- return self.markup
-
- def _to_unicode(self, data, encoding, errors="strict"):
- '''Given a string and its encoding, decodes the string into Unicode.
- %encoding is a string recognized by encodings.aliases'''
- return str(data, encoding, errors)
-
- @property
- def declared_html_encoding(self):
- if not self.is_html:
- return None
- return self.detector.declared_encoding
-
- def find_codec(self, charset):
- value = (self._codec(self.CHARSET_ALIASES.get(charset, charset))
- or (charset and self._codec(charset.replace("-", "")))
- or (charset and self._codec(charset.replace("-", "_")))
- or (charset and charset.lower())
- or charset
- )
- if value:
- return value.lower()
- return None
-
- def _codec(self, charset):
- if not charset:
- return charset
- codec = None
- try:
- codecs.lookup(charset)
- codec = charset
- except (LookupError, ValueError):
- pass
- return codec
-
-
- # A partial mapping of ISO-Latin-1 to HTML entities/XML numeric entities.
- MS_CHARS = {b'\x80': ('euro', '20AC'),
- b'\x81': ' ',
- b'\x82': ('sbquo', '201A'),
- b'\x83': ('fnof', '192'),
- b'\x84': ('bdquo', '201E'),
- b'\x85': ('hellip', '2026'),
- b'\x86': ('dagger', '2020'),
- b'\x87': ('Dagger', '2021'),
- b'\x88': ('circ', '2C6'),
- b'\x89': ('permil', '2030'),
- b'\x8A': ('Scaron', '160'),
- b'\x8B': ('lsaquo', '2039'),
- b'\x8C': ('OElig', '152'),
- b'\x8D': '?',
- b'\x8E': ('#x17D', '17D'),
- b'\x8F': '?',
- b'\x90': '?',
- b'\x91': ('lsquo', '2018'),
- b'\x92': ('rsquo', '2019'),
- b'\x93': ('ldquo', '201C'),
- b'\x94': ('rdquo', '201D'),
- b'\x95': ('bull', '2022'),
- b'\x96': ('ndash', '2013'),
- b'\x97': ('mdash', '2014'),
- b'\x98': ('tilde', '2DC'),
- b'\x99': ('trade', '2122'),
- b'\x9a': ('scaron', '161'),
- b'\x9b': ('rsaquo', '203A'),
- b'\x9c': ('oelig', '153'),
- b'\x9d': '?',
- b'\x9e': ('#x17E', '17E'),
- b'\x9f': ('Yuml', ''),}
-
- # A parochial partial mapping of ISO-Latin-1 to ASCII. Contains
- # horrors like stripping diacritical marks to turn á into a, but also
- # contains non-horrors like turning “ into ".
- MS_CHARS_TO_ASCII = {
- b'\x80' : 'EUR',
- b'\x81' : ' ',
- b'\x82' : ',',
- b'\x83' : 'f',
- b'\x84' : ',,',
- b'\x85' : '...',
- b'\x86' : '+',
- b'\x87' : '++',
- b'\x88' : '^',
- b'\x89' : '%',
- b'\x8a' : 'S',
- b'\x8b' : '<',
- b'\x8c' : 'OE',
- b'\x8d' : '?',
- b'\x8e' : 'Z',
- b'\x8f' : '?',
- b'\x90' : '?',
- b'\x91' : "'",
- b'\x92' : "'",
- b'\x93' : '"',
- b'\x94' : '"',
- b'\x95' : '*',
- b'\x96' : '-',
- b'\x97' : '--',
- b'\x98' : '~',
- b'\x99' : '(TM)',
- b'\x9a' : 's',
- b'\x9b' : '>',
- b'\x9c' : 'oe',
- b'\x9d' : '?',
- b'\x9e' : 'z',
- b'\x9f' : 'Y',
- b'\xa0' : ' ',
- b'\xa1' : '!',
- b'\xa2' : 'c',
- b'\xa3' : 'GBP',
- b'\xa4' : '$', #This approximation is especially parochial--this is the
- #generic currency symbol.
- b'\xa5' : 'YEN',
- b'\xa6' : '|',
- b'\xa7' : 'S',
- b'\xa8' : '..',
- b'\xa9' : '',
- b'\xaa' : '(th)',
- b'\xab' : '<<',
- b'\xac' : '!',
- b'\xad' : ' ',
- b'\xae' : '(R)',
- b'\xaf' : '-',
- b'\xb0' : 'o',
- b'\xb1' : '+-',
- b'\xb2' : '2',
- b'\xb3' : '3',
- b'\xb4' : ("'", 'acute'),
- b'\xb5' : 'u',
- b'\xb6' : 'P',
- b'\xb7' : '*',
- b'\xb8' : ',',
- b'\xb9' : '1',
- b'\xba' : '(th)',
- b'\xbb' : '>>',
- b'\xbc' : '1/4',
- b'\xbd' : '1/2',
- b'\xbe' : '3/4',
- b'\xbf' : '?',
- b'\xc0' : 'A',
- b'\xc1' : 'A',
- b'\xc2' : 'A',
- b'\xc3' : 'A',
- b'\xc4' : 'A',
- b'\xc5' : 'A',
- b'\xc6' : 'AE',
- b'\xc7' : 'C',
- b'\xc8' : 'E',
- b'\xc9' : 'E',
- b'\xca' : 'E',
- b'\xcb' : 'E',
- b'\xcc' : 'I',
- b'\xcd' : 'I',
- b'\xce' : 'I',
- b'\xcf' : 'I',
- b'\xd0' : 'D',
- b'\xd1' : 'N',
- b'\xd2' : 'O',
- b'\xd3' : 'O',
- b'\xd4' : 'O',
- b'\xd5' : 'O',
- b'\xd6' : 'O',
- b'\xd7' : '*',
- b'\xd8' : 'O',
- b'\xd9' : 'U',
- b'\xda' : 'U',
- b'\xdb' : 'U',
- b'\xdc' : 'U',
- b'\xdd' : 'Y',
- b'\xde' : 'b',
- b'\xdf' : 'B',
- b'\xe0' : 'a',
- b'\xe1' : 'a',
- b'\xe2' : 'a',
- b'\xe3' : 'a',
- b'\xe4' : 'a',
- b'\xe5' : 'a',
- b'\xe6' : 'ae',
- b'\xe7' : 'c',
- b'\xe8' : 'e',
- b'\xe9' : 'e',
- b'\xea' : 'e',
- b'\xeb' : 'e',
- b'\xec' : 'i',
- b'\xed' : 'i',
- b'\xee' : 'i',
- b'\xef' : 'i',
- b'\xf0' : 'o',
- b'\xf1' : 'n',
- b'\xf2' : 'o',
- b'\xf3' : 'o',
- b'\xf4' : 'o',
- b'\xf5' : 'o',
- b'\xf6' : 'o',
- b'\xf7' : '/',
- b'\xf8' : 'o',
- b'\xf9' : 'u',
- b'\xfa' : 'u',
- b'\xfb' : 'u',
- b'\xfc' : 'u',
- b'\xfd' : 'y',
- b'\xfe' : 'b',
- b'\xff' : 'y',
- }
-
- # A map used when removing rogue Windows-1252/ISO-8859-1
- # characters in otherwise UTF-8 documents.
- #
- # Note that \x81, \x8d, \x8f, \x90, and \x9d are undefined in
- # Windows-1252.
- WINDOWS_1252_TO_UTF8 = {
- 0x80 : b'\xe2\x82\xac', # €
- 0x82 : b'\xe2\x80\x9a', # ‚
- 0x83 : b'\xc6\x92', # ƒ
- 0x84 : b'\xe2\x80\x9e', # „
- 0x85 : b'\xe2\x80\xa6', # …
- 0x86 : b'\xe2\x80\xa0', # †
- 0x87 : b'\xe2\x80\xa1', # ‡
- 0x88 : b'\xcb\x86', # ˆ
- 0x89 : b'\xe2\x80\xb0', # ‰
- 0x8a : b'\xc5\xa0', # Š
- 0x8b : b'\xe2\x80\xb9', # ‹
- 0x8c : b'\xc5\x92', # Œ
- 0x8e : b'\xc5\xbd', # Ž
- 0x91 : b'\xe2\x80\x98', # ‘
- 0x92 : b'\xe2\x80\x99', # ’
- 0x93 : b'\xe2\x80\x9c', # “
- 0x94 : b'\xe2\x80\x9d', # ”
- 0x95 : b'\xe2\x80\xa2', # •
- 0x96 : b'\xe2\x80\x93', # –
- 0x97 : b'\xe2\x80\x94', # —
- 0x98 : b'\xcb\x9c', # ˜
- 0x99 : b'\xe2\x84\xa2', # ™
- 0x9a : b'\xc5\xa1', # š
- 0x9b : b'\xe2\x80\xba', # ›
- 0x9c : b'\xc5\x93', # œ
- 0x9e : b'\xc5\xbe', # ž
- 0x9f : b'\xc5\xb8', # Ÿ
- 0xa0 : b'\xc2\xa0', #
- 0xa1 : b'\xc2\xa1', # ¡
- 0xa2 : b'\xc2\xa2', # ¢
- 0xa3 : b'\xc2\xa3', # £
- 0xa4 : b'\xc2\xa4', # ¤
- 0xa5 : b'\xc2\xa5', # ¥
- 0xa6 : b'\xc2\xa6', # ¦
- 0xa7 : b'\xc2\xa7', # §
- 0xa8 : b'\xc2\xa8', # ¨
- 0xa9 : b'\xc2\xa9', # ©
- 0xaa : b'\xc2\xaa', # ª
- 0xab : b'\xc2\xab', # «
- 0xac : b'\xc2\xac', # ¬
- 0xad : b'\xc2\xad', #
- 0xae : b'\xc2\xae', # ®
- 0xaf : b'\xc2\xaf', # ¯
- 0xb0 : b'\xc2\xb0', # °
- 0xb1 : b'\xc2\xb1', # ±
- 0xb2 : b'\xc2\xb2', # ²
- 0xb3 : b'\xc2\xb3', # ³
- 0xb4 : b'\xc2\xb4', # ´
- 0xb5 : b'\xc2\xb5', # µ
- 0xb6 : b'\xc2\xb6', # ¶
- 0xb7 : b'\xc2\xb7', # ·
- 0xb8 : b'\xc2\xb8', # ¸
- 0xb9 : b'\xc2\xb9', # ¹
- 0xba : b'\xc2\xba', # º
- 0xbb : b'\xc2\xbb', # »
- 0xbc : b'\xc2\xbc', # ¼
- 0xbd : b'\xc2\xbd', # ½
- 0xbe : b'\xc2\xbe', # ¾
- 0xbf : b'\xc2\xbf', # ¿
- 0xc0 : b'\xc3\x80', # À
- 0xc1 : b'\xc3\x81', # Á
- 0xc2 : b'\xc3\x82', # Â
- 0xc3 : b'\xc3\x83', # Ã
- 0xc4 : b'\xc3\x84', # Ä
- 0xc5 : b'\xc3\x85', # Å
- 0xc6 : b'\xc3\x86', # Æ
- 0xc7 : b'\xc3\x87', # Ç
- 0xc8 : b'\xc3\x88', # È
- 0xc9 : b'\xc3\x89', # É
- 0xca : b'\xc3\x8a', # Ê
- 0xcb : b'\xc3\x8b', # Ë
- 0xcc : b'\xc3\x8c', # Ì
- 0xcd : b'\xc3\x8d', # Í
- 0xce : b'\xc3\x8e', # Î
- 0xcf : b'\xc3\x8f', # Ï
- 0xd0 : b'\xc3\x90', # Ð
- 0xd1 : b'\xc3\x91', # Ñ
- 0xd2 : b'\xc3\x92', # Ò
- 0xd3 : b'\xc3\x93', # Ó
- 0xd4 : b'\xc3\x94', # Ô
- 0xd5 : b'\xc3\x95', # Õ
- 0xd6 : b'\xc3\x96', # Ö
- 0xd7 : b'\xc3\x97', # ×
- 0xd8 : b'\xc3\x98', # Ø
- 0xd9 : b'\xc3\x99', # Ù
- 0xda : b'\xc3\x9a', # Ú
- 0xdb : b'\xc3\x9b', # Û
- 0xdc : b'\xc3\x9c', # Ü
- 0xdd : b'\xc3\x9d', # Ý
- 0xde : b'\xc3\x9e', # Þ
- 0xdf : b'\xc3\x9f', # ß
- 0xe0 : b'\xc3\xa0', # à
- 0xe1 : b'\xa1', # á
- 0xe2 : b'\xc3\xa2', # â
- 0xe3 : b'\xc3\xa3', # ã
- 0xe4 : b'\xc3\xa4', # ä
- 0xe5 : b'\xc3\xa5', # å
- 0xe6 : b'\xc3\xa6', # æ
- 0xe7 : b'\xc3\xa7', # ç
- 0xe8 : b'\xc3\xa8', # è
- 0xe9 : b'\xc3\xa9', # é
- 0xea : b'\xc3\xaa', # ê
- 0xeb : b'\xc3\xab', # ë
- 0xec : b'\xc3\xac', # ì
- 0xed : b'\xc3\xad', # í
- 0xee : b'\xc3\xae', # î
- 0xef : b'\xc3\xaf', # ï
- 0xf0 : b'\xc3\xb0', # ð
- 0xf1 : b'\xc3\xb1', # ñ
- 0xf2 : b'\xc3\xb2', # ò
- 0xf3 : b'\xc3\xb3', # ó
- 0xf4 : b'\xc3\xb4', # ô
- 0xf5 : b'\xc3\xb5', # õ
- 0xf6 : b'\xc3\xb6', # ö
- 0xf7 : b'\xc3\xb7', # ÷
- 0xf8 : b'\xc3\xb8', # ø
- 0xf9 : b'\xc3\xb9', # ù
- 0xfa : b'\xc3\xba', # ú
- 0xfb : b'\xc3\xbb', # û
- 0xfc : b'\xc3\xbc', # ü
- 0xfd : b'\xc3\xbd', # ý
- 0xfe : b'\xc3\xbe', # þ
- }
-
- MULTIBYTE_MARKERS_AND_SIZES = [
- (0xc2, 0xdf, 2), # 2-byte characters start with a byte C2-DF
- (0xe0, 0xef, 3), # 3-byte characters start with E0-EF
- (0xf0, 0xf4, 4), # 4-byte characters start with F0-F4
- ]
-
- FIRST_MULTIBYTE_MARKER = MULTIBYTE_MARKERS_AND_SIZES[0][0]
- LAST_MULTIBYTE_MARKER = MULTIBYTE_MARKERS_AND_SIZES[-1][1]
-
- @classmethod
- def detwingle(cls, in_bytes, main_encoding="utf8",
- embedded_encoding="windows-1252"):
- """Fix characters from one encoding embedded in some other encoding.
-
- Currently the only situation supported is Windows-1252 (or its
- subset ISO-8859-1), embedded in UTF-8.
-
- The input must be a bytestring. If you've already converted
- the document to Unicode, you're too late.
-
- The output is a bytestring in which `embedded_encoding`
- characters have been converted to their `main_encoding`
- equivalents.
- """
- if embedded_encoding.replace('_', '-').lower() not in (
- 'windows-1252', 'windows_1252'):
- raise NotImplementedError(
- "Windows-1252 and ISO-8859-1 are the only currently supported "
- "embedded encodings.")
-
- if main_encoding.lower() not in ('utf8', 'utf-8'):
- raise NotImplementedError(
- "UTF-8 is the only currently supported main encoding.")
-
- byte_chunks = []
-
- chunk_start = 0
- pos = 0
- while pos < len(in_bytes):
- byte = in_bytes[pos]
- if not isinstance(byte, int):
- # Python 2.x
- byte = ord(byte)
- if (byte >= cls.FIRST_MULTIBYTE_MARKER
- and byte <= cls.LAST_MULTIBYTE_MARKER):
- # This is the start of a UTF-8 multibyte character. Skip
- # to the end.
- for start, end, size in cls.MULTIBYTE_MARKERS_AND_SIZES:
- if byte >= start and byte <= end:
- pos += size
- break
- elif byte >= 0x80 and byte in cls.WINDOWS_1252_TO_UTF8:
- # We found a Windows-1252 character!
- # Save the string up to this point as a chunk.
- byte_chunks.append(in_bytes[chunk_start:pos])
-
- # Now translate the Windows-1252 character into UTF-8
- # and add it as another, one-byte chunk.
- byte_chunks.append(cls.WINDOWS_1252_TO_UTF8[byte])
- pos += 1
- chunk_start = pos
- else:
- # Go on to the next character.
- pos += 1
- if chunk_start == 0:
- # The string is unchanged.
- return in_bytes
- else:
- # Store the final chunk.
- byte_chunks.append(in_bytes[chunk_start:])
- return b''.join(byte_chunks)
-
diff --git a/bitbake/lib/bs4/diagnose.py b/bitbake/lib/bs4/diagnose.py
deleted file mode 100644
index 083395fb..00000000
--- a/bitbake/lib/bs4/diagnose.py
+++ /dev/null
@@ -1,216 +0,0 @@
-"""Diagnostic functions, mainly for use when doing tech support."""
-
-__license__ = "MIT"
-
-import cProfile
-from io import StringIO
-from html.parser import HTMLParser
-import bs4
-from bs4 import BeautifulSoup, __version__
-from bs4.builder import builder_registry
-
-import os
-import pstats
-import random
-import tempfile
-import time
-import traceback
-import sys
-import cProfile
-
-def diagnose(data):
- """Diagnostic suite for isolating common problems."""
- print("Diagnostic running on Beautiful Soup %s" % __version__)
- print("Python version %s" % sys.version)
-
- basic_parsers = ["html.parser", "html5lib", "lxml"]
- for name in basic_parsers:
- for builder in builder_registry.builders:
- if name in builder.features:
- break
- else:
- basic_parsers.remove(name)
- print((
- "I noticed that %s is not installed. Installing it may help." %
- name))
-
- if 'lxml' in basic_parsers:
- basic_parsers.append(["lxml", "xml"])
- try:
- from lxml import etree
- print("Found lxml version %s" % ".".join(map(str,etree.LXML_VERSION)))
- except ImportError as e:
- print (
- "lxml is not installed or couldn't be imported.")
-
-
- if 'html5lib' in basic_parsers:
- try:
- import html5lib
- print("Found html5lib version %s" % html5lib.__version__)
- except ImportError as e:
- print (
- "html5lib is not installed or couldn't be imported.")
-
- if hasattr(data, 'read'):
- data = data.read()
- elif os.path.exists(data):
- print('"%s" looks like a filename. Reading data from the file.' % data)
- data = open(data).read()
- elif data.startswith("http:") or data.startswith("https:"):
- print('"%s" looks like a URL. Beautiful Soup is not an HTTP client.' % data)
- print("You need to use some other library to get the document behind the URL, and feed that document to Beautiful Soup.")
- return
- print()
-
- for parser in basic_parsers:
- print("Trying to parse your markup with %s" % parser)
- success = False
- try:
- soup = BeautifulSoup(data, parser)
- success = True
- except Exception as e:
- print("%s could not parse the markup." % parser)
- traceback.print_exc()
- if success:
- print("Here's what %s did with the markup:" % parser)
- print(soup.prettify())
-
- print("-" * 80)
-
-def lxml_trace(data, html=True, **kwargs):
- """Print out the lxml events that occur during parsing.
-
- This lets you see how lxml parses a document when no Beautiful
- Soup code is running.
- """
- from lxml import etree
- for event, element in etree.iterparse(StringIO(data), html=html, **kwargs):
- print(("%s, %4s, %s" % (event, element.tag, element.text)))
-
-class AnnouncingParser(HTMLParser):
- """Announces HTMLParser parse events, without doing anything else."""
-
- def _p(self, s):
- print(s)
-
- def handle_starttag(self, name, attrs):
- self._p("%s START" % name)
-
- def handle_endtag(self, name):
- self._p("%s END" % name)
-
- def handle_data(self, data):
- self._p("%s DATA" % data)
-
- def handle_charref(self, name):
- self._p("%s CHARREF" % name)
-
- def handle_entityref(self, name):
- self._p("%s ENTITYREF" % name)
-
- def handle_comment(self, data):
- self._p("%s COMMENT" % data)
-
- def handle_decl(self, data):
- self._p("%s DECL" % data)
-
- def unknown_decl(self, data):
- self._p("%s UNKNOWN-DECL" % data)
-
- def handle_pi(self, data):
- self._p("%s PI" % data)
-
-def htmlparser_trace(data):
- """Print out the HTMLParser events that occur during parsing.
-
- This lets you see how HTMLParser parses a document when no
- Beautiful Soup code is running.
- """
- parser = AnnouncingParser()
- parser.feed(data)
-
-_vowels = "aeiou"
-_consonants = "bcdfghjklmnpqrstvwxyz"
-
-def rword(length=5):
- "Generate a random word-like string."
- s = ''
- for i in range(length):
- if i % 2 == 0:
- t = _consonants
- else:
- t = _vowels
- s += random.choice(t)
- return s
-
-def rsentence(length=4):
- "Generate a random sentence-like string."
- return " ".join(rword(random.randint(4,9)) for i in range(length))
-
-def rdoc(num_elements=1000):
- """Randomly generate an invalid HTML document."""
- tag_names = ['p', 'div', 'span', 'i', 'b', 'script', 'table']
- elements = []
- for i in range(num_elements):
- choice = random.randint(0,3)
- if choice == 0:
- # New tag.
- tag_name = random.choice(tag_names)
- elements.append("<%s>" % tag_name)
- elif choice == 1:
- elements.append(rsentence(random.randint(1,4)))
- elif choice == 2:
- # Close a tag.
- tag_name = random.choice(tag_names)
- elements.append("</%s>" % tag_name)
- return "<html>" + "\n".join(elements) + "</html>"
-
-def benchmark_parsers(num_elements=100000):
- """Very basic head-to-head performance benchmark."""
- print("Comparative parser benchmark on Beautiful Soup %s" % __version__)
- data = rdoc(num_elements)
- print("Generated a large invalid HTML document (%d bytes)." % len(data))
-
- for parser in ["lxml", ["lxml", "html"], "html5lib", "html.parser"]:
- success = False
- try:
- a = time.time()
- soup = BeautifulSoup(data, parser)
- b = time.time()
- success = True
- except Exception as e:
- print("%s could not parse the markup." % parser)
- traceback.print_exc()
- if success:
- print("BS4+%s parsed the markup in %.2fs." % (parser, b-a))
-
- from lxml import etree
- a = time.time()
- etree.HTML(data)
- b = time.time()
- print("Raw lxml parsed the markup in %.2fs." % (b-a))
-
- import html5lib
- parser = html5lib.HTMLParser()
- a = time.time()
- parser.parse(data)
- b = time.time()
- print("Raw html5lib parsed the markup in %.2fs." % (b-a))
-
-def profile(num_elements=100000, parser="lxml"):
-
- filehandle = tempfile.NamedTemporaryFile()
- filename = filehandle.name
-
- data = rdoc(num_elements)
- vars = dict(bs4=bs4, data=data, parser=parser)
- cProfile.runctx('bs4.BeautifulSoup(data, parser)' , vars, vars, filename)
-
- stats = pstats.Stats(filename)
- # stats.strip_dirs()
- stats.sort_stats("cumulative")
- stats.print_stats('_html5lib|bs4', 50)
-
-if __name__ == '__main__':
- diagnose(sys.stdin.read())
diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
deleted file mode 100644
index 68be42d1..00000000
--- a/bitbake/lib/bs4/element.py
+++ /dev/null
@@ -1,1724 +0,0 @@
-__license__ = "MIT"
-
-import collections.abc
-import re
-import sys
-import warnings
-from bs4.dammit import EntitySubstitution
-
-DEFAULT_OUTPUT_ENCODING = "utf-8"
-PY3K = (sys.version_info[0] > 2)
-
-whitespace_re = re.compile(r"\s+")
-
-def _alias(attr):
- """Alias one attribute name to another for backward compatibility"""
- @property
- def alias(self):
- return getattr(self, attr)
-
- @alias.setter
- def alias(self):
- return setattr(self, attr)
- return alias
-
-
-class NamespacedAttribute(str):
-
- def __new__(cls, prefix, name, namespace=None):
- if name is None:
- obj = str.__new__(cls, prefix)
- elif prefix is None:
- # Not really namespaced.
- obj = str.__new__(cls, name)
- else:
- obj = str.__new__(cls, prefix + ":" + name)
- obj.prefix = prefix
- obj.name = name
- obj.namespace = namespace
- return obj
-
-class AttributeValueWithCharsetSubstitution(str):
- """A stand-in object for a character encoding specified in HTML."""
-
-class CharsetMetaAttributeValue(AttributeValueWithCharsetSubstitution):
- """A generic stand-in for the value of a meta tag's 'charset' attribute.
-
- When Beautiful Soup parses the markup '<meta charset="utf8">', the
- value of the 'charset' attribute will be one of these objects.
- """
-
- def __new__(cls, original_value):
- obj = str.__new__(cls, original_value)
- obj.original_value = original_value
- return obj
-
- def encode(self, encoding):
- return encoding
-
-
-class ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution):
- """A generic stand-in for the value of a meta tag's 'content' attribute.
-
- When Beautiful Soup parses the markup:
- <meta http-equiv="content-type" content="text/html; charset=utf8">
-
- The value of the 'content' attribute will be one of these objects.
- """
-
- CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
-
- def __new__(cls, original_value):
- match = cls.CHARSET_RE.search(original_value)
- if match is None:
- # No substitution necessary.
- return str.__new__(str, original_value)
-
- obj = str.__new__(cls, original_value)
- obj.original_value = original_value
- return obj
-
- def encode(self, encoding):
- def rewrite(match):
- return match.group(1) + encoding
- return self.CHARSET_RE.sub(rewrite, self.original_value)
-
-class HTMLAwareEntitySubstitution(EntitySubstitution):
-
- """Entity substitution rules that are aware of some HTML quirks.
-
- Specifically, the contents of <script> and <style> tags should not
- undergo entity substitution.
-
- Incoming NavigableString objects are checked to see if they're the
- direct children of a <script> or <style> tag.
- """
-
- cdata_containing_tags = set(["script", "style"])
-
- preformatted_tags = set(["pre"])
-
- @classmethod
- def _substitute_if_appropriate(cls, ns, f):
- if (isinstance(ns, NavigableString)
- and ns.parent is not None
- and ns.parent.name in cls.cdata_containing_tags):
- # Do nothing.
- return ns
- # Substitute.
- return f(ns)
-
- @classmethod
- def substitute_html(cls, ns):
- return cls._substitute_if_appropriate(
- ns, EntitySubstitution.substitute_html)
-
- @classmethod
- def substitute_xml(cls, ns):
- return cls._substitute_if_appropriate(
- ns, EntitySubstitution.substitute_xml)
-
-class PageElement(object):
- """Contains the navigational information for some part of the page
- (either a tag or a piece of text)"""
-
- # There are five possible values for the "formatter" argument passed in
- # to methods like encode() and prettify():
- #
- # "html" - All Unicode characters with corresponding HTML entities
- # are converted to those entities on output.
- # "minimal" - Bare ampersands and angle brackets are converted to
- # XML entities: & < >
- # None - The null formatter. Unicode characters are never
- # converted to entities. This is not recommended, but it's
- # faster than "minimal".
- # A function - This function will be called on every string that
- # needs to undergo entity substitution.
- #
-
- # In an HTML document, the default "html" and "minimal" functions
- # will leave the contents of <script> and <style> tags alone. For
- # an XML document, all tags will be given the same treatment.
-
- HTML_FORMATTERS = {
- "html" : HTMLAwareEntitySubstitution.substitute_html,
- "minimal" : HTMLAwareEntitySubstitution.substitute_xml,
- None : None
- }
-
- XML_FORMATTERS = {
- "html" : EntitySubstitution.substitute_html,
- "minimal" : EntitySubstitution.substitute_xml,
- None : None
- }
-
- def format_string(self, s, formatter='minimal'):
- """Format the given string using the given formatter."""
- if not isinstance(formatter, collections.abc.Callable):
- formatter = self._formatter_for_name(formatter)
- if formatter is None:
- output = s
- else:
- output = formatter(s)
- return output
-
- @property
- def _is_xml(self):
- """Is this element part of an XML tree or an HTML tree?
-
- This is used when mapping a formatter name ("minimal") to an
- appropriate function (one that performs entity-substitution on
- the contents of <script> and <style> tags, or not). It's
- inefficient, but it should be called very rarely.
- """
- if self.parent is None:
- # This is the top-level object. It should have .is_xml set
- # from tree creation. If not, take a guess--BS is usually
- # used on HTML markup.
- return getattr(self, 'is_xml', False)
- return self.parent._is_xml
-
- def _formatter_for_name(self, name):
- "Look up a formatter function based on its name and the tree."
- if self._is_xml:
- return self.XML_FORMATTERS.get(
- name, EntitySubstitution.substitute_xml)
- else:
- return self.HTML_FORMATTERS.get(
- name, HTMLAwareEntitySubstitution.substitute_xml)
-
- def setup(self, parent=None, previous_element=None, next_element=None,
- previous_sibling=None, next_sibling=None):
- """Sets up the initial relations between this element and
- other elements."""
- self.parent = parent
-
- self.previous_element = previous_element
- if previous_element is not None:
- self.previous_element.next_element = self
-
- self.next_element = next_element
- if self.next_element:
- self.next_element.previous_element = self
-
- self.next_sibling = next_sibling
- if self.next_sibling:
- self.next_sibling.previous_sibling = self
-
- if (not previous_sibling
- and self.parent is not None and self.parent.contents):
- previous_sibling = self.parent.contents[-1]
-
- self.previous_sibling = previous_sibling
- if previous_sibling:
- self.previous_sibling.next_sibling = self
-
- nextSibling = _alias("next_sibling") # BS3
- previousSibling = _alias("previous_sibling") # BS3
-
- def replace_with(self, replace_with):
- if not self.parent:
- raise ValueError(
- "Cannot replace one element with another when the"
- "element to be replaced is not part of a tree.")
- if replace_with is self:
- return
- if replace_with is self.parent:
- raise ValueError("Cannot replace a Tag with its parent.")
- old_parent = self.parent
- my_index = self.parent.index(self)
- self.extract()
- old_parent.insert(my_index, replace_with)
- return self
- replaceWith = replace_with # BS3
-
- def unwrap(self):
- my_parent = self.parent
- if not self.parent:
- raise ValueError(
- "Cannot replace an element with its contents when that"
- "element is not part of a tree.")
- my_index = self.parent.index(self)
- self.extract()
- for child in reversed(self.contents[:]):
- my_parent.insert(my_index, child)
- return self
- replace_with_children = unwrap
- replaceWithChildren = unwrap # BS3
-
- def wrap(self, wrap_inside):
- me = self.replace_with(wrap_inside)
- wrap_inside.append(me)
- return wrap_inside
-
- def extract(self):
- """Destructively rips this element out of the tree."""
- if self.parent is not None:
- del self.parent.contents[self.parent.index(self)]
-
- #Find the two elements that would be next to each other if
- #this element (and any children) hadn't been parsed. Connect
- #the two.
- last_child = self._last_descendant()
- next_element = last_child.next_element
-
- if (self.previous_element is not None and
- self.previous_element is not next_element):
- self.previous_element.next_element = next_element
- if next_element is not None and next_element is not self.previous_element:
- next_element.previous_element = self.previous_element
- self.previous_element = None
- last_child.next_element = None
-
- self.parent = None
- if (self.previous_sibling is not None
- and self.previous_sibling is not self.next_sibling):
- self.previous_sibling.next_sibling = self.next_sibling
- if (self.next_sibling is not None
- and self.next_sibling is not self.previous_sibling):
- self.next_sibling.previous_sibling = self.previous_sibling
- self.previous_sibling = self.next_sibling = None
- return self
-
- def _last_descendant(self, is_initialized=True, accept_self=True):
- "Finds the last element beneath this object to be parsed."
- if is_initialized and self.next_sibling:
- last_child = self.next_sibling.previous_element
- else:
- last_child = self
- while isinstance(last_child, Tag) and last_child.contents:
- last_child = last_child.contents[-1]
- if not accept_self and last_child is self:
- last_child = None
- return last_child
- # BS3: Not part of the API!
- _lastRecursiveChild = _last_descendant
-
- def insert(self, position, new_child):
- if new_child is None:
- raise ValueError("Cannot insert None into a tag.")
- if new_child is self:
- raise ValueError("Cannot insert a tag into itself.")
- if (isinstance(new_child, str)
- and not isinstance(new_child, NavigableString)):
- new_child = NavigableString(new_child)
-
- position = min(position, len(self.contents))
- if hasattr(new_child, 'parent') and new_child.parent is not None:
- # We're 'inserting' an element that's already one
- # of this object's children.
- if new_child.parent is self:
- current_index = self.index(new_child)
- if current_index < position:
- # We're moving this element further down the list
- # of this object's children. That means that when
- # we extract this element, our target index will
- # jump down one.
- position -= 1
- new_child.extract()
-
- new_child.parent = self
- previous_child = None
- if position == 0:
- new_child.previous_sibling = None
- new_child.previous_element = self
- else:
- previous_child = self.contents[position - 1]
- new_child.previous_sibling = previous_child
- new_child.previous_sibling.next_sibling = new_child
- new_child.previous_element = previous_child._last_descendant(False)
- if new_child.previous_element is not None:
- new_child.previous_element.next_element = new_child
-
- new_childs_last_element = new_child._last_descendant(False)
-
- if position >= len(self.contents):
- new_child.next_sibling = None
-
- parent = self
- parents_next_sibling = None
- while parents_next_sibling is None and parent is not None:
- parents_next_sibling = parent.next_sibling
- parent = parent.parent
- if parents_next_sibling is not None:
- # We found the element that comes next in the document.
- break
- if parents_next_sibling is not None:
- new_childs_last_element.next_element = parents_next_sibling
- else:
- # The last element of this tag is the last element in
- # the document.
- new_childs_last_element.next_element = None
- else:
- next_child = self.contents[position]
- new_child.next_sibling = next_child
- if new_child.next_sibling is not None:
- new_child.next_sibling.previous_sibling = new_child
- new_childs_last_element.next_element = next_child
-
- if new_childs_last_element.next_element is not None:
- new_childs_last_element.next_element.previous_element = new_childs_last_element
- self.contents.insert(position, new_child)
-
- def append(self, tag):
- """Appends the given tag to the contents of this tag."""
- self.insert(len(self.contents), tag)
-
- def insert_before(self, predecessor):
- """Makes the given element the immediate predecessor of this one.
-
- The two elements will have the same parent, and the given element
- will be immediately before this one.
- """
- if self is predecessor:
- raise ValueError("Can't insert an element before itself.")
- parent = self.parent
- if parent is None:
- raise ValueError(
- "Element has no parent, so 'before' has no meaning.")
- # Extract first so that the index won't be screwed up if they
- # are siblings.
- if isinstance(predecessor, PageElement):
- predecessor.extract()
- index = parent.index(self)
- parent.insert(index, predecessor)
-
- def insert_after(self, successor):
- """Makes the given element the immediate successor of this one.
-
- The two elements will have the same parent, and the given element
- will be immediately after this one.
- """
- if self is successor:
- raise ValueError("Can't insert an element after itself.")
- parent = self.parent
- if parent is None:
- raise ValueError(
- "Element has no parent, so 'after' has no meaning.")
- # Extract first so that the index won't be screwed up if they
- # are siblings.
- if isinstance(successor, PageElement):
- successor.extract()
- index = parent.index(self)
- parent.insert(index+1, successor)
-
- def find_next(self, name=None, attrs={}, text=None, **kwargs):
- """Returns the first item that matches the given criteria and
- appears after this Tag in the document."""
- return self._find_one(self.find_all_next, name, attrs, text, **kwargs)
- findNext = find_next # BS3
-
- def find_all_next(self, name=None, attrs={}, text=None, limit=None,
- **kwargs):
- """Returns all items that match the given criteria and appear
- after this Tag in the document."""
- return self._find_all(name, attrs, text, limit, self.next_elements,
- **kwargs)
- findAllNext = find_all_next # BS3
-
- def find_next_sibling(self, name=None, attrs={}, text=None, **kwargs):
- """Returns the closest sibling to this Tag that matches the
- given criteria and appears after this Tag in the document."""
- return self._find_one(self.find_next_siblings, name, attrs, text,
- **kwargs)
- findNextSibling = find_next_sibling # BS3
-
- def find_next_siblings(self, name=None, attrs={}, text=None, limit=None,
- **kwargs):
- """Returns the siblings of this Tag that match the given
- criteria and appear after this Tag in the document."""
- return self._find_all(name, attrs, text, limit,
- self.next_siblings, **kwargs)
- findNextSiblings = find_next_siblings # BS3
- fetchNextSiblings = find_next_siblings # BS2
-
- def find_previous(self, name=None, attrs={}, text=None, **kwargs):
- """Returns the first item that matches the given criteria and
- appears before this Tag in the document."""
- return self._find_one(
- self.find_all_previous, name, attrs, text, **kwargs)
- findPrevious = find_previous # BS3
-
- def find_all_previous(self, name=None, attrs={}, text=None, limit=None,
- **kwargs):
- """Returns all items that match the given criteria and appear
- before this Tag in the document."""
- return self._find_all(name, attrs, text, limit, self.previous_elements,
- **kwargs)
- findAllPrevious = find_all_previous # BS3
- fetchPrevious = find_all_previous # BS2
-
- def find_previous_sibling(self, name=None, attrs={}, text=None, **kwargs):
- """Returns the closest sibling to this Tag that matches the
- given criteria and appears before this Tag in the document."""
- return self._find_one(self.find_previous_siblings, name, attrs, text,
- **kwargs)
- findPreviousSibling = find_previous_sibling # BS3
-
- def find_previous_siblings(self, name=None, attrs={}, text=None,
- limit=None, **kwargs):
- """Returns the siblings of this Tag that match the given
- criteria and appear before this Tag in the document."""
- return self._find_all(name, attrs, text, limit,
- self.previous_siblings, **kwargs)
- findPreviousSiblings = find_previous_siblings # BS3
- fetchPreviousSiblings = find_previous_siblings # BS2
-
- def find_parent(self, name=None, attrs={}, **kwargs):
- """Returns the closest parent of this Tag that matches the given
- criteria."""
- # NOTE: We can't use _find_one because findParents takes a different
- # set of arguments.
- r = None
- l = self.find_parents(name, attrs, 1, **kwargs)
- if l:
- r = l[0]
- return r
- findParent = find_parent # BS3
-
- def find_parents(self, name=None, attrs={}, limit=None, **kwargs):
- """Returns the parents of this Tag that match the given
- criteria."""
-
- return self._find_all(name, attrs, None, limit, self.parents,
- **kwargs)
- findParents = find_parents # BS3
- fetchParents = find_parents # BS2
-
- @property
- def next(self):
- return self.next_element
-
- @property
- def previous(self):
- return self.previous_element
-
- #These methods do the real heavy lifting.
-
- def _find_one(self, method, name, attrs, text, **kwargs):
- r = None
- l = method(name, attrs, text, 1, **kwargs)
- if l:
- r = l[0]
- return r
-
- def _find_all(self, name, attrs, text, limit, generator, **kwargs):
- "Iterates over a generator looking for things that match."
-
- if text is None and 'string' in kwargs:
- text = kwargs['string']
- del kwargs['string']
-
- if isinstance(name, SoupStrainer):
- strainer = name
- else:
- strainer = SoupStrainer(name, attrs, text, **kwargs)
-
- if text is None and not limit and not attrs and not kwargs:
- if name is True or name is None:
- # Optimization to find all tags.
- result = (element for element in generator
- if isinstance(element, Tag))
- return ResultSet(strainer, result)
- elif isinstance(name, str):
- # Optimization to find all tags with a given name.
- result = (element for element in generator
- if isinstance(element, Tag)
- and element.name == name)
- return ResultSet(strainer, result)
- results = ResultSet(strainer)
- while True:
- try:
- i = next(generator)
- except StopIteration:
- break
- if i:
- found = strainer.search(i)
- if found:
- results.append(found)
- if limit and len(results) >= limit:
- break
- return results
-
- #These generators can be used to navigate starting from both
- #NavigableStrings and Tags.
- @property
- def next_elements(self):
- i = self.next_element
- while i is not None:
- yield i
- i = i.next_element
-
- @property
- def next_siblings(self):
- i = self.next_sibling
- while i is not None:
- yield i
- i = i.next_sibling
-
- @property
- def previous_elements(self):
- i = self.previous_element
- while i is not None:
- yield i
- i = i.previous_element
-
- @property
- def previous_siblings(self):
- i = self.previous_sibling
- while i is not None:
- yield i
- i = i.previous_sibling
-
- @property
- def parents(self):
- i = self.parent
- while i is not None:
- yield i
- i = i.parent
-
- # Methods for supporting CSS selectors.
-
- tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
-
- # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
- # \---------------------------/ \---/\-------------/ \-------/
- # | | | |
- # | | | The value
- # | | ~,|,^,$,* or =
- # | Attribute
- # Tag
- attribselect_re = re.compile(
- r'^(?P<tag>[a-zA-Z0-9][-.a-zA-Z0-9:_]*)?\[(?P<attribute>[\w-]+)(?P<operator>[=~\|\^\$\*]?)' +
- r'=?"?(?P<value>[^\]"]*)"?\]$'
- )
-
- def _attr_value_as_string(self, value, default=None):
- """Force an attribute value into a string representation.
-
- A multi-valued attribute will be converted into a
- space-separated stirng.
- """
- value = self.get(value, default)
- if isinstance(value, list) or isinstance(value, tuple):
- value =" ".join(value)
- return value
-
- def _tag_name_matches_and(self, function, tag_name):
- if not tag_name:
- return function
- else:
- def _match(tag):
- return tag.name == tag_name and function(tag)
- return _match
-
- def _attribute_checker(self, operator, attribute, value=''):
- """Create a function that performs a CSS selector operation.
-
- Takes an operator, attribute and optional value. Returns a
- function that will return True for elements that match that
- combination.
- """
- if operator == '=':
- # string representation of `attribute` is equal to `value`
- return lambda el: el._attr_value_as_string(attribute) == value
- elif operator == '~':
- # space-separated list representation of `attribute`
- # contains `value`
- def _includes_value(element):
- attribute_value = element.get(attribute, [])
- if not isinstance(attribute_value, list):
- attribute_value = attribute_value.split()
- return value in attribute_value
- return _includes_value
- elif operator == '^':
- # string representation of `attribute` starts with `value`
- return lambda el: el._attr_value_as_string(
- attribute, '').startswith(value)
- elif operator == '$':
- # string represenation of `attribute` ends with `value`
- return lambda el: el._attr_value_as_string(
- attribute, '').endswith(value)
- elif operator == '*':
- # string representation of `attribute` contains `value`
- return lambda el: value in el._attr_value_as_string(attribute, '')
- elif operator == '|':
- # string representation of `attribute` is either exactly
- # `value` or starts with `value` and then a dash.
- def _is_or_starts_with_dash(element):
- attribute_value = element._attr_value_as_string(attribute, '')
- return (attribute_value == value or attribute_value.startswith(
- value + '-'))
- return _is_or_starts_with_dash
- else:
- return lambda el: el.has_attr(attribute)
-
- # Old non-property versions of the generators, for backwards
- # compatibility with BS3.
- def nextGenerator(self):
- return self.next_elements
-
- def nextSiblingGenerator(self):
- return self.next_siblings
-
- def previousGenerator(self):
- return self.previous_elements
-
- def previousSiblingGenerator(self):
- return self.previous_siblings
-
- def parentGenerator(self):
- return self.parents
-
-
-class NavigableString(str, PageElement):
-
- PREFIX = ''
- SUFFIX = ''
-
- def __new__(cls, value):
- """Create a new NavigableString.
-
- When unpickling a NavigableString, this method is called with
- the string in DEFAULT_OUTPUT_ENCODING. That encoding needs to be
- passed in to the superclass's __new__ or the superclass won't know
- how to handle non-ASCII characters.
- """
- if isinstance(value, str):
- u = str.__new__(cls, value)
- else:
- u = str.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
- u.setup()
- return u
-
- def __copy__(self):
- """A copy of a NavigableString has the same contents and class
- as the original, but it is not connected to the parse tree.
- """
- return type(self)(self)
-
- def __getnewargs__(self):
- return (str(self),)
-
- def __getattr__(self, attr):
- """text.string gives you text. This is for backwards
- compatibility for Navigable*String, but for CData* it lets you
- get the string without the CData wrapper."""
- if attr == 'string':
- return self
- else:
- raise AttributeError(
- "'%s' object has no attribute '%s'" % (
- self.__class__.__name__, attr))
-
- def output_ready(self, formatter="minimal"):
- output = self.format_string(self, formatter)
- return self.PREFIX + output + self.SUFFIX
-
- @property
- def name(self):
- return None
-
- @name.setter
- def name(self, name):
- raise AttributeError("A NavigableString cannot be given a name.")
-
-class PreformattedString(NavigableString):
- """A NavigableString not subject to the normal formatting rules.
-
- The string will be passed into the formatter (to trigger side effects),
- but the return value will be ignored.
- """
-
- def output_ready(self, formatter="minimal"):
- """CData strings are passed into the formatter.
- But the return value is ignored."""
- self.format_string(self, formatter)
- return self.PREFIX + self + self.SUFFIX
-
-class CData(PreformattedString):
-
- PREFIX = '<![CDATA['
- SUFFIX = ']]>'
-
-class ProcessingInstruction(PreformattedString):
-
- PREFIX = '<?'
- SUFFIX = '>'
-
-class Comment(PreformattedString):
-
- PREFIX = '<!--'
- SUFFIX = '-->'
-
-
-class Declaration(PreformattedString):
- PREFIX = '<?'
- SUFFIX = '?>'
-
-
-class Doctype(PreformattedString):
-
- @classmethod
- def for_name_and_ids(cls, name, pub_id, system_id):
- value = name or ''
- if pub_id is not None:
- value += ' PUBLIC "%s"' % pub_id
- if system_id is not None:
- value += ' "%s"' % system_id
- elif system_id is not None:
- value += ' SYSTEM "%s"' % system_id
-
- return Doctype(value)
-
- PREFIX = '<!DOCTYPE '
- SUFFIX = '>\n'
-
-
-class Tag(PageElement):
-
- """Represents a found HTML tag with its attributes and contents."""
-
- def __init__(self, parser=None, builder=None, name=None, namespace=None,
- prefix=None, attrs=None, parent=None, previous=None):
- "Basic constructor."
-
- if parser is None:
- self.parser_class = None
- else:
- # We don't actually store the parser object: that lets extracted
- # chunks be garbage-collected.
- self.parser_class = parser.__class__
- if name is None:
- raise ValueError("No value provided for new tag's name.")
- self.name = name
- self.namespace = namespace
- self.prefix = prefix
- if attrs is None:
- attrs = {}
- elif attrs:
- if builder is not None and builder.cdata_list_attributes:
- attrs = builder._replace_cdata_list_attribute_values(
- self.name, attrs)
- else:
- attrs = dict(attrs)
- else:
- attrs = dict(attrs)
- self.attrs = attrs
- self.contents = []
- self.setup(parent, previous)
- self.hidden = False
-
- # Set up any substitutions, such as the charset in a META tag.
- if builder is not None:
- builder.set_up_substitutions(self)
- self.can_be_empty_element = builder.can_be_empty_element(name)
- else:
- self.can_be_empty_element = False
-
- parserClass = _alias("parser_class") # BS3
-
- def __copy__(self):
- """A copy of a Tag is a new Tag, unconnected to the parse tree.
- Its contents are a copy of the old Tag's contents.
- """
- clone = type(self)(None, self.builder, self.name, self.namespace,
- self.nsprefix, self.attrs)
- for attr in ('can_be_empty_element', 'hidden'):
- setattr(clone, attr, getattr(self, attr))
- for child in self.contents:
- clone.append(child.__copy__())
- return clone
-
- @property
- def is_empty_element(self):
- """Is this tag an empty-element tag? (aka a self-closing tag)
-
- A tag that has contents is never an empty-element tag.
-
- A tag that has no contents may or may not be an empty-element
- tag. It depends on the builder used to create the tag. If the
- builder has a designated list of empty-element tags, then only
- a tag whose name shows up in that list is considered an
- empty-element tag.
-
- If the builder has no designated list of empty-element tags,
- then any tag with no contents is an empty-element tag.
- """
- return len(self.contents) == 0 and self.can_be_empty_element
- isSelfClosing = is_empty_element # BS3
-
- @property
- def string(self):
- """Convenience property to get the single string within this tag.
-
- :Return: If this tag has a single string child, return value
- is that string. If this tag has no children, or more than one
- child, return value is None. If this tag has one child tag,
- return value is the 'string' attribute of the child tag,
- recursively.
- """
- if len(self.contents) != 1:
- return None
- child = self.contents[0]
- if isinstance(child, NavigableString):
- return child
- return child.string
-
- @string.setter
- def string(self, string):
- self.clear()
- self.append(string.__class__(string))
-
- def _all_strings(self, strip=False, types=(NavigableString, CData)):
- """Yield all strings of certain classes, possibly stripping them.
-
- By default, yields only NavigableString and CData objects. So
- no comments, processing instructions, etc.
- """
- for descendant in self.descendants:
- if (
- (types is None and not isinstance(descendant, NavigableString))
- or
- (types is not None and type(descendant) not in types)):
- continue
- if strip:
- descendant = descendant.strip()
- if len(descendant) == 0:
- continue
- yield descendant
-
- strings = property(_all_strings)
-
- @property
- def stripped_strings(self):
- for string in self._all_strings(True):
- yield string
-
- def get_text(self, separator="", strip=False,
- types=(NavigableString, CData)):
- """
- Get all child strings, concatenated using the given separator.
- """
- return separator.join([s for s in self._all_strings(
- strip, types=types)])
- getText = get_text
- text = property(get_text)
-
- def decompose(self):
- """Recursively destroys the contents of this tree."""
- self.extract()
- i = self
- while i is not None:
- next = i.next_element
- i.__dict__.clear()
- i.contents = []
- i = next
-
- def clear(self, decompose=False):
- """
- Extract all children. If decompose is True, decompose instead.
- """
- if decompose:
- for element in self.contents[:]:
- if isinstance(element, Tag):
- element.decompose()
- else:
- element.extract()
- else:
- for element in self.contents[:]:
- element.extract()
-
- def index(self, element):
- """
- Find the index of a child by identity, not value. Avoids issues with
- tag.contents.index(element) getting the index of equal elements.
- """
- for i, child in enumerate(self.contents):
- if child is element:
- return i
- raise ValueError("Tag.index: element not in tag")
-
- def get(self, key, default=None):
- """Returns the value of the 'key' attribute for the tag, or
- the value given for 'default' if it doesn't have that
- attribute."""
- return self.attrs.get(key, default)
-
- def has_attr(self, key):
- return key in self.attrs
-
- def __hash__(self):
- return str(self).__hash__()
-
- def __getitem__(self, key):
- """tag[key] returns the value of the 'key' attribute for the tag,
- and throws an exception if it's not there."""
- return self.attrs[key]
-
- def __iter__(self):
- "Iterating over a tag iterates over its contents."
- return iter(self.contents)
-
- def __len__(self):
- "The length of a tag is the length of its list of contents."
- return len(self.contents)
-
- def __contains__(self, x):
- return x in self.contents
-
- def __bool__(self):
- "A tag is non-None even if it has no contents."
- return True
-
- def __setitem__(self, key, value):
- """Setting tag[key] sets the value of the 'key' attribute for the
- tag."""
- self.attrs[key] = value
-
- def __delitem__(self, key):
- "Deleting tag[key] deletes all 'key' attributes for the tag."
- self.attrs.pop(key, None)
-
- def __call__(self, *args, **kwargs):
- """Calling a tag like a function is the same as calling its
- find_all() method. Eg. tag('a') returns a list of all the A tags
- found within this tag."""
- return self.find_all(*args, **kwargs)
-
- def __getattr__(self, tag):
- #print "Getattr %s.%s" % (self.__class__, tag)
- if len(tag) > 3 and tag.endswith('Tag'):
- # BS3: soup.aTag -> "soup.find("a")
- tag_name = tag[:-3]
- warnings.warn(
- '.%sTag is deprecated, use .find("%s") instead.' % (
- tag_name, tag_name))
- return self.find(tag_name)
- # We special case contents to avoid recursion.
- elif not tag.startswith("__") and not tag=="contents":
- return self.find(tag)
- raise AttributeError(
- "'%s' object has no attribute '%s'" % (self.__class__, tag))
-
- def __eq__(self, other):
- """Returns true iff this tag has the same name, the same attributes,
- and the same contents (recursively) as the given tag."""
- if self is other:
- return True
- if (not hasattr(other, 'name') or
- not hasattr(other, 'attrs') or
- not hasattr(other, 'contents') or
- self.name != other.name or
- self.attrs != other.attrs or
- len(self) != len(other)):
- return False
- for i, my_child in enumerate(self.contents):
- if my_child != other.contents[i]:
- return False
- return True
-
- def __ne__(self, other):
- """Returns true iff this tag is not identical to the other tag,
- as defined in __eq__."""
- return not self == other
-
- def __repr__(self, encoding="unicode-escape"):
- """Renders this tag as a string."""
- if PY3K:
- # "The return value must be a string object", i.e. Unicode
- return self.decode()
- else:
- # "The return value must be a string object", i.e. a bytestring.
- # By convention, the return value of __repr__ should also be
- # an ASCII string.
- return self.encode(encoding)
-
- def __unicode__(self):
- return self.decode()
-
- def __str__(self):
- if PY3K:
- return self.decode()
- else:
- return self.encode()
-
- if PY3K:
- __str__ = __repr__ = __unicode__
-
- def encode(self, encoding=DEFAULT_OUTPUT_ENCODING,
- indent_level=None, formatter="minimal",
- errors="xmlcharrefreplace"):
- # Turn the data structure into Unicode, then encode the
- # Unicode.
- u = self.decode(indent_level, encoding, formatter)
- return u.encode(encoding, errors)
-
- def _should_pretty_print(self, indent_level):
- """Should this tag be pretty-printed?"""
- return (
- indent_level is not None and
- (self.name not in HTMLAwareEntitySubstitution.preformatted_tags
- or self._is_xml))
-
- def decode(self, indent_level=None,
- eventual_encoding=DEFAULT_OUTPUT_ENCODING,
- formatter="minimal"):
- """Returns a Unicode representation of this tag and its contents.
-
- :param eventual_encoding: The tag is destined to be
- encoded into this encoding. This method is _not_
- responsible for performing that encoding. This information
- is passed in so that it can be substituted in if the
- document contains a <META> tag that mentions the document's
- encoding.
- """
-
- # First off, turn a string formatter into a function. This
- # will stop the lookup from happening over and over again.
- if not isinstance(formatter, collections.abc.Callable):
- formatter = self._formatter_for_name(formatter)
-
- attrs = []
- if self.attrs:
- for key, val in sorted(self.attrs.items()):
- if val is None:
- decoded = key
- else:
- if isinstance(val, list) or isinstance(val, tuple):
- val = ' '.join(val)
- elif not isinstance(val, str):
- val = str(val)
- elif (
- isinstance(val, AttributeValueWithCharsetSubstitution)
- and eventual_encoding is not None):
- val = val.encode(eventual_encoding)
-
- text = self.format_string(val, formatter)
- decoded = (
- str(key) + '='
- + EntitySubstitution.quoted_attribute_value(text))
- attrs.append(decoded)
- close = ''
- closeTag = ''
-
- prefix = ''
- if self.prefix:
- prefix = self.prefix + ":"
-
- if self.is_empty_element:
- close = '/'
- else:
- closeTag = '</%s%s>' % (prefix, self.name)
-
- pretty_print = self._should_pretty_print(indent_level)
- space = ''
- indent_space = ''
- if indent_level is not None:
- indent_space = (' ' * (indent_level - 1))
- if pretty_print:
- space = indent_space
- indent_contents = indent_level + 1
- else:
- indent_contents = None
- contents = self.decode_contents(
- indent_contents, eventual_encoding, formatter)
-
- if self.hidden:
- # This is the 'document root' object.
- s = contents
- else:
- s = []
- attribute_string = ''
- if attrs:
- attribute_string = ' ' + ' '.join(attrs)
- if indent_level is not None:
- # Even if this particular tag is not pretty-printed,
- # we should indent up to the start of the tag.
- s.append(indent_space)
- s.append('<%s%s%s%s>' % (
- prefix, self.name, attribute_string, close))
- if pretty_print:
- s.append("\n")
- s.append(contents)
- if pretty_print and contents and contents[-1] != "\n":
- s.append("\n")
- if pretty_print and closeTag:
- s.append(space)
- s.append(closeTag)
- if indent_level is not None and closeTag and self.next_sibling:
- # Even if this particular tag is not pretty-printed,
- # we're now done with the tag, and we should add a
- # newline if appropriate.
- s.append("\n")
- s = ''.join(s)
- return s
-
- def prettify(self, encoding=None, formatter="minimal"):
- if encoding is None:
- return self.decode(True, formatter=formatter)
- else:
- return self.encode(encoding, True, formatter=formatter)
-
- def decode_contents(self, indent_level=None,
- eventual_encoding=DEFAULT_OUTPUT_ENCODING,
- formatter="minimal"):
- """Renders the contents of this tag as a Unicode string.
-
- :param indent_level: Each line of the rendering will be
- indented this many spaces.
-
- :param eventual_encoding: The tag is destined to be
- encoded into this encoding. This method is _not_
- responsible for performing that encoding. This information
- is passed in so that it can be substituted in if the
- document contains a <META> tag that mentions the document's
- encoding.
-
- :param formatter: The output formatter responsible for converting
- entities to Unicode characters.
- """
- # First off, turn a string formatter into a function. This
- # will stop the lookup from happening over and over again.
- if not isinstance(formatter, collections.abc.Callable):
- formatter = self._formatter_for_name(formatter)
-
- pretty_print = (indent_level is not None)
- s = []
- for c in self:
- text = None
- if isinstance(c, NavigableString):
- text = c.output_ready(formatter)
- elif isinstance(c, Tag):
- s.append(c.decode(indent_level, eventual_encoding,
- formatter))
- if text and indent_level and not self.name == 'pre':
- text = text.strip()
- if text:
- if pretty_print and not self.name == 'pre':
- s.append(" " * (indent_level - 1))
- s.append(text)
- if pretty_print and not self.name == 'pre':
- s.append("\n")
- return ''.join(s)
-
- def encode_contents(
- self, indent_level=None, encoding=DEFAULT_OUTPUT_ENCODING,
- formatter="minimal"):
- """Renders the contents of this tag as a bytestring.
-
- :param indent_level: Each line of the rendering will be
- indented this many spaces.
-
- :param eventual_encoding: The bytestring will be in this encoding.
-
- :param formatter: The output formatter responsible for converting
- entities to Unicode characters.
- """
-
- contents = self.decode_contents(indent_level, encoding, formatter)
- return contents.encode(encoding)
-
- # Old method for BS3 compatibility
- def renderContents(self, encoding=DEFAULT_OUTPUT_ENCODING,
- prettyPrint=False, indentLevel=0):
- if not prettyPrint:
- indentLevel = None
- return self.encode_contents(
- indent_level=indentLevel, encoding=encoding)
-
- #Soup methods
-
- def find(self, name=None, attrs={}, recursive=True, text=None,
- **kwargs):
- """Return only the first child of this Tag matching the given
- criteria."""
- r = None
- l = self.find_all(name, attrs, recursive, text, 1, **kwargs)
- if l:
- r = l[0]
- return r
- findChild = find
-
- def find_all(self, name=None, attrs={}, recursive=True, text=None,
- limit=None, **kwargs):
- """Extracts a list of Tag objects that match the given
- criteria. You can specify the name of the Tag and any
- attributes you want the Tag to have.
-
- The value of a key-value pair in the 'attrs' map can be a
- string, a list of strings, a regular expression object, or a
- callable that takes a string and returns whether or not the
- string matches for some custom definition of 'matches'. The
- same is true of the tag name."""
-
- generator = self.descendants
- if not recursive:
- generator = self.children
- return self._find_all(name, attrs, text, limit, generator, **kwargs)
- findAll = find_all # BS3
- findChildren = find_all # BS2
-
- #Generator methods
- @property
- def children(self):
- # return iter() to make the purpose of the method clear
- return iter(self.contents) # XXX This seems to be untested.
-
- @property
- def descendants(self):
- if not len(self.contents):
- return
- stopNode = self._last_descendant().next_element
- current = self.contents[0]
- while current is not stopNode:
- yield current
- current = current.next_element
-
- # CSS selector code
-
- _selector_combinators = ['>', '+', '~']
- _select_debug = False
- def select_one(self, selector):
- """Perform a CSS selection operation on the current element."""
- value = self.select(selector, limit=1)
- if value:
- return value[0]
- return None
-
- def select(self, selector, _candidate_generator=None, limit=None):
- """Perform a CSS selection operation on the current element."""
-
- # Handle grouping selectors if ',' exists, ie: p,a
- if ',' in selector:
- context = []
- for partial_selector in selector.split(','):
- partial_selector = partial_selector.strip()
- if partial_selector == '':
- raise ValueError('Invalid group selection syntax: %s' % selector)
- candidates = self.select(partial_selector, limit=limit)
- for candidate in candidates:
- if candidate not in context:
- context.append(candidate)
-
- if limit and len(context) >= limit:
- break
- return context
-
- tokens = selector.split()
- current_context = [self]
-
- if tokens[-1] in self._selector_combinators:
- raise ValueError(
- 'Final combinator "%s" is missing an argument.' % tokens[-1])
-
- if self._select_debug:
- print('Running CSS selector "%s"' % selector)
-
- for index, token in enumerate(tokens):
- new_context = []
- new_context_ids = set([])
-
- if tokens[index-1] in self._selector_combinators:
- # This token was consumed by the previous combinator. Skip it.
- if self._select_debug:
- print(' Token was consumed by the previous combinator.')
- continue
-
- if self._select_debug:
- print(' Considering token "%s"' % token)
- recursive_candidate_generator = None
- tag_name = None
-
- # Each operation corresponds to a checker function, a rule
- # for determining whether a candidate matches the
- # selector. Candidates are generated by the active
- # iterator.
- checker = None
-
- m = self.attribselect_re.match(token)
- if m is not None:
- # Attribute selector
- tag_name, attribute, operator, value = m.groups()
- checker = self._attribute_checker(operator, attribute, value)
-
- elif '#' in token:
- # ID selector
- tag_name, tag_id = token.split('#', 1)
- def id_matches(tag):
- return tag.get('id', None) == tag_id
- checker = id_matches
-
- elif '.' in token:
- # Class selector
- tag_name, klass = token.split('.', 1)
- classes = set(klass.split('.'))
- def classes_match(candidate):
- return classes.issubset(candidate.get('class', []))
- checker = classes_match
-
- elif ':' in token:
- # Pseudo-class
- tag_name, pseudo = token.split(':', 1)
- if tag_name == '':
- raise ValueError(
- "A pseudo-class must be prefixed with a tag name.")
- pseudo_attributes = re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
- found = []
- if pseudo_attributes is None:
- pseudo_type = pseudo
- pseudo_value = None
- else:
- pseudo_type, pseudo_value = pseudo_attributes.groups()
- if pseudo_type == 'nth-of-type':
- try:
- pseudo_value = int(pseudo_value)
- except:
- raise NotImplementedError(
- 'Only numeric values are currently supported for the nth-of-type pseudo-class.')
- if pseudo_value < 1:
- raise ValueError(
- 'nth-of-type pseudo-class value must be at least 1.')
- class Counter(object):
- def __init__(self, destination):
- self.count = 0
- self.destination = destination
-
- def nth_child_of_type(self, tag):
- self.count += 1
- if self.count == self.destination:
- return True
- if self.count > self.destination:
- # Stop the generator that's sending us
- # these things.
- raise StopIteration()
- return False
- checker = Counter(pseudo_value).nth_child_of_type
- else:
- raise NotImplementedError(
- 'Only the following pseudo-classes are implemented: nth-of-type.')
-
- elif token == '*':
- # Star selector -- matches everything
- pass
- elif token == '>':
- # Run the next token as a CSS selector against the
- # direct children of each tag in the current context.
- recursive_candidate_generator = lambda tag: tag.children
- elif token == '~':
- # Run the next token as a CSS selector against the
- # siblings of each tag in the current context.
- recursive_candidate_generator = lambda tag: tag.next_siblings
- elif token == '+':
- # For each tag in the current context, run the next
- # token as a CSS selector against the tag's next
- # sibling that's a tag.
- def next_tag_sibling(tag):
- yield tag.find_next_sibling(True)
- recursive_candidate_generator = next_tag_sibling
-
- elif self.tag_name_re.match(token):
- # Just a tag name.
- tag_name = token
- else:
- raise ValueError(
- 'Unsupported or invalid CSS selector: "%s"' % token)
- if recursive_candidate_generator:
- # This happens when the selector looks like "> foo".
- #
- # The generator calls select() recursively on every
- # member of the current context, passing in a different
- # candidate generator and a different selector.
- #
- # In the case of "> foo", the candidate generator is
- # one that yields a tag's direct children (">"), and
- # the selector is "foo".
- next_token = tokens[index+1]
- def recursive_select(tag):
- if self._select_debug:
- print(' Calling select("%s") recursively on %s %s' % (next_token, tag.name, tag.attrs))
- print('-' * 40)
- for i in tag.select(next_token, recursive_candidate_generator):
- if self._select_debug:
- print('(Recursive select picked up candidate %s %s)' % (i.name, i.attrs))
- yield i
- if self._select_debug:
- print('-' * 40)
- _use_candidate_generator = recursive_select
- elif _candidate_generator is None:
- # By default, a tag's candidates are all of its
- # children. If tag_name is defined, only yield tags
- # with that name.
- if self._select_debug:
- if tag_name:
- check = "[any]"
- else:
- check = tag_name
- print(' Default candidate generator, tag name="%s"' % check)
- if self._select_debug:
- # This is redundant with later code, but it stops
- # a bunch of bogus tags from cluttering up the
- # debug log.
- def default_candidate_generator(tag):
- for child in tag.descendants:
- if not isinstance(child, Tag):
- continue
- if tag_name and not child.name == tag_name:
- continue
- yield child
- _use_candidate_generator = default_candidate_generator
- else:
- _use_candidate_generator = lambda tag: tag.descendants
- else:
- _use_candidate_generator = _candidate_generator
-
- count = 0
- for tag in current_context:
- if self._select_debug:
- print(" Running candidate generator on %s %s" % (
- tag.name, repr(tag.attrs)))
- for candidate in _use_candidate_generator(tag):
- if not isinstance(candidate, Tag):
- continue
- if tag_name and candidate.name != tag_name:
- continue
- if checker is not None:
- try:
- result = checker(candidate)
- except StopIteration:
- # The checker has decided we should no longer
- # run the generator.
- break
- if checker is None or result:
- if self._select_debug:
- print(" SUCCESS %s %s" % (candidate.name, repr(candidate.attrs)))
- if id(candidate) not in new_context_ids:
- # If a tag matches a selector more than once,
- # don't include it in the context more than once.
- new_context.append(candidate)
- new_context_ids.add(id(candidate))
- if limit and len(new_context) >= limit:
- break
- elif self._select_debug:
- print(" FAILURE %s %s" % (candidate.name, repr(candidate.attrs)))
-
-
- current_context = new_context
-
- if self._select_debug:
- print("Final verdict:")
- for i in current_context:
- print(" %s %s" % (i.name, i.attrs))
- return current_context
-
- # Old names for backwards compatibility
- def childGenerator(self):
- return self.children
-
- def recursiveChildGenerator(self):
- return self.descendants
-
- def has_key(self, key):
- """This was kind of misleading because has_key() (attributes)
- was different from __in__ (contents). has_key() is gone in
- Python 3, anyway."""
- warnings.warn('has_key is deprecated. Use has_attr("%s") instead.' % (
- key))
- return self.has_attr(key)
-
-# Next, a couple classes to represent queries and their results.
-class SoupStrainer(object):
- """Encapsulates a number of ways of matching a markup element (tag or
- text)."""
-
- def __init__(self, name=None, attrs={}, text=None, **kwargs):
- self.name = self._normalize_search_value(name)
- if not isinstance(attrs, dict):
- # Treat a non-dict value for attrs as a search for the 'class'
- # attribute.
- kwargs['class'] = attrs
- attrs = None
-
- if 'class_' in kwargs:
- # Treat class_="foo" as a search for the 'class'
- # attribute, overriding any non-dict value for attrs.
- kwargs['class'] = kwargs['class_']
- del kwargs['class_']
-
- if kwargs:
- if attrs:
- attrs = attrs.copy()
- attrs.update(kwargs)
- else:
- attrs = kwargs
- normalized_attrs = {}
- for key, value in list(attrs.items()):
- normalized_attrs[key] = self._normalize_search_value(value)
-
- self.attrs = normalized_attrs
- self.text = self._normalize_search_value(text)
-
- def _normalize_search_value(self, value):
- # Leave it alone if it's a Unicode string, a callable, a
- # regular expression, a boolean, or None.
- if (isinstance(value, str) or isinstance(value, collections.abc.Callable) or hasattr(value, 'match')
- or isinstance(value, bool) or value is None):
- return value
-
- # If it's a bytestring, convert it to Unicode, treating it as UTF-8.
- if isinstance(value, bytes):
- return value.decode("utf8")
-
- # If it's listlike, convert it into a list of strings.
- if hasattr(value, '__iter__'):
- new_value = []
- for v in value:
- if (hasattr(v, '__iter__') and not isinstance(v, bytes)
- and not isinstance(v, str)):
- # This is almost certainly the user's mistake. In the
- # interests of avoiding infinite loops, we'll let
- # it through as-is rather than doing a recursive call.
- new_value.append(v)
- else:
- new_value.append(self._normalize_search_value(v))
- return new_value
-
- # Otherwise, convert it into a Unicode string.
- # The unicode(str()) thing is so this will do the same thing on Python 2
- # and Python 3.
- return str(str(value))
-
- def __str__(self):
- if self.text:
- return self.text
- else:
- return "%s|%s" % (self.name, self.attrs)
-
- def search_tag(self, markup_name=None, markup_attrs={}):
- found = None
- markup = None
- if isinstance(markup_name, Tag):
- markup = markup_name
- markup_attrs = markup
- call_function_with_tag_data = (
- isinstance(self.name, collections.abc.Callable)
- and not isinstance(markup_name, Tag))
-
- if ((not self.name)
- or call_function_with_tag_data
- or (markup and self._matches(markup, self.name))
- or (not markup and self._matches(markup_name, self.name))):
- if call_function_with_tag_data:
- match = self.name(markup_name, markup_attrs)
- else:
- match = True
- markup_attr_map = None
- for attr, match_against in list(self.attrs.items()):
- if not markup_attr_map:
- if hasattr(markup_attrs, 'get'):
- markup_attr_map = markup_attrs
- else:
- markup_attr_map = {}
- for k, v in markup_attrs:
- markup_attr_map[k] = v
- attr_value = markup_attr_map.get(attr)
- if not self._matches(attr_value, match_against):
- match = False
- break
- if match:
- if markup:
- found = markup
- else:
- found = markup_name
- if found and self.text and not self._matches(found.string, self.text):
- found = None
- return found
- searchTag = search_tag
-
- def search(self, markup):
- # print 'looking for %s in %s' % (self, markup)
- found = None
- # If given a list of items, scan it for a text element that
- # matches.
- if hasattr(markup, '__iter__') and not isinstance(markup, (Tag, str)):
- for element in markup:
- if isinstance(element, NavigableString) \
- and self.search(element):
- found = element
- break
- # If it's a Tag, make sure its name or attributes match.
- # Don't bother with Tags if we're searching for text.
- elif isinstance(markup, Tag):
- if not self.text or self.name or self.attrs:
- found = self.search_tag(markup)
- # If it's text, make sure the text matches.
- elif isinstance(markup, NavigableString) or \
- isinstance(markup, str):
- if not self.name and not self.attrs and self._matches(markup, self.text):
- found = markup
- else:
- raise Exception(
- "I don't know how to match against a %s" % markup.__class__)
- return found
-
- def _matches(self, markup, match_against):
- # print u"Matching %s against %s" % (markup, match_against)
- result = False
- if isinstance(markup, list) or isinstance(markup, tuple):
- # This should only happen when searching a multi-valued attribute
- # like 'class'.
- if (isinstance(match_against, str)
- and ' ' in match_against):
- # A bit of a special case. If they try to match "foo
- # bar" on a multivalue attribute's value, only accept
- # the literal value "foo bar"
- #
- # XXX This is going to be pretty slow because we keep
- # splitting match_against. But it shouldn't come up
- # too often.
- return (whitespace_re.split(match_against) == markup)
- else:
- for item in markup:
- if self._matches(item, match_against):
- return True
- return False
-
- if match_against is True:
- # True matches any non-None value.
- return markup is not None
-
- if isinstance(match_against, collections.abc.Callable):
- return match_against(markup)
-
- # Custom callables take the tag as an argument, but all
- # other ways of matching match the tag name as a string.
- if isinstance(markup, Tag):
- markup = markup.name
-
- # Ensure that `markup` is either a Unicode string, or None.
- markup = self._normalize_search_value(markup)
-
- if markup is None:
- # None matches None, False, an empty string, an empty list, and so on.
- return not match_against
-
- if isinstance(match_against, str):
- # Exact string match
- return markup == match_against
-
- if hasattr(match_against, 'match'):
- # Regexp match
- return match_against.search(markup)
-
- if hasattr(match_against, '__iter__'):
- # The markup must be an exact match against something
- # in the iterable.
- return markup in match_against
-
-
-class ResultSet(list):
- """A ResultSet is just a list that keeps track of the SoupStrainer
- that created it."""
- def __init__(self, source, result=()):
- super(ResultSet, self).__init__(result)
- self.source = source
diff --git a/bitbake/lib/bs4/testing.py b/bitbake/lib/bs4/testing.py
deleted file mode 100644
index 6584ecf3..00000000
--- a/bitbake/lib/bs4/testing.py
+++ /dev/null
@@ -1,686 +0,0 @@
-"""Helper classes for tests."""
-
-__license__ = "MIT"
-
-import pickle
-import copy
-import unittest
-from unittest import TestCase
-from bs4 import BeautifulSoup
-from bs4.element import (
- CharsetMetaAttributeValue,
- Comment,
- ContentMetaAttributeValue,
- Doctype,
- SoupStrainer,
-)
-
-from bs4.builder._htmlparser import HTMLParserTreeBuilder
-default_builder = HTMLParserTreeBuilder
-
-
-class SoupTest(unittest.TestCase):
-
- @property
- def default_builder(self):
- return default_builder()
-
- def soup(self, markup, **kwargs):
- """Build a Beautiful Soup object from markup."""
- builder = kwargs.pop('builder', self.default_builder)
- return BeautifulSoup(markup, builder=builder, **kwargs)
-
- def document_for(self, markup):
- """Turn an HTML fragment into a document.
-
- The details depend on the builder.
- """
- return self.default_builder.test_fragment_to_document(markup)
-
- def assertSoupEquals(self, to_parse, compare_parsed_to=None):
- builder = self.default_builder
- obj = BeautifulSoup(to_parse, builder=builder)
- if compare_parsed_to is None:
- compare_parsed_to = to_parse
-
- self.assertEqual(obj.decode(), self.document_for(compare_parsed_to))
-
- def assertConnectedness(self, element):
- """Ensure that next_element and previous_element are properly
- set for all descendants of the given element.
- """
- earlier = None
- for e in element.descendants:
- if earlier:
- self.assertEqual(e, earlier.next_element)
- self.assertEqual(earlier, e.previous_element)
- earlier = e
-
-class HTMLTreeBuilderSmokeTest(SoupTest):
-
- """A basic test of a treebuilder's competence.
-
- Any HTML treebuilder, present or future, should be able to pass
- these tests. With invalid markup, there's room for interpretation,
- and different parsers can handle it differently. But with the
- markup in these tests, there's not much room for interpretation.
- """
-
- def test_pickle_and_unpickle_identity(self):
- # Pickling a tree, then unpickling it, yields a tree identical
- # to the original.
- tree = self.soup("<a><b>foo</a>")
- dumped = pickle.dumps(tree, 2)
- loaded = pickle.loads(dumped)
- self.assertEqual(loaded.__class__, BeautifulSoup)
- self.assertEqual(loaded.decode(), tree.decode())
-
- def assertDoctypeHandled(self, doctype_fragment):
- """Assert that a given doctype string is handled correctly."""
- doctype_str, soup = self._document_with_doctype(doctype_fragment)
-
- # Make sure a Doctype object was created.
- doctype = soup.contents[0]
- self.assertEqual(doctype.__class__, Doctype)
- self.assertEqual(doctype, doctype_fragment)
- self.assertEqual(str(soup)[:len(doctype_str)], doctype_str)
-
- # Make sure that the doctype was correctly associated with the
- # parse tree and that the rest of the document parsed.
- self.assertEqual(soup.p.contents[0], 'foo')
-
- def _document_with_doctype(self, doctype_fragment):
- """Generate and parse a document with the given doctype."""
- doctype = '<!DOCTYPE %s>' % doctype_fragment
- markup = doctype + '\n<p>foo</p>'
- soup = self.soup(markup)
- return doctype, soup
-
- def test_normal_doctypes(self):
- """Make sure normal, everyday HTML doctypes are handled correctly."""
- self.assertDoctypeHandled("html")
- self.assertDoctypeHandled(
- 'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"')
-
- def test_empty_doctype(self):
- soup = self.soup("<!DOCTYPE>")
- doctype = soup.contents[0]
- self.assertEqual("", doctype.strip())
-
- def test_public_doctype_with_url(self):
- doctype = 'html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"'
- self.assertDoctypeHandled(doctype)
-
- def test_system_doctype(self):
- self.assertDoctypeHandled('foo SYSTEM "http://www.example.com/"')
-
- def test_namespaced_system_doctype(self):
- # We can handle a namespaced doctype with a system ID.
- self.assertDoctypeHandled('xsl:stylesheet SYSTEM "htmlent.dtd"')
-
- def test_namespaced_public_doctype(self):
- # Test a namespaced doctype with a public id.
- self.assertDoctypeHandled('xsl:stylesheet PUBLIC "htmlent.dtd"')
-
- def test_real_xhtml_document(self):
- """A real XHTML document should come out more or less the same as it went in."""
- markup = b"""<?xml version="1.0" encoding="utf-8"?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head><title>Hello.</title></head>
-<body>Goodbye.</body>
-</html>"""
- soup = self.soup(markup)
- self.assertEqual(
- soup.encode("utf-8").replace(b"\n", b""),
- markup.replace(b"\n", b""))
-
- def test_processing_instruction(self):
- markup = b"""<?PITarget PIContent?>"""
- soup = self.soup(markup)
- self.assertEqual(markup, soup.encode("utf8"))
-
- def test_deepcopy(self):
- """Make sure you can copy the tree builder.
-
- This is important because the builder is part of a
- BeautifulSoup object, and we want to be able to copy that.
- """
- copy.deepcopy(self.default_builder)
-
- def test_p_tag_is_never_empty_element(self):
- """A <p> tag is never designated as an empty-element tag.
-
- Even if the markup shows it as an empty-element tag, it
- shouldn't be presented that way.
- """
- soup = self.soup("<p/>")
- self.assertFalse(soup.p.is_empty_element)
- self.assertEqual(str(soup.p), "<p></p>")
-
- def test_unclosed_tags_get_closed(self):
- """A tag that's not closed by the end of the document should be closed.
-
- This applies to all tags except empty-element tags.
- """
- self.assertSoupEquals("<p>", "<p></p>")
- self.assertSoupEquals("<b>", "<b></b>")
-
- self.assertSoupEquals("<br>", "<br/>")
-
- def test_br_is_always_empty_element_tag(self):
- """A <br> tag is designated as an empty-element tag.
-
- Some parsers treat <br></br> as one <br/> tag, some parsers as
- two tags, but it should always be an empty-element tag.
- """
- soup = self.soup("<br></br>")
- self.assertTrue(soup.br.is_empty_element)
- self.assertEqual(str(soup.br), "<br/>")
-
- def test_nested_formatting_elements(self):
- self.assertSoupEquals("<em><em></em></em>")
-
- def test_double_head(self):
- html = '''<!DOCTYPE html>
-<html>
-<head>
-<title>Ordinary HEAD element test</title>
-</head>
-<script type="text/javascript">
-alert("Help!");
-</script>
-<body>
-Hello, world!
-</body>
-</html>
-'''
- soup = self.soup(html)
- self.assertEqual("text/javascript", soup.find('script')['type'])
-
- def test_comment(self):
- # Comments are represented as Comment objects.
- markup = "<p>foo<!--foobar-->baz</p>"
- self.assertSoupEquals(markup)
-
- soup = self.soup(markup)
- comment = soup.find(text="foobar")
- self.assertEqual(comment.__class__, Comment)
-
- # The comment is properly integrated into the tree.
- foo = soup.find(text="foo")
- self.assertEqual(comment, foo.next_element)
- baz = soup.find(text="baz")
- self.assertEqual(comment, baz.previous_element)
-
- def test_preserved_whitespace_in_pre_and_textarea(self):
- """Whitespace must be preserved in <pre> and <textarea> tags."""
- self.assertSoupEquals("<pre> </pre>")
- self.assertSoupEquals("<textarea> woo </textarea>")
-
- def test_nested_inline_elements(self):
- """Inline elements can be nested indefinitely."""
- b_tag = "<b>Inside a B tag</b>"
- self.assertSoupEquals(b_tag)
-
- nested_b_tag = "<p>A <i>nested <b>tag</b></i></p>"
- self.assertSoupEquals(nested_b_tag)
-
- double_nested_b_tag = "<p>A <a>doubly <i>nested <b>tag</b></i></a></p>"
- self.assertSoupEquals(nested_b_tag)
-
- def test_nested_block_level_elements(self):
- """Block elements can be nested."""
- soup = self.soup('<blockquote><p><b>Foo</b></p></blockquote>')
- blockquote = soup.blockquote
- self.assertEqual(blockquote.p.b.string, 'Foo')
- self.assertEqual(blockquote.b.string, 'Foo')
-
- def test_correctly_nested_tables(self):
- """One table can go inside another one."""
- markup = ('<table id="1">'
- '<tr>'
- "<td>Here's another table:"
- '<table id="2">'
- '<tr><td>foo</td></tr>'
- '</table></td>')
-
- self.assertSoupEquals(
- markup,
- '<table id="1"><tr><td>Here\'s another table:'
- '<table id="2"><tr><td>foo</td></tr></table>'
- '</td></tr></table>')
-
- self.assertSoupEquals(
- "<table><thead><tr><td>Foo</td></tr></thead>"
- "<tbody><tr><td>Bar</td></tr></tbody>"
- "<tfoot><tr><td>Baz</td></tr></tfoot></table>")
-
- def test_deeply_nested_multivalued_attribute(self):
- # html5lib can set the attributes of the same tag many times
- # as it rearranges the tree. This has caused problems with
- # multivalued attributes.
- markup = '<table><div><div class="css"></div></div></table>'
- soup = self.soup(markup)
- self.assertEqual(["css"], soup.div.div['class'])
-
- def test_multivalued_attribute_on_html(self):
- # html5lib uses a different API to set the attributes ot the
- # <html> tag. This has caused problems with multivalued
- # attributes.
- markup = '<html class="a b"></html>'
- soup = self.soup(markup)
- self.assertEqual(["a", "b"], soup.html['class'])
-
- def test_angle_brackets_in_attribute_values_are_escaped(self):
- self.assertSoupEquals('<a b="<a>"></a>', '<a b="<a>"></a>')
-
- def test_entities_in_attributes_converted_to_unicode(self):
- expect = '<p id="pi\N{LATIN SMALL LETTER N WITH TILDE}ata"></p>'
- self.assertSoupEquals('<p id="piñata"></p>', expect)
- self.assertSoupEquals('<p id="piñata"></p>', expect)
- self.assertSoupEquals('<p id="piñata"></p>', expect)
- self.assertSoupEquals('<p id="piñata"></p>', expect)
-
- def test_entities_in_text_converted_to_unicode(self):
- expect = '<p>pi\N{LATIN SMALL LETTER N WITH TILDE}ata</p>'
- self.assertSoupEquals("<p>piñata</p>", expect)
- self.assertSoupEquals("<p>piñata</p>", expect)
- self.assertSoupEquals("<p>piñata</p>", expect)
- self.assertSoupEquals("<p>piñata</p>", expect)
-
- def test_quot_entity_converted_to_quotation_mark(self):
- self.assertSoupEquals("<p>I said "good day!"</p>",
- '<p>I said "good day!"</p>')
-
- def test_out_of_range_entity(self):
- expect = "\N{REPLACEMENT CHARACTER}"
- self.assertSoupEquals("�", expect)
- self.assertSoupEquals("�", expect)
- self.assertSoupEquals("�", expect)
-
- def test_multipart_strings(self):
- "Mostly to prevent a recurrence of a bug in the html5lib treebuilder."
- soup = self.soup("<html><h2>\nfoo</h2><p></p></html>")
- self.assertEqual("p", soup.h2.string.next_element.name)
- self.assertEqual("p", soup.p.name)
- self.assertConnectedness(soup)
-
- def test_head_tag_between_head_and_body(self):
- "Prevent recurrence of a bug in the html5lib treebuilder."
- content = """<html><head></head>
- <link></link>
- <body>foo</body>
-</html>
-"""
- soup = self.soup(content)
- self.assertNotEqual(None, soup.html.body)
- self.assertConnectedness(soup)
-
- def test_multiple_copies_of_a_tag(self):
- "Prevent recurrence of a bug in the html5lib treebuilder."
- content = """<!DOCTYPE html>
-<html>
- <body>
- <article id="a" >
- <div><a href="1"></div>
- <footer>
- <a href="2"></a>
- </footer>
- </article>
- </body>
-</html>
-"""
- soup = self.soup(content)
- self.assertConnectedness(soup.article)
-
- def test_basic_namespaces(self):
- """Parsers don't need to *understand* namespaces, but at the
- very least they should not choke on namespaces or lose
- data."""
-
- markup = b'<html xmlns="http://www.w3.org/1999/xhtml" xmlns:mathml="http://www.w3.org/1998/Math/MathML" xmlns:svg="http://www.w3.org/2000/svg"><head></head><body><mathml:msqrt>4</mathml:msqrt><b svg:fill="red"></b></body></html>'
- soup = self.soup(markup)
- self.assertEqual(markup, soup.encode())
- html = soup.html
- self.assertEqual('http://www.w3.org/1999/xhtml', soup.html['xmlns'])
- self.assertEqual(
- 'http://www.w3.org/1998/Math/MathML', soup.html['xmlns:mathml'])
- self.assertEqual(
- 'http://www.w3.org/2000/svg', soup.html['xmlns:svg'])
-
- def test_multivalued_attribute_value_becomes_list(self):
- markup = b'<a class="foo bar">'
- soup = self.soup(markup)
- self.assertEqual(['foo', 'bar'], soup.a['class'])
-
- #
- # Generally speaking, tests below this point are more tests of
- # Beautiful Soup than tests of the tree builders. But parsers are
- # weird, so we run these tests separately for every tree builder
- # to detect any differences between them.
- #
-
- def test_can_parse_unicode_document(self):
- # A seemingly innocuous document... but it's in Unicode! And
- # it contains characters that can't be represented in the
- # encoding found in the declaration! The horror!
- markup = '<html><head><meta encoding="euc-jp"></head><body>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</body>'
- soup = self.soup(markup)
- self.assertEqual('Sacr\xe9 bleu!', soup.body.string)
-
- def test_soupstrainer(self):
- """Parsers should be able to work with SoupStrainers."""
- strainer = SoupStrainer("b")
- soup = self.soup("A <b>bold</b> <meta/> <i>statement</i>",
- parse_only=strainer)
- self.assertEqual(soup.decode(), "<b>bold</b>")
-
- def test_single_quote_attribute_values_become_double_quotes(self):
- self.assertSoupEquals("<foo attr='bar'></foo>",
- '<foo attr="bar"></foo>')
-
- def test_attribute_values_with_nested_quotes_are_left_alone(self):
- text = """<foo attr='bar "brawls" happen'>a</foo>"""
- self.assertSoupEquals(text)
-
- def test_attribute_values_with_double_nested_quotes_get_quoted(self):
- text = """<foo attr='bar "brawls" happen'>a</foo>"""
- soup = self.soup(text)
- soup.foo['attr'] = 'Brawls happen at "Bob\'s Bar"'
- self.assertSoupEquals(
- soup.foo.decode(),
- """<foo attr="Brawls happen at "Bob\'s Bar"">a</foo>""")
-
- def test_ampersand_in_attribute_value_gets_escaped(self):
- self.assertSoupEquals('<this is="really messed up & stuff"></this>',
- '<this is="really messed up & stuff"></this>')
-
- self.assertSoupEquals(
- '<a href="http://example.org?a=1&b=2;3">foo</a>',
- '<a href="http://example.org?a=1&b=2;3">foo</a>')
-
- def test_escaped_ampersand_in_attribute_value_is_left_alone(self):
- self.assertSoupEquals('<a href="http://example.org?a=1&b=2;3"></a>')
-
- def test_entities_in_strings_converted_during_parsing(self):
- # Both XML and HTML entities are converted to Unicode characters
- # during parsing.
- text = "<p><<sacré bleu!>></p>"
- expected = "<p><<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></p>"
- self.assertSoupEquals(text, expected)
-
- def test_smart_quotes_converted_on_the_way_in(self):
- # Microsoft smart quotes are converted to Unicode characters during
- # parsing.
- quote = b"<p>\x91Foo\x92</p>"
- soup = self.soup(quote)
- self.assertEqual(
- soup.p.string,
- "\N{LEFT SINGLE QUOTATION MARK}Foo\N{RIGHT SINGLE QUOTATION MARK}")
-
- def test_non_breaking_spaces_converted_on_the_way_in(self):
- soup = self.soup("<a> </a>")
- self.assertEqual(soup.a.string, "\N{NO-BREAK SPACE}" * 2)
-
- def test_entities_converted_on_the_way_out(self):
- text = "<p><<sacré bleu!>></p>"
- expected = "<p><<sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></p>".encode("utf-8")
- soup = self.soup(text)
- self.assertEqual(soup.p.encode("utf-8"), expected)
-
- def test_real_iso_latin_document(self):
- # Smoke test of interrelated functionality, using an
- # easy-to-understand document.
-
- # Here it is in Unicode. Note that it claims to be in ISO-Latin-1.
- unicode_html = '<html><head><meta content="text/html; charset=ISO-Latin-1" http-equiv="Content-type"/></head><body><p>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</p></body></html>'
-
- # That's because we're going to encode it into ISO-Latin-1, and use
- # that to test.
- iso_latin_html = unicode_html.encode("iso-8859-1")
-
- # Parse the ISO-Latin-1 HTML.
- soup = self.soup(iso_latin_html)
- # Encode it to UTF-8.
- result = soup.encode("utf-8")
-
- # What do we expect the result to look like? Well, it would
- # look like unicode_html, except that the META tag would say
- # UTF-8 instead of ISO-Latin-1.
- expected = unicode_html.replace("ISO-Latin-1", "utf-8")
-
- # And, of course, it would be in UTF-8, not Unicode.
- expected = expected.encode("utf-8")
-
- # Ta-da!
- self.assertEqual(result, expected)
-
- def test_real_shift_jis_document(self):
- # Smoke test to make sure the parser can handle a document in
- # Shift-JIS encoding, without choking.
- shift_jis_html = (
- b'<html><head></head><body><pre>'
- b'\x82\xb1\x82\xea\x82\xcdShift-JIS\x82\xc5\x83R\x81[\x83f'
- b'\x83B\x83\x93\x83O\x82\xb3\x82\xea\x82\xbd\x93\xfa\x96{\x8c'
- b'\xea\x82\xcc\x83t\x83@\x83C\x83\x8b\x82\xc5\x82\xb7\x81B'
- b'</pre></body></html>')
- unicode_html = shift_jis_html.decode("shift-jis")
- soup = self.soup(unicode_html)
-
- # Make sure the parse tree is correctly encoded to various
- # encodings.
- self.assertEqual(soup.encode("utf-8"), unicode_html.encode("utf-8"))
- self.assertEqual(soup.encode("euc_jp"), unicode_html.encode("euc_jp"))
-
- def test_real_hebrew_document(self):
- # A real-world test to make sure we can convert ISO-8859-9 (a
- # Hebrew encoding) to UTF-8.
- hebrew_document = b'<html><head><title>Hebrew (ISO 8859-8) in Visual Directionality</title></head><body><h1>Hebrew (ISO 8859-8) in Visual Directionality</h1>\xed\xe5\xec\xf9</body></html>'
- soup = self.soup(
- hebrew_document, from_encoding="iso8859-8")
- self.assertEqual(soup.original_encoding, 'iso8859-8')
- self.assertEqual(
- soup.encode('utf-8'),
- hebrew_document.decode("iso8859-8").encode("utf-8"))
-
- def test_meta_tag_reflects_current_encoding(self):
- # Here's the <meta> tag saying that a document is
- # encoded in Shift-JIS.
- meta_tag = ('<meta content="text/html; charset=x-sjis" '
- 'http-equiv="Content-type"/>')
-
- # Here's a document incorporating that meta tag.
- shift_jis_html = (
- '<html><head>\n%s\n'
- '<meta http-equiv="Content-language" content="ja"/>'
- '</head><body>Shift-JIS markup goes here.') % meta_tag
- soup = self.soup(shift_jis_html)
-
- # Parse the document, and the charset is seemingly unaffected.
- parsed_meta = soup.find('meta', {'http-equiv': 'Content-type'})
- content = parsed_meta['content']
- self.assertEqual('text/html; charset=x-sjis', content)
-
- # But that value is actually a ContentMetaAttributeValue object.
- self.assertTrue(isinstance(content, ContentMetaAttributeValue))
-
- # And it will take on a value that reflects its current
- # encoding.
- self.assertEqual('text/html; charset=utf8', content.encode("utf8"))
-
- # For the rest of the story, see TestSubstitutions in
- # test_tree.py.
-
- def test_html5_style_meta_tag_reflects_current_encoding(self):
- # Here's the <meta> tag saying that a document is
- # encoded in Shift-JIS.
- meta_tag = ('<meta id="encoding" charset="x-sjis" />')
-
- # Here's a document incorporating that meta tag.
- shift_jis_html = (
- '<html><head>\n%s\n'
- '<meta http-equiv="Content-language" content="ja"/>'
- '</head><body>Shift-JIS markup goes here.') % meta_tag
- soup = self.soup(shift_jis_html)
-
- # Parse the document, and the charset is seemingly unaffected.
- parsed_meta = soup.find('meta', id="encoding")
- charset = parsed_meta['charset']
- self.assertEqual('x-sjis', charset)
-
- # But that value is actually a CharsetMetaAttributeValue object.
- self.assertTrue(isinstance(charset, CharsetMetaAttributeValue))
-
- # And it will take on a value that reflects its current
- # encoding.
- self.assertEqual('utf8', charset.encode("utf8"))
-
- def test_tag_with_no_attributes_can_have_attributes_added(self):
- data = self.soup("<a>text</a>")
- data.a['foo'] = 'bar'
- self.assertEqual('<a foo="bar">text</a>', data.a.decode())
-
-class XMLTreeBuilderSmokeTest(SoupTest):
-
- def test_pickle_and_unpickle_identity(self):
- # Pickling a tree, then unpickling it, yields a tree identical
- # to the original.
- tree = self.soup("<a><b>foo</a>")
- dumped = pickle.dumps(tree, 2)
- loaded = pickle.loads(dumped)
- self.assertEqual(loaded.__class__, BeautifulSoup)
- self.assertEqual(loaded.decode(), tree.decode())
-
- def test_docstring_generated(self):
- soup = self.soup("<root/>")
- self.assertEqual(
- soup.encode(), b'<?xml version="1.0" encoding="utf-8"?>\n<root/>')
-
- def test_xml_declaration(self):
- markup = b"""<?xml version="1.0" encoding="utf8"?>\n<foo/>"""
- soup = self.soup(markup)
- self.assertEqual(markup, soup.encode("utf8"))
-
- def test_real_xhtml_document(self):
- """A real XHTML document should come out *exactly* the same as it went in."""
- markup = b"""<?xml version="1.0" encoding="utf-8"?>
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head><title>Hello.</title></head>
-<body>Goodbye.</body>
-</html>"""
- soup = self.soup(markup)
- self.assertEqual(
- soup.encode("utf-8"), markup)
-
- def test_formatter_processes_script_tag_for_xml_documents(self):
- doc = """
- <script type="text/javascript">
- </script>
-"""
- soup = BeautifulSoup(doc, "lxml-xml")
- # lxml would have stripped this while parsing, but we can add
- # it later.
- soup.script.string = 'console.log("< < hey > > ");'
- encoded = soup.encode()
- self.assertTrue(b"< < hey > >" in encoded)
-
- def test_can_parse_unicode_document(self):
- markup = '<?xml version="1.0" encoding="euc-jp"><root>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</root>'
- soup = self.soup(markup)
- self.assertEqual('Sacr\xe9 bleu!', soup.root.string)
-
- def test_popping_namespaced_tag(self):
- markup = '<rss xmlns:dc="foo"><dc:creator>b</dc:creator><dc:date>2012-07-02T20:33:42Z</dc:date><dc:rights>c</dc:rights><image>d</image></rss>'
- soup = self.soup(markup)
- self.assertEqual(
- str(soup.rss), markup)
-
- def test_docstring_includes_correct_encoding(self):
- soup = self.soup("<root/>")
- self.assertEqual(
- soup.encode("latin1"),
- b'<?xml version="1.0" encoding="latin1"?>\n<root/>')
-
- def test_large_xml_document(self):
- """A large XML document should come out the same as it went in."""
- markup = (b'<?xml version="1.0" encoding="utf-8"?>\n<root>'
- + b'0' * (2**12)
- + b'</root>')
- soup = self.soup(markup)
- self.assertEqual(soup.encode("utf-8"), markup)
-
-
- def test_tags_are_empty_element_if_and_only_if_they_are_empty(self):
- self.assertSoupEquals("<p>", "<p/>")
- self.assertSoupEquals("<p>foo</p>")
-
- def test_namespaces_are_preserved(self):
- markup = '<root xmlns:a="http://example.com/" xmlns:b="http://example.net/"><a:foo>This tag is in the a namespace</a:foo><b:foo>This tag is in the b namespace</b:foo></root>'
- soup = self.soup(markup)
- root = soup.root
- self.assertEqual("http://example.com/", root['xmlns:a'])
- self.assertEqual("http://example.net/", root['xmlns:b'])
-
- def test_closing_namespaced_tag(self):
- markup = '<p xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>20010504</dc:date></p>'
- soup = self.soup(markup)
- self.assertEqual(str(soup.p), markup)
-
- def test_namespaced_attributes(self):
- markup = '<foo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><bar xsi:schemaLocation="http://www.example.com"/></foo>'
- soup = self.soup(markup)
- self.assertEqual(str(soup.foo), markup)
-
- def test_namespaced_attributes_xml_namespace(self):
- markup = '<foo xml:lang="fr">bar</foo>'
- soup = self.soup(markup)
- self.assertEqual(str(soup.foo), markup)
-
-class HTML5TreeBuilderSmokeTest(HTMLTreeBuilderSmokeTest):
- """Smoke test for a tree builder that supports HTML5."""
-
- def test_real_xhtml_document(self):
- # Since XHTML is not HTML5, HTML5 parsers are not tested to handle
- # XHTML documents in any particular way.
- pass
-
- def test_html_tags_have_namespace(self):
- markup = "<a>"
- soup = self.soup(markup)
- self.assertEqual("http://www.w3.org/1999/xhtml", soup.a.namespace)
-
- def test_svg_tags_have_namespace(self):
- markup = '<svg><circle/></svg>'
- soup = self.soup(markup)
- namespace = "http://www.w3.org/2000/svg"
- self.assertEqual(namespace, soup.svg.namespace)
- self.assertEqual(namespace, soup.circle.namespace)
-
-
- def test_mathml_tags_have_namespace(self):
- markup = '<math><msqrt>5</msqrt></math>'
- soup = self.soup(markup)
- namespace = 'http://www.w3.org/1998/Math/MathML'
- self.assertEqual(namespace, soup.math.namespace)
- self.assertEqual(namespace, soup.msqrt.namespace)
-
- def test_xml_declaration_becomes_comment(self):
- markup = '<?xml version="1.0" encoding="utf-8"?><html></html>'
- soup = self.soup(markup)
- self.assertTrue(isinstance(soup.contents[0], Comment))
- self.assertEqual(soup.contents[0], '?xml version="1.0" encoding="utf-8"?')
- self.assertEqual("html", soup.contents[0].next_element.name)
-
-def skipIf(condition, reason):
- def nothing(test, *args, **kwargs):
- return None
-
- def decorator(test_item):
- if condition:
- return nothing
- else:
- return test_item
-
- return decorator
diff --git a/bitbake/lib/bs4/tests/__init__.py b/bitbake/lib/bs4/tests/__init__.py
deleted file mode 100644
index 142c8cc3..00000000
--- a/bitbake/lib/bs4/tests/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-"The beautifulsoup tests."
diff --git a/bitbake/lib/bs4/tests/test_builder_registry.py b/bitbake/lib/bs4/tests/test_builder_registry.py
deleted file mode 100644
index 90cad829..00000000
--- a/bitbake/lib/bs4/tests/test_builder_registry.py
+++ /dev/null
@@ -1,147 +0,0 @@
-"""Tests of the builder registry."""
-
-import unittest
-import warnings
-
-from bs4 import BeautifulSoup
-from bs4.builder import (
- builder_registry as registry,
- HTMLParserTreeBuilder,
- TreeBuilderRegistry,
-)
-
-try:
- from bs4.builder import HTML5TreeBuilder
- HTML5LIB_PRESENT = True
-except ImportError:
- HTML5LIB_PRESENT = False
-
-try:
- from bs4.builder import (
- LXMLTreeBuilderForXML,
- LXMLTreeBuilder,
- )
- LXML_PRESENT = True
-except ImportError:
- LXML_PRESENT = False
-
-
-class BuiltInRegistryTest(unittest.TestCase):
- """Test the built-in registry with the default builders registered."""
-
- def test_combination(self):
- if LXML_PRESENT:
- self.assertEqual(registry.lookup('fast', 'html'),
- LXMLTreeBuilder)
-
- if LXML_PRESENT:
- self.assertEqual(registry.lookup('permissive', 'xml'),
- LXMLTreeBuilderForXML)
- self.assertEqual(registry.lookup('strict', 'html'),
- HTMLParserTreeBuilder)
- if HTML5LIB_PRESENT:
- self.assertEqual(registry.lookup('html5lib', 'html'),
- HTML5TreeBuilder)
-
- def test_lookup_by_markup_type(self):
- if LXML_PRESENT:
- self.assertEqual(registry.lookup('html'), LXMLTreeBuilder)
- self.assertEqual(registry.lookup('xml'), LXMLTreeBuilderForXML)
- else:
- self.assertEqual(registry.lookup('xml'), None)
- if HTML5LIB_PRESENT:
- self.assertEqual(registry.lookup('html'), HTML5TreeBuilder)
- else:
- self.assertEqual(registry.lookup('html'), HTMLParserTreeBuilder)
-
- def test_named_library(self):
- if LXML_PRESENT:
- self.assertEqual(registry.lookup('lxml', 'xml'),
- LXMLTreeBuilderForXML)
- self.assertEqual(registry.lookup('lxml', 'html'),
- LXMLTreeBuilder)
- if HTML5LIB_PRESENT:
- self.assertEqual(registry.lookup('html5lib'),
- HTML5TreeBuilder)
-
- self.assertEqual(registry.lookup('html.parser'),
- HTMLParserTreeBuilder)
-
- def test_beautifulsoup_constructor_does_lookup(self):
-
- with warnings.catch_warnings(record=True) as w:
- # This will create a warning about not explicitly
- # specifying a parser, but we'll ignore it.
-
- # You can pass in a string.
- BeautifulSoup("", features="html")
- # Or a list of strings.
- BeautifulSoup("", features=["html", "fast"])
-
- # You'll get an exception if BS can't find an appropriate
- # builder.
- self.assertRaises(ValueError, BeautifulSoup,
- "", features="no-such-feature")
-
-class RegistryTest(unittest.TestCase):
- """Test the TreeBuilderRegistry class in general."""
-
- def setUp(self):
- self.registry = TreeBuilderRegistry()
-
- def builder_for_features(self, *feature_list):
- cls = type('Builder_' + '_'.join(feature_list),
- (object,), {'features' : feature_list})
-
- self.registry.register(cls)
- return cls
-
- def test_register_with_no_features(self):
- builder = self.builder_for_features()
-
- # Since the builder advertises no features, you can't find it
- # by looking up features.
- self.assertEqual(self.registry.lookup('foo'), None)
-
- # But you can find it by doing a lookup with no features, if
- # this happens to be the only registered builder.
- self.assertEqual(self.registry.lookup(), builder)
-
- def test_register_with_features_makes_lookup_succeed(self):
- builder = self.builder_for_features('foo', 'bar')
- self.assertEqual(self.registry.lookup('foo'), builder)
- self.assertEqual(self.registry.lookup('bar'), builder)
-
- def test_lookup_fails_when_no_builder_implements_feature(self):
- builder = self.builder_for_features('foo', 'bar')
- self.assertEqual(self.registry.lookup('baz'), None)
-
- def test_lookup_gets_most_recent_registration_when_no_feature_specified(self):
- builder1 = self.builder_for_features('foo')
- builder2 = self.builder_for_features('bar')
- self.assertEqual(self.registry.lookup(), builder2)
-
- def test_lookup_fails_when_no_tree_builders_registered(self):
- self.assertEqual(self.registry.lookup(), None)
-
- def test_lookup_gets_most_recent_builder_supporting_all_features(self):
- has_one = self.builder_for_features('foo')
- has_the_other = self.builder_for_features('bar')
- has_both_early = self.builder_for_features('foo', 'bar', 'baz')
- has_both_late = self.builder_for_features('foo', 'bar', 'quux')
- lacks_one = self.builder_for_features('bar')
- has_the_other = self.builder_for_features('foo')
-
- # There are two builders featuring 'foo' and 'bar', but
- # the one that also features 'quux' was registered later.
- self.assertEqual(self.registry.lookup('foo', 'bar'),
- has_both_late)
-
- # There is only one builder featuring 'foo', 'bar', and 'baz'.
- self.assertEqual(self.registry.lookup('foo', 'bar', 'baz'),
- has_both_early)
-
- def test_lookup_fails_when_cannot_reconcile_requested_features(self):
- builder1 = self.builder_for_features('foo', 'bar')
- builder2 = self.builder_for_features('foo', 'baz')
- self.assertEqual(self.registry.lookup('bar', 'baz'), None)
diff --git a/bitbake/lib/bs4/tests/test_docs.py b/bitbake/lib/bs4/tests/test_docs.py
deleted file mode 100644
index d1d76a33..00000000
--- a/bitbake/lib/bs4/tests/test_docs.py
+++ /dev/null
@@ -1,32 +0,0 @@
-"Test harness for doctests."
-
-# pylint: disable-msg=E0611,W0142
-
-__metaclass__ = type
-__all__ = [
- 'additional_tests',
- ]
-
-import doctest
-#from pkg_resources import (
-# resource_filename, resource_exists, resource_listdir, cleanup_resources)
-
-DOCTEST_FLAGS = (
- doctest.ELLIPSIS |
- doctest.NORMALIZE_WHITESPACE |
- doctest.REPORT_NDIFF)
-
-# def additional_tests():
-# "Run the doc tests (README.txt and docs/*, if any exist)"
-# doctest_files = [
-# os.path.abspath(resource_filename('bs4', 'README.txt'))]
-# if resource_exists('bs4', 'docs'):
-# for name in resource_listdir('bs4', 'docs'):
-# if name.endswith('.txt'):
-# doctest_files.append(
-# os.path.abspath(
-# resource_filename('bs4', 'docs/%s' % name)))
-# kwargs = dict(module_relative=False, optionflags=DOCTEST_FLAGS)
-# atexit.register(cleanup_resources)
-# return unittest.TestSuite((
-# doctest.DocFileSuite(*doctest_files, **kwargs)))
diff --git a/bitbake/lib/bs4/tests/test_html5lib.py b/bitbake/lib/bs4/tests/test_html5lib.py
deleted file mode 100644
index a7494ca5..00000000
--- a/bitbake/lib/bs4/tests/test_html5lib.py
+++ /dev/null
@@ -1,98 +0,0 @@
-"""Tests to ensure that the html5lib tree builder generates good trees."""
-
-import warnings
-
-try:
- from bs4.builder import HTML5TreeBuilder
- HTML5LIB_PRESENT = True
-except ImportError as e:
- HTML5LIB_PRESENT = False
-from bs4.element import SoupStrainer
-from bs4.testing import (
- HTML5TreeBuilderSmokeTest,
- SoupTest,
- skipIf,
-)
-
-@skipIf(
- not HTML5LIB_PRESENT,
- "html5lib seems not to be present, not testing its tree builder.")
-class HTML5LibBuilderSmokeTest(SoupTest, HTML5TreeBuilderSmokeTest):
- """See ``HTML5TreeBuilderSmokeTest``."""
-
- @property
- def default_builder(self):
- return HTML5TreeBuilder()
-
- def test_soupstrainer(self):
- # The html5lib tree builder does not support SoupStrainers.
- strainer = SoupStrainer("b")
- markup = "<p>A <b>bold</b> statement.</p>"
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup(markup, parse_only=strainer)
- self.assertEqual(
- soup.decode(), self.document_for(markup))
-
- self.assertTrue(
- "the html5lib tree builder doesn't support parse_only" in
- str(w[0].message))
-
- def test_correctly_nested_tables(self):
- """html5lib inserts <tbody> tags where other parsers don't."""
- markup = ('<table id="1">'
- '<tr>'
- "<td>Here's another table:"
- '<table id="2">'
- '<tr><td>foo</td></tr>'
- '</table></td>')
-
- self.assertSoupEquals(
- markup,
- '<table id="1"><tbody><tr><td>Here\'s another table:'
- '<table id="2"><tbody><tr><td>foo</td></tr></tbody></table>'
- '</td></tr></tbody></table>')
-
- self.assertSoupEquals(
- "<table><thead><tr><td>Foo</td></tr></thead>"
- "<tbody><tr><td>Bar</td></tr></tbody>"
- "<tfoot><tr><td>Baz</td></tr></tfoot></table>")
-
- def test_xml_declaration_followed_by_doctype(self):
- markup = '''<?xml version="1.0" encoding="utf-8"?>
-<!DOCTYPE html>
-<html>
- <head>
- </head>
- <body>
- <p>foo</p>
- </body>
-</html>'''
- soup = self.soup(markup)
- # Verify that we can reach the <p> tag; this means the tree is connected.
- self.assertEqual(b"<p>foo</p>", soup.p.encode())
-
- def test_reparented_markup(self):
- markup = '<p><em>foo</p>\n<p>bar<a></a></em></p>'
- soup = self.soup(markup)
- self.assertEqual("<body><p><em>foo</em></p><em>\n</em><p><em>bar<a></a></em></p></body>", soup.body.decode())
- self.assertEqual(2, len(soup.find_all('p')))
-
-
- def test_reparented_markup_ends_with_whitespace(self):
- markup = '<p><em>foo</p>\n<p>bar<a></a></em></p>\n'
- soup = self.soup(markup)
- self.assertEqual("<body><p><em>foo</em></p><em>\n</em><p><em>bar<a></a></em></p>\n</body>", soup.body.decode())
- self.assertEqual(2, len(soup.find_all('p')))
-
- def test_processing_instruction(self):
- """Processing instructions become comments."""
- markup = b"""<?PITarget PIContent?>"""
- soup = self.soup(markup)
- assert str(soup).startswith("<!--?PITarget PIContent?-->")
-
- def test_cloned_multivalue_node(self):
- markup = b"""<a class="my_class"><p></a>"""
- soup = self.soup(markup)
- a1, a2 = soup.find_all('a')
- self.assertEqual(a1, a2)
- assert a1 is not a2
diff --git a/bitbake/lib/bs4/tests/test_htmlparser.py b/bitbake/lib/bs4/tests/test_htmlparser.py
deleted file mode 100644
index 30a25e67..00000000
--- a/bitbake/lib/bs4/tests/test_htmlparser.py
+++ /dev/null
@@ -1,31 +0,0 @@
-"""Tests to ensure that the html.parser tree builder generates good
-trees."""
-
-import pickle
-from bs4.testing import SoupTest, HTMLTreeBuilderSmokeTest
-from bs4.builder import HTMLParserTreeBuilder
-
-class HTMLParserTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest):
-
- @property
- def default_builder(self):
- return HTMLParserTreeBuilder()
-
- def test_namespaced_system_doctype(self):
- # html.parser can't handle namespaced doctypes, so skip this one.
- pass
-
- def test_namespaced_public_doctype(self):
- # html.parser can't handle namespaced doctypes, so skip this one.
- pass
-
- def test_builder_is_pickled(self):
- """Unlike most tree builders, HTMLParserTreeBuilder and will
- be restored after pickling.
- """
- tree = self.soup("<a><b>foo</a>")
- dumped = pickle.dumps(tree, 2)
- loaded = pickle.loads(dumped)
- self.assertTrue(isinstance(loaded.builder, type(tree.builder)))
-
-
diff --git a/bitbake/lib/bs4/tests/test_lxml.py b/bitbake/lib/bs4/tests/test_lxml.py
deleted file mode 100644
index 6b6cdd07..00000000
--- a/bitbake/lib/bs4/tests/test_lxml.py
+++ /dev/null
@@ -1,70 +0,0 @@
-"""Tests to ensure that the lxml tree builder generates good trees."""
-
-import warnings
-
-try:
- import lxml.etree
- LXML_PRESENT = True
- LXML_VERSION = lxml.etree.LXML_VERSION
-except ImportError as e:
- LXML_PRESENT = False
- LXML_VERSION = (0,)
-
-if LXML_PRESENT:
- from bs4.builder import LXMLTreeBuilder, LXMLTreeBuilderForXML
-
-from bs4 import BeautifulStoneSoup
-from bs4.testing import skipIf
-from bs4.testing import (
- HTMLTreeBuilderSmokeTest,
- XMLTreeBuilderSmokeTest,
- SoupTest,
- skipIf,
-)
-
-@skipIf(
- not LXML_PRESENT,
- "lxml seems not to be present, not testing its tree builder.")
-class LXMLTreeBuilderSmokeTest(SoupTest, HTMLTreeBuilderSmokeTest):
- """See ``HTMLTreeBuilderSmokeTest``."""
-
- @property
- def default_builder(self):
- return LXMLTreeBuilder()
-
- def test_out_of_range_entity(self):
- self.assertSoupEquals(
- "<p>foo�bar</p>", "<p>foobar</p>")
- self.assertSoupEquals(
- "<p>foo�bar</p>", "<p>foobar</p>")
- self.assertSoupEquals(
- "<p>foo�bar</p>", "<p>foobar</p>")
-
- # In lxml < 2.3.5, an empty doctype causes a segfault. Skip this
- # test if an old version of lxml is installed.
-
- @skipIf(
- not LXML_PRESENT or LXML_VERSION < (2,3,5,0),
- "Skipping doctype test for old version of lxml to avoid segfault.")
- def test_empty_doctype(self):
- soup = self.soup("<!DOCTYPE>")
- doctype = soup.contents[0]
- self.assertEqual("", doctype.strip())
-
- def test_beautifulstonesoup_is_xml_parser(self):
- # Make sure that the deprecated BSS class uses an xml builder
- # if one is installed.
- with warnings.catch_warnings(record=True) as w:
- soup = BeautifulStoneSoup("<b />")
- self.assertEqual("<b/>", str(soup.b))
- self.assertTrue("BeautifulStoneSoup class is deprecated" in str(w[0].message))
-
-@skipIf(
- not LXML_PRESENT,
- "lxml seems not to be present, not testing its XML tree builder.")
-class LXMLXMLTreeBuilderSmokeTest(SoupTest, XMLTreeBuilderSmokeTest):
- """See ``HTMLTreeBuilderSmokeTest``."""
-
- @property
- def default_builder(self):
- return LXMLTreeBuilderForXML()
diff --git a/bitbake/lib/bs4/tests/test_soup.py b/bitbake/lib/bs4/tests/test_soup.py
deleted file mode 100644
index 6ad3cb37..00000000
--- a/bitbake/lib/bs4/tests/test_soup.py
+++ /dev/null
@@ -1,479 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Tests of Beautiful Soup as a whole."""
-
-import logging
-import unittest
-import sys
-import tempfile
-
-from bs4 import BeautifulSoup
-from bs4.element import (
- CharsetMetaAttributeValue,
- ContentMetaAttributeValue,
- SoupStrainer,
- NamespacedAttribute,
- )
-import bs4.dammit
-from bs4.dammit import (
- EntitySubstitution,
- UnicodeDammit,
- EncodingDetector,
-)
-from bs4.testing import (
- SoupTest,
- skipIf,
-)
-import warnings
-
-try:
- from bs4.builder import LXMLTreeBuilder, LXMLTreeBuilderForXML
- LXML_PRESENT = True
-except ImportError as e:
- LXML_PRESENT = False
-
-PYTHON_2_PRE_2_7 = (sys.version_info < (2,7))
-PYTHON_3_PRE_3_2 = (sys.version_info[0] == 3 and sys.version_info < (3,2))
-
-class TestConstructor(SoupTest):
-
- def test_short_unicode_input(self):
- data = "<h1>éé</h1>"
- soup = self.soup(data)
- self.assertEqual("éé", soup.h1.string)
-
- def test_embedded_null(self):
- data = "<h1>foo\0bar</h1>"
- soup = self.soup(data)
- self.assertEqual("foo\0bar", soup.h1.string)
-
- def test_exclude_encodings(self):
- utf8_data = "Räksmörgås".encode("utf-8")
- soup = self.soup(utf8_data, exclude_encodings=["utf-8"])
- self.assertEqual("windows-1252", soup.original_encoding)
-
-
-class TestWarnings(SoupTest):
-
- def _no_parser_specified(self, s, is_there=True):
- v = s.startswith(BeautifulSoup.NO_PARSER_SPECIFIED_WARNING[:80])
- self.assertTrue(v)
-
- def test_warning_if_no_parser_specified(self):
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("<a><b></b></a>")
- msg = str(w[0].message)
- self._assert_no_parser_specified(msg)
-
- def test_warning_if_parser_specified_too_vague(self):
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("<a><b></b></a>", "html")
- msg = str(w[0].message)
- self._assert_no_parser_specified(msg)
-
- def test_no_warning_if_explicit_parser_specified(self):
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("<a><b></b></a>", "html.parser")
- self.assertEqual([], w)
-
- def test_parseOnlyThese_renamed_to_parse_only(self):
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("<a><b></b></a>", parseOnlyThese=SoupStrainer("b"))
- msg = str(w[0].message)
- self.assertTrue("parseOnlyThese" in msg)
- self.assertTrue("parse_only" in msg)
- self.assertEqual(b"<b></b>", soup.encode())
-
- def test_fromEncoding_renamed_to_from_encoding(self):
- with warnings.catch_warnings(record=True) as w:
- utf8 = b"\xc3\xa9"
- soup = self.soup(utf8, fromEncoding="utf8")
- msg = str(w[0].message)
- self.assertTrue("fromEncoding" in msg)
- self.assertTrue("from_encoding" in msg)
- self.assertEqual("utf8", soup.original_encoding)
-
- def test_unrecognized_keyword_argument(self):
- self.assertRaises(
- TypeError, self.soup, "<a>", no_such_argument=True)
-
-class TestWarnings(SoupTest):
-
- def test_disk_file_warning(self):
- filehandle = tempfile.NamedTemporaryFile()
- filename = filehandle.name
- try:
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup(filename)
- msg = str(w[0].message)
- self.assertTrue("looks like a filename" in msg)
- finally:
- filehandle.close()
-
- # The file no longer exists, so Beautiful Soup will no longer issue the warning.
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup(filename)
- self.assertEqual(0, len(w))
-
- def test_url_warning(self):
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("http://www.crummy.com/")
- msg = str(w[0].message)
- self.assertTrue("looks like a URL" in msg)
-
- with warnings.catch_warnings(record=True) as w:
- soup = self.soup("http://www.crummy.com/ is great")
- self.assertEqual(0, len(w))
-
-class TestSelectiveParsing(SoupTest):
-
- def test_parse_with_soupstrainer(self):
- markup = "No<b>Yes</b><a>No<b>Yes <c>Yes</c></b>"
- strainer = SoupStrainer("b")
- soup = self.soup(markup, parse_only=strainer)
- self.assertEqual(soup.encode(), b"<b>Yes</b><b>Yes <c>Yes</c></b>")
-
-
-class TestEntitySubstitution(unittest.TestCase):
- """Standalone tests of the EntitySubstitution class."""
- def setUp(self):
- self.sub = EntitySubstitution
-
- def test_simple_html_substitution(self):
- # Unicode characters corresponding to named HTML entites
- # are substituted, and no others.
- s = "foo\u2200\N{SNOWMAN}\u00f5bar"
- self.assertEqual(self.sub.substitute_html(s),
- "foo∀\N{SNOWMAN}õbar")
-
- def test_smart_quote_substitution(self):
- # MS smart quotes are a common source of frustration, so we
- # give them a special test.
- quotes = b"\x91\x92foo\x93\x94"
- dammit = UnicodeDammit(quotes)
- self.assertEqual(self.sub.substitute_html(dammit.markup),
- "‘’foo“”")
-
- def test_xml_converstion_includes_no_quotes_if_make_quoted_attribute_is_false(self):
- s = 'Welcome to "my bar"'
- self.assertEqual(self.sub.substitute_xml(s, False), s)
-
- def test_xml_attribute_quoting_normally_uses_double_quotes(self):
- self.assertEqual(self.sub.substitute_xml("Welcome", True),
- '"Welcome"')
- self.assertEqual(self.sub.substitute_xml("Bob's Bar", True),
- '"Bob\'s Bar"')
-
- def test_xml_attribute_quoting_uses_single_quotes_when_value_contains_double_quotes(self):
- s = 'Welcome to "my bar"'
- self.assertEqual(self.sub.substitute_xml(s, True),
- "'Welcome to \"my bar\"'")
-
- def test_xml_attribute_quoting_escapes_single_quotes_when_value_contains_both_single_and_double_quotes(self):
- s = 'Welcome to "Bob\'s Bar"'
- self.assertEqual(
- self.sub.substitute_xml(s, True),
- '"Welcome to "Bob\'s Bar""')
-
- def test_xml_quotes_arent_escaped_when_value_is_not_being_quoted(self):
- quoted = 'Welcome to "Bob\'s Bar"'
- self.assertEqual(self.sub.substitute_xml(quoted), quoted)
-
- def test_xml_quoting_handles_angle_brackets(self):
- self.assertEqual(
- self.sub.substitute_xml("foo<bar>"),
- "foo<bar>")
-
- def test_xml_quoting_handles_ampersands(self):
- self.assertEqual(self.sub.substitute_xml("AT&T"), "AT&T")
-
- def test_xml_quoting_including_ampersands_when_they_are_part_of_an_entity(self):
- self.assertEqual(
- self.sub.substitute_xml("ÁT&T"),
- "&Aacute;T&T")
-
- def test_xml_quoting_ignoring_ampersands_when_they_are_part_of_an_entity(self):
- self.assertEqual(
- self.sub.substitute_xml_containing_entities("ÁT&T"),
- "ÁT&T")
-
- def test_quotes_not_html_substituted(self):
- """There's no need to do this except inside attribute values."""
- text = 'Bob\'s "bar"'
- self.assertEqual(self.sub.substitute_html(text), text)
-
-
-class TestEncodingConversion(SoupTest):
- # Test Beautiful Soup's ability to decode and encode from various
- # encodings.
-
- def setUp(self):
- super(TestEncodingConversion, self).setUp()
- self.unicode_data = '<html><head><meta charset="utf-8"/></head><body><foo>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</foo></body></html>'
- self.utf8_data = self.unicode_data.encode("utf-8")
- # Just so you know what it looks like.
- self.assertEqual(
- self.utf8_data,
- b'<html><head><meta charset="utf-8"/></head><body><foo>Sacr\xc3\xa9 bleu!</foo></body></html>')
-
- def test_ascii_in_unicode_out(self):
- # ASCII input is converted to Unicode. The original_encoding
- # attribute is set to 'utf-8', a superset of ASCII.
- chardet = bs4.dammit.chardet_dammit
- logging.disable(logging.WARNING)
- try:
- def noop(str):
- return None
- # Disable chardet, which will realize that the ASCII is ASCII.
- bs4.dammit.chardet_dammit = noop
- ascii = b"<foo>a</foo>"
- soup_from_ascii = self.soup(ascii)
- unicode_output = soup_from_ascii.decode()
- self.assertTrue(isinstance(unicode_output, str))
- self.assertEqual(unicode_output, self.document_for(ascii.decode()))
- self.assertEqual(soup_from_ascii.original_encoding.lower(), "utf-8")
- finally:
- logging.disable(logging.NOTSET)
- bs4.dammit.chardet_dammit = chardet
-
- def test_unicode_in_unicode_out(self):
- # Unicode input is left alone. The original_encoding attribute
- # is not set.
- soup_from_unicode = self.soup(self.unicode_data)
- self.assertEqual(soup_from_unicode.decode(), self.unicode_data)
- self.assertEqual(soup_from_unicode.foo.string, 'Sacr\xe9 bleu!')
- self.assertEqual(soup_from_unicode.original_encoding, None)
-
- def test_utf8_in_unicode_out(self):
- # UTF-8 input is converted to Unicode. The original_encoding
- # attribute is set.
- soup_from_utf8 = self.soup(self.utf8_data)
- self.assertEqual(soup_from_utf8.decode(), self.unicode_data)
- self.assertEqual(soup_from_utf8.foo.string, 'Sacr\xe9 bleu!')
-
- def test_utf8_out(self):
- # The internal data structures can be encoded as UTF-8.
- soup_from_unicode = self.soup(self.unicode_data)
- self.assertEqual(soup_from_unicode.encode('utf-8'), self.utf8_data)
-
- @skipIf(
- PYTHON_2_PRE_2_7 or PYTHON_3_PRE_3_2,
- "Bad HTMLParser detected; skipping test of non-ASCII characters in attribute name.")
- def test_attribute_name_containing_unicode_characters(self):
- markup = '<div><a \N{SNOWMAN}="snowman"></a></div>'
- self.assertEqual(self.soup(markup).div.encode("utf8"), markup.encode("utf8"))
-
-class TestUnicodeDammit(unittest.TestCase):
- """Standalone tests of UnicodeDammit."""
-
- def test_unicode_input(self):
- markup = "I'm already Unicode! \N{SNOWMAN}"
- dammit = UnicodeDammit(markup)
- self.assertEqual(dammit.unicode_markup, markup)
-
- def test_smart_quotes_to_unicode(self):
- markup = b"<foo>\x91\x92\x93\x94</foo>"
- dammit = UnicodeDammit(markup)
- self.assertEqual(
- dammit.unicode_markup, "<foo>\u2018\u2019\u201c\u201d</foo>")
-
- def test_smart_quotes_to_xml_entities(self):
- markup = b"<foo>\x91\x92\x93\x94</foo>"
- dammit = UnicodeDammit(markup, smart_quotes_to="xml")
- self.assertEqual(
- dammit.unicode_markup, "<foo>‘’“”</foo>")
-
- def test_smart_quotes_to_html_entities(self):
- markup = b"<foo>\x91\x92\x93\x94</foo>"
- dammit = UnicodeDammit(markup, smart_quotes_to="html")
- self.assertEqual(
- dammit.unicode_markup, "<foo>‘’“”</foo>")
-
- def test_smart_quotes_to_ascii(self):
- markup = b"<foo>\x91\x92\x93\x94</foo>"
- dammit = UnicodeDammit(markup, smart_quotes_to="ascii")
- self.assertEqual(
- dammit.unicode_markup, """<foo>''""</foo>""")
-
- def test_detect_utf8(self):
- utf8 = b"Sacr\xc3\xa9 bleu! \xe2\x98\x83"
- dammit = UnicodeDammit(utf8)
- self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
- self.assertEqual(dammit.unicode_markup, 'Sacr\xe9 bleu! \N{SNOWMAN}')
-
-
- def test_convert_hebrew(self):
- hebrew = b"\xed\xe5\xec\xf9"
- dammit = UnicodeDammit(hebrew, ["iso-8859-8"])
- self.assertEqual(dammit.original_encoding.lower(), 'iso-8859-8')
- self.assertEqual(dammit.unicode_markup, '\u05dd\u05d5\u05dc\u05e9')
-
- def test_dont_see_smart_quotes_where_there_are_none(self):
- utf_8 = b"\343\202\261\343\203\274\343\202\277\343\202\244 Watch"
- dammit = UnicodeDammit(utf_8)
- self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
- self.assertEqual(dammit.unicode_markup.encode("utf-8"), utf_8)
-
- def test_ignore_inappropriate_codecs(self):
- utf8_data = "Räksmörgås".encode("utf-8")
- dammit = UnicodeDammit(utf8_data, ["iso-8859-8"])
- self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
-
- def test_ignore_invalid_codecs(self):
- utf8_data = "Räksmörgås".encode("utf-8")
- for bad_encoding in ['.utf8', '...', 'utF---16.!']:
- dammit = UnicodeDammit(utf8_data, [bad_encoding])
- self.assertEqual(dammit.original_encoding.lower(), 'utf-8')
-
- def test_exclude_encodings(self):
- # This is UTF-8.
- utf8_data = "Räksmörgås".encode("utf-8")
-
- # But if we exclude UTF-8 from consideration, the guess is
- # Windows-1252.
- dammit = UnicodeDammit(utf8_data, exclude_encodings=["utf-8"])
- self.assertEqual(dammit.original_encoding.lower(), 'windows-1252')
-
- # And if we exclude that, there is no valid guess at all.
- dammit = UnicodeDammit(
- utf8_data, exclude_encodings=["utf-8", "windows-1252"])
- self.assertEqual(dammit.original_encoding, None)
-
- def test_encoding_detector_replaces_junk_in_encoding_name_with_replacement_character(self):
- detected = EncodingDetector(
- b'<?xml version="1.0" encoding="UTF-\xdb" ?>')
- encodings = list(detected.encodings)
- assert 'utf-\N{REPLACEMENT CHARACTER}' in encodings
-
- def test_detect_html5_style_meta_tag(self):
-
- for data in (
- b'<html><meta charset="euc-jp" /></html>',
- b"<html><meta charset='euc-jp' /></html>",
- b"<html><meta charset=euc-jp /></html>",
- b"<html><meta charset=euc-jp/></html>"):
- dammit = UnicodeDammit(data, is_html=True)
- self.assertEqual(
- "euc-jp", dammit.original_encoding)
-
- def test_last_ditch_entity_replacement(self):
- # This is a UTF-8 document that contains bytestrings
- # completely incompatible with UTF-8 (ie. encoded with some other
- # encoding).
- #
- # Since there is no consistent encoding for the document,
- # Unicode, Dammit will eventually encode the document as UTF-8
- # and encode the incompatible characters as REPLACEMENT
- # CHARACTER.
- #
- # If chardet is installed, it will detect that the document
- # can be converted into ISO-8859-1 without errors. This happens
- # to be the wrong encoding, but it is a consistent encoding, so the
- # code we're testing here won't run.
- #
- # So we temporarily disable chardet if it's present.
- doc = b"""\357\273\277<?xml version="1.0" encoding="UTF-8"?>
-<html><b>\330\250\330\252\330\261</b>
-<i>\310\322\321\220\312\321\355\344</i></html>"""
- chardet = bs4.dammit.chardet_dammit
- logging.disable(logging.WARNING)
- try:
- def noop(str):
- return None
- bs4.dammit.chardet_dammit = noop
- dammit = UnicodeDammit(doc)
- self.assertEqual(True, dammit.contains_replacement_characters)
- self.assertTrue("\ufffd" in dammit.unicode_markup)
-
- soup = BeautifulSoup(doc, "html.parser")
- self.assertTrue(soup.contains_replacement_characters)
- finally:
- logging.disable(logging.NOTSET)
- bs4.dammit.chardet_dammit = chardet
-
- def test_byte_order_mark_removed(self):
- # A document written in UTF-16LE will have its byte order marker stripped.
- data = b'\xff\xfe<\x00a\x00>\x00\xe1\x00\xe9\x00<\x00/\x00a\x00>\x00'
- dammit = UnicodeDammit(data)
- self.assertEqual("<a>áé</a>", dammit.unicode_markup)
- self.assertEqual("utf-16le", dammit.original_encoding)
-
- def test_detwingle(self):
- # Here's a UTF8 document.
- utf8 = ("\N{SNOWMAN}" * 3).encode("utf8")
-
- # Here's a Windows-1252 document.
- windows_1252 = (
- "\N{LEFT DOUBLE QUOTATION MARK}Hi, I like Windows!"
- "\N{RIGHT DOUBLE QUOTATION MARK}").encode("windows_1252")
-
- # Through some unholy alchemy, they've been stuck together.
- doc = utf8 + windows_1252 + utf8
-
- # The document can't be turned into UTF-8:
- self.assertRaises(UnicodeDecodeError, doc.decode, "utf8")
-
- # Unicode, Dammit thinks the whole document is Windows-1252,
- # and decodes it into "☃☃☃“Hi, I like Windows!”☃☃☃"
-
- # But if we run it through fix_embedded_windows_1252, it's fixed:
-
- fixed = UnicodeDammit.detwingle(doc)
- self.assertEqual(
- "☃☃☃“Hi, I like Windows!”☃☃☃", fixed.decode("utf8"))
-
- def test_detwingle_ignores_multibyte_characters(self):
- # Each of these characters has a UTF-8 representation ending
- # in \x93. \x93 is a smart quote if interpreted as
- # Windows-1252. But our code knows to skip over multibyte
- # UTF-8 characters, so they'll survive the process unscathed.
- for tricky_unicode_char in (
- "\N{LATIN SMALL LIGATURE OE}", # 2-byte char '\xc5\x93'
- "\N{LATIN SUBSCRIPT SMALL LETTER X}", # 3-byte char '\xe2\x82\x93'
- "\xf0\x90\x90\x93", # This is a CJK character, not sure which one.
- ):
- input = tricky_unicode_char.encode("utf8")
- self.assertTrue(input.endswith(b'\x93'))
- output = UnicodeDammit.detwingle(input)
- self.assertEqual(output, input)
-
-class TestNamedspacedAttribute(SoupTest):
-
- def test_name_may_be_none(self):
- a = NamespacedAttribute("xmlns", None)
- self.assertEqual(a, "xmlns")
-
- def test_attribute_is_equivalent_to_colon_separated_string(self):
- a = NamespacedAttribute("a", "b")
- self.assertEqual("a:b", a)
-
- def test_attributes_are_equivalent_if_prefix_and_name_identical(self):
- a = NamespacedAttribute("a", "b", "c")
- b = NamespacedAttribute("a", "b", "c")
- self.assertEqual(a, b)
-
- # The actual namespace is not considered.
- c = NamespacedAttribute("a", "b", None)
- self.assertEqual(a, c)
-
- # But name and prefix are important.
- d = NamespacedAttribute("a", "z", "c")
- self.assertNotEqual(a, d)
-
- e = NamespacedAttribute("z", "b", "c")
- self.assertNotEqual(a, e)
-
-
-class TestAttributeValueWithCharsetSubstitution(unittest.TestCase):
-
- def test_content_meta_attribute_value(self):
- value = CharsetMetaAttributeValue("euc-jp")
- self.assertEqual("euc-jp", value)
- self.assertEqual("euc-jp", value.original_value)
- self.assertEqual("utf8", value.encode("utf8"))
-
-
- def test_content_meta_attribute_value(self):
- value = ContentMetaAttributeValue("text/html; charset=euc-jp")
- self.assertEqual("text/html; charset=euc-jp", value)
- self.assertEqual("text/html; charset=euc-jp", value.original_value)
- self.assertEqual("text/html; charset=utf8", value.encode("utf8"))
diff --git a/bitbake/lib/bs4/tests/test_tree.py b/bitbake/lib/bs4/tests/test_tree.py
deleted file mode 100644
index cf0f1abe..00000000
--- a/bitbake/lib/bs4/tests/test_tree.py
+++ /dev/null
@@ -1,2004 +0,0 @@
-# -*- coding: utf-8 -*-
-"""Tests for Beautiful Soup's tree traversal methods.
-
-The tree traversal methods are the main advantage of using Beautiful
-Soup over just using a parser.
-
-Different parsers will build different Beautiful Soup trees given the
-same markup, but all Beautiful Soup trees can be traversed with the
-methods tested here.
-"""
-
-import copy
-import pickle
-import re
-import warnings
-from bs4 import BeautifulSoup
-from bs4.builder import builder_registry
-from bs4.element import (
- PY3K,
- CData,
- Comment,
- Declaration,
- Doctype,
- NavigableString,
- SoupStrainer,
- Tag,
-)
-from bs4.testing import SoupTest
-
-XML_BUILDER_PRESENT = (builder_registry.lookup("xml") is not None)
-LXML_PRESENT = (builder_registry.lookup("lxml") is not None)
-
-class TreeTest(SoupTest):
-
- def assertSelects(self, tags, should_match):
- """Make sure that the given tags have the correct text.
-
- This is used in tests that define a bunch of tags, each
- containing a single string, and then select certain strings by
- some mechanism.
- """
- self.assertEqual([tag.string for tag in tags], should_match)
-
- def assertSelectsIDs(self, tags, should_match):
- """Make sure that the given tags have the correct IDs.
-
- This is used in tests that define a bunch of tags, each
- containing a single string, and then select certain strings by
- some mechanism.
- """
- self.assertEqual([tag['id'] for tag in tags], should_match)
-
-
-class TestFind(TreeTest):
- """Basic tests of the find() method.
-
- find() just calls find_all() with limit=1, so it's not tested all
- that thouroughly here.
- """
-
- def test_find_tag(self):
- soup = self.soup("<a>1</a><b>2</b><a>3</a><b>4</b>")
- self.assertEqual(soup.find("b").string, "2")
-
- def test_unicode_text_find(self):
- soup = self.soup('<h1>Räksmörgås</h1>')
- self.assertEqual(soup.find(string='Räksmörgås'), 'Räksmörgås')
-
- def test_unicode_attribute_find(self):
- soup = self.soup('<h1 id="Räksmörgås">here it is</h1>')
- str(soup)
- self.assertEqual("here it is", soup.find(id='Räksmörgås').text)
-
-
- def test_find_everything(self):
- """Test an optimization that finds all tags."""
- soup = self.soup("<a>foo</a><b>bar</b>")
- self.assertEqual(2, len(soup.find_all()))
-
- def test_find_everything_with_name(self):
- """Test an optimization that finds all tags with a given name."""
- soup = self.soup("<a>foo</a><b>bar</b><a>baz</a>")
- self.assertEqual(2, len(soup.find_all('a')))
-
-class TestFindAll(TreeTest):
- """Basic tests of the find_all() method."""
-
- def test_find_all_text_nodes(self):
- """You can search the tree for text nodes."""
- soup = self.soup("<html>Foo<b>bar</b>\xbb</html>")
- # Exact match.
- self.assertEqual(soup.find_all(string="bar"), ["bar"])
- self.assertEqual(soup.find_all(text="bar"), ["bar"])
- # Match any of a number of strings.
- self.assertEqual(
- soup.find_all(text=["Foo", "bar"]), ["Foo", "bar"])
- # Match a regular expression.
- self.assertEqual(soup.find_all(text=re.compile('.*')),
- ["Foo", "bar", '\xbb'])
- # Match anything.
- self.assertEqual(soup.find_all(text=True),
- ["Foo", "bar", '\xbb'])
-
- def test_find_all_limit(self):
- """You can limit the number of items returned by find_all."""
- soup = self.soup("<a>1</a><a>2</a><a>3</a><a>4</a><a>5</a>")
- self.assertSelects(soup.find_all('a', limit=3), ["1", "2", "3"])
- self.assertSelects(soup.find_all('a', limit=1), ["1"])
- self.assertSelects(
- soup.find_all('a', limit=10), ["1", "2", "3", "4", "5"])
-
- # A limit of 0 means no limit.
- self.assertSelects(
- soup.find_all('a', limit=0), ["1", "2", "3", "4", "5"])
-
- def test_calling_a_tag_is_calling_findall(self):
- soup = self.soup("<a>1</a><b>2<a id='foo'>3</a></b>")
- self.assertSelects(soup('a', limit=1), ["1"])
- self.assertSelects(soup.b(id="foo"), ["3"])
-
- def test_find_all_with_self_referential_data_structure_does_not_cause_infinite_recursion(self):
- soup = self.soup("<a></a>")
- # Create a self-referential list.
- l = []
- l.append(l)
-
- # Without special code in _normalize_search_value, this would cause infinite
- # recursion.
- self.assertEqual([], soup.find_all(l))
-
- def test_find_all_resultset(self):
- """All find_all calls return a ResultSet"""
- soup = self.soup("<a></a>")
- result = soup.find_all("a")
- self.assertTrue(hasattr(result, "source"))
-
- result = soup.find_all(True)
- self.assertTrue(hasattr(result, "source"))
-
- result = soup.find_all(text="foo")
- self.assertTrue(hasattr(result, "source"))
-
-
-class TestFindAllBasicNamespaces(TreeTest):
-
- def test_find_by_namespaced_name(self):
- soup = self.soup('<mathml:msqrt>4</mathml:msqrt><a svg:fill="red">')
- self.assertEqual("4", soup.find("mathml:msqrt").string)
- self.assertEqual("a", soup.find(attrs= { "svg:fill" : "red" }).name)
-
-
-class TestFindAllByName(TreeTest):
- """Test ways of finding tags by tag name."""
-
- def setUp(self):
- super(TreeTest, self).setUp()
- self.tree = self.soup("""<a>First tag.</a>
- <b>Second tag.</b>
- <c>Third <a>Nested tag.</a> tag.</c>""")
-
- def test_find_all_by_tag_name(self):
- # Find all the <a> tags.
- self.assertSelects(
- self.tree.find_all('a'), ['First tag.', 'Nested tag.'])
-
- def test_find_all_by_name_and_text(self):
- self.assertSelects(
- self.tree.find_all('a', text='First tag.'), ['First tag.'])
-
- self.assertSelects(
- self.tree.find_all('a', text=True), ['First tag.', 'Nested tag.'])
-
- self.assertSelects(
- self.tree.find_all('a', text=re.compile("tag")),
- ['First tag.', 'Nested tag.'])
-
-
- def test_find_all_on_non_root_element(self):
- # You can call find_all on any node, not just the root.
- self.assertSelects(self.tree.c.find_all('a'), ['Nested tag.'])
-
- def test_calling_element_invokes_find_all(self):
- self.assertSelects(self.tree('a'), ['First tag.', 'Nested tag.'])
-
- def test_find_all_by_tag_strainer(self):
- self.assertSelects(
- self.tree.find_all(SoupStrainer('a')),
- ['First tag.', 'Nested tag.'])
-
- def test_find_all_by_tag_names(self):
- self.assertSelects(
- self.tree.find_all(['a', 'b']),
- ['First tag.', 'Second tag.', 'Nested tag.'])
-
- def test_find_all_by_tag_dict(self):
- self.assertSelects(
- self.tree.find_all({'a' : True, 'b' : True}),
- ['First tag.', 'Second tag.', 'Nested tag.'])
-
- def test_find_all_by_tag_re(self):
- self.assertSelects(
- self.tree.find_all(re.compile('^[ab]$')),
- ['First tag.', 'Second tag.', 'Nested tag.'])
-
- def test_find_all_with_tags_matching_method(self):
- # You can define an oracle method that determines whether
- # a tag matches the search.
- def id_matches_name(tag):
- return tag.name == tag.get('id')
-
- tree = self.soup("""<a id="a">Match 1.</a>
- <a id="1">Does not match.</a>
- <b id="b">Match 2.</a>""")
-
- self.assertSelects(
- tree.find_all(id_matches_name), ["Match 1.", "Match 2."])
-
-
-class TestFindAllByAttribute(TreeTest):
-
- def test_find_all_by_attribute_name(self):
- # You can pass in keyword arguments to find_all to search by
- # attribute.
- tree = self.soup("""
- <a id="first">Matching a.</a>
- <a id="second">
- Non-matching <b id="first">Matching b.</b>a.
- </a>""")
- self.assertSelects(tree.find_all(id='first'),
- ["Matching a.", "Matching b."])
-
- def test_find_all_by_utf8_attribute_value(self):
- peace = "םולש".encode("utf8")
- data = '<a title="םולש"></a>'.encode("utf8")
- soup = self.soup(data)
- self.assertEqual([soup.a], soup.find_all(title=peace))
- self.assertEqual([soup.a], soup.find_all(title=peace.decode("utf8")))
- self.assertEqual([soup.a], soup.find_all(title=[peace, "something else"]))
-
- def test_find_all_by_attribute_dict(self):
- # You can pass in a dictionary as the argument 'attrs'. This
- # lets you search for attributes like 'name' (a fixed argument
- # to find_all) and 'class' (a reserved word in Python.)
- tree = self.soup("""
- <a name="name1" class="class1">Name match.</a>
- <a name="name2" class="class2">Class match.</a>
- <a name="name3" class="class3">Non-match.</a>
- <name1>A tag called 'name1'.</name1>
- """)
-
- # This doesn't do what you want.
- self.assertSelects(tree.find_all(name='name1'),
- ["A tag called 'name1'."])
- # This does what you want.
- self.assertSelects(tree.find_all(attrs={'name' : 'name1'}),
- ["Name match."])
-
- self.assertSelects(tree.find_all(attrs={'class' : 'class2'}),
- ["Class match."])
-
- def test_find_all_by_class(self):
- tree = self.soup("""
- <a class="1">Class 1.</a>
- <a class="2">Class 2.</a>
- <b class="1">Class 1.</b>
- <c class="3 4">Class 3 and 4.</c>
- """)
-
- # Passing in the class_ keyword argument will search against
- # the 'class' attribute.
- self.assertSelects(tree.find_all('a', class_='1'), ['Class 1.'])
- self.assertSelects(tree.find_all('c', class_='3'), ['Class 3 and 4.'])
- self.assertSelects(tree.find_all('c', class_='4'), ['Class 3 and 4.'])
-
- # Passing in a string to 'attrs' will also search the CSS class.
- self.assertSelects(tree.find_all('a', '1'), ['Class 1.'])
- self.assertSelects(tree.find_all(attrs='1'), ['Class 1.', 'Class 1.'])
- self.assertSelects(tree.find_all('c', '3'), ['Class 3 and 4.'])
- self.assertSelects(tree.find_all('c', '4'), ['Class 3 and 4.'])
-
- def test_find_by_class_when_multiple_classes_present(self):
- tree = self.soup("<gar class='foo bar'>Found it</gar>")
-
- f = tree.find_all("gar", class_=re.compile("o"))
- self.assertSelects(f, ["Found it"])
-
- f = tree.find_all("gar", class_=re.compile("a"))
- self.assertSelects(f, ["Found it"])
-
- # Since the class is not the string "foo bar", but the two
- # strings "foo" and "bar", this will not find anything.
- f = tree.find_all("gar", class_=re.compile("o b"))
- self.assertSelects(f, [])
-
- def test_find_all_with_non_dictionary_for_attrs_finds_by_class(self):
- soup = self.soup("<a class='bar'>Found it</a>")
-
- self.assertSelects(soup.find_all("a", re.compile("ba")), ["Found it"])
-
- def big_attribute_value(value):
- return len(value) > 3
-
- self.assertSelects(soup.find_all("a", big_attribute_value), [])
-
- def small_attribute_value(value):
- return len(value) <= 3
-
- self.assertSelects(
- soup.find_all("a", small_attribute_value), ["Found it"])
-
- def test_find_all_with_string_for_attrs_finds_multiple_classes(self):
- soup = self.soup('<a class="foo bar"></a><a class="foo"></a>')
- a, a2 = soup.find_all("a")
- self.assertEqual([a, a2], soup.find_all("a", "foo"))
- self.assertEqual([a], soup.find_all("a", "bar"))
-
- # If you specify the class as a string that contains a
- # space, only that specific value will be found.
- self.assertEqual([a], soup.find_all("a", class_="foo bar"))
- self.assertEqual([a], soup.find_all("a", "foo bar"))
- self.assertEqual([], soup.find_all("a", "bar foo"))
-
- def test_find_all_by_attribute_soupstrainer(self):
- tree = self.soup("""
- <a id="first">Match.</a>
- <a id="second">Non-match.</a>""")
-
- strainer = SoupStrainer(attrs={'id' : 'first'})
- self.assertSelects(tree.find_all(strainer), ['Match.'])
-
- def test_find_all_with_missing_atribute(self):
- # You can pass in None as the value of an attribute to find_all.
- # This will match tags that do not have that attribute set.
- tree = self.soup("""<a id="1">ID present.</a>
- <a>No ID present.</a>
- <a id="">ID is empty.</a>""")
- self.assertSelects(tree.find_all('a', id=None), ["No ID present."])
-
- def test_find_all_with_defined_attribute(self):
- # You can pass in None as the value of an attribute to find_all.
- # This will match tags that have that attribute set to any value.
- tree = self.soup("""<a id="1">ID present.</a>
- <a>No ID present.</a>
- <a id="">ID is empty.</a>""")
- self.assertSelects(
- tree.find_all(id=True), ["ID present.", "ID is empty."])
-
- def test_find_all_with_numeric_attribute(self):
- # If you search for a number, it's treated as a string.
- tree = self.soup("""<a id=1>Unquoted attribute.</a>
- <a id="1">Quoted attribute.</a>""")
-
- expected = ["Unquoted attribute.", "Quoted attribute."]
- self.assertSelects(tree.find_all(id=1), expected)
- self.assertSelects(tree.find_all(id="1"), expected)
-
- def test_find_all_with_list_attribute_values(self):
- # You can pass a list of attribute values instead of just one,
- # and you'll get tags that match any of the values.
- tree = self.soup("""<a id="1">1</a>
- <a id="2">2</a>
- <a id="3">3</a>
- <a>No ID.</a>""")
- self.assertSelects(tree.find_all(id=["1", "3", "4"]),
- ["1", "3"])
-
- def test_find_all_with_regular_expression_attribute_value(self):
- # You can pass a regular expression as an attribute value, and
- # you'll get tags whose values for that attribute match the
- # regular expression.
- tree = self.soup("""<a id="a">One a.</a>
- <a id="aa">Two as.</a>
- <a id="ab">Mixed as and bs.</a>
- <a id="b">One b.</a>
- <a>No ID.</a>""")
-
- self.assertSelects(tree.find_all(id=re.compile("^a+$")),
- ["One a.", "Two as."])
-
- def test_find_by_name_and_containing_string(self):
- soup = self.soup("<b>foo</b><b>bar</b><a>foo</a>")
- a = soup.a
-
- self.assertEqual([a], soup.find_all("a", text="foo"))
- self.assertEqual([], soup.find_all("a", text="bar"))
- self.assertEqual([], soup.find_all("a", text="bar"))
-
- def test_find_by_name_and_containing_string_when_string_is_buried(self):
- soup = self.soup("<a>foo</a><a><b><c>foo</c></b></a>")
- self.assertEqual(soup.find_all("a"), soup.find_all("a", text="foo"))
-
- def test_find_by_attribute_and_containing_string(self):
- soup = self.soup('<b id="1">foo</b><a id="2">foo</a>')
- a = soup.a
-
- self.assertEqual([a], soup.find_all(id=2, text="foo"))
- self.assertEqual([], soup.find_all(id=1, text="bar"))
-
-
-
-
-class TestIndex(TreeTest):
- """Test Tag.index"""
- def test_index(self):
- tree = self.soup("""<div>
- <a>Identical</a>
- <b>Not identical</b>
- <a>Identical</a>
-
- <c><d>Identical with child</d></c>
- <b>Also not identical</b>
- <c><d>Identical with child</d></c>
- </div>""")
- div = tree.div
- for i, element in enumerate(div.contents):
- self.assertEqual(i, div.index(element))
- self.assertRaises(ValueError, tree.index, 1)
-
-
-class TestParentOperations(TreeTest):
- """Test navigation and searching through an element's parents."""
-
- def setUp(self):
- super(TestParentOperations, self).setUp()
- self.tree = self.soup('''<ul id="empty"></ul>
- <ul id="top">
- <ul id="middle">
- <ul id="bottom">
- <b>Start here</b>
- </ul>
- </ul>''')
- self.start = self.tree.b
-
-
- def test_parent(self):
- self.assertEqual(self.start.parent['id'], 'bottom')
- self.assertEqual(self.start.parent.parent['id'], 'middle')
- self.assertEqual(self.start.parent.parent.parent['id'], 'top')
-
- def test_parent_of_top_tag_is_soup_object(self):
- top_tag = self.tree.contents[0]
- self.assertEqual(top_tag.parent, self.tree)
-
- def test_soup_object_has_no_parent(self):
- self.assertEqual(None, self.tree.parent)
-
- def test_find_parents(self):
- self.assertSelectsIDs(
- self.start.find_parents('ul'), ['bottom', 'middle', 'top'])
- self.assertSelectsIDs(
- self.start.find_parents('ul', id="middle"), ['middle'])
-
- def test_find_parent(self):
- self.assertEqual(self.start.find_parent('ul')['id'], 'bottom')
- self.assertEqual(self.start.find_parent('ul', id='top')['id'], 'top')
-
- def test_parent_of_text_element(self):
- text = self.tree.find(text="Start here")
- self.assertEqual(text.parent.name, 'b')
-
- def test_text_element_find_parent(self):
- text = self.tree.find(text="Start here")
- self.assertEqual(text.find_parent('ul')['id'], 'bottom')
-
- def test_parent_generator(self):
- parents = [parent['id'] for parent in self.start.parents
- if parent is not None and 'id' in parent.attrs]
- self.assertEqual(parents, ['bottom', 'middle', 'top'])
-
-
-class ProximityTest(TreeTest):
-
- def setUp(self):
- super(TreeTest, self).setUp()
- self.tree = self.soup(
- '<html id="start"><head></head><body><b id="1">One</b><b id="2">Two</b><b id="3">Three</b></body></html>')
-
-
-class TestNextOperations(ProximityTest):
-
- def setUp(self):
- super(TestNextOperations, self).setUp()
- self.start = self.tree.b
-
- def test_next(self):
- self.assertEqual(self.start.next_element, "One")
- self.assertEqual(self.start.next_element.next_element['id'], "2")
-
- def test_next_of_last_item_is_none(self):
- last = self.tree.find(text="Three")
- self.assertEqual(last.next_element, None)
-
- def test_next_of_root_is_none(self):
- # The document root is outside the next/previous chain.
- self.assertEqual(self.tree.next_element, None)
-
- def test_find_all_next(self):
- self.assertSelects(self.start.find_all_next('b'), ["Two", "Three"])
- self.start.find_all_next(id=3)
- self.assertSelects(self.start.find_all_next(id=3), ["Three"])
-
- def test_find_next(self):
- self.assertEqual(self.start.find_next('b')['id'], '2')
- self.assertEqual(self.start.find_next(text="Three"), "Three")
-
- def test_find_next_for_text_element(self):
- text = self.tree.find(text="One")
- self.assertEqual(text.find_next("b").string, "Two")
- self.assertSelects(text.find_all_next("b"), ["Two", "Three"])
-
- def test_next_generator(self):
- start = self.tree.find(text="Two")
- successors = [node for node in start.next_elements]
- # There are two successors: the final <b> tag and its text contents.
- tag, contents = successors
- self.assertEqual(tag['id'], '3')
- self.assertEqual(contents, "Three")
-
-class TestPreviousOperations(ProximityTest):
-
- def setUp(self):
- super(TestPreviousOperations, self).setUp()
- self.end = self.tree.find(text="Three")
-
- def test_previous(self):
- self.assertEqual(self.end.previous_element['id'], "3")
- self.assertEqual(self.end.previous_element.previous_element, "Two")
-
- def test_previous_of_first_item_is_none(self):
- first = self.tree.find('html')
- self.assertEqual(first.previous_element, None)
-
- def test_previous_of_root_is_none(self):
- # The document root is outside the next/previous chain.
- # XXX This is broken!
- #self.assertEqual(self.tree.previous_element, None)
- pass
-
- def test_find_all_previous(self):
- # The <b> tag containing the "Three" node is the predecessor
- # of the "Three" node itself, which is why "Three" shows up
- # here.
- self.assertSelects(
- self.end.find_all_previous('b'), ["Three", "Two", "One"])
- self.assertSelects(self.end.find_all_previous(id=1), ["One"])
-
- def test_find_previous(self):
- self.assertEqual(self.end.find_previous('b')['id'], '3')
- self.assertEqual(self.end.find_previous(text="One"), "One")
-
- def test_find_previous_for_text_element(self):
- text = self.tree.find(text="Three")
- self.assertEqual(text.find_previous("b").string, "Three")
- self.assertSelects(
- text.find_all_previous("b"), ["Three", "Two", "One"])
-
- def test_previous_generator(self):
- start = self.tree.find(text="One")
- predecessors = [node for node in start.previous_elements]
-
- # There are four predecessors: the <b> tag containing "One"
- # the <body> tag, the <head> tag, and the <html> tag.
- b, body, head, html = predecessors
- self.assertEqual(b['id'], '1')
- self.assertEqual(body.name, "body")
- self.assertEqual(head.name, "head")
- self.assertEqual(html.name, "html")
-
-
-class SiblingTest(TreeTest):
-
- def setUp(self):
- super(SiblingTest, self).setUp()
- markup = '''<html>
- <span id="1">
- <span id="1.1"></span>
- </span>
- <span id="2">
- <span id="2.1"></span>
- </span>
- <span id="3">
- <span id="3.1"></span>
- </span>
- <span id="4"></span>
- </html>'''
- # All that whitespace looks good but makes the tests more
- # difficult. Get rid of it.
- markup = re.compile(r"\n\s*").sub("", markup)
- self.tree = self.soup(markup)
-
-
-class TestNextSibling(SiblingTest):
-
- def setUp(self):
- super(TestNextSibling, self).setUp()
- self.start = self.tree.find(id="1")
-
- def test_next_sibling_of_root_is_none(self):
- self.assertEqual(self.tree.next_sibling, None)
-
- def test_next_sibling(self):
- self.assertEqual(self.start.next_sibling['id'], '2')
- self.assertEqual(self.start.next_sibling.next_sibling['id'], '3')
-
- # Note the difference between next_sibling and next_element.
- self.assertEqual(self.start.next_element['id'], '1.1')
-
- def test_next_sibling_may_not_exist(self):
- self.assertEqual(self.tree.html.next_sibling, None)
-
- nested_span = self.tree.find(id="1.1")
- self.assertEqual(nested_span.next_sibling, None)
-
- last_span = self.tree.find(id="4")
- self.assertEqual(last_span.next_sibling, None)
-
- def test_find_next_sibling(self):
- self.assertEqual(self.start.find_next_sibling('span')['id'], '2')
-
- def test_next_siblings(self):
- self.assertSelectsIDs(self.start.find_next_siblings("span"),
- ['2', '3', '4'])
-
- self.assertSelectsIDs(self.start.find_next_siblings(id='3'), ['3'])
-
- def test_next_sibling_for_text_element(self):
- soup = self.soup("Foo<b>bar</b>baz")
- start = soup.find(text="Foo")
- self.assertEqual(start.next_sibling.name, 'b')
- self.assertEqual(start.next_sibling.next_sibling, 'baz')
-
- self.assertSelects(start.find_next_siblings('b'), ['bar'])
- self.assertEqual(start.find_next_sibling(text="baz"), "baz")
- self.assertEqual(start.find_next_sibling(text="nonesuch"), None)
-
-
-class TestPreviousSibling(SiblingTest):
-
- def setUp(self):
- super(TestPreviousSibling, self).setUp()
- self.end = self.tree.find(id="4")
-
- def test_previous_sibling_of_root_is_none(self):
- self.assertEqual(self.tree.previous_sibling, None)
-
- def test_previous_sibling(self):
- self.assertEqual(self.end.previous_sibling['id'], '3')
- self.assertEqual(self.end.previous_sibling.previous_sibling['id'], '2')
-
- # Note the difference between previous_sibling and previous_element.
- self.assertEqual(self.end.previous_element['id'], '3.1')
-
- def test_previous_sibling_may_not_exist(self):
- self.assertEqual(self.tree.html.previous_sibling, None)
-
- nested_span = self.tree.find(id="1.1")
- self.assertEqual(nested_span.previous_sibling, None)
-
- first_span = self.tree.find(id="1")
- self.assertEqual(first_span.previous_sibling, None)
-
- def test_find_previous_sibling(self):
- self.assertEqual(self.end.find_previous_sibling('span')['id'], '3')
-
- def test_previous_siblings(self):
- self.assertSelectsIDs(self.end.find_previous_siblings("span"),
- ['3', '2', '1'])
-
- self.assertSelectsIDs(self.end.find_previous_siblings(id='1'), ['1'])
-
- def test_previous_sibling_for_text_element(self):
- soup = self.soup("Foo<b>bar</b>baz")
- start = soup.find(text="baz")
- self.assertEqual(start.previous_sibling.name, 'b')
- self.assertEqual(start.previous_sibling.previous_sibling, 'Foo')
-
- self.assertSelects(start.find_previous_siblings('b'), ['bar'])
- self.assertEqual(start.find_previous_sibling(text="Foo"), "Foo")
- self.assertEqual(start.find_previous_sibling(text="nonesuch"), None)
-
-
-class TestTagCreation(SoupTest):
- """Test the ability to create new tags."""
- def test_new_tag(self):
- soup = self.soup("")
- new_tag = soup.new_tag("foo", bar="baz")
- self.assertTrue(isinstance(new_tag, Tag))
- self.assertEqual("foo", new_tag.name)
- self.assertEqual(dict(bar="baz"), new_tag.attrs)
- self.assertEqual(None, new_tag.parent)
-
- def test_tag_inherits_self_closing_rules_from_builder(self):
- if XML_BUILDER_PRESENT:
- xml_soup = BeautifulSoup("", "lxml-xml")
- xml_br = xml_soup.new_tag("br")
- xml_p = xml_soup.new_tag("p")
-
- # Both the <br> and <p> tag are empty-element, just because
- # they have no contents.
- self.assertEqual(b"<br/>", xml_br.encode())
- self.assertEqual(b"<p/>", xml_p.encode())
-
- html_soup = BeautifulSoup("", "html.parser")
- html_br = html_soup.new_tag("br")
- html_p = html_soup.new_tag("p")
-
- # The HTML builder users HTML's rules about which tags are
- # empty-element tags, and the new tags reflect these rules.
- self.assertEqual(b"<br/>", html_br.encode())
- self.assertEqual(b"<p></p>", html_p.encode())
-
- def test_new_string_creates_navigablestring(self):
- soup = self.soup("")
- s = soup.new_string("foo")
- self.assertEqual("foo", s)
- self.assertTrue(isinstance(s, NavigableString))
-
- def test_new_string_can_create_navigablestring_subclass(self):
- soup = self.soup("")
- s = soup.new_string("foo", Comment)
- self.assertEqual("foo", s)
- self.assertTrue(isinstance(s, Comment))
-
-class TestTreeModification(SoupTest):
-
- def test_attribute_modification(self):
- soup = self.soup('<a id="1"></a>')
- soup.a['id'] = 2
- self.assertEqual(soup.decode(), self.document_for('<a id="2"></a>'))
- del(soup.a['id'])
- self.assertEqual(soup.decode(), self.document_for('<a></a>'))
- soup.a['id2'] = 'foo'
- self.assertEqual(soup.decode(), self.document_for('<a id2="foo"></a>'))
-
- def test_new_tag_creation(self):
- builder = builder_registry.lookup('html')()
- soup = self.soup("<body></body>", builder=builder)
- a = Tag(soup, builder, 'a')
- ol = Tag(soup, builder, 'ol')
- a['href'] = 'http://foo.com/'
- soup.body.insert(0, a)
- soup.body.insert(1, ol)
- self.assertEqual(
- soup.body.encode(),
- b'<body><a href="http://foo.com/"></a><ol></ol></body>')
-
- def test_append_to_contents_moves_tag(self):
- doc = """<p id="1">Don't leave me <b>here</b>.</p>
- <p id="2">Don\'t leave!</p>"""
- soup = self.soup(doc)
- second_para = soup.find(id='2')
- bold = soup.b
-
- # Move the <b> tag to the end of the second paragraph.
- soup.find(id='2').append(soup.b)
-
- # The <b> tag is now a child of the second paragraph.
- self.assertEqual(bold.parent, second_para)
-
- self.assertEqual(
- soup.decode(), self.document_for(
- '<p id="1">Don\'t leave me .</p>\n'
- '<p id="2">Don\'t leave!<b>here</b></p>'))
-
- def test_replace_with_returns_thing_that_was_replaced(self):
- text = "<a></a><b><c></c></b>"
- soup = self.soup(text)
- a = soup.a
- new_a = a.replace_with(soup.c)
- self.assertEqual(a, new_a)
-
- def test_unwrap_returns_thing_that_was_replaced(self):
- text = "<a><b></b><c></c></a>"
- soup = self.soup(text)
- a = soup.a
- new_a = a.unwrap()
- self.assertEqual(a, new_a)
-
- def test_replace_with_and_unwrap_give_useful_exception_when_tag_has_no_parent(self):
- soup = self.soup("<a><b>Foo</b></a><c>Bar</c>")
- a = soup.a
- a.extract()
- self.assertEqual(None, a.parent)
- self.assertRaises(ValueError, a.unwrap)
- self.assertRaises(ValueError, a.replace_with, soup.c)
-
- def test_replace_tag_with_itself(self):
- text = "<a><b></b><c>Foo<d></d></c></a><a><e></e></a>"
- soup = self.soup(text)
- c = soup.c
- soup.c.replace_with(c)
- self.assertEqual(soup.decode(), self.document_for(text))
-
- def test_replace_tag_with_its_parent_raises_exception(self):
- text = "<a><b></b></a>"
- soup = self.soup(text)
- self.assertRaises(ValueError, soup.b.replace_with, soup.a)
-
- def test_insert_tag_into_itself_raises_exception(self):
- text = "<a><b></b></a>"
- soup = self.soup(text)
- self.assertRaises(ValueError, soup.a.insert, 0, soup.a)
-
- def test_replace_with_maintains_next_element_throughout(self):
- soup = self.soup('<p><a>one</a><b>three</b></p>')
- a = soup.a
- b = a.contents[0]
- # Make it so the <a> tag has two text children.
- a.insert(1, "two")
-
- # Now replace each one with the empty string.
- left, right = a.contents
- left.replaceWith('')
- right.replaceWith('')
-
- # The <b> tag is still connected to the tree.
- self.assertEqual("three", soup.b.string)
-
- def test_replace_final_node(self):
- soup = self.soup("<b>Argh!</b>")
- soup.find(text="Argh!").replace_with("Hooray!")
- new_text = soup.find(text="Hooray!")
- b = soup.b
- self.assertEqual(new_text.previous_element, b)
- self.assertEqual(new_text.parent, b)
- self.assertEqual(new_text.previous_element.next_element, new_text)
- self.assertEqual(new_text.next_element, None)
-
- def test_consecutive_text_nodes(self):
- # A builder should never create two consecutive text nodes,
- # but if you insert one next to another, Beautiful Soup will
- # handle it correctly.
- soup = self.soup("<a><b>Argh!</b><c></c></a>")
- soup.b.insert(1, "Hooray!")
-
- self.assertEqual(
- soup.decode(), self.document_for(
- "<a><b>Argh!Hooray!</b><c></c></a>"))
-
- new_text = soup.find(text="Hooray!")
- self.assertEqual(new_text.previous_element, "Argh!")
- self.assertEqual(new_text.previous_element.next_element, new_text)
-
- self.assertEqual(new_text.previous_sibling, "Argh!")
- self.assertEqual(new_text.previous_sibling.next_sibling, new_text)
-
- self.assertEqual(new_text.next_sibling, None)
- self.assertEqual(new_text.next_element, soup.c)
-
- def test_insert_string(self):
- soup = self.soup("<a></a>")
- soup.a.insert(0, "bar")
- soup.a.insert(0, "foo")
- # The string were added to the tag.
- self.assertEqual(["foo", "bar"], soup.a.contents)
- # And they were converted to NavigableStrings.
- self.assertEqual(soup.a.contents[0].next_element, "bar")
-
- def test_insert_tag(self):
- builder = self.default_builder
- soup = self.soup(
- "<a><b>Find</b><c>lady!</c><d></d></a>", builder=builder)
- magic_tag = Tag(soup, builder, 'magictag')
- magic_tag.insert(0, "the")
- soup.a.insert(1, magic_tag)
-
- self.assertEqual(
- soup.decode(), self.document_for(
- "<a><b>Find</b><magictag>the</magictag><c>lady!</c><d></d></a>"))
-
- # Make sure all the relationships are hooked up correctly.
- b_tag = soup.b
- self.assertEqual(b_tag.next_sibling, magic_tag)
- self.assertEqual(magic_tag.previous_sibling, b_tag)
-
- find = b_tag.find(text="Find")
- self.assertEqual(find.next_element, magic_tag)
- self.assertEqual(magic_tag.previous_element, find)
-
- c_tag = soup.c
- self.assertEqual(magic_tag.next_sibling, c_tag)
- self.assertEqual(c_tag.previous_sibling, magic_tag)
-
- the = magic_tag.find(text="the")
- self.assertEqual(the.parent, magic_tag)
- self.assertEqual(the.next_element, c_tag)
- self.assertEqual(c_tag.previous_element, the)
-
- def test_append_child_thats_already_at_the_end(self):
- data = "<a><b></b></a>"
- soup = self.soup(data)
- soup.a.append(soup.b)
- self.assertEqual(data, soup.decode())
-
- def test_move_tag_to_beginning_of_parent(self):
- data = "<a><b></b><c></c><d></d></a>"
- soup = self.soup(data)
- soup.a.insert(0, soup.d)
- self.assertEqual("<a><d></d><b></b><c></c></a>", soup.decode())
-
- def test_insert_works_on_empty_element_tag(self):
- # This is a little strange, since most HTML parsers don't allow
- # markup like this to come through. But in general, we don't
- # know what the parser would or wouldn't have allowed, so
- # I'm letting this succeed for now.
- soup = self.soup("<br/>")
- soup.br.insert(1, "Contents")
- self.assertEqual(str(soup.br), "<br>Contents</br>")
-
- def test_insert_before(self):
- soup = self.soup("<a>foo</a><b>bar</b>")
- soup.b.insert_before("BAZ")
- soup.a.insert_before("QUUX")
- self.assertEqual(
- soup.decode(), self.document_for("QUUX<a>foo</a>BAZ<b>bar</b>"))
-
- soup.a.insert_before(soup.b)
- self.assertEqual(
- soup.decode(), self.document_for("QUUX<b>bar</b><a>foo</a>BAZ"))
-
- def test_insert_after(self):
- soup = self.soup("<a>foo</a><b>bar</b>")
- soup.b.insert_after("BAZ")
- soup.a.insert_after("QUUX")
- self.assertEqual(
- soup.decode(), self.document_for("<a>foo</a>QUUX<b>bar</b>BAZ"))
- soup.b.insert_after(soup.a)
- self.assertEqual(
- soup.decode(), self.document_for("QUUX<b>bar</b><a>foo</a>BAZ"))
-
- def test_insert_after_raises_exception_if_after_has_no_meaning(self):
- soup = self.soup("")
- tag = soup.new_tag("a")
- string = soup.new_string("")
- self.assertRaises(ValueError, string.insert_after, tag)
- self.assertRaises(NotImplementedError, soup.insert_after, tag)
- self.assertRaises(ValueError, tag.insert_after, tag)
-
- def test_insert_before_raises_notimplementederror_if_before_has_no_meaning(self):
- soup = self.soup("")
- tag = soup.new_tag("a")
- string = soup.new_string("")
- self.assertRaises(ValueError, string.insert_before, tag)
- self.assertRaises(NotImplementedError, soup.insert_before, tag)
- self.assertRaises(ValueError, tag.insert_before, tag)
-
- def test_replace_with(self):
- soup = self.soup(
- "<p>There's <b>no</b> business like <b>show</b> business</p>")
- no, show = soup.find_all('b')
- show.replace_with(no)
- self.assertEqual(
- soup.decode(),
- self.document_for(
- "<p>There's business like <b>no</b> business</p>"))
-
- self.assertEqual(show.parent, None)
- self.assertEqual(no.parent, soup.p)
- self.assertEqual(no.next_element, "no")
- self.assertEqual(no.next_sibling, " business")
-
- def test_replace_first_child(self):
- data = "<a><b></b><c></c></a>"
- soup = self.soup(data)
- soup.b.replace_with(soup.c)
- self.assertEqual("<a><c></c></a>", soup.decode())
-
- def test_replace_last_child(self):
- data = "<a><b></b><c></c></a>"
- soup = self.soup(data)
- soup.c.replace_with(soup.b)
- self.assertEqual("<a><b></b></a>", soup.decode())
-
- def test_nested_tag_replace_with(self):
- soup = self.soup(
- """<a>We<b>reserve<c>the</c><d>right</d></b></a><e>to<f>refuse</f><g>service</g></e>""")
-
- # Replace the entire <b> tag and its contents ("reserve the
- # right") with the <f> tag ("refuse").
- remove_tag = soup.b
- move_tag = soup.f
- remove_tag.replace_with(move_tag)
-
- self.assertEqual(
- soup.decode(), self.document_for(
- "<a>We<f>refuse</f></a><e>to<g>service</g></e>"))
-
- # The <b> tag is now an orphan.
- self.assertEqual(remove_tag.parent, None)
- self.assertEqual(remove_tag.find(text="right").next_element, None)
- self.assertEqual(remove_tag.previous_element, None)
- self.assertEqual(remove_tag.next_sibling, None)
- self.assertEqual(remove_tag.previous_sibling, None)
-
- # The <f> tag is now connected to the <a> tag.
- self.assertEqual(move_tag.parent, soup.a)
- self.assertEqual(move_tag.previous_element, "We")
- self.assertEqual(move_tag.next_element.next_element, soup.e)
- self.assertEqual(move_tag.next_sibling, None)
-
- # The gap where the <f> tag used to be has been mended, and
- # the word "to" is now connected to the <g> tag.
- to_text = soup.find(text="to")
- g_tag = soup.g
- self.assertEqual(to_text.next_element, g_tag)
- self.assertEqual(to_text.next_sibling, g_tag)
- self.assertEqual(g_tag.previous_element, to_text)
- self.assertEqual(g_tag.previous_sibling, to_text)
-
- def test_unwrap(self):
- tree = self.soup("""
- <p>Unneeded <em>formatting</em> is unneeded</p>
- """)
- tree.em.unwrap()
- self.assertEqual(tree.em, None)
- self.assertEqual(tree.p.text, "Unneeded formatting is unneeded")
-
- def test_wrap(self):
- soup = self.soup("I wish I was bold.")
- value = soup.string.wrap(soup.new_tag("b"))
- self.assertEqual(value.decode(), "<b>I wish I was bold.</b>")
- self.assertEqual(
- soup.decode(), self.document_for("<b>I wish I was bold.</b>"))
-
- def test_wrap_extracts_tag_from_elsewhere(self):
- soup = self.soup("<b></b>I wish I was bold.")
- soup.b.next_sibling.wrap(soup.b)
- self.assertEqual(
- soup.decode(), self.document_for("<b>I wish I was bold.</b>"))
-
- def test_wrap_puts_new_contents_at_the_end(self):
- soup = self.soup("<b>I like being bold.</b>I wish I was bold.")
- soup.b.next_sibling.wrap(soup.b)
- self.assertEqual(2, len(soup.b.contents))
- self.assertEqual(
- soup.decode(), self.document_for(
- "<b>I like being bold.I wish I was bold.</b>"))
-
- def test_extract(self):
- soup = self.soup(
- '<html><body>Some content. <div id="nav">Nav crap</div> More content.</body></html>')
-
- self.assertEqual(len(soup.body.contents), 3)
- extracted = soup.find(id="nav").extract()
-
- self.assertEqual(
- soup.decode(), "<html><body>Some content. More content.</body></html>")
- self.assertEqual(extracted.decode(), '<div id="nav">Nav crap</div>')
-
- # The extracted tag is now an orphan.
- self.assertEqual(len(soup.body.contents), 2)
- self.assertEqual(extracted.parent, None)
- self.assertEqual(extracted.previous_element, None)
- self.assertEqual(extracted.next_element.next_element, None)
-
- # The gap where the extracted tag used to be has been mended.
- content_1 = soup.find(text="Some content. ")
- content_2 = soup.find(text=" More content.")
- self.assertEqual(content_1.next_element, content_2)
- self.assertEqual(content_1.next_sibling, content_2)
- self.assertEqual(content_2.previous_element, content_1)
- self.assertEqual(content_2.previous_sibling, content_1)
-
- def test_extract_distinguishes_between_identical_strings(self):
- soup = self.soup("<a>foo</a><b>bar</b>")
- foo_1 = soup.a.string
- bar_1 = soup.b.string
- foo_2 = soup.new_string("foo")
- bar_2 = soup.new_string("bar")
- soup.a.append(foo_2)
- soup.b.append(bar_2)
-
- # Now there are two identical strings in the <a> tag, and two
- # in the <b> tag. Let's remove the first "foo" and the second
- # "bar".
- foo_1.extract()
- bar_2.extract()
- self.assertEqual(foo_2, soup.a.string)
- self.assertEqual(bar_2, soup.b.string)
-
- def test_extract_multiples_of_same_tag(self):
- soup = self.soup("""
-<html>
-<head>
-<script>foo</script>
-</head>
-<body>
- <script>bar</script>
- <a></a>
-</body>
-<script>baz</script>
-</html>""")
- [soup.script.extract() for i in soup.find_all("script")]
- self.assertEqual("<body>\n\n<a></a>\n</body>", str(soup.body))
-
-
- def test_extract_works_when_element_is_surrounded_by_identical_strings(self):
- soup = self.soup(
- '<html>\n'
- '<body>hi</body>\n'
- '</html>')
- soup.find('body').extract()
- self.assertEqual(None, soup.find('body'))
-
-
- def test_clear(self):
- """Tag.clear()"""
- soup = self.soup("<p><a>String <em>Italicized</em></a> and another</p>")
- # clear using extract()
- a = soup.a
- soup.p.clear()
- self.assertEqual(len(soup.p.contents), 0)
- self.assertTrue(hasattr(a, "contents"))
-
- # clear using decompose()
- em = a.em
- a.clear(decompose=True)
- self.assertEqual(0, len(em.contents))
-
- def test_string_set(self):
- """Tag.string = 'string'"""
- soup = self.soup("<a></a> <b><c></c></b>")
- soup.a.string = "foo"
- self.assertEqual(soup.a.contents, ["foo"])
- soup.b.string = "bar"
- self.assertEqual(soup.b.contents, ["bar"])
-
- def test_string_set_does_not_affect_original_string(self):
- soup = self.soup("<a><b>foo</b><c>bar</c>")
- soup.b.string = soup.c.string
- self.assertEqual(soup.a.encode(), b"<a><b>bar</b><c>bar</c></a>")
-
- def test_set_string_preserves_class_of_string(self):
- soup = self.soup("<a></a>")
- cdata = CData("foo")
- soup.a.string = cdata
- self.assertTrue(isinstance(soup.a.string, CData))
-
-class TestElementObjects(SoupTest):
- """Test various features of element objects."""
-
- def test_len(self):
- """The length of an element is its number of children."""
- soup = self.soup("<top>1<b>2</b>3</top>")
-
- # The BeautifulSoup object itself contains one element: the
- # <top> tag.
- self.assertEqual(len(soup.contents), 1)
- self.assertEqual(len(soup), 1)
-
- # The <top> tag contains three elements: the text node "1", the
- # <b> tag, and the text node "3".
- self.assertEqual(len(soup.top), 3)
- self.assertEqual(len(soup.top.contents), 3)
-
- def test_member_access_invokes_find(self):
- """Accessing a Python member .foo invokes find('foo')"""
- soup = self.soup('<b><i></i></b>')
- self.assertEqual(soup.b, soup.find('b'))
- self.assertEqual(soup.b.i, soup.find('b').find('i'))
- self.assertEqual(soup.a, None)
-
- def test_deprecated_member_access(self):
- soup = self.soup('<b><i></i></b>')
- with warnings.catch_warnings(record=True) as w:
- tag = soup.bTag
- self.assertEqual(soup.b, tag)
- self.assertEqual(
- '.bTag is deprecated, use .find("b") instead.',
- str(w[0].message))
-
- def test_has_attr(self):
- """has_attr() checks for the presence of an attribute.
-
- Please note note: has_attr() is different from
- __in__. has_attr() checks the tag's attributes and __in__
- checks the tag's chidlren.
- """
- soup = self.soup("<foo attr='bar'>")
- self.assertTrue(soup.foo.has_attr('attr'))
- self.assertFalse(soup.foo.has_attr('attr2'))
-
-
- def test_attributes_come_out_in_alphabetical_order(self):
- markup = '<b a="1" z="5" m="3" f="2" y="4"></b>'
- self.assertSoupEquals(markup, '<b a="1" f="2" m="3" y="4" z="5"></b>')
-
- def test_string(self):
- # A tag that contains only a text node makes that node
- # available as .string.
- soup = self.soup("<b>foo</b>")
- self.assertEqual(soup.b.string, 'foo')
-
- def test_empty_tag_has_no_string(self):
- # A tag with no children has no .stirng.
- soup = self.soup("<b></b>")
- self.assertEqual(soup.b.string, None)
-
- def test_tag_with_multiple_children_has_no_string(self):
- # A tag with no children has no .string.
- soup = self.soup("<a>foo<b></b><b></b></b>")
- self.assertEqual(soup.b.string, None)
-
- soup = self.soup("<a>foo<b></b>bar</b>")
- self.assertEqual(soup.b.string, None)
-
- # Even if all the children are strings, due to trickery,
- # it won't work--but this would be a good optimization.
- soup = self.soup("<a>foo</b>")
- soup.a.insert(1, "bar")
- self.assertEqual(soup.a.string, None)
-
- def test_tag_with_recursive_string_has_string(self):
- # A tag with a single child which has a .string inherits that
- # .string.
- soup = self.soup("<a><b>foo</b></a>")
- self.assertEqual(soup.a.string, "foo")
- self.assertEqual(soup.string, "foo")
-
- def test_lack_of_string(self):
- """Only a tag containing a single text node has a .string."""
- soup = self.soup("<b>f<i>e</i>o</b>")
- self.assertFalse(soup.b.string)
-
- soup = self.soup("<b></b>")
- self.assertFalse(soup.b.string)
-
- def test_all_text(self):
- """Tag.text and Tag.get_text(sep=u"") -> all child text, concatenated"""
- soup = self.soup("<a>a<b>r</b> <r> t </r></a>")
- self.assertEqual(soup.a.text, "ar t ")
- self.assertEqual(soup.a.get_text(strip=True), "art")
- self.assertEqual(soup.a.get_text(","), "a,r, , t ")
- self.assertEqual(soup.a.get_text(",", strip=True), "a,r,t")
-
- def test_get_text_ignores_comments(self):
- soup = self.soup("foo<!--IGNORE-->bar")
- self.assertEqual(soup.get_text(), "foobar")
-
- self.assertEqual(
- soup.get_text(types=(NavigableString, Comment)), "fooIGNOREbar")
- self.assertEqual(
- soup.get_text(types=None), "fooIGNOREbar")
-
- def test_all_strings_ignores_comments(self):
- soup = self.soup("foo<!--IGNORE-->bar")
- self.assertEqual(['foo', 'bar'], list(soup.strings))
-
-class TestCDAtaListAttributes(SoupTest):
-
- """Testing cdata-list attributes like 'class'.
- """
- def test_single_value_becomes_list(self):
- soup = self.soup("<a class='foo'>")
- self.assertEqual(["foo"],soup.a['class'])
-
- def test_multiple_values_becomes_list(self):
- soup = self.soup("<a class='foo bar'>")
- self.assertEqual(["foo", "bar"], soup.a['class'])
-
- def test_multiple_values_separated_by_weird_whitespace(self):
- soup = self.soup("<a class='foo\tbar\nbaz'>")
- self.assertEqual(["foo", "bar", "baz"],soup.a['class'])
-
- def test_attributes_joined_into_string_on_output(self):
- soup = self.soup("<a class='foo\tbar'>")
- self.assertEqual(b'<a class="foo bar"></a>', soup.a.encode())
-
- def test_accept_charset(self):
- soup = self.soup('<form accept-charset="ISO-8859-1 UTF-8">')
- self.assertEqual(['ISO-8859-1', 'UTF-8'], soup.form['accept-charset'])
-
- def test_cdata_attribute_applying_only_to_one_tag(self):
- data = '<a accept-charset="ISO-8859-1 UTF-8"></a>'
- soup = self.soup(data)
- # We saw in another test that accept-charset is a cdata-list
- # attribute for the <form> tag. But it's not a cdata-list
- # attribute for any other tag.
- self.assertEqual('ISO-8859-1 UTF-8', soup.a['accept-charset'])
-
- def test_string_has_immutable_name_property(self):
- string = self.soup("s").string
- self.assertEqual(None, string.name)
- def t():
- string.name = 'foo'
- self.assertRaises(AttributeError, t)
-
-class TestPersistence(SoupTest):
- "Testing features like pickle and deepcopy."
-
- def setUp(self):
- super(TestPersistence, self).setUp()
- self.page = """<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
-"http://www.w3.org/TR/REC-html40/transitional.dtd">
-<html>
-<head>
-<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
-<title>Beautiful Soup: We called him Tortoise because he taught us.</title>
-<link rev="made" href="mailto:leonardr@segfault.org">
-<meta name="Description" content="Beautiful Soup: an HTML parser optimized for screen-scraping.">
-<meta name="generator" content="Markov Approximation 1.4 (module: leonardr)">
-<meta name="author" content="Leonard Richardson">
-</head>
-<body>
-<a href="foo">foo</a>
-<a href="foo"><b>bar</b></a>
-</body>
-</html>"""
- self.tree = self.soup(self.page)
-
- def test_pickle_and_unpickle_identity(self):
- # Pickling a tree, then unpickling it, yields a tree identical
- # to the original.
- dumped = pickle.dumps(self.tree, 2)
- loaded = pickle.loads(dumped)
- self.assertEqual(loaded.__class__, BeautifulSoup)
- self.assertEqual(loaded.decode(), self.tree.decode())
-
- def test_deepcopy_identity(self):
- # Making a deepcopy of a tree yields an identical tree.
- copied = copy.deepcopy(self.tree)
- self.assertEqual(copied.decode(), self.tree.decode())
-
- def test_unicode_pickle(self):
- # A tree containing Unicode characters can be pickled.
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- dumped = pickle.dumps(soup, pickle.HIGHEST_PROTOCOL)
- loaded = pickle.loads(dumped)
- self.assertEqual(loaded.decode(), soup.decode())
-
- def test_copy_navigablestring_is_not_attached_to_tree(self):
- html = "<b>Foo<a></a></b><b>Bar</b>"
- soup = self.soup(html)
- s1 = soup.find(string="Foo")
- s2 = copy.copy(s1)
- self.assertEqual(s1, s2)
- self.assertEqual(None, s2.parent)
- self.assertEqual(None, s2.next_element)
- self.assertNotEqual(None, s1.next_sibling)
- self.assertEqual(None, s2.next_sibling)
- self.assertEqual(None, s2.previous_element)
-
- def test_copy_navigablestring_subclass_has_same_type(self):
- html = "<b><!--Foo--></b>"
- soup = self.soup(html)
- s1 = soup.string
- s2 = copy.copy(s1)
- self.assertEqual(s1, s2)
- self.assertTrue(isinstance(s2, Comment))
-
- def test_copy_entire_soup(self):
- html = "<div><b>Foo<a></a></b><b>Bar</b></div>end"
- soup = self.soup(html)
- soup_copy = copy.copy(soup)
- self.assertEqual(soup, soup_copy)
-
- def test_copy_tag_copies_contents(self):
- html = "<div><b>Foo<a></a></b><b>Bar</b></div>end"
- soup = self.soup(html)
- div = soup.div
- div_copy = copy.copy(div)
-
- # The two tags look the same, and evaluate to equal.
- self.assertEqual(str(div), str(div_copy))
- self.assertEqual(div, div_copy)
-
- # But they're not the same object.
- self.assertFalse(div is div_copy)
-
- # And they don't have the same relation to the parse tree. The
- # copy is not associated with a parse tree at all.
- self.assertEqual(None, div_copy.parent)
- self.assertEqual(None, div_copy.previous_element)
- self.assertEqual(None, div_copy.find(string='Bar').next_element)
- self.assertNotEqual(None, div.find(string='Bar').next_element)
-
-class TestSubstitutions(SoupTest):
-
- def test_default_formatter_is_minimal(self):
- markup = "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"
- soup = self.soup(markup)
- decoded = soup.decode(formatter="minimal")
- # The < is converted back into < but the e-with-acute is left alone.
- self.assertEqual(
- decoded,
- self.document_for(
- "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"))
-
- def test_formatter_html(self):
- markup = "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"
- soup = self.soup(markup)
- decoded = soup.decode(formatter="html")
- self.assertEqual(
- decoded,
- self.document_for("<b><<Sacré bleu!>></b>"))
-
- def test_formatter_minimal(self):
- markup = "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"
- soup = self.soup(markup)
- decoded = soup.decode(formatter="minimal")
- # The < is converted back into < but the e-with-acute is left alone.
- self.assertEqual(
- decoded,
- self.document_for(
- "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"))
-
- def test_formatter_null(self):
- markup = "<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"
- soup = self.soup(markup)
- decoded = soup.decode(formatter=None)
- # Neither the angle brackets nor the e-with-acute are converted.
- # This is not valid HTML, but it's what the user wanted.
- self.assertEqual(decoded,
- self.document_for("<b><<Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!>></b>"))
-
- def test_formatter_custom(self):
- markup = "<b><foo></b><b>bar</b>"
- soup = self.soup(markup)
- decoded = soup.decode(formatter = lambda x: x.upper())
- # Instead of normal entity conversion code, the custom
- # callable is called on every string.
- self.assertEqual(
- decoded,
- self.document_for("<b><FOO></b><b>BAR</b>"))
-
- def test_formatter_is_run_on_attribute_values(self):
- markup = '<a href="http://a.com?a=b&c=é">e</a>'
- soup = self.soup(markup)
- a = soup.a
-
- expect_minimal = '<a href="http://a.com?a=b&c=é">e</a>'
-
- self.assertEqual(expect_minimal, a.decode())
- self.assertEqual(expect_minimal, a.decode(formatter="minimal"))
-
- expect_html = '<a href="http://a.com?a=b&c=é">e</a>'
- self.assertEqual(expect_html, a.decode(formatter="html"))
-
- self.assertEqual(markup, a.decode(formatter=None))
- expect_upper = '<a href="HTTP://A.COM?A=B&C=É">E</a>'
- self.assertEqual(expect_upper, a.decode(formatter=lambda x: x.upper()))
-
- def test_formatter_skips_script_tag_for_html_documents(self):
- doc = """
- <script type="text/javascript">
- console.log("< < hey > > ");
- </script>
-"""
- encoded = BeautifulSoup(doc, 'html.parser').encode()
- self.assertTrue(b"< < hey > >" in encoded)
-
- def test_formatter_skips_style_tag_for_html_documents(self):
- doc = """
- <style type="text/css">
- console.log("< < hey > > ");
- </style>
-"""
- encoded = BeautifulSoup(doc, 'html.parser').encode()
- self.assertTrue(b"< < hey > >" in encoded)
-
- def test_prettify_leaves_preformatted_text_alone(self):
- soup = self.soup("<div> foo <pre> \tbar\n \n </pre> baz ")
- # Everything outside the <pre> tag is reformatted, but everything
- # inside is left alone.
- self.assertEqual(
- '<div>\n foo\n <pre> \tbar\n \n </pre>\n baz\n</div>',
- soup.div.prettify())
-
- def test_prettify_accepts_formatter(self):
- soup = BeautifulSoup("<html><body>foo</body></html>", 'html.parser')
- pretty = soup.prettify(formatter = lambda x: x.upper())
- self.assertTrue("FOO" in pretty)
-
- def test_prettify_outputs_unicode_by_default(self):
- soup = self.soup("<a></a>")
- self.assertEqual(str, type(soup.prettify()))
-
- def test_prettify_can_encode_data(self):
- soup = self.soup("<a></a>")
- self.assertEqual(bytes, type(soup.prettify("utf-8")))
-
- def test_html_entity_substitution_off_by_default(self):
- markup = "<b>Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!</b>"
- soup = self.soup(markup)
- encoded = soup.b.encode("utf-8")
- self.assertEqual(encoded, markup.encode('utf-8'))
-
- def test_encoding_substitution(self):
- # Here's the <meta> tag saying that a document is
- # encoded in Shift-JIS.
- meta_tag = ('<meta content="text/html; charset=x-sjis" '
- 'http-equiv="Content-type"/>')
- soup = self.soup(meta_tag)
-
- # Parse the document, and the charset apprears unchanged.
- self.assertEqual(soup.meta['content'], 'text/html; charset=x-sjis')
-
- # Encode the document into some encoding, and the encoding is
- # substituted into the meta tag.
- utf_8 = soup.encode("utf-8")
- self.assertTrue(b"charset=utf-8" in utf_8)
-
- euc_jp = soup.encode("euc_jp")
- self.assertTrue(b"charset=euc_jp" in euc_jp)
-
- shift_jis = soup.encode("shift-jis")
- self.assertTrue(b"charset=shift-jis" in shift_jis)
-
- utf_16_u = soup.encode("utf-16").decode("utf-16")
- self.assertTrue("charset=utf-16" in utf_16_u)
-
- def test_encoding_substitution_doesnt_happen_if_tag_is_strained(self):
- markup = ('<head><meta content="text/html; charset=x-sjis" '
- 'http-equiv="Content-type"/></head><pre>foo</pre>')
-
- # Beautiful Soup used to try to rewrite the meta tag even if the
- # meta tag got filtered out by the strainer. This test makes
- # sure that doesn't happen.
- strainer = SoupStrainer('pre')
- soup = self.soup(markup, parse_only=strainer)
- self.assertEqual(soup.contents[0].name, 'pre')
-
-class TestEncoding(SoupTest):
- """Test the ability to encode objects into strings."""
-
- def test_unicode_string_can_be_encoded(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual(soup.b.string.encode("utf-8"),
- "\N{SNOWMAN}".encode("utf-8"))
-
- def test_tag_containing_unicode_string_can_be_encoded(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual(
- soup.b.encode("utf-8"), html.encode("utf-8"))
-
- def test_encoding_substitutes_unrecognized_characters_by_default(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual(soup.b.encode("ascii"), b"<b>☃</b>")
-
- def test_encoding_can_be_made_strict(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertRaises(
- UnicodeEncodeError, soup.encode, "ascii", errors="strict")
-
- def test_decode_contents(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual("\N{SNOWMAN}", soup.b.decode_contents())
-
- def test_encode_contents(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual(
- "\N{SNOWMAN}".encode("utf8"), soup.b.encode_contents(
- encoding="utf8"))
-
- def test_deprecated_renderContents(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- self.assertEqual(
- "\N{SNOWMAN}".encode("utf8"), soup.b.renderContents())
-
- def test_repr(self):
- html = "<b>\N{SNOWMAN}</b>"
- soup = self.soup(html)
- if PY3K:
- self.assertEqual(html, repr(soup))
- else:
- self.assertEqual(b'<b>\\u2603</b>', repr(soup))
-
-class TestNavigableStringSubclasses(SoupTest):
-
- def test_cdata(self):
- # None of the current builders turn CDATA sections into CData
- # objects, but you can create them manually.
- soup = self.soup("")
- cdata = CData("foo")
- soup.insert(1, cdata)
- self.assertEqual(str(soup), "<![CDATA[foo]]>")
- self.assertEqual(soup.find(text="foo"), "foo")
- self.assertEqual(soup.contents[0], "foo")
-
- def test_cdata_is_never_formatted(self):
- """Text inside a CData object is passed into the formatter.
-
- But the return value is ignored.
- """
-
- self.count = 0
- def increment(*args):
- self.count += 1
- return "BITTER FAILURE"
-
- soup = self.soup("")
- cdata = CData("<><><>")
- soup.insert(1, cdata)
- self.assertEqual(
- b"<![CDATA[<><><>]]>", soup.encode(formatter=increment))
- self.assertEqual(1, self.count)
-
- def test_doctype_ends_in_newline(self):
- # Unlike other NavigableString subclasses, a DOCTYPE always ends
- # in a newline.
- doctype = Doctype("foo")
- soup = self.soup("")
- soup.insert(1, doctype)
- self.assertEqual(soup.encode(), b"<!DOCTYPE foo>\n")
-
- def test_declaration(self):
- d = Declaration("foo")
- self.assertEqual("<?foo?>", d.output_ready())
-
-class TestSoupSelector(TreeTest):
-
- HTML = """
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
-"http://www.w3.org/TR/html4/strict.dtd">
-<html>
-<head>
-<title>The title</title>
-<link rel="stylesheet" href="blah.css" type="text/css" id="l1">
-</head>
-<body>
-<custom-dashed-tag class="dashed" id="dash1">Hello there.</custom-dashed-tag>
-<div id="main" class="fancy">
-<div id="inner">
-<h1 id="header1">An H1</h1>
-<p>Some text</p>
-<p class="onep" id="p1">Some more text</p>
-<h2 id="header2">An H2</h2>
-<p class="class1 class2 class3" id="pmulti">Another</p>
-<a href="http://bob.example.org/" rel="friend met" id="bob">Bob</a>
-<h2 id="header3">Another H2</h2>
-<a id="me" href="http://simonwillison.net/" rel="me">me</a>
-<span class="s1">
-<a href="#" id="s1a1">span1a1</a>
-<a href="#" id="s1a2">span1a2 <span id="s1a2s1">test</span></a>
-<span class="span2">
-<a href="#" id="s2a1">span2a1</a>
-</span>
-<span class="span3"></span>
-<custom-dashed-tag class="dashed" id="dash2"/>
-<div data-tag="dashedvalue" id="data1"/>
-</span>
-</div>
-<x id="xid">
-<z id="zida"/>
-<z id="zidab"/>
-<z id="zidac"/>
-</x>
-<y id="yid">
-<z id="zidb"/>
-</y>
-<p lang="en" id="lang-en">English</p>
-<p lang="en-gb" id="lang-en-gb">English UK</p>
-<p lang="en-us" id="lang-en-us">English US</p>
-<p lang="fr" id="lang-fr">French</p>
-</div>
-
-<div id="footer">
-</div>
-"""
-
- def setUp(self):
- self.soup = BeautifulSoup(self.HTML, 'html.parser')
-
- def assertSelects(self, selector, expected_ids):
- el_ids = [el['id'] for el in self.soup.select(selector)]
- el_ids.sort()
- expected_ids.sort()
- self.assertEqual(expected_ids, el_ids,
- "Selector %s, expected [%s], got [%s]" % (
- selector, ', '.join(expected_ids), ', '.join(el_ids)
- )
- )
-
- assertSelect = assertSelects
-
- def assertSelectMultiple(self, *tests):
- for selector, expected_ids in tests:
- self.assertSelect(selector, expected_ids)
-
- def test_one_tag_one(self):
- els = self.soup.select('title')
- self.assertEqual(len(els), 1)
- self.assertEqual(els[0].name, 'title')
- self.assertEqual(els[0].contents, ['The title'])
-
- def test_one_tag_many(self):
- els = self.soup.select('div')
- self.assertEqual(len(els), 4)
- for div in els:
- self.assertEqual(div.name, 'div')
-
- el = self.soup.select_one('div')
- self.assertEqual('main', el['id'])
-
- def test_select_one_returns_none_if_no_match(self):
- match = self.soup.select_one('nonexistenttag')
- self.assertEqual(None, match)
-
-
- def test_tag_in_tag_one(self):
- els = self.soup.select('div div')
- self.assertSelects('div div', ['inner', 'data1'])
-
- def test_tag_in_tag_many(self):
- for selector in ('html div', 'html body div', 'body div'):
- self.assertSelects(selector, ['data1', 'main', 'inner', 'footer'])
-
- def test_tag_no_match(self):
- self.assertEqual(len(self.soup.select('del')), 0)
-
- def test_invalid_tag(self):
- self.assertRaises(ValueError, self.soup.select, 'tag%t')
-
- def test_select_dashed_tag_ids(self):
- self.assertSelects('custom-dashed-tag', ['dash1', 'dash2'])
-
- def test_select_dashed_by_id(self):
- dashed = self.soup.select('custom-dashed-tag[id=\"dash2\"]')
- self.assertEqual(dashed[0].name, 'custom-dashed-tag')
- self.assertEqual(dashed[0]['id'], 'dash2')
-
- def test_dashed_tag_text(self):
- self.assertEqual(self.soup.select('body > custom-dashed-tag')[0].text, 'Hello there.')
-
- def test_select_dashed_matches_find_all(self):
- self.assertEqual(self.soup.select('custom-dashed-tag'), self.soup.find_all('custom-dashed-tag'))
-
- def test_header_tags(self):
- self.assertSelectMultiple(
- ('h1', ['header1']),
- ('h2', ['header2', 'header3']),
- )
-
- def test_class_one(self):
- for selector in ('.onep', 'p.onep', 'html p.onep'):
- els = self.soup.select(selector)
- self.assertEqual(len(els), 1)
- self.assertEqual(els[0].name, 'p')
- self.assertEqual(els[0]['class'], ['onep'])
-
- def test_class_mismatched_tag(self):
- els = self.soup.select('div.onep')
- self.assertEqual(len(els), 0)
-
- def test_one_id(self):
- for selector in ('div#inner', '#inner', 'div div#inner'):
- self.assertSelects(selector, ['inner'])
-
- def test_bad_id(self):
- els = self.soup.select('#doesnotexist')
- self.assertEqual(len(els), 0)
-
- def test_items_in_id(self):
- els = self.soup.select('div#inner p')
- self.assertEqual(len(els), 3)
- for el in els:
- self.assertEqual(el.name, 'p')
- self.assertEqual(els[1]['class'], ['onep'])
- self.assertFalse(els[0].has_attr('class'))
-
- def test_a_bunch_of_emptys(self):
- for selector in ('div#main del', 'div#main div.oops', 'div div#main'):
- self.assertEqual(len(self.soup.select(selector)), 0)
-
- def test_multi_class_support(self):
- for selector in ('.class1', 'p.class1', '.class2', 'p.class2',
- '.class3', 'p.class3', 'html p.class2', 'div#inner .class2'):
- self.assertSelects(selector, ['pmulti'])
-
- def test_multi_class_selection(self):
- for selector in ('.class1.class3', '.class3.class2',
- '.class1.class2.class3'):
- self.assertSelects(selector, ['pmulti'])
-
- def test_child_selector(self):
- self.assertSelects('.s1 > a', ['s1a1', 's1a2'])
- self.assertSelects('.s1 > a span', ['s1a2s1'])
-
- def test_child_selector_id(self):
- self.assertSelects('.s1 > a#s1a2 span', ['s1a2s1'])
-
- def test_attribute_equals(self):
- self.assertSelectMultiple(
- ('p[class="onep"]', ['p1']),
- ('p[id="p1"]', ['p1']),
- ('[class="onep"]', ['p1']),
- ('[id="p1"]', ['p1']),
- ('link[rel="stylesheet"]', ['l1']),
- ('link[type="text/css"]', ['l1']),
- ('link[href="blah.css"]', ['l1']),
- ('link[href="no-blah.css"]', []),
- ('[rel="stylesheet"]', ['l1']),
- ('[type="text/css"]', ['l1']),
- ('[href="blah.css"]', ['l1']),
- ('[href="no-blah.css"]', []),
- ('p[href="no-blah.css"]', []),
- ('[href="no-blah.css"]', []),
- )
-
- def test_attribute_tilde(self):
- self.assertSelectMultiple(
- ('p[class~="class1"]', ['pmulti']),
- ('p[class~="class2"]', ['pmulti']),
- ('p[class~="class3"]', ['pmulti']),
- ('[class~="class1"]', ['pmulti']),
- ('[class~="class2"]', ['pmulti']),
- ('[class~="class3"]', ['pmulti']),
- ('a[rel~="friend"]', ['bob']),
- ('a[rel~="met"]', ['bob']),
- ('[rel~="friend"]', ['bob']),
- ('[rel~="met"]', ['bob']),
- )
-
- def test_attribute_startswith(self):
- self.assertSelectMultiple(
- ('[rel^="style"]', ['l1']),
- ('link[rel^="style"]', ['l1']),
- ('notlink[rel^="notstyle"]', []),
- ('[rel^="notstyle"]', []),
- ('link[rel^="notstyle"]', []),
- ('link[href^="bla"]', ['l1']),
- ('a[href^="http://"]', ['bob', 'me']),
- ('[href^="http://"]', ['bob', 'me']),
- ('[id^="p"]', ['pmulti', 'p1']),
- ('[id^="m"]', ['me', 'main']),
- ('div[id^="m"]', ['main']),
- ('a[id^="m"]', ['me']),
- ('div[data-tag^="dashed"]', ['data1'])
- )
-
- def test_attribute_endswith(self):
- self.assertSelectMultiple(
- ('[href$=".css"]', ['l1']),
- ('link[href$=".css"]', ['l1']),
- ('link[id$="1"]', ['l1']),
- ('[id$="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's2a1', 's1a2s1', 'dash1']),
- ('div[id$="1"]', ['data1']),
- ('[id$="noending"]', []),
- )
-
- def test_attribute_contains(self):
- self.assertSelectMultiple(
- # From test_attribute_startswith
- ('[rel*="style"]', ['l1']),
- ('link[rel*="style"]', ['l1']),
- ('notlink[rel*="notstyle"]', []),
- ('[rel*="notstyle"]', []),
- ('link[rel*="notstyle"]', []),
- ('link[href*="bla"]', ['l1']),
- ('[href*="http://"]', ['bob', 'me']),
- ('[id*="p"]', ['pmulti', 'p1']),
- ('div[id*="m"]', ['main']),
- ('a[id*="m"]', ['me']),
- # From test_attribute_endswith
- ('[href*=".css"]', ['l1']),
- ('link[href*=".css"]', ['l1']),
- ('link[id*="1"]', ['l1']),
- ('[id*="1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's1a2', 's2a1', 's1a2s1', 'dash1']),
- ('div[id*="1"]', ['data1']),
- ('[id*="noending"]', []),
- # New for this test
- ('[href*="."]', ['bob', 'me', 'l1']),
- ('a[href*="."]', ['bob', 'me']),
- ('link[href*="."]', ['l1']),
- ('div[id*="n"]', ['main', 'inner']),
- ('div[id*="nn"]', ['inner']),
- ('div[data-tag*="edval"]', ['data1'])
- )
-
- def test_attribute_exact_or_hypen(self):
- self.assertSelectMultiple(
- ('p[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
- ('[lang|="en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
- ('p[lang|="fr"]', ['lang-fr']),
- ('p[lang|="gb"]', []),
- )
-
- def test_attribute_exists(self):
- self.assertSelectMultiple(
- ('[rel]', ['l1', 'bob', 'me']),
- ('link[rel]', ['l1']),
- ('a[rel]', ['bob', 'me']),
- ('[lang]', ['lang-en', 'lang-en-gb', 'lang-en-us', 'lang-fr']),
- ('p[class]', ['p1', 'pmulti']),
- ('[blah]', []),
- ('p[blah]', []),
- ('div[data-tag]', ['data1'])
- )
-
- def test_unsupported_pseudoclass(self):
- self.assertRaises(
- NotImplementedError, self.soup.select, "a:no-such-pseudoclass")
-
- self.assertRaises(
- NotImplementedError, self.soup.select, "a:nth-of-type(a)")
-
-
- def test_nth_of_type(self):
- # Try to select first paragraph
- els = self.soup.select('div#inner p:nth-of-type(1)')
- self.assertEqual(len(els), 1)
- self.assertEqual(els[0].string, 'Some text')
-
- # Try to select third paragraph
- els = self.soup.select('div#inner p:nth-of-type(3)')
- self.assertEqual(len(els), 1)
- self.assertEqual(els[0].string, 'Another')
-
- # Try to select (non-existent!) fourth paragraph
- els = self.soup.select('div#inner p:nth-of-type(4)')
- self.assertEqual(len(els), 0)
-
- # Pass in an invalid value.
- self.assertRaises(
- ValueError, self.soup.select, 'div p:nth-of-type(0)')
-
- def test_nth_of_type_direct_descendant(self):
- els = self.soup.select('div#inner > p:nth-of-type(1)')
- self.assertEqual(len(els), 1)
- self.assertEqual(els[0].string, 'Some text')
-
- def test_id_child_selector_nth_of_type(self):
- self.assertSelects('#inner > p:nth-of-type(2)', ['p1'])
-
- def test_select_on_element(self):
- # Other tests operate on the tree; this operates on an element
- # within the tree.
- inner = self.soup.find("div", id="main")
- selected = inner.select("div")
- # The <div id="inner"> tag was selected. The <div id="footer">
- # tag was not.
- self.assertSelectsIDs(selected, ['inner', 'data1'])
-
- def test_overspecified_child_id(self):
- self.assertSelects(".fancy #inner", ['inner'])
- self.assertSelects(".normal #inner", [])
-
- def test_adjacent_sibling_selector(self):
- self.assertSelects('#p1 + h2', ['header2'])
- self.assertSelects('#p1 + h2 + p', ['pmulti'])
- self.assertSelects('#p1 + #header2 + .class1', ['pmulti'])
- self.assertEqual([], self.soup.select('#p1 + p'))
-
- def test_general_sibling_selector(self):
- self.assertSelects('#p1 ~ h2', ['header2', 'header3'])
- self.assertSelects('#p1 ~ #header2', ['header2'])
- self.assertSelects('#p1 ~ h2 + a', ['me'])
- self.assertSelects('#p1 ~ h2 + [rel="me"]', ['me'])
- self.assertEqual([], self.soup.select('#inner ~ h2'))
-
- def test_dangling_combinator(self):
- self.assertRaises(ValueError, self.soup.select, 'h1 >')
-
- def test_sibling_combinator_wont_select_same_tag_twice(self):
- self.assertSelects('p[lang] ~ p', ['lang-en-gb', 'lang-en-us', 'lang-fr'])
-
- # Test the selector grouping operator (the comma)
- def test_multiple_select(self):
- self.assertSelects('x, y', ['xid', 'yid'])
-
- def test_multiple_select_with_no_space(self):
- self.assertSelects('x,y', ['xid', 'yid'])
-
- def test_multiple_select_with_more_space(self):
- self.assertSelects('x, y', ['xid', 'yid'])
-
- def test_multiple_select_duplicated(self):
- self.assertSelects('x, x', ['xid'])
-
- def test_multiple_select_sibling(self):
- self.assertSelects('x, y ~ p[lang=fr]', ['xid', 'lang-fr'])
-
- def test_multiple_select_tag_and_direct_descendant(self):
- self.assertSelects('x, y > z', ['xid', 'zidb'])
-
- def test_multiple_select_direct_descendant_and_tags(self):
- self.assertSelects('div > x, y, z', ['xid', 'yid', 'zida', 'zidb', 'zidab', 'zidac'])
-
- def test_multiple_select_indirect_descendant(self):
- self.assertSelects('div x,y, z', ['xid', 'yid', 'zida', 'zidb', 'zidab', 'zidac'])
-
- def test_invalid_multiple_select(self):
- self.assertRaises(ValueError, self.soup.select, ',x, y')
- self.assertRaises(ValueError, self.soup.select, 'x,,y')
-
- def test_multiple_select_attrs(self):
- self.assertSelects('p[lang=en], p[lang=en-gb]', ['lang-en', 'lang-en-gb'])
-
- def test_multiple_select_ids(self):
- self.assertSelects('x, y > z[id=zida], z[id=zidab], z[id=zidb]', ['xid', 'zidb', 'zidab'])
-
- def test_multiple_select_nested(self):
- self.assertSelects('body > div > x, y > z', ['xid', 'zidb'])
-
-
-
diff --git a/bitbake/lib/codegen.py b/bitbake/lib/codegen.py
deleted file mode 100644
index 018b2831..00000000
--- a/bitbake/lib/codegen.py
+++ /dev/null
@@ -1,564 +0,0 @@
-# -*- coding: utf-8 -*-
-"""
- codegen
- ~~~~~~~
-
- Extension to ast that allow ast -> python code generation.
-
- :copyright: Copyright 2008 by Armin Ronacher.
- :license: BSD.
-"""
-from ast import *
-
-BOOLOP_SYMBOLS = {
- And: 'and',
- Or: 'or'
-}
-
-BINOP_SYMBOLS = {
- Add: '+',
- Sub: '-',
- Mult: '*',
- Div: '/',
- FloorDiv: '//',
- Mod: '%',
- LShift: '<<',
- RShift: '>>',
- BitOr: '|',
- BitAnd: '&',
- BitXor: '^'
-}
-
-CMPOP_SYMBOLS = {
- Eq: '==',
- Gt: '>',
- GtE: '>=',
- In: 'in',
- Is: 'is',
- IsNot: 'is not',
- Lt: '<',
- LtE: '<=',
- NotEq: '!=',
- NotIn: 'not in'
-}
-
-UNARYOP_SYMBOLS = {
- Invert: '~',
- Not: 'not',
- UAdd: '+',
- USub: '-'
-}
-
-ALL_SYMBOLS = {}
-ALL_SYMBOLS.update(BOOLOP_SYMBOLS)
-ALL_SYMBOLS.update(BINOP_SYMBOLS)
-ALL_SYMBOLS.update(CMPOP_SYMBOLS)
-ALL_SYMBOLS.update(UNARYOP_SYMBOLS)
-
-def to_source(node, indent_with=' ' * 4, add_line_information=False):
- """This function can convert a node tree back into python sourcecode.
- This is useful for debugging purposes, especially if you're dealing with
- custom asts not generated by python itself.
-
- It could be that the sourcecode is evaluable when the AST itself is not
- compilable / evaluable. The reason for this is that the AST contains some
- more data than regular sourcecode does, which is dropped during
- conversion.
-
- Each level of indentation is replaced with `indent_with`. Per default this
- parameter is equal to four spaces as suggested by PEP 8, but it might be
- adjusted to match the application's styleguide.
-
- If `add_line_information` is set to `True` comments for the line numbers
- of the nodes are added to the output. This can be used to spot wrong line
- number information of statement nodes.
- """
- generator = SourceGenerator(indent_with, add_line_information)
- generator.visit(node)
- return ''.join(generator.result)
-
-
-class SourceGenerator(NodeVisitor):
- """This visitor is able to transform a well formed syntax tree into python
- sourcecode. For more details have a look at the docstring of the
- `node_to_source` function.
- """
-
- def __init__(self, indent_with, add_line_information=False):
- self.result = []
- self.indent_with = indent_with
- self.add_line_information = add_line_information
- self.indentation = 0
- self.new_lines = 0
-
- def write(self, x):
- if self.new_lines:
- if self.result:
- self.result.append('\n' * self.new_lines)
- self.result.append(self.indent_with * self.indentation)
- self.new_lines = 0
- self.result.append(x)
-
- def newline(self, node=None, extra=0):
- self.new_lines = max(self.new_lines, 1 + extra)
- if node is not None and self.add_line_information:
- self.write('# line: %s' % node.lineno)
- self.new_lines = 1
-
- def body(self, statements):
- self.new_line = True
- self.indentation += 1
- for stmt in statements:
- self.visit(stmt)
- self.indentation -= 1
-
- def body_or_else(self, node):
- self.body(node.body)
- if node.orelse:
- self.newline()
- self.write('else:')
- self.body(node.orelse)
-
- def signature(self, node):
- want_comma = []
- def write_comma():
- if want_comma:
- self.write(', ')
- else:
- want_comma.append(True)
-
- padding = [None] * (len(node.args) - len(node.defaults))
- for arg, default in zip(node.args, padding + node.defaults):
- write_comma()
- self.visit(arg)
- if default is not None:
- self.write('=')
- self.visit(default)
- if node.vararg is not None:
- write_comma()
- self.write('*' + node.vararg)
- if node.kwarg is not None:
- write_comma()
- self.write('**' + node.kwarg)
-
- def decorators(self, node):
- for decorator in node.decorator_list:
- self.newline(decorator)
- self.write('@')
- self.visit(decorator)
-
- # Statements
-
- def visit_Assign(self, node):
- self.newline(node)
- for idx, target in enumerate(node.targets):
- if idx:
- self.write(', ')
- self.visit(target)
- self.write(' = ')
- self.visit(node.value)
-
- def visit_AugAssign(self, node):
- self.newline(node)
- self.visit(node.target)
- self.write(BINOP_SYMBOLS[type(node.op)] + '=')
- self.visit(node.value)
-
- def visit_ImportFrom(self, node):
- self.newline(node)
- self.write('from %s%s import ' % ('.' * node.level, node.module))
- for idx, item in enumerate(node.names):
- if idx:
- self.write(', ')
- self.write(item)
-
- def visit_Import(self, node):
- self.newline(node)
- for item in node.names:
- self.write('import ')
- self.visit(item)
-
- def visit_Expr(self, node):
- self.newline(node)
- self.generic_visit(node)
-
- def visit_FunctionDef(self, node):
- self.newline(extra=1)
- self.decorators(node)
- self.newline(node)
- self.write('def %s(' % node.name)
- self.signature(node.args)
- self.write('):')
- self.body(node.body)
-
- def visit_ClassDef(self, node):
- have_args = []
- def paren_or_comma():
- if have_args:
- self.write(', ')
- else:
- have_args.append(True)
- self.write('(')
-
- self.newline(extra=2)
- self.decorators(node)
- self.newline(node)
- self.write('class %s' % node.name)
- for base in node.bases:
- paren_or_comma()
- self.visit(base)
- # XXX: the if here is used to keep this module compatible
- # with python 2.6.
- if hasattr(node, 'keywords'):
- for keyword in node.keywords:
- paren_or_comma()
- self.write(keyword.arg + '=')
- self.visit(keyword.value)
- if hasattr(node, 'starargs') and node.starargs is not None:
- paren_or_comma()
- self.write('*')
- self.visit(node.starargs)
- if hasattr(node, 'kwargs') and node.kwargs is not None:
- paren_or_comma()
- self.write('**')
- self.visit(node.kwargs)
- self.write(have_args and '):' or ':')
- self.body(node.body)
-
- def visit_If(self, node):
- self.newline(node)
- self.write('if ')
- self.visit(node.test)
- self.write(':')
- self.body(node.body)
- while True:
- else_ = node.orelse
- if len(else_) == 1 and isinstance(else_[0], If):
- node = else_[0]
- self.newline()
- self.write('elif ')
- self.visit(node.test)
- self.write(':')
- self.body(node.body)
- else:
- self.newline()
- self.write('else:')
- self.body(else_)
- break
-
- def visit_For(self, node):
- self.newline(node)
- self.write('for ')
- self.visit(node.target)
- self.write(' in ')
- self.visit(node.iter)
- self.write(':')
- self.body_or_else(node)
-
- def visit_While(self, node):
- self.newline(node)
- self.write('while ')
- self.visit(node.test)
- self.write(':')
- self.body_or_else(node)
-
- def visit_With(self, node):
- self.newline(node)
- self.write('with ')
- self.visit(node.context_expr)
- if node.optional_vars is not None:
- self.write(' as ')
- self.visit(node.optional_vars)
- self.write(':')
- self.body(node.body)
-
- def visit_Pass(self, node):
- self.newline(node)
- self.write('pass')
-
- def visit_Print(self, node):
- # XXX: python 2.6 only
- self.newline(node)
- self.write('print ')
- want_comma = False
- if node.dest is not None:
- self.write(' >> ')
- self.visit(node.dest)
- want_comma = True
- for value in node.values:
- if want_comma:
- self.write(', ')
- self.visit(value)
- want_comma = True
- if not node.nl:
- self.write(',')
-
- def visit_Delete(self, node):
- self.newline(node)
- self.write('del ')
- for idx, target in enumerate(node):
- if idx:
- self.write(', ')
- self.visit(target)
-
- def visit_TryExcept(self, node):
- self.newline(node)
- self.write('try:')
- self.body(node.body)
- for handler in node.handlers:
- self.visit(handler)
-
- def visit_TryFinally(self, node):
- self.newline(node)
- self.write('try:')
- self.body(node.body)
- self.newline(node)
- self.write('finally:')
- self.body(node.finalbody)
-
- def visit_Global(self, node):
- self.newline(node)
- self.write('global ' + ', '.join(node.names))
-
- def visit_Nonlocal(self, node):
- self.newline(node)
- self.write('nonlocal ' + ', '.join(node.names))
-
- def visit_Return(self, node):
- self.newline(node)
- self.write('return ')
- self.visit(node.value)
-
- def visit_Break(self, node):
- self.newline(node)
- self.write('break')
-
- def visit_Continue(self, node):
- self.newline(node)
- self.write('continue')
-
- def visit_Raise(self, node):
- # XXX: Python 2.6 / 3.0 compatibility
- self.newline(node)
- self.write('raise')
- if hasattr(node, 'exc') and node.exc is not None:
- self.write(' ')
- self.visit(node.exc)
- if node.cause is not None:
- self.write(' from ')
- self.visit(node.cause)
- elif hasattr(node, 'type') and node.type is not None:
- self.visit(node.type)
- if node.inst is not None:
- self.write(', ')
- self.visit(node.inst)
- if node.tback is not None:
- self.write(', ')
- self.visit(node.tback)
-
- # Expressions
-
- def visit_Attribute(self, node):
- self.visit(node.value)
- self.write('.' + node.attr)
-
- def visit_Call(self, node):
- want_comma = []
- def write_comma():
- if want_comma:
- self.write(', ')
- else:
- want_comma.append(True)
-
- self.visit(node.func)
- self.write('(')
- for arg in node.args:
- write_comma()
- self.visit(arg)
- for keyword in node.keywords:
- write_comma()
- self.write(keyword.arg + '=')
- self.visit(keyword.value)
- if hasattr(node, 'starargs') and node.starargs is not None:
- write_comma()
- self.write('*')
- self.visit(node.starargs)
- if hasattr(node, 'kwargs') and node.kwargs is not None:
- write_comma()
- self.write('**')
- self.visit(node.kwargs)
- self.write(')')
-
- def visit_Name(self, node):
- self.write(node.id)
-
- def visit_Constant(self, node):
- self.write(repr(node.value))
-
- def visit_Tuple(self, node):
- self.write('(')
- idx = -1
- for idx, item in enumerate(node.elts):
- if idx:
- self.write(', ')
- self.visit(item)
- self.write(idx and ')' or ',)')
-
- def sequence_visit(left, right):
- def visit(self, node):
- self.write(left)
- for idx, item in enumerate(node.elts):
- if idx:
- self.write(', ')
- self.visit(item)
- self.write(right)
- return visit
-
- visit_List = sequence_visit('[', ']')
- visit_Set = sequence_visit('{', '}')
- del sequence_visit
-
- def visit_Dict(self, node):
- self.write('{')
- for idx, (key, value) in enumerate(zip(node.keys, node.values)):
- if idx:
- self.write(', ')
- self.visit(key)
- self.write(': ')
- self.visit(value)
- self.write('}')
-
- def visit_BinOp(self, node):
- self.visit(node.left)
- self.write(' %s ' % BINOP_SYMBOLS[type(node.op)])
- self.visit(node.right)
-
- def visit_BoolOp(self, node):
- self.write('(')
- for idx, value in enumerate(node.values):
- if idx:
- self.write(' %s ' % BOOLOP_SYMBOLS[type(node.op)])
- self.visit(value)
- self.write(')')
-
- def visit_Compare(self, node):
- self.write('(')
- self.write(node.left)
- for op, right in zip(node.ops, node.comparators):
- self.write(' %s %%' % CMPOP_SYMBOLS[type(op)])
- self.visit(right)
- self.write(')')
-
- def visit_UnaryOp(self, node):
- self.write('(')
- op = UNARYOP_SYMBOLS[type(node.op)]
- self.write(op)
- if op == 'not':
- self.write(' ')
- self.visit(node.operand)
- self.write(')')
-
- def visit_Subscript(self, node):
- self.visit(node.value)
- self.write('[')
- self.visit(node.slice)
- self.write(']')
-
- def visit_Slice(self, node):
- if node.lower is not None:
- self.visit(node.lower)
- self.write(':')
- if node.upper is not None:
- self.visit(node.upper)
- if node.step is not None:
- self.write(':')
- if not (isinstance(node.step, Name) and node.step.id == 'None'):
- self.visit(node.step)
-
- def visit_ExtSlice(self, node):
- for idx, item in node.dims:
- if idx:
- self.write(', ')
- self.visit(item)
-
- def visit_Yield(self, node):
- self.write('yield ')
- self.visit(node.value)
-
- def visit_Lambda(self, node):
- self.write('lambda ')
- self.signature(node.args)
- self.write(': ')
- self.visit(node.body)
-
- def visit_Ellipsis(self, node):
- self.write('Ellipsis')
-
- def generator_visit(left, right):
- def visit(self, node):
- self.write(left)
- self.visit(node.elt)
- for comprehension in node.generators:
- self.visit(comprehension)
- self.write(right)
- return visit
-
- visit_ListComp = generator_visit('[', ']')
- visit_GeneratorExp = generator_visit('(', ')')
- visit_SetComp = generator_visit('{', '}')
- del generator_visit
-
- def visit_DictComp(self, node):
- self.write('{')
- self.visit(node.key)
- self.write(': ')
- self.visit(node.value)
- for comprehension in node.generators:
- self.visit(comprehension)
- self.write('}')
-
- def visit_IfExp(self, node):
- self.visit(node.body)
- self.write(' if ')
- self.visit(node.test)
- self.write(' else ')
- self.visit(node.orelse)
-
- def visit_Starred(self, node):
- self.write('*')
- self.visit(node.value)
-
- def visit_Repr(self, node):
- # XXX: python 2.6 only
- self.write('`')
- self.visit(node.value)
- self.write('`')
-
- # Helper Nodes
-
- def visit_alias(self, node):
- self.write(node.name)
- if node.asname is not None:
- self.write(' as ' + node.asname)
-
- def visit_comprehension(self, node):
- self.write(' for ')
- self.visit(node.target)
- self.write(' in ')
- self.visit(node.iter)
- if node.ifs:
- for if_ in node.ifs:
- self.write(' if ')
- self.visit(if_)
-
- def visit_excepthandler(self, node):
- self.newline(node)
- self.write('except')
- if node.type is not None:
- self.write(' ')
- self.visit(node.type)
- if node.name is not None:
- self.write(' as ')
- self.visit(node.name)
- self.write(':')
- self.body(node.body)
diff --git a/bitbake/lib/hashserv/__init__.py b/bitbake/lib/hashserv/__init__.py
index 74367eb6..58ab678e 100644
--- a/bitbake/lib/hashserv/__init__.py
+++ b/bitbake/lib/hashserv/__init__.py
@@ -7,12 +7,22 @@ import asyncio
from contextlib import closing
import itertools
import json
+import re
from collections import namedtuple
from urllib.parse import urlparse
from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
+__version__ = "2.20.0"
+
User = namedtuple("User", ("username", "permissions"))
+UNIHASH_REGEX = re.compile(r"^[0-9a-f]{64}$")
+
+
+def is_valid_unihash(value):
+ return isinstance(value, str) and UNIHASH_REGEX.fullmatch(value) is not None
+
+
def create_server(
addr,
dbname,
@@ -25,6 +35,7 @@ def create_server(
anon_perms=None,
admin_username=None,
admin_password=None,
+ reuseport=False,
):
def sqlite_engine():
from .sqlite import DatabaseEngine
@@ -60,9 +71,9 @@ def create_server(
s.start_unix_server(*a)
elif typ == ADDR_TYPE_WS:
url = urlparse(a[0])
- s.start_websocket_server(url.hostname, url.port)
+ s.start_websocket_server(url.hostname, url.port, reuseport=reuseport)
else:
- s.start_tcp_server(*a)
+ s.start_tcp_server(*a, reuseport=reuseport)
return s
diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py
index 0b254bed..b447fb25 100644
--- a/bitbake/lib/hashserv/client.py
+++ b/bitbake/lib/hashserv/client.py
@@ -5,18 +5,262 @@
import logging
import socket
+import asyncio
import bb.asyncrpc
import json
+from abc import abstractmethod
+from collections.abc import AsyncIterable
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
from . import create_async_client
-
logger = logging.getLogger("hashserv.client")
+class AsyncQueue(AsyncIterable):
+ class Shutdown(Exception):
+ pass
+
+ SHUTDOWN_SENTINEL = object()
+
+ def __init__(self, *args, **kwargs):
+ self.__queue = asyncio.Queue()
+ self.__shutdown = False
+ self.__is_done = False
+
+ async def done(self):
+ if self.__is_done:
+ return
+ self.__is_done = True
+ await self.__queue.put(self.SHUTDOWN_SENTINEL)
+
+ async def put(self, item):
+ if self.__is_done:
+ raise self.Shutdown
+ await self.__queue.put(item)
+
+ async def get(self):
+ if self.__shutdown:
+ raise self.Shutdown
+
+ item = await self.__queue.get()
+ if item is self.SHUTDOWN_SENTINEL:
+ self.__shutdown = True
+ raise self.Shutdown
+
+ return item
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ try:
+ return await self.get()
+ except self.Shutdown:
+ raise StopAsyncIteration
+
+
+@dataclass(eq=False, frozen=True)
+class AsyncPipe:
+ send_queue: AsyncQueue
+ recv_queue: AsyncQueue
+
+
+class Stream(AsyncIterable):
+ def __init__(self, pipe):
+ self._pipe = pipe
+
+ async def done(self):
+ await self._pipe.send_queue.done()
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ try:
+ return await self.get_result()
+ except AsyncQueue.Shutdown:
+ raise StopAsyncIteration
+
+ @abstractmethod
+ async def _send_batch_input(self, i):
+ raise NotImplementedError("Not implemented")
+
+ @abstractmethod
+ async def get_result(self):
+ raise NotImplementedError("Not implemented")
+
+ async def batch(self, inputs):
+ """
+ Does a "batch" process of stream messages. This sends the query
+ messages as fast as possible, and simultaneously attempts to read the
+ messages back. This helps to mitigate the effects of latency to the
+ hash equivalence server be allowing multiple queries to be "in-flight"
+ at once
+
+ The input may be a generator or an async generator
+ """
+
+ async def get_inputs():
+ if isinstance(inputs, AsyncIterable):
+ async for i in inputs:
+ yield i
+ else:
+ for i in inputs:
+ yield i
+
+ async def send():
+ try:
+ async for i in get_inputs():
+ await self._send_batch_input(i)
+ finally:
+ await self.done()
+
+ results = []
+
+ async def recv():
+ async for item in self:
+ results.append(item)
+
+ await bb.asyncrpc.TaskGroup.run(send(), recv())
+ return results
+
+
+class GetUnihashStream(Stream):
+ def __init__(self, pipe):
+ super().__init__(pipe)
+
+ async def _send_batch_input(self, i):
+ method, taskhash = i
+ await self.send_query(method, taskhash)
+
+ async def send_query(self, method, taskhash):
+ await self._pipe.send_queue.put(f"{method} {taskhash}")
+
+ async def get_result(self):
+ r = await self._pipe.recv_queue.get()
+ return r if r else None
+
+
+class UnihashExistsStream(Stream):
+ def __init__(self, pipe):
+ super().__init__(pipe)
+
+ async def _send_batch_input(self, i):
+ await self.send_query(i)
+
+ async def send_query(self, unihash):
+ await self._pipe.send_queue.put(unihash)
+
+ async def get_result(self):
+ r = await self._pipe.recv_queue.get()
+ return r == "true"
+
+
+class GcMarkStream(Stream):
+ def __init__(self, pipe, mark):
+ super().__init__(pipe)
+ self.mark = mark
+
+ async def _send_batch_input(self, i):
+ def row_to_dict(row):
+ pairs = row.split()
+ return dict(zip(pairs[::2], pairs[1::2]))
+
+ await self.send_mark(row_to_dict(i))
+
+ async def send_mark(self, where):
+ await self._pipe.send_queue.put(json.dumps({"mark": self.mark, "where": where}))
+
+ async def get_result(self):
+ r = await self._pipe.recv_queue.get()
+ return json.loads(r)
+
+
+class Batch(object):
+ def __init__(self, send_queue, recv_queue):
+ self.send_queue = send_queue
+ self.recv_queue = recv_queue
+ self.fill_done = False
+ self.send_done = False
+ self.cond = asyncio.Condition()
+ self.pending = []
+ self.sent_count = 0
+ self.recv_count = 0
+ self.item = None
+
+ async def recv(self, socket):
+ while True:
+ async with self.cond:
+ await self.cond.wait_for(lambda: self.pending or self.send_done)
+ if not self.pending:
+ if self.send_done:
+ return
+ continue
+
+ m = await socket.recv()
+ await self.recv_queue.put(m)
+
+ async with self.cond:
+ self.recv_count += 1
+ self.pending.pop(0)
+
+ async def fill(self):
+ async for m in self.send_queue:
+ async with self.cond:
+ # Wait for item to be consumed
+ await self.cond.wait_for(lambda: self.item is None)
+ self.item = m
+ self.cond.notify_all()
+
+ async with self.cond:
+ self.fill_done = True
+ self.cond.notify_all()
+
+ async def send(self, socket):
+ # In the event of a restart due to a reconnect, all in-flight
+ # messages need to be resent first to keep to result count in sync
+ async with self.cond:
+ for m in self.pending:
+ await socket.send(m)
+
+ while True:
+ async with self.cond:
+ await self.cond.wait_for(
+ lambda: self.item is not None or self.fill_done
+ )
+ if self.item is None:
+ if self.fill_done:
+ self.send_done = True
+ self.cond.notify_all()
+ return
+ continue
+
+ m = self.item
+
+ await socket.send(m)
+
+ async with self.cond:
+ self.item = None
+ self.pending.append(m)
+ self.sent_count += 1
+ self.cond.notify_all()
+
+ async def stream(self, socket):
+ await bb.asyncrpc.TaskGroup.run(self.send(socket), self.recv(socket))
+
+ def check(self):
+ if self.sent_count != self.recv_count:
+ raise ConnectionError(
+ f"Sent {self.sent_count} messages but only received {self.recv_count}"
+ )
+
+
class AsyncClient(bb.asyncrpc.AsyncClient):
MODE_NORMAL = 0
MODE_GET_STREAM = 1
MODE_EXIST_STREAM = 2
+ MODE_MARK_STREAM = 3
def __init__(self, username=None, password=None):
super().__init__("OEHASHEQUIV", "1.1", logger)
@@ -36,32 +280,57 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
if become:
await self.become_user(become)
- async def send_stream(self, mode, msg):
+ @asynccontextmanager
+ async def send_stream(self, mode):
+ send_queue = AsyncQueue()
+ recv_queue = AsyncQueue()
+ b = Batch(send_queue, recv_queue)
+
async def proc():
await self._set_mode(mode)
- await self.socket.send(msg)
- return await self.socket.recv()
-
- return await self._send_wrapper(proc)
-
- async def invoke(self, *args, **kwargs):
+ await b.stream(self.socket)
+
+ async def process():
+ try:
+ await self._send_wrapper(proc)
+ finally:
+ await recv_queue.done()
+
+ # Create background process to process messages
+ async with bb.asyncrpc.TaskGroup() as group:
+ group.create_task(process())
+ group.create_task(b.fill())
+ try:
+ yield AsyncPipe(send_queue, recv_queue)
+ b.check()
+ except AsyncQueue.Shutdown as e:
+ pass
+ finally:
+ await send_queue.done()
+ await recv_queue.done()
+
+ async def invoke(self, *args, skip_mode=False, **kwargs):
# It's OK if connection errors cause a failure here, because the mode
# is also reset to normal on a new connection
- await self._set_mode(self.MODE_NORMAL)
+ if not skip_mode:
+ await self._set_mode(self.MODE_NORMAL)
return await super().invoke(*args, **kwargs)
async def _set_mode(self, new_mode):
async def stream_to_normal():
+ # Check if already in normal mode (e.g. due to a connection reset)
+ if self.mode == self.MODE_NORMAL:
+ return "ok"
await self.socket.send("END")
return await self.socket.recv()
async def normal_to_stream(command):
- r = await self.invoke({command: None})
+ r = await self.invoke({command: None}, skip_mode=True)
if r != "ok":
+ self.check_invoke_error(r)
raise ConnectionError(
f"Unable to transition to stream mode: Bad response from server {r!r}"
)
-
self.logger.debug("Mode is now %s", command)
if new_mode == self.mode:
@@ -83,16 +352,26 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
await normal_to_stream("get-stream")
elif new_mode == self.MODE_EXIST_STREAM:
await normal_to_stream("exists-stream")
+ elif new_mode == self.MODE_MARK_STREAM:
+ await normal_to_stream("gc-mark-stream")
elif new_mode != self.MODE_NORMAL:
raise Exception("Undefined mode transition {self.mode!r} -> {new_mode!r}")
self.mode = new_mode
async def get_unihash(self, method, taskhash):
- r = await self.send_stream(self.MODE_GET_STREAM, "%s %s" % (method, taskhash))
- if not r:
- return None
- return r
+ async with self.get_unihash_stream() as stream:
+ await stream.send_query(method, taskhash)
+ return await stream.get_result()
+
+ async def get_unihash_batch(self, args):
+ async with self.get_unihash_stream() as stream:
+ return await stream.batch(args)
+
+ @asynccontextmanager
+ async def get_unihash_stream(self):
+ async with self.send_stream(self.MODE_GET_STREAM) as pipe:
+ yield GetUnihashStream(pipe)
async def report_unihash(self, taskhash, method, outhash, unihash, extra={}):
m = extra.copy()
@@ -115,8 +394,18 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
)
async def unihash_exists(self, unihash):
- r = await self.send_stream(self.MODE_EXIST_STREAM, unihash)
- return r == "true"
+ async with self.unihash_exists_stream() as stream:
+ await stream.send_query(unihash)
+ return await stream.get_result()
+
+ async def unihash_exists_batch(self, unihashes):
+ async with self.unihash_exists_stream() as stream:
+ return await stream.batch(unihashes)
+
+ @asynccontextmanager
+ async def unihash_exists_stream(self):
+ async with self.send_stream(self.MODE_EXIST_STREAM) as pipe:
+ yield UnihashExistsStream(pipe)
async def get_outhash(self, method, outhash, taskhash, with_unihash=True):
return await self.invoke(
@@ -216,6 +505,23 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
"""
return await self.invoke({"gc-mark": {"mark": mark, "where": where}})
+ async def gc_mark_batch(self, mark, rows):
+ """
+ Similar to `gc-mark`, but accepts a list of "where" key-value pair
+ conditions. It utilizes stream mode to mark hashes, which helps reduce
+ the impact of latency when communicating with the hash equivalence
+ server.
+ """
+ async with self.gc_mark_stream(mark) as stream:
+ results = await stream.batch(rows)
+
+ return {"count": sum(int(r["count"]) for r in results)}
+
+ @asynccontextmanager
+ async def gc_mark_stream(self, mark):
+ async with self.send_stream(self.MODE_MARK_STREAM) as pipe:
+ yield GcMarkStream(pipe, mark)
+
async def gc_sweep(self, mark):
"""
Finishes garbage collection for "mark". All unihash entries that have
@@ -237,10 +543,12 @@ class Client(bb.asyncrpc.Client):
"connect_tcp",
"connect_websocket",
"get_unihash",
+ "get_unihash_batch",
"report_unihash",
"report_unihash_equiv",
"get_taskhash",
"unihash_exists",
+ "unihash_exists_batch",
"get_outhash",
"get_stats",
"reset_stats",
@@ -259,88 +567,9 @@ class Client(bb.asyncrpc.Client):
"get_db_query_columns",
"gc_status",
"gc_mark",
+ "gc_mark_batch",
"gc_sweep",
)
def _get_async_client(self):
return AsyncClient(self.username, self.password)
-
-
-class ClientPool(bb.asyncrpc.ClientPool):
- def __init__(
- self,
- address,
- max_clients,
- *,
- username=None,
- password=None,
- become=None,
- ):
- super().__init__(max_clients)
- self.address = address
- self.username = username
- self.password = password
- self.become = become
-
- async def _new_client(self):
- client = await create_async_client(
- self.address,
- username=self.username,
- password=self.password,
- )
- if self.become:
- await client.become_user(self.become)
- return client
-
- def _run_key_tasks(self, queries, call):
- results = {key: None for key in queries.keys()}
-
- def make_task(key, args):
- async def task(client):
- nonlocal results
- unihash = await call(client, args)
- results[key] = unihash
-
- return task
-
- def gen_tasks():
- for key, args in queries.items():
- yield make_task(key, args)
-
- self.run_tasks(gen_tasks())
- return results
-
- def get_unihashes(self, queries):
- """
- Query multiple unihashes in parallel.
-
- The queries argument is a dictionary with arbitrary key. The values
- must be a tuple of (method, taskhash).
-
- Returns a dictionary with a corresponding key for each input key, and
- the value is the queried unihash (which might be none if the query
- failed)
- """
-
- async def call(client, args):
- method, taskhash = args
- return await client.get_unihash(method, taskhash)
-
- return self._run_key_tasks(queries, call)
-
- def unihashes_exist(self, queries):
- """
- Query multiple unihash existence checks in parallel.
-
- The queries argument is a dictionary with arbitrary key. The values
- must be a unihash.
-
- Returns a dictionary with a corresponding key for each input key, and
- the value is True or False if the unihash is known by the server (or
- None if there was a failure)
- """
-
- async def call(client, unihash):
- return await client.unihash_exists(unihash)
-
- return self._run_key_tasks(queries, call)
diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py
index 68f64f98..d9c30bdb 100644
--- a/bitbake/lib/hashserv/server.py
+++ b/bitbake/lib/hashserv/server.py
@@ -10,8 +10,10 @@ import math
import time
import os
import base64
+import json
import hashlib
from . import create_async_client
+from . import is_valid_unihash
import bb.asyncrpc
logger = logging.getLogger("hashserv.server")
@@ -172,6 +174,11 @@ def hash_token(algo, salt, token):
return ":".join([algo, salt, h.hexdigest()])
+def validate_unihash(value):
+ if not is_valid_unihash(value):
+ raise bb.asyncrpc.InvokeError("Invalid unihash")
+
+
def permissions(*permissions, allow_anon=True, allow_self_service=False):
"""
Function decorator that can be used to decorate an RPC function call and
@@ -222,6 +229,68 @@ def permissions(*permissions, allow_anon=True, allow_self_service=False):
return wrapper
+class UpstreamQueue(object):
+ UPSTREAM_NONCE = object()
+
+ def __init__(
+ self,
+ logger,
+ queue,
+ get_local_result,
+ send_upstream,
+ get_upstream_result,
+ ):
+ self.logger = logger
+ self.queue = queue
+ self.pending = []
+ self.cond = asyncio.Condition()
+ self.done = False
+ self.get_local_result = get_local_result
+ self.send_upstream = send_upstream
+ self.get_upstream_result = get_upstream_result
+
+ async def process_results(self):
+ try:
+ while True:
+ async with self.cond:
+ await self.cond.wait_for(lambda: self.pending or self.done)
+ if not self.pending:
+ if self.done:
+ return
+ continue
+
+ value, m = self.pending.pop(0)
+
+ if value is self.UPSTREAM_NONCE:
+ value = await self.get_upstream_result(m)
+
+ if value is None:
+ self.logger.error(
+ "None is not allowed as a stream value. Terminating stream"
+ )
+ return
+
+ await self.queue.put(value)
+ finally:
+ await self.queue.put(None)
+
+ async def handler(self, m):
+ if m is None:
+ async with self.cond:
+ self.done = True
+ self.cond.notify_all()
+ return
+
+ value = await self.get_local_result(m)
+ if value is None:
+ await self.send_upstream(m)
+ value = self.UPSTREAM_NONCE
+
+ async with self.cond:
+ self.pending.append((value, m))
+ self.cond.notify_all()
+
+
class ServerClient(bb.asyncrpc.AsyncServerConnection):
def __init__(self, socket, server):
super().__init__(socket, "OEHASHEQUIV", server.logger)
@@ -256,6 +325,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
"backfill-wait": self.handle_backfill_wait,
"remove": self.handle_remove,
"gc-mark": self.handle_gc_mark,
+ "gc-mark-stream": self.handle_gc_mark_stream,
"gc-sweep": self.handle_gc_sweep,
"gc-status": self.handle_gc_status,
"clean-unused": self.handle_clean_unused,
@@ -343,7 +413,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
d = {k: row[k] for k in row.keys()}
elif self.upstream_client is not None:
d = await self.upstream_client.get_taskhash(method, taskhash)
- await self.db.insert_unihash(d["method"], d["taskhash"], d["unihash"])
+ await self.insert_unihash(d["method"], d["taskhash"], d["unihash"])
return d
@@ -375,74 +445,131 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
if data is None:
return
- await self.db.insert_unihash(data["method"], data["taskhash"], data["unihash"])
+ await self.insert_unihash(data["method"], data["taskhash"], data["unihash"])
await self.db.insert_outhash(data)
- async def _stream_handler(self, handler):
+ async def insert_unihash(self, method, taskhash, unihash):
+ validate_unihash(unihash)
+ return await self.db.insert_unihash(method, taskhash, unihash)
+
+ async def _stream_queue_handler(self, handler, queue):
await self.socket.send_message("ok")
- while True:
- upstream = None
+ async def recv():
+ try:
+ while True:
+ m = await self.socket.recv()
+ if not m or m == "END":
+ break
- l = await self.socket.recv()
- if not l:
- break
+ await handler(m)
+ finally:
+ await handler(None)
- try:
- # This inner loop is very sensitive and must be as fast as
- # possible (which is why the request sample is handled manually
- # instead of using 'with', and also why logging statements are
- # commented out.
- self.request_sample = self.server.request_stats.start_sample()
- request_measure = self.request_sample.measure()
- request_measure.start()
-
- if l == "END":
+ async def process():
+ while True:
+ m = await queue.get()
+ if m is None:
break
- msg = await handler(l)
- await self.socket.send(msg)
- finally:
- request_measure.end()
- self.request_sample.end()
+ await self.socket.send(m)
+ await bb.asyncrpc.TaskGroup.run(recv(), process())
await self.socket.send("ok")
+
+ async def _stream_handler(self, handler):
+ queue = asyncio.Queue(1000)
+
+ async def h(m):
+ if m is None:
+ await queue.put(None)
+ else:
+ await queue.put(await handler(m))
+
+ await self._stream_queue_handler(h, queue)
return self.NO_RESPONSE
@permissions(READ_PERM)
async def handle_get_stream(self, request):
- async def handler(l):
- (method, taskhash) = l.split()
- # self.logger.debug('Looking up %s %s' % (method, taskhash))
- row = await self.db.get_equivalent(method, taskhash)
-
- if row is not None:
- # self.logger.debug('Found equivalent task %s -> %s', (row['taskhash'], row['unihash']))
+ async def get_unihash(m):
+ method, taskhash = m.split()
+ if (row := await self.db.get_equivalent(method, taskhash)) is not None:
return row["unihash"]
- if self.upstream_client is not None:
- upstream = await self.upstream_client.get_unihash(method, taskhash)
- if upstream:
- await self.server.backfill_queue.put((method, taskhash))
- return upstream
-
return ""
- return await self._stream_handler(handler)
+ if not self.upstream_client:
+ return await self._stream_handler(get_unihash)
+
+ async with self.upstream_client.get_unihash_stream() as stream:
+
+ async def get_local_result(m):
+ method, taskhash = m.split()
+ if (row := await self.db.get_equivalent(method, taskhash)) is not None:
+ return row["unihash"]
+ return None
+
+ async def send_upstream(m):
+ method, taskhash = m.split()
+ await stream.send_query(method, taskhash)
+
+ async def get_upstream_result(m):
+ unihash = await stream.get_result()
+ if unihash:
+ method, taskhash = m.split()
+ await self.server.backfill_queue.put((method, taskhash))
+ return unihash or ""
+
+ queue = asyncio.Queue()
+ upstream = UpstreamQueue(
+ self.logger,
+ queue,
+ get_local_result,
+ send_upstream,
+ get_upstream_result,
+ )
+
+ await bb.asyncrpc.TaskGroup.run(
+ self._stream_queue_handler(upstream.handler, queue),
+ upstream.process_results(),
+ )
+ return self.NO_RESPONSE
@permissions(READ_PERM)
async def handle_exists_stream(self, request):
- async def handler(l):
+ async def exists_handler(l):
if await self.db.unihash_exists(l):
return "true"
+ return "false"
- if self.upstream_client is not None:
- if await self.upstream_client.unihash_exists(l):
- return "true"
+ if not self.upstream_client:
+ return await self._stream_handler(exists_handler)
- return "false"
+ async with self.upstream_client.unihash_exists_stream() as stream:
- return await self._stream_handler(handler)
+ async def get_local_result(m):
+ if await self.db.unihash_exists(m):
+ return "true"
+ return None
+
+ async def get_upstream_result(m):
+ exists = await stream.get_result()
+ return "true" if exists else "false"
+
+ queue = asyncio.Queue()
+ upstream = UpstreamQueue(
+ self.logger,
+ queue,
+ get_local_result,
+ stream.send_query,
+ get_upstream_result,
+ )
+
+ await bb.asyncrpc.TaskGroup.run(
+ self._stream_queue_handler(upstream.handler, queue),
+ upstream.process_results(),
+ )
+ return self.NO_RESPONSE
async def report_readonly(self, data):
method = data["method"]
@@ -465,6 +592,8 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
# report is made inside the function
@permissions(READ_PERM)
async def handle_report(self, data):
+ validate_unihash(data.get("unihash"))
+
if self.server.read_only or not self.user_has_permissions(REPORT_PERM):
return await self.report_readonly(data)
@@ -507,7 +636,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
if upstream_data is not None:
unihash = upstream_data["unihash"]
- await self.db.insert_unihash(data["method"], data["taskhash"], unihash)
+ await self.insert_unihash(data["method"], data["taskhash"], unihash)
unihash_data = await self.get_unihash(data["method"], data["taskhash"])
if unihash_data is not None:
@@ -523,7 +652,9 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
@permissions(READ_PERM, REPORT_PERM)
async def handle_equivreport(self, data):
- await self.db.insert_unihash(data["method"], data["taskhash"], data["unihash"])
+ validate_unihash(data.get("unihash"))
+
+ await self.insert_unihash(data["method"], data["taskhash"], data["unihash"])
# Fetch the unihash that will be reported for the taskhash. If the
# unihash matches, it means this row was inserted (or the mapping
@@ -583,6 +714,33 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
return {"count": await self.db.gc_mark(mark, condition)}
+ @permissions(DB_ADMIN_PERM)
+ async def handle_gc_mark_stream(self, request):
+ async def handler(line):
+ try:
+ decoded_line = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise bb.asyncrpc.InvokeError(
+ "Could not decode JSONL input '%s'" % line
+ ) from exc
+
+ try:
+ mark = decoded_line["mark"]
+ condition = decoded_line["where"]
+ if not isinstance(mark, str):
+ raise TypeError("Bad mark type %s" % type(mark))
+
+ if not isinstance(condition, dict):
+ raise TypeError("Bad condition type %s" % type(condition))
+ except KeyError as exc:
+ raise bb.asyncrpc.InvokeError(
+ "Input line is missing key '%s' " % exc
+ ) from exc
+
+ return json.dumps({"count": await self.db.gc_mark(mark, condition)})
+
+ return await self._stream_handler(handler)
+
@permissions(DB_ADMIN_PERM)
async def handle_gc_sweep(self, request):
mark = request["mark"]
@@ -603,7 +761,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
@permissions(DB_ADMIN_PERM)
async def handle_gc_status(self, request):
- (keep_rows, remove_rows, current_mark) = await self.db.gc_status()
+ keep_rows, remove_rows, current_mark = await self.db.gc_status()
return {
"keep": keep_rows,
"remove": remove_rows,
@@ -859,7 +1017,12 @@ class Server(bb.asyncrpc.AsyncServer):
method, taskhash = item
d = await client.get_taskhash(method, taskhash)
if d is not None:
- await db.insert_unihash(d["method"], d["taskhash"], d["unihash"])
+ if is_valid_unihash(d.get("unihash")):
+ await db.insert_unihash(
+ d["method"], d["taskhash"], d["unihash"]
+ )
+ else:
+ self.logger.warning("Upstream server returned invalid unihash")
self.backfill_queue.task_done()
def start(self):
diff --git a/bitbake/lib/hashserv/sqlite.py b/bitbake/lib/hashserv/sqlite.py
index da2e844a..976504d7 100644
--- a/bitbake/lib/hashserv/sqlite.py
+++ b/bitbake/lib/hashserv/sqlite.py
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: GPL-2.0-only
#
+from datetime import datetime, timezone
import sqlite3
import logging
from contextlib import closing
@@ -53,6 +54,22 @@ CONFIG_TABLE_DEFINITION = (
CONFIG_TABLE_COLUMNS = tuple(name for name, _, _ in CONFIG_TABLE_DEFINITION)
+def adapt_datetime_iso(val):
+ """Adapt datetime.datetime to UTC ISO 8601 date."""
+ return val.astimezone(timezone.utc).isoformat()
+
+
+sqlite3.register_adapter(datetime, adapt_datetime_iso)
+
+
+def convert_datetime(val):
+ """Convert ISO 8601 datetime to datetime.datetime object."""
+ return datetime.fromisoformat(val.decode())
+
+
+sqlite3.register_converter("DATETIME", convert_datetime)
+
+
def _make_table(cursor, name, definition):
cursor.execute(
"""
diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py
index 0809453c..15ee7ecd 100644
--- a/bitbake/lib/hashserv/tests.py
+++ b/bitbake/lib/hashserv/tests.py
@@ -5,13 +5,13 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-from . import create_server, create_client
+from . import create_server, create_client, create_async_client
from .server import DEFAULT_ANON_PERMS, ALL_PERMISSIONS
from bb.asyncrpc import InvokeError
-from .client import ClientPool
import hashlib
import logging
-import multiprocessing
+from bb import multiprocessing
+import asyncio
import os
import sys
import tempfile
@@ -31,7 +31,7 @@ BIN_DIR = THIS_DIR.parent.parent / "bin"
def server_prefunc(server, idx):
logging.basicConfig(level=logging.DEBUG, filename='bbhashserv-%d.log' % idx, filemode='w',
- format='%(levelname)s %(filename)s:%(lineno)d %(message)s')
+ format='%(levelname)s %(filename)s:%(lineno)d %(message)s', force=True)
server.logger.debug("Running server %d" % idx)
sys.stdout = open('bbhashserv-stdout-%d.log' % idx, 'w')
sys.stderr = sys.stdout
@@ -42,19 +42,15 @@ class HashEquivalenceTestSetup(object):
server_index = 0
client_index = 0
- def start_server(self, dbpath=None, upstream=None, read_only=False, prefunc=server_prefunc, anon_perms=DEFAULT_ANON_PERMS, admin_username=None, admin_password=None):
+ def start_server(self, dbpath=None, upstream=None, read_only=False, prefunc=server_prefunc, anon_perms=DEFAULT_ANON_PERMS, admin_username=None, admin_password=None, addr=None):
self.server_index += 1
+ if addr is None:
+ addr = self.get_server_addr(self.server_index)
+
if dbpath is None:
dbpath = self.make_dbpath()
- def cleanup_server(server):
- if server.process.exitcode is not None:
- return
-
- server.process.terminate()
- server.process.join()
-
- server = create_server(self.get_server_addr(self.server_index),
+ server = create_server(addr,
dbpath,
upstream=upstream,
read_only=read_only,
@@ -64,7 +60,7 @@ class HashEquivalenceTestSetup(object):
server.dbpath = dbpath
server.serve_as_process(prefunc=prefunc, args=(self.server_index,))
- self.addCleanup(cleanup_server, server)
+ self.addCleanup(self.stop_server, server)
return server
@@ -84,6 +80,13 @@ class HashEquivalenceTestSetup(object):
self.server = self.start_server()
return self.server.address
+ def stop_server(self, server):
+ if not server or server.process.exitcode is not None:
+ return
+
+ server.process.terminate()
+ server.process.join()
+
def start_auth_server(self):
auth_server = self.start_server(self.server.dbpath, anon_perms=[], admin_username="admin", admin_password="password")
self.auth_server_address = auth_server.address
@@ -94,9 +97,8 @@ class HashEquivalenceTestSetup(object):
return self.start_client(self.auth_server_address, user["username"], user["token"])
def setUp(self):
- if sys.version_info < (3, 5, 0):
- self.skipTest('Python 3.5 or later required')
-
+ logging.basicConfig(level=logging.DEBUG, filename='bbhashtest.log', filemode='w',
+ format='%(levelname)s %(filename)s:%(lineno)d %(message)s')
self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-hashserv')
self.addCleanup(self.temp_dir.cleanup)
@@ -128,7 +130,7 @@ class HashEquivalenceTestSetup(object):
# Simple test that hashes can be created
taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
outhash = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
- unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+ unihash = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
self.assertClientGetHash(client, taskhash, None)
@@ -182,7 +184,7 @@ class HashEquivalenceCommonTests(object):
# assigned the same unihash
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
@@ -190,7 +192,7 @@ class HashEquivalenceCommonTests(object):
# Report a different task with the same outhash. The returned unihash
# should match the first task
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash, unihash2)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
@@ -200,19 +202,19 @@ class HashEquivalenceCommonTests(object):
# taskhash
taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
outhash = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
- unihash = '218e57509998197d570e2c98512d0105985dffc9'
+ unihash = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertClientGetHash(self.client, taskhash, unihash)
outhash2 = '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
- unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
+ unihash2 = 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b6356290e3'
self.client.report_unihash(taskhash, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash, unihash)
outhash3 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
+ unihash3 = '6842f1f2daccd96ddef15c9154d4e41ac8a2300d781ac9a9db7f8afeb8a96808'
self.client.report_unihash(taskhash, self.METHOD, outhash3, unihash3)
self.assertClientGetHash(self.client, taskhash, unihash)
@@ -272,7 +274,7 @@ class HashEquivalenceCommonTests(object):
# Simple test that hashes can be created
taskhash = 'c665584ee6817aa99edfc77a44dd853828279370'
outhash = '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5b14f7d39db44'
- unihash = '90e9bc1d1f094c51824adca7f8ea79a048d68824'
+ unihash = '06f89b8f329ba8124ff73c56d09ef921b42624747c421277bddaf5e23f136e57'
self.assertClientGetHash(self.client, taskhash, None)
@@ -295,6 +297,36 @@ class HashEquivalenceCommonTests(object):
self.assertEqual(result_outhash['outhash'], outhash)
self.assertEqual(result_outhash['outhash_siginfo'], siginfo)
+ def test_report_rejects_invalid_unihash(self):
+ taskhash = '68a9206490b2321bb033fb3eab013a4ec62c41f9'
+ outhash = 'bf5f2efaf1ca351f3b4c3d079363540ab48f7c58db3d23cfbb069cf4ff1ea8f7'
+ invalid_unihashes = (
+ "${@os.system('true')}",
+ 'a' * 63,
+ 'a' * 65,
+ 'A' * 64,
+ None,
+ )
+
+ for unihash in invalid_unihashes:
+ with self.subTest(unihash=unihash):
+ with self.start_client(self.server_address) as client:
+ with self.assertRaises(InvokeError) as context:
+ client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+
+ self.assertEqual(str(context.exception), "Invalid unihash")
+
+ self.assertClientGetHash(self.client, taskhash, None)
+
+ def test_equivreport_rejects_invalid_unihash(self):
+ taskhash = 'ae6339531895ddf5b67e663e6a374ad8ec71d81c'
+
+ with self.assertRaises(InvokeError) as context:
+ self.client.report_unihash_equiv(taskhash, self.METHOD, "${@os.system('true')}")
+
+ self.assertEqual(str(context.exception), "Invalid unihash")
+ self.assertClientGetHash(self.start_client(self.server_address), taskhash, None)
+
def test_stress(self):
def query_server(failures):
client = Client(self.server_address)
@@ -346,26 +378,31 @@ class HashEquivalenceCommonTests(object):
nonlocal side_client
# check upstream server
+ self.assertTrue(self.client.unihash_exists(unihash))
self.assertClientGetHash(self.client, taskhash, unihash)
# Hash should *not* be present on the side server
+ if old_sidehash and unihash != old_sidehash:
+ self.assertFalse(side_client.unihash_exists(unihash))
self.assertClientGetHash(side_client, taskhash, old_sidehash)
# Hash should be present on the downstream server, since it
# will defer to the upstream server. This will trigger
# the backfill in the downstream server
+ self.assertTrue(down_client.unihash_exists(unihash))
self.assertClientGetHash(down_client, taskhash, unihash)
# After waiting for the downstream client to finish backfilling the
# task from the upstream server, it should appear in the side server
# since the database is populated
down_client.backfill_wait()
+ self.assertTrue(side_client.unihash_exists(unihash))
self.assertClientGetHash(side_client, taskhash, unihash)
# Basic report
taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
outhash = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
- unihash = '218e57509998197d570e2c98512d0105985dffc9'
+ unihash = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
check_hash(taskhash, unihash, None)
@@ -373,7 +410,7 @@ class HashEquivalenceCommonTests(object):
# Duplicated taskhash with multiple output hashes and unihashes.
# All servers should agree with the originally reported hash
outhash2 = '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
- unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
+ unihash2 = 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b6356290e3'
self.client.report_unihash(taskhash, self.METHOD, outhash2, unihash2)
check_hash(taskhash, unihash, unihash)
@@ -381,7 +418,7 @@ class HashEquivalenceCommonTests(object):
# Report an equivalent task. The sideload will originally report
# no unihash until backfilled
taskhash3 = "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1"
- unihash3 = "def64766090d28f627e816454ed46894bb3aab36"
+ unihash3 = "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a20c7dd73458d"
self.client.report_unihash(taskhash3, self.METHOD, outhash, unihash3)
check_hash(taskhash3, unihash, None)
@@ -390,7 +427,7 @@ class HashEquivalenceCommonTests(object):
# propagating to the upstream server
taskhash4 = "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
outhash4 = "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a"
- unihash4 = "3b5d3d83f07f259e9086fcb422c855286e18a57d"
+ unihash4 = "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe"
down_client.report_unihash(taskhash4, self.METHOD, outhash4, unihash4)
down_client.backfill_wait()
@@ -402,18 +439,18 @@ class HashEquivalenceCommonTests(object):
# match which was previously reported to the upstream server
taskhash5 = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
outhash5 = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
- unihash5 = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+ unihash5 = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
result = self.client.report_unihash(taskhash5, self.METHOD, outhash5, unihash5)
taskhash6 = '35788efcb8dfb0a02659d81cf2bfd695fb30fafa'
- unihash6 = 'f46d3fbb439bd9b921095da657a4de906510d2ce'
+ unihash6 = 'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069d36efc933b23'
result = down_client.report_unihash(taskhash6, self.METHOD, outhash5, unihash6)
self.assertEqual(result['unihash'], unihash5, 'Server failed to copy unihash from upstream')
# Tests read through from server with
taskhash7 = '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74'
outhash7 = '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69'
- unihash7 = '05d2a63c81e32f0a36542ca677e8ad852365c538'
+ unihash7 = '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925ab7b4d5'
self.client.report_unihash(taskhash7, self.METHOD, outhash7, unihash7)
result = down_client.get_taskhash(self.METHOD, taskhash7, True)
@@ -424,7 +461,7 @@ class HashEquivalenceCommonTests(object):
taskhash8 = '86978a4c8c71b9b487330b0152aade10c1ee58aa'
outhash8 = 'ca8c128e9d9e4a28ef24d0508aa20b5cf880604eacd8f65c0e366f7e0cc5fbcf'
- unihash8 = 'd8bcf25369d40590ad7d08c84d538982f2023e01'
+ unihash8 = '83386d9385b0bf3ba25693127ddcaaadeaa1c4bf8cb0baecfb5314b9a20072a1'
self.client.report_unihash(taskhash8, self.METHOD, outhash8, unihash8)
result = down_client.get_outhash(self.METHOD, outhash8, taskhash8)
@@ -435,7 +472,7 @@ class HashEquivalenceCommonTests(object):
taskhash9 = 'ae6339531895ddf5b67e663e6a374ad8ec71d81c'
outhash9 = 'afc78172c81880ae10a1fec994b5b4ee33d196a001a1b66212a15ebe573e00b5'
- unihash9 = '6662e699d6e3d894b24408ff9a4031ef9b038ee8'
+ unihash9 = 'cc74784b2c0ad5b378a6b783c74c518d2c46b8b52fba29cb39a8430d742440d7'
self.client.report_unihash(taskhash9, self.METHOD, outhash9, unihash9)
result = down_client.get_taskhash(self.METHOD, taskhash9, False)
@@ -443,6 +480,169 @@ class HashEquivalenceCommonTests(object):
self.assertEqual(result['taskhash'], taskhash9, 'Server failed to copy unihash from upstream')
self.assertEqual(result['method'], self.METHOD)
+ def test_upstream_batch(self):
+ down_server = self.start_server(upstream=self.server.address)
+ down_client = self.start_client(down_server.address)
+
+ taskhash1 = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+ outhash1 = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
+ unihash1 = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
+ self.client.report_unihash(taskhash1, self.METHOD, outhash1, unihash1)
+
+ taskhash2 = "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+ outhash2 = "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a"
+ unihash2 = "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe"
+ self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+
+ taskhash3 = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+ outhash3 = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
+ unihash3 = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
+ self.client.report_unihash(taskhash3, self.METHOD, outhash3, unihash3)
+
+ def query_generator():
+ yield unihash1
+ yield unihash2
+ yield unihash3
+ yield "cc74784b2c0ad5b378a6b783c74c518d2c46b8b52fba29cb39a8430d742440d7"
+
+ results = down_client.unihash_exists_batch(query_generator())
+ self.assertEqual(results, [True, True, True, False])
+
+ def test_upstream_interrupted(self):
+ up_server = self.start_server()
+ down_server = self.start_server(upstream=up_server.address)
+
+ def restart_upstream():
+ nonlocal up_server
+
+ self.stop_server(up_server)
+ up_server = self.start_server(addr=up_server.address, dbpath=up_server.dbpath)
+
+ # Report some hashes
+ with self.start_client(up_server.address) as up_client:
+ taskhash1 = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+ outhash1 = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
+ unihash1 = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
+ up_client.report_unihash(taskhash1, self.METHOD, outhash1, unihash1)
+
+ taskhash2 = "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+ outhash2 = "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a"
+ unihash2 = "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe"
+ up_client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+
+ taskhash3 = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+ outhash3 = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
+ unihash3 = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
+ up_client.report_unihash(taskhash3, self.METHOD, outhash3, unihash3)
+
+ restart_upstream()
+
+ with self.start_client(up_server.address) as up_client:
+ # Verify that reported hashes are correct after restaring server
+ self.assertTrue(up_client.unihash_exists(unihash1))
+ self.assertClientGetHash(up_client, taskhash1, unihash1)
+
+ self.assertTrue(up_client.unihash_exists(unihash2))
+ self.assertClientGetHash(up_client, taskhash2, unihash2)
+
+ async def check_unihashes():
+ async with await create_async_client(down_server.address) as down_client:
+ async with down_client.unihash_exists_stream() as stream:
+ await stream.send_query(unihash1)
+ r = await stream.get_result()
+ self.assertTrue(r)
+
+ restart_upstream()
+
+ await stream.send_query(unihash2)
+ r = await stream.get_result()
+ self.assertTrue(r)
+
+ await stream.send_query(unihash3)
+ r = await stream.get_result()
+ self.assertTrue(r)
+
+ asyncio.run(check_unihashes())
+
+ def test_upstream_lost(self):
+ up_server = self.start_server()
+ down_server = self.start_server(upstream=up_server.address)
+
+ def restart_upstream():
+ nonlocal up_server
+
+ self.stop_server(up_server)
+ up_server = self.start_server(addr=up_server.address, dbpath=up_server.dbpath)
+
+ # Report some hashes
+ with self.start_client(up_server.address) as up_client:
+ taskhash1 = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+ outhash1 = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
+ unihash1 = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
+ up_client.report_unihash(taskhash1, self.METHOD, outhash1, unihash1)
+
+ taskhash2 = "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+ outhash2 = "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a"
+ unihash2 = "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe"
+ up_client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+
+ taskhash3 = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+ outhash3 = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
+ unihash3 = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
+ up_client.report_unihash(taskhash3, self.METHOD, outhash3, unihash3)
+
+ async def check_unihashes():
+ async with await create_async_client(down_server.address) as down_client:
+ with self.assertRaises(ConnectionError):
+ async with down_client.unihash_exists_stream() as stream:
+ await stream.send_query(unihash1)
+ r = await stream.get_result()
+ self.assertTrue(r)
+
+ self.stop_server(up_server)
+
+ await stream.send_query(unihash2)
+ r = await stream.get_result()
+
+ asyncio.run(check_unihashes())
+
+ def test_upstream_get_stream_miss(self):
+ down_server = self.start_server(upstream=self.server.address)
+ down_client = self.start_client(down_server.address)
+
+ # Two hashes present upstream (hits)
+ taskhash1 = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+ outhash1 = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
+ unihash1 = '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380'
+ self.client.report_unihash(taskhash1, self.METHOD, outhash1, unihash1)
+
+ taskhash2 = 'e3da00593d6a7fb435c7e2114976c59c5fd6d561'
+ outhash2 = '1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a'
+ unihash2 = '7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe'
+ self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+
+ # Two taskhashes present nowhere (upstream misses)
+ miss1 = '0000000000000000000000000000000000000001'
+ miss2 = '0000000000000000000000000000000000000002'
+
+ # Miss interleaved with hits: a miss must not truncate the stream
+ results = down_client.get_unihash_batch([
+ (self.METHOD, miss1),
+ (self.METHOD, taskhash1),
+ (self.METHOD, miss2),
+ (self.METHOD, taskhash2),
+ ])
+ self.assertEqual(results, [None, unihash1, None, unihash2])
+
+ # All-miss batch
+ self.assertEqual(
+ down_client.get_unihash_batch([(self.METHOD, miss1), (self.METHOD, miss2)]),
+ [None, None],
+ )
+
+ # Singular get-unihash miss
+ self.assertClientGetHash(down_client, miss1, None)
+
def test_unihash_exsits(self):
taskhash, outhash, unihash = self.create_test_hash(self.client)
self.assertTrue(self.client.unihash_exists(unihash))
@@ -458,7 +658,7 @@ class HashEquivalenceCommonTests(object):
# Report a hash via the read-write server
taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
outhash = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
- unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+ unihash = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
result = rw_client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
@@ -469,7 +669,7 @@ class HashEquivalenceCommonTests(object):
# Ensure that reporting via the read-only server fails
taskhash2 = 'c665584ee6817aa99edfc77a44dd853828279370'
outhash2 = '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5b14f7d39db44'
- unihash2 = '90e9bc1d1f094c51824adca7f8ea79a048d68824'
+ unihash2 = '06f89b8f329ba8124ff73c56d09ef921b42624747c421277bddaf5e23f136e57'
result = ro_client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertEqual(result['unihash'], unihash2)
@@ -555,57 +755,111 @@ class HashEquivalenceCommonTests(object):
# shares a taskhash with Task 2
self.assertClientGetHash(self.client, taskhash2, unihash2)
-
- def test_client_pool_get_unihashes(self):
+ def test_get_unihash_batch(self):
TEST_INPUT = (
# taskhash outhash unihash
- ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'),
+ (
+ '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+ 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e',
+ '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380',
+ ),
# Duplicated taskhash with multiple output hashes and unihashes.
- ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'),
+ (
+ '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+ '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d',
+ 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b6356290e3',
+ ),
# Equivalent hash
- ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"),
- ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"),
- ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'),
- ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'),
- ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'),
+ (
+ "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1",
+ '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d',
+ "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a20c7dd73458d",
+ ),
+ (
+ "e3da00593d6a7fb435c7e2114976c59c5fd6d561",
+ "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a",
+ "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe",
+ ),
+ (
+ '35788efcb8dfb0a02659d81cf2bfd695fb30faf9',
+ '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f',
+ 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9',
+ ),
+ (
+ '35788efcb8dfb0a02659d81cf2bfd695fb30fafa',
+ '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f',
+ 'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069d36efc933b23',
+ ),
+ (
+ '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74',
+ '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69',
+ '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925ab7b4d5',
+ ),
)
EXTRA_QUERIES = (
"6b6be7a84ab179b4240c4302518dc3f6",
)
- with ClientPool(self.server_address, 10) as client_pool:
- for taskhash, outhash, unihash in TEST_INPUT:
- self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
-
- query = {idx: (self.METHOD, data[0]) for idx, data in enumerate(TEST_INPUT)}
- for idx, taskhash in enumerate(EXTRA_QUERIES):
- query[idx + len(TEST_INPUT)] = (self.METHOD, taskhash)
-
- result = client_pool.get_unihashes(query)
-
- self.assertDictEqual(result, {
- 0: "218e57509998197d570e2c98512d0105985dffc9",
- 1: "218e57509998197d570e2c98512d0105985dffc9",
- 2: "218e57509998197d570e2c98512d0105985dffc9",
- 3: "3b5d3d83f07f259e9086fcb422c855286e18a57d",
- 4: "f46d3fbb439bd9b921095da657a4de906510d2cd",
- 5: "f46d3fbb439bd9b921095da657a4de906510d2cd",
- 6: "05d2a63c81e32f0a36542ca677e8ad852365c538",
- 7: None,
- })
+ for taskhash, outhash, unihash in TEST_INPUT:
+ self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
- def test_client_pool_unihash_exists(self):
+
+ result = self.client.get_unihash_batch(
+ [(self.METHOD, data[0]) for data in TEST_INPUT] +
+ [(self.METHOD, e) for e in EXTRA_QUERIES]
+ )
+
+ self.assertListEqual(result, [
+ "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380",
+ "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380",
+ "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380",
+ "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe",
+ "a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9",
+ "a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9",
+ "7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925ab7b4d5",
+ None,
+ ])
+
+ def test_unihash_exists_batch(self):
TEST_INPUT = (
# taskhash outhash unihash
- ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98512d0105985dffc9'),
+ (
+ '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+ 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e',
+ '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689380',
+ ),
# Duplicated taskhash with multiple output hashes and unihashes.
- ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'),
+ (
+ '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+ '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d',
+ 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b6356290e3',
+ ),
# Equivalent hash
- ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e816454ed46894bb3aab36"),
- ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb422c855286e18a57d"),
- ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2cd'),
- ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da657a4de906510d2ce'),
- ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca677e8ad852365c538'),
+ (
+ "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1",
+ '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d',
+ "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a20c7dd73458d",
+ ),
+ (
+ "e3da00593d6a7fb435c7e2114976c59c5fd6d561",
+ "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857cbe688a",
+ "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a9fe",
+ ),
+ (
+ '35788efcb8dfb0a02659d81cf2bfd695fb30faf9',
+ '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f',
+ 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9',
+ ),
+ (
+ '35788efcb8dfb0a02659d81cf2bfd695fb30fafa',
+ '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f',
+ 'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069d36efc933b23',
+ ),
+ (
+ '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74',
+ '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69',
+ '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925ab7b4d5',
+ ),
)
EXTRA_QUERIES = (
"6b6be7a84ab179b4240c4302518dc3f6",
@@ -614,28 +868,24 @@ class HashEquivalenceCommonTests(object):
result_unihashes = set()
- with ClientPool(self.server_address, 10) as client_pool:
- for taskhash, outhash, unihash in TEST_INPUT:
- result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
- result_unihashes.add(result["unihash"])
-
- query = {}
- expected = {}
+ for taskhash, outhash, unihash in TEST_INPUT:
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ result_unihashes.add(result["unihash"])
- for _, _, unihash in TEST_INPUT:
- idx = len(query)
- query[idx] = unihash
- expected[idx] = unihash in result_unihashes
+ query = []
+ expected = []
+ for _, _, unihash in TEST_INPUT:
+ query.append(unihash)
+ expected.append(unihash in result_unihashes)
- for unihash in EXTRA_QUERIES:
- idx = len(query)
- query[idx] = unihash
- expected[idx] = False
- result = client_pool.unihashes_exist(query)
- self.assertDictEqual(result, expected)
+ for unihash in EXTRA_QUERIES:
+ query.append(unihash)
+ expected.append(False)
+ result = self.client.unihash_exists_batch(query)
+ self.assertListEqual(result, expected)
def test_auth_read_perms(self):
admin_client = self.start_auth_server()
@@ -949,14 +1199,14 @@ class HashEquivalenceCommonTests(object):
def test_gc(self):
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -979,17 +1229,59 @@ class HashEquivalenceCommonTests(object):
# First hash is still present
self.assertClientGetHash(self.client, taskhash, unihash)
+ def test_gc_batch(self):
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
+
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+ taskhash3 = 'a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0'
+ outhash3 = '7289c414905303700a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c04f9a53c'
+ unihash3 = 'f5966c8588a69948a89131d3aa63b914c5a63a8cb4820cb1c58694135ba77f0b'
+
+ result = self.client.report_unihash(taskhash3, self.METHOD, outhash3, unihash3)
+ self.assertClientGetHash(self.client, taskhash3, unihash3)
+
+ # Mark the first unihash to be kept
+ ret = self.client.gc_mark_batch("ABC", (f"unihash {h}" for h in [unihash, unihash2]))
+ self.assertEqual(ret, {"count": 2})
+
+ ret = self.client.gc_status()
+ self.assertEqual(ret, {"mark": "ABC", "keep": 2, "remove": 1})
+
+ # Third hash is still there; mark doesn't delete hashes
+ self.assertClientGetHash(self.client, taskhash3, unihash3)
+
+ ret = self.client.gc_sweep("ABC")
+ self.assertEqual(ret, {"count": 1})
+
+ # Hash is gone. Taskhash is returned for second hash
+ self.assertClientGetHash(self.client, taskhash3, None)
+ # First hash is still present
+ self.assertClientGetHash(self.client, taskhash, unihash)
+ # Second hash is still present
+ self.assertClientGetHash(self.client, taskhash2, unihash2)
+
def test_gc_switch_mark(self):
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1027,14 +1319,14 @@ class HashEquivalenceCommonTests(object):
def test_gc_switch_sweep_mark(self):
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1057,7 +1349,7 @@ class HashEquivalenceCommonTests(object):
def test_gc_new_hashes(self):
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
@@ -1074,7 +1366,7 @@ class HashEquivalenceCommonTests(object):
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1373,14 +1665,14 @@ class TestHashEquivalenceClient(HashEquivalenceTestSetup, unittest.TestCase):
def test_gc(self):
taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
- unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+ unihash = '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc61813739a28007fcdf'
result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
outhash2 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
- unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ unihash2 = 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77deaa81a645747'
result = self.client.report_unihash(taskhash2, self.METHOD, outhash2, unihash2)
self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1423,7 +1715,7 @@ class TestHashEquivalenceUnixServerLongPath(HashEquivalenceTestSetup, unittest.T
# Simple test that hashes can be created
taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
outhash = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
- unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+ unihash = 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee33d9'
self.assertClientGetHash(self.client, taskhash, None)
@@ -1438,6 +1730,102 @@ class TestHashEquivalenceTCPServer(HashEquivalenceTestSetup, HashEquivalenceComm
# case it is more reliable to resolve the IP address explicitly.
return socket.gethostbyname("localhost") + ":0"
+ def test_get_stream_upstream_pipelined(self):
+ # Verify that, with an upstream configured, the get-stream handler
+ # pipelines its upstream queries instead of doing one blocking
+ # round-trip per task. A latency proxy injects RTT between this server
+ # and its upstream; a serial (one round-trip per task) implementation
+ # could not beat N * RTT, so finishing far faster proves the queries
+ # are pipelined.
+ import asyncio
+
+ LATENCY = 0.01 # 10ms each direction => ~20ms round trip
+ upstream_host, upstream_port = self.server_address.rsplit(":", 1)
+ upstream_port = int(upstream_port)
+
+ ready = threading.Event()
+ proxy = {}
+
+ def run_proxy():
+ loop = asyncio.new_event_loop()
+ asyncio.set_event_loop(loop)
+
+ async def handle(creader, cwriter):
+ ureader, uwriter = await asyncio.open_connection(upstream_host, upstream_port)
+
+ async def pipe(r, w):
+ try:
+ while True:
+ data = await r.read(65536)
+ if not data:
+ break
+ await asyncio.sleep(LATENCY)
+ w.write(data)
+ await w.drain()
+ except Exception:
+ pass
+ finally:
+ try:
+ w.close()
+ except Exception:
+ pass
+
+ await bb.asyncrpc.TaskGroup.run(pipe(creader, uwriter), pipe(ureader, cwriter))
+
+ stop_event = asyncio.Event()
+ proxy["stop_event"] = stop_event
+ proxy["loop"] = loop
+
+ async def main():
+ server = await asyncio.start_server(handle, "127.0.0.1", 0)
+ proxy["port"] = server.sockets[0].getsockname()[1]
+ ready.set()
+ async with server:
+ await stop_event.wait()
+
+ try:
+ loop.run_until_complete(main())
+ except Exception:
+ pass
+ finally:
+ loop.close()
+
+ proxy_thread = threading.Thread(target=run_proxy, daemon=True)
+ proxy_thread.start()
+ self.assertTrue(ready.wait(10), "latency proxy did not start")
+ self.addCleanup(proxy_thread.join, 10)
+ self.addCleanup(lambda: proxy["loop"].call_soon_threadsafe(proxy["stop_event"].set))
+ proxy_addr = "127.0.0.1:%d" % proxy["port"]
+
+ N = 400
+ expected = []
+ for i in range(N):
+ taskhash = hashlib.sha256(("task%d" % i).encode()).hexdigest()
+ outhash = hashlib.sha256(("out%d" % i).encode()).hexdigest()
+ unihash = hashlib.sha256(("uni%d" % i).encode()).hexdigest()
+ self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ expected.append((self.METHOD, taskhash, unihash))
+
+ # Downstream server with an EMPTY local DB whose upstream is the slow proxy.
+ down_server = self.start_server(upstream=proxy_addr)
+ down_client = self.start_client(down_server.address)
+
+ args = [(m, th) for (m, th, _uh) in expected]
+ start = time.time()
+ results = down_client.get_unihash_batch(args)
+ elapsed = time.time() - start
+
+ self.assertEqual(results, [uh for (_m, _th, uh) in expected])
+
+ # A serial (one round-trip per task) implementation cannot beat N*RTT.
+ # Allow a generous margin to avoid flakiness on loaded CI machines.
+ serial_lower_bound = N * 2 * LATENCY
+ self.assertLess(elapsed, serial_lower_bound / 4,
+ "Upstream queries are not being pipelined "
+ "(%.3fs for %d queries at %.0fms injected RTT)"
+ % (elapsed, N, 2 * LATENCY * 1000))
+
+
class TestHashEquivalenceWebsocketServer(HashEquivalenceTestSetup, HashEquivalenceCommonTests, unittest.TestCase):
def setUp(self):
@@ -1510,4 +1898,3 @@ class TestHashEquivalenceExternalServer(HashEquivalenceTestSetup, HashEquivalenc
def test_auth_get_all_users(self):
self.skipTest("Cannot test all users with external server")
-
diff --git a/bitbake/lib/layerindexlib/__init__.py b/bitbake/lib/layerindexlib/__init__.py
index c3265dda..e693e5c1 100644
--- a/bitbake/lib/layerindexlib/__init__.py
+++ b/bitbake/lib/layerindexlib/__init__.py
@@ -174,15 +174,15 @@ class LayerIndex():
return res
- def load_layerindex(self, indexURI, load=['layerDependencies', 'recipes', 'machines', 'distros'], reload=False):
+ def load_layerindex(self, indexURI, load=[], reload=False):
'''Load the layerindex.
indexURI - An index to load. (Use multiple calls to load multiple indexes)
reload - If reload is True, then any previously loaded indexes will be forgotten.
- load - List of elements to load. Default loads all items.
- Note: plugs may ignore this.
+ load - List of elements to load. By default, an empty list is used to keep things lean.
+ Callers need to specify a minimal set of elements to load, such as ['layerDependencies'] for dependency resolution.
The format of the indexURI:
diff --git a/bitbake/lib/layerindexlib/cooker.py b/bitbake/lib/layerindexlib/cooker.py
index ced3e063..82d77335 100644
--- a/bitbake/lib/layerindexlib/cooker.py
+++ b/bitbake/lib/layerindexlib/cooker.py
@@ -55,10 +55,10 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin):
bb_path = os.path.dirname(bb_path) # .../bitbake/lib/layerindex
bb_path = os.path.dirname(bb_path) # .../bitbake/lib
bb_path = os.path.dirname(bb_path) # .../bitbake
- bb_path = self._run_command('git rev-parse --show-toplevel', os.path.dirname(__file__), default=bb_path)
- bb_branch = self._run_command('git rev-parse --abbrev-ref HEAD', bb_path, default="<unknown>")
- bb_rev = self._run_command('git rev-parse HEAD', bb_path, default="<unknown>")
- for remotes in self._run_command('git remote -v', bb_path, default="").split("\n"):
+ bb_path = self._run_command(['git', 'rev-parse', '--show-toplevel'], os.path.dirname(__file__), default=bb_path)
+ bb_branch = self._run_command(['git', 'rev-parse', '--abbrev-ref', 'HEAD'], bb_path, default="<unknown>")
+ bb_rev = self._run_command(['git', 'rev-parse', 'HEAD'], bb_path, default="<unknown>")
+ for remotes in self._run_command(['git', 'remote', '-v'], bb_path, default="").split("\n"):
remote = remotes.split("\t")[1].split(" ")[0]
if "(fetch)" == remotes.split("\t")[1].split(" ")[1]:
bb_remote = self._handle_git_remote(remote)
@@ -118,14 +118,14 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin):
layerbranch = "<unknown>"
if os.path.isdir(layerpath):
- layerbasepath = self._run_command('git rev-parse --show-toplevel', layerpath, default=layerpath)
+ layerbasepath = self._run_command(['git', 'rev-parse', '--show-toplevel'], layerpath, default=layerpath)
if os.path.abspath(layerpath) != os.path.abspath(layerbasepath):
layersubdir = os.path.abspath(layerpath)[len(layerbasepath) + 1:]
- layerbranch = self._run_command('git rev-parse --abbrev-ref HEAD', layerpath, default="<unknown>")
- layerrev = self._run_command('git rev-parse HEAD', layerpath, default="<unknown>")
+ layerbranch = self._run_command(['git', 'rev-parse', '--abbrev-ref', 'HEAD'], layerpath, default="<unknown>")
+ layerrev = self._run_command(['git', 'rev-parse', 'HEAD'], layerpath, default="<unknown>")
- for remotes in self._run_command('git remote -v', layerpath, default="").split("\n"):
+ for remotes in self._run_command(['git', 'remote', '-v'], layerpath, default="").split("\n"):
if not remotes:
layerurl = self._handle_git_remote(layerpath)
else:
diff --git a/bitbake/lib/layerindexlib/restapi.py b/bitbake/lib/layerindexlib/restapi.py
index 81d99b02..65405cb7 100644
--- a/bitbake/lib/layerindexlib/restapi.py
+++ b/bitbake/lib/layerindexlib/restapi.py
@@ -213,7 +213,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin):
pindex['branches'] = _get_json_response(index.apilinks['branches'] + filter,
username=up.username, password=up.password)
if not pindex['branches']:
- logger.debug("No valid branches (%s) found at url %s." % (branch, url))
+ logger.debug("No valid branches (%s) found at url %s." % (branches, url))
return index
index.add_raw_element("branches", layerindexlib.Branch, pindex['branches'])
diff --git a/bitbake/lib/ply/lex.py b/bitbake/lib/ply/lex.py
deleted file mode 100644
index 182f2e83..00000000
--- a/bitbake/lib/ply/lex.py
+++ /dev/null
@@ -1,1054 +0,0 @@
-# -----------------------------------------------------------------------------
-# ply: lex.py
-#
-# Copyright (C) 2001-2009,
-# David M. Beazley (Dabeaz LLC)
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#
-# * Redistributions of source code must retain the above copyright notice,
-# this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright notice,
-# this list of conditions and the following disclaimer in the documentation
-# and/or other materials provided with the distribution.
-# * Neither the name of the David Beazley or Dabeaz LLC may be used to
-# endorse or promote products derived from this software without
-# specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-# -----------------------------------------------------------------------------
-
-__version__ = "3.3"
-__tabversion__ = "3.2" # Version of table file used
-
-import re, sys, types, copy, os
-
-# This tuple contains known string types
-try:
- # Python 2.6
- StringTypes = (types.StringType, types.UnicodeType)
-except AttributeError:
- # Python 3.0
- StringTypes = (str, bytes)
-
-# Extract the code attribute of a function. Different implementations
-# are for Python 2/3 compatibility.
-
-if sys.version_info[0] < 3:
- def func_code(f):
- return f.func_code
-else:
- def func_code(f):
- return f.__code__
-
-# This regular expression is used to match valid token names
-_is_identifier = re.compile(r'^[a-zA-Z0-9_]+$')
-
-# Exception thrown when invalid token encountered and no default error
-# handler is defined.
-
-class LexError(Exception):
- def __init__(self,message,s):
- self.args = (message,)
- self.text = s
-
-# Token class. This class is used to represent the tokens produced.
-class LexToken(object):
- def __str__(self):
- return "LexToken(%s,%r,%d,%d)" % (self.type,self.value,self.lineno,self.lexpos)
- def __repr__(self):
- return str(self)
-
-# This object is a stand-in for a logging object created by the
-# logging module.
-
-class PlyLogger(object):
- def __init__(self,f):
- self.f = f
- def critical(self,msg,*args,**kwargs):
- self.f.write((msg % args) + "\n")
-
- def warning(self,msg,*args,**kwargs):
- self.f.write("WARNING: "+ (msg % args) + "\n")
-
- def error(self,msg,*args,**kwargs):
- self.f.write("ERROR: " + (msg % args) + "\n")
-
- info = critical
- debug = critical
-
-# Null logger is used when no output is generated. Does nothing.
-class NullLogger(object):
- def __getattribute__(self,name):
- return self
- def __call__(self,*args,**kwargs):
- return self
-
-# -----------------------------------------------------------------------------
-# === Lexing Engine ===
-#
-# The following Lexer class implements the lexer runtime. There are only
-# a few public methods and attributes:
-#
-# input() - Store a new string in the lexer
-# token() - Get the next token
-# clone() - Clone the lexer
-#
-# lineno - Current line number
-# lexpos - Current position in the input string
-# -----------------------------------------------------------------------------
-
-class Lexer:
- def __init__(self):
- self.lexre = None # Master regular expression. This is a list of
- # tuples (re,findex) where re is a compiled
- # regular expression and findex is a list
- # mapping regex group numbers to rules
- self.lexretext = None # Current regular expression strings
- self.lexstatere = {} # Dictionary mapping lexer states to master regexs
- self.lexstateretext = {} # Dictionary mapping lexer states to regex strings
- self.lexstaterenames = {} # Dictionary mapping lexer states to symbol names
- self.lexstate = "INITIAL" # Current lexer state
- self.lexstatestack = [] # Stack of lexer states
- self.lexstateinfo = None # State information
- self.lexstateignore = {} # Dictionary of ignored characters for each state
- self.lexstateerrorf = {} # Dictionary of error functions for each state
- self.lexreflags = 0 # Optional re compile flags
- self.lexdata = None # Actual input data (as a string)
- self.lexpos = 0 # Current position in input text
- self.lexlen = 0 # Length of the input text
- self.lexerrorf = None # Error rule (if any)
- self.lextokens = None # List of valid tokens
- self.lexignore = "" # Ignored characters
- self.lexliterals = "" # Literal characters that can be passed through
- self.lexmodule = None # Module
- self.lineno = 1 # Current line number
- self.lexoptimize = 0 # Optimized mode
-
- def clone(self,object=None):
- c = copy.copy(self)
-
- # If the object parameter has been supplied, it means we are attaching the
- # lexer to a new object. In this case, we have to rebind all methods in
- # the lexstatere and lexstateerrorf tables.
-
- if object:
- newtab = { }
- for key, ritem in self.lexstatere.items():
- newre = []
- for cre, findex in ritem:
- newfindex = []
- for f in findex:
- if not f or not f[0]:
- newfindex.append(f)
- continue
- newfindex.append((getattr(object,f[0].__name__),f[1]))
- newre.append((cre,newfindex))
- newtab[key] = newre
- c.lexstatere = newtab
- c.lexstateerrorf = { }
- for key, ef in self.lexstateerrorf.items():
- c.lexstateerrorf[key] = getattr(object,ef.__name__)
- c.lexmodule = object
- return c
-
- # ------------------------------------------------------------
- # writetab() - Write lexer information to a table file
- # ------------------------------------------------------------
- def writetab(self,tabfile,outputdir=""):
- if isinstance(tabfile,types.ModuleType):
- return
- basetabfilename = tabfile.split(".")[-1]
- filename = os.path.join(outputdir,basetabfilename)+".py"
- tf = open(filename,"w")
- tf.write("# %s.py. This file automatically created by PLY (version %s). Don't edit!\n" % (tabfile,__version__))
- tf.write("_tabversion = %s\n" % repr(__version__))
- tf.write("_lextokens = %s\n" % repr(self.lextokens))
- tf.write("_lexreflags = %s\n" % repr(self.lexreflags))
- tf.write("_lexliterals = %s\n" % repr(self.lexliterals))
- tf.write("_lexstateinfo = %s\n" % repr(self.lexstateinfo))
-
- tabre = { }
- # Collect all functions in the initial state
- initial = self.lexstatere["INITIAL"]
- initialfuncs = []
- for part in initial:
- for f in part[1]:
- if f and f[0]:
- initialfuncs.append(f)
-
- for key, lre in self.lexstatere.items():
- titem = []
- for i in range(len(lre)):
- titem.append((self.lexstateretext[key][i],_funcs_to_names(lre[i][1],self.lexstaterenames[key][i])))
- tabre[key] = titem
-
- tf.write("_lexstatere = %s\n" % repr(tabre))
- tf.write("_lexstateignore = %s\n" % repr(self.lexstateignore))
-
- taberr = { }
- for key, ef in self.lexstateerrorf.items():
- if ef:
- taberr[key] = ef.__name__
- else:
- taberr[key] = None
- tf.write("_lexstateerrorf = %s\n" % repr(taberr))
- tf.close()
-
- # ------------------------------------------------------------
- # readtab() - Read lexer information from a tab file
- # ------------------------------------------------------------
- def readtab(self,tabfile,fdict):
- if isinstance(tabfile,types.ModuleType):
- lextab = tabfile
- else:
- if sys.version_info[0] < 3:
- exec("import %s as lextab" % tabfile)
- else:
- env = { }
- exec("import %s as lextab" % tabfile, env,env)
- lextab = env['lextab']
-
- if getattr(lextab,"_tabversion","0.0") != __version__:
- raise ImportError("Inconsistent PLY version")
-
- self.lextokens = lextab._lextokens
- self.lexreflags = lextab._lexreflags
- self.lexliterals = lextab._lexliterals
- self.lexstateinfo = lextab._lexstateinfo
- self.lexstateignore = lextab._lexstateignore
- self.lexstatere = { }
- self.lexstateretext = { }
- for key,lre in lextab._lexstatere.items():
- titem = []
- txtitem = []
- for i in range(len(lre)):
- titem.append((re.compile(lre[i][0],lextab._lexreflags | re.VERBOSE),_names_to_funcs(lre[i][1],fdict)))
- txtitem.append(lre[i][0])
- self.lexstatere[key] = titem
- self.lexstateretext[key] = txtitem
- self.lexstateerrorf = { }
- for key,ef in lextab._lexstateerrorf.items():
- self.lexstateerrorf[key] = fdict[ef]
- self.begin('INITIAL')
-
- # ------------------------------------------------------------
- # input() - Push a new string into the lexer
- # ------------------------------------------------------------
- def input(self,s):
- # Pull off the first character to see if s looks like a string
- c = s[:1]
- if not isinstance(c,StringTypes):
- raise ValueError("Expected a string")
- self.lexdata = s
- self.lexpos = 0
- self.lexlen = len(s)
-
- # ------------------------------------------------------------
- # begin() - Changes the lexing state
- # ------------------------------------------------------------
- def begin(self,state):
- if not state in self.lexstatere:
- raise ValueError("Undefined state")
- self.lexre = self.lexstatere[state]
- self.lexretext = self.lexstateretext[state]
- self.lexignore = self.lexstateignore.get(state,"")
- self.lexerrorf = self.lexstateerrorf.get(state,None)
- self.lexstate = state
-
- # ------------------------------------------------------------
- # push_state() - Changes the lexing state and saves old on stack
- # ------------------------------------------------------------
- def push_state(self,state):
- self.lexstatestack.append(self.lexstate)
- self.begin(state)
-
- # ------------------------------------------------------------
- # pop_state() - Restores the previous state
- # ------------------------------------------------------------
- def pop_state(self):
- self.begin(self.lexstatestack.pop())
-
- # ------------------------------------------------------------
- # current_state() - Returns the current lexing state
- # ------------------------------------------------------------
- def current_state(self):
- return self.lexstate
-
- # ------------------------------------------------------------
- # skip() - Skip ahead n characters
- # ------------------------------------------------------------
- def skip(self,n):
- self.lexpos += n
-
- # ------------------------------------------------------------
- # opttoken() - Return the next token from the Lexer
- #
- # Note: This function has been carefully implemented to be as fast
- # as possible. Don't make changes unless you really know what
- # you are doing
- # ------------------------------------------------------------
- def token(self):
- # Make local copies of frequently referenced attributes
- lexpos = self.lexpos
- lexlen = self.lexlen
- lexignore = self.lexignore
- lexdata = self.lexdata
-
- while lexpos < lexlen:
- # This code provides some short-circuit code for whitespace, tabs, and other ignored characters
- if lexdata[lexpos] in lexignore:
- lexpos += 1
- continue
-
- # Look for a regular expression match
- for lexre,lexindexfunc in self.lexre:
- m = lexre.match(lexdata,lexpos)
- if not m: continue
-
- # Create a token for return
- tok = LexToken()
- tok.value = m.group()
- tok.lineno = self.lineno
- tok.lexpos = lexpos
-
- i = m.lastindex
- func,tok.type = lexindexfunc[i]
-
- if not func:
- # If no token type was set, it's an ignored token
- if tok.type:
- self.lexpos = m.end()
- return tok
- else:
- lexpos = m.end()
- break
-
- lexpos = m.end()
-
- # If token is processed by a function, call it
-
- tok.lexer = self # Set additional attributes useful in token rules
- self.lexmatch = m
- self.lexpos = lexpos
-
- newtok = func(tok)
-
- # Every function must return a token, if nothing, we just move to next token
- if not newtok:
- lexpos = self.lexpos # This is here in case user has updated lexpos.
- lexignore = self.lexignore # This is here in case there was a state change
- break
-
- # Verify type of the token. If not in the token map, raise an error
- if not self.lexoptimize:
- if not newtok.type in self.lextokens:
- raise LexError("%s:%d: Rule '%s' returned an unknown token type '%s'" % (
- func_code(func).co_filename, func_code(func).co_firstlineno,
- func.__name__, newtok.type),lexdata[lexpos:])
-
- return newtok
- else:
- # No match, see if in literals
- if lexdata[lexpos] in self.lexliterals:
- tok = LexToken()
- tok.value = lexdata[lexpos]
- tok.lineno = self.lineno
- tok.type = tok.value
- tok.lexpos = lexpos
- self.lexpos = lexpos + 1
- return tok
-
- # No match. Call t_error() if defined.
- if self.lexerrorf:
- tok = LexToken()
- tok.value = self.lexdata[lexpos:]
- tok.lineno = self.lineno
- tok.type = "error"
- tok.lexer = self
- tok.lexpos = lexpos
- self.lexpos = lexpos
- newtok = self.lexerrorf(tok)
- if lexpos == self.lexpos:
- # Error method didn't change text position at all. This is an error.
- raise LexError("Scanning error. Illegal character '%s'" % (lexdata[lexpos]), lexdata[lexpos:])
- lexpos = self.lexpos
- if not newtok: continue
- return newtok
-
- self.lexpos = lexpos
- raise LexError("Illegal character '%s' at index %d" % (lexdata[lexpos],lexpos), lexdata[lexpos:])
-
- self.lexpos = lexpos + 1
- if self.lexdata is None:
- raise RuntimeError("No input string given with input()")
- return None
-
- # Iterator interface
- def __iter__(self):
- return self
-
- def next(self):
- t = self.token()
- if t is None:
- raise StopIteration
- return t
-
- __next__ = next
-
-# -----------------------------------------------------------------------------
-# ==== Lex Builder ===
-#
-# The functions and classes below are used to collect lexing information
-# and build a Lexer object from it.
-# -----------------------------------------------------------------------------
-
-# -----------------------------------------------------------------------------
-# get_caller_module_dict()
-#
-# This function returns a dictionary containing all of the symbols defined within
-# a caller further down the call stack. This is used to get the environment
-# associated with the yacc() call if none was provided.
-# -----------------------------------------------------------------------------
-
-def get_caller_module_dict(levels):
- try:
- raise RuntimeError
- except RuntimeError:
- e,b,t = sys.exc_info()
- f = t.tb_frame
- while levels > 0:
- f = f.f_back
- levels -= 1
- ldict = f.f_globals.copy()
- if f.f_globals != f.f_locals:
- ldict.update(f.f_locals)
-
- return ldict
-
-# -----------------------------------------------------------------------------
-# _funcs_to_names()
-#
-# Given a list of regular expression functions, this converts it to a list
-# suitable for output to a table file
-# -----------------------------------------------------------------------------
-
-def _funcs_to_names(funclist,namelist):
- result = []
- for f,name in zip(funclist,namelist):
- if f and f[0]:
- result.append((name, f[1]))
- else:
- result.append(f)
- return result
-
-# -----------------------------------------------------------------------------
-# _names_to_funcs()
-#
-# Given a list of regular expression function names, this converts it back to
-# functions.
-# -----------------------------------------------------------------------------
-
-def _names_to_funcs(namelist,fdict):
- result = []
- for n in namelist:
- if n and n[0]:
- result.append((fdict[n[0]],n[1]))
- else:
- result.append(n)
- return result
-
-# -----------------------------------------------------------------------------
-# _form_master_re()
-#
-# This function takes a list of all of the regex components and attempts to
-# form the master regular expression. Given limitations in the Python re
-# module, it may be necessary to break the master regex into separate expressions.
-# -----------------------------------------------------------------------------
-
-def _form_master_re(relist,reflags,ldict,toknames):
- if not relist: return []
- regex = "|".join(relist)
- try:
- lexre = re.compile(regex,re.VERBOSE | reflags)
-
- # Build the index to function map for the matching engine
- lexindexfunc = [ None ] * (max(lexre.groupindex.values())+1)
- lexindexnames = lexindexfunc[:]
-
- for f,i in lexre.groupindex.items():
- handle = ldict.get(f,None)
- if type(handle) in (types.FunctionType, types.MethodType):
- lexindexfunc[i] = (handle,toknames[f])
- lexindexnames[i] = f
- elif handle is not None:
- lexindexnames[i] = f
- if f.find("ignore_") > 0:
- lexindexfunc[i] = (None,None)
- else:
- lexindexfunc[i] = (None, toknames[f])
-
- return [(lexre,lexindexfunc)],[regex],[lexindexnames]
- except Exception:
- m = int(len(relist)/2)
- if m == 0: m = 1
- llist, lre, lnames = _form_master_re(relist[:m],reflags,ldict,toknames)
- rlist, rre, rnames = _form_master_re(relist[m:],reflags,ldict,toknames)
- return llist+rlist, lre+rre, lnames+rnames
-
-# -----------------------------------------------------------------------------
-# def _statetoken(s,names)
-#
-# Given a declaration name s of the form "t_" and a dictionary whose keys are
-# state names, this function returns a tuple (states,tokenname) where states
-# is a tuple of state names and tokenname is the name of the token. For example,
-# calling this with s = "t_foo_bar_SPAM" might return (('foo','bar'),'SPAM')
-# -----------------------------------------------------------------------------
-
-def _statetoken(s,names):
- nonstate = 1
- parts = s.split("_")
- for i in range(1,len(parts)):
- if not parts[i] in names and parts[i] != 'ANY': break
- if i > 1:
- states = tuple(parts[1:i])
- else:
- states = ('INITIAL',)
-
- if 'ANY' in states:
- states = tuple(names)
-
- tokenname = "_".join(parts[i:])
- return (states,tokenname)
-
-
-# -----------------------------------------------------------------------------
-# LexerReflect()
-#
-# This class represents information needed to build a lexer as extracted from a
-# user's input file.
-# -----------------------------------------------------------------------------
-class LexerReflect(object):
- def __init__(self,ldict,log=None,reflags=0):
- self.ldict = ldict
- self.error_func = None
- self.tokens = []
- self.reflags = reflags
- self.stateinfo = { 'INITIAL' : 'inclusive'}
- self.files = {}
- self.error = 0
-
- if log is None:
- self.log = PlyLogger(sys.stderr)
- else:
- self.log = log
-
- # Get all of the basic information
- def get_all(self):
- self.get_tokens()
- self.get_literals()
- self.get_states()
- self.get_rules()
-
- # Validate all of the information
- def validate_all(self):
- self.validate_tokens()
- self.validate_literals()
- self.validate_rules()
- return self.error
-
- # Get the tokens map
- def get_tokens(self):
- tokens = self.ldict.get("tokens",None)
- if not tokens:
- self.log.error("No token list is defined")
- self.error = 1
- return
-
- if not isinstance(tokens,(list, tuple)):
- self.log.error("tokens must be a list or tuple")
- self.error = 1
- return
-
- if not tokens:
- self.log.error("tokens is empty")
- self.error = 1
- return
-
- self.tokens = tokens
-
- # Validate the tokens
- def validate_tokens(self):
- terminals = {}
- for n in self.tokens:
- if not _is_identifier.match(n):
- self.log.error("Bad token name '%s'",n)
- self.error = 1
- if n in terminals:
- self.log.warning("Token '%s' multiply defined", n)
- terminals[n] = 1
-
- # Get the literals specifier
- def get_literals(self):
- self.literals = self.ldict.get("literals","")
-
- # Validate literals
- def validate_literals(self):
- try:
- for c in self.literals:
- if not isinstance(c,StringTypes) or len(c) > 1:
- self.log.error("Invalid literal %s. Must be a single character", repr(c))
- self.error = 1
- continue
-
- except TypeError:
- self.log.error("Invalid literals specification. literals must be a sequence of characters")
- self.error = 1
-
- def get_states(self):
- self.states = self.ldict.get("states",None)
- # Build statemap
- if self.states:
- if not isinstance(self.states,(tuple,list)):
- self.log.error("states must be defined as a tuple or list")
- self.error = 1
- else:
- for s in self.states:
- if not isinstance(s,tuple) or len(s) != 2:
- self.log.error("Invalid state specifier %s. Must be a tuple (statename,'exclusive|inclusive')",repr(s))
- self.error = 1
- continue
- name, statetype = s
- if not isinstance(name,StringTypes):
- self.log.error("State name %s must be a string", repr(name))
- self.error = 1
- continue
- if not (statetype == 'inclusive' or statetype == 'exclusive'):
- self.log.error("State type for state %s must be 'inclusive' or 'exclusive'",name)
- self.error = 1
- continue
- if name in self.stateinfo:
- self.log.error("State '%s' already defined",name)
- self.error = 1
- continue
- self.stateinfo[name] = statetype
-
- # Get all of the symbols with a t_ prefix and sort them into various
- # categories (functions, strings, error functions, and ignore characters)
-
- def get_rules(self):
- tsymbols = [f for f in self.ldict if f[:2] == 't_' ]
-
- # Now build up a list of functions and a list of strings
-
- self.toknames = { } # Mapping of symbols to token names
- self.funcsym = { } # Symbols defined as functions
- self.strsym = { } # Symbols defined as strings
- self.ignore = { } # Ignore strings by state
- self.errorf = { } # Error functions by state
-
- for s in self.stateinfo:
- self.funcsym[s] = []
- self.strsym[s] = []
-
- if len(tsymbols) == 0:
- self.log.error("No rules of the form t_rulename are defined")
- self.error = 1
- return
-
- for f in tsymbols:
- t = self.ldict[f]
- states, tokname = _statetoken(f,self.stateinfo)
- self.toknames[f] = tokname
-
- if hasattr(t,"__call__"):
- if tokname == 'error':
- for s in states:
- self.errorf[s] = t
- elif tokname == 'ignore':
- line = func_code(t).co_firstlineno
- file = func_code(t).co_filename
- self.log.error("%s:%d: Rule '%s' must be defined as a string",file,line,t.__name__)
- self.error = 1
- else:
- for s in states:
- self.funcsym[s].append((f,t))
- elif isinstance(t, StringTypes):
- if tokname == 'ignore':
- for s in states:
- self.ignore[s] = t
- if "\\" in t:
- self.log.warning("%s contains a literal backslash '\\'",f)
-
- elif tokname == 'error':
- self.log.error("Rule '%s' must be defined as a function", f)
- self.error = 1
- else:
- for s in states:
- self.strsym[s].append((f,t))
- else:
- self.log.error("%s not defined as a function or string", f)
- self.error = 1
-
- # Sort the functions by line number
- for f in self.funcsym.values():
- f.sort(key=lambda x: func_code(x[1]).co_firstlineno)
-
- # Sort the strings by regular expression length
- for s in self.strsym.values():
- if sys.version_info[0] < 3:
- s.sort(lambda x,y: (len(x[1]) < len(y[1])) - (len(x[1]) > len(y[1])))
- else:
- # Python 3.0
- s.sort(key=lambda x: len(x[1]),reverse=True)
-
- # Validate all of the t_rules collected
- def validate_rules(self):
- for state in self.stateinfo:
- # Validate all rules defined by functions
-
-
-
- for fname, f in self.funcsym[state]:
- line = func_code(f).co_firstlineno
- file = func_code(f).co_filename
- self.files[file] = 1
-
- tokname = self.toknames[fname]
- if isinstance(f, types.MethodType):
- reqargs = 2
- else:
- reqargs = 1
- nargs = func_code(f).co_argcount
- if nargs > reqargs:
- self.log.error("%s:%d: Rule '%s' has too many arguments",file,line,f.__name__)
- self.error = 1
- continue
-
- if nargs < reqargs:
- self.log.error("%s:%d: Rule '%s' requires an argument", file,line,f.__name__)
- self.error = 1
- continue
-
- if not f.__doc__:
- self.log.error("%s:%d: No regular expression defined for rule '%s'",file,line,f.__name__)
- self.error = 1
- continue
-
- try:
- c = re.compile("(?P<%s>%s)" % (fname,f.__doc__), re.VERBOSE | self.reflags)
- if c.match(""):
- self.log.error("%s:%d: Regular expression for rule '%s' matches empty string", file,line,f.__name__)
- self.error = 1
- except re.error:
- _etype, e, _etrace = sys.exc_info()
- self.log.error("%s:%d: Invalid regular expression for rule '%s'. %s", file,line,f.__name__,e)
- if '#' in f.__doc__:
- self.log.error("%s:%d. Make sure '#' in rule '%s' is escaped with '\\#'",file,line, f.__name__)
- self.error = 1
-
- # Validate all rules defined by strings
- for name,r in self.strsym[state]:
- tokname = self.toknames[name]
- if tokname == 'error':
- self.log.error("Rule '%s' must be defined as a function", name)
- self.error = 1
- continue
-
- if not tokname in self.tokens and tokname.find("ignore_") < 0:
- self.log.error("Rule '%s' defined for an unspecified token %s",name,tokname)
- self.error = 1
- continue
-
- try:
- c = re.compile("(?P<%s>%s)" % (name,r),re.VERBOSE | self.reflags)
- if (c.match("")):
- self.log.error("Regular expression for rule '%s' matches empty string",name)
- self.error = 1
- except re.error:
- _etype, e, _etrace = sys.exc_info()
- self.log.error("Invalid regular expression for rule '%s'. %s",name,e)
- if '#' in r:
- self.log.error("Make sure '#' in rule '%s' is escaped with '\\#'",name)
- self.error = 1
-
- if not self.funcsym[state] and not self.strsym[state]:
- self.log.error("No rules defined for state '%s'",state)
- self.error = 1
-
- # Validate the error function
- efunc = self.errorf.get(state,None)
- if efunc:
- f = efunc
- line = func_code(f).co_firstlineno
- file = func_code(f).co_filename
- self.files[file] = 1
-
- if isinstance(f, types.MethodType):
- reqargs = 2
- else:
- reqargs = 1
- nargs = func_code(f).co_argcount
- if nargs > reqargs:
- self.log.error("%s:%d: Rule '%s' has too many arguments",file,line,f.__name__)
- self.error = 1
-
- if nargs < reqargs:
- self.log.error("%s:%d: Rule '%s' requires an argument", file,line,f.__name__)
- self.error = 1
-
- for f in self.files:
- self.validate_file(f)
-
-
- # -----------------------------------------------------------------------------
- # validate_file()
- #
- # This checks to see if there are duplicated t_rulename() functions or strings
- # in the parser input file. This is done using a simple regular expression
- # match on each line in the given file.
- # -----------------------------------------------------------------------------
-
- def validate_file(self,filename):
- import os.path
- base,ext = os.path.splitext(filename)
- if ext != '.py': return # No idea what the file is. Return OK
-
- try:
- f = open(filename)
- lines = f.readlines()
- f.close()
- except IOError:
- return # Couldn't find the file. Don't worry about it
-
- fre = re.compile(r'\s*def\s+(t_[a-zA-Z_0-9]*)\(')
- sre = re.compile(r'\s*(t_[a-zA-Z_0-9]*)\s*=')
-
- counthash = { }
- linen = 1
- for l in lines:
- m = fre.match(l)
- if not m:
- m = sre.match(l)
- if m:
- name = m.group(1)
- prev = counthash.get(name)
- if not prev:
- counthash[name] = linen
- else:
- self.log.error("%s:%d: Rule %s redefined. Previously defined on line %d",filename,linen,name,prev)
- self.error = 1
- linen += 1
-
-# -----------------------------------------------------------------------------
-# lex(module)
-#
-# Build all of the regular expression rules from definitions in the supplied module
-# -----------------------------------------------------------------------------
-def lex(module=None,object=None,debug=0,optimize=0,lextab="lextab",reflags=0,nowarn=0,outputdir="", debuglog=None, errorlog=None):
- global lexer
- ldict = None
- stateinfo = { 'INITIAL' : 'inclusive'}
- lexobj = Lexer()
- lexobj.lexoptimize = optimize
- global token,input
-
- if errorlog is None:
- errorlog = PlyLogger(sys.stderr)
-
- if debug:
- if debuglog is None:
- debuglog = PlyLogger(sys.stderr)
-
- # Get the module dictionary used for the lexer
- if object: module = object
-
- if module:
- _items = [(k,getattr(module,k)) for k in dir(module)]
- ldict = dict(_items)
- else:
- ldict = get_caller_module_dict(2)
-
- # Collect parser information from the dictionary
- linfo = LexerReflect(ldict,log=errorlog,reflags=reflags)
- linfo.get_all()
- if not optimize:
- if linfo.validate_all():
- raise SyntaxError("Can't build lexer")
-
- if optimize and lextab:
- try:
- lexobj.readtab(lextab,ldict)
- token = lexobj.token
- input = lexobj.input
- lexer = lexobj
- return lexobj
-
- except ImportError:
- pass
-
- # Dump some basic debugging information
- if debug:
- debuglog.info("lex: tokens = %r", linfo.tokens)
- debuglog.info("lex: literals = %r", linfo.literals)
- debuglog.info("lex: states = %r", linfo.stateinfo)
-
- # Build a dictionary of valid token names
- lexobj.lextokens = { }
- for n in linfo.tokens:
- lexobj.lextokens[n] = 1
-
- # Get literals specification
- if isinstance(linfo.literals,(list,tuple)):
- lexobj.lexliterals = type(linfo.literals[0])().join(linfo.literals)
- else:
- lexobj.lexliterals = linfo.literals
-
- # Get the stateinfo dictionary
- stateinfo = linfo.stateinfo
-
- regexs = { }
- # Build the master regular expressions
- for state in stateinfo:
- regex_list = []
-
- # Add rules defined by functions first
- for fname, f in linfo.funcsym[state]:
- line = func_code(f).co_firstlineno
- file = func_code(f).co_filename
- regex_list.append("(?P<%s>%s)" % (fname,f.__doc__))
- if debug:
- debuglog.info("lex: Adding rule %s -> '%s' (state '%s')",fname,f.__doc__, state)
-
- # Now add all of the simple rules
- for name,r in linfo.strsym[state]:
- regex_list.append("(?P<%s>%s)" % (name,r))
- if debug:
- debuglog.info("lex: Adding rule %s -> '%s' (state '%s')",name,r, state)
-
- regexs[state] = regex_list
-
- # Build the master regular expressions
-
- if debug:
- debuglog.info("lex: ==== MASTER REGEXS FOLLOW ====")
-
- for state in regexs:
- lexre, re_text, re_names = _form_master_re(regexs[state],reflags,ldict,linfo.toknames)
- lexobj.lexstatere[state] = lexre
- lexobj.lexstateretext[state] = re_text
- lexobj.lexstaterenames[state] = re_names
- if debug:
- for i in range(len(re_text)):
- debuglog.info("lex: state '%s' : regex[%d] = '%s'",state, i, re_text[i])
-
- # For inclusive states, we need to add the regular expressions from the INITIAL state
- for state,stype in stateinfo.items():
- if state != "INITIAL" and stype == 'inclusive':
- lexobj.lexstatere[state].extend(lexobj.lexstatere['INITIAL'])
- lexobj.lexstateretext[state].extend(lexobj.lexstateretext['INITIAL'])
- lexobj.lexstaterenames[state].extend(lexobj.lexstaterenames['INITIAL'])
-
- lexobj.lexstateinfo = stateinfo
- lexobj.lexre = lexobj.lexstatere["INITIAL"]
- lexobj.lexretext = lexobj.lexstateretext["INITIAL"]
- lexobj.lexreflags = reflags
-
- # Set up ignore variables
- lexobj.lexstateignore = linfo.ignore
- lexobj.lexignore = lexobj.lexstateignore.get("INITIAL","")
-
- # Set up error functions
- lexobj.lexstateerrorf = linfo.errorf
- lexobj.lexerrorf = linfo.errorf.get("INITIAL",None)
- if not lexobj.lexerrorf:
- errorlog.warning("No t_error rule is defined")
-
- # Check state information for ignore and error rules
- for s,stype in stateinfo.items():
- if stype == 'exclusive':
- if not s in linfo.errorf:
- errorlog.warning("No error rule is defined for exclusive state '%s'", s)
- if not s in linfo.ignore and lexobj.lexignore:
- errorlog.warning("No ignore rule is defined for exclusive state '%s'", s)
- elif stype == 'inclusive':
- if not s in linfo.errorf:
- linfo.errorf[s] = linfo.errorf.get("INITIAL",None)
- if not s in linfo.ignore:
- linfo.ignore[s] = linfo.ignore.get("INITIAL","")
-
- # Create global versions of the token() and input() functions
- token = lexobj.token
- input = lexobj.input
- lexer = lexobj
-
- # If in optimize mode, we write the lextab
- if lextab and optimize:
- lexobj.writetab(lextab,outputdir)
-
- return lexobj
-
-# -----------------------------------------------------------------------------
-# runmain()
-#
-# This runs the lexer as a main program
-# -----------------------------------------------------------------------------
-
-def runmain(lexer=None,data=None):
- if not data:
- try:
- filename = sys.argv[1]
- f = open(filename)
- data = f.read()
- f.close()
- except IndexError:
- sys.stdout.write("Reading from standard input (type EOF to end):\n")
- data = sys.stdin.read()
-
- if lexer:
- _input = lexer.input
- else:
- _input = input
- _input(data)
- if lexer:
- _token = lexer.token
- else:
- _token = token
-
- while 1:
- tok = _token()
- if not tok: break
- sys.stdout.write("(%s,%r,%d,%d)\n" % (tok.type, tok.value, tok.lineno,tok.lexpos))
-
-# -----------------------------------------------------------------------------
-# @TOKEN(regex)
-#
-# This decorator function can be used to set the regex expression on a function
-# when its docstring might need to be set in an alternative way
-# -----------------------------------------------------------------------------
-
-def TOKEN(r):
- def set_doc(f):
- if hasattr(r,"__call__"):
- f.__doc__ = r.__doc__
- else:
- f.__doc__ = r
- return f
- return set_doc
-
-# Alternative spelling of the TOKEN decorator
-Token = TOKEN
-
diff --git a/bitbake/lib/prserv/__init__.py b/bitbake/lib/prserv/__init__.py
index 0e0aa34d..c8c8a982 100644
--- a/bitbake/lib/prserv/__init__.py
+++ b/bitbake/lib/prserv/__init__.py
@@ -4,17 +4,92 @@
# SPDX-License-Identifier: GPL-2.0-only
#
-__version__ = "1.0.0"
-import os, time
-import sys, logging
+__version__ = "2.20.0"
-def init_logger(logfile, loglevel):
- numeric_level = getattr(logging, loglevel.upper(), None)
- if not isinstance(numeric_level, int):
- raise ValueError("Invalid log level: %s" % loglevel)
- FORMAT = "%(asctime)-15s %(message)s"
- logging.basicConfig(level=numeric_level, filename=logfile, format=FORMAT)
+import logging
+logger = logging.getLogger("BitBake.PRserv")
-class NotFoundError(Exception):
- pass
+from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
+
+def create_server(addr, dbpath, upstream=None, read_only=False):
+ from . import serv
+
+ s = serv.PRServer(dbpath, upstream=upstream, read_only=read_only)
+ host, port = addr.split(":")
+ s.start_tcp_server(host, int(port))
+
+ return s
+
+def increase_revision(ver):
+ """Take a revision string such as "1" or "1.2.3" or even a number and increase its last number
+ This fails if the last number is not an integer"""
+
+ fields=str(ver).split('.')
+ last = fields[-1]
+
+ try:
+ val = int(last)
+ except Exception as e:
+ logger.critical("Unable to increase revision value %s: %s" % (ver, e))
+ raise e
+
+ return ".".join(fields[0:-1] + [ str(val + 1) ])
+
+def _revision_greater_or_equal(rev1, rev2):
+ """Compares x.y.z revision numbers, using integer comparison
+ Returns True if rev1 is greater or equal to rev2"""
+
+ fields1 = rev1.split(".")
+ fields2 = rev2.split(".")
+ l1 = len(fields1)
+ l2 = len(fields2)
+
+ for i in range(l1):
+ val1 = int(fields1[i])
+ if i < l2:
+ val2 = int(fields2[i])
+ if val2 < val1:
+ return True
+ elif val2 > val1:
+ return False
+ else:
+ return True
+ return True
+
+def revision_smaller(rev1, rev2):
+ """Compares x.y.z revision numbers, using integer comparison
+ Returns True if rev1 is strictly smaller than rev2"""
+ return not(_revision_greater_or_equal(rev1, rev2))
+
+def revision_greater(rev1, rev2):
+ """Compares x.y.z revision numbers, using integer comparison
+ Returns True if rev1 is strictly greater than rev2"""
+ return _revision_greater_or_equal(rev1, rev2) and (rev1 != rev2)
+
+def create_client(addr):
+ from . import client
+
+ c = client.PRClient()
+
+ try:
+ (typ, a) = parse_address(addr)
+ c.connect_tcp(*a)
+ return c
+ except Exception as e:
+ c.close()
+ raise e
+
+async def create_async_client(addr):
+ from . import client
+
+ c = client.PRAsyncClient()
+
+ try:
+ (typ, a) = parse_address(addr)
+ await c.connect_tcp(*a)
+ return c
+
+ except Exception as e:
+ await c.close()
+ raise e
diff --git a/bitbake/lib/prserv/client.py b/bitbake/lib/prserv/client.py
index 8471ee30..9f5794c4 100644
--- a/bitbake/lib/prserv/client.py
+++ b/bitbake/lib/prserv/client.py
@@ -6,6 +6,7 @@
import logging
import bb.asyncrpc
+from . import create_async_client
logger = logging.getLogger("BitBake.PRserv")
@@ -13,16 +14,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
def __init__(self):
super().__init__("PRSERVICE", "1.0", logger)
- async def getPR(self, version, pkgarch, checksum):
+ async def getPR(self, version, pkgarch, checksum, history=False):
response = await self.invoke(
- {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+ {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
)
if response:
return response["value"]
- async def test_pr(self, version, pkgarch, checksum):
+ async def test_pr(self, version, pkgarch, checksum, history=False):
response = await self.invoke(
- {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum}}
+ {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "history": history}}
)
if response:
return response["value"]
@@ -48,9 +49,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
if response:
return response["value"]
- async def export(self, version, pkgarch, checksum, colinfo):
+ async def export(self, version, pkgarch, checksum, colinfo, history=False):
response = await self.invoke(
- {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo}}
+ {"export": {"version": version, "pkgarch": pkgarch, "checksum": checksum, "colinfo": colinfo, "history": history}}
)
if response:
return (response["metainfo"], response["datainfo"])
@@ -65,7 +66,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
class PRClient(bb.asyncrpc.Client):
def __init__(self):
super().__init__()
- self._add_methods("getPR", "test_pr", "test_package", "importone", "export", "is_readonly")
+ self._add_methods("getPR", "test_pr", "test_package", "max_package_pr", "importone", "export", "is_readonly")
def _get_async_client(self):
return PRAsyncClient()
diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
index eb415081..2da493dd 100644
--- a/bitbake/lib/prserv/db.py
+++ b/bitbake/lib/prserv/db.py
@@ -8,19 +8,13 @@ import logging
import os.path
import errno
import prserv
-import time
+import sqlite3
-try:
- import sqlite3
-except ImportError:
- from pysqlite2 import dbapi2 as sqlite3
+from contextlib import closing
+from . import increase_revision, revision_greater, revision_smaller
logger = logging.getLogger("BitBake.PRserv")
-sqlversion = sqlite3.sqlite_version_info
-if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
- raise Exception("sqlite3 version 3.3.0 or later is required.")
-
#
# "No History" mode - for a given query tuple (version, pkgarch, checksum),
# the returned value will be the largest among all the values of the same
@@ -29,287 +23,232 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
# "History" mode - Return a new higher value for previously unseen query
# tuple (version, pkgarch, checksum), otherwise return historical value.
# Value can decrement if returning to a previous build.
-#
class PRTable(object):
- def __init__(self, conn, table, nohist, read_only):
+ def __init__(self, conn, table, read_only):
self.conn = conn
- self.nohist = nohist
self.read_only = read_only
- self.dirty = False
- if nohist:
- self.table = "%s_nohist" % table
- else:
- self.table = "%s_hist" % table
-
- if self.read_only:
- table_exists = self._execute(
- "SELECT count(*) FROM sqlite_master \
- WHERE type='table' AND name='%s'" % (self.table))
- if not table_exists:
- raise prserv.NotFoundError
- else:
- self._execute("CREATE TABLE IF NOT EXISTS %s \
+ self.table = table
+
+ # Creating the table even if the server is read-only.
+ # This avoids a race condition if a shared database
+ # is accessed by a read-only server first.
+
+ with closing(self.conn.cursor()) as cursor:
+ cursor.execute("CREATE TABLE IF NOT EXISTS %s \
(version TEXT NOT NULL, \
pkgarch TEXT NOT NULL, \
checksum TEXT NOT NULL, \
- value INTEGER, \
- PRIMARY KEY (version, pkgarch, checksum));" % self.table)
-
- def _execute(self, *query):
- """Execute a query, waiting to acquire a lock if necessary"""
- start = time.time()
- end = start + 20
- while True:
- try:
- return self.conn.execute(*query)
- except sqlite3.OperationalError as exc:
- if "is locked" in str(exc) and end > time.time():
- continue
- raise exc
-
- def sync(self):
- if not self.read_only:
+ value TEXT, \
+ PRIMARY KEY (version, pkgarch, checksum, value));" % self.table)
self.conn.commit()
- self._execute("BEGIN EXCLUSIVE TRANSACTION")
- def sync_if_dirty(self):
- if self.dirty:
- self.sync()
- self.dirty = False
+ def _extremum_value(self, rows, is_max):
+ value = None
+
+ for row in rows:
+ current_value = row[0]
+ if value is None:
+ value = current_value
+ else:
+ if is_max:
+ is_new_extremum = revision_greater(current_value, value)
+ else:
+ is_new_extremum = revision_smaller(current_value, value)
+ if is_new_extremum:
+ value = current_value
+ return value
+
+ def _max_value(self, rows):
+ return self._extremum_value(rows, True)
+
+ def _min_value(self, rows):
+ return self._extremum_value(rows, False)
def test_package(self, version, pkgarch):
"""Returns whether the specified package version is found in the database for the specified architecture"""
# Just returns the value if found or None otherwise
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=?;" % self.table,
- (version, pkgarch))
- row=data.fetchone()
- if row is not None:
- return True
- else:
- return False
+ with closing(self.conn.cursor()) as cursor:
+ data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=?;" % self.table,
+ (version, pkgarch))
+ row=data.fetchone()
+ if row is not None:
+ return True
+ else:
+ return False
+
+ def test_checksum_value(self, version, pkgarch, checksum, value):
+ """Returns whether the specified value is found in the database for the specified package, architecture and checksum"""
+
+ with closing(self.conn.cursor()) as cursor:
+ data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and checksum=? and value=?;" % self.table,
+ (version, pkgarch, checksum, value))
+ row=data.fetchone()
+ if row is not None:
+ return True
+ else:
+ return False
def test_value(self, version, pkgarch, value):
"""Returns whether the specified value is found in the database for the specified package and architecture"""
# Just returns the value if found or None otherwise
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and value=?;" % self.table,
- (version, pkgarch, value))
- row=data.fetchone()
- if row is not None:
- return True
- else:
- return False
+ with closing(self.conn.cursor()) as cursor:
+ data=cursor.execute("SELECT value FROM %s WHERE version=? AND pkgarch=? and value=?;" % self.table,
+ (version, pkgarch, value))
+ row=data.fetchone()
+ if row is not None:
+ return True
+ else:
+ return False
- def find_value(self, version, pkgarch, checksum):
+
+ def find_package_max_value(self, version, pkgarch):
+ """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value"""
+
+ with closing(self.conn.cursor()) as cursor:
+ data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=?;" % (self.table),
+ (version, pkgarch))
+ rows = data.fetchall()
+ value = self._max_value(rows)
+ return value
+
+ def find_value(self, version, pkgarch, checksum, history=False):
"""Returns the value for the specified checksum if found or None otherwise."""
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
+ if history:
+ return self.find_min_value(version, pkgarch, checksum)
else:
- return None
+ return self.find_max_value(version, pkgarch, checksum)
- def find_max_value(self, version, pkgarch):
- """Returns the greatest value for (version, pkgarch), or None if not found. Doesn't create a new value"""
- data = self._execute("SELECT max(value) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
- row = data.fetchone()
- if row is not None:
- return row[0]
- else:
- return None
-
- def _get_value_hist(self, version, pkgarch, checksum):
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- #no value found, try to insert
- if self.read_only:
- data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
- row = data.fetchone()
- if row is not None:
- return row[0]
- else:
- return 0
+ def _find_extremum_value(self, version, pkgarch, checksum, is_max):
+ """Returns the maximum (if is_max is True) or minimum (if is_max is False) value
+ for (version, pkgarch, checksum), or None if not found. Doesn't create a new value"""
- try:
- self._execute("INSERT INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
+ with closing(self.conn.cursor()) as cursor:
+ data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=? AND checksum=?;" % (self.table),
+ (version, pkgarch, checksum))
+ rows = data.fetchall()
+ return self._extremum_value(rows, is_max)
- self.dirty = True
+ def find_max_value(self, version, pkgarch, checksum):
+ return self._find_extremum_value(version, pkgarch, checksum, True)
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- raise prserv.NotFoundError
-
- def _get_value_no_hist(self, version, pkgarch, checksum):
- data=self._execute("SELECT value FROM %s \
- WHERE version=? AND pkgarch=? AND checksum=? AND \
- value >= (select max(value) from %s where version=? AND pkgarch=?);"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- row=data.fetchone()
- if row is not None:
- return row[0]
- else:
- #no value found, try to insert
- if self.read_only:
- data = self._execute("SELECT ifnull(max(value)+1, 0) FROM %s where version=? AND pkgarch=?;" % (self.table),
- (version, pkgarch))
- return data.fetchone()[0]
-
- try:
- self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, (select ifnull(max(value)+1, 0) from %s where version=? AND pkgarch=?));"
- % (self.table, self.table),
- (version, pkgarch, checksum, version, pkgarch))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
- self.conn.rollback()
-
- self.dirty = True
-
- data=self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row=data.fetchone()
- if row is not None:
- return row[0]
+ def find_min_value(self, version, pkgarch, checksum):
+ return self._find_extremum_value(version, pkgarch, checksum, False)
+
+ def find_new_subvalue(self, version, pkgarch, base):
+ """Take and increase the greatest "<base>.y" value for (version, pkgarch), or return "<base>.0" if not found.
+ This doesn't store a new value."""
+
+ with closing(self.conn.cursor()) as cursor:
+ data = cursor.execute("SELECT value FROM %s where version=? AND pkgarch=? AND value LIKE '%s.%%';" % (self.table, base),
+ (version, pkgarch))
+ rows = data.fetchall()
+ value = self._max_value(rows)
+
+ if value is not None:
+ return increase_revision(value)
else:
- raise prserv.NotFoundError
+ return base + ".0"
- def get_value(self, version, pkgarch, checksum):
- if self.nohist:
- return self._get_value_no_hist(version, pkgarch, checksum)
- else:
- return self._get_value_hist(version, pkgarch, checksum)
-
- def _import_hist(self, version, pkgarch, checksum, value):
- if self.read_only:
- return None
-
- val = None
- data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row = data.fetchone()
- if row is not None:
- val=row[0]
- else:
- #no value found, try to insert
- try:
- self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table),
+ def store_value(self, version, pkgarch, checksum, value):
+ """Store value in the database"""
+
+ if not self.read_only and not self.test_checksum_value(version, pkgarch, checksum, value):
+ with closing(self.conn.cursor()) as cursor:
+ cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table),
(version, pkgarch, checksum, value))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
+ self.conn.commit()
- self.dirty = True
+ def _get_value(self, version, pkgarch, checksum, history):
- data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=?;" % self.table,
- (version, pkgarch, checksum))
- row = data.fetchone()
- if row is not None:
- val = row[0]
- return val
+ max_value = self.find_package_max_value(version, pkgarch)
- def _import_no_hist(self, version, pkgarch, checksum, value):
- if self.read_only:
- return None
+ if max_value is None:
+ # version, pkgarch completely unknown. Return initial value.
+ return "0"
- try:
- #try to insert
- self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);" % (self.table),
- (version, pkgarch, checksum, value))
- except sqlite3.IntegrityError as exc:
- #already have the record, try to update
- try:
- self._execute("UPDATE %s SET value=? WHERE version=? AND pkgarch=? AND checksum=? AND value<?"
- % (self.table),
- (value, version, pkgarch, checksum, value))
- except sqlite3.IntegrityError as exc:
- logger.error(str(exc))
-
- self.dirty = True
-
- data = self._execute("SELECT value FROM %s WHERE version=? AND pkgarch=? AND checksum=? AND value>=?;" % self.table,
- (version, pkgarch, checksum, value))
- row=data.fetchone()
- if row is not None:
- return row[0]
+ value = self.find_value(version, pkgarch, checksum, history)
+
+ if value is None:
+ # version, pkgarch found but not checksum. Create a new value from the maximum one
+ return increase_revision(max_value)
+
+ if history:
+ return value
+
+ # "no history" mode - If the value is not the maximum value for the package, need to increase it.
+ if max_value > value:
+ return increase_revision(max_value)
else:
- return None
+ return value
+
+ def get_value(self, version, pkgarch, checksum, history):
+ value = self._get_value(version, pkgarch, checksum, history)
+ if not self.read_only:
+ self.store_value(version, pkgarch, checksum, value)
+ return value
def importone(self, version, pkgarch, checksum, value):
- if self.nohist:
- return self._import_no_hist(version, pkgarch, checksum, value)
- else:
- return self._import_hist(version, pkgarch, checksum, value)
+ self.store_value(version, pkgarch, checksum, value)
+ return value
- def export(self, version, pkgarch, checksum, colinfo):
+ def export(self, version, pkgarch, checksum, colinfo, history=False):
metainfo = {}
- #column info
- if colinfo:
- metainfo["tbl_name"] = self.table
- metainfo["core_ver"] = prserv.__version__
- metainfo["col_info"] = []
- data = self._execute("PRAGMA table_info(%s);" % self.table)
+ with closing(self.conn.cursor()) as cursor:
+ #column info
+ if colinfo:
+ metainfo["tbl_name"] = self.table
+ metainfo["core_ver"] = prserv.__version__
+ metainfo["col_info"] = []
+ data = cursor.execute("PRAGMA table_info(%s);" % self.table)
+ for row in data:
+ col = {}
+ col["name"] = row["name"]
+ col["type"] = row["type"]
+ col["notnull"] = row["notnull"]
+ col["dflt_value"] = row["dflt_value"]
+ col["pk"] = row["pk"]
+ metainfo["col_info"].append(col)
+
+ #data info
+ datainfo = []
+
+ if history:
+ sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table
+ else:
+ sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \
+ (SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \
+ WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table)
+ sqlarg = []
+ where = ""
+ if version:
+ where += "AND T1.version=? "
+ sqlarg.append(str(version))
+ if pkgarch:
+ where += "AND T1.pkgarch=? "
+ sqlarg.append(str(pkgarch))
+ if checksum:
+ where += "AND T1.checksum=? "
+ sqlarg.append(str(checksum))
+
+ sqlstmt += where + ";"
+
+ if len(sqlarg):
+ data = cursor.execute(sqlstmt, tuple(sqlarg))
+ else:
+ data = cursor.execute(sqlstmt)
for row in data:
- col = {}
- col["name"] = row["name"]
- col["type"] = row["type"]
- col["notnull"] = row["notnull"]
- col["dflt_value"] = row["dflt_value"]
- col["pk"] = row["pk"]
- metainfo["col_info"].append(col)
-
- #data info
- datainfo = []
-
- if self.nohist:
- sqlstmt = "SELECT T1.version, T1.pkgarch, T1.checksum, T1.value FROM %s as T1, \
- (SELECT version, pkgarch, max(value) as maxvalue FROM %s GROUP BY version, pkgarch) as T2 \
- WHERE T1.version=T2.version AND T1.pkgarch=T2.pkgarch AND T1.value=T2.maxvalue " % (self.table, self.table)
- else:
- sqlstmt = "SELECT * FROM %s as T1 WHERE 1=1 " % self.table
- sqlarg = []
- where = ""
- if version:
- where += "AND T1.version=? "
- sqlarg.append(str(version))
- if pkgarch:
- where += "AND T1.pkgarch=? "
- sqlarg.append(str(pkgarch))
- if checksum:
- where += "AND T1.checksum=? "
- sqlarg.append(str(checksum))
-
- sqlstmt += where + ";"
-
- if len(sqlarg):
- data = self._execute(sqlstmt, tuple(sqlarg))
- else:
- data = self._execute(sqlstmt)
- for row in data:
- if row["version"]:
- col = {}
- col["version"] = row["version"]
- col["pkgarch"] = row["pkgarch"]
- col["checksum"] = row["checksum"]
- col["value"] = row["value"]
- datainfo.append(col)
+ if row["version"]:
+ col = {}
+ col["version"] = row["version"]
+ col["pkgarch"] = row["pkgarch"]
+ col["checksum"] = row["checksum"]
+ col["value"] = row["value"]
+ datainfo.append(col)
return (metainfo, datainfo)
def dump_db(self, fd):
@@ -322,9 +261,8 @@ class PRTable(object):
class PRData(object):
"""Object representing the PR database"""
- def __init__(self, filename, nohist=True, read_only=False):
+ def __init__(self, filename, read_only=False):
self.filename=os.path.abspath(filename)
- self.nohist=nohist
self.read_only = read_only
#build directory hierarchy
try:
@@ -334,14 +272,15 @@ class PRData(object):
raise e
uri = "file:%s%s" % (self.filename, "?mode=ro" if self.read_only else "")
logger.debug("Opening PRServ database '%s'" % (uri))
- self.connection=sqlite3.connect(uri, uri=True, isolation_level="EXCLUSIVE", check_same_thread = False)
+ self.connection=sqlite3.connect(uri, uri=True)
self.connection.row_factory=sqlite3.Row
- if not self.read_only:
- self.connection.execute("pragma synchronous = off;")
- self.connection.execute("PRAGMA journal_mode = MEMORY;")
+ self.connection.execute("PRAGMA synchronous = OFF;")
+ self.connection.execute("PRAGMA journal_mode = WAL;")
+ self.connection.commit()
self._tables={}
def disconnect(self):
+ self.connection.commit()
self.connection.close()
def __getitem__(self, tblname):
@@ -351,7 +290,7 @@ class PRData(object):
if tblname in self._tables:
return self._tables[tblname]
else:
- tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.nohist, self.read_only)
+ tableobj = self._tables[tblname] = PRTable(self.connection, tblname, self.read_only)
return tableobj
def __delitem__(self, tblname):
@@ -359,3 +298,4 @@ class PRData(object):
del self._tables[tblname]
logger.info("drop table %s" % (tblname))
self.connection.execute("DROP TABLE IF EXISTS %s;" % tblname)
+ self.connection.commit()
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index dc4be5b6..e1758863 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -12,6 +12,7 @@ import sqlite3
import prserv
import prserv.db
import errno
+from . import create_async_client, revision_smaller, increase_revision
import bb.asyncrpc
logger = logging.getLogger("BitBake.PRserv")
@@ -41,18 +42,16 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
try:
return await super().dispatch_message(msg)
except:
- self.server.table.sync()
raise
- else:
- self.server.table.sync_if_dirty()
async def handle_test_pr(self, request):
'''Finds the PR value corresponding to the request. If not found, returns None and doesn't insert a new value'''
version = request["version"]
pkgarch = request["pkgarch"]
checksum = request["checksum"]
+ history = request["history"]
- value = self.server.table.find_value(version, pkgarch, checksum)
+ value = self.server.table.find_value(version, pkgarch, checksum, history)
return {"value": value}
async def handle_test_package(self, request):
@@ -68,22 +67,110 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
version = request["version"]
pkgarch = request["pkgarch"]
- value = self.server.table.find_max_value(version, pkgarch)
+ value = self.server.table.find_package_max_value(version, pkgarch)
return {"value": value}
async def handle_get_pr(self, request):
version = request["version"]
pkgarch = request["pkgarch"]
checksum = request["checksum"]
+ history = request["history"]
- response = None
- try:
- value = self.server.table.get_value(version, pkgarch, checksum)
- response = {"value": value}
- except prserv.NotFoundError:
- self.logger.error("failure storing value in database for (%s, %s)",version, checksum)
+ if self.upstream_client is None:
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ return {"value": value}
- return response
+ # We have an upstream server.
+ # Check whether the local server already knows the requested configuration.
+ # If the configuration is a new one, the generated value we will add will
+ # depend on what's on the upstream server. That's why we're calling find_value()
+ # instead of get_value() directly.
+
+ value = self.server.table.find_value(version, pkgarch, checksum, history)
+ upstream_max = await self.upstream_client.max_package_pr(version, pkgarch)
+
+ if value is not None:
+
+ # The configuration is already known locally.
+
+ if history:
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ else:
+ existing_value = value
+ # In "no history", we need to make sure the value doesn't decrease
+ # and is at least greater than the maximum upstream value
+ # and the maximum local value
+
+ local_max = self.server.table.find_package_max_value(version, pkgarch)
+ if revision_smaller(value, local_max):
+ value = increase_revision(local_max)
+
+ if revision_smaller(value, upstream_max):
+ # Ask upstream whether it knows the checksum
+ upstream_value = await self.upstream_client.test_pr(version, pkgarch, checksum)
+ if upstream_value is None:
+ # Upstream doesn't have our checksum, let create a new one
+ value = upstream_max + ".0"
+ else:
+ # Fine to take the same value as upstream
+ value = upstream_max
+
+ if not value == existing_value and not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, value)
+
+ return {"value": value}
+
+ # The configuration is a new one for the local server
+ # Let's ask the upstream server whether it knows it
+
+ known_upstream = await self.upstream_client.test_package(version, pkgarch)
+
+ if not known_upstream:
+
+ # The package is not known upstream, must be a local-only package
+ # Let's compute the PR number using the local-only method
+
+ value = self.server.table.get_value(version, pkgarch, checksum, history)
+ return {"value": value}
+
+ # The package is known upstream, let's ask the upstream server
+ # whether it knows our new output hash
+
+ value = await self.upstream_client.test_pr(version, pkgarch, checksum)
+
+ if value is not None:
+
+ # Upstream knows this output hash, let's store it and use it too.
+
+ if not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, value)
+ # If the local server is read only, won't be able to store the new
+ # value in the database and will have to keep asking the upstream server
+ return {"value": value}
+
+ # The output hash doesn't exist upstream, get the most recent number from upstream (x)
+ # Then, we want to have a new PR value for the local server: x.y
+
+ upstream_max = await self.upstream_client.max_package_pr(version, pkgarch)
+ # Here we know that the package is known upstream, so upstream_max can't be None
+ subvalue = self.server.table.find_new_subvalue(version, pkgarch, upstream_max)
+
+ if not self.server.read_only:
+ self.server.table.store_value(version, pkgarch, checksum, subvalue)
+
+ return {"value": subvalue}
+
+ async def process_requests(self):
+ if self.server.upstream is not None:
+ self.upstream_client = await create_async_client(self.server.upstream)
+ else:
+ self.upstream_client = None
+
+ try:
+ await super().process_requests()
+ finally:
+ if self.upstream_client is not None:
+ await self.upstream_client.close()
async def handle_import_one(self, request):
response = None
@@ -104,9 +191,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
pkgarch = request["pkgarch"]
checksum = request["checksum"]
colinfo = request["colinfo"]
+ history = request["history"]
try:
- (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo)
+ (metainfo, datainfo) = self.server.table.export(version, pkgarch, checksum, colinfo, history)
except sqlite3.Error as exc:
self.logger.error(str(exc))
metainfo = datainfo = None
@@ -117,11 +205,12 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection):
return {"readonly": self.server.read_only}
class PRServer(bb.asyncrpc.AsyncServer):
- def __init__(self, dbfile, read_only=False):
+ def __init__(self, dbfile, read_only=False, upstream=None):
super().__init__(logger)
self.dbfile = dbfile
self.table = None
self.read_only = read_only
+ self.upstream = upstream
def accept_client(self, socket):
return PRServerClient(socket, self)
@@ -134,27 +223,25 @@ class PRServer(bb.asyncrpc.AsyncServer):
self.logger.info("Started PRServer with DBfile: %s, Address: %s, PID: %s" %
(self.dbfile, self.address, str(os.getpid())))
+ if self.upstream is not None:
+ self.logger.info("And upstream PRServer: %s " % (self.upstream))
+
return tasks
async def stop(self):
- self.table.sync_if_dirty()
self.db.disconnect()
await super().stop()
- def signal_handler(self):
- super().signal_handler()
- if self.table:
- self.table.sync()
-
class PRServSingleton(object):
- def __init__(self, dbfile, logfile, host, port):
+ def __init__(self, dbfile, logfile, host, port, upstream):
self.dbfile = dbfile
self.logfile = logfile
self.host = host
self.port = port
+ self.upstream = upstream
def start(self):
- self.prserv = PRServer(self.dbfile)
+ self.prserv = PRServer(self.dbfile, upstream=self.upstream)
self.prserv.start_tcp_server(socket.gethostbyname(self.host), self.port)
self.process = self.prserv.serve_as_process(log_level=logging.WARNING)
@@ -233,7 +320,7 @@ def run_as_daemon(func, pidfile, logfile):
os.remove(pidfile)
os._exit(0)
-def start_daemon(dbfile, host, port, logfile, read_only=False):
+def start_daemon(dbfile, host, port, logfile, read_only=False, upstream=None):
ip = socket.gethostbyname(host)
pidfile = PIDPREFIX % (ip, port)
try:
@@ -249,7 +336,7 @@ def start_daemon(dbfile, host, port, logfile, read_only=False):
dbfile = os.path.abspath(dbfile)
def daemon_main():
- server = PRServer(dbfile, read_only=read_only)
+ server = PRServer(dbfile, read_only=read_only, upstream=upstream)
server.start_tcp_server(ip, port)
server.serve_forever()
@@ -336,6 +423,9 @@ def auto_start(d):
host = host_params[0].strip().lower()
port = int(host_params[1])
+
+ upstream = d.getVar("PRSERV_UPSTREAM") or None
+
if is_local_special(host, port):
import bb.utils
cachedir = (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE"))
@@ -350,7 +440,7 @@ def auto_start(d):
auto_shutdown()
if not singleton:
bb.utils.mkdirhier(cachedir)
- singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port)
+ singleton = PRServSingleton(os.path.abspath(dbfile), os.path.abspath(logfile), host, port, upstream)
singleton.start()
if singleton:
host = singleton.host
diff --git a/bitbake/lib/prserv/tests.py b/bitbake/lib/prserv/tests.py
new file mode 100644
index 00000000..df0c0030
--- /dev/null
+++ b/bitbake/lib/prserv/tests.py
@@ -0,0 +1,388 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2024 BitBake Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from . import create_server, create_client, increase_revision, revision_greater, revision_smaller, _revision_greater_or_equal
+import prserv.db as db
+from bb.asyncrpc import InvokeError
+import logging
+import os
+import sys
+import tempfile
+import unittest
+import socket
+import subprocess
+from pathlib import Path
+
+THIS_DIR = Path(__file__).parent
+BIN_DIR = THIS_DIR.parent.parent / "bin"
+
+version = "dummy-1.0-r0"
+pkgarch = "core2-64"
+other_arch = "aarch64"
+
+checksumX = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4f0"
+checksum0 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a0"
+checksum1 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a1"
+checksum2 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a2"
+checksum3 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a3"
+checksum4 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a4"
+checksum5 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a5"
+checksum6 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a6"
+checksum7 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a7"
+checksum8 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a8"
+checksum9 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4a9"
+checksum10 = "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4ab4aa"
+
+def server_prefunc(server, name):
+ logging.basicConfig(level=logging.DEBUG, filename='prserv-%s.log' % name, filemode='w',
+ format='%(levelname)s %(filename)s:%(lineno)d %(message)s')
+ server.logger.debug("Running server %s" % name)
+ sys.stdout = open('prserv-stdout-%s.log' % name, 'w')
+ sys.stderr = sys.stdout
+
+class PRTestSetup(object):
+
+ def start_server(self, name, dbfile, upstream=None, read_only=False, prefunc=server_prefunc):
+
+ def cleanup_server(server):
+ if server.process.exitcode is not None:
+ return
+ server.process.terminate()
+ server.process.join()
+
+ server = create_server(socket.gethostbyname("localhost") + ":0",
+ dbfile,
+ upstream=upstream,
+ read_only=read_only)
+
+ server.serve_as_process(prefunc=prefunc, args=(name,))
+ self.addCleanup(cleanup_server, server)
+
+ return server
+
+ def start_client(self, server_address):
+ def cleanup_client(client):
+ client.close()
+
+ client = create_client(server_address)
+ self.addCleanup(cleanup_client, client)
+
+ return client
+
+class FunctionTests(unittest.TestCase):
+
+ def setUp(self):
+ self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv')
+ self.addCleanup(self.temp_dir.cleanup)
+
+ def test_increase_revision(self):
+ self.assertEqual(increase_revision("1"), "2")
+ self.assertEqual(increase_revision("1.0"), "1.1")
+ self.assertEqual(increase_revision("1.1.1"), "1.1.2")
+ self.assertEqual(increase_revision("1.1.1.3"), "1.1.1.4")
+ self.assertEqual(increase_revision("9"), "10")
+ self.assertEqual(increase_revision("1.9"), "1.10")
+ self.assertRaises(ValueError, increase_revision, "1.a")
+ self.assertRaises(ValueError, increase_revision, "1.")
+ self.assertRaises(ValueError, increase_revision, "")
+
+ def test_revision_greater_or_equal(self):
+ self.assertTrue(_revision_greater_or_equal("2", "2"))
+ self.assertTrue(_revision_greater_or_equal("2", "1"))
+ self.assertTrue(_revision_greater_or_equal("10", "2"))
+ self.assertTrue(_revision_greater_or_equal("1.10", "1.2"))
+ self.assertFalse(_revision_greater_or_equal("1.2", "1.10"))
+ self.assertTrue(_revision_greater_or_equal("1.10", "1"))
+ self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10"))
+ self.assertFalse(_revision_greater_or_equal("1.10.1", "1.10.2"))
+ self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10.1"))
+ self.assertTrue(_revision_greater_or_equal("1.10.1", "1"))
+ self.assertTrue(revision_greater("1.20", "1.3"))
+ self.assertTrue(revision_smaller("1.3", "1.20"))
+
+ # DB tests
+
+ def test_db(self):
+ dbfile = os.path.join(self.temp_dir.name, "testtable.sqlite3")
+
+ self.db = db.PRData(dbfile)
+ self.table = self.db["PRMAIN"]
+
+ self.table.store_value(version, pkgarch, checksum0, "0")
+ self.table.store_value(version, pkgarch, checksum1, "1")
+ # "No history" mode supports multiple PRs for the same checksum
+ self.table.store_value(version, pkgarch, checksum0, "2")
+ self.table.store_value(version, pkgarch, checksum2, "1.0")
+
+ self.assertTrue(self.table.test_package(version, pkgarch))
+ self.assertFalse(self.table.test_package(version, other_arch))
+
+ self.assertTrue(self.table.test_value(version, pkgarch, "0"))
+ self.assertTrue(self.table.test_value(version, pkgarch, "1"))
+ self.assertTrue(self.table.test_value(version, pkgarch, "2"))
+
+ self.assertEqual(self.table.find_package_max_value(version, pkgarch), "2")
+
+ self.assertEqual(self.table.find_min_value(version, pkgarch, checksum0), "0")
+ self.assertEqual(self.table.find_max_value(version, pkgarch, checksum0), "2")
+
+ # Test history modes
+ self.assertEqual(self.table.find_value(version, pkgarch, checksum0, True), "0")
+ self.assertEqual(self.table.find_value(version, pkgarch, checksum0, False), "2")
+
+ self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "3"), "3.0")
+ self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "1"), "1.1")
+
+ # Revision comparison tests
+ self.table.store_value(version, pkgarch, checksum1, "1.3")
+ self.table.store_value(version, pkgarch, checksum1, "1.20")
+ self.assertEqual(self.table.find_min_value(version, pkgarch, checksum1), "1")
+ self.assertEqual(self.table.find_max_value(version, pkgarch, checksum1), "1.20")
+
+class PRBasicTests(PRTestSetup, unittest.TestCase):
+
+ def setUp(self):
+ self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv')
+ self.addCleanup(self.temp_dir.cleanup)
+
+ dbfile = os.path.join(self.temp_dir.name, "prtest-basic.sqlite3")
+
+ self.server1 = self.start_server("basic", dbfile)
+ self.client1 = self.start_client(self.server1.address)
+
+ def test_basic(self):
+
+ # Checks on non existing configuration
+
+ result = self.client1.test_pr(version, pkgarch, checksum0)
+ self.assertIsNone(result, "test_pr should return 'None' for a non existing PR")
+
+ result = self.client1.test_package(version, pkgarch)
+ self.assertFalse(result, "test_package should return 'False' for a non existing PR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR")
+
+ # Add a first configuration
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR")
+
+ result = self.client1.test_package(version, pkgarch)
+ self.assertTrue(result, "test_package should return 'True' for an existing PR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series")
+
+ # Check that the same request gets the same value
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: asking for the same PR a second time in a row should return the same value.")
+
+ # Add new configurations
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "1", "getPR: second PR of a package should be '1'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum1)
+ self.assertEqual(result, "1", "test_pr should return '1' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "1", "max_package_pr should return '1' in the current test series")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "2", "getPR: second PR of a package should be '2'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum2)
+ self.assertEqual(result, "2", "test_pr should return '2' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "2", "max_package_pr should return '2' in the current test series")
+
+ result = self.client1.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "3", "getPR: second PR of a package should be '3'")
+
+ result = self.client1.test_pr(version, pkgarch, checksum3)
+ self.assertEqual(result, "3", "test_pr should return '3' here, matching the result of getPR")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "3", "max_package_pr should return '3' in the current test series")
+
+ # Ask again for the first configuration
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "4", "getPR: should return '4' in this configuration")
+
+ # Ask again with explicit "no history" mode
+
+ result = self.client1.getPR(version, pkgarch, checksum0, False)
+ self.assertEqual(result, "4", "getPR: should return '4' in this configuration")
+
+ # Ask again with explicit "history" mode. This should return the first recorded PR for checksum0
+
+ result = self.client1.getPR(version, pkgarch, checksum0, True)
+ self.assertEqual(result, "0", "getPR: should return '0' in this configuration")
+
+ # Check again that another pkgarg resets the counters
+
+ result = self.client1.test_pr(version, other_arch, checksum0)
+ self.assertIsNone(result, "test_pr should return 'None' for a non existing PR")
+
+ result = self.client1.test_package(version, other_arch)
+ self.assertFalse(result, "test_package should return 'False' for a non existing PR")
+
+ result = self.client1.max_package_pr(version, other_arch)
+ self.assertIsNone(result, "max_package_pr should return 'None' for a non existing PR")
+
+ # Now add the configuration
+
+ result = self.client1.getPR(version, other_arch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.test_pr(version, other_arch, checksum0)
+ self.assertEqual(result, "0", "test_pr should return '0' here, matching the result of getPR")
+
+ result = self.client1.test_package(version, other_arch)
+ self.assertTrue(result, "test_package should return 'True' for an existing PR")
+
+ result = self.client1.max_package_pr(version, other_arch)
+ self.assertEqual(result, "0", "max_package_pr should return '0' in the current test series")
+
+ result = self.client1.is_readonly()
+ self.assertFalse(result, "Server should not be described as 'read-only'")
+
+class PRUpstreamTests(PRTestSetup, unittest.TestCase):
+
+ def setUp(self):
+
+ self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv')
+ self.addCleanup(self.temp_dir.cleanup)
+
+ dbfile2 = os.path.join(self.temp_dir.name, "prtest-upstream2.sqlite3")
+ self.server2 = self.start_server("upstream2", dbfile2)
+ self.client2 = self.start_client(self.server2.address)
+
+ dbfile1 = os.path.join(self.temp_dir.name, "prtest-upstream1.sqlite3")
+ self.server1 = self.start_server("upstream1", dbfile1, upstream=self.server2.address)
+ self.client1 = self.start_client(self.server1.address)
+
+ dbfile0 = os.path.join(self.temp_dir.name, "prtest-local.sqlite3")
+ self.server0 = self.start_server("local", dbfile0, upstream=self.server1.address)
+ self.client0 = self.start_client(self.server0.address)
+ self.shared_db = dbfile0
+
+ def test_upstream_and_readonly(self):
+
+ # For identical checksums, all servers should return the same PR
+
+ result = self.client2.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0'")
+
+ result = self.client1.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)")
+
+ result = self.client0.getPR(version, pkgarch, checksum0)
+ self.assertEqual(result, "0", "getPR: initial PR of a package should be '0' (same as upstream)")
+
+ # Now introduce new checksums on server1 for, same version
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "0.0", "getPR: first PR of a package which has a different checksum upstream should be '0.0'")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "0.1", "getPR: second PR of a package that has a different checksum upstream should be '0.1'")
+
+ # Now introduce checksums on server0 for, same version
+
+ result = self.client1.getPR(version, pkgarch, checksum1)
+ self.assertEqual(result, "0.2", "getPR: can't decrease for known PR")
+
+ result = self.client1.getPR(version, pkgarch, checksum2)
+ self.assertEqual(result, "0.3")
+
+ result = self.client1.max_package_pr(version, pkgarch)
+ self.assertEqual(result, "0.3")
+
+ result = self.client0.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "0.3.0", "getPR: first PR of a package that doesn't exist upstream should be '0.3.0'")
+
+ result = self.client0.getPR(version, pkgarch, checksum4)
+ self.assertEqual(result, "0.3.1", "getPR: second PR of a package that doesn't exist upstream should be '0.3.1'")
+
+ result = self.client0.getPR(version, pkgarch, checksum3)
+ self.assertEqual(result, "0.3.2")
+
+ # More upstream updates
+ # Here, we assume no communication between server2 and server0. server2 only impacts server0
+ # after impacting server1
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum5), "1")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum6), "1.0")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "1.1")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum8), "1.1.0")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum9), "1.1.1")
+
+ # "history" mode tests
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum0, True), "0")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum2, True), "0.1")
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum3, True), "0.3.0")
+
+ # More "no history" mode tests
+
+ self.assertEqual(self.client2.getPR(version, pkgarch, checksum0), "2")
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum0), "2") # Same as upstream
+ self.assertEqual(self.client0.getPR(version, pkgarch, checksum0), "2") # Same as upstream
+ self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), "3") # This could be surprising, but since the previous revision was "2", increasing it yields "3".
+ # We don't know how many upstream servers we have
+ # Start read-only server with server1 as upstream
+ self.server_ro = self.start_server("local-ro", self.shared_db, upstream=self.server1.address, read_only=True)
+ self.client_ro = self.start_client(self.server_ro.address)
+
+ self.assertTrue(self.client_ro.is_readonly(), "Database should be described as 'read-only'")
+
+ # Checks on non existing configurations
+ self.assertIsNone(self.client_ro.test_pr(version, pkgarch, checksumX))
+ self.assertFalse(self.client_ro.test_package("unknown", pkgarch))
+
+ # Look up existing configurations
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0), "3") # "no history" mode
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0, True), "0") # "history" mode
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3), "3")
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3, True), "0.3.0")
+ self.assertEqual(self.client_ro.max_package_pr(version, pkgarch), "2") # normal as "3" was never saved
+
+ # Try to insert a new value. Here this one is know upstream.
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum7), "3")
+ # Try to insert a completely new value. As the max upstream value is already "3", it should be "3.0"
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum10), "3.0")
+ # Same with another value which only exists in the upstream upstream server
+ # This time, as the upstream server doesn't know it, it will ask its upstream server. So that's a known one.
+ self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum9), "3")
+
+class ScriptTests(unittest.TestCase):
+
+ def setUp(self):
+
+ self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-prserv')
+ self.addCleanup(self.temp_dir.cleanup)
+ self.dbfile = os.path.join(self.temp_dir.name, "prtest.sqlite3")
+
+ def test_1_start_bitbake_prserv(self):
+ try:
+ subprocess.check_call([BIN_DIR / "bitbake-prserv", "--start", "-f", self.dbfile])
+ except subprocess.CalledProcessError as e:
+ self.fail("Failed to start bitbake-prserv: %s" % e.returncode)
+
+ def test_2_stop_bitbake_prserv(self):
+ try:
+ subprocess.check_call([BIN_DIR / "bitbake-prserv", "--stop"])
+ except subprocess.CalledProcessError as e:
+ self.fail("Failed to stop bitbake-prserv: %s" % e.returncode)
diff --git a/bitbake/lib/pyinotify.py b/bitbake/lib/pyinotify.py
deleted file mode 100644
index 3c5dab03..00000000
--- a/bitbake/lib/pyinotify.py
+++ /dev/null
@@ -1,2326 +0,0 @@
-#
-# pyinotify.py - python interface to inotify
-# Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
-#
-# SPDX-License-Identifier: MIT
-#
-"""
-pyinotify
-
-@author: Sebastien Martini
-@license: MIT License
-@contact: seb@dbzteam.org
-"""
-
-class PyinotifyError(Exception):
- """Indicates exceptions raised by a Pyinotify class."""
- pass
-
-
-class UnsupportedPythonVersionError(PyinotifyError):
- """
- Raised on unsupported Python versions.
- """
- def __init__(self, version):
- """
- @param version: Current Python version
- @type version: string
- """
- PyinotifyError.__init__(self,
- ('Python %s is unsupported, requires '
- 'at least Python 3.0') % version)
-
-
-# Check Python version
-import sys
-if sys.version_info < (3, 0):
- raise UnsupportedPythonVersionError(sys.version)
-
-
-# Import directives
-import threading
-import os
-import select
-import struct
-import fcntl
-import errno
-import termios
-import array
-import logging
-import atexit
-from collections import deque
-from datetime import datetime, timedelta
-import time
-import re
-import glob
-import locale
-import subprocess
-
-try:
- from functools import reduce
-except ImportError:
- pass # Will fail on Python 2.4 which has reduce() builtin anyway.
-
-try:
- import ctypes
- import ctypes.util
-except ImportError:
- ctypes = None
-
-try:
- import inotify_syscalls
-except ImportError:
- inotify_syscalls = None
-
-
-__author__ = "seb@dbzteam.org (Sebastien Martini)"
-
-__version__ = "0.9.6"
-
-
-# Compatibity mode: set to True to improve compatibility with
-# Pyinotify 0.7.1. Do not set this variable yourself, call the
-# function compatibility_mode() instead.
-COMPATIBILITY_MODE = False
-
-
-class InotifyBindingNotFoundError(PyinotifyError):
- """
- Raised when no inotify support couldn't be found.
- """
- def __init__(self):
- err = "Couldn't find any inotify binding"
- PyinotifyError.__init__(self, err)
-
-
-class INotifyWrapper:
- """
- Abstract class wrapping access to inotify's functions. This is an
- internal class.
- """
- @staticmethod
- def create():
- """
- Factory method instanciating and returning the right wrapper.
- """
- # First, try to use ctypes.
- if ctypes:
- inotify = _CtypesLibcINotifyWrapper()
- if inotify.init():
- return inotify
- # Second, see if C extension is compiled.
- if inotify_syscalls:
- inotify = _INotifySyscallsWrapper()
- if inotify.init():
- return inotify
-
- def get_errno(self):
- """
- Return None is no errno code is available.
- """
- return self._get_errno()
-
- def str_errno(self):
- code = self.get_errno()
- if code is None:
- return 'Errno: no errno support'
- return 'Errno=%s (%s)' % (os.strerror(code), errno.errorcode[code])
-
- def inotify_init(self):
- return self._inotify_init()
-
- def inotify_add_watch(self, fd, pathname, mask):
- # Unicode strings must be encoded to string prior to calling this
- # method.
- assert isinstance(pathname, str)
- return self._inotify_add_watch(fd, pathname, mask)
-
- def inotify_rm_watch(self, fd, wd):
- return self._inotify_rm_watch(fd, wd)
-
-
-class _INotifySyscallsWrapper(INotifyWrapper):
- def __init__(self):
- # Stores the last errno value.
- self._last_errno = None
-
- def init(self):
- assert inotify_syscalls
- return True
-
- def _get_errno(self):
- return self._last_errno
-
- def _inotify_init(self):
- try:
- fd = inotify_syscalls.inotify_init()
- except IOError as err:
- self._last_errno = err.errno
- return -1
- return fd
-
- def _inotify_add_watch(self, fd, pathname, mask):
- try:
- wd = inotify_syscalls.inotify_add_watch(fd, pathname, mask)
- except IOError as err:
- self._last_errno = err.errno
- return -1
- return wd
-
- def _inotify_rm_watch(self, fd, wd):
- try:
- ret = inotify_syscalls.inotify_rm_watch(fd, wd)
- except IOError as err:
- self._last_errno = err.errno
- return -1
- return ret
-
-
-class _CtypesLibcINotifyWrapper(INotifyWrapper):
- def __init__(self):
- self._libc = None
- self._get_errno_func = None
-
- def init(self):
- assert ctypes
-
- try_libc_name = 'c'
- if sys.platform.startswith('freebsd'):
- try_libc_name = 'inotify'
-
- libc_name = None
- try:
- libc_name = ctypes.util.find_library(try_libc_name)
- except (OSError, IOError):
- pass # Will attemp to load it with None anyway.
-
- self._libc = ctypes.CDLL(libc_name, use_errno=True)
- self._get_errno_func = ctypes.get_errno
-
- # Eventually check that libc has needed inotify bindings.
- if (not hasattr(self._libc, 'inotify_init') or
- not hasattr(self._libc, 'inotify_add_watch') or
- not hasattr(self._libc, 'inotify_rm_watch')):
- return False
-
- self._libc.inotify_init.argtypes = []
- self._libc.inotify_init.restype = ctypes.c_int
- self._libc.inotify_add_watch.argtypes = [ctypes.c_int, ctypes.c_char_p,
- ctypes.c_uint32]
- self._libc.inotify_add_watch.restype = ctypes.c_int
- self._libc.inotify_rm_watch.argtypes = [ctypes.c_int, ctypes.c_int]
- self._libc.inotify_rm_watch.restype = ctypes.c_int
- return True
-
- def _get_errno(self):
- assert self._get_errno_func
- return self._get_errno_func()
-
- def _inotify_init(self):
- assert self._libc is not None
- return self._libc.inotify_init()
-
- def _inotify_add_watch(self, fd, pathname, mask):
- assert self._libc is not None
- # Encodes path to a bytes string. This conversion seems required because
- # ctypes.create_string_buffer seems to manipulate bytes internally.
- # Moreover it seems that inotify_add_watch does not work very well when
- # it receives an ctypes.create_unicode_buffer instance as argument.
- pathname = pathname.encode(sys.getfilesystemencoding())
- pathname = ctypes.create_string_buffer(pathname)
- return self._libc.inotify_add_watch(fd, pathname, mask)
-
- def _inotify_rm_watch(self, fd, wd):
- assert self._libc is not None
- return self._libc.inotify_rm_watch(fd, wd)
-
-
-# Logging
-def logger_init():
- """Initialize logger instance."""
- log = logging.getLogger("pyinotify")
- console_handler = logging.StreamHandler()
- console_handler.setFormatter(
- logging.Formatter("[%(asctime)s %(name)s %(levelname)s] %(message)s"))
- log.addHandler(console_handler)
- log.setLevel(20)
- return log
-
-log = logger_init()
-
-
-# inotify's variables
-class ProcINotify:
- """
- Access (read, write) inotify's variables through /proc/sys/. Note that
- usually it requires administrator rights to update them.
-
- Examples:
- - Read max_queued_events attribute: myvar = max_queued_events.value
- - Update max_queued_events attribute: max_queued_events.value = 42
- """
- def __init__(self, attr):
- self._base = "/proc/sys/fs/inotify"
- self._attr = attr
-
- def get_val(self):
- """
- Gets attribute's value.
-
- @return: stored value.
- @rtype: int
- @raise IOError: if corresponding file in /proc/sys cannot be read.
- """
- with open(os.path.join(self._base, self._attr), 'r') as file_obj:
- return int(file_obj.readline())
-
- def set_val(self, nval):
- """
- Sets new attribute's value.
-
- @param nval: replaces current value by nval.
- @type nval: int
- @raise IOError: if corresponding file in /proc/sys cannot be written.
- """
- with open(os.path.join(self._base, self._attr), 'w') as file_obj:
- file_obj.write(str(nval) + '\n')
-
- value = property(get_val, set_val)
-
- def __repr__(self):
- return '<%s=%d>' % (self._attr, self.get_val())
-
-
-# Inotify's variables
-#
-# Note: may raise IOError if the corresponding value in /proc/sys
-# cannot be accessed.
-#
-# Examples:
-# - read: myvar = max_queued_events.value
-# - update: max_queued_events.value = 42
-#
-for attrname in ('max_queued_events', 'max_user_instances', 'max_user_watches'):
- globals()[attrname] = ProcINotify(attrname)
-
-
-class EventsCodes:
- """
- Set of codes corresponding to each kind of events.
- Some of these flags are used to communicate with inotify, whereas
- the others are sent to userspace by inotify notifying some events.
-
- @cvar IN_ACCESS: File was accessed.
- @type IN_ACCESS: int
- @cvar IN_MODIFY: File was modified.
- @type IN_MODIFY: int
- @cvar IN_ATTRIB: Metadata changed.
- @type IN_ATTRIB: int
- @cvar IN_CLOSE_WRITE: Writtable file was closed.
- @type IN_CLOSE_WRITE: int
- @cvar IN_CLOSE_NOWRITE: Unwrittable file closed.
- @type IN_CLOSE_NOWRITE: int
- @cvar IN_OPEN: File was opened.
- @type IN_OPEN: int
- @cvar IN_MOVED_FROM: File was moved from X.
- @type IN_MOVED_FROM: int
- @cvar IN_MOVED_TO: File was moved to Y.
- @type IN_MOVED_TO: int
- @cvar IN_CREATE: Subfile was created.
- @type IN_CREATE: int
- @cvar IN_DELETE: Subfile was deleted.
- @type IN_DELETE: int
- @cvar IN_DELETE_SELF: Self (watched item itself) was deleted.
- @type IN_DELETE_SELF: int
- @cvar IN_MOVE_SELF: Self (watched item itself) was moved.
- @type IN_MOVE_SELF: int
- @cvar IN_UNMOUNT: Backing fs was unmounted.
- @type IN_UNMOUNT: int
- @cvar IN_Q_OVERFLOW: Event queued overflowed.
- @type IN_Q_OVERFLOW: int
- @cvar IN_IGNORED: File was ignored.
- @type IN_IGNORED: int
- @cvar IN_ONLYDIR: only watch the path if it is a directory (new
- in kernel 2.6.15).
- @type IN_ONLYDIR: int
- @cvar IN_DONT_FOLLOW: don't follow a symlink (new in kernel 2.6.15).
- IN_ONLYDIR we can make sure that we don't watch
- the target of symlinks.
- @type IN_DONT_FOLLOW: int
- @cvar IN_EXCL_UNLINK: Events are not generated for children after they
- have been unlinked from the watched directory.
- (new in kernel 2.6.36).
- @type IN_EXCL_UNLINK: int
- @cvar IN_MASK_ADD: add to the mask of an already existing watch (new
- in kernel 2.6.14).
- @type IN_MASK_ADD: int
- @cvar IN_ISDIR: Event occurred against dir.
- @type IN_ISDIR: int
- @cvar IN_ONESHOT: Only send event once.
- @type IN_ONESHOT: int
- @cvar ALL_EVENTS: Alias for considering all of the events.
- @type ALL_EVENTS: int
- """
-
- # The idea here is 'configuration-as-code' - this way, we get our nice class
- # constants, but we also get nice human-friendly text mappings to do lookups
- # against as well, for free:
- FLAG_COLLECTIONS = {'OP_FLAGS': {
- 'IN_ACCESS' : 0x00000001, # File was accessed
- 'IN_MODIFY' : 0x00000002, # File was modified
- 'IN_ATTRIB' : 0x00000004, # Metadata changed
- 'IN_CLOSE_WRITE' : 0x00000008, # Writable file was closed
- 'IN_CLOSE_NOWRITE' : 0x00000010, # Unwritable file closed
- 'IN_OPEN' : 0x00000020, # File was opened
- 'IN_MOVED_FROM' : 0x00000040, # File was moved from X
- 'IN_MOVED_TO' : 0x00000080, # File was moved to Y
- 'IN_CREATE' : 0x00000100, # Subfile was created
- 'IN_DELETE' : 0x00000200, # Subfile was deleted
- 'IN_DELETE_SELF' : 0x00000400, # Self (watched item itself)
- # was deleted
- 'IN_MOVE_SELF' : 0x00000800, # Self (watched item itself) was moved
- },
- 'EVENT_FLAGS': {
- 'IN_UNMOUNT' : 0x00002000, # Backing fs was unmounted
- 'IN_Q_OVERFLOW' : 0x00004000, # Event queued overflowed
- 'IN_IGNORED' : 0x00008000, # File was ignored
- },
- 'SPECIAL_FLAGS': {
- 'IN_ONLYDIR' : 0x01000000, # only watch the path if it is a
- # directory
- 'IN_DONT_FOLLOW' : 0x02000000, # don't follow a symlink
- 'IN_EXCL_UNLINK' : 0x04000000, # exclude events on unlinked objects
- 'IN_MASK_ADD' : 0x20000000, # add to the mask of an already
- # existing watch
- 'IN_ISDIR' : 0x40000000, # event occurred against dir
- 'IN_ONESHOT' : 0x80000000, # only send event once
- },
- }
-
- def maskname(mask):
- """
- Returns the event name associated to mask. IN_ISDIR is appended to
- the result when appropriate. Note: only one event is returned, because
- only one event can be raised at a given time.
-
- @param mask: mask.
- @type mask: int
- @return: event name.
- @rtype: str
- """
- ms = mask
- name = '%s'
- if mask & IN_ISDIR:
- ms = mask - IN_ISDIR
- name = '%s|IN_ISDIR'
- return name % EventsCodes.ALL_VALUES[ms]
-
- maskname = staticmethod(maskname)
-
-
-# So let's now turn the configuration into code
-EventsCodes.ALL_FLAGS = {}
-EventsCodes.ALL_VALUES = {}
-for flagc, valc in EventsCodes.FLAG_COLLECTIONS.items():
- # Make the collections' members directly accessible through the
- # class dictionary
- setattr(EventsCodes, flagc, valc)
-
- # Collect all the flags under a common umbrella
- EventsCodes.ALL_FLAGS.update(valc)
-
- # Make the individual masks accessible as 'constants' at globals() scope
- # and masknames accessible by values.
- for name, val in valc.items():
- globals()[name] = val
- EventsCodes.ALL_VALUES[val] = name
-
-
-# all 'normal' events
-ALL_EVENTS = reduce(lambda x, y: x | y, EventsCodes.OP_FLAGS.values())
-EventsCodes.ALL_FLAGS['ALL_EVENTS'] = ALL_EVENTS
-EventsCodes.ALL_VALUES[ALL_EVENTS] = 'ALL_EVENTS'
-
-
-class _Event:
- """
- Event structure, represent events raised by the system. This
- is the base class and should be subclassed.
-
- """
- def __init__(self, dict_):
- """
- Attach attributes (contained in dict_) to self.
-
- @param dict_: Set of attributes.
- @type dict_: dictionary
- """
- for tpl in dict_.items():
- setattr(self, *tpl)
-
- def __repr__(self):
- """
- @return: Generic event string representation.
- @rtype: str
- """
- s = ''
- for attr, value in sorted(self.__dict__.items(), key=lambda x: x[0]):
- if attr.startswith('_'):
- continue
- if attr == 'mask':
- value = hex(getattr(self, attr))
- elif isinstance(value, str) and not value:
- value = "''"
- s += ' %s%s%s' % (output_format.field_name(attr),
- output_format.punctuation('='),
- output_format.field_value(value))
-
- s = '%s%s%s %s' % (output_format.punctuation('<'),
- output_format.class_name(self.__class__.__name__),
- s,
- output_format.punctuation('>'))
- return s
-
- def __str__(self):
- return repr(self)
-
-
-class _RawEvent(_Event):
- """
- Raw event, it contains only the informations provided by the system.
- It doesn't infer anything.
- """
- def __init__(self, wd, mask, cookie, name):
- """
- @param wd: Watch Descriptor.
- @type wd: int
- @param mask: Bitmask of events.
- @type mask: int
- @param cookie: Cookie.
- @type cookie: int
- @param name: Basename of the file or directory against which the
- event was raised in case where the watched directory
- is the parent directory. None if the event was raised
- on the watched item itself.
- @type name: string or None
- """
- # Use this variable to cache the result of str(self), this object
- # is immutable.
- self._str = None
- # name: remove trailing '\0'
- d = {'wd': wd,
- 'mask': mask,
- 'cookie': cookie,
- 'name': name.rstrip('\0')}
- _Event.__init__(self, d)
- log.debug(str(self))
-
- def __str__(self):
- if self._str is None:
- self._str = _Event.__str__(self)
- return self._str
-
-
-class Event(_Event):
- """
- This class contains all the useful informations about the observed
- event. However, the presence of each field is not guaranteed and
- depends on the type of event. In effect, some fields are irrelevant
- for some kind of event (for example 'cookie' is meaningless for
- IN_CREATE whereas it is mandatory for IN_MOVE_TO).
-
- The possible fields are:
- - wd (int): Watch Descriptor.
- - mask (int): Mask.
- - maskname (str): Readable event name.
- - path (str): path of the file or directory being watched.
- - name (str): Basename of the file or directory against which the
- event was raised in case where the watched directory
- is the parent directory. None if the event was raised
- on the watched item itself. This field is always provided
- even if the string is ''.
- - pathname (str): Concatenation of 'path' and 'name'.
- - src_pathname (str): Only present for IN_MOVED_TO events and only in
- the case where IN_MOVED_FROM events are watched too. Holds the
- source pathname from where pathname was moved from.
- - cookie (int): Cookie.
- - dir (bool): True if the event was raised against a directory.
-
- """
- def __init__(self, raw):
- """
- Concretely, this is the raw event plus inferred infos.
- """
- _Event.__init__(self, raw)
- self.maskname = EventsCodes.maskname(self.mask)
- if COMPATIBILITY_MODE:
- self.event_name = self.maskname
- try:
- if self.name:
- self.pathname = os.path.abspath(os.path.join(self.path,
- self.name))
- else:
- self.pathname = os.path.abspath(self.path)
- except AttributeError as err:
- # Usually it is not an error some events are perfectly valids
- # despite the lack of these attributes.
- log.debug(err)
-
-
-class ProcessEventError(PyinotifyError):
- """
- ProcessEventError Exception. Raised on ProcessEvent error.
- """
- def __init__(self, err):
- """
- @param err: Exception error description.
- @type err: string
- """
- PyinotifyError.__init__(self, err)
-
-
-class _ProcessEvent:
- """
- Abstract processing event class.
- """
- def __call__(self, event):
- """
- To behave like a functor the object must be callable.
- This method is a dispatch method. Its lookup order is:
- 1. process_MASKNAME method
- 2. process_FAMILY_NAME method
- 3. otherwise calls process_default
-
- @param event: Event to be processed.
- @type event: Event object
- @return: By convention when used from the ProcessEvent class:
- - Returning False or None (default value) means keep on
- executing next chained functors (see chain.py example).
- - Returning True instead means do not execute next
- processing functions.
- @rtype: bool
- @raise ProcessEventError: Event object undispatchable,
- unknown event.
- """
- stripped_mask = event.mask & ~IN_ISDIR
- # Bitbake hack - we see event masks of 0x6, i.e., IN_MODIFY & IN_ATTRIB.
- # The kernel inotify code can set more than one of the bits in the mask,
- # fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATTRIB,
- # IN_MODIFY and IN_ACCESS can arrive together.
- # This breaks the code below which assume only one mask bit is ever
- # set in an event. We don't care about attrib or access in bitbake so
- # drop those.
- if stripped_mask & IN_MODIFY:
- stripped_mask &= ~(IN_ATTRIB | IN_ACCESS)
-
- maskname = EventsCodes.ALL_VALUES.get(stripped_mask)
- if maskname is None:
- raise ProcessEventError("Unknown mask 0x%08x" % stripped_mask)
-
- # 1- look for process_MASKNAME
- meth = getattr(self, 'process_' + maskname, None)
- if meth is not None:
- return meth(event)
- # 2- look for process_FAMILY_NAME
- meth = getattr(self, 'process_IN_' + maskname.split('_')[1], None)
- if meth is not None:
- return meth(event)
- # 3- default call method process_default
- return self.process_default(event)
-
- def __repr__(self):
- return '<%s>' % self.__class__.__name__
-
-
-class _SysProcessEvent(_ProcessEvent):
- """
- There is three kind of processing according to each event:
-
- 1. special handling (deletion from internal container, bug, ...).
- 2. default treatment: which is applied to the majority of events.
- 3. IN_ISDIR is never sent alone, he is piggybacked with a standard
- event, he is not processed as the others events, instead, its
- value is captured and appropriately aggregated to dst event.
- """
- def __init__(self, wm, notifier):
- """
-
- @param wm: Watch Manager.
- @type wm: WatchManager instance
- @param notifier: Notifier.
- @type notifier: Notifier instance
- """
- self._watch_manager = wm # watch manager
- self._notifier = notifier # notifier
- self._mv_cookie = {} # {cookie(int): (src_path(str), date), ...}
- self._mv = {} # {src_path(str): (dst_path(str), date), ...}
-
- def cleanup(self):
- """
- Cleanup (delete) old (>1mn) records contained in self._mv_cookie
- and self._mv.
- """
- date_cur_ = datetime.now()
- for seq in (self._mv_cookie, self._mv):
- for k in list(seq.keys()):
- if (date_cur_ - seq[k][1]) > timedelta(minutes=1):
- log.debug('Cleanup: deleting entry %s', seq[k][0])
- del seq[k]
-
- def process_IN_CREATE(self, raw_event):
- """
- If the event affects a directory and the auto_add flag of the
- targetted watch is set to True, a new watch is added on this
- new directory, with the same attribute values than those of
- this watch.
- """
- if raw_event.mask & IN_ISDIR:
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- created_dir = os.path.join(watch_.path, raw_event.name)
- if watch_.auto_add and not watch_.exclude_filter(created_dir):
- addw = self._watch_manager.add_watch
- # The newly monitored directory inherits attributes from its
- # parent directory.
- addw_ret = addw(created_dir, watch_.mask,
- proc_fun=watch_.proc_fun,
- rec=False, auto_add=watch_.auto_add,
- exclude_filter=watch_.exclude_filter)
-
- # Trick to handle mkdir -p /d1/d2/t3 where d1 is watched and
- # d2 and t3 (directory or file) are created.
- # Since the directory d2 is new, then everything inside it must
- # also be new.
- created_dir_wd = addw_ret.get(created_dir)
- if ((created_dir_wd is not None) and (created_dir_wd > 0) and
- os.path.isdir(created_dir)):
- try:
- for name in os.listdir(created_dir):
- inner = os.path.join(created_dir, name)
- if self._watch_manager.get_wd(inner) is not None:
- continue
- # Generate (simulate) creation events for sub-
- # directories and files.
- if os.path.isfile(inner):
- # symlinks are handled as files.
- flags = IN_CREATE
- elif os.path.isdir(inner):
- flags = IN_CREATE | IN_ISDIR
- else:
- # This path should not be taken.
- continue
- rawevent = _RawEvent(created_dir_wd, flags, 0, name)
- self._notifier.append_event(rawevent)
- except OSError as err:
- msg = "process_IN_CREATE, invalid directory: %s"
- log.debug(msg % str(err))
- return self.process_default(raw_event)
-
- def process_IN_MOVED_FROM(self, raw_event):
- """
- Map the cookie with the source path (+ date for cleaning).
- """
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- path_ = watch_.path
- src_path = os.path.normpath(os.path.join(path_, raw_event.name))
- self._mv_cookie[raw_event.cookie] = (src_path, datetime.now())
- return self.process_default(raw_event, {'cookie': raw_event.cookie})
-
- def process_IN_MOVED_TO(self, raw_event):
- """
- Map the source path with the destination path (+ date for
- cleaning).
- """
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- path_ = watch_.path
- dst_path = os.path.normpath(os.path.join(path_, raw_event.name))
- mv_ = self._mv_cookie.get(raw_event.cookie)
- to_append = {'cookie': raw_event.cookie}
- if mv_ is not None:
- self._mv[mv_[0]] = (dst_path, datetime.now())
- # Let's assume that IN_MOVED_FROM event is always queued before
- # that its associated (they share a common cookie) IN_MOVED_TO
- # event is queued itself. It is then possible in that scenario
- # to provide as additional information to the IN_MOVED_TO event
- # the original pathname of the moved file/directory.
- to_append['src_pathname'] = mv_[0]
- elif (raw_event.mask & IN_ISDIR and watch_.auto_add and
- not watch_.exclude_filter(dst_path)):
- # We got a diretory that's "moved in" from an unknown source and
- # auto_add is enabled. Manually add watches to the inner subtrees.
- # The newly monitored directory inherits attributes from its
- # parent directory.
- self._watch_manager.add_watch(dst_path, watch_.mask,
- proc_fun=watch_.proc_fun,
- rec=True, auto_add=True,
- exclude_filter=watch_.exclude_filter)
- return self.process_default(raw_event, to_append)
-
- def process_IN_MOVE_SELF(self, raw_event):
- """
- STATUS: the following bug has been fixed in recent kernels (FIXME:
- which version ?). Now it raises IN_DELETE_SELF instead.
-
- Old kernels were bugged, this event raised when the watched item
- were moved, so we had to update its path, but under some circumstances
- it was impossible: if its parent directory and its destination
- directory wasn't watched. The kernel (see include/linux/fsnotify.h)
- doesn't bring us enough informations like the destination path of
- moved items.
- """
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- src_path = watch_.path
- mv_ = self._mv.get(src_path)
- if mv_:
- dest_path = mv_[0]
- watch_.path = dest_path
- # add the separator to the source path to avoid overlapping
- # path issue when testing with startswith()
- src_path += os.path.sep
- src_path_len = len(src_path)
- # The next loop renames all watches with src_path as base path.
- # It seems that IN_MOVE_SELF does not provide IN_ISDIR information
- # therefore the next loop is iterated even if raw_event is a file.
- for w in self._watch_manager.watches.values():
- if w.path.startswith(src_path):
- # Note that dest_path is a normalized path.
- w.path = os.path.join(dest_path, w.path[src_path_len:])
- else:
- log.error("The pathname '%s' of this watch %s has probably changed "
- "and couldn't be updated, so it cannot be trusted "
- "anymore. To fix this error move directories/files only "
- "between watched parents directories, in this case e.g. "
- "put a watch on '%s'.",
- watch_.path, watch_,
- os.path.normpath(os.path.join(watch_.path,
- os.path.pardir)))
- if not watch_.path.endswith('-unknown-path'):
- watch_.path += '-unknown-path'
- return self.process_default(raw_event)
-
- def process_IN_Q_OVERFLOW(self, raw_event):
- """
- Only signal an overflow, most of the common flags are irrelevant
- for this event (path, wd, name).
- """
- return Event({'mask': raw_event.mask})
-
- def process_IN_IGNORED(self, raw_event):
- """
- The watch descriptor raised by this event is now ignored (forever),
- it can be safely deleted from the watch manager dictionary.
- After this event we can be sure that neither the event queue nor
- the system will raise an event associated to this wd again.
- """
- event_ = self.process_default(raw_event)
- self._watch_manager.del_watch(raw_event.wd)
- return event_
-
- def process_default(self, raw_event, to_append=None):
- """
- Commons handling for the followings events:
-
- IN_ACCESS, IN_MODIFY, IN_ATTRIB, IN_CLOSE_WRITE, IN_CLOSE_NOWRITE,
- IN_OPEN, IN_DELETE, IN_DELETE_SELF, IN_UNMOUNT.
- """
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- if raw_event.mask & (IN_DELETE_SELF | IN_MOVE_SELF):
- # Unfornulately this information is not provided by the kernel
- dir_ = watch_.dir
- else:
- dir_ = bool(raw_event.mask & IN_ISDIR)
- dict_ = {'wd': raw_event.wd,
- 'mask': raw_event.mask,
- 'path': watch_.path,
- 'name': raw_event.name,
- 'dir': dir_}
- if COMPATIBILITY_MODE:
- dict_['is_dir'] = dir_
- if to_append is not None:
- dict_.update(to_append)
- return Event(dict_)
-
-
-class ProcessEvent(_ProcessEvent):
- """
- Process events objects, can be specialized via subclassing, thus its
- behavior can be overriden:
-
- Note: you should not override __init__ in your subclass instead define
- a my_init() method, this method will be called automatically from the
- constructor of this class with its optionals parameters.
-
- 1. Provide specialized individual methods, e.g. process_IN_DELETE for
- processing a precise type of event (e.g. IN_DELETE in this case).
- 2. Or/and provide methods for processing events by 'family', e.g.
- process_IN_CLOSE method will process both IN_CLOSE_WRITE and
- IN_CLOSE_NOWRITE events (if process_IN_CLOSE_WRITE and
- process_IN_CLOSE_NOWRITE aren't defined though).
- 3. Or/and override process_default for catching and processing all
- the remaining types of events.
- """
- pevent = None
-
- def __init__(self, pevent=None, **kargs):
- """
- Enable chaining of ProcessEvent instances.
-
- @param pevent: Optional callable object, will be called on event
- processing (before self).
- @type pevent: callable
- @param kargs: This constructor is implemented as a template method
- delegating its optionals keyworded arguments to the
- method my_init().
- @type kargs: dict
- """
- self.pevent = pevent
- self.my_init(**kargs)
-
- def my_init(self, **kargs):
- """
- This method is called from ProcessEvent.__init__(). This method is
- empty here and must be redefined to be useful. In effect, if you
- need to specifically initialize your subclass' instance then you
- just have to override this method in your subclass. Then all the
- keyworded arguments passed to ProcessEvent.__init__() will be
- transmitted as parameters to this method. Beware you MUST pass
- keyword arguments though.
-
- @param kargs: optional delegated arguments from __init__().
- @type kargs: dict
- """
- pass
-
- def __call__(self, event):
- stop_chaining = False
- if self.pevent is not None:
- # By default methods return None so we set as guideline
- # that methods asking for stop chaining must explicitely
- # return non None or non False values, otherwise the default
- # behavior will be to accept chain call to the corresponding
- # local method.
- stop_chaining = self.pevent(event)
- if not stop_chaining:
- return _ProcessEvent.__call__(self, event)
-
- def nested_pevent(self):
- return self.pevent
-
- def process_IN_Q_OVERFLOW(self, event):
- """
- By default this method only reports warning messages, you can overredide
- it by subclassing ProcessEvent and implement your own
- process_IN_Q_OVERFLOW method. The actions you can take on receiving this
- event is either to update the variable max_queued_events in order to
- handle more simultaneous events or to modify your code in order to
- accomplish a better filtering diminishing the number of raised events.
- Because this method is defined, IN_Q_OVERFLOW will never get
- transmitted as arguments to process_default calls.
-
- @param event: IN_Q_OVERFLOW event.
- @type event: dict
- """
- log.warning('Event queue overflowed.')
-
- def process_default(self, event):
- """
- Default processing event method. By default does nothing. Subclass
- ProcessEvent and redefine this method in order to modify its behavior.
-
- @param event: Event to be processed. Can be of any type of events but
- IN_Q_OVERFLOW events (see method process_IN_Q_OVERFLOW).
- @type event: Event instance
- """
- pass
-
-
-class PrintAllEvents(ProcessEvent):
- """
- Dummy class used to print events strings representations. For instance this
- class is used from command line to print all received events to stdout.
- """
- def my_init(self, out=None):
- """
- @param out: Where events will be written.
- @type out: Object providing a valid file object interface.
- """
- if out is None:
- out = sys.stdout
- self._out = out
-
- def process_default(self, event):
- """
- Writes event string representation to file object provided to
- my_init().
-
- @param event: Event to be processed. Can be of any type of events but
- IN_Q_OVERFLOW events (see method process_IN_Q_OVERFLOW).
- @type event: Event instance
- """
- self._out.write(str(event))
- self._out.write('\n')
- self._out.flush()
-
-
-class ChainIfTrue(ProcessEvent):
- """
- Makes conditional chaining depending on the result of the nested
- processing instance.
- """
- def my_init(self, func):
- """
- Method automatically called from base class constructor.
- """
- self._func = func
-
- def process_default(self, event):
- return not self._func(event)
-
-
-class Stats(ProcessEvent):
- """
- Compute and display trivial statistics about processed events.
- """
- def my_init(self):
- """
- Method automatically called from base class constructor.
- """
- self._start_time = time.time()
- self._stats = {}
- self._stats_lock = threading.Lock()
-
- def process_default(self, event):
- """
- Processes |event|.
- """
- self._stats_lock.acquire()
- try:
- events = event.maskname.split('|')
- for event_name in events:
- count = self._stats.get(event_name, 0)
- self._stats[event_name] = count + 1
- finally:
- self._stats_lock.release()
-
- def _stats_copy(self):
- self._stats_lock.acquire()
- try:
- return self._stats.copy()
- finally:
- self._stats_lock.release()
-
- def __repr__(self):
- stats = self._stats_copy()
-
- elapsed = int(time.time() - self._start_time)
- elapsed_str = ''
- if elapsed < 60:
- elapsed_str = str(elapsed) + 'sec'
- elif 60 <= elapsed < 3600:
- elapsed_str = '%dmn%dsec' % (elapsed / 60, elapsed % 60)
- elif 3600 <= elapsed < 86400:
- elapsed_str = '%dh%dmn' % (elapsed / 3600, (elapsed % 3600) / 60)
- elif elapsed >= 86400:
- elapsed_str = '%dd%dh' % (elapsed / 86400, (elapsed % 86400) / 3600)
- stats['ElapsedTime'] = elapsed_str
-
- l = []
- for ev, value in sorted(stats.items(), key=lambda x: x[0]):
- l.append(' %s=%s' % (output_format.field_name(ev),
- output_format.field_value(value)))
- s = '<%s%s >' % (output_format.class_name(self.__class__.__name__),
- ''.join(l))
- return s
-
- def dump(self, filename):
- """
- Dumps statistics.
-
- @param filename: filename where stats will be dumped, filename is
- created and must not exist prior to this call.
- @type filename: string
- """
- flags = os.O_WRONLY|os.O_CREAT|os.O_NOFOLLOW|os.O_EXCL
- fd = os.open(filename, flags, 0o0600)
- os.write(fd, bytes(self.__str__(), locale.getpreferredencoding()))
- os.close(fd)
-
- def __str__(self, scale=45):
- stats = self._stats_copy()
- if not stats:
- return ''
-
- m = max(stats.values())
- unity = scale / m
- fmt = '%%-26s%%-%ds%%s' % (len(output_format.field_value('@' * scale))
- + 1)
- def func(x):
- return fmt % (output_format.field_name(x[0]),
- output_format.field_value('@' * int(x[1] * unity)),
- output_format.simple('%d' % x[1], 'yellow'))
- s = '\n'.join(map(func, sorted(stats.items(), key=lambda x: x[0])))
- return s
-
-
-class NotifierError(PyinotifyError):
- """
- Notifier Exception. Raised on Notifier error.
-
- """
- def __init__(self, err):
- """
- @param err: Exception string's description.
- @type err: string
- """
- PyinotifyError.__init__(self, err)
-
-
-class Notifier:
- """
- Read notifications, process events.
-
- """
- def __init__(self, watch_manager, default_proc_fun=None, read_freq=0,
- threshold=0, timeout=None):
- """
- Initialization. read_freq, threshold and timeout parameters are used
- when looping.
-
- @param watch_manager: Watch Manager.
- @type watch_manager: WatchManager instance
- @param default_proc_fun: Default processing method. If None, a new
- instance of PrintAllEvents will be assigned.
- @type default_proc_fun: instance of ProcessEvent
- @param read_freq: if read_freq == 0, events are read asap,
- if read_freq is > 0, this thread sleeps
- max(0, read_freq - (timeout / 1000)) seconds. But if
- timeout is None it may be different because
- poll is blocking waiting for something to read.
- @type read_freq: int
- @param threshold: File descriptor will be read only if the accumulated
- size to read becomes >= threshold. If != 0, you likely
- want to use it in combination with an appropriate
- value for read_freq because without that you would
- keep looping without really reading anything and that
- until the amount of events to read is >= threshold.
- At least with read_freq set you might sleep.
- @type threshold: int
- @param timeout: see read_freq above. If provided, it must be set in
- milliseconds. See
- https://docs.python.org/3/library/select.html#select.poll.poll
- @type timeout: int
- """
- # Watch Manager instance
- self._watch_manager = watch_manager
- # File descriptor
- self._fd = self._watch_manager.get_fd()
- # Poll object and registration
- self._pollobj = select.poll()
- self._pollobj.register(self._fd, select.POLLIN)
- # This pipe is correctely initialized and used by ThreadedNotifier
- self._pipe = (-1, -1)
- # Event queue
- self._eventq = deque()
- # System processing functor, common to all events
- self._sys_proc_fun = _SysProcessEvent(self._watch_manager, self)
- # Default processing method
- self._default_proc_fun = default_proc_fun
- if default_proc_fun is None:
- self._default_proc_fun = PrintAllEvents()
- # Loop parameters
- self._read_freq = read_freq
- self._threshold = threshold
- self._timeout = timeout
- # Coalesce events option
- self._coalesce = False
- # set of str(raw_event), only used when coalesce option is True
- self._eventset = set()
-
- def append_event(self, event):
- """
- Append a raw event to the event queue.
-
- @param event: An event.
- @type event: _RawEvent instance.
- """
- self._eventq.append(event)
-
- def proc_fun(self):
- return self._default_proc_fun
-
- def coalesce_events(self, coalesce=True):
- """
- Coalescing events. Events are usually processed by batchs, their size
- depend on various factors. Thus, before processing them, events received
- from inotify are aggregated in a fifo queue. If this coalescing
- option is enabled events are filtered based on their unicity, only
- unique events are enqueued, doublons are discarded. An event is unique
- when the combination of its fields (wd, mask, cookie, name) is unique
- among events of a same batch. After a batch of events is processed any
- events is accepted again. By default this option is disabled, you have
- to explictly call this function to turn it on.
-
- @param coalesce: Optional new coalescing value. True by default.
- @type coalesce: Bool
- """
- self._coalesce = coalesce
- if not coalesce:
- self._eventset.clear()
-
- def check_events(self, timeout=None):
- """
- Check for new events available to read, blocks up to timeout
- milliseconds.
-
- @param timeout: If specified it overrides the corresponding instance
- attribute _timeout. timeout must be sepcified in
- milliseconds.
- @type timeout: int
-
- @return: New events to read.
- @rtype: bool
- """
- while True:
- try:
- # blocks up to 'timeout' milliseconds
- if timeout is None:
- timeout = self._timeout
- ret = self._pollobj.poll(timeout)
- except select.error as err:
- if err.args[0] == errno.EINTR:
- continue # interrupted, retry
- else:
- raise
- else:
- break
-
- if not ret or (self._pipe[0] == ret[0][0]):
- return False
- # only one fd is polled
- return ret[0][1] & select.POLLIN
-
- def read_events(self):
- """
- Read events from device, build _RawEvents, and enqueue them.
- """
- buf_ = array.array('i', [0])
- # get event queue size
- if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1:
- return
- queue_size = buf_[0]
- if queue_size < self._threshold:
- log.debug('(fd: %d) %d bytes available to read but threshold is '
- 'fixed to %d bytes', self._fd, queue_size,
- self._threshold)
- return
-
- try:
- # Read content from file
- r = os.read(self._fd, queue_size)
- except Exception as msg:
- raise NotifierError(msg)
- log.debug('Event queue size: %d', queue_size)
- rsum = 0 # counter
- while rsum < queue_size:
- s_size = 16
- # Retrieve wd, mask, cookie and fname_len
- wd, mask, cookie, fname_len = struct.unpack('iIII',
- r[rsum:rsum+s_size])
- # Retrieve name
- bname, = struct.unpack('%ds' % fname_len,
- r[rsum + s_size:rsum + s_size + fname_len])
- # FIXME: should we explictly call sys.getdefaultencoding() here ??
- uname = bname.decode()
- rawevent = _RawEvent(wd, mask, cookie, uname)
- if self._coalesce:
- # Only enqueue new (unique) events.
- raweventstr = str(rawevent)
- if raweventstr not in self._eventset:
- self._eventset.add(raweventstr)
- self._eventq.append(rawevent)
- else:
- self._eventq.append(rawevent)
- rsum += s_size + fname_len
-
- def process_events(self):
- """
- Routine for processing events from queue by calling their
- associated proccessing method (an instance of ProcessEvent).
- It also does internal processings, to keep the system updated.
- """
- while self._eventq:
- raw_event = self._eventq.popleft() # pop next event
- if self._watch_manager.ignore_events:
- log.debug("Event ignored: %s" % repr(raw_event))
- continue
- watch_ = self._watch_manager.get_watch(raw_event.wd)
- if (watch_ is None) and not (raw_event.mask & IN_Q_OVERFLOW):
- if not (raw_event.mask & IN_IGNORED):
- # Not really sure how we ended up here, nor how we should
- # handle these types of events and if it is appropriate to
- # completly skip them (like we are doing here).
- log.warning("Unable to retrieve Watch object associated to %s",
- repr(raw_event))
- continue
- revent = self._sys_proc_fun(raw_event) # system processings
- if watch_ and watch_.proc_fun:
- watch_.proc_fun(revent) # user processings
- else:
- self._default_proc_fun(revent)
- self._sys_proc_fun.cleanup() # remove olds MOVED_* events records
- if self._coalesce:
- self._eventset.clear()
-
- def __daemonize(self, pid_file=None, stdin=os.devnull, stdout=os.devnull,
- stderr=os.devnull):
- """
- pid_file: file where the pid will be written. If pid_file=None the pid
- is written to /var/run/<sys.argv[0]|pyinotify>.pid, if
- pid_file=False no pid_file is written.
- stdin, stdout, stderr: files associated to common streams.
- """
- if pid_file is None:
- dirname = '/var/run/'
- basename = os.path.basename(sys.argv[0]) or 'pyinotify'
- pid_file = os.path.join(dirname, basename + '.pid')
-
- if pid_file and os.path.lexists(pid_file):
- err = 'Cannot daemonize: pid file %s already exists.' % pid_file
- raise NotifierError(err)
-
- def fork_daemon():
- # Adapted from Chad J. Schroeder's recipe
- # @see http://code.activestate.com/recipes/278731/
- pid = os.fork()
- if (pid == 0):
- # parent 2
- os.setsid()
- pid = os.fork()
- if (pid == 0):
- # child
- os.chdir('/')
- os.umask(0o022)
- else:
- # parent 2
- os._exit(0)
- else:
- # parent 1
- os._exit(0)
-
- fd_inp = os.open(stdin, os.O_RDONLY)
- os.dup2(fd_inp, 0)
- fd_out = os.open(stdout, os.O_WRONLY|os.O_CREAT, 0o0600)
- os.dup2(fd_out, 1)
- fd_err = os.open(stderr, os.O_WRONLY|os.O_CREAT, 0o0600)
- os.dup2(fd_err, 2)
-
- # Detach task
- fork_daemon()
-
- # Write pid
- if pid_file:
- flags = os.O_WRONLY|os.O_CREAT|os.O_NOFOLLOW|os.O_EXCL
- fd_pid = os.open(pid_file, flags, 0o0600)
- os.write(fd_pid, bytes(str(os.getpid()) + '\n',
- locale.getpreferredencoding()))
- os.close(fd_pid)
- # Register unlink function
- atexit.register(lambda : os.unlink(pid_file))
-
- def _sleep(self, ref_time):
- # Only consider sleeping if read_freq is > 0
- if self._read_freq > 0:
- cur_time = time.time()
- sleep_amount = self._read_freq - (cur_time - ref_time)
- if sleep_amount > 0:
- log.debug('Now sleeping %d seconds', sleep_amount)
- time.sleep(sleep_amount)
-
- def loop(self, callback=None, daemonize=False, **args):
- """
- Events are read only one time every min(read_freq, timeout)
- seconds at best and only if the size to read is >= threshold.
- After this method returns it must not be called again for the same
- instance.
-
- @param callback: Functor called after each event processing iteration.
- Expects to receive the notifier object (self) as first
- parameter. If this function returns True the loop is
- immediately terminated otherwise the loop method keeps
- looping.
- @type callback: callable object or function
- @param daemonize: This thread is daemonized if set to True.
- @type daemonize: boolean
- @param args: Optional and relevant only if daemonize is True. Remaining
- keyworded arguments are directly passed to daemonize see
- __daemonize() method. If pid_file=None or is set to a
- pathname the caller must ensure the file does not exist
- before this method is called otherwise an exception
- pyinotify.NotifierError will be raised. If pid_file=False
- it is still daemonized but the pid is not written in any
- file.
- @type args: various
- """
- if daemonize:
- self.__daemonize(**args)
-
- # Read and process events forever
- while 1:
- try:
- self.process_events()
- if (callback is not None) and (callback(self) is True):
- break
- ref_time = time.time()
- # check_events is blocking
- if self.check_events():
- self._sleep(ref_time)
- self.read_events()
- except KeyboardInterrupt:
- # Stop monitoring if sigint is caught (Control-C).
- log.debug('Pyinotify stops monitoring.')
- break
- # Close internals
- self.stop()
-
- def stop(self):
- """
- Close inotify's instance (close its file descriptor).
- It destroys all existing watches, pending events,...
- This method is automatically called at the end of loop().
- Afterward it is invalid to access this instance.
- """
- if self._fd is not None:
- self._pollobj.unregister(self._fd)
- os.close(self._fd)
- self._fd = None
- self._sys_proc_fun = None
-
-
-class ThreadedNotifier(threading.Thread, Notifier):
- """
- This notifier inherits from threading.Thread for instanciating a separate
- thread, and also inherits from Notifier, because it is a threaded notifier.
-
- Note that every functionality provided by this class is also provided
- through Notifier class. Moreover Notifier should be considered first because
- it is not threaded and could be easily daemonized.
- """
- def __init__(self, watch_manager, default_proc_fun=None, read_freq=0,
- threshold=0, timeout=None):
- """
- Initialization, initialize base classes. read_freq, threshold and
- timeout parameters are used when looping.
-
- @param watch_manager: Watch Manager.
- @type watch_manager: WatchManager instance
- @param default_proc_fun: Default processing method. See base class.
- @type default_proc_fun: instance of ProcessEvent
- @param read_freq: if read_freq == 0, events are read asap,
- if read_freq is > 0, this thread sleeps
- max(0, read_freq - (timeout / 1000)) seconds.
- @type read_freq: int
- @param threshold: File descriptor will be read only if the accumulated
- size to read becomes >= threshold. If != 0, you likely
- want to use it in combination with an appropriate
- value set for read_freq because without that you would
- keep looping without really reading anything and that
- until the amount of events to read is >= threshold. At
- least with read_freq you might sleep.
- @type threshold: int
- @param timeout: see read_freq above. If provided, it must be set in
- milliseconds. See
- https://docs.python.org/3/library/select.html#select.poll.poll
- @type timeout: int
- """
- # Init threading base class
- threading.Thread.__init__(self)
- # Stop condition
- self._stop_event = threading.Event()
- # Init Notifier base class
- Notifier.__init__(self, watch_manager, default_proc_fun, read_freq,
- threshold, timeout)
- # Create a new pipe used for thread termination
- self._pipe = os.pipe()
- self._pollobj.register(self._pipe[0], select.POLLIN)
-
- def stop(self):
- """
- Stop notifier's loop. Stop notification. Join the thread.
- """
- self._stop_event.set()
- os.write(self._pipe[1], b'stop')
- threading.Thread.join(self)
- Notifier.stop(self)
- self._pollobj.unregister(self._pipe[0])
- os.close(self._pipe[0])
- os.close(self._pipe[1])
-
- def loop(self):
- """
- Thread's main loop. Don't meant to be called by user directly.
- Call inherited start() method instead.
-
- Events are read only once time every min(read_freq, timeout)
- seconds at best and only if the size of events to read is >= threshold.
- """
- # When the loop must be terminated .stop() is called, 'stop'
- # is written to pipe fd so poll() returns and .check_events()
- # returns False which make evaluate the While's stop condition
- # ._stop_event.isSet() wich put an end to the thread's execution.
- while not self._stop_event.isSet():
- self.process_events()
- ref_time = time.time()
- if self.check_events():
- self._sleep(ref_time)
- self.read_events()
-
- def run(self):
- """
- Start thread's loop: read and process events until the method
- stop() is called.
- Never call this method directly, instead call the start() method
- inherited from threading.Thread, which then will call run() in
- its turn.
- """
- self.loop()
-
-
-class TornadoAsyncNotifier(Notifier):
- """
- Tornado ioloop adapter.
-
- """
- def __init__(self, watch_manager, ioloop, callback=None,
- default_proc_fun=None, read_freq=0, threshold=0, timeout=None,
- channel_map=None):
- """
- Note that if later you must call ioloop.close() be sure to let the
- default parameter to all_fds=False.
-
- See example tornado_notifier.py for an example using this notifier.
-
- @param ioloop: Tornado's IO loop.
- @type ioloop: tornado.ioloop.IOLoop instance.
- @param callback: Functor called at the end of each call to handle_read
- (IOLoop's read handler). Expects to receive the
- notifier object (self) as single parameter.
- @type callback: callable object or function
- """
- self.io_loop = ioloop
- self.handle_read_callback = callback
- Notifier.__init__(self, watch_manager, default_proc_fun, read_freq,
- threshold, timeout)
- ioloop.add_handler(self._fd, self.handle_read, ioloop.READ)
-
- def stop(self):
- self.io_loop.remove_handler(self._fd)
- Notifier.stop(self)
-
- def handle_read(self, *args, **kwargs):
- """
- See comment in AsyncNotifier.
-
- """
- self.read_events()
- self.process_events()
- if self.handle_read_callback is not None:
- self.handle_read_callback(self)
-
-
-class AsyncioNotifier(Notifier):
- """
-
- asyncio/trollius event loop adapter.
-
- """
- def __init__(self, watch_manager, loop, callback=None,
- default_proc_fun=None, read_freq=0, threshold=0, timeout=None):
- """
-
- See examples/asyncio_notifier.py for an example usage.
-
- @param loop: asyncio or trollius event loop instance.
- @type loop: asyncio.BaseEventLoop or trollius.BaseEventLoop instance.
- @param callback: Functor called at the end of each call to handle_read.
- Expects to receive the notifier object (self) as
- single parameter.
- @type callback: callable object or function
-
- """
- self.loop = loop
- self.handle_read_callback = callback
- Notifier.__init__(self, watch_manager, default_proc_fun, read_freq,
- threshold, timeout)
- loop.add_reader(self._fd, self.handle_read)
-
- def stop(self):
- self.loop.remove_reader(self._fd)
- Notifier.stop(self)
-
- def handle_read(self, *args, **kwargs):
- self.read_events()
- self.process_events()
- if self.handle_read_callback is not None:
- self.handle_read_callback(self)
-
-
-class Watch:
- """
- Represent a watch, i.e. a file or directory being watched.
-
- """
- __slots__ = ('wd', 'path', 'mask', 'proc_fun', 'auto_add',
- 'exclude_filter', 'dir')
-
- def __init__(self, wd, path, mask, proc_fun, auto_add, exclude_filter):
- """
- Initializations.
-
- @param wd: Watch descriptor.
- @type wd: int
- @param path: Path of the file or directory being watched.
- @type path: str
- @param mask: Mask.
- @type mask: int
- @param proc_fun: Processing callable object.
- @type proc_fun:
- @param auto_add: Automatically add watches on new directories.
- @type auto_add: bool
- @param exclude_filter: Boolean function, used to exclude new
- directories from being automatically watched.
- See WatchManager.__init__
- @type exclude_filter: callable object
- """
- self.wd = wd
- self.path = path
- self.mask = mask
- self.proc_fun = proc_fun
- self.auto_add = auto_add
- self.exclude_filter = exclude_filter
- self.dir = os.path.isdir(self.path)
-
- def __repr__(self):
- """
- @return: String representation.
- @rtype: str
- """
- s = ' '.join(['%s%s%s' % (output_format.field_name(attr),
- output_format.punctuation('='),
- output_format.field_value(getattr(self,
- attr))) \
- for attr in self.__slots__ if not attr.startswith('_')])
-
- s = '%s%s %s %s' % (output_format.punctuation('<'),
- output_format.class_name(self.__class__.__name__),
- s,
- output_format.punctuation('>'))
- return s
-
-
-class ExcludeFilter:
- """
- ExcludeFilter is an exclusion filter.
- """
- def __init__(self, arg_lst):
- """
- Examples:
- ef1 = ExcludeFilter(["/etc/rc.*", "/etc/hostname"])
- ef2 = ExcludeFilter("/my/path/exclude.lst")
- Where exclude.lst contains:
- /etc/rc.*
- /etc/hostname
-
- Note: it is not possible to exclude a file if its encapsulating
- directory is itself watched. See this issue for more details
- https://github.com/seb-m/pyinotify/issues/31
-
- @param arg_lst: is either a list of patterns or a filename from which
- patterns will be loaded.
- @type arg_lst: list of str or str
- """
- if isinstance(arg_lst, str):
- lst = self._load_patterns_from_file(arg_lst)
- elif isinstance(arg_lst, list):
- lst = arg_lst
- else:
- raise TypeError
-
- self._lregex = []
- for regex in lst:
- self._lregex.append(re.compile(regex, re.UNICODE))
-
- def _load_patterns_from_file(self, filename):
- lst = []
- with open(filename, 'r') as file_obj:
- for line in file_obj.readlines():
- # Trim leading an trailing whitespaces
- pattern = line.strip()
- if not pattern or pattern.startswith('#'):
- continue
- lst.append(pattern)
- return lst
-
- def _match(self, regex, path):
- return regex.match(path) is not None
-
- def __call__(self, path):
- """
- @param path: Path to match against provided regexps.
- @type path: str
- @return: Return True if path has been matched and should
- be excluded, False otherwise.
- @rtype: bool
- """
- for regex in self._lregex:
- if self._match(regex, path):
- return True
- return False
-
-
-class WatchManagerError(Exception):
- """
- WatchManager Exception. Raised on error encountered on watches
- operations.
- """
- def __init__(self, msg, wmd):
- """
- @param msg: Exception string's description.
- @type msg: string
- @param wmd: This dictionary contains the wd assigned to paths of the
- same call for which watches were successfully added.
- @type wmd: dict
- """
- self.wmd = wmd
- Exception.__init__(self, msg)
-
-
-class WatchManager:
- """
- Provide operations for watching files and directories. Its internal
- dictionary is used to reference watched items. When used inside
- threaded code, one must instanciate as many WatchManager instances as
- there are ThreadedNotifier instances.
-
- """
- def __init__(self, exclude_filter=lambda path: False):
- """
- Initialization: init inotify, init watch manager dictionary.
- Raise OSError if initialization fails, raise InotifyBindingNotFoundError
- if no inotify binding was found (through ctypes or from direct access to
- syscalls).
-
- @param exclude_filter: boolean function, returns True if current
- path must be excluded from being watched.
- Convenient for providing a common exclusion
- filter for every call to add_watch.
- @type exclude_filter: callable object
- """
- self._ignore_events = False
- self._exclude_filter = exclude_filter
- self._wmd = {} # watch dict key: watch descriptor, value: watch
-
- self._inotify_wrapper = INotifyWrapper.create()
- if self._inotify_wrapper is None:
- raise InotifyBindingNotFoundError()
-
- self._fd = self._inotify_wrapper.inotify_init() # file descriptor
- if self._fd < 0:
- err = 'Cannot initialize new instance of inotify, %s'
- raise OSError(err % self._inotify_wrapper.str_errno())
-
- def close(self):
- """
- Close inotify's file descriptor, this action will also automatically
- remove (i.e. stop watching) all its associated watch descriptors.
- After a call to this method the WatchManager's instance become useless
- and cannot be reused, a new instance must then be instanciated. It
- makes sense to call this method in few situations for instance if
- several independant WatchManager must be instanciated or if all watches
- must be removed and no other watches need to be added.
- """
- os.close(self._fd)
-
- def get_fd(self):
- """
- Return assigned inotify's file descriptor.
-
- @return: File descriptor.
- @rtype: int
- """
- return self._fd
-
- def get_watch(self, wd):
- """
- Get watch from provided watch descriptor wd.
-
- @param wd: Watch descriptor.
- @type wd: int
- """
- return self._wmd.get(wd)
-
- def del_watch(self, wd):
- """
- Remove watch entry associated to watch descriptor wd.
-
- @param wd: Watch descriptor.
- @type wd: int
- """
- try:
- del self._wmd[wd]
- except KeyError as err:
- log.error('Cannot delete unknown watch descriptor %s' % str(err))
-
- @property
- def watches(self):
- """
- Get a reference on the internal watch manager dictionary.
-
- @return: Internal watch manager dictionary.
- @rtype: dict
- """
- return self._wmd
-
- def __format_path(self, path):
- """
- Format path to its internal (stored in watch manager) representation.
- """
- # path must be a unicode string (str) and is just normalized.
- return os.path.normpath(path)
-
- def __add_watch(self, path, mask, proc_fun, auto_add, exclude_filter):
- """
- Add a watch on path, build a Watch object and insert it in the
- watch manager dictionary. Return the wd value.
- """
- path = self.__format_path(path)
- if auto_add and not mask & IN_CREATE:
- mask |= IN_CREATE
- wd = self._inotify_wrapper.inotify_add_watch(self._fd, path, mask)
- if wd < 0:
- return wd
- watch = Watch(wd=wd, path=path, mask=mask, proc_fun=proc_fun,
- auto_add=auto_add, exclude_filter=exclude_filter)
- # wd are _always_ indexed with their original unicode paths in wmd.
- self._wmd[wd] = watch
- log.debug('New %s', watch)
- return wd
-
- def __glob(self, path, do_glob):
- if do_glob:
- return glob.iglob(path)
- else:
- return [path]
-
- def add_watch(self, path, mask, proc_fun=None, rec=False,
- auto_add=False, do_glob=False, quiet=True,
- exclude_filter=None):
- """
- Add watch(s) on the provided |path|(s) with associated |mask| flag
- value and optionally with a processing |proc_fun| function and
- recursive flag |rec| set to True.
- All |path| components _must_ be str (i.e. unicode) objects.
- If |path| is already watched it is ignored, but if it is called with
- option rec=True a watch is put on each one of its not-watched
- subdirectory.
-
- @param path: Path to watch, the path can either be a file or a
- directory. Also accepts a sequence (list) of paths.
- @type path: string or list of strings
- @param mask: Bitmask of events.
- @type mask: int
- @param proc_fun: Processing object.
- @type proc_fun: function or ProcessEvent instance or instance of
- one of its subclasses or callable object.
- @param rec: Recursively add watches from path on all its
- subdirectories, set to False by default (doesn't
- follows symlinks in any case).
- @type rec: bool
- @param auto_add: Automatically add watches on newly created
- directories in watched parent |path| directory.
- If |auto_add| is True, IN_CREATE is ored with |mask|
- when the watch is added.
- @type auto_add: bool
- @param do_glob: Do globbing on pathname (see standard globbing
- module for more informations).
- @type do_glob: bool
- @param quiet: if False raises a WatchManagerError exception on
- error. See example not_quiet.py.
- @type quiet: bool
- @param exclude_filter: predicate (boolean function), which returns
- True if the current path must be excluded
- from being watched. This argument has
- precedence over exclude_filter passed to
- the class' constructor.
- @type exclude_filter: callable object
- @return: dict of paths associated to watch descriptors. A wd value
- is positive if the watch was added sucessfully, otherwise
- the value is negative. If the path was invalid or was already
- watched it is not included into this returned dictionary.
- @rtype: dict of {str: int}
- """
- ret_ = {} # return {path: wd, ...}
-
- if exclude_filter is None:
- exclude_filter = self._exclude_filter
-
- # normalize args as list elements
- for npath in self.__format_param(path):
- # Require that path be a unicode string
- if not isinstance(npath, str):
- ret_[path] = -3
- continue
-
- # unix pathname pattern expansion
- for apath in self.__glob(npath, do_glob):
- # recursively list subdirs according to rec param
- for rpath in self.__walk_rec(apath, rec):
- if not exclude_filter(rpath):
- wd = ret_[rpath] = self.__add_watch(rpath, mask,
- proc_fun,
- auto_add,
- exclude_filter)
- if wd < 0:
- err = ('add_watch: cannot watch %s WD=%d, %s' % \
- (rpath, wd,
- self._inotify_wrapper.str_errno()))
- if quiet:
- log.error(err)
- else:
- raise WatchManagerError(err, ret_)
- else:
- # Let's say -2 means 'explicitely excluded
- # from watching'.
- ret_[rpath] = -2
- return ret_
-
- def __get_sub_rec(self, lpath):
- """
- Get every wd from self._wmd if its path is under the path of
- one (at least) of those in lpath. Doesn't follow symlinks.
-
- @param lpath: list of watch descriptor
- @type lpath: list of int
- @return: list of watch descriptor
- @rtype: list of int
- """
- for d in lpath:
- root = self.get_path(d)
- if root is not None:
- # always keep root
- yield d
- else:
- # if invalid
- continue
-
- # nothing else to expect
- if not os.path.isdir(root):
- continue
-
- # normalization
- root = os.path.normpath(root)
- # recursion
- lend = len(root)
- for iwd in self._wmd.items():
- cur = iwd[1].path
- pref = os.path.commonprefix([root, cur])
- if root == os.sep or (len(pref) == lend and \
- len(cur) > lend and \
- cur[lend] == os.sep):
- yield iwd[1].wd
-
- def update_watch(self, wd, mask=None, proc_fun=None, rec=False,
- auto_add=False, quiet=True):
- """
- Update existing watch descriptors |wd|. The |mask| value, the
- processing object |proc_fun|, the recursive param |rec| and the
- |auto_add| and |quiet| flags can all be updated.
-
- @param wd: Watch Descriptor to update. Also accepts a list of
- watch descriptors.
- @type wd: int or list of int
- @param mask: Optional new bitmask of events.
- @type mask: int
- @param proc_fun: Optional new processing function.
- @type proc_fun: function or ProcessEvent instance or instance of
- one of its subclasses or callable object.
- @param rec: Optionally adds watches recursively on all
- subdirectories contained into |wd| directory.
- @type rec: bool
- @param auto_add: Automatically adds watches on newly created
- directories in the watch's path corresponding to |wd|.
- If |auto_add| is True, IN_CREATE is ored with |mask|
- when the watch is updated.
- @type auto_add: bool
- @param quiet: If False raises a WatchManagerError exception on
- error. See example not_quiet.py
- @type quiet: bool
- @return: dict of watch descriptors associated to booleans values.
- True if the corresponding wd has been successfully
- updated, False otherwise.
- @rtype: dict of {int: bool}
- """
- lwd = self.__format_param(wd)
- if rec:
- lwd = self.__get_sub_rec(lwd)
-
- ret_ = {} # return {wd: bool, ...}
- for awd in lwd:
- apath = self.get_path(awd)
- if not apath or awd < 0:
- err = 'update_watch: invalid WD=%d' % awd
- if quiet:
- log.error(err)
- continue
- raise WatchManagerError(err, ret_)
-
- if mask:
- wd_ = self._inotify_wrapper.inotify_add_watch(self._fd, apath,
- mask)
- if wd_ < 0:
- ret_[awd] = False
- err = ('update_watch: cannot update %s WD=%d, %s' % \
- (apath, wd_, self._inotify_wrapper.str_errno()))
- if quiet:
- log.error(err)
- continue
- raise WatchManagerError(err, ret_)
-
- assert(awd == wd_)
-
- if proc_fun or auto_add:
- watch_ = self._wmd[awd]
-
- if proc_fun:
- watch_.proc_fun = proc_fun
-
- if auto_add:
- watch_.auto_add = auto_add
-
- ret_[awd] = True
- log.debug('Updated watch - %s', self._wmd[awd])
- return ret_
-
- def __format_param(self, param):
- """
- @param param: Parameter.
- @type param: string or int
- @return: wrap param.
- @rtype: list of type(param)
- """
- if isinstance(param, list):
- for p_ in param:
- yield p_
- else:
- yield param
-
- def get_wd(self, path):
- """
- Returns the watch descriptor associated to path. This method
- presents a prohibitive cost, always prefer to keep the WD
- returned by add_watch(). If the path is unknown it returns None.
-
- @param path: Path.
- @type path: str
- @return: WD or None.
- @rtype: int or None
- """
- path = self.__format_path(path)
- for iwd in self._wmd.items():
- if iwd[1].path == path:
- return iwd[0]
-
- def get_path(self, wd):
- """
- Returns the path associated to WD, if WD is unknown it returns None.
-
- @param wd: Watch descriptor.
- @type wd: int
- @return: Path or None.
- @rtype: string or None
- """
- watch_ = self._wmd.get(wd)
- if watch_ is not None:
- return watch_.path
-
- def __walk_rec(self, top, rec):
- """
- Yields each subdirectories of top, doesn't follow symlinks.
- If rec is false, only yield top.
-
- @param top: root directory.
- @type top: string
- @param rec: recursive flag.
- @type rec: bool
- @return: path of one subdirectory.
- @rtype: string
- """
- if not rec or os.path.islink(top) or not os.path.isdir(top):
- yield top
- else:
- for root, dirs, files in os.walk(top):
- yield root
-
- def rm_watch(self, wd, rec=False, quiet=True):
- """
- Removes watch(s).
-
- @param wd: Watch Descriptor of the file or directory to unwatch.
- Also accepts a list of WDs.
- @type wd: int or list of int.
- @param rec: Recursively removes watches on every already watched
- subdirectories and subfiles.
- @type rec: bool
- @param quiet: If False raises a WatchManagerError exception on
- error. See example not_quiet.py
- @type quiet: bool
- @return: dict of watch descriptors associated to booleans values.
- True if the corresponding wd has been successfully
- removed, False otherwise.
- @rtype: dict of {int: bool}
- """
- lwd = self.__format_param(wd)
- if rec:
- lwd = self.__get_sub_rec(lwd)
-
- ret_ = {} # return {wd: bool, ...}
- for awd in lwd:
- # remove watch
- wd_ = self._inotify_wrapper.inotify_rm_watch(self._fd, awd)
- if wd_ < 0:
- ret_[awd] = False
- err = ('rm_watch: cannot remove WD=%d, %s' % \
- (awd, self._inotify_wrapper.str_errno()))
- if quiet:
- log.error(err)
- continue
- raise WatchManagerError(err, ret_)
-
- # Remove watch from our dictionary
- if awd in self._wmd:
- del self._wmd[awd]
- ret_[awd] = True
- log.debug('Watch WD=%d (%s) removed', awd, self.get_path(awd))
- return ret_
-
-
- def watch_transient_file(self, filename, mask, proc_class):
- """
- Watch a transient file, which will be created and deleted frequently
- over time (e.g. pid file).
-
- @attention: Currently under the call to this function it is not
- possible to correctly watch the events triggered into the same
- base directory than the directory where is located this watched
- transient file. For instance it would be wrong to make these
- two successive calls: wm.watch_transient_file('/var/run/foo.pid', ...)
- and wm.add_watch('/var/run/', ...)
-
- @param filename: Filename.
- @type filename: string
- @param mask: Bitmask of events, should contain IN_CREATE and IN_DELETE.
- @type mask: int
- @param proc_class: ProcessEvent (or of one of its subclass), beware of
- accepting a ProcessEvent's instance as argument into
- __init__, see transient_file.py example for more
- details.
- @type proc_class: ProcessEvent's instance or of one of its subclasses.
- @return: Same as add_watch().
- @rtype: Same as add_watch().
- """
- dirname = os.path.dirname(filename)
- if dirname == '':
- return {} # Maintains coherence with add_watch()
- basename = os.path.basename(filename)
- # Assuming we are watching at least for IN_CREATE and IN_DELETE
- mask |= IN_CREATE | IN_DELETE
-
- def cmp_name(event):
- if getattr(event, 'name') is None:
- return False
- return basename == event.name
- return self.add_watch(dirname, mask,
- proc_fun=proc_class(ChainIfTrue(func=cmp_name)),
- rec=False,
- auto_add=False, do_glob=False,
- exclude_filter=lambda path: False)
-
- def get_ignore_events(self):
- return self._ignore_events
-
- def set_ignore_events(self, nval):
- self._ignore_events = nval
-
- ignore_events = property(get_ignore_events, set_ignore_events,
- "Make watch manager ignoring new events.")
-
-
-class RawOutputFormat:
- """
- Format string representations.
- """
- def __init__(self, format=None):
- self.format = format or {}
-
- def simple(self, s, attribute):
- if not isinstance(s, str):
- s = str(s)
- return (self.format.get(attribute, '') + s +
- self.format.get('normal', ''))
-
- def punctuation(self, s):
- """Punctuation color."""
- return self.simple(s, 'normal')
-
- def field_value(self, s):
- """Field value color."""
- return self.simple(s, 'purple')
-
- def field_name(self, s):
- """Field name color."""
- return self.simple(s, 'blue')
-
- def class_name(self, s):
- """Class name color."""
- return self.format.get('red', '') + self.simple(s, 'bold')
-
-output_format = RawOutputFormat()
-
-class ColoredOutputFormat(RawOutputFormat):
- """
- Format colored string representations.
- """
- def __init__(self):
- f = {'normal': '\033[0m',
- 'black': '\033[30m',
- 'red': '\033[31m',
- 'green': '\033[32m',
- 'yellow': '\033[33m',
- 'blue': '\033[34m',
- 'purple': '\033[35m',
- 'cyan': '\033[36m',
- 'bold': '\033[1m',
- 'uline': '\033[4m',
- 'blink': '\033[5m',
- 'invert': '\033[7m'}
- RawOutputFormat.__init__(self, f)
-
-
-def compatibility_mode():
- """
- Use this function to turn on the compatibility mode. The compatibility
- mode is used to improve compatibility with Pyinotify 0.7.1 (or older)
- programs. The compatibility mode provides additional variables 'is_dir',
- 'event_name', 'EventsCodes.IN_*' and 'EventsCodes.ALL_EVENTS' as
- Pyinotify 0.7.1 provided. Do not call this function from new programs!!
- Especially if there are developped for Pyinotify >= 0.8.x.
- """
- setattr(EventsCodes, 'ALL_EVENTS', ALL_EVENTS)
- for evname in globals():
- if evname.startswith('IN_'):
- setattr(EventsCodes, evname, globals()[evname])
- global COMPATIBILITY_MODE
- COMPATIBILITY_MODE = True
-
-
-def command_line():
- """
- By default the watched path is '/tmp' and all types of events are
- monitored. Events monitoring serves forever, type c^c to stop it.
- """
- from optparse import OptionParser
-
- usage = "usage: %prog [options] [path1] [path2] [pathn]"
-
- parser = OptionParser(usage=usage)
- parser.add_option("-v", "--verbose", action="store_true",
- dest="verbose", help="Verbose mode")
- parser.add_option("-r", "--recursive", action="store_true",
- dest="recursive",
- help="Add watches recursively on paths")
- parser.add_option("-a", "--auto_add", action="store_true",
- dest="auto_add",
- help="Automatically add watches on new directories")
- parser.add_option("-g", "--glob", action="store_true",
- dest="glob",
- help="Treat paths as globs")
- parser.add_option("-e", "--events-list", metavar="EVENT[,...]",
- dest="events_list",
- help=("A comma-separated list of events to watch for - "
- "see the documentation for valid options (defaults"
- " to everything)"))
- parser.add_option("-s", "--stats", action="store_true",
- dest="stats",
- help="Display dummy statistics")
- parser.add_option("-V", "--version", action="store_true",
- dest="version", help="Pyinotify version")
- parser.add_option("-f", "--raw-format", action="store_true",
- dest="raw_format",
- help="Disable enhanced output format.")
- parser.add_option("-c", "--command", action="store",
- dest="command",
- help="Shell command to run upon event")
-
- (options, args) = parser.parse_args()
-
- if options.verbose:
- log.setLevel(10)
-
- if options.version:
- print(__version__)
-
- if not options.raw_format:
- global output_format
- output_format = ColoredOutputFormat()
-
- if len(args) < 1:
- path = '/tmp' # default watched path
- else:
- path = args
-
- # watch manager instance
- wm = WatchManager()
- # notifier instance and init
- if options.stats:
- notifier = Notifier(wm, default_proc_fun=Stats(), read_freq=5)
- else:
- notifier = Notifier(wm, default_proc_fun=PrintAllEvents())
-
- # What mask to apply
- mask = 0
- if options.events_list:
- events_list = options.events_list.split(',')
- for ev in events_list:
- evcode = EventsCodes.ALL_FLAGS.get(ev, 0)
- if evcode:
- mask |= evcode
- else:
- parser.error("The event '%s' specified with option -e"
- " is not valid" % ev)
- else:
- mask = ALL_EVENTS
-
- # stats
- cb_fun = None
- if options.stats:
- def cb(s):
- sys.stdout.write(repr(s.proc_fun()))
- sys.stdout.write('\n')
- sys.stdout.write(str(s.proc_fun()))
- sys.stdout.write('\n')
- sys.stdout.flush()
- cb_fun = cb
-
- # External command
- if options.command:
- def cb(s):
- subprocess.Popen(options.command, shell=True)
- cb_fun = cb
-
- log.debug('Start monitoring %s, (press c^c to halt pyinotify)' % path)
-
- wm.add_watch(path, mask, rec=options.recursive, auto_add=options.auto_add, do_glob=options.glob)
- # Loop forever (until sigint signal get caught)
- notifier.loop(callback=cb_fun)
-
-
-if __name__ == '__main__':
- command_line()
diff --git a/bitbake/lib/toaster/bldcollector/admin.py b/bitbake/lib/toaster/bldcollector/admin.py
index feaa888c..e5677a2a 100644
--- a/bitbake/lib/toaster/bldcollector/admin.py
+++ b/bitbake/lib/toaster/bldcollector/admin.py
@@ -8,6 +8,7 @@ from django import forms
import django.db.models as models
+@admin.register(BitbakeVersion)
class BitbakeVersionAdmin(admin.ModelAdmin):
# we override the formfield for db URLField
@@ -20,18 +21,17 @@ class BitbakeVersionAdmin(admin.ModelAdmin):
db_field, **kwargs)
+@admin.register(Release)
class ReleaseAdmin(admin.ModelAdmin):
pass
+@admin.register(ToasterSetting)
class ToasterSettingAdmin(admin.ModelAdmin):
pass
+@admin.register(Layer_Version)
class LayerVersionsAdmin(admin.ModelAdmin):
pass
-admin.site.register(Layer_Version, LayerVersionsAdmin)
-admin.site.register(BitbakeVersion, BitbakeVersionAdmin)
-admin.site.register(Release, ReleaseAdmin)
-admin.site.register(ToasterSetting, ToasterSettingAdmin)
diff --git a/bitbake/lib/toaster/bldcontrol/admin.py b/bitbake/lib/toaster/bldcontrol/admin.py
index 1754bc11..04dd4e16 100644
--- a/bitbake/lib/toaster/bldcontrol/admin.py
+++ b/bitbake/lib/toaster/bldcontrol/admin.py
@@ -5,7 +5,7 @@
from django.contrib import admin
from .models import BuildEnvironment
+@admin.register(BuildEnvironment)
class BuildEnvironmentAdmin(admin.ModelAdmin):
pass
-admin.site.register(BuildEnvironment, BuildEnvironmentAdmin)
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 577e765f..4e0a6e90 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -6,6 +6,11 @@
# SPDX-License-Identifier: GPL-2.0-only
#
+# DEBUGGING NOTE: if you need to add debug statements into the files within a
+# Toaster-imported layer, you must disable the "git fetch ..." command after
+# the initial clone else any changes will be automatically replaced with the
+# original code
+
import os
import re
import shutil
@@ -35,7 +40,8 @@ class LocalhostBEController(BuildEnvironmentController):
def __init__(self, be):
super(LocalhostBEController, self).__init__(be)
- self.pokydirname = None
+ self.bitbake_dirname = None
+ self.oecore_dirname = None
self.islayerset = False
def _shellcmd(self, command, cwd=None, nowait=False,env=None):
@@ -45,7 +51,10 @@ class LocalhostBEController(BuildEnvironmentController):
env=os.environ.copy()
logger.debug("lbc_shellcmd: (%s) %s" % (cwd, command))
- p = subprocess.Popen(command, cwd = cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+ if isinstance(command, str):
+ p = subprocess.Popen(command, cwd = cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+ else:
+ p = subprocess.Popen(command, cwd = cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
if nowait:
return
(out,err) = p.communicate()
@@ -95,7 +104,7 @@ class LocalhostBEController(BuildEnvironmentController):
# 1. get a list of repos with branches, and map dirpaths for each layer
gitrepos = {}
- # if we're using a remotely fetched version of bitbake add its git
+ # We are using a remotely fetched version of bitbake - add its git
# details to the list of repos to clone
if bitbake.giturl and bitbake.commit:
gitrepos[(bitbake.giturl, bitbake.commit)] = []
@@ -130,7 +139,7 @@ class LocalhostBEController(BuildEnvironmentController):
cached_layers = {}
try:
- for remotes in self._shellcmd("git remote -v", self.be.sourcedir,env=git_env).split("\n"):
+ for remotes in self._shellcmd(['git', 'remote', '-v'], self.be.sourcedir, env=git_env).split("\n"):
try:
remote = remotes.split("\t")[1].split(" ")[0]
if remote not in cached_layers:
@@ -149,17 +158,24 @@ class LocalhostBEController(BuildEnvironmentController):
clone_total=len(gitrepos.keys())
self.setCloneStatus(bitbake,'Started',clone_total,clone_count,'')
for giturl, commit in gitrepos.keys():
- self.setCloneStatus(bitbake,'progress',clone_total,clone_count,gitrepos[(giturl, commit)][0][0])
+ layer_name = gitrepos[(giturl, commit)][0][0]
+ self.setCloneStatus(bitbake,'progress',clone_total,clone_count,layer_name)
clone_count += 1
localdirname = os.path.join(self.be.sourcedir, self.getGitCloneDirectory(giturl, commit))
logger.debug("localhostbecontroller: giturl %s:%s checking out in current directory %s" % (giturl, commit, localdirname))
+ # Capture local paths to release's bitbake and openembedded-core layers
+ if 'bitbake' == layer_name:
+ self.bitbake_dirname = localdirname
+ # Capture local path to release's openembedded-core layer (for 'oe-init-build-env')
+ if 'openembedded-core' == layer_name:
+ self.oecore_dirname = localdirname
+
# see if our directory is a git repository
if os.path.exists(localdirname):
try:
- localremotes = self._shellcmd("git remote -v",
- localdirname,env=git_env)
+ localremotes = self._shellcmd(['git' ,'remote', '-v'], localdirname, env=git_env)
# NOTE: this nice-to-have check breaks when using git remaping to get past firewall
# Re-enable later with .gitconfig remapping checks
#if not giturl in localremotes and commit != 'HEAD':
@@ -172,28 +188,21 @@ class LocalhostBEController(BuildEnvironmentController):
else:
if giturl in cached_layers:
logger.debug("localhostbecontroller git-copying %s to %s" % (cached_layers[giturl], localdirname))
- self._shellcmd("git clone \"%s\" \"%s\"" % (cached_layers[giturl], localdirname),env=git_env)
- self._shellcmd("git remote remove origin", localdirname,env=git_env)
- self._shellcmd("git remote add origin \"%s\"" % giturl, localdirname,env=git_env)
+ self._shellcmd(['git', 'clone', cached_layers[giturl], localdirname], env=git_env)
+ self._shellcmd(['git', 'remote', 'remove', 'origin'], localdirname, env=git_env)
+ self._shellcmd(['git', 'remote', 'add', 'origin', giturl], localdirname, env=git_env)
else:
logger.debug("localhostbecontroller: cloning %s in %s" % (giturl, localdirname))
- self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname),env=git_env)
+ self._shellcmd(['git', 'clone', giturl, localdirname], env=git_env)
# branch magic name "HEAD" will inhibit checkout
if commit != "HEAD":
logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
ref = commit if re.match('^[a-fA-F0-9]+$', commit) else 'origin/%s' % commit
- self._shellcmd('git fetch && git reset --hard "%s"' % ref, localdirname,env=git_env)
-
- # take the localdirname as poky dir if we can find the oe-init-build-env
- if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
- logger.debug("localhostbecontroller: selected poky dir name %s" % localdirname)
- self.pokydirname = localdirname
-
- # make sure we have a working bitbake
- if not os.path.exists(os.path.join(self.pokydirname, 'bitbake')):
- logger.debug("localhostbecontroller: checking bitbake into the poky dirname %s " % self.pokydirname)
- self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % (bitbake.commit, bitbake.giturl, os.path.join(self.pokydirname, 'bitbake')),env=git_env)
+ # DEBUGGING NOTE: this is the 'git fetch" to disable after the initial clone to
+ # prevent inserted debugging commands from being lost
+ self._shellcmd(['git', 'fetch'], localdirname, env=git_env)
+ self._shellcmd(['git', 'reset', '--hard', ref], localdirname, env=git_env)
# verify our repositories
for name, dirpath, index in gitrepos[(giturl, commit)]:
@@ -208,21 +217,11 @@ class LocalhostBEController(BuildEnvironmentController):
self.setCloneStatus(bitbake,'complete',clone_total,clone_count,'')
logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
- # Resolve self.pokydirname if not resolved yet, consider the scenario
- # where all layers are local, that's the else clause
- if self.pokydirname is None:
- if os.path.exists(os.path.join(self.be.sourcedir, "oe-init-build-env")):
- logger.debug("localhostbecontroller: selected poky dir name %s" % self.be.sourcedir)
- self.pokydirname = self.be.sourcedir
- else:
- # Alternatively, scan local layers for relative "oe-init-build-env" location
- for layer in layers:
- if os.path.exists(os.path.join(layer.layer_version.layer.local_source_dir,"..","oe-init-build-env")):
- logger.debug("localhostbecontroller, setting pokydirname to %s" % (layer.layer_version.layer.local_source_dir))
- self.pokydirname = os.path.join(layer.layer_version.layer.local_source_dir,"..")
- break
- else:
- logger.error("pokydirname is not set, you will run into trouble!")
+ # If self.bitbake_dirname still missing (e.g. 'local' build), use local bitbake
+ if not self.bitbake_dirname:
+ self.bitbake_dirname = os.environ.get('BBBASEDIR','')
+ if not self.bitbake_dirname:
+ logger.error("bitbake_dirname is not set, you will run into trouble!")
# 5. create custom layer and add custom recipes to it
for target in targets:
@@ -319,13 +318,37 @@ class LocalhostBEController(BuildEnvironmentController):
layers = self.setLayers(bitbake, layers, targets)
is_merged_attr = bitbake.req.project.merged_attr
+ # Is this build for a Bitbake Setup installation?
+ isBBSetup = False
+ release_description = bitbake.req.project.release.description
+ if 'master' in release_description:
+ # This release going forward is Bitbake Setup
+ isBBSetup = True
+ else:
+ try:
+ # Yocto Project 5.3 "Whinlatter"
+ params = release_description.split(' ')
+ # Work past YP "10.1"
+ release_params = params[2].split('.')
+ if int(release_params[0]) > 5:
+ isBBSetup = True
+ if (release_params[0] == '5') and (int(release_params[1]) >= 3):
+ isBBSetup = True
+ except:
+ # If release schema not recognized, then this release is in the future
+ isBBSetup = True
+
git_env = os.environ.copy()
+ toaster_run_dir = git_env['BUILDDIR']
# (note: add custom environment settings here)
try:
# insure that the project init/build uses the selected bitbake, and not Toaster's
del git_env['TEMPLATECONF']
- del git_env['BBBASEDIR']
del git_env['BUILDDIR']
+ # Insure that oe-init finds the right bitbake (e.g. 'oe-buildenv-internal')
+ git_env['BBBASEDIR'] = self.bitbake_dirname
+ git_env['BITBAKEDIR'] = self.bitbake_dirname
+
except KeyError:
pass
@@ -334,29 +357,45 @@ class LocalhostBEController(BuildEnvironmentController):
builddir = bitbake.req.project.builddir
else:
builddir = '%s-toaster-%d' % (self.be.builddir, bitbake.req.project.id)
- oe_init = os.path.join(self.pokydirname, 'oe-init-build-env')
- # init build environment
+ oe_init = os.path.join(self.oecore_dirname, 'oe-init-build-env')
+ setup_init = os.path.join(builddir, 'init-build-env')
+ # init build environment and build directory structure
try:
custom_script = ToasterSetting.objects.get(name="CUSTOM_BUILD_INIT_SCRIPT").value
custom_script = custom_script.replace("%BUILDDIR%" ,builddir)
self._shellcmd("bash -c 'source %s'" % (custom_script),env=git_env)
except ToasterSetting.DoesNotExist:
- self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
- self.be.sourcedir,env=git_env)
+ # Normal setup
+ if isBBSetup:
+ # Create an 'init-build-env' for the build directory that points to respective cloned bitbake
+ os.makedirs(builddir, exist_ok = True)
+ build_init_script = os.path.join(builddir,'init-build-env')
+ os.system(f"cp {os.path.join(toaster_run_dir,'init-build-env')} {build_init_script}")
+ os.system(f"sed -i -e 's|^cd .*$|cd {self.oecore_dirname}|g' {build_init_script}")
+ os.system(f"sed -i -e 's|^set .*$|set {builddir}|g' {build_init_script}")
+ # Execute the init
+ self._shellcmd(f"bash -c 'source {build_init_script}'", env=git_env)
+ else:
+ self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
+ self.be.sourcedir,env=git_env)
# update bblayers.conf
if not is_merged_attr:
bblconfpath = os.path.join(builddir, "conf/toaster-bblayers.conf")
with open(bblconfpath, 'w') as bblayers:
bblayers.write('# line added by toaster build control\n'
- 'BBLAYERS = "%s"' % ' '.join(layers))
+ 'BBLAYERS = "%s"\n' % ' '.join(layers))
# write configuration file
confpath = os.path.join(builddir, 'conf/toaster.conf')
with open(confpath, 'w') as conf:
+ if isBBSetup:
+ # Force TOPDIR
+ conf.write(f'# Force TOPDIR\n')
+ conf.write(f'TOPDIR = "{builddir}"\n\n')
for var in variables:
- conf.write('%s="%s"\n' % (var.name, var.value))
- conf.write('INHERIT+="toaster buildhistory"')
+ conf.write('%s = "%s"\n' % (var.name, var.value))
+ conf.write('INHERIT += "toaster buildhistory"\n')
else:
# Append the Toaster-specific values directly to the bblayers.conf
bblconfpath = os.path.join(builddir, "conf/bblayers.conf")
@@ -432,10 +471,10 @@ class LocalhostBEController(BuildEnvironmentController):
# run bitbake server from the clone if available
# otherwise pick it from the PATH
- bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
+ bitbake = os.path.join(self.bitbake_dirname, 'bin', 'bitbake')
if not os.path.exists(bitbake):
logger.info("Bitbake not available under %s, will try to use it from PATH" %
- self.pokydirname)
+ self.bitbake_dirname)
for path in os.environ["PATH"].split(os.pathsep):
if os.path.exists(os.path.join(path, 'bitbake')):
bitbake = os.path.join(path, 'bitbake')
@@ -443,15 +482,25 @@ class LocalhostBEController(BuildEnvironmentController):
else:
logger.error("Looks like Bitbake is not available, please fix your environment")
+ # NOTE: BB SETUP requires a 'bblayers.conf' even though Toaster immediately overrides it with 'toaster-bblayers.conf'
+ bblayers = os.path.join(builddir,"conf/bblayers.conf")
toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf")
- if not is_merged_attr:
- self._shellcmd('%s bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s --read %s '
- '--server-only -B 0.0.0.0:0\"' % (env_clean, oe_init,
- builddir, bitbake, confpath, toasterlayers), self.be.sourcedir)
+ if isBBSetup:
+ # Use 'init-build-env' model
+ if not is_merged_attr:
+ self._shellcmd(f'{env_clean} bash -c \"source {setup_init}; BITBAKE_UI="knotty" {bitbake} --read {confpath} --read {bblayers} --read {toasterlayers} '
+ '--server-only -B 0.0.0.0:0\"', self.be.sourcedir,env=git_env)
+ else:
+ self._shellcmd(f'{env_clean} bash -c \"source {setup_init}; BITBAKE_UI="knotty" {bitbake} '
+ '--server-only -B 0.0.0.0:0\"', self.be.sourcedir,env=git_env)
else:
- self._shellcmd('%s bash -c \"source %s %s; BITBAKE_UI="knotty" %s '
- '--server-only -B 0.0.0.0:0\"' % (env_clean, oe_init,
- builddir, bitbake), self.be.sourcedir)
+ # Use 'oe-init-build-env' model
+ if not is_merged_attr:
+ self._shellcmd(f'{env_clean} bash -c \"source {oe_init} {builddir}; BITBAKE_UI="knotty" {bitbake} --read {confpath} --read {bblayers} --read {toasterlayers} '
+ '--server-only -B 0.0.0.0:0\"', self.be.sourcedir,env=git_env)
+ else:
+ self._shellcmd(f'{env_clean} bash -c \"source {oe_init} {builddir}; BITBAKE_UI="knotty" {bitbake} '
+ '--server-only -B 0.0.0.0:0\"', self.be.sourcedir,env=git_env)
# read port number from bitbake.lock
self.be.bbport = -1
@@ -498,19 +547,15 @@ class LocalhostBEController(BuildEnvironmentController):
local_bitbake = os.path.join(os.path.dirname(os.getenv('BBBASEDIR')),
'bitbake')
if not is_merged_attr:
- self._shellcmd(['%s bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:%s" '
- '%s %s -u toasterui --read %s --read %s --token="" >>%s 2>&1;'
- 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:%s %s -m)&\"' \
- % (env_clean, brbe, self.be.bbport, local_bitbake, bbtargets, confpath, toasterlayers, log,
- self.be.bbport, bitbake,)],
+ self._shellcmd(f'{env_clean} bash -c \"(TOASTER_BRBE="{brbe}" BBSERVER="0.0.0.0:{self.be.bbport}" '
+ f'{bitbake} {bbtargets} -u toasterui --read {confpath} --read {bblayers} --read {toasterlayers} --token="" >>{log} 2>&1;'
+ f'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:{self.be.bbport} {bitbake} -m)&\"',
builddir, nowait=True)
else:
- self._shellcmd(['%s bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:%s" '
- '%s %s -u toasterui --token="" >>%s 2>&1;'
- 'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:%s %s -m)&\"' \
- % (env_clean, brbe, self.be.bbport, local_bitbake, bbtargets, log,
- self.be.bbport, bitbake,)],
+ self._shellcmd(f'{env_clean} bash -c \"(TOASTER_BRBE="{brbe}" BBSERVER="0.0.0.0:{self.be.bbport}" '
+ f'{local_bitbake} {bbtargets} -u toasterui --token="" >>{log} 2>&1;'
+ f'BITBAKE_UI="knotty" BBSERVER=0.0.0.0:{self.be.bbport} {bitbake} -m)&\"',
builddir, nowait=True)
logger.debug('localhostbecontroller: Build launched, exiting. '
- 'Follow build logs at %s' % log)
+ f'Follow build logs at {log}')
diff --git a/bitbake/lib/toaster/orm/fixtures/check_fixtures.py b/bitbake/lib/toaster/orm/fixtures/check_fixtures.py
new file mode 100755
index 00000000..b33c093d
--- /dev/null
+++ b/bitbake/lib/toaster/orm/fixtures/check_fixtures.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2026 Linux Foundation
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import json
+import urllib.request
+
+import gen_fixtures as fixtures
+
+RELEASE_URL = "https://dashboard.yoctoproject.org/releases.json"
+LAYER_INDEX_URL = "https://layers.openembedded.org/layerindex/api/branches/"
+
+# Load a JSON file via REST
+def fetch_json(url):
+ with urllib.request.urlopen(url) as response:
+ if response.getcode() == 200:
+ data = response.read().decode("utf-8")
+ return(json.loads(data))
+ else:
+ print("Couldn't access %s: %s" % (url, reponse.getcode()))
+ exit(1)
+
+# Grab the existing release branches in the Layer Index
+branches = fetch_json(LAYER_INDEX_URL)
+index_branches = [
+ e["name"].lower() for e in branches
+]
+
+# Grab the recent release branches and add master, so we can ignore old branches
+releases = fetch_json(RELEASE_URL)
+active_releases = []
+active_releases_bitbake = {}
+active_but_not_index = []
+for e in releases:
+ if e["series"] != "current":
+ continue
+ release = e["release_codename"].lower()
+ if not release in index_branches:
+ active_but_not_index.append(release)
+ else:
+ active_releases.append(release)
+ active_releases_bitbake[release] = e["bitbake_version"]
+active_releases.append("master")
+active_releases.append("head")
+active_releases.sort()
+
+if active_but_not_index:
+ print(f"Note: Active releases that are not yet in the Layer Index: {active_but_not_index}" )
+
+# Get the list of releases instantiated in the Toaster fixtures
+fixtures_releases = []
+fixtures_releases_bitbake = {}
+for fixture in fixtures.current_releases:
+ release = fixture[0].lower()
+ fixtures_releases.append(release)
+ fixtures_releases_bitbake[release] = fixture[6]
+fixtures_releases.sort()
+
+# Report the resulting status
+ok = True
+for release in active_releases:
+ if release in ("master","head"):
+ continue
+ if active_releases_bitbake[release] != fixtures_releases_bitbake[release]:
+ print(f"ERROR: Bitbake version mismatch for '{release}': {active_releases_bitbake[release]} versus {fixtures_releases_bitbake[release]}")
+ ok = False
+if set(active_releases) != set(fixtures_releases):
+ print(f"WARNING: Active releases don't match toaster configured releases, the difference is: {set(active_releases).difference(set(fixtures_releases))}")
+ print(f"Active releases: {sorted(active_releases)}")
+ print(f"Toaster configured releases: {sorted(fixtures_releases)}")
+ ok = False
+if ok:
+ print("Success, configuration matches")
+
diff --git a/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py b/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py
index 71afe391..58961d8e 100755
--- a/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py
+++ b/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py
@@ -4,12 +4,12 @@
#
# Generate Toaster Fixtures for 'poky.xml' and 'oe-core.xml'
#
-# Copyright (C) 2022 Wind River Systems
+# Copyright (C) 2026 Wind River Systems
# SPDX-License-Identifier: GPL-2.0-only
#
# Edit the 'current_releases' table for each new release cycle
#
-# Usage: ./get_fixtures all
+# Usage: ./get_fixtures --all
#
import os
@@ -32,22 +32,28 @@ verbose = False
# It is also policy to include all active LTS releases.
#
-# [Codename, Yocto Project Version, Release Date, Current Version, Support Level, Poky Version, BitBake branch]
+# [0=Codename, 1=Yocto Project Version, 2=Release Date, 3=Current Version, 4=Support Level, 5=Poky Version, 6=BitBake branch]
current_releases = [
# Release slot #1
- ['Kirkstone','4.0','April 2022','4.0.8 (March 2023)','Stable - Long Term Support (until Apr. 2024)','','2.0'],
+ ['Blacksail','6.1','October 2026','6.1 (October 2026)','Support for 7 months (until April 2027)','','2.20'],
# Release slot #2 'local'
- ['HEAD','HEAD','','Local Yocto Project','HEAD','','HEAD'],
+ ['Wrynose','6.0','April 2026','6.0 (April 2026)','Long Term Support (until April 2030)','','2.18'],
# Release slot #3 'master'
- ['Master','master','','Yocto Project master','master','','master'],
+ ['HEAD','HEAD','','Local Yocto Project','HEAD','','HEAD'],
# Release slot #4
- ['Mickledore','4.2','April 2023','4.2.0 (April 2023)','Support for 7 months (until October 2023)','','2.4'],
-# ['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for 7 months (until May 2023)','','2.2'],
-# ['Honister','3.4','October 2021','3.4.2 (February 2022)','Support for 7 months (until May 2022)','26.0','1.52'],
-# ['Hardknott','3.3','April 2021','3.3.5 (March 2022)','Stable - Support for 13 months (until Apr. 2022)','25.0','1.50'],
-# ['Gatesgarth','3.2','Oct 2020','3.2.4 (May 2021)','EOL','24.0','1.48'],
- # Optional Release slot #5
- ['Dunfell','3.1','April 2020','3.1.23 (February 2023)','Stable - Long Term Support (until Apr. 2024)','23.0','1.46'],
+ ['Master','master','','Yocto Project master','master','','master'],
+ # Release slot #5
+ ['Scarthgap','5.0','April 2024','5.0 (April 2024)','Long Term Support (until April 2028)','','2.8'],
+ #['Whinlatter','5.3','October 2025','5.3 (October 2024)','Support for 7 months (until May 2026)','','2.14'],
+ #['Styhead','5.1','November 2024','5.1.0 (November 2024)','Support for 7 months (until May 2025)','','2.10'],
+ #['Nanbield','4.3','November 2023','4.3.0 (November 2023)','Support for 7 months (until May 2024)','','2.6'],
+ #['Mickledore','4.2','April 2023','4.2.0 (April 2023)','Support for 7 months (until October 2023)','','2.4'],
+ #['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for 7 months (until May 2023)','','2.2'],
+ #['Kirkstone','4.0','April 2022','4.0 (March 2023)','Stable - Long Term Support (until Apr. 2024)','','2.0'],
+ #['Honister','3.4','October 2021','3.4.2 (February 2022)','Support for 7 months (until May 2022)','26.0','1.52'],
+ #['Hardknott','3.3','April 2021','3.3.5 (March 2022)','Stable - Support for 13 months (until Apr. 2022)','25.0','1.50'],
+ #['Gatesgarth','3.2','Oct 2020','3.2.4 (May 2021)','EOL','24.0','1.48'],
+ #['Dunfell','3.1','April 2020','3.1.23 (February 2023)','Stable - Long Term Support (until Apr. 2024)','23.0','1.46'],
]
default_poky_layers = [
@@ -77,15 +83,15 @@ prolog_template = '''\
bitbakeversion_poky_template = '''\
<object model="orm.bitbakeversion" pk="{{bitbake_id}}">
<field type="CharField" name="name">{{name}}</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">{{branch}}</field>
- <field type="CharField" name="dirpath">bitbake</field>
+ <field type="CharField" name="dirpath"></field>
</object>
'''
bitbakeversion_oecore_template = '''\
<object model="orm.bitbakeversion" pk="{{bitbake_id}}">
<field type="CharField" name="name">{{name}}</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">{{bitbakeversion}}</field>
</object>
'''
@@ -220,7 +226,7 @@ def generate_poky():
params['release'] = params['release'][0].lower() + params['release'][1:]
params['name'] = params['release']
params['bitbake_id'] = str(i+1)
- params['branch'] = params['release']
+ params['branch'] = release[6] # Bitbake branch
print_template(bitbakeversion_poky_template,params,fd)
print_str('',fd)
@@ -243,7 +249,7 @@ def generate_poky():
params['description'] = 'Local Yocto Project'
params['name'] = 'local'
else:
- params['help_source'] = 'using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/{{h_release}}">Yocto Project {{Release}} branch</a>'
+ params['help_source'] = 'using the tip of the <a href="https://git.yoctoproject.org/meta-yocto/log/{{h_release}}">Yocto Project {{Release}} branch</a>'
params['description'] = 'Yocto Project {{release_version}} "{{Release}}"'
if 'master' == params['release']:
params['h_release'] = ''
@@ -276,10 +282,17 @@ def generate_poky():
params = {}
params['layer'] = layer
params['layer_id'] = str(i+1)
- params['vcs_url'] = 'git://git.yoctoproject.org/poky'
- params['vcs_web_url'] = 'https://git.yoctoproject.org/cgit/cgit.cgi/poky'
- params['vcs_web_tree_base_url'] = 'https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%'
- params['vcs_web_file_base_url'] = 'https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%'
+ if 'openembedded-core' == layer:
+ params['vcs_url'] = 'https://git.openembedded.org/openembedded-core'
+ params['vcs_web_url'] = 'https://git.openembedded.org/openembedded-core'
+ params['vcs_web_tree_base_url'] = 'https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
+ params['vcs_web_file_base_url'] = 'https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
+ else:
+ # Yocto Project layers: meta-poky, meta-yocto-bsp
+ params['vcs_url'] = 'https://git.yoctoproject.org/meta-yocto'
+ params['vcs_web_url'] = 'https://git.yoctoproject.org/meta-yocto/tree'
+ params['vcs_web_tree_base_url'] = 'https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%'
+ params['vcs_web_file_base_url'] = 'https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%'
if i:
print_str('',fd)
@@ -388,10 +401,10 @@ def generate_oe_core():
params = {}
params['layer'] = layer
params['layer_id'] = str(i+1)
- params['vcs_url'] = 'git://git.openembedded.org/openembedded-core'
- params['vcs_web_url'] = 'https://cgit.openembedded.org/openembedded-core'
- params['vcs_web_tree_base_url'] = 'https://cgit.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
- params['vcs_web_file_base_url'] = 'https://cgit.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
+ params['vcs_url'] = 'https://git.openembedded.org/openembedded-core'
+ params['vcs_web_url'] = 'https://git.openembedded.org/openembedded-core'
+ params['vcs_web_tree_base_url'] = 'https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
+ params['vcs_web_file_base_url'] = 'https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%'
if i:
print_str('',fd)
print_template(layer_oe_core_template,params,fd)
@@ -402,6 +415,68 @@ def generate_oe_core():
print_str(epilog_template,fd)
fd.close()
+#################################
+# Update test project list
+#
+
+def generate_test():
+ this_path = os.path.dirname(os.path.abspath(__file__))
+ test_script = os.path.join(this_path,"../../tests/functional/test_create_new_project.py")
+
+ is_copy = True
+ out_lines = ""
+ with open(test_script, 'r') as file_fd:
+ lines = file_fd.readlines()
+ for line in lines:
+ if 'test_create_new_project_without_name' in line:
+ is_copy = True
+ out_lines += '\n'
+ if is_copy:
+ out_lines += line
+ if line.startswith('class TestCreateNewProject'):
+ is_copy = False
+ # Insert new project entries
+
+ for id,release in enumerate(current_releases):
+ release_id = id + 1
+ release_name = release[0]
+ release_merge = 'True'
+ release_version = ''
+ if 'Master' == release_name:
+ release_name = 'Master'
+ release_title = 'Yocto Project master'
+ release_merge = 'False'
+ elif 'HEAD' == release_name:
+ release_name = 'Local'
+ release_title = 'Local Yocto Project'
+ else:
+ release_title = f'Yocto Project {release[1]} "{release_name}"'
+ release_version = f'{release[1]} '
+ # Insert project
+ test_release_def = f'''
+ def test_create_new_project_{release_name.lower()}(self):
+ """ Test create new project using:
+ - Project Name: Any string
+ - Release: Yocto Project {release_version}"{release_name}" (option value: {release_id})
+ - Merge Toaster settings: {release_merge}
+ """
+ release = '{release_id}'
+ release_title = '{release_title}'
+ project_name = 'project{release_name.lower()}'
+ self.create_new_project(
+ project_name,
+ release,
+ release_title,
+ {release_merge},
+ )
+'''
+ out_lines += test_release_def
+
+ with open('test_create_new_project.py', 'w') as file_fd:
+ file_fd.write(out_lines)
+ print(f"Output: 'test_create_new_project.py'")
+ os.system(f"mv -f test_create_new_project.py {test_script}")
+
#################################
# Help
#
@@ -420,8 +495,9 @@ def main(argv):
global verbose
parser = argparse.ArgumentParser(description='gen_fixtures.py: table generate the fixture files')
- parser.add_argument('--poky', '-p', action='store_const', const='poky', dest='command', help='Generate the poky.xml file')
- parser.add_argument('--oe-core', '-o', action='store_const', const='oe_core', dest='command', help='Generate the oe-core.xml file')
+ parser.add_argument('--poky', '-p', action='store_const', const='poky', dest='command', help="Generate the 'poky.xml' file")
+ parser.add_argument('--oe-core', '-o', action='store_const', const='oe_core', dest='command', help="Generate the 'oe-core.xml' file")
+ parser.add_argument('--test', '-t', action='store_const', const='test', dest='command', help="Update the 'test_create_new_project.py' file")
parser.add_argument('--all', '-a', action='store_const', const='all', dest='command', help='Generate all fixture files')
parser.add_argument('--list', '-l', action='store_const', const='list', dest='command', help='List the release table')
parser.add_argument('--verbose', '-v', action='store_true', dest='verbose', help='Enable verbose debugging output')
@@ -432,11 +508,12 @@ def main(argv):
generate_poky()
elif 'oe_core' == args.command:
generate_oe_core()
+ elif 'test' == args.command:
+ generate_test()
elif 'all' == args.command:
generate_poky()
generate_oe_core()
- elif 'all' == args.command:
- list_releases()
+ generate_test()
elif 'list' == args.command:
list_releases()
diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
index 950f2a98..ccd8457a 100644
--- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
+++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
@@ -8,66 +8,66 @@
<!-- Bitbake versions which correspond to the metadata release -->
<object model="orm.bitbakeversion" pk="1">
- <field type="CharField" name="name">kirkstone</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
- <field type="CharField" name="branch">2.0</field>
+ <field type="CharField" name="name">blacksail</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.20</field>
</object>
<object model="orm.bitbakeversion" pk="2">
+ <field type="CharField" name="name">wrynose</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.18</field>
+ </object>
+ <object model="orm.bitbakeversion" pk="3">
<field type="CharField" name="name">HEAD</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">HEAD</field>
</object>
- <object model="orm.bitbakeversion" pk="3">
+ <object model="orm.bitbakeversion" pk="4">
<field type="CharField" name="name">master</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">master</field>
</object>
- <object model="orm.bitbakeversion" pk="4">
- <field type="CharField" name="name">mickledore</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
- <field type="CharField" name="branch">2.4</field>
- </object>
<object model="orm.bitbakeversion" pk="5">
- <field type="CharField" name="name">dunfell</field>
- <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
- <field type="CharField" name="branch">1.46</field>
+ <field type="CharField" name="name">scarthgap</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.8</field>
</object>
<!-- Releases available -->
<object model="orm.release" pk="1">
- <field type="CharField" name="name">kirkstone</field>
- <field type="CharField" name="description">Openembedded Kirkstone</field>
+ <field type="CharField" name="name">blacksail</field>
+ <field type="CharField" name="description">Openembedded Blacksail</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">kirkstone</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=kirkstone\">OpenEmbedded Kirkstone</a> branch.</field>
+ <field type="CharField" name="branch_name">blacksail</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=blacksail\">OpenEmbedded Blacksail</a> branch.</field>
</object>
<object model="orm.release" pk="2">
+ <field type="CharField" name="name">wrynose</field>
+ <field type="CharField" name="description">Openembedded Wrynose</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">2</field>
+ <field type="CharField" name="branch_name">wrynose</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=wrynose\">OpenEmbedded Wrynose</a> branch.</field>
+ </object>
+ <object model="orm.release" pk="3">
<field type="CharField" name="name">local</field>
<field type="CharField" name="description">Local Openembedded</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">2</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
<field type="CharField" name="branch_name">HEAD</field>
<field type="TextField" name="helptext">Toaster will run your builds with the version of OpenEmbedded that you have cloned or downloaded to your computer.</field>
</object>
- <object model="orm.release" pk="3">
+ <object model="orm.release" pk="4">
<field type="CharField" name="name">master</field>
<field type="CharField" name="description">OpenEmbedded core master</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field>
<field type="CharField" name="branch_name">master</field>
<field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded master</a> branch.</field>
</object>
- <object model="orm.release" pk="4">
- <field type="CharField" name="name">mickledore</field>
- <field type="CharField" name="description">Openembedded Mickledore</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field>
- <field type="CharField" name="branch_name">mickledore</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=mickledore\">OpenEmbedded Mickledore</a> branch.</field>
- </object>
<object model="orm.release" pk="5">
- <field type="CharField" name="name">dunfell</field>
- <field type="CharField" name="description">Openembedded Dunfell</field>
+ <field type="CharField" name="name">scarthgap</field>
+ <field type="CharField" name="description">Openembedded Scarthgap</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">5</field>
- <field type="CharField" name="branch_name">dunfell</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=dunfell\">OpenEmbedded Dunfell</a> branch.</field>
+ <field type="CharField" name="branch_name">scarthgap</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"https://cgit.openembedded.org/openembedded-core/log/?h=scarthgap\">OpenEmbedded Scarthgap</a> branch.</field>
</object>
<!-- Default layers for each release -->
@@ -96,10 +96,10 @@
<!-- Layer for the Local release -->
<object model="orm.layer" pk="1">
<field type="CharField" name="name">openembedded-core</field>
- <field type="CharField" name="vcs_url">git://git.openembedded.org/openembedded-core</field>
- <field type="CharField" name="vcs_web_url">https://cgit.openembedded.org/openembedded-core</field>
- <field type="CharField" name="vcs_web_tree_base_url">https://cgit.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
- <field type="CharField" name="vcs_web_file_base_url">https://cgit.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_url">https://git.openembedded.org/openembedded-core</field>
+ <field type="CharField" name="vcs_web_url">https://git.openembedded.org/openembedded-core</field>
+ <field type="CharField" name="vcs_web_tree_base_url">https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_web_file_base_url">https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
</object>
<object model="orm.layer_version" pk="1">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml b/bitbake/lib/toaster/orm/fixtures/poky.xml
index 121e52fd..d4c1ddfd 100644
--- a/bitbake/lib/toaster/orm/fixtures/poky.xml
+++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
@@ -8,72 +8,72 @@
<!-- Bitbake versions which correspond to the metadata release -->
<object model="orm.bitbakeversion" pk="1">
- <field type="CharField" name="name">kirkstone</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="branch">kirkstone</field>
- <field type="CharField" name="dirpath">bitbake</field>
+ <field type="CharField" name="name">blacksail</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.20</field>
+ <field type="CharField" name="dirpath"></field>
</object>
<object model="orm.bitbakeversion" pk="2">
+ <field type="CharField" name="name">wrynose</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.18</field>
+ <field type="CharField" name="dirpath"></field>
+ </object>
+ <object model="orm.bitbakeversion" pk="3">
<field type="CharField" name="name">HEAD</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">HEAD</field>
- <field type="CharField" name="dirpath">bitbake</field>
+ <field type="CharField" name="dirpath"></field>
</object>
- <object model="orm.bitbakeversion" pk="3">
+ <object model="orm.bitbakeversion" pk="4">
<field type="CharField" name="name">master</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
<field type="CharField" name="branch">master</field>
- <field type="CharField" name="dirpath">bitbake</field>
- </object>
- <object model="orm.bitbakeversion" pk="4">
- <field type="CharField" name="name">mickledore</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="branch">mickledore</field>
- <field type="CharField" name="dirpath">bitbake</field>
+ <field type="CharField" name="dirpath"></field>
</object>
<object model="orm.bitbakeversion" pk="5">
- <field type="CharField" name="name">dunfell</field>
- <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="branch">dunfell</field>
- <field type="CharField" name="dirpath">bitbake</field>
+ <field type="CharField" name="name">scarthgap</field>
+ <field type="CharField" name="giturl">https://git.openembedded.org/bitbake</field>
+ <field type="CharField" name="branch">2.8</field>
+ <field type="CharField" name="dirpath"></field>
</object>
<!-- Releases available -->
<object model="orm.release" pk="1">
- <field type="CharField" name="name">kirkstone</field>
- <field type="CharField" name="description">Yocto Project 4.0 "Kirkstone"</field>
+ <field type="CharField" name="name">blacksail</field>
+ <field type="CharField" name="description">Yocto Project 6.1 "Blacksail"</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">kirkstone</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=kirkstone">Yocto Project Kirkstone branch</a>.</field>
+ <field type="CharField" name="branch_name">blacksail</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/meta-yocto/log/?h=blacksail">Yocto Project Blacksail branch</a>.</field>
</object>
<object model="orm.release" pk="2">
+ <field type="CharField" name="name">wrynose</field>
+ <field type="CharField" name="description">Yocto Project 6.0 "Wrynose"</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">2</field>
+ <field type="CharField" name="branch_name">wrynose</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/meta-yocto/log/?h=wrynose">Yocto Project Wrynose branch</a>.</field>
+ </object>
+ <object model="orm.release" pk="3">
<field type="CharField" name="name">local</field>
<field type="CharField" name="description">Local Yocto Project</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">2</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
<field type="CharField" name="branch_name">HEAD</field>
<field type="TextField" name="helptext">Toaster will run your builds with the version of the Yocto Project you have cloned or downloaded to your computer.</field>
</object>
- <object model="orm.release" pk="3">
+ <object model="orm.release" pk="4">
<field type="CharField" name="name">master</field>
<field type="CharField" name="description">Yocto Project master</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
- <field type="CharField" name="branch_name">master</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto Project Master branch</a>.</field>
- </object>
- <object model="orm.release" pk="4">
- <field type="CharField" name="name">mickledore</field>
- <field type="CharField" name="description">Yocto Project 4.2 "Mickledore"</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field>
- <field type="CharField" name="branch_name">mickledore</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=mickledore">Yocto Project Mickledore branch</a>.</field>
+ <field type="CharField" name="branch_name">master</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/meta-yocto/log/">Yocto Project Master branch</a>.</field>
</object>
<object model="orm.release" pk="5">
- <field type="CharField" name="name">dunfell</field>
- <field type="CharField" name="description">Yocto Project 3.1 "Dunfell"</field>
+ <field type="CharField" name="name">scarthgap</field>
+ <field type="CharField" name="description">Yocto Project 5.0 "Scarthgap"</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">5</field>
- <field type="CharField" name="branch_name">dunfell</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=dunfell">Yocto Project Dunfell branch</a>.</field>
+ <field type="CharField" name="branch_name">scarthgap</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="https://git.yoctoproject.org/meta-yocto/log/?h=scarthgap">Yocto Project Scarthgap branch</a>.</field>
</object>
<!-- Default project layers for each release -->
@@ -146,135 +146,135 @@
<object model="orm.layer" pk="1">
<field type="CharField" name="name">openembedded-core</field>
<field type="CharField" name="layer_index_url"></field>
- <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="vcs_web_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky</field>
- <field type="CharField" name="vcs_web_tree_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
- <field type="CharField" name="vcs_web_file_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_url">https://git.openembedded.org/openembedded-core</field>
+ <field type="CharField" name="vcs_web_url">https://git.openembedded.org/openembedded-core</field>
+ <field type="CharField" name="vcs_web_tree_base_url">https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_web_file_base_url">https://git.openembedded.org/openembedded-core/tree/%path%?h=%branch%</field>
</object>
<object model="orm.layer_version" pk="1">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">kirkstone</field>
+ <field type="CharField" name="branch">blacksail</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer_version" pk="2">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">2</field>
- <field type="CharField" name="branch">HEAD</field>
- <field type="CharField" name="commit">HEAD</field>
+ <field type="CharField" name="branch">wrynose</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer_version" pk="3">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
- <field type="CharField" name="branch">master</field>
+ <field type="CharField" name="branch">HEAD</field>
+ <field type="CharField" name="commit">HEAD</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer_version" pk="4">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">mickledore</field>
+ <field type="CharField" name="branch">master</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer_version" pk="5">
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">5</field>
- <field type="CharField" name="branch">dunfell</field>
+ <field type="CharField" name="branch">scarthgap</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer" pk="2">
<field type="CharField" name="name">meta-poky</field>
<field type="CharField" name="layer_index_url"></field>
- <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="vcs_web_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky</field>
- <field type="CharField" name="vcs_web_tree_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
- <field type="CharField" name="vcs_web_file_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_url">https://git.yoctoproject.org/meta-yocto</field>
+ <field type="CharField" name="vcs_web_url">https://git.yoctoproject.org/meta-yocto/tree</field>
+ <field type="CharField" name="vcs_web_tree_base_url">https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_web_file_base_url">https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%</field>
</object>
<object model="orm.layer_version" pk="6">
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">kirkstone</field>
+ <field type="CharField" name="branch">blacksail</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer_version" pk="7">
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">2</field>
- <field type="CharField" name="branch">HEAD</field>
- <field type="CharField" name="commit">HEAD</field>
+ <field type="CharField" name="branch">wrynose</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer_version" pk="8">
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
- <field type="CharField" name="branch">master</field>
+ <field type="CharField" name="branch">HEAD</field>
+ <field type="CharField" name="commit">HEAD</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer_version" pk="9">
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">mickledore</field>
+ <field type="CharField" name="branch">master</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer_version" pk="10">
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">5</field>
- <field type="CharField" name="branch">dunfell</field>
+ <field type="CharField" name="branch">scarthgap</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer" pk="3">
<field type="CharField" name="name">meta-yocto-bsp</field>
<field type="CharField" name="layer_index_url"></field>
- <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="vcs_web_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky</field>
- <field type="CharField" name="vcs_web_tree_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
- <field type="CharField" name="vcs_web_file_base_url">https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_url">https://git.yoctoproject.org/meta-yocto</field>
+ <field type="CharField" name="vcs_web_url">https://git.yoctoproject.org/meta-yocto/tree</field>
+ <field type="CharField" name="vcs_web_tree_base_url">https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%</field>
+ <field type="CharField" name="vcs_web_file_base_url">https://git.yoctoproject.org/meta-yocto/tree/%path%?h=%branch%</field>
</object>
<object model="orm.layer_version" pk="11">
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">kirkstone</field>
+ <field type="CharField" name="branch">blacksail</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
<object model="orm.layer_version" pk="12">
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">2</field>
- <field type="CharField" name="branch">HEAD</field>
- <field type="CharField" name="commit">HEAD</field>
+ <field type="CharField" name="branch">wrynose</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
<object model="orm.layer_version" pk="13">
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">3</field>
- <field type="CharField" name="branch">master</field>
+ <field type="CharField" name="branch">HEAD</field>
+ <field type="CharField" name="commit">HEAD</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
<object model="orm.layer_version" pk="14">
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">mickledore</field>
+ <field type="CharField" name="branch">master</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
<object model="orm.layer_version" pk="15">
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">5</field>
- <field type="CharField" name="branch">dunfell</field>
+ <field type="CharField" name="branch">scarthgap</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
</django-objects>
diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
index 6d64830e..1f36906e 100644
--- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
+++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
@@ -105,7 +105,8 @@ class Command(BaseCommand):
url_branches = ";branch=%s" % ','.join(allowed_branch_names)
else:
url_branches = ""
- layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches))
+ layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches),
+ load=['layerDependencies', 'recipes', 'machines', 'distros'])
http_progress.stop()
@@ -138,9 +139,7 @@ class Command(BaseCommand):
l.summary = index.layerItems[id].summary
l.description = index.layerItems[id].description
- if created:
- # predefined layers in the fixtures (for example poky.xml)
- # always preempt the Layer Index for these values
+ if True: # (if created:) always override fixture defaults now that poky distro is stripped
l.vcs_url = index.layerItems[id].vcs_url
l.vcs_web_url = index.layerItems[id].vcs_web_url
l.vcs_web_tree_base_url = index.layerItems[id].vcs_web_tree_base_url
@@ -179,7 +178,7 @@ class Command(BaseCommand):
index.layerBranches[id].layer_id)
continue
- if created:
+ if True: # (if created:) always override fixture defaults now that poky distro is stripped
lv.release = li_branch_id_to_toaster_release[index.layerBranches[id].branch_id]
lv.up_date = index.layerBranches[id].updated
lv.commit = index.layerBranches[id].actual_branch
@@ -227,14 +226,18 @@ class Command(BaseCommand):
total = len(index.distros)
for i, id in enumerate(index.distros):
- distro, created = Distro.objects.get_or_create(
- name=index.distros[id].name,
- layer_version=Layer_Version.objects.get(
- pk=li_layer_branch_id_to_toaster_lv_id[index.distros[id].layerbranch_id]))
- distro.up_date = index.distros[id].updated
- distro.name = index.distros[id].name
- distro.description = index.distros[id].description
- distro.save()
+ try:
+ distro, created = Distro.objects.get_or_create(
+ name=index.distros[id].name,
+ layer_version=Layer_Version.objects.get(
+ pk=li_layer_branch_id_to_toaster_lv_id[index.distros[id].layerbranch_id]))
+ distro.up_date = index.distros[id].updated
+ distro.name = index.distros[id].name
+ distro.description = index.distros[id].description
+ distro.save()
+ except KeyError as e:
+ logger.warning("Couldn't process %s (%s)" % (i, repr(e)))
+ continue
self.mini_progress("distros", i, total)
# update machines
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 19c96862..e2f488ed 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -79,7 +79,6 @@ if 'sqlite' in settings.DATABASES['default']['ENGINE']:
# end of HACK
class GitURLValidator(validators.URLValidator):
- import re
regex = re.compile(
r'^(?:ssh|git|http|ftp)s?://' # http:// or https://
r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' # domain...
@@ -1500,7 +1499,7 @@ class Layer_Version(models.Model):
# code lifted, with adaptations, from the layerindex-web application
# https://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/
def _handle_url_path(self, base_url, path):
- import re, posixpath
+ import posixpath
if base_url:
if self.dirpath:
if path:
diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
index 393be754..ce108e2f 100644
--- a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
+++ b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
@@ -27,7 +27,9 @@ from selenium.webdriver.common.by import By
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.common.exceptions import NoSuchElementException, \
StaleElementReferenceException, TimeoutException, \
- SessionNotCreatedException
+ SessionNotCreatedException, WebDriverException, \
+ ElementNotInteractableException
+from selenium.webdriver.common.action_chains import ActionChains
def create_selenium_driver(cls,browser='chrome'):
# set default browser string based on env (if available)
@@ -90,7 +92,7 @@ class Wait(WebDriverWait):
Subclass of WebDriverWait with predetermined timeout and poll
frequency. Also deals with a wider variety of exceptions.
"""
- _TIMEOUT = 10
+ _TIMEOUT = 20
_POLL_FREQUENCY = 0.5
def __init__(self, driver, timeout=_TIMEOUT, poll=_POLL_FREQUENCY):
@@ -98,7 +100,7 @@ class Wait(WebDriverWait):
self._POLL_FREQUENCY = poll
super(Wait, self).__init__(driver, self._TIMEOUT, self._POLL_FREQUENCY)
- def until(self, method, message=''):
+ def until(self, method, message='', scroll=None):
"""
Calls the method provided with the driver as an argument until the
return value is not False.
@@ -107,13 +109,21 @@ class Wait(WebDriverWait):
end_time = time.time() + self._timeout
while True:
try:
+ if scroll:
+ scroll(self._driver)
value = method(self._driver)
if value:
return value
+
+ except ElementNotInteractableException:
+ pass
except NoSuchElementException:
pass
except StaleElementReferenceException:
pass
+ except WebDriverException:
+ # selenium.common.exceptions.WebDriverException: Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Node with given id does not belong to the document"}
+ pass
time.sleep(self._poll)
if time.time() > end_time:
@@ -183,7 +193,7 @@ class SeleniumTestCaseBase(unittest.TestCase):
self.driver.get(abs_url)
try: # Ensure page is loaded before proceeding
- self.wait_until_visible("#global-nav", poll=3)
+ self.wait_until_visible("#global-nav")
except NoSuchElementException:
self.driver.implicitly_wait(3)
except TimeoutException:
@@ -208,36 +218,59 @@ class SeleniumTestCaseBase(unittest.TestCase):
""" Return the element which currently has focus on the page """
return self.driver.switch_to.active_element
- def wait_until_present(self, selector, poll=0.5):
+ def wait_until_present(self, selector, timeout=Wait._TIMEOUT):
""" Wait until element matching CSS selector is on the page """
is_present = lambda driver: self.find(selector)
msg = 'An element matching "%s" should be on the page' % selector
- element = Wait(self.driver, poll=poll).until(is_present, msg)
- if poll > 2:
- time.sleep(poll) # element need more delay to be present
+ element = Wait(self.driver, timeout=timeout).until(is_present, msg)
return element
- def wait_until_visible(self, selector, poll=1):
+ def wait_until_element_visible(self, element, timeout=Wait._TIMEOUT):
+ """ Wait until element matching CSS selector is visible on the page """
+ is_visible = lambda driver: element.is_displayed()
+ msg = 'An element should be visible'
+ Wait(self.driver, timeout=timeout).until(is_visible, msg)
+ return
+
+ def wait_until_visible(self, selector, timeout=Wait._TIMEOUT):
""" Wait until element matching CSS selector is visible on the page """
is_visible = lambda driver: self.find(selector).is_displayed()
msg = 'An element matching "%s" should be visible' % selector
- Wait(self.driver, poll=poll).until(is_visible, msg)
- time.sleep(poll) # wait for visibility to settle
+ Wait(self.driver, timeout=timeout).until(is_visible, msg)
return self.find(selector)
- def wait_until_clickable(self, selector, poll=1):
+ def wait_until_not_visible(self, selector, timeout=Wait._TIMEOUT):
+ """ Wait until element matching CSS selector is not visible on the page """
+ is_visible = lambda driver: self.find(selector).is_displayed()
+ msg = 'An element matching "%s" should be visible' % selector
+ Wait(self.driver, timeout=timeout).until_not(is_visible, msg)
+ return self.find(selector)
+
+ def wait_until_clickable(self, selector, timeout=Wait._TIMEOUT):
""" Wait until element matching CSS selector is visible on the page """
- WebDriverWait(
- self.driver,
- Wait._TIMEOUT,
- poll_frequency=poll
- ).until(
- EC.element_to_be_clickable((By.ID, selector.removeprefix('#')
- )
- )
- )
+ WebDriverWait(self.driver, timeout=timeout).until(lambda driver: self.driver.execute_script("return jQuery.active == 0"))
+ is_clickable = lambda driver: (self.find(selector).is_displayed() and self.find(selector).is_enabled())
+ msg = 'An element matching "%s" should be clickable' % selector
+ Wait(self.driver, timeout=timeout).until(is_clickable, msg)
return self.find(selector)
+ def wait_until_finder_clickable(self, finder, timeout=Wait._TIMEOUT):
+ """ Wait until element is clickable """
+ WebDriverWait(self.driver, timeout=timeout).until(lambda driver: self.driver.execute_script("return jQuery.active == 0"))
+ is_clickable = lambda driver: (finder(driver).is_displayed() and finder(driver).is_enabled())
+ msg = 'A matching element never became be clickable'
+ Wait(self.driver, timeout=timeout).until(is_clickable, msg)
+ return finder(self.driver)
+
+ def wait_until_element_clickable(self, element, timeout=Wait._TIMEOUT):
+ """ Wait until element is clickable """
+ WebDriverWait(self.driver, timeout=timeout).until(lambda driver: self.driver.execute_script("return jQuery.active == 0"))
+ is_clickable = lambda driver: (element.is_displayed() and element.is_enabled())
+ scroll = lambda driver: ActionChains(driver).move_to_element(element).perform()
+ msg = 'A matching element never became be clickable'
+ Wait(self.driver, timeout=timeout).until(is_clickable, msg, scroll=scroll)
+ return
+
def wait_until_focused(self, selector):
""" Wait until element matching CSS selector has focus """
is_focused = \
diff --git a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
index b9356a03..9ab81fb1 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
@@ -200,6 +200,7 @@ class TestAllBuildsPage(SeleniumTestCase):
# should see a rebuild button for non-command-line builds
self.wait_until_visible('#allbuildstable tbody tr')
+ self.wait_until_visible('.rebuild-btn')
selector = 'div[data-latest-build-result="%s"] .rebuild-btn' % build1.id
run_again_button = self.find_all(selector)
self.assertEqual(len(run_again_button), 1,
@@ -224,7 +225,7 @@ class TestAllBuildsPage(SeleniumTestCase):
url = reverse('all-builds')
self.get(url)
- self.wait_until_visible('#allbuildstable', poll=3)
+ self.wait_until_visible('#allbuildstable')
# get the project name cells from the table
cells = self.find_all('#allbuildstable td[class="project"]')
@@ -257,7 +258,7 @@ class TestAllBuildsPage(SeleniumTestCase):
url = reverse('all-builds')
self.get(url)
- self.wait_until_visible('#allbuildstable', poll=3)
+ self.wait_until_visible('#allbuildstable')
# test recent builds area for successful build
element = self._get_build_time_element(build1)
@@ -452,7 +453,7 @@ class TestAllBuildsPage(SeleniumTestCase):
def test_show_rows(row_to_show, show_row_link):
# Check that we can show rows == row_to_show
show_row_link.select_by_value(str(row_to_show))
- self.wait_until_visible('#allbuildstable tbody tr', poll=3)
+ self.wait_until_visible('#allbuildstable tbody tr')
# check at least some rows are visible
self.assertTrue(
len(self.find_all('#allbuildstable tbody tr')) > 0
diff --git a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
index 9ed1901c..05e12892 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
@@ -81,7 +81,7 @@ class TestAllProjectsPage(SeleniumTestCase):
def _get_row_for_project(self, project_name):
""" Get the HTML row for a project, or None if not found """
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
rows = self.find_all('#projectstable tbody tr')
# find the row with a project name matching the one supplied
@@ -236,7 +236,7 @@ class TestAllProjectsPage(SeleniumTestCase):
self.get(url)
# Chseck search box is present and works
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
search_box = self.find('#search-input-projectstable')
self.assertTrue(search_box.is_displayed())
@@ -244,7 +244,7 @@ class TestAllProjectsPage(SeleniumTestCase):
search_box.send_keys('test project 10')
search_btn = self.find('#search-submit-projectstable')
search_btn.click()
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
rows = self.find_all('#projectstable tbody tr')
self.assertTrue(len(rows) == 1)
@@ -290,7 +290,7 @@ class TestAllProjectsPage(SeleniumTestCase):
)
url = reverse('all-projects')
self.get(url)
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
# Check edit column
edit_column = self.find('#edit-columns-button')
@@ -313,7 +313,7 @@ class TestAllProjectsPage(SeleniumTestCase):
def test_show_rows(row_to_show, show_row_link):
# Check that we can show rows == row_to_show
show_row_link.select_by_value(str(row_to_show))
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
# check at least some rows are visible
self.assertTrue(
len(self.find_all('#projectstable tbody tr')) > 0
@@ -321,7 +321,7 @@ class TestAllProjectsPage(SeleniumTestCase):
url = reverse('all-projects')
self.get(url)
- self.wait_until_visible('#projectstable tbody tr', poll=3)
+ self.wait_until_visible('#projectstable tbody tr')
show_rows = self.driver.find_elements(
By.XPATH,
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
index d838ce36..82367108 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
@@ -162,7 +162,7 @@ class TestBuildDashboardPage(SeleniumTestCase):
"""
url = reverse('builddashboard', args=(build.id,))
self.get(url)
- self.wait_until_visible('#global-nav', poll=3)
+ self.wait_until_visible('#global-nav')
def _get_build_dashboard_errors(self, build):
"""
diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py b/bitbake/lib/toaster/tests/browser/test_landing_page.py
index 8fe5fea4..210359d5 100644
--- a/bitbake/lib/toaster/tests/browser/test_landing_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_landing_page.py
@@ -34,6 +34,7 @@ class TestLandingPage(SeleniumTestCase):
def test_icon_info_visible_and_clickable(self):
""" Test that the information icon is visible and clickable """
self.get(reverse('landing'))
+ self.wait_until_visible('#toaster-version-info-sign')
info_sign = self.find('#toaster-version-info-sign')
# check that the info sign is visible
@@ -43,6 +44,7 @@ class TestLandingPage(SeleniumTestCase):
# and info modal is appearing when clicking on the info sign
info_sign.click() # click on the info sign make attribute 'aria-describedby' visible
info_model_id = info_sign.get_attribute('aria-describedby')
+ self.wait_until_visible(f'#{info_model_id}')
info_modal = self.find(f'#{info_model_id}')
self.assertTrue(info_modal.is_displayed())
self.assertTrue("Toaster version information" in info_modal.text)
@@ -50,6 +52,7 @@ class TestLandingPage(SeleniumTestCase):
def test_documentation_link_displayed(self):
""" Test that the documentation link is displayed """
self.get(reverse('landing'))
+ self.wait_until_visible('#navbar-docs')
documentation_link = self.find('#navbar-docs > a')
# check that the documentation link is visible
@@ -65,6 +68,7 @@ class TestLandingPage(SeleniumTestCase):
def test_openembedded_jumbotron_link_visible_and_clickable(self):
""" Test OpenEmbedded link jumbotron is visible and clickable: """
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check OpenEmbedded
@@ -76,6 +80,7 @@ class TestLandingPage(SeleniumTestCase):
def test_bitbake_jumbotron_link_visible_and_clickable(self):
""" Test BitBake link jumbotron is visible and clickable: """
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check BitBake
@@ -88,6 +93,7 @@ class TestLandingPage(SeleniumTestCase):
def test_yoctoproject_jumbotron_link_visible_and_clickable(self):
""" Test Yocto Project link jumbotron is visible and clickable: """
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check Yocto Project
@@ -101,6 +107,7 @@ class TestLandingPage(SeleniumTestCase):
if visible and clickable
"""
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check Big magenta button
@@ -119,6 +126,7 @@ class TestLandingPage(SeleniumTestCase):
Layer_Version.objects.create(layer=layer)
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check Big Blue button
@@ -132,6 +140,7 @@ class TestLandingPage(SeleniumTestCase):
def test_toaster_manual_link_visible_and_clickable(self):
""" Test Read the Toaster manual link jumbotron is visible and clickable: """
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check Read the Toaster manual
@@ -145,6 +154,7 @@ class TestLandingPage(SeleniumTestCase):
def test_contrib_to_toaster_link_visible_and_clickable(self):
""" Test Contribute to Toaster link jumbotron is visible and clickable: """
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
jumbotron = self.find('.jumbotron')
# check Contribute to Toaster
@@ -161,6 +171,7 @@ class TestLandingPage(SeleniumTestCase):
=> should see the landing page
"""
self.get(reverse('landing'))
+ self.wait_until_visible('.jumbotron')
self.assertTrue(self.LANDING_PAGE_TITLE in self.get_page_source())
def test_default_project_has_build(self):
@@ -193,6 +204,7 @@ class TestLandingPage(SeleniumTestCase):
user_project.save()
self.get(reverse('landing'))
+ self.wait_until_visible('#projectstable')
elements = self.find_all('#projectstable')
self.assertEqual(len(elements), 1, 'should redirect to projects')
@@ -213,7 +225,7 @@ class TestLandingPage(SeleniumTestCase):
self.get(reverse('landing'))
- self.wait_until_visible("#latest-builds", poll=3)
+ self.wait_until_visible("#latest-builds")
elements = self.find_all('#allbuildstable')
self.assertEqual(len(elements), 1, 'should redirect to builds')
content = self.get_page_source()
diff --git a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
index 5c29548b..476a8fa5 100644
--- a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
@@ -64,7 +64,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
args=(self.project.pk,
self.imported_layer_version.pk))
- def _edit_layerdetails(self):
+ def test_edit_layerdetails_page(self):
""" Edit all the editable fields for the layer refresh the page and
check that the new values exist"""
@@ -78,17 +78,20 @@ class TestLayerDetailsPage(SeleniumTestCase):
self.wait_until_visible("#layer-git-repo-url")
# Open every edit box
- for btn in self.find_all("dd .glyphicon-edit"):
- btn.click()
+ self.wait_until_element_clickable(self.find("#edit-layer-description"))
+ self.click("#edit-layer-description")
+ self.wait_until_element_clickable(self.find("#edit-layer-summary"))
+ self.click("#edit-layer-summary")
# Wait for the inputs to become visible after animation
- self.wait_until_visible("#layer-git input[type=text]")
- self.wait_until_visible("dd textarea")
+ self.wait_until_visible("#layer-git-repo-url")
self.wait_until_visible("dd .change-btn")
+ self.wait_until_element_clickable(self.find("#layer-description"))
+ self.wait_until_element_clickable(self.find("#layer-summary"))
# Edit each value
for inputs in self.find_all("#layer-git input[type=text]") + \
- self.find_all("dd textarea"):
+ [self.find("#layer-summary"), self.find("#layer-description")]:
# ignore the tt inputs (twitter typeahead input)
if "tt-" in inputs.get_attribute("class"):
continue
@@ -100,24 +103,19 @@ class TestLayerDetailsPage(SeleniumTestCase):
(self.initial_values, value))
# Make sure the input visible beofre sending keys
- self.wait_until_visible("#layer-git input[type=text]")
+ self.wait_until_element_clickable(inputs)
inputs.send_keys("-edited")
# Save the new values
for save_btn in self.find_all(".change-btn"):
save_btn.click()
- try:
- self.wait_until_visible("#save-changes-for-switch", poll=3)
- btn_save_chg_for_switch = self.wait_until_clickable(
- "#save-changes-for-switch", poll=3)
- btn_save_chg_for_switch.click()
- except ElementClickInterceptedException:
- self.skipTest(
- "save-changes-for-switch click intercepted. Element not visible or maybe covered by another element.")
- except TimeoutException:
- self.skipTest(
- "save-changes-for-switch is not clickable within the specified timeout.")
+ self.wait_until_visible("#save-changes-for-switch")
+ # Ensure scrolled into view
+ self.driver.execute_script('window.scrollTo({behavior: "instant", top: 0, left: 0})')
+ btn_save_chg_for_switch = self.wait_until_clickable(
+ "#save-changes-for-switch")
+ btn_save_chg_for_switch.click()
self.wait_until_visible("#edit-layer-source")
@@ -147,17 +145,10 @@ class TestLayerDetailsPage(SeleniumTestCase):
new_dir = "/home/test/my-meta-dir"
dir_input.send_keys(new_dir)
- try:
- self.wait_until_visible("#save-changes-for-switch", poll=3)
- btn_save_chg_for_switch = self.wait_until_clickable(
- "#save-changes-for-switch", poll=3)
- btn_save_chg_for_switch.click()
- except ElementClickInterceptedException:
- self.skipTest(
- "save-changes-for-switch click intercepted. Element not properly visible or maybe behind another element.")
- except TimeoutException:
- self.skipTest(
- "save-changes-for-switch is not clickable within the specified timeout.")
+ self.wait_until_visible("#save-changes-for-switch")
+ btn_save_chg_for_switch = self.wait_until_clickable(
+ "#save-changes-for-switch")
+ btn_save_chg_for_switch.click()
self.wait_until_visible("#edit-layer-source")
@@ -168,12 +159,6 @@ class TestLayerDetailsPage(SeleniumTestCase):
"Expected %s in the dir value for layer directory" %
new_dir)
- def test_edit_layerdetails_page(self):
- try:
- self._edit_layerdetails()
- except ElementClickInterceptedException:
- self.skipTest(
- "ElementClickInterceptedException occured. Element not visible or maybe covered by another element.")
def test_delete_layer(self):
""" Delete the layer """
@@ -211,6 +196,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
self.get(self.url)
# Add the layer
+ self.wait_until_clickable("#add-remove-layer-btn")
self.click("#add-remove-layer-btn")
notification = self.wait_until_visible("#change-notification-msg")
@@ -218,12 +204,17 @@ class TestLayerDetailsPage(SeleniumTestCase):
expected_text = "You have added 1 layer to your project: %s" % \
self.imported_layer_version.layer.name
- self.assertTrue(expected_text in notification.text,
+ self.assertIn(expected_text, notification.text,
"Expected notification text %s not found was "
" \"%s\" instead" %
(expected_text, notification.text))
+ hide_button = self.find('#hide-alert')
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
+
# Remove the layer
+ self.wait_until_clickable("#add-remove-layer-btn")
self.click("#add-remove-layer-btn")
notification = self.wait_until_visible("#change-notification-msg")
@@ -231,7 +222,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
expected_text = "You have removed 1 layer from your project: %s" % \
self.imported_layer_version.layer.name
- self.assertTrue(expected_text in notification.text,
+ self.assertIn(expected_text, notification.text,
"Expected notification text %s not found was "
" \"%s\" instead" %
(expected_text, notification.text))
diff --git a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
index 9f0b6397..bf0304db 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
@@ -90,7 +90,7 @@ class TestNewCustomImagePage(SeleniumTestCase):
"""
url = reverse('newcustomimage', args=(self.project.id,))
self.get(url)
- self.wait_until_visible('#global-nav', poll=3)
+ self.wait_until_visible('#global-nav')
self.click('button[data-recipe="%s"]' % self.recipe.id)
diff --git a/bitbake/lib/toaster/tests/browser/test_new_project_page.py b/bitbake/lib/toaster/tests/browser/test_new_project_page.py
index 458bb653..e50f236c 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_project_page.py
@@ -47,7 +47,7 @@ class TestNewProjectPage(SeleniumTestCase):
url = reverse('newproject')
self.get(url)
- self.wait_until_visible('#new-project-name', poll=3)
+ self.wait_until_visible('#new-project-name')
self.enter_text('#new-project-name', project_name)
select = Select(self.find('#projectversion'))
@@ -58,7 +58,7 @@ class TestNewProjectPage(SeleniumTestCase):
# We should get redirected to the new project's page with the
# notification at the top
element = self.wait_until_visible(
- '#project-created-notification', poll=3)
+ '#project-created-notification')
self.assertTrue(project_name in element.text,
"New project name not in new project notification")
@@ -79,7 +79,7 @@ class TestNewProjectPage(SeleniumTestCase):
url = reverse('newproject')
self.get(url)
- self.wait_until_visible('#new-project-name', poll=3)
+ self.wait_until_visible('#new-project-name')
self.enter_text('#new-project-name', project_name)
@@ -89,12 +89,10 @@ class TestNewProjectPage(SeleniumTestCase):
radio = self.driver.find_element(By.ID, 'type-new')
radio.click()
- self.click("#create-project-button")
-
- self.wait_until_present('#hint-error-project-name', poll=3)
+ self.wait_until_visible('#hint-error-project-name')
element = self.find('#hint-error-project-name')
- self.assertTrue(("Project names must be unique" in element.text),
+ self.assertIn("Project names must be unique", element.text,
"Did not find unique project name error message")
# Try and click it anyway, if it submits we'll have a new project in
diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py b/bitbake/lib/toaster/tests/builds/buildtest.py
index cacfccd4..e54d5613 100644
--- a/bitbake/lib/toaster/tests/builds/buildtest.py
+++ b/bitbake/lib/toaster/tests/builds/buildtest.py
@@ -128,7 +128,7 @@ class BuildTest(unittest.TestCase):
if os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"):
ProjectVariable.objects.get_or_create(
name="SSTATE_MIRRORS",
- value="file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH",
+ value="file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH",
project=project)
ProjectTarget.objects.create(project=project,
diff --git a/bitbake/lib/toaster/tests/functional/functional_helpers.py b/bitbake/lib/toaster/tests/functional/functional_helpers.py
index 7c20437d..9bcaac61 100644
--- a/bitbake/lib/toaster/tests/functional/functional_helpers.py
+++ b/bitbake/lib/toaster/tests/functional/functional_helpers.py
@@ -12,14 +12,19 @@ import logging
import subprocess
import signal
import re
+import requests
+from django.urls import reverse
from tests.browser.selenium_helpers_base import SeleniumTestCaseBase
from selenium.webdriver.common.by import By
+from selenium.webdriver.support.select import Select
from selenium.common.exceptions import NoSuchElementException
logger = logging.getLogger("toaster")
toaster_processes = []
+MASTER_PROJECT_ID = '4'
+
class SeleniumFunctionalTestCase(SeleniumTestCaseBase):
wait_toaster_time = 10
@@ -136,3 +141,86 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase):
except NoSuchElementException:
return False
return element
+
+ def create_new_project(
+ self,
+ project_name,
+ release,
+ release_title,
+ merge_toaster_settings,
+ ):
+ """ Create/Test new project using:
+ - Project Name: Any string
+ - Release: Any string
+ - Merge Toaster settings: True or False
+ """
+
+ # Obtain a CSRF token from a suitable URL
+ projs = requests.get(self.live_server_url + reverse('newproject'))
+ csrftoken = projs.cookies.get('csrftoken')
+
+ # Use the projects typeahead to find out if the project already exists
+ req = requests.get(self.live_server_url + reverse('xhr_projectstypeahead'), {'search': project_name, 'format' : 'json'})
+ data = req.json()
+ # Delete any existing projects
+ for result in data['results']:
+ del_url = reverse('xhr_project', args=(result['id'],))
+ del_response = requests.delete(self.live_server_url + del_url, cookies={'csrftoken': csrftoken}, headers={'X-CSRFToken': csrftoken})
+ self.assertEqual(del_response.status_code, 200)
+
+ self.get(reverse('newproject'))
+ self.wait_until_visible('#new-project-name')
+ self.driver.find_element(By.ID,
+ "new-project-name").send_keys(project_name)
+
+ select = Select(self.find('#projectversion'))
+ select.select_by_value(release)
+
+ # check merge toaster settings
+ checkbox = self.find('.checkbox-mergeattr')
+ if merge_toaster_settings:
+ if not checkbox.is_selected():
+ checkbox.click()
+ else:
+ if checkbox.is_selected():
+ checkbox.click()
+
+ self.wait_until_clickable('#create-project-button')
+
+ self.driver.find_element(By.ID, "create-project-button").click()
+
+ element = self.wait_until_visible('#project-created-notification')
+ self.assertTrue(
+ self.element_exists('#project-created-notification'),
+ f"Project:{project_name} creation notification not shown"
+ )
+ self.assertTrue(
+ project_name in element.text,
+ f"New project name:{project_name} not in new project notification"
+ )
+
+ # Use the projects typeahead again to check the project now exists
+ req = requests.get(self.live_server_url + reverse('xhr_projectstypeahead'), {'search': project_name, 'format' : 'json'})
+ data = req.json()
+ self.assertGreater(len(data['results']), 0, f"New project:{project_name} not found in database")
+
+ project_id = data['results'][0]['id']
+
+ self.wait_until_visible('#project-release-title')
+
+ # check release
+ if release_title is not None:
+ self.assertTrue(re.search(
+ release_title,
+ self.driver.find_element(By.XPATH,
+ "//span[@id='project-release-title']"
+ ).text),
+ 'The project release is not defined')
+
+ return project_id
+
+ def load_projects_page_helper(self):
+ self.wait_until_present('#projectstable')
+ # Need to wait for some data in the table too
+ self.wait_until_present('td[class="updated"]')
+
diff --git a/bitbake/lib/toaster/tests/functional/test_create_new_project.py b/bitbake/lib/toaster/tests/functional/test_create_new_project.py
index 94d90459..294b6bc9 100644
--- a/bitbake/lib/toaster/tests/functional/test_create_new_project.py
+++ b/bitbake/lib/toaster/tests/functional/test_create_new_project.py
@@ -11,125 +11,84 @@ import pytest
from django.urls import reverse
from selenium.webdriver.support.select import Select
from tests.functional.functional_helpers import SeleniumFunctionalTestCase
-from orm.models import Project
from selenium.webdriver.common.by import By
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
class TestCreateNewProject(SeleniumFunctionalTestCase):
- def _create_test_new_project(
- self,
- project_name,
- release,
- release_title,
- merge_toaster_settings,
- ):
- """ Create/Test new project using:
+ def test_create_new_project_blacksail(self):
+ """ Test create new project using:
- Project Name: Any string
- - Release: Any string
- - Merge Toaster settings: True or False
+ - Release: Yocto Project 6.1 "Blacksail" (option value: 1)
+ - Merge Toaster settings: True
"""
- self.get(reverse('newproject'))
- self.wait_until_visible('#new-project-name', poll=3)
- self.driver.find_element(By.ID,
- "new-project-name").send_keys(project_name)
-
- select = Select(self.find('#projectversion'))
- select.select_by_value(release)
-
- # check merge toaster settings
- checkbox = self.find('.checkbox-mergeattr')
- if merge_toaster_settings:
- if not checkbox.is_selected():
- checkbox.click()
- else:
- if checkbox.is_selected():
- checkbox.click()
-
- self.driver.find_element(By.ID, "create-project-button").click()
-
- element = self.wait_until_visible('#project-created-notification', poll=3)
- self.assertTrue(
- self.element_exists('#project-created-notification'),
- f"Project:{project_name} creation notification not shown"
- )
- self.assertTrue(
- project_name in element.text,
- f"New project name:{project_name} not in new project notification"
- )
- self.assertTrue(
- Project.objects.filter(name=project_name).count(),
- f"New project:{project_name} not found in database"
- )
-
- # check release
- self.assertTrue(re.search(
+ release = '1'
+ release_title = 'Yocto Project 6.1 "Blacksail"'
+ project_name = 'projectblacksail'
+ self.create_new_project(
+ project_name,
+ release,
release_title,
- self.driver.find_element(By.XPATH,
- "//span[@id='project-release-title']"
- ).text),
- 'The project release is not defined')
+ True,
+ )
- def test_create_new_project_master(self):
+ def test_create_new_project_wrynose(self):
""" Test create new project using:
- Project Name: Any string
- - Release: Yocto Project master (option value: 3)
- - Merge Toaster settings: False
+ - Release: Yocto Project 6.0 "Wrynose" (option value: 2)
+ - Merge Toaster settings: True
"""
- release = '3'
- release_title = 'Yocto Project master'
- project_name = 'projectmaster'
- self._create_test_new_project(
+ release = '2'
+ release_title = 'Yocto Project 6.0 "Wrynose"'
+ project_name = 'projectwrynose'
+ self.create_new_project(
project_name,
release,
release_title,
- False,
+ True,
)
- def test_create_new_project_kirkstone(self):
+ def test_create_new_project_local(self):
""" Test create new project using:
- Project Name: Any string
- - Release: Yocto Project 4.0 "Kirkstone" (option value: 1)
+ - Release: Yocto Project "Local" (option value: 3)
- Merge Toaster settings: True
"""
- release = '1'
- release_title = 'Yocto Project 4.0 "Kirkstone"'
- project_name = 'projectkirkstone'
- self._create_test_new_project(
+ release = '3'
+ release_title = 'Local Yocto Project'
+ project_name = 'projectlocal'
+ self.create_new_project(
project_name,
release,
release_title,
True,
)
- def test_create_new_project_dunfell(self):
+ def test_create_new_project_master(self):
""" Test create new project using:
- Project Name: Any string
- - Release: Yocto Project 3.1 "Dunfell" (option value: 5)
+ - Release: Yocto Project "Master" (option value: 4)
- Merge Toaster settings: False
"""
- release = '5'
- release_title = 'Yocto Project 3.1 "Dunfell"'
- project_name = 'projectdunfell'
- self._create_test_new_project(
+ release = '4'
+ release_title = 'Yocto Project master'
+ project_name = 'projectmaster'
+ self.create_new_project(
project_name,
release,
release_title,
False,
)
- def test_create_new_project_local(self):
+ def test_create_new_project_scarthgap(self):
""" Test create new project using:
- Project Name: Any string
- - Release: Local Yocto Project (option value: 2)
+ - Release: Yocto Project 5.0 "Scarthgap" (option value: 5)
- Merge Toaster settings: True
"""
- release = '2'
- release_title = 'Local Yocto Project'
- project_name = 'projectlocal'
- self._create_test_new_project(
+ release = '5'
+ release_title = 'Yocto Project 5.0 "Scarthgap"'
+ project_name = 'projectscarthgap'
+ self.create_new_project(
project_name,
release,
release_title,
@@ -172,8 +131,10 @@ class TestCreateNewProject(SeleniumFunctionalTestCase):
"import-project-dir").send_keys(wrong_path)
self.driver.find_element(By.ID, "create-project-button").click()
+ self.wait_until_visible('.alert-danger')
+
# check error message
self.assertTrue(self.element_exists('.alert-danger'),
- 'Allert message not shown')
+ 'Alert message not shown')
self.assertTrue(wrong_path in self.find('.alert-danger').text,
"Wrong path not in alert message")
diff --git a/bitbake/lib/toaster/tests/functional/test_functional_basic.py b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
index e4070fbb..71999ccf 100644
--- a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
+++ b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
@@ -10,152 +10,139 @@
import re
from django.urls import reverse
import pytest
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase, MASTER_PROJECT_ID
from orm.models import Project
from selenium.webdriver.common.by import By
from tests.functional.utils import get_projectId_from_url
-@pytest.mark.django_db
-@pytest.mark.order("second_to_last")
class FuntionalTestBasic(SeleniumFunctionalTestCase):
"""Basic functional tests for Toaster"""
project_id = None
+ project_url = None
def setUp(self):
super(FuntionalTestBasic, self).setUp()
if not FuntionalTestBasic.project_id:
- self._create_slenium_project()
- current_url = self.driver.current_url
- FuntionalTestBasic.project_id = get_projectId_from_url(current_url)
-
-# testcase (1514)
- def _create_slenium_project(self):
- project_name = 'selenium-project'
- self.get(reverse('newproject'))
- self.wait_until_visible('#new-project-name', poll=3)
- self.driver.find_element(By.ID, "new-project-name").send_keys(project_name)
- self.driver.find_element(By.ID, 'projectversion').click()
- self.driver.find_element(By.ID, "create-project-button").click()
- element = self.wait_until_visible('#project-created-notification', poll=10)
- self.assertTrue(self.element_exists('#project-created-notification'),'Project creation notification not shown')
- self.assertTrue(project_name in element.text,
- "New project name not in new project notification")
- self.assertTrue(Project.objects.filter(name=project_name).count(),
- "New project not found in database")
- return Project.objects.last().id
+ FuntionalTestBasic.project_id = self.create_new_project('selenium-project', MASTER_PROJECT_ID, None, False)
# testcase (1515)
def test_verify_left_bar_menu(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_present('#config-nav')
self.assertTrue(self.element_exists('#config-nav'),'Configuration Tab does not exist')
project_URL=self.get_URL()
self.driver.find_element(By.XPATH, '//a[@href="'+project_URL+'"]').click()
- self.wait_until_present('#config-nav', poll=10)
try:
+ self.wait_until_present('#config-nav')
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'customimages/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Custom images",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'Custom images information is not loading properly')
+ self.wait_until_present('#filter-modal-customimagestable')
except:
self.fail(msg='No Custom images tab available')
+ self.assertTrue(re.search("Custom images",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'Custom images information is not loading properly')
try:
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'images/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Compatible image recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible image recipes information is not loading properly')
+ self.wait_until_present('#filter-modal-imagerecipestable')
except:
self.fail(msg='No Compatible image tab available')
+ self.assertTrue(re.search("Compatible image recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible image recipes information is not loading properly')
try:
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'softwarerecipes/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Compatible software recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible software recipe information is not loading properly')
+ self.wait_until_present('#filter-modal-softwarerecipestable')
except:
self.fail(msg='No Compatible software recipe tab available')
+ self.assertTrue(re.search("Compatible software recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible software recipe information is not loading properly')
try:
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'machines/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Compatible machines",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible machine information is not loading properly')
+ self.wait_until_present('#filter-modal-machinestable')
except:
self.fail(msg='No Compatible machines tab available')
+ self.assertTrue(re.search("Compatible machines",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible machine information is not loading properly')
try:
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'layers/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Compatible layers",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible layer information is not loading properly')
+ self.wait_until_present('#filter-modal-layerstable')
except:
self.fail(msg='No Compatible layers tab available')
+ self.assertTrue(re.search("Compatible layers",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Compatible layer information is not loading properly')
try:
self.driver.find_element(By.XPATH, "//*[@id='config-nav']/ul/li/a[@href="+'"'+project_URL+'configuration"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Bitbake variables",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Bitbake variables information is not loading properly')
+ self.wait_until_present('#configvar-list')
except:
self.fail(msg='No Bitbake variables tab available')
+ self.assertTrue(re.search("Bitbake variables",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Bitbake variables information is not loading properly')
# testcase (1516)
def test_review_configuration_information(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
project_URL=self.get_URL()
- self.wait_until_present('#config-nav', poll=10)
+
+ # Machine section of page
+ self.wait_until_visible('#machine-section')
+ self.assertTrue(self.element_exists('#machine-section'),'Machine section for the project configuration page does not exist')
+ self.assertTrue(re.search("qemux86-64",self.driver.find_element(By.XPATH, "//span[@id='project-machine-name']").text),'The machine type is not assigned')
try:
- self.assertTrue(self.element_exists('#machine-section'),'Machine section for the project configuration page does not exist')
- self.assertTrue(re.search("qemux86-64",self.driver.find_element(By.XPATH, "//span[@id='project-machine-name']").text),'The machine type is not assigned')
self.driver.find_element(By.XPATH, "//span[@id='change-machine-toggle']").click()
- self.wait_until_visible('#select-machine-form', poll=10)
- self.wait_until_visible('#cancel-machine-change', poll=10)
+ self.wait_until_visible('#select-machine-form')
+ self.wait_until_visible('#cancel-machine-change')
self.driver.find_element(By.XPATH, "//form[@id='select-machine-form']/a[@id='cancel-machine-change']").click()
except:
self.fail(msg='The machine information is wrong in the configuration page')
+ # Most built recipes section
+ self.wait_until_visible('#no-most-built')
try:
self.driver.find_element(By.ID, 'no-most-built')
except:
self.fail(msg='No Most built information in project detail page')
- try:
- self.assertTrue(re.search("Yocto Project master",self.driver.find_element(By.XPATH, "//span[@id='project-release-title']").text),'The project release is not defined')
- except:
- self.fail(msg='No project release title information in project detail page')
+ # Project Release title
+ self.assertTrue(re.search("Yocto Project master",self.driver.find_element(By.XPATH, "//span[@id='project-release-title']").text), 'The project release is not defined in the project detail page')
+ # List of layers in project
+ self.wait_until_visible('#layer-container')
+ self.driver.find_element(By.XPATH, "//div[@id='layer-container']")
+ self.assertTrue(re.search("3",self.driver.find_element(By.ID, "project-layers-count").text),'There should be 3 layers listed in the layer count')
try:
- self.driver.find_element(By.XPATH, "//div[@id='layer-container']")
- self.assertTrue(re.search("3",self.driver.find_element(By.ID, "project-layers-count").text),'There should be 3 layers listed in the layer count')
layer_list = self.driver.find_element(By.ID, "layers-in-project-list")
layers = layer_list.find_elements(By.TAG_NAME, "li")
- for layer in layers:
- if re.match ("openembedded-core",layer.text):
- print ("openembedded-core layer is a default layer in the project configuration")
- elif re.match ("meta-poky",layer.text):
- print ("meta-poky layer is a default layer in the project configuration")
- elif re.match ("meta-yocto-bsp",layer.text):
- print ("meta-yocto-bsp is a default layer in the project configuratoin")
- else:
- self.fail(msg='default layers are missing from the project configuration')
except:
self.fail(msg='No Layer information in project detail page')
+ for layer in layers:
+ if re.match ("openembedded-core", layer.text):
+ print ("openembedded-core layer is a default layer in the project configuration")
+ elif re.match ("meta-poky", layer.text):
+ print ("meta-poky layer is a default layer in the project configuration")
+ elif re.match ("meta-yocto-bsp", layer.text):
+ print ("meta-yocto-bsp is a default layer in the project configuratoin")
+ else:
+ self.fail(msg='default layers are missing from the project configuration')
+
# testcase (1517)
def test_verify_machine_information(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_visible('#machine-section')
+ self.assertTrue(self.element_exists('#machine-section'),'Machine section for the project configuration page does not exist')
+ self.wait_until_visible('#project-machine-name')
+ self.assertTrue(re.search("qemux86-64",self.driver.find_element(By.ID, "project-machine-name").text),'The machine type is not assigned')
try:
- self.assertTrue(self.element_exists('#machine-section'),'Machine section for the project configuration page does not exist')
- self.assertTrue(re.search("qemux86-64",self.driver.find_element(By.ID, "project-machine-name").text),'The machine type is not assigned')
self.driver.find_element(By.ID, "change-machine-toggle").click()
- self.wait_until_visible('#select-machine-form', poll=10)
- self.wait_until_visible('#cancel-machine-change', poll=10)
+ self.wait_until_visible('#select-machine-form')
+ self.wait_until_visible('#cancel-machine-change')
self.driver.find_element(By.ID, "cancel-machine-change").click()
except:
self.fail(msg='The machine information is wrong in the configuration page')
@@ -163,83 +150,95 @@ class FuntionalTestBasic(SeleniumFunctionalTestCase):
# testcase (1518)
def test_verify_most_built_recipes_information(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_present('#config-nav')
project_URL=self.get_URL()
+
+ self.wait_until_visible('#no-most-built')
+ self.assertTrue(re.search("You haven't built any recipes yet",self.driver.find_element(By.ID, "no-most-built").text),'Default message of no builds is not present')
try:
- self.assertTrue(re.search("You haven't built any recipes yet",self.driver.find_element(By.ID, "no-most-built").text),'Default message of no builds is not present')
self.driver.find_element(By.XPATH, "//div[@id='no-most-built']/p/a[@href="+'"'+project_URL+'images/"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
- self.assertTrue(re.search("Compatible image recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Choose a recipe to build link is not working properly')
except:
self.fail(msg='No Most built information in project detail page')
+ self.wait_until_visible('#config-nav')
+ self.assertTrue(re.search("Compatible image recipes",self.driver.find_element(By.XPATH, "//div[@class='col-md-10']").text),'The Choose a recipe to build link is not working properly')
# testcase (1519)
def test_verify_project_release_information(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
-
- try:
- self.assertTrue(re.search("Yocto Project master",self.driver.find_element(By.ID, "project-release-title").text),'The project release is not defined')
- except:
- self.fail(msg='No project release title information in project detail page')
+ self.wait_until_visible('#project-release-title')
+ self.assertTrue(re.search("Yocto Project master",self.driver.find_element(By.ID, "project-release-title").text), 'No project release title information in project detail page')
# testcase (1520)
def test_verify_layer_information(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_present('#config-nav')
project_URL=self.get_URL()
+ self.wait_until_visible('#layer-container')
+ self.driver.find_element(By.XPATH, "//div[@id='layer-container']")
+ self.wait_until_visible('#project-layers-count')
+ self.assertTrue(re.search("3",self.driver.find_element(By.ID, "project-layers-count").text),'There should be 3 layers listed in the layer count')
+
try:
- self.driver.find_element(By.XPATH, "//div[@id='layer-container']")
- self.assertTrue(re.search("3",self.driver.find_element(By.ID, "project-layers-count").text),'There should be 3 layers listed in the layer count')
layer_list = self.driver.find_element(By.ID, "layers-in-project-list")
layers = layer_list.find_elements(By.TAG_NAME, "li")
+ except:
+ self.fail(msg='No Layer information in project detail page')
- for layer in layers:
- if re.match ("openembedded-core",layer.text):
- print ("openembedded-core layer is a default layer in the project configuration")
- elif re.match ("meta-poky",layer.text):
- print ("meta-poky layer is a default layer in the project configuration")
- elif re.match ("meta-yocto-bsp",layer.text):
- print ("meta-yocto-bsp is a default layer in the project configuratoin")
- else:
- self.fail(msg='default layers are missing from the project configuration')
+ for layer in layers:
+ if re.match ("openembedded-core",layer.text):
+ print ("openembedded-core layer is a default layer in the project configuration")
+ elif re.match ("meta-poky",layer.text):
+ print ("meta-poky layer is a default layer in the project configuration")
+ elif re.match ("meta-yocto-bsp",layer.text):
+ print ("meta-yocto-bsp is a default layer in the project configuratoin")
+ else:
+ self.fail(msg='default layers are missing from the project configuration')
+ try:
self.driver.find_element(By.XPATH, "//input[@id='layer-add-input']")
self.driver.find_element(By.XPATH, "//button[@id='add-layer-btn']")
self.driver.find_element(By.XPATH, "//div[@id='layer-container']/form[@class='form-inline']/p/a[@id='view-compatible-layers']")
self.driver.find_element(By.XPATH, "//div[@id='layer-container']/form[@class='form-inline']/p/a[@href="+'"'+project_URL+'importlayer"'+"]")
except:
- self.fail(msg='No Layer information in project detail page')
+ self.fail(msg='Layer configuration controls missing')
# testcase (1521)
def test_verify_project_detail_links(self):
self.get(reverse('all-projects'))
- self.wait_until_present('#projectstable', poll=10)
+ self.load_projects_page_helper()
self.find_element_by_link_text_in_table('projectstable', 'selenium-project').click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_present('#config-nav')
project_URL=self.get_URL()
self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li[@id='topbar-configuration-tab']/a[@href="+'"'+project_URL+'"'+"]").click()
- self.wait_until_present('#config-nav', poll=10)
+ self.wait_until_visible('#topbar-configuration-tab')
self.assertTrue(re.search("Configuration",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li[@id='topbar-configuration-tab']/a[@href="+'"'+project_URL+'"'+"]").text), 'Configuration tab in project topbar is misspelled')
try:
self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'builds/"'+"]").click()
- self.wait_until_visible('#project-topbar', poll=10)
- self.assertTrue(re.search("Builds",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'builds/"'+"]").text), 'Builds tab in project topbar is misspelled')
+ except:
+ self.fail(msg='Builds tab information is not present')
+
+ self.wait_until_visible('#project-topbar')
+ self.assertTrue(re.search("Builds",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'builds/"'+"]").text), 'Builds tab in project topbar is misspelled')
+ try:
self.driver.find_element(By.XPATH, "//div[@id='empty-state-projectbuildstable']")
except:
self.fail(msg='Builds tab information is not present')
try:
self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'importlayer"'+"]").click()
- self.wait_until_visible('#project-topbar', poll=10)
- self.assertTrue(re.search("Import layer",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'importlayer"'+"]").text), 'Import layer tab in project topbar is misspelled')
+ except:
+ self.fail(msg='Import layer tab not loading properly')
+
+ self.wait_until_visible('#project-topbar')
+ self.assertTrue(re.search("Import layer",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'importlayer"'+"]").text), 'Import layer tab in project topbar is misspelled')
+ try:
self.driver.find_element(By.XPATH, "//fieldset[@id='repo-select']")
self.driver.find_element(By.XPATH, "//fieldset[@id='git-repo']")
except:
@@ -247,11 +246,12 @@ class FuntionalTestBasic(SeleniumFunctionalTestCase):
try:
self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'newcustomimage/"'+"]").click()
- self.wait_until_visible('#project-topbar', poll=10)
- self.assertTrue(re.search("New custom image",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'newcustomimage/"'+"]").text), 'New custom image tab in project topbar is misspelled')
- self.assertTrue(re.search("Select the image recipe you want to customise",self.driver.find_element(By.XPATH, "//div[@class='col-md-12']/h2").text),'The new custom image tab is not loading correctly')
except:
self.fail(msg='New custom image tab not loading properly')
+ self.wait_until_visible('#project-topbar')
+ self.assertTrue(re.search("New custom image",self.driver.find_element(By.XPATH, "//div[@id='project-topbar']/ul[@class='nav nav-tabs']/li/a[@href="+'"'+project_URL+'newcustomimage/"'+"]").text), 'New custom image tab in project topbar is misspelled')
+ self.assertTrue(re.search("Select the image recipe you want to customise",self.driver.find_element(By.XPATH, "//div[@class='col-md-12']/h2").text),'The new custom image tab is not loading correctly')
+
diff --git a/bitbake/lib/toaster/tests/functional/test_project_config.py b/bitbake/lib/toaster/tests/functional/test_project_config.py
index dbee36aa..8c8450cf 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_config.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_config.py
@@ -7,20 +7,16 @@
#
import string
-import random
import pytest
from django.urls import reverse
from selenium.webdriver import Keys
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import TimeoutException
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase, MASTER_PROJECT_ID
from selenium.webdriver.common.by import By
from .utils import get_projectId_from_url
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
class TestProjectConfig(SeleniumFunctionalTestCase):
project_id = None
PROJECT_NAME = 'TestProjectConfig'
@@ -28,42 +24,6 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
INVALID_PATH_CHAR_TEXT = 'The directory path cannot include spaces or ' \
'any of these characters'
- def _create_project(self, project_name):
- """ Create/Test new project using:
- - Project Name: Any string
- - Release: Any string
- - Merge Toaster settings: True or False
- """
- self.get(reverse('newproject'))
- self.wait_until_visible('#new-project-name', poll=2)
- self.find("#new-project-name").send_keys(project_name)
- select = Select(self.find("#projectversion"))
- select.select_by_value('3')
-
- # check merge toaster settings
- checkbox = self.find('.checkbox-mergeattr')
- if not checkbox.is_selected():
- checkbox.click()
-
- if self.PROJECT_NAME != 'TestProjectConfig':
- # Reset project name if it's not the default one
- self.PROJECT_NAME = 'TestProjectConfig'
-
- self.find("#create-project-button").click()
-
- try:
- self.wait_until_visible('#hint-error-project-name', poll=2)
- url = reverse('project', args=(TestProjectConfig.project_id, ))
- self.get(url)
- self.wait_until_visible('#config-nav', poll=3)
- except TimeoutException:
- self.wait_until_visible('#config-nav', poll=3)
-
- def _random_string(self, length):
- return ''.join(
- random.choice(string.ascii_letters) for _ in range(length)
- )
-
def _get_config_nav_item(self, index):
config_nav = self.find('#config-nav')
return config_nav.find_elements(By.TAG_NAME, 'li')[index]
@@ -72,16 +32,14 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
""" Navigate to project BitBake variables page """
# check if the menu is displayed
if TestProjectConfig.project_id is None:
- self._create_project(project_name=self._random_string(10))
- current_url = self.driver.current_url
- TestProjectConfig.project_id = get_projectId_from_url(current_url)
- else:
- url = reverse('projectconf', args=(TestProjectConfig.project_id,))
- self.get(url)
- self.wait_until_visible('#config-nav', poll=3)
+ TestProjectConfig.project_id = self.create_new_project(self.PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+
+ url = reverse('projectconf', args=(TestProjectConfig.project_id,))
+ self.get(url)
+ self.wait_until_visible('#config-nav')
bbv_page_link = self._get_config_nav_item(9)
bbv_page_link.click()
- self.wait_until_visible('#config-nav', poll=3)
+ self.wait_until_visible('#config-nav')
def test_no_underscore_iamgefs_type(self):
"""
@@ -90,13 +48,13 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self._navigate_bbv_page()
imagefs_type = "foo_bar"
- self.wait_until_visible('#change-image_fstypes-icon', poll=2)
+ self.wait_until_visible('#change-image_fstypes-icon')
self.click('#change-image_fstypes-icon')
self.enter_text('#new-imagefs_types', imagefs_type)
- element = self.wait_until_visible('#hintError-image-fs_type', poll=2)
+ element = self.wait_until_visible('#hintError-image-fs_type')
self.assertTrue(("A valid image type cannot include underscores" in element.text),
"Did not find underscore error message")
@@ -110,7 +68,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
imagefs_type = "btrfs"
- self.wait_until_visible('#change-image_fstypes-icon', poll=2)
+ self.wait_until_visible('#change-image_fstypes-icon')
self.click('#change-image_fstypes-icon')
@@ -129,22 +87,20 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
"""
self._navigate_bbv_page()
- self.wait_until_visible('#change-image_fstypes-icon', poll=2)
-
+ self.wait_until_visible('#change-image_fstypes-icon')
self.click('#change-image_fstypes-icon')
checkboxes_selector = '.fs-checkbox-fstypes'
- self.wait_until_visible(checkboxes_selector, poll=2)
+ self.wait_until_visible(checkboxes_selector)
checkboxes = self.find_all(checkboxes_selector)
for checkbox in checkboxes:
if checkbox.get_attribute("value") == "cpio":
checkbox.click()
+ self.wait_until_visible('#new-imagefs_types')
element = self.driver.find_element(By.ID, 'new-imagefs_types')
- self.wait_until_visible('#new-imagefs_types', poll=2)
-
self.assertTrue(("cpio" in element.get_attribute('value'),
"Imagefs not added into the textbox"))
checkbox.click()
@@ -160,20 +116,19 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
# activate the input to edit download dir
try:
- change_dl_dir_btn = self.wait_until_visible('#change-dl_dir-icon', poll=2)
+ change_dl_dir_btn = self.wait_until_visible('#change-dl_dir-icon')
except TimeoutException:
# If download dir is not displayed, test is skipped
change_dl_dir_btn = None
if change_dl_dir_btn:
- change_dl_dir_btn = self.wait_until_visible('#change-dl_dir-icon', poll=2)
change_dl_dir_btn.click()
# downloads dir path doesn't start with / or ${...}
- input_field = self.wait_until_visible('#new-dl_dir', poll=2)
+ input_field = self.wait_until_visible('#new-dl_dir')
input_field.clear()
self.enter_text('#new-dl_dir', 'home/foo')
- element = self.wait_until_visible('#hintError-initialChar-dl_dir', poll=2)
+ element = self.wait_until_visible('#hintError-initialChar-dl_dir')
msg = 'downloads directory path starts with invalid character but ' \
'treated as valid'
@@ -183,7 +138,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self.driver.find_element(By.ID, 'new-dl_dir').clear()
self.enter_text('#new-dl_dir', '/foo/bar a')
- element = self.wait_until_visible('#hintError-dl_dir', poll=2)
+ element = self.wait_until_visible('#hintError-dl_dir')
msg = 'downloads directory path characters invalid but treated as valid'
self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text), msg)
@@ -191,7 +146,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self.driver.find_element(By.ID,'new-dl_dir').clear()
self.enter_text('#new-dl_dir', '${TOPDIR}/down foo')
- element = self.wait_until_visible('#hintError-dl_dir', poll=2)
+ element = self.wait_until_visible('#hintError-dl_dir')
msg = 'downloads directory path characters invalid but treated as valid'
self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text), msg)
@@ -219,10 +174,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self._navigate_bbv_page()
try:
- btn_chg_sstate_dir = self.wait_until_visible(
- '#change-sstate_dir-icon',
- poll=2
- )
+ btn_chg_sstate_dir = self.wait_until_visible('#change-sstate_dir-icon')
self.click('#change-sstate_dir-icon')
except TimeoutException:
# If sstate_dir is not displayed, test is skipped
@@ -230,10 +182,10 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
if btn_chg_sstate_dir: # Skip continuation if sstate_dir is not displayed
# path doesn't start with / or ${...}
- input_field = self.wait_until_visible('#new-sstate_dir', poll=2)
+ input_field = self.wait_until_visible('#new-sstate_dir')
input_field.clear()
self.enter_text('#new-sstate_dir', 'home/foo')
- element = self.wait_until_visible('#hintError-initialChar-sstate_dir', poll=2)
+ element = self.wait_until_visible('#hintError-initialChar-sstate_dir')
msg = 'sstate directory path starts with invalid character but ' \
'treated as valid'
@@ -243,7 +195,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self.driver.find_element(By.ID, 'new-sstate_dir').clear()
self.enter_text('#new-sstate_dir', '/foo/bar a')
- element = self.wait_until_visible('#hintError-sstate_dir', poll=2)
+ element = self.wait_until_visible('#hintError-sstate_dir')
msg = 'sstate directory path characters invalid but treated as valid'
self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text), msg)
@@ -251,7 +203,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
self.driver.find_element(By.ID,'new-sstate_dir').clear()
self.enter_text('#new-sstate_dir', '${TOPDIR}/down foo')
- element = self.wait_until_visible('#hintError-sstate_dir', poll=2)
+ element = self.wait_until_visible('#hintError-sstate_dir')
msg = 'sstate directory path characters invalid but treated as valid'
self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text), msg)
@@ -275,13 +227,14 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
var_name, field, btn_id, input_id, value, save_btn, *_ = kwargs.values()
""" Change bitbake variable value """
self._navigate_bbv_page()
- self.wait_until_visible(f'#{btn_id}', poll=2)
+ self.wait_until_visible(f'#{btn_id}')
if kwargs.get('new_variable'):
self.find(f"#{btn_id}").clear()
self.enter_text(f"#{btn_id}", f"{var_name}")
else:
self.click(f'#{btn_id}')
- self.wait_until_visible(f'#{input_id}', poll=2)
+
+ self.wait_until_visible(f'#{input_id}')
if kwargs.get('is_select'):
select = Select(self.find(f'#{input_id}'))
diff --git a/bitbake/lib/toaster/tests/functional/test_project_page.py b/bitbake/lib/toaster/tests/functional/test_project_page.py
index adbe3587..8f67c420 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_page.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_page.py
@@ -7,8 +7,8 @@
#
import os
-import random
import string
+import time
from unittest import skip
import pytest
from django.urls import reverse
@@ -16,64 +16,23 @@ from django.utils import timezone
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import TimeoutException
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase, MASTER_PROJECT_ID
from orm.models import Build, Project, Target
from selenium.webdriver.common.by import By
from .utils import get_projectId_from_url, wait_until_build, wait_until_build_cancelled
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
-class TestProjectPage(SeleniumFunctionalTestCase):
+class TestProjectPageBase(SeleniumFunctionalTestCase):
project_id = None
PROJECT_NAME = 'TestProjectPage'
- def _create_project(self, project_name):
- """ Create/Test new project using:
- - Project Name: Any string
- - Release: Any string
- - Merge Toaster settings: True or False
- """
- self.get(reverse('newproject'))
- self.wait_until_visible('#new-project-name')
- self.find("#new-project-name").send_keys(project_name)
- select = Select(self.find("#projectversion"))
- select.select_by_value('3')
-
- # check merge toaster settings
- checkbox = self.find('.checkbox-mergeattr')
- if not checkbox.is_selected():
- checkbox.click()
-
- if self.PROJECT_NAME != 'TestProjectPage':
- # Reset project name if it's not the default one
- self.PROJECT_NAME = 'TestProjectPage'
-
- self.find("#create-project-button").click()
-
- try:
- self.wait_until_visible('#hint-error-project-name')
- url = reverse('project', args=(TestProjectPage.project_id, ))
- self.get(url)
- self.wait_until_visible('#config-nav', poll=3)
- except TimeoutException:
- self.wait_until_visible('#config-nav', poll=3)
-
- def _random_string(self, length):
- return ''.join(
- random.choice(string.ascii_letters) for _ in range(length)
- )
-
def _navigate_to_project_page(self):
# Navigate to project page
- if TestProjectPage.project_id is None:
- self._create_project(project_name=self._random_string(10))
- current_url = self.driver.current_url
- TestProjectPage.project_id = get_projectId_from_url(current_url)
- else:
- url = reverse('project', args=(TestProjectPage.project_id,))
- self.get(url)
+ if TestProjectPageBase.project_id is None:
+ TestProjectPageBase.project_id = self.create_new_project(self.PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+
+ url = reverse('project', args=(TestProjectPageBase.project_id,))
+ self.get(url)
self.wait_until_visible('#config-nav')
def _get_create_builds(self, **kwargs):
@@ -81,14 +40,14 @@ class TestProjectPage(SeleniumFunctionalTestCase):
# parameters for builds to associate with the projects
now = timezone.now()
self.project1_build_success = {
- 'project': Project.objects.get(id=TestProjectPage.project_id),
+ 'project': Project.objects.get(id=TestProjectPageBase.project_id),
'started_on': now,
'completed_on': now,
'outcome': Build.SUCCEEDED
}
self.project1_build_failure = {
- 'project': Project.objects.get(id=TestProjectPage.project_id),
+ 'project': Project.objects.get(id=TestProjectPageBase.project_id),
'started_on': now,
'completed_on': now,
'outcome': Build.FAILED
@@ -133,7 +92,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
list_check_box_id: list
):
# Check edit column
- edit_column = self.find(f'#{edit_btn_id}')
+ finder = lambda driver: self.find(f'#{edit_btn_id}')
+ edit_column = self.wait_until_finder_clickable(finder)
self.assertTrue(edit_column.is_displayed())
edit_column.click()
# Check dropdown is visible
@@ -192,7 +152,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
def test_show_rows(row_to_show, show_row_link):
# Check that we can show rows == row_to_show
show_row_link.select_by_value(str(row_to_show))
- self.wait_until_visible(f'#{table_selector} tbody tr', poll=3)
+ self.wait_until_visible(f'#{table_selector} tbody tr')
# check at least some rows are visible
self.assertTrue(
len(self.find_all(f'#{table_selector} tbody tr')) > 0
@@ -222,34 +182,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
rows = self.find_all(f'#{table_selector} tbody tr')
self.assertTrue(len(rows) > 0)
- def test_create_project(self):
- """ Create/Test new project using:
- - Project Name: Any string
- - Release: Any string
- - Merge Toaster settings: True or False
- """
- self._create_project(project_name=self.PROJECT_NAME)
-
- def test_image_recipe_editColumn(self):
- """ Test the edit column feature in image recipe table on project page """
- self._get_create_builds(success=10, failure=10)
-
- url = reverse('projectimagerecipes', args=(TestProjectPage.project_id,))
- self.get(url)
- self.wait_until_present('#imagerecipestable tbody tr')
-
- column_list = [
- 'get_description_or_summary', 'layer_version__get_vcs_reference',
- 'layer_version__layer__name', 'license', 'recipe-file', 'section',
- 'version'
- ]
-
- # Check that we can hide the edit column
- self._mixin_test_table_edit_column(
- 'imagerecipestable',
- 'edit-columns-button',
- [f'checkbox-{column}' for column in column_list]
- )
+class TestProjectPage(TestProjectPageBase):
def test_page_header_on_project_page(self):
""" Check page header in project page:
@@ -272,8 +205,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
logo_img = logo.find_element(By.TAG_NAME, 'img')
self.assertTrue(logo_img.is_displayed(),
'Logo of Yocto project not found')
- self.assertTrue(
- '/static/img/logo.png' in str(logo_img.get_attribute('src')),
+ self.assertIn(
+ '/static/img/logo.png', str(logo_img.get_attribute('src')),
'Logo of Yocto project not found'
)
# "Toaster"+" Information icon", clickable
@@ -282,34 +215,34 @@ class TestProjectPage(SeleniumFunctionalTestCase):
"//div[@class='toaster-navbar-brand']//a[@class='brand']",
)
self.assertTrue(toaster.is_displayed(), 'Toaster not found')
- self.assertTrue(toaster.text == 'Toaster')
+ self.assertEqual(toaster.text, 'Toaster')
info_sign = self.find('.glyphicon-info-sign')
self.assertTrue(info_sign.is_displayed())
# "Server Icon" + "All builds"
all_builds = self.find('#navbar-all-builds')
all_builds_link = all_builds.find_element(By.TAG_NAME, 'a')
- self.assertTrue("All builds" in all_builds_link.text)
- self.assertTrue(
- '/toastergui/builds/' in str(all_builds_link.get_attribute('href'))
+ self.assertIn("All builds", all_builds_link.text)
+ self.assertIn(
+ '/toastergui/builds/', str(all_builds_link.get_attribute('href'))
)
server_icon = all_builds.find_element(By.TAG_NAME, 'i')
- self.assertTrue(
- server_icon.get_attribute('class') == 'glyphicon glyphicon-tasks'
+ self.assertEqual(
+ server_icon.get_attribute('class'), 'glyphicon glyphicon-tasks'
)
self.assertTrue(server_icon.is_displayed())
# "Directory Icon" + "All projects"
all_projects = self.find('#navbar-all-projects')
all_projects_link = all_projects.find_element(By.TAG_NAME, 'a')
- self.assertTrue("All projects" in all_projects_link.text)
- self.assertTrue(
- '/toastergui/projects/' in str(all_projects_link.get_attribute(
+ self.assertIn("All projects", all_projects_link.text)
+ self.assertIn(
+ '/toastergui/projects/', str(all_projects_link.get_attribute(
'href'))
)
dir_icon = all_projects.find_element(By.TAG_NAME, 'i')
- self.assertTrue(
- dir_icon.get_attribute('class') == 'icon-folder-open'
+ self.assertEqual(
+ dir_icon.get_attribute('class'), 'icon-folder-open'
)
self.assertTrue(dir_icon.is_displayed())
@@ -317,23 +250,23 @@ class TestProjectPage(SeleniumFunctionalTestCase):
toaster_docs_link = self.find('#navbar-docs')
toaster_docs_link_link = toaster_docs_link.find_element(By.TAG_NAME,
'a')
- self.assertTrue("Documentation" in toaster_docs_link_link.text)
- self.assertTrue(
- toaster_docs_link_link.get_attribute('href') == 'http://docs.yoctoproject.org/toaster-manual/index.html#toaster-user-manual'
+ self.assertIn("Documentation", toaster_docs_link_link.text)
+ self.assertEqual(
+ toaster_docs_link_link.get_attribute('href'), 'http://docs.yoctoproject.org/toaster-manual/index.html#toaster-user-manual'
)
book_icon = toaster_docs_link.find_element(By.TAG_NAME, 'i')
- self.assertTrue(
- book_icon.get_attribute('class') == 'glyphicon glyphicon-book'
+ self.assertEqual(
+ book_icon.get_attribute('class'), 'glyphicon glyphicon-book'
)
self.assertTrue(book_icon.is_displayed())
# AT RIGHT -> button "New project"
new_project_button = self.find('#new-project-button')
self.assertTrue(new_project_button.is_displayed())
- self.assertTrue(new_project_button.text == 'New project')
+ self.assertEqual(new_project_button.text, 'New project')
new_project_button.click()
- self.assertTrue(
- '/toastergui/newproject/' in str(self.driver.current_url)
+ self.assertIn(
+ '/toastergui/newproject/', str(self.driver.current_url)
)
def test_edit_project_name(self):
@@ -348,7 +281,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
# click on "Edit" icon button
self.wait_until_visible('#project-name-container')
- edit_button = self.find('#project-change-form-toggle')
+ finder = lambda driver: self.find('#project-change-form-toggle')
+ edit_button = self.wait_until_finder_clickable(finder)
edit_button.click()
project_name_input = self.find('#project-name-change-input')
self.assertTrue(project_name_input.is_displayed())
@@ -358,8 +292,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
# check project name is changed
self.wait_until_visible('#project-name-container')
- self.assertTrue(
- 'New Name' in str(self.find('#project-name-container').text)
+ self.assertIn(
+ 'New Name', str(self.find('#project-name-container').text)
)
def test_project_page_tabs(self):
@@ -376,10 +310,10 @@ class TestProjectPage(SeleniumFunctionalTestCase):
# check "configuration" tab
self.wait_until_visible('#topbar-configuration-tab')
config_tab = self.find('#topbar-configuration-tab')
- self.assertTrue(config_tab.get_attribute('class') == 'active')
- self.assertTrue('Configuration' in str(config_tab.text))
- self.assertTrue(
- f"/toastergui/project/{TestProjectPage.project_id}" in str(self.driver.current_url)
+ self.assertEqual(config_tab.get_attribute('class'), 'active')
+ self.assertIn('Configuration', str(config_tab.text))
+ self.assertIn(
+ f"/toastergui/project/{TestProjectPageBase.project_id}", str(self.driver.current_url)
)
def get_tabs():
@@ -392,9 +326,9 @@ class TestProjectPage(SeleniumFunctionalTestCase):
def check_tab_link(tab_index, tab_name, url):
tab = get_tabs()[tab_index]
tab_link = tab.find_element(By.TAG_NAME, 'a')
- self.assertTrue(url in tab_link.get_attribute('href'))
- self.assertTrue(tab_name in tab_link.text)
- self.assertTrue(tab.get_attribute('class') == 'active')
+ self.assertIn(url, tab_link.get_attribute('href'))
+ self.assertIn(tab_name, tab_link.text)
+ self.assertEqual(tab.get_attribute('class'), 'active')
# check "Builds" tab
builds_tab = get_tabs()[1]
@@ -402,7 +336,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
check_tab_link(
1,
'Builds',
- f"/toastergui/project/{TestProjectPage.project_id}/builds"
+ f"/toastergui/project/{TestProjectPageBase.project_id}/builds"
)
# check "Import layers" tab
@@ -411,7 +345,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
check_tab_link(
2,
'Import layer',
- f"/toastergui/project/{TestProjectPage.project_id}/importlayer"
+ f"/toastergui/project/{TestProjectPageBase.project_id}/importlayer"
)
# check "New custom image" tab
@@ -420,7 +354,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
check_tab_link(
3,
'New custom image',
- f"/toastergui/project/{TestProjectPage.project_id}/newcustomimage"
+ f"/toastergui/project/{TestProjectPageBase.project_id}/newcustomimage"
)
# check search box can be use to build recipes
@@ -428,13 +362,17 @@ class TestProjectPage(SeleniumFunctionalTestCase):
search_box.send_keys('core-image-minimal')
self.find('#build-button').click()
self.wait_until_visible('#latest-builds')
- lastest_builds = self.driver.find_elements(
- By.XPATH,
- '//div[@id="latest-builds"]',
- )
- last_build = lastest_builds[0]
- self.assertTrue(
- 'core-image-minimal' in str(last_build.text)
+ buildtext = "Loading"
+ while "Loading" in buildtext:
+ time.sleep(1)
+ lastest_builds = self.driver.find_elements(
+ By.XPATH,
+ '//div[@id="latest-builds"]',
+ )
+ last_build = lastest_builds[0]
+ buildtext = last_build.text
+ self.assertIn(
+ 'core-image-minimal', str(last_build.text)
)
def test_softwareRecipe_page(self):
@@ -446,7 +384,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
"""
self._navigate_to_config_nav('softwarerecipestable', 4)
# check title "Compatible software recipes" is displayed
- self.assertTrue("Compatible software recipes" in self.get_page_source())
+ self.assertIn("Compatible software recipes", self.get_page_source())
# Test search input
self._mixin_test_table_search_input(
input_selector='search-input-softwarerecipestable',
@@ -455,12 +393,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
table_selector='softwarerecipestable'
)
# check "build recipe" button works
- rows = self.find_all('#softwarerecipestable tbody tr')
- image_to_build = rows[0]
- build_btn = image_to_build.find_element(
- By.XPATH,
- '//td[@class="add-del-layers"]//a[1]'
- )
+ finder = lambda driver: self.find_all('#softwarerecipestable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]/a')
+ build_btn = self.wait_until_finder_clickable(finder)
build_btn.click()
build_state = wait_until_build(self, 'queued cloning starting parsing failed')
lastest_builds = self.driver.find_elements(
@@ -468,11 +402,10 @@ class TestProjectPage(SeleniumFunctionalTestCase):
'//div[@id="latest-builds"]/div'
)
self.assertTrue(len(lastest_builds) > 0)
- last_build = lastest_builds[0]
- cancel_button = last_build.find_element(
- By.XPATH,
- '//span[@class="cancel-build-btn pull-right alert-link"]',
- )
+ # Find the latest builds, the last build and then the cancel button
+
+ finder = lambda driver: driver.find_elements(By.XPATH, '//div[@id="latest-builds"]/div')[0].find_element(By.XPATH, '//span[@class="cancel-build-btn pull-right alert-link"]')
+ cancel_button = self.wait_until_finder_clickable(finder)
cancel_button.click()
if 'starting' not in build_state: # change build state when cancelled in starting state
wait_until_build_cancelled(self)
@@ -510,7 +443,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
"""
self._navigate_to_config_nav('machinestable', 5)
# check title "Compatible software recipes" is displayed
- self.assertTrue("Compatible machines" in self.get_page_source())
+ self.assertIn("Compatible machines", self.get_page_source())
# Test search input
self._mixin_test_table_search_input(
input_selector='search-input-machinestable',
@@ -519,17 +452,13 @@ class TestProjectPage(SeleniumFunctionalTestCase):
table_selector='machinestable'
)
# check "Select machine" button works
- rows = self.find_all('#machinestable tbody tr')
- machine_to_select = rows[0]
- select_btn = machine_to_select.find_element(
- By.XPATH,
- '//td[@class="add-del-layers"]//a[1]'
- )
- select_btn.send_keys(Keys.RETURN)
- self.wait_until_visible('#config-nav')
+ finder = lambda driver: self.find_all('#machinestable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]')
+ select_btn = self.wait_until_finder_clickable(finder)
+ select_btn.click()
+ self.wait_until_visible('#project-machine-name')
project_machine_name = self.find('#project-machine-name')
- self.assertTrue(
- 'qemux86-64' in project_machine_name.text
+ self.assertIn(
+ 'qemux86-64', project_machine_name.text
)
# check "Add layer" button works
self._navigate_to_config_nav('machinestable', 5)
@@ -540,16 +469,23 @@ class TestProjectPage(SeleniumFunctionalTestCase):
searchBtn_selector='search-submit-machinestable',
table_selector='machinestable'
)
- self.wait_until_visible('#machinestable tbody tr', poll=3)
- rows = self.find_all('#machinestable tbody tr')
- machine_to_add = rows[0]
- add_btn = machine_to_add.find_element(By.XPATH, '//td[@class="add-del-layers"]')
+
+ self.wait_until_visible('#machinestable tbody tr')
+ # Locate a machine to add button
+ finder = lambda driver: self.find_all('#machinestable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]')
+ add_btn = self.wait_until_finder_clickable(finder)
add_btn.click()
self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have added 1 layer to your project' in str(change_notification.text)
+ self.assertIn(
+ f'You have added 1 layer to your project', str(change_notification.text)
)
+
+ finder = lambda driver: self.find('#hide-alert')
+ hide_button = self.wait_until_finder_clickable(finder)
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
+
# check Machine table feature(show/hide column, pagination)
self._navigate_to_config_nav('machinestable', 5)
column_list = [
@@ -580,7 +516,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
"""
self._navigate_to_config_nav('layerstable', 6)
# check title "Compatible layers" is displayed
- self.assertTrue("Compatible layers" in self.get_page_source())
+ self.assertIn("Compatible layers", self.get_page_source())
# Test search input
input_text='meta-tanowrt'
self._mixin_test_table_search_input(
@@ -590,42 +526,44 @@ class TestProjectPage(SeleniumFunctionalTestCase):
table_selector='layerstable'
)
# check "Add layer" button works
- self.wait_until_visible('#layerstable tbody tr', poll=3)
- rows = self.find_all('#layerstable tbody tr')
- layer_to_add = rows[0]
- add_btn = layer_to_add.find_element(
- By.XPATH,
- '//td[@class="add-del-layers"]'
- )
+ self.wait_until_visible('#layerstable tbody tr')
+ finder = lambda driver: self.find_all('#layerstable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]/a[@data-directive="add"]')
+ add_btn = self.wait_until_finder_clickable(finder)
add_btn.click()
# check modal is displayed
- self.wait_until_visible('#dependencies-modal', poll=3)
+ self.wait_until_visible('#dependencies-modal')
list_dependencies = self.find_all('#dependencies-list li')
# click on add-layers button
- add_layers_btn = self.driver.find_element(
- By.XPATH,
- '//form[@id="dependencies-modal-form"]//button[@class="btn btn-primary"]'
- )
+ finder = lambda driver: self.driver.find_element(By.XPATH, '//form[@id="dependencies-modal-form"]//button[@class="btn btn-primary"]')
+ add_layers_btn = self.wait_until_finder_clickable(finder)
add_layers_btn.click()
self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have added {len(list_dependencies)+1} layers to your project: {input_text} and its dependencies' in str(change_notification.text)
+ self.assertIn(
+ f'You have added {len(list_dependencies)+1} layers to your project: {input_text} and its dependencies', str(change_notification.text)
)
+
+ finder = lambda driver: self.find('#hide-alert')
+ hide_button = self.wait_until_finder_clickable(finder)
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
+
# check "Remove layer" button works
- self.wait_until_visible('#layerstable tbody tr', poll=3)
- rows = self.find_all('#layerstable tbody tr')
- layer_to_remove = rows[0]
- remove_btn = layer_to_remove.find_element(
- By.XPATH,
- '//td[@class="add-del-layers"]'
- )
+ self.wait_until_visible('#layerstable tbody tr')
+ finder = lambda driver: self.find_all('#layerstable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]/a[@data-directive="remove"]')
+ remove_btn = self.wait_until_finder_clickable(finder)
remove_btn.click()
- self.wait_until_visible('#change-notification', poll=2)
+ self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have removed 1 layer from your project: {input_text}' in str(change_notification.text)
+ self.assertIn(
+ f'You have removed 1 layer from your project: {input_text}', str(change_notification.text)
)
+
+ finder = lambda driver: self.find('#hide-alert')
+ hide_button = self.wait_until_finder_clickable(finder)
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
+
# check layers table feature(show/hide column, pagination)
self._navigate_to_config_nav('layerstable', 6)
column_list = [
@@ -656,7 +594,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
"""
self._navigate_to_config_nav('distrostable', 7)
# check title "Compatible distros" is displayed
- self.assertTrue("Compatible Distros" in self.get_page_source())
+ self.assertIn("Compatible Distros", self.get_page_source())
# Test search input
input_text='poky-altcfg'
self._mixin_test_table_search_input(
@@ -666,17 +604,14 @@ class TestProjectPage(SeleniumFunctionalTestCase):
table_selector='distrostable'
)
# check "Add distro" button works
- rows = self.find_all('#distrostable tbody tr')
- distro_to_add = rows[0]
- add_btn = distro_to_add.find_element(
- By.XPATH,
- '//td[@class="add-del-layers"]//a[1]'
- )
+ self.wait_until_visible(".add-del-layers")
+ finder = lambda driver: self.find_all('#distrostable tbody tr')[0].find_element(By.XPATH, '//td[@class="add-del-layers"]')
+ add_btn = self.wait_until_finder_clickable(finder)
add_btn.click()
- self.wait_until_visible('#change-notification', poll=2)
+ self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have changed the distro to: {input_text}' in str(change_notification.text)
+ self.assertIn(
+ f'You have changed the distro to: {input_text}', str(change_notification.text)
)
# check distro table feature(show/hide column, pagination)
self._navigate_to_config_nav('distrostable', 7)
@@ -699,7 +634,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
)
def test_single_layer_page(self):
- """ Test layer page
+ """ Test layer details page using meta-poky as an example (assumes is added to start with)
- Check if title is displayed
- Check add/remove layer button works
- Check tabs(layers, recipes, machines) are displayed
@@ -708,45 +643,62 @@ class TestProjectPage(SeleniumFunctionalTestCase):
- Check layer summary
- Check layer description
"""
- url = reverse("layerdetails", args=(TestProjectPage.project_id, 8))
- self.get(url)
+ self._navigate_to_config_nav('layerstable', 6)
+ layer_link = self.driver.find_element(By.XPATH, '//tr/td[@class="layer__name"]/a[contains(text(),"meta-poky")]')
+ layer_link.click()
self.wait_until_visible('.page-header')
# check title is displayed
self.assertTrue(self.find('.page-header h1').is_displayed())
- # check add layer button works
- remove_layer_btn = self.find('#add-remove-layer-btn')
+ # check remove layer button works
+ finder = lambda driver: self.find('#add-remove-layer-btn')
+ remove_layer_btn = self.wait_until_finder_clickable(finder)
remove_layer_btn.click()
- self.wait_until_visible('#change-notification', poll=2)
+ self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have removed 1 layer from your project' in str(change_notification.text)
+ self.assertIn(
+ f'You have removed 1 layer from your project', str(change_notification.text)
)
- # check add layer button works, 18 is the random layer id
- add_layer_btn = self.find('#add-remove-layer-btn')
+ finder = lambda driver: self.find('#hide-alert')
+ hide_button = self.wait_until_finder_clickable(finder)
+ hide_button.click()
+ # check add layer button works
+ self.wait_until_not_visible('#change-notification')
+ finder = lambda driver: self.find('#add-remove-layer-btn')
+ add_layer_btn = self.wait_until_finder_clickable(finder)
add_layer_btn.click()
self.wait_until_visible('#change-notification')
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have added 1 layer to your project' in str(change_notification.text)
+ self.assertIn(
+ f'You have added 1 layer to your project', str(change_notification.text)
)
+ finder = lambda driver: self.find('#hide-alert')
+ hide_button = self.wait_until_finder_clickable(finder)
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
# check tabs(layers, recipes, machines) are displayed
tabs = self.find_all('.nav-tabs li')
self.assertEqual(len(tabs), 3)
# Check first tab
tabs[0].click()
- self.assertTrue(
- 'active' in str(self.find('#information').get_attribute('class'))
+ self.assertIn(
+ 'active', str(self.find('#information').get_attribute('class'))
)
- # Check second tab
+ # Check second tab (recipes)
+ self.wait_until_visible('.nav-tabs')
+ # Ensure page is scrolled to the top
+ self.driver.execute_script('window.scrollTo({behavior: "instant", top: 0, left: 0})')
tabs[1].click()
- self.assertTrue(
- 'active' in str(self.find('#recipes').get_attribute('class'))
+ self.assertIn(
+ 'active', str(self.find('#recipes').get_attribute('class'))
)
- # Check third tab
+ # Check third tab (machines)
+ self.wait_until_visible('.nav-tabs')
+ # Ensure page is scrolled to the top
+ self.driver.execute_script('window.scrollTo({behavior: "instant", top: 0, left: 0})')
tabs[2].click()
- self.assertTrue(
- 'active' in str(self.find('#machines').get_attribute('class'))
+ self.assertIn(
+ 'active', str(self.find('#machines').get_attribute('class'))
)
# Check left section is displayed
section = self.find('.well')
@@ -755,9 +707,13 @@ class TestProjectPage(SeleniumFunctionalTestCase):
section.find_element(By.XPATH, '//h2[1]').is_displayed()
)
# Check layer summary
- self.assertTrue("Summary" in section.text)
+ self.assertIn("Summary", section.text)
# Check layer description
- self.assertTrue("Description" in section.text)
+ self.assertIn("Description", section.text)
+
+@pytest.mark.django_db
+@pytest.mark.order("last")
+class TestProjectPageRecipes(TestProjectPageBase):
def test_single_recipe_page(self):
""" Test recipe page
@@ -767,7 +723,12 @@ class TestProjectPage(SeleniumFunctionalTestCase):
- Check recipe: name, summary, description, Version, Section,
License, Approx. packages included, Approx. size, Recipe file
"""
- url = reverse("recipedetails", args=(TestProjectPage.project_id, 53428))
+ # Use a recipe which is likely to exist in the layer index but not enabled
+ # in poky out the box - xen-image-minimal from meta-virtualization
+ self._navigate_to_project_page()
+ prj = Project.objects.get(pk=TestProjectPageBase.project_id)
+ recipe_id = prj.get_all_compatible_recipes().get(name="xen-image-minimal").pk
+ url = reverse("recipedetails", args=(TestProjectPageBase.project_id, recipe_id))
self.get(url)
self.wait_until_visible('.page-header')
# check title is displayed
@@ -782,11 +743,33 @@ class TestProjectPage(SeleniumFunctionalTestCase):
section.find_element(By.XPATH, '//h2[1]').is_displayed()
)
# Check recipe sections details info are displayed
- self.assertTrue("Summary" in section.text)
- self.assertTrue("Description" in section.text)
- self.assertTrue("Version" in section.text)
- self.assertTrue("Section" in section.text)
- self.assertTrue("License" in section.text)
- self.assertTrue("Approx. packages included" in section.text)
- self.assertTrue("Approx. package size" in section.text)
- self.assertTrue("Recipe file" in section.text)
+ self.assertIn("Summary", section.text)
+ self.assertIn("Description", section.text)
+ self.assertIn("Version", section.text)
+ self.assertIn("Section", section.text)
+ self.assertIn("License", section.text)
+ self.assertIn("Approx. packages included", section.text)
+ self.assertIn("Approx. package size", section.text)
+ self.assertIn("Recipe file", section.text)
+
+ def test_image_recipe_editColumn(self):
+ """ Test the edit column feature in image recipe table on project page """
+ self._get_create_builds(success=10, failure=10)
+
+ url = reverse('projectimagerecipes', args=(TestProjectPageBase.project_id,))
+ self.get(url)
+ self.wait_until_present('#imagerecipestable tbody tr')
+
+ column_list = [
+ 'get_description_or_summary', 'layer_version__get_vcs_reference',
+ 'layer_version__layer__name', 'license', 'recipe-file', 'section',
+ 'version'
+ ]
+
+ # Check that we can hide the edit column
+ self._mixin_test_table_edit_column(
+ 'imagerecipestable',
+ 'edit-columns-button',
+ [f'checkbox-{column}' for column in column_list]
+ )
+
diff --git a/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py b/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py
index eb905ddf..0c1100c5 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py
@@ -7,72 +7,27 @@
#
import string
-import random
+import time
import pytest
from django.urls import reverse
from selenium.webdriver import Keys
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import ElementClickInterceptedException, NoSuchElementException, TimeoutException
-from orm.models import Project
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase, MASTER_PROJECT_ID
from selenium.webdriver.common.by import By
from .utils import get_projectId_from_url, wait_until_build, wait_until_build_cancelled
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
-class TestProjectConfigTab(SeleniumFunctionalTestCase):
+class TestProjectConfigTabBase(SeleniumFunctionalTestCase):
PROJECT_NAME = 'TestProjectConfigTab'
project_id = None
- def _create_project(self, project_name, **kwargs):
- """ Create/Test new project using:
- - Project Name: Any string
- - Release: Any string
- - Merge Toaster settings: True or False
- """
- release = kwargs.get('release', '3')
- self.get(reverse('newproject'))
- self.wait_until_visible('#new-project-name')
- self.find("#new-project-name").send_keys(project_name)
- select = Select(self.find("#projectversion"))
- select.select_by_value(release)
-
- # check merge toaster settings
- checkbox = self.find('.checkbox-mergeattr')
- if not checkbox.is_selected():
- checkbox.click()
-
- if self.PROJECT_NAME != 'TestProjectConfigTab':
- # Reset project name if it's not the default one
- self.PROJECT_NAME = 'TestProjectConfigTab'
-
- self.find("#create-project-button").click()
-
- try:
- self.wait_until_visible('#hint-error-project-name', poll=3)
- url = reverse('project', args=(TestProjectConfigTab.project_id, ))
- self.get(url)
- self.wait_until_visible('#config-nav', poll=3)
- except TimeoutException:
- self.wait_until_visible('#config-nav', poll=3)
-
- def _random_string(self, length):
- return ''.join(
- random.choice(string.ascii_letters) for _ in range(length)
- )
-
def _navigate_to_project_page(self):
# Navigate to project page
- if TestProjectConfigTab.project_id is None:
- self._create_project(project_name=self._random_string(10))
- current_url = self.driver.current_url
- TestProjectConfigTab.project_id = get_projectId_from_url(
- current_url)
- else:
- url = reverse('project', args=(TestProjectConfigTab.project_id,))
- self.get(url)
+ if TestProjectConfigTabBase.project_id is None:
+ TestProjectConfigTabBase.project_id = self.create_new_project(self.PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+ url = reverse('project', args=(TestProjectConfigTabBase.project_id,))
+ self.get(url)
self.wait_until_visible('#config-nav')
def _create_builds(self):
@@ -88,8 +43,8 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
'//div[@id="latest-builds"]/div',
)
last_build = lastest_builds[0]
- self.assertTrue(
- 'foo' in str(last_build.text)
+ self.assertIn(
+ 'foo', str(last_build.text)
)
last_build = lastest_builds[0]
try:
@@ -114,6 +69,8 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
config_nav = self.find('#config-nav')
return config_nav.find_elements(By.TAG_NAME, 'li')[index]
+class TestProjectConfigTab(TestProjectConfigTabBase):
+
def test_project_config_nav(self):
""" Test project config tab navigation:
- Check if the menu is displayed and contains the right elements:
@@ -138,48 +95,48 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
def check_config_nav_item(index, item_name, url):
item = _get_config_nav_item(index)
- self.assertTrue(item_name in item.text)
- self.assertTrue(item.get_attribute('class') == 'active')
- self.assertTrue(url in self.driver.current_url)
+ self.assertIn(item_name, item.text)
+ self.assertEqual(item.get_attribute('class'), 'active')
+ self.assertIn(url, self.driver.current_url)
# check if the menu contains the right elements
# COMPATIBLE METADATA
compatible_metadata = _get_config_nav_item(1)
- self.assertTrue(
- "compatible metadata" in compatible_metadata.text.lower()
+ self.assertIn(
+ "compatible metadata", compatible_metadata.text.lower()
)
# EXTRA CONFIGURATION
extra_configuration = _get_config_nav_item(8)
- self.assertTrue(
- "extra configuration" in extra_configuration.text.lower()
+ self.assertIn(
+ "extra configuration", extra_configuration.text.lower()
)
# Actions
actions = _get_config_nav_item(10)
- self.assertTrue("actions" in str(actions.text).lower())
+ self.assertIn("actions", str(actions.text).lower())
conf_nav_list = [
# config
[0, 'Configuration',
- f"/toastergui/project/{TestProjectConfigTab.project_id}"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}"],
# custom images
[2, 'Custom images',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/customimages"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/customimages"],
# image recipes
[3, 'Image recipes',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/images"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/images"],
# software recipes
[4, 'Software recipes',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/softwarerecipes"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/softwarerecipes"],
# machines
[5, 'Machines',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/machines"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/machines"],
# layers
[6, 'Layers',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/layers"],
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/layers"],
# distro
[7, 'Distros',
- f"/toastergui/project/{TestProjectConfigTab.project_id}/distros"],
- # [9, 'BitBake variables', f"/toastergui/project/{TestProjectConfigTab.project_id}/configuration"], # bitbake variables
+ f"/toastergui/project/{TestProjectConfigTabBase.project_id}/distros"],
+ # [9, 'BitBake variables', f"/toastergui/project/{TestProjectConfigTabBase.project_id}/configuration"], # bitbake variables
]
for index, item_name, url in conf_nav_list:
item = _get_config_nav_item(index)
@@ -253,7 +210,7 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
def test_show_rows(row_to_show, show_row_link):
# Check that we can show rows == row_to_show
show_row_link.select_by_value(str(row_to_show))
- self.wait_until_visible('#imagerecipestable tbody tr', poll=3)
+ self.wait_until_visible('#imagerecipestable tbody tr')
# check at least some rows are visible
self.assertTrue(
len(self.find_all('#imagerecipestable tbody tr')) > 0
@@ -299,9 +256,11 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
- meta-poky
- meta-yocto-bsp
"""
- # Create a new project for this test
- project_name = self._random_string(10)
- self._create_project(project_name=project_name)
+ project_id = self.create_new_project(self.PROJECT_NAME + "-ST", MASTER_PROJECT_ID, None, True)
+ url = reverse('project', args=(project_id,))
+ self.get(url)
+ self.wait_until_visible('#config-nav')
+
# check if the menu is displayed
self.wait_until_visible('#project-page')
block_l = self.driver.find_element(
@@ -313,7 +272,7 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
def check_machine_distro(self, item_name, new_item_name, block_id):
block = self.find(f'#{block_id}')
title = block.find_element(By.TAG_NAME, 'h3')
- self.assertTrue(item_name.capitalize() in title.text)
+ self.assertIn(item_name.capitalize(), title.text)
edit_btn = self.find(f'#change-{item_name}-toggle')
edit_btn.click()
self.wait_until_visible(f'#{item_name}-change-input')
@@ -324,12 +283,15 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
change_btn.click()
self.wait_until_visible(f'#project-{item_name}-name')
project_name = self.find(f'#project-{item_name}-name')
- self.assertTrue(new_item_name in project_name.text)
+ self.assertIn(new_item_name, project_name.text)
# check change notificaiton is displayed
change_notification = self.find('#change-notification')
- self.assertTrue(
- f'You have changed the {item_name} to: {new_item_name}' in change_notification.text
+ self.assertIn(
+ f'You have changed the {item_name} to: {new_item_name}', change_notification.text
)
+ hide_button = self.find('#hide-alert')
+ hide_button.click()
+ self.wait_until_not_visible('#change-notification')
# Machine
check_machine_distro(self, 'machine', 'qemux86-64', 'machine-section')
@@ -338,97 +300,51 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
# Project release
title = project_release.find_element(By.TAG_NAME, 'h3')
- self.assertTrue("Project release" in title.text)
- self.assertTrue(
- "Yocto Project master" in self.find('#project-release-title').text
+ self.assertIn("Project release", title.text)
+ self.assertIn(
+ "Yocto Project master", self.find('#project-release-title').text
)
# Layers
title = layers.find_element(By.TAG_NAME, 'h3')
- self.assertTrue("Layers" in title.text)
+ self.assertIn("Layers", title.text)
+ self.wait_until_clickable('#layer-add-input')
# check at least three layers are displayed
# openembedded-core
# meta-poky
# meta-yocto-bsp
- layers_list = layers.find_element(By.ID, 'layers-in-project-list')
- layers_list_items = layers_list.find_elements(By.TAG_NAME, 'li')
+ layer_list_items = []
+ starttime = time.time()
+ while len(layer_list_items) < 3:
+ layers_list = self.driver.find_element(By.ID, 'layers-in-project-list')
+ layer_list_items = layers_list.find_elements(By.TAG_NAME, 'li')
+ if time.time() > (starttime + 30):
+ self.fail("Layer list didn't contain at least 3 items within 30s (contained %d)" % len(layer_list_items))
+
# remove all layers except the first three layers
- for i in range(3, len(layers_list_items)):
- layers_list_items[i].find_element(By.TAG_NAME, 'span').click()
+ for i in range(3, len(layer_list_items)):
+ layer_list_items[i].find_element(By.TAG_NAME, 'span').click()
+
# check can add a layer if exists
add_layer_input = layers.find_element(By.ID, 'layer-add-input')
add_layer_input.send_keys('meta-oe')
self.wait_until_visible('#layer-container > form > div > span > div')
- dropdown_item = self.driver.find_element(
- By.XPATH,
- '//*[@id="layer-container"]/form/div/span/div'
- )
- try:
- dropdown_item.click()
- except ElementClickInterceptedException:
- self.skipTest(
- "layer-container dropdown item click intercepted. Element not properly visible.")
+ self.wait_until_visible('.dropdown-menu')
+ finder = lambda driver: driver.find_element(By.XPATH, '//*[@id="layer-container"]/form/div/span/div/div/div')
+ dropdown_item = self.wait_until_finder_clickable(finder)
+ dropdown_item.click()
+ self.wait_until_clickable('#add-layer-btn')
add_layer_btn = layers.find_element(By.ID, 'add-layer-btn')
add_layer_btn.click()
self.wait_until_visible('#layers-in-project-list')
- # check layer is added
- layers_list_items = layers_list.find_elements(By.TAG_NAME, 'li')
- self.assertTrue(len(layers_list_items) == 4)
- def test_most_build_recipes(self):
- """ Test most build recipes block contains"""
- def rebuild_from_most_build_recipes(recipe_list_items):
- checkbox = recipe_list_items[0].find_element(By.TAG_NAME, 'input')
- checkbox.click()
- build_btn = self.find('#freq-build-btn')
- build_btn.click()
- self.wait_until_visible('#latest-builds')
- wait_until_build(self, 'queued cloning starting parsing failed')
- lastest_builds = self.driver.find_elements(
- By.XPATH,
- '//div[@id="latest-builds"]/div'
- )
- self.assertTrue(len(lastest_builds) >= 2)
- last_build = lastest_builds[0]
- try:
- cancel_button = last_build.find_element(
- By.XPATH,
- '//span[@class="cancel-build-btn pull-right alert-link"]',
- )
- cancel_button.click()
- except NoSuchElementException:
- # Skip if the build is already cancelled
- pass
- wait_until_build_cancelled(self)
- # Create a new project for remaining asserts
- project_name = self._random_string(10)
- self._create_project(project_name=project_name, release='2')
- current_url = self.driver.current_url
- TestProjectConfigTab.project_id = get_projectId_from_url(current_url)
- url = current_url.split('?')[0]
-
- # Create a new builds
- self._create_builds()
-
- # back to project page
- self.driver.get(url)
-
- self.wait_until_visible('#project-page', poll=3)
-
- # Most built recipes
- most_built_recipes = self.driver.find_element(
- By.XPATH, '//*[@id="project-page"]/div[1]/div[3]')
- title = most_built_recipes.find_element(By.TAG_NAME, 'h3')
- self.assertTrue("Most built recipes" in title.text)
- # check can select a recipe and build it
- self.wait_until_visible('#freq-build-list', poll=3)
- recipe_list = self.find('#freq-build-list')
- recipe_list_items = recipe_list.find_elements(By.TAG_NAME, 'li')
- self.assertTrue(
- len(recipe_list_items) > 0,
- msg="Any recipes found in the most built recipes list",
- )
- rebuild_from_most_build_recipes(recipe_list_items)
- TestProjectConfigTab.project_id = None # reset project id
+ # check layer is added
+ layer_list_items = []
+ starttime = time.time()
+ while len(layer_list_items) < 4:
+ layers_list = self.driver.find_element(By.ID, 'layers-in-project-list')
+ layer_list_items = layers_list.find_elements(By.TAG_NAME, 'li')
+ if time.time() > (starttime + 30):
+ self.fail("Layer list didn't contain at least 4 items within 30s (contained %d)" % len(layer_list_items))
def test_project_page_tab_importlayer(self):
""" Test project page tab import layer """
@@ -466,42 +382,42 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
layers = block_l.find_element(By.ID, 'layer-container')
layers_list = layers.find_element(By.ID, 'layers-in-project-list')
layers_list_items = layers_list.find_elements(By.TAG_NAME, 'li')
- self.assertTrue(
- 'meta-fake' in str(layers_list_items[-1].text)
+ self.assertIn(
+ 'meta-fake', str(layers_list_items[-1].text)
)
def test_project_page_custom_image_no_image(self):
""" Test project page tab "New custom image" when no custom image """
- project_name = self._random_string(10)
- self._create_project(project_name=project_name)
- current_url = self.driver.current_url
- TestProjectConfigTab.project_id = get_projectId_from_url(current_url)
+ project_id = self.create_new_project(self.PROJECT_NAME + "-CustomImage", MASTER_PROJECT_ID, None, True)
+ url = reverse('project', args=(project_id,))
+ self.get(url)
+ self.wait_until_visible('#config-nav')
+
# navigate to "Custom image" tab
custom_image_section = self._get_config_nav_item(2)
custom_image_section.click()
self.wait_until_visible('#empty-state-customimagestable')
# Check message when no custom image
- self.assertTrue(
- "You have not created any custom images yet." in str(
+ self.assertIn(
+ "You have not created any custom images yet.", str(
self.find('#empty-state-customimagestable').text
)
)
div_empty_msg = self.find('#empty-state-customimagestable')
link_create_custom_image = div_empty_msg.find_element(
By.TAG_NAME, 'a')
- self.assertTrue(TestProjectConfigTab.project_id is not None)
- self.assertTrue(
- f"/toastergui/project/{TestProjectConfigTab.project_id}/newcustomimage" in str(
+ self.assertTrue(TestProjectConfigTabBase.project_id is not None)
+ self.assertIn(
+ f"/toastergui/project/{project_id}/newcustomimage", str(
link_create_custom_image.get_attribute('href')
)
)
- self.assertTrue(
- "Create your first custom image" in str(
+ self.assertIn(
+ "Create your first custom image", str(
link_create_custom_image.text
)
)
- TestProjectConfigTab.project_id = None # reset project id
def test_project_page_image_recipe(self):
""" Test project page section images
@@ -526,3 +442,66 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
self.wait_until_visible('#imagerecipestable tbody tr')
rows = self.find_all('#imagerecipestable tbody tr')
self.assertTrue(len(rows) > 0)
+
+@pytest.mark.django_db
+@pytest.mark.order("last")
+class TestProjectConfigTabDB(TestProjectConfigTabBase):
+
+ def test_most_build_recipes(self):
+ """ Test most build recipes block contains"""
+ def rebuild_from_most_build_recipes(recipe_list_items):
+ checkbox = recipe_list_items[0].find_element(By.TAG_NAME, 'input')
+ checkbox.click()
+ build_btn = self.find('#freq-build-btn')
+ build_btn.click()
+ self.wait_until_visible('#latest-builds')
+ wait_until_build(self, 'queued cloning starting parsing failed')
+ lastest_builds = self.driver.find_elements(
+ By.XPATH,
+ '//div[@id="latest-builds"]/div'
+ )
+ self.assertTrue(len(lastest_builds) >= 2)
+ last_build = lastest_builds[0]
+ try:
+ cancel_button = last_build.find_element(
+ By.XPATH,
+ '//span[@class="cancel-build-btn pull-right alert-link"]',
+ )
+ cancel_button.click()
+ except NoSuchElementException:
+ # Skip if the build is already cancelled
+ pass
+ wait_until_build_cancelled(self)
+
+ # Create a new project for remaining asserts
+ project_id = self.create_new_project(self.PROJECT_NAME + "-MostBuilt", MASTER_PROJECT_ID, None, True)
+ url = reverse('project', args=(project_id,))
+ self.get(url)
+ self.wait_until_visible('#config-nav')
+
+ current_url = self.driver.current_url
+ url = current_url.split('?')[0]
+
+ # Create a new builds
+ self._create_builds()
+
+ # back to project page
+ self.driver.get(url)
+
+ self.wait_until_visible('#project-page')
+
+ # Most built recipes
+ most_built_recipes = self.driver.find_element(
+ By.XPATH, '//*[@id="project-page"]/div[1]/div[3]')
+ title = most_built_recipes.find_element(By.TAG_NAME, 'h3')
+ self.assertIn("Most built recipes", title.text)
+ # check can select a recipe and build it
+ self.wait_until_visible('#freq-build-list')
+ recipe_list = self.find('#freq-build-list')
+ recipe_list_items = recipe_list.find_elements(By.TAG_NAME, 'li')
+ self.assertTrue(
+ len(recipe_list_items) > 0,
+ msg="No recipes found in the most built recipes list",
+ )
+ rebuild_from_most_build_recipes(recipe_list_items)
+
diff --git a/bitbake/lib/toaster/tests/functional/utils.py b/bitbake/lib/toaster/tests/functional/utils.py
index 7269fa18..72345aef 100644
--- a/bitbake/lib/toaster/tests/functional/utils.py
+++ b/bitbake/lib/toaster/tests/functional/utils.py
@@ -8,7 +8,7 @@
from time import sleep
-from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException, TimeoutException
+from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException, TimeoutException, WebDriverException
from selenium.webdriver.common.by import By
from orm.models import Build
@@ -36,7 +36,7 @@ def wait_until_build(test_instance, state):
if 'failed' in str(build_state).lower():
break
except NoSuchElementException:
- continue
+ pass
except TimeoutException:
break
start_time += 1
@@ -48,7 +48,6 @@ def wait_until_build_cancelled(test_instance):
"""
timeout = 30
start_time = 0
- build = None
while True:
try:
if start_time > timeout:
@@ -64,19 +63,17 @@ def wait_until_build_cancelled(test_instance):
if 'failed' in str(build_state).lower():
break
if 'cancelling' in str(build_state).lower():
- # Change build state to cancelled
- if not build: # get build object only once
- build = Build.objects.last()
- build.outcome = Build.CANCELLED
- build.save()
+ pass
if 'cancelled' in str(build_state).lower():
break
- except NoSuchElementException:
- continue
- except StaleElementReferenceException:
- continue
except TimeoutException:
break
+ except NoSuchElementException:
+ pass
+ except StaleElementReferenceException:
+ pass
+ except WebDriverException:
+ pass
start_time += 1
sleep(1) # take a breath and try again
diff --git a/bitbake/lib/toaster/tests/toaster-tests-requirements.txt b/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
index 71cc0834..86ba9b42 100644
--- a/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
+++ b/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
@@ -1,7 +1,9 @@
selenium>=4.13.0
-pytest==7.4.2
+pytest>=9.0.3
pytest-django==4.5.2
pytest-env==1.1.0
pytest-html==4.0.2
pytest-metadata==3.0.0
pytest-order==1.1.0
+requests
+
diff --git a/bitbake/lib/toaster/tests/views/test_views.py b/bitbake/lib/toaster/tests/views/test_views.py
index e1adfcf8..5e64e819 100644
--- a/bitbake/lib/toaster/tests/views/test_views.py
+++ b/bitbake/lib/toaster/tests/views/test_views.py
@@ -27,7 +27,7 @@ import toastergui
from toastergui.tables import SoftwareRecipesTable
import json
-from bs4 import BeautifulSoup
+from bb._vendor.bs4 import BeautifulSoup
import string
PROJECT_NAME = "test project"
diff --git a/bitbake/lib/toaster/toastergui/templates/layerdetails.html b/bitbake/lib/toaster/toastergui/templates/layerdetails.html
index 923ca3bf..2f23156a 100644
--- a/bitbake/lib/toaster/toastergui/templates/layerdetails.html
+++ b/bitbake/lib/toaster/toastergui/templates/layerdetails.html
@@ -328,12 +328,12 @@
<span class="current-value">{{layerversion.layer.summary|default_if_none:''}}</span>
<form style="display:none; margin-bottom:20px; margin-top:5px;">
<div class="form-group">
- <textarea class="form-control" rows="2">{% if layerversion.layer.summary %}{{layerversion.layer.summary}}{% endif %}</textarea>
+ <textarea id="layer-summary" class="form-control" rows="2">{% if layerversion.layer.summary %}{{layerversion.layer.summary}}{% endif %}</textarea>
</div>
<button class="btn btn-default change-btn" data-layer-prop="summary" type="button">Save</button>
<a href="#" class="btn btn-link cancel">Cancel</a>
</form>
- <span class="glyphicon glyphicon-edit"></span>
+ <span id="edit-layer-summary" class="glyphicon glyphicon-edit"></span>
<span class="glyphicon glyphicon-trash delete-current-value" data-toggle="tooltip" title="Delete"></span>
</dd>
<dt>
@@ -344,12 +344,12 @@
<span class="current-value">{{layerversion.layer.description|default_if_none:''}}</span>
<form style="display:none; margin-bottom:20px; margin-top:5px;">
<div class="form-group">
- <textarea class="form-control" rows="6">{% if layerversion.layer.description %}{{layerversion.layer.description}}{% endif %}</textarea>
+ <textarea id="layer-description" class="form-control" rows="6">{% if layerversion.layer.description %}{{layerversion.layer.description}}{% endif %}</textarea>
</div>
<button class="btn btn-default change-btn" data-layer-prop="description" type="button" >Save</button>
<a href="#" class="btn btn-link cancel">Cancel</a>
</form>
- <span class="glyphicon glyphicon-edit"></span>
+ <span id="edit-layer-description" class="glyphicon glyphicon-edit"></span>
<span class="glyphicon glyphicon-trash delete-current-value" data-toggle="tooltip" title="Delete"></span>
</dd>
{% if layerversion.layer_source == layer_source.TYPE_LAYERINDEX %}
diff --git a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
index bd398f00..aee9bbcd 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
@@ -233,7 +233,6 @@ def filter_sizeovertotal(package_object, total_size):
return '{:.1%}'.format(float(size)/float(total_size))
-from django.utils.safestring import mark_safe
@register.filter
def format_vpackage_rowclass(size):
if size == -1:
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index 40aed265..3192b727 100644
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -8,6 +8,7 @@
import ast
import re
+import shlex
import subprocess
import sys
@@ -372,7 +373,6 @@ def _get_parameters_values(request, default_count, default_order):
# set cookies for parameters. this is usefull in case parameters are set
# manually from the GET values of the link
def _set_parameters_values(pagesize, orderby, request):
- from django.urls import resolve
current_url = resolve(request.path_info).url_name
request.session['%s_count' % current_url] = pagesize
request.session['%s_orderby' % current_url] =orderby
@@ -699,7 +699,6 @@ class LazyEncoder(json.JSONEncoder):
return super(LazyEncoder, self).default(obj)
from toastergui.templatetags.projecttags import filtered_filesizeformat
-import os
def _get_dir_entries(build_id, target_id, start):
node_str = {
Target_File.ITYPE_REGULAR : '-',
@@ -1558,7 +1557,7 @@ if True:
# perform the final actions for the project specific page
def project_specific_finalize(cmnd, pid):
project = Project.objects.get(pk=pid)
- callback = project.get_variable(Project.PROJECT_SPECIFIC_CALLBACK)
+ callback = shlex.split(project.get_variable(Project.PROJECT_SPECIFIC_CALLBACK) or "")
if "update" == cmnd:
# Delete all '_PROJECT_PREPARE_' builds
for b in Build.objects.all().filter(project=project):
@@ -1571,16 +1570,15 @@ if True:
management.call_command('builddelete', str(b.id), interactive=False)
# perform callback at this last moment if defined, in case Toaster gets shutdown next
default_target = project.get_variable(Project.PROJECT_SPECIFIC_DEFAULTIMAGE)
- if callback:
- callback = callback.replace("<IMAGE>",default_target)
+ for i in range(len(callback)):
+ callback[i] = callback[i].replace("<IMAGE>", default_target)
if "cancel" == cmnd:
- if callback:
- callback = callback.replace("<IMAGE>","none")
- callback = callback.replace("--update","--cancel")
+ for i in range(len(callback)):
+ callback[i] = callback[i].replace("<IMAGE>", "none")
+ callback[i] = callback[i].replace("--update", "--cancel")
# perform callback at this last moment if defined, in case this Toaster gets shutdown next
- ret = ''
if callback:
- ret = os.system('bash -c "%s"' % callback)
+ subprocess.run(callback, shell=False)
project.set_variable(Project.PROJECT_SPECIFIC_CALLBACK,'')
# Delete the temp project specific variables
project.set_variable(Project.PROJECT_SPECIFIC_ISNEW,'')
diff --git a/bitbake/lib/toaster/toastermain/settings.py b/bitbake/lib/toaster/toastermain/settings.py
index e06adc5a..9d2dbc46 100644
--- a/bitbake/lib/toaster/toastermain/settings.py
+++ b/bitbake/lib/toaster/toastermain/settings.py
@@ -229,8 +229,8 @@ from os.path import dirname as DN
SITE_ROOT=DN(DN(os.path.abspath(__file__)))
import subprocess
-TOASTER_BRANCH = subprocess.Popen('git branch | grep "^* " | tr -d "* "', cwd = SITE_ROOT, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
-TOASTER_REVISION = subprocess.Popen('git rev-parse HEAD ', cwd = SITE_ROOT, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()[0]
+TOASTER_BRANCH = subprocess.check_output(['git', 'branch', '--show-current'], cwd=SITE_ROOT, text=True).strip()
+TOASTER_REVISION = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=SITE_ROOT, text=True).strip()
ROOT_URLCONF = 'toastermain.urls'
@@ -298,7 +298,6 @@ SOUTH_TESTS_MIGRATE = False
# We automatically detect and install applications here if
# they have a 'models.py' or 'views.py' file
-import os
currentdir = os.path.dirname(__file__)
for t in os.walk(os.path.dirname(currentdir)):
modulename = os.path.basename(t[0])
diff --git a/bitbake/packaging-pypi/bitbake-setup/BUILD.md b/bitbake/packaging-pypi/bitbake-setup/BUILD.md
new file mode 100644
index 00000000..adb8d183
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/BUILD.md
@@ -0,0 +1,67 @@
+# Development Instructions
+
+## Requirements
+
+- Python >= 3.9
+- pip >= 19 (for installation)
+
+## Testing
+
+To lint the `bitbake-setup` pypi packaging, run the ruff tool.
+```bash
+ruff check bin/bitbake-setup packaging-pypi/bitbake-setup
+```
+
+The steps to build and test the `bitbake-setup` pypi packaging have been automated with the `bitbake-selftest` tool. This tool automatically creates a Python virtual environment for you.
+
+Run the bitbake-selftest
+```bash
+BB_SKIP_PYPI_TESTS=no bin/bitbake-selftest -v bb.tests.setup.PyPIPackagingTest
+```
+
+## Packaging
+
+### Create the development sandbox
+
+To create the development sandbox run:
+```bash
+packaging-pypi/bitbake-setup/package-bitbake-setup.py
+cd packaging-workspace
+```
+
+### Updating the vendored modules
+
+You may use the Python vendoring module to update the vendored modules. Its configuration can be found in the pyproject.toml under the tool.vendoring section.
+
+This information includes:
+- the destination directory where the vendored modules are stored (e.g. lib/bb/_vendor)
+- the module names and versions to vendor are found in the requirements file vendor.txt
+- the namespace to place the modules in (eg. bb._vendor)
+- the patches to apply to the pristine source code (eg. vendor/patches)
+
+To run the tool you may install it from pypi and run it from the top-level directory.
+
+```bash
+python3 -m pip install vendoring
+vendoring sync .
+```
+
+### Building the package
+
+To install the development tools manually run:
+```bash
+python3 -m pip install -e '.[dev]'
+```
+
+To build a wheel (.whl) then use:
+```bash
+python3 -m build
+```
+
+This produces a wheel (.whl) file in the dist directory. This may be installed using pip.
+
+### Installing the package
+
+```bash
+python3 -m pip install dist/bitbake_setup-*-py3-none-any.whl
+```
diff --git a/bitbake/packaging-pypi/bitbake-setup/LICENSE b/bitbake/packaging-pypi/bitbake-setup/LICENSE
new file mode 100644
index 00000000..4b4c7367
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/LICENSE
@@ -0,0 +1,23 @@
+bitbake-setup is licensed under the GNU General Public License version 2.0. See
+LICENSE.GPL-2.0-only for further details.
+
+Individual files contain the following style tags instead of the full license text:
+
+ SPDX-License-Identifier: GPL-2.0-only
+
+This enables machine processing of license information based on the SPDX
+License Identifiers that are here available: http://spdx.org/licenses/
+
+The bitbake-setup package vendorizes the following modules:
+
+beautifulsoup4
+ SPDX-License-Identifier: MIT
+
+typing_extensions
+ SPDX-License-Identifier: PSF-2.0
+
+progressbar:
+ SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
+
+simplediff:
+ SPDX-License-Identifier: Zlib
diff --git a/bitbake/packaging-pypi/bitbake-setup/README.md b/bitbake/packaging-pypi/bitbake-setup/README.md
new file mode 100644
index 00000000..aaa70278
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/README.md
@@ -0,0 +1,42 @@
+# bitbake-setup
+
+This package provides the `bitbake-setup` command and the Python modules
+required to support BitBake setup and configuration.
+
+## Usage
+
+Instructions on uses of bitbake-setup can be found in
+[Setting Up the Environment With bitbake-setup](https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-environment-setup.html) from the Yocto Project manual.
+
+List the available configurations;
+```bash
+bitbake-setup list
+```
+
+Show the help for the bitbake-setup init subcommand:
+```bash
+bitbake-setup init --help
+usage: bitbake-setup init [-h] [--non-interactive] [--source-overrides SOURCE_OVERRIDES] [--setup-dir-name SETUP_DIR_NAME] [--skip-selection SKIP_SELECTION] [-L SOURCE_NAME PATH]
+ [config ...]
+
+positional arguments:
+ config path/URL/id to a configuration file (use 'list' command to get available ids), followed by configuration options. Bitbake-setup will ask to choose from available
+ choices if command line doesn't completely specify them.
+
+options:
+ -h, --help show this help message and exit
+ --non-interactive Do not ask to interactively choose from available options; if bitbake-setup cannot make a decision it will stop with a failure.
+ --source-overrides SOURCE_OVERRIDES
+ Override sources information (repositories/revisions) with values from a local json file.
+ --setup-dir-name SETUP_DIR_NAME
+ A custom setup directory name under the top directory.
+ --skip-selection SKIP_SELECTION
+ Do not select and set an option/fragment from available choices; the resulting bitbake configuration may be incomplete.
+ -L SOURCE_NAME PATH, --use-local-source SOURCE_NAME PATH
+ Symlink local source into a build, instead of getting it as prescribed by a configuration (useful for local development).
+```
+
+To initialize a workspace for the Poky reference distro using the development branch (ie. "master") of OpenEmbedded:
+```bash
+bitbake-setup init poky-master
+```
\ No newline at end of file
diff --git a/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py b/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
new file mode 100755
index 00000000..f5468dc4
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import shutil
+from pathlib import Path
+
+
+def create_packaging_workspace(directory):
+ # Create the directory for packaging workspace
+ if len(directory or "") > 0:
+ workspace_dir = Path(directory)
+ else:
+ # This script is located in packaging-pypi/bitbake-setup/package-bitbake-setup.py
+ workspace_dir = Path(__file__).parents[2] / "packaging-workspace"
+
+ if not workspace_dir.exists():
+ logging.debug(f"Created packaging workspace at: {workspace_dir}")
+ workspace_dir.mkdir(exist_ok=True)
+ else:
+ logging.debug(f"Packaging workspace already exists at: {workspace_dir}")
+
+ # Copy packaging files to the workspace
+ files_to_copy = [
+ "packaging-pypi/bitbake-setup/pyproject.toml",
+ "packaging-pypi/bitbake-setup/README.md",
+ "packaging-pypi/bitbake-setup/LICENSE",
+ "LICENSE.MIT",
+ "LICENSE.GPL-2.0-only"
+ ]
+
+ for file in files_to_copy:
+ src_path = Path(__file__).parents[2] / file
+ dest_path = workspace_dir / Path(file).name
+
+ if src_path.is_dir():
+ shutil.copytree(src_path, dest_path, dirs_exist_ok=True)
+ logging.debug(f"Copied directory: {src_path} to {dest_path}")
+ else:
+ shutil.copy2(src_path, dest_path)
+ logging.debug(f"Copied file: {src_path} to {dest_path}")
+
+
+ # Copy necessary modules to the workspace
+ modules_to_bundle = [
+ "lib/bb",
+ ]
+
+ for module in modules_to_bundle:
+ src_path = Path(__file__).parents[2] / module
+ dest_path = workspace_dir / "src" / Path(module).name
+
+ if src_path.is_dir():
+ shutil.copytree(src_path, dest_path, dirs_exist_ok=True)
+ logging.debug(f"Bundled module directory: {src_path} to {dest_path}")
+ else:
+ shutil.copy2(src_path, dest_path)
+ logging.debug(f"Bundled module file: {src_path} to {dest_path}")
+
+ # Create bitbake_setup module
+ bitbake_setup_dir = Path(workspace_dir / "src" / "bitbake_setup")
+ bitbake_setup_dir.mkdir(exist_ok=True)
+ Path(bitbake_setup_dir / "__init__.py").touch()
+ shutil.copy2(Path(__file__).parents[2] / "bin" / "bitbake-setup", str(bitbake_setup_dir / "__main__.py"))
+
+
+if __name__ == "__main__":
+ logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+ parser = argparse.ArgumentParser(description='Package bitbake-setup for PyPI')
+ parser.add_argument('-v', '--verbose', action='store_true', help='increase output verbosity.')
+ parser.add_argument('-d', '--directory', type=str, help='specify the directory to create the packaging workspace in.')
+
+ args = parser.parse_args()
+
+ if args.verbose:
+ logging.getLogger().setLevel(logging.DEBUG)
+
+ create_packaging_workspace(args.directory)
diff --git a/bitbake/packaging-pypi/bitbake-setup/pyproject.toml b/bitbake/packaging-pypi/bitbake-setup/pyproject.toml
new file mode 100644
index 00000000..90ed1ffb
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/pyproject.toml
@@ -0,0 +1,92 @@
+[build-system]
+requires = [
+ "setuptools>=64",
+]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "bitbake-setup"
+dynamic = ["version"]
+description = "bitbake-setup"
+readme = "README.md"
+requires-python = ">=3.9"
+license = "GPL-2.0-only AND MIT AND PSF-2.0 AND Zlib AND LGPL-2.1-or-later OR BSD-3-Clause-Clear"
+authors = [
+ { name = "OpenEmbedded BitBake Developers", email = "bitbake-devel@lists.openembedded.org" }
+]
+classifiers = [
+ "Programming Language :: Python :: 3",
+ "Operating System :: POSIX :: Linux",
+]
+
+dependencies = [
+ # bitbake-setup is mostly self-contained
+]
+
+[project.optional-dependencies]
+test = [
+ "pytest>=7",
+]
+
+lint = [
+ "ruff>=0.3",
+ "mypy>=1.8",
+]
+
+dev = [
+ "pytest>=7",
+ "ruff>=0.3",
+ "mypy>=1.8",
+ "build",
+ "twine",
+]
+
+[project.scripts]
+bitbake-setup = "bitbake_setup.__main__:main"
+
+[project.urls]
+Homepage = "https://git.openembedded.org/bitbake/"
+Documentation = "https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-environment-setup.html"
+Repository = "https://git.openembedded.org/bitbake/"
+
+[tool.mypy]
+python_version = "3.9"
+warn_unused_configs = true
+ignore_missing_imports = true
+no_implicit_optional = true
+check_untyped_defs = false
+
+[tool.ruff]
+target-version = "py39"
+line-length = 88
+src = ["src"]
+
+[tool.ruff.lint]
+select = [
+ "E", # pycodestyle
+ "F", # pyflakes
+ "B", # flake8-bugbear
+ "I", # import sorting
+]
+ignore = [
+ "E501", # line length (handled by formatter)
+]
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+zip-safe = false
+include-package-data = true
+
+[tool.setuptools.dynamic]
+version = {attr = "bb.__version__"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = [
+ "bb*",
+ "bitbake_setup"
+]
diff --git a/bitbake/pyproject.toml b/bitbake/pyproject.toml
new file mode 100644
index 00000000..50b6436d
--- /dev/null
+++ b/bitbake/pyproject.toml
@@ -0,0 +1,16 @@
+[project]
+name = "bitbake"
+version = "2.20.0"
+requires-python = ">=3.9"
+
+[tool.vendoring]
+destination = "lib/bb/_vendor"
+requirements = "vendor.txt"
+namespace = "bb._vendor"
+patches-dir = "vendor/patches"
+
+[tool.vendoring.license.directories]
+ply = "vendor/licenses/ply/LICENSE"
+
+[tool.vendoring.license.fallback-urls]
+ply = "https://raw.githubusercontent.com/dabeaz/ply/refs/heads/master/src/ply/lex.py"
diff --git a/bitbake/setup-schema/bitbake-setup.schema.json b/bitbake/setup-schema/bitbake-setup.schema.json
new file mode 100644
index 00000000..bb10a925
--- /dev/null
+++ b/bitbake/setup-schema/bitbake-setup.schema.json
@@ -0,0 +1,211 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "description": "Schema for bitbake-setup configuration files",
+ "examples" : [
+ {
+ "description": "Example bitbake-setup configuration file",
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "rev": "master"
+ },
+ "path": "bitbake"
+ },
+ "openembedded-core": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/openembedded-core",
+ "rev": "master"
+ },
+ "path": "openembedded-core"
+ }
+ },
+ "bitbake-setup": {
+ "configurations": [
+ {
+ "name" : "qemux86-64-nodistro",
+ "description": "OpenEmbedded 'nodistro' build for qemux86-64",
+ "setup-dir-name": "oe-nodistro-master",
+ "bb-layers": ["openembedded-core/meta"],
+ "oe-fragments": ["machine/qemux86-64"]
+ }
+ ]
+ },
+ "version": "1.0"
+ }
+ ],
+ "title": "bitbake-setup configuration file",
+ "type": "object",
+ "required": [
+ "description",
+ "bitbake-setup",
+ "version"
+ ],
+ "properties": {
+ "description": {
+ "type": "string",
+ "description": "Description of the bitbake-setup configuration file"
+ },
+ "sources": {
+ "$ref": "layers.schema.json#/properties/sources"
+ },
+ "expires": {
+ "type": "string",
+ "description": "End of life date for this configuration, in ISO 8601 format (YYYY-MM-DD)"
+ },
+ "bitbake-setup": {
+ "type": "object",
+ "description": "BitBake-setup configurations",
+ "required": [
+ "configurations"
+ ],
+ "properties": {
+ "configurations": {
+ "type": "array",
+ "minItems": 1,
+ "$anchor": "configurations",
+ "items": {
+ "type": "object",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Name of the configuration"
+ },
+ "description": {
+ "type": "string",
+ "description": "Human-readable description of the configuration"
+ },
+ "notes": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ }
+ ],
+ "description": "Extra notes that populate conf-notes.txt when bb-layers is used"
+ },
+ "bb-layers": {
+ "type": "array",
+ "description": "List of BitBake layer paths to include, relative to the layers download directory",
+ "items": {
+ "type": "string"
+ }
+ },
+ "bb-layers-file-relative": {
+ "type": "array",
+ "description": "List of BitBake layers paths to include, relative to the directory with the configuration file",
+ "items": {
+ "type": "string"
+ }
+ },
+ "oe-template": {
+ "type": "string",
+ "description": "OE-template configuration"
+ },
+ "oe-fragments": {
+ "$anchor": "oe-fragments",
+ "type": "array",
+ "description": "List of BitBake configuration fragments to enable",
+ "items": {
+ "anyOf": [
+ {
+ "type": "string",
+ "description": "Configuration fragment name"
+ },
+ {
+ "type": "object",
+ "description": "Dictionary of BitBake configuration fragments to enable (with description)",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Configuration fragment name"
+ },
+ "description": {
+ "type": "string",
+ "description": "Human-readable description of the fragment"
+ }
+ }
+ }
+ ]
+ }
+ },
+ "oe-fragments-one-of": {
+ "type": "object",
+ "description": "Mutually exclusive bitbake configuration fragment",
+ "patternProperties": {
+ ".*": {
+ "type": "object",
+ "required": [
+ "description",
+ "options"
+ ],
+ "properties": {
+ "description": {
+ "type": "string",
+ "description": "Human-readable description of the fragment category"
+ },
+ "options": {
+ "$ref": "#oe-fragments"
+ }
+ },
+ "additionalProperties": false
+ }
+ },
+ "additionalProperties": false
+ },
+ "configurations": {
+ "$ref": "#configurations"
+ },
+ "bb-env-passthrough-additions": {
+ "type": "array",
+ "description": "List of environment variables to include in BB_ENV_PASSTHROUGH_ADDITIONS",
+ "items": {
+ "type": "string"
+ }
+ },
+ "setup-dir-name": {
+ "type": "string",
+ "description": "A suggestion for the setup directory name, $-prefixed keys from oe-fragments-one-of will be substituted with user selections."
+ },
+ "install-buildtools": {
+ "type": "object",
+ "description": "Settings for downloading buildtools via bb.fetch",
+ "properties": {
+ "url": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Full URL to the buildtools SDK installer."
+ },
+ "sha256sum": {
+ "type": "string",
+ "minLength": 1,
+ "description": "SHA256 checksum of the buildtools installer."
+ }
+ },
+ "required": [
+ "url",
+ "sha256sum"
+ ],
+ "additionalProperties": false
+ }
+ },
+ "additionalProperties": false
+ }
+ }
+ },
+ "additionalProperties": false
+ },
+ "version": {
+ "description": "The version of this document; currently '1.0'",
+ "enum": [
+ "1.0"
+ ]
+ }
+ },
+ "additionalProperties": false
+}
diff --git a/bitbake/setup-schema/layers.schema.json b/bitbake/setup-schema/layers.schema.json
new file mode 100644
index 00000000..3ed40d08
--- /dev/null
+++ b/bitbake/setup-schema/layers.schema.json
@@ -0,0 +1,107 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "description": "A schema for a manifest of layers to setup",
+ "title": "OpenEmbedded Layer Setup Schema",
+ "examples" : [
+ {
+ "sources": {
+ "bitbake": {
+ "git-remote": {
+ "uri": "https://git.openembedded.org/bitbake",
+ "rev": "master"
+ },
+ "path": "bitbake"
+ }
+ },
+ "version": "1.0"
+ }
+ ],
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "version"
+ ],
+ "properties": {
+ "version": {
+ "description": "The version of this document; currently '1.0'",
+ "enum": ["1.0"]
+ },
+ "sources": {
+ "description": "The dict of layer sources",
+ "type": "object",
+ "patternProperties": { ".*" : {
+ "type": "object",
+ "description": "The upstream source from which a set of layers may be fetched",
+ "additionalProperties": false,
+ "properties": {
+ "path": {
+ "description": "The path where this layer source will be placed when fetching",
+ "type": "string"
+ },
+ "contains_this_file": {
+ "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then",
+ "type": "boolean"
+ },
+ "git-remote": {
+ "description": "A remote git source from which to fetch",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "rev"
+ ],
+ "properties": {
+ "branch": {
+ "description": "The git branch to fetch (optional)",
+ "type": "string"
+ },
+ "rev": {
+ "description": "The git revision to checkout",
+ "type": "string"
+ },
+ "describe": {
+ "description": "The output of 'git describe' (human readable description of the revision using tags in revision history)",
+ "type": "string"
+ },
+ "uri": {
+ "description": "Specifies the git URI, can be used instead of the more elaborate 'remotes' property when there is only a single URI",
+ "type": "string"
+ },
+ "remotes": {
+ "description": "The dict of git remotes to add to this repository",
+ "type": "object",
+ "patternProperties": { ".*" : {
+ "description": "A git remote",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "uri"
+ ],
+ "properties": {
+ "uri": {
+ "description": "The URI for the remote, using git URL syntax",
+ "type": "string"
+ }
+ }
+ }}
+ }
+ }
+ },
+ "local": {
+ "description": "A local directory that should be made available for builds via symlinking",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "path"
+ ],
+ "properties": {
+ "path": {
+ "description": "The path to the directory",
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ }}
+ }
+}
diff --git a/bitbake/toaster-requirements.txt b/bitbake/toaster-requirements.txt
index d8e48b7f..6941719f 100644
--- a/bitbake/toaster-requirements.txt
+++ b/bitbake/toaster-requirements.txt
@@ -1,4 +1,4 @@
-Django>4.2,<4.3
+Django>5.2,<5.3
beautifulsoup4>=4.4.0
pytz
django-log-viewer==1.1.7
diff --git a/bitbake/vendor.txt b/bitbake/vendor.txt
new file mode 100644
index 00000000..8d5dcad7
--- /dev/null
+++ b/bitbake/vendor.txt
@@ -0,0 +1,6 @@
+beautifulsoup4==4.15.0
+ply==3.10
+progressbar==2.5
+simplediff==1.1
+typing_extensions==4.15.0
+tomli==2.4.1
diff --git a/bitbake/vendor/licenses/ply/LICENSE b/bitbake/vendor/licenses/ply/LICENSE
new file mode 100644
index 00000000..d050b2ef
--- /dev/null
+++ b/bitbake/vendor/licenses/ply/LICENSE
@@ -0,0 +1,32 @@
+# -----------------------------------------------------------------------------
+# ply
+#
+# Copyright (C) 2001-2018
+# David M. Beazley (Dabeaz LLC)
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+# * Neither the name of the David Beazley or Dabeaz LLC may be used to
+# endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -----------------------------------------------------------------------------
diff --git a/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch b/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch
new file mode 100644
index 00000000..e352473e
--- /dev/null
+++ b/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch
@@ -0,0 +1,40 @@
+From 8e444cd9913d1ee0672b5583e263e5927c3221df Mon Sep 17 00:00:00 2001
+From: Richard Purdie <richard.purdie@linuxfoundation.org>
+Date: Fri, 31 May 2024 13:09:44 +0100
+Subject: [PATCH 10/11] lib/bs4: Avoid soupsieve warning
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+---
+ lib/bs4/css.py | 10 ++--------
+ 1 file changed, 2 insertions(+), 8 deletions(-)
+
+diff --git a/lib/bb/_vendor/bs4/css.py b/lib/bb/_vendor/bs4/css.py
+index 245ac6010..cd1fd2df8 100644
+--- a/lib/bb/_vendor/bs4/css.py
++++ b/lib/bb/_vendor/bs4/css.py
+@@ -24,23 +24,14 @@
+ Optional,
+ TYPE_CHECKING,
+ )
+-import warnings
+ from bs4._typing import _NamespaceMapping
+
+ if TYPE_CHECKING:
+- from soupsieve import SoupSieve
+ from bs4 import element
+ from bs4.element import ResultSet, Tag
+
+-soupsieve: Optional[ModuleType]
+-try:
+- import soupsieve
+-except ImportError:
+- soupsieve = None
+- warnings.warn(
+- "The soupsieve package is not installed. CSS selectors cannot be used."
+- )
+-
++# We don't use soupsieve
++soupsieve = None
+
+ class CSS(object):
+ """A proxy object against the ``soupsieve`` library, to simplify its
diff --git a/bitbake/vendor/patches/bs4-remove-double-imports.patch b/bitbake/vendor/patches/bs4-remove-double-imports.patch
new file mode 100644
index 00000000..2bbfa937
--- /dev/null
+++ b/bitbake/vendor/patches/bs4-remove-double-imports.patch
@@ -0,0 +1,23 @@
+commit 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d
+Author: Michael Estner <michaelestner@web.de>
+Date: Tue Feb 25 19:34:25 2025 +0100
+
+ lib: Remove double imports
+
+ * Remove double imports mentioned by pylint
+
+ Signed-off-by: Michael Estner <michaelestner@web.de>
+ Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+diff --git a/lib/bb/_vendor/bs4/__init__.py b/lib/bb/_vendor/bs4/__init__.py
+index d8ad5e1dc..725203d94 100644
+--- a/lib/bb/_vendor/bs4/__init__.py
++++ b/lib/bb/_vendor/bs4/__init__.py
+@@ -1173,7 +1173,5 @@
+
+ # If this file is run as a script, act as an HTML pretty-printer.
+ if __name__ == "__main__":
+- import sys
+-
+ soup = BeautifulSoup(sys.stdin)
+ print((soup.prettify()))
diff --git a/bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch b/bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch
new file mode 100644
index 00000000..97dd5ca9
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-progress-support.patch
@@ -0,0 +1,125 @@
+From 0d275fc5b6531957a6189069b04074065bb718a0 Mon Sep 17 00:00:00 2001
+From: Paul Eggleton <paul.eggleton@linux.intel.com>
+Date: Thu, 23 Jun 2016 22:59:05 +1200
+Subject: [PATCH 1/6] lib: implement basic task progress support
+
+For long-running tasks where we have some output from the task that
+gives us some idea of the progress of the task (such as a percentage
+complete), provide the means to scrape the output for that progress
+information and show it to the user in the default knotty terminal
+output in the form of a progress bar. This is implemented using a new
+TaskProgress event as well as some code we can insert to do output
+scanning/filtering.
+
+Any task can fire TaskProgress events; however, if you have a shell task
+whose output you wish to scan for progress information, you just need to
+set the "progress" varflag on the task. This can be set to:
+ * "percent" to just look for a number followed by a % sign
+ * "percent:<regex>" to specify your own regex matching a percentage
+ value (must have a single group which matches the percentage number)
+ * "outof:<regex>" to look for the specified regex matching x out of y
+ items completed (must have two groups - first group needs to be x,
+ second y).
+We can potentially extend this in future but this should be a good
+start.
+
+Part of the implementation for [YOCTO #5383].
+
+Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+---
+ lib/progressbar/progressbar.py | 16 +++++++++++----
+ lib/progressbar/widgets.py | 36 ++++++++++++++++++++++++++++++++++
+ 2 files changed, 48 insertions(+), 4 deletions(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index 0b9dcf763..2873ad6ca 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -3,6 +3,8 @@
+ # progressbar - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# (With some small changes after importing into BitBake)
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+@@ -261,12 +263,14 @@ class ProgressBar(object):
+ now = time.time()
+ self.seconds_elapsed = now - self.start_time
+ self.next_update = self.currval + self.update_interval
+- self.fd.write(self._format_line() + '\r')
++ output = self._format_line()
++ self.fd.write(output + '\r')
+ self.fd.flush()
+ self.last_update_time = now
++ return output
+
+
+- def start(self):
++ def start(self, update=True):
+ """Starts measuring time, and prints the bar at 0%.
+
+ It returns self so you can use it like this:
+@@ -289,8 +293,12 @@ class ProgressBar(object):
+ self.update_interval = self.maxval / self.num_intervals
+
+
+- self.start_time = self.last_update_time = time.time()
+- self.update(0)
++ self.start_time = time.time()
++ if update:
++ self.last_update_time = self.start_time
++ self.update(0)
++ else:
++ self.last_update_time = 0
+
+ return self
+
+diff --git a/lib/bb/_vendor/progressbar/widgets.py b/lib/bb/_vendor/progressbar/widgets.py
+index 6434ad559..77285ca7a 100644
+--- a/lib/bb/_vendor/progressbar/widgets.py
++++ b/lib/bb/_vendor/progressbar/widgets.py
+@@ -353,3 +353,39 @@ class BouncingBar(Bar):
+ if not self.fill_left: rpad, lpad = lpad, rpad
+
+ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
++
++
++class BouncingSlider(Bar):
++ """
++ A slider that bounces back and forth in response to update() calls
++ without reference to the actual value. Based on a combination of
++ BouncingBar from a newer version of this module and RotatingMarker.
++ """
++ def __init__(self, marker='<=>'):
++ self.curmark = -1
++ self.forward = True
++ Bar.__init__(self, marker=marker)
++ def update(self, pbar, width):
++ left, marker, right = (format_updatable(i, pbar) for i in
++ (self.left, self.marker, self.right))
++
++ width -= len(left) + len(right)
++ if width < 0:
++ return ''
++
++ if pbar.finished: return '%s%s%s' % (left, width * '=', right)
++
++ self.curmark = self.curmark + 1
++ position = int(self.curmark % (width * 2 - 1))
++ if position + len(marker) > width:
++ self.forward = not self.forward
++ self.curmark = 1
++ position = 1
++ lpad = ' ' * (position - 1)
++ rpad = ' ' * (width - len(marker) - len(lpad))
++
++ if not self.forward:
++ temp = lpad
++ lpad = rpad
++ rpad = temp
++ return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch b/bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch
new file mode 100644
index 00000000..fd8ccaa3
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch
@@ -0,0 +1,79 @@
+From ff237c33337f4da2ca06c3a2c49699bc26608a6b Mon Sep 17 00:00:00 2001
+From: Richard Purdie <richard.purdie@linuxfoundation.org>
+Date: Tue, 30 Apr 2019 11:05:26 +0100
+Subject: [PATCH 2/6] bitbake: Add initial pass of SPDX license headers to
+ source code
+
+This adds the SPDX-License-Identifier license headers to the majority of
+our source files to make it clearer exactly which license files are under.
+
+The bulk of the files are under GPL v2.0 with one found to be under V2.0
+or later, some under MIT and some have dual license. There are some files
+which are potentially harder to classify where we've imported upstream code
+and those can be handled specifically in later commits.
+
+The COPYING file is replaced with LICENSE.X files which contain the full
+license texts.
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+---
+ lib/progressbar/__init__.py | 2 ++
+ lib/progressbar/compat.py | 2 ++
+ lib/progressbar/progressbar.py | 2 ++
+ lib/progressbar/widgets.py | 2 ++
+ 4 files changed, 8 insertions(+)
+
+diff --git a/lib/bb/_vendor/progressbar/__init__.py b/lib/bb/_vendor/progressbar/__init__.py
+index fbab744ee..c545a6275 100644
+--- a/lib/bb/_vendor/progressbar/__init__.py
++++ b/lib/bb/_vendor/progressbar/__init__.py
+@@ -4,6 +4,8 @@
+ # progressbar - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/compat.py b/lib/bb/_vendor/progressbar/compat.py
+index a39f4a1f4..9804e0b51 100644
+--- a/lib/bb/_vendor/progressbar/compat.py
++++ b/lib/bb/_vendor/progressbar/compat.py
+@@ -3,6 +3,8 @@
+ # progressbar - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index 2873ad6ca..e2b6ba108 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -5,6 +5,8 @@
+ #
+ # (With some small changes after importing into BitBake)
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/widgets.py b/lib/bb/_vendor/progressbar/widgets.py
+index 77285ca7a..0772aa536 100644
+--- a/lib/bb/_vendor/progressbar/widgets.py
++++ b/lib/bb/_vendor/progressbar/widgets.py
+@@ -3,6 +3,8 @@
+ # progressbar - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch b/bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch
new file mode 100644
index 00000000..aca9e0ed
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-value-over-initial-maxval.patch
@@ -0,0 +1,54 @@
+From 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4 Mon Sep 17 00:00:00 2001
+From: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
+Date: Thu, 22 Feb 2024 16:21:53 +0100
+Subject: [PATCH 3/6] bitbake: progressbar: accept value over initial maxval
+
+There is a very rare case where the maxval is improperly computed
+initially for cache loading progress, and the value will go over.
+
+Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:progress()
+ # we might have calculated incorrect total size because a file
+ # might've been written out just after we checked its size
+
+In that case, progressbar will receive a value over the initial maxval.
+This results in a ValueError stack trace as well as bitbake returning 1.
+ Traceback (most recent call last):
+ File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main
+ cacheprogress.update(event.current)
+ File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in update
+ raise ValueError('Value out of range')
+ ValueError: Value out of range
+
+This fix mirrors the behavior of MulticonfigCache and accepts the new
+value as the new maxval. This is also what the percentage printout
+is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:percentage()
+
+I encountered this issue randomly while working on a project with
+VSCode saving files while commands where fired.
+
+Note: This file is a fork from python-progressbar. It hasn't been
+refreshed in 8 years. We did only two commits, 5 years ago with minor
+modifications. This new change is also not how the upstream project is
+behaving.
+
+Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+---
+ lib/progressbar/progressbar.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index e2b6ba108..d4da10ab7 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -253,7 +253,7 @@ class ProgressBar(object):
+ if (self.maxval is not widgets.UnknownLength
+ and not 0 <= value <= self.maxval):
+
+- raise ValueError('Value out of range')
++ self.maxval = value
+
+ self.currval = value
+
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch b/bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch
new file mode 100644
index 00000000..1bf0718f
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch
@@ -0,0 +1,53 @@
+From f8c76eb89d52b1c28407f0b52dfe4318faa47cd2 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Benjamin=20Sz=C5=91ke?= <egyszeregy@freemail.hu>
+Date: Sat, 25 Jan 2025 13:27:49 +0100
+Subject: [PATCH 4/6] progressbar: Add self._fd_console to use for
+ self._handle_resize()
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Introduce self._fd_console as a dedicated attribute of self._handle_resize().
+
+Signed-off-by: Benjamin Szőke <egyszeregy@freemail.hu>
+Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
+---
+ lib/progressbar/progressbar.py | 11 +++++++++--
+ 1 file changed, 9 insertions(+), 2 deletions(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index d4da10ab7..eccc45849 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -110,12 +110,20 @@ class ProgressBar(object):
+ self.widgets = widgets
+ self.fd = fd if fd is not None else sys.stderr
+ self.left_justify = left_justify
++ self._fd_console = None
+
+ self.signal_set = False
+ if term_width is not None:
+ self.term_width = term_width
+ else:
+ try:
++ # Check if given file descriptor is resizable for example belong
++ # to a terminal/console as STDOUT or STDERR. If file descriptor
++ # is resizable, let's allow to use for self._handle_resize()
++ # in a dedicated self._fd_console in order to be able to set
++ # temporarily/permanently self.fd to any StringIO or other
++ # file descriptor later.
++ self._fd_console = fd
+ self._handle_resize()
+ signal.signal(signal.SIGWINCH, self._handle_resize)
+ self.signal_set = True
+@@ -182,7 +189,7 @@ class ProgressBar(object):
+ def _handle_resize(self, signum=None, frame=None):
+ """Tries to catch resize signals sent from the terminal."""
+
+- h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
++ h, w = array('h', ioctl(self._fd_console, termios.TIOCGWINSZ, '\0' * 8))[:2]
+ self.term_width = w
+
+
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch b/bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch
new file mode 100644
index 00000000..c8592355
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-correctly-with-maxval-0.patch
@@ -0,0 +1,31 @@
+From 736b5b33a5d656a6d5cb6a3180ab529832f77b16 Mon Sep 17 00:00:00 2001
+From: Peter Kjellerstedt <pkj@axis.com>
+Date: Sat, 11 Oct 2025 06:23:10 +0200
+Subject: [PATCH 5/6] progressbar: Make bars show correctly with maxval == 0
+
+While it can be argued how useful a progressbar created with 0 as
+maximum value is, it should still show two states, started (empty) and
+finished (full). Setting the maxval to _DEFAULT_MAXVAL instead will
+accomplish this.
+
+Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
+Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
+---
+ lib/progressbar/progressbar.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index eccc45849..a8e2dc09c 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -106,7 +106,7 @@ class ProgressBar(object):
+ if widgets is None:
+ widgets = list(self._DEFAULT_WIDGETS)
+
+- self.maxval = maxval
++ self.maxval = maxval if maxval != 0 else self._DEFAULT_MAXVAL
+ self.widgets = widgets
+ self.fd = fd if fd is not None else sys.stderr
+ self.left_justify = left_justify
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch b/bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch
new file mode 100644
index 00000000..96868c76
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch
@@ -0,0 +1,31 @@
+From e827db50d8f61f0d04c9bd0753c05aeb1ed0715f Mon Sep 17 00:00:00 2001
+From: Richard Purdie <richard.purdie@linuxfoundation.org>
+Date: Tue, 24 Mar 2026 21:37:25 +0000
+Subject: [PATCH 6/6] progressbar/knotty: Allow mixing log messages and
+ progress bars
+
+If we try and print a log message in the middle of progress bar, the display
+can be corrupted. Add a clear() method to progress bars allowing them to be
+removed for the log message, then reprinted using update().
+
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+---
+ lib/progressbar/progressbar.py | 3 +++
+ 1 file changed, 3 insertions(+)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/progressbar/progressbar.py
+index a8e2dc09c..1562774ba 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -278,6 +278,9 @@ class ProgressBar(object):
+ self.last_update_time = now
+ return output
+
++ def clear(self):
++ self.fd.write(" " * self.term_width + '\r')
++ self.fd.flush()
+
+ def start(self, update=True):
+ """Starts measuring time, and prints the bar at 0%.
+--
+2.49.0
--
2.56.0
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20261109091802.202521-9-clara.kowalsky%40siemens.com.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-11-09 9:19 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-09 9:17 [PATCH 0/8] Update BitBake in Isar to v2.20.0 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 1/8] base: replace BitBake's outdated persist_data 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 2/8] isar-events: preserve cleanup for default multiconfig 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 3/8] isar-events: close /proc/mounts after cleanup 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 4/8] meta: migrate fetcher references to bb.fetch 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 5/8] wic: run the Python driver outside the imaging chroot 'Clara Kowalsky' via isar-users
2026-11-09 9:17 ` [PATCH 6/8] bitbake.conf: explicitly exclude export and func from signatures 'Clara Kowalsky' via isar-users
2026-11-09 9:18 ` [PATCH 7/8] meta: add missing whitespace around assignment operators 'Clara Kowalsky' via isar-users
2026-11-09 9:18 ` [PATCH 8/8] bitbake: update to v2.20.0 'Clara Kowalsky' via isar-users
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox